To identify the different machine learning (ML) techniques that have been applied to automate physician competence assessment and evaluate how these techniques can be used to assess different ...competence domains in several medical specialties.
In May 2017, MEDLINE, EMBASE, PsycINFO, Web of Science, ACM Digital Library, IEEE Xplore Digital Library, PROSPERO, and Cochrane Database of Systematic Reviews were searched for articles published from inception to April 30, 2017. Studies were included if they applied at least one ML technique to assess medical students', residents', fellows', or attending physicians' competence. Information on sample size, participants, study setting and design, medical specialty, ML techniques, competence domains, outcomes, and methodological quality was extracted. MERSQI was used to evaluate quality, and a qualitative narrative synthesis of the medical specialties, ML techniques, and competence domains was conducted.
Of 4,953 initial articles, 69 met inclusion criteria. General surgery (24; 34.8%) and radiology (15; 21.7%) were the most studied specialties; natural language processing (24; 34.8%), support vector machine (15; 21.7%), and hidden Markov models (14; 20.3%) were the ML techniques most often applied; and patient care (63; 91.3%) and medical knowledge (45; 65.2%) were the most assessed competence domains.
A growing number of studies have attempted to apply ML techniques to physician competence assessment. Although many studies have investigated the feasibility of certain techniques, more validation research is needed. The use of ML techniques may have the potential to integrate and analyze pragmatic information that could be used in real-time assessments and interventions.
Surgical expertise demands technical and nontechnical skills. Traditionally, surgical trainees acquired these skills in the operating room; however, operative time for residents has decreased with ...duty hour restrictions. As in other professions, video analysis may help maximize the learning experience.
To develop and evaluate a postoperative video-based coaching intervention for residents.
In this mixed methods analysis, 10 senior (postgraduate year 4 and 5) residents were videorecorded operating with an attending surgeon at an academic tertiary care hospital. Each video formed the basis of a 1-hour one-on-one coaching session conducted by the operative attending; although a coaching framework was provided, participants determined the specific content collaboratively. Teaching points were identified in the operating room and the video-based coaching sessions; iterative inductive coding, followed by thematic analysis, was performed.
Teaching points made in the operating room were compared with those in the video-based coaching sessions with respect to initiator, content, and teaching technique, adjusting for time.
Among 10 cases, surgeons made more teaching points per unit time (63.0 vs 102.7 per hour) while coaching. Teaching in the video-based coaching sessions was more resident centered; attendings were more inquisitive about residents' learning needs (3.30 vs 0.28, P = .04), and residents took more initiative to direct their education (27% 198 of 729 teaching points vs 17% 331 of 1977 teaching points, P < .001). Surgeons also more frequently validated residents' experiences (8.40 vs 1.81, P < .01), and they tended to ask more questions to promote critical thinking (9.30 vs 3.32, P = .07) and set more learning goals (2.90 vs 0.28, P = .11). More complex topics, including intraoperative decision making (mean, 9.70 vs 2.77 instances per hour, P = .03) and failure to progress (mean, 1.20 vs 0.13 instances per hour, P = .04) were addressed, and they were more thoroughly developed and explored. Excerpts of dialogue are presented to illustrate these findings.
Video-based coaching is a novel and feasible modality for supplementing intraoperative learning. Objective evaluation demonstrates that video-based coaching may be particularly useful for teaching higher-level concepts, such as decision making, and for individualizing instruction and feedback to each resident.
Commentary: Another Dutch treat Zenati, Marco A.; Yule, Steven J.
Journal of thoracic and cardiovascular surgery/The Journal of thoracic and cardiovascular surgery/The journal of thoracic and cardiovascular surgery,
05/2020, Letnik:
159, Številka:
5
Journal Article
This paper addresses the issue of occupational safety and how the process of globalisation can potentially influence the attitudes, beliefs and behaviour of disparate ‘national’ workforces working ...across the globe for the same multi-national company. The paper reviews published literature on cross-cultural differences in attitudes, perceptions and beliefs regarding safety and presents details of a study examining the relationship between Hofstede’s Hofstede, G., 1984. Culture’s Consequences; International Differences in Work-Related Values, Abridged edition. Sage Publications, London, Hofstede, G., 1991. Culture and Organisations; Software of the Mind. McGraw Hill, Maidenhead cultural values dimensions (i.e., Power Distance, Individualism/Collectivism, Masculinity/Femininity); safety climate (perceived management commitment to safety) and risk-taking behaviour in workforce members of a multi-national engineering organisation operating in six countries. The results suggest that more proximal influences such as perceived management commitment to safety and the efficacy of safety measures exert more impact on workforce behaviour and subsequent accident rates than fundamental national values.
To evaluate the current evidence for surgical sabermetrics: digital methods of assessing surgical nontechnical skills and investigate the implications for enhancing surgical performance.
Surgeons ...need high-quality, objective, and timely feedback to optimize performance and patient safety. Digital tools to assess nontechnical skills have the potential to reduce human bias and aid scalability. However, we do not fully understand which of the myriad of digital metrics of performance assessment have efficacy for surgeons.
A systematic review was conducted by searching PubMed, EMBASE, CINAHL, and PSYCINFO databases following PRISMA-ScR guidelines. MeSH terms and keywords included "Assessment," "Surgeons," and "Technology". Eligible studies included a digital assessment of nontechnical skills for surgeons, residents, and/or medical students within an operative context.
From 19,229 articles screened, 81 articles met the inclusion criteria. The studies varied in surgical specialties, settings, and outcome measurements. A total of 122 distinct objective, digital metrics were utilized. Studies digitally measured at least 1 category of surgical nontechnical skill using a single (n=54) or multiple objective measures (n=27). The majority of studies utilized simulation (n=48) over live operative settings (n=32). Surgical Sabermetrics has been demonstrated to be beneficial in measuring cognitive load (n=57), situation awareness (n=24), communication (n=3), teamwork (n=13), and leadership (n=2). No studies measured intraoperative decision-making.
The literature detailing the intersection between surgical data science and operative nontechnical skills is diverse and growing rapidly. Surgical Sabermetrics may provide a promising modifiable technique to achieve desirable outcomes for both the surgeon and the patient. This study identifies a diverse array of measurements possible with sensor devices and highlights research gaps, including the need for objective assessment of decision-making. Future studies may advance the integration of physiological sensors to provide a holistic assessment of surgical performance.
Display omitted
•The operating room (OR) is a high-risk and complex environment.•Cognitive overload imposed by surgical tasks may impair team performance.•This study describes a novel approach to ...measure team cognitive load.•Physiological synchronization and entropy can be used to measure team cognitive load.
The operating room (OR) is a high-risk and complex environment, where multiple specialized professionals work as a team to effectively care for patients in need of surgical interventions. Surgical tasks impose high cognitive demands on OR staff and cognitive overload may have deleterious effects on team performance and patient safety. The aim of the present study was to investigate the feasibility and describe a novel methodological approach to characterize dynamic changes in team cognitive load by measuring synchronization and entropy of heart rate variability parameters during real-life cardiac surgery. Cognitive load was measured by capturing interbeat intervals (IBI) from three team members (surgeon, anesthesiologist and perfusionist) using an unobtrusive wearable heart rate sensor and transmitted in real-time to a smartphone application. Clinical data and operating room audio/video recordings were also collected to provide behavioral and contextual information. We developed symbolic representations of the transient cognitive state of individual team members (Individual Cognitive State – ICS), and overall team (Team Cognitive State – TCS) by comparing IBI data from each team member with themselves and with others. The distribution of TCS symbols during surgery enabled us to display and analyze temporal states and dynamic changes of team cognitive load. Shannon’s entropy was calculated to estimate the changing levels of team organization and to detect fluctuations resulting from a variety of cognitive demands and/or specific situations (e.g. medical error, emergency, flow disruptions). An illustrative example from a real cardiac surgery team shows how cognitive load patterns shifted rapidly after an actual near-miss medication event, leading the team to a more organized and synchronized state. The methodological approach described in this study provides a measurement technique for the assessment of team physiological synchronization, which can be applied to many other team-based environments. Future research should gather additional validity evidence to support the proposed methods for team cognitive load measurement.
Mini-abstract: Surgical sabermetrics is advanced analytics of digitally recorded surgical training and operative procedures to enhance insight, support professional development, and optimize clinical ...and safety outcomes. This perspectives article illustrates how surgery can leverage data science approaches in athletics and industry to transform individual and team performance in the operating room.
Background The importance of leadership is recognized in surgery, but the specific impact of leadership style on team behavior is not well understood. In other industries, leadership is a ...well-characterized construct. One dominant theory proposes that transactional (task-focused) leaders achieve minimum standards and transformational (team-oriented) leaders inspire performance beyond expectations. Study Design We videorecorded 5 surgeons performing complex operations. Each surgeon was scored on the Multifactor Leadership Questionnaire, a validated method for scoring transformational and transactional leadership style, by an organizational psychologist and a surgeon researcher. Independent coders assessed surgeons' leadership behaviors according to the Surgical Leadership Inventory and team behaviors (information sharing, cooperative, and voice behaviors). All coders were blinded. Leadership style (Multifactor Leadership Questionnaire) was correlated with surgeon behavior (Surgical Leadership Inventory) and team behavior using Poisson regression, controlling for time and the total number of behaviors, respectively. Results All surgeons scored similarly on transactional leadership (range 2.38 to 2.69), but varied more widely on transformational leadership (range 1.98 to 3.60). Each 1-point increase in transformational score corresponded to 3 times more information-sharing behaviors (p < 0.0001) and 5.4 times more voice behaviors (p = 0.0005) among the team. With each 1-point increase in transformational score, leaders displayed 10 times more supportive behaviors (p < 0.0001) and displayed poor behaviors 12.5 times less frequently (p < 0.0001). Excerpts of representative dialogue are included for illustration. Conclusions We provide a framework for evaluating surgeons' leadership and its impact on team performance in the operating room. As in other fields, our data suggest that transformational leadership is associated with improved team behavior. Surgeon leadership development, therefore, has the potential to improve the efficiency and safety of operative care.
Background
Previous research has shown that surgeons’ intraoperative non-technical skills are related to surgical outcomes. The aim of this study was to evaluate the reliability of the NOTSS ...(Non-technical Skills for Surgeons) behavior rating system. Based on task analysis, the system incorporates five categories of skills for safe surgical practice (Situation Awareness, Decision Making, Task Management, Communication & Teamwork, and Leadership).
Methods
Consultant (attending) surgeons (
n
= 44) from five Scottish hospitals attended one of six experimental sessions and were trained to use the NOTSS system. They then used the system to rate consultant surgeons’ behaviors in six simulated operating room scenarios that were presented using video. Surgeons’ ratings of the behaviors demonstrated in each scenario were compared to expert ratings (“accuracy”), and assessed for inter-rater reliability and internal consistency.
Results
The NOTSS system had a consistent internal structure. Although raters had minimal training, rating “accuracy” for acceptable/unacceptable behavior was above 60% for all categories, with mean of 0.67 scale points difference from reference (expert) ratings (on 4-point scale). For inter-rater reliability, the mean values of within-group agreement (
r
wg
) were acceptable for the categories Communication & Teamwork (.70), and Leadership (.72), but below a priori criteria for other categories. Intra-class correlation coefficients (ICC) indicated high agreement using average measures (values were .95–.99).
Conclusions
With the requisite training, the prototype NOTSS system could be used reliably by surgeons to observe and rate surgeons’ behaviors. The instrument should now be tested for usability in the operating room.
PDF
