Surgical workflow recognition has numerous potential medical applications, such as the automatic indexing of surgical video databases and the optimization of real-time operating room scheduling, ...among others. As a result, surgical phase recognition has been studied in the context of several kinds of surgeries, such as cataract, neurological, and laparoscopic surgeries. In the literature, two types of features are typically used to perform this task: visual features and tool usage signals. However, the used visual features are mostly handcrafted. Furthermore, the tool usage signals are usually collected via a manual annotation process or by using additional equipment. In this paper, we propose a novel method for phase recognition that uses a convolutional neural network (CNN) to automatically learn features from cholecystectomy videos and that relies uniquely on visual information. In previous studies, it has been shown that the tool usage signals can provide valuable information in performing the phase recognition task. Thus, we present a novel CNN architecture, called EndoNet, that is designed to carry out the phase recognition and tool presence detection tasks in a multi-task manner. To the best of our knowledge, this is the first work proposing to use a CNN for multiple recognition tasks on laparoscopic videos. Experimental comparisons to other methods show that EndoNet yields state-of-the-art results for both tasks.
Accurate surgery duration estimation is necessary for optimal OR planning, which plays an important role in patient comfort and safety as well as resource optimization. It is, however, challenging to ...preoperatively predict surgery duration since it varies significantly depending on the patient condition, surgeon skills, and intraoperative situation. In this paper, we propose a deep learning pipeline, referred to as RSDNet, which automatically estimates the remaining surgery duration (RSD) intraoperatively by using only visual information from laparoscopic videos. The previous state-of-the-art approaches for RSD prediction are dependent on manual annotation, whose generation requires expensive expert knowledge and is time-consuming, especially considering the numerous types of surgeries performed in a hospital and the large number of laparoscopic videos available. A crucial feature of RSDNet is that it does not depend on any manual annotation during training, making it easily scalable to many kinds of surgeries. The generalizability of our approach is demonstrated by testing the pipeline on two large datasets containing different types of surgeries: 120 cholecystectomy and 170 gastric bypass videos. The experimental results also show that the proposed network significantly outperforms a traditional method of estimating RSD without utilizing manual annotation. Further, this paper provides a deeper insight into the deep learning network through visualization and interpretation of the features that are automatically learned.
Purpose
Real-time surgical tool tracking is a core component of the future intelligent operating room (OR), because it is highly instrumental to analyze and understand the surgical activities. ...Current methods for surgical tool tracking in videos need to be trained on data in which the spatial positions of the tools are manually annotated. Generating such training data is difficult and time-consuming. Instead, we propose to use solely binary presence annotations to train a tool tracker for laparoscopic videos.
Methods
The proposed approach is composed of a CNN + Convolutional LSTM (
ConvLSTM
) neural network trained end to end, but weakly supervised on tool binary presence labels only. We use the ConvLSTM to model the temporal dependencies in the motion of the surgical tools and leverage its spatiotemporal ability to smooth the class peak activations in the localization heat maps (
Lh-maps
).
Results
We build a baseline tracker on top of the CNN model and demonstrate that our approach based on the ConvLSTM outperforms the baseline in tool presence detection, spatial localization, and motion tracking by over
5.0
%
,
13.9
%
, and
12.6
%
, respectively.
Conclusions
In this paper, we demonstrate that binary presence labels are sufficient for training a deep learning tracking model using our proposed method. We also show that the ConvLSTM can leverage the spatiotemporal coherence of consecutive image frames across a surgical video to improve tool presence detection, spatial localization, and motion tracking.
This work presents a novel approach for the early recognition of the type of a laparoscopic surgery from its video. Early recognition algorithms can be beneficial to the development of "smart" OR ...systems that can provide automatic context-aware assistance, and also enable quick database indexing. The task is however ridden with challenges specific to videos belonging to the domain of laparoscopy, such as high visual similarity across surgeries and large variations in video durations. To capture the spatio-temporal dependencies in these videos, we choose as our model a combination of a convolutional neural network (CNN) and long short-term memory (LSTM) network. We then propose two complementary approaches for improving early recognition performance. The first approach is a CNN fine-tuning method that encourages surgeries to be distinguished based on the initial frames of laparoscopic videos. The second approach, referred to as " Future-State Predicting LSTM," trains an LSTM to predict information related to future frames, which helps in distinguishing between the different types of surgeries. We evaluate our approaches on a large dataset of 425 laparoscopic videos containing nine types of surgeries ( Laparo425 ), and achieve on average an accuracy of 75% having observed only the first 10 min of a surgery. These results are quite promising from a practical standpoint and also encouraging for other types of image-guided surgeries.
Background
Clinical evaluation of the demarcation line separating ischemic from non-ischemic liver parenchyma may be challenging. Hyperspectral imaging (HSI) is a noninvasive imaging modality, which ...combines a camera with a spectroscope and allows quantitative imaging of tissue oxygenation. Our group developed a software to overlay HSI images onto the operative field, obtaining HSI-based enhanced reality (HYPER). The aim of the present study was to evaluate the accuracy of HYPER to identify the demarcation line after a left vascular inflow occlusion during an anatomical left hepatectomy.
Materials and methods
In the porcine model (
n
= 3), the left branches of the hepatic pedicle were ligated. Before and after vascular occlusion, HSI images based on tissue oxygenation (StO
2
), obtained through the Near-Infrared index (NIR index), were regularly acquired and superimposed onto RGB video. The demarcation line was marked on the liver surface with electrocautery according to HYPER. Local lactates were measured on blood samples from the liver surface in both ischemic and perfused segments using a strip-based device. At the same areas, confocal endomicroscopy was performed.
Results
After ligation, HSI demonstrated a significantly lower oxygenation (NIR index) in the left medial lobe (LML) (0.27% ± 0.21) when compared to the right medial lobe (RML) (58.60% ± 12.08;
p
= 0.0015). Capillary lactates were significantly higher (3.07 mmol/L ± 0.84 vs. 1.33 ± 0.71 mmol/L;
p
= 0.0356) in the LML versus RML, respectively. Concordantly, confocal videos demonstrated the absence of blood flow in the LML and normal perfusion in the RML.
Conclusions
HYPER has made it possible to correctly identify the demarcation line and quantify surface liver oxygenation. HYPER could be an intraoperative tool to guide perfusion-based demarcation line assessment and segmentation.
Background
Fluorescence-based enhanced reality (FLER) is a computer-based quantification method of fluorescence angiographies to evaluate bowel perfusion. The aim of this prospective trial was to ...assess the clinical feasibility and to correlate FLER with metabolic markers of perfusion, during colorectal resections.
Methods
FLER analysis and visualization was performed in 22 patients (diverticulitis
n
= 17; colorectal cancer
n
= 5) intra- and extra-abdominally during distal and proximal resection, respectively. The fluorescence signal of indocyanine green (0.2 mg/kg) was captured using a near-infrared camera and computed to create a virtual color-coded cartography. This was overlaid onto the bowel (enhanced reality). It helped to identify regions of interest (ROIs) where samples were subsequently obtained. Resections were performed strictly guided according to clinical decision. On the surgical specimen, samplings were made at different ROIs to measure intestinal lactates (mmol/L) and mitochondria efficiency as acceptor control ratio (ACR).
Results
The native (unquantified) fluorescent signal diffused to obvious ischemic areas during the distal appreciation. Proximally, a lower diffusion of ICG was observed. Five anastomotic complications occurred. The expected values of local capillary lactates were correlated with the measured values both proximally (3.62 ± 2.48 expected vs. 3.17 ± 2.8 actual; rho 0.89;
p
= 0.0006) and distally (4.5 ± 3 expected vs. 4 ± 2.5 actual; rho 0.73;
p
= 0.0021). FLER values correlated with ACR at the proximal site (rho 0.76;
p
= 0.04) and at the ischemic zone (rho 0.71;
p
= 0.01). In complicated cases, lactates at the proximal resection site were higher (5.8 ± 4.5) as opposed to uncomplicated cases (2.45 ± 1.5;
p
= 0.008). ACR was reduced proximally in complicated (1.3 ± 0.18) vs. uncomplicated cases (1.68 ± 0.3;
p
= 0.023).
Conclusions
FLER allows to image the quantified fluorescence signal in augmented reality and provides a reproducible estimation of bowel perfusion (NCT02626091).
Objective: Minimally invasive surgical interventions in the gastrointestinal tract, such as endoscopic submucosal dissection (ESD), are very difficult for surgeons when performed with standard ...flexible endoscopes. Robotic flexible systems have been identified as a solution to improve manipulation. However, only a few such systems have been brought to preclinical trials as of now. As a result, novel robotic tools are required. Methods: We developed a telemanipulated robotic device, called STRAS, which aims to assist surgeons during intraluminal surgical endoscopy. This is a modular system, based on a flexible endoscope and flexible instruments, which provides 10 degrees of freedom (DoFs). The modularity allows the user to easily set up the robot and to navigate toward the operating area. The robot can then be teleoperated using master interfaces specifically designed to intuitively control all available DoFs. STRAS capabilities have been tested in laboratory conditions and during preclinical experiments. Results: We report 12 colorectal ESDs performed in pigs, in which large lesions were successfully removed. Dissection speeds are compared with those obtained in similar conditions with the manual Anubiscope platform from Karl Storz. We show significant improvements (p= 0.01). Conclusion: These experiments show that STRAS (v2) provides sufficient DoFs, workspace, and force to perform ESD, that it allows a single surgeon to perform all the surgical tasks and those performances are improved with respect to manual systems. Significance: The concepts developed for STRAS are validated and could bring new tools for surgeons to improve comfort, ease, and performances for intraluminal surgical endoscopy.
Background
Augmented reality (AR) in surgery consists in the fusion of synthetic computer-generated images (3D virtual model) obtained from medical imaging preoperative workup and real-time patient ...images in order to visualize unapparent anatomical details. The 3D model could be used for a preoperative planning of the procedure. The potential of AR navigation as a tool to improve safety of the surgical dissection is outlined for robotic hepatectomy.
Materials and methods
Three patients underwent a fully robotic and AR-assisted hepatic segmentectomy. The 3D virtual anatomical model was obtained using a thoracoabdominal CT scan with a customary software (VR-RENDER®, IRCAD). The model was then processed using a VR-RENDER® plug-in application, the Virtual Surgical Planning (VSP®, IRCAD), to delineate surgical resection planes including the elective ligature of vascular structures. Deformations associated with pneumoperitoneum were also simulated. The virtual model was superimposed to the operative field. A computer scientist manually registered virtual and real images using a video mixer (MX 70; Panasonic, Secaucus, NJ) in real time.
Results
Two totally robotic AR segmentectomy V and one segmentectomy VI were performed. AR allowed for the precise and safe recognition of all major vascular structures during the procedure. Total time required to obtain AR was 8 min (range 6–10 min). Each registration (alignment of the vascular anatomy) required a few seconds. Hepatic pedicle clamping was never performed. At the end of the procedure, the remnant liver was correctly vascularized. Resection margins were negative in all cases. The postoperative period was uneventful without perioperative transfusion.
Conclusions
AR is a valuable navigation tool which may enhance the ability to achieve safe surgical resection during robotic hepatectomy.
An expert recommendation conference was conducted to identify factors associated with adverse events during laparoscopic cholecystectomy (LC) with the goal of deriving expert recommendations for the ...reduction of biliary and vascular injury. Nineteen hepato-pancreato-biliary (HPB) surgeons from high-volume surgery centers in six countries comprised the Research Institute Against Cancer of the Digestive System (IRCAD) Recommendations Group. Systematic search of PubMed, Cochrane, and Embase was conducted. Using nominal group technique, structured group meetings were held to identify key items for safer LC. Consensus was achieved when 80% of respondents ranked an item as 1 or 2 (Likert scale 1-4). Seventy-one IRCAD HPB course participants assessed the expert recommendations which were compared to responses of 37 general surgery course participants. The IRCAD recommendations were structured in seven statements. The key topics included exposure of the operative field, appropriate use of energy device and establishment of the critical view of safety (CVS), systematic preoperative imaging, cholangiogram and alternative techniques, role of partial and dome-down (fundus-first) cholecystectomy. Highest consensus was achieved on the importance of the CVS as well as dome-down technique and partial cholecystectomy as alternative techniques. The put forward IRCAD recommendations may help to promote safe surgical practice of LC and initiate specific training to avoid adverse events.
PDF
