Agile maneuvers are essential for robot-enabled complex tasks such as surgical procedures. Prior explorations on surgery autonomy are limited to feasibility study of completing a single task without ...systematically addressing generic manipulation safety across different tasks. We present an integrated planning and control framework for 6-DoF robotic instruments for pipeline automation of surgical tasks. We leverage the geometry of a robotic instrument and propose the nodal state space to represent the robot state in SE(3) space. Each elementary robot motion could be encoded by regulation of the state parameters via a dynamical system. This theoretically ensures that every in-process trajectory is globally feasible and stably reached to an admissible target, and the controller is of closed-form without computing 6-DoF inverse kinematics. Then, to plan the motion steps reliably, we propose an interactive (instant) goal state of the robot that transforms manipulation planning through desired path constraints into a goal-varying manipulation (GVM) problem. We detail how GVM could adaptively and smoothly plan the procedure (could proceed or rewind the process as needed) based on on-the-fly situations under dynamic or disturbed environment. Finally, we extend the above policy to characterize complete pipelines of various surgical tasks. Simulations show that our framework could smoothly solve twisted maneuvers while avoiding collisions. Physical experiments using the da Vinci Research Kit validates the capability of automating individual tasks including tissue debridement, dissection, and wound suturing. The results confirm good task-level consistency and reliability compared to state-of-the-art automation algorithms.
In this paper, we present a new feedback method to automatically servo-control the 3-D shape of soft objects with robotic manipulators. The soft object manipulation problem has recently received a ...great deal of attention from robotics researchers because of its potential applications in, e.g., food industry, home robots, medical robotics, and manufacturing. A major complication to automatically control the shape of an object is the estimation of its deformation properties, which determines how the manipulator's motion actively transforms into deformations. Note that these properties are rarely known beforehand, and its offline parametric identification is difficult and/or impractical to conduct in many applications. To cope with this issue, we developed a new algorithm that computes in real time the unknown deformation parameters of a soft object; this algorithm provides a valuable adaptive behavior to the deformation controller, something we cannot achieve with traditional fixed-model approaches. In contrast with most controllers in the literature, our new method can explicitly servo-control 3-D deformations (and not just 2-D image projections) in an entirely model-free way. To validate the proposed adaptive controller, we present a detailed experimental study with robotic manipulators.
Deformable object manipulation (DOM) with point clouds has great potential as nonrigid 3D shapes can be measured without detecting and tracking image features. However, robotic shape control of ...deformable objects with point clouds is challenging due to: the unknown point correspondences and the noisy partial observability of raw point clouds; the modeling difficulties of the relationship between point clouds and robot motions. To tackle these challenges, this paper introduces a novel modal-graph framework for the model-free shape servoing of deformable objects with raw point clouds. Unlike the existing works studying the object’s geometry structure, we propose a modal graph to describe the low-frequency deformation structure of the DOM system, which is robust to the measurement irregularities. The modal graph enables us to directly extract low-dimensional deformation features from raw point clouds without extra processing of registrations, refinements, and occlusion removal. It also preserves the spatial structure of the DOM system to inverse the feature changes into robot motions. Moreover, as the framework is built with unknown physical and geometric object models, we design an adaptive robust controller to deform the object toward the desired shape while tackling the modeling uncertainties, noises, and disturbances online. The system is proved to be input-to-state stable (ISS) using Lyapunov-based methods. Extensive experiments are conducted to validate our method using linear, planar, tubular, and volumetric objects under different settings.
Despite successful clinical applications, teleoperated robotic surgical systems face particular limitations in the functional endoscopic sinus surgery (FESS) in terms of incompatible instrument ...dimensions and robot set-up. The endoscope remains manually handled by an assistant when the surgeon performs bimanual operations. This paper introduces the development of the Foot-controlled Robot-Enabled EnDOscope Manipulator (FREEDOM) designed for FESS. The system features clinical considerations that inform the design for providing reliable and safe endoscope positioning with minimal obstruction to the routine practice. The robot structure is modular and compact to ensure coaxial instrument manipulation through the nostril for manual procedures. To avoid rigid endoscope motions, a new compliant endoscope holder is proposed that passively limits the lens-tissue contact forces under collisions for patient-side protection. To facilitate hands-free endoscope manipulation that imposes minimal distractions to the surgeon, a foot-wearable interface is further designed to relieve the assistant's workload. The foot control method owns a short learning curve (mean 3.4 mins), and leads the task to be more ergonomic and surgeon-centered. Cadaver and clinical studies were both conducted to evaluate the surgical applicability of the FREEDOM to assist endoscope manipulation in FESS. The system was validated to be safe (IEC-60601-1) and easy for set up (mean 3.6 mins), from which the surgeon could perform various three-handed procedures alone in FESS without disrupting the routine practice.
A major issue for needle insertion into soft tissue during suturing is the induced tissue deformation that hinders the minimization of tip-target positioning error. In this letter, we present a new ...robot control framework to solve target deviation by integrating active deformation control. We characterize the motion behavior of the desired target under needle-tissue interaction by introducing the needle-induced deformation matrix. Note that the modeling does not require the exact knowledge of tissue or needle insertion properties. The unknown parameters are online updated during the insertion procedure by an adaptive estimator via sensor-based measurement. A closed-loop controller is then proposed for dualarm robotic execution upon image guidance. The dual-arm control aims to regulate a feature vector concerning the tip-target alignment to ensure target reachability. The feasibility of the proposed algorithm is studied via simulations and experiments on different biological tissues to simulate robotic minimally-invasive suturing using the da Vinci Research Kit as the control platform.
Conventional robot hand-eye calibration methods are impractical for localizing robotic instruments in minimally-invasive surgeries under intra-corporeal workspace after preoperative set-up. In this ...letter, we present a new approach to autonomously calibrate a robotic instrument relative to a monocular camera without recognizing calibration objects or salient features. The algorithm leverages interactive manipulation (IM) of the instrument for tracking its rigid-body motion behavior subject to the remote center-of-motion constraint. An adaptive controller is proposed to regulate the IM-induced instrument trajectory, using visual feedback, within a 3D plane which is observable from both the robot base and the camera. The eye-to-hand orientation and position are then computed via a dual-stage process allowing parameter estimation in low-dimensional spaces. The method does not require the exact knowledge of instrument model or large-scale data collection. Results from simulations and experiments on the da Vinci Research Kit are demonstrated via a laparoscopy resembled set-up using the proposed framework.
In this letter, we propose a new method to fully control complete 4-image-DoF manipulation of laparoscopic instruments with remote center of motion (RCM) mechanism based on the geometric features of ...a designed marker in a 2-D image. Our marker encodes the configuration of the instruments by computing geometric features among the projected image points from segmented areas in hue-saturation-value (HSV) space. We can then construct an image geometric feature vector to locally characterize the configuration of a laparoscopic instrument. Furthermore, we design an image-based kinematic controller to asymptotically track a planned trajectory using the constructed feature vector as the feedback. We evaluate our integration of rotation distinguishing marker and kinematic controller by several experiments in terms of illumination-invariance, rotation angle accuracy, and controller performance.
In this paper, we address the problem of the image-based 3D pose computation of a semi-circle suturing needle using monocular image feedback for laparoscopy. We propose a constrained ...two-degree-of-freedom (2-DOF) geometry-based modelling method to parametrise the needle's 6-DOF pose, including depth information. The modelling solely relies on the simultaneous observation of the needle's apparent tip and junction. No external markers are needed for extra constraints. An adaptive controller combining gradient descent and vector-flow method is introduced to iteratively guide the needle's initial guessing pose to its real pose by minimizing image-based position errors. Experiments have been conducted using both numerical simulations and simulated laparoscopic scenarios to evaluate the performance of the algorithm.
Shape control of deformable objects is a challenging and important robotic problem. This article proposes a model-free controller using novel 3-D global deformation features based on modal analysis. ...Unlike most existing controllers using geometric features, our controller employs physically based deformation features designed by decoupling global deformation into low-frequency modes. Although modal analysis is widely adopted in computer vision and simulation, its usage in robotic deformation control is still an open topic. We develop a new model-free framework for the modal-based deformation control. Physical interpretation of the modes enables us to formulate an analytical deformation Jacobian matrix mapping the robot manipulation onto changes of the modal features. In the Jacobian matrix, unknown geometric and physical models of the object are treated as low-dimensional modal parameters, which can be used to linearly parameterize the closed-loop system. Thus, an adaptive controller with proven stability can be designed to deform the object while online estimating the modal parameters. Simulations and experiments are conducted using linear, planar, and volumetric objects under different settings. The results not only confirm the superior performance of our controller, but also demonstrate its advantages over the baseline method.
PDF
