We investigate whether a robot arm can learn to pick and throw arbitrary rigid objects into selected boxes quickly and accurately. Throwing has the potential to increase the physical reachability and ...picking speed of a robot arm. However, precisely throwing arbitrary objects in unstructured settings presents many challenges: from acquiring objects in grasps suitable for reliable throwing, to handling varying object-centric properties (e.g., mass distribution, friction, shape) and complex aerodynamics. In this work, we propose an end-to-end formulation that jointly learns to infer control parameters for grasping and throwing motion primitives from visual observations (RGB-D images of arbitrary objects in a bin) through trial and error. Within this formulation, we investigate the synergies between grasping and throwing (i.e., learning grasps that enable more accurate throws) and between simulation and deep learning (i.e., using deep networks to predict residuals on top of control parameters predicted by a physics simulator). The resulting system, TossingBot , is able to grasp and successfully throw arbitrary objects into boxes located outside its maximum reach range at 500+ mean picks per hour (600+ grasps per hour with 85% throwing accuracy); and generalizes to new objects and target locations.
We present a survey of recent work on robot manipulation and sensing of deformable objects, a field with relevant applications in diverse industries such as medicine (e.g. surgical assistance), food ...handling, manufacturing, and domestic chores (e.g. folding clothes). We classify the reviewed approaches into four categories based on the type of object they manipulate. Furthermore, within this object classification, we divide the approaches based on the particular task they perform on the deformable object. Finally, we conclude this survey with a discussion of the current state-of-the-art approaches and propose future directions within the proposed classification.
Cables are complex, high-dimensional, and dynamic objects. Standard approaches to manipulate them often rely on conservative strategies that involve long series of very slow and incremental ...deformations, or various mechanical fixtures such as clamps, pins, or rings. We are interested in manipulating freely moving cables, in real time, with a pair of robotic grippers, and with no added mechanical constraints. The main contribution of this paper is a perception and control framework that moves in that direction, and uses real-time tactile feedback to accomplish the task of following a dangling cable. The approach relies on a vision-based tactile sensor, GelSight, that estimates the pose of the cable in the grip, and the friction forces during cable sliding. We achieve the behavior by combining two tactile-based controllers: (1) cable grip controller, where a PD controller combined with a leaky integrator regulates the gripping force to maintain the frictional sliding forces close to a suitable value; and (2) cable pose controller, where an linear–quadratic regulator controller based on a learned linear model of the cable sliding dynamics keeps the cable centered and aligned on the fingertips to prevent the cable from falling from the grip. This behavior is possible with the use of reactive gripper fitted with GelSight-based high-resolution tactile sensors. The robot can follow 1 m of cable in random configurations within two to three hand regrasps, adapting to cables of different materials and thicknesses. We demonstrate a robot grasping a headphone cable, sliding the fingers to the jack connector, and inserting it. To the best of the authors’ knowledge, this is the first implementation of real-time cable following without the aid of mechanical fixtures. Videos are available at http://gelsight.csail.mit.edu/cable/
Robotic manipulation aims at combining the versatility and flexibility of mobile robot platforms with manipulation capabilities of robot manipulators. This survey paper comprehensively reviews the ...state-of-the-art development of collaborative robotic manipulation from the perspective of modelling, control and optimization. Then, the recent results in this field can be categorized into coordination of multiple fixed manipulators, mobile robots and mobile manipulators, respectively. A classification and comparison of various issues and promising approaches is given. Finally, a short discussion section is given to summarize existing research and to point out several future research directions.
In this paper, we propose Multi-View Dreaming, a novel reinforcement learning agent for integrated recognition and control from multi-view observations by extending Dreaming. Most current ...reinforcement learning method assumes a single-view observation space, and this imposes limitations on the observed data, such as lack of spatial information and occlusions. This makes obtaining ideal observational information from the environment difficult and is a bottleneck for real-world robotics applications. In this paper, we use contrastive learning to train a shared latent space between different viewpoints and show how the Products of Experts approach can be used to integrate and control the probability distributions of latent states for multiple viewpoints. We also propose Multi-View DreamingV2, a variant of Multi-View Dreaming that uses a categorical distribution to model the latent state instead of the Gaussian distribution. Experiments show that the proposed method outperforms simple extensions of existing methods in a realistic robot control task.
Grasping and assembly are essential tasks in high-precision robotic manipulation for industrial manufacturing as well as for home service applications. Many efforts have been devoted to this area in ...an attempt to meet the increasing precision requirement of the task. However, it remains a problematic objective to fulfill high precision, high reliability, high speed, and high flexibility all at once during one robotic manipulation task. To find answers to the above-mentioned problem, this article tries to categorize, review, and compare the recent works focusing on robotic grasping and assembly tasks to reveal some potential trends in this research area. The approaches will be divided into five groups based on the difference in the utilization of sensing or constraints. For each part, robotic grasping and assembly will be treated as practical cases to illustrate the concrete work in that area. This article could give the readers some knowledge on the current developments in robotic manipulation, and provide new thoughts on future direction in this area-inspiring new designs, structures, and systems to meet new requirements in applications in industrial manufacturing and home service.
The robotic manipulation of deformable linear objects (DLOs) has broad application prospects in many fields. However, a key issue is to obtain the exact deformation models (i.e., how robot motion ...affects DLO deformation), which are hard to theoretically calculate and vary among different DLOs. Thus, the shape control of DLOs is challenging, especially for large deformation control that requires global and more accurate models. In this article, we propose a coupled offline and online data-driven method for efficiently learning a global deformation model, allowing for both accurate modeling through offline learning and further updating for new DLOs via online adaptation. Specifically, the model approximated by a neural network is first trained offline on random data, then seamlessly migrated to the online phase, and further updated online during actual manipulation. Several strategies are introduced to improve the model's efficiency and generalization ability. We propose a convex-optimization-based controller and analyze the system's stability using the Lyapunov method. Detailed simulations and real-world experiments demonstrate that our method can efficiently and precisely estimate the deformation model and achieve the large deformation control of untrained DLOs in 2-D and 3-D dual-arm manipulation tasks better than the existing methods. It accomplishes all 24 tasks with different desired shapes on different DLOs in the real world, using only simulation data for the offline learning.
PDF
