Rapid and reliable robot grasping of a wide variety of objects remains a Grand Challenge for robotics due to sensor noise, imprecise control, and partial observability. Deep neural networks trained on datasets of human-labeled or self-supervised grasps can be used to rapidly plan grasps across a diverse set of objects, but data collection is tedious and performance may asymptote with training dataset size. To reduce data collection time, I propose to generate synthetic training datasets of millions of 3D point clouds and robot grasps using geometric models of grasp success and image formation. In this talk I present the Dexterity-Network (Dex-Net), a framework for generating datasets by analyzing mechanical models of contact forces and torques under stochastic perturbations across thousands of 3D object CAD models. I describe generative models for training policies to lift and transport objects from a tabletop or cluttered bin using a parallel-jaw (two-finger) or suction cup gripper. I explore methods for learning robust policies that transfer from simulation to reality and that can decide which gripper to use. To substantiate the method, I describe thousands of experimental trials on a physical robot which suggest that deep learning on synthetic Dex-Net datasets can be used to rapidly plan successful grasps across a diverse set of novel objects with a 95% success rates on heaps of 25 objects.

Autonomous systems, such as self-driving cars, are becoming tangible technologies that will soon impact the human experience. However, the desirable impacts of autonomy are only achievable if the underlying algorithms can handle the unique challenges humans present: People tend to defy expected behaviors and do not conform to many of the standard assumptions made in robotics. To design safe, trustworthy autonomy, we must transform how intelligent systems interact, influence, and predict human agents. In this work, we’ll use tools from robotics, artificial intelligence, and control to explore and uncover structure in complex human-robot systems to create more intelligent, interactive autonomy. In this talk, I'll present on robust prediction methods that allow us to predict driving behavior over long time horizons with very high accuracy. These methods have been applied to intervention schemes for semi-autonomous vehicles and to autonomous planning that considers nuanced interactions during cooperative maneuvers. I’ll also present a new framework for multi-agent perception that uses people as sensors to improve mapping. By observing the actions of human agents, we demonstrate how we can make inferences about occluded regions and, in turn, improve control. Finally, I’ll present on recent efforts on validating stochastic systems, merging learning and control, and implementing these algorithms on a fully equipped test vehicle that can operate safely on the road.

For the past several decades, consumer applications of robotics have been more science fiction than reality. However, recent developments in deep-learning, cloud AI, and plummeting prices of both computation and sensing have created the necessary components for a rapidly growing consumer robotics industry to finally emerge. In this talk, I’ll discuss the evolution of Anki from 3 Ph.Ds and a kitchen table prototype, to a global company that has quickly become the 2nd largest producer of consumer robots in the world. I’ll share many of the successes and challenges of producing robots at million+ unit scale, and the important trends that will impact both academia and industry. I’ll talk about the importance of emotion and character for building a great user experience, and some surprising findings about human-robot interaction. I’ll also discuss Anki’s unique “bottom’s up approach" to robotics, and show how with an increasingly complicated series of low-cost mass-market robots, we’ve created a virtuous cycle that’s driving growth in the industry. Lastly, I'll discuss Vector, a new robot Anki just announced that is an important step towards a future with useful, emotive, robot companions for every home.

In this talk, Jur will share his passion for the specific application of autonomous vehicle technology to moving freight on highways, as it meaningfully downscopes the technical challenges of autonomous mobility in multiple ways, while at the same time allowing for a compelling business case. He will explain in detail what the specific technical challenges are to letting trucks drive themselves safely on highways, and what it takes to get a self-driving product on the road, for real.

Although wildly successful, deep learning systems are also extremely brittle; this is evidenced for example by the widespread possibility of adversarial attacks, specially crafted inputs meant to fool deep classifiers. This talk will discuss our recent work in develop deep classifiers that are provably robust to (certain classes of) perturbation attacks. Our methods work by considering a convex relaxation of the "adversarial polytope", the set of last-layer activations achievable under some norm-bounded perturbation of the input, and using these to derive very efficient methods for computing (and then minimizing) upper bounds the adversarial loss that can be suffered under such attacks. The method leads to some of the largest verified networks of which we are currently aware, including a convolutional MNIST classifier with a provable bound of 3.7% error under L_infinity perturbations of size epsilon=0.1. I'll relate our work to similar ongoing directions, and also discuss the main challenges that it faces: the task of scaling to significantly larger networks, e.g. at ImageNet scale, and the task of better characterizing the "correct" set of perturbations that we would like to be robust to. I'll also connect the work to efforts in proving properties about deep networks in other settings, such as control and general verification.

Continuum manipulators are a class of long, slender soft robots that can be employed for minimally invasive surgical procedures such are cardiac cath eterization, colonoscopy, and bronchoscopy.  The soft nature of these devices introduces uncertainty in modeling both due to the deformable nature of the manip ulator and the environment. To deal with uncertainty in continuum manipulators, my group has developed two approaches to control that greatly reduce the need f or modeling and instead rely on continuous measurement of the device.  In addition, we are beginning to work on the problem of dealing with uncertainties in th e environment by performing continuous real-time registration to moving anatomy, using image feedback and deep learning.  I will discuss our control and locali zation research in this talk, working toward the ultimate goal of automating minimally invasive surgery.

Biological systems such as birds and humans are able to move with great agility, efficiency, and robustness in a wide range of environments.  Endowing machines with similar capabilities requires designing controllers that address the challenges of high-degree-of-freedom, high-degree-of-underactuation, nonlinear & hybrid dynamics, as well as input, state, and safety-critical constraints in the presence of model and sensing uncertainty.  In this talk, I will present the design of planning and control algorithms for (i) dynamic legged locomotion over discrete terrain that requires enforcing safety-critical constraints in the form of precise foot placements; and (ii) dynamic aerial manipulation through cooperative transportation of a cable-suspended payload using multiple aerial robots with safety-critical constraints on manifolds.  We will use the tools of control Lyapunov and control barrier functions for enforcing stability and safety while combining it with deep learning for visual perception.

My main goal in this talk is to discuss some manipulation capabilities that I believe are essential to develop practical autonomous robotic manipulation systems. These are manipulation skills that I wish my robots had. I’ll start the talk by briefing on recent work by Team MIT-Princeton in the Amazon Robotics Challenge to develop a robotic pick-and-place system capable of grasping and recognizing novel objects in cluttered environments. I’ll also describe the challenges derived from the lack of practical solutions to exploit feedback and contact sensing, which make of grasping a yet unsolved problem. In the second part of the presentation I will share ideas on what I think are some of the missing capabilities to build systems with dexterity. I will finish by describing current efforts in my group to develop perception, planning, and control algorithms that explicitly embrace contact: 1) by exploiting contacts with the environment; 2) by fusing tactile and vision sensing in real time; and 3) by enabling fast reactive control.

There is a pressing demand to integrate unmanned aerial vehicles, commonly termed as drones, in the national airspace to enable a plethora of services such as package delivery, infrastructure monitoring, news and sports coverage, and precision agriculture. Recent research and development efforts in industry, Government and academia have focused on designing a dedicated traffic management system for unmanned aerial vehicles, operating at low altitude. Unlike the existing manned air traffic management, the unmanned air traffic management (UTM) system is envisioned to have higher degree of autonomy, and must be provably safe while guaranteeing high throughput. As a networked cyber-physical system, UTM has many unique design challenges which essentially stem from the fact that different groups of drones may have conflicting business needs, and yet are required to share the same physical space-time resource. This talk will outline a UTM motion protocol to guarantee real-time safety, which relies on timely computation of reachable tubes of the drones subject to measurement and wind uncertainties. We will propose algorithms for fast computation of outer approximation of these tubes to enable this motion protocol, and will provide numerical results.

This talk will discuss insights gathered over nearly thirty years of research on medical robotics and computer-integrated interventional medicine (CIIM), both at IBM and at Johns Hopkins University. The goal of this research has been the creation of a three-way partnership between physicians, technology, and information to improve treatment processes. CIIM systems combine innovative algorithms, robotic devices, imaging systems, sensors, and human-machine interfaces to work cooperatively with surgeons in the planning and execution of surgery and other interventional procedures. For individual patients, CIIM systems can enable less invasive, safer, and more cost-effective treatments. Since these systems have the ability to act as “flight data recorders” in the operating room, they can enable the use of statistical methods to improve treatment processes for future patients and to promote physician training. We will illustrate these themes with examples from our past and current work and will offer some thoughts about future research opportunities and system evolution.

Human-robot interactions (HRI) have been recognized to be a key element of future robots in manufacturing and transportation, which entail huge social and economical impacts. Technically, it is challenging to design the behavior of these robots, as they need to operate in highly unstructured and stochastic environments. The fundamental problem to address in this talk is how to ensure these robots operate efficiently and safely in human-involved dynamic uncertain environments. The microscopic aspect of the problem, i.e. the design of the behavior system for single robot through learning, motion planning and control in the framework of adaptive optimal control, will be discussed first. A novel real-time non-convex optimization solver that handles nonlinear robot dynamics and non-convex safety constraints will be introduced to ensure timely responses of the robot during interactions. Then the macroscopic aspect of the problem, i.e. the analysis and synthesis of the human-robot system from a game theoretical multi-agent system perspective, will be discussed. The performance of the method will be illustrated through human-in-the-loop experiments on industrial collaborative robot manipulators and automated vehicles. These works demonstrate a promising way to build intelligent robots that can better serve, assist and collaborate with people in our daily lives across work, home and leisure.

The nascent field of soft robotics has emerged as an exciting area of research that stands to revolutionize our interaction with machines. Soft robots possess many attributes that are difficult, if not impossible, to achieve with conventional robots composed of rigid materials. Yet, despite recent advances, soft robots generally require rigid components for power, control, and energy regulation, thus they remain either tethered or soft-rigid hybrid systems. Recent work has explored possible soft analogs for these standard rigid components. Elimination of rigid components allows the shrinking of length scales and development of new form factors impossible with traditional motors, batteries, and electronic controllers. We look at the use of a novel soft controller, alternate fuel source, and soft actuator and explore possible form factors and applications. These components are brought together via a design and rapid fabrication approach, which lays the foundation for a new class of completely soft, autonomous robots. I will describe the development of Octobot, the first entirely soft, untethered robot. I will then present work in EMB3D printing soft somatosensitive actuators innervated with multiple conductive features for haptic, proprioceptive, and thermoceptive sensing in soft robotic end effectors. This integrated design, materials, and manufacturing approach can be readily extended to other soft robotic systems that are entirely soft, require somatosensory feedback for improved control, or cannot be made with traditional manufacturing methods.

Have you ever wondered what it’s like to be a robot? While others are wildly speculating about what the future of robots will look like, we actually already know quite a bit about what it’s like to live and work around robots. We also know a lot about what it’s like to telecommute to work everyday via telepresence robot. Coming from a human-robot interaction perspective, I’ll be sharing some of those experiences and lessons with you. Over the past several years, I’ve collaborated with remote colleagues via robotic telepresence systems that enabled them to drive themselves around the office, join in those impromptu hallway meetings, pounce on us when we didn’t respond to emails, and ultimately build stronger working relationships. I’ll present the research lessons learned from several years of fielding prototype telepresence robots in multiple companies and running quantitative user studies in the lab to figure out how to better support remote collaboration.

Future human-robot teams need to function as effectively, preferably better than their equivalent human-only teams. Developing robot teammates that can contribute effectively to the team in order to elevate the entire team requires robots to understand and adapt to their human counterparts. This adaptive capability is particularly important in high-risk, uncertain and dynamic environments that expose humans to dangerous situations, such as first response to a large manmade disaster. Human performance can be negatively impacted in a significant manner in these more demanding scenarios. Robots, however, are not susceptible to these same performance degradations. Further, as robot capabilities improve, robots are will reason over team modifications, such as interaction methods or task assignments, in an optimized manner. Ultimately, robot teammates need to adapt to the changes in the human teammates’ performance in order to ensure the team succeeds. This presentation will provide a road map and our progress towards to developing such capabilities. In particular, how can human performance be modeled and objectively measured given the domain constraints, in what ways can human-robot teaming be adapted, and how can the measures of human performance be analyzed to identify overload and underload states in real-time.

Capable mobile manipulation will be a cornerstone for the future of marine robotics. Compliant underactuated hands are a useful solution for manipulation in unstructured environments, allowing a range of grasp types and affording physical robustness without the complexity of a fully-actuated design. However, these devices often trade off dexterity and simplicity. In this talk, I will address the design of a hand that is versatile and strong, while also being light and resilient. Ocean One is a submersible humanoid robot intended to bring intuitive telepresence to delicate subsea environments. Its adaptive, multi-finger, tendon-driven hands can perform both precision pinches and strong wrap grasps with a single actuator. This design was field-tested as part of Ocean One’s maiden voyage, during which it acquired a vase from the La Lune shipwreck site at a depth of 91m in the Mediterranean Sea. The hands use fingers with preloaded nonlinear flexure joints and a spring-loaded transmission for controllable grasp force distribution, such that the hand can be relatively soft for handling delicate objects and stiff for tasks requiring strength. We also explore the addition of gentle suction flow to the fingertips as a way to enhance grasp acquisition and stability and to sense contact. As Ocean One's hands are controlled from a haptic console, evaluation of performance is intended to guide teleoperator decisions.

Toyota Research Institute (TRI), established in 2015, has four initial mandates: 1) enhance the safety of automobiles, 2) increase access to cars to those who otherwise cannot drive, 3) translate Toyota’s expertise in creating products for outdoor mobility into products for indoor mobility, and 4) accelerate scientific discovery by applying techniques from artificial intelligence and machine learning. TRI is based in the United States, with offices in Los Altos, Calif., Cambridge, Mass., and Ann Arbor, Mich.  In this talk, Eric Krotkov will provide a high-level overview of the research and development activities at TRI.  He will go on to present recent approaches and results in robotics, with a focus on manipulation.

In the animal world there is no WiFi—agents collaborate and compete by sensing and predicting the actions of teammates, rivals, predators, and prey.  Likewise, in the engineered world, many of the most promising applications for autonomous robots require them to interact with other agents in the world by sensing and predicting their actions.  Autonomous driving in traffic, collision avoidance for UAVs, and human-robot teaming are key examples where a wireless network either cannot exist, or will not exist for some time.  In competitive scenarios, such as racing or pursuit-evasion, agents would not want to communicate even if they could.  In this talk I will describe several recent examples from my lab of algorithms enabling multiple robotic agents to interact, both collaboratively and competitively, without a communication network.  I will discuss a communication-free multi-robot manipulation algorithm by which many simple robots cooperate to transport a payload too large for any one of them to move alone.  I will describe a highly scalable collision avoidance strategy, and a related pursuit-evasion strategy, that only requires agents to sense the positions of nearby neighbors.  Finally, I will present a game theoretic receding horizon control algorithm for autonomous drone racing, in which drones sense each other's position with a monocular camera.  I will show results from hardware experiments with ground robots, scale autonomous cars, and quadrotor UAVs collaborating and competing in the scenarios above.

Robotics is experiencing a period of explosive growth in academia and industry. As robots are assigned more and more complex tasks to be performed in a variety of situations, it becomes essential being able to pursue multiple objectives at once while coping with uncertainty and possible failures. In this talk I will present some of our recent results in risk-aware multi objective planning leveraging the theory of constrained Markov Decision Processes. I will show how this approach can be used to tackle a variety of problems, how to manage large state spaces, and how to close the loop between theory and applications.

Laziness is defines as "the quality of being unwilling to work". It is a common approach used in many algorithms (and by many graduate students) where work, or computation, is delayed until absolutely necessary. In the context of motion planning, this idea has been frequently used to reduce the computational cost of testing if a robot collides with obstacles, an operation that governs the running time of many motion-planning algorithms. In this talk I will describe and analyze several algorithms that use this simple, yet effective idea, to dramatically improve over the state-of-the-art. A by-product of lazily performing collision detection is a shift in the computational weight in motion-planning algorithms from collision detection to nearest-neighbor search or to graph search. This induces new challenges which I will also address in my talk—Can we employ application-specific nearest-neighbor data structures tailored for lazy motion-planning algorithms? Do we need to be completely lazy (with respect to collision detection) or should we balance laziness with, say graph operations?

Early robots often occupied tightly controlled environments, e.g., factory floors, designed to segregate robots and humans for safety. In the near future, robots will "live" with humans, providing a variety of services at homes, in workplaces, or on the road. To become effective and trustworthy collaborators, robots must adapt to human preferences and more interestingly, adapt to changing human preferences, as humans adapt as well. I will discuss our recent work, covering mathematical models that leverage estimation of human intention for robot adaptation, planning algorithms that connect robot perception with decision making, and learning algorithms that enable robots to adapt to human preferences without a prior model.  The discussion, I hope, will  spur greater interest towards principled approaches that integrate perception, planning, and learning for fluid human-robot collaboration.