Rapid and reliable robot grasping of a wide variety of objects remains a Grand Challenge for robotics due to sensor noise, imprecise control, and partial observability. Deep neural networks trained on datasets of human-labeled or self-supervised grasps can be used to rapidly plan grasps across a diverse set of objects, but data collection is tedious and performance may asymptote with training dataset size. To reduce data collection time, I propose to generate synthetic training datasets of millions of 3D point clouds and robot grasps using geometric models of grasp success and image formation. In this talk I present the Dexterity-Network (Dex-Net), a framework for generating datasets by analyzing mechanical models of contact forces and torques under stochastic perturbations across thousands of 3D object CAD models. I describe generative models for training policies to lift and transport objects from a tabletop or cluttered bin using a parallel-jaw (two-finger) or suction cup gripper. I explore methods for learning robust policies that transfer from simulation to reality and that can decide which gripper to use. To substantiate the method, I describe thousands of experimental trials on a physical robot which suggest that deep learning on synthetic Dex-Net datasets can be used to rapidly plan successful grasps across a diverse set of novel objects with a 95% success rates on heaps of 25 objects.

Autonomous systems, such as self-driving cars, are becoming tangible technologies that will soon impact the human experience. However, the desirable impacts of autonomy are only achievable if the underlying algorithms can handle the unique challenges humans present: People tend to defy expected behaviors and do not conform to many of the standard assumptions made in robotics. To design safe, trustworthy autonomy, we must transform how intelligent systems interact, influence, and predict human agents. In this work, we’ll use tools from robotics, artificial intelligence, and control to explore and uncover structure in complex human-robot systems to create more intelligent, interactive autonomy. In this talk, I'll present on robust prediction methods that allow us to predict driving behavior over long time horizons with very high accuracy. These methods have been applied to intervention schemes for semi-autonomous vehicles and to autonomous planning that considers nuanced interactions during cooperative maneuvers. I’ll also present a new framework for multi-agent perception that uses people as sensors to improve mapping. By observing the actions of human agents, we demonstrate how we can make inferences about occluded regions and, in turn, improve control. Finally, I’ll present on recent efforts on validating stochastic systems, merging learning and control, and implementing these algorithms on a fully equipped test vehicle that can operate safely on the road.

For the past several decades, consumer applications of robotics have been more science fiction than reality. However, recent developments in deep-learning, cloud AI, and plummeting prices of both computation and sensing have created the necessary components for a rapidly growing consumer robotics industry to finally emerge. In this talk, I’ll discuss the evolution of Anki from 3 Ph.Ds and a kitchen table prototype, to a global company that has quickly become the 2nd largest producer of consumer robots in the world. I’ll share many of the successes and challenges of producing robots at million+ unit scale, and the important trends that will impact both academia and industry. I’ll talk about the importance of emotion and character for building a great user experience, and some surprising findings about human-robot interaction. I’ll also discuss Anki’s unique “bottom’s up approach" to robotics, and show how with an increasingly complicated series of low-cost mass-market robots, we’ve created a virtuous cycle that’s driving growth in the industry. Lastly, I'll discuss Vector, a new robot Anki just announced that is an important step towards a future with useful, emotive, robot companions for every home.

In this talk, Jur will share his passion for the specific application of autonomous vehicle technology to moving freight on highways, as it meaningfully downscopes the technical challenges of autonomous mobility in multiple ways, while at the same time allowing for a compelling business case. He will explain in detail what the specific technical challenges are to letting trucks drive themselves safely on highways, and what it takes to get a self-driving product on the road, for real.

Although wildly successful, deep learning systems are also extremely brittle; this is evidenced for example by the widespread possibility of adversarial attacks, specially crafted inputs meant to fool deep classifiers. This talk will discuss our recent work in develop deep classifiers that are provably robust to (certain classes of) perturbation attacks. Our methods work by considering a convex relaxation of the "adversarial polytope", the set of last-layer activations achievable under some norm-bounded perturbation of the input, and using these to derive very efficient methods for computing (and then minimizing) upper bounds the adversarial loss that can be suffered under such attacks. The method leads to some of the largest verified networks of which we are currently aware, including a convolutional MNIST classifier with a provable bound of 3.7% error under L_infinity perturbations of size epsilon=0.1. I'll relate our work to similar ongoing directions, and also discuss the main challenges that it faces: the task of scaling to significantly larger networks, e.g. at ImageNet scale, and the task of better characterizing the "correct" set of perturbations that we would like to be robust to. I'll also connect the work to efforts in proving properties about deep networks in other settings, such as control and general verification.

The robotics research community has long relied on the developments in computing technology for consumer electronics in order to integrate complex, performant algorithms into robotics applications. However, the slowdown in the improvements in general computing systems have led the consumer electronics industry to develop application-specific chips from the ground up for specific applications, ranging from face detection to augmented reality. This shift in computing technology presents substantial opportunities and significant technical challenges for next-generation embedded systems for robotics. In the first part of this talk, we discuss ultra-high-throughput embedded systems for robotics applications. We envision fast, agile vehicles equipped with high-rate, high-resolution sensing and powerful embedded computing systems. We discuss high-dimensional control design using compressed continuous computation with tensor decompositions, high-throughput visual-inertial navigation using feature selection with sub-modular optimization, and other problems at the intersection of motion planning and perception. We propose new algorithms with provable guarantees on performance. We demonstrate some of the algorithms in an autonomous drone racing scenario, reaching more than 6.7m/s (roughly 15mph) in a room that is 7m (roughly 20 feet) in size, in a virtual reality environment.In the second part of the talk, we discuss ultra-low-energy embedded systems for robotics. We envision insect-size vehicles with complex perception and decision-making capabilities. We argue that, in order to enable vehicles at such scale, their computers must be designed from the ground up together with the algorithms. We discuss our latest application-specific integrated circuit (ASIC) for visual-inertial odometry with bundle adjustment. The new chip requires a mere 2 milliWatts on average, and can process camera images up to 170 frames per second. 

This talk provides an overview of autonomy and robotics research at the United Technologies Research Center (UTRC), from its past heritage in autonomous vehicle technology development for Sikorsky Aircraft, to its present day focus areas on human-robot collaboration for a broad range of commercial and aerospace applications within United Technologies Corporation (UTC). This includes a review of the role of UTRC in maturing and transitioning key technologies in the areas of autonomy architecture, planning, perception, and human-machine systems which led to successful flight demonstrations on full-sized rotorcraft platforms. The talk will also describe how similar capabilities continue to be carried forward within the context of the current robotics initiative at UTRC, with applications in the areas of advanced manufacturing, inspections, service, and maintenance. A selection of active research topics will also be presented, including hierarchical planning, collaborative autonomy, and human robot interaction. This is accompanied by concept demonstration results on a range of UAV, UGV, and robotic manipulation platforms.

Recent advances in robot learning for control have shown increasing success for tasks in real-world human applications. However, we believe capable robots are still too expensive to allow for widespread consumer adoption for applications within elder care, and homes. We discuss considerations for a new design paradigm for low cost capable robotic manipulators, and follow through with a fully functional realization of this new paradigm.

Recent demonstrations of continuous state reachability in thousands of dimensions are impressive, but to maximize control authority while ensuring safety for human-in-the-loop systems we need not just to identify the existence of a safe trajectory, but to characterize the set of safe controls.  In the first part of the talk I will describe a new algorithm for constructing under-approximations of robust controlled invariant or viable sets of uncertain linear systems using an efficient convex optimization.  In the second part I will tackle a challenge for robust analyses: The a priori model uncertainty and hence safe control authority may be overly conservative.  I will describe an algorithm which addresses this challenge by detecting and efficiently removing the effect of erroneous uncertainty at runtime, thereby improving control authority when it is safe to do so.  The algorithms will be demonstrated on a nonlinear model of a quadrotor and automated delivery of anesthesia respectively.

The promise of driverless vehicles to transform the urban mobility experience is contingent on them behaving well within the human driving and pedestrian ecosystem.  Driverless vehicles must behave in a socially acceptable manner.  This talk will unpack what this means and explore how the technologies for achieving this.

Several decades ago in the early days of MEMS, micro robots seemed like they were just around the corner.  The first IEEE MEMS conference was the ambitiously-named "Micro Robots and Teleoperators Workshop". But every piece of building micro robots turned out to be hard.  Motors, mechanisms, sensing, computation, communication, and power all required significant improvement to support the creation of autonomous robots on a sub-centimeter size scale.  After several decades of progress, it appears that we are finally at the point where we will be able to build such systems, making our own silicon insects and other robots with no natural analog. This talk will cover the status of our mesh-networked silicon ant, flea, and ionocraft efforts, with some hints at future needs for control, path planning, navigation, swarm control, etc.

New advances in parallel real-time scheduling theory and concurrency platforms are enabling a new generation of cyber-physical systems, which can support a challenging combination of (1) significant computational demands, (2) stringent timing constraints, and (3) dynamic and substantial changes in how resources are allocated, at run-time.  This talk will describe a series of recent advances in parallel real-time systems research, including both theoretical and practical results, and how each of them impacts the specific domain of real-time hybrid simulation, a cyber-physical approach to high-fidelity testing of structures at scale that is increasingly relevant for earthquake engineering and other related fields.

Several nonlinear control analysis and design problems, such as stability and stabilization problems, adaptive control problems, model reduction and observer design, optimal/robust control and game theory problems, rely upon the computation of solutions of partial differential equations (PDEs). These could be linear PDEs, such as those encountered in stability, stabilization, and adaptive control and observer design; nonlinear PDES, such as the Hamilton Jacobi PDEs; or coupled PDES, such as those arising in game theory and in mean field games. In this talk we present a method to replace PDEs with algebraic equations at the expense of sacrificing performance (in a measurable way). The method is shown to be sufficiently flexible to deal with a wide range of problems and to provide solutions which are more accurate than those obtained using series expansion methods or other numerical schemes for the solutions of PDEs. Examples in the areas of engine control, multi agent systems and state estimations illustrate the theory.

Automated driving is breaking new ground where "smart", connected and eventually collaborating machines are deployed for increasingly complex tasks in unstructured environments. This provides a wonderful stage with numerous opportunities and challenges, especially considering the societal impact and broad variety of applications, from automated machines in mines to SAE level 4 automated cars. This stage is representative of an ongoing technological shift, with similar trends in many other domains. However, current methodologies are not well prepared for such future Cyber-Physical Systems (CPS), requiring new systems and safety engineering approaches to be established. In this talk I will address limitations of existing engineering methodologies, and in particular those of safety engineering. Traditional safety engineering is focused on structured environments and risk reduction, which based on risk assessment and the definition of corresponding risk reducing measures. For CPS and so called functional safety, the approaches are to a large extent process-based and treat software as deterministic. Common safety engineering patterns include the use of "safety functions" that are separate from the  nominal system and that are simple enough to be "certified" w.r.t. the identified risk level. Using this context, I will discuss the design and assurance of highly automated vehicles, elaborating a number of perspectives to the challenges and directions, emphasizing the role of controlled experimental environments, providing minimal performance requirements, and accelerating the development of safety engineering guidelines. I will describe our ongoing work on so called safety supervisor architectures and their design. A conceptual architectural design of a fault-tolerant autonomous driving intelligence will be presented, encompassing a nominal and a safety supervisor channel. I will discuss hazardous events, their sources, the design space in terms of redundancy and diversity, the division of responsibilities among the channels, when the supervisor should take over, and remaining open challenges. In architecting future automated vehicles it is essential that viewpoints and solution strategies from controls, computer science, computer engineering, AI/ML, safety and classical vehicle engineering are brought together.

Self-organization is a pervasive phenomenon in nature, which has inspired the development of multi-robot systems that can mimick their biological counterparts. As we consider larger groups of autonomous agents or swarms, new theoretical challenges appear that are associated with both their size and robotic-system limitations.  In this talk, we outline recent work on two complementary problems related to the control of large swarms. First, we consider a deployment objective by which robots are to be shaped into certain density profile. Under the assumption that agents can obtain measurements of the local density, but do not have access to absolute position information, we propose a PDE-based feedback control strategy that includes the distributed computation of diffeomorphisms. Then we discuss how to handle complementary set of limitations such as lack of access to position information, noisy actuation, or optimal transport under distributed gradient flows. To finish, we present preliminary results on the identification of subsets of nodes that are critical for the performance of spatial consensus algorithms in large groups.

Continuum manipulators are a class of long, slender soft robots that can be employed for minimally invasive surgical procedures such are cardiac cath eterization, colonoscopy, and bronchoscopy.  The soft nature of these devices introduces uncertainty in modeling both due to the deformable nature of the manip ulator and the environment. To deal with uncertainty in continuum manipulators, my group has developed two approaches to control that greatly reduce the need f or modeling and instead rely on continuous measurement of the device.  In addition, we are beginning to work on the problem of dealing with uncertainties in th e environment by performing continuous real-time registration to moving anatomy, using image feedback and deep learning.  I will discuss our control and locali zation research in this talk, working toward the ultimate goal of automating minimally invasive surgery.

Biological systems such as birds and humans are able to move with great agility, efficiency, and robustness in a wide range of environments.  Endowing machines with similar capabilities requires designing controllers that address the challenges of high-degree-of-freedom, high-degree-of-underactuation, nonlinear & hybrid dynamics, as well as input, state, and safety-critical constraints in the presence of model and sensing uncertainty.  In this talk, I will present the design of planning and control algorithms for (i) dynamic legged locomotion over discrete terrain that requires enforcing safety-critical constraints in the form of precise foot placements; and (ii) dynamic aerial manipulation through cooperative transportation of a cable-suspended payload using multiple aerial robots with safety-critical constraints on manifolds.  We will use the tools of control Lyapunov and control barrier functions for enforcing stability and safety while combining it with deep learning for visual perception.

My main goal in this talk is to discuss some manipulation capabilities that I believe are essential to develop practical autonomous robotic manipulation systems. These are manipulation skills that I wish my robots had. I’ll start the talk by briefing on recent work by Team MIT-Princeton in the Amazon Robotics Challenge to develop a robotic pick-and-place system capable of grasping and recognizing novel objects in cluttered environments. I’ll also describe the challenges derived from the lack of practical solutions to exploit feedback and contact sensing, which make of grasping a yet unsolved problem. In the second part of the presentation I will share ideas on what I think are some of the missing capabilities to build systems with dexterity. I will finish by describing current efforts in my group to develop perception, planning, and control algorithms that explicitly embrace contact: 1) by exploiting contacts with the environment; 2) by fusing tactile and vision sensing in real time; and 3) by enabling fast reactive control.

There is a pressing demand to integrate unmanned aerial vehicles, commonly termed as drones, in the national airspace to enable a plethora of services such as package delivery, infrastructure monitoring, news and sports coverage, and precision agriculture. Recent research and development efforts in industry, Government and academia have focused on designing a dedicated traffic management system for unmanned aerial vehicles, operating at low altitude. Unlike the existing manned air traffic management, the unmanned air traffic management (UTM) system is envisioned to have higher degree of autonomy, and must be provably safe while guaranteeing high throughput. As a networked cyber-physical system, UTM has many unique design challenges which essentially stem from the fact that different groups of drones may have conflicting business needs, and yet are required to share the same physical space-time resource. This talk will outline a UTM motion protocol to guarantee real-time safety, which relies on timely computation of reachable tubes of the drones subject to measurement and wind uncertainties. We will propose algorithms for fast computation of outer approximation of these tubes to enable this motion protocol, and will provide numerical results.

This talk will discuss insights gathered over nearly thirty years of research on medical robotics and computer-integrated interventional medicine (CIIM), both at IBM and at Johns Hopkins University. The goal of this research has been the creation of a three-way partnership between physicians, technology, and information to improve treatment processes. CIIM systems combine innovative algorithms, robotic devices, imaging systems, sensors, and human-machine interfaces to work cooperatively with surgeons in the planning and execution of surgery and other interventional procedures. For individual patients, CIIM systems can enable less invasive, safer, and more cost-effective treatments. Since these systems have the ability to act as “flight data recorders” in the operating room, they can enable the use of statistical methods to improve treatment processes for future patients and to promote physician training. We will illustrate these themes with examples from our past and current work and will offer some thoughts about future research opportunities and system evolution.

Human-robot interactions (HRI) have been recognized to be a key element of future robots in manufacturing and transportation, which entail huge social and economical impacts. Technically, it is challenging to design the behavior of these robots, as they need to operate in highly unstructured and stochastic environments. The fundamental problem to address in this talk is how to ensure these robots operate efficiently and safely in human-involved dynamic uncertain environments. The microscopic aspect of the problem, i.e. the design of the behavior system for single robot through learning, motion planning and control in the framework of adaptive optimal control, will be discussed first. A novel real-time non-convex optimization solver that handles nonlinear robot dynamics and non-convex safety constraints will be introduced to ensure timely responses of the robot during interactions. Then the macroscopic aspect of the problem, i.e. the analysis and synthesis of the human-robot system from a game theoretical multi-agent system perspective, will be discussed. The performance of the method will be illustrated through human-in-the-loop experiments on industrial collaborative robot manipulators and automated vehicles. These works demonstrate a promising way to build intelligent robots that can better serve, assist and collaborate with people in our daily lives across work, home and leisure.

The nascent field of soft robotics has emerged as an exciting area of research that stands to revolutionize our interaction with machines. Soft robots possess many attributes that are difficult, if not impossible, to achieve with conventional robots composed of rigid materials. Yet, despite recent advances, soft robots generally require rigid components for power, control, and energy regulation, thus they remain either tethered or soft-rigid hybrid systems. Recent work has explored possible soft analogs for these standard rigid components. Elimination of rigid components allows the shrinking of length scales and development of new form factors impossible with traditional motors, batteries, and electronic controllers. We look at the use of a novel soft controller, alternate fuel source, and soft actuator and explore possible form factors and applications. These components are brought together via a design and rapid fabrication approach, which lays the foundation for a new class of completely soft, autonomous robots. I will describe the development of Octobot, the first entirely soft, untethered robot. I will then present work in EMB3D printing soft somatosensitive actuators innervated with multiple conductive features for haptic, proprioceptive, and thermoceptive sensing in soft robotic end effectors. This integrated design, materials, and manufacturing approach can be readily extended to other soft robotic systems that are entirely soft, require somatosensory feedback for improved control, or cannot be made with traditional manufacturing methods.

Have you ever wondered what it’s like to be a robot? While others are wildly speculating about what the future of robots will look like, we actually already know quite a bit about what it’s like to live and work around robots. We also know a lot about what it’s like to telecommute to work everyday via telepresence robot. Coming from a human-robot interaction perspective, I’ll be sharing some of those experiences and lessons with you. Over the past several years, I’ve collaborated with remote colleagues via robotic telepresence systems that enabled them to drive themselves around the office, join in those impromptu hallway meetings, pounce on us when we didn’t respond to emails, and ultimately build stronger working relationships. I’ll present the research lessons learned from several years of fielding prototype telepresence robots in multiple companies and running quantitative user studies in the lab to figure out how to better support remote collaboration.

Future human-robot teams need to function as effectively, preferably better than their equivalent human-only teams. Developing robot teammates that can contribute effectively to the team in order to elevate the entire team requires robots to understand and adapt to their human counterparts. This adaptive capability is particularly important in high-risk, uncertain and dynamic environments that expose humans to dangerous situations, such as first response to a large manmade disaster. Human performance can be negatively impacted in a significant manner in these more demanding scenarios. Robots, however, are not susceptible to these same performance degradations. Further, as robot capabilities improve, robots are will reason over team modifications, such as interaction methods or task assignments, in an optimized manner. Ultimately, robot teammates need to adapt to the changes in the human teammates’ performance in order to ensure the team succeeds. This presentation will provide a road map and our progress towards to developing such capabilities. In particular, how can human performance be modeled and objectively measured given the domain constraints, in what ways can human-robot teaming be adapted, and how can the measures of human performance be analyzed to identify overload and underload states in real-time.

Capable mobile manipulation will be a cornerstone for the future of marine robotics. Compliant underactuated hands are a useful solution for manipulation in unstructured environments, allowing a range of grasp types and affording physical robustness without the complexity of a fully-actuated design. However, these devices often trade off dexterity and simplicity. In this talk, I will address the design of a hand that is versatile and strong, while also being light and resilient. Ocean One is a submersible humanoid robot intended to bring intuitive telepresence to delicate subsea environments. Its adaptive, multi-finger, tendon-driven hands can perform both precision pinches and strong wrap grasps with a single actuator. This design was field-tested as part of Ocean One’s maiden voyage, during which it acquired a vase from the La Lune shipwreck site at a depth of 91m in the Mediterranean Sea. The hands use fingers with preloaded nonlinear flexure joints and a spring-loaded transmission for controllable grasp force distribution, such that the hand can be relatively soft for handling delicate objects and stiff for tasks requiring strength. We also explore the addition of gentle suction flow to the fingertips as a way to enhance grasp acquisition and stability and to sense contact. As Ocean One's hands are controlled from a haptic console, evaluation of performance is intended to guide teleoperator decisions.

Toyota Research Institute (TRI), established in 2015, has four initial mandates: 1) enhance the safety of automobiles, 2) increase access to cars to those who otherwise cannot drive, 3) translate Toyota’s expertise in creating products for outdoor mobility into products for indoor mobility, and 4) accelerate scientific discovery by applying techniques from artificial intelligence and machine learning. TRI is based in the United States, with offices in Los Altos, Calif., Cambridge, Mass., and Ann Arbor, Mich.  In this talk, Eric Krotkov will provide a high-level overview of the research and development activities at TRI.  He will go on to present recent approaches and results in robotics, with a focus on manipulation.

In the animal world there is no WiFi—agents collaborate and compete by sensing and predicting the actions of teammates, rivals, predators, and prey.  Likewise, in the engineered world, many of the most promising applications for autonomous robots require them to interact with other agents in the world by sensing and predicting their actions.  Autonomous driving in traffic, collision avoidance for UAVs, and human-robot teaming are key examples where a wireless network either cannot exist, or will not exist for some time.  In competitive scenarios, such as racing or pursuit-evasion, agents would not want to communicate even if they could.  In this talk I will describe several recent examples from my lab of algorithms enabling multiple robotic agents to interact, both collaboratively and competitively, without a communication network.  I will discuss a communication-free multi-robot manipulation algorithm by which many simple robots cooperate to transport a payload too large for any one of them to move alone.  I will describe a highly scalable collision avoidance strategy, and a related pursuit-evasion strategy, that only requires agents to sense the positions of nearby neighbors.  Finally, I will present a game theoretic receding horizon control algorithm for autonomous drone racing, in which drones sense each other's position with a monocular camera.  I will show results from hardware experiments with ground robots, scale autonomous cars, and quadrotor UAVs collaborating and competing in the scenarios above.

Robotics is experiencing a period of explosive growth in academia and industry. As robots are assigned more and more complex tasks to be performed in a variety of situations, it becomes essential being able to pursue multiple objectives at once while coping with uncertainty and possible failures. In this talk I will present some of our recent results in risk-aware multi objective planning leveraging the theory of constrained Markov Decision Processes. I will show how this approach can be used to tackle a variety of problems, how to manage large state spaces, and how to close the loop between theory and applications.

Laziness is defines as "the quality of being unwilling to work". It is a common approach used in many algorithms (and by many graduate students) where work, or computation, is delayed until absolutely necessary. In the context of motion planning, this idea has been frequently used to reduce the computational cost of testing if a robot collides with obstacles, an operation that governs the running time of many motion-planning algorithms. In this talk I will describe and analyze several algorithms that use this simple, yet effective idea, to dramatically improve over the state-of-the-art. A by-product of lazily performing collision detection is a shift in the computational weight in motion-planning algorithms from collision detection to nearest-neighbor search or to graph search. This induces new challenges which I will also address in my talk—Can we employ application-specific nearest-neighbor data structures tailored for lazy motion-planning algorithms? Do we need to be completely lazy (with respect to collision detection) or should we balance laziness with, say graph operations?

Early robots often occupied tightly controlled environments, e.g., factory floors, designed to segregate robots and humans for safety. In the near future, robots will "live" with humans, providing a variety of services at homes, in workplaces, or on the road. To become effective and trustworthy collaborators, robots must adapt to human preferences and more interestingly, adapt to changing human preferences, as humans adapt as well. I will discuss our recent work, covering mathematical models that leverage estimation of human intention for robot adaptation, planning algorithms that connect robot perception with decision making, and learning algorithms that enable robots to adapt to human preferences without a prior model.  The discussion, I hope, will  spur greater interest towards principled approaches that integrate perception, planning, and learning for fluid human-robot collaboration.