Rebecca Pankow and John Oberlin programmed Baxter to pick petals off of a daisy during my graduate seminar last semester, Topics in Grounded Language for Robotics. The robot localizes the petal on the daisy using synthetic photography based on light fields, then plucks each petal off of the daisy. It looks for the largest open space when selecting the next petal to pick. It keeps track of the parity of the petals picked so it can either nod and smile (if the answer is, “he loves me”) or frown (if the answer is, “he loves me not.”) This project was recently featured in the New Yorker!
Humans communicating with other humans use a feedback loop that enables errors to be detected and fixed, increasing overall interaction success. We aim to enable robots to participate in this feedback loop so that they elicit additional information from the person when they are confused and use that information to resolve ambiguity and infer the person’s needs. This technology will enable robots to interact fluidly with untrained users who communicate with them using language and gestures. People from all walks of life can benefit from robotic help with physical tasks, ranging from assisting a disabled veteran in his home by fetching objects to a captain coordinating with a robotic assistant on a search-and-rescue mission.
Our latest paper defines a mathematical framework for an item-fetching domain that allows a robot to increase the speed and accuracy of its ability to interpret a person’s requests y reasoning about its own uncertainty as well as processing implicit information (implicatures). We formalize the item delivery domain as a Partially Observable Markov Decision Process (POMDP), and approximately solve this POMDP in real time. Our model improves speed and accuracy of fetching tasks by asking relevant clarifying questions only when necessary. To measure our model’s improvements, we conducted a real world user study with 16 participants. Our model is 2.17 seconds faster (25% faster) than state-of-the-art baseline, while being 2.1% more accurate.
You can see the system in action in this video: when the user is close to the robot, it is able to interpret the gesture and immediately selects the correct object without asking a question. However when the user is farther away, the pointing gesture is more ambiguous. The robot asks a targeted question. After the user answers the question, the robot selects the correct object. For more information, see our paper, which was accepted into ICRA 2017!
This summer, we had some great high school students work on projects involving our robots. Anisha Agarwal was one of those students. She built a bowling routine for Baxter. Here is here project!
With the ability to pick and place objects comes a surprising amount of power. Picking up objects and placing them down is the basis for setting a table, drawing, building block structures, playing numerous games and more. We decided to use this power to teach Baxter how to bowl. The bowling program sets up bowling pins and knocks them down by rolling a ball towards them. Baxter sets up 3 bowling pins from the home area into an area on the other end of the table. Baxter also picks up a “bowling” ball (although, for our purposes, a golf ball worked better), swings its arm and releases the ball towards the upright pins.
Occasionally, Baxter accidentally knocks down a pin in its attempt to place another one nearby. Also, a very specific gripper setting is necessary, such that the grippers are wide enough for the ball, but slim enough to grasp the thinnest portion of the bowling pins. Also, since all 3 pins and the bowling ball are presented to Baxter at once, it can be difficult to arrange them so they aren’t close enough together to confuse the robot, but also not so far apart that certain pieces are outside of the range where the arm can reach.
Despite these limitations, it’s exciting to watch Baxter setting up and knocking down pins!
Hey! So here’s a cool video of us using an Amazon Echo to control one of our Baxters. The Echo definitely outperforms our previous speech-to-text methods, and we like using it.
If you’re interested in finding out about how we did it, check the code here, or shoot me an email at email@example.com.