Using light field methods, we can use Baxter’s monocular camera to localize the metal 0.24′ nut and corresponding bolt. The localization is precise enough to allow the robot to use the estimated poses to then perform an open-loop pick, place, and screw to put the nut on the bolt. The precise pose estimation enables complex routines to be quickly encoded, because once the robot knows where the parts are, it can perform accurate grasps and placement actions. You can read more about how it works in our RSS 2017 paper.
Rebecca Pankow and John Oberlin programmed Baxter to pick petals off of a daisy during my graduate seminar last semester, Topics in Grounded Language for Robotics. The robot localizes the petal on the daisy using synthetic photography based on light fields, then plucks each petal off of the daisy. It looks for the largest open space when selecting the next petal to pick. It keeps track of the parity of the petals picked so it can either nod and smile (if the answer is, “he loves me”) or frown (if the answer is, “he loves me not.”) This project was recently featured in the New Yorker!
Our group was featured in this New Yorker article, showcasing Rebecca Pankow and John Oberlin’s work programming Baxter to pick petals from a daisy, as well as some of my thoughts on inequality and automation. I was thrilled with Sheelah’s work on this very important issue, focusing on the effects of automation and our changing economy.
Slashdot wrote a little article about us! Neato! The title may be a bit inaccurate, but it does talk about a cool collaboration we have with Cornell, particularly our fellow roboticist Ashutosh Saxena.