Teach your very young children to program without screens, and without literacy!I created a game called “Ready Robots.” Any kid can play as long as they are old enough to do “if you are happy and you know it clap your hands.” They don’t need to be able to read. You can make the game yourself by drawing your own cards, or you can download and print my cards.

To play, ask kids to lay out the cards in a particular order, then say “ready robots? Execute!” Then you and the kids should do each action in the sequence in order. Take turns making different programs for different people. Kids love making a program for their grownups to execute. See a video of the game in action!

I’ve played this game with kids for years. Even very young children can do it by imitating you and looking at the pictures on the cards. It can even build literacy skills by pairing words on the cards with actions. When I add a real robot, like an Aibo robot dog or Sphero or Baxter, I have the kids make the program with cards, then quickly type it into an Ein command line, to make the physical robot execute it. When using an Aibo robot dog, I first ask them to run the program (Sit! Wag your tail! Bark!) and then we have the robot do the same program.

For older children, you can add “if this then that.” Make little programs like “if I stomp my feet then you beep your nose.” Put the cards up on a white board or wall and add more and more rules. It is kind of like Simon Says but more confusing because you might have 4 or 5 different rules.

Most robots can’t pick up most objects most of the time. This is for a variety of reasons. The robot is turned off. The object is out of reach. The object won’t fit in the gripper. The robot doesn’t know where the object is, because it doesn’t have a sensor at all, or because the sensor isn’t aimed at the object, or because if it is aimed at the object, the object is transparent or reflective. Or if it is aimed at the object, and it is easy to see, it might not have a detector or pose estimator for that object that can localize and predict grasps accurately enough to pick things up.

Existing work by Robb Platt, Ken Goldberg, Sergey Levine, and Chelsea Finn is working towards more robust grasping. But doing things on real robots in the real world is hard, and this won’t change any time soon. What is needed is an integrated perception/planning/motion stack with POMDPs and learned object models that can run on a robust robot with a capable, flexible general-purpose manipulator.

I am a computer science professor and also a mom and an aunt. I have taken the opportunity of the COVID pandemic to take a deep dive into computer science with my son and his cousins, ages 7-11. I wanted to share some of the things we’ve tried and what worked and didn’t work.

Scratch Club

One of the first activities we started, right after schools closed, was Scratch Club. We met several times per week for one hour. The first 10-30 minutes, the kids take turns showing each other their programs. They screen-share to demo their program and explain how it works. They get feedback and ideas from the other kids. We talk about giving constructive criticism, calling attention to neat features of their programs, and getting them to say what they plan to make their program do next. The rest of the time, they program on their own, on whatever program they want. I have them split into two separate calls with a more advanced and less advanced group. Then I check in which each student individually and try to help them with their programs. Sometimes I do individual instruction/teaching. Sometimes I help them find an online tutorial to do. (We really like Warfame and Griffpatch!) If they get stuck in the tutorial, I help them find bugs. I’ve observed some really beautiful peer-teaching, where one of them helps another with a problem. This peer teaching is really good for both, because they are practicing teaching/communication skills, as well as programming.

Unfortunately the Scratch programming framework makes several big technical blunders. First they do not provide functions. It is not possible to write a function that returns a value; there are only blocks that take actions. You can hack around this by defining variables to hold the return value, but is this really the sort of programming we want to model? Second, they do not let you have lists of lists. For example, we wanted to do tic-tac-toe and the natural representation is a list of lists to represent a board, but Scratch does not allow this.

Snap is a similar framework that addresses all these problems. But what Scratch gets really really right is the social angle. All our friends use Scratch and not Snap. All the best tutorials use Scratch and not Snap. Scratch has a large user base and the coolest games are really really cool. (Check out Griffpatch’s Cloud Platform Multiplayer game!) And you can remix all those cool games and make your own variations. So together these features tied us to Scratch, even though we poked around at Snap a bit.

What surprised me was that in many cases these limitations in Scratch turned into opportunities to discuss CS concepts we would not have gotten to otherwise. Scratch doesn’t let you store strings or lists as cloud variables. So the kids learned to encode and decode strings into integers, which was super cool to see them doing.

Turing Tumble

Turing Tumble is a toy for programming with marbles. We played with it a lot when he was at a younger age, (and I had to help a lot.) But then I got asked “how do we program the computer to do addition? if we don’t already have addition?” Turing Tumble is a great way to answer this question, because it goes step by step how to build an adder using a marble computer. The curriculum is very well designed so that each puzzle builds on the next. This activity is also nice because it does not require a screen.

Unity

Unity is a widely used framework for making 2D and 3D games, so there are tons of tutorials and resources available on the web. It can be used to make phone and tablet games. I really like the idea of teaching kids the tools that are used to program games they already know and like. It turns them from a passive consumer of games to a producer, which is great not only because they can make games, but because they can look at the games they already play with a critical and creative eye. It also exposes kids to a compiled language, and the idea of rigid body physics. The bad: it is really complicated. It requires a heavyweight machine – no Chromebooks. And it’s easy for mysterious things to go wrong that mysterious check boxes fix. My son is having fun but it takes a pretty big help from me to get things going and fix things when he is stuck. There are lots of fun Youtube videos about how to make Unity games, although following the step-by-step tutorials is somewhat tricky as the Unity versions keep changing. Don’t go this route unless you are willing to do heavy lifting to help when things get stuck.

Khan Academy

This was quite good as far as it went, but it has a ceiling. The Khan Academy Javascript lessons require no software installed and run right in your web browser. It was completely internationalized so my son was doing the lessons in Polish, which was cool to see. The graphics angle was a good hook, and my son enjoyed the visual effects he could create. It was also a good introduction to a text-based language. However the ceiling is rather low; the coolest games in the Khan Academy ecosystem were not as cool as the coolest games in Scratch. They also don’t have a sharing/social aspect, or substantial built-in graphics. These features in Scratch made it possible to make a cool game faster than in Khan Academy.

This post describes steps for fixing a robot that is broken. Since robots are computers, many of the steps also apply to fixing a broken phone, tablet or PC, although the details differ.

Hardware

The first step is to determine if the hardware itself is working. The hardware subsystems that most commonly fail are 1) the power subsystem 2) the compute subsystem (RAM/CPU/motherboard 3) the long-term storage subsystem (hard drive or SD card) . You want to therefore systematically check each subsystem (in that order) to verify that it is working correctly. Sometimes it is necessary to take the robot apart to check these subsystems. While it is rarely the case, it is possible that dust build-up inside your robot is causing ventilation issues. If you are pulling your robot apart and see a lot of dust, take a second to clear it out! Also, always make sure that when fixing your robot and the power is on, that you are near the e-stop. Be careful when taking the robot apart. Pieces can be very fragile and delicate and aren’t made for outside conditions. static charge can build on you, which is dangerous when getting near power systems to fix them. You should always wear an antistatic wrist strap when fixing power systems to ensure you don’t discharge any static into the robot. When doing this, be sure to keep track of all the pieces you unscrew, and it’s always good to have backups in case. Be careful with unscrewing, you don’t want to strip the screws.

Power Subsystem. The most common symptom of a power problem is that the robot or device just won’t turn on. This could be because the power supply failed, because the battery died, or maybe your wall outlet doesn’t have power. (Batteries have a limited number of charging cycles; they wear out and need to be replaced!) Most robots or devices have some kind of LED that indicates that power is being transmitted to the device, and the location, color, and meanings of these LEDs is often documented in the device spec sheet or on the board itself. For example, the Raspberry Pi Model B+ has two LEDs, labeled ACT (activity) and PWD (Power), which you can see in this picture. If you can’t verify power is coming to the device, get a multimeter and check that the voltage coming from the battery or power supply is what is specified in the datasheet. You might have to take the device apart to access the power supply; you might have to do some digging to find the data sheet. For example, here is the information on the voltages required to power the Raspberry Pi. Don’t forget to check the wall! Maybe the wall outlet isn’t receiving power, and your robot is fine. It’s also possible a short-circuit has occurred, and something is fried. In this case, you’ll most likely need to replace a fuse, wire, etc. One useful way to quickly check for this is the “smell test”: if the inside of your robot smells slightly smokey or burned, you most likely had a short and should look for something that looks burned.

Compute Subystem. The compute subsystem fails less frequently than the other two. However it is next in the debugging process because it is necessary for checking if your long-term storage system is working. Once a computer is powered on, it conducts a power-on self-test (POST). This POST occurs immediately after powering on, before you boot into your operating system (which requires access to long-term storage). It verifies that each of the hardware components of the machine are working. A PC that fails its post will make strange BIOS beep codes, and you need to figure out what the beep codes mean. This requires figuring out exactly which BIOS you have by looking up the specifications for that PC, or opening it up to look at the motherboard. Then the documentation for the BIOS will indicate what the beep codes mean. The Raspberry PI doesn’t have beep codes; instead it has LED flash codes to indicate different errors.

Of course, the best way to get more information about what’s going on with the POST is to plug in a monitor and keyboard into the computer. This may already be true if you are working with a PC, but if you are working with a robot, it may not have a monitor plugged in by default. The BIOS beep codes and LED flashes indicate problems without needing a monitor, but plugging in a monitor will show exactly what’s going on. Most robots support this somehow; for example our MOVO robot has an HDMI port on the bag to plug into. The Raspberry PI Model B+ also has an HDMI port, and many times my students are surprised to realize that you can plug in a monitor and keyboard and suddenly their quadrotor drone is a PC! Once you’ve verified the POST ran, you can be fairly confident the CPU, motherboard, and RAM are working. Or if not, it will indicate what’s wrong and you can try replacing those components.

Long-term Storage Subystem. If the POST tests succeed, the next step is for the computer to boot into its operating system, which is held on the long-term storage device. Hard drives and SD cards are another common failure point. Both have a limited number of read/writes before they will fail. Hard drives can have bad sectors, parts of the disk that are permanently bad, and can fail entirely. You want to verify that your robot is able to read the long-term storage and boot into its operating system. One of the simplest ways to do this is to plug it into a keyboard and mouse and see if it boots up! This allows you to watch the whole boot process and see if it completes successfully and enables checking other problems too, like networking and software issues. Fortunately the hard drive or SD card is relatively easy to replace, if you backed up your data. In many cases there may exist a standard disk image for the system you are working with; for example the SD card in our Kuri robot failed, but we were able to restore it using an image we could download from their website.

Networking

The next major goal is to connect to your robot over the network. You don’t want to start working on this until you are reasonably sure the robot is passing its POST test and booting into its operating system. The robot’s wifi can be configured in either Master mode or Managed mode. In Master mode, it will act as its own wifi hot spot. You will see an SSID on your base station laptop or desktop that corresponds to the robot’s network. In this mode if you connect to the robot (or PC), it will give your base station an IP address. You can find the robot’s IP address by using a tool like nmap to scan, or route to look at the gateway machine, or look at your base station’s logs to find the IP address of the DHCP server. Second, it can be configured in Managed mode, where the robot is looking for a wireless network. Typically in this case it is configured to look for a network with a particular SSID, and this configuration can be stored on the hard drive. Sometimes for commercial robots, there is a process where this is configured via an app. For example, I recently installed the Mysa smart thermostat. First my phone asked which network I wanted the Mysa to connect to. Then, it connected to the Mysa wireless network, where the Mysa itself was the AP master, and neither my phone nor Mysa had internet access. Then the phone tells Mysa what SSID and password to use, and then both devices connect to that SSID (the “house” internet), and can talk to each other. The Kuri robot works the same way. Third, the robot can be configured to use a fixed, static IP address. Many home routers work this way; in this case the static IP should hopefully be documented in the robot’s or device’s documentation. This is the cause of many issues: setting a static IP but the IP of the robot gets dynamically changed via DHCP server. A simple test is to make sure you can ping your robot, and that your robot can ping you. Sometimes the networking problem comes from firewalls. Make sure the firewalls aren’t stopping networking traffic your robot needs, but don’t just take them completely down; a full discussion of this is beyond the scope of this article, but note that we scanned the internet for robots and found a lot! You can configure your base station to use a different static IP address in the same subnet, and they should be able to connect. You can also snoop by using the ARP cache or network snooping tools, but that is beyond the scope of this article. If your network setup uses ethernet cables, make sure there are no issue with the wires. They can break quite easily without it being obvious. You should also be aware of your networking bandwidth/latency. If your robot is jittering around or sending incomplete sensor data, you may need to throttle things or swap from wifi to a wired connection.

Software

Next you want to make sure the software on your robot is working. The details of this are also beyond the scope of this article, but at a high level, you want to make sure that 1) the software to make the robot move started up without errors and 2) that it can connect to whatever it needs to connect to to do its job. Typically this means using tools like ps to show running processes, checking log files to see if there are errors on startup, and writing and running client programs to see if each of the programs is running. The key is to be systematic; check each sensor or actuator subsystem on the robot to verify it started and is working, because you may find other hardware issues in this process. Typically each sensor comes with its own drivers and minipackages for checking if they work, and these drivers are used within a larger system like ROS to connect things together. Always make sure that the drivers for your sensors work on your robot platform first. For example, a common problem on our drones is being unable to connect to the flight controller so the drone can’t send a command to arm or spin its motors, because the USB socket has failed. ROS does not make this debugging process easy, because people often configure their robot to use a single roslaunch file to start the entire robot stack, making it easy to miss errors in one subsystem or another, and hard to audit a substack. For example, to debug our MOVO robot, we manually started the MoveIt stack from the command line to see the error logs and try different configurations to fix problems.

Software Dependencies.

Calibration. A very common source of errors is miscalibration of some kind. For example, our MOVO robot has a Kinect 2 sensor mounted on its head. Even though we carefully calibrated the sensor about one year ago, it got knocked or moved in that time, and when we checked it yesterday it was off by quite a bit. If your robot does not know where its sensors are relative to its base frame, then it cannot effectively use the sensor data. To check this, use a tool such as RVis to visualize the sensor data overlayed on the robot’s model, or use some kind of fiducial such as AprilTags to check the calibration or recalibrate.

Many of us are preparing to virtualize our classes in response to the COVID-19 outbreak that is spreading in the United States. I wanted to share my experiences teaching my Introduction to Robotics class with video lectures last semester. My motivation for doing online lectures last fall had nothing to do with the COVID-19 outbreak. First, I felt that the time I spent preparing for lecture each class had to be spent again each year. Although I reused slides, I still needed to refresh myself on the slides before each lecture and prepare mentally for what I was going to say and when I was going to say it. Second, it seemed that blasting out a lecture at a group of students who are mostly passively was not the best way to use my face time with students. I would of course ask the group questions, and mix in pair-wise exercises where the students talk with a partner. However most of the time was me blasting technical material at them, and I wanted more informal small group interactions in the class period. Third, as we expand our drone program, it seemed useful to expand access to the lecture materials beyond the students who are on Brown’s campus.

My starting point was a curriculum, syllabus, and set of lecture slides that I developed for the course over the past two years. I previously helped out a bit with the Udacity Flying Car Nanondegree, and one of the key lessons from that experience was to focus on small blocks of content. Based on the Sheriden center’s advice, I started out each lecture by defining learning objectives. The verbs in this page were a helpful starting point. I decided to use the edge.edx.org platform to host the online lecture content because they take care of the hosting and it has a framework for hosting videos as well as asking auto-graded questions, and Youtube to host the videos themselves. Defining learning objectives already put a lot more structure into my slides than I had before. (Yes I know I should have been doing this all along!)

Next I divided each slide deck into small chunks that I could cover in two minutes or less. The lectures I created consisted of a short two minute video followed by a few multiple choice questions. A crucial benefit of the online modality is the ability to ask every student questions after each block. Each question can be automatically graded, allowing students to get immediate feedback on what they understand and don’t understand. At first I worried that I wouldn’t be able to ask substantive questions, and it is true the format does not lend itself to extended discussions. However a large lecture itself is hard to have substantive discussions; when it happens, it’s often a few people talking and most people passively listening. Instead I tried to ask multiple choice questions that were tied to the learning objectives for each section.

We experimented with more sophisticated types of questions, such as creating a drag-and-drop question for the different blocks of a PID controller, but ultimately stuck with multiple choice questions. More complicated questions ended up being more work for incremental benefit. We also considered having coding questions for example in some kind of Jupyter notebook. However we ultimately decided that the best way to assess coding was to use our existing projects and assignments, where students program on their own drone, on their own time through structured assignments and projects. This simplified the creation of online lectures since they only needed to assess content knowledge rather than the ability to write programs or derive equations.

I used a screen capture program to record myself talking over the slides. If I made a mistake, I stopped recording and restarted from the beginning. Since each planned video was only two minutes, this process was easier and faster than editing videos later. Merely the act of recording improved things because I would find bugs in my slides, stop and fix them, and then record a clean version. The ability to stop and rerecord when something came out wrong meant that the version the students eventually saw was easier to understand, and more information-dense. As a result I was able to cover the material I wanted to cover in each lecture in far fewer minutes of recorded content than the 80 minutes I had planned for lecture.

Once I had a process set up it took me two to three hours to record content for the 80 minute lecture from my previously existing slides. The first block of time I would use to define learning objectives (which I probably should have been doing before, but wasn’t), as well as create questions. The second block of time I used to record. I then uploaded the videos to Youtube. I used the built-in editing tool to crop the beginning and the end when I was starting and stopping the screen grabber. Then I put them in EdX and entered the questions after each video.

During class, students could choose to build their drone, work on assignments, or watch the lectures. Students would do some of each. I used the class time to answer individual questions, help students debug their drones, and perform checkoffs. I felt that I got to know the individual students better with this in-class style than when I lectured to them, and they got more individual interactions from me. Effectively I used the class time as extended office hours. Of course this sort of interaction is hard to make happen now, remotely, but hopefully this experience will help those who are transitioning their classes online!

The only way to really understand a paper is to implement that paper. Obviously we don’t have time or resources to do that for most of the papers we read. So most of the papers we read, we won’t really understand. But that’s okay!

When reading a technical paper, it is important to first pay attention to the problem the paper is solving. What are the inputs, and what are the outputs? What can the robot do that it couldn’t do before? You do not need to understand HOW the paper is solving the problem in order to understand WHAT problem it is solving. Hopefully the authors of the paper made this clear in the introduction and abstract. When it’s not clear, sometimes I flip straight to the evaluation, not to look at the numbers, but to understand at the end of the day, what the system is doing.

Once you understand what problem the paper is solving, it is time to decide if it is worth digging deeper. Why is this paper important to your research? There are several common use cases. If the paper does not match any of these, it is okay to decide not to read further! You can’t possibly even skim every paper, much less implement them all so you are always trying to decide how much resources to invest in trying to understand them.

First, you could be interested in using this paper as a subcomponent of your own research system. For example, if your research is about making robots understand natural language commands, and this paper is about making robots pick up objects, you may be interested in using the ideas, algorithms and/or code presented in the paper as part of the language understanding system. If this is the case, you want to understand the inputs and outputs to the system and how it will connect to your planned system. You may also want to understand how well the system works; if you are aiming to produce a user study or collect a dataset, you need more reliability than if you are aiming to evaluate in simulation and only use this system for the video.

A second reason is that the paper may be solving a similar problem to the one you are solving, or even the same problem. This situation occurs frequently in NLP when many people work on the same corpus and report results even using the same training/testing split. For example, the GeoQuery dataset has been used by many papers to evaluate semantic parsing on a common task, asking questions about a geographic database. If this is why you are reading the paper, then it is important to understand at a technical level how their approach to solving the problem differs from yours. It is also important to characterize the performance of your work relative to the other paper. Do you outperform them because you are using a more advanced algorithm? Or leveraging more training data? Maybe you perform similarly to them, but you are using less training data or less annotation, or perform better on an interesting subset of the dataset. Ultimately you want to make a determination of whether this paper can be cited in related work as solving a different problem, or if it is solving the same problem and therefore needs to be compared as a baseline to your approach.

A third reason to read the paper is that it may be solving a different problem that is only distantly related to your problem, but using a particular technical tool or approach that you would like to apply to your problem. For example, you might wish to apply a technique that was translating from English to French to translate between English and the symbolic language your robot is using. Then you want to pay attention to the similarities and differences between your problem and the problem this paper is solving. What changes will you need to make to your input to use the technique described in this paper? Do you expect it to work as well? Better? Worse? How long will it take to train or implement?

A final reason to read the paper is to understand open problems in the field, and figure out what problem you would like to work on. This is often the situation when you are starting out in research. There is sort of a chicken-and-egg problem here in that you don’t know what problem to work on so you read the paper, but you will be more effective at reading papers when you know what problem you are working on. This feeling is okay and normal. In this situation, the most important thing to pay attention to is the problem the paper is solving. Ask yourself if it is an interesting problem. What are the important problems in your field? Is this paper describing one of them? Why or why not?

Any of these reasons may lead to your decision to implement the paper (even if an implementation already exists!). If you are using the paper as a subcomponent of your system, you may wish to own the implementation to make sure it works the way you expect and manage dependencies, although more frequently in this situation you may find yourself using the author’s implementation (when it exists!) If you are comparing to this paper as a baseline, sometimes you will implement the baseline algorithm in order to make sure you understand it. It is important, when implementing, to first reproduce the published numbers on the published dataset before trying it on your new dataset. If you are using an established dataset, many times people will not even run the algorithm on that dataset but use the previously published numbers. If you are using the method as a subcomponent of yours, you will frequently find yourself implementing it yourself in order to deeply understand the method and integrate it with your system. If you are not sure what to work on, implementing an existing seminal paper is a great way to get started. It helps you understand the core tools if your field, and often leads to a new paper of your own when you fix a problem, identify a new use case, or define a new, related problem.

We are lucky enough to have a pack of AIBOs in our lab, leftover from Robocup days of old. My student John Oberlin resurrected them, and we use them as one of our outreach platform.  Another student, Eric Rosen, just had a tutorial session teaching students how to use them for outreach.  My AIBO, Ella, is my main outreach platform that I use when I go into classrooms of very young children.  I just finished four hours of robot activities at my son’s kindergarten classroom with Ella.

AIBO was invented by Sony.  The first consumer model retailed in 1999 and retailed for $3,000.  Too much for the consumer market to really take off.  New models were released every year until 2006, when they were discontinued.  Now they are robot antiques, and new ones are on ebay for $6,000, although you can get a used one for $500.   Our AIBOs are equipped with a rich sensor package, including a camera on the snout, a microphone, two IR sensors, and joint encoders.  They are fully quadrupedal and each leg has three degrees of freedom.  Additionally you can actuate the ears, the tail, and the mouth, and the robot has a speaker for playing sounds such as barks.  You can program the robot to sit, to lie down, to walk forward or backward, to turn left and right, wag its tail, or bark, even opening its mouth.  It was added to the CMU Robot Hall of Fame because it “represents the most sophisticated product ever offered in the consumer robot marketplace.”

Sony shipped around 150,000 AIBOs before discontinuing production.  We are super excited to see the new reboot.

I realized a long time ago that the best age range for me to target for outreach activities is whatever age my son currently is.   (He’s five now!)  So when he got old enough for preschool, I started visiting preschool classrooms of him and his cousins, and the AIBO was a natural choice to take.  The first time I took AIBO to a classroom, it was clear it was a hit.I did the demo during circle time at Jay’s preschool.   The silvery color, the deco rubber hollowed out ears, the wiggly tail, the buttons just capture the essence of robot to a three year old.  As soon as I started unwrapping the robot, the kids crowd around, vying for who gets to turn her on.   We do a programming activity where kids “program” Ella by laying out cards, like “Sit”, “Stand”, “Bark” or “Lie down.”  Then I type in whatever program they did, and then she executes it!  My favorite part is asking the kids to be the robot.  I say “Kindergarten Execute!” and they all pretend to be puppy dogs.

We would like to share this Thanksgiving Pie baked by Rebecca Pankow!