Monthly Archives: August 2022

Teaching Kids to Program Without Screens

Teach your very young children to program without screens, and without literacy!I created a game called “Ready Robots.” Any kid can play as long as they are old enough to do “if you are happy and you know it clap your hands.” They don’t need to be able to read. You can make the game yourself by drawing your own cards, or you can download and print my cards.

To play, ask kids to lay out the cards in a particular order, then say “ready robots? Execute!” Then you and the kids should do each action in the sequence in order. Take turns making different programs for different people. Kids love making a program for their grownups to execute. See a video of the game in action!

I’ve played this game with kids for years. Even very young children can do it by imitating you and looking at the pictures on the cards. It can even build literacy skills by pairing words on the cards with actions. When I add a real robot, like an Aibo robot dog or Sphero or Baxter, I have the kids make the program with cards, then quickly type it into an Ein command line, to make the physical robot execute it. When using an Aibo robot dog, I first ask them to run the program (Sit! Wag your tail! Bark!) and then we have the robot do the same program.

For older children, you can add “if this then that.” Make little programs like “if I stomp my feet then you beep your nose.” Put the cards up on a white board or wall and add more and more rules. It is kind of like Simon Says but more confusing because you might have 4 or 5 different rules.

Most Robots Can’t Pick up Most Objects Most of the Time

Most robots can’t pick up most objects most of the time. This is for a variety of reasons. The robot is turned off. The object is out of reach. The object won’t fit in the gripper. The robot doesn’t know where the object is, because it doesn’t have a sensor at all, or because the sensor isn’t aimed at the object, or because if it is aimed at the object, the object is transparent or reflective. Or if it is aimed at the object, and it is easy to see, it might not have a detector or pose estimator for that object that can localize and predict grasps accurately enough to pick things up.

Existing work by Robb Platt, Ken Goldberg, Sergey Levine, and Chelsea Finn is working towards more robust grasping. But doing things on real robots in the real world is hard, and this won’t change any time soon. What is needed is an integrated perception/planning/motion stack with POMDPs and learned object models that can run on a robust robot with a capable, flexible general-purpose manipulator.

Computer Science Activities for the Pandemic Parent

I am a computer science professor and also a mom and an aunt. I have taken the opportunity of the COVID pandemic to take a deep dive into computer science with my son and his cousins, ages 7-11. I wanted to share some of the things we’ve tried and what worked and didn’t work.

Scratch Club

One of the first activities we started, right after schools closed, was Scratch Club. We met several times per week for one hour. The first 10-30 minutes, the kids take turns showing each other their programs. They screen-share to demo their program and explain how it works. They get feedback and ideas from the other kids. We talk about giving constructive criticism, calling attention to neat features of their programs, and getting them to say what they plan to make their program do next. The rest of the time, they program on their own, on whatever program they want. I have them split into two separate calls with a more advanced and less advanced group. Then I check in which each student individually and try to help them with their programs. Sometimes I do individual instruction/teaching. Sometimes I help them find an online tutorial to do. (We really like Warfame and Griffpatch!) If they get stuck in the tutorial, I help them find bugs. I’ve observed some really beautiful peer-teaching, where one of them helps another with a problem. This peer teaching is really good for both, because they are practicing teaching/communication skills, as well as programming.

Unfortunately the Scratch programming framework makes several big technical blunders. First they do not provide functions. It is not possible to write a function that returns a value; there are only blocks that take actions. You can hack around this by defining variables to hold the return value, but is this really the sort of programming we want to model? Second, they do not let you have lists of lists. For example, we wanted to do tic-tac-toe and the natural representation is a list of lists to represent a board, but Scratch does not allow this.

Snap is a similar framework that addresses all these problems. But what Scratch gets really really right is the social angle. All our friends use Scratch and not Snap. All the best tutorials use Scratch and not Snap. Scratch has a large user base and the coolest games are really really cool. (Check out Griffpatch’s Cloud Platform Multiplayer game!) And you can remix all those cool games and make your own variations. So together these features tied us to Scratch, even though we poked around at Snap a bit.

What surprised me was that in many cases these limitations in Scratch turned into opportunities to discuss CS concepts we would not have gotten to otherwise. Scratch doesn’t let you store strings or lists as cloud variables. So the kids learned to encode and decode strings into integers, which was super cool to see them doing.

Turing Tumble

Turing Tumble is a toy for programming with marbles. We played with it a lot when he was at a younger age, (and I had to help a lot.) But then I got asked “how do we program the computer to do addition? if we don’t already have addition?” Turing Tumble is a great way to answer this question, because it goes step by step how to build an adder using a marble computer. The curriculum is very well designed so that each puzzle builds on the next. This activity is also nice because it does not require a screen.


Unity is a widely used framework for making 2D and 3D games, so there are tons of tutorials and resources available on the web. It can be used to make phone and tablet games. I really like the idea of teaching kids the tools that are used to program games they already know and like. It turns them from a passive consumer of games to a producer, which is great not only because they can make games, but because they can look at the games they already play with a critical and creative eye. It also exposes kids to a compiled language, and the idea of rigid body physics. The bad: it is really complicated. It requires a heavyweight machine – no Chromebooks. And it’s easy for mysterious things to go wrong that mysterious check boxes fix. My son is having fun but it takes a pretty big help from me to get things going and fix things when he is stuck. There are lots of fun Youtube videos about how to make Unity games, although following the step-by-step tutorials is somewhat tricky as the Unity versions keep changing. Don’t go this route unless you are willing to do heavy lifting to help when things get stuck.

Khan Academy

This was quite good as far as it went, but it has a ceiling. The Khan Academy Javascript lessons require no software installed and run right in your web browser. It was completely internationalized so my son was doing the lessons in Polish, which was cool to see. The graphics angle was a good hook, and my son enjoyed the visual effects he could create. It was also a good introduction to a text-based language. However the ceiling is rather low; the coolest games in the Khan Academy ecosystem were not as cool as the coolest games in Scratch. They also don’t have a sharing/social aspect, or substantial built-in graphics. These features in Scratch made it possible to make a cool game faster than in Khan Academy.

Learning to Follow Directions With Less Supervision

A person should be able to give a complex natural language command to a robot drone or self-driving car in a cityscale environment and have it be understood, such as “”walk along third street until the intersection with main street, then walk until you reach the charles river.” Existing approaches represent commands like these as expressions in Linear Temporal Logic, which can represent constraints such as “eventually X” and “avoid X”. This representation is powerful, but to use it requires a large dataset of language paired with LTL expressions for training the model. This paper represents the first ever framework for learning to map from English to LTL without requiring any LTL annotations at training time. We learn a semantic parsing model that does not require paired data of language and LTL logical forms, but instead learns from trajectories as a proxy. To collect trajectories on a large scale over a range

We release this data as well as the data collection procedure to simulate paths in large environments. We see the benefits of using a more expressive language such as LTL in instructions that require temporal ordering, and also see that the path taken with our approach more closely follows constraints specified in natural language. This dataset consists of 10 different environments, with up to 2,458 samples in each environment, giving us a total of 18,060 samples. To the best of our knowledge this is the largest dataset of temporal commands in existence.

You can read the paper here!