The only way to really understand a paper is to implement that paper. Obviously we don’t have time or resources to do that for most of the papers we read. So most of the papers we read, we won’t really understand. But that’s okay!

When reading a technical paper, it is important to first pay attention to the problem the paper is solving. What are the inputs, and what are the outputs? What can the robot do that it couldn’t do before? You do not need to understand HOW the paper is solving the problem in order to understand WHAT problem it is solving. Hopefully the authors of the paper made this clear in the introduction and abstract. When it’s not clear, sometimes I flip straight to the evaluation, not to look at the numbers, but to understand at the end of the day, what the system is doing.

Once you understand what problem the paper is solving, it is time to decide if it is worth digging deeper. Why is this paper important to your research? There are several common use cases. If the paper does not match any of these, it is okay to decide not to read further! You can’t possibly even skim every paper, much less implement them all so you are always trying to decide how much resources to invest in trying to understand them.

First, you could be interested in using this paper as a subcomponent of your own research system. For example, if your research is about making robots understand natural language commands, and this paper is about making robots pick up objects, you may be interested in using the ideas, algorithms and/or code presented in the paper as part of the language understanding system. If this is the case, you want to understand the inputs and outputs to the system and how it will connect to your planned system. You may also want to understand how well the system works; if you are aiming to produce a user study or collect a dataset, you need more reliability than if you are aiming to evaluate in simulation and only use this system for the video.

A second reason is that the paper may be solving a similar problem to the one you are solving, or even the same problem. This situation occurs frequently in NLP when many people work on the same corpus and report results even using the same training/testing split. For example, the GeoQuery dataset has been used by many papers to evaluate semantic parsing on a common task, asking questions about a geographic database. If this is why you are reading the paper, then it is important to understand at a technical level how their approach to solving the problem differs from yours. It is also important to characterize the performance of your work relative to the other paper. Do you outperform them because you are using a more advanced algorithm? Or leveraging more training data? Maybe you perform similarly to them, but you are using less training data or less annotation, or perform better on an interesting subset of the dataset. Ultimately you want to make a determination of whether this paper can be cited in related work as solving a different problem, or if it is solving the same problem and therefore needs to be compared as a baseline to your approach.

A third reason to read the paper is that it may be solving a different problem that is only distantly related to your problem, but using a particular technical tool or approach that you would like to apply to your problem. For example, you might wish to apply a technique that was translating from English to French to translate between English and the symbolic language your robot is using. Then you want to pay attention to the similarities and differences between your problem and the problem this paper is solving. What changes will you need to make to your input to use the technique described in this paper? Do you expect it to work as well? Better? Worse? How long will it take to train or implement?

A final reason to read the paper is to understand open problems in the field, and figure out what problem you would like to work on. This is often the situation when you are starting out in research. There is sort of a chicken-and-egg problem here in that you don’t know what problem to work on so you read the paper, but you will be more effective at reading papers when you know what problem you are working on. This feeling is okay and normal. In this situation, the most important thing to pay attention to is the problem the paper is solving. Ask yourself if it is an interesting problem. What are the important problems in your field? Is this paper describing one of them? Why or why not?

Any of these reasons may lead to your decision to implement the paper (even if an implementation already exists!). If you are using the paper as a subcomponent of your system, you may wish to own the implementation to make sure it works the way you expect and manage dependencies, although more frequently in this situation you may find yourself using the author’s implementation (when it exists!) If you are comparing to this paper as a baseline, sometimes you will implement the baseline algorithm in order to make sure you understand it. It is important, when implementing, to first reproduce the published numbers on the published dataset before trying it on your new dataset. If you are using an established dataset, many times people will not even run the algorithm on that dataset but use the previously published numbers. If you are using the method as a subcomponent of yours, you will frequently find yourself implementing it yourself in order to deeply understand the method and integrate it with your system. If you are not sure what to work on, implementing an existing seminal paper is a great way to get started. It helps you understand the core tools if your field, and often leads to a new paper of your own when you fix a problem, identify a new use case, or define a new, related problem.