Coders, 3 Year Olds, and Unit Tests

Anyone who has kids knows that three-year olds are starting to learn that the world is full of complex rules. Don’t go on red. Look before you cross the street. You can’t just watch TV all day. What every three year old wants to know is “why?”. I answer “why?” to my kid about twenty times a day. I’ve found that from all the teams I’ve worked on, coders are the same. Tell a coder that they need to fill out a new bug report form or write unit tests and they will ask “why”. Now, like a three year old you can threaten a coder (usually with a bad performance review rather than a timeout), you can bribe a coder (maybe a bonus or award instead of candy, or maybe just candy), or you can convince that it’s in their best interest to it. Like three year olds, it’s best to convince coders that doing something is in their best interest rather than threatening or bribing.

When it’s time to discuss unit testing, what most managers do is bring out some slides about quality and fill them with buzzwords about “quality metrics” or even worse, they bring up some industry standard that promises lots of paper work like CMM or ISO9000. I’ve got to be honest, while almost every coder I know cares about the product and the customer experience, most I know could care less about something like CMM or ISO9000.

So what is the right way to make coders care about unit tests? From my personal experience, unit tests give me four key benefits:

  1. Prove that the API is not broken
  2. Increase speed of execution
  3. Catch issues before they get shipped
  4. Allow me to really understand an API/project

Prove That the API is Not Broken

I really like writing unit tests that test an external API. I can use my tests to ensure that the API is staying stable. This is important not just to catch places where someone adds an argument to a function, but when someone makes a change that modifies the behavior of the API. Once an API ships, people start relying on the behavior of it, whether it was specified or not (wrongly or not). For example, some accomplishments-daemon code will throw a KeyError if you pass in an invalid key to a dictionary. So regardless as to whether this was specified in the documentation, someone may have code that relies on that, and so I test it. (I know that relying on implicit, undocumented behavior of an API is bad, but almost everyone does it.) If the code is changed next week to catch the KeyError instead and return None, the unit tests will at least tell us and we can decide whether at the very least the API version needs to roll.

Increase Speed of Execution

This is my favorite reason for writing unit tests. Well written tests let you isolate testing of one function all by itself and make it easier to debug an algorithm without all the setup and teardown. In a previous job I was writing software to get show listings from a TiVo and download shows. I could not make much progress unless I was in my office on the same subnet with my three TiVo devices. Also these devices were sometimes taken by the test group or marketing for various reasons, delaying my work. So what I ended up doing was writing unit tests and making a fake TiVo device, a mock, that behaved exactly like a real device (from an API stand-point). This not only allowed me to work anywhere, it let me skip the tedious setup, device discovery, and remote debugging required (my code ran on a device other than my dev box). I’d estimate that setting up each debug run would have taken me 5 minutes with a real h/w and it was more like seconds with unit tests. Over the course of three months, this saved hundreds of hours of my time.

When my first release of the TiVo code wrapped up, I was in-between assignments, so I wrote a “season subscription” feature. This feature would periodically scan the device for new shows and download them if they were part of a season that you wanted. By this time, the test team and marketing had taken my TV and all my TiVOs. I ended up spending 3 days writing all the code and only validating with new unit tests. I then just emailed the new installer to the test group and everything worked fine on the hardware.

Catch Issues Before They Get Shipped

This is the one part of unit testing that you will likely hear from a manager armed with a “quality” slide. It is in fact true. Bugs are cheaper to fix earlier, customers are happier, and better yet, bugs or quirks in your API don’t become an implicit part of the API.

Allow Me to Really Understand an API/Project

If you are new to a project and really want to understand how the API works, there’s no better way than writing unit tests. You can really learn the API, the “quirks” of the code, and help the project out at the same time.

Broken Unit Tests

Unit tests are only useful if test failures are taken seriously, and they’re only taken seriously if the build (hopefully continuous) breaks when they fail. Never underestimate the ability of people to forget (intentionally or otherwise) to run unit tests or to ignore failures. The tests I worked on for the Ubuntu Accomplishments Daemon had been broken for months and were so outdated they had to be scrapped completely. The new tests will run during every build which is run at least daily. If the tests fail, the deb is not built.

Ubuntu Accomplishments Daemon Unit Tests

I’ve spend the last week or so writing unit tests for the Ubuntu Accomplishments Daemon. You can see the pile of tests here in the tests.py file. While writing the tests we found about 5 bugs in the accomplishments-daemon, most were small fixes, but most of them also caused the API to abort, which would kill the daemon in a real environment. Making the daemon more resilient will make Ubuntu Accomplishments much friendlier to end-users.

There are still plenty of APIs that are not yet tested, so if you’d like to dive in, please pull the code and look at the to do list at the top of tests.py.

Conclusion

There are entire books written about unit testing, about how to do it, and about why. However, if you’re like me, you’ll need to see the benefits first-hand to really want to write unit-tests. So go read the books, and then give it a try, I promise it will reduce your bug count and improve your speed of execution.

Tagged , , ,

2 thoughts on “Coders, 3 Year Olds, and Unit Tests

  1. Thomas Kluyver says:

    I’d add one point to your list of reasons: having a good suite of tests means you can make changes to the code much more confidently, because you have the reassurance of knowing that if you break something critical, you’ll find out about it quickly. It’s not a guarantee, of course, but it’s a big psychological benefit when you want to work on the code.

    For IPython, we have a script to test a Github pull request and post the results as a comment. We aim to make sure that any code changes pass the tests on Python 2 & 3 before they get merged.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>