A theme that has come up rather often in my career is how to test candidates for developer positions. Most of the tests that I have seen have been pretty bad; many of them are language-specific, and feel more like tests of the candidates’ ability to memorize libraries rather than their ability to program. All too often, companies find themselves hiring people who can spew out reams of knowledge about various frameworks, but who fail even the simple FizzBuzz test.

I am part of the hiring process again, and the topic came up to try to find a good test for candidates. When I first looked at Codility I was dubious, but now I am sold.

What makes Codility different

Codility allows you to pick and choose a number of different programming exercises to give to your candidates. They complete the small applications, and the system runs them through with what amounts to a number of unit tests, and provides you with a score from 1 to 100 (with 100 being the best). Along with the score is an exact explanation of how that score was determined and the code that was submitted. Each question is timed and limited to 30 minutes, and it is suggested that you do not use more than four questions (which is good advice in my opinion). There is a wide range of languages available for the candidates to use (including Java, C#, Ruby, and PHP), and you may restrict which languages the candidates can use based on your needs.

My first impression of Codility

At first, I was leery of the tests; the “medium” tests seemed fairly trivial, and the “easy” tests looked like assignments from my first month or two of programming classes in my freshman year of high school. I gave it a shot anyway, and my opinion quickly changed.

I keep forgetting just how unprepared many developers — even those who have been working for some time — are for actually writing software. Not only has Codility done a great job at filtering out bad candidates, but it has given me insight into how candidates approach development. Looking at how someone tries to solve a problem, regardless of whether they arrived at a working solution, is an invaluable tool. Codility does not do anything that I could not do myself during the interview, but it is a great way to keep bad candidates from getting to the interview stage.

One issue with Codility

My one issue with Codility is the nature of most of the problems. Because it uses automated scoring, the kinds of programs that lend themselves well to Codility are often math-oriented. While someone in a very traditional Computer Science program should have no problem with that, many people who are self-taught or went to a less traditional Computer Science program may spend too much time just trying to understand the question. There were a few questions that had me scratching my head unnecessarily.

How to get the most out of Codility

I highly recommend that you keep the questions to a maximum of four. Originally I thought the 30 minute time limit was far too generous (I would be shocked if any of the “easy” questions took me more than a few minutes to complete), but after putting a few people through tests, I think the 30 minute time limit is sensible. It is easy for me to forget that I have been developing software for quite a while, and that I always have done well on these types of “Comp Sci 101” questions.

I also suggest that you stick with the “medium” and “easy” questions for entry-level developers. The “hard” questions will frustrate them.

Another idea is to do two rounds: one with two or three “easy” questions and one “medium” question to filter out the bottom of the barrel folks, and another with a mix of “medium” and “hard” questions to separate the better from the average folks. I admit that I haven’t tried this idea out yet.

Finally, read the questions. If you cannot easily understand what the question is asking, your candidates may get hung up on it too.


Keep your engineering skills up to date by signing up for TechRepublic’s free Software Engineer newsletter, delivered each Tuesday.