IT Employment

Test developer candidates with Codility

Codility does a great job at filtering out bad developer candidates, and it will give you insight into how candidates approach development.

A theme that has come up rather often in my career is how to test candidates for developer positions. Most of the tests that I have seen have been pretty bad; many of them are language-specific, and feel more like tests of the candidates' ability to memorize libraries rather than their ability to program. All too often, companies find themselves hiring people who can spew out reams of knowledge about various frameworks, but who fail even the simple FizzBuzz test.

I am part of the hiring process again, and the topic came up to try to find a good test for candidates. When I first looked at Codility I was dubious, but now I am sold.

What makes Codility different

Codility allows you to pick and choose a number of different programming exercises to give to your candidates. They complete the small applications, and the system runs them through with what amounts to a number of unit tests, and provides you with a score from 1 to 100 (with 100 being the best). Along with the score is an exact explanation of how that score was determined and the code that was submitted. Each question is timed and limited to 30 minutes, and it is suggested that you do not use more than four questions (which is good advice in my opinion). There is a wide range of languages available for the candidates to use (including Java, C#, Ruby, and PHP), and you may restrict which languages the candidates can use based on your needs.

My first impression of Codility

At first, I was leery of the tests; the "medium" tests seemed fairly trivial, and the "easy" tests looked like assignments from my first month or two of programming classes in my freshman year of high school. I gave it a shot anyway, and my opinion quickly changed.

I keep forgetting just how unprepared many developers -- even those who have been working for some time -- are for actually writing software. Not only has Codility done a great job at filtering out bad candidates, but it has given me insight into how candidates approach development. Looking at how someone tries to solve a problem, regardless of whether they arrived at a working solution, is an invaluable tool. Codility does not do anything that I could not do myself during the interview, but it is a great way to keep bad candidates from getting to the interview stage.

One issue with Codility

My one issue with Codility is the nature of most of the problems. Because it uses automated scoring, the kinds of programs that lend themselves well to Codility are often math-oriented. While someone in a very traditional Computer Science program should have no problem with that, many people who are self-taught or went to a less traditional Computer Science program may spend too much time just trying to understand the question. There were a few questions that had me scratching my head unnecessarily.

How to get the most out of Codility

I highly recommend that you keep the questions to a maximum of four. Originally I thought the 30 minute time limit was far too generous (I would be shocked if any of the "easy" questions took me more than a few minutes to complete), but after putting a few people through tests, I think the 30 minute time limit is sensible. It is easy for me to forget that I have been developing software for quite a while, and that I always have done well on these types of "Comp Sci 101" questions.

I also suggest that you stick with the "medium" and "easy" questions for entry-level developers. The "hard" questions will frustrate them.

Another idea is to do two rounds: one with two or three "easy" questions and one "medium" question to filter out the bottom of the barrel folks, and another with a mix of "medium" and "hard" questions to separate the better from the average folks. I admit that I haven't tried this idea out yet.

Finally, read the questions. If you cannot easily understand what the question is asking, your candidates may get hung up on it too.

J.Ja

Keep your engineering skills up to date by signing up for TechRepublic's free Software Engineer newsletter, delivered each Tuesday.

About

Justin James is the Lead Architect for Conigent.

20 comments
EHankinson
EHankinson

Hello, I am one that has been victim to this thing. Even after this discussion this test was still used in a hiring process I recently went through. Do these questions seem like entry level developer questions? I have an associates in Computer Information Systems and have not seen anything similar to this before the test.

Slayer_
Slayer_

[b] A zero-indexed array A consisting of N integers is given. An equilibrium index of this array is any integer P such that 0 P < N and the sum of elements of lower indices is equal to the sum of elements of higher indices, i.e. A[0] + A[1] + ... + A[P1] = A[P+1] + ... + A[N2] + A[N1]. Sum of zero elements is assumed to be equal to 0. This can happen if P = 0 or if P = N1. For example, consider the following array A consisting of N = 7 elements: A[0] = -7 A[1] = 1 A[2] = 5 A[3] = 2 A[4] = -4 A[5] = 3 A[6] = 0 P = 3 is an equilibrium index of this array, because: A[0] + A[1] + A[2] = A[4] + A[5] + A[6] P = 6 is also an equilibrium index, because: A[0] + A[1] + A[2] + A[3] + A[4] + A[5] = 0 and there are no elements with indices greater than 6. P = 7 is not an equilibrium index, because it does not fulfill the condition 0 P < N. Write a function Private Function equi ( A As Integer() ) as Integer that, given a zero-indexed array A consisting of N integers, returns any of its equilibrium indices. The function should return 1 if no equilibrium index exists. Assume that: N is an integer within the range [0..10,000,000]; each element of array A is an integer within the range [2,147,483,648..2,147,483,647]. For example, given array A such that A[0] = -7 A[1] = 1 A[2] = 5 A[3] = 2 A[4] = -4 A[5] = 3 A[6] = 0 the function may return 3 or 6, as explained above. Complexity: expected worst-case time complexity is O(N); expected worst-case space complexity is O(N), beyond input storage (not counting the storage required for input arguments). Elements of input arrays can be modified. [/b] My eyes glossed over trying to read this.... I suppose my first problem is I don't know what an equilibrium indices is. I guess I suck at programming. Or probably reading comprehension. I tried this [b] try return 6 catch return 3 end try [/b] I figured it should be right at least half the time, but it gave me a zero :(

waltersokyrko
waltersokyrko

Software development in the real world requires many skills: defining, documenting, and approving requirements; designing user interface; designing software; integrating software into existing system; writing software; testing software; debugging software. Writing software is often a small part of the overall project. In my last project, writing code took 1 week out of a 3 month project. Yes, knowing how to write code is important. Focusing on writing code when hiring a software developer may not get the best candidate.

xcxcxcxcxc
xcxcxcxcxc

I did their demo assignment, and it was indeed a nice challenge. It took me a couple of minutes to get used to the interface, the way to use the tests, and the response. I am quicte used to Eclipse and I am very spoiled by the auto complete and other handy little things. Obviously, none of this is available, you have to do it all alone. But that is not a bad thing! For a next hire I will defenitely use this product.

RMSx32767
RMSx32767

Supports, per their website: Java, C++, C, C#, Python, PHP, Ruby, JavaScript, Pascal, Objective-C, Perl, VB.NET, Lua, SQL Pascal? There are some folks who will state, quite forcefully, that Pascal was never meant for the real world. No BASIC, Fortran, CoBOL, APL, Ada, PL/1, etc? There is much so-called legacy code in need of support. Care to share some of the easy, or medium, questions?

phscnp
phscnp

These tests are necessary, but should not be put before a phone interview. The interview process is a two way street. The candidate wants to know what you really need. Over the years I have had many short and sweet (under 3 minute) phone interviews. At that point I was able to say, "Thank you. I'm not the guy you're looking for, and here is why." To do that, I had to talk, yes talk, to the person doing the hiring. Not the recruiter, not a test. Requiring a half hour investment up front (4x30 minute questions is half an hour for a qualified candidate) eliminates candidates who budget their time. Take a test and learn nothing about the position, or go outside and play? Easy choice.

Justin James
Justin James

... some of the questions were not good, because it is easy to get hung up on the question itself... this is an example of one such question. :( J.Ja

andrew232006
andrew232006

Unfortunately those answers are only right for the example and the actual answer can be in the range [2,147,483,648..2,147,483,647]. But if that try catch loop works you may have a way to hack through many of the questions. You need to find the array index where the sum of all the elements before it matches the sum of all the elements after it. If A[0] + A[1] + A[2] = A[4] + A[5] +A[6] the answer is 3

RMSx32767
RMSx32767

Try? There is no "try", there is "DO WHILE SUCCESSFUL", or "REPEAT UNTIL FAIL"

RMSx32767
RMSx32767

Been there myself; face to face first interview which lasted not very long. I had the skills and could do the job but it was clear after the intro by interviewer I was not the person they were seeking, and I said so. I was thanked for my honesty and that was it.

Slayer_
Slayer_

Quick, before any company actually uses this thing.

Slayer_
Slayer_

I'll have to think about that.... -Edit This is what I figured out, I don't like that it doesn't include the current index. I think the real answer is 5, but to get 3 I exclude the current index in the math. I wrote in in VB6 just cause I had the IDE open at the time [b] Option Explicit Private Sub Form_Load() Dim aTest(0 To 6) As Double Dim i As Double Dim iAnswer As Double aTest(0) = -7 aTest(1) = 1 aTest(2) = 5 aTest(3) = 2 aTest(4) = -4 aTest(5) = 3 aTest(6) = 0 iAnswer = 1 For i = LBound(aTest) + 1 To UBound(aTest) - 1 Step 1 If SumArray(aTest, LBound(aTest), i - 1) = SumArray(aTest, i + 1, UBound(aTest)) Then iAnswer = i Exit For End If Next i Call MsgBox(iAnswer) End Sub Private Function SumArray(ByRef arr() As Double, iFromIndex As Double, iToIndex As Double) As Double Dim i As Double Dim sum As Double For i = iFromIndex To iToIndex Step 1 sum = sum + arr(i) Next i SumArray = sum End Function [/b]

Slayer_
Slayer_

Do loop until 1= 2 Yay! If I put that it, I wonder if the web service would hang trying to get a result from that.

bedeA
bedeA

@climons Yes, this is a correct answer, but it has the worst case complexity O(N).

Slayer_
Slayer_

Can you write some code for it? I'd like to see what you mean.

climons
climons

yes, of course - depending on array size. But we can easily convert it to linear sequence with the same params set - no big deal...

Slayer_
Slayer_

You get points for geek factor, but lose points for the inherent problem recursion has, you risk running out of stack space.

climons
climons

really fits into single recursive function. It's like imaginary slider moving from the start to the end of an array. We have 2 sums - Before and After the slider index. Every time slider moves, value of its element is added to the Before sum and is taken from After. Naturally, at the start Before = 0, After = sum of all the elements. That the only time we need to calculate the sum of number of the elements. After that would be only single subtraction/addition. Function keeps calling itself with parameters of slider index and Before and After sums. The function compares sums, increments number of equilibrium indices, if they're equal, and adds slider index value to resulting collection, again, if satisfied. Slider index is incremented, Before and After sums are updated as well. Then the function is called again from itself. Condition of leaving recursion is when slider index reaches the end of an array.