Software Development

Interview coding tests should measure more

There's more to good programming than being able to hack together a solution that "works", even if it's efficient. Consider the security angle when testing your job applicants.

Articles, Weblog postings, and forum discussions about hiring practices and interview techniques tend to draw a lot of attention and spawn lengthy, passionate debates. This is especially true of references to interview questions that people think should be asked in programmer interviews. As pointed out in How do you interview security experts?, though, I believe the sort of questions that should be asked are those for which there may be many "correct" answers, but the way an applicant answers them is the really important factor:

You need questions that invite explanations, not just simple answers. It's harder to explain why an answer is right than to just give that right answer itself. If your candidate gives a "right answer", your question should be phrased so that it requires more than just a few words to satisfy the answer. If the candidate gives a "wrong answer", tell him or her what answer you were expecting and ask why the answer you got was different. Have a conversation about the difference between the given answer and the expected answer, and make your decision based on the quality of the explanation and the thought process behind it, rather than whether the answer was what you expected.

Coding challenges, where you ask a programmer applicant to whip up some quick code on the spot to demonstrate a minimal grasp of software development (and perhaps of the specific language(s) for which you're hiring), may seem to require more objectively answerable questions. A popular subject of debate on the Internet a couple years ago was the FizzBuzz challenge, used to weed out the absolute worst coders applying for a job. Reginald Braithwaite (Internet-famous programmer) made the incredible claim that 199 out of 200 applicants for a programming job can't write code worth a damn, and will probably never be competent programmers.

I agree with Mr. Braithwaite's assessment of the FizzBuzz problem -- that it (or an equivalent programming problem, preferably one that hasn't been solved 600 different ways on the Internet) is useful to help weed out the 199 out of 200 applicants who aren't really programmers, no matter what their resumes say. I agree that, sometimes, such a very basic approach to assessing the most minimal level of competency is necessary before you can move on to more challenging problems. I just think it is a crying shame that the scope of the way this question is asked is so limited.

Let us consider another FizzBuzz level problem that is often used to test minimal programming skill: Fibonacci numbers. The idea is that you write something like this on the whiteboard in the room where you are conducting your interview:

  F(n) = F(n-1) + F(n-2), where F(0) = 0 and F(1) = 1
(The following examples will use Scheme. If you want to follow along on your own computer, running each example and experimenting with it, Ypsilon is an excellent cross-platform compatible implementation of R6RS, the most current iteration of the Scheme standard.)

Then, you say something like "Write a function that solves this problem." Let's assume for argument's sake that you let the applicant choose the implementation language, and he goes with Scheme (Justin James points out that this may be a good sign, in Write a resume that will land you a programming job). If the applicant has encountered this in college at all, he can probably dredge this from memory, as the "naive recursive implementation":

  (define (fib n)

(cond ((= n 0) 0)

((= n 1) 1)

(else (+ (fib (- n 1)) (fib (- n 2))))))

This should get your candidate past the FizzBuzz gate, proving he knows how to implement a clearly specified requirement in code. This question could be so much more, though, with so little effort -- and without eliminating its usefulness for weeding out the most worthless candidates. For instance, you could ask the applicant to write an implementation that is suitable for "real-world use".

This is where the naiveté of that algorithm rears its ugly head, and where most people with any real-world experience would start thinking about efficiency and give you an implementation that doesn't explode in your face when you start trying to calculate large Fibonacci numbers. For example, it calculates the Fibonacci number for an input of 20 pretty much instantaneously, but it hangs for a while for an input of 50, because that's a tree-recursive algorithm that isn't tail-call optimizable. In English, this algorithm increases the number of operations that must be performed at an exponential rate, which also tends to increase the time it takes to calculate an answer at an exponential rate.

If the applicant is asked to produce an algorithm that is suitable for "real world" use, there are a number of ways to improve the efficiency of the algorithm, including memoization, strictly iterative operation, or reordering the thing to take advantage of Scheme's tail-call optimization. A tail-recursive version, with an essentially iterative form, might look like this:

  (define (fib a b count)

(if (= count 0)

b

(fib (+ a b) a (- count 1))))

(define (fibonacci n)

(fib 1 0 n))

Now you're getting somewhere -- right? This is about as far as most developers, and thus most technical interviewers, are likely to get. There's another concern that is just as important, however; security. If you are a programmer, you might notice a potential for impressive failures in practical use of this code. The problem is that calculating Fibonacci numbers assumes you're using positive integer values as inputs, and this code assumes it will only receive positive integer values as inputs too. If you receive input that looks like any of the following, your program is going to blow up in your face, if the above code is all there is for dealing with arbitrary inputs:

  • -3
  • 2.7
  • 10/9

Let us imagine that your applicant, upon hearing you say that this code might need to serve a real-world purpose, decides to implement input validation:

  (define (fib a b count)

(if (= count 0)

b

(fib (+ a b) a (- count 1)))))

(define (fibonacci n)

(fib 1 0 n))

(define (fib-validate n)

(if (and (integer? n) (>= n 0))

(fibonacci n)

#f))

All else being equal, this is the candidate for the job you want to hire. He not only thought about efficiency, but also about security. In fact, in many cases, I would be inclined to select a candidate whose code for "real world" use employed input validation but did not improve the efficiency of the core implementation over an applicant who improved efficiency but did not employ any input validation, like so:

  (define (fib n)

(cond ((= n 0) 0)

((= n 1) 1)

(else (+ (fib (- n 1)) (fib (- n 2))))))

(define (fib-validate n)

(if (and (integer? n) (>= n 0))

(fib n)

#f))

Any reasonably experienced programmer on your team should be able to recognize efficiency issues, and fix them, if needed. Considering how rare it is for coders today to think of security issues first, however, the candidate whose initial instinct is to validate input when you start talking about deploying code in the real world is worth his weight in gold.

(I owe thanks to Sterling Camden, TechRepublic's own IT Consulting guru, for his input while I worked on the basis for this article.)

About

Chad Perrin is an IT consultant, developer, and freelance professional writer. He holds both Microsoft and CompTIA certifications and is a graduate of two IT industry trade schools.

26 comments
mikifinaz1
mikifinaz1

I have passed and failed all sorts of tests and in neither case has it lead to any great insight. I remember in one case I failed a hiring test, OK by me, I just went on to another job. Well, later on this company fell in dire straights. One of the people they hired, I had worked with earlier in our careers; pleaded with them to hire me since he knew me and my rather eccentric and eclectic mix of skills. Well, over and above an enormous amount of grumbling from HR he engaged me and I saved their bacon. THEN they were falling all over themselves to hire me. The unfortunate thing was that in the interim (since their first rejection and hiring me) I had become a consultant and had the enviable position of turning them down and I cited their love of tests as one of my reasons. A clear case of karma, what goes around comes around.

Tony Hopkinson
Tony Hopkinson

while not in scheme is marked on and specifically mentions input validation, even to trapping going out of range on the integer datatype you use. Sometimes we give the same test with other similar examples e.g. factorisation. There's fair few things we look for, but reading the requirement and then being able to explain the why of the implementation are key. We call it our sanity check, and it's identified more than few people who need a rubber room.

apotheon
apotheon

Any reasonably experienced programmer on your team should be able to recognize efficiency issues, and fix them, if needed. Of course, if you're working at the scale of Google, you might want to use a more efficient core algorithm (like my more-efficient version, or something more explicitly iterative), and use memoization, and maybe see if you can find a magic wand to help out too. On the other hand, if you're Microsoft, perhaps neither efficiency nor security is even a concern. You should probably stay away from Scheme at Microsoft, too.

Sterling chip Camden
Sterling chip Camden

By which you mean, "recurse until resources are exhausted and the interpreter or system responds with some punitive action." Non-numeric inputs would get a more immediate negative response. Just in case anyone's lost, in each of the multi-function examples above, Chad intends the last one defined to be the publicly executed function. Thanks for the hat tip, Chad!

Tony Hopkinson
Tony Hopkinson

What's the fifth property down of class X in IDE Y.. Or do this, oh and we've disabled help and intellisense. You can get decent ones, give me a crap test, and I'll asuume it's a crap job. Find the bugs is a good one, testing understanding of fundmanentals as per Apotheon, also a mini project you can prepare in advance and then discuss.

apotheon
apotheon

Most interview tests are poorly constructed and used to measure things they can't reliably measure. Some are actually useful, however. Chances are good you fell afoul of a test that wasn't useful.

Sterling chip Camden
Sterling chip Camden

used to ask candidates to write a simple implementation of a linked list, in any language they chose. You'd be surprised how many self-styled programmers don't even know what that is, much less how to write one. I was often asked to review the submissions. One candidate offered a version in C that was not only correct, it was brilliantly simple. My review consisted of two words: "Hire him!"

apotheon
apotheon

Input validation is far too often overlooked in coding. It's good to hear you're working somewhere that considers it an important part of software development.

Sterling chip Camden
Sterling chip Camden

Google it and most of the results are about corporate schemes, with a couple of Powerpoint schemes thrown in. One notable exception: this article by Don Box on MSDN called Scheme is Love. Of course, the only thing he says about Microsoft's involvement in anything resembling Scheme is: "This melding of code and data is central to all dialects of Lisp, and is fundamental to the way Microsoft is integrating multiple expression languages (most notably SQL) in future versions of the Microsoft? .NET Framework." Pretty lame -- I bet the marketing department made him add that.

apotheon
apotheon

I guess I could have been more clear about which function is meant to actually be executed, with the others being accessed by the "publicly executed" function. Hopefully people won't be lost in general just because of the choice of demonstration language.

jmgarvin
jmgarvin

Is to find out what kind of coder you are hiring. Are you hiring a hacker, a debugger, or something in the middle?

jmgarvin
jmgarvin

I've seen interview tests as trivial as doing a simple select statement in SQL to as complex as solving the Traveling Salesman problem...Yet there seems to be too few in the middle that are actually decent. With that being said, I'm a big fan of interview tests that are: A) Relevant to the job at hand B) Vary in difficulty from trivial to difficult. A simple select is good, something like a left outer join is decent, and maybe a complex stored procedure with triggers is the top of the difficulty. C) With a layered approach, don't expect every question to be answered and also don't expect them to be 100% correct, save for the easier questions. On the difficult questions judge the question answers based on approach, technical skill, and though process...

Tony Hopkinson
Tony Hopkinson

Can't rememeber the last time I had to implement a list raw It's even longer since how it was implemented had any relevance. You want to see puzzled frowns asked the boogers to implement a Deque. Last time I did that was easy 15 years ago.

apotheon
apotheon

$ ypsilon Ypsilon 0.9.6-update3 Copyright (c) 2008 Y.Fujita, LittleWing Company Limited. > For those who don't immediately get the above joke, Ypsilon is an implementation of Scheme, which has linked lists built into the language itself. The "code" sample actually shows what happens when I launch the interactive Ypsilon Scheme interpreter (aka, the REPL) from a shell prompt.

apotheon
apotheon

Yeah, I have yet to see anything remotely LISPy about anything Microsoft does. Of course, LISP doesn't "meld" code and data -- it just recognizes that code is data, and doesn't bother to insert artificial divisions there between code and other types of data. Meanwhile, Microsoft's integration of things went flying out of control somewhere back in the '80s, and trying to paint that as a good thing seems quixotic at best.

Sterling chip Camden
Sterling chip Camden

translated the "registered trademark" character into a question mark. I love it!

apotheon
apotheon

If you want to demonstrate stack overflow, just write a tree-recursive function in Forth with no terminating condition (such as a Fibonacci function that doesn't do input validation; demonstrate stack overflow by giving it a negative number). I mention Forth because it's a stack-based language, so you'll know that the limiting factor will be a stack overflow. Less easily, I guess you could implement your own stack in any language that does recursion.

Tony Hopkinson
Tony Hopkinson

Public int MyInt {get;set;} and it will fill it in for you, I'm struggling to develop the habit though. :( Still the main point was to illustrate what stack overflow is, without having to write Towers of Hannoi with infinite sticks and disks.

Sterling chip Camden
Sterling chip Camden

All the scaffolding required to implement the "wrap every data element in a property" standard allows errors to hide easily in all that meaningless code.

Tony Hopkinson
Tony Hopkinson

Thought I'd done it right for a minute Setter Should be _myInt = value; Don't worry, I've had stack overflow from that one a few times....

Sterling chip Camden
Sterling chip Camden

... when you plan to add validation to the setter. But I agree -- I don't know how many times I've seen properties wrapping every variable in an application that don't add anything but overhead.

Tony Hopkinson
Tony Hopkinson

Learning machine code first meant having to be very familiar with the stack. When people tell me you don't need to know that sort of thing anymore, I just fold my arms and wait for them to figure out what's up with this. class MyClass { private int _myInt = 0; public int MyInt { get { return _myInt; } set { MyInt = value; } } :D

Sterling chip Camden
Sterling chip Camden

Even though a program stack is an extremely useful abstraction, its mechanism is hidden and therefore often misunderstood or taken for granted.

Tony Hopkinson
Tony Hopkinson

Even better, explicitly using a stack you can demonstrate that nice overflow you get when trying to implement infinite recursion (usually inadvertantly :p )

Sterling chip Camden
Sterling chip Camden

How about "use a stack to simulate recursion in a language that doesn't provide it?" These new-fangled languages take all the challenge out of programming. Of course, Lisp has had lists and recursion for 50 years. It's just taken the rest of the world until now to almost catch up.

Sterling chip Camden
Sterling chip Camden

All Lisp dialects come with lists as a primary component of the language. You don't have to write them.

Editor's Picks