Creating Good SDE Interview Questions - Effective Interviewing

This post covers the anatomy of a good interview question and is part of my series on effective interviewing.I have already written about planning ahead for interviews, and a key part of that is having good questions.  There are lots of sites online with questions you can pull from, it is also good to keep track of questions you hear other’s ask, or even questions people ask you when you are interviewing.After you have been interviewing for a while, you should aim to come up with 2-5 questions that are your questions.  Ones that you clearly know how a candidate should do, have seen most of the good/bad/ugly solutions and know when to help and hint.  In general this post is very focused on interviewing software engineers, web developers - you know, the people who write code and build software for a living - although some of the lessons definitely apply to other roles and interviews.When it comes to a picking your questions here are some things that I look for in a truly good interview question:

Tests more than one competency.

You questions should aim to cover multiple different disciplines, or facets of the candidate’s skill set and knowledge.  For example, asking a question like “What project has been your greatest success?” you can learn the background on one of the projects they worked on, as well as a good culture fit question on what motivates them or what they see as a metric for success/accomplishments.  You should also plan to start with that question and ask some follow up questions to get more details, like:

• What size was the team and what was your role?
• What was the duration of the project?
• What technologies and process did you use?
• Looking back would you do anything differently?

For technical questions this can be easier, since most questions naturally probe several areas.  For example any sort of project that involves both choosing a data structure and then coding the solution allows you to evaluate them on two areas.

Easy to understand and explain

Make sure that the question is straightforward.  Ideally it should be something that the candidate is familiar with, and presented in language that is clear and succinct. This is especially true for any question you ask over the phone (since sometimes you could be talking with candidates where English is a second a language).  Ideally (especially for phone) the problem statement should be one sentence.  If you problem involves diagrams to explain it, then I would only ask those in person where it will be easy to verify the candidate understands what you are trying to lay out.Before you get too far though, make sure the candidate clearly understands the problem.  I have certainly gone through enough interviews where we go 5 minutes in before I realize that they are solving a totally different problem than I intended.  A good tactic is to ask them to repeat it back to you or give an example.  You should plan to have some examples to illustrate the problem and solution if the candidate is having trouble.  I know one interviewer who would always ask the candidate to come up with 2 examples and their solutions – that way he could ensure that both of them were on the same page.I always err on the side of simple problems, that don’t involve specific knowledge (for example, I used to always ask about chess boards, or black jack games, but ran into too many candidates that didn’t understand those concepts so I moved to more common objects like elevators and parking meters).

Solvable in 10-15 minutes

When it comes to interview schedules, I like to cover a lot of ground.  This approach gives candidates the best shot, in the event that they are really good, but perhaps have a stumbling block or issues with one of my questions.  As a result I try to aim for questions that are solvable in 10-15 minutes.I know some people who like to spend the whole interview working on one problem, but generally the people that do this ask a question with a lot of sub-problems or angles to it so they are effectively testing different facets of the candidates skill set.   On example of this type of question is to ask someone to design a bit component – like a garbage collector, or stock ticker, and then break it down and go deeper into specific parts (so for the garbage collector this might be the sweep() function, or for the stock ticker this might be API to power the hourly/daily/yearly stock charts).If you have a great question, but it takes too long, try breaking it down into more manageable pieces, and have the candidate focus on one area.  Or have them solve the problem for a limited use case.  Either way, by distilling it down it allows you to spend more time on other things.  After all, if they aren’t getting the solution, you can always give them the option to come back to it at the end of the interview.

Has a brute force, or otherwise approachable solution

The best questions are typically ones that can be solved by most people in a straightforward, or brute force way.  There is a solution that even the “no hire” candidates can discover and code up.  This sort of problem gives candidates the best shot to come up with a solution versus getting stuck when they can’t discover the trick (this is actually the biggest reason I am not a fan of the “puzzle” type questions – most of time they aren’t solvable without an elegant solution).Once the candidate has uncovered the brute force solution, then these problems evolve into more elegant, or more efficient solutions.  One of my favorites of these questions is the permutations question (where the candidate is asked to compute all the permutations of the string).  Since there are n! combinations (where n is the length of the string), it is hard to do much better, and most people can come up with a basic workable solution.  However, as the interview, you can then evolve the problem and say “now assume that the string has many duplicate characters, how do you optimize your code to only execute the minimum number of times?”Other examples are problems that can be solved by looping through the entire data set, but then can be made more efficient using dynamic programming (like the Fibonacci sequence – where the normal recursive one is the brute force answer, the iterative result is more efficient, and then adding dynamic programming/memoziation can make it even more efficient).

The solution is terse and concise – no multi white board answers

Another key thing that I like to look for is problems that have straightforward coding solutions.  I don’t like to have the solution involve lots of language specifics (that would be hard to do without their environment or IDE), or involve a ton of logic.  I try to find problems where the good solution is elegant and relatively trivial amount of code.I know some interviewers that like to see a lot of code in interviews, but I am mostly just trying to determine can they solve a problem and express that solution in a programming language.  I tend to be less focused on syntax, and am looking for logical correctness (although I do want to see that they initialize variables and don’t leak memory).  In general their job won’t be to write any code on white boards, so I don’t want to spend a lot of my precious interviewing time watching them write a ton of code.If you aren’t sure how long a solution should be, solve the problem yourself (on a whiteboard) and see how long the solution is and how long it takes you just to write it.  If it is a lot of code to get it right (when you likely know the solution) then maybe you should scale down the scope of the problem or choose another question.

It has interesting follow up questions. 

Like the previous trait of having an approachable solution that can be evolved into better, more optimal solutions, I also like questions that have interesting follow up questions that will allow you to get more out of your question.Here are some that I like and can be applied to most coding questions:

• Analyze the run time of your solution.
• How could you optimize it and make is faster?
• If this code were to be checked in and deployed to production, how would you change your code?
• What (if anything) would you change to make this solution scale?  Run on multiple servers?
• How would you test this solution?

Each of these questions will help test or assess another part of the candidate’s skill and also serve to make a basic question a little more interesting (which is important for the really great “hire” candidates).Do you have any other suggestions or facets that make interview questions fall into the great category?  Or do you disagree with any of mine?  If so, definitely put them in the comments since I would love to hear your thoughts.