Technical Interview Questions for Software Development Managers

February 2, 2012

One of the more popular posts on this blog has been the one on <a href="http://katemats.com/2011/12/15/interview-questions-for-software-development-managers-and-leaders/">how to interview your manager</a>.And one of the questions I am often asked is how technical does a VP engineering, or software development manager need to be if they aren’t writing coding day to day? And moreover how do you assess that in an interview?This post outlines an interview question that could take 1-2 hours depending on the depth of the candidate (and of course the interviewer). However, I think it does a good job covering technical skills, and more importantly how those technical skills play into the real role of a software development manager or leader.<em>P.S. Like all interviews though, this is not a one size fits all so please be sure to follow up on the questions and facets relevant to your company and role.</em> <h1><strong>Evolution of a Product Problem</strong></h1><h2><strong>Step 1: Stage the ambiguous BIG idea</strong></h2>Pick a big software system with relatively clear functionality. Ideally something you understand, but that neither you nor the candidate have a ton of expertise in (that way you should clearly be able to assess is this person way better than I would be – which should be something true of a manager). Some of my favorites are:<ul><li>An image hosting application (like <a href="http://www.flickr.com/">Flickr</a> – people upload images and others view them)</li><li>A cloud storage system (like <a href="http://docs.amazonwebservices.com/AmazonS3">Amazon’s S3</a> – people upload files and access them)</li><li>A simple ecommerce website (like <a href="http://www.lululemon.com">Lululemon</a> – one vendor of products, relatively simple check out process)</li></ul>But you can really choose anything as long as (a) the candidate can understand the functionality easily (you don’t want to spend the interview explaining the nuances of the problem) and (b) it has the potential to grow in terms of traffic, usage, items, data, etc.<h2><strong>Step 2: Ask them to define it.</strong></h2>Once you have explained the big problem, throw the ball back into their court to define and articulate the requirements. This is a great way to get a sense of their general product know-how, and intuition on product design and implementation.For the sake of this article, let’s say we chose the image-hosting example. So in this case I would say something like:<blockquote>Imagine you were tasked with managing the creation of an image hosting service like Flickr, or Photobucket. What are the high-level use cases and requirements for such a service?</blockquote>As they answer pay attention to how the approach the problem. Do they start with the customer first? What kind of questions do they ask up front? If you tell them to ignore the wireframes and design and just focus on APIs, what are the interfaces they define for the service? What parameters do they specify? Do they think about security or external/internal consumption of those APIs? Feel free to ask probing questions and give suggestions until you are happy with the result.So for the image-hosting example their answers might be something like the following:Upload content:<pre><strong>POST</strong> <a href="http://api.site.com/services/upload/">http://api.site.com/services/upload/ </a><strong>Arguments:</strong> photo file, title, description, tags, content-type<strong>Return value:</strong> photo id or error code</pre>Updating existing files:<pre><strong>POST</strong> <a href="http://api.site.com/services/update/">http://api.site.com/services/update/</a><strong>Arguments</strong>: photo id, photo file<strong>Return value:</strong> success or error code response</pre>Fetch or Retrieve the file:<pre><strong>GET</strong> <a href="http://api.site.com/services/get">http://api.site.com/services/get </a><strong>Arguments:</strong> photo id<strong>Return value:</strong> photo or error code response</pre>Delete a file:<pre><strong>POST</strong> <a href="http://api.site.com/services/delete">http://api.site.com/services/delete </a><strong>Arguments:</strong> photo id<strong>Return value:</strong> success or error code response</pre><h2><strong>Step 3: How does the system actually work?</strong></h2>Once you've gone through the requirement problem you got a feel for the candidate's ability to identify customer requirements it is time to move onto the actual technical design (which is what you are probably used to for senior engineer system design type questions).Using their interfaces from Step 2, ask the candidate to actually draw out the logical pieces of the system needed to support these requests. You should cover the following things:<ul><li>How are requests services?</li><li>Where is data is stored?</li><li>What needs to be backed up and when?</li><li>How do these different pieces interact together? What are the services needed to support the system?</li></ul>At this point, they have probably drawn an architecture diagram that may look something like the following:<a href="http://katemats.com/2012/02/02/technical-interview-questions-for-software-development-managers/imageservicesbackup/" rel="attachment wp-att-1004"><img class="size-medium wp-image-1004 alignnone" title="simple service with redundant storage" src="http://katemats.com/wp-content/uploads/2012/02/ImageServicesBackup-400x202.png" alt="simple service oriented system architecture with redundant storage layer" width="400" height="202" /></a><h2><strong>Step 4: Scaling </strong></h2>Then comes the good part of the problem. With their diagram ask them about scaling and growth.<ul><li>Where are the current bottlenecks?</li><li>How would things need to change to support 1000x of the amount of traffic?</li><li>How could you support more customers? Would you need more security and authentication?</li><li>What are some different partitioning schemes?</li></ul>Things to look for as hot spots are any place where you have a lot of incoming requests (client uploads and requests for images), outgoing requests (think of all the places you might want caches), and where data can grow beyond the capacity of one server (image files for example). Also, for long running jobs like uploads, you will want to make sure they understand one server can only maintain so many open connections at a time, etc. You get the idea.Asking these questions will give you a good feel for the candidate’s knowledge and ability to diagnose and understand potential bottlenecks and issues as the services grows (which hopefully your company is doing!). It can also be a good indicator if the candidate really understands the system architecture and the way the pieces go together and support one another.Maybe their diagram will end up something like this one:<a href="http://katemats.com/2012/02/02/technical-interview-questions-for-software-development-managers/imageservicespartition/" rel="attachment wp-att-1005"><img class="alignnone size-medium wp-image-1005" title="ImageServicesPartition" src="http://katemats.com/wp-content/uploads/2012/02/ImageServicesPartition-400x221.png" alt="service oriented architecture scaling with part ions" width="400" height="221" /></a><em>Typically for great well-rounded candidates this first 4 steps will take almost an hour to go through in detail. They will be asking good questions, clarifying use cases (be prepared for that), and talking you through tradeoffs and potential solutions to problems.</em><h2><strong>Step 5: Building </strong></h2>Once you're confident that they've done a great job with the architecture then it is time to move on to an important skill any software manager needs to nail – how to build a system. I like the follow up with something like the following:<blockquote>Imagine you're tasked with actually building this system.</blockquote><ul><li>How would you staff the team?</li><li>What technologies would you use and why?</li><li>Which parts would you build first and in what order?</li><li>How long do you think it would take to get a basic version shipped? How did you determine that estimate?</li></ul>While there are certainly an infinite number of ways to build and approach this problem, you are really looking to dive into their thought process and philosophy around resourcing and staffing projects. You want to understand the way they structure and think about software and the build out of a complex system.I once had someone tell me, “Well I would let my senior engineer make those decisions”, and maybe that would be true, but push the candidate to go through the full exercise as if they didn’t have a senior engineer and had to make those decisions on their own.A great follow-up on this question is to ask the candidate if they had to build the system as fast as possible what do is the fastest they could build and launch it? How much time would it take? How many people would they need to make this happen? What are the greatest areas of risk?This line of questions can provide a picture of how pragmatic and realistic the candidate can be, and perhaps some of their creative resourcing ideas. Hopefully they will pepper their answer with anecdotes from their past experience, and if not you can always prompt them and ask, “Have you ever tried that before? How did it go?” or “How to ensure your team hits deadlines and ships software on time?”<h2><strong>Step 6: Deployment and Releases</strong></h2>Once you've gone through a basic resourcing and costing exercise in step 5, a great follow up is to get a feel for how they would actually handle the deployment and launch of such a system built from scratch. There are two sections to this step – the infrastructure and the process.<h3>Step 6a: The Infrastructure</h3>In this portion of the interview you are trying to determine how well your candidate understands hardware, storage, and general costs of operating and maintaining a large software website.A great line of questioning that works well is as follows:<ul><li>What is the minimum amount of hardware needed to operate this system?</li><li>As you scale where would you add hardware first?</li><li>What is the incremental cost at each step?</li><li>Are there optimizations you could make to reduce your hardware spend? (good answers are things like compression, archival/cold storage, etc)</li></ul>You are looking to get a feel for the candidate’s sense of cost and spend for hardware and hosting (which hopefully is something they have managed or dealt with at some point in their career – even if it was just their own personal website).<h3>Step 6b: The Deployment</h3>Now that both of you have explored the way the software might reside and operate in the physical environment a great follow up is how they would actually update or deploy the software. Fair questions are things like:<ul><li>What kind of tools do you use?</li><li>How would you roll out to new users? (good answers are typically staged roll outs, betas, feature flags, etc.)</li><li>How do you balance the “splashiness” of a big marketing launch (and peak traffic) with a brand new system?</li><li>How can you mitigate risks to ensure a successful launch?</li></ul>Hopefully they have some good ideas and can pull from some past experiences (including mistakes where launches went terribly bad – since <a href="http://devblog.seomoz.org/2011/06/software-launch-failures/">most of us have experience with those</a>).<h2><strong>Step 7: Operations</strong></h2>Since most of us are not so lucky to be able to build and operate products we created from scratch, understanding the operation of this type of system is an important thing to assess in the interview. You should be sure to touch on philosophies on operations and support, but also make sure they understand best practices. Here are some questions to help you probe these areas:<h4>Monitoring:</h4><ul><li>What types of monitoring do you need for this system?</li><li>How often would you look at your instrumentation?</li><li>What sort of systems have you used in the past? How did they work? What did you like and not like about them?</li></ul><h4>Operations:</h4><ul><li>How many people do you think are needed to maintain and operate this system?</li><li>How have you handled operational issues in the past? Did it work well? Would you change it if you could? How did the team feel about it?</li><li>What kind of process or feedback loop should be in place for customer feedback/issues?</li></ul><h4>Business Metrics:</h4><ul><li>What kind of business metrics do you want to measure or track for this project?</li><li>How often and in what medium should they be reported?</li><li>How have you tracked these types of business metrics in the past? Did it work well? Why or why not?</li></ul>With each area of these questions you are trying to get a grasp of the candidate’s experience and thoughts with the various aspects of running a production system. Hopefully they will provide great answers that align with your company culture and resonate with you and your team.<div> -------------------------------</div>This sort of problem should give you a lot of ground to cover in a lot of different areas; and for most software management roles it should many aspects of the role. Of course you can pick and choose each step based on your team’s needs and the responsibilities of the position. You can also add more steps too – things like hiring and recruiting talent, documentation, managing biz dev partnerships and APIs, etc. You get the idea. Hopefully this helps get you started, and if you have thoughts or comments don’t hesitate to add them to the comments :)Extra references, helpful to everybody reading this post:Microsoft Office Home & Student Mortgage Marketing Automation Software + Email Marketing Software Solution