Check it out, it is very cool!

Not all that many months ago we were working towards a deadline (for those of you new to this blog I work at a phenomenal startup, Delve, where I lead the engineering organization) and we started ordering dinners to provide an incentive for people to stay late. Food would arrive between 5-6pm and most people would stay until 7 or 8. Almost everyone ordered meals. When people might normally leave around 6, we found most people staying at least 1-2 hours longer than they had in the past because of the dinners (part of this was the guilt trip that if you order food you really ought to be staying late).

One day we were pushing against a tight deadline and the whole team was planning on getting up to go to lunch. So I said “Hey guys, let’s work through lunch and I will go pick up food for everyone.” The team was excited by this and everyone stayed through lunch (and later dinner) excited by the lure of free food. We decided to keep doing lunches for the next couple of weeks until our milestone.

We were able to hit our deadline with ease (something that we thought was unlikely) and it quickly came apparent that these meals were drastically increasing our productivity as a team. Of course part of it was because people were working more hours, but lunches had a much bigger impact than dinner ever did.

Why was free lunch so important? Well our team would all go pick up or eat lunch together. Even if it wasn’t the whole team it was usually more than 2-3 people, and that is a lot when we only have a 10 person team. By moving lunches in the office everyone would stay. Everyone would work through lunch and even though they were eating at their desks, people would eat together and socialize a little bit (we sit in an open space so it is easy to visit one another without getting up). Not having people leave the office to go pick up food made a huge impact on our productivity as a team.

So after our glowing success I considered suspending the program (since getting food from restaurants isn’t always cheap). However when I started working the economics, I reconsidered:

Developer Salary: about $75k or $35/hr
Average cost of a meal: $20

Cost savings in 1 year: $15 * 5 days * 48 weeks (assuming 4 weeks vacation) = $3600
With two meals: $7,200
over a team of 10: $72,000 or about 1 full time developer

So even though the meals are an increased cost, I believe that doing them is giving us close to the benefit of an extra head.

Although besides the cost benefit it also makes the developers really happy. They always talk about free meals being a great perk of the job. I also thinks it helps team unity since people spend more of their social time together as a team (lubricating relationships and making collaboration more efficient).

I would strongly suggest that you try it in your office with your team–at least for a while before a milestone and see if it works for you. I would have never believed it made this much of a difference if I hadn’t seen the impact with my own eyes (we have been doing this for about 9 months now with much success).

I am a big believer in cloud computing and feel strongly that all new software should be built leveraging these new architectures, but when they have outages (and of course they will) it is so important to build your application to handle these sorts of issues–in a sense to fail gracefully.  Regardless of if you use cloud computing or you host your own servers most outages are inevitable.  There will always be the chance that there are natural disasters or that hardware could fail–both of which are impossible to predict.  By being able to trust and leverage a service that has experts in storage it allows you to build your business on that expertise in a cost effective way.  Also chances are that a company with a large operations team is going to be able to respond and diagnose issues quicker.  In addition in Amazon’s case–they have several data centers and redundancy that is only available to businesses on a very large scale.  There are very fewer startups that would want to spend their precious capital buying hardware in two geographical locations for increased redundancy (nor would this be a smart use of one’s capital).  I would like to believe that because of Amazon’s large infrastructure and expertise in storage that their outages would be fewer and far between than if I was relying on my own storage solution.  Although only time will tell if this is the case.

One thing though, is that this does present opportunity.  If someone could build storage on top of amazon’s or could promise higher up-time some companies would pay a premium for that higher guarantee.  This could be a good business.  And one could leverage a bunch of different cloud computing offerings underneath their abstraction layer.  Google has a storage service, Nirvanix is another competitor, and as this model becomes more popular I am sure others will emerge.  There is still work to be done with cloud computing but it is an economical and game changing development in technology.  I think that in a few years there will be computing services, storage services, etc and only a few companies will control all the hardware.

So what do you do if you have built your application using S3 and it goes down?  Make sure you address your customers’ experience.  This is the most important thing you can do.  If your application is inoperable, provide your customers a means to reach your tech support.  If part of your application can still work, make sure it does.  Try to segment your applicaiton so your customers aren’t blocked.  And finally, if possible, build a back up method for storage.  Maybe it is using another web service, or your own local servers to keep things functioning.  Then customers can still add new data, they just may not be able to access the old data.  Or if you cache some stuff, try to write a smart service that will look in the cache first.  The main thing is: focus on the user experience and make sure that they don’t encounter a dead application.

What you know:
We want to spend as little as possible per GB.
We know we are going to be using more bandwidth month over month.
We don’t want to be locked into a contract that requires a high usage per month though, in the event we don’t hit those numbers.
We need to offer a reliable solution to our customers

Here are the elements of the contract to consider:

Contract Length
- Our usage is going to change as our business grows so we need to make sure our contract allows us to take advantage of that.
- We also do not want to be locked into a fixed amount of usage in the event we are not able to ramp up quickly enough.

Minimums
- For certain prices most CDNs have minimum usage requirements.
- Some will offer roll over (just like cell phone minutes) where you have a 12 month period to meet your minimums (which works nicely as your usage and bandwidth grows)
- Some allow ramp up times before you are required to meet your minimums (so you only pay for what you use for the first 90 days as an example)

Cash Flow — what is the payment plan associated—do we pay monthly, quarterly, etc.

Per Gig Pricing — Obvious the lower per GB the better.

Quality
- CDN quality is hard to quantify (there aren’t studies or things you can reference and all of them seem to have tip top keynote metrics).
- Once we have implemented a CDN we should run our own set of tests and determine their ability to deliver high quality content.
- We can look at examples of their other video customers, etc to get a feel for different levels of quality.
- Do they have different “quality” pricing? This is not an issue now, but could be at much higher usage levels.

Reliability
- Backup vendors: Do we need to offer a fall back CDN to customers? If so, is it worth employing a pay-for-what-you-use relationship with a CDN to fall back on if our primary CDN goes down?

Multiple vendor
- If we use two vendors then it is harder for us to achieve a bulk discount for higher volume.
- Using multiple vendors can allow us to get better results—comparing quality post implementation.
- More referrals from vendors for more customer leads.

SLAs
-Evaluate and negotiate on penalties and ramifications for SLA and service failures.

Termination
- Make sure we understand the conditions under what we can terminate the contract.

Feel free to leave comments if you think of anything I might have missed!

When I was at Windows this was one of the problems in developing Windows 2000.  We had opened up our platform to allow hardware vendors to write software and hardware.  They were supposed to use our finite set of APIs to access things (such as writing to the registry), however since APIs were limited in early versions of windows, people would  muck around the code and figure out how to achieve what they needed with their application (or sometimes were even given those instructions from Microsoft employees so we would have more hardware that was compatible with our operating system).  This later lead to lets of problems because applications built on top of the OS could overwrite bits, leak memory, and other bad things all of which could cause things like the blue screen of death.  Microsoft took control of this situation by creating more APIs, more documentation, and creating “trusted”software programs that assured a user the software they were about to install was a trusted application.  At the time I worked in Windows (many years ago), one of the selling points (say above a Macintosh) was that it was compatible with so much software and hardware.  This was because Windows (whether it meant to be or not) was largely an open platform.

Apple on the other hand is the King of User Experience.  I use an  Apple Powerbook and it seldom ever gets restarted.  I rarely have a problem with any application I run.  Apple is known for being a closed platform.  Apple gives developers a finite set of APIs to work with–APIs  that are well protected so they can ensure applications don’t affect the over all Apple experience.  They are doing this with the iPhone now too.  Developers sometimes complain that they are limited with what is possible on the Apple platform, but as a user I like the simplicity and consistency between applications I use and install.

In my opinion, web services are the crux of the next generation of software–where you never install anything and everything runs seamlessly through your broadband connection.  With this new technology and new paradigm for building software the whole open platform idea has a whole new life of its own.  Amazon AWS opens up the basic things: computation power, storage, and messaging–what you need to build any web application.  Increasingly more and more companies are building their software on services.  Using Flex (the technology we use at my company for our front end development) one of the primary ways to power an application is web services based.  So then as a business if your software is built this way, when do you open it up for innovation?  How do you let users build on top of your platform so that is controlled and you still have a great user experience?

Facebook got lots of hype and developer fanfare over their open platform, but most of the applications annoy me–when I logon and see a bunch of junk.  While I have a bunch of them installed on my profile, it was more because of apathy for removing them than the fact I actually use them.  Facebook is a great distraction and I like the entertaining aspect of social networks, but I do not find the rate your friends, vote for who is most popular (maybe because I never was?), or kick your friends in the head applications all my friends seem to have installed.  But with each of these applications people are spending more time on Facebook, and increased engagement means more money.  So are they really wrong here?

So how do you get it right?  I think the jury is still out.  The debate is interesting though and definitely something I am thinking about a lot.

Objection #1: We are working in Java, so I need someone who can be productive quickly and write code in Java.

Yes, if you have someone who has experience with the technology they may be able to ramp up a bit quicker (typically measured in weeks) than a candidate without (even though, in my experience, this has not always been the case).  However, they will still have ramp up time (learning your business, code and technology base, any specialized software, developer tools, etc.).  If you aim to hire someone smart, they will be able to learn on the job (since any new hire will have to do get up to speed quickly).  Candidates who have experience with different technologies are often able to pick up new ones quickly as they are able to apply patterns or methodologies that are familiar to them from past experiences.  This is particularly important because technology changes so much over time.  Ten years ago C was the language to know (followed shortly by c++), whereas now, most interviewers are interested in C# or Java.  You have to be able to jive with the times and continually stay abreast of the tools and technologies that make your skills better.  It is like that saying “It does no good to have great knife skills in a gun fight”.

Objection #2: Interviews should be given in the language we use.

I always have to ask “Why?”  What is the reason that a person needs to write Java because you use Java?  Most interview questions that require candidates to write code are all about the candidate being able to demonstrate how they think.  I am almost always most interested in learning how a candidate solves a problem (are they smart enough to figure out an answer without getting flustered) and then is able to translate their idea into code. Since I know a bunch of different languages I can almost always understand what they write (every once and while there will be the smart ass candidate who wants to write a solution in some obscure language–but thankfully I have dabbled in a bit of those too).  By letting a candidate write a solution in a language they are most comfortable in, chances are they are going to produce a better solution.  And when you are trying to judge a thought process and general smarts–doesn’t it make more sense to let the candidate use the tools they are most proficient in?  So as long as you are smart enough to read the solution, you should let the candidate use the language they think they can perform the answer best.

Objection #3: Why should I pay for someone to learn on my dime?

Well, first making an investment in your people is *never* a bad thing.  But most of all, good hires are hard to find!  There is a reason the recruiting industry is as profitable as it is.  By limiting your frame of vision and looking for such a specific candidate you may be overlooking and missing some incredible talent right beneath your nose.  It is worth training your team on how to interview and calibrate for super bright and talented candidates.  You would be amazed at how many more “qualified” candidates you might find.

There is no reason not to use AMZN’s web services platform if you are building any sort of website. They have made their platform secure, easy to use (and if you don’t think it is easy they now have premium support), and because you only pay for what you use it is very cost effective. In the past when someone has wanted to start a website or build an application you had to commission servers, and you paid the same rate whether they were in use or not. Moreover you had to have a system administrator, or member of your development team, be responsible for monitoring and regulating the health of those boxes. This means you could end up paying for hardware and people that you didn’t always needs. This is especially true for any startup who is just building out their service. Until you have lots of users you are essentially paying for idle resources–which is never a good thing when you are watching every dollar.

Using a web service platform also gives you a lot of free things you can’t get with just using your own boxes. You get things like scaling for free. Need to store more data, just upload to S3–you don’t need to worry about how many boxes the data is on, or what happens when a hard drive fails. Amazon handles all of that for you. This is one less thing for you to monitor (which means freeing up resources to do other things). And Amazon has a whole army of people with pagers watching these things for you–all for about the same cost as the hardware you would end up paying for anyway. While Amazon’s web services doesn’t take away the need for monitoring (and you should *always* be monitoring your end user’s experience with your application), it means you don’t have to worry as much about hardware failures, network outages, data centers losing power, etc. Amazon offers you the commercial level reliability that was once only available to enterprise websites like their own website, Amazon.com. And now that they are broadcasting their uptime one can be more confident building their business on top of their platform.

Needless to say I am big fan of Amazon’s web services and we use them extensively where I work. We have been nothing short of happy with using them, because we can have 100 servers at a moment’s notice, and we only pay for what we use. So if you are thinking of building any sort of new technology on the web, look into one of the web services platforms for your application–that way you will be able to get the most bang for your buck

For any person in a software engineer role, I would expect that a substantial amount of their time would be spent writing code. I am not going to get into the philosophical argument about design time versus coding time (I can save that for another entry), but I will say that building their piece of the product is a basic I expectation. And in the case of my team, our product is software, and that means any engineer better be able to write code and make their contribution. So I believe it is fairly clear the ability to write code is a job requirement and therefore should be tested, measured, or quantified in some way during the interview process.

There are a series of objections one typically hears about coding in interviews:

• Writing code on a whiteboard, or notepad isn’t comparable to writing code at a terminal as one would do in a job.
• When you are writing code at your job you have all sorts of resources (books, Google, your peers, etc.) at your disposal.
• If a person has accomplished so much on their resume, and they have xxxx certifications, then they can do the job.

It is a true statement that if a person can write code at their computer it is much different than standing up nervous in front of a whiteboard. That is also why I don’t ask people to implement a metrics system in an interview. I ask them to solve simple problems—most of which can be solved in one white board (so not more than 20 lines of code max) to illustrate how they translate their thought into code (I promise to post some of my favorite interview questions at some point). If they can algorithmically solve a problem, and have a solid understanding of data structures, then isn’t it important that they be able to turn those solutions into something that can be compiled?

Sometimes people are concerned that the standing up thing throws people off their game and hurts their performance, which is why I adapted the write the code on a pad of paper approach. This is also great because I can take their code sample into our debrief meeting and discuss their approach with the other interviews. The questions that are asked in these coding interviews do not involve complicated libraries—they require the syntactical knowledge that most students receive in an Introduction to Programming course. Furthermore, I will encourage students to write it in the language we use to do development (because then you can also assess at a basic level that person’s knowledge and style with a particular and relevant technology) but I always let them pick their language (I do try to make it object oriented though, since that is the type of development we do here) and I do not fault them on choice. This also allows them to be comfortable and be able to perform the best they can on a problem. But as an interviewer it means sometimes brushing up on languages you don’t love to use (for me that would be Python and Ruby).

As for resources, if a person wants to use a library but can’t remember the syntax (as is often the case for hash tables in interviews) I tell them to write out the APIs as they would implement them in the library. This means that the candidate doesn’t get blocked on the fact they can’t remember the parameters to pass into some API they barely remember, which allows you to focus on the problem at hand—but it also demonstrates how they might design an interface. Often times, senior candidates can even articulate all the interesting aspects of interface design and what are the pros and cons of their approach. Another thing you can ask candidates is where they might go to find information. Resourceful people are the type of people I want on my team. In other words, the resources aspect of the argument is easily overcome if you are a savvy interviewer and make sure to remove this as an obstacle for your candidate.

And finally, regardless of what a resume says, you should always approach candidates with the assumption everything isn’t always true. It sounds mean to assume that people would lie—but many do, and many more embellish. Understanding what is on the resume and how it translates to your business and your team is an important and necessary part of the interview process. Plus, if worse comes to worse you can compare what all those accomplishments really mean in comparison to other candidates and your own team on a set of baseline problems. In my opinion there is never a reason a candidate shouldn’t be able to write code. Your job as an interviewer is to set the stage in such a way so they can clearly demonstrate their ability and put their best foot forward.

Another stylistic thing I do is I almost always ask them (once they are completed with their first pass at the code solution) to explain it to me (since being able to explain your code is an important skill). Then if there are bugs I try to see if they will catch them, if they don’t I will suggest they walk through an example, if they still don’t I will let them know there is a bug. I like to see that people can identify and fix their own mistakes with a little help (since this is a reality in the life of a software engineer—everyone should know the basics of debugging). I then follow up with a question about testing. Good software engineers should have some basic ideas on how they would test their solution. And then finally, since the code is usually messy (because of the medium), I ask them what they would do if it were production code (some people say comments, formatting, moving methods around etc.). A lot of people will also delve into running time, performance, space usage, etc—all informative lines of questions that can help you understand more and more about the candidate, their approach to problems and their experience thinking about their code.

There is a ton of value in doing this line of questioning. I would not judge a candidate on these type of questions alone (since it takes a lot more than being able to write code to be a successful engineer), but I believe any good software engineer should meet a minimum bar.