On Hiring Developers

As the CTO of TrackAbout, one of my jobs is to hire and retain awesome developers. We used to be horrible at this, and today I’m happy to say we don’t suck. This post looks back at the evolution of our approach to recruiting.

We’re growing, and while that’s very exciting for us, it also means a lot of recruiting. All of our developers work from home, which means we get to hire people anywhere in the country. It also means that we can get buried with applicants.

Ball of email

Years ago, when we hired our very first developers, our process was terrible. We’d broadcast a job to the typical job boards (free ones, whenever possible), then sift through a deluge of emailed resumes. We were looking for clones of ourselves, and if someone used the exact same technologies we were using, we’d invite them to interview. One interview, maybe two tops. Sometimes we’d fly a candidate in, sometimes we wouldn’t. The interview questions were hastily compiled, often disregarded, and we’d make calls using our gut. Predictably, the results were awful.

TrackAbout is a company that, at its core, believes in continuous improvement. We are passionate about getting better and leaner in every facet. Compared to 10 years ago, our recruiting process has progressed by leaps and bounds.

Woman studying

I’m in a unique and fortunate position in that my wife, Shari, happens to be a human resources professional with a focus in Organizational Development, and who has done a lot of tech recruiting. We’ve relied on her sage advice at many stages along our journey.

When we realized we weren’t asking the right questions in our interviews, we studied behavioral-based interviewing techniques like Targeted Selection and Top Grading. Now we feel like we have the tools to get at what’s really important in the limited time available in an interview.

In November 2009, we got the entire development team involved in the recruiting process. They deserved a say in who they got to work with. We devised a series of technical interviews that require candidates to discuss specific programming practices and patterns, look for code smells, and actually code right there in the interview. These interviews are conducted via Skype and screen sharing. Today, a candidate has to pass through two levels of technical interviews involving four different developers before moving on.

Around that time, we also stopped flying in candidates for interviews. Our developers work from home 100% of the time. It seemed silly and wasteful to require someone to fly across the country and take an entire day to prove to us in person that they would be successful working from home. I won’t deny that there’s value in talking to someone face-to-face, reading their body language, and giving them an opportunity to meet people in the office. But in the end, we decided we were getting what we needed through our other methods, so we nixed the in-office visit unless the candidate was Pittsburgh-local.

Using email to manage job candidates causes loss of appetite, dry tongue and general malaise. In March 2010, our CFO Tom Patton introduced us to a local Pittsburgh startup he’d helped advise called Jazz (formerly The Resumator). In short, it’s awesome and we’ve been using it ever since to manage the flow of incoming applicants, the pre-screening process, and much of the communications with candidates. It’s made our lives much easier.

We were frustrated trying to pick the best place to post among sites like Monster, Dice and CareerBuilder. We had devolved to round robin, paying gobs of money to one, then the other, then the next, trying to figure out which was yielding the best candidates. We lamented what it must be like to be on the other end, as a job-seeker having to make the rounds to all these sites, trying not to miss the right opportunity.

Careers 2.0 launched in in February 2011. We had tremendous respect for what Joel Spolsky and Jeff Atwood had already achieved with Stack Overflow. We knew that the best programmers (heck, all the programmers) go to Stack Overflow to ask and answer questions, and we wanted to get in front of them. So we decided to post our job openings to Careers before all other sites.

What we’ve found is that the quality of the applicants from Careers is consistently higher than those we get from other sources. I’m not saying we don’t get any good applicants from other sources or bad ones from Careers, we do. But by the numbers, Careers is ahead. We’ve also found the Careers people to be awesome to work with, and they’ve bent over backwards to help us get our job postings right (thanks Nick!).

Trello screenshot

Trello was released in September 2011, and we took to it like flies on rice (yes) for all manner of tasks. The Storyboard model is a natural for us, as we already use a Kanban board to manage our development flow as part of our agile methodology. Our candidate-tracking Trello board allows the entire company to visualize the pipeline. Each card is a candidate, the lists are the steps in the workflow. In a card, we embed the candidate’s resume, contact info, links to their web sites, blogs, LinkedIn, twitter feed, etc. Our developers can volunteer to interview a candidate with a click. Within a card, we can discuss what we’re learning along the way. It cuts down on communication overhead and allows us to focus on the candidate, not the process. We could have used Resumator for this, but Trello is free and visual.

Scheduling interviews is a huge time suck. In November 2011, I became aware of a scheduling service called tungle.me. It integrated with Google Apps calendar and would let people outside the company schedule a meeting whenever I had free time. I can’t emphasize how awesome it is to be able to send a candidate a link and have him or her choose a few time slots, which I can accept or decline. tungle.me was bought by RIM (you already know where this is going) and they recently announced they are sun-setting the service. So I signed up with a similar provider, ScheduleOnce. My developers started to use Doodle instead, because it allowed them to more easily coordinate three people for a tech interview.

Did I mention I’ve been doing a lot of recruiting lately? Since posting our most recent developer opening about eight weeks ago, I’ve personally phone screened 50 candidates. I’ve paid a recruiter hourly in the past to do phone screens and it went ok, I may do it again. During a phone screen, I focus on fit, passion, attitude, communication skills and resonance with our core values. Being a programmer myself, I probe at the candidate’s knowledge of various programming principles and patterns.

One thing I have neglected to focus on in my past phone screens is whether candidates can actually code.

2 + 2 = 5?

Jeff Atwood of Coding Horror and Stack Overflow fame has written a handful of great blog posts [1, 2, 3] on hiring programmers. Earlier this week, I reviewed these posts after conducting five exhausting back-to-back phone screens. In my weakened state, Jeff’s advice took on a particular resonance, and inspired me to go sign up for Codility. Two of my developers had recommended Codility recently. So starting yesterday, we’re using Codility to weed out those “programmers” who can’t actually program. This also allowed me to clear 7 hours of phone screen appointments off my calendar this week alone.

One of the common fears of putting something like Codility first in your process is that it might be off-putting or intimidating. I think Codility addresses that quite effectively in its FAQ (edited for brevity):

Q: I am hiring a senior software engineer, don’t you think that top-shelf candidates will get offended if I ask them to solve a simple programming test?
A: Codility tests are fundamental, not necessarily simple. People who get offended too easily are probably those who you don’t want to hire (emphasis mine). …[S]erious candidates like to be treated seriously … good engineers enjoy interesting problems and are excited to discuss them.

If a developer is going to be intimidated or is too proud to prove that he or she can, in fact, code, then I don’t think that person is going to be a very good fit for TrackAbout.

The final step before the offer stage is checking references, which I do myself for lack of anyone better to pawn it off on. We all know that candidates only provide references who will say glowing things about them, right? You’d be surprised. I’ve had references tell me not to hire someone. Lots of people who agree to be a reference may do so begrudgingly out of some sense of obligation, or perhaps are just uncomfortable saying “no”. When speaking to a reference, I listen for the things they aren’t saying about a candidate. I probe for specific examples where a candidate exceeded expectations, or didn’t, put in full effort, or didn’t. During the initial phone screen process with a candidate, I ask them to tell me what their references would say if I were to ask them. In this stage, I get to hear it from the horse’s mouth.

Even though we’ve come a long way, it’s still challenging to hire great developers fast enough. I’m sure we’re not yet done tweaking this process. After all, there’s always room for improvement.

Comments (19)

I think you use a lot of interesting tools that obviously are helping you manage the process a lot better. I’m also glad to see that, during the phone screening, you focus on (what I consider) the right things – fit, passion, communication, etc.

One thing I would point out, though, is that I feel you’re doing the phone screening / coding test process backwards, and here’s why.

I’m basing my argument on the position that a company and its employees operate in a partnership – the company cannot exist without people to do the work and the employees are provided for by the company (through salary, benefits, etc). I also feel that an interview goes both ways – the company is determining if the prospective employee is right for them while the job seeker is determining the same thing about the prospective employer.

With that said, I feel that putting the coding test before the phone call puts the organization in a position of power above the job seeker. It’s like saying your time is much more valuable than their time. You don’t want to waste your time with talking to someone on the phone who doesn’t have the technical skills (understandable), but the job seeker doesn’t want to waste their time doing coding exercises for a company that could be the completely wrong cultural fit.

It’s not that the developer is “too easily offended,” as Codility puts it. Instead, it’s that they prefer to be treated as equals, and not as prospective coding monkeys. There’s nothing wrong with programming tests, those are fine. As Codility says, serious candidates like to be taken seriously and top talent enjoys solving interesting problems (though most code tests I’ve seen aren’t exactly interesting problems). The error here is with your process (i.e. putting the coding test first).

I suggest a compromise (based on an assumption that may be incorrect). Right now, I’m assuming that the phone screening call is not a short call, which means that each phone screening call is a large investment of your time. I would say have a preliminary call, limited to 10 to 15 minutes, to introduce yourself, provide a bit of information about the company, and explain the interview process to them. Only after that preliminary call should the coding exercise be sent. This will make it so that the job seeker doesn’t feel like they have to jump through your hoops just for the privilege of speaking to you.

This compromise assumes that the coding exercise is sent before any phone conversations take place (my conclusion drawn from your statement that coding tests have cleared 7 hours of phone calls from your calendar).

Russell – Thanks for the very thoughtful response. I appreciate the time it took to compose it. You make some strong points. Certainly I don’t want TrackAbout to appear unapproachable, cold or unfriendly.

I’ve been reading “The Lean Startup” by Eric Ries, which stresses the importance of experimentation and measurement in an organization. Introducing Codility two days ago is an experiment. As a leader in a learning organization, I’m always open to better ideas and approaches.

I’ll be in touch with all the candidates that I’ve sent through Codility. I’ll be asking questions about how it made them feel, and learning from their responses.

It’s too early to judge, but so far the experience has been positive. I’ve had some really gung-ho candidates who couldn’t wait to try their hand at the tests and emailed me immediately after to tell me what they thought.

when talking to a company about a job i want the coding out of the way or better, be able to use that as a discussion point in the interview.

if codility tests are not to your liking, check out hackerhires.com
they design tests specifically for the company where you are applying, so those tests potentially give insight at what kind of work you might be doing.

i like your suggestion for a compromise though.

I have no issue with Codility or the concept of testing a developer. It’s the order presented that I disagree with. I feel like the test should come after the initial phone call or meeting – not before.

I tried the first test. Nope. There’s no way I even could waste 30 minutes of my life on what amounts to solving a crossword puzzle. Best to use that time adding some features to my application that’s used by 1000s of global users, including IBM. That demo tests whether you are a cold-blooded machine. I would want to hire developers with a heart, not pot-noodle eating, jolt drinking nerds who can tell you how many seconds have elapsed since they never had a girlfriend.

Hans-Christian Boos

I could not disagree more. Programming is a skill, not an art. And on top of hiring cool guys you want to hire people with the right skills. And having a brain does not contradict the notion of having sex.

No, programming is an art. I’ve worked with PhD. programmers, really smart guys who could solve the tests at Codility in 5 minutes, but who have no idea of application architecture, usability, unit testing, creativity, improvisation, and communication skills. Working for a company who relied on such tests sounds to me like a job for a robot, not a human.

“Technology alone is not enough. It’s technology married with
liberal arts, married with humanities, that yields the results that make
our hearts sing.”

Steve Jobs

This type of thing was my initial concern with the approach. The fact that you have to be tested before talking to them is, in my mind, a huge barrier for many people.

Do you feel that the test may have been more tolerable after you had talked to the company and found that you liked the people and principles of the organization, and actually wanted to work for them?

It’s good to be concerned over it. I worry often. I build systems for non-profits in my spare time. Azure web services, Xamarin.IOS, Xamarin.Android. I don’t do it, because I haven’t talked to them first. Wish you luck though.

The challenge is not writing the code, but solving the problem. Once the solution is known translating it to code is trivial.

What is the best way for an early stage startup to hire web designer?…

Check out this post I ran across yesterday. http://blog.trackabout.com/2012/10/03/on-hiring-developers/ I hope it helps your search. As you ramp up your search for other employees, you may want to check out some applicant tracking systems to help manag…

>So starting yesterday, we’re using Codility to weed out those “programmers” who can’t actually program.

This phrase only shows that you can’t actually program, neither you have basic knowledge of what is required to program.

>This also allowed me to clear 7 hours of phone screen appointments off my calendar this week alone.

You can clear 24×7 hours by not taling to anyone at all

If you’re entrusting determining someone’s ability to code to a piece of software that’s limited to run in a web browser, that will tell them more about you than you than it could ever tell you about them. There’s a reason we developers all have PCs on our desks loaded up with actual real-life IDEs. We need the colour-coding, syntax-checking-as-you-type, ability to navigate code structures (e.g., being able to determine at will what other objects are using a particular class or property), etc, etc. People that use inappropriate tools like Codility regularly fail to appreciate this. There’s a mass market developed around producing tools that allow gullible non-technical people to feel that they’re meaningfully assessing experienced developers who have skills the test purchaser does not have. No-one that is actually experienced in a given language would ever try to do something as foolish ask a fellow-professional to write code using a web browser. It’s just too far removed from the real IDEs people use on the actual job. You might as well try to determine someone’s ability to drive a lorry by taking them on the Dodgem cars at a funfair. The only way to determine actual coding ability is to attach a projector to a laptop and watch the candidate solve a reasonably-sized problem in a short amount of time using all of the tools they’ll have available on the job, including the internet. If you’re incapable of doing that because you wouldn’t understand what you’re looking at, that’s a pretty sure sign you’re Codility’s Ideal Customer. But it’s also a sure sign you shouldn’t ever try to hire software developers.

Ah, the elusive anonymous troll. Very well.

You seem to have misunderstood the blog post. Codility is just one step of many in our hiring process. It’s a “smoke test”. We also have phone screens, behavior-based interviews, two rounds of technical interviews in which candidates must code in front of us, and reference checks. You seem to be laboring under the false impression that we hire anyone who can pass a Codility test. Clearly not so.

When using Codility, you’re free to write your code using your “actual real-life” IDE and paste in your solution. I think most developers of sufficient intellect can figure that out. Those who know their chosen programming language well enough and who aren’t crippled without syntax highlighting, intellisense, etc, can choose to simply bang out their code in the web browser if they like.

So, anyone that calls you out for needing to use an automagically-marked test that you couldn’t pass yourself instead of your own actual skill to assess whether the professionals you hope to attract to work for you is a “troll”? OK. Here’s me thinking that label was about people that don’t have an actual point to make, rather than just having a point that you happen to disagree with. Psychologists would call what you’re doing “rationalising”, but you’re entitled to justify your own inability to judge ability as a software developer by actually *talking* to the people you purport to be capable of managing however makes you feel most comfortable – it’s a free country, after all. Me?, I’m just an experienced software developer that knows when a hiring manager is using these tests it invariably means they don’t have a clue about software development themselves. The only way to tell whether the individual in front of you is a clueless moron that has read the same two whole Wikipedia articles on technology ‘X’ as you, or a Code Monkey that copies solutions from the internet, or an actual x10 software developer that knows what they’re doing and is capable of turning your business around is to attach a projector to a laptop loaded up with all the tools they have on the job, and watch them solve a meaningful problem in real time. But since that type of meaningful interview/test would require you to actually know what you were looking at, something tells me you won’t be taking that approach any time soon.

Once again, as pointed out in the blog, our interview process includes live programming challenges with flesh-and-blood team members vetting the candidate’s capabilities. Codility is just an early step of a much larger process.

We have a phenomenal development team at TrackAbout. I’m proud of the team I’ve built, and Codility has been a small part of that.

If you’re ever responsible for hiring developers, you can design a process that works for you.

I am responsible for hiring developers within my own organisation, thanks. And from that experience I know that if I used Codility as an initial screening tool, I’d never get to see the best people inside my interview room, since they’d be the ones smart enough to say “thanks, but no thanks” to such a ‘test’ of their skill. But, as you say Codility has enabled you to to elevate your own selection process from “horrible” to “not sucking”. If you’re happy with that level of acheivement, that’s all that matters for you.

You clearly have strong opinions on what you think works when hiring developers. Should you ever write something describing your hiring process, I hope you’ll share the link here so we can all benefit from your experience.

Leave a comment