The Benefits of Arbitrary Questions

We recently hired a new office manager at SeatGeek. It’s possible to find good office manager candidates by posting on sites like Craigslist and Monster, but there’s a big signal-to-noise problem–a single Craigslist post can yield hundreds of resumes, many from unqualified applicants. The cost of posting on Craigslist isn’t $25; the true cost is your time.

Many applicants didn’t know anything about our company or the particulars of the job. Perhaps they did a search on Craigslist for “office manager” and applied to every job they found in rapid succession, hoping to get a few hits. These folks reduce the amount of time I can spend with folks who actually want to work at SeatGeek.

So during the past hiring process, I tried something a little unusual: I required that all applicants complete a short Excel challenge when applying [1]. Importantly, the job involves almost no work with Excel. I was upfront about this in the job post:

Please note that using Excel is not a major component of this job. So why the Excel challenge? We believe that applicants who do well on the challenge are more likely to be smart, computer literate, and good at following instructions.

I got some interesting results. About 50% of applicants sent emails with no Excel attachment. Among the remaining 50%, about half hard-coded answers to the questions rather than using formulas, which the instructions explicitly warned against. So I was able to eliminate 75% of applicants in under 20 seconds. That meant I had more time to spend with the folks who were most qualified.

Among 25% of “good” applicants, I cared little about their answers. I didn’t expect people to get everything right–the final two questions were quite difficult; only 3 out of 500+ got all six questions correct. As long as the person applying had answers that weren’t completely wacky, I considered them to have passed. If they were in the 25% group it meant they were better than most other applicants at following basic instructions.

I’ve been doing something similar when posting freelancing jobs on sites like oDesk. After
you post a job on oDesk, freelancers respond with a bid and a few paragraphs about why they should be selected. So, at the end of my posts, I say something like “This job must be completed by July 20. In your response, please explicitly state the date by which you think you can finish.” If a bidder doesn’t include a date, I assume she didn’t bother to read all the way through.

The person we ended up hiring as our office manager got 4 of 6 Excel questions right. But the email she sent us included a “Notes on the Excel Test” section that described some of the issues she had dealt with. She didn’t need to include that to get an interview, but as soon as I saw it I was completely sold. More than a test of Excel skills, this was a test of how much someone wanted to work at SeatGeek.

[1] One downside of this is that it takes applicants longer to apply, thus potentially scaring off qualified folks. I tried to mitigate this by telling people not to bother writing a cover letter.