In the world of candidate interviews, lemons (i.e. ineffective, broken, bad) are prevalent.
These “lemon” interviews are littered with insightful, inequitable, and uninspiring questions. The interview experience is laced with bias or disinterest or numerous details that give an uncomfortable, unwelcoming message to certain individuals or demographics. And/or the interview structure lack the consistency required to equitably compare candidates.
For example, does this sound familiar?
“Walk me through your resume.”
“Tell me about a time when you showed strong communication skills.”
“As a [demographic], tell me how you work with teams of [different demographic].”
Questions like these leave a lot to be desired. They lack clarity so candidates don’t know how to effectively answer and employers are left without a consistent, measurable scale to compare answers. They lack inclusion and equity, as those who have access or knowledge to prepare for the questions or have insight into what the employer is actually looking for will outperform those who don’t (independent of skill or ability) and that doesn’t take into account the exclusion that occurs when demographics are singled out.
Great questions, on the other hand, are clear measurable, relevant, and insightful. They provide employers the information they need to effectively and equitably compare candidate’s skills and abilities, and facilitate selecting the best fit candidates for a job. In other words, great questions lead to great hires.
But what do you do if you’re not sure if your questions are excellent or just ‘lemons’?
Wisdom of ‘try it before you buy it’
Before buying a car or investing in a new technology, you try it first. Experience how it handles, check for flaws, get a sense of how well it works for you. The same wisdom goes for interview questions.
Before engaging candidates with a set of interview questions, try them first. Validate that the questions are clear, measurable, relevant and insightful and that they provide the information you need to effectively and equitably evaluate your candidates.
To do this:
Step 1: Select the test group
Find volunteers to go through the mock interview process. This includes both the hiring team (those asking the questions) and the interviewees (those answering the questions).
For the interviewers, the volunteers must be a true hiring team – one that will be interviewing for a specific position or set of positions. This can include the hiring manager, recruiter, subject matter experts, and anyone else typically involved.
For the interviewees, every volunteer must have at least a working knowledge of the position and skills involved (so they can answer the questions). Ideally the interviewee volunteers include those who are qualified and those who are ‘not quite’ qualified (i.e. enough knowledge/skills that they would get through the initial steps, but would not be the right choice for the job).
For both interviewers and interviewees, the more diversity within the volunteer group, the better (in any and every meaning of diversity). Diversity will test the range of how questions are asked, answered, and interpreted. It will also add perspective to the experience itself to ensure it is as inclusive as possible.
Step 2: Set up the mock interview process
Set up a scenario that mimics the true interview – including the space you will use (virtual or physical environment), the processes and the technology (i.e. how you will ask the questions and record / rate the responses).
Explain the expectations to everyone involved.
For interviewers, the process is no different than an actual interview – how they ask, follow-up, record responses, and evaluate candidates, should all be exactly as they would for real candidates (as this is what is being tested).
For candidates, depending on how many you have and how well they align to the role, there are options:
Have everyone ‘do their best’ as though they were in a true interview. This is best when you have a mix of qualified and unqualified volunteers.
Role-play: Select a few volunteers to have something about them that would disqualify them from the position (for example, lacking a key skill or character trait). Ask them to answer all the questions as best as they can but within the boundaries of the ‘thing’ they are missing.
Plant a difficult person: Select one or more volunteers to go out of their way to try to answer every question in a way that is technically correct, but without being at all helpful.
Step 3: Hold the mock interviews
Everything you need to know in this one is in the title – hold the mock interviews.
Step 4: Debrief
After completing the mock interviews, collect all the results from the interviewers – the questions, answers notes, evaluations, and selection.
Evaluate the results for the following:
Did the questions work? Were the interviewers able to effectively differentiate and evaluate the candidates in a consistent way.
Were there any discrepancies in the results that may point to an equity challenge? For example, one demographic or human attribute scoring disproportionately better than another.
Gather feedback, such as:
What did the interviewers think of the questions and why? Were they effective in evaluating the candidates?
For the interviewers - what should they have asked, but didn’t?
What did the interviewees think of the questions and why?
For the interviewees – what should they have been asked but weren’t?
How inclusive was the process? Did the interviewers and interviewees feel engaged and fairly evaluated against the intended criteria.
What recommendations, if any, do the volunteers have for improving the questions, process, and/or experience?
Step 5: Apply the learnings
Apply any learnings gathered back to the interview questions (or the process around them). If there are major adjustments, such as completely rewriting the questions, consider starting back at step 1 to validate new questions.
Bringing it all together
When making a major investment, such as a new car, a new technology, or a new employee, the more trying before buying can mean the difference between excellence and getting stuck with a lemon. While it takes a little time, test driving your interview questions before using them is a worthy investment. It validates that you are effectively and equitably evaluating your candidates. As a bonus, a good ‘test drive’ process will also serve as training and confidence building for the hiring team and organization. #NoBias #diversityandinclusion