Structured Interviewing

You might also like

Download as pdf or txt
Download as pdf or txt
You are on page 1of 17

9 -4 2 0 -0 3 2

REV: MARCH 6, 2020

ETHAN BERNSTEIN

AMY ROSS

Note on Structured Interviewing


Bad hiring decisions are expensive. 1 In addition to the significant financial cost, a, 2 bad hiring
decisions slow progress on critical agendas, reduce morale, and increase churn. 3 For the person being
hired, family life is disrupted and reputations are needlessly affected. In short, the stakes are high when
making hiring choices.

Yet most future leaders, despite having invested heavily in interviewing skills as a candidate
seeking a job, invest disproportionately less in becoming a highly-qualified interviewer to recruit
others. This note addresses that imbalance by helping the reader acquire the skills necessary to be an
effective interviewer and to champion best-practice interviewing in his or her organization.

Larry Bossidy, former CEO of AlliedSignal, famously called the unstructured interview “the most
flawed process in American business.” 4 Because hiring decisions are made with limited information
under time pressure, research has found them to be even more susceptible to unconscious bias than
most other human capital decisions. 5 Therefore, without corrective action, research and experience
suggest traditional, unstructured forms of interviewing to be a highly undependable form of candidate
assessment. 6 On the other hand, more and more organizations are proving that structured interviewing
is capable of reducing that bias, and thereby providing more reliable insights into a candidate’s ability
to do the job. 7

Nonetheless, in part because structured interviews are so “structured,” many people initially find
them unnatural and even uncomfortable. Some may even find them robotic or transactional. 8 But with
practice, people gain not only comfort with structured interviews but also a deeper appreciation of the
primary benefit: a more thorough and accurate sense of the candidate in a short amount of time. 9

Because the greatest barrier to becoming a better interviewer is often the discomfort involved with
early attempts at structured interviewing, this note is focused on practice. It provides a three-stage
roadmap to follow when conducting structured interviews, from preparation to interaction and finally
assessment, and articulates concrete actions at each stage for interviewers to practice in order to
develop themselves into more skilled and effective evaluators of future talent.

a The US Department of Labor (2003) estimates that the average cost of a bad hire is 30% of that individual’s first year earning
potential (US Department of Labor, 2003); additionally, a 2012 CareerBuilder report that surveyed 2,700 individuals found that
41% believed a bad hire cost $25,000 and a quarter believed that a bad hire cost $50,000 or more (HBR, 2012).

HBS Professor Ethan Bernstein and independent researcher Amy Ross (McKinsey & Company) prepared this note as the basis for class discussion.

Copyright © 2019, 2020 President and Fellows of Harvard College. To order copies or request permission to reproduce materials, call 1-800-545-
7685, write Harvard Business School Publishing, Boston, MA 02163, or go to www.hbsp.harvard.edu. This publication may not be digitized,
photocopied, or otherwise reproduced, posted, or transmitted, without the permission of Harvard Business School.

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
420-032 Note on Structured Interviewing

Common Interview Approaches


Good interviews are predictive. The primary goal of any interviewing process should be a high
degree of confidence about whether a candidate will be successful in the role. That prediction should
not depend on who conducts the interview; the decision should be the same regardless of the
interviewer. That consistency, and reliability of prediction, comes from the interview approach.

Brainteasers and “Think on Your Feet” Questions


“Describe the color yellow to someone who’s blind.” “If you were asked to unload a 747 full of jelly
beans, what would you do?” “How many square feet of pizza are eaten in the U.S. each year?” “Why
are manholes round?” “How would you solve problems if you were from Mars?” “Who would win in
a fight between Spiderman and Batman?” “If you were a pizza delivery man, how would you benefit
from scissors?” “You’ve been given an elephant. You can’t give it away or sell it. What would you do
with the elephant?” “What do you think of garden gnomes?”

There is no shortage of interviews built around impossible brainteasers or out-of-the-blue


questions. (Exhibit 1 has more examples of typical and atypical questions that have been used in
interviews.) Unfortunately, while such questions are intended to make interviewees think on their feet
and produce unprepared answers (and may, indeed, succeed in doing so), research has shown rather
unequivocally that they fail to predict future performance in role. Laszlo Bock, former senior vice
president of People Operations at Google, said Google “found that brainteasers are a complete waste
of time. They don’t predict anything. They serve primarily to make the interviewer feel smart.” 10

“Fit” Interviews
We have all heard the cliché that a person “knows” in the first five minutes--or first five seconds--
whether a candidate is a good fit for the company/role. Unfortunately, research has shown that this
‘knowing’ is grounded in unreliable unconscious processing. Organizational psychologist Uco
Wiersma described these first moments of a fit interview as “Stage 1” of a four-stage interview process:

Stage 1 begins when interviewers and candidates meet eye-to-eye and ends with a
handshake. Strangers assess one another on the two universal dimensions of warmth and
competence when they first meet, 11 and people judge how trustworthy a stranger’s face
looks within one-tenth of a second. 12 This stage lasts only a few seconds, however
premature biases about job candidates, which are unrelated to potential job performance,
cause interviewers to hire candidates who perform poorly on the job or to dismiss
candidates who could have done the job quite well. Appearance and the handshake are
two sources of bias during Stage 1. Interview decisions are heavily affected by candidate
appearance during unstructured, informal interviews but interviewers barely consider
appearance when the interview is structured. Moreover, appearance does not correlate
with subsequent job performance. 13

Here is an example of the pitfalls of “fit” interviews. Imagine that you interview someone with a
background or personal style similar to your own. After just a few minutes, you have an instant rapport
with this candidate and conclude they would be excellent in the role. What is going on? First, you are
responding to your own personal preferences – you like this person and can imagine having fun
working with them. Second, you may be influenced by one or more unconscious biases – perhaps
Confirmation Bias (everything they say confirms your initial impression) or Stability Bias (we look to
hire people just like us to preserve the status quo).

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
Note on Structured Interviewing 420-032

Structured Interviews
Structured interviews are very different from brain teasers and think-on-your-feet questions. While
the latter assume that the silver bullet to making an interview predictive is finding the perfect question,
structured interviewing assumes that predictability drives from the structure, not the question itself.
Research has confirmed this claim, demonstrating that structure improves interviewer judgments to
be “more predictive of job performance than those from unstructured interviews.” 14 In short, “adding
structure to the interview process can enhance the reliability and validity of interviewer evaluations.” 15

Structured interviews are also very different from “fit” interviews. Because of the biases involved
in evaluating fit, the interviewee’s actual capacity to successfully fulfill the requirements of the role is
never evaluated objectively—whether she or he will be successful is largely left to a question of luck.
Structured interviewing, in contrast, does not depend on luck. In fact, it is grounded in the opposite
assumption: what one ‘knows’ from the first five minutes is completely unreliable, and so interviews
should be sufficiently structured to put guardrails on an interviewer forming such a judgment and
focused more on evaluating skills through the surfacing of objective information about the candidate
than on likeability/fit. As a result, structured interviews have been shown to help interviewers detect
the difference between honest responses and deceptive impression management better than
interviewing experience alone, 16 reduce bias against pregnant women 17 and overweight job
candidates 18, and reduce both gender and race similarity effects between interviewers and
interviewees. 19, 20

If an interviewer is focused on the skills required to succeed in the role – and intent on confining
impressions of fit and likeability (and the preferences/biases inherent in them) to marketing and
outreach events – then the interview will look very different. It will look like a structured interview.

Nonetheless, unstructured interviews remain prevalent. 21 In part, that may be because interviewers
must adopt a very different mindset to successfully use a structured interviewing approach (Table 1).

Table 1 Mindset Shift Required for Interviewers Transitioning to Structured Interviewing

From: To:
Interviewer is looking mainly for a good cultural fit Interviewer is looking for skills necessary to
succeed in the position
A fluid conversation between interviewer and Interviewer directs the discussion and interrupts
candidate when necessary
Interviewer does not have to prepare Interviewer is highly trained and does some
preparation before each interview
Interviewers rely on first impressions Interviewer ignores preferences and focuses on
demonstration of skills
Candidates like the interview, and by association, the Candidates think the process is fair and that they
organization had a chance to demonstrate their skills

Source: Casewriter adapted from McKinsey & Company partner interviewer training.

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
420-032 Note on Structured Interviewing

A User’s Manual to Structured Interviewing


A large part of what makes an interview structured is the preparation of the hiring manager,
possibly with support from a human resources (HR) or a people/talent function, before bringing in
candidates. Perhaps because we are all busy with our normal roles, interviewing often ends up as a
‘magic time’ responsibility. There is the perception that interviewing just involves showing up, briefly
reviewing a person’s CV, and then talking with the candidate for 60 minutes. For organizations who
take structured interviewing seriously, however, that could not be further from the truth.

Put simply, a structured interview is intentional—that is, the competencies being assessed and
evaluation criteria are well-defined ahead of time, as are the interview design and the interviewer’s
questions. The note now turns to each of those building blocks in turn.

Building Block #1: Align on “What Good Looks Like”


For most organizations, structured interviews start with identifying the knowledge, skills,
abilities, and other characteristics (KSAOs) required to drive success in a particular role. The amount
of effort taken to identify KSAOs will vary depending on the role and the organization’s capacity.
Often, job descriptions or review criteria can serve as the starting point, but one may also choose to use
much more rigorous methods – from focus groups to robust statistical analysis on success factors.

For example, imagine you are looking to fill a sales manager role. KSAOs would likely include
knowledge of the industry, product or consumer; skills related to using a salesforce management tool,
writing effective, legally-compliant contracts; and abilities that include motivating a team, setting
challenging goals for self and others, disaggregating work, and effectively responding to customer
requests.

KSAOs can be challenging to determine. Some organizations find it helpful to work with outside
vendors who specialize in job analysis, while others prefer to keep the process in-house. If determining
KSAOs in-house, be sure to form teams comprised of both HR and other functional professionals, and
consider working in a test-and-learn fashion to arrive at the optimal set of a KSAOs. You will also need
to review and adjust the KSAOs over time; as the environment in which the organization acts changes,
new KSAOs may be required while others may no longer be needed.

After identifying “what” is required for the role, the next step is creating a scoring rubric, which is
necessary to objectively describe candidates’ abilities relative to the KSAOs. Typically, an organization
will choose a rating scale based on what is most practical. A more straightforward scale provides less
opportunity for differentiation but is easier to use and is more likely to enjoy enthusiastic adoption.
The key features of a scoring rubric (see Figure 1 for an example) include:

• Down the left side, the competencies identified earlier (e.g., motivates others) are listed. This
is not an exhaustive list by any means but is practical, as interviewers can get a confident
read of only so many things in an interview.
• Across the top, there is a straightforward rating scale from 1-5, which will ensure some
degree of differentiation while still being easy to use.
• What sets this rubric apart from others is the descriptors for each scoring option. The words
are chosen carefully to reflect things that the interviewer is able to observe over the course of
their interview. Indeed, the purpose of these detailed descriptions is to have a clear sense of
the likely success of the candidate upon entry and the support required (e.g., Can they be on
their own with customers? What additional training is required?)

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
Note on Structured Interviewing 420-032

Figure 1 Sample Scoring Rubric (for a Sales Manager Role)

Source: Casewriter adapted from McKinsey & Company partner interviewer training.

The more specific you are when describing competencies on the left-hand side of the scoring rubric,
the more specific you can be in your anchor descriptors, and thus the fairer your assessment will be.
For example, take communication – an important skill for almost all jobs. If you label the competency
as “communicates well,” there is a good chance that your anchor descriptors will similarly be very
generic. A better alternative might be “communicates effectively with all levels of stakeholders both
verbally and in writing.”

Building Block #2: Select a Consistent and Structured Interview Design


At this point, it is time to choose the interviewing technique. The three most common structured
interviewing techniques are:

Experience-Based Interviews. Here, the interviewer asks a candidate to share something from their
past that is related to the specific skill, behavior, or area of expertise required for the job (“Tell me about
a time when…”) These prompts are typically pre-determined and used with all candidates.

Scenario-Based Interviews. In these interviews, the interviewer asks a candidate to respond to a


specific, job-related scenario (“What would you do if…”). The scenario is typically pre-determined and
used with all candidates. Case interviews, where candidates are given a specific business problem and
asked to respond in real-time to different issues, are the best-known example of scenario-based
interviews. Case interviews are structured because they follow a standard format, use scripted
information and questions, and apply clearly-defined evaluation criteria.

Real-time Skill Test Interviews. Typically, these are tests of job skills (e.g. coding, analysis) that
are easy to structure with pre-written questions and an answer key. Sometimes, job simulations or
role-plays are used, but in order to be structured, these would need to be painstakingly scripted.

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
420-032 Note on Structured Interviewing

All three types of structured interviews are reliable in predicting job performance 22 and have
demonstrated added validity beyond cognitive tests. 23 Sometimes, certain roles or competencies lend
themselves well to a given technique, but typically, the choice of technique depends of the relative
weighting of available preparation time and interviewer skill level. There are several considerations
when choosing an interviewing technique (see Exhibit 2).

Once the technique has been chosen, then it is time to develop the interview guide. When using
experience-based interviews, develop questions ahead of time that frame the kind of past behavior the
organization is looking for. In the sales manager example, the question allows the candidate to choose
a suitable example that demonstrates his or her skill level against the competency. When using job
simulation exercises, such as case interviews, there is significant time required up-front to develop and
prepare the case, but then it can be used repeatedly and is very reliable (meaning, most interviewers
will come to a similar evaluation and will not rely on preferences in their ratings).

Finally, the organization needs to select interviewers. Interviewers should be specifically chosen
for the candidates they are seeing. There are several important considerations:

• Should interviewers be very familiar with the role? That may indicate choosing employees
who are likely to be peers of the candidate.
• Should some interviewers have a tenured perspective on the KSAOs? Best to include senior
team members.
• Is the would-be interviewer in good standing? Do they have the interest and capacity to be
trained and to conduct a good number of interviews (important to build skills and to be
calibrated)?
• Finally, if bias is a concern, is it possible to also include employees well-removed from the
particular role, with relatively little at stake?
Hiring managers and HR play an important role on the interview day to prepare the interviewers
before, during, and after the interview. Training can vary in format and investment required. Whatever
the decision, the objective of interviewer training is to ensure that the interviewer is familiar with the
technique, understands the competencies to be assessed, is well calibrated with their ratings, and has
good interviewing technique. The “how” of interviewing technique is just as important to train on as
the “what.” 24

Some organizations choose to conduct classroom-type trainings (in-person or remote), often


devoting a half-day or full-day training to this, similar to any other type of skill-building training. An
effective curriculum would include a mix of theory and practice. Many organizations continue the
training throughout the interviewer’s career. For example, many include shadowing, where a newly-
trained interviewer first observes, and then is observed, conducting actual interviews. They receive
feedback on their technique, as well as their ratings. Organizations are even establishing new practices
of holding refresher trainings, often based on analytics that measure an interviewer’s calibration, or
predictiveness. Responsibility for training is best housed in a people/talent function, but in a smaller
organization, can be handled by the hiring team.

In addition to the formal training, the interview coordinator should send a note and/or hold a
briefing with interviewers prior to meeting candidates so that each interviewer knows exactly who
they are seeing (e.g., candidate CV and background, role description) and what their responsibilities
are (e.g., what they are supposed to be evaluating, what questions they should use, how they are going
to differentiate a good interview from a bad one). Those with experience being the interviewee may
have observed that many interviewers arrive with what appears to be a briefing book, or a well-loaded

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
Note on Structured Interviewing 420-032

tablet containing CVs, interviewing guides and evaluation forms. All of that is in place before an
interviewer even sees a candidate. This is how an interviewer invests in candidate experience—by
being prepared and respecting their time, rather than “winging it.”

Building Block #3: Incorporate Best Practice Questioning Techniques


A common misconception about interviewing is that interviewers should mostly be passive
listeners, allowing candidates to “take charge” since this is their time to make their case. Instead,
consider that interviewers have a very limited amount of time to determine if the candidate has the
skills required to be successful in the role. With that in mind, an interviewer must be directive.

Imagine an interview between a sales manager candidate, Wah, and an interviewer Merri. Merri
has asked Wah to share an experience in the past that seems related to the type of situation they would
encounter in the new role. Here is an excerpt of this discussion – the “before” version. As you read,
consider the following questions:

• What kinds of questions is the interviewer asking?


• What is the interviewer learning (that is reliable) about the candidate?

Merri: So, I’d like to talk to you about the topic of motivation. Is there anything you’ve done recently that
you’d like to share?

Wah: Ah, yes, quite recently I was leading a special project for my company.

Merri: Ok, and what about that project had to do with motivation?

Wah: Well, there was a guy on the team who just wasn’t pulling his weight. He was doing an
important part of the work, and every time we met as a team, he didn’t have anything done on his
part. He would always come up with some excuse – computer trouble, client emergency, that sort
of thing.

Merri: Why do you think he wouldn’t admit that he hadn’t done it?

Wah: Why? Hmmm. Well, I know he has a lot of new accounts and he plays on our company’s
softball team, so he probably had lots of other things to do I suppose. He was probably too
embarrassed or ashamed to say that those things were more important I suppose.

Merri: I see. And so how many times had this happened before you felt like you had to motivate him? A couple
of times? Over a period of several weeks or months?

Wah: Well, by the third time, we thought that this had probably gone too far so we decided to do
something about it.

Merri: What did you do? For example, did you go and speak to him about it? Did you try different ways to
motivate him?

Wah: Ah, yeah, yeah, I did, I caught him after the meeting and spoke to him about it.

Merri: And what happened? Did it work?

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
420-032 Note on Structured Interviewing

Wah: I had a word with him about how things aren’t working very well and how we need his input
for the project. I think I really got through to him because he said he was sorry, and he’d do better
next time.

Merri: That’s great. And did he bring it the next time?

Wah: Yeah, we gave him a little bit that he could manage given all the other things he had going on
in his life.

Merri: It sounds like you really worked with him and what he was going through. Did it work out in the end?

Wah: Yeah, it did actually. We got the project delivered on time. And by the way, I got a promotion
partly due to the fact that this helped land one of our biggest new accounts.

Merri: Oh, that’s great. What did you think you learned from this experience? I mean, what did this teach you
about motivating people?

Wah: I learned that you need to understand people below the surface – find out what they have
going on in their life. I didn’t realize how much time the new account was taking, but once I did, I
was able to scale back his commitment to our project. Motivating people really comes down to
meeting people where they are.

What did we learn about Wah from this exchange? If we are being perfectly honest, not very much.
While we heard about Wah’s grade and scholarship, presumably those were two things were already
featured in his CV. We heard some things related to working on a team, but Wah spoke in such
generalities that it would be difficult to have much confidence in our evaluation; we don’t know exactly
how he might behave on the job. To that, you will need to apply some best practices.

• Be specific about the kind of experience sought. Merri should have started with a narrow
question like, “‘Please describe an example that you’re really proud of when you motivated a
person on your team.”
• Interrupt if the discussion is going off track. Even with this narrow question, it is possible
for the candidate to speak at length. To direct the interview towards topics important to you,
consider interrupting with prompts like, “Let’s be sure to focus on those times when the two
of you were working together.”
• Ask open-ended questions. Merri asked several leading questions like, “So what did you do,
did you go and speak to him?” This is inserting yourself too much in the interview! Better to
keep an open mind and probe deeply into the experience. This way, you will learn much
more about what the candidate did. Further, in an interview setting, it is very tempting for
the candidate to follow your lead – if you suggest a course of action (“Did you go and speak
to him?”), there is a risk that the candidate will agree with you, whether he did this action or
not.

Also remember: in organizations that take structured interviewing seriously, interviewers are
specifically chosen and trained. See Exhibit 3 for some specific best practices on which companies train
their interviewers. The goal? Not to make the interviewer seem robotic, but to make the interviewer
reliable in assessing a candidate. There is still a lot of room to be human in a structured interview, but
the structure does constrain the behaviors that are most likely to make an interview assessment
unreliable. If an interviewer keeps defaulting to a non-structured approach to interviewing, those who
coordinate structured interviews should not be afraid to “deselect” them for future rounds.

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
Note on Structured Interviewing 420-032

Building Block #4: Encourage Evidence-Based Assessment


The real test of any interviewing process is how well it assesses candidates and mitigates (and
corrects for) non-evidence-based preferences and biases. There are several techniques that can be
employed to improve the chances of achieving that.

First, it is important to deliberately and carefully determine how much information the
interviewer should have about the candidate prior to the interview. Organizations should be
intentional with how they use, for example, a CV; a candidate’s background and experience is why
they are being interviewed and is a way to start a discussion with a candidate, but they should not be
used to form opinions about a candidate’s skills. Only the interview will allow you to do that. Seeing a
candidate’s “file” may help an interviewer connect with the interviewee, but it may come at the cost of
priming the interviewer in ways that make the interview assessment less unbiased and predictive.
After all, the organization already knows everything on a candidate’s CV—if the interview focuses on
that, the organization learns little more of value from the interview.

It is also worth noting that—as tempting as it may be—interviewers should never conduct an
internet or social media search about a candidate prior to an interview. Under the latest General Data
Protection Regulation (GDPR) requirements, such searches (without proper consent) can violate a
candidate’s privacy and cost the organization millions in fines. Such searches, when conducted in an
ad hoc fashion, are also unlikely to be accurate.

Second, it is important to explicitly counsel interviewers on how they should interpret certain
candidate behaviors. For example, many candidates are nervous, and some reveal this more than
others. While nervousness can sometimes indicate that the candidate wants to do well, there is no way
to know for sure. So, attempting to draw conclusions from the anxiety a candidate exhibits is a waste
of time. Never draw inferences; instead, probe deeply for evidence of skills.

Third, it is important to explicitly counsel interviewers on how they should consider


communication style. The short answer is: they should not (unless the interview is specifically
evaluating their communication style—and, of course, is therefore using a structured technique to do
so). Candidates themselves have preferences (speaking quickly, speaking softly, asking questions,
concentrating on the task, etc.) and oftentimes these preferences are based on societal or cultural norms,
some of which an interviewer may not be aware of. Well-trained interviewers meet candidates where
they are and do not attempt to draw inferences. For example, if a candidate speaks slowly, the
interviewer should slow down as well and leave lots of silence for them; the interviewer should not
infer that they are slow thinking or that they are not interested in the job.

Fourth, it is important for organizations to explicitly counsel interviewers on how they should
measure the success of the interview process. Interviewers often perceive, consciously or
subconsciously, a tension between two different measures of success for the interviews they conduct:
(1) getting a reliable, predictive assessment of a candidate’s performance in the role; and (2) providing
the interviewee with a great experience. Some interviewers want the interviewee to leave saying, “Ah,
those people, they are just so nice—I just love those people from that interview,” and there is evidence
that interviewees prefer general questions over situational or patterned behavior description questions.
Interviewers who want to be liked therefore tend to prefer general questions. But organizations that
conduct structured interviews well think about it differently. They have social events where they hope
a candidate will get to know them and walk away liking them, but the interview is kept separate and
distinct. When they ask for candidate feedback on the interview, which they always do to improve the
structure of the process, they hope the candidate’s feedback describes the interview as fair. Success

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
420-032 Note on Structured Interviewing

would be if a candidate responded, “Ok, that was rigorous, but I feel like they really know me and my
skills.” That is the true measure of success for a structured interview.

See Exhibit 4 for a one-page summary of these four building blocks of structured interviewing.

Looking Ahead
In the next decade and beyond, skills in structured interviewing are likely to become more valuable,
not less. As the world becomes more digital, more interviews are likely to be virtual, and a set of
researchers recently found that structure benefits virtual interviews even more than face-to-face
interviews, because structure helps to compensate for the challenge of processing highly-complex
interactions over a relatively lower-bandwidth communication medium. 25

That said, structured interviewing is unlikely to exist in a bubble. While structured interviewing
can be a powerful tool in any talent acquisition toolkit, the future is likely to bring a number of other
tools that may be used in combination with structured interviewing. Increasingly, gaming platforms
and virtual reality environments are providing opportunities for applicants to be assessed on their
knowledge, skills, or cognitive capabilities. Such technologies are also providing increasingly real
simulations of future jobs such that organizations can actually test, rather than simply try to predict,
how a candidate will do in a particular role or at a particular task.

Nonetheless, as talent assessment evolves, those who understand the principles of structured
interviewing and the means of doing it well are likely to build new approaches that make such
structured processes even more predictive; those who fail to do so, in contrast, are likely to replicate
the poorer outcomes of unstructured interview processes.

10

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
Note on Structured Interviewing 420-032

Exhibit 1 Types of Interview Brain Teasers

1) The “How Many [Things] Are There in [Location]?” Question:


a. How many gas stations are there in Europe?
b. How many taxi cabs are there in London? How many tuk-tuks are there in Bangkok?
How many bicycles are there in Copenhagen?
c. How many square feet of pizza are eaten in the U.S. each year?
d. How many barbers are there in Tokyo?
2) The “How Many [Things] Could Fit in [Container]?” Question:
a. How many ping pong balls could fit in a Boeing 747?
b. How many pennies would fit in this room?
c. How many gallons of paint does it take to paint the outside of the White House?
d. How many trees are there in NYC’s Central Park?
e. How much would you charge to wash every window in Toronto?
3) The “Do Some Quick Math” Question:
a. What is the sum of the numbers one to 100?
b. What is the angle between the hour-hand and minute-hand of a clock at [time]?
c. If I roll two dice, what is the probability the sum of the amounts is nine?
d. If you had 5,623 participants in a tournament, how many games would need to be
played to determine the winner?
4) The “Why Is [Common Item] [the Way Common Item Is]” Question:
a. Why are manhole covers round?
b. Why is a tennis ball fuzzy?
c. Describe the benefits of wearing a seatbelt.
5) The “Explain [Concept] to a [Difficult-to-Explain-Concept Person]?” Question:
a. Explain the internet to someone coming out of a 30-year coma.
b. Describe the color yellow to a blind person.
c. Teach me how to make an omelet.
d. What is the color of money?
6) The “Solve This Mystery” Question:
a. A windowless room has three light bulbs. You are outside the room with three switches,
each controlling one of the light bulbs. If you can only enter the room one time, how
can you determine which switch controls which light bulb?
b. You have a birthday cake and have exactly 3 slices to cut it into 8 equal pieces. How do
you do it?
7) The “How Would You Do Something Ridiculous” Question:
a. You’ve been given an elephant. You can’t give it away or sell it. What would you do
with the elephant?
b. If you were asked to unload a 747 full of jelly beans, what would you do?
c. How would you fight a bear?
d. How would you kill a giraffe?
e. How would you catch a kangaroo?
f. How would you design a spice rack for the blind?
g. How would you test a calculator?
Source: Adapted by casewriter, from Alyse Kalish, The Muse, 7 Insane Brain Teasers You Could Actually Encounter in an
Interview, https://www.themuse.com/advice/7-insane-brain-teasers-you-could-actually-encounter-in-an-interview

11

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
420-032 Note on Structured Interviewing

Exhibit 2 Considerations When Choosing an Interview Technique

Source: Casewriter adapted from McKinsey & Company partner interviewer training.

12

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
Note on Structured Interviewing 420-032

Exhibit 3 Best Practice Questioning Techniques

Think of your role as an interviewer as that of a lawyer, detective, or journalist; you need to ask
many questions, probing progressively deeper into a candidate’s responses, in order to uncover “the
truth” about a candidate’s competencies. To do so, great interviewers utilize the following tactics:

1) Set the stage: By all means, set the expectation early on with your interviewee that you will be
interrupting him/her. You can explicitly say at the start of the interview, “I will interrupt or
redirect the conversation frequently in order to get at the information I need. Do not be alarmed when
this happens; I do this with all candidates!” That way, the candidate is not taken aback or
wondering whether the interview is going poorly. Instead, you have set the stage early on for
how the interview will proceed.

2) Explain what you are looking for and why: Ideally, before the candidate even comes in for an
interview, s/he has received some information on what the interview will consist of and what
skills you will be looking to assess. From there, as you begin to assess a particular competency,
spend some time explaining why it is important to the company/for the role. For example, after
initial pleasantries, “I am going to shift now to a part of the interview where I am going to talk about
something in your past and explore that in some detail. As you know, for this sales manager role, you
will be expected to manage some clients, but you will also be expected to manage a team. For us, having
our employees be excited on the job and motivated to do their best is really important, and we see you as
being the key person to drive that. So, what I am most interested in is learning about is a time in your
past when you were successful doing just that. The specific question is: can you share an experience with
me where you truly inspired your team to achieve something?”

3) Give the interviewee time: Allow your interviewee a chance to think of the best, most relevant
example from their past to discuss with you. Say, “Take a minute” and then pause for a few
moments while s/he thinks.

4) Start with context: Once the interviewee has an example, ask him/her to begin by explaining
the context of the situation. For example, “Ok, so before we get started discussing this experience,
can you give me maybe a minute of context? Just a flavor for what the team situation was and what you
were trying to achieve?”

5) Demonstrate active listening: As your interviewee talks, show that you are following along.
For example, “Ok… ok… yep… ok… alright… got it.”

6) Confirm understanding: Periodically check in to confirm you understand what the interviewee
is saying. For example, “Ok, so let me play this back to you [or, “Ok, just to recap”]: It sounds like the
way you inspired this team is that you harkened back to the financial crisis and said there would be an
opportunity to actually talk to consumers and to do things in a new entrepreneurial way. Is that right?
[or, “Does that capture it?]”

7) Continually probe deeper to assess the interviewee’s role: In order to accurately assess a
candidate’s skills, you need to understand what s/he thought, did, etc. in a situation, rather than
any of his/her colleagues. Any time the candidate uses generalities (“we,” “everyone,” “they”),
bring the conversation back to him/her specifically. For example:

- “Can I understand a little bit about your role in each of these things? In terms of where the idea
came from, or how this played out specifically with you working with the team?”

13

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
420-032 Note on Structured Interviewing

- “Ok, that sounds like a great example. Why don’t we start with the first part about the financial
crisis – can you bring me into that, in terms of how you approached your team with this idea?”

- “Excuse me for interrupting, but one thing that would really help me is, can you recall a
particular meeting where you discussed this with members of your team?”

- “Ok, so who was at the kickoff?”

- “Ok and so, as if you were re-living this, before you went into the meeting, what was going
through your head?”

- “Ok, tell me more about that.”

- “Ok, was anything else going through your mind?”

- “Alright, so how did you kickoff the meeting?”

- “Ok, can you remember the messages you delivered at the meeting?”

- “Talk to me about the reaction – do you remember anyone’s reaction specifically?”

- “Right, but tell me how you perceived that this was the heart of the meeting?”

- “Ok, does anything stand out in particular?”

- “Yep… ok…. So what did you do next?”

- “How exactly did you communicate that everyone was going to be an author?”

- “Ok, tell me more about it.”

- “And what was the reaction by the team?”

- “Any reactions stand out for you in that meeting?”

- “You said that “everyone wanted to do everything” – can you give me an example of that?”

- “What was the reaction of the other team members when she signed up?”

- “And what went through your head?”

- “Very interesting – tell me more about what was going through your head.”

- “Give me an example.”

- “Sorry to interrupt again, but you used the term “falling apart.” Can you give me a little more
detail about that?”

While that may seem like a lot of questioning, it would actually ideally continue for even longer!
You want to spend about 15-20 minutes assessing a particular competency in order to really
understand it (so depending on the interview length, each interviewer should be assigned to assess
only 2-4 competencies each). As you saw, do not hesitate to pause or redirect the conversation in
order to get the information you need to form an accurate assessment of the candidate.
Source: Casewriter.

14

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
Note on Structured Interviewing 420-032

Exhibit 4 One-Page Summary: User’s Manual to Structured Interviewing via Four Building Blocks

1) Align on “What Good Looks Like”


a. Identify the knowledge, skills, abilities, and other characteristics (KSAOs) required to
drive success in a particular role
b. Create a scoring rubric to objectively describe candidates’ abilities relative to the KSAOs

2) Select a Consistent and Structured Interview Design


a. Choose the interviewing technique (or portfolio of techniques) your interviewers will use
(e.g., case interviews, scenario-based interviews, or experience-based interviews)
b. Develop the interview guide to produce reliable outcomes (i.e., most interviewers will come
to a similar evaluation of a candidate and will not rely on preferences in their ratings)
c. Select interviewers chosen specifically for the candidates they are seeing (and deselect
interviewers who, over time, prove unable or unwilling to stick to the agreed-upon
structured interviewing technique)
d. Prepare the interviewers with substantial training (e.g., a full day for first-time interviewers)
and briefing materials

3) Incorporate Best Practice Questioning Techniques


a. Your mindset is important—take charge. Think about it – you have about an hour to
determine if a candidate will be successful in your organization. This is the not the time to be
overly polite and or to be checking your phone.
b. Be specific about the kind of experience sought. Make every minute count by asking
narrow questions that get beneath surface-level answers (“tell me more about that”).
c. Interrupt if the discussion is going off-track. Sometimes you will need to interrupt or re-
direct a candidate. This is uncomfortable, but is a critical skill to develop in order to optimize
your time with a candidate.
d. Ask open-ended questions. Don’t ask leading questions. You will learn much more about
what the candidate did rather than having them tell you what they think (or, more
specifically, what they think you want to hear).

4) Encourage Evidence-Based Assessment


a. Determine how much information the interviewer should have about the candidate prior
to the interview. Be careful of priming the interviewer with more than necessary, causing the
interview to be more about what you already know about the candidate (i.e., the CV) than
what you don’t.
b. Counsel interviewers on how the should interpret certain candidate behaviors. Do not
draw inferences from anxiety or other behaviors; probe deeply for skills.
c. Counsel interviewers on how they should consider communication style. Do not consider
communication style unless that is what you are testing for in the interview.
d. Counsel interviewers on how they should measure the success of the interview process.
The goal is to be perceived as fair, not as “nice” or “likeable”.

Source: Casewriter.

15

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
420-032 Note on Structured Interviewing

Endnotes

1 Rebekah Cardenas, “What’s the Real Cost of a Bad Hire?”, HR Exchange Network, April 2, 2014,
https://www.hrexchangenetwork.com/hr-talent-aquisition/articles/what-s-the-real-cost-of-a-bad-hire, accessed August 2019.
2 David K. Williams and Mary Michelle Scott, “Seven ‘Non-Negotiables’ to Prevent a Bad Hire,” Harvard Business Review, May
31, 2012, https://hbr.org/2012/05/7-non-negotiables-to-prevent-a, accessed August 2019.
3 Falon Fatemi, “The True Cost of a Bad Hire – It’s More Than You Think,” Forbes, September 28, 2016,
https://www.forbes.com/sites/falonfatemi/2016/09/28/the-true-cost-of-a-bad-hire-its-more-than-you-think/, accessed
August 2019. Michael Housman and Dylan Minor, “Toxic Workers," Harvard Business School Strategy Unit Working Paper,
No. 16-057, November 16, 2015.
4 Larry Bossidy, “The Job No CEO Should Delegate,” Harvard Business Review, March 2001, https://hbr.org/2001/03/the-job-
no-ceo-should-delegate, accessed August 2019.
5 Marc Bendick, Jr., and Ana P. Nunes, 2012. Developing the research basis for controlling bias in hiring. Journal of Social Issues,
68(2), p.238-262.
6 Sharon Segrest Purkiss, Pamela L. Perrewé, Treena L. Gillespie, Bronston T. Mayes, and Gerald R. Ferris, 2006. Implicit
sources of bias in employment interview judgments and decisions. Organizational Behavior and Human Decision Processes, 101(2),
p. 152-167.
7 Michael A. Campion, David K. Palmer, James E. Campion, 1997. A review of structure in the selection interview. Personnel
Psychology, 50(3), p.655-702.
8 Karen I. van der Zee, Arnold B. Bakker, and Paulien Bakker, 2002. Why are structured interviews so rarely used in personnel
selection? Journal of Applied Psychology, 87(1), p. 176.
9 Jason Dana, Robyn Dawes, Nathanial Peterson, 2013. Belief in the unstructured interview: The persistence of an illusion.
Judgment and Decision Making, 8(5), p. 512.
10 Adam Pasick, “Google Finally Admits That Its Infamous Brainteasers Were Completely Useless for Hiring,” The Atlantic,
June 20, 2013, https://www.theatlantic.com/business/archive/2013/06/google-finally-admits-that-its-infamous-brainteasers-
were-completely-useless-for-hiring/277053/, accessed August 2019.
11 Susan T. Fiske, Amy J.C. Cuddy, and Peter Glick, 2007. Universal dimensions of social cognition: Warmth and competence.
Trends in cognitive sciences, 11(2), p.77-83.
12 Janine Willis and Alexander Todorov, 2006. First impressions: Making up your mind after a 100-ms exposure to a face.
Psychological science, 17(7), p.592-598.
13 Uco Jillert Wiersma, 2016. The four stages of the employment interview: Helping interviewers put two and two together.
Evidence-based HRM: a Global Forum for Empirical Scholarship, 4(3), p.232-239. (Barrick et. al., 2009)
14 Therese Macan, 2009. The employment interview: A review of current studies and directions for future research. Human
Resource Management Review, 19(3), p.203-218.
15 Therese Macan, 2009. The employment interview: A review of current studies and directions for future research. Human
Resource Management Review, 19(3), p.203-218. (e.g., Conway et. al., 1995; Huffcutt & Arthur, 1994; Huffcutt & Woehr, 1999;
Posthuma et. al., 2002).
16 Nicolas Roulin, Adrian Bangerter, and Julia Levashina, 2015. Honest and deceptive impression management in the
employment interview: Can it be detected and how does it impact evaluations? Personnel Psychology, 68(2), p.395–444.
17 Jennifer DeNicolis Bragger, Eugene Kutcher, John Morgan, and Patricia Firth, 2002. The effects of the structured interview
on reducing biases against pregnant job applicants. Sex Roles, 46(7), p.215–226. https://doi.org/10.1023/A:1019967231059.
18 Eugene J. Kutcher and Jennifer DeNicolis Bragger, 2004. Selection interviews of overweight job applicants: Can structure
reduce the bias? Journal of Applied Social Psychology, 34(10), p.1993–2022.
19 Julie M. McCarthy, Chad H. Van Iddekinge, and Michael A. Campion, 2010. Are highly structured job interviews resistant to
demographic similarity effects? Personnel Psychology, 63(2), p.325–59.

16

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.
Note on Structured Interviewing 420-032

20 Uco Jillert Wiersma, 2016. The four stages of the employment interview: Helping interviewers put two and two together.
Evidence-based HRM: a Global Forum for Empirical Scholarship, 4(3), p.232-239.
21 Karen I. van der Zee, Arnold B. Bakker, and Paulien Bakker, 2002. Why are structured interviews so rarely used in personnel
selection? Journal of Applied Psychology, 87(1), p. 176.
22 Gary C. Oliphant, Katharine Hansen, and Becky J. Oliphant, 2008. Predictive validity of a behavioral interview technique.
Marketing Management Journal, 18(2), p.93-105.
23 Michael A. Campion, James E. Campion, and J. Peter Hudson, Jr., 1994. Structured interviewing: A note on incremental validity
and alternative question types. Journal of Applied Psychology, 79(6), p.998.
24 James M. Conway and Gina M. Peneno, 1999. Comparing structured interview question types: Construct validity and applicant
reactions. Journal of Business and Psychology, 13(4), pp.485-506.
25 Julia Levashina, Christopher J. Hartwell, Frederick P. Morgeson, and Michael A. Campion, 2014. The structured employment
interview: Narrative and quantitative review of the research literature. Personnel Psychology, 67(1), p.241-293.

17

This document is authorized for use only in Taya Cohen's 46-740 M4 Spring 2024 at Carnegie Mellon University from Mar 2024 to Aug 2024.

You might also like