The Leadership Assessment Fallacy

In Brief

• By the time a shortlist of candidates are being interviewed for a leadership role, they’ll have been through a fairly crude selection process from a long list of candidates.
• Companies believe their interview processes are good because they almost always result in a great candidate being hired.
• Psychometric tests that claim to peak beyond the curtains of a candidate’s psyche may lead to apparently compelling conclusions, yet they are inherently flawed and deceptive.
• Correlation does not equal causation. Just because a candidate goes on to do well, this doesn’t mean that the interview and psychometrics tests were optimal. Often, they can be misleading.

Leadership hiring at all companies appears to work just fine. Great leaders are still being hired into companies, and they go on to do well, so what’s the problem? The problem is the leadership assessment fallacy. It’s not the assessment process that’s good; the initial screening process is sufficiently good and the candidates on the shortlist to interview are both good and motivated to do well in the new role.

By the time five candidates are interviewed for a leadership role, they will probably have been shortlisted from 20-50 candidates, perhaps more or less. This shortlisting can be as crude as looking at where they worked, in what roles, in what size of leadership positions, and over what extended period of time. The selection process has to start somewhere. The debate about the crudeness of this process can be left for another discussion.

Let’s imagine a large global bank like J.P. Morgan is looking for a new CFO, and they’ve decided to look to the external market. They may hire an external headhunter or headhunt directly and put out a job ad for the role. Let’s say there are 50 candidates on the long list. Most of those will fall under a few categories ranging from ‘not relevant’ to ‘not senior enough.’ J.P. Morgan will likely want to see very senior candidates who have worked at one of the other large global banks, like HSBC, Citi, Goldman Sachs, or others, in a heavily matrixed international role. Then the criteria may be that the candidates must have either been in the same level of CFO role before or been in another very senior position where they developed a similar skillset at a similar level of seniority.

Let’s say we now have a short list of five candidates crudely selected by these criteria of comparable experience at competitor organizations. Think for a moment about the career trajectory of these five candidates before we even get a chance to interview them. It’s likely they will have gone to some of the best universities and started their career on the best grad schemes. They then spent twenty-plus years competing with their peers to climb the career ladder from Associate Vice President (AVP) to Vice President (VP) to Executive Director (ED) to Managing Director (MD) and then further up the various levels of MD seniority.

These are all incredibly successful candidates in their own right. They’re all motivated, they roughly know what they’re doing, and there’s compelling evidence they’ve succeeded in similar roles before. The chances of any of these four candidates being unable to do the role are slim, although it will come down to how they present themselves and how they interact with the assessors during the process. The right candidate certainly cannot be deciphered by their responses to psychometric tests or competency questions.

This notion that a psychometric test can peek behind the curtains of a candidate’s psyche and discover ‘what they’re really like’ instead of how they act in everyday life is not very useful. What people are really like is how they act in situations relevant to the environment for which they’re being assessed. In this case, we are looking at and considering how they act in the workplace.

We can run a thought experiment on this. We have three options to assess a leadership candidate.

1. Do a fully structured interview where we can get a complete biographical account of a candidate’s life and career from an early age onwards. This would be a structured interview like the Bremnus method, not the behavioral competency interviews used by most companies today. All competency interviews give the interviewer is a highlight reel of great examples from one’s career, which doesn’t tell us the whole story.
2. Conduct a psychometric test only and decide who to hire based on this test without any interviewing.
3. Combine a structured interview with a psychometric test to optimize the rigor of the process.

It is typical in larger organizations to combine interviews and psychometrics, as in the case of option three. Those who believe this is optimal will say that the psychometric tests augment the interview process and help them understand more about the candidate that they wouldn’t otherwise get to understand by just doing an interview. We’ll come back to this point.

Let’s consider option two. The first issue with psychometric tests is there are so many of them to choose from, so we first need to contextualize the problem. Let’s set aside psychometrics like Myers Briggs and DiSC for a moment. Anyone still using these roughly 80-year-old assessment systems should look at the literature on the Big Five personality traits, which supersedes these methods and brings some scientific rigor that isn’t evident at all in these old Jungian offspring. Besides, on the Myers Briggs website, it even reads, “It is not ethical to use the MBTI instrument for hiring or for deciding job assignments.” This is a disclaimer to help them avoid getting sued for claiming it’s a job assessment tool but they know and we know that companies are still using it anyway. If you look deep enough behind the many positive headlines churned out by the company behind Myers-Briggs to fill out the first few pages of Google searches, you’ll see why.

Okay, so let’s take the Big Five Personality traits. This is the best-known most-valid taxonomy of personality there is. Unlike Myers Briggs and DiSC, it has five personality traits, and we all exist on a spectrum of the five, which is relatively stable over time. With Myers Briggs and DiSC, there is no spectrum, which is both limiting and not representative of personalities at all. The test for the Big Five is a battery of questions where you see a statement and have to select from one to five ranging from strongly disagree to strongly agree and increments in between. In the comfort of your own home, trying to discover something about yourself, this may be a beneficial test as only you will know the answers, so you are more likely to answer them truthfully. However, in a job assessment, when asked to agree or disagree with statements like, “I work well in teams” and “I tend to be messy,” just how honest are people going to be? Really, think about that. You don’t have to have spent ten years at university getting a Ph.D. in psychology to see how misleading the results of such a test can be under job interview conditions. So, the test is already flawed under such conditions. However, this is not the main issue.

The main issue with option three from our thought experiment is this: If a company is going to conduct an interview and a Big Five personality test to broaden the understanding of the candidate, they’re going to run into issues. If the personality test miraculously uncovers traits that have never manifested in any part of the candidate’s career, how useful is it to know what these newly discovered traits are? For example, if you interview someone and discover they’re a tough authoritarian leader that people don’t tend to get along with, yet in their personality test, it says they’re highly agreeable (one of the Big Five traits), what do you infer from that? That they cheated on the test, or that they’re working on themselves, and this is who they’ll turn into if they’re given the job.

It’s utterly useless to observe any traits that are counter to all of the observable patterns of traits that we extract from the stories of their life and career in the interview. They’re not relevant if they’re not reflected in the candidate’s actions. Such a personality test may be more suitable for someone in their early career where they’re developing other technical skills, but by the time they get to leadership roles, it’s mostly about behaviors, and behaviors leave clues in all aspects of a leader’s work and life.

Then you may say, “Well, if we find similar traits to those we find in the interview, then this personality test is useful in augmenting the interview process.” No, it’s not. In this case, it’s telling you something you already know, as evidenced by their actions and the conclusions you reached from the structured interview. To now conduct a personality test rendered invalid by the nature of the interview conditions and the ease with which it can be manipulated will only lead you to more questions than answers. What we must do instead is stick with a very well-structured interview like option one from the thought experiment.

Besides, there’s another thought experiment you can run with the three options. In this case, let’s assume we don’t even have resumes to review. We just took the first three names in the database of applicants. Remember, most aren’t even relevant. Option one was an interview only, option two was a psychometric test only, and option three was to do both. Here’s the thought experiment. Choose from the following options:

1. Do a properly structured interview with no personality test.
2. Do a blind personality test with no interview.
3. Do both.

Option three will give you the same as option one except for some additional irrelevant and potentially distracting traits that may trigger some confusing biases. Therefore we are wise to cancel number three as an option.

Doing option two, what if one of the applicants was a junior candidate who did work in finance but wasn’t a senior CFO? Personalities are relatively stable over time, so you may find a great personality fit for the role, but had you done an interview, you’d see this person was just not senior enough. Option one is the only sensible option here. It has to be an adequately structured interview, though.

Those of you who are big fans of psychometric tests may be reading this and thinking, “But they really work for us.” Well, exactly what do you mean by “work”? How do you know? If it’s because the successful candidate went on to do a great job, there are many reasons for that. If your thoughts are, “We found out so much more about the person than just doing an interview,” well, clearly, your interview wasn’t sufficiently structured with the right questions. If it was, everything that is useful to know about that candidate will be evident in how they’ve gone about their work and life to date. So the fact they work for you has more to do with the leadership assessment fallacy than the effectiveness of these psychometrics.

Conclusion

The leadership assessment fallacy is the affirmation of leadership assessment processes because chosen candidates go on to do well. Cue the classic ‘correlation doesn’t equal causation’ argument. Just because more people drown in the ocean on hot days, because more people are out there, and more people eat ice cream on such days, we cannot infer that eating too much ice cream carries a risk of drowning.

The reality is most candidates who are sufficiently qualified and experienced, especially in a senior role in a global company where other competent people surround them, will do well. So, why have a structured interview process like the Bremnus Method if it doesn’t really matter how we assess candidates anyway? Just because something cannot be perfected, it can still be vastly improved. The Bremnus interview method is a question structure designed to help the candidate tell as much of their true story as possible. This is entirely different from behavioral competency interviews.

With a complete biography of the candidate’s story, it is then up to the trained interviewer to spot the patterns of relevant traits to make an informed decision, rather than asking the candidate to give them examples of traits, as in the case of competency interviews.

The Big Five personality traits have helped us to universalize a taxonomy of personality, so we better understand how we all sit across a spectrum of these five factors. That does not make it or any of the outdated or subordinate psychometric tests a diagnostic tool in leadership assessment, though. At best, the Big Five provides a grounding of trait patterns we should be trying to observe when interviewing, but the tests themselves can be so easily manipulated as to render them misleading.

We forget the power of intuition and wisdom that we build as leaders over years and decades of experience that is most useful in selecting candidates. The narrative of the past two decades about how all biases are corrupt and must be dispensed with is just not true. Those making such claims don’t know the literature on biases and intuition and how these two cannot be untangled. In short, the correct biases are helpful, and the bad ones are not. Leaders using intuition in decision-making under conditions of uncertainty is not just the right thing to do; it’s often the only option they have due to a lack of information or time available. That’s just part of what leadership is – leading and making decisions in moments of limited information and time.

The leadership assessment fallacy will persist in companies who insist on continuing to do what they’ve always done – combining behavioral competency interviews and psychometric tests. It will persist in companies that just choose to ignore the facts. It will not just be the companies that suffer. Still, more importantly, great candidates will fail to advance their careers because some obscure and invalid test or some silly and irrelevant competency question did not elicit the response the uninformed interviewer thought they were looking for. We all know the story about those who continue to do what they’ve always done. They’ll only ever get where they’ve always been. It’s time for a change.

ABOUT THE AUTHOR

Fraser Hill is the founder of the leadership consulting and assessment company, Bremnus, as well as the founder and creator of Extraview.io, an HR software company aimed at experienced hire interview and selection in corporates and executive search firms. His 20+ year career has brought him to London, Hong Kong, Eastern Europe, Canada, and now the US, where he lives and works. His new book is The CEO’s Greatest Asset – The Art and Science of Landing Leaders.