Mariusz Pietrzyk (wijet)

How The Power Law Changes Everything You Thought You Knew About Knowledge Worker Recruitment

Fri, 15 Jun 2018 00:00:00 +0200

After describing in a recent blog the odds of hiring the top 10% of talents, I would like to delve deeper into our prejudices and biases about how the world works. What’s the real difference between an extraordinary candidate versus someone who seems only slightly less qualified?

Let’s start with a little bit of math (but not too much).

The distribution of a statistical data set (or a population) is a listing or function showing all the possible values (or intervals) of the data and how often they occur¹.

The most obvious distribution that comes to mind is normal distribution (or Gaussian distribution). It’s all around us. Let’s take human height², blood pressure or points on a test, for example. They all follow a symmetric bell-shaped distribution.

Normal distribution influences our world view

Have a look at the graph below, which shows human height.

https://sugarandslugs.wordpress.com/2011/02/13/sex-differences

The average height is the most common in a population. Smaller groups of people are considerably shorter or taller than the average height.

When we remind ourselves of our school days, we can notice quite similar patterns in test scores. There were a few extraordinary children, a few who challenged the education system and a whole lot of mediocre ones in the middle.

So when we think about the performance of an employee, we often perceive it in a similar way. The vast majority of people are in the middle, there are a few underperformers and a few exceeding our expectations.

Based on this view, we draw the following conclusions:

Most people are in the average zone.
An equal number of people are below and above the average zone.
The difference between the average and the best is not huge. As seen with height, the average male height is around 175cm while the tallest is around 200 cm, so the difference is only approximately 12.5%.

Let’s see how this model works for outlining employee performance distribution.

Performance distribution for blue-collar workers

Studies have shown that the performance of blue-collar workers can indeed be described with a bell-curve (e.g. it follows normal distribution). The best performers are about 20% more efficient than the average ³.

Have a look at the graph below. Most people are in the middle (around the mean), and there are equivalent groups of people with lower and higher performance levels: more or less like the distribution of human height.

However, this model was more useful in the Industrial Age, when most people were performing simple jobs, often on an assembly line. In the case of white-collar workers (knowledge workers) this is no longer the case.

Performance distribution for white-collar workers

When analysing the performance of a white-collar worker, a myriad of factors come into play. It’s rarely only a matter of one’s speed or knowledge; rather, it’s very often a mixture of one’s knowledge, experience, motivation, perseverance and probably hundreds of other factors.

The complexity of a job makes differences in people’s performance staggering. One employee can be as productive in terms of delivered business value as a few other employees combined.

Researchers say that a top life insurance sales person can be 240% more productive than the average one. In case of a software developer, the difference might be as high as 1,200%³.

In 2012, a paper titled ‘The best and the rest: Revisiting the norm of normality of individual performance’ was published by Ernest O’Boyle Jr. and Herman Aguinis. They conducted five studies on the subject involving 198 samples including 633,263 researchers, entertainers, politicians and amateur and professional athletes. They made the following observation:

“Results are remarkably consistent across industries, types of jobs, types of performance measures, and time frames and indicate that individual performance is not normally distributed – instead, it follows a Paretian (power law) distribution.”

Have a look at the graphs below showing results of the studies.

Power law distribution

Now look at the graph below, showing performance distribution of knowledge workers.

There are two main differences from normal distribution:

More people are below the mean than in normal distribution
More people exceed the average performance by a factor of several times (so called ‘long tail’) compared to normal distribution

From this we can draw the following conclusion:

When the complexity of a job increases, performance distribution starts to follow the power law distribution rather than a natural distribution.

How power law distribution affects the recruitment process

Thinking in terms of the normal distribution model and applying it to knowledge worker candidates causes us to dramatically underestimate the differences between an average candidate and an extraordinary one.

Without knowing how the performance is distributed in the candidate pool, we might feel that someone is only slightly worse, when in reality he might be several times worse.

Looking at power law distribution, we quickly notice that a very large number of people are average performers and a much smaller number of people are above the average. More productive people, often several times more productive, are much less frequent in the population.

Clearly, the recruitment process has to be adjusted to account for a greater polarisation in the candidates’ performances than we thought.

Revising our evaluation techniques, however, is not enough to improve our accuracy. Marketing (employer branding) needs to be changed, as well. If you want to hire the people who are X times more efficient, you must target them specifically—and they usually have much different requirements for the job than the average Joe.

Conclusion

When evaluating someone, it’s good to keep in mind that individual performance of knowledge workers varies significantly, to the point where one person can be as efficient as three other people combined. Think about it next time, before deciding to hire someone only slightly less qualified than required.

Let me know what do you think; do you also see massive differences between people when recruiting them? Ping me on Twitter at @wijet.

Notes

What the distribution tells you about a statistical data set ↩
Although there are some discussion on the internet that human height doesn’t follow a normal distribution. Have a look here ‘Why heights are not normally distributed’ ↩
‘Great People Decisions: Why They Matter So Much, Why They are So Hard, and How You Can Master Them’ by Claudio Fernández-Aráoz ↩ ↩²

How To Be Better At Recruitment Than A Coin-Toss

Mon, 28 May 2018 00:00:00 +0200

We tend to overestimate our potential and abilities. The more complex a task is and the further away in time the consequences of our decisions are, the harder it is for us to accurately evaluate our performance.

It’s not bad will or lack of judgment, it’s a natural bias people have.

This fact is clearly visible in the hiring process, because it’s hard to evaluate someone’s knowledge, motives, potential and cultural fit based on a few calls. Whether a candidate was actually a good fit can sometimes come out only after several months.

For small companies a wrong hire can be a real hiccup

Each employee contributes to the company culture in a unique way. In a small company, not shackled by politics, the impact one person can make is more profound than in a large company, and a person who doesn’t fit can make the rest of the team miserable.

Recruitment is a lengthy process. It can take one to two months to hire a person, then another two to three months for the trial period. If it was a wrong hire, the process must be repeated to hire a replacement and hope that this time things will be different. The whole process can take six months or more.

If you happen to be an agency, as we are, and the hire is a developer, your client won’t be very happy, either, if you need to replace him.

Let’s not forget that with a bad hire, we also waste the candidate’s time. Since you know the company, it’s your responsibility to help the candidate discover whether he or she is a good match or not.

This is not a trivial decision.

At Ragnarson, every successful recruitment involves from 8 to 12 of our developers. Each developer has to be aware of the math, the rules and the mechanics of the process. There are multiple factors that are not obvious at first glance. That’s why last month we carried out a hiring workshop for our developers.

You, too, need to know your factors. Being aware of the facts and statistics and how to improve them will increase your chances of recruiting top talent.

Your personal evaluation skills

If you happen to be God, you have an accuracy of 1. This means that you’re right 100% of the time.

The very best professional interviewers have an accuracy of 0.7. Their evaluations are 70% correct. Most people fall in the 0.3 range, so they are right only 30% of the time¹. Not surprisingly, when deciding whether a candidate at hand is a match or not, they’re wrong 70% of the time².

It doesn’t matter where you land on this scale. Even if you’re better than the industry’s ‘crème de la crème’ and you have an accuracy of 0.9, the odds are still against you. And here is why.

What are your chances when aiming for the top 10% of talent?

It’s not uncommon that during any two year period, an HR specialist will have gone through hundreds of job candidates. So, let’s assume that you have a group of 100 people, and in this group, 10 are top talents and 90 are the rest.

Every company prides itself on hiring only the best, so, naturally, you aim for exactly those top 10%. However, you don’t know who these people are. So, what will be your error rate in picking the top 10%?

The harsh reality is 50%. Your error rate will be the same as a coin-toss. Keeping in mind that you have an unrealistic accuracy of evaluating a candidate at hand of 90%.

Let me explain that. If we have 10 top talents, 90 suboptimal choices and your accuracy is 90%, you’ll pick 9 top talents from the group of 10 and dismiss one of them. From the group of 90, again, you’ll rightly eliminate 81 candidates as not top 10% and wrongly pick 9. You end up with 9 right and 9 wrong choices that you think are the top 10%.

So you’re correct in 50% of the cases.

Sadly, this is not the end of the bad news. The calculations assume that your evaluations are 90% correct, which is hardly possible. If you’re in a group of the industry’s best interviewers, you’re only 70% accurate. Presented with the same group of 100 people, your error rate will top almost 80%. Have a look at the figure below.

If you have an accuracy of 30%, your error rate is close to 96%, which means you would be better off doing the opposite of what your gut tells you.

How can we improve our odds?

Two things come swiftly to mind.

We could simply improve the quality of the pool of candidates. If we had a group of 100 people in which 60 would be top performers and your accuracy were 70%, we would have an error rate of 22% instead of 80%. Not bad, but of course, that’s easier said than done.

How, then, do we improve our odds if we can’t dramatically improve the quality of the pool?

We can add filters to the recruitment process, or simply put, include more decision-makers. Have a look at the table below.

Imagine our starting scenario with a group of 100 people: 10 top performers, 90 suboptimal choices and three people evaluating them with an accuracy of 70% each. We would have an error rate of 80% on the first person (as previously mentioned). The second person would have a 61% error rate because it would only work on the pool previously filtered by the first person, and the third person would have an error rate of 41%.

We managed to get from an 80% error rate to 41%.

However, the decision-makers (the filters) don’t just sit together and come up with a democratic verdict. Each filter is applied after the other.

So, multiple interviews?

You may be thinking, well, you didn’t exactly spill the beans, here. Every recruitment process has several interviews, filters or stages: first call, homework exercise, pair programming session and a meeting with the management.

You’re right, but if those interviewers influence each other’s decisions, they all act as one filter instead of many. For example, some interviewers (like managers) might have a stronger incentive to hire because they have more work than their current team can handle.

Moreover, you don’t know who should remain as an interviewer because of excellent accuracy and who just throws spaghetti at the wall.

Interviewers’ accuracy varies

As we saw in the table with filters, three interviewers, each with a 0.3 accuracy, have an overall error rate of 99%. To get reasonable results, we still need to strive for the highest possible evaluation accuracy of each interviewer.

Following the advice from K. Pearson and P. Drucker, ‘That which is measured, improves’, you need to start keeping individual scores of each interviewer.

At Ragnarson, we do exactly that. If a newly recruited employee leaves the company without so much as a ‘goodbye’ or we need to fire someone, we consider that person a bad hire. This will affect the accuracy rate of the team-members who approved the hire.

Engage the team

Besides looking at the personal accuracy, we encourage the people who will work with a newly hired employee to take part in the process.

They’re the ones spending the most time with a newcomer, they’re the ones understanding role requirements and the company from the inside, so they’re the ones who should make the final call.

Conclusion

Everyone wants to work with the best, and sometimes we let ourselves get carried away by guesses, hunches and wishes. In these moments, it’s good to know what the odds are. Multiple interviews working as independent filters and a simple scoring system for each interviewer could increase the effectiveness of the recruitment process.

I would love to hear about your ideas for increasing the odds of a good hire. Feel free to ping me on Twitter at @wijet.

Notes

Insights and figures in this article come from a great book on recruitment called It’s Not the How or the What But the Who: Succeed by Surrounding Yourself with the Best by Claudio Fernández-Aráoz. I recommend it to anyone dealing with hiring.

Here is the Google spreadsheet with the error rate calculations.

According to the Claudio Fernández-Aráoz the author of It’s Not the How or the What But the Who: Succeed by Surrounding Yourself with the Best ↩
To simplify calculations, we assume that people have the same accuracy in evaluating whether someone matches the profile as when they don’t. ↩