The holy grail of selection is no longer

To hire the top candidate for a job, we require selection procedures that provide us with a sound assessment of who will perform best on the job. Classic examples of selection procedures include resume screening and job interviews – arguably also the most used selection procedures. But what does the outcome of those procedures tell us about the potential of a candidate? In other words, what predictive value do they have concerning the future performance of a candidate in a given job?

Already in 1998, Frank Schmidt and John Hunter provided answers to these questions. Based on the results from their landmark study, cognitive ability (i.e. general intelligence) measures were considered as the holy grail of selection procedures. Cognitive ability tests ranked above all other selection procedures in predicting future job performance. Moreover, only small increments in predictive value were to be expected when combining other procedures with cognitive ability measures.

However, in a very recent re-evaluation of the validity of selection procedures, Paul Sackett, Charlene Zhang, Christopher Berry, and Filip Lievens provide compelling arguments why the predictive value regarding job performance of many procedures is lower than initially thought. In scientific terms, we call this predictive value ‘operational validity’. This re-evaluation includes thousands of studies, comprising hundreds of thousands of participants. What it comes down to (without getting too technical) is that previous validity estimates have been overcorrected. This resulted in an overestimation of the predictive value of several procedures.

The landscape of selection procedures is shaken up. The study of Sackett and colleagues sheds new light on the ranking of selection procedures. In what follows, I will give a brief account of these new developments. I can already reveal one thing: the holy grail of selection is no longer.

The strength of structured interviews

The top five selection procedures are essentially reshuffled. Structured interviews – not unstructured interviews – replace cognitive ability measures as the top-ranked selection procedure. Asking about past behaviour will, in fact, give you information about future behaviour. This does not mean that cognitive ability measures are no longer relevant or useful – yet, it does appear that their predictive value has been substantially overstated.

Structured interviews are followed by job knowledge tests and empirically-keyed biodata (i.e. information related to your working life as well as more personal yet oftentimes less relevant and more sensitive details). Work sample tests and, tied for fifth, cognitive ability and integrity tests complete the top. Notice, at the bottom of the ranking, that just counting the years of relevant work experience is barely predictive and thus not a good practice.

Graph showing the ranking of selection procedures as predictors of job performance. The most remarkable finding is that structured interviews now supersede cognitive ability measures as top-ranked selection procedure.

Not a one-size-fits-all

The results seem clear-cut: structured interviews outperform all other selection procedures in terms of predicting job performance. Thus, we should just commit to this ‘new holy grail’. Well, in consulting, one would say: “It depends.”. It appears that the predictive value regarding job performance of several selection procedures varies substantially across studies and, hence, across job types and organisational contexts. It is not a one-size-fits-all.

Graph showing the uncertainty around validity estimates of selection procedures as predictors of job performance. The predictive value of several procedures can vary substantially across organisational contexts.

Building further on the example of structured interviews, you probably already asked yourself what such an interview would look like. There is no straightforward answer to this question. A lot depends on, for example, which elements are important or relevant to the job or the employer, who drafts the interview, or how structured the interview actually is. There is no such thing as ‘the structured interview’.

The issue of applicant diversity

Until this point, I have only written about selection procedures as predictors of job performance. Nevertheless, employers might not just be on the lookout for the ‘unicorn candidate’ that ticks all the boxes but might also want to increase their organisational diversity – team diversity and team performance go hand in hand. The latter statement is especially true for functional and educational background. Therefore, it is essential to know the potential adverse effects of selection procedures.

Adverse effects arise when a selection procedure consistently disfavours a certain group of applicants, resulting in overall discrimination at the group level. A recent meta-analysis (preprint), in which I was personally involved, demonstrates the persistence of discrimination in the resume screening process. This discrimination is not limited to ethnicity but also includes age, disability and physical appearance, among other discrimination grounds. Early exclusion of qualified minority candidates from the application process results in a needless waste of talent.

The recent study of Sackett and colleagues also provides information on these adverse effects in the form of standardised mean group differences between Black and White applicants. While cognitive ability measures are no longer top-ranked in terms of operational validity, they are (still) highly susceptible to ethnicity-based adverse effects. These measures are followed by work sample tests, job knowledge tests and assessment centers. Counting the years of relevant work experience completes the top five (although I hope that, by now, you already vouched never to solely rely on this procedure again). At the other end of the spectrum, we have personality measures and… structured interviews.

Graph showing the ranking of selection procedures by adverse impact potential. Cognitive ability and work samples have the highest adverse impact potential in selection, while personality measures and structured interviews are better to maximise (ethnic) diversity potential.

The best candidate in a diverse organisation

A final takeaway from the study of Sackett and colleagues is that increased predictive value concerning job performance often comes at the expense of decreased (ethnic) diversity potential in selection. This is the case for cognitive ability measures, for example. On the other hand, structured interviews, integrity tests, measures of conscientiousness, and personality-based emotional intelligence measures do seem to outperform other procedures in terms of validity-diversity tradeoffs.

Graph showing the tradeoff between operational validity and adverse impact potential of selection procedures. Increased operational validity of selection procedures often comes at the expense of decreased (ethnic) diversity potential.

Using procedures with a high adverse impact potential is not out of the question but does require thorough reflection. As a general rule, when using high adverse impact procedures, try outbalancing their negative effects by also adding low adverse impact procedures to the mix. Specifically when opting for cognitive ability measures, you could consider focusing on sufficiently specific and relevant cognitive ability subtests (e.g. language-free) to minimise the adverse effects.

In the end, it is up to the employer to determine the most useful selection procedures based on their selection goals and organisational needs, and, of course, the time and resources at hand. Following these new insights, it remains a balancing exercise between selecting the best candidate and selecting a diverse set of candidates. Most probably, the solution lies in using an appropriate combination of selection procedures maximising operational validity while still cultivating a diverse organisation.



This post also appeared on LinkedIn. Data, code, and key references related to this post can be found in this repository. This page was last updated on 19 January 2022.