Bias in Measures of Implicit Racial Bias

It is claimed that implicit association tests, or IATs, reveal unconscious biases against racial and ethnic minorities and other stigmatized groups. The tests are simple and their results appear to be straightforward to interpret: if you are quicker to associate positive words (or other positive stimuli) with the non-stigmatized group (e.g., whites) and quicker to associate negative words with the stigmatized group (e.g., blacks), you have an implicit preference for the former and against the latter. Moreover, it has been shown that IAT scores are (modestly) related to arguably discriminatory behaviors. Given that the IAT scores of most people suggest that they are biased against stigmatized groups, it has been claimed that implicit biases explain discriminatory behaviors in the real world.

Hart Blanton, a long-term critic of various theoretical and methodological absurdities in the IAT paradigm, has written, with some colleagues, a paper challenging a key assumption of the IAT. Re-analyzing several published implicit bias studies, they found that the standard IAT scoring procedure will typically label as implicitly biased people whose observed behavior is neutral and unbiased. IAT researchers assume that individuals who associate positive and negative IAT stimuli with different groups with equal ease are unbiased, but the research by Blanton et al. suggests that such individuals tend to be biased in favor of the stigmatized group. In other words, the zero point of the IAT scale is not associated with behavioral neutrality.

The results of Blanton et al. are pretty straightforward, but not necessarily easy to understand, so I’ll try to clarify them a bit.

On an IAT — you can take one here — respondents are shown stimuli on a computer screen and asked to categorize them correctly as quickly as possible. For example, on an IAT purporting to measure implicit racial bias, the respondent is asked first, in the “compatible trials”, to categorize white faces with positive words and black faces with negative words. Then, in the “incompatible trials”, they are asked to categorize black faces with positive words and white faces with negative words. The respondent’s IAT score is calculated based on the difference between their average response latencies (or reaction times) on the incompatible and compatible trials. If their response latencies, measured in milliseconds, are longer on the incompatible trials, indicating that they found it harder to associate blacks with good words and whites with bad words than the other way round, their IAT score will be positive and is interpreted as evidence of an implicit preference for whites and bias against blacks.

The IAT scale is seemingly easy to interpret. It appears to have a “rational zero point”: If the average response latencies on both types of trials are the same, then there is no evidence of bias. If one of the latencies is longer, it’s evidence of an implicit bias in favor of one of the groups. However, for the IAT scale to work this way, several assumptions must hold. At the very least, unconscious biases must actually exist and the IAT must reliably tap into them. As it happens, it is far from clear that this is the case.

The standard way to score IATs is known as the D-score. It is computed by dividing the raw difference between the incompatible and compatible trials in milliseconds by the within-individual standard deviation of response latencies calculated across both types of trials. The D-score was adopted by the IAT people in an attempt to eliminate certain undesirable properties of previous scoring algorithms, such as the fact that people with faster reaction times in general would “automatically” get lower IAT scores. As Blanton and colleagues have shown in another paper, the D-score does not fix such flaws and introduces new problems, but I won’t get into that here.

You get a D-score of 0 if your response latencies are the same on the incompatible and compatible trials. The IAT people have adopted cut-off points on the D-scale that indicate various levels of implicit bias. When the absolute value of a D-score is 0.15 or higher it is labeled “slight preference”, 0.35 or higher is “moderate preference”, and 0.64 or higher is “strong preference.” If you do a test on the IAT website, your “bias level” that is reported after the test is based on these values. The problem with these cut-offs is that they are arbitrary and are not based on any research. Furthermore, the underlying assumption that a D-score of 0 indicates unbiasedness has not been properly tested.

Blanton et al. attempt to rectify the arbitrariness of the D-scale by linking it to observed behavioral indicators of bias. They assume that if an individual is observed to behave in a neutral or unbiased manner, then their IAT D-score should be zero. If the D-score of such an individual is non-zero, then the true zero point of the D-scale is not zero but is equal to the D-score of such an individual (on average). To put it more technically, they test the hypothesis that if you regress IAT scores on behavioral indicators of bias, the intercept will be zero. If the intercept is non-zero, then behavioral neutrality is associated with non-zero IAT scores.

Blanton et al. were able to obtain the raw data of a handful of IAT studies and regressed IAT scores from these studies on the various behavioral indicators of bias included in them. These behavioral indicators, which were generally modestly correlated with IAT scores, were:

– the quality of interactions (e.g., eye contact, body language) of participants with Italian versus African or German confederates

– how participants judge the quality of a black person’s writing performance

– choices of white participants in a Prisoner’s Dilemma Game when they believed they were competing with a white versus black partner

– friendliness of participants in interactions with black and white experimenters

– participants’ evaluations of hypothetical white and black job applicants

If a D-score of zero really reflects behavioral neutrality, then you would expect that when D-scores are regressed on behavioral indicators of bias, the intercepts would be distributed randomly on both sides of zero, with an average of around zero. However, Blanton et al. found that the intercepts were generally positive: behavioral neutrality towards a stigmatized group is typically associated with an IAT score that suggest that the person is biased against the stigmatized group.

As they had access to raw data, Blanton et al. were able to adjust the D-scores in each experiment by subtracting the values of the intercept from the scores. As a result of this recalibration, behavioral neutrality corresponded to a D-score of zero. Using these new D-values, they recalculated the frequency distributions of racial bias categories (strong, moderate, slight, none) in each experiment. The original (“standard”) and corrected frequency distributions are shown in the following tables:

It can be seen that while the original frequency distributions generally suggest the presence of moderate to strong bias in favor of non-stigmatized groups, the corrected distributions suggest that negative and positive biases towards stigmatized (blacks, generally) and non-stigmatized (whites, generally) groups are about equally common. The corrected modal IAT category in most studies is “no bias.” Only in Ziegert & Hanges’s experiments is this not true, probably reflecting the fact that they used a very different scoring algorithm than the others.

Raw data were available only from that handful of studies, so Blanton et al. sought to extend their analysis by including IAT studies whose descriptive statistics made it possible to calculate the values of the intercept when IAT scores are regressed on observed behavioral and attitudinal variables. This yielded about 40 published experiments, with the variables, their correlations with IAT scores, and the associated intercepts shown in the table below:

As can be seen, intercepts were positive in all of these studies. The average intercept in studies using the D-score was 0.23. This means that observed behavioral or attitudinal neutrality towards different groups is associated with implicit bias scores that are typically interpreted as showing evidence of a slight bias in favor of the non-stigmatized or majority group (0.15 being the cut-off for slight bias).

The results of Blanton et al. suggest that if you are the sort of person that treats, say, blacks and whites with equal consideration, IAT will probably deem you to be biased against blacks. The way to get in IAT’s good books is to be somewhat biased in favor of blacks.

Why is it then that the zero point on the standard IAT scale is not associated with behavioral or attitudinal neutrality? One explanation is that whites and other majorities are, deep down, biased against blacks and other minorities, but that they are aware of this bias and consciously try to act nicer towards them. This, however, contradicts the idea that IAT reveals unconscious biases. Another explanation is that the D-score is a hopeless measure from a psychometric perspective, as Blanton et al. have suggested. The interpretation of IAT scores is further complicated by the fact that they are confounded by other psychological variables, including IQ, due to the correlation between IQ and reaction time.

In any case, if you hear it claimed that a huge percentage of Americans are unconsciously biased against blacks or some other group, take it with a grain of salt. At the limit, a group of people may be deemed to be harboring implicit biases against some disadvantaged group even when every member of that group is biased in favor of the other group in their everyday dealings.

P.S. I have a vague recollection of having skimmed some big, recent, many-labs IAT study that also found that the IAT zero point is not associated with behavioral neutrality. I think it was coauthored by one of the main IAT guys (Brian Nosek perhaps?), but I am not able to find it. Does anyone remember this paper?

P.P.S. I located the Nosek paper I alluded to above. In that study, they found that the average white had a pro-white bias according to measures of both explicit and implicit attitudes, but that the average white’s behavior (admissions decisions to an academic honor society) was nevertheless pro-black. The effect was replicated across multiple samples and experimental conditions, corroborating the findings of Blanton et al.

References

Blanton, H., et al. (2015). Implications of the Implicit Association Test D-Transformation for Psychological Assessment. Assessment, 22, 429-440.

Blanton, H., et al. (2015). Toward a Meaningful Metric of Implicit Prejudice. Journal of Applied Psychology. Advance online publication.