Researchers Disagree on Accuracy of Well-Known Bias Test

Last year, a team of researchers at Harvard made headlines with an experiment testing unconscious bias at hospitals. Doctors were shown the picture of a 50-year-old man - sometimes black, sometimes white - and asked how they would treat him if he arrived at the emergency room with chest pains indicating a possible heart attack. Then the doctors took a computer test intended to reveal unconscious racial bias.

The doctors who scored higher on the bias test were less likely than the other doctors to give clot-busting drugs to the black patients, according to the researchers, who suggested addressing the problem by encouraging doctors to test themselves for unconscious bias. The results were hailed by other psychologists as some of the strongest evidence that unconscious bias leads to harmful discrimination.

But then two other researchers, Neal Dawson and Hal Arkes, pointed out a curious pattern in the data. Even though most of the doctors registered some antiblack bias, as defined by the researchers, on the whole doctors ended up prescribing the clot-busting drugs to blacks just as often as to whites. The doctors scoring low on bias had a pronounced preference for giving the drugs to blacks, while high-scoring doctors had a relatively small preference for giving the drugs to whites - meaning that the more "biased" doctors actually treated blacks and whites more equally.

Does this result really prove dangerous bias in the emergency room? Or, as critics suggest, does it illustrate problems with the way researchers have been using split-second reactions on a computer test to diagnose an epidemic of racial bias?

In a series of scathing critiques, some psychologists have argued that this computerized tool, the Implicit Association Test, or I.A.T., has methodological problems and uses arbitrary classifications of bias. If Barack Obama's victory seemed surprising, these critics say, it's partly because social scientists helped create the false impression that three-quarters of whites are unconsciously biased against blacks.

The I.A.T., which has been taken by millions of people on an academic Web site, measures respondents' reaction times as they follow instructions to associate words like "joy" or "awful" with either blacks or whites. It generally takes whites longer to associate positive words with blacks than with whites, although some do show no bias.

The test is widely used in research, and some critics acknowledge that it's a useful tool for detecting unconscious attitudes and studying cognitive processes. But they say it's misleading for I.A.T. researchers to give individuals ratings like "slight," "moderate" or "strong" - and advice on dealing with their bias - when there isn't even that much consistency in the same person's scores if the test is taken again.

"One can decrease racial bias scores on the I.A.T. by simply exposing people to pictures of African-Americans enjoying a picnic," says Hart Blanton, a psychologist at Texas A&M. "Yet respondents who take this test on the Web are given feedback suggesting that some enduring quality is being assessed." He says that even the scoring system itself has been changed arbitrarily in recent years. "People receiving feedback about their 'strong' racial biases," Dr. Blanton says, "are encouraged in sensitivity workshops to confront these tendencies as some ugly reality that has meaning in their daily lives. But unbeknownst to respondents who take this test, the labels given to them were chosen by a small group of people who simply looked at a distribution of test scores and decided what terms seemed about right. This is not how science is done."

Two of the leading I.A.T. researchers, Anthony Greenwald of the University of Washington and Mahzarin Banaji of Harvard, say that some of the past criticism about their measurement techniques has been useful. But they dismiss most of the current objections as moot because the I.A.T.'s validity has been confirmed repeatedly.

In a new a meta-analysis of more than 100 studies, Dr. Greenwald, Dr. Banaji and fellow psychologists conclude that scores on I.A.T. reliably predict people's behavior and attitudes, and that the test is a better predictor of interracial behavior than self-description. Their critics reach a different conclusion after reanalyzing the data in some of those studies, which they say are inconsistent and sometimes demonstrate the reverse of what has been reported. They have suggested addressing the scientific dispute over bias - and the researchers' arguments about the legal implications for affirmative-action policies - by having the two sides join in an "adversarial collaboration."

One critic, Philip Tetlock, a psychologist at the University of California, Berkeley, said he had found prominent research groups and scholars willing to mediate joint experiments. But so far nothing has happened - and each side blames the other. Dr. Greenwald says he tried proposing a joint experiment to Dr. Tetlock only to have it rejected. Dr. Tetlock says that he tried a counterproposal and offered to work out a compromise, but that the I.A.T. researchers had refused two invitations to sit down with independent mediators.

After all the mutual invective in the I.A.T. debate, maybe it's unrealistic to expect the two sides to collaborate. But these social scientists are supposed to be experts in overcoming bias and promoting social harmony. If they can't figure out how to get along with their own colleagues, how seriously should we take their advice for everyone else?

Walter

Harvard University, Texas A&M, University of Washington, University of California, Berkeley