"One of my mentors told me that my real mistake was trying to replicate my work. He told me doing that was just setting myself up for disappointment."

I fell behind in my New Yorker reading in December, so I just got to this intriguing Jonah Lehrer article about a puzzling problem for scientific researchers: the inability to replicate their landmark findings in subsequent studies. It seems that researchers regularly avoid rechecking their results because they know future tests may call their findings into question. Does that mean that their original studies were unintentionally biased, subjective in some way that they don’t understand? The troubling occurrence is called the “decline effect.” One of the subjects Lehrer discusses the situation with is Jonathan Schooler, a highly self-aware psychology professor at the University of Santa Barbara. An excerpt:

“Jonathan Schooler was a young graduate student at the University of Washington in the nineteen-eighties when he discovered a surprising new fact about language and memory. At the time, it was widely believed that the act of describing our memories improved them. But, in a series of clever experiments, Schooler demonstrated that subjects shown a face and asked to describe it were much less likely to recognize the face when shown it later than those who had simply looked at it. Schooler called the phenomenon ‘verbal overshadowing.’

The study turned him into an academic star. Since its initial publication, in 1990, it has been cited more than four hundred times. Before long, Schooler had extended the model to a variety of other tasks, such as remembering the taste of a wine, identifying the best strawberry jam, and solving difficult creative puzzles. In each instance, asking people to put their perceptions into words led to dramatic decreases in performance.

But while Schooler was publishing these results in highly reputable journals, a secret worry gnawed at him: it was proving difficult to replicate his earlier findings. ‘I’d often still see an effect, but the effect just wouldn’t be as strong,’ he told me. ‘It was as if verbal overshadowing, my big new idea, was getting weaker.’ At first, he assumed that he’d made an error in experimental design or a statistical miscalculation. But he couldn’t find anything wrong with his research. He then concluded that his initial batch of research subjects must have been unusually susceptible to verbal overshadowing. (John Davis, similarly, has speculated that part of the drop-off in the effectiveness of antipsychotics can be attributed to using subjects who suffer from milder forms of psychosis which are less likely to show dramatic improvement.) ‘It wasn’t a very satisfying explanation,’ Schooler says. ‘One of my mentors told me that my real mistake was trying to replicate my work. He told me doing that was just setting myself up for disappointment.’”