My second scientific bombshell

I was a young graduate student when I first read a book by Thomas Kuhn which convinced me that scientific fact could never be absolutely certain.  The reason is that a new theory might always displace an earlier theory, which then put everything we thought we “knew” into a different perspective.  And indeed this has happened more than once in the last four centuries.

As a result of understanding this, I have always been comfortable with the changing landscape and fundamental potential uncertainties of science.  But a friend has just sent me an article published last month in the New Yorker magazine that is, for me at least, as big a bombshell to the certainties of scientific fact as Kuhn was.

The article is about what is called the “Decline Effect.”  Fundamentally, this is a phenomenon in which scientific findings get progressively weaker as they are replicated.  The effects of a new generation of anti-psychotic drugs, for instance, which twenty years ago seemed to reduce psychotic symptoms dramatically, have bit by bit dribbled away.

The Decline Effect, though, has not just appeared in relation to research into the effectiveness of new drugs and medicines.  It has shown up in psychology, in biology, in physics, in fact in almost every area of science.

What in heaven’s name is going on?  Two holy grails of the scientific method – replication and statistical analysis – were supposed to prevent this from happening.  Are they breaking down?

Replication has been intrinsic to the scientific method since it was first formulated and refined five hundred years ago.  Around the world, it has  been almost an article of faith that scientific results must be repeated by other scientists, doing their own observations and producing their own data independently.  This seemed to be the cast-iron assurance against not only outright fraud but against careless research or against the inevitable bias that any individual might, in all innocence, impose on his or her research.

The essence of statistical analysis is deciding whether some event probably happened by chance or not.  Although analyses have become extraordinarily sophisticated with the increasing computer power in the modern world, the principle remains the same.

For instance, walking down the street, there is one chance in 365 that the next person I see will have been born on the same day of the year as I was.  There is one chance in two that if I flip a penny, it will come up heads.  Science uses statistical analysis to decide if the effect connected to some variable happened by chance or is what is called “statistically significant.”

Did this group of people, for instance, who took an aspirin every day for the last ten years have lower levels of cancer than a similar group of people who didn’t regularly take aspirin?  And if the first group did have lower levels of cancer, was that merely a chance difference or were the differences so great that the probability that it was chance are miniscule?

What the decline effect is suggesting is that results that at first look as if could almost certainly not have happened by chance gradually seem to look more and more like chance with repeated replication.

The Decline Effect, of course, creates a cloud of doubt over huge swathes of accepted scientific findings.  And scientists are not yet totally agreed as to its causes or its cures.

But there are some potential explanations, and their exploration will almost certainly have an influence on the way science is practiced in the future.  More about what they might be in my next post on the subject.