In my post on evidence-based medicine, I said that “when valid scientific principles are not followed, no valid conclusion can be reached.”
Why not? Why don’t anecdotal claims (such as personal experience) count as evidence? Here is an example of what I mean:
A young woman who was a Christian Scientist told me that various relatives of hers have been healed by prayer. For example, her sister was revived from death by the prayer of her family. I have two comments about her story:
- The observations that she told me about (that her sister was physiological dead, that her family prayed for her, and that she was revived) are probably true.
- There was no dishonesty on her part: she sincerely believed her story
Despite accepting her observation, I disagreed with her conclusion about the causal relationship between prayer and health. How could I know that interpretation is wrong? For one, I was not there. Furthermore, after I expressed skepticism at this story, the young woman gave me many more examples, all from her direct personal experience of various friends and family being healed by prayer. On what basis could I reject them all without any personal experience on my own?
This was the essence of my reply:
I cannot object to the events you observed, as I was not there. But this does not mean that I must accept your causal explanation for those events. I have three reasons for this:
- I have a certain understanding about the nature of the universe and of the means by which things happen. We call this cause and effect. In my experience, cause and effect happens according to certain rules, which we formally call the “laws of nature.” If someone presents an explanation that is inconsistent with my basic understanding of the laws of nature, they should have overwhelming evidence. Otherwise, I must conclude that their understanding of causality is wrong, even if their observations are true. This is especially true in observations regarding human health.
- There are good reasons for scientists to reject personal experience and informal observations as sufficient basis for conclusions. There are many forms of cognitive bias can we can honestly make unless we follow strict rules to eliminate errors. In many fields (such as fixing a car) trial and error is good enough. There are no lives depending on a car working perfectly. In others, such as human physiology anecdotal evidence cannot lead to correct conclusions no matter how honest or smart you are. The forms of error take many forms: availability bias, post hoc ergo propter hoc, hasty generalization, placebo effects, selection bias, regression to the mean, bias by prior beliefs, social influence, etc. Even if you are aware of the biases, you cannot fully escape their effects; only try to structure your research to minimize them. In other words, when trial and error is not good enough, there is no substitute for proper science.
- I categorize causal explanations into three kinds: true, false, and arbitrary. True explanations correspond with what I know. False explanations contradict what I know, but with further evidence may be proven true. Arbitrary explanations are neither true nor false because they do not refer to anything. They are “magical theories” because there is nothing that we can point to as the causal mechanism. How does prayer work? It just does – no mechanism is possible because it by definition exists outside of causality. Because arbitrary claims they cannot be proven or disproved, once identified as such, we can only dismiss them from consideration. We should be extremely skeptical of anyone who makes causal claims based on arbitrary/magical explanations. In the rare cause they point out a true causal connection, it is only by accident, and has no value to anyone as we have no more basis to believe that idea than any of their other claims.
If personal anecdotes are not acceptable as evidence, what is? A good theory:
- is supported by many different kinds of observations
- is consistent with existing knowledge
- is possible to confirm by repeating the observations
- has high predictive value: it should predict what will happen as well as what will not happen
“A theory is a good theory if it satisfies two requirements: It must accurately describe a large class of observations on the basis of a model that has only a few arbitrary elements, and it must make definite predictions about the results of future observations.” (Stephen Hawking)
Further reading:
- The Role of Anecdotes in Science-Based Medicine
- How Anecdotal Evidence Can Undermine Scientific Results
