At Science Is Not The Answer, William M. Briggs says we should never trust stories headlined with “research shows X” (although if you’re a regular reader, you probably already know this):
Research Shows headlines are generated from papers by academics, and these all have explicit or tacit claims of cause, all purporting to explain some set of observations (whether gathered in history, the world, or by experiment). To explain is to state or to tacitly point to a cause.
The problem is that the methods science has developed to affirm or claim cause are often wrong: they are not right; they are in error; they are incorrect; they are fallacious; they sometimes make the right decisions, but only accidentally. By which I mean, cause is arrived at not by the methods, but by other means, yet the methods are credited.
I hope it is clear when I say that these methods are not to be used.
But are.
The worst tool, and one whose use is always and every instance a fallacy, is the so-called hypothesis test. We have done (“wee Ps”) in Class so many times, we’re sick of it (or I am). But I want to prove to you “tests” are fallacious another way, using one familiar example and one common situation. And with no math!
The idea is simple: a researcher makes observations, runs a “test”, and makes a pronouncement the cause he thought of is the one correct explanation for the observations.
He might be right about this, but it will only be accidentally, and not because of the “test”. For that same “test” could be used in support of an infinite number of other possible causes. That is the proof against “tests”: that they can support anything.
This will always be the case. As in every time. As in it is inescapable.
Our familiar example comes from A Global Enlightenment: Western Progress and Chinese Science by Alexander Statman (his surname guaranteeing I would read the book). It is mostly a review of the 18th Century Jesuit mission to China, and those Jesuits’ interactions with major figures of the so-called Enlightenment.
An important observation Statman makes is man’s proclivity to look for wisdom in the past or the future. One believes those in the past had superior knowledge, knew more secrets and could communicate with God (or the gods) with greater ease and facility, yet somehow that knowledge was lost (possibly wiped away in the flood; China, having the oldest extant civilization was thought to hold vast repositories). Or one believes those to come will be better than we, will know more, and will lead easier and happier lives, if only they are not held back by those who look to that past (China was also by others thought to have stagnated and could only copy their betters).
