The Myth of the Scientific Method

I referenced Tom Naughton’s excellent presentation “Science for Smart People” in my “What makes a scientist?” post. He calls out the steps of the scientific method to describe the difference between data from an observational study vs. a controlled double-blind experiment. He rightly chastises scientists for not following the scientific method and relying on incomplete observations to determine what is scientific fact. All good science should be done orderly, carefully, and reproducibly. That’s how every important advance in science has been conducted, right? No, not really.

On the whole, scientists share the belief that science is worthwhile, reliable, progressive. They share the belief that it is good to be scientific, even about matters outside of science. They share the belief—the illusion really—that the virtues of science, of being scientific, stem from using the so-called scientific method. That is to say, scientists by and large share the common, naive, and misconceived view of science; and further, scientists share the belief that they not only know what the scientific method is but actually use it.¹

Where do new scientific discoveries come from?

Modern science is the result of systematic, widespread cooperation between scientists who work independently to achieve their research goals. History is rife with stories of scientific advancement through experimentation. Often these experiments were designed to test hypothesis that ran counter to the accepted dogma. How do good ideas become part of the accepted mainstream and bad ones cast aside? Does every scientist simply check off all the boxes required of the scientific method before every experiment to ensure the results will be agreeable?

Research, or frontier science, is by no means a disciplined, homogeneous activity in which all the participants agree with one another about most things. Very far from it. Scientific research is a medley of all sorts of attempts to gain new knowledge, in every way that human ingenuity can conceive—by cutting corners, by doing “quick-and-dirty” experiments, not just carefully systematic ones, by following hunches or “just playing around,” as well as by trying carefully thought-out things. After all, research is done by people who differ from one another in all the usual sorts of ways: in having noble or ignoble motives, great or little energy, many or few ideas, and so on.²

There has never been a shortage of people who will claim to have an answer to a problem plaguing society. Sometimes those claims are at odds with the conventional wisdom and the prevailing scientific opinion. These renegade views are pushed by people that have varying degrees of competence, persistence, and honesty. How does science separate the wheat from the chaff?

Scientific knowledge—which aspires to be clearly expressed, with assumptions and limitations made explicit, reliable because well tested—can be pictured as gleaned from a mess of all sorts of suggestions, claims, and beliefs by progressive refining as errors and inadequacies are filtered out.³

The Knowledge Filter. In stages, deficiencies are eliminated by virtue of the social institutions that science has evolved, peer review in particular.⁴

To borrow a phrase from Longfellow, “Though the mills of science grind slowly, yet they grind exceeding small.” New hypothesizes are scrutinized iteratively through successively more critical and stringent reviews. This knowledge filter can be both a virtue and a curse. New ideas that conflict with the current orthodoxy may never gain traction. Pseudoscience, fraud, and pure lunacy get dismissed right away, but so might genuinely revolutionary ideas.

Science is now—compared to a century or two or three ago—professionalized; it has evolved undergraduate and graduate training that are virtual necessities for those who seek to make valid scientific contributions. So what enters the knowledge filter of science is much constrained by what students learn before they become practicing scientists and generate new knowledge. One of the most important things they learn is to take account of their fellows in the scientific community, those who will review their proposals and decide whether and how much to support their research. Individuals therefore learn to curb or modify their wilder inclinations, to pay some heed to the consensus that prevails around them, and thereby much naivete and sheer nonsense are nipped in the bud almost before they can enter into science. Only those who have learned the current state of the art are taken seriously, and only when they comport themselves in reasonably disciplined fashion.⁵

Published research is not (yet) scientific knowledge

It’s a common mistake by laymen (and some professionals who should know better) to jump on news from the results of a published paper. Mass media outlets (e.g., newspapers, magazines, etc.) are not part of the knowledge filter. Most articles by journalists do an extremely poor job of communicating the true nature of the research being reported. I’ve seen far too many examples of nonsense spewed forth by the fourth estate when it comes to science.

“Piled Higher and Deeper” by Jorge Cham www.phdcomics.com

Publishing work does not make it fact. It simply means that it is now widely available. If it’s of interest to other scientists, it can be discovered and further research can occur. If not, it will fade into obscurity. Further testing allows the hypothesis to be verified, modified, or extended.

So any piece of scientific work that becomes widely cited and therefore well known is unlikely to be fraudulent or to incorporate obvious mistakes; and dull, uninteresting, or scientifically incompetent stuff never becomes widely known. Only what has stood some test of time, as interesting and useful and not obviously wrong, becomes incorporated into the secondary literature of review articles, monographs, and graduate-level textbooks; and this then represents something like the prevailing consensus in the various research communities. It is pretty reliable stuff, mostly. But it is known in detail only to people who work actively in that particular field or in closely related ones. The scientific knowledge that has widest currency is that contained in undergraduate textbooks, and there is an appreciable time-lag between something making its way into reviews or monographs and finding its way into textbooks. During that time, some things become obsolete and others turn out to be mistaken after all, so that what does get incorporated into the texts is mostly very reliable indeed. In this tertiary literature we find the textbook science that we learn through typically dogmatic teaching in school and undergraduate college and that we tend to believe unreservedly. Yet if we could look ahead, say, a hundred years, we would find the content of the textbooks different in significant ways: more coherent with other fields, less inadequate in a variety of ways, using viewpoints that are sometimes quite different (say, particle ideas differ from wave ideas).⁶

The vetting process of science may appear to outsiders as an ivory tower of academics handing down decisions like some kind of supreme court of the natural world. The reality is that thousands of scientists act independently and autonomously to perform research. This bottom-up approach is more in line with a free market than a group of statist overseers deciding what works and what doesn’t. Yes, there will always be politicized areas of research like climate change, pollution, and nutrition that will attract outside influences who try to corrupt the process. This is even more reason to be mindful of the quality of evidence presented by evaluating how much filtering has been performed on the evidence discussed.

Overall, then, the raw stuff of frontier science has those characteristics of uncertainty, subjectivity, and lack of discipline that one should surely expect whenever human beings try to do what has never been done before. But after successive filterings through the institutions that science has evolved over the centuries, what remains easily gives the appearance of being objective and true. In point of fact, what remains is (relatively) impersonal rather than strictly objective, and it is hugely reliable and trustworthy rather than warranted true for all time; but in practice one rarely or never notices the difference—nor does that usually matter. John Ziman has ventured the guess that in physics, textbook science may be about 90 percent right, whereas the primary literature is probably 90 percent wrong.⁷

Old ideas die hard

Max Plank famously said, “Science advances one funeral at a time.” Old hypothesis do not always get replaced by newer, better ones even if the evidence warrants it. Even textbooks are not immune despite their position at the end of the knowledge filter. Cognitive dissonance occurs there, too. Dave Dixon wrote a great article about this back in 2008, exactly 6 years to the date I happened to write this post. (Weird!) You can read it here: http://sparkofreason.blogspot.com/2008_05_01_archive.html

Conclusion

Not every area of science is amenable to controlled double-blind experiments to verify hypothesis. Much of the commonly accepted dogma in nutrition, for example, is based on softer evidence of observational studies rather the filtered facts from a biochemistry textbook. Evidence that bolsters or contradicts hypothesis should come from many different areas of science. Anthropology, chemistry, psychology, etc. all provide a different view of the same universe. Hopefully, all paths converge to the same answer. When they don’t, a hard look is in order to determine if a cherished belief is perhaps not as valid as once thought.

Reference

1. Bauer, Henry H. “How Science Really Works.” Scientific Literacy and the Myth of the Scientific Method. Urbana: University of Illinois Press, 1992. 42. Print.
2. p. 46.
3. p. 44.
4. p. 45.
5. p. 46.
6. p. 47.
7. p. 47.