Through out this blog, in my numerous debates on climate change, I have referenced parts of the scientific process such as peer-review, or the rationality of listening to the relevant experts, but I have never really fully explained how science (and more generally rational thought) works. Thankfully John Mashey writing over at Deltoid has done a wonderful job of explaining how science works in a manner intelligible to everyone, and equally importantly, how to identify when someone isn’t arriving at conclusions based on science and/or rational thought.
Motivation and Approach to Science
By John Mashey
1.1 Why This?
I’m always curious when people with decent-or-better educational backgrounds strongly espouse conclusions directly opposite that of mainstream science. Is the mainstream wrong? Have they not yet done sufficient study? Or are there extra-science reasons?…
1.2 Ideas, Hypotheses, Theories in Science
To paraphrase Stanford Professor Stephen Schneider, in any scientific discipline, ideas can be roughly categorized as:
(S3) Some things are well-established – strong proof is required to overturn [strong theory]
(S2) Some things (especially measured effects) have competing explanations. [hypotheses]
(S1) Some things are speculation. [ideas]
S3 includes:
(a) simple statements (b) statements about probability distributions of some measured or projected quantity, often expressed with confidence intervals. Many people are far more comfortable with single numbers than distributions, and are not accustomed to error bars on measurements.
Sometimes, S2 hypotheses gain strength because new measurements shrink confidence intervals small enough that competing hypotheses can be ruled out.
Example: (a) cigarette smoking increases the risk of disease, now sufficiently understood in the US that few adults start smoking. (b) Numerous studies provide statistical measures of the likelihood of developing various diseases. Of course, controlled lab experiments on humans are not feasible.
Example: (a) CO2 is a greenhouse gas and the recent increase in CO2 is substantially due to human activities, (b) The temperature sensitivity to doubling CO2 (from 280ppm to 560ppm) is believed to be in range ~1.5C to ~4.5C, not exactly 3C.
While some technical disciplines allow/require exact yes/no answers, this is rare among complex observational disciplines for which simple lab experiments or mathematical proofs do not exist. Natural scientists in particular require substantial ambiguity tolerance.
People new to a specific topic are often confused by fierce arguments on S2 or S1 and think they are about S3, especially if just reading abstracts of research papers. Benny Peiser and KM Schulte both displayed especially severe cases of this error in attacking Naomi Oreskes’ 2004 essay in Science.
It is important for a newcomer to read, not just the current mainstream view, but enough history to understand its development, and understand where a given idea fits and how long it has been there.
If one studies science histories, one can find obvious progressions, which might be roughly categorized as follows, and based only on peer-reviewed science, since nothing else counts for much:
Case 1 S1 -> nowhere
Case 2 S1 -> S2, but refuted fairly quickly, via errors, new data
Case 3 S1 -> S2…, but competing explanations persist, long battles
Case 4 S1 -> S2 -> S3 wins over competing explanations
Case 1: someone publishes something that is not very interesting, seems to have no way to be falsified, or turns out to have serious errors.
Peer-review only says “We didn’t find obvious errors in a quick review and this might be worth reading”. Passing peer-review does not prove correctness or importance. Not being able to get papers through peer-review should be a major red flag for the reader. Scientists with new major results do not just write OpEds or web pages, or publish them in Energy&Environment or Journal for Scientific Exploration. They send their results to Science, Nature, or other credible journals.
Case 2: an interesting new idea appears and gets attention, but then fairly quickly (within a few years) gets refuted, or claims of strong effects get weakened.
Example: Richard Lindzen’s “Iris” hypothesis attracted interest, but did not gain widespread scientific support, as substantial conflicting evidence existed. There may still be some interest, but this paper did not suddenly invalidate AGW as some wanted to think.
Case 3: multiple hypotheses arise and persist for some time, gathering support, being modified, sometimes combining, or failing to accumulate evidence. An issue can stay open decades, and then quickly be resolved if the right new data or explanation appears.
Example: Geologists argued fiercely for many decades over Alfred Wegener’s hypothesis of continental drift, but when enough new kinds of data appeared following World War II, most geologists quickly accepted it.
Case 4: some hypothesis has gained additional supporting data from multiple research efforts, and is accepted as a strong theory. This may well take decades, and there is often a continuous transition from hypothesis to well-supported theory, not a sudden jump, although the latter occasionally happens. Any theory is just an approximation to reality, and a good new theory must explain everything a previous good theory did, plus be a better approximation.
Example: it took many years to accumulate data that showed the health effects of tobacco. Some chemicals in cigarette smoke are known to be carcinogens without knowing the exact biochemical processes that cause them to be so.
Example: Newton’s laws of motion work pretty well on Earth, well enough to launch satellites. Einstein’s work better, and are needed for GPS satellites. Relativity is often revered, not just because it explained existing awkward data, but because it made many correct predictions of effects that were not yet observable.
Example: the idea that H. Pylori bacteria caused some peptic ulcers went from an odd idea to being well-accepted in few years, and of course, Warren and Marshall got a Nobel for their work. Important wacko ideas that turn out to be true are big wins.
The publication cycle of the most credible peer-reviewed journals is long enough that a non-expert should be prepared to be wary of any paper only 1-2 years old, especially if it has novel implications counter to mainstream established science. It normally takes at least several years to reach Case 3, and many more for Case 4.
Some people fasten on any new paper without understanding this, and in some cases, persist for years in referencing papers that have long since been refuted.
Sometimes, a good, or even great, scientist will become fixated on some idea, and will fight on in its behalf … forever. Most scientists change their minds when the balance of evidence becomes clear, but some do not.
Example: Sir Fred Hoyle was a great astrophysicist, but fought for the “steady-state universe” long after overpowering evidence had accumulated for the Big Bang. Halton Arp has done fine observational astronomy, but also has not accepted the Big Bang.
Example: Sir Ronald Fisher was a great statistician, but never accepted the statistical evidence for the smoking-cancer connection.
Current example of real science one can watch happening: my favorite example of scientific process in visible action can be found in William Ruddiman’s “Plows, Plagues, and Petroleum” plus surrounding papers, arguments, counter-arguments, modifications, to-and-fro-ing. Bill offers several somewhat surprising hypotheses (early CO2, early CH4, and more recent plague effects on CO2), with enough evidence from a highly-regarded researcher to make it to S2, but not yet (and maybe never) part of S3.
Unlike many arguments, this set is actually understandable to non-experts (like me), and one can actually watch science in progress. These hypotheses may linger with insufficient evidence to confirm or deny (Case 3), may get refuted later, or may turn out to be brilliant multidisciplinary theories accepted as the best explanations for otherwise puzzling data.
1.3 Metaphor: The Great Wall of Science
Think of progress in a scientific discipline as building a large structure, like the Great Wall of China (well, Olympics has been on :-), with multiple segments (sub-disciplines) each building upwards, but also trying to connect to form one consistent, connected whole. The Wall is so big that no one can see the whole thing. Quite often really exciting work happens in the gaps between well-established segments. Following are examples of the various Cases.
Case 1: Someone places a brick (S1), but no one else cares, so it never connects with the Wall.
Case 2: Someone else quickly appears and kicks the brick away (Case 2).
Case 3: The brick looks OK, and others start working with it, perhaps moving it around, or placing more bricks and mortar atop it (S2). Perhaps other groups do the same thing with competing alternative segments of the Wall.
Case 4: Sooner or later, some segment acquires enough bricks, and mortar, and even steel rebars, at which point it is enough stronger than the alternates, that the latter are abandoned.
A new brick anywhere is not yet mortared in, and probably takes a few years, even if it’s atop the existing wall, i.e., a refinement of the mainstream. If a new brick is a bit wobbly, that doesn’t mean the Wall collapses. Nobody cares very much about a brick until it has been tested, and mortared with others. In particular, a tall stack of bricks erected by one worker alone, with no connections, may carry very little interest, and falls over easily. Important papers in science get cited positively by other people, not just by the authors and colleagues, and not just to refute.
Measurement errors happen. ARGO buoys or weather balloons are found to have calibration problems. After years of use, computer programs for satellite temperature calculations are found to have simple sign errors. That’s life.
A new brick placed far away from the Wall has to be very compelling to pull efforts in that direction (H. pylori and peptic ulcers). Scientists are strongly motivated to establish such new directions, not just add another brick to the Wall, as it’s a good way to get a Nobel, as happened in that case.
Sometimes a well-established part of a Wall runs into a height limit, and needs a whole new level, i.e., Newton -> Einstein. The lower level is fine as far as I goes, but the second level is a better approximation. People have many hypotheses for the next level, but there is as yet no agreement.
For some people, usually not those directly involved in Wall-building, the appearance of a single brick anywhere else is enough to declare its collapse. Some may cite collections of old discarded bricks, cherry-pick specific bricks, or ignore inconvenient recent bricks, mortar and even steel, and claim the whole Wall is down. People routinely claim to have disproved long-established major laws of physics, which might be considered steel-reinforced concrete (like laws of Thermodynamics), and others then publicize such claims as proof of collapse.
It is very rare for long-established, rebarred Wall segments to be torn down or even reworked in major ways. When it does happen, it is almost always done by people experienced in the field, not by amateurs. Long-established AGW Walls are not demolished in a few months of part-time effort by 15-year-old students without much knowledge of physics and statistics. It is sad but true that most scientific breakthroughs are not generated by unknown lone scientists working alone in their basements.
It is easy for a non-expert, starting with the wrong book, website, or blog, to become convinced that AGW is all wrong, especially with a snapshot at one point in time, and especially if they get pulled into a self-reinforcing group that knows this.
(Ruddiman calls this an amazing “alternate universe” in which “most of the basic findings of mainstream science are rejected or ignored.”)
It takes time for a non-expert to assess authors. If someone claims the Wall is wrong, but relies mostly on workers who contributed little, or cites long-discarded bricks, or changes their reasons every few years, then one can assume they are doing anti-science. It is well worth going back 5-20 years and seeing how people did or did not change their views. It is worth checking the publication records of those referenced. One must be especially careful when an expert builder on one segment retires, and then suddenly starts opining on a completely different segment in directions totally opposite the current workers there.
Real scientists would describe the Wall by how well-established each element was. If a non-expert backtracks what good scientists say, they’ll find that wrong ideas get discarded, good ones progress and gain support, and understanding improves.
The IPCC is especially explicit about its confidence levels.
1.4 Extraordinary and Non-extraordinary Claims
Extraordinary Claims
Carl Sagan was known for saying “Extraordinary claims require extraordinary evidence”, although this was more likely occasioned by his thoughts about parapsychology and other “interesting” ideas. I’ve read plenty of these for fun, and because every once in a while, something crazy turns out to be a better approximation … but hardly ever.
Non-extraordinary Position On AGW
I subscribe to the mainstream science position that global warming is real, is substantially caused by humans, and will very likely cause serious problems under Business-As-Usual (BAU) assumptions.
Clouds, ocean heat exchange, and aerosols, especially contribute uncertainty, but I believe the IPCC’s uncertainty bounds are reasonable, and I know smart people are working hard to tighten the bounds. Like many, I do worry about inherently-conservative IPCC forecasts in the presence of potentially non-linear effects/tipping points and I think there is easily enough evidence to require action. (I say “inherently” given the nature of the IPCC process, as discussed with a useful number of IPCC authors.)
Any non-expert (like me) could arrive at that same non-extraordinary position in two different ways:
(a) Default Acceptance of Mainstream
One either doesn’t know enough physics/math/statistics, or doesn’t want to spend the time to study AGW deeply, so one assumes the professional mainstream opinion is the best approximation of reality available.
Modern science is so huge that nobody can know everything, even in a specific discipline, so (a) is what most people have to do most of the time on most topics. One would find a few credible sources, understand the position, perhaps talk to a few experts, and that would be sufficient. [In Part 3, see Peter Darbee for an example of a smart non-technical person’s approach.]
(b) Deeper Inspection by Interested (Classical) Skeptic
One conditionally assumes the mainstream, but really wants to study the topic to be sure, to be able to discuss the topic intelligently, and to give the objections every reasonable chance.
One generates a list of concerns about the mainstream position, studies each one in depth, and see whether the list grows or shrinks. Scientists think about preponderance of evidence, and are usually alert to contradictory data that might be right, since when found, such often lead to advances. Quite often, contradictory data turns out to be erroneous, or when fixed, is well within the plausible range.
This requires studying a selection of those disagreeing with the mainstream, and carefully assessing what they say, and giving them every possible chance to prove their cases.
However, it also requires some familiarity with well-established non-science methods of attacking science, and ability to assess credibility of sources’ biases in any direction, possibly from ideology or economics,
One might go even further into studying anti-science memes, how they spread, who spreads them, how they work, their psychology and demographics, etc. (For me, this is a continuation of a long interest in science vs non-science issues.)
(c) Anti-Mainstream Position
Suppose an educated person takes the position:
“Even though I’m no expert, I’m almost certain mainstream scientists are wrong and these others are right.”
I’d call that an extraordinary claim, in which case I ask my usual questions to try to understand why someone takes this position. It might be that they know more than the professionals, and thus vastly more than I do, in which case I could learn something new … but, HARDLY EVER.
Since AGW is non-extraordinary mainstream science, I’m not sure it needs a lot of justification…
Leave a Reply