Errors in the IPCC and perspective

It was inevitable. The IPCC AR4 is over 3000 pages long, there are bound to be some errors contained within. But some perspective is needed when they are found; jumping to the conclusion that any error in the IPCC is proof that climate science is bunk, or that global warming is a sham is absurd.

The New York Times has errors literally every single day, and sometimes they are significant errors. But news media don’t act as if the NYT is totally unreliable and built on a hoax. -Brad Johnson

Yet deniers do this all the time, and the media falls for it.

Here is another examples from James Hrynyshyn:

Here’s IPCC author Phil Duffy, whose thoughts on the subject inspired mine:

Things happen, but let’s react appropriately. Medical doctors make mistakes every day. (In fact, medical errors in the US alone kill hundreds of people daily–the equivalent of a jumbo-jet crash.) And no doubt many of these errors happen because established procedures are ignored, sometimes knowingly. Does this mean the entire edifice of western medicine is wrong, or prejudiced, or the product of a conspiracy, and should be rejected? Of course not. Furthermore, the medical profession as a whole is still held in high regard, as it should be.

No one worth listening to is calling for a massive inquiry into the science underpinning modern medicine, or the engineering foundations of the car industry. But pseudoskeptics argue that the IPCC is systematically fraudulent simply because a couple of statements among thousands of pages of heavily edited and re-editing (and re-re-edited) documents cite gray literature instead of the peer-reviewed literature that supplied the science in the first place…

To thrown out anthropogenic global warming because of such missteps is the climatological analog of dismissing an entire faculty of medicine because someone correctly diagnosed a patient because of a story they read in New Scientist instead of the medical journal article on which the story was based. Bad judgment? Yes. Fatal error? No.

And Marc Roberts (creator of the above comic) explains why all of this is nothing more than an attempt to muddy the waters:

So much mud is being thrown that the slightest speck sticking to a labcoat seems enough to persuade Joe Public that the bigger scientific picture is irrelevant and that it must, therefore, all boil down to moustache twirling evil-doers Mwa-haar-haaring under ill-gotten showers of public gold. We do love a soundbite. So I guess it must be OK if the glaciers start to disappear in 50 years, not 20, and the African harvests collapse in 40 years, not 20, and if a quarter of Holland is lost to the sea, not a half. That’ll be fine. Nothing to worry about. Party on dudes.

Lest we forget – it was the IPCC and the UEA themselves that brought their own errors to the attention of their peers.

Errors will happen; that is a fact of life. They will happen no matter how many peers review your writing. But when they happen they need to be addressed honestly, and with perspective. An error determining when Himalayan glaciers will disappear, does not undermine the whole IPCC.  Neither does citing the WWF instead of the peer-reviewed literature.

Anyone who wishes to demonstrate that the science is wrong or fraudulent needs much more than a few errors. Anyone who claims otherwise is either dishonest, or lacks perspective.

UPDATE: Steve Easterbrook has tried to answer the question How many errors should we expect in the IPCC reports? by comparing the IPCC review process to NASA’s code review process for the space shuttle flight code, which is the most reviewed code in existence.

If the IPCC review process is as thorough as NASA’s, then we should still expect around 30 significant errors made it through the review process. We’ve heard of two recently – does this mean we have to endure another 28 stories, spread out over the next few months, as the drone army of denialists toils through trying to find more mistakes? Actually, it’s probably worse than that…

The IPCC writing, editing and review processes are carried out entirely by unpaid volunteers. They don’t have automated testing and static analysis tools to help – human reviewers are the only kind of review available. So they’re bound to do much worse than NASA’s flight software. I would expect there to be 100s of errors in the reports, even with the best possible review processes in the world. Somebody point me to a technical review process anywhere that can do better than this, and I’ll eat my hat. Now, what was the point of all those newspaper stories again? Oh, yes, sensationalism sells.

Again this highlights the importance of having perspective when errors are found. They are inevitable, and their mere existence doesn’t automatically discount the rest of the IPCC. Each error needs to be looked at to determine what if any effect it has on the overall picture.

UPDATE 2: Climatologist Andrew Weaver expands on this whole issue:

UPDATE 3: RealClimate has done an excellent and thorough write-up of the errors in the IPCC and their significance (or rather lack thereof) to the big picture:

IPCC errors: facts and spin

Currently, a few errors –and supposed errors– in the last IPCC report (“AR4″) are making the media rounds – together with a lot of distortion and professional spin by parties interested in discrediting climate science.  Time for us to sort the wheat from the chaff: which of these putative errors are real, and which not? And what does it all mean, for the IPCC in particular, and for climate science more broadly?

Let’s start with a few basic facts about the IPCC.  The IPCC is not, as many people seem to think, a large organization. In fact, it has only 10 full-time staff in its secretariat at the World Meteorological Organization in Geneva, plus a few staff in four technical support units that help the chairs of the three IPCC working groups and the national greenhouse gas inventories group. The actual work of the IPCC is done by unpaid volunteers – thousands of scientists at universities and research institutes around the world who contribute as authors or reviewers to the completion of the IPCC reports. A large fraction of the relevant scientific community is thus involved in the effort.  The three working groups are:

Working Group 1 (WG1), which deals with the physical climate science basis, as assessed by the climatologists, including several of the Realclimate authors.

Working Group 2 (WG2), which deals with impacts of climate change on society and ecosystems, as assessed by social scientists, ecologists, etc.

Working Group 3 (WG3) , which deals with mitigation options for limiting global warming, as assessed by energy experts, economists, etc.

Assessment reports are published every six or seven years and writing them takes about three years. Each working group publishes one of the three volumes of each assessment. The focus of the recent allegations is the Fourth Assessment Report (AR4), which was published in 2007.  Its three volumes are almost a thousand pages each, in small print. They were written by over 450 lead authors and 800 contributing authors; most were not previous IPCC authors. There are three stages of review involving more than 2,500 expert reviewers who collectively submitted 90,000 review comments on the drafts. These, together with the authors’ responses to them, are all in the public record (see here and here for WG1 and WG2 respectively).

Errors in the IPCC Fourth Assessment Report (AR4)

As far as we’re aware, so far only one–or at most two–legitimate errors have been found in the AR4:

Himalayan glaciers: In a regional chapter on Asia in Volume 2, written by authors from the region, it was erroneously stated that 80% of Himalayan glacier area would very likely be gone by 2035. This is of course not the proper IPCC projection of future glacier decline, which is found in Volume 1 of the report. There we find a 45-page, perfectly valid chapter on glaciers, snow and ice (Chapter 4), with the authors including leading glacier experts (such as our colleague Georg Kaser from Austria, who first discovered the Himalaya error in the WG2 report).  There are also several pages on future glacier decline in Chapter 10 (“Global Climate Projections”), where the proper projections are used e.g. to estimate future sea level rise. So the problem here is not that the IPCC’s glacier experts made an incorrect prediction. The problem is that a WG2 chapter, instead of relying on the proper IPCC projections from their WG1 colleagues, cited an unreliable outside source in one place. Fixing this error involves deleting two sentences on page 493 of the WG2 report.

Sea level in the Netherlands: The WG2 report states that “The Netherlands is an example of a country highly susceptible to both sea-level rise and river flooding because 55% of its territory is below sea level”. This sentence was provided by a Dutch government agency – the Netherlands Environmental Assessment Agency, which has now published a correction stating that the sentence should have read “55 per cent of the Netherlands is at risk of flooding; 26 per cent of the country is below sea level, and 29 per cent is susceptible to river flooding”. It surely will go down as one of the more ironic episodes in its history when the Dutch parliament last Monday derided the IPCC, in a heated debate, for printing information provided by … the Dutch government. In addition, the IPCC notes that there are several definitions of the area below sea level. The Dutch Ministry of Transport uses the figure 60% (below high water level during storms), while others use 30% (below mean sea level). Needless to say, the actual number mentioned in the report has no bearing on any IPCC conclusions and has nothing to do with climate science, and it is questionable whether it should even be counted as an IPCC error.

Some other issues

African crop yields: The IPCC Synthesis Report states: “By 2020, in some countries, yields from rain-fed agriculture could be reduced by up to 50%.” This is properly referenced back to chapter 9.4 of WG2, which says: “In other countries, additional risks that could be exacerbated by climate change include greater erosion, deficiencies in yields from rain-fed agriculture of up to 50% during the 2000-2020 period, and reductions in crop growth period (Agoumi, 2003).”  The Agoumi reference is correct and reported correctly. The Sunday Times, in an article by Jonathan Leake,labels this issue “Africagate” – the main criticism being that Agoumi (2003) is not a peer-reviewed study (see below for our comments on “gray” literature), but a report from the International Institute for Sustainable Development and the Climate Change Knowledge Network, funded by the US Agency for International Development. The report, written by Morroccan climate expert Professor Ali Agoumi, is a summary of technical studies and research conducted to inform Initial National Communications from three countries (Morocco, Algeria and Tunisia) to the United Nations Framework Convention on Climate Change, and is a perfectly legitimate IPCC reference.

It is noteworthy that chapter 9.4 continues with “However, there is the possibility that adaptation could reduce these negative effects (Benhin, 2006).”  Some examples thereof follow, and then it states: “However, not all changes in climate and climate variability will be negative, as agriculture and the growing seasons in certain areas (for example, parts of the Ethiopian highlands and parts of southern Africa such as Mozambique), may lengthen under climate change, due to a combination of increased temperature and rainfall changes (Thornton et al., 2006). Mild climate scenarios project further benefits across African croplands for irrigated and, especially, dryland farms.” (Incidentally, the Benhin and Thornton references are also “gray”, but nobody has complained about them. Could there be double standards amongst the IPCC’s critics?)

Chapter 9.4 to us sounds like a balanced discussion of potential risks and benefits, based on the evidence available at the time–hardly the stuff for shrill “Africagate!” cries. If the IPCC can be criticized here, it is that in condensing these results for its Synthesis Report, important nuance and qualification were lost – especially the point that the risk of drought (defined as a 50% downturn in rainfall) “could be exacerbated by climate change”, as chapter 9.4 wrote – rather than being outright caused by climate change.

Trends in disaster losses: Jonathan Leake (again) in The Sunday Times accused the IPCC of wrongly linking global warming to natural disasters. The IPCC in a statement points out errors in Leake’s “misleading and baseless story”, and maintains that the IPCC provided “a balanced treatment of a complicated and important issue”. While we agree with the IPCC here, WG2 did include a debatable graph provided by Robert Muir-Wood (although not in the main report but only as Supplementary Material). It cited a paper by Muir-Wood as its source although that paper doesn’t include the graph, only the analysis that it is based on. Muir-Wood himself has gone on record to say that the IPCC has fairly represented his research findings and that it was appropriate to include them in the report. In our view there is no IPCC error here; at best there is a difference of opinion. Obviously, not every scientist will always agree with assessments made by the IPCC author teams.

Amazon forest dieback: Leake (yet again), with “research” by skeptic Richard North, has also promoted “Amazongate” with a story regarding a WG2 statement on the future of Amazonian forests under a drying climate.  The contested IPCC statement reads: “Up to 40% of the Amazonian forests could react drastically to even a slight reduction in precipitation; this means that the tropical vegetation, hydrology and climate system in South America could change very rapidly to another steady state, not necessarily producing gradual changes between the current and the future situation (Rowell and Moore, 2000).”  Leake’s problem is with the Rowell and Moorereference, a WWF report.

The roots of the story are in two blog pieces by North, in which he first claims that the IPCC assertions attributed to the WWF report are not actually in that report. Since this claim was immediately shown to be false,  North then argued that the WWF report’s basis for their statement (a 1999 Nature article by Nepstad et al.) dealt only with the effects of logging and fire –not drought– on Amazonian forests. To these various claims Nepstad has now responded, noting that the IPCC statement is in fact correct. The only issue is that the IPCC cited the WWF report rather than the underlying peer-reviewed papers by Nepstad et al. These studies actually provide the  basis for the IPCC’s estimate on Amazonian sensitivity to drought. Investigations of the correspondence between Leake, scientists, and a BBC reporter (see here and here and here) show that Leake ignored or misrepresented explanatory information given to him by Nepstad and another expert, Simon Lewis, and published his incorrect story anyway. This “issue” is thus completely without merit.

Gray literature: The IPCC cites 18,000 references in the AR4; the vast majority of these are peer-reviewed scientific journal papers. The IPCC maintains a clear guideline on the responsible use of so-called “gray” literature, which are typically reports by other organizations or governments. Especially for Working Groups 2 and 3 (but in some cases also for 1) it is indispensable to use gray sources, since many valuable data are published in them: reports by government statistics offices, the International Energy Agency, World Bank, UNEP and so on. This is particularly true when it comes to regional impacts in the least developed countries, where knowledgeable local experts exist who have little chance, or impetus, to publish in international science journals.

Reports by non-governmental organizations like the WWF can be used (as in the Himalaya glacier and Amazon forest cases) but any information from them needs to be carefully checked (this guideline was not followed in the former case). After all, the role of the IPCC is to assessinformation, not just compile anything it finds.  Assessment involves a level of critical judgment, double-checking, weighing supporting and conflicting pieces of evidence, and a critical appreciation of the methodology used to obtain the results. That is why leading researchers need to write the assessment reports – rather than say, hiring graduate students to compile a comprehensive literature review.

Media distortions

To those familiar with the science and the IPCC’s work, the current media discussion is in large part simply absurd and surreal. Journalists who have never even peeked into the IPCC report are now outraged that one wrong number appears on page 493 of Volume 2. We’ve met TV teams coming to film a report on the IPCC reports’ errors, who were astonished when they held one of the heavy volumes in hand, having never even seen it. They told us frankly that they had no way to make their own judgment; they could only report what they were being told about it. And there are well-organized lobby forces with proper PR skills that make sure these journalists are being told the “right” story. That explains why some media stories about what is supposedly said in the IPCC reports can easily be falsified simply by opening the report and reading. Unfortunately, as a broad-based volunteer effort with only minimal organizational structure the IPCC is not in a good position to rapidly counter misinformation.

One near-universal meme of the media stories on the Himalaya mistake was that this was “one of the most central predictions of the IPCC” – apparently in order to make the error look more serious than it was.  However, this prediction does not appear in any of the IPCC Summaries for Policy Makers, nor in the Synthesis Report (which at least partly explains why it went unnoticed for years). None of the media reports that we saw properly explained that Volume 1 (which is where projections of physical climate changes belong) has an extensive and entirely valid discussion of glacier loss.

What apparently has happened is that interested quarters, after the Himalyan glacier story broke, have sifted through the IPCC volumes with a fine-toothed comb, hoping to find more embarrassing errors. They have actually found precious little, but the little they did find was promptly hyped into Seagate, Africagate, Amazongate and so on. This has some similarity to the CRU email theft, where precious little was discovered from among thousands of emails, but a few sentences were plucked out of context, deliberately misinterpreted (like “hide the decline”) and then hyped into “Climategate”.

As lucidly analysed by Tim Holmes, there appear to be a few active leaders of this misinformation parade in the media. Jonathan Leake is carrying the ball on this, but his stories contain multiple errors, misrepresentations and misquotes. There also is a sizeable contingent of me-too journalism that is simply repeating the stories but not taking the time to form a well-founded view on the topics. Typically they report on various “allegations”, such as these  against the IPCC, similar to reporting that the CRU email hack lead to “allegations of data manipulation”. Technically it isn’t even wrong that there were such allegations. But isn’t it the responsibility of the media to actually investigate whether allegations have any merit before they decide to repeat them?

Leake incidentally attacked the scientific work of one of us (Stefan) in a Sunday Times article in January. This article was rather biased and contained some factual errors that Stefan asked to be corrected. He has received no response, nor was any correction made. Two British scientists quoted by Leake – Jonathan Gregory and Simon Holgate – independently wrote to Stefan after the article appeared to say they had been badly misquoted. One of them wrote that the experience with Leake had made him “reluctant to speak to any journalist about any subject at all”.

Does the IPCC need to change?

The IPCC has done a very good job so far, but certainly there is room for improvement. The review procedures could be organized better, for example. Until now, anyone has been allowed to review any part of the IPCC drafts they liked, but there was no coordination in the sense that say, a glacier expert was specifically assigned to double-check parts of the WG2 chapter on Asia. Such a practice would likely have caught the Himalayan glacier mistake. Another problem has been that reports of all three working groups had to be completed nearly at the same time, making it hard for WG2 to properly base their discussions on the conclusions and projections from WG1. This has already been improved on for the AR5, for which the WG2 report can be completed six months after the WG1 report.

Also, these errors revealed that the IPCC had no mechanism to publish errata. Since a few errors will inevitably turn up in a 2800-page report, obviously an avenue is needed to publish errata as soon as errors are identified.

Is climate science sound?

In some media reports the impression has been given that even the fundamental results of climate change science are now in question, such as whether humans are in fact changing the climate, causing glacier melt, sea level rise and so on. The IPCC does not carry out primary research, and hence any mistakes in the IPCC reports do not imply that any climate research itself is wrong. A reference to a poor report or an editorial lapse by IPCC authors obviously does not undermine climate science. Doubting basic results of climate science based on the recent claims against the IPCC is particularly ironic since none of the real or supposed errors being discussed are even in the Working Group 1 report, where the climate science basis is laid out.

To be fair to our colleagues from WG2 and WG3, climate scientists do have a much simpler task. The system we study is ruled by the well-known laws of physics, there is plenty of hard data and peer-reviewed studies, and the science is relatively mature. The greenhouse effect was discovered in 1824 by Fourier, the heat trapping properties of CO2 and other gases were first measured by Tyndall in 1859, the climate sensitivity to CO2 was first computed in 1896 by Arrhenius, and by the 1950s the scientific foundations were pretty much understood.

Do the above issues suggest “politicized science”, deliberate deceptions or a tendency towards alarmism on the part of IPCC? We do not think there is any factual basis for such allegations. To the contrary, large groups of (inherently cautious) scientists attempting to reach a consensus in a societally important collaborative document is a prescription for reaching generally “conservative” conclusions. And indeed, before the recent media flash broke out, the real discussion amongst experts was about the AR4 having underestimated, not exaggerated, certain aspects of climate change. These include such important topics as sea level rise and sea ice decline (see the sea ice and sea level chapters of the Copenhagen Diagnosis), where the data show that things are changing faster than the IPCC expected.

Overall then, the IPCC assessment reports reflect the state of scientific knowledge very well. There have been a few isolated errors, and these have been acknowledged and corrected. What is seriously amiss is something else: the public perception of the IPCC, and of climate science in general, has been massively distorted by the recent media storm. All of these various “gates” – Climategate, Amazongate, Seagate, Africagate, etc., do not represent scandals of the IPCC or of climate science. Rather, they are the embarrassing battle-cries of a media scandal, in which a few journalists have misled the public with grossly overblown or entirely fabricated pseudogates, and many others have naively and willingly followed along without seeing through the scam. It is not up to us as climate scientists to clear up this mess – it is up to the media world itself to put this right again, e.g. by publishing proper analysis pieces like the one of Tim Holmes and by issuing formal corrections of their mistaken reporting. We will follow with great interest whether the media world has the professional and moral integrity to correct its own errors.

PS. A new book by Realclimate-authors David Archer and Stefan Rahmstorf critically discussing the main findings of the AR4 (all three volumes) is just out: The Climate Crisis. None of the real or alleged errors are in this book, since none of those contentious statements plucked from the thousands of pages appeared to be “main findings” that needed to be discussed in a 250-page summary.

PPS. Same thing for Mike’s book Dire Predictions: Understanding Global Warming, which bills itself as “The illustrated guide to the findings of the IPCC”. Or Gavin’s “Climate Change: Picturing the Science” – which does include a few pictures of disappearing glaciers though!