has the world's temperature data really gone missing?

The climate change deniers are raising another stink. Global Warming ate my data, frets Andrew Orlowski in the Register. The Dog Ate Global Warming, pines Patrick J. Michaels in the National Review.

What exactly has been eaten here?

Michaels asks us to:

Imagine if there were no reliable records of global surface temperature. Raucous policy debates such as cap-and-trade would have no scientific basis, Al Gore would at this point be little more than a historical footnote, and President Obama would not be spending this U.N. session talking up a (likely unattainable) international climate deal in Copenhagen in December.

Orlowski, on the other hand, gets right to the point:

The world's source for global temperature record admits it's lost or destroyed all the original data that would allow a third party to construct a global temperature record. The destruction (or loss) of the data comes at a convenient time for the Climatic Research Unit (CRU) in East Anglia - permitting it to snub FoIA requests to see the data.

On the face of it, this sounds preposterous. Could it really be true that all the world's historical temperature records from thousands of measurement sites were stored in one location only, and have suddenly been destroyed in one fell swoop?

Michaels is a little more careful; he's not making such absolute statements without some hedging:

Steel yourself for the new reality, because the data needed to verify the gloom-and-doom warming forecasts have disappeared.

Or so it seems. Apparently, they were either lost or purged from some discarded computer. Only a very few people know what really happened, and they aren’t talking much. And what little they are saying makes no sense.

What data are they talking about, and where did it go?

Much of what we know about climate change is due to the efforts of two organizations that have spent decades collecting and aggregating temperature measurements from around the world. The Climate Research Unit, headed by Phil Jones of the University of East Anglia in Norwich, has been collecting climate data from around the world since 1978. The Goddard Institute for Space Studies (GISS), a division of NASA headed by James Hansen, has been gathering climate data since 1961.

Apparently the data controversy began in 2004 (per Orlowski) or 2005 (per Michaels), when one Warwick Hughes — a "free lance earth scientist" (his own description) — requested raw climate data from Phil Jones. According to Michaels:

Jones’s response to a fellow scientist attempting to replicate his work was, “We have 25 years or so invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it?”

If the question sounds harsh, it's perhaps because Warwick Hughes has a long history of antagonism toward Phil Jones. On his website and his blog, Hughes devotes numerous pages to criticism of Jones. For example, in an Ongoing review of 1986 Jones et al papers compiling global temperature trends that now define "IPCC global warming", Hughes calls on Jones to "measure global temperature trends using data that does not include many hundreds of temperature records contaminated by local urban heat islands."

The underlying issue here is that buildings, roads, and other human-built structures trap heat during the daytime. At night, these structures release the heat back into the air. The result is nighttime temperatures that are warmer than the surrounding countryside. A true picture of global temperatures must either measure both urban and rural temperatures, or adjust the urban temperatures to account for the urban heat island (UHI) effect.

Hughes's website also includes an open letter to all authors of the Jones et al 1986 papers and DoE documentation books. In this one, Hughes objects to the inclusion of temperature data from a station in Atlanta:

Personally, I would have thought there was an abundance of evidence in the early 1980's that Atlanta was UHI affected and that evidence has compounded since with NASA web pages portraying satellite thermal imagery defining the Atlanta UHI which they characterise as Hotlanta.

Hughes includes the reply he received from Phil Jones. Dr Jones states, in part:

The Atlanta station you refer to is one of 22 sites within the grid box (30-35N, 80-85W) where Atlanta is located. So even if the data have become more urban affected through time, the effect on the grid-box average would be minor. For the 1985/1986 papers/reports you refer to all the stations were assessed for homogeneity problems.

UHI appears to be a major issue for Hughes. He seems not to be satisfied with the adjustments to the climate aggregations to account for the UHI factor. In fact, Hughes gives his game away in his "ongoing review," where he insists that CRU not adjust, but eliminate, any measurements "contaminated by local urban heat islands".

Perhaps it is the UHI issue that drove Hughes to request the raw data from Jones, or perhaps it was something else. Either way, Jones was unable to supply it.

The problem is, the raw data does not belong to Jones. What Jones and CRU have produced over the last quarter century are a series of aggregates of monthly temperatures collected from more than 3000 sites around the world. The most current dataset is known as CRUTEM3, and is available for download from the CRU website. CRU has obtained these temperature readings from a variety of sources, some of whom are happy to share their data with anyone who asks, and some of whom have strict policies concerning its reuse.

There is nothing sinister about either the agreements CRU signed to obtain the data from disparate sources, or the way CRU has built their aggregation of the measurements. And certainly CRU is being open in sharing its own data.

But the spark lit by Hughes was kindled in 2007 when statistician and mineral exploration expert Stephen McIntyre filed a Freedom of Information Act (FoIA) request for the raw source data. Jones again refused; his curt explanation for not having to supply the data was, as Orlowski's article notes:

Information accessible to applicant via other means. Some information is publicly available on external websites.

In plain English, the data does not belong to CRU, but can be obtained from the original sources. Some of it is quite easy to obtain; just go to the appropriate websites. What Jones leaves unsaid is that some of the data may be difficult to obtain from the sources. But he has gone to great lengths to gather as much data as he possibly could, and anybody else who wants to put forth the effort can obtain the same data.

Jones alludes to some of the difficulty in his later remarks, also reported in Orlowski's Register article:

We are not in a position to supply data for a particular country not covered by the example agreements referred to earlier, as we have never had sufficient resources to keep track of the exact source of each individual monthly value. Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues.

This is in no way inconsistent with his reply to the FoIA request. Again, all the data should be retrievable from the sources, and some of the sources make it quite easy to do so. Some don't, and obtaining their data could be difficult.

Michaels grouses in his National Review article that keeping the data shouldn't have been that difficult for CRU, even with the limited storage available in the 1980s.

The statement about “data storage” is balderdash. They got the records from somewhere. The files went onto a computer. All of the original data could easily fit on the 9-inch tape drives common in the mid-1980s. I had all of the world’s surface barometric pressure data on one such tape in 1979.

But Michaels misses the point. CRU, in fact, did merge the records to fit everything into the available storage. But in merging, CRU lost the distinctions among the original sources. It's not, as Michaels insinuates, that the temperature data has been lost or destroyed. CRU has processed the data and kept the results. The original data can still be obtained from the original sources.

Furthermore, Jones and CRU are not the only ones who have collected and aggregated all this data. In the United States, GISS has been keeping independent climate records since before CRU was established. GISS also makes their data available from their website, so even if CRU had destroyed their data, we would still have global temperature datasets for review. And if GISS and CRU can do it, others can build their own data collections if they are willing to put forth the effort.

The problem is, the climate change deniers don't want to do the work. They simply want to criticize and complain. Because they are too lazy to go to all the sources of temperature measurements around the world, they question whether the measurements even exist. And because organizations like the Register and the National Review give them a free platform for making allegations, they just might confuse a lot of people into thinking there is still some controversy over the reality of global warming.