Iain Murray wrote an excellent piece at Pajamas Media regarding the three things you must know about Climategate (the hacked CRU email and data). Despite being excellent, I think there’s one more to add. While the emails got a lot of attention, a file called HARRY_READ_ME.txt is finally getting some attention. And wow, is it interesting. Even CBS has taken notice: (H/T: Hot Air)
As the leaked messages, and especially the HARRY_READ_ME.txt file, found their way around technical circles, two things happened: first, programmers unaffiliated with East Anglia started taking a close look at the quality of the CRU’s code, and second, they began to feel sympathetic for anyone who had to spend three years (including working weekends) trying to make sense of code that appeared to be undocumented and buggy, while representing the core of CRU’s climate model.
The link has some good excecrpts, but The Devil’s Kitchen has more, plus commentary. Frankly, I encourage you to read the original file. Whoever this Harry person is, he at least knows how to keep an entertaining log. Some fun bits:
It’s Sunday evening, I’ve worked all weekend, and just when I thought it was done I’m hitting yet another problem that’s based on the hopeless state of our databases. There is no uniform data integrity, it’s just a catalogue of issues that continues to grow as they’re found.
Back to the gridding. I am seriously worried that our flagship gridded data product is produced by Delaunay triangulation – apparently linear as well.
As far as I can see, this renders the station counts totally meaningless.
So.. we don’t have the coefficients files (just .eps plots of something). But what are all those monthly files? DON’T KNOW, UNDOCUMENTED. Wherever I look, there are data files, no info about what they are other than their names. And that’s useless.. take the above example, the filenames in the _mon and _ann directories are identical, but the contents are not. And the only difference is that one directory is apparently ‘monthly’ and the other ‘annual’ – yet both contain monthly files.
19. Here is a little puzzle. If the latest precipitation database file contained a fatal data error (see 17. above), then surely it has been altered since Tim last used it to produce the precipitation grids? But if that’s the case, why is it dated so early?
But, (Lord how many times have I used ‘however’ or ‘but’ in this file?!!)
First problem: there is no program to convert sun percentage to cloud percentage. I can do sun percentage to cloud oktas or sun hours to cloud percentage! So what the hell did Tim do?!! As I keep asking.
So what’s the fourth take home message from Climategate? This: CRU’s temperature profile is an incoherent mess. Harry’s basically trying to fit the programming to published results to see how they did it, and can’t. Along the way, he discovers garbage data, horrible code, undocumented files, unexplained paranormal phenomena, and piles upon piles of errors. Hadley earlier this year stated they can’t release the original data because they lost it. If this file is true, then the current code is useless too, as it contains too much garbage, fudging, and improper procedures.
Or, to put it succinctly: CRU’s data is not in a usable form, and should not be used for further investigation. OK, so I can’t say that for certain yet, but that seems to be the implication of all this.
So what does that mean? Given the collusions going on in global warming research, we ought to demand full access to GISS and NCDC temperature profiles and the method, data, and code used to create them. If they are as big of a mess as CRU, then they too should be tossed.
Which would mean all we’d have left is satellite data. Which means the global warming models would be based on satellite data. And given the differences between satellite data and surface temperatures, suddenly global warming won’t look as bad as they claim.
We’re a long way from that point. We have no proof that GISS and NCDC are as messed up as this appears to be. But I think this is reason enough to start some FOI requests on historical temperature data. And if things turn out bad, then those that said this whole climategate issue means nothing in regards to the actual science may turn out to be very, very wrong.