RealClimate

Comments

RSS feed for comments on this post.

  1. The problem is more than just access to raw data. It is knowing which data has been used in which papers. It is knowing how the data has been manipulated and the justification of that manipulation. This is particularly true in a field that requires as much statistical analysis as paleoclimatology.

    Replication of results is key to the scientific method. Data and data analysis should be released with all published papers. It should be released when requested for academic study. And it should definitely be released when demanded under the FOIA.

    Comment by JPSobel — 27 Nov 2009 @ 9:59 PM

  2. Just a question. Is there anywhere one can view the purloined emails to form one’s own opinion about how damaging they may be? I am deeply concerned for example by Monbiot’s opinions on the content and would like to judge for myself. Thanks.

    [Response: The easiest way is through here. But some of the emails are strangely truncated, not sure why. Please note that these are illegally obtained, though curiosity is a powerful driver.... - gavin]

    Comment by Dr. Alan Keller — 27 Nov 2009 @ 10:10 PM

  3. Agree totally with #1 JP Sobel – I’ve requested data from Briffa and never had a reply to numerous emails. I have never experienced that before in my entire scientific career in the many science fields I work in. I’ve always had an email back in reply with data given freely and in a friendly, helpful manner. In fact, one of the past posts on this site about this subject now reflects poorly on the group as a whole in light of the leaked CRU emails. Better to post all data, all algorithms and all methodology at time of publication so others can replicate with ease. Then, when all of it has to released under FOI request, there will be no emails, papers or statements that can be “taken out of context”.

    [Response: There will always be people to take things out of context - whether these things are released via FOI or stolen. If you want Briffa's data, go to the NOAA Paleoclimate link. - gavin]

    Comment by Richard — 27 Nov 2009 @ 10:19 PM

  4. I recommend including a link to the world data center for greenhouse gases:

    http://gaw.kishou.go.jp/wdcgg/

    Comment by tamino — 27 Nov 2009 @ 10:42 PM

  5. if you *really* want to see how scientists talk behind closed doors . . . . see the following video!

    http://www.youtube.com/watch?v=-VRBWLpYCPY&feature=player_embedded

    Gavin is the co-author on the far left.

    Comment by Jeffrey Park — 27 Nov 2009 @ 10:47 PM

  6. Gavin,
    all you have to do is go open source–no need to hide and invent excuses, you are obviously not very good at that anyway.
    Release all the data (you can keep the 2% of inexplicably protected data that you might have) even if it is just 98% of the total and also release the codes that were used to create/prove the conclusions/graphs that were published.
    It is very simple … people do this [public release of specific work] daily .. see linux and OSS, if you don’t know how to do it and how it works.

    The context is quite clear when it comes to the e-mails of the CRU folks … the only way out, for your gang, is to go open source.

    [Response: GISTEMP has. Why aren't you interested in that? - gavin]

    Comment by donQ — 27 Nov 2009 @ 10:51 PM

  7. Gavin,
    thank you for [Response: GISTEMP has. Why aren't you interested in that? - gavin] …
    unfortunately this is insufficient.
    1) There is no source code provided
    2) There is no coherent data provided (within the context of how such data is used to provide results)

    I also have gone through the codes available in the CRU FOI2009.zip — those source files betray astounding level of incompetence from several angles … 1) management 2) computer-science 3) quality assurance.

    So … again, release the codes that according to you are used to generate your results.

    Perhaps you weren’t aware of it, but the page you provide links to http://data.giss.nasa.gov/gistemp/sources/GISTEMP_sources.tar.gz which is not available according to your web-site. So again, where is the source code?

    [Response: Not sure why that isn't up, it was before. Maybe they are updating for some recent changes in source data. In the mean time, you can see the code here: http://code.google.com/p/open-gistemp/ - gavin]

    Comment by donQ — 27 Nov 2009 @ 11:11 PM

  8. Wow – and they used to call me a flat earther…

    Barney Rubble (my best friend) says that if we used techniques like the CRU during our quest for fire (taxing the roots and berries of all our friends for years to fund our preconceived notion that fire is caused by a complete lack of water) then we would have miserably failed.

    Instead, we just stoned our detractors regularly. Eventually, Barney threw a piece of flint at some poor guy, and started a fire. Voila. I credit our discovery not only to the regular stonings of detractors (a technique CRU did apparently use) but mostly to one thing we had back then that seems to be completely lacking at East Anglia – minds that are open to alternate theories.

    See, even during the stone age our approaches and techniques were vastly superior to the CRU’s.

    - Cheers,
    - Fred Flintstone

    Comment by F. Flintstone — 27 Nov 2009 @ 11:22 PM

  9. Could you also add links for radiosonde data?

    [Response: Good idea. I added RAOBCORE and HadAT - but I think there are a couple more sources. Links welcome. - gavin]

    Comment by Chad — 27 Nov 2009 @ 11:45 PM

  10. Slightly off topic again but u know me. Has anyone read the preliminary warning from University of Manitoba researcher David Barber. He has recently been around arctica including the areas that appear white on the satellite that was believed to be intact multi-year ice. According to his research it’s only solid on the surface beneath that it’s completely ‘rotten’, indicating to me that the warmer arctic ocean currents are contributing far more to the arctic ice loss that air temps. There is virtually no more solid multi year ice left. When he was there he witnessed a massive multi year ice floe break up before his eyes. His prediction is that within 6 years there will be no summer sea ice in the arctic. This data should be rushed through to the IPCC for evaluation and presented in Copenhagen ASAP!

    Comment by Lawrence Coleman — 27 Nov 2009 @ 11:55 PM

  11. Gavin,
    thank you. The google link has code. ;-)
    The total amount of it [code + support files] is 5 MB ... is this all that you have?

    [Response: Me personally? If you want my code, go here. If you mean why isn't there more data for processing the temperature stations, I'm not sure why you want more. It's not that hard (and I'm sure a determined team to cut it down radically). - gavin]

    [PS: The other link to the GISTEMP sources (up to date) is now live again. It's all about who you know.... ;) - gavin]

    Comment by donQ — 27 Nov 2009 @ 11:57 PM

  12. Long time lurker. Great site to find info to smite Deniers with. Not that they ever realize they’ve been smited, unfortunately.

    Anyway, with the links provided, could a skeptic/denier perform an analysis that would disprove man made global warming? (If, in fact, man made global warming WAS just a liberal conspiracy?) If so, why haven’t they already done so? Is there any data missing that would be needed? If so, are there teams of denialists out there working their butts off to gather such data?

    Or are FOI requests the only way they know how to do research?

    Comment by Groucho48 — 28 Nov 2009 @ 12:05 AM

  13. Here are some more links for the radiosondes
    http://weather.uwyo.edu/upperair/sounding.html
    http://www.ncdc.noaa.gov/oa/climate/ratpac/
    http://www.ccrc.unsw.edu.au/staff/profiles/sherwood/radproj/index.html

    [Response: thanks! - gavin]

    Comment by Chad — 28 Nov 2009 @ 12:08 AM

  14. Lawrence Coleman – Barber is obviously mistaken (those scientists, always believing what their senses and sensors tell them) because a CRU staff member said something rude in an email. Just ask George Monbiot, or the 2000 or so responses to his mea culpa (well, thea culpa, really I suppose) – global warming is all over. Pouff, never existed, gone, just like a bad dream. All the fault of Phil Jones and his vivid imagination. We were so close to a one world government by communists – just as well someone leaked emails before we were all living in ice caves again.

    Comment by David Horton — 28 Nov 2009 @ 12:08 AM

  15. Gavin, have you ever looked into software to support “reproducible
    research” and “literate programming”? There are some good packages out there, like Sweave, org-babel, python.sty, and Literate Haskell. The basic idea is go make a .tar.gz which has code + latex + data + a build script. The paper plus all the figures can then be rebuilt in one command by anyone. This technique is used by a lot of people in bioinformatics and statistics and I think it could be the answer to a lot of problems.

    Comment by Assad — 28 Nov 2009 @ 12:14 AM

  16. http://mathsci.ucd.ie/~rca/climt/ CliMT
    http://www.ncdc.noaa.gov/oa/ncdc.html
    http://www.pyclimate.org/ for processing ??
    maybe a link gistemp
    this is a VERY good step forward

    Comment by jl — 28 Nov 2009 @ 12:20 AM

  17. re: donQ: Patience of a saint, Gavin.

    GISP2
    ftp://ftp.ncdc.noaa.gov/pub/data/paleo/icecore/greenland/summit/gisp2/isotopes/gisp2_temp_accum_alley2000.txt

    I’m not sure if this has value …
    NCAR CCM3
    http://www.cgd.ucar.edu/cms/ccm3/source.shtml

    Comment by Ron Broberg — 28 Nov 2009 @ 12:24 AM

  18. When this first broke, I was absolutely horrified. Being a true believer myself, I was crestfallen to find out that the science behind a major and popular theory was “fudged” and had gone unchecked. Even worse were the comments about peer reviewed literature. I felt is if I had been betrayed. I was angry and fearful.

    But this controversy seems to be unraveling by the day. The more I read, the more context that is provided about what actually happened…the more the motivations of the deniers and internet foamers become readily apparent.

    The longer this controversy drags on it seems that the accusations of fudging data seem to be sticking to the deniers.

    I’m really happy that they’ve released the raw data.

    Hey kiddos, the data is out there now. BTW You are not the first to recompile it or analyze it. A few DIYers have recompiled the data with their own methods and produced similar results…and yes I’m “hiding” that link from you….because I forgot what page I found it on earlier today. (the conspiracy continues!!!)

    This controversy has actually turned out to be good for the AGW crowd. I suspect 2 months from now, this will be barely a blip on the radar screen.

    Comment by matt — 28 Nov 2009 @ 12:26 AM

  19. Monthly global sea level and rate reconstruction from tide gauges with errors
    http://www.pol.ac.uk/psmsl/author_archive/jevrejeva_etal_gsl/

    useful software for examining coherence between time series based on wavelets.
    http://www.pol.ac.uk/home/research/waveletcoherence/

    Comment by john moore — 28 Nov 2009 @ 12:56 AM

  20. Hey thanks for consolidating all of this in one place. I was wondering about where to get the raw world data the CRU uses. I’m assuming that is the GHCN v.2?

    Gavin, it is unfortunate that people believe NASA is hiding data and code. I have never found this to be the case. Hopefully that will get cleared up.

    Could at some point you speak to what sources are not publicly available? If that is stated clearly in a list, it won’t seem like anyone is trying to hide the data. Obviously you can’t cover everything, but if there is some set of major importance that is not publicly available, it might be worthwhile to address it.

    Comment by Jonathan Fischoff — 28 Nov 2009 @ 12:58 AM

  21. Looks like a mountain of data to me. Could the deniers possibly have anything in common with the people who say that there is no evidence for evolution?

    Comment by Edward Greisch — 28 Nov 2009 @ 1:03 AM

  22. How about adding a link to NGDC (as distinct from NCDC) http://www.ngdc.noaa.gov/ngdcinfo/onlineaccess.html
    - has many paleodata. As editor of Marine Micropaleontology, my co-editor and I request all authors to submit all raw data in addition to statistically treated data, in downloadable form (mainly excel files). I wonder how people can say that they can’t get data – I’ve always been able to find what I want on-line at one of the data centers.

    Comment by Ellen Thomas — 28 Nov 2009 @ 1:16 AM

  23. Has anybody added up how many terabytes of data that is in total? I’m sure I couldn’t afford that much hard drive space. I’m forwarding your email to my friends.

    Comment by Edward Greisch — 28 Nov 2009 @ 1:40 AM

  24. “But this controversy seems to be unraveling by the day. The more I read, the more context that is provided about what actually happened…the more the motivations of the deniers and internet foamers become readily apparent.”

    I think we get so used to lies and non-science in the denialosphere so often, we forget just how much comes from there, the latest being the New Zealand non scandal. They aren’t interested in genuine enquiry, debate and advancement of science, but tearing it all down because the implications upset them.

    Comment by lumpy — 28 Nov 2009 @ 1:52 AM

  25. Reasonable requests for data (the actual data used, plus the code used to process it) have been fobbed off for years, with Real Climate fully supporting and justifying the refusals to release data (usually with a very arrogant tone). Now all of a sudden you’re very interested in appearing to be as open and user-friendly as possible (while still not releasing or encouraging others to release the actual datasets used to support their papers).

    If you really want to limit the damage to your credibility in the eyes of the public (and many in the scientific community), you’ll have to do better than this. Pretending all the data has been available all along won’t cut it.

    Comment by Annabelle — 28 Nov 2009 @ 1:56 AM

  26. But this controversy seems to be unraveling by the day. — matt @ 18

    Well, the main stream media seems to have yet picked up on the unravelling (I exclude FOX as it wouldn’t report the unravelling in any event). NPR ran with a story the other day, picking on a particularly miffed climatologist that his paper showing snowfall in the Sierras (I believe; haven’t gone back to check the story on that yet) hasn’t diminished despite what he claims are the consensus predictions, hinting at dark conspiracies — and the NPR reporter went with this in spite of his own reporting in the same story that the paper was merely cumulative of others that had already pointed out the “anomoly”!

    I also want to say that, as a lay person, I find it sometimes very hard to follow what the posts are saying as there tends to be a lapse into technical jargon, higher math, or just a focus on one specific component of climate modelling. I appreciate what I can understand, and am grateful for it, but, please, remember that this is something that affects the lives of people without Phds and they need to be able to cut through the blizzard of talking points and sound bites spewing forth from Washington and the main stream media (and this time I include FOX). So, please, everybody, even in the comments, try to leave the tech language in the lab for the sake of the millions, nay billions upon whose lives this issue impacts. Thank you.

    Comment by JohnW — 28 Nov 2009 @ 2:09 AM

  27. If I could also make a suggestion – it’s not much good putting all that code up without a description of how to interpret it.

    I decided to wrap my head around how a GCM worked two years ago. I downloaded code (modelE as it happens) and could not understand it. I thought documentation was the problem, so I got the NCAR documentation and model and could not understand it. The various IPCC reports might be fine literature reviews but they are lousy texts.

    Then I ordered Washington and Parkinson’s textbook – with that (and EdGCM – no longer free sadly) finally I was getting somewhere.

    For people who are actually interested in finding out how this stuff works in detail a short reading list of 3-5 textbooks would be a *lot* more useful than a mass of code. Perhaps a link to your “books” page and a brief reading list would be a good idea?

    Also the “Giovanni” link looks broken.

    Comment by Patrick Caldon — 28 Nov 2009 @ 2:46 AM

  28. The “data sources” page is a valuable resource.
    Small correction: The link to Sealevel (U. Colorado) does not work like this, it is sealevel.colorado.edu, without the “www.” bit.

    Comment by Kees van der Leun — 28 Nov 2009 @ 2:47 AM

  29. http://www.cru.uea.ac.uk/cru/data/
    (This website is currently being served from the CRU Emergency Webserver.
    Some pages may be out of date.)

    Comment by Hank Roberts — 28 Nov 2009 @ 2:49 AM

  30. Perhaps you could clarify in the main post that all this data has been available for a long, long time? From some of the comments, it would seem that many still can’t get their head around the idea that there has always been oceans of data out there waiting for the deniers to analyse.

    I can’t imagine what’s holding them back.

    Comment by Didactylos — 28 Nov 2009 @ 3:06 AM

  31. Gavin, this is exactly the right approach — you are taking the issue seriously and patiently, without legitimizing spurious accusations. Well done.

    Of course hardcore deniers will never be satisfied by any amount of data, as that is not what they are after, but I think most reasonable people, following an exchange like yours above with donQ, can easily tell who is being reasonable and who is not.

    That this meme of “hiding the data” is so successful is because it is an instance of the broader meme that scientists are an elite cabal of liberal intellectuals bent on controlling everybody’s lives. Your patience and honesty in responding to this does much to counter that meme. I think this is the right approach and should be continued.

    This is just one instance of what I think is a crisis of credibility for science in general. It is in the interest of some who benefit from the status quo to isolate scientists from the general population and vice versa. Scientists ignore or allow this isolation to the detriment of all. The best bet, IMHO, is to open up the conversation frankly and honestly and patiently as you have done.

    Thank you. You rock. Your modesty in earlier threads is becoming — you stated that you are not engaged in “saving the world,” only in producing good science, which is an admirable response. However, for your tireless efforts to communicate that science to the general public, I think it is possible that history will not remember you as a hero only because real heroes are often forgotten. Just ask the last guy who saved the world.

    Keep going with this approach, I think it will work in the long term.

    Comment by Eli Snyder — 28 Nov 2009 @ 3:11 AM

  32. Gavin,

    Calling it a powerful meme still sounds like you are in the denial stages.

    It may be that the e-mails have been hacked “illegally” as you put it – much in the same way that the UK Parliamentary expenses were “stolen and leaked”. The reaction there was that Michael Martin instituted a police enquiry and eventually he became the first Speaker of the House to be forced out of office for 300 years.

    The e-mail chain reveals a disturbing pattern of obfuscation etc. etc. instead of opennesss. Coupled with patters in a major media organisation here that seems to want to push only one argument and stream.

    Personally I have a background in engineering with two Masters behind me – one involving computer simulation of a very simple eletrical system. I have 25+ years systems and software engineering experience, and enough wit to know how difficult it is to model any sort of physical system.

    The science of modelling is a really complex field that I don’t have the bandwidth to fully understand (day job and family commitments too), but I do have an interest in it, and a very natural suspicion when I am told to simply trust models etc. etc. A very useful suspicion that has served me well professionally in the software industry in assisting to build and work on various parts of system reliability etc.

    There has been far too much “the science is settled – trust us – we are the ones to trust”. In fact – it sounds rather like another religious institution in Ireland that has just got a well deserved pasting for what was actually very serious criminal activity. I’m not putting you guys in the same category as the irish bishops – but just to say, the manner of your communication has very similar patterns to their way of handling a serious issue.

    Sorry if this has rambled over a number of different thoughts.

    It is a very important topic, and you need to do justice to legitimate concerns from various fields that might not all be attached to or paid for by big oil !!!!!

    Comment by John O'Connor — 28 Nov 2009 @ 3:55 AM

  33. “His prediction is that within 6 years there will be no summer sea ice in the arctic. ”

    Just what Big Pil wants!

    Anyone that thinks for a second that Big Oil doesn’t want global warming is a fool. Imagine how much money they can make drilling all that oil in the artic now that it will be ice free! Hook, line, sinker.

    Heck, their effort to convince people that global warming ISNT’T true plays right into their hands, making sure nothing is done about it! These people aren’t stupid, just the drones that believe the nonsense they are spewing. It’s quite ingenious if you ask me.

    Here is what they are after: http://en.wikipedia.org/wiki/Azolla_event

    Comment by Garrett — 28 Nov 2009 @ 3:55 AM

  34. This is how the denialist blogs will spin this: Thanks to their speaking truth to power, the intrepid hackers who liberated the secret emails, and the irresistible pressure on the elitist scientist cabal in recent days, Gavin has finally backed down and ordered his minions to release the closely guarded secret data and code.

    [Response: And I would have got away with it too, if it hadn't been for those pesky kids.... ;) - gavin]

    Comment by wagdog — 28 Nov 2009 @ 4:44 AM

  35. Just trying to get a rough idea of what the raw ghcnv2 data looks like (I think that is the only raw temp data there, right?).

    I tried averaging all the monthly data that isn’t invalid per year and looking at that, but didn’t get anything that looked significant (I understood that’s totally crude, just wanted a rough idea).

    I then realized the problem looking closer at the data — the test points are not the same at all over time. For one, there seems to be a huge drop off in station count in recent times. eg 1955 has over 7000 while 2004 is more like 2300. Any idea why there are so many fewer stations now that before? I would have thought it would be the opposite. Is there a selection for stations in this data? If so, is there a place where all stations data is available (selected and discarded)? I’m assuming they didn’t shut down stations that were reporting regularly in the past, but maybe they just don’t include them for some reason.

    I understand there will be places that will have warmed, and others that have cooled, but there should be a large number (majority) of places that show the dramatic warming we see in IPCC graphs (and some more extreme even I assume).

    I will keep trying… thanks for posting these links. It would be very helpful to see how you get from this data to those graphs as well, hopefully that can get posted at some time.

    Comment by debreuil — 28 Nov 2009 @ 4:51 AM

  36. I’ve been maintaining my own links to data now for a long time; I do a bit of amateur analysis of my own from time to time. But it is always useful to have another convenient directory for finding stuff.

    Above in comment at 28 Nov 2009 12:26 AM Matt says:

    I’m really happy that they’ve released the raw data.

    Hey kiddos, the data is out there now.

    It’s worth emphasizing — and I suspect Matt actually knows this — that this directory is not some new release of data or a renewed openness in response to the hack affair. These are links to already available data. And there always has been a power of work going on to make data available and accessible.

    You may want to add to the radiosonde sources the following:

    Radiosonde Temperature Anomalies in the Troposphere and Lower Stratosphere for the Globe, Hemispheres, and Latitude Zones. Cite as: Sterin, A.M., 2007. Radiosonde Temperature Anomalies in the Troposphere and Lower Stratosphere for the Globe, Hemispheres, and Latitude Zones. In Trends Online: A Compendium of Data on Global Change. Carbon Dioxide Information Analysis Center, Oak Ridge National Laboratory, U.S. Department of Energy, Oak Ridge, Tennessee, U.S.A. doi: 10.3334/CDIAC/cli.004

    Global, Hemispheric, and Zonal Temperature Deviations Derived From Radiosonde Records. CITE AS: Angell, J.K. 2009. Global, hemispheric, and zonal temperature deviations derived from radiosonde records. In Trends Online: A Compendium of Data on Global Change. Carbon Dioxide Information Analysis Center, Oak Ridge National Laboratory, U.S. Department of Energy, Oak Ridge, Tennessee, U.S.A. doi: 10.3334/CDIAC/cli.005

    I don’t think they are in your current radiosonde list.

    Comment by Duae Quartunciea — 28 Nov 2009 @ 5:21 AM

  37. Regarding releasing supporting code and data and paper in one package, I’ve amused myself for quite some time playing with the interactive pages that Dr. Hansen and other GISS researchers often put up alongside their papers e.g. http://data.giss.nasa.gov/modelE/transient/dangerous.html ISTR another where one could tweak forcings and plot the output from GISS Model E, for those without the time to download and compile the fortran…, but a quick Google doesn’t find it. Maybe its been taken down or memory is playing tricks…

    also http://data.giss.nasa.gov/modelforce/ghgases/ ,of course.

    I repeat my thanks to Gavin and Eric for assembling a 1800 comment archive of rebuttals and context in a matter of days. Impressive.

    Comment by pjclarke — 28 Nov 2009 @ 5:35 AM

  38. Lots of processed data at:
    http://www.hadobs.org

    Ed.

    Comment by Ed Hawkins — 28 Nov 2009 @ 5:41 AM

  39. I am relatively new to the climate politics and science discussion and it was the recent CRU email affair that caught my attention. It seems that this has brought out the worst all around. The sceptics are making idiots out of themselves as can be seen by the rude comments posted everywhere, I expect that, but it is interesting how the mainstream scientific community is handling this – at first not very well. In fact, I have been disappointed a number of times by a downright defensive attitude which, I find, quite unbecoming.

    It does look like some people are rising to the occasion and realizing that perhaps an more inclusive view of “community” needs to be adopted. Large parts of the software world has benefited greatly by adopting an open source methodology and general quality has reached higher levels. Perhaps the scientific community can also benefit from this approach. The challenge is to learn to live and work with people who are on very different levels.

    I am glad to see postings such as this “Where’s the data?” It is both a good idea and a good gesture. The matter of there still (as I understand) being proprietary data out there is a serious problem that can probably not be addressed from the scientific community, but needs to be dealt with by legislators when they concern themselves with copyright laws and public service funding.

    Comment by Ole Juul — 28 Nov 2009 @ 5:53 AM

  40. I can fully appreciate the concerns about releasing raw data to those who might not know how to use it properly, and particularly if it requires a lot of extra work to adequately describe the variables and potential variations in their collection.

    Maybe someone should explain what sort of data processing happens to raw climate data to get it into usable form? I suspect that some deniers simply do not understand that all data need some form of processing, and I would guess that this is particularly the case for proxy data.

    As an aside, The Daily Mash provided an amusing summary:
    http://www.thedailymash.co.uk/news/environment/climate-change-emails-stop-glaciers-from-melting-200911252254/

    Comment by Lucibee — 28 Nov 2009 @ 5:59 AM

  41. First you state that, it is just a “powerful meme” that access to raw data is somehow restricted. And then you tell us that, you are “actively working to increase accessibility and usability of the data.”

    Obviously there is a contradiction here and you might consider dropping the pejorative word “meme” from the first paragraph. Your objective is not to label those calling for openness as just parrots, is it?

    [Response: huh? Not at all. Free speech concerns are also a powerful meme - witness the furor when Hansen was prevented from speaking to the media in 2005/2006. Many people were outraged by that even if they hadn't been paying attention to the details. You only need to look at the comment threads to see that this has touched a nerve. In this case, while the concern is real and needs addressing, the real attitude or practices of the climate science community are very different from that perceived. - gavin]

    Comment by Matti Virtanen — 28 Nov 2009 @ 6:06 AM

  42. OT–Can anyone there point me to an equation or method for determining the extinction coefficient at different wavelengths due to Rayleigh scattering? Or a table of such values, or a paper which mentions it? I want to go to full-scale use of the complete ERT in my RCMs and I need to calculate, rather than parameterize, Rayleigh scattering.

    Comment by Barton Paul Levenson — 28 Nov 2009 @ 6:32 AM

  43. Sorry to post an off topic question, but here goes nevertheless.
    (IF this is a no brainer, or I am stupid or something please don’t shoot me down in flames)
    As we know a standard ic engine converts about 30% of the available energy in a given fossil fuel into motion energy.
    The remaining 60 to 70 % is “lost” as heat, ie dumped into the atmosphere. Multiply that individual car, truck. diesel locomotive , stationary engines and what have you by the millions of vehicles out there.
    On top of that also add the waste heat from electric power stations, and even the thermal pollution from nuke plants (Even though they do not emit co2) and airplanes and I reckon that is a lot of heat energy that used to be a form of carbon now warming up the atmosphere.
    So my question is , is this total amount of “waste ” heat negligible compared with the total solar radiation received by the earth, or is this large enough to play a role in influencing the climate and thus has to be included in the models?
    Thanks to anyone who can enlighten me

    [Response: Not a dumb question at all. Locally - in heavily urban environments for instance - this can be an issue, but on a global basis it is about 2 orders of magnitude smaller than the warming effect of increasing CO2 (1.7 W/m2). This is discussed here, but the calculation is easy - global energy use is about 15 TW, area of the planet is 5.1x10^14 m2, therefore the heat flux is ~0.03 W/m2. - gavin]

    Comment by uncle pete — 28 Nov 2009 @ 6:51 AM

  44. Please add the IPCC data distribution centre to your Data Sources page (http://www.realclimate.org/index.php/data-sources/). The URL is http://www.ipcc-data.org/

    Thanks and I use this opportunity to thank you in general for the good work you do @ RealClimate

    Comment by Andreas Fischlin — 28 Nov 2009 @ 8:01 AM

  45. Thanks RC! I’m sure you realise that the innumerable trolls and astroturfer’s who claim they cannot find ANY climate data will never find this page either.

    Hat tip to your web developers too, I imagine you guys are getting more hack attempts than NASA right now.

    Comment by Alan of Oz — 28 Nov 2009 @ 8:01 AM

  46. OT, but just quick thank-you; Realclimate is an incredibly useful site. The levels of dedication and patience shown are exemplary. Your book’s on my Christmas wish-list as a small token of my appreciation.

    Comment by Tim Croker — 28 Nov 2009 @ 8:13 AM

  47. This is quite lovely. I hope established bloggers will link to it and make it come up on searches. You want data? Here you go.

    You also might include mitgcm.org for a model link, though I’m not sure which climate papers it’s been used for.

    I’m curious about how putting GISS data fit into a welcome trend of making geophysical data easily available on the web. It seems like a lot of uncontroversial data sets (MODIS products, HYCOM output to name two) have been put in relatively user-friendly form the past several years — were you going in this direction regardless?

    Thanks so much for all your work.

    [Response: Yes. Regardless of the disproportionate amount of public attention being given to the paleo-reconstructions, the big drive for more efficient access to data sets is simply the size and complexity of the models' output and satellite retrievals. It became apparent years ago that the groups who produced this data were not on their own capable of doing all the analysis or synthesis that needs to (or just could) be done. The creation of the CMIP3 data base back in 2004 (and planned well before that) and the huge number of external groups who have used that data for all sorts of research ideas (500 papers and counting), way beyond what the IPCC used, showed to all the model groups how much value-added work can be done with unrestricted access. - gavin]

    Comment by afeman — 28 Nov 2009 @ 8:15 AM

  48. Reanalysis-2 (NCEP/DOE):
    http://www.cpc.noaa.gov/products/wesley/reanalysis2/

    Comment by Andreas — 28 Nov 2009 @ 8:30 AM

  49. Thanks Gavin, this is an awesome link to send to the automatons out there repeating the “they won’t release the data” meme. Unfortunately, as I’m sure you realize, most of them are just trying to score rhetorical points, so whether they have the data or not doesn’t really matter. Now that they’ve been proven wrong about the availability of the data, they will continue to bitch about how it’s not pretty or user friendly so they don’t have to address the fact that they have no clue what to do with it now that they know where to find it.

    Now you just have to provide them with GUIs for everything, instruction books with full-color illustrations, and a 24-hour helpline in the interest of full scientific transparency. ;)

    Comment by Mike G — 28 Nov 2009 @ 9:08 AM

  50. Hi Folks, a skeptic here

    I am saddened by the tone of many of the comments. I just can not understand why the sarcastic comment and calling us names and suggesting we are just the same as the creationists

    Surely this is how you got in the mess you have.

    I am a skeptic simply because I have developed some doubts. I became more skeptical when I started to understand that much of the data and models seemed to be getting witheld. I was not sure why the secrecy but was always opne to an explanation.

    Then I just kept hearing things about maybe there was some cooling going on. So, I thought, well that sounds like worth doing some serious checking on. Then I started to hear the old “it’s all settled” line. Gosh, I have never known anything to be settled in my long life.

    So, this just kepy adding up and then the name calling and all of that started me thinking, could group think and politics really be in play.

    Som the CRU emails come along. I have read most now and looked at a good amount of the code and the read me files. Of course, I am shocked and, of course, I am even more skeptical.

    So, here I am on your site and I see a mix of people being reasoned and others attacking people like me. Then there is some folks saying that it is a storm in a teacup and it will all go away. Surely they can not be serious.

    I think that the reasoned people on yoru ‘side’ need to take control of this baby. I can;t see how attacking us with names is going to work. I think it will only make the “alarmist” tag stick.

    That is just my thoughts, please don’t attack me, I have done nothing wrong, nothing more than question the dominant paradigm and have doubts, that is not a sin or a crime.

    Peace

    Comment by Jimi Bostock — 28 Nov 2009 @ 9:09 AM

  51. http://coast.gkss.de/staff/zorita/

    [edit - link available]

    [Response: Unfortunately, this episode is being seen as an opportunity by some to imbue their personal and professional conflicts with particular researchers with a greater importance than they have. I am not going to comment on the history of tension between certain people, nor doubt the sincerity of people's clearly deeply held views. But talk of blacklisting scientists from assessment bodies is, at best, foolish. These panels require a full spectrum of the community to take part in order to constructively come up with language that all can accept. Excluding people because they have criticised your work in the past (and vice versa) is not the way to go. Lindzen took part in the 2001 IPCC and the NAS 2002, John Christy was on the CCSP panel on tropsohperic trends - excluding them because of a history of disagreements or perceived personal failings would have been a mistake. The same goes for the scientists mentioned in the above link - especially since one of them at least has no apparent connection to any of the issues raised by these emails. This is not a topic for further discussion. Sorry. - gavin]

    Comment by Silke — 28 Nov 2009 @ 9:31 AM

  52. I hold a degree in computer science and have been a software engineer for 20yrs. All the talk about making code available to reproduce the reults is utter nonesense.

    Sure it’s nice to have the code but the code itself is irrelevant to the science. Software configuration practices and OSS are not particularly usefull for reproducing results in the scientific sense since the same code will always produce the same results and you will be stuck with the impossible task of demonstrating the code is bug free.

    Reproducing the same results with the same data and different code is a much more powerfull test because it provides confidence that the results are independent of a particular implementation of the methods.

    I find it more than a little ironic that climate scientists have to educate programmers on the benifits of multiple independent source trees. It’s not a hard concept to grasp, double entry bookeeping works on the same principle and even an accountant can do that ;)

    Comment by Alan of Oz — 28 Nov 2009 @ 9:34 AM

  53. Fair is fair. I demand that the Heartland Institute, George Marshall Institute, American Petroleum Institute, Western Fuels Association, Sen Inhofe, McIntyre, Singer, Ball, Michaels, and all other organizations and individuals attempting to influence public policy in regards to climate change IMMEDIATELY make their entire email archives available for public scrutiny and analysis.

    Comment by Jiminmpls — 28 Nov 2009 @ 9:40 AM

  54. There are three GCMs with source code available from the University of Hamburg here http://www.mi.uni-hamburg.de/Projekte.209.0.html?&L=3
    They are SAM, PUMA and PLASIM.

    Comment by Alastair McDonald — 28 Nov 2009 @ 9:42 AM

  55. This is “feeling” better, Gavin…. written as a lay person who felt badly trashed by this site a bit ago when
    I pleaded for openness.

    I do not understand — care, even — about the data links you’ve provided now. What is confusing to me is
    why, if it is now this easy, has it taken so long for
    Steve McIntyre, who runs a pretty disciplined,
    gentlemanly, albeit technical site, to have questions
    answereed?

    Wasn’t there a Congressional hearing some years ago
    where Mann tried to withhold data, but was compelled
    to free some…? I don’t know, except it seemed that
    quiet Steve McIntyre “won” that exchange, altho Mann’s
    side claimed victory, also.

    I can’t believe it is this easy now… was there all
    along….

    And I did try to read the poor CRU programmer’s Harry Read-Me file with a sigh and a laugh.

    And remember: this isn’t about “winning.” This is
    about advancing science.

    Comment by Esmeralda Dangerfield — 28 Nov 2009 @ 9:49 AM

  56. Bottom of the page under “Much bigger indexes of data sources:” the last item reads “Comments Off” which leads to an Error 404 page.
    So, smartypantses. what are you trying to hide?
    J/K Great work all. Thank you.

    [Response: Fixed. thanks! - gavin]

    Comment by chris — 28 Nov 2009 @ 9:59 AM

  57. debreuil #35:

    Any idea why there are so many fewer stations now that before? I would have thought it would be the opposite.

    As I understand it, reporting delays. Climatology is not the first priority for most of these stations.

    [Response: That is true for a few months, but the reason why there is a big drop off in the 1990s is that around that time there had been a coordinated push to gather data from all the different stations and digitise a lot of stuff that hadn't been available before. Many of those stations were subsequently closed as met services consolidated their networks. However, absent another big push to gather neglected data (something that is possibly overdue), the current station network is determined almost exclusively by what the met services choose to put out on the CLIMAT network. This is a summary service where just the monthly means - calculated in a specific way - are sent out to the met services. It is a separate network from the SYNOP notifications that are used daily for weather forecasting purposes and for which there are many more stations. If people were looking for a 'citizen science' project to work on, coming up with a way for the SYNOP data (available via WeatherUnderground etc.) to be made commensurate with the CLIMAT data (available via GHCN), would be a great one. There are some subtleties involved (definitions of daily and monthly means vary among providers), but that would provide an interesting back-up and comparison to the CLIMAT-derived summaries from GISTEMP, HadCRU or NCDC. - gavin]

    Comment by Martin Vermeer — 28 Nov 2009 @ 10:17 AM

  58. Clever way to make it appear that you are complying with the requests to “free the data, free the code”. What you have posted is nothing new. These are just link to data and code that always been public.

    [Response: Yes. That's is exactly the point. - gavin]

    Comment by PatrickG — 28 Nov 2009 @ 10:22 AM

  59. Gavin…. Thanks for posting my comment, typo and all.

    I am *very* concerned about the future — forget the AGW problem; I’ll be dead…smile — the future of the quality of the minds and the education of future scientists, if this is perceived by newbees as a rigged game, only for the follower-type lesser lights.

    There have been some true greats. Hank Stommel, with
    less than stellar peer-review credentials was one of
    them. And a friend.

    I have been thinking and w/ some creative ideas might
    be able to help reform this process.

    If you wish, we might talk. I could consolidate the
    email to a central professional-only intl system and
    accept and transfer data streams through a central
    point, making data available post-publication.
    I think…. smile.

    I would need some fairly modest help.

    If you would like send me a phone number and I will
    call you.

    This comment is not for attrition. …Ez
    PS: My name isn’t Esmeralda Dangerfield. smile…

    Comment by Esmeralda Dangerfield — 28 Nov 2009 @ 10:35 AM

  60. I think that it’s important to point out that computer model results are *not* the most important evidence linking CO2 and other greenhouse gases to large-scale climate-change. The most important evidence is the physical evidence left in the Earth’s crust by major climatic events like the PETM. The real smoking gun evidence linking carbon emissions to global-warming has been produced by paleontologists, not computer modelers.

    If every single climate model were thrown away, there still would be a mountain of (direct physical) evidence linking carbon emissions to large-scale climate-change.

    Comment by caerbannog — 28 Nov 2009 @ 10:43 AM

  61. Tides:
    http://tidesandcurrents.noaa.gov/sltrends/sltrends_global.shtml

    Sunspots:
    http://sidc.oma.be/sunspot-data/
    http://www.ngdc.noaa.gov/stp/SOLAR/ftpsunspotnumber.html

    Code/Tools:
    http://www.ncl.ucar.edu/

    I’m going to rebuild a Ubuntu box and see where I left off with the GISTEMP reconstruction.

    Comment by Ron Broberg — 28 Nov 2009 @ 10:47 AM

  62. Clever way to make it appear that you are complying with the requests to “free the data, free the code”. What you have posted is nothing new. These are just link to data and code that always been public.

    My irony meter just exploded.

    However, I think this is going to be pretty much the standard response among followers of Steve McIntyre’s “disciplined, gentlemanly” web site …

    Comment by dhogaza — 28 Nov 2009 @ 10:48 AM

  63. Patrick G,

    All the raw data and methods needed to reproduce GISS’ global temperature estimates have publicly available for years. Over at the GHCN repository (google is your friend here), not only are there all the raw and homogenized temperature data, but there’s also source-code that will parse and extract data from the data files. You can compile the code (out of the box) with the free gfortran compiler. It will run on a Linux box (and almost certainly Macs and Windows boxes with cygwin installed.)

    If you wanted to, you could perform your own completely independent check of GISS’s work for no more than the cost of a laptop and a two-dollar cup of coffee (for access to the coffee-house’s WIFI access point).

    And you have had that ability for years (whether or not you realized it).

    Comment by caerbannog — 28 Nov 2009 @ 10:59 AM

  64. @caerbannog#60: I tell people the mirror image. That you can through away all the paleo-data and you still haven’t disproved AGW since it is a physical theory arising from the radiative properties of the CO2 molecule.

    And that’s the point, isn’t it. Skeptics think that AGW is a house of cards and if they can just pull one card out, the whole thing collapses. They have no idea of the explanatory power of CO2 warming/cooling and the diversity of data supporting it.

    Comment by Ron Broberg — 28 Nov 2009 @ 11:00 AM

  65. “But talk of blacklisting scientists from assessment bodies is, at best, foolish. These panels require a full spectrum of the community to take part in order to constructively come up with language that all can accept. Excluding people because they have criticised your work in the past (and vice versa) is not the way to go”

    Well, isn’t that rich?

    Comment by PaulinMI — 28 Nov 2009 @ 11:01 AM

  66. I hope this well thought out gesture is picked up on by the mainstream media. Ideally, it would then be Issue Resolved, except for the real extremists, who always believe that government scientists are “hiding” something.

    Most likely the deniers will attempt another line of attack in the next few months. They certainly have the time and money to do so. I’m sensing that a lot of people (including myself) are getting a little frayed from the constant battling lately. Since evidence does not always prevail, I wish I knew the answer about how to address the deniers in the future.

    As many have pointed out, they have become noisier as the evidence continues to make their basic positions completely untenable. This leads one into a strange world of arguing about someone’s fantasies. The truth always prevails eventually. I fear that this time it could be in the form of further deterioration, not increasing enlightenment.

    Comment by mike roddy — 28 Nov 2009 @ 11:13 AM

  67. Great work as always Gavin. I anxiously anticipate the next attempt to move the goalposts.

    Alan of Oz brings up a great point, code and data availability isn’t going to help. Replication of studies is the only thing that will confirm of deny those studies.

    [Response: Oh the irony... - gavin]

    Comment by Rod — 28 Nov 2009 @ 11:14 AM

  68. For satellite records of snowcover. A key source is the Rutgers Global Snow lab

    Comment by mauri pelto — 28 Nov 2009 @ 11:17 AM

  69. Forgive my ignorance, but I am an American Citizen having a hard time sifting through all of the ‘data’ myself. I have no way of looking at any of this with real objectivity. I have created the cardinal sin, and listened to what many of the deniers are saying. What makes it easy for me to listen the skeptics? When I hear that we are going to have crazy hurricane activity, or no arctic ice, or global warming killed 300,000 people last year, or we will warm 6C – 10C in the next 100 years… and then I look at reality… can you blame me? I know not all agw climate science is extreme. Anyone care to show me one prediction that came true from your models? What is your biggest fear that has been realized up to this point? I don’t mean to be rude, or take up your time. I am sincerely looking for information. Thank you.

    [Response: Read more mainstream stuff from actual scientists. David's book is good, mine or Mike's would do as nicely. The IPCC FAQs are good too. There is none of the hysteria that you are rightly wary of, but lots more detail on how the science really works, what scientists really do and why they've come to the conclusions they have. -gavin]

    Comment by Jason S — 28 Nov 2009 @ 11:24 AM

  70. Datasets for stations in Europe, both raw and processed:
    http://eca.knmi.nl/

    Comment by HenkL — 28 Nov 2009 @ 11:26 AM

  71. Would it be useful for another nation to set up a similar facility as the CRU?

    eg. China maybe?

    It seems like it would be a good idea to have a more dispersed setup. If CRU is going to be out of action for a bit, we seem to be down to two American establishments for the near future.

    Comment by Paul UK — 28 Nov 2009 @ 11:32 AM

  72. On our web site we have a lot “pre-digested” climate data converted to CSV in a format that allows for easy plotting and anlysis. The data all come from respected sources and the original reference is given. It is at:
    http://www.climatedata.info

    Comment by Ron — 28 Nov 2009 @ 11:48 AM

  73. Real Climate,

    Many of the Hadley CRU e-mails and the infamous HARRY-READ_ME.txt include deeply disturbing content. Almost as troubling is the under-reaction of Real Climate as this scandal expands. This scandal deserves an immediate and fully transparent response from the climate science community, not the “circling of the wagons” approach cited by some.

    While I am not a professional climate science, I design atmospheric monitoring instruments and have used them to create a 20-year time series of calibrated measurements of the ozone layer, column water vapor, aerosol optical depth, direct UV-B and various other parameters from South Texas. I have worked under various assignments for NASA GSFC in the US and Brazil and have just completed a major book for NOAA. I was a co-PI for GLOBE for 6 years and have been a USDA UV-B network site manager for 6 years. My findings have been published in the refereed literature, and I have served as a reviewer for several leading scholarly journals. I have also reviewed an array of technical books for McGraw-Hill, Prentice-Hall and Academic Press.

    I trust that this background will establish that I and others who practice traditional, objective climate science–and the public that has paid many of our expenses–are fully entitled to a prompt end to the blacklisting, withholding of data, destruction of e-mails, ad hominems, threats and other misconduct so blatantly displayed in many of the leaked Hadley CRU documents by scientists who somehow found time to communicate with one another between their many trips around the world. Real Climate was created by some of those whose troubling correspondence is now before the world. So Real Climate is where reform should begin.

    QUESTIONS:

    1. Late last night I received a poignant inquiry from a student requesting advice about being a “climate science grad student in the IPCC era.” Some climate scientists have written that they, too, are receiving similar questions from students who are troubled by the content of the leaked e-mails and certain other documents and who are even being asked to “tweak” their data to fit preconceived notions. What steps can be taken by societies, journals, universities and government agencies to protect scientific integrity and peer review, permit dissenting views, end blacklisting, provide transparent access to methods and data, and restore public trust in climate science? Will Real Climate sincerely advocate and openly promote these and other essential reforms?

    2. Your recent posts are a welcome improvement from what can be found in even a cursory scan through Real Climate. You also seem to be allowing comments and questions from a much broader audience, and this is also very good news. My concern, however, is that the tone and ad hominem nature of some Real Climate comments before last week are disturbingly similar to that of some of the leaked e-mails. (This is not surprising in view of the tone of some e-mails from a Real Climate founder in the leaked CRU e-mails.) Will Real Climate hereafter follow the friendlier and more open approach that you have recently exhibited and end the disparagement of those whose positions Real Climate views as incorrect? Will Real Climate follow the lead of Judith Curry in responding to questions from the climate science community in general and grad students in particular? Will Real Climate sever its relationship with scientists who have damaged climate science by advocating the misconduct revealed in many of the Hadley CRU e-mails and documents?

    Thank you for considering these questions. Based on a lengthy telephone conversation this morning, I am far from the only practicing climate observer who will be highly interested in your responses.

    Forrest

    Forrest M. Mims III
    Editor, THE CITIZEN SCIENTIST (www.sas.org/tcs)
    http://www.forrestmims.org
    http://www.sunandsky.org
    twitter.com/fmims

    Comment by Forrest M. Mims III — 28 Nov 2009 @ 11:55 AM

  74. Esmerelda Dangerfield (#55) tells us that she “felt badly trashed by this site a bit ago when I pleaded for openness.”

    Now she says, “I do not understand — care, even — about the data links you’ve provided now.”

    The conclusion is obvious.

    Comment by tamino — 28 Nov 2009 @ 12:06 PM

  75. The entire underlying basis for AGW is called into question by this paper in the International Journal of Modern Physics
    http://arxiv.org/PS_cache/arxiv/pdf/0707/0707.1161v4.pdf

    In short the authors conclude the whole idea of a CO2 driven green house effect violates the laws of physics.

    [Response: Something which has apparently eluded all other physicists since Fourier? Really. And you think this is credible? (Hint). - gavin]

    Comment by Glenn — 28 Nov 2009 @ 12:14 PM

  76. #10 Lawrence Coleman:
    Here is a media report about David Barber:

    “…His findings, to be published in the journal Geophysical Research Letters, suggest a transformative change in Arctic ecosystems is accelerating and that safe shipping in polar waters during the summer and fall will begin much sooner than many experts predict…”

    http://www.canada.com/technology/Arctic+meltdown+remains+severe+report/2276659/story.html

    Comment by Holly Stick — 28 Nov 2009 @ 12:14 PM

  77. Surely the key data set for dealing with denial is just CO2 at Mauna Loa. That data, plus the physics which tells us that more CO2 in the atmosphere implies more retained heat, tells us that we are en route to a problem which will, sooner or later, become very serious. All the rest is about how serious? how soon?

    Since I have never understood the word ‘believe’, I am always a sceptic. The proper attitude for a sceptic is to go by the evidence, not by the assertions. Those who call themselves ‘sceptics’ over global warming are either confused or are going by the assertions; the evidence is against them.

    Unfortunately, as in many a scientific controversy, there are scientists in this field who act for at least part of the time like believers – preferring assertion to evidence. Insisting on making the evidence available to any serious enquirer is the traditional and effective way of checking those impulses.

    Comment by David Heigham — 28 Nov 2009 @ 12:33 PM

  78. Gavin,

    As others have noted, you definitely have a dismissive tone. Terms like “meme” and “denial” or referring to the hacked emails as “illegal” coming from a publication that was actually mentioned in those same emails for being a part of the scam, is disheartening.

    It seems like you want to appear open and honest but without really acknowledging that the problems here are larger than just mere context. You’re clearly having a hard time understanding just how serious the revelations by way of the CRU emails really is. I’m no scientist and I have no understanding about the numbers. But I can tell when someone is BSing me. And my BS meter is in the red right now.

    Regards,

    The Obnoxious American

    P.S. making jokes about my character, or my name only reinforces my point.

    [Response: I'm clearly not the only one who PR-challenged then. - gavin]

    Comment by The Obnoxious American — 28 Nov 2009 @ 12:52 PM

  79. GLIMS (Global Land Ice Measurements from Space) is a project designed to monitor the world’s glaciers primarily using data from optical satellite instruments.

    General GLIMS website:
    http://www.glims.org/

    GLIMS glacier database:
    http://glims.colorado.edu/glacierdata/

    Comment by Jim Torson — 28 Nov 2009 @ 1:04 PM

  80. Just for fun, shall we start keeping a count of the denialist posts that claim victory and the moral high ground while utterly faililing to deal with the evidence that contradicts them?

    Comment by Ray Ladbury — 28 Nov 2009 @ 1:18 PM

  81. Thanks for taking the time to put all these links together. But I would like to echo the point a couple of people already made that this is NOT new data – it’s been available for ages to anyone who bothered to spend a few minutes on Google (obviously too much like hard work for the “skeptic” community).

    It might be worth highlighting this fact in the blog entry in case anyone gets the impression you are suddenly rushing to release data as a result of the email hack.

    Certainly the CMIP3 model data has been available for download for years (since 2004 IIRC, and CMIP2 before that) and KNMI climate explorer has been around – providing an ever-growing amount of model and observational data – since 2000 I think. I’m not sure how long the proxy stuff / GISTEMP station data has been online for, but I expect it’s been up for a similarly long time.

    And many thanks to Gavin for patiently putting up with all this rubbish, you deserve a medal.

    Comment by cer — 28 Nov 2009 @ 1:19 PM

  82. The Obnoxious American: Terms like “meme” and “denial” or referring to the hacked emails as “illegal” coming from a publication that was actually mentioned in those same emails for being a part of the scam, is disheartening.

    BPL: Read my lips, TOA. Hacking into the CRU computers without CRU’s permissions was ILLEGAL. AGAINST THE LAW. A CRIME. If they can find out who did it, those people are going to jail. That’s what “illegal” MEANS.

    Comment by Barton Paul Levenson — 28 Nov 2009 @ 1:20 PM

  83. The people asking for the unreleased data they needed to attempt to reproduce CRU’s work already knew about these links.

    Comment by David — 28 Nov 2009 @ 2:23 PM

  84. “Forgive my ignorance, but I am an American Citizen having a hard time sifting through all of the ‘data’ myself” – quite right AC, it’s appalling that these so-called scientists insist on having all their data available and then expecting you to do something with it.

    Gavin there is some other stuff that has been kept hidden all these years – long past time that you made available full lists of all universities around the world that have courses/degrees in climatology, physics, biology, geology, oceanography, chemistry, mathematics, ecology, meteorology. There is no doubt that all of the denialists could have done these degrees themselves if they had only known they existed, I mean, how hard could they be. But in the meantime don’t you go thinking that releasing data and expecting people to “sift through it themselves” gets you off the hook. Why don’t you ever think of having a web site where all sorts of information (and data) and ideas about climate science could be published and discussed? There, that’s brought you up short, smarty pants, didn’t think of that, did you!

    Comment by David Horton — 28 Nov 2009 @ 2:23 PM

  85. I’ve made a tour of the denial blog triumphalist sites and found that this graph is commonly used to refute the whole AGW theory.

    http://wattsupwiththat.files.wordpress.com/2009/02/eia_co2_contributions_table3.png

    771,000 metric tons of CO2 from natural sources and 23,000 from humans. For them, end of story. Watts put it up for this reason alone.

    The one-liner response would be?

    [Response: Note the absorption number and the net accumulation in the air - around 50% of the human addition. And this is the same thing, year after year. - gavin]

    Comment by Mark A. York — 28 Nov 2009 @ 2:24 PM

  86. “some of the emails are strangely truncated”

    The content has been altered. Since this was done entirely anonymously, we have no way of judging the honesty of the criminals who broke in, or the people who have posted the archives. Serious accusations are being leveled at scientists based on anonymous, unsourced, unreliable texts. I wonder, really, that there is not more shame and doubt on the part of the people who are trying to use the archives as evidence. And why do more people not wonder at the criminals who did this? It is unlikely they are in it for anyone else’s health!

    [Response: Still not sure about this - though there is some investigation of this. My comment referred to the website linked only where there are some discrepancies with the emails in the original zip file. - gavin]

    Comment by The Raven — 28 Nov 2009 @ 2:31 PM

  87. Gavin can you confirm that the CRU raw data that is at the center of the dispute has not been released In this post?

    Comment by Hansen — 28 Nov 2009 @ 2:32 PM

  88. Jimi Bostock (50) — There are plenty of resources for actually learning climatology. You might care to start with “The Discovery of Global Warming” by Spencer Weart:
    http://www.aip.org/history/climate/index.html
    after reading Andy Revkin’s review:
    http://query.nytimes.com/gst/fullpage.html?res=9F04E7DF153DF936A35753C1A9659C8B63

    Comment by David B. Benson — 28 Nov 2009 @ 2:40 PM

  89. here is the “DATA”:

    1. The term “climate change” itself. Thirty years ago, the term “climate change” would have meant natural climate change, which is what climate scientists mostly studied before that time. Today, it has come to mean human-caused climate change. The public, and especially the media, now think that “climate change” implies WE are responsible for it. Mother Nature, not Al Gore, invented real climate change.

    2. “Climate change denier”. A first cousin to the first annoyance. Again, thirty years ago, “climate change denier” would have meant someone who denied that the Medieval Warm Period ever happened. Or that the Little Ice Age ever happened. What a kook fringe thing to believe that would have been! And now, those of us who still believe in natural climate change are called “climate change deniers”?? ARGHH.

    3. The appeal to peer-reviewed and published research. I could go on about this for pages. Yes, it is important to have scientific research peer-reviewed and published. But as the Climategate e-mails have now exposed (and what many scientists already knew), we skeptics of human-caused climate change have “peers” out there who have taken it upon themselves to block our research from being published whenever possible. We know there are editors of scientific journals who assist in this by sending our papers to these gatekeepers for the purpose of killing the paper. We try not to complain too much when it happens because it is difficult to prove motivation. I believe the day is approaching when it will be time to make public the evidence of biased peer review.

    4. Appeal to authority. This is the last refuge of IPCC scientists. Even when we skeptics get research published, it is claimed that our research is contradicted by other research the IPCC has encouraged, helped to get funded, and cherry-picked to support its case. This is dangerous for the progress of science. If the majority opinion of scientists was always assumed to be correct, then most major scientific advances would not have occurred. The appeal to authority is also a standard propaganda technique.

    5. Unwillingness to debate. I have lectured to many groups where the organizers could not find anyone from the IPCC side who would present the IPCC’s side of the story. I would be happy to debate any of the IPCC experts on the central issues of human-caused versus natural climate change, and feedbacks in the climate system. They know where to find me. (For the most common tactic used by the IPCC in a debate, see annoyance #4.)

    6. A lack of common sense. Common sense can be misleading, of course. But when there is considerable uncertainty, sometimes it is helpful to go ahead and use a little anyway. Example: It is well known that the net effect of clouds is to cool the Earth in response to radiant heating by the sun. But when it comes to global warming, all climate models do just the opposite…change clouds in ways that amplify radiative warming. While this is theoretically possible, it is critical to future projections of global warming that the reasons why models do this be thoroughly understood. Don’t believe it just because group think within the climate modeling community has decided it should be so.

    7. Use of climate models as truth. Because there are not sufficient high-quality, globally-distributed, and long term observations of climate fluctuations to study and better understand the climate system with, computerized climate models are now regarded as truth. The modelers’ belief that climate models represent truth is evident from the language they use: climate models are not “tested” with real data, but instead “validated”. The implication is clear: if the data do not agree with the models, it must be the data’s fault.

    8. Claims that climate models have been tested. A hallmark of a good theory is that it should predict something which, upon further investigation, turns out to be correct. To my knowledge, climate models have not yet forecasted anything of significance. And even if they did, models are ultimately being relied upon to forecast global warming (aka ‘climate change’). As far as I can tell, there is no good way to test them in this regard. And please don’t tell me they can now replicate the seasons quite well. Even the public could predict the seasons before there were climate models. Predicting future warming (or cooling) is slightly more difficult, but not by much: a flip a coin will be correct 50% of the time.

    9. The claim that the IPCC is unbiased. The IPCC was formed for the explicit purpose of building the case for global warming being our fault, not for investigating the possibility that it is just part of a natural cycle in the climate system. Their accomplices in government have bought off the scientific community for the purpose of achieving specific policy goals.

    10. The claim that reducing CO2 emissions is the right thing to do anyway. Oh, really? What if life on Earth (which requires CO2 for its existence) is actually benefiting from more CO2? Nature is always changing anyway…why must we always assume that every single change that humans cause is necessarily a bad thing? Even though virtually all Earth scientists believe this, too, it is not science, but religion. I’m all for religion…but not when it masquerades as science.

    Comment by Silke — 28 Nov 2009 @ 2:59 PM

  90. I seem to recall quite a while ago somebody on the intertubes having a list of major scientific institutions with statements supporting the science of AGW. Does anyone know of such a list? One argument I’ve been making is that CRU is only one of many science groups.

    [Response: Here. - gavin]

    Comment by Holly Stick — 28 Nov 2009 @ 3:05 PM

  91. Thank you. It’s good to see responses to accusations quickly presented in a concise and comprehensive way.

    It may not make a dent in the truly obstinate, but it should help quell some of the inflamed nonsense. FWIW, the tack on talk radio (they move like a herd of hooligans) is now to refer to AGW as “crank science.” Climate scientists are portrayed as hateful fools covered in egg, back pedaling behind their “new mantra that it’s OK to be a skeptic but not a denialist.” This is followed by derisive laughter, since it’s assumed to be self-evidently ridiculous in its immense stupidity.

    In other words, “I’m rubber and you’re glue. What you say bounces off me and sticks to you.” It’s a sad commentary that adults can make a living behaving this way.

    Comment by Radge Havers — 28 Nov 2009 @ 3:07 PM

  92. DH: “There, that’s brought you up short, smarty pants, didn’t think of that, did you!”

    At a time when clear thought and considered action are necessary to counteract the AGM Denier noise machine, here is more infantile hissy-fit behavior as a distraction. I worry that there really is no way to counteract the damage done in the minds of people who are caught in the middle in all of this. Simply publishing the raw data does nothing to ease the minds of non-scientists. Public outreach to these people is incredibly important. An information kit that puts activities such as statistical analysis in clear language would be a good start.

    Comment by Roger LaPrelle — 28 Nov 2009 @ 3:12 PM

  93. I’m guessing that if someone had hacked into Bernie Madoff’s email account and discovered he was fudging the numbers, people wouldn’t be screaming, “Leave Bernie alone! You’re a thief, what you did is illegal, and your GOING TO JAIL!!!”

    This is a straw man argument to take away form the substance of the email correspondence that there is a coordinated attempt among the climate change community to only present data that shows an expected, and desired, outcome: AGW.

    Comment by Tom Franklin — 28 Nov 2009 @ 3:12 PM

  94. Um, Roger (#91), it was satire … and a bit of irony.

    Comment by David Horton — 28 Nov 2009 @ 3:30 PM

  95. Mark York,

    One lines responses regarding human addition of CO2 being only a small portion of the carbon cycle:

    Same difference as between turnover and profit. It’s the latter that matters to the growth of the company, even though it’s usually much smaller than the former.

    If two elephants are sitting on a wip-wap, it’s kept in equilibrium (i.e. natural fluxes of carbon). Now you (the human contribution) go sit on top of one of the elephants. What happens, and why? Who is to blame for your side of the wip wap going down, the elephant or you? And who is heavier?

    [Response: Note: wip-wap in this context is a see-saw. Do not look up the other definition in urban dictionary. - gavin]

    Comment by Bart Verheggen — 28 Nov 2009 @ 3:37 PM

  96. Is anybody else amused by the fact that certain people have become so convinced that the CRU emails contain indisputable evidence of scientific fraud on a massive scale that they no longer feel the need (if they ever did) to provide any evidence that this is the case?

    Comment by Dave C — 28 Nov 2009 @ 3:45 PM

  97. Gavin, comment #89 by Silke is a copy&paste of this page on Roy Spencer’s blog:

    http://www.drroyspencer.com/2009/11/my-top-10-annoyances-in-the-climate-change-debate/

    [Response: Thanks for spotting. - gavin]

    Comment by Alfio Puglisi — 28 Nov 2009 @ 3:49 PM

  98. Here is a great post from, as it happens, one of the smartest people I’ve ever met (I had no idea he was an expert in climate science too). I think he sums up the main problem with all this pretty well here — continually presenting your data in a skewed way erodes your credibility.

    http://nierenbergclimate.blogspot.com/2009/11/copenhagen-diagnosis-is-dissapointing.html

    I think it would go a long way towards recovery if a) that stopped and b) people from here would step up and rebut some of the more extreme wild claims you hear in the media. I see that happening a wee bit in some of the comment replies here, but maybe a ‘what we are not saying’ post would help everyone delineate where the science stops and cheerleading begins.

    Just a thought…

    Comment by debreuil — 28 Nov 2009 @ 3:51 PM

  99. Mark A. York,

    Since you know so much, perhaps you could tell me why the oceans have not been warming after 2003 — or if they have why it’s a very slight warming contrary to the projections of the IPCC’s models. Also, do you think that ocean heat reconstructions based on ARGO array data will show ocean heat catching up to IPCC projections because of the current El Nino (because of bias effects introduced by there being fewer sensors at the lower depths)?

    Comment by David — 28 Nov 2009 @ 3:55 PM

  100. Thanks, Gavin. It turns out wikipedia has what looks like a more recent version of the scientific opinion, plus a small page on economic opinion.

    http://en.wikipedia.org/wiki/Scientific_opinion_on_climate_change

    http://en.wikipedia.org/wiki/Economic_opinion_on_climate_change

    Comment by Holly Stick — 28 Nov 2009 @ 3:58 PM

  101. >FWIW, the tack on talk radio (they move like a herd of hooligans) is now to refer to AGW as “crank science.”

    This is a larger and more dangerous issue than just the one at hand. I spend as much if not more energy with my *left* leaning friends counteracting anti-vax crap, suspicion of modern medicine, and the post-modernist brain rot they picked up in school. In fact I believe it’s “deconstruction” as promoted by supposedly progressive academics that leaked out of the universities and enabled things to get this bad. If you’ve ever argued with one, the techniques aren’t quite as frat-boy ugly, but they are similar.

    The fact is that the scientific method, and the body of knowledge built up, constrains people’s scope of plausible beliefs. This rankles them, as any constraint might.

    Re the open source code, what might be really interesting is to take the source code and data and make a simulation “game” for the layperson. Input a set of assumptions, and show outcomes with best-guess probabilities, including side-paths to see the mechanisms involved. The same “game” could have a toolkit to show the impact of lifestyle choices and technology improvements.

    Comment by Bruce the Canuck — 28 Nov 2009 @ 4:08 PM

  102. Re the Briffa data- I searched for “Yamal” in the Tree Ring page you link to, and found two items. With a total of ~150 rows of undocumented data between them. :
    Here’s the top page:
    http://www.ncdc.noaa.gov/paleo/indextree.html
    and the two reports:
    http://www.ncdc.noaa.gov/paleo/metadata/noaa-tree-4770.html
    http://www.ncdc.noaa.gov/paleo/metadata/noaa-tree-4769.html

    These don’t seem to have any of the source data at all?

    Comment by AC — 28 Nov 2009 @ 4:20 PM

  103. @Gavin,

    I have nearly finished reading the Copenhagen Diagnosis. Could you re-open that thread so I can see what the regulars and you think of it? I so often learn more from those discussions than the more formal papers. Thanks.

    Comment by Trent1492 — 28 Nov 2009 @ 4:21 PM

  104. This is a straw man argument to take away form the substance of the email correspondence that there is a coordinated attempt among the climate change community to only present data that shows an expected, and desired, outcome: AGW.

    Tom Franklin, you will of course be able to provide us exact reference to what data has been repressed, since the mails show this to be the case.

    At the same time, you will of course also be able to provide evidence of what laws of physics the scientists have changed, to fit with AGW. This would be necessary, as AGW follows straightforward from the known physics – those dastardly scientists!

    Comment by Kristjan Wager — 28 Nov 2009 @ 4:31 PM

  105. 50. “Then I just kept hearing things about maybe there was some cooling going on. So, I thought, well that sounds like worth doing some serious checking on. Then I started to hear the old “it’s all settled” line. Gosh, I have never known anything to be settled in my long life.”

    The world is not cooling:
    http://www.huffingtonpost.com/wires/2009/10/26/ap-impact-statisticians-r_ws_333941.html

    The skeptics were mathematically wrong with the entire global cooling nonsense that they were trying to convince people on. The math was wrong, and it was intentionally wrong.

    30 “Perhaps you could clarify in the main post that all this data has been available for a long, long time? From some of the comments, it would seem that many still can’t get their head around the idea that there has always been oceans of data out there waiting for the deniers to analyse.
    I can’t imagine what’s holding them back.”

    People see whatever they want to see.

    Comment by EL — 28 Nov 2009 @ 4:42 PM

  106. Wow! That’s a lot more data than I had found (or even know what to do with). Next time someone says, “They need to release the data” I will link them here. But first I will get them to specify which data they think hasn’t been released so they can’t move the goalpost as easily by claiming this isn’t the data they’re looking for.

    O/T: I just read Idiot America. It has a chapter on global warming. A fun read unless you get most of your “science” from AM radio talk shows.

    Comment by Ken — 28 Nov 2009 @ 4:48 PM

  107. Hi Gavin,
    It might be useful to some if you add a list of commonly used data analysis tools to the master list. Here’s a few (mostly for netCDF or OPeNDAP formatted data, which is one of the most commonly used formats for gridded climate data) that I find useful and might be good to start with:

    Climate Data Operators
    netCDF operators (NCO)
    Ferret
    GrADS
    Pydap
    Climate Data Analysis Tools

    - Joe S

    Comment by Joe S — 28 Nov 2009 @ 4:50 PM

  108. I think we’re seeing the effect of loosening the moderation…Gavin or the others, if you were planning to answer Silke later, just remove this post.

    @Silke:
    1. Climate change is climate change, regardless of the source. We’re part of nature, too, so if you want you can always call the current climate change “natural”…

    2. People who are called “climate change deniers” are those that neglect an enormous body of research, misinterpret and overinterpret some other work, and subsequently claim that “it’s all natural” (but see point 1)

    3. It’s funny you mention the peer-review process, since the e-mails discuss several papers that are fundamentally flawed, and of which one has very likely entered the literature because of a biased peer-review process. It was a paper by ‘skeptics’ (Soon and Baliunas), and bad processing by a ‘skeptic’ editor.

    4. The IPCC does not fund research, and the fact that you do not know this is quite telling. The IPCC frequently does call for certain factors to investigate more, which notably include points brought in by skeptics (land use changes, solar influences, etc.). Majority opinions in science change rapidly when the minority opinion provides a better explanation. So far, the minority opinion is found extremely wanting, yet it is still discussed in the IPCC reports.

    5. Debates are popularity contests. On top of that, all the ‘skeptic’ needs to do is create doubt. With a largely scientifically illiterate audience on the topic at hand, this is really easy, especially if there is something at stake. It is the preferred tactic of the creationists in debates with evolutionists.

    6. “Common sense” isn’t always right. In the case of clouds, if we take your ‘common sense’, it is simply wrong. Real “common sense” actually would say that clouds should have both a cooling and warming effect: they reflect sunlight, but also trap heat.

    7. Regarding model testing and validation, you can get some basic understanding here:
    http://www.ccsm.ucar.edu/working_groups/Software/dev_guide/dev_guide/node14.html
    The terminology is the same for ANY model, climate or not.

    8. I guess Gavin can best enlighten you here. I can give you a pointer:
    http://en.wikipedia.org/wiki/Global_climate_model (and references therein)

    9. The IPCC was set up with the explicit goal to find out how much influence anthropogenic influence had on global climate. The claim that it was set up to blame humans and that scientists were bought is outright false, and indicates NWO conspiracy thinking.

    10. Life on earth depends on many things. Without nitrous oxide, much of life will die out. Too much nitrous oxide, and it will die out, too. The idea that more CO2 is good for life needs a LOT of qualifiers. For starters, there are a lot of plants that are evolutionary optimised for low amounts of CO2 (this includes quite a few foodstuffs on which most people are dependent). With increasing CO2, other forms of plants take over, also exerting evolutionary pressure on the animal kingdom. We know that humans have only encountered low CO2-life, and thus can only hope (and pray, if you are religious) that the high CO2-life isn’t too deadly for us. The claim that all earth scientists believe that anything humans do is bad, is simply untrue. Earth scientists, however, are (hopefully) aware that there are many interlaced processes. Altering one part *will* have effects elsewhere. Sometimes calling something “good” or “bad” really depends on the desired outcome. If your desired outcome includes creating a new eco-system, making a dam for hydropower is fine. If you like to maintain an eco-system, building a dam is not a good idea. I can come with hundreds of such examples.

    Comment by Marco — 28 Nov 2009 @ 4:52 PM

  109. Oh, I see. Thank you. Still, it would be a lot of work to verify that the archives match the actual e-mails, this work has not been done, the source is criminal, yet here we are with people using these e-mails as evidence for very serious charges.

    Comment by The Raven — 28 Nov 2009 @ 4:53 PM

  110. Data? What data?

    http://www.timesonline.co.uk/tol/news/environment/article6936289.ece

    Comment by Pops — 28 Nov 2009 @ 4:58 PM

  111. If I could also make a suggestion – it’s not much good putting all that code up without a description of how to interpret it.

    I decided to wrap my head around how a GCM worked two years ago. I downloaded code (modelE as it happens) and could not understand it. I thought documentation was the problem, so I got the NCAR documentation and model and could not understand it. The various IPCC reports might be fine literature reviews but they are lousy texts.

    Then I ordered Washington and Parkinson’s textbook – with that (and EdGCM – no longer free sadly) finally I was getting somewhere.

    For people who are actually interested in finding out how this stuff works in detail a short reading list of 3-5 textbooks would be a *lot* more useful than a mass of code. Perhaps a link to your “books” page and a brief reading list would be a good idea?

    What you write above is exactly the logic that proves those claiming everything is hidden, so everything is suspect are basically lying.

    First, you point out most of those casting stones have no clue *how* to sue the data. This is clearly shown by the request, as you make above, for info on *how* to use this material. And do they think they can become experts in climate modeling in the next week or two? Or even a year or two?

    Second, the above point is augmented by the request for info showing EXACTLY how model results were achieved, how the data was created, how it was analyzed, etc. Yet, if you simply replicate exactly what others have already done, what have you gained? No new info. It is more useful and effective to do it on your own. This adds robustness to the data gathering.

    If you suspect the work already done is in error, then examine that as is and show where the errors are. This is far more efficient than simply saying 100 other people must replicate exactly what others have already done.

    This is a simple logical argument based on the fact that many, many lines of inquiry are used to elucidate Anthropogenic Climate Change. This focus on climate models is just an extension of the lie that climate models are all wrong, all suspect, and can’t do what they claim to do. This is a lie. The lie is proven by the FACT that the models do replicate past climate and do give us a sense of what is happening and will happen. All the more so because they are generally **underestimating** the change… which is another logical bullet hole through the denialists’ harangues. How can they claim the models don’t show warming, and can’t, when the models are showing less than observed? If anything, our models are even better than we think because they are obviously finding the correct trend even though we obviously need to improve the models to capture the extreme rate of change.

    This is not to say replicating exactly what has been done has no value, it does. But with so many lines of inquiry all showing the same thing, this is not vital. It is more important at this point to increase our understanding, not repeat the last 100 years of science because some people who have an agenda want to slow things down even more than they already have.

    Third, as alluded already, observation trumps all else and is the basis and bedrock of all climate science. Scientists don’t think the globe is warming because they made a model and it told them so, they think it is because they are looking at and measuring changes in the environment and THEN making models to help them understand why, how fast, and what the effects might be.

    But this point is studiously ignored by the anti-AGW conspirators. (Yes, conspiracy is a historical fact, unlike the fantasy of a pro-AGW conspiracy which has no basis in fact, no evidence, no proof.)

    It is good the scientists are going even further than they already have to prove their honesty, but it is a shame they have to. Simple logic proves AGW. All you have to do is open your eyes.

    These people will not be able to disprove the science by having the (already mostly long available) code. They WILL use it via the GIGO method: they will claim THEIR data is better – though provided by garage scientists – and THEIR results prove AGW is bunk. Then the true scientists will have to go through and prove their work is badly done and wrong. Then the denialists will simply claim it’s all a conspiracy and that their bad science is being suppressed.

    That is, Gavin, et al., are wasting their time. They should be left to do the science that has given us a chance to avert catastrophic, or even highly destructive and/or disruptive change. Instead, they must refute lies and distortions long ago manufactured by think tanks that knew they were lying and distorting the science even then, back in the 1990′s.

    Let Anthony Watts, et al. have the data, but know hey will do nothing useful or positive with it. If I’m not mistaken, data has been supplied before to such people then never used. After all, delay is the point, isn’t it? (<– Rhetorical question.)

    Comment by ccpo — 28 Nov 2009 @ 5:04 PM

  112. Re: Tom
    “I’m guessing that if someone had hacked into Bernie Madoff’s email account and discovered he was fudging the numbers, people wouldn’t be screaming, “Leave Bernie alone! You’re a thief, what you did is illegal, and your GOING TO JAIL!!!””

    I would expect both of them to take responsibility for their actions. If what they did was illegal, they should accept the consequences. The ends do not justify the means

    personally, I see nothing illegal in the emails or code. I based my opinion on the scientific evidence. Everything the scientists have said in the way of explanations is plausible and the science backs up what they say. Personality issues have no bearing on the science.

    Comment by jerryg — 28 Nov 2009 @ 5:06 PM

  113. The Raven (109) — It is not yet clear the extent to which the e-mail messages were doctored or just made up…

    Comment by David B. Benson — 28 Nov 2009 @ 5:08 PM

  114. Silke #89, parroting the usual lies:

    3. The appeal to peer-reviewed and published research. I could go on about this for pages. Yes, it is important to have scientific research peer-reviewed and published. But as the Climategate e-mails have now exposed (and what many scientists already knew), we skeptics of human-caused climate change have “peers” out there who have taken it upon themselves to block our research from being published whenever possible.

    Indeed the quotation marks are appropriate, as what they try to block, as proper part of their job as “peers”, is junk papers. You can find this out; the CRU stolen mails refer to several such. If you produce such junk, you deserve to be relegated to “journals” where peer review is a standing joke. Or to blog science. There are dozens of reputable journals to choose from. Claiming that they are all centrally controlled is classic conspiracy theory.

    I believe the day is approaching when it will be time to make public the evidence of biased peer review.

    That day has already come, see the CRU stolen emails. Editors resigned because of it.

    4. Appeal to authority.

    You mean: appeal to expertise. Denying the reality of expertise is shamefully anti-intellectual. You may not want to admit it, but science is hard. It takes effort to learn. You have to be good at it to get it right, most of the time. And when you don’t, you have your peers to tell you so. Learn to listen to them.

    5. Unwillingness to debate.

    Debating is for lawyers, politicians and moncktons, not for those honestly interested in the truth. The proper process for that takes place on the pages of the peer reviewed literature, like it or not. Dozens of journals to choose from; their reputation hangs on their record of publishing science rather than junk.

    6. A lack of common sense.

    What you are missing is that you need to look at the change in cloud cover that comes with a change in temperature. Not the absolute quantities. So much for common sense…

    As for the rest of your falsehoods, I’m running out of time and stamina. Just to note that you’re rather overestimating the weight of general circulation models in the package of multiple independent lines of evidence on which our current understanding of the reality and seriousness of man-made climate change is based. And, from the all-too-much-transparency department, did you have to mention Al Gore?!?

    Comment by Martin Vermeer — 28 Nov 2009 @ 5:17 PM

  115. David Benson (113) – Come on now, not even the participants in these exchanges are claiming that the emails are doctored or made up.

    Comment by Benjamin — 28 Nov 2009 @ 5:23 PM

  116. Forrest M. Mims III says:
    28 November 2009 at 11:55 AM

    Real Climate,

    Many of the Hadley CRU e-mails and the infamous HARRY-READ_ME.txt include deeply disturbing content. Almost as troubling is the under-reaction of Real Climate as this scandal expands. This scandal deserves an immediate and fully transparent response from the climate science community, not the “circling of the wagons” approach cited by some…

    **screed filled with blatantly false claims deleted**

    Forrest

    Forrest M. Mims III

    Dear Forrest,

    There is nothing in the e-mails that support your false characterizations, which I suspect is why you’ve received no answer. Why should they reply to propaganda?

    Given there is nothing in the e-mails that undermines the climate science, let us put our time to better use. There is zero evidence, let alone proof, of an agenda, let alone a conspiracy, to promote AGW. There is, however, proof conclusive of the conspiracy to hide the science.

    Please respond to the following with the outrage it deserves:

    The American Denial of Global Warming
    http://www.youtube.com/watch?v=2T4UF_Rmlio

    ExxonMobil’s Tobacco-like Disinformation Campaign on Global Warming Science
    http://www.ucsusa.org/global_warming/science_and_impacts/global_warming_contrarians/exxonmobil-report-smoke.html

    Industry Ignored Its Scientists on Climate
    http://www.nytimes.com/2009/04/24/science/earth/24deny.html?_r=2

    The Denial Machine
    http://video.google.com/videoplay?docid=522784499045867811&q=denial+machine#

    In those links you will find evidence/proof of payment for distorted science, a memo that proves the anti-AGW lobby knew the science was solid – from their own scientists – back in 1995, an explanation showing how the cigarette denialists became the climate denialists (and if they are proven liars already, are supposed to trust them now?), and of how the government itself intentionally distorted the truth.

    What you won’t find there, or anywhere else, is anything even remotely approaching a conspiracy to **support** AGW.

    Please respond to the above evidence/proof of conspiracy by the anti-AGW crowd. That is far more settled, far better evidenced, and is actually proven.

    Deal with that, if you’ve the ethical and moral fortitude.

    Comment by ccpo — 28 Nov 2009 @ 5:44 PM

  117. Times Online has posted an article claiming that RC pulled their own postings:
    “It was a powerful and controversial mix — far too powerful for some. Real Climate is a website designed for scientists who share Jones’s belief in man-made climate change. Within hours the file had been stripped from the site.

    Several hours later, however, it reappeared — this time on an obscure Russian server. Soon it had been copied to a host of other servers, first in Saudi Arabia and Turkey and then Europe and America.”
    http://www.timesonline.co.uk/tol/news/environment/article6936289.ece

    [Response: The zip file was temporarily available (about 30 min) from a link to this site until we shut it down and then removed it. It was never posted as a blog entry. - gavin]

    Comment by MarkusR — 28 Nov 2009 @ 5:48 PM

  118. “Mark A. York,

    Since you know so much, perhaps you could tell me why the oceans have not been warming after 2003 — or if they have why it’s a very slight warming contrary to the projections of the IPCC’s models.”

    Sounds like an either/or fallacy. To my knowledge oceans are warming. That would be first time I’ve seen such a claim. Sounds like a clone of the warming stopped in 1998 offhand.

    Comment by Mark A. York — 28 Nov 2009 @ 5:48 PM

  119. Can somebody recommend a good paper that gives a basic overview of the physics in the line-by-line radiation transfer codes? Turns out I’m writing a model (for an entirely field) that has some of the same features (absorption, scattering and emission at multiple wavelengths), and I want to make sure I’m doing it right. I assume Beer’s Law is at the heart of the absorption description?

    Comment by tharanga — 28 Nov 2009 @ 6:06 PM

  120. Mark A. York,

    No, it’s not an either-or fallacy. NPR had an article last year about global warming’s missing heat.

    http://www.npr.org/templates/story/story.php?storyId=88520025

    Comment by David — 28 Nov 2009 @ 6:12 PM

  121. The Mystery of Global Warming’s Missing Heat
    http://www.npr.org/templates/story/story.php?storyId=88520025

    I think I forgot to give the link in my last comment.

    Comment by David — 28 Nov 2009 @ 6:14 PM

  122. You might also put up a link to the NEMO Ocean model. Both the source, documentation, version control and bug database is available with registration:

    http://www.nemo-ocean.eu/

    GFDL also has put up plenty of data (http://data1.gfdl.noaa.gov/) and with registration, bits of their model (http://www.gfdl.noaa.gov/fms).

    Comment by jon pipitone — 28 Nov 2009 @ 6:21 PM

  123. For Mark York:
    I just pasted his question into Google, and this is near the top of the first page of results:

    http://global-warming.accuweather.com/2008/03/oceans_are_not_warming_1.html

    Likely that (or icecap or one of the other you know what sites) is the source of that idea, which comes probably from Domingues (see below)

    Accuweather points to the Argo homepage,
    where there is a pointer to a nearly empty Argo blog that includes one very old topic on temperatures (Domingues paper, see below)

    That also points to a main ARGO page that CSIRO has been hosting, which provides weekly updated T/S profiles for the global Argo array:

    http://www.marine.csiro.au/~ttchen/argo/gmap.htm
    From there you can look up each individual float and see its data set and a recent summary.

    I’d imagine there’s more there to find; I gave it about four minutes’ worth of poking around just for the exercise.

    See also:
    http://www.realclimate.org/index.php/archives/2008/06/ocean-heat-content-revisions/
    http://www.realclimate.org/index.php/archives/2009/07/more-bubkes/

    As always (for any new reader here who hasn’t heard this):
    I’m just another reader here, I know nothing about this except what an amateur can do to look things up. Nothing I post is more than a “here’s what I found” example. The real scientists can correct or improve on what I find. So can you, whoever you are, just take a few more minutes — read the papers, read the footnotes, look up the more recent papers.

    The latter is key. The “oceans not warming” thing was probably from a paper more than a year ago, and — peer review is only the beginning. Look for discussion of it to see if the idea has held up.

    Comment by Hank Roberts — 28 Nov 2009 @ 6:25 PM

  124. Can you comment on the attached graph here http://i44.tinypic.com/29dwsj7.gif ?

    This appears to have the US temperature anomaly as reported by GISS in 2000 and 2009, and a large difference between them. What is the reason for these differences and is the source code as used in 2000 available?

    [Response: The differences are well documented in the Hansen et al (2001) paper and are due to needed corrections for various problems (such as changes in the timing of observations) provided by NOAA. - gavin]

    Comment by Alw — 28 Nov 2009 @ 6:33 PM

  125. A PS for Gavin, if there is (or will be) a FAQ on the tools in use and how data is collected, this is marvelous:
    http://smart-machines.blogspot.com/2008/02/slocum-glider-green-energy-underwater.html

    The “Slocum glider” design was first published as science fiction, here:
    http://www.tos.org/oceanography/issues/issue_archive/issue_pdfs/2_1/2.1_stommel.pdf

    (Someone in a thread here mentioned the author in the last few days — THANK YOU! I looked him up. Oh, wow. Science. It really does work wonders.)

    Here’s one way (there are other ways) that they’re in use:
    http://www2.sese.uwa.edu.au/~hollings/anfog/
    Detail:
    http://www2.sese.uwa.edu.au/~hollings/anfog/index.php?page=deployments

    An example from an ARGO device page I picked at random from the huge map:
    http://www.marine.csiro.au/remotesensing/oceancurrents/profiles/5901161/latest.html

    Comment by Hank Roberts — 28 Nov 2009 @ 6:36 PM

  126. In addition to what Hank posted, see this article from the NASA site:

    “In mid-2008, however, a team of scientists led by Catia Domingues and John Church from Australia’s CSIRO, and Peter Gleckler, from Lawrence Livermore National Laboratory in California, revised long-term estimates of ocean warming based on the corrected XBT data. Since the revision, says Willis, the bumps in the graph have largely disappeared, which means the observations and the models are in much better agreement. “That makes everyone happier,” Willis says.”

    (Note that this postdates the NPR story.)

    From:

    http://earthobservatory.nasa.gov/Features/OceanCooling/

    Comment by Kevin McKinney — 28 Nov 2009 @ 6:40 PM

  127. Gavin,
    your code (as found here http://data.giss.nasa.gov/gistemp/sources/GISTEMP_sources.tar.gz) is rather flaky and lacks basic documentation to make it usable.
    I spent time reading through the code since executing it in any meaningful manner seems arduous. The reading of the code points to it being nothing special … most of it is data conversion and rudimentary data filtering, but outside of that there is nothing/(very little) that would make this code interesting.

    Can you provide better documentation on what this code is supposed to do and how can one verify that the output (after running all steps) is valid?
    For the sake of simplicity, I imagine you have this code working with some dataset … is it possible for you to publish the input parameters that you are using and also the resulting output?

    Thank you.

    [Response: I never said it was perfect. It is the working code that they use (and it isn't my project just fyi). It isn't that complicated because most of it is database stuff plus some corrections for rural/urban differences plus some interpolation. What were you expecting? If you want to know more about it, read the references. The input is the public files (like v2.mean from GHCN) and the output is on the GISTEMP website. There are intermediate results available in the 'Stations' pages. - gavin]

    Comment by donQ — 28 Nov 2009 @ 6:45 PM

  128. For the processed data sets, could you put links to the code used to process those data sets? It looks like you have the code in a different section, it might be nice to have those side-by-side.

    Thanks again for the data!

    Comment by Jere Krischel — 28 Nov 2009 @ 6:56 PM

  129. Gavin,

    Re 124, I have just had a scan through the Hansen reference and one line sticks out:

    “Some prefatory comments about adjustments to the temperature records are in order. The aim of
    adjustments is to make the temperature record more “homogeneous,” i.e., a record in which the temperature change
    is due only to local weather and climate. However, caution is required in making adjustments, as it is possible to
    make the long-term change less realistic even in the process of eliminating an unrealistic short-term discontinuity.
    Indeed, if the objective is to obtain the best estimate of long-term change, it might be argued that in the absence of
    metadata defining all changes, it is better not to adjust discontinuities.”

    So it would seem the pre 2001 version is a better reference for long term trends?

    [Response: Only if you didn't have any relevant metadata. If you know that people used to take mesurements a 5am, and then they change to noon, that needs to be corrected for. And that's what done. - gavin]

    Comment by Alw — 28 Nov 2009 @ 7:11 PM

  130. Gavin, I wish you hadn’t told me not to look up the (other) definition of wip-wap. On the other hand, this gives me an idea: maybe you should tell the denialati not to read a good textbook on climate physics, and see if it has the same effect.

    Comment by S. Molnar — 28 Nov 2009 @ 7:15 PM

  131. Gavin,
    it is true that you didn’t say “it [the code] was perfect”. It also true that from computer science this code is very, very far from perfect. Rudimentary error handling is not in place. Someone has to copy the “temp results” folder from one step to the next. There is no way to ensure/validate if this “temp results” folder contains meaningful data.

    I’m sure that someone is using this software for something … but apart from the simple utility that can be provided by this code, I can’t see how this code comes anywhere near “computer modeling” software. It is, at beast, translation utility.

    What I was expecting is to see (or at least have a peek at) the modeling software, and the heuristic prowess contained therein.

    BTW, I did read the references, within the source code package, and there isn’t much to talk about … I don’t know how using the software I can arrive at conclusion that, for example, “Earth is overheating based on human activities that produce C02″ — as far as I’m concerned this rise in temperature might very well be true, precisely because of CO2. However the software should be at least able to model, unambiguously, the fact that Earth is warming up (regardless of cause), but it doesn’t seem that this software can be used to show even that. So again, this software seems to be just a conversion utility and nothing more. (?!)

    BTW,
    I realize that this project is not YOURS. I also understand that even if it was yours you would rather not admit it. ;-) (I wouldn’t admit it either.)

    [Response: Who said this was computer modelling? Go to computer models for that! -gavin]

    Comment by donQ — 28 Nov 2009 @ 7:18 PM

  132. For those who have a particular interest in the GISTEMP code base, our Clear Climate Code project is in the process of re-implementing GISTEMP, with a particular emphasis on clarity. We’re using Python, chosen specifically as a language in which it is easy to be clear. There are various stages to the reimplementation, starting with running the original GISTEMP code, then ripping out the ksh dependency, then step-by-step reimplementing in Python (this part is mostly done now). Once we have an all-Python implementation producing the same results, we’re going to look at further improvements and clarifications (for instance, replacing the rather cumbersome file-to-file staging with faster and clearer iterator pipelining). It’s taking a long time because the people involved all have day jobs, families, etc.
    The process has revealed some bugs in GISTEMP, all of which have been reported to GISS and none of which make any material difference to the results (we think we’ve recently found some similarly inconsequential problems in STEP5, which we will be communicating to GISS as soon as we’ve convinced ourselves).
    The code in in Google Code, at
    The discussion mailing list is at
    We are reworking the website, so there’s nothing to see there right now.

    Comment by Nick Barnes — 28 Nov 2009 @ 7:27 PM

  133. Argh, link breakage, sorry. Google for ccc-gistemp or ccc-gistemp-discuss to find our code/resources/mailing list.

    Comment by Nick Barnes — 28 Nov 2009 @ 7:29 PM

  134. Benjamin (115) — It seems that some have rather suspicious time zones attached and some are strangely truncated; too many chare the same uid and gid. I reiterate that nobody has authenticated all of the e-mail messages wihtout asserting a claim that all or even any were fabricated, just that some doubts about complete authenticity have arisen.

    Comment by David B. Benson — 28 Nov 2009 @ 7:32 PM

  135. Slightly related: I just sent an email to CNN about their coverage of Copenhagen tonight… instead of getting moderate people to come on and answer the questions Malveax was asking (what do you think Obama can accomplish, etc), they had Carville and of all people, Ben Stein, who naturally sat there editorializing with disinformation, including the insistence that data is being withheld due to malfeasance. I don’t know if you can watch the clip online or not, but anyone who has seen it, or the presumable re-run of it later this evening, please email CNN and let them know how appalling their coverage of CC has been and continues to be. They didn’t need to have a complete ideologue on this issue, especially when they are supposedly moving away from that. If they wanted both sides, they could have picked Lindsey Graham or someone else with a moderate approach. According to their own poll, shown in that segment, 51% of Republicans believe it’s happening, so don’t tell me they couldn’t have found anyone. CNN has consistently failed on this issue, and I couldn’t be much angrier right now! Their entire staff is completely uneducated in this area, while it’s a huge part of our reality and of news. Here is the link for the comment form for the Situation Room, so if you’ve seen it and are disgusted, please let them know! http://www.cnn.com/feedback/forms/form5.html?65

    Comment by Shirley — 28 Nov 2009 @ 7:38 PM

  136. “Silke” (#89) appears to have lifted text from Roy Spencer — see http://www.drroyspencer.com/, “My Top 10 Annoyances…”. Marco #108 and Martin Vermeer #114 don’t seem to read widely on these issues and appear to be responding to a troll. :-)

    Comment by BillyRuffn — 28 Nov 2009 @ 7:44 PM

  137. Alan of Oz #50

    Sure it’s nice to have the code but the code itself is irrelevant to the science.

    I can’t believe you posted that. The source code for developing models is irrelevent? Why even have code then? I’m puzzled by where you were going with that.

    Software configuration practices and OSS are not particularly usefull for reproducing results in the scientific sense since the same code will always produce the same results and you will be stuck with the impossible task of demonstrating the code is bug free.

    There’s nothing stopping people from having different projects that examine the same data. In fact that’s what has been done in linux development — the scheduler for example was done in two different ways.

    Reproducing the same results with the same data and different code is a much more powerfull test because it provides confidence that the results are independent of a particular implementation of the methods.

    That’s is 100% correct and is why climate models / code needs to be open and freely available. Without open access to other’s codes how can one find out why the results of two programs differ? How can others expect to become better without looking at each other’s work? And finally how can one find out what’s going on in a code based without a history (in checkins and comments) of the code?

    I find it more than a little ironic that climate scientists have to educate programmers on the benifits of multiple independent source trees. It’s not a hard concept to grasp, double entry bookeeping works on the same principle and even an accountant can do that ;)

    Why does open source mean one repository only? The open source world is full of forks to projects that have either gone dead or not the way people wanted them to go.

    Comment by BlogReader — 28 Nov 2009 @ 7:50 PM

  138. Re:76 Holly Stick, 33 Garrett. I saw that as well. During summer even now the channels are effectively open, not even sure you would need an icebreaker, just vessels with a reinforced bow should be ok.
    I think Big Oil can see the bginning of the end…the global accord and spirit of cooperation growing in leuw of Copenhagen is very impressive exept for the US who again is being led rather than taking a leading position.
    Does anyone know what happening to the polar bears in the arctic region..I haven’t heard about them for quite a while.

    Comment by Lawrence Coleman — 28 Nov 2009 @ 8:24 PM

  139. DonQ, it’s research code, rather than a published widely-used application. It’s the software equivalent of a lab setup. To understand it, you have to read the docs, probably read the code, go to the published papers, and perhaps spend time as a student.

    In another field, I’ve done some of the work of making research code into production code. It’s hugely time- consuming. Writing documentation takes at least as long as writing research code and isn’t nearly as much fun. Developing user interfaces also takes much time, and the skills required are usually different than those of developing research code. Perhaps worthwhile work but so far GISS hasn’t taken it one.

    Comment by The Raven — 28 Nov 2009 @ 8:40 PM

  140. The Raven says “…we have no way of judging the honesty of the criminals who broke in…”

    OK, I want to make sure everybody sees this. I damn near fell out of my chair. By all means, we want to make sure we are dealing with honest thieves…

    Comment by Ray Ladbury — 28 Nov 2009 @ 8:45 PM

  141. Hey thanks Hank. Why is he asking me is my question? I mean I did a herring cruise off the coast of Maine once but that isn’t my specialty if I have one at all. When on couches a question like that with “if you know so much” then…it’s hard to take it seriously. It sounded like a contrarian talking point to me. Seems like it is. They don’t do nuance well.

    Comment by Mark A. York — 28 Nov 2009 @ 8:46 PM

  142. under the listings for RAW data is the following:
    Climate data (raw)

    GHCN v.2 (Global Historical Climate Network: weather station records from around the world, temperature and precipitation)

    When you look at the readme for that file it is:

    The purpose of this file is to provide a description of and the format for the
    following USHCN data files:

    hcn_doe_max_data.Z Areal Edited, Time of Observation, and Filnet Adjusted
    Maximum Monthly Temperature
    hcn_calc_mean_data.Z Time of Observation and Filnet Adjusted Mean Monthly
    Temperature (Calculated from hcn_doe_max_data.Z and hcn_doe_min_data.Z)
    hcn_doe_mean_data.Z Areal Edited, Time of Observation, and Filnet Adjusted
    Mean Monthly Temperature
    hcn_doe_min_data.Z Areal Edited, Time of Observation, and Filnet Adjusted
    Minimum Monthly Temperature
    hcn_doe_pcp_data.Z Areal Edited, Time of Observation, and Filnet Adjusted
    Monthly Precipitation
    station.history.Z Station History
    station.inventory.Z Station Inventory
    station.landuse.Z Station Area Land Use and Land Cover
    urban_max_fahr.Z Urban Heat Adjusted Maximum Monthly Temperature
    urban_calc_mean_fahr.Z Urban Heat Adjusted Mean Monthly Temperature
    (Calculated from urban.max.Z and urban.min.Z)
    urban_mean_fahr.Z Urban Heat Adjusted Mean Monthly Temperature
    urban_min_fahr.Z Urban Heat Adjusted Minimum Monthly Temperature

    Not exactly raw data!

    Comment by Alw — 28 Nov 2009 @ 8:47 PM

  143. Gee, 2 weeks of denialists posting and so far not one has actually taken issue with or even mentioned the evidence. Hmm. Why would that be?

    Comment by Ray Ladbury — 28 Nov 2009 @ 8:49 PM

  144. Post 116, in response to post 76:
    >Dear Forrest…Deal with that, if you’ve the ethical and moral fortitude.

    Ok, sorry but I call foul. By his linked website, Forrest is somewhat right leaning, and perhaps “on the other side” – but he’s *also* an admirable science geek. When in doubt, be the one toning it down. Straight rational argument works best when dealing with non-dittoheads (of course if they don’t respond with same, then disengage or mock…)

    As I posted above, there is a larger fight in society against anti-science within which the smarter skeptics on AGW are our *allies*. Many are clearly strong on basic enlightenment values and the scientific method. Many are science geeks. They are in fact even in the best position to tone down the crazy of the real denialists through basic education about the mechanisms involved.

    If there is a lesson in this mess, it is neither in the interests of the cause of reducing CO2 emmissions, nor the greater cause of upholding science itself, to assume all skeptics are Denialists. If someone starts throwing accusations around, even if it’s 99% lies, don’t let your ego stop you from improving your own behavior as a response to the criticism. As per the original the subject of this thread.

    Comment by Bruce the Canuck — 28 Nov 2009 @ 8:56 PM

  145. Stage 1: If you don’t make all the data and model code available, I refuse to believe you.
    Stage 2: If you won’t teach me how to analyze the data and run the models, I refuse to believe you.
    Stage 3?: This stuff is too difficult — it can’t be true. (“All I really need to know I learned in kindergarten.” — Robert Fulghum)

    Comment by Jerry Steffens — 28 Nov 2009 @ 9:10 PM

  146. >your code (as found here http://data.giss.nasa.gov/gistemp/sources/GISTEMP_sources.tar.gz) is rather flaky and lacks basic documentation to make it usable.

    It seems to me that this comment goes right to the heart of the issue. Documentation of raw data takes time (and staff) away from other projects. Even writing email in a context free style (for public consumption) takes a lot of effort. If only there was a way for scientific staff to devote resources to getting certain material in publishable shape, while keeping other materials under wraps!

    Comment by Jeremy Erwin — 28 Nov 2009 @ 10:03 PM

  147. ” Barton Paul Levenson says:

    Read my lips, TOA. Hacking into the CRU computers without CRU’s permissions was ILLEGAL. AGAINST THE LAW. A CRIME. If they can find out who did it, those people are going to jail. That’s what “illegal” MEANS.”

    It is not a CRIME if it is shown that the people who did the deed can show that the public interest in disclosure of the information outweighs the public interest in non-disclosure. A public interest defense is a defense which allows a defendant who disclosed classified or protected information to avoid criminality, if (s)he can establish that the public interest in disclosure of the information outweighs the public interest in non-disclosure. This defense is part of UK law and is specifically aimed at protecting whistleblowers of governmental misconduct.

    http://freedom.libdems.org.uk/the-freedom-bill/13-public-interest-defence-for-whistleblowers/

    Now you do not know how 12 members of a British jury will act, when presented with such a defense, any more than you know about the amount of snow there will be at the North Pole in 20 years time; but the lack of knowledge has not stopped you talking out of your arse on both subjects.

    Comment by DocMartyn — 28 Nov 2009 @ 10:03 PM

  148. I’m waiting for “open sourced” emails from American Enterprise Institute, the Heritage Foundation, the Cato Institute, Competitive Enterprise Institutes, the Reason Foundation, et al

    After all, what’s good for the goose. . .

    Comment by Steve — 28 Nov 2009 @ 10:17 PM

  149. “This is a powerful meme and one that has clear resonance far beyond the people who are actually interested in analysing data themselves.”

    That statement is arrogant and condescending and is contrary to the spirit of free inquiry. At pity indeed that realclimate.org will reject this post. Shame on you!

    Comment by VGD — 28 Nov 2009 @ 10:19 PM

  150. “I’m no scientist and I have no understanding about the numbers. But I can tell when someone is BSing me.”
    So if you can’t do the numbers, what do you use instead – Tarot cards? flipping a coin? Rolling dice? Trust Glenn Beck or Rush Limbaugh tell you who’s BSing? Maybe you find an aristocratic British accent like Lord Monckton’s convincing? Or perhaps the erudite, sophisticated, content free vocabulary of George Will tells you what you want to know?
    If you can’t tell why someone wants to run a 40 year Fourier convolution filter on one set of temperature data, but draws straight lines through two year peaks of Arctic ice data in order to claim warming isn’t happening, you might as well use some of the “proxies” suggested above. I would recommend flipping a coin, as this would give you the best (50%) odds of actually being right.

    Comment by Brian Dodge — 28 Nov 2009 @ 10:21 PM

  151. Excellent article Kevin KcKinney. Thanks! That sure explains the ocean warming/cooling dichotomy and is a great example of how science is self-correcting. Our contrarian friends stay with the headlines, which can be written by editors who know little about the story. And thanks Gavin for that last answer.

    Comment by Mark A. York — 28 Nov 2009 @ 10:21 PM

  152. I keep a directory of online scientific data resources at Element List (http://www.elementlist.com/lnx/scientific_data/).

    Comment by Jackie — 28 Nov 2009 @ 10:30 PM

  153. Thanks David B. Benson, I am having a look now. Of course, much of it is beyond me. It is not a bad thing to admit, I am just a normal guy trying to make sense of it all.

    But may I say that I am heartened that my comments were posted. I think your site has a wonderful opportunity to play a role in the healing.

    I am certainly lobbying within the so called denier camp that we should refrain from using words lile “alarmist”. I think you should try to stop using the word “denier”.

    I suggest this not just from a human perspective but from a PR angle. Yes, I know that is not a wonderful angle but it is vital. I simply think that using that word is going to rebound in light of the CRU emails. I think it will be too easy for us to turn that around and make it about the denials that are revealed in the emails.

    To me, that would be a bad outcome. I liked someone saying that this is not about winners and losers.

    So, all power to you Gavin. I am urging you to play a big part in changing the tone. SOme of your own posts here show that you are trying but you are slipping into old ways.

    Peace

    Comment by Jimi Bostock — 28 Nov 2009 @ 10:31 PM

  154. I keep a large list of online scientific databases at Element List (http://www.elementlist.com/lnx/scientific_data/).

    Comment by Jackie — 28 Nov 2009 @ 10:32 PM

  155. #50 Jimi, I found this site a couple of years ago after seeing the Great Global Warming Swindle, and I’ve followed the climate change discussions, both scientific (as best I can) and political, quite closely ever since.

    I suggest you look back over some of the years of posts here and get a sense of the culture and of events from before the email hack.

    Over and over again I’ve seen scientists answer the same questions and self-described “climate skeptics” ignore the answers. Over and over again I’ve seen “climate skeptics” attack the motives and integrity of scientists and ignore the content of their work. Over and over again I’ve seen open discussion and dispute about details of climate science distorted by “climate skeptics” into statements that global warming is all wrong.

    I don’t mean to suggest that you’re a phony skeptic like most of the bloggers I’ve read. All I’m suggesting is that you ask yourself how you might talk to colleagues or friends about requests and attacks from people who had demonstrated bad faith many times over.

    Comment by Susanne — 28 Nov 2009 @ 10:40 PM

  156. Gavin, I DEMAND that you teach me how to be a climate scientist IMMEDIATELY, so that I can expose this giant con you are peddling!

    Seriously though Gavin, that you actually still appear to be sane and rational after reading some of the recent comments is nothing short of a miracle.

    I am eagerly awaiting the stony cold silence which will accompany the first competent (if that’s not wishful thinking) sceptical “re-analysis” of the data.

    Comment by Mike of Oz — 28 Nov 2009 @ 10:58 PM

  157. For anyone who knows anything about science, the refusal to release data, codes, methods, proxies, all of it, until forced to do so is hard to comprehend.

    The excuse is that CRU has confidentiality agreements with the providers of data, but that data sets that are public correlate 98% with the data set used by CRU.

    This brings up two issues. One, if the data sets are substantially the same, why did CRU pay for something that is free elsewhere? Two, why would anyone involved in the search for the unbiased truth sign a confidentiality agreement that would prevent others from having access to the raw data, along with all of the “value added” statistical adjustments?

    As a corollary, when physicists were debating competing theories for origin of the universe (big bang vs steady state) one side did not make information unavailable to the other, and the debate between the two views was every bit as contentious as this one. Ultimately, the overwhelming evidence proved big bang…until something else comes along.

    Comment by Bill S — 28 Nov 2009 @ 11:09 PM

  158. This story from the London Times forthrightly states that the raw data that Phil Jones and his team depends upon has been lost.
    http://www.timesonline.co.uk/tol/news/environment/article6936328.ece
    Can you clarify whether this is true or not.
    Thanks

    [Response: No. The original data is curated at the met services where it originated. - gavin]

    Comment by Bernie — 28 Nov 2009 @ 11:34 PM

  159. Re #67 – “Alan of Oz brings up a great point, code and data availability isn’t going to help.”

    To clarify my point, access to data and methods is required for reproduction, access to code is irrelevant.

    Comment by Alan of Oz — 28 Nov 2009 @ 11:41 PM

  160. “I’m no scientist and I have no understanding about the numbers. But I can tell when someone is BSing me.”

    Really? If you don’t understand the science or math, than how can you possibly know who’s BSing who? You can’t. You probably inherently trust someone merely because they agree with you in some unrelated realm (e.g. politics or religious). Sadly that’s not a very good mechanism for determining the truth.
    If you lack the science and math to understand this complex field, then the rational thing to do would be to accept what the vast majority of practicing climate scientists are saying (unless you buy into some silly conspirasy theory or discount all of them as liars).

    Comment by Ken W — 29 Nov 2009 @ 12:01 AM

  161. ccpo –

    Not sure what I wrote that justifies the long winded reply. I pretty much agree with much of you wrote, but I might employ a less strident tone to express it.

    I’m really not sure where I said anything fundamental has been done in error.

    Indeed my criticisms of the “climate change project” would boil down to saying that the sea-ice modeling looks pretty conservative, and I’d like to see lots more work on biofeedbacks which look poorly characterized. I’m really interested in whitecap albedo, and the papers I’ve read on this have not been clear. Further I’m not sure why there’s such an emphasis on all this dendrochonology stuff – to me it’s academically interesting but at the end of the day doesn’t really answer the more fundamental question: “How does emitting a bejesus-load of GHG alter the planet?”

    That’s just my amateur judgment. None of this says it’s a good idea to emit a bejesus-load of GHGs. At the end of the day not much turns on it. If you disagree I’d love to hear why.

    My point was that linking batches of code is pretty useless as a pedagogical activity, and I learned a lot more from reading a textbook. Of course I don’t think what Gavin is doing is a pedagogical activity – it’s more making a rhetorical point, and not a bad one at that. But pedagogy never hurts, and a few textbooks might be a nice addition to the site. I said the Washington and Parkinson’s text is pretty good. As a suggestion to the site I said it would therefore be nice to have a few textbook references on the “books” page.

    Comment by Patrick Caldon — 29 Nov 2009 @ 12:12 AM

  162. Gavin,

    I saw your link to the NSIDC but to make collection a bit easier here is the direct link to the submarine upward looking sonar data.

    http://nsidc.org/data/g01360.html

    Mac

    Comment by Mac — 29 Nov 2009 @ 12:17 AM

  163. Is the data raw or corrected. If it is corrected, where did the corrections primarily come from, and where would one find the raw data with a listing of the corrected algos( if aplicable)?

    Thanks!

    Comment by Joe V. — 29 Nov 2009 @ 12:21 AM

  164. What’s required is that the data, metadata, program source code and program output for each published paper be put together for review by skeptics and allies alike. Then the code can be run on the data, the output confirmed, and any mistakes identified. Then modifications to the programs can be run, by skeptics and allies alike, and the differences caused by the modifications can be clearly identified. That way, assumptions that are made, and decisions that are made, can be clearly shown to determine, or not determine, specific claims made about the results. Put differently, for each published paper there needs to be a complete and accurate data audit trail.

    That is what ClimateAudit has persistenly requested. What RealClimate has supplied instead is a direction such as “We used the data at such and such a website.” That would never work for a patent application, or FDA approval.

    Comment by Matthew — 29 Nov 2009 @ 12:21 AM

  165. donQ says:
    28 November 2009 at 6:45 PM

    Gavin,
    your code (as found here http://data.giss.nasa.gov/gistemp/sources/GISTEMP_sources.tar.gz) is rather flaky and lacks basic documentation to make it usable.

    What a hoot! Folks have demand, DEMAND! access to the code, and then have to ask for tutorials on what it is and does. Almost 30 years ago, Fred Brooks observed, in The Mythical Man Month that a software product is three times harder (read expensive) than a program. So what should our civil servants be doing with our money, building programs that are good enough to do their day-to-day jobs, or writing products that allow complete noobs to do the job too, looking over our civil servants’ shoulders? This is a serious question. According to the CRU emails, the CRU grants for the last 19 years average, roughly, a million USD per year. In the Washington, DC area I could cover the salary, benefits, overhead, and G&A for 4 good software product developers. I doubt that I could cover 5.I won’t take the folks who are demanding better code seriously until they also start demanding that the budget for the climate folks be tripled to pay for turning programs into products.

    Comment by Tim McDermott — 29 Nov 2009 @ 12:48 AM

  166. 116: If Forrest Mims is not considered someone worth listening to here (**screed filled with blatantly false claims deleted**) then I’m at a loss. I don’t know what else to say.

    Comment by debreuil — 29 Nov 2009 @ 1:00 AM

  167. [Response: No. The original data is curated at the met services where it originated. - gavin]

    No one seems to know this but you… CRU is not pointing people to “the met.” Could you offer greater confirmation of this claim?

    Here is the relevant CRU statement regarding this issue:

    “We, therefore, do not hold the original raw data but only the value-added data.”

    Comment by Adlac — 29 Nov 2009 @ 1:11 AM

  168. This brings up two issues. One, if the data sets are substantially the same, why did CRU pay for something that is free elsewhere?

    They didn’t pay. They got the data, for free, under an agreement (“contract”, just in case you’re obtuse) that they’d only use it for research purposes and not release it without permission from those who provided it.

    Two, why would anyone involved in the search for the unbiased truth sign a confidentiality agreement that would prevent others from having access to the raw data, along with all of the “value added” statistical adjustments?

    Because they want as much data as they can get. More data is better. A couple of decades ago they never imagined that someone would be fabricating a political McCarthy-ist attack on their research over something that’s not at all uncommon in science. They never imagined they’d have to defend themselves against extremist anti-science right-wingnut attacks.

    Comment by dhogaza — 29 Nov 2009 @ 1:18 AM

  169. That is what ClimateAudit has persistenly requested. What RealClimate has supplied instead is a direction such as “We used the data at such and such a website.” That would never work for a patent application

    This person actually believes that the patent office requires applicants to provide working source code and ACTUALLY RUNS IT?

    The mind boggles. The earth must be flat, after all, and the sky green and the sea sweet and my tap water salty.

    Comment by dhogaza — 29 Nov 2009 @ 1:21 AM

  170. Whoah, debreuil — 116 is just ccpo, quoting part of what Forrest Mims wrote much earlier at http://www.realclimate.org/index.php/archives/2009/11/wheres-the-data/comment-page-2/#comment-145290

    Nothing has been deleted; ccpo used the word “deleted” but meant “elided” or “omitted from quoting” — notice the timestamp is provided so you can find the original for yourself.

    Forrest Mims doesn’t specify what he’s referring to:
    “… advocating the misconduct revealed in many of the Hadley CRU e-mails” could be a specific accusation if he were specific; without specificity, it’s only an opinion that there must be something wrong somewhere.

    Everyone wants to be judge and jury, but none of us get to decide what really happened; all we have is “randomly selected” (whatever _that_ means) bits from some larger stolen email file.

    Why aren’t people demanding that the crackers reveal the rest of what they’re hiding, so we can know the whole truth?

    Oh, wait, that won’t work well for them, will it?

    Comment by Hank Roberts — 29 Nov 2009 @ 1:30 AM

  171. Bernie says:

    “This story from the London Times forthrightly states that the raw data that Phil Jones and his team depends upon has been lost.
    http://www.timesonline.co.uk/tol/news/environment/article6936328.ece
    Can you clarify whether this is true or not.
    Thanks

    [Response: No. The original data is curated at the met services where it originated. - gavin]”

    Gavin, the original data may indeed be at the met services where it originated, along with a lot of other data that CRU chose not to use for its global temperature record. So unless CRU released a list of which stations it used and which stations it chose not to use (which it has not done) saying that the original data is at the local met services is like saying that the answer to the “Who wants to be a millionaire” final question is in the encyclopedia. I’m sure it is, but the “trick” is: where?

    Comment by Sean — 29 Nov 2009 @ 1:31 AM

  172. I have a question about whether (and if so, how) gridded temperature averages are weighted.

    I am not talking about station temperature weight based on distance to the grid point – that’s a separate topic.

    What I want to know is, when gridded average temperatures are themselves averaged to determine regional/global averages, if each grid square ,ie. 2 or 5 degrees, is weighted for its actual surface area. I looked at the zonav.f code file, and I didn’t see any code that weighted grid squares by area.

    On this topic, recent articles refer to previous articles going back to Hansen et al. 1981. Hansen et al 1981 states “The results shown were obtained with 40 equal-area boxes in each hemisphere … and the global trend was obtained by area-weighting the trends for all latitude zones.” But there is no detailed explanation or formula for how these boxes were weighted or even if “equal-area boxes” means equal by surface area or if it means 2 X 2 degree or 5 X 5 degree squares as is used currently.

    Since a grid square at, say, 60 degrees latitude is half the size of a square at the equator, it seems that heating or cooling away from the equator should receive less weight per gridded square. Does that happen in the calculations? If so, can you point me towards details? Thanks in advance for your help with this.

    [Response: If you do a global mean from gridded data, then yes, you weight for the area of the box. A quick estimate is to use the cos(lat) for the mid-point of the box. A lat-lon grid (i.e. 2x2) definitely needs this, but you can have 'equal area' grids that have each grid box have the same area, in which case the global mean is simply the unweighted average. - gavin]

    Comment by Mike — 29 Nov 2009 @ 1:39 AM

  173. Alan of Oz To clarify my point, access to data and methods is required for reproduction, access to code is irrelevant.

    Why? Why shouldn’t the exact means for which a paper is being based on not be freely available?

    I don’t understand what you’re trying to accomplish here. yes the code might have warts in it, yes it might be ugly. Who cares if the most efficient way of sorting an array wasn’t used, just make the code available.

    Can you clarify what you mean by “irrelevant”? You state that the “methods” should be open, so you mean describing the code but without releasing it? What if there was a mistake in there that many eyeballs would catch?

    [Response: We discussed this above. The replication that is required in an observational science like climate is the replication of the conclusion. Perfect code, perfectly executed is worthless if the basic assumption that went in was flawed. Independent approaches using either the same base data, or completely different data are much better at assessing this than checking other peoples' code line-by-line. Two different climate models that give the same result are much more informative than taking one of them and looking for bugs (which undoubtedly exist). Given limited resources, scientists focus mostly on issues that matter - ie. verifying/challenging conclusions rather than checking code. Sometimes there are problems, but these usually emerge because someone's independent but nominally similar approach gives a different result (such as with the MSU records) - at that point it is worth looking more closely to see why that should be. - gavin]

    Comment by BlogReader — 29 Nov 2009 @ 1:39 AM

  174. donQ,

    The request was for “basic documentation.” Documenting one’s code is a software-development best practice. See HARRY_READ_ME.txt for an example of what happens if you don’t have adequate documentation.
    http://www.anenglishmanscastle.com/HARRY_READ_ME.txt

    Yes, there’s merit to the argument that even if adequate documentation was provided, a layperson may not understand it. However, Harry is (I assume) an expert, and he seemed despairing of ever getting this code to work, to say nothing of reconciling the databases’ issues.

    Comment by David — 29 Nov 2009 @ 1:55 AM

  175. “Esmeralda Dangerfield” said:

    This comment is not for attrition

    Then scientists will find it a welcome change from vexatious Freedom of Information requests and spurious insinuations of data fiddling.

    (attrition: “the act of weakening or exhausting by constant harassment, abuse, or attack” – Merriam-Webster)

    Comment by CM — 29 Nov 2009 @ 2:13 AM

  176. David Benson – I guess you’re calling me a sockpuppet. I’ve got a link to a blog in my username, I’m a contributor on that blog. I came here trying to get information and most of what I see in these comments is not info but snark and scorn. Depressing.

    Comment by Benjamin — 29 Nov 2009 @ 2:35 AM

  177. Tim McDermott, why sir, being a programmer and a hobbyist software developer, I am quite pleased by the ownage you have displayed. A hat tip to you!

    This discussion reminds me of the creation of the PDS. The PDS (Planetary Data System) is a system that NASA set up a while ago that all non-Earth science space related data paid for by a government grant (either NSF or NASA) must be released in to. Note, the PDS was possibly created due to conspiracy mongers begging for data and claiming falsification, though more likely for academia, only shutting up the conspiracy freaks coincidently; I put denialists in the same basic camp as moon hoaxers or Mars alien artifact freaks when it comes to claims for data and accusations against scientists.

    Comment by Josh Cryer — 29 Nov 2009 @ 3:34 AM

  178. Re 165: Agreed. The arrogance and wide-eyed naivety that is driving the (irrelevant) critsisims of code quality is unfortunately well known to experienced developers, a quote from an old friend of mine nails the behaviour very well – “Source code is like sh*t, everybody else’s stinks”.

    As for costs, I work in a group of about 25 developers/testers on an esoteric set of data tools that I doubt anyone here has heard of, it not the largest project I’ve worked on by a long shot but our annual budget is twice that of the IPCC! Like most of the commercial code I have seen over the last 20yrs source level comments are virtually non-existant and when they do appear are often misleading because they are not maintained.

    Comment by Alan of Oz — 29 Nov 2009 @ 3:43 AM

  179. Bruce the Canuck says:
    28 November 2009 at 8:56 PM

    Post 116, in response to post 76:
    >Dear Forrest…Deal with that, if you’ve the ethical and moral fortitude.

    Ok, sorry but I call foul. By his linked website, Forrest is somewhat right leaning, and perhaps “on the other side” – but he’s *also* an admirable science geek.

    And debreuil says:
    29 November 2009 at 1:00 AM

    116: If Forrest Mims is not considered someone worth listening to here (**screed filled with blatantly false claims deleted**) then I’m at a loss. I don’t know what else to say.

    Sorry, but I don’t know either of your stances – whether anti-AGW or not – but did you read what the guy wrote? His background means nothing to me. His actions do. Did you, again, read what he wrote? Based on the responses by Gavin, et al., here and others elsewhere, some of his claims were, as I’ve already said, blatantly false. That he listed his resume before his screed may have fooled you with an Appeal to Authority, but it did not me. He actually asked RealClimate to stop doing science with the people involved in the e-mails! What in the world do you think that implies?

    When in doubt, be the one toning it down.

    I wasn’t aware I was in doubt.

    …it is neither in the interests of the cause of reducing CO2 emmissions, nor the greater cause of upholding science itself, to assume all skeptics are Denialists.

    Why? They are. If they are so ill informed as to not know the very obvious, even as laypersons, perhaps they should not be posting, eh? Since the evidence is incredibly one-sided, and even the scientists here and elsewhere don’t dispute my contention posted here and elsewhere that there is not even one paper that in any way refutes any of the underpinnings of AGW, how can they be but a denialist?

    Asking me to be “nice” will get you nowhere. We’ve been too nice, and the nuts are winning.

    The only thing that will stop these pugilists is for them to have consequences. They need to find themselves in criminal or civil court.

    Cheers

    Comment by ccpo — 29 Nov 2009 @ 3:49 AM

  180. When people say they want to know where the data is, I assume they mean the data that may have been deleted, or not made public – not the data that has always been out there, or the data that you have not deleted.

    Just a thought.

    Comment by Mk — 29 Nov 2009 @ 3:49 AM

  181. Seeing a lot of comments on the blogosphere about software quality.

    As a software engineer that worked on a commercial open source software product (Gamebryo) I would say the time difference between just making something that works and making a standard conforming documented program, is about 10/1.

    Raw data and code are necessary, however I don’t think scientists need to create great code.

    All we need is the ability to do

    c:>runAnalysis.exe input.dat output.dat

    Can’t say if we are there or not, but that seems like an appropriate minimum.

    Comment by Jonathan Fischoff — 29 Nov 2009 @ 3:49 AM

  182. Patrick,

    You simply misread my intent. You highlighted the silliness of non-climate scientists asking for all the code and the data, is all.

    Comment by ccpo — 29 Nov 2009 @ 3:54 AM

  183. Even with the best software developers in the world, there is still the problem that they must understand the mathematics and algorithms to the level that they can think independently and discuss things at a high enough level with the climate scientists. That’s the *really* tricky bit: finding some software developers who are capable of crossing the scientific divide and still being damned good developers. Chances are, if they are that good at the science they will be in jobs doing the science.

    Comment by Donald Oats — 29 Nov 2009 @ 4:35 AM

  184. I once read on a programming support forum the following relating to many of the questions posed on said forum.

    “…it seems that a lot of the posts in this forum are from people who say things like: I want to learn how to do ray tracing in cobol on a palm pilot; I don’t know anything about programming yet, and clearly I’m not able to use a search engine either, but I want to be able to complete my project by noon tomorrow”.

    The “enquires” from the denialists on the last few RC posts remind me alot of this (of course with an obligatory reference to the scam added on).

    If you want to know the first thing about climate science, you may actually need to study it. Flipping through blog posts does not count as studying. You may actually need to crack a book, read a paper and do some friggin’ work before it’ll make much sense.

    Of course the vast majority of armchair “skeptics” have no intention of doing any work. If they had, this post would have been unnecessary, as instead of crowing “where is the data”, they would have spent the 10 minutes on google that are required to have found it themselves.

    Thanks for your efforts Gavin. Australian Bureau of Meteorology data can be obtained here: http://www.bom.gov.au/climate (raw AWS/METAR/SYNOP data is often unavailable… the Bureau sells it). Monthly SOI data going back to 1876 is available here http://www.bom.gov.au/climate/current/soihtm1.shtml

    Comment by ChrisC — 29 Nov 2009 @ 4:36 AM

  185. David: Since you know so much, perhaps you could tell me why the oceans have not been warming after 2003 — or if they have why it’s a very slight warming contrary to the projections of the IPCC’s models.

    BPL: Look again:

    Domingues, C.M., J.A. Church, N.J. White, P.J. Gleckler, S.E. Wijffels, P.M. Barker, and J.R. Dunn 2008. “Improved Estimates of Upper-Ocean Warming and Multi-Decadal Sea-Level Rise.” Nature 453, 1090-1093.

    Levitus et al. 2009. “Global ocean heat content 1955–2008 in light of recently revealed instrumentation problems” Geophys. Res. Lett. 36, L07608.

    [Response: I'll put up an updated figure on this shortly. - gavin]

    Comment by Barton Paul Levenson — 29 Nov 2009 @ 5:43 AM

  186. Re Ocean Heat Content:

    Levitus et al (GRL 2009) seem to use these OHC data (http://www.nodc.noaa.gov/OC5/3M_HEAT_CONTENT/)
    whichseem to show a levelling off since 2003/2004. Based on the amount of noise in the whole signal (see e.g. the relative flatline in OHC from the late 70s to the late 80s, when global air temperatures climbed steeply) I wouldn’t make too much out of it, but perhaps someone with more knowledge on the OHC content could provide a perspective?

    Re my comment 95 (see saw), guess it goes to show that I’m not a native speaker…

    Comment by Bart Verheggen — 29 Nov 2009 @ 6:08 AM

  187. Gavin et al.,

    Here is an idea:

    We all know that you are never going to satisfy the denialists, no matter how much data you provide, and how much code you supply. There will always be something more that they want.

    The fundamental issue is that proving AGW is actually a relatively straightforward matter of physics, combined with a few observations.

    So, how about stripping out everything except the bare essentials? Jettison tree-rings altogether. Ignore sea ice measurements. Ignore tide gauges. My guess as a physicist (and yes I know that we assume cows are spherical and isotropic :-)) would be that you could make do with a fairly small dataset, e.g.

    – GISTEMP
    – satellite measurements of sea level and temperature
    – CO2 measurements from Mauna Loa
    – baseline natural CO2 from ice cores
    – satellite measurements of TSI

    Then, build a really simple climate model. I guess the most complex input will be the detailed line spectra of CO2, etc, you can read this in from MODTRAN or whatever you use. Make the code high quality and freely available. Make all the databases freely available.

    Next, collaborate with 20 or more climate scientists and write a peer-reviewed paper describing the above, together with the necessary physics and any additional assumptions you need (e.g., radioactive heat output of the earth, albedo, average volcanic activity) and show convincingly that it proves AGW.

    Now, this isn’t going to be easy, and may well require additional complexity (clouds, aerosols, etc). But the basic idea is to provide the simplest, most robust, proof of AGW. With the code and data all available, anyone can try running the model, and try to find faults.

    My outline above may not be practical, but I think that something close to it should be. If it could be done, it would short-circuit all the criticism of Yamal, HARRY, etc.

    Comment by Michael Ashley — 29 Nov 2009 @ 6:54 AM

  188. Bill S wrote:

    “For anyone who knows anything about science, the refusal to release data, codes, methods, proxies, all of it, until forced to do so is hard to comprehend.”

    Perhaps you missed the point that the vast majority of data, codes and proxies have been freely available for a long time without anybody being “forced” into it. As for the methods, they are all in the published literature, that being what journals are there for.

    “The excuse is that CRU has confidentiality agreements with the providers of data, but that data sets that are public correlate 98% with the data set used by CRU.

    This brings up two issues. One, if the data sets are substantially the same, why did CRU pay for something that is free elsewhere?”

    Er, they didn’t pay for what is free, they paid for the 2% that’s not free. The reason for doing that even though it makes no difference to the global average temperature is that a) you can’t know whether something will make a difference or not without examining it and b) it doesn’t make a difference globally, but it provides a more complete small-scale / regional picture of temperature trends.

    Are you really suggesting that Phil Jones should have said “eh, well, I know there’s some extra data out there but I’m not going to include it in my work because I’m pretty sure it won’t change much”. Can you imagine the “skeptics” uproar? “AGW alarmists refuse to include new data sources – what are they afraid of?!”

    “Two, why would anyone involved in the search for the unbiased truth sign a confidentiality agreement that would prevent others from having access to the raw data, along with all of the “value added” statistical adjustments?”

    Because if you don’t sign the agreement you don’t get access to the data and you can’t include it, rendering your analysis less complete as a result? I would’ve thought that’d be obvious by simple common sense.

    Comment by cer — 29 Nov 2009 @ 7:14 AM

  189. RE Matthew

    What’s required is that the data, metadata, program source code and program output for each published paper be put together for review by skeptics and allies alike. Then the code can be run on the data, the output confirmed, and any mistakes identified. Then modifications to the programs can be run, by skeptics and allies alike, and the differences caused by the modifications can be clearly identified. That way, assumptions that are made, and decisions that are made, can be clearly shown to determine, or not determine, specific claims made about the results. Put differently, for each published paper there needs to be a complete and accurate data audit trail.

    That is what ClimateAudit has persistenly requested. What RealClimate has supplied instead is a direction such as “We used the data at such and such a website.” That would never work for a patent application, or FDA approval.

    What a way to completely slow down the review process. An important part of peer review is selecting qualified reviewers, and I don’t see this happening with this selection. Allies are probably other climatologists who have enough to do.

    Now it so happens that I have experience with both patent applications and FDA submissions. My patents were based on material that I had published, and except for making sure I had everything in my (company-owned) notebooks in case of challenge, there wasn’t anything I put in the application that wasn’t found in my publications.

    The FDA will accept manuscripts as supporting data. What you are writing about are the clinical studies and GLP animal safety studies. I can tell you that:

    1. Getting set up for GLP studies (having SOPs, validation, QA review) is onerous. There’s a reason research labs, even corporate research labs, are not GLP. Allow for about a year in down time and about $300,000 for each institute to start, and about $150,000-200,000/yr for maintenance. Who will pay for this? Allow for 6-month to 1-year delays in the publication of each study.

    2. All auditing is internal by employees or contractors/consultants of the company. For submitted data, the FDA can go through the books and do their own checking. What’s the climactic version of the FDA?

    3. The raw data for these studies is not released to the public (or rival companies), except under terms of any journal that publishes a paper. Publication is not a precondition for FDA review; in fact tables in an Investigator’s Brochure are fine with the FDA. This may be why many clinical studies are reported only in scientific meeting abstracts.

    Science has a method for validation of results – independent confirmation. Scientists are a tough audience. For example: the temperature reconstructions by MBH led to other reconstructions. More data gathering and experience from almost ten years of publications, meetings, comments led to Mann, et al. 2008. I will admit that I believe the NAS findings helped Mann, et al. Not sure that these points would not have been raised without the criticisms from the outside or not. Of course, lab animal care has improved over the last 30 years I’ve been involved in research; that doesn’t mean that all lab animal protocols and data should be reviewed by PeTA first.

    Each finding is a piece of the puzzle. Nothing is entirely perfect, but the picture becomes clearer with each new study. Like the fossil record, scientists study what nature can show them to get the best idea of what has happened and is happening. We should at least admit that they know their field better than we do and that they know what they are doing.

    Comment by Deech56 — 29 Nov 2009 @ 7:27 AM

  190. Matthew says “What’s required is that the data, metadata, program source code and program output for each published paper be put together for review by skeptics and allies alike.”

    Uh, no. This is not required except by tin-foil-hat-and-black-helicopter conspiracy theorists. What is required by scientists is validation of the conclusions and methodology of the work. If you want to come up with another endeavor, fine, but don’t pretend it’s science. Last I saw, science was working just fine.

    Comment by Ray Ladbury — 29 Nov 2009 @ 8:24 AM

  191. DonQ, as a start, go to David Archer’s post, listen to his lectures, read his book. That will perhaps put you in a much better position to deal with the GCM code.

    (Why yes, I am standing on one foot. But I’m a bird and this is not difficult.)

    Comment by The Raven — 29 Nov 2009 @ 8:47 AM

  192. As much as it pains me to say it, as I have great respect for Forrest Mims work popularizing electronics, has anyone pointed out that Mims is a Fellow at the Discovery Institute? This hardly bespeaks any strong understanding of scientific method, and given the anti-science activities of said Institute, I think we are safe in thanking Gavin for sparing us the screed.

    Comment by Ray Ladbury — 29 Nov 2009 @ 8:47 AM

  193. Composite continuous 200 to 250-year resolution sea-level record for the last 500,000 years on EDC3 chronology (with the various component parts from which the composite was formed), for use alongside ice-core CO2 and temperature records (Rohling, E.J., Grant, K., Bolshaw, M., Roberts, A.P., Siddall, M., Hemleben, Ch., and Kucera, M., Antarctic temperature and global sea level closely coupled over the past five glacial cycles, Nature Geoscience, 2, 500-504, 2009). Composite record is shown in columns AR (EDC3 age) and AS (Relative Sea Level). Link: http://www.soes.soton.ac.uk/staff/ejr/Rohling-papers/Rohling%20et%20al%20Nat%20Geosc_%20data%20supplement.xls

    Comment by eelco rohling — 29 Nov 2009 @ 9:28 AM

  194. If you want scientists to review and accept your work, then there is no need to make your data understandable to the general public. They are welcome to take the risk of citing your work if they believe in it. They assume the risk of later being proven wrong.
    If you want the general public to understand your work and alter their behaviour accordingly, you had better be ready to educate them on how your science works. A fatherly pat on the head and a “trust me” will not suffice.
    This would require a complete open source model which can be operated by the public, with all raw data open sourced.
    Welcome to the 21st century.

    Comment by David Wright — 29 Nov 2009 @ 9:28 AM

  195. There is a website where the owner purports to have been analyzing the code used to convert GHCN data to GIStemp. He believes he has found multiple methodology errors, such as adding “false accuracy” of multiple decimal places to arithmetic means of integer data, and errors due to the use of distant reference stations that do not experience the same climate to lengthen the temperature series of current stations.

    He’s not a climatologist, but is obviously a very experienced programmer. He also states in at least one comment that he has attempted to post comments here at RC, only to have them round-filed.

    It looks like he’s been at this since at least Apr 09. I haven’t read and digested everything, but so far, his assertions appear plausible: that on the way to becoming GIStemp, the temperature data has been bent, folded and mutilated. The website is http://chiefio.wordpress.com/gistemp/

    [Response: Any coding errors found should of course be notified to the GISTEMP people - they will be happy to fix these things immediately and post a brief description of the impact it has on the results (as they have many times before). However, a brief perusal of this person's site shows that he is pretty confused about where the data comes from, the motivations of choices made in the analysis and has a woeful lack of understanding of what is going on in a calculation of the average temperature anomaly. His insistence that GISTEMP is dropping high altitude sites to increase the mean temperature of the stations is just nonsense. The mean anomaly is calculated by averaging anomalies across stations and so only if there was a systematically smaller anomaly in high altitude regions would the stations shift through time have this affect. In fact though, the evidence is that anomalies are of larger magnitude at altitude, and so it would have the opposite effect to that claimed. Combined with the fact that GISTEMP doesn't control what gets into the GHCN data in the first place, those kinds of criticisms have very little merit. However, I stress that any actual coding errors should of course be reported. - gavin]

    Comment by Paul Lindsey — 29 Nov 2009 @ 10:24 AM

  196. Of course the vast majority of armchair “skeptics” have no intention of doing any work. If they had, this post would have been unnecessary, as instead of crowing “where is the data”, they would have spent the 10 minutes on google that are required to have found it themselves.

    Perhaps it’s time for the RC folks to post a short piece devoted to soliciting informal research proposals from the skeptics here who have been demanding more access to data. These folks who have been loudly demanding full access to the data must be “chomping at the bit” to get started on their own independent research projects. As the ever-growing “Data Sources” page attests, there is an incredible variety of freely available raw and processed climatic data available to them. If what’s already out there isn’t sufficient, then the skeptics could detail what additional data their research projects would require and perhaps others here could help them get their hands on that data (all the while respecting IP guidelines, of course.)

    A proposal could be along the lines of “I don’t feel that UHI has been addressed adequately, so I am going to perform my own comparison of urban and rural temperature data to demonstrate that”, along with some followup details about temperature station selection, etc. Nothing terribly detailed and formal, mind you, just some quick “out there” ideas that skeptics feel haven’t been looked out closely enough.

    And after the RC folks here have collected the skeptics’ research proposals, they might want to set up a page here devoted to publishing the skeptics’ results.

    Comment by caerbannog — 29 Nov 2009 @ 10:26 AM

  197. I finally got curious enough to download the GISS Model E from the NASA website.

    In my opinion, people who say this is inadequately documented are simply stupid. The gzip has a how-to document in HTML that walks you through running the model. The code is FORTRAN. About every 4th line is a comment describing what the program is doing. The variable names even (largely) make sense. What the @#$* else could you possibly want for a one-off piece of research software written in FORTRAN?

    I haven’t written a large FORTRAN program in close to 20 years (I program in SAS). But this is perfectly readable code, even for somebody as out-of-practice as I am.

    If I had a month or two, I could probably understand it in a fair level of detail. But if you think you’re going to be able to download it and find an error in a couple of hours, you don’t have your head on straight.

    Comment by Christopher Hogan — 29 Nov 2009 @ 10:36 AM

  198. Perhaps it’s time for the RC folks to post a short piece devoted to soliciting informal research proposals from the skeptics here who have been demanding more access to data.

    Somewhere in one of these overburdened threads Gavin mentioned a “citizen science” project that would be “truly useful”.

    Perhaps a list of “useful citizen science” projects that could be undertaken by people with a genuine interest might be interesting. As you say, “there is an incredible variety of freely available raw and processed climatic data available” … it could be a list of “useful armchair citizen science” projects, no field work required :)

    I guess the difference between what you’re proposing and what I’m saying is that I’m suggesting ideas flow from researchers to citizens, rather than skeptics to researchers. I fear the latter case would result in a short list mostly boiling down to “I want to prove climate science is a fraud”.

    Comment by dhogaza — 29 Nov 2009 @ 10:46 AM

  199. I’m not sure if it was mentioned allready, but rimfrost.no got a great collection of surface temperature.

    Comment by Hans Olav Hygen — 29 Nov 2009 @ 11:00 AM

  200. “And after the RC folks here have collected the skeptics’ research proposals, they might want to set up a page here devoted to publishing the skeptics’ results.”

    A forum for crank research? Yeah, there’s a crying need on the web for that.

    Comment by Jeffrey Davis — 29 Nov 2009 @ 11:18 AM

  201. I am glad to read of the ccc-gistemp project, Alan Barnes above.

    For an example of what is required for adequate disclosure, check out the paper by Mann et al in the current Science magazine. The supporting on line material comes to 22 MB, data and source code. It can be downloaded by any member of AAAS. For example, I downloaded it. Yes, this slows down the review process, but it makes the result more reliable, and it is required as a condition of publication by an increasing number of journals.

    Comment by Matthew — 29 Nov 2009 @ 11:18 AM

  202. Hey, Dr. Schmidt,

    You may want to add in the SkyRad data sets that were available at arm.gov…

    Cheers!
    Dave Cooke

    Comment by L. David Cooke — 29 Nov 2009 @ 11:22 AM

  203. deech56: Now it so happens that I have experience with both patent applications and FDA submissions. My patents were based on material that I had published, and except for making sure I had everything in my (company-owned) notebooks in case of challenge, there wasn’t anything I put in the application that wasn’t found in my publications.

    When I wrote reports in support of intellectual property, I included the source code. When I published papers, I created directories that had the source code, complete unedited output, and the data used (with a reference to the data that they had been selected from), and wrote the directory to a backup CD. The programs included references to the sources of the equations if I hadn’t thought of them myself. That was in case it was requested. Even you had supporting material in case of challenge.

    As you note, there is no equivalent of the FDA in climate science. That’s why full public disclosure, exemplified by the Mann et al paper in the current Science, is necessary. It is the process of intense scrutiny of everything by skeptics that gives science its strength. This is one reason why full disclosure is increasingly required as a condition of receiving federal grant money.

    Comment by Matthew — 29 Nov 2009 @ 11:40 AM

  204. CER wrote:
    > they didn’t pay for what is free, they paid
    > for the 2% that’s not free.

    CER, you misunderstood: some countries sell their meteorological data to businesses; the same data is shared with researchers who agree not to give it away — so they don’t undercut the market for selling it to businesses.

    Businesses buy such country weather records to do business in that country; knowing growing season, rainy season, things like that affect business plans.

    > if they are that good at the science
    > they will be in jobs doing the science.

    Or really, really wish they were: http://www.xkcd.com/664/
    I watched a really clever Excel expert fall out of her chair laughing upon seeing that particular one.

    Comment by Hank Roberts — 29 Nov 2009 @ 11:49 AM


  205. A forum for crank research? Yeah, there’s a crying need on the web for that.

    Actually, I don’t expect to see *anything* like that from the skeptics here. Even “crank” research would be too much work for those guys.

    However, a link to a blank page listing all research proposals/results from the folks who have been demanding “more data” might be a useful rhetorical tool. ;)

    Comment by caerbannog — 29 Nov 2009 @ 11:56 AM

  206. “All we need is the ability to do

    c:>runAnalysis.exe input.dat output.dat”

    Well, no. That says nothing except that the code, given A, produces B. Who knows what that means? At a certain point, and it’s reached early on in the scientific process, a degree of literacy in the field, is needed. The code is a abstraction, a facsimile, of the process described in the study. It isn’t the process.

    I wouldn’t presume to any interest in the code and data resources being assembled here since I wouldn’t know if the data were apt for the programs or if the processes studied were apt for the science.

    There’s a huge element of bunkum in the request for data and code. If you were a scientist, you’d know where the data were and you would know how to code to test the hypothesis in the study. You’d know roughly if the hypothesis described was worth the effort to study and if the experiment was properly designed to test its hypothesis. The code is more akin to an illustration than substance.

    It’s a lot like a dog chasing a school bus except the dog knows he doesn’t know how to drive.

    Comment by Jeffrey Davis — 29 Nov 2009 @ 11:57 AM

  207. Ray Ladbury says: 29 November 2009 at 8:47 AM
    > Forrest Mims … Gavin … sparing us the screed

    NOT

    Ray, you were confused on two points there.

    Forrest Mims posted in the thread early on; the full text is there.
    Gavin did nothing to what Forrest Mims posted. It’s all there.

    “ccpo” quoted _part_ and said he/she had “deleted” [WRONG WORD] ‘a screed’

    Comment by Hank Roberts — 29 Nov 2009 @ 12:11 PM

  208. (testing the spamfilter)

    —> omitted, elided, left out <— [RIGHT WORD for omitting something]

    Comment by Hank Roberts — 29 Nov 2009 @ 12:14 PM

  209. Im getting really tired of these deniers. Its as if a global conspiracy engulfed the world. They are simply conspiracy theorist!

    And they also hide themselves. If you truly believe in this conspiracy why dont you show yourselves by name? I wish there was a badge these denialists were forced to wear on their breast!! Like a bright yellow fluorescent D, for denialist, or something.

    Comment by Henrik Nerbo — 29 Nov 2009 @ 12:24 PM

  210. If I pay taxes, then my meteorological services at all levels should provide me with the data for free as a citizen.

    Comment by The Bert Man — 29 Nov 2009 @ 12:46 PM

  211. Hank, BPL, Keven,

    Thanks for the explanations you’ve given – so basically, the ‘missing’ ocean heat was never really missing and only thought to be missing due to bad data that’s since been corrected. Is that right?

    Mark A. York, since you inquired as to why I asked you the question, it was to see if you knew as much as you presented yourself as knowing. Since you felt the need to smear the editor of the article to cover for not being aware of the ‘missing heat’ a puzzle which, as you note, made the headlines, you might want to go back and see the scientists he quoted and the substantial portion of the article that consisted of direct quotes from those scientists. At the time the article was written, it was genuinely a puzzle. It was good reporting by the NPR.

    Comment by David — 29 Nov 2009 @ 1:06 PM

  212. David (others), GISTEMP code doesn’t do much, regardless of its faulty structure and implementation. (Harry, or whomever, might very well go crazy trying to fix this code.)
    Out of all the people here who have downloaded I doubt there is even one who has managed to run this software to its completion, I doubt that even Gavin is using it for anything. Then, after it has been run, I don’t see how to verify what the output is.

    It is great that someone is working (from scratch) to make things better, presumably rewriting the whole thing.

    @ Christopher Hogan,
    in GISTEMP_sources there isn’t a single html file and the code is not well documented. I’m not sure what you are looking at, can you provide a link?

    Comment by donQ — 29 Nov 2009 @ 1:11 PM

  213. If I pay taxes, my intelligence agencies at all levels should provide me with the data as a citizen.

    Comment by Petro — 29 Nov 2009 @ 1:31 PM

  214. “Since you felt the need to smear the editor of the article”

    David what’s chip on your shoulder about? I have no idea what you are talking about. I know what I know from study. It’s not all-encompassing though.

    Comment by Mark A. York — 29 Nov 2009 @ 1:33 PM

  215. Gavin: We discussed this above. The replication that is required in an observational science like climate is the replication of the conclusion.

    I am baffled by the claims that making code available is neither standard practice nor desirable. In the social and life sciences, anyways, it is standard practice at many journals to require that code be made publicly available, and hopefully this standard will become universal in the next few years. There are several high-profile examples of coding errors that have been caught only because authors made code available. Making code available also makes it completely transparent what steps were taken to clean or otherwise process data, and similarly clarifies the statistical analysis.

    It isn’t that either “replication of the conclusion” by recreating a study from scratch is important or verification is important, it’s rather that both are important. As far as I am aware, that is not a controversial opinion in any quantitative discipline.

    [Response: I never said it wasn't desirable. The maximum openness possible is certainly a goal worth striving for. However, you may be mistaken in thinking that a) it will change many of the skeptics mind or b) reduce the number of attacks on climate scientists. Nor will it make much difference (if any) to the robustness of conclusions. It might reduce the spin-up time for new people into the field, but access to code is not a substitute for knowing what to do with it. You will still learn much more by doing something yourself than in starting off with someone else's code. Much to many people's disappointment, global warming is not caused by an arithmetical error. Just ask the glaciers. - gavin]

    Comment by Chris Auld — 29 Nov 2009 @ 1:48 PM

  216. Petro says: 29 November 2009 at 1:31 PM
    > If I pay taxes, my intelligence agencies at all levels
    > should provide me with the data as a citizen.

    Chuckle. Let us know, when you ask them to deliver it, how that works out.

    Comment by Hank Roberts — 29 Nov 2009 @ 1:50 PM

  217. David says: 29 November 2009 at 1:06 PM

    > the ‘missing’ ocean heat was … due to bad data that’s
    > since been corrected. Is that right?

    > Mark York … I asked … to see if you knew as much …
    > the need to smear the editor of the article

    Follow the science. There’s no single point at which it’s “right.”
    Posting faux-naive questions with outdated information to try to test your fellow reader wastes everyone’s time who bothers to try to be helpful here.

    You’re using a common name as a userid — whoever you are, people won’t know if the next “David” along who asks a naive question is from you again, playing games, or from someone with an honest question.

    I think I need more coffee. I’m grumpy this morning.

    Comment by Hank Roberts — 29 Nov 2009 @ 2:03 PM

  218. The Bert Man
    29 November 2009 at 12:46 PM

    I pay all my taxes. An extract of the population register is not free. Requesting MY OWN name & address costs money!

    I do not see any harm in letting the people who use the data pay for it. After all, most is requested by companies that make money of the data. Why should government hand it out for free so others can make money of tax-payer funded data?

    Damned if you do, damned if you don’t.

    Comment by Anne van der Bom — 29 Nov 2009 @ 2:04 PM

  219. Chris Auld (#216) says: “I am baffled by the claims that making code available is neither standard practice nor desirable. In the social and life sciences, anyways, it is standard practice at many journals to require that code be made publicly available, and hopefully this standard will become universal in the next few years.”

    In the areas of physics that I have worked in, it is quite unusual to make the code publicly available. In fact, I have published papers where my corporate employer would never allow me to make the code publicly available (nor even give the code to a specific person who asked for it). Certainly one effect of a journal requiring one to make code publicly available would be a severe drop-off in submissions from people who work in corporate R&D labs, which may not be too relevant for climate science where a fairly small fraction of the work is done in not done in the corporate world but would be for other fields. There would likely be a huge drop-off in submissions for journals like Applied Physics Letters.

    Comment by Joel Shore — 29 Nov 2009 @ 2:15 PM

  220. In a lecture (sept 2009) at the University of Strasbourg Vincent Courtillot (director of the Geophysics Institute of Paris), mentions that he has asked CRU for several years for the data underlying the error estimate of the air temperatures above the Ocean in the period 1850 – today. He claims that this data was never given to him, despite repeated requests. Can someone point out where on the Internet he can find this data?

    The lecture (in french) can be watched on dailymotion.com . The particular claim is between minute 13 and 15 in the first part (of six).

    [Response: It would almost certainly come from the ICOADS data (which is the collection of all the temperature measurements by ships and buoys). This is processed by the Hadley Centre to produce the HadISST gridded product, and their papers would be the ones to read on how the uncertainties are estimated. In neither case has this anything to do with CRU. - gavin]

    Comment by Frank Tuijnman — 29 Nov 2009 @ 2:27 PM

  221. “214.If I pay taxes, my intelligence agencies at all levels should provide me with the data as a citizen.”

    Are met workers in danger of having their covers blown, losing their lives over a disclosure?

    Comment by David Wright — 29 Nov 2009 @ 2:48 PM

  222. Hank Roberts wrote:

    CER, you misunderstood: some countries sell their meteorological data to businesses; the same data is shared with researchers who agree not to give it away — so they don’t undercut the market for selling it to businesses.

    Sorry, I should’ve been more specific – I wasn’t sure if the extra data CRU used was only from met offices or also from private/corporate sources. If CRU got it for free then I guess it was the former. Certainly when we did a regional study on Siberia we obtained a dataset from some Russian source, which we definitely had to pay for (but then it was partly for use with an industry project so I guess it didn’t qualify as purely academic purposes).

    Chris Auld wrote:

    I am baffled by the claims that making code available is neither standard practice nor desirable.

    Well I can’t claim to be expert on every field of science, but it’s definitely not just climate science where making code available isn’t standard practice. I used to work in a particle physics lab and certainly there was no “open-source code” flying around. It worked exactly the same was as climate science – if you wanted to replicate someone else’s work, you read the relevant publications, understood their methodology and then wrote your own code to do the same thing. In fact my first supervisor wouldn’t let me have a copy of his own code at first, because he wanted me to check that his method was working correctly, so he had me write my own *completely independent* code to do the same calculation, because that’s the only way of really being sure the results are sound. No amount of “verification” or bug-fixing can provide the level of validation that a new, separate analysis can.

    Comment by cer — 29 Nov 2009 @ 3:11 PM

  223. RE: post #215: Gavin, I agree with you that making code available is unlikely to reduce baseless attacks, substitute for substantive knowledge, or overthrow results that have been arrived at from multiple research strategies.

    But I disagree that making code available has little or no effect on the robustness of particular results—as I noted, there are famous examples of specific results in the primary literature which were artifacts of coding errors. Making code available implies complete transparency in methods, allows others to catch coding errors, and perhaps reduces the frequency of such errors in the first place.

    For these reasons many journals and societies in the observational and other sciences either require or encourage authors to make code available. I think you underestimate how important it is to make code available in general, not just in the context of climate research.

    Comment by Chris Auld — 29 Nov 2009 @ 3:13 PM

  224. @donQ#212: Out of all the people here who have downloaded I doubt there is even one who has managed to run this software to its completion…

    I have just today run STEPS 0, 1, 2, 3, and 5 on a Ubuntu 9.10 x86 laptop.

    Step 4 is an optional step where surface data is combined with sea surface data. But the sea surface data is BIGENDIAN, and requires special compiler flags to run on linux/x86 computers. I hope to crack that nut soon.

    After generating my own output, I have compared it with the official output of the file located here:
    http://data.giss.nasa.gov/gistemp/tabledata/GLB.Ts.txt

    There are some differences in the last significant digit. I will explore these as time is available. This is a hobby. I suspect that some of the differences have to do with variations in precision in various compilers on different processors.

    DonQ don’t generalize your own lack of skills. I am nowhere near the first person to accomplish this. There are more than a few similar successes reported on various blogs. I am working on creating a tarball and instructions so that anyone capable of running a Linux workstation can duplicate this “feat.”

    Comment by Ron Broberg — 29 Nov 2009 @ 3:22 PM

  225. Benjamin (176) — Huh? I’m not even sure what “sockpuppetry” is…

    Comment by David B. Benson — 29 Nov 2009 @ 3:28 PM

  226. ** I’m in the top ten percent of wage earners (in my country -USA), and we pay more taxes than the lower 90% combined, and I demand that access to the data, and lawmakers, and government regulators, and university researchers, and other policy makers be apportioned according to how much we pay in to the system. Now that I think about it, I already get that. **
    ** = air quotes. I just made most of that up, and explicitly included “air quotes” and this explanation for the potential parody challenged readers. I did make a 5 figure donation to fund an ongoing fellowship at a local University, and I get a Christmas card from the dean each year; he either answers or return my occasional phone call. I once used a free “student version” of commercial modeling software, and wanted to include some materials properties that weren’t in the free version’s data file. I wrote the company asking if the material properties were available to me, and politely explained what I was using it for – basically farting around for my own edification and enjoyment – and that I wasn’t interested in and didn’t need their full, commercially valuable, material database. I received a polite reply that the only data that could be supplied to users with a student license was included with the free download. They also noted that the files were updated regularly, and perhaps I should download the latest free student version of the material data file, and gave me the link to do so. When I downloaded the (3 day) newer version of the material property file, I found, **no doubt entirely by coincidence**, that the material I was interested in had been added to the latest data. I wouldn’t want to imply that Dr. Jones did give access to data that was restricted under commercial license to people outside the CRU, but it wouldn’t surprise me if some emails that were selectively omitted from the hackfile showed that the **”hidden, secret, now destroyed, smoking gun that disproves global warming”** data were shared with independent scientists who didn’t start off with accusations of fraud, conspiracy, hippy liberal eco-nazi soc – ialist power grabs, and that Al gore is fat.
    Google search results
    “Results 1 – 10 of about 313 from climateaudit.org for fraud. ”
    “Results 1 – 10 of about 361 from climateaudit.org for conspiracy”
    “Results 1 – 10 of about 63 from climateaudit.org for gore fat. ”
    “Your search – hippy liberal eco-nazi site:climateaudit.org – did not match any documents.” (do I have to put ** around EVERY exaggeration?)

    In the interest of full disclosure, from WUWT discussion on google hits –
    “However, the point of this blog entry is still with merit and significance. Google has historically made pro-AGW material much more accessible via their system than material which challenges the IPCC, AGW supporters, and Gore.”
    “I think it’s crystal clear that having climategate ”go missing” in the Google suggestions list was very definitely and deliberately ”thought about”, from the partisan AGW agenda perspective.
    As another WUWT commenter astutely pointed out on another thread:
    Al Gore sits (or at least used to sit) on the Google BOD. ‘nough said.”

    Comment by Brian Dodge — 29 Nov 2009 @ 4:00 PM

  227. Are met workers in danger of having their covers blown, losing their lives over a disclosure?

    Professor Jones was given police protection due to threats received after the hacked e-mails were released.

    Comment by dhogaza — 29 Nov 2009 @ 4:11 PM

  228. With this avalanche of climate data (most of which was already available on the web), I am going to hold my breath while waiting for the first sceptical re-analysis which conclusively demonstrates how it was manipulated and the results fabricated.

    Is this a good idea, do you think?

    Comment by Mike of Oz — 29 Nov 2009 @ 4:13 PM

  229. My state in the USA used to complete information requests free of charge. But too many crackpot groups abused the system. So now requestors must also pay expenses. Requests for information have gone way down.

    Comment by Pete W — 29 Nov 2009 @ 4:47 PM

  230. Gavin: Much to many people’s disappointment, global warming is not caused by an arithmetical error. Just ask the glaciers.

    Yes, but the forecasts for the future may be based on some sort of programming error.

    However, you may be mistaken in thinking that a) it will change many of the skeptics mind or b) reduce the number of attacks on climate scientists.

    I think that you underestimate the effect that openness will have among the skeptics, though you are surly right if you mean that the “deniers” will be unaffected. It seems (your information may be different) that the ranks of the skeptics have grown in the last few years, and especially in the last few days. A major effort to win back the skeptics might be worthwhile.

    Nor will it make much difference (if any) to the robustness of conclusions.

    That presumes that there are no mistakes or irregularities. Only a completely open review can establish that, I think, in light of the hacks.

    Thorough open scrutiny of everything by skeptics isn’t fun for the participants, but it is one of the principal strengths of scientific research.

    Comment by Matthew — 29 Nov 2009 @ 5:15 PM

  231. Thanks for putting up all the links to the info, code & data
    - it’s a useful repository

    From what I’ve seen of the Giss Model E, it seems well documented, and the code well written & commented
    - so I’m not sure what the complaint above was about..

    Perhaps you could encourage CRU to adopt a similar coding & documentation process, and then publish some of their work in the web so that no one would be in doubt about which bits of code were run for which results etc!

    As an aside, I would have thought an OO language such as C++ would be great for climate/weather models, as you could describe the atmosphere boxes as objects, each with their own state variables, and functions to process the data for the next timeslice.
    - I guess converting 100K lines of Fortan to another language is a no-go though – but it might not be as hard as you think!

    Comment by Phil M — 29 Nov 2009 @ 5:17 PM

  232. Matthew (230) — All whatever the number, 23?, GCMs all give very close to the same answer. Amazing they all contain exactly the same programming error, is it not?

    In fact, using just a single equation and assuming BAU, I can do quite a good job of predicting future temperatures; I’ve checked this against historical data. Amazing, is it not?

    I leave out feedbacks thought to be accelerating in the future; I doubt these are in the GCMs yet either. But none of that is strictly necessary as one can use paleodata with profit here; Mark Lynas has done so in his research leading to his book, “Six Degrees”. Here is a link to a review:
    http://www.timesonline.co.uk/tol/news/uk/science/article1480669.ece

    Comment by David B. Benson — 29 Nov 2009 @ 5:46 PM

  233. In the late 80′s and early 90′s I did ad hoc data analysis and developed a major new database we needed. My comments were minimum in the ad hoc analysis, because they were modified versions of other programs I wrote for a different, one-time data request. The reports I generated periodically did have more comments so that I could eventually hand it off for someone else to run. The edits, displays, etc. for the database I wrote were well documented, because these programs were run by users at the entities required to report the data. The code was available to the users of the system so they knew exactly what we were checking. I also wrote a three inch Data Dictionary (with error code definitions). The database took about three years before it went into production. Most of the ad hoc reports took from 5 minutes to an hour for the more difficult ones, though some complex questions could take several hours from start to printer.

    To have heavily commented my ad hoc programs and data tools would have been a waste of time and my efficiency would have been dismal. The edits, displays, and other user tools for the database used script files in the form of \runAnalysis entity abbreviation, with a predetermined input file name format so that the documentation could be easily followed even by a computer illiterate PhD. These added layers for a simple user interface, user documentation, comments and meetings increased the development time by at least of a factor of 10 compared to bareback programs.

    These are scientist trying to analyze data in a timely manner. To require a c:>runAnalysis.exe input.dat output.dat” end product would be a complete waste of time needed for other projects. Having said this, I almost cried when I read the HARRY_READ_ME.txt. I once had to comment and update an old program (which contained no comments) written by my boss’s boss – he was trying to hand it off to me…. It was a nightmare similar to the HARRY_READ.ME file. Some commenting is necessary when others will need to access to the code.

    I really do feel for the people whose email was made public, it’s just wrong. I’ve read a very small percent of the email and only have one problem with what I’ve read so far: these scientist are suppose to be concerned with AGW, yet they seem to spend half their time jetting around and around the world then write about it almost as if there was a competition (i.e. whose frequent flyer account is bigger). Does CRU or the other organizations have any type of offsetting program to balance the CO2 produced by these frequent intercontinental flights?

    Comment by Jason — 29 Nov 2009 @ 6:02 PM

  234. Tom Franklin (93), but if they had actually done that Madoff would now probably be a free man…

    Comment by Rod B — 29 Nov 2009 @ 6:04 PM

  235. For this to be an intellectually honest exercise, you would want to list all the major data/models that went into IPCC. Then, provide links to the data/models that are available and explanations/requests for the ones that are not.

    Comment by David Kane — 29 Nov 2009 @ 6:24 PM

  236. There are some severe misapprehensions about both software quality and about scientific method. In both cases, commenters seem to think that there is ONE approach.

    Software quality can be achieved in a few ways, only one of which is to try to make a single piece of software as perfect as possible. Its a nice, simple approach that you can see in the comments, more or less “lets have everything and we will run it and find some bugs”, but in fact the bugs are just as likely to be in the choices of smoothing techniques, in the assumptions about errors or somewhere in the basic algorithms and software review will never find those. Indeed, review of code, even testing of it, has a characteristic defect discovery efficiency that is never 100%, so even the best code review and testing does not eliminate all bugs. My favorite program, one that is run hundreds of thousands of times all over the world every day, is two lines long and has had three defects raised against it in 45 years. Was that a failure of code review? Version one of the program had ONE line of code (its IEFBR14 for the dinosaurs out there who will recognise the name).

    An alternative approach, one that is used in the software industry where extremely high reliability is required in very complex environments, is also the fundamental approach in published science, is what is called a voting strategy, where independently developed systems evaluate the same data and compare results. The winning answer is the one that is reproduced independently by multiple systems. Sounds awfully like what happens in scientific journals and conferences. The choices people make about what they try to reproduce are driven by factors other than formal methods, they are driven by interest, by intuition and by ego (there are lots more).

    The constant whining about conforming to THE scientific method is begging the question, assuming a weirdly naive Popperianism, when science is much messier than turn-the-crank fantasies about scientists collecting data in some completely wide-eyed-innocent way, evaluating it with no preconceptions, and being surprised by what pops out. We might be able to write genetic algorithms that work that way on complex data sets (I actually think that that is likely to be a fruitful approach in some areas of computing like system management) but science is not like that. Its just too wildly inefficient for people to work that way, people have to make choices about what to focus on, and the successful ones do not choose to focus on the low-level stuff of commenting their code or making it easy for idiots to run, they are trying to get a result that they can feed into the science process: inventing, competing, reviewing, and re-creating others results.

    Comment by grumpy software architect — 29 Nov 2009 @ 6:28 PM

  237. If I may make a suggestion, any chance for some coverage of that climate paper recently published in Science? As necessary as it was and as much as I appreciate that RC has turned into the primary place to go for the discussion of the CRU hack, it might be fun to get the inside scoop on some real data and give Gavin a well-earned break.

    Comment by Deech56 — 29 Nov 2009 @ 6:29 PM

  238. @Ron Broberg:224
    Ron,
    you seem confused about what data and code means.
    You claim that you must use special compiler flags to deal with big-endian data .. however Big-endian data can be used without problems on little-enidan machines regardless of compiler flags — and compiler flags are used to change how the code is interpreted/compiled. Your message is so confused, while attempting to use basic computer-science terms, that I’m inclined to think you have not done anything at all and are just trolling.

    (btw, within this thread you are, so far, the first one to claim to have ran this software with some level of success … congratulations. You just have to make it a bit more believable.)

    Comment by donQ — 29 Nov 2009 @ 6:52 PM

  239. Quick question guys.

    This article has raised concerns for me http://www.timesonline.co.uk/tol/news/environment/article6936328.ece, just wondering what the stance is on this, specifically “We do not hold the original raw data but only the value-added (quality controlled and homogenised) data.”

    Is this true?

    Comment by Ben — 29 Nov 2009 @ 6:53 PM

  240. Here is a link for the ETCCDI Climate Extreme Indices dataset.

    This is particularly relevant to the matter of sharing local station data from many different countries which have different regulations and expectations for the sharing of their data. See especially the following paper (currently open access):

    Thomas C. Peterson and Michael J. Manton (2008) Monitoring Changes in Climate Extremes: A Tale of International Collaboration, in BAMS 89(9) Sept 2008, pp 1266–1271, doi:10.1175/2008BAMS2501.1

    In this case, rather than store complete station records, which would have been a problem for many countries, data is given as major climate indices. The description says this is a global land-based climate extremes dataset produced through the coordination of the ETCCDMI. It comprises of 27 indices of temperature and precipitation computed from daily station data using the RClimDex software. Data is in the form of zip files for each participating country, plus a gridded version and pointers to related sources.

    Comment by Duae Quartunciea — 29 Nov 2009 @ 7:02 PM

  241. Matthew, there are no true skeptics left–merely the ignorant, the wilfully ignorant, the denialists and the wingnuts.

    Dave Benson’s point is important–you have 23 models, all developed independently, which agree to a very large degree. This is strong evidence that the models are properly coded and have more or less the right physics.

    If each team had had access to the others’ code we could not with confidence conclude that the models were independent, and your supposition about a programming error could not be dismissed as easily. To me, I see this as a strong argument for not sharing code or making code public. Independence is an essential criterion for most statistical analyses–and for good reason.

    Comment by Ray Ladbury — 29 Nov 2009 @ 7:07 PM

  242. In reply to: Ben says: 29 November 2009 at 6:53 PM
    > Quick question

    Ben, Google:

    http://www.google.com/search?q=We+do+not+hold+the+original+raw+data+but+only+the+value-added+%28quality+controlled+and+homogenised%29+site%3Arealclimate.org

    Comment by Hank Roberts — 29 Nov 2009 @ 7:16 PM

  243. @donQ#238: I’m inclined to think you have not done anything at all and are just trolling.

    Well, at least we understand each other – I’m also inclined to think that you have not done anything at all and are just trolling. It’s probably pretty obvious to most of us, in fact.

    Or have you in fact contributed something to this thread that I missed?

    Comment by Ron Broberg — 29 Nov 2009 @ 7:36 PM

  244. 237, deech56: If I may make a suggestion, any chance for some coverage of that climate paper recently published in Science?

    It’s is not easy to address in the short comment format of blogs, except for a few points. I commented that it illustrated what ought to have been done all along, namely putting everything together in the Supporting Online Material for independent, public skeptical review. Also, they have decided to give a new name to the Medieval Warm Period, namely “Medieval Climate Anomaly”, without justifying the claim that it was “anomalous”. Also, it appears that the GCM models that they tested were unable to reproduce the difference between the Medieval Warm Period and the Little Ice Age, perhaps calling into question the accuracy of those models.

    Since all (23?) of the GCMs failed to predict the apparent end/reduction of warming since 1999, and since other studies show that they can not reproduce any recent changes, it would seem that all of them are suspect. Of course it is possible that they all contain a common core of flaws.

    241, Ray Ladbury: Matthew, there are no true skeptics left–merely the ignorant, the wilfully ignorant, the denialists and the wingnuts. Well, that’s one opinion. Mine is, as I stated, that the number of true skeptics has increased.

    Comment by Matthew — 29 Nov 2009 @ 8:25 PM

  245. Wai’minnit here:

    A few days ago new poster “donQ” was suggesting Gavin was hiding the code;
    now “donQ” 29 November 2009 at 6:52 PM suggests Ron Broberg 29 November 2009 at 3:22 PM might be a liar or troll for a post about partial success running the code, partly refuting donQ’s suggestion at 212 that nobody’s succeeded.

    Ron, you could cite/point to reports of success; that’d let others check the fact. Otherwise it’s just an argument by assertion on both sides.

    I’ll leave the question about big-endian for the programmers; surely there’s somewhere else to pursue that.

    Useful: http://www.catb.org/~esr/faqs/smart-questions.html
    (Ironic–on climate he doesn’t practice what he preaches–but I appreciate good preaching despite preachers’ personal peccadilloes in practice.)

    Comment by Hank Roberts — 29 Nov 2009 @ 8:33 PM

  246. In #241 Ray Ladbury says there are 23 independently developed climate models. I do not think that is entirely true – I understand many of them are derived from a smaller set of older models, and groups share algorithms and even code. I don’t know how one could quantify the degree of independence. In any case, I disagree with the case for not disclosing code and data, and apparently so does Gavin in his recent comments.

    Comment by Blair Dowden — 29 Nov 2009 @ 9:15 PM

  247. My set of links compiled a few months ago on availability of source code for *all* the GCMs used in IPCC AR4:
    http://www.easterbrook.ca/steve/?p=667

    Comment by Steve — 29 Nov 2009 @ 9:30 PM

  248. Gavin, there’s sea level and ocean heat content data here: http://www.cmar.csiro.au/sealevel/sl_data_cmar.html (from a variety of international sources).

    Thanks for making all this stuff easier to find.

    This maximum openness demand is marvellous, especially as many in the denial camp are free market extremists who believe in closing government down (the same people who are pressuring academia to be more like business, and charge for everything on a commercial basis).

    Although I understand why CRU people may not have felt motivated to supply data to openly hostile denialists, this is a matter of damned if you do, damned if you don’t (remember the huge fuss made over the correction to GISStemp that had no significant effect on the overall trend)? On balance the NASA approach of making everything freely available is better because it helps those genuinely interested in the science.

    What I find particularly amusing about the “hide the decline” issue is that we’ve known for more than 10 years that tree ring data diverges from the late 20th century temperature record. If the denialists had evenly marginally competent scientists on their side, they could have used this openly known fact to sow confusion. So much easier than making up garbage statistical methods.

    Comment by Philip Machanick — 29 Nov 2009 @ 9:59 PM

  249. “[Response: And I would have got away with it too, if it hadn't been for those pesky kids.... ;) - gavin]”

    Thank you Gavin, that made my day :-)

    Comment by PeterW — 29 Nov 2009 @ 10:08 PM

  250. Re: 210
    “If I pay taxes, then my meteorological services at all levels should provide me with the data for free as a citizen.”

    If that’s the way you feel, then be prepared to have your income taxes tripled or quadrupled! Civil Servants are hired for specific jobs to help various elements of the government to function. There are chains of commands for each one (up to Congress, or the White House). These chains do NOT pass through every tax paying citizen, just because they think their personal request should be the top priority of the entire government. You have the right to vote, if you don’t like how government functions, but you do not have the right to dictate how Civil Servants spend their time. That comes from the top down.

    Comment by Ken W — 29 Nov 2009 @ 10:19 PM

  251. Re 241: Ray,
    True, the models are consistent with each other. However, how consistent are they with nature? How many of those models forecast the timing and extent of the recent and ongoing Arctic sea ice decline? How many suggested that (GRACE) might be detecting loss of Antarctic ice in 2009?

    I suggest that the failure to forecast Arctic sea ice decline is symbolic of the model’s inherent inability to deal with initial ice melt as a rather local process, rather than as a larger volumetric process.

    The problem is that even small amounts of ice melt set up albedo and water vapor feedback process that result the abrupt melt of adjacent ice. More worrisome is that local melt can set up mechanical stresses that result in progressive structural collapse.

    I assert that all ice melt processes resulting from global warming will occur much more abruptly than suggested by any of the GCM.

    Each of the disciplines had its own literature, they exchanged ideas. They knew the kinds of approaches that other groups were using for their models, and nobody took a dramatically different approach. The builders of each model were drawing on the peer reviewed literature. None of the 23 models were built by “lone wolf” scientists. Nobody pulled a “Feynman” and made a point of not reading the literature. Thus, similar conceptual flaws are in each model.

    Comment by Aaron Lewis — 29 Nov 2009 @ 11:12 PM

  252. @Christopher Hogan.

    I also downloaded and ‘played’ with http://www.giss.nasa.gov/tools/modelE/ code.
    This code is considerably different and has sufficient (actually quite good) documentation along with it.
    If you compare modelE with http://data.giss.nasa.gov/gistemp/sources/GISTEMP_sources.tar.gz (which a source code that was suggested very early in this thread) you will see/understand the ‘frustration’ and the criticisms.

    I’d rather ‘play’ with modeE …

    Regards.

    Comment by donQ — 29 Nov 2009 @ 11:14 PM

  253. Where IS the data, indeed?

    I’m looking for the evidence to support the claim made by “some prominent climate scientists” in the New York Times, as presented by Andy Revkin:

    “[Some prominent climate scientists] say that the e-mail messages…have damaged the public’s trust in the evidence that humans are dangerously warming the planet”

    What basis is there for claiming that the public’s trust in this evidence has been damaged?

    What basis is there for attributing that alleged damage to the email messages, rather than, for instance, to misleading media on the issue?

    Thanks.

    Comment by paulina — 29 Nov 2009 @ 11:30 PM


  254. “If I pay taxes, then my meteorological services at all levels should provide me with the data for free as a citizen.”

    Then why did I have to pay 25 bucks for my driver’s license renewal? A hundred bucks for my passport?

    Comment by caerbannog — 30 Nov 2009 @ 12:12 AM

  255. Fast, cheap, good. Pick two.

    Click here for fast and cheap.
    http://rhinohide.cx/co2/gistemp/

    I’ll update this later in the week as I beat out installation and configuration documentation.

    Comment by Ron Broberg — 30 Nov 2009 @ 12:25 AM

  256. I want to thank Gavin for supporting this discussion. On the whole, I think that it has been fruitful.

    I especially appreciated the comment by Nick Barnes about the project using Python.

    My PhD was in statistics, and I have experience modeling nonstationary multivariate biological (behavioral and physiological) time series using nonlinear dynamic models in SAS and Fortran-77. These projects, as I’ll call them, are much smaller in scale than the climate modeling.

    Comment by Matthew — 30 Nov 2009 @ 12:25 AM

  257. I’ll leave the question about big-endian for the programmers; surely there’s somewhere else to pursue that.

    Well, I’m a big-endian programmer, but what do you expect, seeing as I’m 55 years old? (sob :( )

    (which end is big is left as an exercise for the reader, QED).

    Comment by dhogaza — 30 Nov 2009 @ 12:32 AM

  258. INTERNATIONAL JOURNAL OF CLIMATOLOGY
    Int. J. Climatol. 25: 1055–1079 (2005)
    Published online in Wiley InterScience
    Data:

    (www.interscience.wiley.com). DOI: 10.1002/joc.1148
    AIR TEMPERATURES AT ARMAGH OBSERVATORY, NORTHERN IRELAND,
    FROM 1796 TO 2002
    C. J. BUTLER,* A. M. GARCI´A SUA´ REZ, A. D. S. COUGHLIN and C. MORRELL
    Armagh Observatory, College Hill, Armagh BT61 9DG, Northern Ireland, UK
    Received 5 May 2004
    Revised 15 November 2004
    Accepted 15 November 2004

    Comment by information = data — 30 Nov 2009 @ 1:12 AM

  259. I am a self uneducated retiree with some scientific background.

    Does anybody on either side of this argument claim to be able to foresee the future? As far as I can see, computer models have the same predictive ability as chicken entrails, albeit using chicken entrails you get to eat a chicken as a side benefit.

    Although it is tempting to believe that the past can be extrapolated into the future, it just ain’t so.

    What usually happens is that people (not just scientists) confuse “assumptions” with “predictions”. Newton’s thoughts on gravity. Einstein’s thoughts on gravity etc., were all deemed to be practically useful at the time. Good enough to “predict” eclipses, demolish the luminiferous ether (believed in by a majority of scientists, I believe), and so on.

    The future is unknowable. Prove that you can foresee even the next 5 seconds repeatedly with one hundred percent accuracy, and I will “predict” that civilisation as we know it, will end. You can easily work out why.

    I won’t hold my breath.

    [Response: Fortune telling is a class-B misdemeanor in NY state, and obviously no-one here is doing that! However, predictions are the lifeblood of science. Given certain, well-defined, conditions, scientific theories will predict what the result of a specific experiment will be. If the conditions are not met (the temperature is different, or it wasn't a vaccuum etc.), the prediction doesn't work. With respect to the future of the climate system, there are some external forces that are quite predictable - the orbit of the Earth, and its variations, the continuing rise in CO2 emissions, etc. Another example would be the climate consequences of a big volcano. In each case, you can use climate models to project the impacts on temperature, circulation etc. In fact, people have done this. In 1991 Hansen et al predicted the temperature drop that was expected from the Pinatubo eruption well in advance of it happening. In 1988, the same team projected how temperatures would increase under 3 scenarios of rising greenhouse gases. For the scenario that came closest to being realised, the trends were a pretty good match to what actually happened. Other groups have made skillful projections for the El Niño events and their consequences months in advance. So finding useful information about the future is not impossible, even if it isn't fortune telling as described in the NYS penal code. - gavin]

    Comment by Mike Flynn — 30 Nov 2009 @ 1:12 AM

  260. Ben,
    “We do not hold the original raw data but only the value-added (quality controlled and homogenised) data.”
    I’m afraid that it is true, and it appears to be even worse that it sounds.
    One would think that they could simply reverse the adjustments, and recalculate the original data. If you look at the annotations of the programmers doing the coding, however, it appears that they have lost track of how the original data has been adjusted.

    [Response: The issue is not any adjustments, but simply that they maintain a master database with all of the data sets merged together. Having done something similar (though smaller scale) with multi-sourced data - some I digitised myself, some done by others, gathered over a 10 year period, including calibration adjustments and typo and meta data corrections, I think I have a pretty good idea why you can't just break it down into constituent parts. As with CRU, all I have to give out is the merged product. - gavin]

    Comment by Cardin Drake — 30 Nov 2009 @ 1:19 AM

  261. Where is the raw HadCRU, GISTEMP, and NOAA NCDC data. Saying that it comes from GHCN doesn’t help. We have to know which stations over what time periods are being used. Reproduction is also going to require a site history for the surface stations. For example, if a site is moved, or if there is a change of time for data sampling. There seems to be a shortage of this kind of information – especially for HadCRU. Then we need all of the adjustment algorithms. This is probably more important than the code. Given the algorithms, we can quickly write our own code. Then, having the raw data, the meta-data, and the algorithims, reproduction should be possible. Excuses like, “we did this and you didn’t” won’t work. If we didn’t do what the sources did, it’s because the sources didn’t supply the information about what was done. Then if reproduction isn’t possible, there is a problem. I hope that this is not just one more example of Real Climate trying to appear to be transparent while at the same time hiding the important stuff.

    [Response: Oh, I don't know. Perhaps you could start by reading the papers? Or looking at the GISTEMP code? You keep saying stuff is being hidden, but when the whole thing is in front of you, you don't look. Someone else in the comments has already downloaded the GISTEMP code and got it to work in the time that this post has been up, while you have been complaining for months that it's all some secret. Who do you think has more credibility? - gavin]

    Comment by Tilo Reber — 30 Nov 2009 @ 1:57 AM

  262. Gavin, I hate to ask you to add one more thing to your already sizeable list, but since you’ve done an outstanding job dealing with the craziness so far I thought I’d suggest it here.

    Could you put do a FAQ of the “ClimateGate” scandal, like you did for the objections to AGW? “I read that the CRU deleted the data.” “No, no data was released. See here and here for more info.” That type of thing.

    If I knew enough about everything….

    Comment by Jody — 30 Nov 2009 @ 2:45 AM

  263. “Anyway, with the links provided, could a skeptic/denier perform an analysis that would disprove man made global warming?”

    The comment above appeared in an earlier post. I suggest that anyone putting asking questions like that should google Betrand Russell’s teapot, or just go here: http://en.wikipedia.org/wiki/Russell%27s_teapot

    Comment by Neal Asher — 30 Nov 2009 @ 3:31 AM

  264. Couldn’t not contribute an aerosol data source ;-)
    AERONET (http://aeronet.gsfc.nasa.gov/)

    Comment by James Allan — 30 Nov 2009 @ 4:56 AM

  265. The Data Sources page looks like an excellent resource. Will it be maintained, or is this a one-off exercise?

    [Response: It's set up as a special page, so it won't go off the front page, and we will aim to keep it up to date. There's been a lot of suggestions of data portals even we didn't know about, so it probably is going to be useful. - gavin]

    Comment by AndyL — 30 Nov 2009 @ 6:01 AM

  266. Mike Flynn:

    As far as I can see, computer models have the same predictive ability as chicken entrails

    BPL: Do a little more research. Start here:

    http://BartonPaulLevenson.com/ModelsReliable.html

    Comment by Barton Paul Levenson — 30 Nov 2009 @ 6:37 AM

  267. There’s plenty of climate model output available here: http://results.cpdn.org.
    It is requested that those downloading the data make a note of what they would like it for but that’s the only requirement.

    Comment by Milo — 30 Nov 2009 @ 7:15 AM

  268. Wow, what a major storm in a teacup. Although I find it disturbing that CRU appears to have destroyed a lot of the raw data, the availability of satellite measurements over the period of the most recent warming mean that there is plenty of data out there that is independent of any possible CRU ‘fiddling’.

    As a layman, I have no way of using the data supplied in the links here. For most people it is more interesting to see graphical representations. I find the following site very useful for comparing the various records as it has them all, including the most recent data, all in the same place. It appears to be mainly factual, although do I detect a very slight sceptical slant? It also has some links detailing some interesting stories from history showing how climate change has affected real people in the past.

    http://www.climate4you.com/GlobalTemperatures.htm

    particularly this chart which overlays the various data sets:

    http://www.climate4you.com/GlobalTemperatures.htm#Comparing%20global%20temperature%20estimates

    As an aside, can historical records be used in any scientific way to inform climate models? For instance, is it possible to know whether the climatic conditions that generated the following fascinating story are in any way indicative of global conditions, or can these kinds of multi-year cooling events be a purely local phenomenon?

    http://www.climate4you.com/ClimateAndHistory%201600-1699.htm#1658: Swedish ice-bourne invasion of Denmark; the Treaty of Roskilde

    [Response: I detect the same thing. There are some rather odd methodological choices, a distinct lack of understanding the concepts of significance and some weird editorialising. So not climate4me! - gavin]

    Comment by Matthew L. — 30 Nov 2009 @ 7:18 AM

  269. Neal, you apparently either don’t understand Russell’s teapot or you don’t understand the basis for AGW (or both). The point of Russell’s analogy is that it’s impossible to prove a negative for a belief that is based on personal credulity rather than evidence. The analogy does not apply to claims based on empirical evidence.

    We know from direct measurements of the absorption spectra that CO2 is in fact a greenhouse gas. We know from direct measurement that it is increasing in concentration in the atmosphere. We know from direct measurements of the isotopic ratios of that CO2 that most of the increase is due to fossil fuel use and land use changes rather than natural sources like volcanoes. We know from measurements that the stratosphere is cooling while the troposphere warms, and that the warming is greatest at night, both observations which are consistent with greenhouse warming and inconsistent with almost any other mechanism.

    Performing an analysis disproving anthropogenic warming would mean showing that either MANY of these measurements are in serious error or that there other mechanisms exist which can explain the evidence better. In either case, the analysis would prove a positive- the existence of errors, or more powerful mechanisms.

    Comment by Mike G — 30 Nov 2009 @ 8:49 AM

  270. Ron at #255

    The README.TXT file indicates successful completion of Steps 0-3.

    Is that correct or does the README.TXT file need to be updated?

    Thanks

    Comment by Dan Hughes — 30 Nov 2009 @ 8:56 AM

  271. @Dan Hughes#270: Is that correct or does the README.TXT file need to be updated?

    The README.txt is correct. Until I can either produce the SBBX.HadR2 file in STEP4 or consume it in STEP5, I won’t claim success for the STEPs 4 and 5. I think some of my problem is just a confusion of what STEP4 does. Looking closer, it appears to update an existing SBBX.HadR2 file with data from oiv2mon files. But if I start with an existing SBBX.HadR2 downloaded from GISS, I bump into the bigendian/littleendian issue. This doesn’t seem insurmountable … but I have a day job. I’ll see what I can do tonight.

    If anyone has constructive suggestions, they can email at ronbroberg in the company of yahoo.com.

    Comment by Ron Broberg — 30 Nov 2009 @ 9:42 AM

  272. The new Data Page is a wonderful idea. One small correction: Under Climate data (processed): Cloud and radiation products (ISCCP, CERES-ERBE), I suggest changing the CERES-ERBE link to the following: http://eosweb.larc.nasa.gov/project/ceres/table_ceres.html

    Comment by Norman Loeb — 30 Nov 2009 @ 10:06 AM

  273. RE: 50 Jimi Bostock

    If you really want skeptics to embrace your movement (theories), you should do better than attack them. Most people I know, when challenged, try to explain their position, backed by FACTS. Your name-calling and arrogance don’t help your cause. Those who attack generally do so from fear. Why don’t you take time, retrench; and present fully your methods, data, and code. Perhaps then the argument will be settled, by removing the emotional controversy. (But, you need to release everything – warts and all.)

    Comment by dale — 30 Nov 2009 @ 10:46 AM

  274. This data sets page is a great resource. There are some here I did not know about. Thanks for putting it together. Do you mind if I link to it?

    A good source of data is NCAR http://www.ncar.ucar.edu/tools/datasets/
    I’m not sure how you would list this in your hierarchy (there is much
    data available, perhaps only some of it relative to climate research.)
    Some of this data is available to everybody, some of it is restricted to people with NCAR computing accounts (mostly due to restrictions placed on the data when they receive it, and other practicalities.) Lets face it, WMO40 was/is
    a dual edged sword – it gave us access to some information that was previously
    difficult (or impossible) to obtain – at the same time it restricted
    what we could do with the data. I’d love to see a time when all data
    collected for weather and climate research could be freely exchanged,
    but I think that day is sometime in the future, and may never happen.
    We used to make global hourly SYNOP observations available, but the
    Irish Met office was kind enough to let us know that only 6 hourly
    observations were covered by WMO resolution 40. Everything else
    was considered “special” data – and every country had their own
    rules on “special” data. We don’t have staff to figure out
    which is which, and to separate it, so we had to stop making
    the data available.
    For those interested, a quick google search found
    http://www.nws.noaa.gov/im/wmocovr.htm
    It would be nice if the WMO, or some other body, made available
    an archive of all available data. Maybe someday this will happen.
    I’m sure it just a matter of money.

    My understanding is the data that CRU “lost” was in the form of paper
    output and obsolete tapes. This apparently happened in the mid 1980′s.
    Sorry I cannot provide a reference for this; this “understanding”
    occurred over the last few days from reading various web sites
    and blogs – and may not be correct. I know that we have had to
    recycle stacks of paper, and magnetic tape, because it became
    impractical to store, and resources were not available to put
    them on more modern storage. The reality is that no University
    has the capability, or the responsibility, to make any data
    they may have once had available to others. People say that the
    data should have been kept because now important (and expensive)
    decisions are being made based on the analysis of that data.
    But in 1985 that was not known.

    If anybody here has influence with IPCC or WMO, perhaps you can encourage
    them to make such a freely available data repository available?
    But again that would cost money.

    Since the “science” has apparently been
    more or less settled, perhaps that money should go
    toward possible mitigation efforts? But I still think an
    international data repository would be very valuable.
    While I’m tempted to suggest that all climate scientists
    retire, or move on, and no new ones be made, I think that would also be
    a mistake. To store and make the data available, and
    to retain climate scientists, is a lot cheaper than
    trying to resolve the mess we have (or might have) found
    ourselves in. If the data is available to everybody,
    perhaps one of them might find a mistake, or something new.
    The reality is, like it or not, the science has not
    been settled – sure there is some evidence the Earth
    might be warming, and CO2 is an obvious cause, but
    if you really think we know what is going on, I suggest
    you might be wrong (of course, you could be right).

    I have concerns about how the data might have been analyzed. But I
    have no interest in reanalyzing it. Well, I have the interest,
    but not the time. Enough people have done this – there might
    be problems looking for a signal in so much noise, but I think
    the people who have done this work did it as carefully and properly
    as possible.

    Climate scientists find themselves in an awkward position. They
    have found themselves thrust in to a position of guiding policy.
    This was in part because they had no choice – they found something
    that may be a real concern – in order to get that concern heard
    they had to publicize it. There is some concern that they trust
    this into the public view in order to enhance their career, or their
    funding. This might be true in some cases, but I have no doubt
    that there is a real concern – and when you are concerned
    about something you have to try to do something about it.
    Scientists have been wrong before, and MAYBE they are this time.
    But they would be negligent if they saw a potential problem,
    and did not try to fix it, even if it meant they have to
    delve into politics (where most of them have little skill)

    You have to admit, when we can’t predict the weather for next week,
    the public has to wonder about our ability to predict the weather
    (or climate, which is after all just the average of weather)
    for the next decade. Yes I know they are different problems, but
    most of the public does not see that, and after all they are mostly
    based on similar sets of data and models.

    Comment by David — 30 Nov 2009 @ 11:05 AM

  275. Just heard the Diane Rehm panel discussion on Copenhagen, which included Michael Mann along with a “resident scholar” from the American Enterprise Institute. I commend it to anyone, especially in the skeptic community, who still has questions about the email incident as well as the reality of the recent warming.

    Comment by Walter Pearce — 30 Nov 2009 @ 11:06 AM

  276. The medical profession has discovered that a poison contained in a United Nations variant of fudge was the cause of a debilitating brain disease among scientists, now termed Climate Wars Syndrome (CWS).

    The disease was secretly suspected by sceptical scientists to have spread rapaciously among the scientific community for two decades and to have taken a terrible grip over the reasoning powers of many. Victims can be identified by their green and alarmist complexion. Other side effects include an irrational hatred of mankind and a Tourette syndrome-like verbal abuse of anyone who uses fossil fuels. Threats of violence may occur. The world first learned of these sensational developments from the Internet on Friday November 20th 2009. The story broke that both the underlying cause of CWS and an effective treatment had been discovered by the due diligence of one man working at the UK’s Climate Research Unit (CRU). A vast community of Internet surfers soon memorialised these profound events by naming them, ‘Climategate.’

    From leaked documents we understand that the catalyst for this epoch change in science occurred when a climatologist and self-taught computer programmer known only as ‘Harry’ was sat at his laboratory computer chewing on some fudge. Only after three long years working on this problem and in a sudden eureka moment, did it finally dawn on him. In Harry’s hands was the cause of brain fog mystery.

    “F**k! It’s the fudge! It’s serial!” he cried.

    Inadvertently, Harry has become the hero the public associate with solving one of the great mysteries of modern science. Since those findings have appeared on the Internet the world has quickly accepted that it was the UN’s foul fudge that caused scientists to suffer this dreadful disease.

    Meanwhile, epidemiologists and clinicians have been quick to identify the hallucinagenic properties of the offending fudge to further unravel the mystery. Incredibly, the fudge has been found to contain a psychotropic substance that acts primarily upon the central nervous system where it alters brain function, resulting in changes in perception, mood, consciousness and behavior leading patients to feel delusions of grandeur and a sense of spiritual purpose in their lives.

    It appears lone-wolf Harry, wiling away his time in the CRU laboratory subliminally faced the truth and by a process of ‘cognitive dissonance,’ shocked himself out of the effects of the psychotropic intoxicant, a drug now known to cause the hallucinogenic appearance of a mythical beast known as, ‘Man-Bear-Pig’ (MBP). Other experts who have replicated Harry’s experiments confirm the efficacy of the cognitive dissonance reasoning process as a cure. Apparently, most recovering ‘addicts’ (for this fudge-eating was clearly an addiction) soon notice a change starting with improvements in the appearance of their eyes which lose their tainted green colouration.

    Other convalescing climatologists, that body of scientists identified as the worst fudge sufferers, are reporting the same side effects as Harry. Symptoms include anxiety, guilt, shame, anger, embarrassment, stress, and other negative emotional states that torment the patient. Epidemiologists have coined the name ’Climate War Syndrome’ (CWS) to describe the fudge-induced malady. Both ‘Climategate’ and ’Climate War Syndrome’ (CWS) have fast entered common usage giving a new handle on what was one of the great mysteries of our time.

    Of course, like any serious disease, there will always be patients who won’t respond well to treatment. Those worst cases permeated with the deepest shade of green are believed to be James Hansen, Michael ‘upside down’ Mann and Phil Jones whom, its feared, may all need to be quarantined in isolation for several years.

    [Response: Not really appropriate, but a step above the, umm... let's say, less cerebral sceptic offerings... - gavin]

    Comment by John O'Sullivan — 30 Nov 2009 @ 11:11 AM

  277. In the “documents” section of the FOIA2009 archive, there are several IDL programs that add “artificial corrections” to temperature data (for a list of files that match “artificial” see here). One such program is briffa_sep98_e.pro. This graph shows tree-ring data before and after the corrections defined in the file (for more background see here. Doesn’t the corrected graph look like they hockey stick? Were such corrections used in the construction of the hockey stick?

    [Response: No. These "corrections" were never used anywhere. See comments passim. - gavin]

    Comment by Kevan Hashemi — 30 Nov 2009 @ 11:34 AM

  278. “As with CRU, all I have to give out is the merged product. – gavin]”

    How do you know that the merged product represents the original product. Unless you either have the original, or enough metafile data to point you to the places where you got the pieces of your merged products, then the idea that your merged product represents reality is simply unsupportable.

    [Response: You are welcome to go back to the original sources and do it again yourself. The point is that there is only one product (the current database), but the raw data still exists (in it's less than perfectly accessible form). - gavin]

    Comment by Tilo Reber — 30 Nov 2009 @ 12:25 PM

  279. @Ron Broberg
    it is good to see that you are trying to make this GISTEMP work.

    Your first attempt appears to have been less successful than you originally thought and someone had to call you on it. So while you were not trolling (sorry for that) you were over excited and produced personal, as it were, attack based on your confusion.

    Hopefully you will get it to work and in the process create good documentation and truly become the first person in this thread to have ran GISTEMP.

    Comment by donQ — 30 Nov 2009 @ 12:32 PM

  280. I just saw an article in the UK Times that claims the orignal temp data (ie: raw unadjusted in any way) has been destroyed/mislaid during a move of facilaties. Further according to this article, the only thing now in possession of the CRU is the smothed and corrected data produce in accordance with CRU’s model. Could you comment as to whether this is true and on a hypothetical, were it in fact true would the research in fact be repeatable, in the of a good experemnet can be repeated and arrive at the same answer? Sorry some one may haved this before just did not have the heart to wade through all the coments. Thanks.

    [Response: The raw data is in the custody of the met services who originated it. CRU is just a collation, not a temperature measuring organisation. - gavin]

    Comment by Garrett Jones — 30 Nov 2009 @ 1:10 PM

  281. Ron, if you are using GNU Fortran, the enddianness magic can be done by setting GFORTRAN_CONVERT_UNIT before invoking the executable. This can set the endianness of any “units” (like file IDs) to big-endian, little-endian, platform-endian, swap-endian, etc.

    On the ccc-gistemp project I hacked out all the ksh driver scripts a long time ago – replacing them with a single /bin/sh script – and rearranged all the files into consistent directories, and so on, so my code and scripts aren’t the same as the released GISTEMP any more, but the relevant part of my run.sh file reads like this:

    echo “====> STEP 5 ====”
    # GISS binary data files are big-endian, so ours are too, including
    # the intermediate file generated by step3.py and the SBBX.HadR2 ocean
    # file which comes out of STEP4. We use GFORTRAN_CONVERT_UNIT to tell
    # our code to treat all these files as big-endian. This Will Not Work
    # if the Fortran compiler is not GNU Fortran.

    GFORTRAN_CONVERT_UNIT=”big_endian:10,11,12″ bin/SBBXotoBX.exe 100 0 > log/SBBXotoBX.log
    GFORTRAN_CONVERT_UNIT=”big_endian:10,11″ bin/zonav.exe > log/zonav.Ts.ho2.GHCN.CL.PA.log
    GFORTRAN_CONVERT_UNIT=”big_endian:10,11,12″ bin/annzon.exe > log/annzon.Ts.ho2.GHCN.CL.PA.log

    Hoping this makes it through the RC comment filter. If this interests you, feel free to join us on ccc-gistemp.

    Comment by Nick Barnes — 30 Nov 2009 @ 1:28 PM

  282. After days of reading through this, the more tempting it is to simply throw up my hands in frustration. The denialist will still continue to deny AGW, regardless of how much data you give/throw/otherwise convey to them. The primary forces opposed to AGW are simply too well financed; their pockets are very deep, they have control over numerous outlets and are quite crafty at picking-and-choosing-and-releasing data that supports their claims that AGW is not real, and that while climate change MAY BE real (and that is a big, fat MAY BE), it is most certainly not human in origin. This is not science against science here any longer; it is a clash of belief systems. The funny thing is that, like most belief systems, it is easier to say that the side with data supporting their claims has turned their science into a near religion. This is what happens with Creationists; their number one charge against evolutionists is that the evolutionists have turned their “theories” into religion.
    The media outlets that show a strong pro-industry bias are almost always controlled, in some fashion, by parties that stand to lose big if any mandates are handed down. Of course they are going to fight this, as this is going to be the biggest paradigm shift in recent human history.
    I fear that there will soon be a new group emerging from this; the GCC embracers. These are former denialists who now accept that AGW is real, but alas, it is too late to do anything about, so we simply need to just move along and accept our new, somewhat warmer and regionally flooded world.
    Just a few thoughts…

    Comment by VagabondAstronomer — 30 Nov 2009 @ 1:52 PM

  283. Before the “emails” were released Micheal Mann had a paper in press. It appeared in the recent issue of Science.

    Global Signatures and Dynamical Origins of the Little Ice Age and Medieval Climate Anomaly
    Michael E. Mann et al. Science 27 November 2009:
    Vol. 326. no. 5957, pp. 1256 – 1260

    Mann is one of the folks being lambasted for a lack of transparency. This paper is a case study in transparency and data availablilty.

    From the text of the paper:

    “Further details of reconstruction procedure, associated statistical validation and skill assessments, uncertainty estimation procedures, data used, and MATLAB source codes for the analysis procedures are provided in the Materials and Methods.”

    When you go to the online supplement, you will find this:

    “Computer codes and data. Computer codes (MATLAB), data, and supporting information for analysis in main paper. File are packaged as a compressed archive, in *.zip format; users should download the compressed file to their machine and decompress the file on their local hard drive, using the instructions below.

    multiproxySpatial09.zip (22 MB) [This is a link in the online version”

    And should we forget were we could be spending our time and energy, it’s a very interesting paper and well worth a read.

    Comment by Peter Houlihan — 30 Nov 2009 @ 2:18 PM

  284. Re: “You’re using a common name as a userid — whoever you are, people won’t know if the next “David” along who asks a naive question is from you again, playing games, or from someone with an honest question.”

    That’s a fair point, and I apologize. I won’t do it again, no matter how much I may be bothered by the tone of some of the posts here.

    Comment by David — 30 Nov 2009 @ 2:26 PM

  285. @Ron Broberg,
    using your instructions + code provided and starting from scratch, while running Ubuntu 9.10 x64 in Parallels and using this ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.mean.Z after “./do_all.sh” produces:

    start STEP0

    At line 10 of file sorts.f (unit = 1, file = ‘antarc.txt’)
    Fortran runtime error: End of file
    Bringing Antarctic tables closer to input_files/v2.mean format
    collecting surface station data
    … and autom. weather stn data
    … and australian data
    replacing ‘-’ by -999.9, blanks are left alone at this stage
    adding extra Antarctica station data to input_files/v2.mean
    0.00 not ok
    0.00 not ok
    created v2.meanx from v2_antarct.dat and input_files/v2.mean
    removing pre-1880 data:

    GHCN data:
    removing data before year 1880.0000
    created v2.meany from v2.meanx

    Replacing USHCN station data in v2.mean by USHCN_noFIL data (Tobs+maxmin adj+SHAPadj+noFIL)
    reformat USHCN to v2.mean format
    replacing USHCN station data in by USHCN_V2 data (all adjustments, but ignoring fill-ins)
    unifying station-ids
    ./unify_us_ids[3]: 9641C_200907_F52.avg: cannot open [No such file or directory]
    mv: cannot stat `9641C_200907_F52.avg.1′: No such file or directory
    <<>>

    Finding offset caused by adjustments
    extracting US data from GHCN set
    removing data before year 1980.0000

    Getting USHCN data:
    removing data before year 1880.0000
    done with ushcn

    Created ushcn-ghcn_offset_noFIL
    At line 21 of file cmb2.ushcn.v2.f (unit = 3, file = ‘ushcn-ghcn_offset_noFIL’)
    Fortran runtime error: End of file
    created v2.meanz

    Replacing Hohenspeissenberg data in v2.mean by more complete data (priv.comm.)
    disregard pre-1880 data:

    —-

    Is this similar to the output that you are seeing?

    Comment by donQ — 30 Nov 2009 @ 2:28 PM

  286. Interested in direct links to cosmic ray count data, AMO, PDO, and the major Nino SST indices?

    Comment by Brian Angliss — 30 Nov 2009 @ 2:32 PM

  287. 53 Jiminmpls said:

    “28 November 2009 at 9:40 AM
    Fair is fair. I demand that the Heartland Institute, George Marshall Institute, American Petroleum Institute, Western Fuels Association, Sen Inhofe, McIntyre, Singer, Ball, Michaels, and all other organizations and individuals attempting to influence public policy in regards to climate change IMMEDIATELY make their entire email archives available for public scrutiny and analysis.”

    According to most of the people posting here none of those organisations have anything at all worth saying so what would be the point?

    Comment by Tonyb — 30 Nov 2009 @ 2:57 PM

  288. The Antarctic Glaciological Data Center (AGDC) at NSIDC houses a collection of Paleoclimate/Ice Core data
    (http://nsidc.org/agdc/data/paleoclimate.html) collected under the NSF Antarctic Glaciology program.

    cheers,
    Rob

    Comment by Rob Bauer — 30 Nov 2009 @ 3:12 PM

  289. There’s a data-processing program buried in the FOIA archive, called briffa_sep98_e.pro. It adds hard-coded corrections to raw tree-ring data. The effect is like this. Did this program play a part in the composition of the Hockey Stick graph?

    [Response: No. See comments passim. - gavin]

    Comment by Kevan Hashemi — 30 Nov 2009 @ 4:16 PM

  290. Ray Ladbury (241) says, “Matthew, there are no true skeptics left–merely the ignorant, the wilfully ignorant, the denialists and the wingnuts.”

    So what do you call people who deny the reality of true skeptics (your self serving definition aside.)

    Comment by Rod B — 30 Nov 2009 @ 4:40 PM

  291. Not sure if it would be data overload, but RCM code is pretty easy to find too. I know from NARCCAP, at least, MM5, WRF, and RegCM3 have code available. I think WRF might make you do a quick registration first, though. It’s probably not worth listing RCM output since it’s highly individualized.

    Comment by Jason Patton — 30 Nov 2009 @ 4:48 PM

  292. I can confirm what Ron Broberg (#271) says. I also arrived at step 4 and met following message:

    iyrbeg-new 1880 iyrbeg-old 1476853760

    where the latter number is “1880″ in reverse endianess. I am using Ubuntu 32bit on intel.

    By the way, it took me between a little more than one hour to make the thing run, including download of everything, installing most of the needed tools (my Ubuntu box didn’t even had Ksh and the Fortran compiler installed) and rewriting the Python extension installer script. Not out-of-the-box, but not so hard either :-)

    Comment by Alfio Puglisi — 30 Nov 2009 @ 5:36 PM

  293. Re 129

    Gavin the quote from Hansen 2001 stated metadata defining “all” changes, not as you put it “any metadata”.

    Whilst the change of time of measurement is one of the multitude of changes, it does not comprise metadata defining all of the changes.

    Comment by Alw — 30 Nov 2009 @ 6:14 PM

  294. Okay, here’s a question about the Times missing raw data story beyond where the data still is: what did the CRU mean when they said:

    “Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues.”

    Wouldn’t the cost of keeping a few more numbers on tapes (or whatever) be trivial compared to cost of the whole enterprise? Or was something other than cost the issue?

    Comment by IJH — 30 Nov 2009 @ 6:27 PM

  295. Rod B., A true skeptic must acquaint him or herself with the evidence, acknowledge it and seek plausible alternative explanations. Do you know of any true skeptics Rod? I don’t. If you know of any, perhaps you can tell me their theories on how the troposphere can warm and the stratosphere cool via a mechanism other than a greenhouse one. Or what credible alternative energy mechanism accounts for 30 years of warming and the loss of trillions of tons of ice. That’s for starters.

    Comment by Ray Ladbury — 30 Nov 2009 @ 6:39 PM

  296. Gavin, thanks for your reassurance that the corrected data was not used. I’m looking for “comments passim” that you mentioned. Perhaps I’m just a blind old fool, but I’m not finding them in this list of comments. Can you direct me to your previous answer? Thank you for your patience. Kevan

    Comment by Kevan Hashemi — 30 Nov 2009 @ 6:43 PM

  297. @donQ#285: Is this similar to the output that you are seeing?

    Yes. Next update, I’ll post an output log.
    Also, I scripted a diff for the NASA GISTEMP output and my file. The largest diffs are in the endpoints – 1880 – 1900 or so and are likely related to the Hohenpeissenberg file. Currently I am downloading the one hosted in NL rather than use the one provided in the NASA tarball. I’ll switch back to NASA’s and rerun tonight.

    @Nick Barnes:
    Thanks for the tip. I’ll give them a try tonight.

    Comment by Ron Broberg — 30 Nov 2009 @ 7:29 PM

  298. #50–”I have never known anything to be settled in my long life”–so you go to bed hoping that you won’t fall of the earth before you wake up, gravity being one of those things that still isn’t settled. You hear there is some cooling going on–when it’s shaping up to be the hottest decade ever–each of the last 10 year have been among the 15 hottest years. Yes, 1998 was hot, but the trend still seems to be up. Maybe you should look a little more closely at what the sceptics say and see if it really holds water–that’s the basis of the camparison of deniers to creationists–they both ignore the evidence of their surroundings.

    Comment by Dave E — 30 Nov 2009 @ 9:03 PM

  299. Gavin, I’m sorry to bother you with more nonsense, but I’d trying to deal with some people making a fuss about the Times UK article about the magnetic tapes and paper notes that were thrown out to save space. I know I saw something here at RC on this, but haven’t been able to find any clarification despite some fancy “ctrl-F work in the posts. Also, in the advent of that article, it might be a time/sanity saver to update something about that in the body of this or the “…context” blog, to clarify quantity, quality and perhaps extent of documentation available to reconstruct (ie: how much methodology was written up if anyone wanted to backwards engineer the oldest data to extrapolate what would have been on the tapes) or how that data might have compared to other data sets from the same time/now. I personally see data on 1980s magnetic tapes as valuable as data scribbled on cocktail napkins late at night in the dark, but I’m only an undergrad… I guess I’d better not lose any of my notebook pages from now until I die or I might be in trouble. Anyway, I’d like to have something to tell people who think this is some big meaningful plot or omission of science to put it into perspective.

    Comment by Shirley — 30 Nov 2009 @ 9:12 PM

  300. On the question as to whether some Briffa data were used in one of the hockey stick graphs you say “No, see comments passim”. I’ve searched the comments for “Briffa” and “hockey stick” and not found anything. What comments answering the question are you referring to?

    Comment by Mark V Wilson — 30 Nov 2009 @ 9:42 PM

  301. #50–I apologize for perhaps being a little harsh in my previous reply to you. I admit that climate change is probably not as well settled as gravity, although I think that there is little doubt that it is occurring. I did use the term deniers in the comparison to creationists, and I do think that is a reasonably valid comparison, in that they are both denying something for which there is ocnsiderable evidence. You said you are a skeptic and despite my previous post, I wouldn’t want to put honest skeptics in with deniers. If you really are a skeptic, I hope that you will examine what the skeptical web sites are saying and look at what this site and others are saying about what is happening. I think if you really look into it, you will find that a lot of the skeptical sites are distorting what the real situation is.

    Comment by Dave E — 30 Nov 2009 @ 10:08 PM

  302. About #278 John o’Sullivan – Didn’t Michael Crichton get his big break by writing fiction about an alien microbe?

    Comment by Holly Stick — 30 Nov 2009 @ 10:54 PM

  303. Mike Flynn #259: judging from the examples you cite, I wouldn’t venture out of my home, if I wanted that degree of certainty. I certainly wouldn’t cross a bridge or drive a car.

    Kevan Hashemi #289: there’s a lengthy discussion on Steve McIntyre’s Climate Audit site predating the theft of the emails and documents, where they picked up the fact that the tree ring data ends in 1960, but somehow failed to read numerous mentions of the fact that this data becomes unreliable after 1960. While I have some sympathy with complaints that raw data wasn’t available, it’s hard to believe McIntyre is a serious scientist if he fails to mention (possibly failed to read?) the obvious explanation contained in the papers he is attacking. One example, from a paper cited on that same Climate Audit page: K. R. Briffa, T. J. Osborn, F. H. Schweingruber, Large-scale temperature inferences from tree rings: a review, Global and Planetary Change, Volume 40, Issues 1-2, Global Climate Changes during the Late Quaternary, January 2004, Pages 11-26, ISSN 0921-8181, DOI: 10.1016/S0921-8181(03)00095-X:

    However, in many tree-ring chronologies, we do not observe the expected rate of ring density increases that would be compatible with observed late 20th century warming. This changing climate sensitivity may be the result of other environmental factors that have, since the 1950s, increasingly acted to reduce tree-ring density below the level expected on the basis of summer temperature changes.

    The strongest conclusion I can draw from all this is that if anyone is being deliberately dishonest, its McIntyre.

    Comment by Philip Machanick — 30 Nov 2009 @ 10:58 PM


  304. Just heard the Diane Rehm panel discussion on Copenhagen, which included Michael Mann along with a “resident scholar” from the American Enterprise Institute. I commend it to anyone, especially in the skeptic community, who still has questions about the email incident as well as the reality of the recent warming.

    The whole episode is available as a podcast here: http://podcastdownload.npr.org/anon.npr-podcasts/podcast/305/510071/120952159/WAMU_120952159.mp3

    Basically, Kenneth Green of the AEI served up clueless talking-points and Michael Mann smacked them down quite nicely. The scientist (Mann) was able to get in the last word on most of the exchanges with the political operator (Green), a refreshing departure from the typical talk-radio format.

    Folks here who have “fence-sitting” friends/relatives should make certain that their friends/relatives download this podcast to their iPods/iPhones/iWhatevers and give it a listen.

    It probably won’t change many minds that have been made up, but I’m confident that it will convince many that haven’t. Dr. Mann did a bang-up job on that interview. I hope he does more.

    Comment by caerbannog — 1 Dec 2009 @ 12:54 AM

  305. Re: Comment by dale — 30 November 2009 @ 10:46 AM

    Hypocrisy knows no bounds, apparently:

    http://mediamatters.org/blog/200911290004
    “Capital Punishment for James Hansen. Climategate is high treason.”

    Comment by ccpo — 1 Dec 2009 @ 1:48 AM

  306. Well what I can conclude is that if climate science is somehow unfairly restricting access to raw data upon which scientific conclusions are based, there definitely has to be certain reasons to it. Raw data is data not formatted maybe they have a different process before making data available to all.

    Comment by dsi r4 — 1 Dec 2009 @ 2:26 AM

  307. I was a little disappointed not to see my climatology page listed in the resources… I list a lot of time series data and other useful tables. Oh, well.

    And can anyone point me to extinction coefficients for Rayleigh scattering? I’ve looked on Google Scholar, but I must be using the wrong search words. I know it’s proportionate to wavelength to the negative fourth power, but that doesn’t help me with ke = ka + ks.

    Comment by Barton Paul Levenson — 1 Dec 2009 @ 6:54 AM

  308. Philip Machanick #303: Okay, let me make sure I get this straight.

    1. The tree-ring data deviates from surface thermometers after 1960, and post-1960 data was not used at all in making the hockey stick graph.

    Many people have been saying the surface temperatures are wrong because of urban heating, and indeed I see evidence of urban heating when I reconstruct the surface trend myself. And yet, tree rings were rejected because they disagreed with the rise in temperature measured by surface thermometers. The fact that this disagreement has a name, “The Divergence Problem”, and that it is “Widely Known”, does not detract from the fact that most of us scientists would not react to a data conflict by rejecting one data stream and declaring the other to be flawless.

    [Response: Logical fail. That actual temperatures have been rising is unequivocal and demonstrated by ocean temperatures, retreating glaciers, melting snow, etc. etc. etc. The surface met station data do not need to be flawless (and of course they aren't) for you to know that a big decrease in one particular set of tree ring proxies is not representative of temperature. - gavin]

    2. The tree-ring data was used to show temperature stability over the previous four centuries.

    I guess different scientists have different styles, but when I find that a measurement method does not work, I don’t use it. In this case, it appears that the tree-ring data, having been discredited after 1960, was trusted over the past 1000 years to show remarkable stability of climate.

    [Response: Other tree data and other proxies work fine for this. - gavin]

    3. Many widely-publicized graphs, like this one, show tree-ring data from Briffa et al. right up to 2000.

    How can it be that we have tree-ring graphs going right up to 2000, and yet the raw tree-ring data itself is not being used? [edit]

    [Response: This actually only goes up to 1999 (since that was when it was made), and it can hardly be said to be well-publicised since no-one had noticed it until last week. And, as you know, this uses instrumental data for the modern period (alebit in a not clearly communicated way at the time). - gavin]

    Each of these three numbered statements can be defended in isolation, but taken together they are hard to explain. I am, however, open to persuasion.

    Comment by Kevan Hashemi — 1 Dec 2009 @ 8:59 AM

  309. Gavin #308: Thank you for your answers. When you say “the actual temperatures” I guess I’m not sure what you mean. Do you mean that the surface temperature measurements are the final word in global temperature? If so, I accept your faith in them, but I hope you can accept my lack thereof. The number of stations used to obtain the surface plots dropped from around 8000 stations in 1960 to 1800 in 2000 (see here), which happens to be the period during which you reject the tree rings for disagreeing with the thermometers, and the period during which the world experienced unprecedented urban growth. As to melting ice: if I put a big enough lump in the garden, it will take a thousand years to melt (I live in Boston, not LA), and I would not trust the melt rate as a thermometer, let alone a +-0.1C trend-measuring thermometer. That’s just me, however, but please note that measuring temperature is one of my specialities. So instead of “logical fail”, I’d rather say “disagreement about data reliability”, which seems more dignified.

    [Response: Try again. I was not specifically not claiming that the temperature data were flawless (which of course they aren't). But I was rejecting your claim that the post-1960 MXD proxy being thought wrong was a statement that the surface temperatures are perfect. That is a problem in logic, not a disagreement about data quality. The station number issue is a red herring since there are more than enough stations to characterise global temperature anomalies on an annual basis (something like 100 good ones is all that would be required), and again, there is plenty of supporting evidence for continued warming since 1990 - satellites anyone? I suggest you try your backyard ice block experiment - a thousand years isn't even close. - gavin]

    You say that other tree ring data and other proxies also show that the climate has been stable for the past thousand years. But work like this, which uses every proxy except tree rings, shows a distinct medieval warm period and a subsequent mini ice-age. So I’m not sure I could defend your statement if I was asked to.

    [Response: I never claimed that climate was stable for a thousand years. But Loehle's reconstruction is crap. - gavin]

    In your final response, you flatter me by saying that I know how instrumental data is used in the graph I linked to. The truth is: I don’t know, but I’m listening. How do you get 1960-1999 values in the Briffa plot when you have thrown away the post-1960 data? I just want the twenty-word summary.

    [Response: The smooth was calculated using instrumental data past 1960. (8 words). The caption stated that both proxy records and instrumental data were plotted but wasn't clear enough about how that had been done. The recent IPCC figures are much clearer in that respect. - gavin]

    Comment by Kevan Hashemi — 1 Dec 2009 @ 9:41 AM

  310. Tilo Reber says:
    30 November 2009 at 12:25 PM
    “As with CRU, all I have to give out is the merged product. – gavin]”

    How do you know that the merged product represents the original product. Unless you either have the original, or enough metafile data to point you to the places where you got the pieces of your merged products, then the idea that your merged product represents reality is simply unsupportable.

    [Response: You are welcome to go back to the original sources and do it again yourself. The point is that there is only one product (the current database), but the raw data still exists (in it's less than perfectly accessible form). - gavin]

    Also if you read the UKMet Office conditions of use for their data which has been posted here you’ll see that users are precluded from keeping their own copies of the raw data and from passing such data on to third parties. If other MOs have similar rules then it’s clear that no compiler of derived statistics would be able to keep the raw data on file.

    Comment by Phil. Felton — 1 Dec 2009 @ 9:56 AM

  311. Ow. I made the mistake of looking at Kevan Hashemi’s website: he using a method for averaging the anomalies of all weather stations in a dataset, and claims that because his average looks similar to the CRU record, his method is a valid proxy for that record. Because he doesn’t account for geographical siting of weather stations, and because he has determined that his method is a valid proxy, he therefore claims that the CRU record does not account for geographical siting either.

    Speaking of logical fails…

    Comment by Marcus — 1 Dec 2009 @ 10:14 AM

  312. How about the Coral Reef Temperature Anomaly Database (CorTAD): http://www.nodc.noaa.gov/SatelliteData/Cortad/

    Comment by John Bruno — 1 Dec 2009 @ 10:24 AM

  313. In response to all the skeptics still complaining about the lack of climate data availability, hammering Gavin about it, etc, what would happen if you tried looking for said data on the internet? Lets see.

    I just googled “climate change data”

    Result; over 46,400,000 hits. That is right. FORTY SIX MILLION AND FOUR HUNDRED THOUSAND hits. (and it only took 0.22 seconds)

    Wow, that was easy! And I didn’t even have to file a FOIA to get those selfish, corrupt scientists to spit out their coveted data.

    On page one of my search results (you can view the output here; http://www.climateshifts.org/?p=3619) there are 10 results including some newspaper stories arguing that scientists don’t share data and (ironically) also several portals where one can easily download climate data such as;
    the Climate Change Data Portal, http://sdwebx.worldbank.org/climateportal/
    a NASA master data directory, http://gcmd.nasa.gov/
    the NOAA climate Program office, http://www.climate.noaa.gov/cpo_pa/ccdd/
    the IPCC data distribution center, http://www.ipcc-data.org/
    and the NOAA National Climate Data Center, http://www.ncdc.noaa.gov/oa/climate/globalwarming.html

    At least five of the first ten results contain a wealth of climate data. Free for anyone to download, use and abuse. Extrapolating, without any justification whatsoever, to the all the results, means that there are 23,200,000 online data repositories. Obviously, there are not that many. But one could spend several lifetimes analyzing the data from just the first five results.

    To paraphrase from Ray Pierrehumbert’s education of Steve Levitt, it is amazing what you can learn with just a tiny bit of curiosity and an internet connection.

    And to speak to all the folks claiming they wrote such and such and that mean scientist didn’t respond; there could be hundreds or thousands of people asking Briffa, etc for their data. it just isn’t realistic to expect scientists to respond to EVERYONE who asks if they get popular, are being targeted, etc. I share my data regularly, but if I was getting more than a request or two day, it just wouldn’t be possible to handle the correspondence.

    I think open access data is a great idea. Believe me, since scientists are the primary consumers of data, this is the way we want it! But it takes a lot of time and $ to share data in a sophisticated way. What Gavin has set up here; http://www.giss.nasa.gov/tools/modelE/modelEsrc/
    is mind blowing. And far beyond the norm in environmental sciences. Lets give him some credit. And recognize that as currently structured NGOs, universities and many governmental organizations reward scientists for science, not for data sharing.

    Comment by John Bruno — 1 Dec 2009 @ 10:45 AM

  314. Gavin #309: Okay, you used “instrumental data” to obtain the post-1960 Briffa tree-ring graph, and there was some “smoothing” involved. Which instruments supplied the “instrumental data” that you used in the Briffa graph after 1960? (PS. Please forgive my request for twenty-word summary, which I meant as a way of showing respect for how busy you are, but I think comes across as some kind of challenge.)

    Comment by Kevan Hashemi — 1 Dec 2009 @ 11:08 AM

  315. Marcus #311: I am honored that you took the time to go to my site, and regret that you did not enjoy the visit. Perhaps you could explain in more detail why “integrated derivative” analysis of the station data is less reliable than CRU’s, by posting your critique here (If I am the fool you say I am, then my analysis is not worthy of this audience). All assaults upon my arguments are welcome and indeed encouraged.

    Comment by Kevan Hashemi — 1 Dec 2009 @ 11:19 AM

  316. Tilo Reber says:
    > How do you know that the merged product
    > represents the original product.

    And if CRU had indeed given over a file that they claimed was their copy of the “original product” — you’d trust their copy to be correct?
    Or would you be claiming that they had to show you certified copies of each original, and proof the people who certified them were certified as well?

    Comment by Hank Roberts — 1 Dec 2009 @ 11:27 AM

  317. Re: #71

    In addition to the CRU, NOAA, and NASA global temperature records there is data from the Japanese Meteorological Agency (JMA):
    http://ds.data.jma.go.jp/tcc/tcc/products/gwp/temp/ann_wld.html

    (hat tip to Michael Schlesinger via Andy Revkin: http://dotearth.blogs.nytimes.com/2009/11/30/more-on-the-climate-files-and-climate-trends/?ref=science )

    Comment by R Simmon — 1 Dec 2009 @ 11:29 AM

  318. Rather amusingly, a new lobby group launched in the UK last week, the Global Warming Policy Foundation, which has been amongst the most vocal critics of UEA during the current controversy has now admitted that it had introduced an error into CRU data that it has published on its website to show ‘global cooling’: http://timesonline.typepad.com/science/2009/12/climate-sceptics-get-it-wrong-1.html

    Comment by Bob Ward — 1 Dec 2009 @ 11:43 AM

  319. Hank Says:

    “Posting faux-naive questions with outdated information to try to test your fellow reader wastes everyone’s time who bothers to try to be helpful here.”

    Yet this is EXACTLY WHAT I SAID OF BURGY!!!!

    FFS.

    Two faced rissoles.

    Comment by Mark — 1 Dec 2009 @ 12:07 PM

  320. HAT-CRUd DATASET TAMPERING SCANDAL

    In a sensational development that has set the world of client science in a turmoil, climate change skeptic activists have published photographs of leading climatologists showing several of them afflicted with a severe case of HAT-CRUd, or, in layman’s terms, dandruff. Versions of the same photographs had already been published in leading scientific journals, but, according to the activists, “photoshopped so that the HAT-CRUd is dishonestly camoflaged”.

    “How can you trust people who have such questionable personal hygiene?” is one of the more printable lines of their press release. “These people clearly have no credibility, and they want to confiscate my SUV?”

    One of the pictured scientists, when contacted, claimed that the white flecks in the pirated photos were “merely artifacts of a lossy data compression algorithm”, but nobody listened to the geeky slob.

    Comment by Alistair Connor — 1 Dec 2009 @ 12:11 PM

  321. Seems to me that it might help to have clear definitions when discussing so-called climate skeptics.
    ‘Skepticism’ is certainly a misnomer, since it implies a certain amount of open-mindedness.
    ‘Cynicism’ is the correct word, since climate cynics consistently accuse scientists of wrong doing.
    From the dictionary:
    “Cynicism: 1. An attitude of scornful or jaded negativity, especially a general distrust of the integrity or professed motives of others”
    In contrast to:
    “Skepticism: 1. A doubting or questioning attitude or state of mind; dubiety.”
    I think it’s time to be more precise in distinguishing between skeptics & cynics in the public debate, as anyone claiming fraud is clearly a cynic, not a skeptic.

    Comment by Dennis Sweitzer — 1 Dec 2009 @ 12:15 PM

  322. I have some thoughts about clear definitions when discussing so-called climate skeptics.
    ‘Skepticism’ is certainly a misnomer, since it implies a certain amount of open-mindedness.
    ‘Cynicism’ is the correct word, since climate cynics consistently accuse scientists of wrong doing.
    From the dictionary:
    “Cynicism: 1. An attitude of scornful or jaded negativity, especially a general distrust of the integrity or professed motives of others”
    In contrast to:
    “Skepticism: 1. A doubting or questioning attitude or state of mind; dubiety.”
    I think it’s time to be more precise in distinguishing between skeptics & cynics in the public debate, as anyone claiming fraud is clearly a cynic, not a skeptic.

    Comment by Dennis Sweitzer — 1 Dec 2009 @ 12:26 PM

  323. Ray (295), by (normal) definition, a skeptic is NOT required to have a fully developed alternative.

    Comment by Rod B — 1 Dec 2009 @ 3:07 PM

  324. Gavin: don’t get too frustrated by the high number of newbies asking old questions. I don’t know what’s happened to traffic to RealClimate, but my blog has had a huge increase in visits mostly via RC (Google Analytics), and I haven’t been posting anything remarkable here, so the most plausible explanation is a lot more new curious readers on RC.

    Kevan Hashemi <a href="http://www.hashemifamily.com/Kevan/Climate/"#309: if I can add a little to Gavin’s responses, if any stations are removed from the total, the overall average has to be calculated taking into account the area they used to cover. One approach is to use neighbouring sites as approximations for the missing sites. This is one of the reasons that a temperature anomaly is used rather than absolute temperature. Neighbouring sites may have different absolute temperatures but will usually have highly correlated variation. I don’t know how this is actually done, but the obvious thing to do when losing a site is to find neighbours with variation most highly correlated to the missing site, and weight those up to replace the lost site.

    On the Briffa allegations, my approach has been to read his papers on the subject, and they generally explain the divergence problem, so claiming that there has been dishonesty based on finding computer code to generate graphs has no basis in reality. We have no idea what a particular program in this archive actually represents. It may be a what-if thought experiment, a test program to compare actual values with constants to check that scaling is right, a version modified since publication to compare against another data set, … I have a PhD in computer science, not clairvoyance, so there’s a limit to what I can read into a snippet of code with no context. Particularly when the code concerned was stolen under circumstances yet to be revealed. Had there been computer code showing a faked graph and that graph was published as representing the actual reconstructed temperature, then I would have been irate, as I suspect would most other readers of RC (and I hope contributors).

    Comment by Philip Machanick — 1 Dec 2009 @ 5:16 PM

  325. The courteous way you are handling critical post is a credit to you. But, gentlemen, if you do not treat the scientists that have betrayed us harshly then this field of study will be placed alongside Phrenology and Astrology.

    Comment by Tom Johnson — 1 Dec 2009 @ 6:33 PM

  326. Nice to see that efforts are made to compile the GISTEMP code and verify the output. I took a different approach by constructing a global temperature index from scratch, using the same observations used in GISTEMP along with data from other sources. First results, very crude (e.g. without station series homogenization), look quite similiar to the existing estimates, especially from 1960 onward. More details at the link, input and ideas are most welcome (e-mail can be found on the page as well).

    Comment by Ben Lankamp — 1 Dec 2009 @ 7:19 PM

  327. Gavin, would it be possible to open up the raw data for plotting the O2/N2 atmospheric mixing ratio? The data is currently protected on this web site: http://bluemoon.ucsd.edu/data.html

    If there is an alternate location where this data can be publicly accessed then please advise. Thank you.

    Comment by Daniel H — 1 Dec 2009 @ 7:51 PM

  328. Oops!
    http://timesonline.typepad.com/science/2009/12/climate-sceptics-get-it-wrong-1.html

    Comment by Moira Kemp — 1 Dec 2009 @ 8:07 PM

  329. #303 – Philip. I believe I said the following: -
    ” I am a self uneducated retiree with some scientific background.

    Does anybody on either side of this argument claim to be able to foresee the future? As far as I can see, computer models have the same predictive ability as chicken entrails, albeit using chicken entrails you get to eat a chicken as a side benefit.

    Although it is tempting to believe that the past can be extrapolated into the future, it just ain’t so.

    What usually happens is that people (not just scientists) confuse “assumptions” with “predictions”. Newton’s thoughts on gravity. Einstein’s thoughts on gravity etc., were all deemed to be practically useful at the time. Good enough to “predict” eclipses, demolish the luminiferous ether (believed in by a majority of scientists, I believe), and so on.

    The future is unknowable. Prove that you can foresee even the next 5 seconds repeatedly with one hundred percent accuracy, and I will “predict” that civilisation as we know it, will end. You can easily work out why.”

    I can’t see the “examples” to which you refer. Maybe you looked at another post.

    Gavin –

    You say : Fortune telling is a class-B misdemeanor in NY state, and obviously no-one here is doing that!

    I say : Really? Based on the past, my chicken entrails predict iceless Antarctica, grapevines flourishing in Northern England, Greenland supporting a farming population big enough for Rome to send a Bishop, permafrost thawing, with animals grazing on previously frozen ground.

    I then foretell the future beyond this. An Ice Age will descend upon the world. Glaciers will reshape the landscape. A time of cold and dark will prevail!

    As I say, you can’t predict the future any better than I with my entrails. (Actually I cannot lie. I don’t use entrails, I just “predict” the past as the future.) Alas, I am forced to confess. Mea culpa.

    You say (G) :

    However, predictions are the lifeblood of science. Given certain, well-defined, conditions, scientific theories will predict what the result of a specific experiment will be. If the conditions are not met (the temperature is different, or it wasn’t a vaccuum etc.), the prediction doesn’t work.

    I say (M) : My point exactly. You use “prediction” instead of “assumption”. And we have to wait until the predicted event either occurs, or doesn’t. As in Newton’s “laws” being found not quite able to explain some subsequent observations. Einstein’s work filled in some of the holes. However, there are some areas of Einstein’s work that don’t quite accord with observations. Oh well, most predictions were close enough. The science is settled?

    G.: With respect to the future of the climate system, there are some external forces that are quite predictable – the orbit of the Earth, and its variations, the continuing rise in CO2 emissions, etc. Another example would be the climate consequences of a big volcano. In each case, you can use climate models to project the impacts on temperature, circulation etc.

    M. : I don’t believe you! One the one hand, you predict growing CO2 emissions. On the other, you believe they should be lowered until they no longer grow. Please examine your models and predict CO2 levels in the year 2050? Lower or higher and by how much? Maybe this is one of the areas where the prediction doesn’t work.

    G: In fact, people have done this. In 1991 Hansen et al predicted the temperature drop that was expected from the Pinatubo eruption well in advance of it happening. In 1988, the same team projected how temperatures would increase under 3 scenarios of rising greenhouse gases. For the scenario that came closest to being realised, the trends were a pretty good match to what actually happened. Other groups have made skillful projections for the El Niño events and their consequences months in advance.

    M: I am not a trained “climatologist”, but I assume that a rather large cloud of particulate (aerosol) matter might result in lower air temperatures under the cloud. Is this what happened? NASA said “The predicted decrease in the global surface air temperature by 0.5 C by the end of 1992 with recovery to normal by 1995 has proven to be remark-ably accurate, though not all for the right reasons.” I cannot find a free copy of the paper (strange, that). Other citations use words like “closely”, “maybe”, “possibly”, and “somewhat less than predicted.” One thing that I DID notice was that the Pinatubo eruption reversed the effect of AGW for a year or so. Maybe that’s the answer – flood the atmosphere with aerosols of the right type. Skillful projection is what I do. Can’t bag anyone else for doing the same thing. Would you rather have a factually answer for the wrong reasons or a well reasoned, peer reviewed, ideally referenced incorrect answer?

    If you are as scientist, I don’t have to point out the incorrect scientific assumptions over the years.

    G: So finding useful information about the future is not impossible, even if it isn’t fortune telling as described in the NYS penal code. – gavin]

    M: Yes it is. It’s just disguised as “science”. The best climate predictions

    1. Are scary.

    2. Won’t occur until the present Government of (insert country of choice here) has finished its current term.

    3. Preferably after the predictor retires

    4. Generate maximum grant funding for the predictor.

    The list goes on.

    Don’t get me wrong. The AGW theory may be right. But based on past history, the Earth has to get at least as hot as it has been in the past without human involvement, before I will become any more than moderately concerned. So call me after the Antarctic ice cover is all gone. Obviously, if the ice cores only go back x years, there wasn’t any ice before that time.

    I wouldn’t blame you for moderating me out of existence, but let an old man get some enjoyment from his few remaining years.

    Regards to all. Live well and prosper

    Comment by Mike Flynn — 1 Dec 2009 @ 8:16 PM

  330. @Rod B#320: by (normal) definition, a skeptic is NOT required to have a fully developed alternative.

    In my reading of the history of science, (normal) skepticism doesn’t get very far. Bad science is tossed out not by skeptics, but by better science. Einstein’s skepticism of quantum theory didn’t amount to a hill of beans because he had nothing better to offer in that field. On the other hand, he revolutionized physics, not by being skeptical of Newton, but by offering something better (more complete).

    Comment by Ron Broberg — 1 Dec 2009 @ 8:30 PM

  331. RE: caerbannog

    Basically, Kenneth Green of the AEI served up clueless talking-points and Michael Mann smacked them down quite nicely. The scientist (Mann) was able to get in the last word on most of the exchanges with the political operator (Green), a refreshing departure from the typical talk-radio format.

    Mike Mann did very well – good to see another good spokesperson for the science. He was very good at framing questions to make the bigger point, in the right context. I think he also acquitted himself well with the policy/big picture questions.

    Caerbannog, Diane Rehm is one of the better interviewers out there. She’s always been able to bring in some interesting people and ask great questions. Besides the regular government people, she brings in some heavy hitters from NIH for health issues.

    Comment by Deech56 — 1 Dec 2009 @ 9:28 PM

  332. “Interested in direct links to cosmic ray count data,” Brian Angliss — 30 November 2009 @ 2:32 PM
    http://cosmicrays.oulu.fi/ ” The full database (since 1964)”

    Comment by Brian Dodge — 1 Dec 2009 @ 9:32 PM

  333. Rod B @ 320, by definition, a skeptic is not required to have a fully developed alternative… this is true, but skeptics should be able to articulate a fact-based reason as to why they are skeptical. Most “skeptics” in the realm of AGW don’t do this. They simply dismiss the 30+ years of collected data, research, and scientific literature with a hand wave. Those are the actions of a denier, not a skeptic, and at worst is intellectually lazy and dishonest.

    Comment by Karl K. — 1 Dec 2009 @ 9:45 PM

  334. Ray (295), by (normal) definition, a skeptic is NOT required to have a fully developed alternative.

    Sorry, but that’s simply not true, not in science, anyway. You’re saying that “la-la-la I don’t believe you!” is sufficient, which come to think of it, pretty much sums up your contribution to this site over the last few years.

    Comment by dhogaza — 2 Dec 2009 @ 12:12 AM

  335. Cool, raw data. As an electrical engineer with 30 years of remote sensing experience, and having taken statistics and probability way back in college, I was easily go through and process raw data in Excel until it couldn’t hold any more data. Then I could have written a simple program to do the same analysis, but being lazy, downloaded one from the web.
    The biggest headache was simply reading through the readme’s so I knew how to read the raw data. (For example on the v2 files knowing to divide by 100 to convert the integers to degrees.)
    But after going through this (I only processed data that had the measured temperatures), I don’t get any Hockey Sticks, and seems to go down for the last 10 years. What I get matches the Sunspot cycle length vs temp models.
    Now I am admittedly not a climatologist, so I don’t know the validity of converting from anything but measured values, but I can do the mathematics. But I also did not keep track of which pieces of data I used, since some of it has missing measurements, or other funkiness, and it was more an exercise in curiosity.

    Many of the other posts have sighted availability of data all over the web, but almost all of these are data that is expressed as anomalies and furthermore the ones that I looked at trace back to the ones that you have sited here. Which in terms of raw data sources is not all that huge. But it is a lot of data.
    People like me need to be able to follow the math from the raw data (not process it), or we will be skeptical, and I can’t get there from the raw data (so I tried to process it). When I read that people like as Freeman Dyson , Fred Singer, Ian Plimar, John Coleman, John Christy, Richard Lindzen, Henk Tennekes, and many, many others have similar issues that doesn’t help, and is exacerbated by the email scandal.
    If I hear “Peer Review” as proof, I want to puke. Peer review is something that you should do to make sure your proof will hold up to everyone else reviewing what you have done.
    At present, as I look out the window and read my outdoor thermometer, I think that the astrophysical models of Milankovitch and Sunspot cycle models is more accurate, which should give C02 being absorbed as the oceans cool. But maybe the CO2 models are correct and we can use it to fend of the ice age.

    Comment by Mike S. — 2 Dec 2009 @ 12:21 AM

  336. @Ben Lankamp#326: I took a different approach by constructing a global temperature index from scratch, using the same observations used in GISTEMP along with data from other sources.

    Nice work. I hope we can your methodology in more detail later. Your work is more valuable than a simple port of GISTEMP.

    A snippet of something I wrote elsewhere about levels of confirmation/corroboration.

    1) Given the exact data and exact code, an audit can be made on implementation errors.

    2) Given the same data but using independent methods, results of a first team can be bolstered/undermined by a second team.

    3) Using independent data and independent methods, results of a first team can be bolstered/undermined by a second team.

    Each has its place, but #3 has the most scientific value.

    Comment by Ron Broberg — 2 Dec 2009 @ 12:41 AM

  337. Dear Gavin, Your tireless moderation is much appreciated. I would like to print or copy text from the comments (eg. your response to comment 57), but find I cannot do this as I can’t seem to select text from this pop-up. Any advice?

    Comment by Hugh R — 2 Dec 2009 @ 4:08 AM

  338. “So call me after the Antarctic ice cover is all gone…. let an old man get some enjoyment from his few remaining years.”

    Thanks, ‘old man’ but I’d prefer to leave a planet for my children, and their children, that is capable of supporting human civilisation in roughly the same degree of luxury as it does now. I don’t predict
    extinction (the science doesn’t support that!) but it does support significant damages to agriculture,
    water supplies, lose of fish, increased disease vectors, migration etc. that are very alarming.

    There is ZERO evidence that you can increase CO2 in the atmosphere to 550ppm and see no significant temperature rise. If you can prove this, please do so. It would make me happier.

    The is HEAPS (literally heaps and heaps) of evidence that IS NOT BASED ON MODELS that shows that if you increase CO2 in the atmosphere to 550ppm, temperature will increase by 3 degrees, give or take a degree and a half.

    Based on THE SCIENTIFIC EVIDENCE which of the following statements do you believe are true?

    a) Our current fossil fuel based economy will cause a significant increase in global temperatures over the course of the next century if we fail to moderate CO2 emissions

    b) Nothing to see here folks. Carry on as usual

    Comment by Silk — 2 Dec 2009 @ 5:15 AM

  339. Re: the website Climate4You (#268),

    it’s run by Ole Humlum, a U. of Oslo physical geographer of Danish origin, who’s done glacier work on Greenland and Svalbard (e. g. in Holocene 15(3)). He’s got a rambling 34-page essay online in Norwegian, a hodgepodge of historical factoids on climate with some hand-waving about Henry’s law and isotope ratios to suggest that the atmospheric CO2 increase is mainly from natural sources. He recently featured in a Norwegian newspaper debate with Eystein Jansen about the IPCC and the skeptics. Humlum, and a co-author who also ought to know better, wrote inter alia:

    Recently the UN’s environment organization, which is behind the [IPCC], announced that it’s wrong that the temperature on our planet has risen so steeply the last hundred years. After 11 years, a cornerstone argument of the [IPCC] that it’s never been warmer than today for thousands of years, is falling. It is now well-documented that the Middle Ages [sic!] was a global phenomenon and considerably warmer than the last decade, which is only marginally warmer than the 1980s. [Aftenposten, 16 Oct 09, transl. CM]

    ‘Nuff said.

    Comment by CM — 2 Dec 2009 @ 6:04 AM

  340. What’s the point of posting links that require you to log in with a password? I want to examine how the stability problem in GCMs is actually dealt with.

    [Response: What stability problem? However, if you are looking for explanations, you are better of reading the papers than the code. -gavin]

    Comment by 3D modeler — 2 Dec 2009 @ 6:17 AM

  341. Mike Flynn: if you don’t want to accept any projection or prediction or whatever you want to call it, take my advice and never leave your home again.

    Are you confused about what an assumption is? An assumption is something you make up (possibly plausibly, but it’s something you toss in without having to justify). Any reasonable scientific theory is not based on assumptions, but on propositions that are verifiable against reality.

    Mike S.: you will not get much traction by citing the likes of Fred Singer and “Ian Plimar” (even if you spell his name right) as authorities. Singer has a long history of attacking science to support causes like tobacco, and Plimer is not a climate scientist. He’s an economic geologist, and his widely publicized (in Australia) book is riddled with errors, indicating he isn’t even a good geologist. Rather than citing bogus authority, stick to understanding the science, even if it’s hard. The people who discovered the role of CO_2 in the climate are also by and large those who first understood how Milankovitch cycles worked. If you think sunspots explain the modern climate trend, I have news for you.

    Mike, this so-called email scandal is nothing compared with the concerted effort of industry shills and hangers-on to confuse the public about the science. It’s an old game started with industries like tobacco and asbestos. The only puzzle is why more people haven’t seen through it by now.

    Comment by Philip Machanick — 2 Dec 2009 @ 6:32 AM

  342. Mike Flynn,

    are you the fellow who wrote Eifelheim?

    Regards,

    Martin

    Comment by Martin — 2 Dec 2009 @ 8:02 AM

  343. Philip (#324), If you apply our method of integrated derivatives to any set of station data, it will generate almost exactly the same curve as the CRU or NCDC methods. That’s our claim. Our existing graph is all the evidence we need to make the claim, just as a graph relating mercury column height to temperature from 0C to 100C is all we need to claim that mercury column height is a measure for temperature. You are welcome to disprove our claim by applying our method (it’s simple, here’s the code) to any other set of station data, calculate the integrated derivative curve, and compare it to an existing CRU or NCDC curve obtained from the same data. If they differ significantly, I’m wrong. But no amount of talk about smoothing and grids and compensation is going to shift me from my position. Why should it? I’m a scientist, I don’t have to be convinced by fancy talk. Now, given that we can generate the same curve with a calculation that takes no account of distribution, disappearance, nor urban heating, we can say with certainty that one of two things must be true for each of these sources of error: either it does not exist in the data, or CRU and NCDC failed to compensate for it. But we show that disappearing stations do have a strong effect upon the resulting trend. Therefore, the CRU and NCDC curves have failed to account for the loss off 80% of their weather stations during the period 1970 to 2000, which is precisely the period when their curve shows such sustained warming.

    Comment by Kevan Hashemi — 2 Dec 2009 @ 10:41 AM

  344. For anyone still reading this thread, and interested in GISTEMP:
    David Jones and I have finally started the Clear Climate Code website, hosted by John Keyes. The first couple of blog entries describe the overall project and the progress on CCC-GISTEMP.

    Comment by Nick Barnes — 2 Dec 2009 @ 12:20 PM

  345. Thanks for making these links available – very helpful.

    Can anyone shed light on the following file I came across in the NGRIP site – http://www.gfy.ku.dk/~www-glac/data/ddjtemp.txt ? This is a set of data showing historic temperatures in Greenland based on ice-cores. The following summary accompanies the data:

    In Greenland:
    The warmth of the climatic optimum during the European stone age 5000 years ago is clearly seen, just as the cool period during the Roman age and the
    relative warmth of the Viking age (where the Norsemen settled in Iceland and
    Greenland) and the two cold periods of the “little ice age” at 1600 AD and 1875 AD. The warming in Greenland in the 20th century only lasts until 1950 AD. After
    that it has become colder.

    This doesn’t seem to fit with everything else I’ve read about unprecedented temperatures today – and how can Greenland be getting cooler if the ice sheets have melted so dramatically?

    Comment by Jonathan — 2 Dec 2009 @ 4:23 PM

  346. Kevan Hashemi #343: Believe it or not, I’m also a scientist, though I don’t work in climate science. I can also read Pascal, though that was an interesting retro experience. I last used the language about 20 years ago.

    Remember what we are trying to do: measure a worldwide trend, i.e., drift in the long-term average. We need to do that in way that minimises the risk of losing data, since the number of weather stations is not fixed over time. If you have your own personal methodology that you show to be highly sensitive to changes in the number of stations, you have only established that your method is flawed, not that someone else’s method is flawed.

    Before you run an experiment, you need to set up your hypothesis, and determine what will support it and what will knock it down. You can’t just do an arbitrary calculation then claim it means something. First, using absolute temperatures rather than anomalies means removing or adding weather stations will have an effect that is unrelated to the trend. Here’s an experiment for you to try. Remove every Arctic weather station after a given year, and the average will shoot up after that. Secondly, the derivative method you use will delete any inter-annual trend. Why would you want to do that? It makes no sense whatsoever.

    I’m not sure how the fact that your different data manipulations produce a particular result have any relationship to anyone else’s graph. The only thing you can say with any certainty is that coincidences are possible if you try enough random data manipulations. You do one set of manipulations that is in no way similar to those of the other bunch and get a vaguely similar graph. You change the data, and get a different graph. Totally meaningless unless you have a solid justification for your approach.

    Try this. Use the same methodology as the original approach, i.e., use temperature anomalies rather than absolute temperatures, and don’t do any strange manipulations like integration and derivatives that produce artefacts not present in the original data (do you have any idea what effect these manipulations have on the statistics of the data set? Are you introducing autocorrelation?). Calculating anomalies is not that hard. Choose the baseline period (any range of years long enough to smooth out differences, 30 years should be enough: NASA uses 1951-1980, CRU uses 1961-90). Average temperatures at each station over that period, then subtract each station’s average from the actual temperature reading. Graph the result. Others on this site will let you know if I have this wrong.

    Once you have this straight you can do a proper analysis of the effect of this missing stations. I suggest you read the academic literature on the urban heat island effect before you do this. Google Scholar is your friend.

    NASA GISS has a good discussion of why anomalies are used. Read it.

    Comment by Philip Machanick — 2 Dec 2009 @ 5:52 PM

  347. Jonathan, pasting that cite (it’s from 11 years ago) into Google Scholar turns up papers published since then that mention it. Try some of these:

    http://scholar.google.com/scholar?hl=en&q=D.Dahl-Jensen+et+al%2C+SCIENCE+282%2C+1998%2C+p.+268-271&lr=lang_en&as_sdt=2001&as_ylo=&as_vis=1

    Comment by Hank Roberts — 2 Dec 2009 @ 8:58 PM

  348. Here is a link to some raw data – US Station Daily Data: Access by State http://www.esrl.noaa.gov/psd/data/usstations/. “the user can examine the temperature and precipitation station data for continental US. All stations available have data from 1950 until 1999 for daily maximum and minimum temperature and for precipitation”. Gavin, mabye you could add a subdirectory for raw data.

    Thank you!

    Comment by Joe V. — 2 Dec 2009 @ 10:14 PM

  349. Jonathan, since the Arctic melt allows low pressures with thick cloud cover to go in a more northerly route, the Arctic high pressure area is located more commonly over Greenland, allowing it to cool more at nights, or so I’ve understood this. To get a more detailed answer I should try to look to some model results, but I think this is the trick to get at least a partial explanation to the observed discrepancy in trends between Greenland temperatures and the rest of the Arctic. It is colder in winter/nights in there that it has been before.

    Comment by jyyh — 2 Dec 2009 @ 11:23 PM

  350. @Philip Machanick

    I apologise for not communicating more clearly. I didn’t complete High School, so my feeble attempt to draw a distinction between an unproven (but workable) assumption, and a Scientific “prediction” or “forecast” made by people who cheerfully admit that nobody understands the complete workings of the Earth’s climatic processes sufficiently to accurately explain the past, let alone the future, was obviously doomed to failure.

    I am not sure why I should never leave my home again. Unless you can be a bit more specific, I will carry on life as usual. So far, so good – I’m still alive.

    @Martin.

    No, sorry. I didn’t write Eifelheim.

    @Silk

    I am unaware of any scientists who don’t accept that the Earth has experienced glacial periods, followed by (naturally enough) interglacial periods. I assume that the ice melts because the surrounding atmosphere gets warmer.

    Conversely, areas of permafrost that were once soil capable of growing grass and similar stuff, (I don’t know what it would be called, but mammoths got quite large eating it), got a lot colder quite suddenly.

    I also believe the research that shows that prior to the late Ordivician glacial period, atmospheric CO2 probably reached 4000 ppm, or thereabouts. The exact quantum may be irrelevant.

    My query has always been “What mechanism explains the glacial/interglacial temperature differences?”

    The Earth may be warming, may be cooling, may be about to lurch one way or the other next week/month/year.

    Could you please me know what will happen if CO2 reaches 1000 ppm within the next two years?

    Or maybe 7000 ppm. I believe that the atmosphere has reached this level, but I can’t vouch for it from personal experience. I await your guidance.

    @Jonathan

    Good for you. Parts of the Earth warm up, parts of the Earth cool down. Nobody really seems to know why. They’ll still predict the future for you, for a fistful of dollars! Sorry, couldn’t help myself. Must be my lack of education showing through.

    @Gavin

    I really WOULD like to know how the atmosphere and climate work. I worked for the OZ Met Bureau a long time ago. Had an interesting experience when the crew of a rather high flying Vulcan confirmed my radar observations of “cloud” tops. Unfortunately, the scientific assessment of the qualified meteorologists was that my observation was theoretically impossible according to published papers by experts in the field.

    Some time later, I believe a new cloud growth mechanism was adopted, which fitted better with observation. Possibly RAAF pilots carry more weight than humble Observers.

    Some experts continued to disbelieve observed data which conflicted with their theories, and discarded it. Maybe it still happens. I really don’t know.

    Regards to all. Live long and prosper.

    Comment by Mike Flynn — 2 Dec 2009 @ 11:32 PM

  351. Also for Jonathan:
    http://www.globalwarmingart.com/wiki/File:Holocene_Temperature_Variations_Rev_png

    Comment by Hank Roberts — 3 Dec 2009 @ 12:20 AM

  352. Kevin Hashemi,

    Your conclusions are very important if true. Please write them up in proper academic format and submit them to a climate journal, or perhaps Nature or Science.

    Comment by Barton Paul Levenson — 3 Dec 2009 @ 7:40 AM

  353. Jonathan (#345), some suggestions from an amateur reader:

    I’d keep in mind that rising global mean temperatures do not that local or regional temperatures must rise evenly in space or time. Greenland and Antarctica, for instance, have tended to go opposite ways over the long run. And the ups and downs of Greenland winter temps, for instance, are tied to the regional variation known as the North Atlantic Oscillation.

    Next I’d note that the time series you linked to were published a decade ago, so they only go up to 1984 and 1995. Greenland temps have been rising since sometime in the 1990s, and so has melt area. What you hear about accelerated melting pertains especially to this recent period (though 1987 and 1991 were big melt years, and there’s an upward trend in melt area for the whole period since 1979, see the Copenhagen Diagnosis, figure 9).

    I’d want to see these ice-core-based reconstructions in context with others using different methodologies. For the recent period, you would want to see how it fits with the actual Greenland instrumental temperature record, e. g. here (Vinter et al. 2006).

    One thing I think I see in Vinter et al. is that the temperature decline after the mid-20th century was most pronounced in the winter and spring months, not in summer when it actually gets above freezing. So it might not make much difference to ice melt? – I could be wrong about this.

    (Anyway, if you look at the ice-core time series you linked, you’ll see that they reached their warmest point since the 13th century and the 10th century in 1953 and 1966, respectively. Though they show declining temperatures after that, those temperatures were still actually fairly high, the highest since an 18th-century peak which was, again, the highest since the 13th century or so.)

    Oh, and if you were worried that these studies might have gone unnoticed, don’t be; they’re both cited in the 2007 IPCC report.

    Comment by CM — 3 Dec 2009 @ 8:30 AM

  354. Mike F, relevant to many of your queries is the release below (and you are of course free to follow up to the actual papers.)

    (The ultraquick version: at the end of the Ordivician, rapid geologic uplift (AKA the mountain-building episode that created the Appalachians) resulted in a pulse of silicate weathering, which in turn crashed CO2 levels (in “only” 7-8 million years) thus driving glaciation. Saltzman 2005 reconciles the chronology.)

    Link to release:
    http://earthobservatory.nasa.gov/Newsroom/view.php?id=31182

    Much of this material is covered in David Archer’s book, The Long Thaw, which I’ve reviewed here:

    Or, better, you might want to check out Dr. Archer’s U. of Chicago material via the portal elsewhere on this site.

    Comment by Kevin McKinney — 3 Dec 2009 @ 8:40 AM

  355. Mike Flynn:

    My query has always been “What mechanism explains the glacial/interglacial temperature differences?”

    Google “Milankovic Cycles.”

    I really WOULD like to know how the atmosphere and climate work

    Then I suggest you read a book like Dennis Hartmann’s “Global Physical Climatology” and follow it up with one like John T. Houghton’s “The Physics of Atmospheres.” And work all the problems.

    Comment by Barton Paul Levenson — 3 Dec 2009 @ 1:52 PM

  356. #350

    “I am unaware of any scientists who don’t accept that the Earth has experienced glacial periods, followed by (naturally enough) interglacial periods. I assume that the ice melts because the surrounding atmosphere gets warmer.”

    I’m not sure what the relevance of this is to what I posted. If you think it is relevant, could you explain how?

    “Conversely, areas of permafrost that were once soil capable of growing grass and similar stuff, (I don’t know what it would be called, but mammoths got quite large eating it), got a lot colder quite suddenly.”

    ditto

    “I also believe the research that shows that prior to the late Ordivician glacial period, atmospheric CO2 probably reached 4000 ppm, or thereabouts. The exact quantum may be irrelevant.”

    I’m no climatologist, but yes, there were periods in the past where CO2 was very much higher than today. That’s where the carbon now in coal came from. But I still don’t understand what this has to do with anything. What point are you trying to make?

    “My query has always been “What mechanism explains the glacial/interglacial temperature differences?””

    There is an awful lot of information on this site that explains that. The “FAQ” on this site has a question “What Caused the Ice Ages and Other Important Climate Changes Before the Industrial Era?” which explains.

    And the IPCC reports are freely available online. WG1 Report will explain all this, and provides references.

    Or use google.

    “The Earth may be warming, may be cooling, may be about to lurch one way or the other next week/month/year.”

    No. The earth does not ‘lurch’. All the data shows that changes between states are gradual and take hundreds or thousands of years.

    But yes, the natural cycle will take us into cooler and warmer cycles. The fact that there is a natural cycle has absolutely nothing to do with this argument whatsoever, however.

    “Could you please me know what will happen if CO2 reaches 1000 ppm within the next two years?”

    Not physically possible. The amount of coal and trees you’d have to burn is immense. Even if you deforested the entire global in two years, it still wouldn’t reach 1000ppm.

    However, what I can say is that (all other things staying equal) if CO2 steadily increases this century and reaches 1100ppm and then stops increasing but doesn’t go down, global mean temperature will increase by around 6 degrees. I think I’m correct in saying it’s 95% likely that the increase would be in the range 3 degrees to 9 degrees but I’m happy to be corrected if I’ve got the confidence interval wrong. I also think I’m correct in saying that there’s a lag, and you wouldn’t see the full temperature rise until about 50 years later. But it would be irreversible (excepting nuclear winter) at that point.

    “I await your guidance”

    I hope that was useful.

    Trying to predict what is actually on your mind is of course impossible, but allow me to point out a couple of other things.

    1 – AGW theory IS NOT based on observed temperature increase since the 1980s. Scientists DID NOT see an increase and come up with a mechanism. The mechanism was predicted BEFORE the temperature increase was observed. And you can quite easily show (based on a set of OBSERVATIONS that are reproducible and independent) that climate sensitivity is 3 degrees without using a climate model, or ANY temperature data from the last 30 years.

    2 – Please, please, please don’t go down the route of questioning whether the increase in atmospheric CO2 observed since 1750 is caused by man. It is. No serious scientist, not even the most skeptical of climate skeptics, denies this. It’s as factual as “HIV is a virus that attacks the immune system” or “the earth orbits the sun”. Again, IPCC AR4 WG1 explains how we know this. http://www.ipcc.ch

    Comment by Silk — 3 Dec 2009 @ 6:12 PM

  357. You can access australian climate data here
    http://www.bom.gov.au/climate/data/weather-data.shtml

    Comment by HR — 3 Dec 2009 @ 6:17 PM

  358. Thanks to Hank, jyyh, Mike and CM for your responses.

    I hadn’t heard of the Climactic Optimum until I read the summary on the NGRIP site. I see it’s also referred to in the temperature chart which Hank referred me to: http://www.globalwarmingart.com/wiki/File:Holocene_Temperature_Variations_Rev_png .

    Were temperatures (at least in the Northern hemisphere) warmer than today approx 5,000 years ago? Does this mean we should be less worried by e.g. potential for significant methane release from Siberia if this happened in the relatively recent past?

    ps does anyone know why it’s called the Optimum? I sometimes wonder what the optimum temperature is for the planet. A lot of current debate seems to assume that we’re at the optimum point today and so any warming (or cooling) from that would be detrimental – or maybe it’s just the current speed of change which is the problem.

    Comment by Jonathan — 3 Dec 2009 @ 7:25 PM

  359. Mike Flynn (350) — I recommend beginning with climatologist W.F. Ruddiman’s popular “Plows, Plagues and Petroleum” and then his textbook, “Earth’s Climate: Past and Future”.

    Comment by David B. Benson — 3 Dec 2009 @ 7:30 PM

  360. Jonathan, what matters is the rate of change.

    Look at those long-term temperature charts — the increase in the past century goes straight up, at the right margin. It’s so steep it’s almost invisible.

    If the increase in CO2 were slow, natural processes would take care of it, plants would migrate to more appropriate temperature bands over time, bird migrations cued by temperature would change over generations by slow natural selection.

    http://www.ucar.edu/learn/1_4_1.htm

    http://www.eea.europa.eu/data-and-maps/figures/rate-of-change-of-global-average-temperature-1850-2007-in-oc-per-decade-1

    The rate of temperature change in the 20th century is four times greater than the average rate of change over the previous four centuries. …
    http://www.usgcrp.gov/usgcrp/seminars/980112DD.html

    Comment by Hank Roberts — 3 Dec 2009 @ 8:19 PM

  361. @Kevin McKinney. Thank you for your helpful suggestions (unlike some).

    Your first link doesn’t do me a lot of good. I have difficulty taking concrete actions based on “may have caused”, “have suspected”, “reinforced the notion”, “scientists believe”, “strongly suggests”, “the likely cause”, and “in the Himalayas, the process would have been the same.”

    Follow those up with this : “The rise and subsequent weathering of the Himalayas may have caused our current Ice Age, the one that began 40 million years ago.” MAY?

    I read your review. The best part for me, obviously, was your statement (relating to some prediction or other) ” . . . it is not possible to predict, which if any of these things could happen, or when.” My point exactly. I couldn’t have said it better. Maybe I am starting to understand science after all.

    The author of the book should reread Lorenz, Mandelbrot and Feigenbaum. He might be a little less certain about declaring that he can make accurate predictions in the long term, while at the same time the near future is a mystery to him. Maybe I misunderstand chaos as it relates to dynamic (non linear?) systems.

    On to the next –
    http://geoflop.uchicago.edu/forecast/docs/Projects/hubbert.doc.html

    “Hubbert
    This is not really a mechanistic model at all, but a curve fit. Marion King Hubbert proposed bell curves as good fits to the rate of oil extraction. Hubbert forecast, back in 1956, that U.S. production would peak between 1965 and 1972. Test his forecast, and make your own for global oil production.”

    Comment: I like this one. It leads to a button called “Tell the future”. Obviously no real scientist would lay himself open to charges of fortune telling, so it must be a joke.

    However, persevering, I change the peak year to 2010, and opt for World Oil Production. Gee whiz! Quel surprise! The “prediction” and the data are not even in the same ballpark, as the Americans say.

    So once again, sorry. If the first thing I go to is useless to me, I look elsewhere for some facts. I’m not sure what you do – maybe the same as me. Or like some people, keep hitting the ESC key in the hope that it may work eventually.

    Please don’t take offense. You and I obviously have the same thoughts. I just have them more often and more consistently.

    @Barton Paul Levenson.

    Well, I tried.

    Google it yourself and than tell me that Milankovich cycles fully explain glacial/interglacial cycles. And then explain away the periods that don’t fit. You are not allowed to use the words “maybe”, “probably”, possibly”, “if”, “may”, “might”, “suggest”, “believe”, “notion” or any other imprecise language. After all, we are supposedly talking about the past. It is fixed, no ifs buts or maybe. Just because you can’t explain it, doesn’t mean it didn’t happen.

    I don’t wish to annoy you, but I have interpreted your suggestions as supercilious throwaways, taking advantage of my admitted lack of education. Just because I’m uneducated doesn’t necessarily mean that I am stupid. But I might be. Surely it is easy enough for you to peer into the future, and tell me what the average temperature will be In Darwin Australia for the year 2015. I can then decide to stay, or flee to Tasmania.

    Hartmann lost me a little when he made the statement ” . . . this basic mechanism must explain why. . .”.

    MUST? Are there no other explanations? I like my authors to back up their statements. He may well have a reason, but his phraseology leads one to think “I wrote it, therefore it is.”

    On to Houghton.

    Unfortunately, at around page 3, the author discovers that his calculations don’t seem to fit Jupiter very well. He can only account for half of the energy. The answer is simple. “. . . The other half therefore, must be internally generated”. And the other planets, like Earth don’t generate heat internally? Not what I’ve observed.
    I pressed on. I stopped after this statement – “Since there are dynamical constraints on the motions which may occur in the atmosphere . . . it is not clear that processes for attaining such a reference state are dynamically possible. In fact . . . it may be more realistic . . .”

    And no, I didn’t work all the problems. If “. . . it is not clear . . .”, I have better things to do with my time.

    Now I notice that there have been rather extreme climatic events, both cooling and warming, which seem to occurred very quickly, in years rather than eons. It seems (and I’m a believer, I guess), that the Younger Dryas cooling occurred within a decade. Temperature drop, 5 deg C, maybe more. According to NOAA – GISP2 – there was a rapid warming – about 7 deg C. Other sources (which appear to be scientific to me) give a shorter time period for the change, and a greater temperature swing.

    I mean no offense to anyone, but I still maintain the future is unknowable. The past is gone. Whether the “data” is good, bad or indifferent is most likely irrelevant.

    @Silk.

    Sorry. Just noticed your post now.

    In my own bumblefooted way, I was trying to suggest that the earth has been far warmer in the past, and the atmosphere contained a much greater percentage of CO2. I thought this was relevant inasmuch as there seems to be alarm about the Earth warming (whatever the cause). So I tried (not successfully) to point out that increases of temperature in excess of those which seem to cause you concern (3 to 9 deg C?) may be “natural”.

    I am sorry about the “lurch”. I meant a sudden change in climatic conditions, such as (fortuitously enough) as may be seen in ice core research. How about a 7 deg C rise in less than 10 years? Sudden enough? And, yes, I realise the whole Earth may not have warmed this much this quickly, but who knows? Not me, and not you, I warrant.

    As for “Not physically possible.”, I wonder how it was possible for CO2 levels to be a lot higher in the past. But I accept your assertion.

    If you are right, and global mean temperature rises by 6 deg (Celsius I’m guessing) what then? Do we all die? Your guidance is not useful.

    Now your points.

    1. I’m confused. “The mechanism was predicted BEFORE the temperature increase was observed . . .”? What mechanism ? There have been a lot of theories about the atmosphere in the last few thousand years. To which particular one do you refer?

    I am not sure what “. . . climate sensitivity is 3 degrees without using a climate model, or ANY temperature data from the last 30 years.” means. Are you saying that global temperatures have shown a rise of 3 degrees which correlates exactly to CO2 rises, during the period to 1980? Please explain.

    2. I am quite happy to go down the route of questioning whether the current changing climate owes its existence to man, nature or a combination of both. I am, as you are probably aware, not a scientist. Facts are not decided by democratic vote, or articles published in learned journals.

    Next you’ll be saying that no serious scientist would challenge Newton, or Einstein, or Lorenz . . . In fact many did. Thank goodness!

    You must have read a different IPCC AR4 WG1 than I did.”Large uncertainties remain in many issues. . .”,” . . . model”, “. . . model. . .” “. . . projection. . .”. Bald statements “The Earth is a sphere.” No it ain’t. Not important? If you say so. If this is the best the IPCC can come up with, I understand why some scientists are less than impressed.

    We don’t even know what we don’t know!

    @David B Benson.

    Thanks. I remember reading “Plows plagues and petroleum.” I thought it was “Ploughs”, but I won’t fight you over it. An interesting read.

    Earth’s Climate? I think I will back my bucket of chicken entrails against his predictions of global temperature rises. Have quick look at all the question marks on some of the graphics. Making predictions is difficult, particularly where the future is concerned.

    @All. Sorry about the length of the post. Kind regards. Live well and prosper.

    Comment by Mike Flynn — 3 Dec 2009 @ 8:49 PM

  362. Jonathan (358) — In that usage, “optimum” is an old-fashioned term for local maximum. The optimum global temperature for humans is that which is least likely to cause much sea rise (or fall) and promote the best growing seasons on the most productive soils. I estimate that temperatures prevailing about 1900–1920 ought to be our goal.

    In some localities it is abundantly clear that temperatures are now higher than over 5000 years ago, for example
    http://researchnews.osu.edu/archive/quelcoro.htm
    but various conditions in Siberia were clearly much different so long ago (no black carbon, few people, more wild animals, …) so I suggest treating the recent research news about Siberian methane releases rather soberly.

    Comment by David B. Benson — 3 Dec 2009 @ 8:52 PM

  363. theford — the first link under Science will get you to the answers you’re asking for. Basic reading — you’re asking for words, which are only approximations; it took big computers to work out the detail you’re asking about.

    Short answer (remember this is just words): lower in the atmosphere most of the interaction between gases is by collisions which occur far more often than a photon gets emitted and absorbed (by those that can do that, the ‘greenhouse’ gases). In the upper atmosphere, molecules are so much farther apart that it’s more likely one will interact with a photon than with another molecule. This changes a whole lot of things.

    —-
    Context: well, the anthropologists too have their differences expressed in strong words:

    http://chronicle.com/article/Rebuttal-of-Decade-Old/49320/

    Comment by Hank Roberts — 3 Dec 2009 @ 9:17 PM

  364. Ron Broberg (330), Kark K (333), dhogaza (334), I fully agree that the skeptic who has a well-developed alternative theory is more likely to be more productive and helpful to the cause, but, to the original point, it is not necessary to be a skeptic. Though I would further agree that a skeptic ought to be able to articulate at some level his rational which requires some level of understanding but far short of full comprehension. Nor should it be a broad brush. “La-la-la I don’t believe you!” is not sufficient; while dhogaza claims that is all I say, in fact he has never heard me say that.

    Gell-Mann thought the skeptic/contrarian/iconoclast was the most important member in the science community, though he understood they were seldom successful.

    Gavin pretty much covered it in another thread (163 – CRU Hack-3). He’s correct when he blames some of the “skeptics” themselves because some can be dorks — as can some AGW proponents. Of course SecularAnimist, in the same thread (184), takes it to ridiculous extremes

    Comment by Rod B — 3 Dec 2009 @ 10:58 PM

  365. (sorry if this appears twice – delete away)

    Mike Flynn,

    It is hard to make recommendations when you dismiss a science text for containing the words “may” or “might”. When you don’t accept “must” either, you’ve covered all your bases. You need to work on your problem with modal verbs before anyone can help you with the science.

    PS. Barton is just extending the courtesy to others of believing them to be as capable and desirous of educating themselves as he is. I for one am not, but I also don’t call people supercilious for offering me reading tips.

    Comment by CM — 4 Dec 2009 @ 5:19 AM

  366. Mike Flynn:

    I have interpreted your suggestions as supercilious throwaways, taking advantage of my admitted lack of education. Just because I’m uneducated doesn’t necessarily mean that I am stupid.

    No. But it is stupid to militantly refuse to learn. For example, you assume that because Jupiter and Earth generate internal heat, the amount should be the same. It isn’t, by many, many orders of magnitude, because completely different processes are involved. Jupiter is the largest gas-giant planet in the system, Earth is the largest terrestrial planet. The difference in their masses is a ratio of 317.9. On the other hand, Earth receives 27 times more sunlight than Jupiter does, which makes a difference to the relative proportions and would even if Jupiter DID generate only as much internal heat as Earth does. Earth is made mostly of rock and metal, Jupiter is made mostly of gaseous and liquid hydrogen and helium. That makes a difference too.

    I did not advise you to flip through the books, find statements that didn’t make sense in the state of your admitted lack of education, and abandon the project. I advised you, if I wasn’t clear before, to READ the books AND WORK THE PROBLEMS. DO THE MATH. If you don’t have enough math, read up on that first.

    Comment by Barton Paul Levenson — 4 Dec 2009 @ 5:49 AM

  367. Jonathan,

    yes, it’s the rate of change, but it is also about the absolute temperatures we are headed for in the near future, which will exceed the band in which we developed human civilization. Hence the worries about the infrastructure and agriculture we have developed to support a human population of nearly 7 billion and counting. We have a lot more resources to deal with climate change today than our Neolithic ancestors did. But we also have a lot more people to support, it’s harder for us to up stakes and move, and there’s not that many places to go any more, whether for us or for other species.

    On “optimum”, there’s a tendency to avoid the term these days as being value-loaded, as well as for suggesting a single general experience. The IPCC cites evidence for /local/ periods warmer than the last decades in the early to mid-Holocene, but not all at the same time, and possibly offset by cooler tropical oceans in some parts. Modeling of the climate 6,000 years ago suggested the global mean temperature did not change much. Hence expressions such as the “mid-Holocene thermal optimum” are not “globally relevant”, they said.

    On this basis I’d guess that Holocene temperature reconstructions do not provide much of a guide to the future of methane locked up in Siberia. (I’d prefer being wrong, though.)

    There are some older posts on RC where David Archer lays out what may or may not happen with methane hydrates and the methane cycle in general, with the disturbing conclusion that for all we know about the methane cycle we don’t have a clue about the near-term future.

    Comment by CM — 4 Dec 2009 @ 7:27 AM

  368. @Barton Paul Levenson.

    I assumed no such thing, and I would be most grateful if you could substantiate your bald assertion.

    I accept that you see textbooks (or at least this one, as gospel.) I don’t.

    You will notice that the author makes some pretty awe inspiring assumptions, that we are supposed to accept uncritically. Say for example, the effective albedo of Jupiter, at the wavelengths relevant to a discussion of the greenhouse effect, was 0.343 +/- 0.032 rather than 0.58, (I’m not saying it is, mind you), you might find that the projected temperature at the cloud tops is different. And, yes, the author does indicate that although the column is titled “Mean surface temperature”, in the case of Jupiter, it isn’t the mean surface temperature at all.

    The author accepts that the processes involved in atmospheric behavior are not clear. I agree. He is doing his best to provide a base for students to proceed from. I may be wrong, but you know more than I about the author.

    As to your advice, respect is earned, not demanded as far as I am concerned. I merely stated I would like to know how the atmosphere works. So, I believe, would other experts in the field. It appears nothing much has changed since the last time I was curious about the workings of the atmosphere, a longish time ago.

    This was written a few years ago by R Davis of the Berkeley Wireless Research Center : -

    “Henon’s strange attractor allowed explaining and predicting the chaotic motion of stars. Unfortunately, verifying the hypothesis would take several billion years of experimentation.”

    What’s the relevance to the current discussion about climate change? Well, previous assumptions about physical behaviour on both the macro and micro scale based on Newtonian or Einsteinian (I like that one!) physics may well be wrong. If the behaviour of the atmosphere is chaotic, in the sense of sensitive dependence on initial conditions and so on, then accurate predictions as to the state of the chaotic system at any point in the future are probably impossible.

    So pardon me if I take your advice under advisement. I have done enough mathematics in the past to assume I am not going to learn anything fresh from the problems set. The mathematics involved is pretty basic, you would agree?

    @CM

    You’re right. My point exactly. I have no problem with all the “waffle words” if the user also allows that the opposite is equally valid. As in ” . . . it is not possible to predict, which if any of these things could happen, or when.” Some of the assumptions made in science texts are staggeringly impressive. Texts change all the time, as advances occur. Science changes all the time. I used to believe, along with all the medical scientists, that there were various reasons for gastric ulcers, bacteria not generally recognised as being of any significance.

    We were all wrong. Helicobacter pylori was discovered. The discovery was rejected by the majority of scientists, until the proof was incontrovertible. I changed my thinking.

    As to modal verbs, I understand “must” implies certainty, or near certainty. Now, “may”, “might”, “possibly”, express the writer’s confidence based on the writer’s assumptions of fact. You and I may differ on the use of modal verbs, but I am intrigued by the notion that you assume that I have a problem with modal verbs. I don’t believe I do, but I am willing to be proved wrong.

    I like your certainty in your response to Jonathan (#367) where you say the temperature in the near future “. . . will exceed the band . . .” You then go on to “guess”, and then finish up by referring to David Archer. I would take advice from David Archer if you quote him correctly, as he apparently lays out what “may or may not happen”, and that we “don’t have clue about the near term future.” I don’t necessarily have to accept anything else he says that contradicts his thoughts as stated by yourself.

    So. Where does that leave us? You hold a strong unprovable opinion about the future (in the sense that you cannot test the truth of the hypotheses until you have compared the predicted results with observation), whereas I am not as sure.

    As I don’t believe that you can make meaningful and useful “predictions” based on the past any better than me with or without a bucket of chicken entrails, you haven’t convinced me that I should alter my behaviour one iota.

    The reverse is probably true. Maybe, might be, could be – take your pick.

    I’ll let you get back to trying to figure out the past. Once you can reliably predict the past (it should be easy to compare the prediction with the observation, as it has already happened), I will give serious consideration to your ability to predict the future with more certainty than I.

    @moderator.

    Sorry for taking up your space and time. For me, the scientific method is observe, think, theorise, “predict”, observe. Seems a lot of people stop at the “predict” stage, and say “Well, there you are. Proven”.

    @All

    Regards. Live well and prosper.

    Comment by Mike Flynn — 4 Dec 2009 @ 9:17 PM

  369. Is there a good writeup about Sulfur Hexafluoride (SF6) that’s accessible by ordinary people? Supposedly 1 lb of SF6 has the same global warming potential as 12 tons of CO2. That means 1 molecule of SF6 has more global warming potential than 100,000 molecules of CO2. I’m interested in what this means exactly, and how this has been measured. I am confused about what “global warming potential” means, since they give different values for different numbers of years.

    My understanding is there is a heat-trapping effect, and then a half-life in the atmosphere, and some calculation involving the two yields the global warming potential. How the heat-trapping effect is measured, and how the half-life is measured under realistic environmental conditions would be interesting.

    Your data section is very helpful.

    Comment by Peter Baldo — 5 Dec 2009 @ 1:12 AM

  370. #361

    “In my own bumblefooted way, I was trying to suggest that the earth has been far warmer in the past, and the atmosphere contained a much greater percentage of CO2. I thought this was relevant inasmuch as there seems to be alarm about the Earth warming (whatever the cause). So I tried (not successfully) to point out that increases of temperature in excess of those which seem to cause you concern (3 to 9 deg C?) may be “natural”.”

    It is in noi way disputed that the earth has had many different states. So in that sense CO2 at 1000ppm in the past was ‘natural’

    However, there were no humans around back then.

    The issue we are dealing with today is

    a) Is human activity likely to cause CO2 to rise a lot – Answer “Yes, this is already happening”
    b) Would it be more damaging to humans to let CO2 rise, or to take steps to stop the human activities causing the rise – Answer “The impact of rising CO2 on HUMANS will be very bad and it would be much better to avoid that rise by investing in emissions reductions” For more on this, see the Stern Review.

    “I am sorry about the “lurch”. I meant a sudden change in climatic conditions, such as (fortuitously enough) as may be seen in ice core research. How about a 7 deg C rise in less than 10 years? Sudden enough? And, yes, I realise the whole Earth may not have warmed this much this quickly, but who knows? Not me, and not you, I warrant.”

    I’m no climatologist, but I think it could be shown that the amount of energy required to do this in such a short period of time (think how much energy you need to heat the oceans) could not be provided.

    I’m not aware of an ice core research that suggests 7 degree rises in 10 years. Perhaps you could direct me?

    “As for “Not physically possible.”, I wonder how it was possible for CO2 levels to be a lot higher in the past. But I accept your assertion. ”

    It is possible for CO2 to rise to very high levels OVER TIME. It is not physically possible to do this in two years, because of the time contraint. In much the same way as it is physically possible for me to drink 10 pints of beer, but not physically possible for me to do that in 3 seconds.

    “If you are right, and global mean temperature rises by 6 deg (Celsius I’m guessing) what then? Do we all die? Your guidance is not useful. ”

    IPCC Working Group 2 sets out what is likely to happen. http://www.ipcc.ch Summary for policymakers will sort you out.

    “1. I’m confused. “The mechanism was predicted BEFORE the temperature increase was observed . . .”? What mechanism ? There have been a lot of theories about the atmosphere in the last few thousand years. To which particular one do you refer?”

    That increasing GHG concentrations in the atmosphere will cause increased global temperatures.

    “I am not sure what “. . . climate sensitivity is 3 degrees without using a climate model, or ANY temperature data from the last 30 years.” means. Are you saying that global temperatures have shown a rise of 3 degrees which correlates exactly to CO2 rises, during the period to 1980? Please explain.”

    No. What I’m saying is that the paleoclimate data shows that past temperature and CO2 trends are consistent with an earth where, if you double CO2 in the atmosphere, temperature goes up 3 degrees. To clarify further, this data IS NOT consistent with an earth where, if you double CO2 in the atmosphere, temperature goes up by LESS THAN 1.5 degrees or MORE THAN 4.5 degrees (to 95% certainty)

    This is no way relies on global temperatures since 1980 (but, of course, is consistent with observed temperatures. If it were not consistent, we’d have a problem)

    “2. I am quite happy to go down the route of questioning whether the current changing climate owes its existence to man, nature or a combination of both. I am, as you are probably aware, not a scientist. Facts are not decided by democratic vote, or articles published in learned journals.”

    Facts are decided by articles pubished in journals. How else are they decided?

    I am NOT suggesting you cease to question climate change. What I SPECIFICALLY suggested is that you do not question whether increasing CO2 in the atmosphere is due to human activity. If you do question it, we will eventually arrive at the situation “Yes, it is” but it will waste our time. I’d ask you to take that on trust, or (if you really want to get into it) read the stuff on Carbon Cycles in WG1 report of the IPCC.

    “Next you’ll be saying that no serious scientist would challenge Newton, or Einstein, or Lorenz . . . In fact many did. Thank goodness!”

    No. I wasn’t saying that.

    “You must have read a different IPCC AR4 WG1 than I did.”Large uncertainties remain in many issues. . .”,” . . . model”, “. . . model. . .” “. . . projection. . .”. Bald statements “The Earth is a sphere.” No it ain’t. Not important? If you say so. If this is the best the IPCC can come up with, I understand why some scientists are less than impressed.”

    This is garbage. You’ve pulled out a few phrases and quoted them without any context. I’m afraid I can’t respond to that. If you have some specific criticisms, please level them and I can address.

    “We don’t even know what we don’t know!”

    Could you go into more detail?

    We /know/ that temperature will go up by between 1.5 and 4.5 degrees Celsius if CO2 goes to 550ppm or higher.

    We /know/ this BASED ON OBSERVATION with, AND WITHOUT, models.

    We /know/ that human activity is increasing CO2 in the atmopshere and this will continue (indeed accelerate) in the current economy.

    We /know/ this will be damaging to humankind.

    We /know/ it doesn’t stop there and that CO2 at 650ppm would be worse etc. etc.

    What do we /not/ know, in order to make informed policy decisions about mitigation?

    I’d be very keen for you to answer that last question.

    Comment by Silk — 5 Dec 2009 @ 6:05 AM

  371. “Sorry for taking up your space and time. For me, the scientific method is observe, think, theorise, “predict”, observe. Seems a lot of people stop at the “predict” stage, and say “Well, there you are. Proven”.”

    I’ve no idea what other sciences do this, but this is certainly not a valid representation of climate science.

    For example, http://bartonpaullevenson.com/ModelsReliable.html sets out SEVENTEEN (count them!) observations that validate climate models.

    I don’t know how much is spent on climate observation globally, but it’s a lot (those satellites aren’t cheap!) and ALL that data is constantly being analysed to check, refine and improve climate science.

    There is ZERO inconsistency between the following two statements

    1 – “We know enough about man’s impact on climate to know we have to act to reduce emissions in order to avoid significant damages to humankind”

    2 – “There’s a lot more to know, and we should spend more time, efforts and money on collecting and analysing data, checking the models and doing new science”

    1 is why people are in Copenhagen next week.

    2 is why there are, right now probably, people in a bog, or glacier, or lakebed, taking samples, working them up, etc.

    Comment by Silk — 5 Dec 2009 @ 6:14 AM

  372. Mike Flynn:

    If the behaviour of the atmosphere is chaotic, in the sense of sensitive dependence on initial conditions and so on, then accurate predictions as to the state of the chaotic system at any point in the future are probably impossible.

    BPL: Sorry, that does not follow. Which hand of bl ackj ack will turn up next or how the ro ull ette ball will land is absolutely unpredictable for all practical purposes, yet ca si nos stay in business anyway.

    Climate is not an initial-values problem. It is a boundary-values problem. Weather is an initial-values problem.

    So pardon me if I take your advice under advisement. I have done enough mathematics in the past to assume I am not going to learn anything fresh from the problems set. The mathematics involved is pretty basic, you would agree?

    BPL: See, this is what I call militant ignorance. You _refuse_ to learn anything more about the subject, because you think you know enough already.

    Is the math in Houghton pretty basic? Yes, if you confine yourself to the first chapter, which you apparently did. Later on he uses such things as vector calculus. I don’t expect you to know that, but it would nice if you went as far as you could with the problems before bailing. But I don’t expect you to do that, either. I expect you to maintain your present views no matter what, and lecture people about it who actually know far more than you do. Kind of like a guy who has never put two pieces of rock together in his life going up to a 20-year union stonemason building a wall, and telling him, “You’re doing that all wrong!”

    Comment by Barton Paul Levenson — 5 Dec 2009 @ 11:28 AM

  373. An additional source for CCSM (and the older PCM) model run output is the NCAR portal for the Earth System Grid, at

    http://www.earthsystemgrid.org

    All the CCSM and PCM AR4 runs are there (including many more fields than were requested for the AR4) as well as many more runs.

    Comment by Gary Strand — 5 Dec 2009 @ 3:41 PM

  374. @ Silk

    Rising levels of CO2 = more abundant plant life = more food.
    If rising CO2 levels = rise in temperature, then permafrost regions become fertile again. The temperature will have to rise by a lot more than current forecasts to achieve this. Vast regions of the Earth’s surface become much more amenable to human occupation. If you wish to go back to the “Dark Ages” (called that for a very good reason), be my guest.

    As to the rapid warming, I may have been a little conservative. It now appears that the temperature rise could have been as much as 10C +/- 4C, and in as little as a year. I refer you to the various ice core studies, in particular GISP 2 can be researched at NOAA ( I hope this is reputable enough for you.) If this is fact, then you may need to redo your energy distribution calcualtions. I haven’t seen your workings, so I can’t be of much help.

    IPCC. “Virtually certain – Agriculture etc – Increased yields. ” As I said.”Human health – Reduced human mortality.” I had forgotten that – I’m old. Sorry. “Industry etc. – Reduced energy demand for heating. Reduced disruption to transport.” Oh well, I’ll stop cherry picking.

    I notice your prediction of temperature rises due to GHG (I’m guessing GHG means CO2, I hope it doesn’t mean Global HydroCarbons, otherwise we’d all be in deep trouble) is based on ” . . . consistent with an earth where, if you double CO2 in the atmosphere, temperature goes up 3 degrees.” Fine but that must be another Earth. This Earth has had CO2 levels around 7000 ppm down to about 200 ppm, without the associated variations in temperature. You may well have an explanation, but you didn’t tell me. If you keep me in the dark, how can you lose?

    Facts are decided by journal publication? Care to comment Gavin? How about making it simple – facts are now to be decided by democratic vote.

    Where did I question rising CO2 levels were due to human activity? Don’t believe I did. May have said there was a chance that global warming was natural.

    May I ask you what aspects of climatological prediction are poorly or incompletely understood? Are there any other factors affecting climatological predictions of which we may be unaware? Oh, you don’t know?

    Now to your last :

    Until a scientist has observed something, he doesn’t know. If the observation doesn’t coincide with the theory,
    - new theory = scientific progress.

    What do I not know? How to predict the future. You can, I accept. Woe, woe, thrice woe!

    Your version of the scientific method is yours. Good luck with it.

    I am glad that you accept that a lot of money is being spent to make observations, check the models, and improve the science. Hopefully, they will find out if the current models work or not, and whether the “new science” shows them what they didn’t know under the “old science”. I agree totally.

    Politicians are in Copenhagen. I am not sanguine about politicians’ ability to foresee the future. Based on past performance, you wouldn’t bet your life on it. Canute’s advisers convinced him he could hold the tide back, too.

    Oh well.

    @Barton Paul Levenson.

    What has black – jack or roul – ette got to do with the science of chaos? You are no doubt aware that it was Lorenz
    who kicked off the study of chaos. Not applicable to climate? -
    1976 Nondeterministic theories of climate change. Quaternary Research. Vol.6 -guess who wrote it. You guessed it!

    And a heap of others – published in peer reviewed journals, I might add.

    Vector calculus (and yes, I have had more than a nodding acquaintance with it), is old hat, but useful nevertheless, for some things. You might like to tell me what else I don’t know. In spite of my lack of formal education, I have lectured at University level, amongst other interesting things.

    As I have demonstrated before, if somebody produces facts to prove me wrong, I change my view. What do you do?

    I am not lecturing anyone. I apologise if it seems that way.

    Hopefully enough.

    Live well and prosper.

    Comment by Mike Flynn — 5 Dec 2009 @ 9:55 PM

  375. Alfio Puglisi says (28 November 2009 at 3:49 PM)
    Gavin, comment #89 by Silke is a copy&paste of this page on Roy Spencer’s blog: http://www.drroyspencer.com/2009/11/my-top-10-annoyances-in-the-climate-change-debate/
    [Response: Thanks for spotting. - gavin]

    Does this somehow invalidate the points?

    Comment by Tuomo — 5 Dec 2009 @ 10:27 PM

  376. “Rising levels of CO2 = more abundant plant life = more food.
    If rising CO2 levels = rise in temperature, then permafrost regions become fertile again. The temperature will have to rise by a lot more than current forecasts to achieve this. Vast regions of the Earth’s surface become much more amenable to human occupation. If you wish to go back to the “Dark Ages” (called that for a very good reason), be my guest. ”

    Not sure what your “dark ages” comment is refering to, so I’ll ignore.

    The rest of this statement falls down at the first hurdle. Rising CO2 DOES NOT increase food productivity in areas were CO2 is not the contraining factor.

    The best available science (i.e. something better than your guestimate about the impacts of more available land from melting versus increased deserts) suggests that overall global food production will decrease once temperatures go above a certain level (can’t remember the number but you can look it up if you care) and that, if you were unfortunate enough to reach 6 degrees, it would decline very very significantly. Large areas of current fertile land in Africa, the USA, Australia and Europe would become infertile, which would not be offset by increases in Russia and Canada.

    And your point completely ignores that fact that the permafrost is pretty much exclusively in Russia and Canada. Do you think that the Africans, Spanish and Mexicans are just going to be able to move there once their own countries are barren? (Don’t answer that one – Rhetorical question)

    “As to the rapid warming, I may have been a little conservative. It now appears that the temperature rise could have been as much as 10C +/- 4C, and in as little as a year. I refer you to the various ice core studies, in particular GISP 2 can be researched at NOAA ( I hope this is reputable enough for you.) If this is fact, then you may need to redo your energy distribution calcualtions. I haven’t seen your workings, so I can’t be of much help.”

    Will take a look. I’m absolutely sure you are wrong (if you implying that global mean temperature can change by 10 degrees over the period of a year) but I’m happy to take a look and explain why. As to why I’m sure you are wrong – Well, what energy source could provide sufficient energy to increase the oceans by 10 degrees over the course of a year? And what would that energy input do to life on earth. But like I say, I’ll look at it.

    “IPCC. “Virtually certain – Agriculture etc – Increased yields. ” As I said.”

    The IPCC DOES NOT say that. It says that, in some regions, increased yields are virtually certain. And that in others, decreased yields are virtually certain. And that overall, at the more extreme ends of climate change (+5 degrees or less) the overall impact is decreased.

    “”Human health – Reduced human mortality.” I had forgotten that – I’m old.”

    Again, that’s a partial take.

    ““Industry etc. – Reduced energy demand for heating.”

    And increased energy demand for cooling.

    ” Reduced disruption to transport.” Oh well, I’ll stop cherry picking.”

    Indeed, you are. Overall, impacts are negative, especially at the high end of temperature change.

    “I notice your prediction of temperature rises due to GHG (I’m guessing GHG means CO2, I hope it doesn’t mean Global HydroCarbons, otherwise we’d all be in deep trouble) is based on ” . . . consistent with an earth where, if you double CO2 in the atmosphere, temperature goes up 3 degrees.” Fine but that must be another Earth.”

    No. that’s this earth. That’s what the paleoclimate data shows.

    “This Earth has had CO2 levels around 7000 ppm down to about 200 ppm, without the associated variations in temperature.”

    Incorrect. The earth has shown values of 200ppm (ish) up to 2000ppm (ish, I think, not 7000ppm) and the OBSERVED INCREASE IN T is how we KNOW climate change is happening.

    See

    http://julesandjames.blogspot.com/2006/03/climate-sensitivity-is-3c.html

    “You may well have an explanation, but you didn’t tell me. If you keep me in the dark, how can you lose?”

    If we are ‘keeping you in the dark’ why would we have RealClimate? If you read the FAQs, or indeed, the IPCC report I already pointed you to, you could see EXACTLY how we arise at the value of climate sensitivity we do.

    But like I said

    http://julesandjames.blogspot.com/2006/03/climate-sensitivity-is-3c.html

    “Facts are decided by journal publication? Care to comment Gavin? How about making it simple – facts are now to be decided by democratic vote. ”

    Nope.

    “Where did I question rising CO2 levels were due to human activity? Don’t believe I did.”

    I never said you did. I merely asked you not too, assuming you might go down that route. If you weren’t headed in that direction, apologies, my mistake.

    “May I ask you what aspects of climatological prediction are poorly or incompletely understood? Are there any other factors affecting climatological predictions of which we may be unaware? Oh, you don’t know?”

    No. I don’t know. Would you care to explain to me if we know everything about the bevhaviour of metals? Or fuels? Or radioactive isotopes? Or economics? Or disease vectors? Or surgery? Or the mind?

    Yet somehow we manage to practice engineering, and medicine, and everything else, just fine.

    The fact that we don’t know EVERYTHING about climate in no way invalidates climate models.

    “Until a scientist has observed something, he doesn’t know. If the observation doesn’t coincide with the theory, – new theory = scientific progress.”

    Wrong. A scientist NEVER knows. All he has is a theory. He can use his theory to do useful things (for example, I can use Newtonian mechanics to build an arch). Later on, I might find out Newtonian mechanics is wrong. Doesn’t mean my arch will fall down, mind you.

    “What do I not know? How to predict the future. You can, I accept. Woe, woe, thrice woe!”

    Human beings have been able to predict climate for 10,000s of years. Since the advent of agriculture, or before.

    Indeed, virtually all animals can predict climate.

    “Your version of the scientific method is yours. Good luck with it. ”

    I live in a world where you do science to do work. We have very good climate science. We can now put it to work to avoid some very nasty things happening in the future.

    “I am glad that you accept that a lot of money is being spent to make observations, check the models, and improve the science.”

    Jolly good.

    “Hopefully, they will find out if the current models work or not, and whether the “new science” shows them what they didn’t know under the “old science”. I agree totally.”

    Not sure there’s any “new science” happening here. Greenhouse effect has been around for a long time.

    Certainly the observations are consistent with the models, so that’s good. The models look reliable.

    You didn’t answer my main question though. I’d really like you to do so, as it cuts to the quick of the arguement.

    “Given that

    We /know/ that temperature will go up by between 1.5 and 4.5 degrees Celsius if CO2 goes to 550ppm or higher.

    We /know/ this BASED ON OBSERVATION with, AND WITHOUT, models.

    We /know/ that human activity is increasing CO2 in the atmopshere and this will continue (indeed accelerate) in the current economy.

    We /know/ this will be damaging to humankind.

    We /know/ it doesn’t stop there and that CO2 at 650ppm would be worse etc. etc.

    What do we /not/ know, in order to make informed policy decisions about mitigation?”

    Comment by Silk — 6 Dec 2009 @ 2:50 PM

  377. A couple of important boundary layer C data sites:

    FLUXNET (with links to regional flux tower networks):
    http://www.fluxnet.ornl.gov/fluxnet/index.cfm

    Ameriflux:
    http://public.ornl.gov/ameriflux/dataproducts.shtml

    And the JAXA-GOSAT data is due to become publicly available starting next month:
    http://www.jaxa.jp/projects/sat/gosat/index_e.html

    Comment by Jim Bouldin — 6 Dec 2009 @ 3:46 PM

  378. You don’t seem to have the MAGICC model at http://www.cgd.ucar.edu/cas/wigley/magicc/

    Comment by John Hunter — 6 Dec 2009 @ 8:09 PM

  379. It’s a non-issue. Why? Seems both sides have forgotten a wee fact: alternative, non-carbon involved technologies, of which there are many. Al Gore and companies carbon tax should be considered as an unintended incentive. The smart businessman, like Dean Kamen, simply looks at what’s available-”no carbon producing, or meet emission standards? Sure-oh look, there’s all these nifty technologies, virtually all except for hydrogen, that are seriously adaptable to third world education levels, and technology.” I suspect that more liberal warmers will respond with a ‘hell yeah! let’s do it!’ and the hardcore warmers will be less likely. The carbon tax is, well, incidental, in that light. And as far as projected outcomes, with warming? We easily have all the knowledge and tech to that, including water-as well as removing salt from salt water. As well as having many kinds of tech on standby to deal with food shortages. I’m not worried in the slightest, because we can handle some discomfort.

    Comment by bryce ramussen — 6 Dec 2009 @ 8:17 PM

  380. What about the comment that we do not understand why there has been no warming in the last few years and that this is a travesty? This comment was made by one of the leaders in this field of research, correct? How much did the IPCC report rely on the data set from the CRU? If there was much reliance at all, there is a problem because the documentation reveals that this data set would only be regarded as valid by someone with no scientific standards. In my field of research, actions such as the ones revealed in the released documents would produce universal and vocal condemnation and no defensiveness. Why hasn’t that been the case in the climate research community? It seems to me that convincing people that there are plenty of good data out there will require an admission that the CRU data are flawed and that conclusions derived from them suspect. Also, if there is no explanation of the pause in temperature increase, that should be admitted, but I have seen no comments on it from members of the climate research community.

    [Response: The quote about the 'travesty' is taken completely out of context. Trenberth was talking about how we don't know specifically what is happening during a short period of time, and why we need more observations stations. The fact that we can't specifically explain exactly what happened during one ten-year period is not the same as saying that 10-year periods of trends different than the long term average are unexpected. He was not suggesting there was some fundamental problem with our understanding. As we already wrote, you need to read Trenberth's recent paper (here). In any event the myth about 'global warming stopping' is not even true, as explained succinctly in the document at copenhagendiagnosis.org, as well as at several recent posts here at Realclimate (e.g. here and here--eric]

    Comment by Stephen Pruett — 6 Dec 2009 @ 10:15 PM

  381. @ Silk.

    This is a bit like writing a computer program. Never perfect, but you have to stop somewhere.

    Can I point out that increasing available CO2 to most food plants increases yield. Indeed, the concept of increase demands that the present natural CO2 constraint be overridden. This fact is widely used by plant growers around the world (where cost effective) to increase yields. Not really rocket science, but to put it in general terms – more food (mainly CO2) plus sunlight etc.- more carbon based sugars and other compounds which comprise plants. Believe, don’t believe – the plants don’t care, they just grow better.

    The IPCC says it , alright. Like both sides in the warming debate, I just left some bits out.

    With regard to CO2 levels, I won’t bother giving you an authoritative reference (published books, papers etc) because you “think” it can’t be true. I wasn’t there at the time (late Cambrian), and even the authors allow there is uncertainty in the measurements. Have a look, maybe you will change your mind. I don’t know, and I don’t much care.

    I still don’t know what your GHG is, but it’s irrelevant. With regard to the 3 Deg C rise, so what? I notice that poor old James seems to be copping some stick from others in the field. Who’s right?

    You asked me what I don’t know that I don’t know (kind of cool, that). I turned the question round to get you to admit that you (and everyone else), doesn’t know what they don’t know, otherwise they would know what they don’t know, and could then attempt to find find some useful answers.

    If you say people can predict climate, I will believe you. You will no doubt be able to show who predicted the Medieval Warming Period, the Little Ice Age, the Year without Summer, and when the next Ice Age (assuming there is one,) will occur. No? I thought not.

    “2 – “There’s a lot more to know, and we should spend more time, efforts and money on collecting and analysing data, checking the models and doing new science”” Your quote. Note the words “new” and “science”. I thought you meant what you wrote.

    Your question contains a lot of givens

    In spite of that, I believe I can still give you a meaningful answer to your question “What do we not know, in order to make informed policy decisions about mitigation?” Well, assuming your givens, we do not know the consequences of any actions we may take, inasmuch as the cure may end up being worse than the disease.

    A couple of minor examples. All compact flouro lamps contain mercury (almost all, anyway. Certainly those generally sold to the public.) Conservatively, around 4mg. People tend to buy the cheapest CFLs, which have shorter lives than might be expected. So, every 1000 CFLs disposed of = 4 x 1000 mg Hg. Not much, 4 gm.
    Every million, 4 Kg. Liquid mercury, seeping into the groundwater. More every year, as the population increases.

    I remember Minimata Bay. Ever wonder about the Mad Hatter in Alice in Wonderland? Mad as a hatter ring a bell?

    How about we flood the upper atmosphere with aerosols? Agreed as an effective way of overcoming global warming. Given that present aerosols have a reasonably short half life, use something that Brownian motion will suspend indefinitely. Then start praying you haven’t inadvertently brought on the next Ice Age a bit quickly.

    There are many example of “good things” that were soundly science based at the time, but turned out to be “bad things” due to unforeseen circumstances. Believe, don’t believe. The past is fact.

    So. I know there are a couple of points I didn’t cover. Blame it on old age, and a care factor of zero.

    Anyway I applaud your certainty that you can think facts away, and that we are all doomed if something (I don’t know what you propose) isn’t done immediately. In any case, I wonder why what YOU “think” is fact, and what I “think” is rubbish.

    As a last thought, consider this. Pick up any text on climatology. There will probably be a short introduction concerning the Solar System, radiant energy, the planets, and energy balance. Depending on the text, there may well be assumptions and simplifications like these : -

    “The Earth’s orbit is elliptical” – no it’s not. Close, but no cigar.
    “The Sun’s output in x units is y.” – no it’s not. It varies.
    “The Earth’s albedo is a”. – no it’s not. It varies – clouds, absorption reradiation wavelengths, atmospheric composition at any given moment.
    “The average surface temperature of the Earth is b.” – no it’s not, unless you have made a really good guess. Average over what period? What type of average? What observed maximum and minimum did you use? How do you know the extremes recorded were the real maximum and minimum? In any case the average tells very little about the temperatures in Omsk in winter, or Death Valley in summer. If you say it doesn’t matter, I will believe you.

    If each statement is even a little bit out, the cumulative effect can be devastating. Or you could be lucky. Further, as many climatologists agree (not all) that the workings of the atmosphere are not amenable to deterministic algorithms, then it may well be that chaos theory may be more applicable.

    This contains a problem, inasmuch as “proving” for example, the existence and nature of attractors may take a very long time. It is easier to accept that there is a natural state of climate equilibrium, and any disturbances will revert to the mean – however we have no way of establishing what the “mean” should be.

    I do not share your certainty about the future. You may well be right, or not.

    Thanks to all. Live long and prosper.

    Comment by Mike Flynn — 6 Dec 2009 @ 10:42 PM

  382. Kevan Hashemi #343, I looked at your graph and your code (I used to code in Pascal in a previous life), and what can I say… dear oh dear. Do you know what an areal average is? Ever heard of spatial autocorrelation?

    Comment by Martin Vermeer — 7 Dec 2009 @ 2:45 AM

  383. Sea Level data – the source you give is purely British. A better zssource is surely http://www.gloss-sealevel.org/data/

    Comment by Philip Lloyd — 7 Dec 2009 @ 9:02 AM

  384. > if there is no explanation of the pause in temperature increase …

    You haven’t been reading here very long, or looked at any of the older topics, or checked ‘start here’ at the top of the page, or read anything about trends and noise in time series — right? Would you be willing to read a bit? How about Google? See the site search box, top of page?

    > I have seen no comments on it

    What _have_ you been reading? would you trust some guy on a blog if you don’t know how to look things up for yourself?

    Seriously, there is a great thing called the public library, and people called librarians.

    Lending books is not so cial ism, and librarianship is not a subtle form of indoctrination by the conspiracy against freedom.

    But why would you believe me?

    http://www.google.com/search?q=site%3Arealclimate.org+pause+in+temperature+increase

    Memo to self: no more blogging before the coffee perks. Just can’t do it.

    Comment by Hank Roberts — 7 Dec 2009 @ 10:38 AM

  385. Mike Flynn – I’d like to thank you. If you hadn’t pointed out those large temperature increases I’d probablyy have remained ignorant. There is a very good description of them here http://www.realclimate.org/index.php/archives/2006/11/revealed-secrets-of-abrupt-climate-shifts/

    These events are extremely rapid (though I can’t see any evidence for them occuring in a single year) and, most interestingly, they do not lead to global increases in mean temperature, but rather result from a change in ocean circulation.

    So yes, there have been significant local changes in temperature in the past, and yes, these were natural. This in no way invalidates the statement that a increase of more than 2 degrees in global mean temperature would be very dangerous for humankind.

    Comment by Silk — 7 Dec 2009 @ 11:22 AM

  386. Silk (376), a nit pik (maybe) clarification: when and who OBSERVED temperature going up 1.5-4.5 degrees following CO2 concentration rise to 550 ppm?

    Comment by Rod B — 7 Dec 2009 @ 11:36 AM

  387. Incidentally, Mike, I looked at the GISP2 record and the largest temperature increase I saw was 10.4 degrees, which is massive. But occurred over a period of 163 years, not a year.

    The reconstruction at ftp://ftp.ncdc.noaa.gov/pub/data/paleo/icecore/greenland/summit/gisp2/isotopes/gisp2_temp_accum_alley2000.txt

    gives Temperature in central Greenland (degrees C), not global mean temperature.

    Very very interesting, but not particularly relevant to the issue of whether or not we should substaintially reduce our CO2 emissions over the next 30 years.

    Comment by Silk — 7 Dec 2009 @ 1:37 PM

  388. Rod B: when and who OBSERVED temperature going up 1.5-4.5 degrees following CO2 concentration rise to 550 ppm?

    BPL: No one, since it hasn’t happened yet. BTW, when and who OBSERVED you dying when you stepped off the Empire State Building? Maybe you could do that safely. There are no observations to contradict it.

    Comment by Barton Paul Levenson — 7 Dec 2009 @ 5:51 PM

  389. Rod B (386) — Go study PETM:
    http://en.wikipedia.org/wiki/Paleocene%E2%80%93Eocene_Thermal_Maximum

    Comment by David B. Benson — 7 Dec 2009 @ 6:03 PM

  390. @Silk.

    Glad to help. The NOAA data you have is a bit out of date. The cores have been archived, and as you probably notice, researchers are continuing to try to unravel the information the ice cores hold.

    More up to date research seems to indicate that the temperature change MAY have occurred within less than 4 years. I don’t know how or why, but it goes along with recent research that indicates (possibly with more certainty) that glaciation MAY have occurred with equal (or, believe it or not, greater) rapidity.

    With regard to energy budgets, I don’t know. As you noticed, local temperatures are not global means. Have a look at the other ice core studies if you wish. I think the problem is that no one is really sure if the climate has homogeneous “global” effects. Depending on the model, there can be, say, a 5 deg C “global” rise/fall, but a substantial difference in the amount of rise at the poles vs the Equator. Additionally, finding “proof” of the past can be as difficult as finding petrified wood by looking randomly.

    I could search for years in an unsuitable area, and conclude that petrified wood didn’t exist because I personally didn’t find any. I understand the difficulty in reconciling ice core data. Lots of variables, and lots of assumptions. Could be a bit like an ET trying to derive the Earths present climate after landing at Vostok or Death Valley USA. I would say “good luck with that”.

    Maybe someone else can help.

    In any case, I am not dismissing the possibility of AGW. I tend to the “possibility” rather than “virtual certainty” .

    I wish you all the best, and now withdraw my presence. It’s been interesting.

    To all. Live long and prosper.

    Comment by Mike Flynn — 7 Dec 2009 @ 6:05 PM

  391. Mike Flynn says “I am not lecturing anyone.”

    No, you are not. You are bullshitting. And what is more, since that majority of posters here have put in quite a few hours studying REAL climate science, they know it.

    http://xkcd.com/54/

    Comment by Ray Ladbury — 7 Dec 2009 @ 7:08 PM

  392. BlogReader (#137) Please read what is said in post #50. Post #50 clearly states that it does not matter whether the code is in Fortran, C, C++ whatever. What is important is how the algorithm is implemented and whether it was implemented correctly. As has been repeatedly demonstrated, regardless of what algorithm is used, the result science is the same.

    Please make sure when you make a statement

    That’s is 100% correct and is why climate models / code needs to be open and freely available. Without open access to other’s codes how can one find out why the results of two programs differ? How can others expect to become better without looking at each other’s work?

    At least make sure it that it a minimal amount of truth to it. A quick google search shows that NCAR’s Community Climate System Model (CCSM) has been open source since June 1996 (13 YEARS), as was the data.

    Of course multiple models are good, that’s why there are so many of them. What is even better is that all of these different approaches all produce nearly identical results. The Program Climate Model Diagnosis and Inter-comparison was created (www-pcmdi.llnl.gov). That’s why NOAA, NASA, CRU, NCAR etc all have their own analysis tools, but yet they all produce nearly identical results

    Please don’t hold up the linux model of development as something to be aspired to! As both a meteorologist and computer scientist I find linux and OSS software in general to be a bane on my existence. The software written by a Master’s student trying to get the code for their thesis to work 3 days before their final defense is a model of good software practice when compared to Linux/OSS software.

    What you have done is to create a strawman and then attempt to knock it down. You are neither the first and unfortunately probably not the last to try and use this long discredited line of argument

    Comment by mommycalled — 7 Dec 2009 @ 9:04 PM

  393. @381: Off topic: Mercury and compact fluorescent bulbs. If the bulbs are powered by nuclear fission, yes. If they are powered by fossil fuels then there is less mercury released into the air from that burning than there would be to provide an equivalent amount of light from incandescent bulbs.

    Comment by Adrian Midgley — 7 Dec 2009 @ 9:23 PM

  394. BPL (388), Silk said it was OBSERVED. Why are you fussing at me?

    David B. Benson (389), I found nothing in your reference that addressed my specific question.

    Comment by Rod B — 7 Dec 2009 @ 11:19 PM

  395. Right, Rod, he means the proxy has been observed. Now you can quibble about the validation of the proxy. Please cite your sources.

    Or would you like to … no, of course not.

    Comment by Hank Roberts — 7 Dec 2009 @ 11:43 PM

  396. Just in case you hadn’t heard – the UK Met Office have just released (some of) the
    station data used to create HadCRUT. See http://www.metoffice.gov.uk/corporate/pressoffice/2009/pr20091208a.html for press release and links to the data.

    Cheers,
    Dan

    Comment by Dan Hodson — 8 Dec 2009 @ 6:24 AM

  397. As a PhD student in biology, I have witnessed the absurd tactics of creationists throughout my career (they were particularly annoying at U.T. Austin, where I could get them to admit that one of their talking points wasn’t valid, but they’d come back next week and repeat it again).

    I imagine that climate research scientists face aggrivations many times the intensity of the average biologist defending evolution against a creationist. Not only does climatology span many disciplines like evolution (so you have to know chemistry, meterology, geology, physics, astronomy, and biology) which makes creationist shotgun-style attacks very easy to carry out, but the AGW-denier crowd is far better funded, and lobbies the hell out of basically every U.S. politician (whereas the creationist/ID crowd generally has to rely on the “true believer” crowd in politics, i.e., Sarah Palin, instead of the much more persuasive route of simply showering them with money).

    So I can definently understand a climatologist’s defensiveness when a snarky, angry Rush Limbaugh listener (yes, I’ve listend to him enough to hear him claim that because we still have cold spots in winter, global warming must be false) who has no intention of analyzing the data or reviewing the source code or unit tests writes/emails/calls demanding data he or she will never use.

    Comment by BigErn — 8 Dec 2009 @ 9:17 AM

  398. Hank, well, if he meant the we have projected proxies, he should have said that, instead of saying it was OBSERVED (his caps)…

    Comment by Rod B — 11 Dec 2009 @ 11:43 AM

  399. No, Rod,

    You toss the word “projected” in now and, by adding yet another word to misunderstand, increase the possibility of confusing someone who doesn’t know your subtle ways. You’re good at this.

    He meant we have the paleo records — such as the PETM event, much discussed here already.
    But you do too know what he meant.

    Comment by Hank Roberts — 11 Dec 2009 @ 12:05 PM

  400. Hi,

    a lot of station data, some with very long time series, for the Alpine region:
    http://www.zamg.ac.at/histalp/index.html

    Comment by Feli — 11 Dec 2009 @ 5:13 PM

  401. Hank (499), you might have a logical point, but the assertion was that we have OBSERVED the rise in temperature following the CO2 concentration rise to 550 (if I recall his number correctly) — I don’t think that has been done in paleo studies. Even so proxies, projected or not are not the same as OBSERVATIONS — though this might be getting picky.

    Comment by Rod B — 11 Dec 2009 @ 9:50 PM

  402. Rod B, the paleo work is done from observations of proxies: “a window on the past.”

    Would you only use the word “observation” for watching a thermometer? Sure you can rule anything out if you split the hairs finely enough and argue definitions. You know where that leads.

    Even daily records are mostly obtained nowadays via proxies, and the data has to be worked on to get a signal from the noise.

    This is about data sets — all sorts.

    Chad and JL mentioned NCDC with links earlier.

    NCDC has pages on specific subjects, in very straightforward form, along with extensive references and links to the data. Here, for example, is the page on abrupt climate change:

    http://www.ncdc.noaa.gov/paleo/abrupt/references.html

    Comment by Hank Roberts — 12 Dec 2009 @ 9:47 PM

  403. World Monthly Surface Station Climatology, 1738-cont
    http://dss.ucar.edu/datasets/ds570.0/

    Comment by Ron Broberg — 17 Dec 2009 @ 9:38 AM

  404. Hank Roberts says:
    12 December 2009 at 9:47 PM
    Rod B, the paleo work is done from observations of proxies: “a window on the past.”

    Would you only use the word “observation” for watching a thermometer? Sure you can rule anything out if you split the hairs finely enough and argue definitions. You know where that leads.

    Even daily records are mostly obtained nowadays via proxies, and the data has to be worked on to get a signal from the noise.

    By the picky standards evidenced here even a thermometer is a proxy, you’re actually observing the expansion of a fluid not temperature!

    Comment by Phil. Felton — 17 Dec 2009 @ 10:09 AM

  405. Happened on this collection while looking up something else.
    I can’t vouch for its contents in any way.
    It says UPDATED with a blink tag on some items.

    http://climate.geog.udel.edu/~climate/html_pages/archive.html

    Comment by Hank Roberts — 19 Dec 2009 @ 5:54 PM

  406. http://www.mccip.org.uk/summaryaims.html

    The primary aim of the MCCIP is to provide a co-ordinating framework within the UK for the transfer of high-quality marine climate change impacts evidence and advice to policy advisors and decision-makers. In particular, the Partnership will act as the primary focus for the supply of evidence and advice to partners to enable them to individually and collectively plan for the challenges and opportunities presented by the impacts of climate change in the marine environment.
    Relationships with other UK marine monitoring groups ….
    http://www.mccip.org.uk/images/MCCIPrelationships.gif

    Comment by Hank Roberts — 23 Dec 2009 @ 1:45 PM

  407. I am not a climate scientist and have never seen any climate data until today. But I do know how to think.

    A post today on Broadstuff OPEN DATA VS GLOBAL WARMING caught my eye and just seemed suspicious. The poster though he had analyzed the “open data” provided by the Met Office and saw no trend in his graph so concluded “Its an inconvenient truth, but the benefits of Open Data cut many ways, pulling down myths with little respect for whose they are.”

    The graphon his website was a confusing mishmash produced by a site http://ww.geo.me. It is no wonder he could discern no pattern.

    Having never seen a climate chart or any climate data, but with fundamental math skills and the ability to read the simple source material, I was able to come up with a graph that showed what to me was a more obvious trend chart of the anomalies at the Port Elizabeth South Africa land stationin less that an hour.

    This is perhaps the problem when so many in the country lack fundamental analytical skills.

    Comment by david sanger — 25 Dec 2009 @ 12:32 AM

Sorry, the comment form is closed at this time.

Close this window.

1.039 Powered by WordPress