RealClimate logo


The CRU hack: Context

Filed under: — gavin @ 23 November 2009

This is a continuation of the last thread which is getting a little unwieldy. The emails cover a 13 year period in which many things happened, and very few people are up to speed on some of the long-buried issues. So to save some time, I’ve pulled a few bits out of the comment thread that shed some light on some of the context which is missing in some of the discussion of various emails.

  • Trenberth: You need to read his recent paper on quantifying the current changes in the Earth’s energy budget to realise why he is concerned about our inability currently to track small year-to-year variations in the radiative fluxes.
  • Wigley: The concern with sea surface temperatures in the 1940s stems from the paper by Thompson et al (2007) which identified a spurious discontinuity in ocean temperatures. The impact of this has not yet been fully corrected for in the HadSST data set, but people still want to assess what impact it might have on any work that used the original data.
  • Climate Research and peer-review: You should read about the issues from the editors (Claire Goodess, Hans von Storch) who resigned because of a breakdown of the peer review process at that journal, that came to light with the particularly egregious (and well-publicised) paper by Soon and Baliunas (2003). The publisher’s assessment is here.

Update: Pulling out some of the common points being raised in the comments.

  • HARRY_read_me.txt. This is a 4 year-long work log of Ian (Harry) Harris who was working to upgrade the documentation, metadata and databases associated with the legacy CRU TS 2.1 product, which is not the same as the HadCRUT data (see Mitchell and Jones, 2003 for details). The CSU TS 3.0 is available now (via ClimateExplorer for instance), and so presumably the database problems got fixed. Anyone who has ever worked on constructing a database from dozens of individual, sometimes contradictory and inconsistently formatted datasets will share his evident frustration with how tedious that can be.
  • “Redefine the peer-reviewed literature!” . Nobody actually gets to do that, and both papers discussed in that comment – McKitrick and Michaels (2004) and Kalnay and Cai (2003) were both cited and discussed in Chapter 2 of 3 the IPCC AR4 report. As an aside, neither has stood the test of time.
  • “Declines” in the MXD record. This decline was hidden written up in Nature in 1998 where the authors suggested not using the post 1960 data. Their actual programs (in IDL script), unsurprisingly warn against using post 1960 data. Added: Note that the ‘hide the decline’ comment was made in 1999 – 10 years ago, and has no connection whatsoever to more recent instrumental records.
  • CRU data accessibility. From the date of the first FOI request to CRU (in 2007), it has been made abundantly clear that the main impediment to releasing the whole CRU archive is the small % of it that was given to CRU on the understanding it wouldn’t be passed on to third parties. Those restrictions are in place because of the originating organisations (the various National Met. Services) around the world and are not CRU’s to break. As of Nov 13, the response to the umpteenth FOI request for the same data met with exactly the same response. This is an unfortunate situation, and pressure should be brought to bear on the National Met Services to release CRU from that obligation. It is not however the fault of CRU. The vast majority of the data in the HadCRU records is publicly available from GHCN (v2.mean.Z).
  • Suggestions that FOI-related material be deleted … are ill-advised even if not carried out. What is and is not responsive and deliverable to an FOI request is however a subject that it is very appropriate to discuss.
  • Fudge factors (update) IDL code in the some of the attached files calculates and applies an artificial ‘fudge factor’ to the MXD proxies to artificially eliminate the ‘divergence pattern’. This was done for a set of experiments reported in this submitted 2004 draft by Osborn and colleagues but which was never published. Section 4.3 explains the rationale very clearly which was to test the sensitivity of the calibration of the MXD proxies should the divergence end up being anthropogenic. It has nothing to do with any temperature record, has not been used in any published reconstruction and is not the source of any hockey stick blade anywhere.

Further update: This comment from Halldór Björnsson of the Icelandic Met. Service goes right to the heart of the accessibility issue:

Re: CRU data accessibility.

National Meteorological Services (NMSs) have different rules on data exchange. The World Meteorological Organization (WMO) organizes the exchange of “basic data”, i.e. data that are needed for weather forecasts. For details on these see WMO resolution number 40 (see http://bit.ly/8jOjX1).

This document acknowledges that WMO member states can place restrictions on the dissemination of data to third parties “for reasons such as national laws or costs of production”. These restrictions are only supposed to apply to commercial use, the research and education community is supposed to have free access to all the data.

Now, for researchers this sounds open and fine. In practice it hasn’t proved to be so.

Most NMSs also can distribute all sorts of data that are classified as “additional data and products”. Restrictions can be placed on these. These special data and products (which can range from regular weather data from a specific station to maps of rain intensity based on satellite and radar data). Many nations do place restrictions on such data (see link for additional data on above WMO-40 webpage for details).

The reasons for restricting access is often commercial, NMSs are often required by law to have substantial income from commercial sources, in other cases it can be for national security reasons, but in many cases (in my experience) the reasons simply seem to be “because we can”.

What has this got to do with CRU? The data that CRU needs for their data base comes from entities that restrict access to much of their data. And even better, since the UK has submitted an exception for additional data, some nations that otherwise would provide data without question will not provide data to the UK. I know this from experience, since my nation (Iceland) did send in such conditions and for years I had problem getting certain data from the US.

The ideal, that all data should be free and open is unfortunately not adhered to by a large portion of the meteorological community. Probably only a small portion of the CRU data is “locked” but the end effect is that all their data becomes closed. It is not their fault, and I am sure that they dislike them as much as any other researcher who has tried to get access to all data from stations in region X in country Y.

These restrictions end up by wasting resources and hurting everyone. The research community (CRU included) and the public are the victims. If you don’t like it, write to you NMSs and urge them to open all their data.

I can update (further) this if there is demand. Please let me know in the comments, which, as always, should be substantive, non-insulting and on topic.

Comments continue here.


1,074 Responses to “The CRU hack: Context”

  1. 901
    EL says:

    “But what would we have to fear if CO2 and temperature actually increased? … A warmer world is a better world.”

    Red List:
    http://data.iucn.org/dbtw-wpd/edocs/RL-2009-001.pdf

    Because the world is warming faster then biodiversity can keep up.

  2. 902
    s graves says:

    Gavin: You have done an admirable job addressing many of the issues raised by the documents. Is there anything in the emails or other documents that indicates untoward behavior or poor science on the part of those involved with respect to their behavior, the data or processes…or only a few unfortunate instances of poor wording…from your point of view?

  3. 903
    Patrick 027 says:

    Re 887 Anand Rajan KD –

    What point has been fixed that should not have been, and what is being bent around it, exactly?

    Is it your contention that anyone who cares about what happens regarding anthropogenic climate change should be barred from studying anthropogenic climate change in any official capacity?

    If your are proposing that it would be better to let AGW (anthropogenic global warming) continue up to the limits of fossil fuel scarcity vs lack thereof of alternatives + technological development in the absence of AGW-mitigating policy, etc, because adaptation would be less costly than mitigation, then you still have to deal with the inequities involved and an emissions tax or other regulation could still be justified. Anyway, that a particular combination of adaptation and mitigation, etc, is better than the rest has not been taken for granted – there are people who study that subject matter as well; is it a crime for people who study the climate physics/ecology to be aware of knowledge coming from other fields of inquiry?

  4. 904
    Patrick 027 says:

    Re 885 John P. Reisman (OSS Foundation) – nice work!

    Re Howard C. Hayden -
    “A warmer world begets more precipitation.”

    General global average tendency; regional variations are VERY important, as are temporal distributions.

    “All computer models predict a smaller temperature gradient between the poles and the equator. Necessarily, this would mean fewer and less violent storms.”

    Tropical cyclones would do better with smaller horizontal temperature gradients.

    Changes in temperature gradients are not evently distributed across all latitudes and longitudes, and seasons, and is not the same at all heights. It might matter how locations of gradient changes align with storm tracks. We get severe weather in summer when the pole-to-equator variation is least, and in the Northern Hemisphere, the summer temperature variation may increase.

    Latent heating and static stability are important.

    “The melting point of ice is 0 ºC in Antarctica, just as it is everywhere else. The highest recorded temperature at the South Pole is -14 ºC, and the lowest is -117 ºC. How, pray, will a putative few degrees of warming melt all the ice and inundate Florida, as is claimed by the warming alarmists?”

    Temperature change is not evenly distributed. Glaciers flow and ice sheets can be in mass balance when accumulation still occurs in parts.

  5. 905
    John says:

    Hank Roberts,

    this is John, regarding the hockey stick question.

    Your answer assumes that I am way off base, However, Manns private Email seem to indicate that I am a great deal more correct than you are really giving me credit for. Lets hold off on stats 101 suggestions, and talk about this interpretation of the significance of the error bars on the Hockey stick, and the assumption that its creator personally harbors that it in fact is a great deal less certain in its implication than a first glance would indicate, and that one would do well to pay attention to the fact that its mostly based on data so unreliable that it often strays from their projections by several standard deviations at a time.

    >>In one of the pre-interviews they asked about the “Hockey Stick”. I
    >>told them of my doubts about the intercentury precision of the record,
    >>especially the early part, and that other records suggested the period
    >>1000 years ago was warmer. I remember saying that “you must give the
    >>author credit for including the large error bars for that time series in
    >>the figure.” I also specifically said that the most precise record of
    >>century scale precision, Greenland Borehole temps, was very important to
    >>note but that the figure was not in the IPCC. I then looked quickly at
    >>the IPCC reference list and saw the citation of Dahl-Jensen and assumed
    >>that it was at least commented on in the 1000 year time series material
    >>and told ABC as much.
    >>
    >>ABC called back a few days later and said they couldn’t find a reference
    >>to the Greenland stuff in the IPCC discussion of the past 1000 years.
    >>So I read the final version, and ABC was right. I said this was an
    >>omission that should not have happened – and that I take part of the
    >>blame because I had mentioned it at each of our Lead Author meetings.
    >>
    >>Last Thursday night, I was one of the guys flown to NY City for the
    >>taping of the show. There was only one question on this particular
    >>issue (it was even after Stossel had left the room) and I gave much the
    >>same answer as I indicated above (as best as I can remember)- that the
    >>”Hockey Stick” (I don’t think I used the term “Hockey Stick”, and I’m
    >>almost positive I did not mention your name at any point) is one
    >>realization of temperatures but that other data are not included and
    >>that I had thought the “other” data were clearly mentioned in the IPCC,
    >>but weren’t. I mentioned the large error bars (as a credit to you) and
    >>that I was partly to blame for this omission. If they use my remark,
    >>they could slice and dice it to make it as provocative as possible.
    >>
    >>Four of us were taped for almost 2 hours, and from this they will select
    >>about 8 minutes, so I doubt my remarks will make the show. When Stossel
    >>came back in after all was said and done, he said to me that I might be
    >>a good scientist but I didn’t have the emotion and passion necessary to
    >>excite the audience. In one way, that is a compliment I suppose. I
    >>think Pat M. will have a good chunk of air time (I don’t remember
    >>whether he added any comments on the 1000-year time series, but he may
    >>have).
    >>
    >>Whatever is shown, just keep it in context. There is no way a clear
    >>scientific point with all the caveats and uncertainties can come across
    >>in such venues. However, I do agree with Stossel’s premise (though I
    >>don’t know what the piece will actually look like so I may be
    >>disappointed) that the dose of climate change disasters that have been
    >>dumped on the average citizen is designed to be overly alarmist and
    >>could lead us to make some bad policy decisions. (I’ve got a good story
    >>about the writers of the TIME cover piece a couple of months ago that
    >>proves they were not out to discuss the issue but to ignore science and
    >>influence government.)

    [Response: The full context is important. Maybe it seems surprising to you that Christy and Mann could have had a civilised discussion on these issues (both were authors on the TAR chapter). - gavin]

  6. 906
    Benjamin says:

    Gavin – your comment above:

    “This happens in other sciences too – quantum physicists are bombarded with crank stuff – but in no other field are these cranks held up in public as the new Galileos by people with very strong vested interests in the status quo and who have access to very effective media conduits.”

    Your comment policy linked in the sidebar says not to attack people’s motives. Does this not apply to the contributors?

  7. 907

    #897 David

    Natural cycle forcing ranges from around thermal equilibrium +/- .3 W/m2, or so. Thermal equilibrium being 0 W/m2 and an ice age being around .3.4 W/m2

    We are currently around 1.6 W/m2 (3.6 W/m2 -2.0 W/m2 (negative forcing anthropogenic combined and -0.2W/m2 for bottom of Schwabe cycle.)

    I am confident an ice age will occur again but don’t know enough to estimate how long it might take. Depends on how the feedback mechanisms interact and how long it takes the system to mitigate the human contribution.

  8. 908

    #904 Patrick 027

    Thank you :)

    I was a bit sloppy on a few points and very much appreciate your additional contexts.

  9. 909

    #907

    Correction: ice ages being around -3.4 W/m2, not 3.4

  10. 910
    Ed Parton says:

    To me it looks like some of the “human” (anthropogenic) caused global warming stats were manipulated by naysayers (to global warming) just to make the “science” of the data and reporting procedures look really, really, bad. There is no global warming, the sky is not falling, but tons of other resources and reputable scientists dismiss this “breakin” as old news. Some of the emails and information (if not altered and are really original) are over 10 years old and do not reflect reputable current statistics and fact gathering, by diverse independent worldwide experts that use data that is taken from other than “peer reviewed” resources. (i.e. “closed” resources) National Meteriological Services do not release the best factual data because it is a “National Security” issue…

    Who do you believe??

  11. 911

    #910 Ed Parton

    I don’t know anyone holding climate data as a national security issue? Can you point me to references? Maybe there is a country out there withholding climate data as a national security issue, but is that important. Scientists around the world have collected a tremendous amount of information.There are those holding data for commercial purposes as far as I know.

    As to the emails, think about it realistically. Place yourself in their place (the scientists). Think about their contexts. Any sentence, any point, when placed in context, no longer looks like a grand conspiracy and resembles normal discussion and argument.

    Any point out of context is pretty much worthless. Saying a molehill is a mountain does not make the molehill a mountain.

    Try a thought experiment. Since we are past the peak of forcing from the Milankovitch cycles, which would certainly indicate we should be slowly heading back toward an ice age… then why is the Arctic ice volume dropping so fast and why are the mountain glaciers around the world melting? Why are we warming?

    Who do you believe??

    It’s not about belief, it’s about evidence, observation and understanding.

    http://www.ossfoundation.us/projects/environment/global-warming
    http://www.ossfoundation.us/projects/environment/global-warming/milankovitch-cycles
    http://www.ossfoundation.us/projects/environment/global-warming/natural-cycle
    http://www.ossfoundation.us/projects/environment/global-warming/myths
    http://www.ossfoundation.us/projects/environment/global-warming/myths/images/glacier-retreat

  12. 912
    Hank Roberts says:

    Thanks for the full context pointer, Gavin.
    (John, with any science material, it’s always useful to point to the original and describe what you think you see there, rather than paste chunks, particularly without attribution. Others need to see the original in context.)

    The link Gavin gave also shows what individual people are searching at the moment you click the page, which makes me wish to see their whole tally of searches and times (I suppose it’s done for the paying advertisers’ use tho’).
    When I clicked it said:

    “What are others looking for?
    * falsify
    * briffa
    * New World Order
    * alter
    * password
    * mike
    * global
    * denier
    * bolt
    * sunspots
    * watson
    * change
    * mcintyre
    * svensmark
    * zorita
    * singer
    * overpeck
    * jouzel
    * grant
    * political”

    (reload to update, it changes constantly — and note someone searching for “password” there — cautionary)

  13. 913
    David B. Benson says:

    David (897) & John P. Reisman (OSS Foundation) (907, 909) — Baring AGW, the next attempt at a stade (massive ice sheets) would not be for at least 20,000 more years (AR4 states 30,000 years). The orbital forcing at that time is rather small with a much better one around 50,000 years from now. As for AGW effects, see David Archer’s “The Long Thaw”.

    If this seems long in comparison to the shorter duration of previous interglacials, note that the current orbital forcing is more similar to that during MIS 11, when a long interglacial occurred.

  14. 914
    RaymondT says:

    Gavin, Thanks for your reply (listed below) to my message “‘One thing that surprises me when I read the Fourth IPCC report is that the history match of the temperature anomaly is not very good in the period 1901 to 1955 shown in Figs. TS.22 (Technical Summary) schematically for the global temperature and especially the global ocean temperature and in more detail in Figures TS.23 a) and b).”

    [Previous Response: The further back you go, the more uncertainties there are. Aerosol forcing is one of them, but so is the solar change, and even the scaling for the volcanoes. GHGs are pretty well characterised. - gavin]

    1) Which radiative forcings would you have to change to obtain a better history match of the 1901 to 1955 period ? 2) Could the mechanisms which caused the increase in temperature 1901 to 1955 also explain in part the temperature increase in the years 1965 to 2005 ? 3) Could the poorer history match of the 1901 to 1955 also be due to a poorer modelling of the ocean circulation ?

  15. 915
    Ray Ladbury says:

    Anand Rajan asks, “Why dont you stand your ground, and debate?”

    I presume, since we are talking about a scientific issue, yours is an invitation to scientific debate. Fine, how am I to debate a matter scientifically when the opposition refuses to deal in evidence but thinks character assassination will carry the day? How is that worthy of any response but utter contempt?

    And as to my “approach,” it is actually quite easy to get along with me. All you have to do is avoid levying unsubstantiated allegations of fraud against individual scientists just trying to do their job and against science itself. Is that really so hard? As long as you avoid that, I actually try to be helpful if I can to those trying to understand the science.

    My objection to your thesis is that you seem to think one must stop being human to practice science. It’s not true. Science has managed to return a very good approximation of truth for 300 years despite the fact that all of its practitioners have been human (well mostly, anyway). In that time, there have been epic rivalries, feuds–hell, even folks driven to suicide. Even so, the truth has won out. I will not defend Phil Jones or Mike Mann or any other researcher because 1)they do not require my defense and 2)I don’t require them to be saints. All they have to do to be card carrying scientists is go with the preponderance of the evidence, and this they have done.

    As to defending my ideology, there isn’t much to defend. I believe in what works–in science most of all. However, I also believe in free markets and regulations to keep them that way, in democracy, in trial by jury and in eternal vigilance. And finally, because I believe that continuation of human civilization is better than allowing the fall of human civilization, I believe that everyone, scientists included, has to step up to the plate, put aside self-delusion and reach a sustainable global economy.

  16. 916

    Response: Without any context, you have no basis for your statements. What was this for? what paper did it appear in? was it even ever used? Track that down and then we can talk. – gavin

    —————-

    So there’s an array of fudge factors in the CRU code and your only response is that maybe it wasn’t “used?” How about your code that WAS used? Do we get to see whether or not the code in your models and data analyses that WERE used was of similar “quality?”

    I have a Ph.D. in Computer Science, and ~30 years of experience as a software engineer. From what I’ve seen of the CRU’s code and its software management techniques, I wouldn’t trust your results to determine anything of any importance whatsoever, to say nothing of a major reorganization of the world’s economy.

    Your only hope of regaining any credibility is to open source all of your code and data IMMEDIATELY. The only example we have of the quality of your organization’s work is APPALLING.

    [Response: I have NO IDEA who you are, but if the CODE WAS NEVER USED in a publication, then it is not really relevant to any ASSESSMENTS that ONLY RELY on the published literature. CODE THAT WAS NEVER INTENDED TO BE PUBLIC should probably be EXEMPT from YOUR STANDARDS FOR PUBLISHED CODE. Try having your system hacked and then defend every stray piece of code in it. - gavin]

  17. 917
    MoVa says:

    So, the Earth may not be flat after all…

    Do you know what this is called? It’s Social Proof. Read about it and try to enlighten a little, before it’s too late:

    http://www.copyblogger.com/social-media-change/

  18. 918
    MoVa says:

    Social proof, also known as informational social influence, is a psychological phenomenon that occurs in ambiguous social situations when people are unable to determine the appropriate mode of behavior. Making the assumption that surrounding people possess more knowledge about the situation, they will deem the behavior of others as appropriate or better informed. Since observation of others usually provides only inconclusive information about what behavior is most profitable, the term ‘informational social influence’ is superior. Social influence in general can lead to conformity of large groups of individuals in either correct or mistaken choices, a phenomenon sometimes referred to as herd behavior. Although informational social influence at least in part reflects a rational motive to take into account the information of others, formal analysis shows that it can cause people to converge too quickly upon a single choice, so that decisions of even large groups of individuals may reflect very little information (see information cascades).

  19. 919
    Ron R. says:

    Anand:“But to claim that it is actually not possible to remain objective in any scenario in life, including science, and therefore plunge headlong into subjectivism…is that correct?”

    OK, here’s what I actually said:

    Anand, just so you know, I do agree with your basic premise that science needs to always strive to be objective and accurate. That its conclusions should not be driven by pre-conceived notions nor influenced by emotion. The thing is that climate science meets this criterion as best I can tell, and no one, despite enormous effort, has been able to show otherwise. And it’s science is such that it has, so far, stood the test of time. Does that mean, though, that climate scientists should (even if they could) be emotionless automatons? That they shouldn’t care about the wonderful planet that they have learned, and continue to learn, so much about? Of course not. Every breakthrough in science was, I’m sure, welcomed by it’s discoverers with emotion, all cared about their work, and that care did not lessen the science one bit.

    Not exactly what you said I said is it?

    “Gavin would point out repeatedly about the gigatons of carbon being pumped into the atmosphere raising sea levels and wiping out human infrastructure, the very infrastructure that is causing the warming in the first place. Thus we find a AGW proponent supporting preemptive measures to sustain coastal and inland industrial/military human infrastructure. How did that come about?”

    Come on, you’re being ridiculous here. What is the main consideration in protecting coastlines from flooding due to climate change? How many times have you heard those concerned say that they are worried that because this or that polluting industry might be destroyed? The issue is that lives, and yes, the local infrastructure that they depend on are at stake. Whether that be homes, towns, or cities. Schools, hospitals, farms, what have you. That some industries that are not beneficial might happen to fall within this area and also benefit is merely a (unwelcome) byproduct.

    About the Alex Jones’ type conspiracy comments about Futerra, I don’t know anything about it nor do I have the time to get into it.

    Anand, you claim to be an advocate of robotic disinterest, yet I would submit that you are not being honest with yourself. Your dogged insistance on your opinion is due to your subjectivity. In fact, to the extent that they see the world through their own eyes everyone is subjective to some degree. But again, that has no bearing on the fact that a lot of evidence is pointing to climate change and here it is: http://www.realclimate.org/index.php/data-sources/. If you can find fault with it then you have a case.

    Here’s the way I look at it: Even if at some point something came along that undermined the science of climate change that would not in the least, contrary to what the skeptics would have us believe, mean that those studying climate change had nefarious motives. They are doing the best that they can do with the facts and tools they have at hand. It’s clear to me that the nature haters in the rightwing are oh so dearly hoping that the science is wrong. Why? So that they can then say that ALL of environmental science must similarly be wrong. Then we can dump the endangered species act, auction off national parks to the highest bidder (true, the Bush administration actually had people on board that wanted to do that, look up Terry Anderson), and throw of the restraints of government regulations on polluting or otherwise environmentally destructive industries. So what’s at stake, unfortunately, is actually bigger than just climate change. The idea seems to be to force scientists to get defensive and make the most positive statements they can, something that they hope to be able to throw in their faces later.

    I’m a believer in “considering the source”. When I consider the source I see the font of denialistspeak, EXXON/Mobile and Big Coal, rightwing think tanks which in addition taking money to deny climate change have taken money from the tobacco companies to deny the harmful effects of smoking etc, etc, and politicians like Inhofe who coincidentally is one of the largest recipients of Big Oil/Big Coal money.

    http://tinyurl.com/yf8n97t

    Those facts should send up very large red flags for anyone. That they don’t for so many is testament to the weird, pretzel-like twisting of right and wrong that is so prevalent today.

    “Doubt is our product, since it is the best means of competing with the ‘body of fact’ that exists in the minds of the general public. It is also the means of establishing controversy.” -Brown & Williamson Tobacco co.
    http://tinyurl.com/cltcx
    http://tinyurl.com/2ozbrc

  20. 920

    [Response: Have these savants realised that 'Harry' was working on CRU TS 3.0, a completely different product than the HadCRUT? Have they never worked on databases that bring together multiple sources of incompatible source data? Or ever debugged code (over a 4 year period)? Or occasionally got frustrated? Hmmm... - gavin]

    —————-

    The massive institutional incompetence on display in HARRY_READ_ME.txt, can’t be “denied” by implying that “everyone does it.” You had rank amateurs working on code with huge social consequences. Your programmers never heard of source code control systems? Version control?

    Look, for instance, at Harry expressing his concern about the sum-of-squares parameter going negative (line 309 of HARRY_READ_ME.txt). He seems to have a dim grasp on the idea of the maximum positive value for a single precision IEEE-754 floating point value, something that is routinely taught to second year undergraduate computer science students (I know this because I teach that class).

    As Harry may not know, that maximum positive value is approximately 10 to the 38th power. Is that software computing values that are SUPPOSED to be near 10 to the 38th? And if it is, why weren’t those numbers scaled to keep them away from that value and the potential for floating point overflow errors?

    ALL OF YOUR CODE, all of the data analysis and climate modeling code, needs to be open sourced IMMEDIATELY so that competent software engineers and computer scientists can do a peer review on what folks like Harry have been doing all these years.

    There are large consequences associated with the results of your software. The light of day needs to shine on what you folks have been doing in your computers.

  21. 921
    sHx says:

    @#815 Marco, my understanding of Freedom of Information Acts (at least those in Australia) is that they do not concern themselves how the information requested is to be used. If I file an FOI with a government department requesting information about the funding of a particular project, then they are obliged to provide it to me regardless of what I intend to do with it. Whether it is ‘use’ or ‘abuse’ is subjective and irrelevant. I am curious to know whether “2% of the data, which carries very limited extra information” is fact or speculation.

    @#819 Probably, there would not have been a flood of FOI requests mostly asking for the same data, had the CRU responded to the first one. Administrative inconvenience is not an excuse not to comply with a FOI request. The CRU could still provide whatever data not covered by confidentiality agreements AND the names of the national meteorological services which prevented them from providing the rest of the data. Other interested parties could then purchase the remaining data from whatever NMSs that were selling it.

    @#824 Paul, the words “seems” and “suggest” are useful if you want to express some uncertainty or readiness to offer the benefit of doubt. In the specific post you are refering to (#812), I have not used the word “seem”, but I have used the word “suggest” once in this sentence: “Instead, as the emails suggest, his [Phil Jones's] intentions were to do completely the opposite.” In hindsight, I should not have expressed any uncertainty that Phil Jones had no intention to comply with the FOI requests. Therefore, I would like to revise that sentence and replace the word “suggest” with the word “prove”.

    @#849 I am sorry for not anticipating that my mention of the Human Genome Project would cause a few replies that would distract from the general debate and the point that I was trying to make. If the secretive nature of the human genome project set back medical advances by 10 years, then I shudder to think how much the entire civilisation will be set back by policy responses to the AGW theory, which was developed in part by data still kept in secrecy.

    It is astounding to find out that we do not currently have an international “open data” agreement whereby raw data from all national meteorological services will be collected in a single location and freely accessible by all researchers. This would save all Phil Joneses and Steve McIntyres of the world from wasting their research time hassling each other over unavailable data. Climatologists and meteorologists should have pushed decades ago for the funding and establishment of an institution that would gather all of the raw temperature data from around the world and make it available to anyone with a PC and internet connection. I note and commend Real Climate for setting a new page that does exactly that, but, in my view, it is too little too late.

  22. 922
    Robert Campbell says:

    Since so much of the analysis, and therefore the conclusions, depend on data that has been gathered and analyzed by many, it seems to me that all of the data available should be available for download in machine-readable format by anyone who is interested. Furthermore, anyone publishing analyses or reports should be required to cite the data sets used and fully describe any “adjustments” to that data.

    Part of the peer review process should be that papers will not be accepted for publlication unless the data is cited and available to the reviewers and any new data will be published upon publication of the papers.

    It might be appropriate for some recognized agency to maintain a web site with links to all of the data and reports.

  23. 923
    Patrick 027 says:

    More re warming a good thing? not so fast…

    See Ch 10 of the IPCC AR4 WGI, specifically p.764 – …

    Precipitation: generally a drying trend in the subtropics, with increasing precipitation at lower and higher latitudes, with longitudinal variations in/from that pattern. (PS To some extent, the dry regions of the subtropics are expected to expand poleward – I think this is related to a general poleward shift in midlatitude storm tracks (lack of ‘storms’ not necessarily a good thing!!)).

    Note that evaporation changes are also important in determining changes to water resources and ecosystems.

    Surface temperatures:

    Specifically, the zonally averaged (averaged over longitudes for each latitude) temperature changes are expected to be greatest in the high northern latitudes (surface albedo feedbacks) and least in the southern mid-high latitudes (mainly ocean; wind-driven upwelling cools the surface and the source water will take a long time (how long exactly, I don’t know; I’d naively guess several centuries?) to recieve the climate change signal, and as surface water at other latitudes warms, the increased temperature gradient from about 30 deg S to 40 or 50 deg S will tend to increase the winds (via extratropical storm tracks, as I understand it) that drive the upwelling. Antarctica is expected to warm more than the surrounding southern mid-high latitude ocean, but not as much as the Arctic ocean at least in part due to the smaller albedo feedback of the ice sheet – which may remain the case even with losses in ice mass. (PS to what extent Antarctica might melt if given sufficient time, I don’t know, or for that matter, if it might reach a new equilibrium that would still have significant ice cover even if cooling off to the same global external forcing from warmer conditions would fail to grow much ice cover (hysteresis) (?).)

    There will be a reduction in pole-to-equator temperature gradients (zonal annual average) in general on land outside the roughly 30 deg S to 55 deg S belt. Some of this occurs between the tropics in both hemispheres over land, but the greatest changes over land and water are between the midlatitudes and polar latitudes – specifically, from roughly 55 deg S to 70 deg S and in a similar belt in the Northern hemisphere, with some differences between land and water. Interestingly, the gradient would increase between about 40 and 50 deg N over the ocean, with no overall change in temperature variation between roughly 38-40 deg N and 57 deg N (eyeballed) over the ocean.

    If the midlatitude stormtracks are centered around 40 to 50 deg latitude, then it seems the reduced temperature gradients are mainly poleward of those storm tracks.

    Of course, this could be partly due to dynamic feedback – a change in radiative and latent heating gradients that would tend to reduce the temperature gradient would tend to cause changes in storm track activity that would reduce horizontal thermal mixing and act as a negative feedback on the gradient change.

    However, the focus of the temperature gradient reduction may also indicate the boundaries of greatest surface albedo feedback. If that is the case, it makes sense that, within the northern midlatitudes, there is some greater reduction in the temperature gradient on land relative to the ocean because of snow cover feedback (and changes in leaf loss and growth dates, etc). Note that, at least as I understand it, snow cover changes (outside year-round ice cover) should cause warming mainly in winter – reasons why seasonal land snow cover would have less dramatic effects than sea ice are: 1. loss of winter snow has less incident solar radiation to act on than loss of summer sea ice (although the higher latitude of sea ice could partly compensate, and I think the sea ice cycle lags the seasonal forcing more than snow cover (snow cover doesn’t peak in March)) 2. seasonal snow’s albedo effect is muted by the texture of underylying vegetation or protruding vegetation, and the greater albedo of the uncovered surface relative to open water. Note however that sea ice albedo feedback has the greatest impact on temperature in the winter (or cold season in general) when the open water puts heat back into the air as it cools and possibly freezes, whereas already frozen ice partly decouples the surface and air temperature from the underlying water temperature.

    The decrease in northern subpolar temperature gradient is concentrated in the cold part of the year, when the mid-latitude storm tracks are farther south (in the Northern Hemisphere). My understanding is that the Northern subpolar temperature gradient change may be in the opposite direction during at least a portion of the warm season (might this partly be the land-sea contrast between midlatitude continents and polar ocean?), when the midlatitude storm tracks are at relatively higher latitudes.

    Moreover, in the upper troposphere at most latitudes (and farther down into the mid-troposphere poleward of the tropics) and in lower stratosphere outide a lower-latitude band that widens with height (? – figure 10.7 doesn’t show exactly how the tropopause slopes and is in linear pressure coordinates, so it’s a bit hard to tell some details at higher levels) the tendency in both hemispheres is an increase in the equator to pole temperature decline (or reduction of the increase in those parts of the stratosphere).

    From other sources (browsed the data page at RC, will have to specify where later), there is an increase in westerly winds near the storm tracks that increases from the surface upward…, thus increased wind shear… out of time for now.

  24. 924
    huxley says:

    Ray Ladbury:

    You said: “What would it take to make you actually look at the evidence? After all, shouldn’t science be about evidence rather than personalities? It is not as complicated as you think. What would it take to get you to look at evidence rather than anywhere but.”

    I have looked at the evidence, as an intelligent layman, and I have read more than most people I know on AGW. In fact I have defended AGW to my more skeptical friends — not that that would seem to matter to you.

    However, what you and others in the AGW camp don’t seem to understand is that the onus is on you to convince the rest of us of the validity of your efforts and necessity to take action, not for us to passively accept for your pronouncements as gospel and take large hits on our incomes and ways of life, just because you are so all-fired certain.

    I am old enough to remember Dr. Paul Ehrlich and The Club of Rome. They said that billions were going to starve and we were going to run out of most of our important resources. Didn’t happen. Didn’t happen at all. And The Club of Rome even used … computer models.

    Why should we believe the advocates of AGW are any different, esepcially when their predictions for the past ten years and counting have been wrong? And now the CRU Hack which reveals blatant seamy scientific misconduct….

    A few years ago I listened to Gavin Schmidt and other AGW advocates defend AGW against Lindzen and other AGW skeptics. (http://www.npr.org/templates/story/story.php?storyId=9082151) The AGW advocates not only lost the debate, they lost when the majority of the audience pre-debate agreed with the AGW position. In other words, the AGW side would have been better off staying at home rather than arguing on their behalf.

    AGW advocates are practically their own worst enemies. Like it or not, the CRU Hack only validates an impression that has brewing for a long time.

  25. 925
    Hank Roberts says:

    I’m old enough to remember Paul Ehrlich lecturing too, on coevolution
    http://www.jstor.org/stable/2406212

    He’s been right far more than he’s been wrong. He’s gotten onto areas very early and brought them to other scientists’ attention as worthwhile to study, which is a very important contribution in science. Work that brings attention to an interesting area often gets corrected later — that is valuable, and it’s cited for its value as well as to correct early mistakes.

    Hunger and shortages? If you think it’s not happened already you aren’t paying attention. Know what two of the most significant shortages are?
    Iron, and Iodine. Look at them on these two maps:
    http://www.fao.org/DOCREP/005/Y8346M/y8346m01.jpg
    http://www.fao.org/DOCREP/005/Y8346M/y8346m02.jpg

    As to the past 10 years of climate work, well, why am I bothering—if you can’t see the difference between the Second and Fourth IPCC Reports, you’re really not looking at the science. Handwaving about predictions is just Blog Science, except without the science.

  26. 926
    Rattus Norvegicus says:

    Well, since nobody has done this yet, here is what I found by looking at all of the emails (note that this is a qualitative and quantitative analysis):

    ~95% are completely innocent and include the following subjects

    + IPCC meeting announcement
    + Discussions of the wording of IPCC chapter contents
    + Discussions of how best to show findings in IPCC 4AR graphs
    + Meeting announcements for other scientific meetings
    + Discussions, often vigorous of results
    + Discussions of papers in progress by collaborators
    + Wife and kids sort of stuff

    The other ~5% consists of the following:

    + Discussions of the perceived failure of peer review
    + FOI requests and how to deal with them
    + A single message about “hide the decline”
    + Stuff slagging McIntyre

    The discussions of perceived failure of peer review relate mainly to the publication of Soon and Baliunas 2003 by Climate Research. In isolation these discussions look pretty bad, if you do not know the backstory.

    Climate Research (CR) had published several papers which were seriously flawed culminating with the publication of SB2003, a paper which made claims which could not be supported by the analysis contained in the paper. The editor who handled this paper, Chris deFrietas, is a well known contrarian gave the go ahead to publish this paper as he had on several other bad (in the sense that they were not scientifically rigorous) papers. Subsequently, although why is not clear, the publishers decided to appoint Has von Storch as Editor in Chief. Von Storch wanted to publish an editorial announcing there was a new sheriff in town, but the publisher would not let him run it. This led to the resignation of von Storch and 4 other members of the editorial board.

    Now, the discussion of the state of editing at CR may have led to the appointment of von Storch as EIC, something that it is clear that the CRU people would not have welcomed. The resignations were not due to this. The resignations were due to the fact that von Storch was not given the authority that would be expected given his title. The publisher clearly was not willing to clean up the problems with the peer review system in place at CR.

    The comments about Saiers at GRL really don’t go anywhere. Saiers was not removed through an internal AGU process and AGU journals have not stopped publishing questionable papers (I think the paper being complained about
    was MM2005). In fact there was a rather devastating comment on MM2005 which appeared in GRL by Huybers. Steve was not happy about that.

    The final “peer review” complaint has to do with McKitrick and Michaels and another bad paper (sorry, I can’t recall) and “redefining the peer review literature”. Both papers were discussed in 4AR. And you were saying? It should be noted that McKitrick is funded by the oil industry through the Fraser Institute and that Micheals is funded by the oil industry through the CEI. Yeah I know that this is an ad hom, but in these days of post normal science, you need to consider the source.

    The FOI issues are more troubling. However the initial message from Phil Jones is in 2005 and he knows at this point that with IP he can fend off any requests. Note that this is before *any* FOI requests were made. The thing to remember is that he already has experience with McIntyre and the unending requests that he is likely to make. Subsequent events show him to be prescient. The best way to illustrate this is the fact that from July 24 to July 29 CRU got 58 FOI requests from McIntyre or other people associated with CA. I believe that these would be considered “vexatious” requests. Indeed, one of the messages indicated that the denial of access was entirely aimed at CA and McIntyre’s acolytes for reasons which should be clear. This is not to say that other requests would not have been denied, indeed they would because of IP agreements. However, McIntyre knew that his requests would have been denied because he had already been informed of the limitations on the data and was just using the FOI denials as ammunition to call into question the validity of the HadCRU instrumental record. The problem here is that there is no indication that the CRU surface record is corrupted as it is backed up by GISTEMP, UAH and RSS — all methods using different datasets and analysis techniques.

    The “hide the decline” comment has been well discussed here. The associated “divergence problem” is an area of active research. At this point it is well established that divergence is not a problem which is associated with all dendro records, but it is an interesting question.

    Finally, we come to what I refer to as “Steve slagging”. This relates directly to Judy Curry’s circling the wagons criticism. I really do not see much evidence of this until Steve started to get some traction with the press in late 2005. In this time frame Steve started to step up his accusations of incompetence and fraud. Well, he often just insinuated this, and let his followers and the conservative press run with it. Thus Mann’s remark about plausible deniability — referring to Steve’s techniques, rather than to some dark conspiracy on the part of climate scientists. Given what was happening I can certainly understand the circling the wagons mindset.

    Who comes out worst in this? My vote goes to McIntyre. Keenen filed a baseless charge of scientific fraud against Wei-Chyung Wang with SUNY Albany. Wang was cleared. Steve comes off as lazy and wanting someone else to do his homework in his “audit” of Santer’s criticism of Douglass, et. al. in the IJoC. All of the raw data necessary to replicate Santer’s analysis was available at PCMDI, and he could have taken the same approach as Santer did in replicating the Douglass, et. al. study. This is exactly the sort of behavior which Jones had run into. The best way to look at the way the people who do this for a living look at Steve is this:

    login: steve
    password: tosser

    The final nail in Steve’s coffin is his abuse of his position as an invited reviewer for 4AR. His attempt to get data for d’Ariggo, et. al. by writing to the editor of the journal in which this study was to be published is ugly. You can be sure that he will not be invited back.

    My best guess of how this will all end up? Steve is going to lose a lot of credibility and will not get called by the likes of Andy Revkin. This will be devastating for him since he seems to love the attention he has been getting.

  27. 927
    Rattus Norvegicus says:

    I forgot to mention my theory of why these were the emails we got to see. First a short list of the mailboxes which were included:

    + Phil Jones
    + Keith Briffa
    + Tim Osborn
    + Mike Hulme
    + Mick Kelly
    + Asher Minns

    I suspect that most of them were selected though a keyword search (grep for you Linux/Unix sorts). Keywords were probably: IPCC, Yamal, Mann, FOI (or Freedom of Information). There may have been some others, but these seem to cover around 90% of the mails.

  28. 928
    PeterK says:

    Re: Gavin “….the onslaught of nonsense that has been thrown at the climate science community from people who don’t know their greenhouse gas from a hole in the ground ….in no other field are these cranks held up in public as the new Galileos by people with very strong vested interests in the status quo and who have access to very effective media conduits”.

    One can certainly sympathise with experts being criticised by citizens who don’t know a GHG from a hole in the ground. But we should also note that the biggest vested interest all – by several orders of magnitude in brute spending and propaganda output – is the state. Governments everywhere with out-of-control spending – certainly including the US and the EU – would love nothing better than a credible new reason to crank up taxes. And this is of course exactly what the climate experts are providing. Experts selected and employed and financed by the very same states wanting to tax more.

    So sympathy is due too to citizens and taxpayers for having real concerns about those who *do* know a GHG from a hole in the ground, but some of whose objectivity and motives may well be less than hoped for. Concerns reinforced by the CRU emails, and comments such as some above about how ‘post-normal’ science – ditching objectivity for ‘correct’ ideological commitments such as higher taxes and world government (quote below) – is just fine and dandy.


    Christine Stewart, former Canadian Minister of the Environment: “No matter if the science of global warming is all phony… climate change provides the greatest opportunity to bring about justice and equality in the world.”.

    [Response: The science of climate change is not affected by co-benefits that anyone else might see in reducing emissions. Designing cities to reduce the need and consequences of road traffic reduce CO2 emissions, but also ground level ozone, traffic fatalities and increase quality of life. First off, I don't see why the existence of co-benefits makes the climate science less valuable, and second, politicians looking for co-benefits is perfectly natural and indeed, sensible. If you come up with two policies that reduce emissions equally - wouldn't you prefer the policy that made the most sense for other reasons too? Your implicit assumption that the co-benefits for politicians is driving the science is simply nonsense. - gavin]

  29. 929
    David Horton says:

    It must be a relief to the defence forces of America, Canada, Russia, Sweden, Norway to learn that as a result of the CRU leaked emails they will no longer need to plan and invest huge resources in appropriating an ice free Arctic; and a relief to the big energy companies that they will not need to invest resources in planning, exploring, and then drilling for oil in an ice free Arctic. What were they thinking? Why oh why didn’t they listen to the denialists all along?

  30. 930

    AJ,

    Your professor Hayden hasn’t bothered to familiarize himself with the actual science involved. To pick out just one point, we know the new CO2 is mainly from fossil-fuel burning because of its radioisotope signature, a point first tentatively made by Suess in 1955 and pinned down by Revelle and Suess in 1957. [edit - calm down]

  31. 931

    If the Mayans were such great prognosticators, it’s a shame they somehow missed the destruction of their own culture due to drought about the year 800 AD.

  32. 932
    Marco says:

    @Huxley:
    http://www.csiro.au/files/files/plje.pdf
    On the forecasts of the Club of Rome.
    Back to you!

    @sHx:
    I’m afraid that even publically available raw data won’t stop the McIntyres of this world. Remember that he HAD the data that Briffa used, and still complained. Remember also that he asked Ben Santer for publically available data.
    Regarding the 2% of data: this was notably McIntyre’s own comment in the FOI request. And yes, FOI does not specify anything about the use of the data. However, I’d say that abuse of the data (however subjective this may be) would go against the spirit of the law. But maybe that’s just me.

    @Robert Campbell:
    Many scientists would not mind, just double or triple their research budgets. Because that’s what it is going to take in many instances to have such an audit trail. Scientists generally want to do science, not push paper around.

  33. 933
    marco b says:

    > However, what you and others in the AGW camp don’t seem to understand is that the onus is on you to convince the rest of us of the validity of your efforts and necessity to take action, not for us to passively accept for your pronouncements as gospel and take large hits on our incomes and ways of life, just because you are so all-fired certain.

    My first and last comment on this bitter discussion:

    You’re half right in that. But they already do, while deniers don’t. Because they cannot, so they only repeat their sceptical questions again and again, although they’ve already been answered. Why?

    Why should we face decreasing incomes, if AGW is acted upon? On what facts do you base these predictions? You have to weigh the cost of not acting in face of a 100% (=90% politically reduced) certainty against the costs of a) necessary investments to keep the old stuff running and b) the known consequences of not acting. And your way of life: those who do not adapt to changes in their ecological sphere will not make it, whether it heats up more or less or on behalf of whatever causes, whether they deny the facts or not.

    > Why should we believe the advocates of AGW are any different, esepcially when their predictions for the past ten years and counting have been wrong?

    Because they say so, if they find out, that they were wrong (good ones) or they are proven wrong by the scientific community (bad ones). Because science is about working on a thesis and correcting it, until it best matches reality. Because ‘doubt’ is sciences’ best friend, in order to get a better and better picture. It’s not about setting up a worldwide conspiracy of scientists who want to see the world and its inhabitants suffer for their relatively underpaid jobs. It’s not that easy to get two scientists to an agreement, so how about hundreds, worldwide?

    And if you ask yourself the question: cui bono? Who profits from denial of, who profits from accepting AGW-theory, who would profit from global warming, once it hits us hard or not at all? For whom would you consider to be more money in nowadays and more profits in the future? Science?

    The whole discussion is so stupid, the only accepted ‘proof’ of a prediction by deniers is the actual event itself. Everybody’ll get their personal 100% certainty, sooner or later. The better climate models we have, the earlier.

    Thanks to everybody, who’s working on this, be it a real critic or scientist. I read this crap since the early eighties and I really hope, my daughter won’t have to do so, once she’s grown up.

  34. 934
    Ray Ladbury says:

    PeterK, I fail to see how the fact that the Canadian Environment Minister is a moron affects in any way the overwhelming evidence that shows we are warming the planet. In my opinion, the fact that she is even entertaning the proposition that the evidence is false makes here as idiotic as the average denialist.

  35. 935
    Edward O'Malley says:

    Thanks for responding to my question. The reason I classified Lindzen as being in disagreement is not that he disputes the temperature record, but that he disputes the majority of models used by the IPCC and their ability to accurately predict future temperatures and that the alarmism spreading around the world is unfounded.

    Thanks,

    Ed

  36. 936
    John Murphy says:

    At *883* it was suggested by Thomas:-
    <>

    Actually, I love solving problems, but I like to be sure what they really are before starting to solve them… I’m not a scientist, but from what I see, there are lots of different ‘climate models’ being referenced- all pretty much saying the same thing, but none of them reflecting the actual answer.

    I had a slide-rule like that at school- sadly I never had the ear of the examining board, so they marked me up as ‘wrong’. Little did I know how profitable that may have been, if only I had pursued it :-(

    My Thanks to Mr Reisman at *885* and *892* for clarifying some items and

    Barton @ *931* says
    <>

    How true, however I don’t suppose their ‘climate models’ were any more reliable than the ones we seem to have today.

    Lol, sorry, I know its a serious discussion, but I couldn’t help myself :-)

  37. 937

    #920 Michael Trigoboff

    I think the light of awareness needs to be turned on so people can see how fast the glaciers and ice volume in the Arctic is dropping.

    #922 Robert Campbell

    I think some important conclusions can be drawn without the data (not saying data is unimportant).

  38. 938

    #931 Barton Paul Levenson

    My thoughts exactly.

  39. 939
    John Murphy says:

    At 936, I’m not sure where the items quoted went, they were copied and pasted, but didn’t survive the transfer. :-(

  40. 940
    J. Bob says:

    11/28/2009, Arctic sea ice extent still “nip & tuck” with 2005 & 2003. Above 2007 & 2008.
    http://www.ijis.iarc.uaf.edu/en/home/seaice_extent.htm

    While a lot of the discussions have been about personalities, the refusal to release the climate data by government agencies is one of the worst mistakes of this whole mess. If for some reason, these government agencies have a problem with releasing the data for cost reasons ( of course then we have to pay twice for it), simply publicize the cost for the data. Then the excuse “We don’t have time” would not exist.

  41. 941
    David Gordon says:

    I’m back with more questions, following up on the CO2 topic.

    1) Reading the papers, I see an estimate of 2.9 deg C +/- 1.5 deg C for climate sensitivity to doubling of CO2. With a margin of error of over 50%, what are the grounds for confidence in the statement that CO2 causes warming?

    [Response: Umm... the entire range is well above zero? - gavin]

    2) What measures have been taken to rule out the possibility that the planet has self-correcting mechanisms (e.g. plant growth to take up excess CO2) that will mitigate or eliminate warming?

    [Response: Carbon-cycle changes are irrelevant for the basic sensitivity issue (ie. the warming associated with a doubling of CO2). The fact that the CO2 levels are continuing to rise rapidly implies that the natural carbon cycle cannot sequester more than about half of the emitted CO2 over short time scales. - gavin]

    3) What exactly has been the temperature trend since 1998? Has the climate been cooling or warming in the past decade? (Getting lots of conflicting info on this point.)

    [Response: See for yourself. - gavin]

    4) Given that climate is a mathematically chaotic system, what use is a necessarily incomplete computer model at all, and what models have shown to be accurate predictors of climate?

    [Response: The answer to the first part is implied in the answer to the second. Many models have shown skill (at predicting the response to a big volcano, predicting multi-decadal trends, matching 20th C and paleo-climate data). That doesn't make them perfect, but it does show that they are useful despite the underlying complexity. The overall constraints on the system do actually constrain things. - gavin]

    Thanks again!

  42. 942
    Hank Roberts says:

    J. Bob says: 29 November 2009 at 11:37 AM
    “… the refusal to release… If for some reason … problem with releasing the data for cost reasons ( of course then we have to pay twice for it) ….”

    Nonsense.

    Some data came free, but its use is limited–for research use only–from countries that rely on resale of their meteorological data for income.
    Some guy named Reagan or Adam Smith came up with that bright idea. Yeah, it’s stupid, it ought to be free. But how do you make them give it away?

    You’ve paid nothing, not even the time to learn what you’re talking about.

  43. 943
    Neal J. King says:

    In reading the discussion in RC and a scattering of other websites concerning the implications of the break-in to Hadley CRU, I see a conflict of two perspectives:

    a) The perspective of the scientific workers, who have established working methods and procedures supportive of, and adequate to, providing information and documentation on experimental methods, data, and data processing sufficient for the conduct and analysis of the work; and for explanation to external colleagues.

    b) The perspective of auditors, who are doubtful of the results of the work, suspicious of the methods, and unconvinced as to the competence and consistency of the workmanship.

    Under approach a), working methods have been somewhat informal. Information provided for informal inquiries can assume familiarity with the technical and scientific background of the data. The documentation need only explain the work.

    But under approach b), the working methods should be capable of not only explaining the work, but justifying it against a presumption of error – almost to a legal standard. Documentation provided to external inquiries must explain the procedures, explain how to access and utilize the data, explain what is missing (and how to get it), in the interest of total transparency to an audience which cannot be expected to have colleague-level expertise. Silver-platter/silver-spoon treatment.

    It would be great if this whole thing blows over, and people can operate with the relatively low overhead of paradigm a). But if it does not, scientific agencies and institutions involved with potentially controversial topics may have to deal with a dramatic shift in documentation and working methods to transition to paradigm b), which will require a much more approachable system of documentation, as well as dedicated staff to respond to external inquiries with flexibility and grace.

    Documentation must take on a public-relations aspect – in practice and probably in organization.

  44. 944
    Deech56 says:

    RE Anand Rajan KD

    But Gavin would point out repeatedly about the gigatons of carbon being pumped into the atmosphere raising sea levels and wiping out human infrastructure, the very infrastructure that is causing the warming in the first place. Thus we find a AGW proponent supporting preemptive measures to sustain coastal and inland industrial/military human infrastructure. How did that come about?

    I spent a few years researching approaches to HIV and Lyme disease vaccines. For the record, I wanted these diseases to be controlled as much as possible. Researchers are funny like that. I also thought that people should take preventative measures. Maybe climatologists see trends that worry them as people.

  45. 945

    #940 J. Bob

    Don’t rely on short term ice extent but rather long-term ice thickness/volume/mass and ‘long-term’ ice extent to get a reasoned view.

    http://ossfoundation.us/projects/environment/global-warming/myths/images/arctic

    As far as hidden data, here’ s a bunch of it

    http://www.realclimate.org/index.php/data-sources/

    hmm… but if it’s here, then it isn’t hidden… silly me. Mind you, much of this has been available for a long time but thanks to the RC team there is a nice consolidated page :)

  46. 946
    huxley says:

    marco b:

    You said [933]: “Because they say so, if they find out, that they were wrong (good ones) or they are proven wrong by the scientific community (bad ones). Because science is about working on a thesis and correcting it, until it best matches reality. Because ‘doubt’ is sciences’ best friend, in order to get a better and better picture. It’s not about setting up a worldwide conspiracy of scientists who want to see the world and its inhabitants suffer for their relatively underpaid jobs. It’s not that easy to get two scientists to an agreement, so how about hundreds, worldwide?”

    I’m not positing a worldwide conspiracy. I’m recognizing that scientists are human. They make mistakes and can be caught up in groupthink and in turf wars for funding and recognition.

    As to scientific method, exactly, and that’s all I’m asking for here: that all the data and all the methodology be made available for inspection and replication.

    No more refusals to answer FOI requests. No more threats to destroy emails and data or requests along that line, No more games with peer-reviewed journals. No more destruction of raw data.

    As the WSJ said, “The public has every reason to ask why [AGW advocates] felt the need to rig the game if their science is as indisputable as they claim.”

    You said: “And if you ask yourself the question: cui bono? Who profits from denial of, who profits from accepting AGW-theory, who would profit from global warming, once it hits us hard or not at all? For whom would you consider to be more money in nowadays and more profits in the future? Science?”

    Cui bono cuts both ways. Billions of dollars in funding have sluiced into a small scientific speciality since the concerns over AGW have become so intense. Reputations have been made. Michael Mann is arguably now one of the most famous scientists on the planet. Governments are looking forward to hefty carbon taxes. There is a bonanza ahead for green industries if cap-and-trade type measures are implemented.

    Again, all I’m asking for here — that all the data and all the methodology be made available for inspection and replication.

    [Response: I await your audit of the GISTEMP code and output with baited breath. - gavin]

  47. 947
    huxley says:

    “I await your audit of the GISTEMP code and output with baited breath. – gavin”

    Gavin: I await your response and that of your colleagues to conform to the scientific method with equally baited breath.

    Otherwise I don’t see how your response above is relevant.

    [Response: Of course you don't. Your point is to insist on some impossible standard so that you can avoid paying any attention to the results. My point is that there are completely open projects that if you were really concerned with evaluating you would do so. That you don't understand my point, simply underlines the issue. - gavin]

  48. 948
    mondo says:

    What you guys seem to miss is that the reason that the sceptics want to have a look at the data and methods behind climate data is that it appears that all of the ‘adjustments’ to past climate records are in the direction of making it all worse – OMGIWTWT.

    Of course, anybody who has been around for a while understands that corrections and adjustments are often necessary in any line of work. However, usually some of the corrections and adjustments are in one direction, and some in the other. In climate science, it seems that all of the adjustments and corrections seem to be in the one direction. How can that be do you suppose?

    [Response: Well, since it isn't actually the case, the point is moot. Some examples, corrections for UHI reduce surface temperature trends and the correction for the bucket measurements pre-WW2 reduced trends in the SST records. Corrections for radio-sonde biases in the stratosphere reduces cooling trends there, corrections for post-glacial rebound reduce estimates of mass loss on Antarctica. Corrections for water vapour overlap reduced the amount of forcing associated with CO2 changes. I could go on. - gavin]

    Also, in several places in the above threads you refer to other sources of data signifying AGW even if the dendrochronology work is ignored. In particular you refer to ice cores [edit] and glacier retreat.

    [Response: Go to the World Glacier Monitoring Service and then come back and tell me what you think about glacier retreat. Or visit the Exit Glacier in Alaska, the soon-to-be-renamed Glacier National Park, the Alps, the Andes, the Himalayas or Kibo. Pretty common pattern there. - gavin]

    Re the latter, it is worth having a look at Cliff Ollier’s review of “Himalayan Glaciers – A State-of-Art Review of Glacial Studies, Glacial Retreat and Climate Change” by V.K. Raina.

    Ollier remarks: “The IPCC’s 2007 Working Group II report asserted that Himalayan glaciers “are receding faster than in any other part of the world and, if the present rate continues, the likelihood of them disappearing by the year 2035 and perhaps sooner is very high if the Earth keeps warming at the current rate”. Such claims are unsupported, unscientific and wrong.

    [Response: Try actually reading the report. There is no statement that the glaciers aren't receeding - the photographic evidence and current monitoring is actually quite solid. - gavin]

    Surely it is up to the AGW proponents to offer proof for their special case theory. If the AGW camp wish to assert that glacier retreat is due to global warming, they need to demonstrate how such warming affects the glacier budget.

    [Response: Gosh that's tricky. Let me make a stab at at it though. As temperatures increase, more of a glacier melts or ablates. Not too terrible as a theory I guess.... - gavin]

    When is it supposed that AGW set in, and how can it be demonstrated?”

  49. 949
    Andrew says:

    “As to scientific method, exactly, and that’s all I’m asking for here: that all the data and all the methodology be made available for inspection and replication.”

    Oddly enough, custody or freedom of data and method has never actually been part of the scientific method. Never. Replication and confirmation are best when they are independent; which means independent of the apparatus, data, and methods of the original scientists.

    There are entire fields of science where no you cannot just ask for data and expect it to be given. Don’t think so? Go to the university nearest you active in medical research and ask for access to all the data.

    Or, go to the research labs of the oil and chemical companies. Ask them to give you all their methods, code, and data. See exactly what that gets you. Take one of those television commercials where the company has a little parade of scientists with animated equations or molecular models talking about how their company is advancing science; pick up the phone and call the people in the commercial. Ask for all their data, methods, and code.

    Get back to me when you have it.

    And it’s been like this for a long, long time. The excellent book “The Demon Under the Microscope” gives insight into Bayer’s research between the world wars that led to the development of sulfa drugs – including a Nobel prize (Gerhard Domagk) and plenty of political intrigue – first the French, then the Nazis. Lots and lots of unpublished data, scientists trying to get hold of other people’s data.

    Go Google “ceiinosssttuu” and ask what that tells you about the history of information sharing in science.

    So yeah, making information available is often good. But it should really not be promoted to some sort of sine qua non.

  50. 950
    Deech56 says:

    Rattus – you went through the e-mails? Nice finds. The dates for the FOI requests do match up with this call for spamming UEA.


Switch to our mobile site