RealClimate logo


The CRU hack

Filed under: — group @ 20 November 2009

As many of you will be aware, a large number of emails from the Climatic Research Unit (CRU) at the University of East Anglia webmail server were hacked recently (Despite some confusion generated by Anthony Watts, this has absolutely nothing to do with the Hadley Centre which is a completely separate institution). As people are also no doubt aware the breaking into of computers and releasing private information is illegal, and regardless of how they were obtained, posting private correspondence without permission is unethical. We therefore aren’t going to post any of the emails here. We were made aware of the existence of this archive last Tuesday morning when the hackers attempted to upload it to RealClimate, and we notified CRU of their possible security breach later that day.

Nonetheless, these emails (a presumably careful selection of (possibly edited?) correspondence dating back to 1996 and as recently as Nov 12) are being widely circulated, and therefore require some comment. Some of them involve people here (and the archive includes the first RealClimate email we ever sent out to colleagues) and include discussions we’ve had with the CRU folk on topics related to the surface temperature record and some paleo-related issues, mainly to ensure that posting were accurate.

Since emails are normally intended to be private, people writing them are, shall we say, somewhat freer in expressing themselves than they would in a public statement. For instance, we are sure it comes as no shock to know that many scientists do not hold Steve McIntyre in high regard. Nor that a large group of them thought that the Soon and Baliunas (2003), Douglass et al (2008) or McClean et al (2009) papers were not very good (to say the least) and should not have been published. These sentiments have been made abundantly clear in the literature (though possibly less bluntly).

More interesting is what is not contained in the emails. There is no evidence of any worldwide conspiracy, no mention of George Soros nefariously funding climate research, no grand plan to ‘get rid of the MWP’, no admission that global warming is a hoax, no evidence of the falsifying of data, and no ‘marching orders’ from our socialist/communist/vegetarian overlords. The truly paranoid will put this down to the hackers also being in on the plot though.

Instead, there is a peek into how scientists actually interact and the conflicts show that the community is a far cry from the monolith that is sometimes imagined. People working constructively to improve joint publications; scientists who are friendly and agree on many of the big picture issues, disagreeing at times about details and engaging in ‘robust’ discussions; Scientists expressing frustration at the misrepresentation of their work in politicized arenas and complaining when media reports get it wrong; Scientists resenting the time they have to take out of their research to deal with over-hyped nonsense. None of this should be shocking.

It’s obvious that the noise-generating components of the blogosphere will generate a lot of noise about this. but it’s important to remember that science doesn’t work because people are polite at all times. Gravity isn’t a useful theory because Newton was a nice person. QED isn’t powerful because Feynman was respectful of other people around him. Science works because different groups go about trying to find the best approximations of the truth, and are generally very competitive about that. That the same scientists can still all agree on the wording of an IPCC chapter for instance is thus even more remarkable.

No doubt, instances of cherry-picked and poorly-worded “gotcha” phrases will be pulled out of context. One example is worth mentioning quickly. Phil Jones in discussing the presentation of temperature reconstructions stated that “I’ve just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (ie from 1981 onwards) and from 1961 for Keith’s to hide the decline.” The paper in question is the Mann, Bradley and Hughes (1998) Nature paper on the original multiproxy temperature reconstruction, and the ‘trick’ is just to plot the instrumental records along with reconstruction so that the context of the recent warming is clear. Scientists often use the term “trick” to refer to a “a good way to deal with a problem”, rather than something that is “secret”, and so there is nothing problematic in this at all. As for the ‘decline’, it is well known that Keith Briffa’s maximum latewood tree ring density proxy diverges from the temperature records after 1960 (this is more commonly known as the “divergence problem”–see e.g. the recent discussion in this paper) and has been discussed in the literature since Briffa et al in Nature in 1998 (Nature, 391, 678-682). Those authors have always recommend not using the post 1960 part of their reconstruction, and so while ‘hiding’ is probably a poor choice of words (since it is ‘hidden’ in plain sight), not using the data in the plot is completely appropriate, as is further research to understand why this happens.

The timing of this particular episode is probably not coincidental. But if cherry-picked out-of-context phrases from stolen personal emails is the only response to the weight of the scientific evidence for the human influence on climate change, then there probably isn’t much to it.

There are of course lessons to be learned. Clearly no-one would have gone to this trouble if the academic object of study was the mating habits of European butterflies. That community’s internal discussions are probably safe from the public eye. But it is important to remember that emails do seem to exist forever, and that there is always a chance that they will be inadvertently released. Most people do not act as if this is true, but they probably should.

It is tempting to point fingers and declare that people should not have been so open with their thoughts, but who amongst us would really be happy to have all of their email made public?

Let he who is without PIN cast the the first stone.

Update: The official UEA statement is as follows:

“We are aware that information from a server used for research information
in one area of the university has been made available on public websites,”
the spokesman stated.

“Because of the volume of this information we cannot currently confirm
that all of this material is genuine.”

“This information has been obtained and published without our permission
and we took immediate action to remove the server in question from
operation.”

“We are undertaking a thorough internal investigation and we have involved
the police in this enquiry.”

Update II: Please comment on the next thread.


1,092 Responses to “The CRU hack”

  1. 1051
    CB says:

    http://www.realclimate.org/index.php/archives/2009/11/the-cru-hack/comment-page-21/#comment-143539

    Leonard Herchen #1028

    You asked about splicing temperature data onto a proxy record. Do you recall the specifics of the the study? As someone not directly involved in the particular research I believe you refer to I can only make a general comment.

    If I had a long proxy dataset that overlapped with instrumental data, and I could demonstrate that during the period of overlap there was a strong relationship between the two, then it might be reasonable to merge the two and present them as a single timeseries in a figure – particularly if the instrumental data extended further than the proxy series and the long-term trends were important to show. If I did so I would want to follow standard practice and clearly identify the contributions of the proxy and the instrumental data to the figure – perhaps by using a different color for each and being sure that it was clear in any accompanying text how the figure was constructed. It is perhaps more common to simply overlay the two timeseries on a plot showing the similarity during the period of overlap. Does this speak to your question?

  2. 1052
    Rod B says:

    Dan Hughes (way back in 878; tough keeping up…) I feel obligated to correct a fellow skeptic.

    Employers do have legal access to their employee’s emails. However, that is not the same as public access.

    The public at large does not have a right to access carte blanche a public employee’s work product.

    Writing emails to personal contacts within the subject matter of one’s endeavors is not the same as “personal use” within the meaning ‘can’t make personal use of employer’s equipment.’ Even some pure personal use can not be prohibited. Most employees are allowed to use the employer’s facilities — in the redneck meaning of the term, e.g.

  3. 1053
    EL says:

    738 – On models and prediction

    Interpolation and extrapolation are the two main ways one can use a model. Interpolation is used for modeling within a set of data points, and extrapolation is used for modeling the future. Models are by their nature better with interpolation than extrapolation. Extrapolation usually comes with greater uncertainty than interpolation when even remotely complex systems are being modeled. Mathematicians are being asked by the climate science community to help better the extrapolation abilities of the models, but extrapolation will still come with some degree of uncertainty that mathematicians will not be able to remove. In other words, after mathematicians have went through the models with a fine comb, the models are still going to have some degree of uncertainty when predicting the future.

    747 – On Global Cooling.

    Comment number 747 asks “Answer as honestly as you might. What do think the state of climate change would be today if all of the grant money was allocated to “global cooling”

    If grant money was allocated for the study of global cooling, the researchers would arrive at a contradiction.

  4. 1054
    Ricki (Australia) says:

    RC – you are doing it tough at present, but time will moderate the response. It is just a matter of time before the climate projections become obvious to all and sundry to have been essntially correct. The real problem is that we are running out of time to react to the problem.

    In my opinion, we have already started melting of the ice sheets and this will continue for thousands of years. The real question is just how much further we will push this climate before we do something real to stop it.

    Good on you Gavin, keep your chin up.

  5. 1055
    Brewster says:

    J #1035 – as Gavin said, you aren’t even trying, and it IS unworthy.

    Beyond that, it’s obvious you didn’t read your own reference.

    In the Discussion, one statement stands out:

    “This result is essentially consistent with the findings reported by Intergovernmental Panel on Climate Change [2001] and Hansen et al. [2002] suggesting that almost all warming since 1950 had an anthropogenic
    origin.”

    Hard to say that disproves AGW.

  6. 1056
    J says:

    And here’s Scafetta response to Benestad and Schmidt:

    http://climatesci.org/2009/08/03/nicola-scafetta-comments-on-solar-trends-and-global-warming-by-benestad-and-schmidt/

    Climate is an extremely complex, chaotic system, accurate prediction has yet to be accomplished. Scientists disagree with each other. Those who disagree with you are deemed “unworthy.”

    That’s the point.

    [Response: No it isn't. Scafetta could have done a proper analysis that wasn't guaranteed to give the answer he first thought of, and which fails miserably in cases where we know the right answer. This was "unworthy" regardless of the conclusion drawn. - gavin]

    If you don’t wish “ANY scientific paper, or set of scientific papers, that disprove, to any degree at all, the anthropogenic influence on climate, with particular regard to warming since 1850″ then don’t ask for them.

    If you really do want them posted, here’s another:

    Stephen Schwartz, Journal of Geophysical Research; Climate is 1/3 as sensitive to carbon dioxide vs. the IPCC:

    http://www.ecd.bnl.gov/steve/pubs/HeatCapacity.pdf

    [Response: Another terrible choice. What you are demonstrating is that bad papers are a commonplace in the mythology you have built up that you are using to prove to yourself that AGW is not a real problem. - gavin]

  7. 1057
    Jeff Id says:

    #922, I missed nothing. The question regards the chopping of data simply because it doesn’t relate to temp. More specifically it regards the validity of the rest of the series if the important bit (which is claimed to prove the data is temp) doesn’t match.

    Giving it a title does not give it validity nor does it make any difference to resolving the issue.

    I don’t see why climate science needs this data anyway. Just do the right thing and say it ain’t temp and find another set of data.

  8. 1058
    john byatt says:

    vukcevic #957

    do you mean like creation scientists debating actual scientists?

  9. 1059
    Michael says:

    re: ccpo 1031

    Doesn’t the IPCC say there is a 90% chance that the change in temps is anthropogenic? Therefore doesn’t this mean there is a 10% chance it isn’t. Is that 10% chance of it not being anthropogenic “to any degree at all” for you?

    BTW I didn’t mention ethics I mentioned moral equivalence, you didn’t understand my question.

    Regards

    Michael

  10. 1060
    Hank Roberts says:

    J, be skeptical, do what we all have learned to do. If someone tells you a paper is conclusive proof of something — do the right thing.

    Look it up.

    Paste the citation into Google Scholar and look for citing papers and later work.

    Once you get the abstract, you’ll have a fair idea if the paper you were told about is really a good one, or if someone’s blowing smoke.

    In the example you gave, the citing paper Tim pointed you to is open access, you can read the whole thing. Did you bother? Look back for the post:

    “… Timothy Chase says: 22 novembre 2009 at 9:32 PM”

  11. 1061
    S Jones says:

    Hi all, I am a somewhat regular reader who has never posted before, but I would like to give a little of my own perspective in general, not responding to specific posts. First I am a biochemist, I have not made up my mind personally on global warming (a topic I don’t feel qualified to lecture others on), but I would like to make a few general points about science and how people are reacting to these emails.

    First the authors are clearly trying to sex up their graphs (this means they are trying to make them more dramatic), this is not good scientific practice, and the authors should a little ashamed by this. But it is also not uncommon in science, and many other fields, Nate Silver at fivethirtyeight.com spoke more eloquently about this than I can, but the underling point is that sexing the data is very different than making it up, withholding, or improperly manipulating it. The underling science is not changed by what the authors did, there papers will not need to be retracted, and I would be very surprised if anyone within the scientific community is really that surprised or upset by this whichever side of the debate they are on.

    Second the “trick” issue. Several posters have very helpfully posted the definition of trick to show how these scientists are being dishonest. However it seems pretty clear to me from my reading of the emails that trick is being used in the place of words like protocol, method, and technique. This is a very common thing for scientists to say. For curiosities sake I just flipped through 25 of my most recent work related emai’s and found that I had done this four times in that many emails. In every one of those cases it was when I meant technique or protocol such as

    Hey ##### I was thinning we should try Wu’s trick for measuring particle concentration by intravital microscopy.

    In that line I am not implying that Wu et. al. has a deceitful way of measuring particle concentration, I am saying that they have a very cool and clever protocol for doing this that we should use.

    I think in general the problem with these emails is that when two scientists have a personal discussion they are liable to drop into a scientific short hand that to an outsider could be easily misunderstood. I think if you all think about this the professionals amongst might realize that this could easily happen in other fields, economists might sound like they are trying to mislead investors, lawyers might sound like they are trying to obstruct justice, and is some cases they probably are. But unless you are fluent in that short hand it is hard to tell if there is a true problem or not.

    For me the take home message is that 1 this changes nothing about the climate debate in either direction, 2 it is shocking the lengths that some people will go too to help there own cause (like hacking a personal email), 3 that personal emails that were never supposed to be published probably should not be published!

    Sorry for the length of this post.

  12. 1062
    gluggle says:

    Of course, we could just look at the actual weather – the vicious heatwaves and deaths that are already sweeping Australia. To the skeptics, these heat deaths are just healthy and normal, as are the new infections with dengue fever etc.

    Perhaps the ‘skeptics’ (lets face they are usually lying animals employed by the fossil fuel industries) could visit the families of the dead to tell them that their relatives died from healthy normal weather.

    I do so hope in years to come as the evidence comes in on AGW that the skeptics get the full, rich reward they so rightly deserve.

  13. 1063
    Michael Anderson says:

    Hi folks;

    Not only did those hackers commit crime which crosses numerous juridictions, it shows the base level at which they are willing to stoop to defend their failing positions .

    The question in my mind is who put them up to this? Who is behind this ridiculous act that will certainly backfire on them.

    This is my position
    Any scientist whom is not a skeptic is a fanatic… And vice versa regarding the public as well on many issues.

    There are times that I read a research paper and go whoa! Where did that conclusion come from? How did they get it? and where’s the basis for it?

    There are outstanding scientists, technicians, and public volunteers out there gathering the data and working to come to understandings as to what nature [the climate] is doing,

    At most, I can concentrate on the areas I am working in. Science has thousands of specialties that folks focus their energies on, from nematodes, to atmospheric physics, to biology.

    Science is not about finding ultimate truths as some have been sadly led to imagine.

    It is about quantifying aspects of nature and reaching an understanding of those aspects, and if I quantify some aspect of nature fine.
    If not, I then try to understand why my inquiry doesn’t reach the conclusion I expected.

    Nature, and scientific investigation in its simplest acceptations, is a collective name for all facts, actual or possible, or (to speak more accurately) a name for specific phenomena or process, partly known to us and partly unknown, in which all things take place and henceforth observing, gathering data and processing a complete knowledge of them as is possible, to which conception and understanding as the aim of science to raise itself by successive steps of generalization, hypothesis, observation, theory based on experience, father observation, experimentation, validation and proof to quantify and understand a particular aspect of nature.

    Some answers are best guess given the data and our ability to interpret it and form conclusions that are consistent with the data, not with politics, faith, business philosophies, or dogmas. It has to be fact based, straight forward and verifiable in as best we can.

    There are so many disciplines’ working on climate the amount of information is astounding. If I tried to delve into every area, I would not be able to focus on anything. It may be easy for some; however, with a brain injury my focus is critical to my work.

    However, the phenomena and processes of the human effects on the climate have presented themselves, they have been and are being observed, data collected, and understandings developed which are evolving hour by hour.

    In my mind working these models, the thought of the oceanic dynamics being the 1000 pound gorilla has been prevalent.
    These facts of science confirm to us that other unseen processes are underway.
    I am of the opinion that we are far beyond the postulation of such immense changes. Now

    One cannot add chemistries and other man-made cause and effect dynamics to any system without subsequent dynamic stoichometric consequences.

    Any contribution to ecosystems on a micro-macro scale has consequential ramifications on a scale of magnitude yet to be quantified and fully understood; and is propagating through adaptation, and evolution the creation of new species of bacteria and viruses as well as contributing to the genetic adaptation and alteration of existing known bacterial and viral pathogens].

    Anderson’s First rule of ecosystem consequence dynamics.
    “Any physical, chemical, thermal, biological, or mechanical contribution to any ecosystem other than those derived through natural processes; forces exponential dynamic adaptation, evolution, and creation of new species and the extinction of existing species.” [Anderson, et al]

    After nearly two centuries of industrialization, and millenia of human contribution to this dynamic process of hyper-adaptation and evolution; we are at a crossroad of action versus inaction on changing this mechanism

    These new emerging and adapting species present a risk to ultimate human existence.

    The conseqences of bio-adapation and evolution will present the growing risk on a scale unimaginable.

    I have seen the effects of sea-level rise on arctic shores, reduced sea ice and shorter freeze cycles, in the permafrost, the fauna, changes in the wildlife patterns, dislocation of indigenous villages inland, and deaths of polar bears and potential of their extinction.

    I for one would prefer not to taunt that 1000 llb gorilla and not see those potential consequences. At 750 ppm we have seen from studies of drill cores from Madagascar that there was a change in the climate.

    I have read the fine research,

    The Copenhagen meeting will determine whether or not we taunt that gorilla. I’d like to bet that we make an option as a global society not to taunt it any further than we have.
    In essence, despite the debunkers, skeptics and other what nots; we will continue to remaiin the contributing architects of our own potential demise .

    To my colleagues:

    Best wishes,

    Michael

  14. 1064
    Connor says:

    Sorry Gavin,

    But regarding the h-index. This is the full body of the text, I inadvertently left some out

    Email 1: Mike to Phil
    Anyway, I wanted to check w/ you on two things:

    1. One thing that people sometimes like to know is the maximum value
    of
    “N” where “N” is the number of papers an individual
    authored/co-authored
    that have more than N citations. N=40 (i.e., an individual has
    published
    at least 40 papers that have each been cited at least 40 times) is
    supposedly an important threshold for admission in the U.S. National
    Academy of Sciences. I’m guessing your N is significantly greater than
    that, and it would be nice to cite that if possible. Would you mind
    figuring out that number and sending–I think it would be useful is
    really sealing the case.

    Email 2: Phil to Mike
    On 1, this is what people call the H index. I’ve tried working
    this out and there is software for it on the web of science.

    Problem is my surname. I get a number of 62 if I just use the
    software, but I have too many papers. I then waded through
    and deleted those in journals I’d never heard of and got
    52. I think this got rid of some biologist from the 1970s/1980s,
    so go with 52.

    Email 3: Mike to Phil
    OK–thanks, I’ll just go w/ the H=62. That is an impressive number and
    almost certainly higher than the vast majority of AGU Fellows

    Sorry to take up your time with this s**t but its really hard to counter denialist claims because I (like them, except that I can actually admit it) don’t understand a lot of what is being discussed nor the context it is being discussed in. If you could find a few moments to quickly explain this it’d be greatly appreciated.

    [Response: What do you want explained? Mike obviously made a mistake or misread the email or something. What difference does it make to anything? 52 is still a great number. - gavin]

  15. 1065
    J says:

    [Response: You're welcome. Scaffetta doesn't disprove AGW to any degree. Hint: if you assume that only one factor matters..]

    What he does speak to is the larger amount of one non-Anthropomorphic factor and that factor’s influence on others. And this goes to disprove the degree of anthropogenic influence on climate.

    The request was for: “ANY scientific paper, or set of scientific papers, that disprove, to any degree at all, the anthropogenic influence on climate..”

    So Scafetta does apply.

    [Response: "To any degree" would imply something greater than zero. -gavin]

  16. 1066
    Terry says:

    Gavin…Congratulations on your patience and dedication in responding to the huge volume of posts in what has been a deluge. Im sure it has been a massive job and I trust they gave you a few days off in lieu. Personally I belong in the skeptical camp (not denier) but I admire your efforts on this one. Cheers.

  17. 1067
    Richard Barrett says:

    Rod B wrote: “Employers do have legal access to their employee’s emails. However, that is not the same as public access. The public at large does not have a right to access carte blanche a public employee’s work product…”

    Well, perhaps when you’ve sorted out the lies that government ministers tell the public, you can then come back to condemning the public for being suspicious of anyone working for the government.

    WMD that can be deployed in 45 minutes, anyone? No money for a social safety net, but plenty for the banks.

    It’s time for the public to be heard, not once again silenced and treated like illiterate peasants. The public ARE the employers of all who work in government.

  18. 1068
    Leonard Herchen says:

    1050 CB
    Thanks for responding to my question. A few comments:
    1. Because the proxy covers millenia and teh temperature record only a century or less, it is impossible to demonstrate a relationship between the two (other than a theory that says they are related, which begs the question)
    2. It is my understanding the assertion is that the data were spliced to become a one colour line on the graph so that the character of one curve in the last 1/2 of this century changed.
    3. Your comment on standard practice is exactly my question. The second 1/2 of the assertion on CA is that they state that the very scientists who did the splicing stated in no uncertain words that this was not standard practice and shouldn’t be done.

    I’d like to keep discussing this as well it would be nice if Gavin addressed it, as it seems fairly straightforward yea or nea type question.

    Leonard

    [Response: It was clearly not standard practice because Jones' WMO graph is the only one in 10 years AFAIK that showed it. And prior to Thursday, no-one was even aware of this one. I'm not a dogmatic type and I never say never, but my general rule is that captions should describe clearly what was done in a particular figure. The caption on this one was incomplete and had I seen this at the time, I would have suggested amending it. But why do you care so much about one 10 year old graph when there must have been hundreds of very similar graphs made by dozens of people in dozens of publications? If you want to talk about what the community has conferred over and decided to present to the world, the graphs in the IPCC report are much more relevant. Unless of course you want to start talking about some of the extreme fakery (not just a badly described captions) from Monckton, Michaels, Beck, Durkin, and the like. - gavin]

  19. 1069
    George says:

    Gavin in your response to 187 you say that CRU has not lost or destroyed any data. I was wondering where you get this information as on Aug 11 2009 CRU had the following on their age on data availability (now not accessible):

    “Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data.”

    [Response: No data has been destroyed, the original files and numbers are with the national weather services that provided them. Removing a copy of a original file because it is not useful for my purposes is not 'deleting data' - gavin]

    I have to say that I find this attitude astonishing. The single most valuable asset in my data shop are the original data files. With our documentation everything can be re-created from them if necessary (it would be ugly, but it could be done). Without them, however, we have absolutely no way to determine if modified and more immediately “useful” files contain computational or coding errors. Original data files in our shop are ALWAYS read only, cannot be modified, and are backed up in several locations.

    Why on Earth would you cast aside these data?

  20. 1070
    joshv says:

    “The is not ‘assuming the answer’, it is accurately predicting future observations. ”

    Tell me Gavin, how does one gauge the accuracy of a predicted future observation? Warm up the flux capacitor in the DeLorean?

    [Response: You wait and see. As was done for Pinatubo, or the predictions made in the 1980s. What would you rather we do? - gavin]

  21. 1071
    J. Bob says:

    #981 Hank, so you do go to WUWT. – OK, as in any analysis, it’s a good idea to cross check results with other methods. The following figure is a cross check on the E. England data, and Fourier filtered result.
    http://www.imagenerd.com/uploads/ave1-raw-smoothed-5rhsb.gif

    Included are three filtering methods, MOV, Fourier and “filtfilt” (reverse) recursive 2 pole Chebushev (0.025 yr. Cut-off). The “filtfilt is described in the MATLAB Signal conditioning Toolbox, and is basically a filter that runs forward and backward in time.
    The resultant is a 4 pole filter. In this case, the recursive and Fourier filter track each other fairly well except at the end points, where the “filtfilt” breaks down. The 40 year MOV filter by it’s nature, cuts off 20 years at the beginning and the end points, which are the most interest.

    However another method of cross checking is to compare a EMD (Empirical Mode Decomposition) filter against the Fourier filter. In a paper “On the trend, detrending, and variabilty of nonlinear and nonstationary time series” by Wu, Huang, et. al., they use the CRU data from 1855 to 2003. The figure below compares the EMD and Fourier method, using the CRU data. In this case, the results agree quite well over the full range.
    http://www.imagenerd.com/uploads/cru-fig-6-NMyC0.gif
    It would seem to indicate that the Fourier filter does very well in evaluating results at the end points of the time series. A plot of global composite temperatures from Climate4you.com, and superimposed Hadcet and E. England data is shown below. The plot shows a flattening of temperatures over the past 6-8 years, the same as indicated by the ave1 figure above.

    http://www.imagenerd.com/uploads/climate4u-lt-temps-Ljbug.gif
    I guess I would go with the Fourier convolution method, it seems closer to reality.

  22. 1072
    Jere Krischel says:

    [Response: Which basic premise is that? The reason you aren't seeing debates in the emails over whether CO2 is increasing because of human activity is because it's done with. We know the answer. Why don't the debate whether CO2 is a greenhouse gas? Because that too is over. What people debate now are not the 'basic premise', but the details - and as you can read there is plenty of debate about that. - gavin]

    The basic premise that increased CO2 emissions from human activity is going to cause catastrophic warming. More specifically:

    1) what % of CO2 increasing is due to natural variability, and what % is due to human activity – put a number on it, and tell me what would change your mind about your hypothesis.

    [Response: 100% of the ~110ppmv increase. This would be hard to shift because of the wealth of evidence. You'd have to show there was something wrong with the isotope studies, the emissions budgets, the accumulation in the ocean, the accumulation on land, the decrease in oxygen etc. - gavin]

    2) what is the maximum greenhouse gas effect possible with CO2? With faked “forcings” and all. What data would change your mind about that part of the hypothesis?

    [Response: There is no relevant maximum effect. Venus would have to be shown to be much cooler than we think, and thousands of spectral measurements would have to be revised away. - gavin]

    We can both agree that CO2 is released by humans, and that CO2 is a greenhouse gas, but taking the next step and asserting that the amount of CO2 released by humans is going to cause a runaway greenhouse gas effect does not follow. This is the “basic premise” that the theory of human released CO2 catastrophic global warming that is not challenged in any serious way by pro-AGW proponents.

    [Response: Now you are jumping to conclusions. Who said 'runaway' and what do *you* mean when you use that term? If you want to know why increasing CO2 is a problem, I wrote it down in six easy steps for you. - gavin]

    I’d like to see a pro-AGWer, an honest pro-AGWer, say something like, “I believe that 50% of the increase in CO2 is due to human activity, and this will cause a catastrophic 5C increase in temperatures in 100 years. You can prove me wrong by showing (a) or showing (b).” Where (a) and (b) could be something like, “find any point in the historical record where CO2 levels went from 200ppm to 400ppm within the course of 100 years when human activity was low”, or “find any point in the historical record where CO2 levels were at 1000ppm but temperatures were lower”. Fix (a) and (b) as you like, but please, something falsifiable is what makes it real science.

    [Response: Please read the IPCC reports (and not just the summary). All of these steps are outlined. - gavin]

  23. 1073
    Larry Lidar says:

    (piggy-backing on earlier comments by Jen (#753) and Deech56 (#836)…)

    many thanks for providing such measured and informative responses to the huge number of posts that have appeared here over the last few days

    and FWIW, after a bit of perusal, I too gotta believe that the “random selection” of e-mails has been carefully vetted by someone with a political rather than scientific agenda

  24. 1074
    Jere Krischel says:

    #926: If one produces a study that goes against the theory, it doesn’t negate the theory because of the large volume of studies that support it. If a large number of studies go against the theory it is weakened, but it probably won’t be thrown out until the aha moment when a new theory explains more of the data than the old one.

    Neither does a large volume of studies that claim to support a theory prove it, especially when these studies in peer reviewed journals are carefully screened by people who do not want a skeptical view of their theory.

    If one produces a study, which falsifies a premise of a theory, and that study is repeatable, we do know that the theory is wrong at least in some way. Whether or not to throw out the theory depends on whether it can be modified, applying occam’s razor of course, in such a way to explain the contrary study. It is not necessary to have a new theory before discarding the old one. It is quite possible that climate is simply not predictable, and that it behaves in a chaotic enough manner to defy a unified one-input/one-output theory such as AGW.

  25. 1075
    RaymondT says:

    Gavin, How do climatologists first identify the ENSO and then remove it as referenced in the text below?

    Excerpt from [1255553034.txt]
    At the risk of overload, here are some notes of mine on the
    recent lack of warming. I look at this in two ways. The first is to
    look at the difference between the observed and expected anthropogenic
    trend relative to the pdf for unforced variability. The second
    is to remove ENSO, volcanoes and TSI variations from the
    observed data.
    Both methods show that what we are seeing is not unusual. The
    second method leaves a significant warming over the past decade.
    These sums complement Kevin’s energy work.
    Kevin says … “The fact is that we can’t account for the lack
    of warming at the moment and it is a travesty that we can’t”. I
    do not agree with this.
    [end excerpt]

    [Response: See here for an example. - gavin]

  26. 1076
    Peter Backes says:

    Still screening out the people who find it interesting to wish various nasty diseases upon us, encourage us not to breathe anymore, or whose fantasies regarding prison life reveal a little more information about them than I’d ever want to know. – gavin

    Coming from the Fox News contingent no doubt.

    Yes, a grand conspiracy clearly originates in our institutions of higher learning – NOT.

    To the reality-challenged: Governments and corporations are already staking out oil claims in the rapidly de-icing Arctic. Shipping companies are anticipating being able to reliably navigate the famed Northwest passage in the next few years. Clearly they (and the folks at RC) get it – why don’t you?

  27. 1077
    Mark says:

    Obviously scientific consensus agrees with warming due to man. A skeptic can’t really argue with those facts. A skeptic can only argue how dangerous it might turn out to be. The unknown future is the only thing that skeptics are really banking on. The argument that climate model forecasts won’t fortune tell rises in temperatures with much accuracy. Perhaps that is a reasonable thing to have doubt for. Verification is certainly not something held with much accountability. Anyone can predict recklessly that the Arctic will melt in 2030 or 2015, or that temperatures will rise 1-4.5*C. The IPCC makes great indications of knowhow and scientific justification, which I embrace. But I certainly will not bank on their predictions with absolute faith, even though global warming could and may turn out to be, in fact, “dangerous.”

  28. 1078
    Sloop says:

    Response to Steve Fish at 739,

    I’m a little disappointed at nobody venturing into the questions I’ve raised; but as I follow along, I’m gleaning a little material here and there so that I can address them myself.

    I’ve worked in water resources management and policy (along with work in wastewater engineering) for about 23 yrs, and currently work in state government. I offer the following tentative synopsis of the progression of science, governance, and management in an attempt to establish broader insight as to where we are now and what steps should be taken to deal with the policy issues of atmospheric/oceanic/climate change.

    Climate change started receiving serious attention in the late 80′s among coastal/ocean scientists and managers when I was in grad school (Ph.D.’s in coastal environmental science and policy). There was healthy professional skepticism on the part of many as to the ultimate cause of what was being observed by diverse monitoring systems and as to the validity/robustness of the global climate models that were starting to taking shape. For much of the 80′s to 90′s the stance of many managers and scientists was, “whatever the ultimate cause, there are things going on environmentally (habitat alterations, sea-level rise rate accelerations, etc.) that I have to understand more in order to start addressing them.” (It was also clear to many that there were good reasons other than climate change to start transforming humanity’s fossil-fuel based economy and society.) In sum, within the realms of applied environmental science and management, the biological, ecological, chemical, and physical symptoms or consequences of global atmosphere/ocean/climate change started to be widely appreciated over two decades ago.

    Consequently, starting in the 70’s and continuing to this day, the advancement of climatology was accelerated by both the scientific community and science funding, policy, and management institutions globally. The fruits of that acceleration have been harvested in the last ten years or so and will continue to come in more substantively.

    Governments grapple with many difficult problems/issues that far exceed our collective capacities to address them. The “back-burner issue” (The issues you haven’t been paying as much attention could at anytime become the most serious one on yr agenda.) is a common and worrisome dynamic. Hence, the strategic managerial stance adopted is to de-prioritize issues that are relatively less certain scientifically (or possess trans-scientific properties). If climate science were truly controversial scientifically, then executive authorities would be strongly disposed to push AGW down the agenda because there are so many urgent problems already. AGW has been doing the opposite—forcing itself up the agenda as the science advances.

    Thus, the public (and scientists who focus on basic research) needs to understand better that executive agencies cultivate and apply expert judgment on which issues are truly urgent based upon evaluations of the state of the science, monitoring (despite its inadequacy across the spectrum), technology development, risk assessment, etc. Executive agencies could always perform such evaluative judgments better. But we do not need or seek these problems simply to justify more funding for our orgs or to advance a particular set of values. The obvious, dynamic, and diverse challenges of environmental protection, restoration, and sustainable development we face, even if AGW didn’t exist, ensures that we’re a growth industry already.

    Because on the “frontlines” we’ve been racing to address the existing and expanding consequences of global atm./ocean/climate change, many managers have not been paying attention to what has been going on re: ‘denialists’ and the climatologists. It simply is not a useful dispute for us.

    The timing of this hacking incident as the Copenhagen meeting is about to convene and as the US Congress finalizes its climate change legislation is not coincidental or unexpected.

    An analysis of the denialists’ arguments, communications campaigns, and political assets will reveal more fully their purpose and motivations and should afford development of better ‘counter-insurgency’ initiatives. Fundamentally, many see that their interests center not upon the science, but upon the public and achieving desired outcomes in democratic governance processes.

    After a few weeks of following this, I offer some initial points and conclusions (though not really the above mentioned analysis). One is that this is a sophisticated and deadly serious long-term campaign being executed by highly motivated and intelligent people, most of whom are not evident, not kooks with a delusional agenda. Two, there is a lot of potential payoff for certain interests simply in delaying the inevitable government actions; inevitabilities they see too. Three, the climatologists who have been targeted have been too long fighting this themselves. They have allies, some of whom need to pivot and provide support and cover.

    Apologies if the militaristic language is distasteful. It’s merited.

    Dr. G. Schmidt: well-done. Perhaps some rest is in order?

  29. 1079
    ccpo says:

    Rational criticism must be ENCOURAGED, not suppressed or ridiculed.

    Comment by Hans Scundal — 22 November 2009 @ 2:01 PM

    Show us some rational criticism. I’ve yet to see any from any quarter. See my challenge above.

    It is plain to me as a layman who is trying to educate myself about AGW that I can no longer trust at all relevant research is published because of the endemic censorship within flagship climate institutes.

    Comment by Allan — 22 November 2009 @ 4:31 PM

    You have proven you are a denier in two easy steps:

    1. You repeat the crap posted by deniers all over the web that those e-mails somehow show scientists can’t be trusted and cast all science in question.

    2. You claim to be an objective observer, but disprove that with #1.

    This tired charade has been used by most deniers trying to present the false indication that there is legitimate dissent about AGW.

    “[Response: Preventing an ice age is a long way from being a problem worthy of immediate attention. - gavin]”

    As is, maybe, AGW?

    [Response: Sorry, but no. - gavin]

    Comment by conversefive — 22 November 2009 @ 4:53 PM

    I disagree with you, Gavin. Given that fossil fuels would be quite useful in mitigating a coming ice age, keeping them in the ground till needed might be wise, regardless of how far in the future. One of the reasons we are in global overshoot is the failure to think in terms of the far future.

    [Response: If you wanted to quickly increase the greenhouse effect you'd use HFCs, not CO2. Much cheaper. - gavin]

  30. 1080
    Jere Krischel says:

    #1041: To reject anthropogenic causation, you must turn your back on all those mountains of evidence. What better example of denial is there?

    Mountains of circumstantial evidence can be refuted, and to reject that as even a possibility is a better example of denial.

    I’ll admit, it’s much easier to poke holes in someone else’s fragile theory (especially when it comes to light that the data has been fudged, the code hasn’t been maintained, and the results are pushed in a pre-determined direction) than it is to come up with a more predictive theory, but frankly, that’s simply not the point of science. Knowing what we don’t know is as important as knowing what we do, and if the long and short of it is that AGW can be wrong without anyone coming up with a better explanation. That uncertainty might frighten some people, but it is the nature of complex and chaotic systems.

    The sad part of the whole debacle is that the people doing real science, auditing results and acting as the skeptical eye for this theory, have been marginalized, conspired against, and abused. Phil Jones should’ve become McIntyre’s best buddy, and offered him a research position at his institution to keep people honest.

  31. 1081
    Leonard Herchen says:

    1050 CB
    More to your question:
    On this page in RC,
    http://www.realclimate.org/index.php/archives/2004/12/myths-vs-fact-regarding-the-hockey-stick/
    Mann states clearly that the instrumental record is clearly distinguished from the proxy record see discussion in comment 4. Over on the CA site they state that the “hide the decline” comment proves that this was disingenuous,

    As I understand it, if you go to this site:
    http://www.climateaudit.org/?p=1553#comment-340175 and here
    http://www.climateaudit.org/?p=7810#more-7810
    They say that they can only duplicate some graphs that show up in the IPCC report by grafting the temperature record on to a proxy.

    That leaves me with the simple question, “was the shape of a proxy record changed by including instrumental temperature record in in way that the proxy record was shifted from where it would have been in the 20th century?”

    [Response: I am looking at the last IPCC AR4 report, figure 6.10, and I can see no evidence of this at all. The description of the smooth and the end-point padding in the caption is unambiguous. - gavin]

    Obviously, if the answer is yes, and this was bad science, then why was it done?

    If you read the comments at CA, some people think this is some kind of smoking gun, but I don’t think the whole AGW case rests on this graph. It probably is a looks worse than it is kind of thing. However, I would like to know if the assertion is true from some on on the RC website. I think it would clear the air a lot.
    Leonard

    [Further response: The point that has been made a number of times is that the reconstruction (the raw annual values through 1980) has never been presented with the instrumental values (available after the end of the proxy record in 1980) as a single "grafted" curve by Mann and collaborators (here). Indeed, the instrumental values and proxy-reconstructed temperature values have always been demarcated and clearly labeled as distinct (e.g. in Mann et al '98 and the extension back to AD 1000 in Mann et al '99) (data)

    However, there is some ambiguity regarding the smoothed curves used to indicate the long-term variations in the record, as the boundary conditions have not always been stated. Boundary conditions are sometimes used at the end of a time series so that the smoothed line can come up to the ends. The methods used for this end-point problem in smoothing are problematic, often ambiguous and various alternative approaches have been used in the literature. They were used in the last two IPCC reports and even in the most recent IPCC report, different methods are adopted in different chapters. Over the past 5 years (Mann 2004; Mann et al, 2008), we have favored an "optimal boundary condition" approach which minimizes the misfit of the smoothed data and raw data with respect to combinations of alternative boundary constraints.

    In some earlier work though (Mann et al, 1999), the boundary condition for the smoothed curve (at 1980) was determined by padding with the mean of the subsequent data (taken from the instrumental record). This does make a small difference near the end of the series. It doesn't effect any of the conclusions drawn in the paper though. These were based on comparisons of the individual reconstructed annual values (individual years and decadal averages over 10 consecutive years) from AD 1000-1980, with those from the recent instrumental record (1981-1998), and centered on the fact that the recent instrumental values were outside the error range of the reconstructed values over the past 1000 years and were not related to the smoothed curve. This figure shows the comparison of the originally published result with an alternative smoothing based on our more recent approach which does not use any instrumental data.

    Thanks for encouraging us to go back and check this. - mike]

  32. 1082
    Doug says:

    Can someone put some realclimate spin on what Phil Jones meant by the following email to Mann regarding skeptic research?

    “will keep them out somehow — even if we have to redefine what the peer-review literature is!”

    Thanks.

    [Response: I'll do my best even if I have to redefine spin to do it! - gavin]

  33. 1083
    J says:

    Re: Hank Roberts: “J, be skeptical..”

    Thanks Hank, I think I am being. We now (if my last post was accepted) have the paper, it’s rebuttal and the response to the rebuttal.

    I ask that you bear in mind the discussion this is part of. It’s was in response to a request for: “ANY scientific paper, or set of scientific papers, that disprove, to any degree at all, the anthropogenic influence on climate..”

    Now, if you logically arrive at the requests assumption you have: “There is NO scientific paper, or set of scientific papers, that disprove, to any degree at all, the anthropogenic influence on climate..”

    As you can see, this is far from your sensible request to be skeptical.

  34. 1084
    Bobby says:

    #1047
    Hank. Thanks for the explanation – it does make sense assuming that when the “invented” figures were worked into the 1995 data set, it was pointed out that December figures were extrapolated/estimated and not from the measured data. Clearly if “full year” figures for 1995 were published in Dec 1995, then anyone using the figures should have realised that at least some of the December figures had to be estimates.

    Interestingly I did a search on 1995 temperatures to see if/how this was disclosed and came up with the following:

    http://www.fumento.com/chills.html

    March 1996

    “In early January, the Climatic Research Unit at England’s University of East Anglia made headlines, including the New York Times, with a preliminary report saying 1995 was the hottest year on record. But their data were for only 11 months. Rather than risk December’s temperatures spoiling everything, they jumped the gun and sent out their press releases.

    Their fears were fulfilled when December’s average temperature came in at the lowest in 17 years. “It was a pretty ordinary year,” said NASA scientist John Christy, who has been analyzing satellite data on temperature since 1979. And James Hansen of NASA’s Goddard Institute for Space Studies, who pretty much started the whole global-warming scare, admits his study of land areas – where the effects of global warming would be most severe – revealed that 1995 was about 0.02 degrees celsius cooler than 1990. ”

    Clearly their “invented” temperatures turned out to not be very accurate and as such the email discussed how they didn’t want to make the same mistake the next year.

    The incident does highlight the difficulties in estimating future temperatures though – even just a month out.

  35. 1085
    ccpo says:

    [Response: you aren't even trying. This is indeed unworthy - see benetsad and schmidt (2009) for starters. - gavin]

    I fulfilled the request for: “ANY scientific paper, or set of scientific papers, that disprove, to any degree at all, the anthropogenic influence on climate, with particular regard to warming since 1850.” You deem it unworthy.

    Thanks for proving the point.

    [Response: You're welcome. Scaffetta doesn't disprove AGW to any degree. Hint: if you assume that only one factor matters, then it is rather predictable that this is what you think is important. Wait 1, 2, 3, .... for someone to take that out of context. -gavin]

    Comment by J — 22 November 2009 @ 9:13 PM

    J, you continue to perform as expected. 1. You claim to have met the challenge when the challenge clearly used the term “disprove” as opposed to what you seem to want it to mean, which is “challenge.” Any idiot can make a challenge; only an intelligent, skilled scientist is likely to disprove any part of AGW science.

    The paper was deemed unworthy because it was faulty. This is what happens when you are paid for your opinion rather than your work. It is also what happens when you violate basic scientific principles and look for proof of your hypothesis rather than simply testing the hypothesis.

    I repeat, simply because it must be, ad nauseum: AGW deniers have not one paper that has survived the peer review process, been published and been left intact. Of those that have been published, particularly in journals of good standing, I know of none that has survived post-publication review. Not one.

    Gavin, et al., do you know of any? Seriously, even one, that in any way challenges anthropogenic warming as the cause of warming since 1850?

  36. 1086
    Hank Roberts says:

    Nice detailed take-apart of email including references to “fraud” here:

    http://timpanogos.wordpress.com/2009/11/22/smoking-guns-in-the-clr-stolen-e-mails-a-real-tale-of-real-ethics-in-science/

    “Sure enough, with just a few minutes of searching the e-mails, I found references to ethical breaches in cooking of data, and a discussion about how to talk about the data and the issue in public.”

  37. 1087
    Steve Fish says:

    Dan Hughes — 22 November 2009 @ 9:16 PM:

    I am sorry if I misunderstood you but, if so, what were you saying that has anything to do with this thread. What are you talking about regarding “Climate ‘science’ attempts to invoke yet another exemption..?”

    The scientists in question were doing what all university scientists do in their intra/internet communications with the expectation that, short of illegal activity, it was protected by the university. For my university appointments I was routinely supplied with a single account that I was allowed to use for both personal and work activities.

    Also, where is the strawman?

    Steve

  38. 1088
    J says:

    [Response: "To any degree" would imply something greater than zero. -gavin]

    The argument is that the influence of solar activity is greater and greenhouse effect’s is less (rather that there are stronger interactions than the models predict). Since AGW concerns the influence of manmade CO2/greenhouse effect on warming, AGW’s influence is reduced – by an amount greater than zero.

    [Response: That would only be true if they had in fact produced convincing evidence that solar effects were larger than thought. They did not, and so the conclusion does not follow. - gavin]

  39. 1089
    Hank Roberts says:

    J. Bob writes:
    > so you do go to WUWT.
    Nope. Google does, however.

  40. 1090
    Leonard Herchen says:

    1068 Gavin:
    Thanks for responding. There are several reasons I care about the 10 year old graph, but the foremost one is simply credibility. I’m reading charge and counter charge in these websites, and my degree is in physics, not statistics or climate. So unless I can duplicate the work that you or MM do, I have to depend somewhat on my interpretation of credibility to decide who to beleive. Just like, if I was sick, I would go to the credible doctor, instead of getting my own medical degree. . The CA website threads ongoingly assert that climate science is closed system, doesn’t correct its errors and shouts down criticism. I don’t know if it is true or just internet breeze that I should ignore. So, I thought I would simply ask you the direct question and get the answer from you directly.

    Even without a degree in climate science, I can see that mixing proxy data onto temperature data yields results that may not mean much, and the CA guys are making a big deal about this. So I wanted to check if it really is that simple. That is, was it done and not clearly acknowledged? And if so, why?

    Again, I appreciate very much your attention to my comment.
    Leonard

  41. 1091
    Tom Fuller says:

    Dr. Schmidt at 1078: I cannot disagree more with your characterization of the skeptic/lukewarmer response as highly organised or well-funded. There are funded skeptics and skeptic websites. They are pretty much ignored. They act as corporate stop-points. The sites that are getting all the traffic–WUWT, Climate Audit, etc., are run on the site owner’s own dime, and although there is a good sense of collegiality, there is no agenda. In fact you’ll notice that they are as much reactive as proactive, if not more so.

    Gavin, you’ve done good work on this thread. Well done.

    [Response: Thanks. But I don't know what comment response you are referring to, and your claim that WUWT and CA have no agenda is laughable. - gavin]

  42. 1092
    gavin says:

    I’ve closed this thread, please go to the next thread to comment.


Switch to our mobile site