Much of the discussion in recent days has been motivated by the idea that climate science is somehow unfairly restricting access to raw data upon which scientific conclusions are based. This is a powerful meme and one that has clear resonance far beyond the people who are actually interested in analysing data themselves. However, many of the people raising this issue are not aware of what and how much data is actually available.
Therefore, we have set up a page of data links to sources of temperature and other climate data, codes to process it, model outputs, model codes, reconstructions, paleo-records, the codes involved in reconstructions etc. We have made a start on this on a new Data Sources page, but if anyone has other links that we’ve missed, note them in the comments and we’ll update accordingly.
The climate science community fully understands how important it is that data sources are made as open and transparent as possible, for research purposes as well as for other interested parties, and is actively working to increase accessibility and usability of the data. We encourage people to investigate the various graphical portals to get a feel for the data and what can be done with it. The providers of these online resources are very interested in getting feedback on any of these sites and so don’t hesitate to contact them if you want to see improvements.
Update: Big thank you to all for all the additional links given below. Keep them coming!
Re 241: Ray,
True, the models are consistent with each other. However, how consistent are they with nature? How many of those models forecast the timing and extent of the recent and ongoing Arctic sea ice decline? How many suggested that (GRACE) might be detecting loss of Antarctic ice in 2009?
I suggest that the failure to forecast Arctic sea ice decline is symbolic of the model’s inherent inability to deal with initial ice melt as a rather local process, rather than as a larger volumetric process.
The problem is that even small amounts of ice melt set up albedo and water vapor feedback process that result the abrupt melt of adjacent ice. More worrisome is that local melt can set up mechanical stresses that result in progressive structural collapse.
I assert that all ice melt processes resulting from global warming will occur much more abruptly than suggested by any of the GCM.
Each of the disciplines had its own literature, they exchanged ideas. They knew the kinds of approaches that other groups were using for their models, and nobody took a dramatically different approach. The builders of each model were drawing on the peer reviewed literature. None of the 23 models were built by “lone wolf” scientists. Nobody pulled a “Feynman” and made a point of not reading the literature. Thus, similar conceptual flaws are in each model.
@Christopher Hogan.
I also downloaded and ‘played’ with http://www.giss.nasa.gov/tools/modelE/ code.
This code is considerably different and has sufficient (actually quite good) documentation along with it.
If you compare modelE with http://data.giss.nasa.gov/gistemp/sources/GISTEMP_sources.tar.gz (which a source code that was suggested very early in this thread) you will see/understand the ‘frustration’ and the criticisms.
I’d rather ‘play’ with modeE …
Regards.
Where IS the data, indeed?
I’m looking for the evidence to support the claim made by “some prominent climate scientists” in the New York Times, as presented by Andy Revkin:
“[Some prominent climate scientists] say that the e-mail messages…have damaged the public’s trust in the evidence that humans are dangerously warming the planet”
What basis is there for claiming that the public’s trust in this evidence has been damaged?
What basis is there for attributing that alleged damage to the email messages, rather than, for instance, to misleading media on the issue?
Thanks.
“If I pay taxes, then my meteorological services at all levels should provide me with the data for free as a citizen.”
Then why did I have to pay 25 bucks for my driver’s license renewal? A hundred bucks for my passport?
Fast, cheap, good. Pick two.
Click here for fast and cheap.
http://rhinohide.cx/co2/gistemp/
I’ll update this later in the week as I beat out installation and configuration documentation.
I want to thank Gavin for supporting this discussion. On the whole, I think that it has been fruitful.
I especially appreciated the comment by Nick Barnes about the project using Python.
My PhD was in statistics, and I have experience modeling nonstationary multivariate biological (behavioral and physiological) time series using nonlinear dynamic models in SAS and Fortran-77. These projects, as I’ll call them, are much smaller in scale than the climate modeling.
Well, I’m a big-endian programmer, but what do you expect, seeing as I’m 55 years old? (sob :( )
(which end is big is left as an exercise for the reader, QED).
INTERNATIONAL JOURNAL OF CLIMATOLOGY
Int. J. Climatol. 25: 1055–1079 (2005)
Published online in Wiley InterScience
Data:
(www.interscience.wiley.com). DOI: 10.1002/joc.1148
AIR TEMPERATURES AT ARMAGH OBSERVATORY, NORTHERN IRELAND,
FROM 1796 TO 2002
C. J. BUTLER,* A. M. GARCI´A SUA´ REZ, A. D. S. COUGHLIN and C. MORRELL
Armagh Observatory, College Hill, Armagh BT61 9DG, Northern Ireland, UK
Received 5 May 2004
Revised 15 November 2004
Accepted 15 November 2004
I am a self uneducated retiree with some scientific background.
Does anybody on either side of this argument claim to be able to foresee the future? As far as I can see, computer models have the same predictive ability as chicken entrails, albeit using chicken entrails you get to eat a chicken as a side benefit.
Although it is tempting to believe that the past can be extrapolated into the future, it just ain’t so.
What usually happens is that people (not just scientists) confuse “assumptions” with “predictions”. Newton’s thoughts on gravity. Einstein’s thoughts on gravity etc., were all deemed to be practically useful at the time. Good enough to “predict” eclipses, demolish the luminiferous ether (believed in by a majority of scientists, I believe), and so on.
The future is unknowable. Prove that you can foresee even the next 5 seconds repeatedly with one hundred percent accuracy, and I will “predict” that civilisation as we know it, will end. You can easily work out why.
I won’t hold my breath.
[Response: Fortune telling is a class-B misdemeanor in NY state, and obviously no-one here is doing that! However, predictions are the lifeblood of science. Given certain, well-defined, conditions, scientific theories will predict what the result of a specific experiment will be. If the conditions are not met (the temperature is different, or it wasn’t a vaccuum etc.), the prediction doesn’t work. With respect to the future of the climate system, there are some external forces that are quite predictable – the orbit of the Earth, and its variations, the continuing rise in CO2 emissions, etc. Another example would be the climate consequences of a big volcano. In each case, you can use climate models to project the impacts on temperature, circulation etc. In fact, people have done this. In 1991 Hansen et al predicted the temperature drop that was expected from the Pinatubo eruption well in advance of it happening. In 1988, the same team projected how temperatures would increase under 3 scenarios of rising greenhouse gases. For the scenario that came closest to being realised, the trends were a pretty good match to what actually happened. Other groups have made skillful projections for the El Niño events and their consequences months in advance. So finding useful information about the future is not impossible, even if it isn’t fortune telling as described in the NYS penal code. – gavin]
Ben,
“We do not hold the original raw data but only the value-added (quality controlled and homogenised) data.”
I’m afraid that it is true, and it appears to be even worse that it sounds.
One would think that they could simply reverse the adjustments, and recalculate the original data. If you look at the annotations of the programmers doing the coding, however, it appears that they have lost track of how the original data has been adjusted.
[Response: The issue is not any adjustments, but simply that they maintain a master database with all of the data sets merged together. Having done something similar (though smaller scale) with multi-sourced data – some I digitised myself, some done by others, gathered over a 10 year period, including calibration adjustments and typo and meta data corrections, I think I have a pretty good idea why you can’t just break it down into constituent parts. As with CRU, all I have to give out is the merged product. – gavin]
Where is the raw HadCRU, GISTEMP, and NOAA NCDC data. Saying that it comes from GHCN doesn’t help. We have to know which stations over what time periods are being used. Reproduction is also going to require a site history for the surface stations. For example, if a site is moved, or if there is a change of time for data sampling. There seems to be a shortage of this kind of information – especially for HadCRU. Then we need all of the adjustment algorithms. This is probably more important than the code. Given the algorithms, we can quickly write our own code. Then, having the raw data, the meta-data, and the algorithims, reproduction should be possible. Excuses like, “we did this and you didn’t” won’t work. If we didn’t do what the sources did, it’s because the sources didn’t supply the information about what was done. Then if reproduction isn’t possible, there is a problem. I hope that this is not just one more example of Real Climate trying to appear to be transparent while at the same time hiding the important stuff.
[Response: Oh, I don’t know. Perhaps you could start by reading the papers? Or looking at the GISTEMP code? You keep saying stuff is being hidden, but when the whole thing is in front of you, you don’t look. Someone else in the comments has already downloaded the GISTEMP code and got it to work in the time that this post has been up, while you have been complaining for months that it’s all some secret. Who do you think has more credibility? – gavin]
Gavin, I hate to ask you to add one more thing to your already sizeable list, but since you’ve done an outstanding job dealing with the craziness so far I thought I’d suggest it here.
Could you put do a FAQ of the “ClimateGate” scandal, like you did for the objections to AGW? “I read that the CRU deleted the data.” “No, no data was released. See here and here for more info.” That type of thing.
If I knew enough about everything….
“Anyway, with the links provided, could a skeptic/denier perform an analysis that would disprove man made global warming?”
The comment above appeared in an earlier post. I suggest that anyone putting asking questions like that should google Betrand Russell’s teapot, or just go here: http://en.wikipedia.org/wiki/Russell%27s_teapot
Couldn’t not contribute an aerosol data source ;-)
AERONET (http://aeronet.gsfc.nasa.gov/)
The Data Sources page looks like an excellent resource. Will it be maintained, or is this a one-off exercise?
[Response: It’s set up as a special page, so it won’t go off the front page, and we will aim to keep it up to date. There’s been a lot of suggestions of data portals even we didn’t know about, so it probably is going to be useful. – gavin]
Mike Flynn:
BPL: Do a little more research. Start here:
http://BartonPaulLevenson.com/ModelsReliable.html
There’s plenty of climate model output available here: http://results.cpdn.org.
It is requested that those downloading the data make a note of what they would like it for but that’s the only requirement.
Wow, what a major storm in a teacup. Although I find it disturbing that CRU appears to have destroyed a lot of the raw data, the availability of satellite measurements over the period of the most recent warming mean that there is plenty of data out there that is independent of any possible CRU ‘fiddling’.
As a layman, I have no way of using the data supplied in the links here. For most people it is more interesting to see graphical representations. I find the following site very useful for comparing the various records as it has them all, including the most recent data, all in the same place. It appears to be mainly factual, although do I detect a very slight sceptical slant? It also has some links detailing some interesting stories from history showing how climate change has affected real people in the past.
http://www.climate4you.com/GlobalTemperatures.htm
particularly this chart which overlays the various data sets:
http://www.climate4you.com/GlobalTemperatures.htm#Comparing%20global%20temperature%20estimates
As an aside, can historical records be used in any scientific way to inform climate models? For instance, is it possible to know whether the climatic conditions that generated the following fascinating story are in any way indicative of global conditions, or can these kinds of multi-year cooling events be a purely local phenomenon?
http://www.climate4you.com/ClimateAndHistory%201600-1699.htm#1658: Swedish ice-bourne invasion of Denmark; the Treaty of Roskilde
[Response: I detect the same thing. There are some rather odd methodological choices, a distinct lack of understanding the concepts of significance and some weird editorialising. So not climate4me! – gavin]
Neal, you apparently either don’t understand Russell’s teapot or you don’t understand the basis for AGW (or both). The point of Russell’s analogy is that it’s impossible to prove a negative for a belief that is based on personal credulity rather than evidence. The analogy does not apply to claims based on empirical evidence.
We know from direct measurements of the absorption spectra that CO2 is in fact a greenhouse gas. We know from direct measurement that it is increasing in concentration in the atmosphere. We know from direct measurements of the isotopic ratios of that CO2 that most of the increase is due to fossil fuel use and land use changes rather than natural sources like volcanoes. We know from measurements that the stratosphere is cooling while the troposphere warms, and that the warming is greatest at night, both observations which are consistent with greenhouse warming and inconsistent with almost any other mechanism.
Performing an analysis disproving anthropogenic warming would mean showing that either MANY of these measurements are in serious error or that there other mechanisms exist which can explain the evidence better. In either case, the analysis would prove a positive- the existence of errors, or more powerful mechanisms.
Ron at #255
The README.TXT file indicates successful completion of Steps 0-3.
Is that correct or does the README.TXT file need to be updated?
Thanks
@Dan Hughes#270: Is that correct or does the README.TXT file need to be updated?
The README.txt is correct. Until I can either produce the SBBX.HadR2 file in STEP4 or consume it in STEP5, I won’t claim success for the STEPs 4 and 5. I think some of my problem is just a confusion of what STEP4 does. Looking closer, it appears to update an existing SBBX.HadR2 file with data from oiv2mon files. But if I start with an existing SBBX.HadR2 downloaded from GISS, I bump into the bigendian/littleendian issue. This doesn’t seem insurmountable … but I have a day job. I’ll see what I can do tonight.
If anyone has constructive suggestions, they can email at ronbroberg in the company of yahoo.com.
The new Data Page is a wonderful idea. One small correction: Under Climate data (processed): Cloud and radiation products (ISCCP, CERES-ERBE), I suggest changing the CERES-ERBE link to the following: http://eosweb.larc.nasa.gov/project/ceres/table_ceres.html
RE: 50 Jimi Bostock
If you really want skeptics to embrace your movement (theories), you should do better than attack them. Most people I know, when challenged, try to explain their position, backed by FACTS. Your name-calling and arrogance don’t help your cause. Those who attack generally do so from fear. Why don’t you take time, retrench; and present fully your methods, data, and code. Perhaps then the argument will be settled, by removing the emotional controversy. (But, you need to release everything – warts and all.)
This data sets page is a great resource. There are some here I did not know about. Thanks for putting it together. Do you mind if I link to it?
A good source of data is NCAR http://www.ncar.ucar.edu/tools/datasets/
I’m not sure how you would list this in your hierarchy (there is much
data available, perhaps only some of it relative to climate research.)
Some of this data is available to everybody, some of it is restricted to people with NCAR computing accounts (mostly due to restrictions placed on the data when they receive it, and other practicalities.) Lets face it, WMO40 was/is
a dual edged sword – it gave us access to some information that was previously
difficult (or impossible) to obtain – at the same time it restricted
what we could do with the data. I’d love to see a time when all data
collected for weather and climate research could be freely exchanged,
but I think that day is sometime in the future, and may never happen.
We used to make global hourly SYNOP observations available, but the
Irish Met office was kind enough to let us know that only 6 hourly
observations were covered by WMO resolution 40. Everything else
was considered “special” data – and every country had their own
rules on “special” data. We don’t have staff to figure out
which is which, and to separate it, so we had to stop making
the data available.
For those interested, a quick google search found
http://www.nws.noaa.gov/im/wmocovr.htm
It would be nice if the WMO, or some other body, made available
an archive of all available data. Maybe someday this will happen.
I’m sure it just a matter of money.
My understanding is the data that CRU “lost” was in the form of paper
output and obsolete tapes. This apparently happened in the mid 1980’s.
Sorry I cannot provide a reference for this; this “understanding”
occurred over the last few days from reading various web sites
and blogs – and may not be correct. I know that we have had to
recycle stacks of paper, and magnetic tape, because it became
impractical to store, and resources were not available to put
them on more modern storage. The reality is that no University
has the capability, or the responsibility, to make any data
they may have once had available to others. People say that the
data should have been kept because now important (and expensive)
decisions are being made based on the analysis of that data.
But in 1985 that was not known.
If anybody here has influence with IPCC or WMO, perhaps you can encourage
them to make such a freely available data repository available?
But again that would cost money.
Since the “science” has apparently been
more or less settled, perhaps that money should go
toward possible mitigation efforts? But I still think an
international data repository would be very valuable.
While I’m tempted to suggest that all climate scientists
retire, or move on, and no new ones be made, I think that would also be
a mistake. To store and make the data available, and
to retain climate scientists, is a lot cheaper than
trying to resolve the mess we have (or might have) found
ourselves in. If the data is available to everybody,
perhaps one of them might find a mistake, or something new.
The reality is, like it or not, the science has not
been settled – sure there is some evidence the Earth
might be warming, and CO2 is an obvious cause, but
if you really think we know what is going on, I suggest
you might be wrong (of course, you could be right).
I have concerns about how the data might have been analyzed. But I
have no interest in reanalyzing it. Well, I have the interest,
but not the time. Enough people have done this – there might
be problems looking for a signal in so much noise, but I think
the people who have done this work did it as carefully and properly
as possible.
Climate scientists find themselves in an awkward position. They
have found themselves thrust in to a position of guiding policy.
This was in part because they had no choice – they found something
that may be a real concern – in order to get that concern heard
they had to publicize it. There is some concern that they trust
this into the public view in order to enhance their career, or their
funding. This might be true in some cases, but I have no doubt
that there is a real concern – and when you are concerned
about something you have to try to do something about it.
Scientists have been wrong before, and MAYBE they are this time.
But they would be negligent if they saw a potential problem,
and did not try to fix it, even if it meant they have to
delve into politics (where most of them have little skill)
You have to admit, when we can’t predict the weather for next week,
the public has to wonder about our ability to predict the weather
(or climate, which is after all just the average of weather)
for the next decade. Yes I know they are different problems, but
most of the public does not see that, and after all they are mostly
based on similar sets of data and models.
Just heard the Diane Rehm panel discussion on Copenhagen, which included Michael Mann along with a “resident scholar” from the American Enterprise Institute. I commend it to anyone, especially in the skeptic community, who still has questions about the email incident as well as the reality of the recent warming.
The medical profession has discovered that a poison contained in a United Nations variant of fudge was the cause of a debilitating brain disease among scientists, now termed Climate Wars Syndrome (CWS).
The disease was secretly suspected by sceptical scientists to have spread rapaciously among the scientific community for two decades and to have taken a terrible grip over the reasoning powers of many. Victims can be identified by their green and alarmist complexion. Other side effects include an irrational hatred of mankind and a Tourette syndrome-like verbal abuse of anyone who uses fossil fuels. Threats of violence may occur. The world first learned of these sensational developments from the Internet on Friday November 20th 2009. The story broke that both the underlying cause of CWS and an effective treatment had been discovered by the due diligence of one man working at the UK’s Climate Research Unit (CRU). A vast community of Internet surfers soon memorialised these profound events by naming them, ‘Climategate.’
From leaked documents we understand that the catalyst for this epoch change in science occurred when a climatologist and self-taught computer programmer known only as ‘Harry’ was sat at his laboratory computer chewing on some fudge. Only after three long years working on this problem and in a sudden eureka moment, did it finally dawn on him. In Harry’s hands was the cause of brain fog mystery.
“F**k! It’s the fudge! It’s serial!” he cried.
Inadvertently, Harry has become the hero the public associate with solving one of the great mysteries of modern science. Since those findings have appeared on the Internet the world has quickly accepted that it was the UN’s foul fudge that caused scientists to suffer this dreadful disease.
Meanwhile, epidemiologists and clinicians have been quick to identify the hallucinagenic properties of the offending fudge to further unravel the mystery. Incredibly, the fudge has been found to contain a psychotropic substance that acts primarily upon the central nervous system where it alters brain function, resulting in changes in perception, mood, consciousness and behavior leading patients to feel delusions of grandeur and a sense of spiritual purpose in their lives.
It appears lone-wolf Harry, wiling away his time in the CRU laboratory subliminally faced the truth and by a process of ‘cognitive dissonance,’ shocked himself out of the effects of the psychotropic intoxicant, a drug now known to cause the hallucinogenic appearance of a mythical beast known as, ‘Man-Bear-Pig’ (MBP). Other experts who have replicated Harry’s experiments confirm the efficacy of the cognitive dissonance reasoning process as a cure. Apparently, most recovering ‘addicts’ (for this fudge-eating was clearly an addiction) soon notice a change starting with improvements in the appearance of their eyes which lose their tainted green colouration.
Other convalescing climatologists, that body of scientists identified as the worst fudge sufferers, are reporting the same side effects as Harry. Symptoms include anxiety, guilt, shame, anger, embarrassment, stress, and other negative emotional states that torment the patient. Epidemiologists have coined the name ’Climate War Syndrome’ (CWS) to describe the fudge-induced malady. Both ‘Climategate’ and ’Climate War Syndrome’ (CWS) have fast entered common usage giving a new handle on what was one of the great mysteries of our time.
Of course, like any serious disease, there will always be patients who won’t respond well to treatment. Those worst cases permeated with the deepest shade of green are believed to be James Hansen, Michael ‘upside down’ Mann and Phil Jones whom, its feared, may all need to be quarantined in isolation for several years.
[Response: Not really appropriate, but a step above the, umm… let’s say, less cerebral sceptic offerings… – gavin]
In the “documents” section of the FOIA2009 archive, there are several IDL programs that add “artificial corrections” to temperature data (for a list of files that match “artificial” see here). One such program is briffa_sep98_e.pro. This graph shows tree-ring data before and after the corrections defined in the file (for more background see here. Doesn’t the corrected graph look like they hockey stick? Were such corrections used in the construction of the hockey stick?
[Response: No. These “corrections” were never used anywhere. See comments passim. – gavin]
“As with CRU, all I have to give out is the merged product. – gavin]”
How do you know that the merged product represents the original product. Unless you either have the original, or enough metafile data to point you to the places where you got the pieces of your merged products, then the idea that your merged product represents reality is simply unsupportable.
[Response: You are welcome to go back to the original sources and do it again yourself. The point is that there is only one product (the current database), but the raw data still exists (in it’s less than perfectly accessible form). – gavin]
@Ron Broberg
it is good to see that you are trying to make this GISTEMP work.
Your first attempt appears to have been less successful than you originally thought and someone had to call you on it. So while you were not trolling (sorry for that) you were over excited and produced personal, as it were, attack based on your confusion.
Hopefully you will get it to work and in the process create good documentation and truly become the first person in this thread to have ran GISTEMP.
I just saw an article in the UK Times that claims the orignal temp data (ie: raw unadjusted in any way) has been destroyed/mislaid during a move of facilaties. Further according to this article, the only thing now in possession of the CRU is the smothed and corrected data produce in accordance with CRU’s model. Could you comment as to whether this is true and on a hypothetical, were it in fact true would the research in fact be repeatable, in the of a good experemnet can be repeated and arrive at the same answer? Sorry some one may haved this before just did not have the heart to wade through all the coments. Thanks.
[Response: The raw data is in the custody of the met services who originated it. CRU is just a collation, not a temperature measuring organisation. – gavin]
Ron, if you are using GNU Fortran, the enddianness magic can be done by setting GFORTRAN_CONVERT_UNIT before invoking the executable. This can set the endianness of any “units” (like file IDs) to big-endian, little-endian, platform-endian, swap-endian, etc.
On the ccc-gistemp project I hacked out all the ksh driver scripts a long time ago – replacing them with a single /bin/sh script – and rearranged all the files into consistent directories, and so on, so my code and scripts aren’t the same as the released GISTEMP any more, but the relevant part of my run.sh file reads like this:
echo “====> STEP 5 ====”
# GISS binary data files are big-endian, so ours are too, including
# the intermediate file generated by step3.py and the SBBX.HadR2 ocean
# file which comes out of STEP4. We use GFORTRAN_CONVERT_UNIT to tell
# our code to treat all these files as big-endian. This Will Not Work
# if the Fortran compiler is not GNU Fortran.
GFORTRAN_CONVERT_UNIT=”big_endian:10,11,12″ bin/SBBXotoBX.exe 100 0 > log/SBBXotoBX.log
GFORTRAN_CONVERT_UNIT=”big_endian:10,11″ bin/zonav.exe > log/zonav.Ts.ho2.GHCN.CL.PA.log
GFORTRAN_CONVERT_UNIT=”big_endian:10,11,12″ bin/annzon.exe > log/annzon.Ts.ho2.GHCN.CL.PA.log
Hoping this makes it through the RC comment filter. If this interests you, feel free to join us on ccc-gistemp.
After days of reading through this, the more tempting it is to simply throw up my hands in frustration. The denialist will still continue to deny AGW, regardless of how much data you give/throw/otherwise convey to them. The primary forces opposed to AGW are simply too well financed; their pockets are very deep, they have control over numerous outlets and are quite crafty at picking-and-choosing-and-releasing data that supports their claims that AGW is not real, and that while climate change MAY BE real (and that is a big, fat MAY BE), it is most certainly not human in origin. This is not science against science here any longer; it is a clash of belief systems. The funny thing is that, like most belief systems, it is easier to say that the side with data supporting their claims has turned their science into a near religion. This is what happens with Creationists; their number one charge against evolutionists is that the evolutionists have turned their “theories” into religion.
The media outlets that show a strong pro-industry bias are almost always controlled, in some fashion, by parties that stand to lose big if any mandates are handed down. Of course they are going to fight this, as this is going to be the biggest paradigm shift in recent human history.
I fear that there will soon be a new group emerging from this; the GCC embracers. These are former denialists who now accept that AGW is real, but alas, it is too late to do anything about, so we simply need to just move along and accept our new, somewhat warmer and regionally flooded world.
Just a few thoughts…
Before the “emails” were released Micheal Mann had a paper in press. It appeared in the recent issue of Science.
Global Signatures and Dynamical Origins of the Little Ice Age and Medieval Climate Anomaly
Michael E. Mann et al. Science 27 November 2009:
Vol. 326. no. 5957, pp. 1256 – 1260
Mann is one of the folks being lambasted for a lack of transparency. This paper is a case study in transparency and data availablilty.
From the text of the paper:
“Further details of reconstruction procedure, associated statistical validation and skill assessments, uncertainty estimation procedures, data used, and MATLAB source codes for the analysis procedures are provided in the Materials and Methods.”
When you go to the online supplement, you will find this:
“Computer codes and data. Computer codes (MATLAB), data, and supporting information for analysis in main paper. File are packaged as a compressed archive, in *.zip format; users should download the compressed file to their machine and decompress the file on their local hard drive, using the instructions below.
multiproxySpatial09.zip (22 MB) [This is a link in the online version”
And should we forget were we could be spending our time and energy, it’s a very interesting paper and well worth a read.
Re: “You’re using a common name as a userid — whoever you are, people won’t know if the next “David” along who asks a naive question is from you again, playing games, or from someone with an honest question.”
That’s a fair point, and I apologize. I won’t do it again, no matter how much I may be bothered by the tone of some of the posts here.
@Ron Broberg,
using your instructions + code provided and starting from scratch, while running Ubuntu 9.10 x64 in Parallels and using this ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.mean.Z after “./do_all.sh” produces:
—
start STEP0
At line 10 of file sorts.f (unit = 1, file = ‘antarc.txt’)
Fortran runtime error: End of file
Bringing Antarctic tables closer to input_files/v2.mean format
collecting surface station data
… and autom. weather stn data
… and australian data
replacing ‘-‘ by -999.9, blanks are left alone at this stage
adding extra Antarctica station data to input_files/v2.mean
0.00 not ok
0.00 not ok
created v2.meanx from v2_antarct.dat and input_files/v2.mean
removing pre-1880 data:
GHCN data:
removing data before year 1880.0000
created v2.meany from v2.meanx
Replacing USHCN station data in v2.mean by USHCN_noFIL data (Tobs+maxmin adj+SHAPadj+noFIL)
reformat USHCN to v2.mean format
replacing USHCN station data in by USHCN_V2 data (all adjustments, but ignoring fill-ins)
unifying station-ids
./unify_us_ids[3]: 9641C_200907_F52.avg: cannot open [No such file or directory]
mv: cannot stat `9641C_200907_F52.avg.1’: No such file or directory
<<>>
Finding offset caused by adjustments
extracting US data from GHCN set
removing data before year 1980.0000
Getting USHCN data:
removing data before year 1880.0000
done with ushcn
Created ushcn-ghcn_offset_noFIL
At line 21 of file cmb2.ushcn.v2.f (unit = 3, file = ‘ushcn-ghcn_offset_noFIL’)
Fortran runtime error: End of file
created v2.meanz
Replacing Hohenspeissenberg data in v2.mean by more complete data (priv.comm.)
disregard pre-1880 data:
—-
Is this similar to the output that you are seeing?
Interested in direct links to cosmic ray count data, AMO, PDO, and the major Nino SST indices?
53 Jiminmpls said:
“28 November 2009 at 9:40 AM
Fair is fair. I demand that the Heartland Institute, George Marshall Institute, American Petroleum Institute, Western Fuels Association, Sen Inhofe, McIntyre, Singer, Ball, Michaels, and all other organizations and individuals attempting to influence public policy in regards to climate change IMMEDIATELY make their entire email archives available for public scrutiny and analysis.”
According to most of the people posting here none of those organisations have anything at all worth saying so what would be the point?
The Antarctic Glaciological Data Center (AGDC) at NSIDC houses a collection of Paleoclimate/Ice Core data
(http://nsidc.org/agdc/data/paleoclimate.html) collected under the NSF Antarctic Glaciology program.
cheers,
Rob
There’s a data-processing program buried in the FOIA archive, called briffa_sep98_e.pro. It adds hard-coded corrections to raw tree-ring data. The effect is like this. Did this program play a part in the composition of the Hockey Stick graph?
[Response: No. See comments passim. – gavin]
Ray Ladbury (241) says, “Matthew, there are no true skeptics left–merely the ignorant, the wilfully ignorant, the denialists and the wingnuts.”
So what do you call people who deny the reality of true skeptics (your self serving definition aside.)
Not sure if it would be data overload, but RCM code is pretty easy to find too. I know from NARCCAP, at least, MM5, WRF, and RegCM3 have code available. I think WRF might make you do a quick registration first, though. It’s probably not worth listing RCM output since it’s highly individualized.
I can confirm what Ron Broberg (#271) says. I also arrived at step 4 and met following message:
iyrbeg-new 1880 iyrbeg-old 1476853760
where the latter number is “1880” in reverse endianess. I am using Ubuntu 32bit on intel.
By the way, it took me between a little more than one hour to make the thing run, including download of everything, installing most of the needed tools (my Ubuntu box didn’t even had Ksh and the Fortran compiler installed) and rewriting the Python extension installer script. Not out-of-the-box, but not so hard either :-)
Re 129
Gavin the quote from Hansen 2001 stated metadata defining “all” changes, not as you put it “any metadata”.
Whilst the change of time of measurement is one of the multitude of changes, it does not comprise metadata defining all of the changes.
Okay, here’s a question about the Times missing raw data story beyond where the data still is: what did the CRU mean when they said:
“Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues.”
Wouldn’t the cost of keeping a few more numbers on tapes (or whatever) be trivial compared to cost of the whole enterprise? Or was something other than cost the issue?
Rod B., A true skeptic must acquaint him or herself with the evidence, acknowledge it and seek plausible alternative explanations. Do you know of any true skeptics Rod? I don’t. If you know of any, perhaps you can tell me their theories on how the troposphere can warm and the stratosphere cool via a mechanism other than a greenhouse one. Or what credible alternative energy mechanism accounts for 30 years of warming and the loss of trillions of tons of ice. That’s for starters.
Gavin, thanks for your reassurance that the corrected data was not used. I’m looking for “comments passim” that you mentioned. Perhaps I’m just a blind old fool, but I’m not finding them in this list of comments. Can you direct me to your previous answer? Thank you for your patience. Kevan
@donQ#285: Is this similar to the output that you are seeing?
Yes. Next update, I’ll post an output log.
Also, I scripted a diff for the NASA GISTEMP output and my file. The largest diffs are in the endpoints – 1880 – 1900 or so and are likely related to the Hohenpeissenberg file. Currently I am downloading the one hosted in NL rather than use the one provided in the NASA tarball. I’ll switch back to NASA’s and rerun tonight.
@Nick Barnes:
Thanks for the tip. I’ll give them a try tonight.
#50–“I have never known anything to be settled in my long life”–so you go to bed hoping that you won’t fall of the earth before you wake up, gravity being one of those things that still isn’t settled. You hear there is some cooling going on–when it’s shaping up to be the hottest decade ever–each of the last 10 year have been among the 15 hottest years. Yes, 1998 was hot, but the trend still seems to be up. Maybe you should look a little more closely at what the sceptics say and see if it really holds water–that’s the basis of the camparison of deniers to creationists–they both ignore the evidence of their surroundings.
Gavin, I’m sorry to bother you with more nonsense, but I’d trying to deal with some people making a fuss about the Times UK article about the magnetic tapes and paper notes that were thrown out to save space. I know I saw something here at RC on this, but haven’t been able to find any clarification despite some fancy “ctrl-F work in the posts. Also, in the advent of that article, it might be a time/sanity saver to update something about that in the body of this or the “…context” blog, to clarify quantity, quality and perhaps extent of documentation available to reconstruct (ie: how much methodology was written up if anyone wanted to backwards engineer the oldest data to extrapolate what would have been on the tapes) or how that data might have compared to other data sets from the same time/now. I personally see data on 1980s magnetic tapes as valuable as data scribbled on cocktail napkins late at night in the dark, but I’m only an undergrad… I guess I’d better not lose any of my notebook pages from now until I die or I might be in trouble. Anyway, I’d like to have something to tell people who think this is some big meaningful plot or omission of science to put it into perspective.
On the question as to whether some Briffa data were used in one of the hockey stick graphs you say “No, see comments passim”. I’ve searched the comments for “Briffa” and “hockey stick” and not found anything. What comments answering the question are you referring to?