RealClimate logo


Please, show us your code

Filed under: — rasmus @ 17 December 2009

The 1991 Science paper by Friis-Christensen & Lassen, work by Henrik Svensmark (Physical Review Letters), and calculations done by Scafetta & West (in the journals Geophysical Research Letters, Journal of Geophysical Research, and Physics Today) have inspired the idea that the recent warming is due to changes in the sun, rather than greenhouse gases.

We have discussed these papers before here on RealClimate (here, here, and here), and I think it’s fair to say that these studies have been fairly influential one way or the other. But has anybody ever seen the details of the methods used, or the data? I believe that a full disclosure of their codes and data would really boost the confidence in their work, if they were sound. So if they believe so strongly that their work is solid, why not more transparency?

There is a recent story in the British paper The Independent, where Friis-Christensen and Svensmark responded to the criticism forwarded by Peter Laut (here). All this would perhaps be unnecessary if they had disclosed their codes and data.

Gavin and I published a paper in Journal of Geophysical Research, where we tested the general approach used by Scafetta & West, and tried to repeat their analysis. We were up-front about our lack of success in a 100% replication of their work, but we argue that the any pronounced effect – as claimed by Scafetta & West – should be detectable even if the set-up is not 100% identical.

However, Scafetta does not accept our analysis and has criticized me for lacking knowledge about wavelet analysis – he tells me to read the text books. So I asked him to post his code openly on the Internet so that others could repeat our test with their code. That should settle our controversy.

After repeated requests, he told me that he doesn’t really understand why I’m not able to write my own program to reproduce the calculations (actually, I did in the paper together with Gavin, but Scafetta wouldn’t accept our analysis), and keeps insulting me by telling me to take a course on wavelet analysis. Furthermore, he stated that there “are several other and even more serious problems” in our work. I figure then that the easiest way to get to the bottom of this issue it to repeat our tests with his code.

A replication in general doesn’t require full disclosure of source code because the description in the paper should be sufficient, though in this case it clearly wasn’t. So to both save having us do it again and perhaps miss some other little detail – in addition to using an algorithm that Scafetta is happy with – it’s worth getting the code with which to validate our efforts.

It should be a common courtesy to provide methods requested by other scientists in order to speedily get to the essence of the issue, and not to waste time with the minutiae of which year is picked to end the analysis.

The reason why Gavin and I were not able to repeat Scafetta’s analysis in exact details is that his papers didn’t disclose all the necessary details. The first point he raised was that we used periodic instead of reflection boundaries. The fact that the paper referred to the expression ’1/2 A sin (2 pi t)’ to describe the temperatures or solar forcing would normally suggest that they used periodic rather than reflection boundaries. There was no information in the paper about reflection boundary. But this is no big deal, as we have subsequently repeated the analysis with reflection boundary, and that doesn’t alter our conclusions.

After further communication, we found out that Scafetta re-sampled the data in such a way that the center of the wavelet band pass filter was located exactly on the 11 and 22 year solar cycles, which were the frequencies of interest. He also informed me that a reasonable choice of the year when the reflection boudary was made should be the year 2002-3 when the sun experienced a maximum for both the 11 and 22 year cycles. This information was not provided in the papers.

I’m no psychic, so I couldn’t have guessed that all this was needed to reproduce his result. But since Scafetta has lost faith in my ability to repeat his work, I think it’s even a greater reason to disclose his code so that others can have a go.

For the record, we did not just use wavelets to filter the data – we obtained the same conclusion with an ordinary band-pass filter.


233 Responses to “Please, show us your code”

  1. 201
    Rod B says:

    Tim McDermott (196), I meant no disparagement of Gavin and Co. I have no idea if they are conspirators or not — though I certainly doubt it; and if they can make $153,000 plus sideline extras I think that’s great — and I’m certain they didn’t sign away their soul and ethics to earn it. I was only commenting on the “climategate” documents and the seriously negative connotation that much of the public perceives. In politics (I know — the scientists usually abhor that arena), and this is to you too, Ray (197), perception, not evidence, is reality.

  2. 202
    Rod B says:

    Hank, not wrong. For a molecule to absorb IR directly into rotational bands the molecule must have a permanent polar moment. To absorb into vibration bands there must be a natural change in the polar moment, going from no moment in a quiescent state (methane) to seeing briefly a moment by virtue of the bond movement. Methane has one level of any import which is mixture of bending and stretching at 1306 cm^-1. True the absorption is allowed by virtue of the wiggling, but only so long as the wiggling produces an EM moment. For wiggling without creating a moment (N2) the IR couldn’t care less. BTW, the ones that can absorb vibration energy can also absorb rotation even though they don’t fall in the basic rule.

    It ain’t the wiggling that directly allows IR to be absorbed.

  3. 203
    Marcus says:

    Hank Roberts: #199: Dipole moments are also important: in the field of fluorescent absorption, they are pretty key. Additionally, my recollection is that the _creation_ of dipole moments is an important facet of absorption by various vibrational modes – I seem to remember (and I should probably go back and review my quantum theory) that this is why diatomic molecules don’t have any absorption, because the only way they can vibrate is to stretch and contract, and that doesn’t form any dipoles.

    On the other hand, methane, which is non-polar in its typically drawn state, does form a dipole moment briefly during various stretches.

    So it is possible that Rod B was not so much completely mistaken as just having incomplete recall. Aha – and the Google tells me: “Infrared (IR) and Raman spectroscopy both measure the vibrational energies of molecules but these method rely only different selection rules. Recall that for a vibrational motion to be IR active, the dipole moment of the molecule must change. Therefore, the symmetric stretch in carbon dioxide is not IR active because there is not change in the dipole moment. The asymmetric stretch is IR active due to a change in dipole moment.” http://carbon.cudenver.edu/public/chemistry/classes/chem4538/raman.htm

  4. 204
    Hank Roberts says:

    Marcus, that’s the point of the answers I quoted.

    Rod misremembered “dipole” as “polar” — so made a wrong statement.

    I think he’s also wrong about methane, but his ideas about radiation physics have consumed so many words on so many topics here that I’m inclined to say “well in your physics I’m sure it’s true” and leave it so.

    I found similar statements (“polar”) from confused people posting on denial sites, with unique theories about how the physics works. That’s why I emphasize looking this kind of thing up instead of relying on memory.

  5. 205
    Hank Roberts says:

    > if they can make $153,000 plus sideline extras

    Note the problem with Rod’s memory of numbers and understanding of finance.

  6. 206
    Hank Roberts says:

    Rod, to avoid another long digression into radiation physics that avoids the basic question, sure there are other vibrational modes and photons of higher energy can affect molecules. That’s not the greenhouse effect and it’s not what you were talking about when you made the mistake. Get clear please on the basics because you can very easily confuse yourself again.

    H20 is a polar molecule (uneven distribution of charge)
    CO2 is not a polar molecule
    CH4 is not a polar molecule

    Infrared photons are relatively low energy and affect some but not all bonds in some but not all possible ways. The greenhouse effect is this small subset of the many ways radiation interacts with molecules.

    http://www.windows.ucar.edu/tour/link=/physical_science/chemistry/co2_molecule_vibrate_modes_big_gif_image.html
    “Vibration modes of carbon dioxide. Mode (a) is symmetric and results in no net displacement of the molecule’s “center of charge”, and is therefore not associated with the absorption of IR radiation. Modes (b) and (c) do displace the “center of charge”, creating a “dipole moment”, and therefore are modes that result from EM radiation absorption, and are thus responsible for making CO2 a greenhouse gas. — Credit: Martin C. Doege”
    http://www.windows.ucar.edu/tour/link=/earth/climate/greenhouse_effect_gases.html

    “CO2 and O3 have “floppy” vibration motions whose quantum states can be excited by collisions at energies encountered in the atmosphere. For example, carbon dioxide is a linear molecule, but it has an important vibrational mode in which the molecule bends with the carbon in the middle moving one way and the oxygen atoms on the ends moving the other way, creating some charge separation, a dipole moment, and thus carbon dioxide molecules can absorb IR radiation. Collisions will immediately transfer this energy to heating the surrounding gas. On the other hand, other CO2 molecules will be vibrationally excited by collisions. Roughly 5% of CO2 molecules are vibrationally excited at room temperature and it is this 5% that radiates. A substantial part of the greenhouse effect due to carbon dioxide exists because this vibration is easily excited by infrared radiation. CO2 has two other vibrational modes. The symmetric stretch does not radiate, and the asymmetric stretch is at too high a frequency to be effectively excited by atmospheric temperature collisions, although it does contribute to absorption of IR radiation.

    The vibrational modes of water are at too high energies to effectively radiate, but do absorb higher frequency IR radiation. Water vapor has a bent shape. It has a permanent dipole moment ….”

    http://www.newworldencyclopedia.org/entry/Greenhouse_effect

    —–
    Point being you can look this stuff up, and similar words mean very different things, and it’s very easy to confuse yourself and others about how the greenhouse effect works.

    Look, here’s just one of the examples I found of some guy on a blog confusing himself and his readers about how the greenhouse effect works, starting from this misconception. You know how to find this stuff:

    “… CO2 is a polar molecule, the science behind that statement is very stale, but what that means is that the molecule is capable of absorbing infrared radiation. In a prefect world with no CO2, energy ….”

    And from there he goes on to slam Al Gore and physics etcetera. He’s a nitwit, and he could have gotten his physics right before starting to rant.

    Don’t encourage this kind of mistake by making excuses. Get it right.
    It isn’t hard to get this kind of simple statement right and make it clear.

  7. 207
    Completely Fed Up says:

    “CO2 is not a polar molecule”

    Uh, I believe it is.

    the electron will spend longer at the C than the O and that uneven distribution makes a dipole. Whether this is enough to make a large contribution to IR absorption I cannot remember.

    And when excited the triatom vibrates like a V-shaped tuning fork when truck. Making another dipole.

  8. 208
    Completely Fed Up says:

    “For a molecule to absorb IR directly into rotational bands the molecule must have a permanent polar moment.”

    And I don’t think that is right either.

    It needs enough mobility of the electrons to allow it to react.

    If your idea was right, then solids wouldn’t be able to absorb photons unless they were metallic…

  9. 209
    Completely Fed Up says:

    #199

    Ah, that’s right. Forgot that it’s O-C-O therefore there is no permanent dipole unless strained. But one can be induced: O–C-O to O-C–O.

  10. 210
    Silk says:

    “For a molecule to absorb IR directly into rotational bands the molecule must have a permanent polar moment.”

    My PhD is in this field, so I’m think I’m qualified to comment.

    Yes. Water can absorb directly into rotational modes in the far-IR. For heavier molecules, this type of absorption is in the microwave region.

    CO2 cannot absorb directly into rotational modes but can undergo ro-vibrational transitions.

    All this is well understood, easily measured (by Freshmen) and factored into climate models.

    I admit I’ve lost the thread a bit. How does this impact on the work of RealClimate?

  11. 211
    Rod B says:

    Hank, well, maybe my chosen word was not precise, but one can not have a polar moment without a dipole. Some say tomäto, some say tomato. [that probably won't work.]

    I’m wrong about methane how? Or does it not matter to you?

    My memory of finances and numbers are wrong how? Or does it not matter to you?

  12. 212
    Hank Roberts says:

    > How does this impact on the work of RealClimate?
    Barely (sigh). Thanks, Silk, for checking in — always better when a real scientist addresses this stuff. Did I get the distinction between “polar molecule” and dipole ok?

    I posted first just to note that beliefs can be checked. Belabored after seeing claims elsewhere that only polar molecules can be a greenhouse gases, so Al Gore is etcetera.

  13. 213
    Brian Macker says:

    Yes they should provide everything as should you and every other climatologist. I’m skeptical of both.

  14. 214
    Completely Fed Up says:

    “I’m skeptical of both.”

    Which can be solved (if you’re willing to change) by reading the G&T paper and reading the IPCC chapter on the science of AGW.

    G&T: Completely laughable
    IPCC: Maybe not verifiable, but not obviously wrong

    If you’re willing to learn, that will teach you what denialists will think is a good science paper.

    That should be enough for anyone who isn’t predisposed to a preconceived idea.

  15. 215
    Rod B says:

    Hank (206), your criticism is perplexing. I said nothing different from anything you say in #206. So where is it that I’m wrong? I might have used “allowed” instead of “result from” in your reference quote, “….and therefore are modes that result from EM radiation absorption,” though either could work depending on how prissy one is. What is it you want me to “get right,” “basics” or otherwise?

    If your claiming I said something similar to your “nitwit” reference, I talked about a polar moment (or bipolar moment if you prefer) being created from CO2′s base configuration molecular state, which I never said was polar.

    FWIW, I don’t think this is just a “digression” away from the basic question. This aspect of radiation physics is pivotal in climate change, yet I detect a fair amount of uncertainty (if not ignorance) and a bunch of conflicting opinions. This is significant and odd.

    Sidebar: I thought Ray’s estimate of vibrationally excited CO2 was more like 1% rather than 5%.

    Completely Fed Up, In its basic configuration CO2 is not a polar molecule, in the context being discussed here. Electronic mobility has nothing to do with IR absorption. And, to repeat, “For a molecule to absorb IR directly into rotational bands the molecule must have a permanent polar moment.” The context here is gas, as in greenhouse gas; I don’t know about solids (other than I know microwaves excite the H2O molecules to transfer energy/heat to the rest of the meatloaf.) Also the key word is direct — as Silk points out.

  16. 216
    Hank Roberts says:

    > talked about a polar moment (or bipolar moment if you
    > prefer) being created from CO2’s base configuration
    > molecular state

    Yeah, that’s what I called a wiggle. Not a rotation, not a stretch, but a wiggle — back and forth– after absorbing the infrared photon, as per the pictures illustrating it.

  17. 217
    t_p_hamilton says:

    Rod B says:”FWIW, I don’t think this is just a “digression” away from the basic question. This aspect of radiation physics is pivotal in climate change, yet I detect a fair amount of uncertainty (if not ignorance) and a bunch of conflicting opinions. This is significant and odd.”

    How is the fact that a bunch of comments on a blog APPEAR to be conflicting significant?
    Or odd? People have different levels of expertise in particular areas.

    What is odd is people projecting lack of knowledge onto experts, and concluding that the imputed uncertainty is “significant”.

    For what it is worth, IR absorption (and emission) intensities depend on the dipole moment derivatives. Rotational absorption (and emission) depends on there being a permanent dipole moment, but is in the microwave range. Then again, a radiation physicist would just use the experimental spectra which implicitly contains these factors.

  18. 218
    Geoff Wexler says:

    Re #215
    Uncertainty about radiation transfer theory?
    It is only part of the story, but I should imagine it is one of the least uncertain parts.

  19. 219
    Eli Rabett says:

    t_p what you are seeing here is a bunch of different models for the same phenomenon. Some of the language is awkward, but most is useful. Errors are getting pointed out. Eli suspects most of the differences come from the place where the various posters learned about IR spectroscopy. Much of it appears to correlate with what is taught in organic/general chemistry. Dipole moment derivatives is more physical chemistry speak.

    Finally, to Geoff Wexler Eli would point out that what you are watching is a bunch of people trying to understand radiation and collisional transfer.

  20. 220
    bob says:

    gavin, i’m sorry but i am confused. you have argued repeatedly on your blog against sharing code+data. now you are for it? please could you post one clear statement of what you believe to be best practices regarding reproducibility in computational science?

    [Response: On Replication - gavin]

  21. 221
    vic says:

    Gavin/ Eric

    we might take you more seriously if you dispensed with editing the blog posts, as in not publishing posts that you do not like. sounds pretty much like your ilk’s approach to peer review.
    I challange you to have an open forum discussion – in public view, no editing no deletion of posts- complete transparency.
    announce it and inform MM and others to post and have a no editing policy, and let the facts and logic take us where it will.
    i fully expect this post to not get published, given you past history.

  22. 222
    vic says:

    Thanks for publishing that Gavin

    For anyone interested enough to spend a couple of hours reading
    here are some links that at the very least cast some doubts
    If you have an open mind and a somewhat analytical temperament, give it a try.

    [edit - no random lists of link nonsense. If you want to discuss something specific, do so]

    Do you like green eggs and ham…..
    You do not like them
    so you say
    Try them Try them
    and you may
    Try them and you may I say!

    A response/ rebuttal- (and no straw man rebuttal) from gavin etc appreciated

  23. 223
    Silk says:

    “FWIW, I don’t think this is just a “digression” away from the basic question. This aspect of radiation physics is pivotal in climate change, yet I detect a fair amount of uncertainty (if not ignorance) and a bunch of conflicting opinions. This is significant and odd.”

    It is neither odd, nor is it significant.

    It is not odd, because blogs are generally full of doo-doo.

    It is not significant, because neither Hank, nor you, nor I, are climate modellers.

  24. 224
    Completely Fed Up says:

    “yet I detect a fair amount of uncertainty (if not ignorance) and a bunch of conflicting opinions. This is significant and odd.”

    Not really.

    Definitions.

    “It just got lower here”

    Does that mean (since we’re talking about climate) the temperatures got lower? Or, since ice melting has been discussed, that the ice levels have got lower? Does it mean I’m crouching down?

  25. 225
    Completely Fed Up says:

    “i fully expect this post to not get published, given you past history.”

    Given you past history you is not really asking you is leading questions.

    Innit.

  26. 226
    Completely Fed Up says:

    Hank:
    “> if they can make $153,000 plus sideline extras

    Note the problem with Rod’s memory of numbers and understanding of finance.”

    Yup, it’s as if he’s got sufficient research powers to find this stuff but doesn’t apply it unless the result is “AGW is wrong, m’kay?”.

  27. 227
    dhogaza says:

    we might take you more seriously if you dispensed with editing the blog posts, as in not publishing posts that you do not like.

    Do you take the same attitude towards all of science, or is there something special about climatology that makes it necessary for climate scientists to run an unmoderated blog in order to validate their work?

    Do you apply the same standards to physics? Evolutionary biology? Medicine? If not, why not? If so, please name the unmoderated blog that, in your mind, validates those fields of science.

    And please explain why an unmoderated blog is more important to the doing of good science than good ‘ole fashioned peer-reviewed publication of research results.

  28. 228

    Vic,

    I do not like green eggs and ham.
    I do not like them, Sam-I-am.

  29. 229
    Rod B says:

    Completely Fed Up, your post 226 has some clever sounding wording, but makes zero sense.

  30. 230
    bob says:

    gavin this is bob from #220 again. i’m sorry but now i’m even *more* confused. the long posting from february shows how you, not unlike “harry” from HARRY_READ_ME.txt, wasted hours of your life trying to reproduce something that could easily have been done in seconds had the authors released code. even more confusing is how at the end of the post you conclude that this laborious process — forensic inference — is somehow a good use of our time. so, first off I would say that i disagree with you that forensic inference is a good idea, and secondly i am now even more confused at the difference between your tone in these two posts. now that someone you clearly disagree with is not posting code, posting code becomes preferable to forensic inference? i remind you that you’re the good guys, and to act like it. having different sets of standards for how much the authors annoy you is not quite what i’d call avoiding the appearance of impropriety.

    i think your stories, along with the events of november, reveal the need for serious climate scientists such as yourself to commit explicitly to reproducible research, in a statement to that effect, which is certainly not what your feb posting amounts to. if you believe your results, don’t act like you have something to hide or that you tolerate those who do (for an example of what happens when scientists who cook their results to get the results that benefit them are exposed to the harsh light of forensic computational science, see here: http://bit.ly/8Nx3Tt

  31. 231
    Brian says:

    “Gavin
    Off subject, I just read tis article about a paper that you coauthored and I have a question
    http://www.physorg.com/news176058147.html
    are you saying that the warming caused by all the non-carbon dioxde GHG is equal to the warming cause by CO2?

    [Response: No, but close. Non-CO2 GHGs are about 40% of the total GHG effect. If you include black carbon and ozone, then CO2 while still the biggest single term, is slightly less than half (with some uncertainty). But note that CO2 is currently the fastest growing factor and is the only one that under BAU has the potential for really big impacts in the future. -gavin]”

    am I understanding you right. about 50% of the warming we’ve seen because of AWG is because of co2 and the rest is becasue of non-co2 ghg, black carbon and ozone . but since co2 is the fastest growing factor, it has the most potential for temp. increase in the future?

    [Response: Yes. - gavin]

  32. 232
    Eli Rabett says:

    Eli is not too sure that you want Scafetta’s code. He doesn’t even appear to know how to follow the directions on the boxtop.

  33. 233
    Sergei Rostov says:

    #48

    3) Exactly why our tax money goes to the Army to fund such research is unclear to me. I have often worked with military folks who at least seemed to be spending the money on what you’d expect, but this seems “odd”.

    It has to do with how the military must think ahead: future allocation of resources, maintenance and storage of current and future equipment, what new technologies may be needed, map changes (battle strategies based on geography), predictions of where social and/or economic unrest may occur, etc. etc. etc.


Switch to our mobile site