RealClimate logo


From blog to Science

Filed under: — gavin @ 13 February 2011

There is a lot of talk around about why science isn’t being done on blogs. It can happen though, and sometimes blog posts can even end up as (part of) a real Science paper. However, the process is non-trivial and the relatively small number of examples of such a transition demonstrate clearly why blog science is not going to replace the peer-reviewed literature any time soon.

Way back in April 2005, I wrote a post on RC on the role of water vapour in the greenhouse effect and why it is considered a feedback and not a forcing in the IPCC sense. It was a basic enough exposition, and in lieu of finding a comprehensive paper on the components of the atmospheric greenhouse effect, I did a few very simple (even simplistic) GCM experiments to show what I was talking about. The bottom line was that CO2 was indeed an important contributor to the present day greenhouse effect, and depending on how you calculated the percentage, could account for between 9 and 26% of the effect.

This proved useful, and soon the page was being quoted quite widely. But the calculations were not very sophisticated and I started to be concerned that they were being given more credibility than they deserved – not that they were necessarily wrong (they weren’t), but because a blog post doesn’t give enough context. For instance, some people incorrectly thought that the range 9-26% was the uncertainty in the calculation, rather than two different conceptual estimates. So I started to look for ‘proper’ references for these kinds of calculations.

I had already seen a few papers that calculated the importance of water vapour, CO2 and clouds for a one-dimensional standard ‘profile’ (Ramanathan and Coakley, 1978 for instance), and I was pointed to a section in Kiehl and Trenberth (1997) that turns out to be the most useful reference. They too had used a single ‘typical’ profile. Both of these references (and a few others – like Ray Pierrehumbert’s 2007 paper which used the NCEP global distribution of water vapour and temperature) generally calculated the importance of CO2 in one of two ways – either by looking at what happened when you removed CO2, or by looking at what happened when only CO2 was operating (though rarely both) (because of the spectral overlaps between the different absorbers, the second number is always larger than the first). Invariably, the treatment of clouds was highly simplified or neglected.

What I didn’t find was any justification in the literature for the most widely quoted ‘contrarian’ view of the issue that CO2 was ‘only 2%’ of the effect. I traced this back to a book review that Lindzen wrote about the first IPCC report, but never found any actual reasoning in support of this.

So in putting together a real paper there were a number of necessary steps that went beyond what was appropriate for a blog post. First, the previous literature had to be collated and their results reported in a consistent way. Second, there were a number of differences between the more serious calculations done for the paper and the calculations done casually for the blog. We used a longer period of time (a full annual cycle rather than a single time step) to avoid a bias towards a particular part of the year. Then we rechecked that the radiation code was still giving good results at very low CO2 levels…. and it turned out that it wasn’t – and so we needed to update the code via comparisons with a more complete line-by-line model so that all the tests we were doing were within the validated range of the radiative-transfer code. Finally, we did many more tests – more combinations, different baselines – to try and ensure that the results were robust.

When it came time to submit the paper, we first tried pitching it to BAMS as a popular science piece that would try and explain the concept and clear the air (so to speak). However, for various reasons this didn’t work out (two rounds of unsatisfying reviews). I’d say it was mainly due to the draft not really being pitched at the right level for BAMS. One amusing aspect of the process was that one of the referees initially suggested that our paper wasn’t necessary because it was common knowledge that the attribution to CO2 was between 9 and 26% (sound familiar?). As it turns out, they were reading a page from UCAR which was quoting (without attribution!) from my original blog post.

There was one other interesting (and highly critical) review which objected to the criticism of Lindzen’s 1991 comment, though it is perhaps worth noting that they considered the 1991 comment to be ‘formally incorrect’.

After a period in which I was a little tired of the whole exercise (it happens), we then submitted the paper to JGR, where it had an easier passage. At the same time, the simulations we had done for the paper were also used as part of a broader paper that Andy Lacis wanted to put together for Science. Both these papers appeared in October 2010 – some five years after the initial post, 3 and a half years after the first journal submission, 5 rewrites and 11 reviews.

So why bother to turn ideas from blog posts into real papers? Well, first off, you get to do a much more thorough job. You have the time and space to check multiple variations of the method, and you can take the time to do a proper literature review. And because you have put more effort into it, it is rightly seen as more credible. It’s worth noting that in our case the paper benefited from comments from all reviewers (even the very critical one) – language was tightened up, a broader literature search was done, concepts were clarified and many of the additional issues raised were dealt with.

In turn, the more credible work on the topic forms a stable point around which to craft a critique (if desired), and hopefully provides more of substance to critique (versus a series of blog posts that can be laced with distracting commentary, conceptual errors and moving targets).

To be clear, it is not only the reviews by peers that makes a peer-reviewed paper better than a blog post. Since it is known ahead of time that there is an effort required to get past the peer-review hurdle, the resulting work is usually more reflective, more interesting, more concise and more of a serious contribution – even before it gets to the editor.

Thus when scientists who find themselves criticised in the blogosphere quite often ask their critics to submit their points for peer-review, the point is not to dismiss a critique, but rather to encourage the critics to make the critique as well formulated and a propos as possible. This doesn’t always work of course, but is it nonetheless worthwhile. The alternative, especially for high profile issues, is to try and deal with a multi-headed hydra of critiques that range from the ill-informed to the excellent. Unfortunately, technical commentary does not work well at the ‘speed of blog’ and conversations often take a personal turn (which only rarely happens in the literature).

The many existing critiques of peer review as a system (for instance by Richard Smith, ex-editor of the BMJ, or here, or in the British Academy report), sometimes appear to assume that all papers arrive at the journals fully formed and appropriately written. They don’t. The mere existence of the peer review system elevates the quality of submissions, regardless of who the peer reviewers are or what their biases might be. The evidence for this is in precisely what happens in venues like E&E that have effectively dispensed with substantive peer review for any papers that follow the editor’s political line – you end up with a backwater of poorly presented and incoherent contributions that make no impact on the mainstream scientific literature or conversation. It simply isn’t worth wading through the dross in the hope of finding something interesting.

In the end of course, the science will win out. No single paper is ever the last word on an issue, and there are always new approaches to try and new data to assimilate. But the papers will endure long after the plug has been pulled on a blog. I certainly think that blogs can be of tremendous value in bringing up more context and dispelling the various mis-apprehensions that exist, but as a venue for actually doing science, they cannot replace the peer-reviewed paper – however painful that publishing process might be.

207 Responses to “From blog to Science

  1. 201
    Septic Matthew says:

    160, gavin in comment: There is variability in the Holocene, but it is very small compared to glacial times (Stage 3). Greenland is not the world and most of the other proxies for this time period are pretty static and often out of phase with greenland (see Wanner et al (2008)).

    What you have provided is evidence for spatio-temporal structure in the oscillations, of the sort commonly found in dissipative non-linear systems; for reference, Kondepudi and Prigogine, chapters 18 and (esp.) 19. That is not evidence for staionarity. The Greenland ice core data show peaks and troughs, with interpeak intervals of about 1,000 years; equally importantly, the peaks show a declining trend, indicative (if not evidence) of a slight imbalance in insolation with a sign opposite what you hypothesized. With the number of cycles observed in the Holocene, it’s impossible to judge the issue of stationarity with confidence (unless you have a large prior for stationarity), but the evidence to date makes it look more non-stationary than stationary.

    Should the hypothesis of stationarity have a high prior probability? I learned of the Digital Orrery in a book by Ruelle. According to some accurate simulations, the revolutions of the planets (including Earth) about the sun are never periodic. The effect on Earth climate of the evolution of the revolutions (sorry, couldn’t resist) could be slight, but the result discredits the assumption of stationarity.

    Of the extant models, which do you think is likely, on present evidence, to make the most accurate predictions for the future? If I had to bet, which as a voter I pretty much have to, I’d bet on the average (you have heard of Bayesian model averaging, probably) of the models of Latif and Tsonis. I might bet on the average of 5 models (including Hansen, 1988 or a recent update if there is one that Hansen likes best), with most probability assigned to the models of Latif and Tsonis.

    [Response: Tsonis doesn’t have a model in the GCM sense. The analysis he and Swanson did used the standard IPCC AR4 simulations. The Latif model is, I presume, what they published in Keenlyside et al? But here you are confusing a configuration (initiallised decadal predictions vs. free-running coupled models) with the model itself (which can be run in multiple different ways). The Keenlyside et al predictions have been way off so far, and are likely to stay so, therefore I’m not clear why you would want to weight them heavily. – gavin]

  2. 202
    Septic Matthew says:

    201, gavin in comment: The Keenlyside et al predictions have been way off so far, and are likely to stay so, therefore I’m not clear why you would want to weight them heavily

    Thank you for the reply. You are right. I got a little ahead of myself; I need to examine their predictions more closely, on an annual basis. I mean to weight them proportionally to the inverse of the accumulated mean square error, by year, beginning with the the year 2011 for model predictions made before 2010. As time goes by, the most accurate models should dominate the predictions.

    The alternative is to start with some priors on the models, and to compute the posterior probabilities annually from the observed likelihoods, and then to make the prediction from a weighted mean with the posterior probabilities of the models. Bayesian model averaging is seldom accompanied by a really convincing exposition of how to select the prior weights.

  3. 203
    David B. Benson says:

    Bob (Sphaerica) @193 —

    :-)

  4. 204
    Brian Dodge says:

    “…the bad guys are going to win.”

    They may think they’re winning, and may indeed win a few political battles, but they are fighting the wrong foe – science.

    Reality will win the war. The main question is who, and how many, are casualties along the way.

  5. 205

    Bob 194,

    That’s a pretty good idea. I’ll see what I can do, but it might take me a while.

  6. 206
    Horatio Algeranon says:

    Two blogs diverged in a warming world
    And sorry I could not reconcile both
    And be one blogger, with Italian flag unfurled,
    I looked down one to where it quarreled
    With the gravity of warming and CO2 growth…

    The Blog Not Taken

  7. 207

    Horatio,

    I just read the whole thing–ROFLMAO!!! I love it. My English-major wife got it at the first line. Good show.