{"id":15551,"date":"2013-06-28T16:31:45","date_gmt":"2013-06-28T21:31:45","guid":{"rendered":"http:\/\/www.realclimate.org\/?p=15551"},"modified":"2014-12-07T15:18:00","modified_gmt":"2014-12-07T20:18:00","slug":"a-new-experiment-with-science-publication","status":"publish","type":"post","link":"https:\/\/www.realclimate.org\/index.php\/archives\/2013\/06\/a-new-experiment-with-science-publication\/","title":{"rendered":"A new experiment with science publication"},"content":{"rendered":"
\n

A while ago, I received a request to publish a paper on a post that I had written here on RealClimate, exposing the flaws in the analysis of Humlum et al., (2011)<\/a><\/span>. <\/p>\n

Instead of writing a comment to one paper, however, I thought it might be useful to collect a sample of papers that I found unconvincing (usual suspects), and that have had a fairly high public profile.<\/p>\n

<\/p>\n

The first step was to examine the selection of papers and repeat their analysis. For this, I created an R-package in the vein of Pebesma et al., 2012<\/a> (‘replicationDemos<\/a>‘). After having repeated the analysis in these papers, I felt that I had a good understanding of how their methods worked.<\/p>\n

The main focus was on the methods, and I could test them to see how robust the results were and see if the conclusions depended on the exact set-up of the analysis. <\/p>\n

This process also made a number of logical flaws visible, and showed that some of the papers had applied methods in a na\u00efve fashion, neglecting tacit knowledge about their limitations. <\/p>\n

There were other papers too, where the flaw was obvious, e.g. mixing up statistical concepts or inconsistent physics.<\/p>\n

Finally, I wrote the paper together with some colleagues of mine, and we tried to publish the synthesis of this work together with an essay on the connection to public view on climate change and the peer review system.<\/p>\n

An interesting and new dimension to this was to see the common denominator found in some of these papers. They did not appear to be just some random cases which happened to be in the lower part of a statistical distribution describing a hypothetical quality of papers. These papers came more in clusters – perhaps like minds attract each other?<\/p>\n

Our efforts met strong resistance, with one reviewer dead against our paper (others more favourable), and after a couple of rejections, we proceeded to publish the work as a ‘discussion paper’ in Earth System Dynamics (European Geophysical Union). There it is open for comments (Agnotology: learning from mistakes<\/a>). <\/p>\n

The interesting part is to read the comments – all very negative in one way or the other. <\/p>\n

One commenter revealed that he had been the negative reviewer in one of the previous submissions, but since our paper is criticizing his own work, one could hardly avoid conflict of interest. His comment fits nicely into our discussion, however, as it shows how little substance his arguments had in the first place (they were more or less repeated in the comment to the discussion paper).<\/p>\n

There are quite a few gems in the comments, which I will not disclose here (just follow the link<\/a>).<\/p>\n

I don’t know if the paper will get published in the end, but this is nevertheless an interesting experiment. I don’t think there are many similar papers like our out there – if anyone knows, it would be nice to have a list in the commentary field below.<\/p>\n

update: Dana Nuccitelli has also written a blog about our paper on SkepticalScience<\/a>. Another thing I’d be interested to hear is if anybody find the cases discussed in the paper too superficial\/not convincing. Or alternatively which is most persuasive.<\/strong><\/p>\n

References<\/h2>\n
    \n
  1. <\/a>\nO. Humlum, J. Solheim, and K. Stordahl, \"Identifying natural contributions to late Holocene climate change\", Global and Planetary Change<\/i>, vol. 79, pp. 145-156, 2011. http:\/\/dx.doi.org\/10.1016\/j.gloplacha.2011.09.005<\/a>\n\n\n<\/li>\n<\/ol>\n\n<\/div> ","protected":false},"excerpt":{"rendered":"

    A while ago, I received a request to publish a paper on a post that I had written here on RealClimate, exposing the flaws in the analysis of . Instead of writing a comment to one paper, however, I thought it might be useful to collect a sample of papers that I found unconvincing (usual […]<\/p>\n","protected":false},"author":11,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","footnotes":""},"categories":[1,28,38,34],"tags":[],"class_list":["post-15551","post","type-post","status-publish","format-standard","category-climate-science","category-reviews","category-scientific-practice","category-skeptics","entry"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t