Someone C.A.R.E.S.

Do we need a new venue for post-publication comments and replications?

Social media is full of commentary (of varying degrees of seriousness) on the supposed replication crisis in science. Whether this is really a crisis, or just what is to be expected at the cutting edge is unclear (and may well depend on the topic and field). But one thing that is clear from all the discussion is that it’s much too hard to publish replications, or even non-replications, in the literature. Often these efforts have to be part of a new paper that has to make its own independent claim to novelty before it can get in the door and that means that most attempted replications don’t get published at all.

This is however just a subset of the difficulty that exists in getting any kind of comment on published articles accepted. Having been involved in many attempts – in the original journal or as a new paper – some successful, many not, it has become obvious to me that the effort to do so is wholly disproportionate to the benefits for the authors, and is thus very effectively discouraged.

The overall mismatch between the large costs/minimal benefit for the commenters, compared to the real benefits for the field, suggests that something really needs to change.

I have thought for a long time that an independent journal venue for comments would be a good idea, but a tweet by Katharine Hayhoe last weekend made me realize that the replication issue might be well served by a similar approach. So, here’s a proposal for a new journal.

Commentary And Replication in Earth Science (C.A.R.E.S.)

It is well known that existing approaches to post-publication reviews of science are hampered by long delays while responses are gathered, the inbuilt disincentives of journals to want to publish commentary that is (even implicitly) critical of their editorial decisions, the high bar for what is deemed a worthwhile criticism, and the difficulty in tracking commentary that occurs informally (on social media, blogs, conference remarks). Indeed, key journals in Earth Sciences – Nature, Science, PNAS, GRL – have very limited or no comment facility at all.

Some recent attempts to remedy this have been useful – particularly PubPeer (as discussed here). But that effort is based on anonymous comments, and has had it’s greatest success in finding dubious imagery/claims in biological journals. For whatever reason, it has not taken off as a well-used commentary platform in Earth Sciences. Nonetheless, there are some very useful innovations that have arisen from this effort – notably the PubPeer browser plugin that highlights PubPeer comments that have been made on any doi quoted. For instance, if you have the plugin, you should be able to see a link to the Pubpeer comments below on a recent Nature editorial on post-publication review. Also, the automatic notification to authors of comments being made is sensible.

A new journal could provide several clear advantages over the status quo if it was able to reduce the barriers to publication while maintaining quality. For this to work it would have to have low overhead, so would be online only. Comments would be accepted related to any published paper in the field of Earth Sciences (broadly conceived). Editorial peer review would be present, but should be minimal (basically for tone and sanity). There would not be a requirement for a comment to be equivalent to a full paper. To aid recognition of efforts, comments/replications would be published and assigned a doi straight away and responses (and perhaps even conversations) would follow over time as part of the same (evolving) page. A browser plugin and notifications like for PubPeer would be extremely useful.

I envisage this journal being used in multiple ways:

Page 1 of 2 | Next page

References

  1. G. Foster, J.D. Annan, G.A. Schmidt, and M.E. Mann, "Comment on “Heat capacity, time constant, and sensitivity of Earth's climate system” by S. E. Schwartz", Journal of Geophysical Research, vol. 113, 2008. http://dx.doi.org/10.1029/2007JD009373
  2. G.A. Schmidt, "Spurious correlations between recent warming and indices of local economic activity", International Journal of Climatology, vol. 29, pp. 2041-2048, 2009. http://dx.doi.org/10.1002/joc.1831
  3. "Post-publication criticism is crucial, but should be constructive", Nature, vol. 540, pp. 7-8, 2016. http://dx.doi.org/10.1038/540007b