Do we need a new venue for post-publication comments and replications?
Social media is full of commentary (of varying degrees of seriousness) on the supposed replication crisis in science. Whether this is really a crisis, or just what is to be expected at the cutting edge is unclear (and may well depend on the topic and field). But one thing that is clear from all the discussion is that it’s much too hard to publish replications, or even non-replications, in the literature. Often these efforts have to be part of a new paper that has to make its own independent claim to novelty before it can get in the door and that means that most attempted replications don’t get published at all.
This is however just a subset of the difficulty that exists in getting any kind of comment on published articles accepted. Having been involved in many attempts – in the original journal or as a new paper – some successful, many not, it has become obvious to me that the effort to do so is wholly disproportionate to the benefits for the authors, and is thus very effectively discouraged.
The overall mismatch between the large costs/minimal benefit for the commenters, compared to the real benefits for the field, suggests that something really needs to change.
I have thought for a long time that an independent journal venue for comments would be a good idea, but a tweet by Katharine Hayhoe last weekend made me realize that the replication issue might be well served by a similar approach. So, here’s a proposal for a new journal.
Commentary And Replication in Earth Science (C.A.R.E.S.)
It is well known that existing approaches to post-publication reviews of science are hampered by long delays while responses are gathered, the inbuilt disincentives of journals to want to publish commentary that is (even implicitly) critical of their editorial decisions, the high bar for what is deemed a worthwhile criticism, and the difficulty in tracking commentary that occurs informally (on social media, blogs, conference remarks). Indeed, key journals in Earth Sciences – Nature, Science, PNAS, GRL – have very limited or no comment facility at all.
Some recent attempts to remedy this have been useful – particularly PubPeer (as discussed here). But that effort is based on anonymous comments, and has had it’s greatest success in finding dubious imagery/claims in biological journals. For whatever reason, it has not taken off as a well-used commentary platform in Earth Sciences. Nonetheless, there are some very useful innovations that have arisen from this effort – notably the PubPeer browser plugin that highlights PubPeer comments that have been made on any doi quoted. For instance, if you have the plugin, you should be able to see a link to the Pubpeer comments below on a recent Nature editorial on post-publication review. Also, the automatic notification to authors of comments being made is sensible.
A new journal could provide several clear advantages over the status quo if it was able to reduce the barriers to publication while maintaining quality. For this to work it would have to have low overhead, so would be online only. Comments would be accepted related to any published paper in the field of Earth Sciences (broadly conceived). Editorial peer review would be present, but should be minimal (basically for tone and sanity). There would not be a requirement for a comment to be equivalent to a full paper. To aid recognition of efforts, comments/replications would be published and assigned a doi straight away and responses (and perhaps even conversations) would follow over time as part of the same (evolving) page. A browser plugin and notifications like for PubPeer would be extremely useful.
I envisage this journal being used in multiple ways:
- To make quick corrections to published work, that perhaps the original authors are unaware of, or deem too trivial or too unimportant to bother with (note that an author’s view on this is not necessarily universally shared).
- To replicate analyses (or not) using analogous data and perhaps different methods to demonstrate robustness (or not) of a result.
- To make ‘standard’ comments that take issue with some claim or conclusion of a paper.
- To be somewhere where original authors can append additional analysis that wouldn’t merit a new full paper, but that might be a relevant and useful appendix to the original paper.
Personally, I am aware of multiple examples of papers where data errors, incorrect equations, or unsupported conclusions have been made and that remain unchallenged and/or uncorrected in the literature. Often, the people who found these issues have not bothered with a formal comment because of the time and hassle of doing so, or they are under the (usually mistaken) impression that ‘everyone’ already knows. I have also had examples where supplemental material on a paper might be useful, but has never seen the light of day.
I don’t think that a journal or comments are the right venue for issues of misconduct or fraud. Journals and institutions have procedures for this (however imperfect), and trying to peer-review suggestions of misconduct without access to lab notebooks/machine logs/correspondence etc. is a job for investigative committees, not an online journal.
There are some potential pitfalls. It is possible that this journal could be used to target high profile papers for non-scientific reasons, and so occasionally a stricter peer review protocol might be needed to separate out genuine issues from worthless politicized critiques. Given that each comment would be reviewed and have a doi, some people might decide to bulk up their publication list by submitting a lot of comments. I don’t actually think that is too bad though – assessors of their research output should be able to see through this, if indeed the comments are of little worth.
As is well known, comments and responses can become contentious and editors would need to have the means to tone down inflammatory rhetoric, disallow questions of motivation and remove unsupported accusations of misconduct. Again, the experience of PubPeer with this kind of issue is instructive.
To help potential commenters, it might be useful to have a guide to effective commenting/replications, though it would be mostly common sense i.e. stick to the point, don’t add extraneous criticisms, don’t discuss motivations, assume good faith, don’t overreach etc.
Funding for this journal would need to be thought about. While #openaccess for the submissions would be ideal, page charges would be a disincentive to commenting, but might be modest enough for short contributions to not be too much of a hindrance. Author typesetting using Word or LaTeX templates might reduce costs further.
So, what do people think? Is this worth pursuing formally? Anyone want to sign up for the future editorial board? If you are a publisher, is this something you’d support?
Perhaps it should go without saying, but there is an obvious branding advantage for any publisher that takes this on (with the possible exception of the World Health Organization). I mean, wouldn’t you want to be the publisher or society that CARES?
- G. Foster, J.D. Annan, G.A. Schmidt, and M.E. Mann, "Comment on “Heat capacity, time constant, and sensitivity of Earth's climate system” by S. E. Schwartz", Journal of Geophysical Research, vol. 113, 2008. http://dx.doi.org/10.1029/2007JD009373
- G.A. Schmidt, "Spurious correlations between recent warming and indices of local economic activity", International Journal of Climatology, vol. 29, pp. 2041-2048, 2009. http://dx.doi.org/10.1002/joc.1831
- "Post-publication criticism is crucial, but should be constructive", Nature, vol. 540, pp. 7-8, 2016. http://dx.doi.org/10.1038/540007b