Scientific practice

A paper in the scientific literature has to have some minimum level of content to be worth publishing (and regrettably, the search for the ‘Least Publishable Unit’ (LPU) of work is occasionally apparent for those wishing to pad their CVs). But what happens when someone has something worth saying that falls below that level? This is might be an update to an earlier paper, with modifications to figure, one extra sensitivity test, or some other minor addition, that could be of interest to readers of the original, but it doesn’t really get to the point where one would write a whole new paper. Some new journals (such as Geoscientific Model Development, GMD) have set up mechanisms to provide versioning of papers so that small updates can be made relatively easily, but this is rather uncommon.

A common reason to want to add additional material is in the light of subsequent commentary. In the usual case of an official submitted comment, the required response provides a good opportunity to give further details, add justification, or even agree with the comment (this last one doesn’t happen very often, but it does occur). Comments are often hard to publish for all sorts of non-scientific reasons, see Rick Trebino’s appalling story for instance, and note GRL’s 2010 decision to stop accepting comments altogether. Unfortunately not all comment/response pairs that do get published are worthwhile, but I still think they can be useful.

Where criticism occurs on a blog, there is no necessity to respond (as there would be for a submitted comment), but it is possible that there is something worth addressing (not everything is of course). Responses posted to that material – either in blog comments or in other blogs are however a little unsatisfying since the blog commentary and response are not tied to the actual paper (though mechanisms like that used by ‘Research Blogging‘ or JournalTalk could conceivably be used), and can quite frequently spiral out of control (with additional criticisms, responses, and often vast amounts of irrelevant commentary). As a useful archive of a discussion, this leaves much to be desired, nonetheless, the determined reader can usually find some nuggets.

But there is a third case where the comment/response effectively never sees the light of day. For a number of reasons critics will sometimes decide not to submit a comment, but rather a whole new paper. This might be because they want to include more information than a comment would allow or are making a comment on a previous work as part of a larger paper. Or it might be that they (correctly) note that comments are not as useful on the CV as a ‘proper’ paper, or indeed, a journal does not want to accept a comment for some reason. Less nobly, comments are sometimes avoided to try to prevent the original authors from having the last word. However, there is a risk that this paper never gets published at all (perhaps because it has less than one LPU, or it isn’t very good, or it clearly nothing more than a comment on a previous paper, or the authors lose enthusiasm). In that case, the criticism, and any response to it from the original authors (if they were asked to respond), simply disappears from sight. While possible, in my experience it is very rare that the critics then turn back to the official comment route.

It is very unusual for any scientific paper to the last word on anything, and there are almost always things that, in retrospect, one would have done differently. So it is not surprising that questions get raised through all this that the original authors might want to tackle without themselves submitting a whole other paper. Theoretically most papers would benefit from a well-refereed post-publication commentary. Yet, without a formal mechanism to shepherd this process, this material generally falls through the cracks.

As readers might have surmised, this is leading up to something.

Two papers that I was an author or co-author on in recent years generated a fair amount of blog commentary – Schmidt (2009) in IJoC and Benestad and Schmidt (2009) in JGR – mostly because they were explorations of issues raised by authors critical of the mainstream view of climate science. Despite the blog discussions, however, in neither case was a comment/response pair published. A comment was submitted on Schmidt (2009) by Jos de Laat, but this did not pass peer review (rightly I think) and no more has been heard of it. In both cases however, other authors whose work was criticised (specifically Ross McKitrick and Nicola Scafetta) submitted new papers to the same journal that were effectively just extended comments. A couple of valid points were made, but much in the draft texts was either wrong or irrelevant. I was asked to respond to both submissions by the editors involved, and did so in the form of a signed draft response as if the papers had indeed been submitted as comments. In both cases however, the papers were eventually rejected. A similar paper by McKitrick and Nierenberg appeared in another journal, while Scafetta’s paper has not been seen again.

So why bring this up now?

The fact is well-crafted comments and responses on both these papers would have been a useful contribution to the literature, and it is a shame that this didn’t happen (as I stated in at least one of the responses). That the authors were unwilling to submit ‘just’ a comment for whatever reason is part of the story (but it is not clear that any actual comment would have stuck to the points I thought worth making, or that the comment/response would have passed peer review either). But both papers have been mentioned recently in various contexts and it was apparent that the conversations might have been at least a little more interesting if the at least some of the unpublished correspondence on the papers had been available.

It strikes me that these are unlikely to be unique circumstances (or at least I would be very surprised if this kind of thing hasn’t happened to other people). So is there interest in RC providing space for these kinds of discussions? Authors, with something extra to say that they don’t ever think they’ll put in a new paper, could add some extra analysis and tie up some of the loose ends. I’d be happy to start the ball rolling by dealing with some issues relating to the papers mentioned above, but is this something else anyone would care to do? Should we instead be advocating for a rapid reaction online journal specifically for worthwhile comments and replies, especially since some journals have stopped accepting them? Thoughts welcome.

Abuse of the UK libel laws is so commonplace as to require no real introduction (but see the Campaign for libel reform for more details). Because of the ridiculous costs and pro-plaintiff assumptions, it has been (ab)used by many and fought against successfully only by a few. In the realm of discussions about science, Simon Singh’s triumph over a libel suit brought by the British Chiropractors Association stands out, as does Ben Goldacre’s successful £500,000 defense against Matthias Rath – a vitamin salesman peddling bogus AIDS cures. But despite that, it remains (for now) a potent threat to throw around if you want to try to intimidate a critic.

We received this letter on Friday:

From: Bill Hughes
Cc: Sonja Boehmer-Christiansen
Subject:: E&E libel
Date: 02/18/11 10:48:01

Gavin, your comment about Energy & Environment which you made on RealClimate has been brought to my attention:

“The evidence for this is in precisely what happens in venues like E&E that have effectively dispensed with substantive peer review for any papers that follow the editor’s political line. ”

To assert, without knowing, as you cannot possibly know, not being connected with the journal yourself, that an academic journal does not bother with peer review, is a terribly damaging charge, and one I’m really quite surprised that you’re prepared to make. And to further assert that peer review is abandoned precisely in order to let the editor publish papers which support her political position, is even more damaging, not to mention being completely ridiculous.

At the moment, I’m prepared to settle merely for a retraction posted on RealClimate. I’m quite happy to work with you to find a mutually satisfactory form of words: I appreciate you might find it difficult.

I look forward to hearing from you.

With best wishes
Bill Hughes
Director
Multi-Science Publsihing [sic] Co Ltd

The comment in question was made in the post “From blog to Science” and the full context was:

The many existing critiques of peer review as a system (for instance by Richard Smith, ex-editor of the BMJ, or here, or in the British Academy report), sometimes appear to assume that all papers arrive at the journals fully formed and appropriately written. They don’t. The mere existence of the peer review system elevates the quality of submissions, regardless of who the peer reviewers are or what their biases might be. The evidence for this is in precisely what happens in venues like E&E that have effectively dispensed with substantive peer review for any papers that follow the editor’s political line – you end up with a backwater of poorly presented and incoherent contributions that make no impact on the mainstream scientific literature or conversation. It simply isn’t worth wading through the dross in the hope of finding something interesting.

The point being that if the ‘peer-review’ bar gets lowered, the result is worse submissions, less impact and a declining reputation. Something that fits E&E in spades. This conclusion is based on multiple years of evidence of shoddy peer-review at E&E and, obviously, on the statements of the editor, Sonja Boehmer-Christiansen. She was quoted by Richard Monastersky in the Chronicle of Higher Education (3 Sep 2003) in the wake of the Soon and Baliunas fiasco:

The journal’s editor, Sonja Boehmer-Christiansen, a reader in geography at the University of Hull, in England, says she sometimes publishes scientific papers challenging the view that global warming is a problem, because that position is often stifled in other outlets. “I’m following my political agenda — a bit, anyway,” she says. “But isn’t that the right of the editor?”

So the claim that the ‘an editor publishes papers based on her political position’ while certainly ‘terribly damaging’ to the journal’s reputation is, unfortunately, far from ridiculous.

Other people have investigated the peer-review practices of E&E and found them wanting. Greenfyre, dissecting a list of supposedly ‘peer-reviewed’ papers from E&E found that:

A given paper in E&E may have been peer reviewed (but unlikely). If it was, the review process might have been up to the normal standards for science (but unlikely). Hence E&E’s exclusion from the ISI Journal Master list, and why many (including Scopus) do not consider E&E a peer reviewed journal at all.

Further, even the editor states that it is not a science journal and that it is politically motivated/influenced. Finally, at least some of what it publishes is just plain loony.

Also, see comments from John Hunter and John Lynch. Nexus6 claimed to found the worst climate paper ever published in its pages, and that one doesn’t even appear to have been proof-read (a little like Bill’s email). A one-time author, Roger Pielke Jr, said “…had we known then how that outlet would evolve beyond 1999 we certainly wouldn’t have published there. “, and Ralph Keeling once asked, “Is it really the intent of E&E to provide a forum for laundering pseudo-science?”. We report, you decide.

We are not surprised to find that Bill Hughes (the publisher) is concerned about his journal’s evidently appalling reputation. However, perhaps the way to fix that is to start applying a higher level of quality control rather than by threatening libel suits against people who publicly point out the problems? Is being known as the journal who tries to sue critics of their editorial policies (or worse, tries to intimidate critics by threatening libel suits) really going to help?

As a final note, if you think that threatening unjustifiable UK libel suits against valid criticism is an appalling abuse, feel free to let Bill Hughes know (but please be polite), and add your support to the Campaign for libel reform in the UK which looks to be making great headway. In the comments, feel free to list your examples of the worst papers ever published in E&E.

Bill, if you are reading, you can take this ‘form of words’ as a full and complete response to your email.

Update: The Guardian reports on the story, and Bill Hughes sends another email.

Despite what is often claimed, climate scientists aren’t “just in it for the money”. But what scientists actually do to get money and how the funding is distributed is rarely discussed. Since I’ve spent time as a reviewer and on a number of panels for various agencies that provide some of the input into those decisions, I thought it might be interesting to discuss some of the real issues that arise and the real tensions that exist in this process. Obviously, I’m not going to discuss specific proposals, calls, or even the agencies involved, but there are plenty of general insights worth noting.
[Read more…] about Reflections on funding panels