Recently, a Financial Times op-ed criticised the IPCC for having contributors and peers drawn from a narrow professional circle. I don’t think this is fair, unless one regards a whole discipline as ‘narrow’. Furthermore, recent public disclosure of both comments and response suggests a different story to the allegations in the FT op-ed of ‘refusing to disclose data and methods’. The IPCC has no control over the independent publication, but the disclosure of the comments and response at least enhances the openness for the synthesis of the report.
But it is pretty unusual in common scientific peer review that reviewer comments, such as those for the two first drafts of the IPCC Working Group I report (the scientific basis), are available on line – usually the reviewers’ comments are anonymous and the process concerning manuscript reviews are closed to the wider community. However, this does provide a good demonstration that the process leading up to the IPCC report has indeed been transparent and accountable.
For some, this release of reviewers’ comments may appear as a change of heart by the IPCC, and that it may look like the IPCC did not originally intend to do this. However, the fact that all the reviewers’ comments are available on-line, proves that all the information was indeed stored and organised for the benefit of the outside community (who ever they were intended to be). The proof of the pudding is in the eating.
An inspection of the comments shows that there has been a large number of reviewers. Much higher number than a paper review usually relies on (and don’t forget, the assessment is made of papers published in peer-reviewed literature and have already been reviewed for the individual journals).
One should also keep in mind that the main point of the reviewing exercise was to provide critical voices to the IPCC process. That is how science works and how it is supposed to work. Hence, it should not come as a surprise that one may find critical comments.
When comparing the IPCC process to normal peer-review, one should note that there are substantial differences: normal peer-review is not only anonymous but also non-public; in normal peer-review the reviewers do the whole paper not just bits; in normal peer-review the authors have a different balance of power with the reviewers.
Also the review process was very open – practically anyone could comment. There were a whole pile of comments from various government reviewers for the second round. In fact there is a lot to be said on the differences, however, this is not really the focus of this post.
Some have used these critical comments to create the impression that there are critical views to Al Gore’s film. Such framing displays a misunderstanding of the whole endeavour, since after all, it’s the final version of the report that really matters.
An important question is, however, how these critical comments were dealt with. There are several cases where obvious nonsense was rejected, but also plenty of examples where critical stuff was accepted.
The authors of the report used the input from the reviewers to improve the report. In some cases, the authors may disagree with the comments – after all, it is them who are the authors of the report; not the reviewers.
And finally, one should not forget that it is the final version that counts. In summary, the IPCC is indeed an open and transparent process and there is plenty of criticism from all quarters during the review process and the idea that it is just a closed forum is completely bogus.
Update: old link doesn’t work. Here is a new link.