RealClimate logo

Science is self-correcting: Lessons from the arsenic controversy

Filed under: — group @ 29 December 2010

Recent attention to NASA’s announcement of ‘arsenic-based life’ has provided a very public window into how science and scientists operate. Debate surrounds the announcement of any controversial scientific finding. In the case of arseno-DNA, the discussion that is playing out on the blogs is very similar to the process that usually plays out in conferences and seminars. This discussion is a core process by which science works.

The arseno-DNA episode has displayed this process in full public view. If anything, this incident has demonstrated the credibility of scientists, and should promote public confidence in the scientific establishment.

The story begins with a long-standing scientific consensus backed by an enormous amount of data: DNA is made with a phosphate backbone. Alternative backbones, such as arsenate, have long been considered unlikely for theoretical reasons.

Nonetheless, despite this consensus, reputable scientists have promoted the study of alternatives challenging the prevailing view. And NASA has willingly funded these studies.

Lesson one: Major funding agencies willingly back studies challenging scientific consensus.

The research team, Felisa Wolfe-Simon and colleagues, behind this study collected data and concluded that they had sufficient evidence to demonstrate incorporation of arsenate into bacterial DNA. Although the data were preliminary in nature, Science accepted the manuscript (pdf). With a high profile, potentially groundbreaking paper about to be published, NASA announced a press conference to publicize the findings.

Lesson two: Most everyone would be thrilled to overturn the consensus. Doing so successfully can be a career-making result. Journals such as Science and Nature are more than willing to publish results that overturn scientific consensus, even if data are preliminary – and funding agencies are willing to promote these results.

Within days of the arsenic paper’s publication, strong criticism of the study began to appear on scientific blogs. These blogs attracted the attention of the mainstream scientific press. Soon thereafter, media reported the wide skepticism within the scientific community – with some scientists going so far as to say that the paper should not have been published.

These scientific criticisms opened the door to those wishing to discredit science and the peer-review process, with the contrarian blogs suggesting that this study demonstrates that peer-review is “broken”. A comment on Watts’ blog summarizes their thinking:

It’s amazing how fast the scientific community came out to attack NASA for what they claim is plainly flawed science. Then again, NASA isn’t funding any of the attackers.

In the Climategate mess however, we still have heard very little from an awful lot of so-called scientists who should have been saying a lot more about flawed science but are too afraid to lose their grant money.

This raises an interesting question: just who is critiquing the NASA study? It turns out that many of the critics are also NASA-funded. In fact, many prominent critics of this study are funded by the NASA Astrobiology Institute – the very same program that funded the arsenic study.

Carl Zimmer gives us several examples:

  • Norm Pace offers the critique: “Low levels of phosphate in growth media, naive investigators and bad reviewers are the stories here”.
  • Shelley Copley suggested, “this paper should not have been published”
  • Roger Summons remarked that a critical experiment was left undone, and backed the critical blog analysis of his NASA-funded former student.
  • Michael Russell agreed with blogosphere critics, and offered his own critique of the study based on cosmic ratios of phosphorus to arsenic. Russell is a member of the Astrobiology Institute, as well as an employee of NASA’s Jet Propulsion Laboratory.
  • Forest Rohwer observed, “the experimental evidence in the paper is pretty weak.”
  • George Cody says he “cannot accept this claim until such an experiment [mass spectrometry] (easily done) is performed.”
  • Steven Benner was an early skeptic. To NASA’s credit, they invited him to present his criticisms at the press conference. He has said “we are not expecting this result to survive”.

Each of these scientists is affiliated with NASA Astrobiology.

Lesson three: Scientists offer opinions based on their scientific knowledge and a critical interpretation of data. Scientists willingly critique what they think might be flawed or unsubstantiated science, because their credibility – not their funding – is on the line.

Regardless of whether or not ‘arseno-DNA’ survives the test of time and further study, scientists have shown that they will rigorously criticize science perceived as flawed, with no fear of reprisal from funding agencies.

This is the key lesson to take from this incident, and it applies to all scientific disciplines: peer-review continues after publication. Challenges to consensus are seriously entertained – and are accepted when supported by rigorous data. Poorly substantiated studies may inspire further study, but will be scientifically criticized without concern for funding opportunities. Scientists are not “afraid to lose their grant money”.

Finally, there is the issue of how scientists who publish papers that generate credible blog reactions should in turn react. In times past, it was simple to wait for properly crafted letters and comments to be sent in to the journal. This gave fixed targets to deal with and allowed for considered reflection and response; discussions would perhaps be published 6 months to year later. But today, serious criticisms can arrive immediately (as seen above). Nature (perhaps with a little schadenfreude) had an op-ed suggesting that the authors on this (Science) paper should be more strongly engaged in the reaction, while Science had a plea from the lead author for a little patience, since they were clearly a little overwhelmed.

In our view, this needs to be thought about clearly on a case by case basis. Some criticisms (that for instance accuse the authors of deliberate fraud or misconduct based on a dislike of the conclusions) are not worth rapidly responding to, but it is worth trying to head off any misinterpretations that might be emerging. Short form papers (even with copious supplementary information) do not provide full context for the results in themselves, and so putting together a response to frequently asked questions is certainly useful (as Dr. Wolfe-Simon and colleagues have). This doesn’t replace the need for technical commentary to pass via the peer-review process though. In the end, that is what people will refer back to.

203 Responses to “Science is self-correcting: Lessons from the arsenic controversy”

  1. 201
    Nathanael says:

    “Any comments on “the decline effect” as described in this recent New Yorker article?

    Interesting that most of the examples were from the field of neuroscience.”

    There’s a lot of obvious causes, and they’re all in the article. It’s a great lesson in the problems of probability and statistics. When you have any sort of pervasive bias, they become very hard to use. And we have known psychological biases, which create publication bias, selective reporting, and other biases, preventing random sampling. Even then, chance generates lots of noise — using the 95% confidence interval, 1/20 study results will be nonsense, even *with* a truly random sample.

    It’s a good article and Ionallis gives good advice at how to fight the biases. The Journal of Negative Results is also an excellent project.

    But this leads me to another point.

    Because of the extreme political hostility to the results of real climate science, it is one of the few subjects which is *not* going to be subject to such publication bias. Any solid negative result *will* get publicity. Anyone who designs a study well, trying to get a negative result, *knows* they can get funded for it. Every climate scientist *wants* to get results which show that global warming isn’t as bad as we thought, because if it is this bad, all our kids will be in for a nightmarish future.

    The biases are *against* finding evidence of severe manmade global warming, which is probably why even though the principles have been known since the 1970s, and there was evidence already at that time, nobody really sounded the alarm until the 1990s, and to the general public not until the 2000s.

  2. 202
    Nathanael says:

    “There are lots of cases where guidance to policy is needed”

    Well, really, the climate science tells us, if we want a better life for our kids, basically, we (the whole world) should go on a wartime austerity footing regarding fossil fuels, and start a massive crash program to build a carbon-negative economy as fast as possible. Draft everyone into the solar panel and wind farm and electric car and train factories, build any hydro which is still buildable, and wave farms, and geothermal, insulate everything, plant massive numbers of trees, fund every form of artificial carbon fixation research, etc.

    But it’s politically “impossible”. Unfortunately the science of psychology is in its infancy and nobody knows how to change the minds of the people who could make it happen. Heck, our governments are still fighting utterly pointless wars which are also deeply unpopular. Getting them to actually pay attention to real national (and world) security threats? I don’t have the slightest idea how to do it, as there isn’t a….

  3. 203
    Nathanael says:

    Septic wrote at 149: “I don’t want to put words in your mouth, but you seem to be asserting that Cargill, ADM, GE, Siemens, former V.P. Al Gore, Sharp and others all have been silent these last 5 – 10 years.”

    The problem is that they are outfunded by first, the oil industry and second, a much worse group.

    I’m having trouble describing the much worse group. [edit – please dial it down. This is a forum for discussing climate science, not your theories about the presumed pathologies of company directors]