The vast majority of mainstream media items about science are related to new hot-off-the-press studies, often in high profile journals, that report a new breakthrough, or that purportedly overturn previous ideas. However, while these are exciting news items, this preponderance of coverage given to these state-of-the-art studies compared to assessments such as from the National Academies, can give a misleading impression about the state of a scientific knowledge. The more mature and solid a field, the less controversy there is, and thus the fewer news stories. Ironically, this means the public is told the least about the most solid aspects of science.
One effect of this tendency is that quite often news stories are focused on claims that turn out to be wrong, or if not actually wrong, heavily reduced in importance by the time the dust settles. This is not deliberate, but merely how science works at the frontier. People push measurements to the limit of their accuracy (and sometimes beyond) and theories are used slightly out of their domain of applicability. In recognition of that, Richard Feynman had a useful rule of thumb that the last data point on any graph should be discounted because, if it had been easy to obtain, there would have been another one further along.
Scientists are of course aware of this, and are usually very cautious about adopting new studies that appear unless they’ve been confirmed independently. The blogosphere and Marc Morano? not so much. There have been many times when seemingly dramatic new climate studies have been widely (and wildly) touted where RealClimate (amongst others) have suggested that more caution is warranted. That caution is usually based on what can be loosely describe as “Bayesian priors” – the accumulated wisdom that underpins most of the ‘balance of evidence’ arguments supporting how we understand the real world. Conclusions that aren’t built on single pieces of evidence, but are the most likely explanation to fit a whole set of observations, are generally pretty robust to revisions to single parts of the evidence.
In the news last week was an attempted replication of the surprising finding in 2007 by Pope et al. that a key reaction rate related to polar ozone depletion was very different to what had been supposed. In a guest posting here, Drew Shindell cautioned that this was unlikely to stand since it would make good observations of related effects almost impossible to explain. And so it has proved. New methods to pin down this reaction rate have placed it very close to where people thought it was before the 2007 paper. This is not of course the final word (despite the Nature News headline), but it does illustrate how science deals with the inevitable anomalies that always crop up.
Another example of this was a paper back in 2005 that described a possibly dramatic slowing down of the North Atlantic circulation over the last 30 years. Our prior knowledge about the likely impacts of such a thing (cooling in the North Atlantic for instance) which hadn’t been seen, made us cautious about the interpretation of such a result. And a couple of years later, that caution was vindicated by much higher frequency sampling of the data which revealed a great deal more variability than had been expected, implying that the signal in the earlier paper was more likely to have been an artifact.
Similarly, in 2006 we reported being baffled by a story on plants producing methane, which lead to an unfortunate rash of erroneous ‘Trees cause global warming‘ stories (which weren’t justified even if you fully accepted the paper’s results). A few years and many attempted replications later, the potential for methane fluxes from living plants has been dialed down almost to zero.
These are just a few examples, but all of them illustrate the importance of context in interpreting new results. Not every surprising new finding will be wrong, but seeing what other scientists say about it is a good indication of how likely it is to stand. The key importance of this back and forth though is, as Marcus Rex points out in the Nature News piece on the ozone chemistry story: “Even though [the] results now seem to be wrong, his study did prompt other groups to think about alternative approaches”.
And that is a big part of what scientific research is about.
#41 Ike Solem
Just to add a bit to the PDO thing, re the Easterbrook 2-3 decade thing
http://www.ossfoundation.us/projects/environment/global-warming/pacific-decadal-oscillation
http://www.esr.org/pdo_index.html
I don’t know how anyone can look at the PDO and say with a straight face that it will be negative for 2-3 decades. That thing is all over the place.
“Al Gore will go probably down in history as the guy who claimed to have invented the Internet”
Al Gore never claimed to have invented the internet…
But nobody really cares about the truth. They only care about what they want or fear.
From the OP: One effect of this tendency is that quite often news stories are focused on claims that turn out to be wrong, or if not actually wrong, heavily reduced in importance by the time the dust settles. This is not deliberate, but merely how science works at the frontier. People push measurements to the limit of their accuracy (and sometimes beyond) and theories are used slightly out of their domain of applicability…
Scientists are of course aware of this, and are usually very cautious about adopting new studies that appear unless they’ve been confirmed independently.
Sort of tangentially related is this interview on the ABC a couple of days back, with Dr Simon Singh from the UK:
http://www.abc.net.au/lateline/content/2008/s2618503.htm
To cut a long stranscript short, the interview with Singh focussed on a libel suit being launched by the British Chiropractic Society against Singh for a piece he wrote that criticised some chiropractors in the UK for claiming they could treat some childhood conditions like colic. Singh points out that other suits have been taken against other scientists in the UK alleging defamation, even though the science that was reported was well-founded and the conclusions robust. Singh, as a science journalist, believes that defamation suits like these mean “…that good articles are often withdrawn, good articles are often gutted so that the meat of the content is removed before it’s published and sometimes articles that should be published aren’t even published or written in the first place because of the fear of libel laws in England.”
The interviewer asked Singh if the impending case hindered his ability to be openly critical (of therapies in this instance), to which Singh replied: “I think it does to a very large extent because there are things – there’s a reverse burden of proof. So, a science writer such as myself is guilty until proven innocent. The sheer costs involved. A libel case can cost 1 million pounds, and that might be over damages that are just a few thousand pounds. And so it turns journalism into a sort of high stakes poker game. Somebody threatens you with a libel suit, do you have the absolute confidence and financial resources to see it through to the end. If you don’t, then you’d probably be smarter backing off immediately and apologising, even if you think you’re right. So, it does have this intimidation effect on journalism.”
More on this at the Grauniad:
http://www.guardian.co.uk/science/2009/jun/04/simon-singh-libel-british-chiropractic-association-bca
Sorry if the subject is o/t. But I thought the wider point – that science journalism, and consequently the quality of information and legitimate criticism in the public domain, can be thwarted by fear of reprisal – was worth flagging.
Gavin,
What is the stance on the Deep Solar Minimum here? My simple mind sees it as a factor in this summers temperatures. Since you cut my post, I’m curious about your stance and why you have it.
[Response: I don’t have a stance. The sun is obviously a key boundary condition, which we clearly don’t have much predictive ability for. – gavin]
Doug Bostrom,
Thanks for this Don Easterbrook quote:
He’s brilliant. He discovered what the point of stopping CO2 emissions is to not affect the climate.
Also in the news recently was a Large-Scale
Assessment of the Effect of Popularity on the Reliability of
Research in PLoS One. Pfeiffer and Hoffmann refer to theoretical
concerns that research in “popular” fields may be less reliable (1)
because of stronger incentives to “manufacture” positive results and
(2) because multiple independent testing increases the chances of at
least one group getting a positive result. And indeed, data-mining the
literature for claims about yeast protein interactions they find inter
alia that “that individual results … become less reliable with
increasing popularity of the interacting proteins”. They suggest
intensifying efforts at post-publication evaluation through wikis and
suchlike. (No mention of the sterling work of RC, though.) This being
hot off the press, by your argument we should perhaps reserve
judgment, but: Any thoughts about the applicability to climate
science?
Thanks, Vincent!
Ed Davies wrote in to point out that I’d made a mistake in describing the nucleogenesis of carbon-14 in the lag paper–that has since been corrected.
Case in point: any thoughts on this scary story about rainforest die-back?
http://www.washingtonmonthly.com/features/2009/0907.brown.html
re CM (55):
I haven’t read the entire PLoS study (yet), and although the authors likely have a point, I would argue that whatever mistakes/errors are made in research on “popular” topics is relatively quickly identified as such, because of the number of people examining the topic. More problematic are those topics which are considered to be “backwater” areas, with relatively few people working on them. There, faulty methodological approaches, perhaps established early on, can go unexamined for long periods, such that mistakes of inference are perpetuated. This then becomes problematic whenever the less popular topic takes on a new importance because of new-found connections to other topics that are more popular or societally relevant, because others will take the results of the former and run with them. I’m dealing with this issue on two different fronts right now.
#52 EL:
Dr. Easterbrook knows he’s taking a contrarian stance with attendant risks, he’s proud of it as we can tell by his war stories. It’s remarkable how cavalier he is about torching his own credibility with careless yet illuminating remarks of the kind I highlighted. I suppose he may not understand how poorly compartmentalized access to information is these days.
What would be nice is if– when the New York Times or other popular media hand contrarians or anybody else a megaphone– they’d take a moment to help their readers assess those voices by means of a general credibility check. A general audience cannot necessarily process the science in play, but if an author is down on record disseminating untruths in his own words that’s a helpful hint about the author’s credibility and judgment.
Unfortunately popular media has been crippled by the need of investors to get money without doing work, so that sort of depth of reporting is not available.
Ike Solem (41) & EL (54) — You might care to read Tung & Cabin (2008), available from Prof. Tung’s UW website, regarding the effect of the sunspot cycle on global surface temperatures. They find a more pronounced effect than climatologists before them
Jim Boulding (#59), thanks for your thoughts. It makes good sense to me that findings in “popular” fields would also attract more scrutiny tending to rectify errors quickly. As far as I can see this possible effect is not discussed in the PLoS article. Makes me wonder if their data-mining approach could be refined to identify not only what claims have been published but also whether and how fast such claims have been contested in the literature or even retracted.
a compliment of sorts: I haven’t been a reader of RC forever and could have missed them, but, I was surprised and it struck me that this thread, drew a lot of comments from people who seemed very capable and who I’ve never heard of or seen here before. Tells me there are a bunch of credible folks who monitor RC even though they’re not active posters. That they would do this speaks highly of RC, IMO.
Also, is the PLoS article also being affected by the problem the PLoS article discusses?
Sort of OT, but:
So, global warming is caused by waste heat?
Jim (#65),
Here is a link to the article. http://www.inderscience.com/filter.php?aid=27100
It seems to have some pretty sloppy thinking. For example, they claim that they have underestimated both their global heat accumulation and their net heat “emissions” so that their “missing” heat is underestimated. Hoever, increasing global heat accumulation would increase the “missing” heat so they seem to have one of their conclusions wrong in a rather obvious way.
Increased radiative forcing from GHGs is much larger than our heat production, so it should not be too hard to find their main blunders.
Opps, that should have been “so their ‘missing’ heat is overestimated”
I’m sure you guys are busy, but if you wouldn’t mind, it would be great if you could respond to this article: “Could we be wrong about global warming?” http://blogs.usatoday.com/sciencefair/2009/07/could-we-be-wrong-about-global-warming.html
This essay you wrote already addresses it generally, but more specific responses would be really helpful.
Jim (#65),
Looking a little closer at the paper you mentioned, there are some deep flaws in the authors’ thinking. They consider it to a common notion that the energy we produce is ultimately radiated to space but argue that this happens only rarely when a source is hot (street lighting?) but otherwise is mixed to ambient temperature and retained. They do not seem to understand is that it is heat at ambient temperature that is lost to space, rapidly, and that it is radiation from the Sun which replenishes that loss. Thus, contrary to their statement, one does want to compare heat we generate from fossil energy use with incoming solar energy. Upon comparison, it is seen that our energy use in negligible.
Their core confusion seems to be in the idea of “retaining” heat. They go to some (sloppy) effort to calculate the extra heat in the earth system owing to warming using the heat capacity of the ground, oceans and air and treating ice melting as an effective heat capacity. But this holding of heat, the existence of a heat capacity, is not what is warming the surface. The surface is warming because heat can’t escape as readily owning to the chemical change in the atmosphere from introducing more greenhouse gases. Heat is being retained in the way a blanket retains heat, not in the way a thermal bath holds heat. It is being impeded in its flow. The manner of storage, while having an effect on how quickly we reach a new equilibrium temperature, is not important to the fact that the temperature is changing.
Since they have misunderstood the problem, the numerical coincidence that they find between total fossil energy use, and heat capacity filled so far is merely that. Our energy use cannot account for the increase in surface temperature and thus can’t drive the storage of heat. There is no causation without the chemical effect on the atmosphere.
Chris, thanks for the summary.
http://blogs.nature.com/news/thegreatbeyond/2009/07/global_warming_is_shrinking_yo.html
….
“… a new paper in Biology Letters comments on the fact that the larger sheep also tend to have darker coats than the smaller sheep. There is, say Shane Maloney, of the University of Western Australia, and colleagues, apparently a genetic link between coat colour and body size.
This may be another reason for the decline of the larger, black sheep: “While in the past a dark coat has offset the metabolic costs of thermoregulation by absorbing solar radiation, the selective advantage of a dark coat may be waning as the climate warms in the North Atlantic,” the researchers write…..”
Just a heads-up: CBC Radio just aired an hour-long interview with someone named Lawrence Solomon on their show called “Ideas” last week. You can download the podcast here until August 21:
http://www.cbc.ca/ideas/podcast.html (Episode is called “The Deniers”, July 21). Solomon trotted out almost every play in the denier book, including global warming on Mars, discrediting the hockey stick, grapes in England, recovery from the Little Ice Age, cosmic rays . . .
I’m in the process of writing a lengthy letter of complaint to the CBC ombudsman. I’m waiting for some feedback from colleagues before I send it, but I would be happy to post it somewhere if anyone wants to read/critique it.