It is a truism that all models are wrong. Just as no map can capture the real landscape and no portrait the true self, numerical models by necessity have to contain approximations to the complexity of the real world and so can never be perfect replications of reality. Similarly, any specific observations are only partial reflections of what is actually happening and have multiple sources of error. It is therefore to be expected that there will be discrepancies between models and observations. However, why these arise and what one should conclude from them are interesting and more subtle than most people realise. Indeed, such discrepancies are the classic way we learn something new – and it often isn’t what people first thought of.
The first thing to note is that any climate model-observation mismatch can have multiple (non-exclusive) causes which (simply put) are:
- The observations are in error
- The models are in error
- The comparison is flawed
In climate science there have been multiple examples of each possibility and multiple ways in which each set of errors has arisen, and so we’ll take them in turn.
1. Observational Error
These errors can be straight-up mistakes in transcription, instrument failure, or data corruption etc., but these are generally easy to spot and so I won’t dwell on this class of error. More subtly, most of the “observations” that we compare climate models to are actually syntheses of large amounts of raw observations. These data products are not just a function of the raw observations, but also of the assumptions and the “model” (usually statistical) that go into building the synthesis. These assumptions can relate to space or time interpolation, corrections for non-climate related factors, or inversions of the raw data to get the relevant climate variable. Examples of these kinds of errors being responsible for a climate model/observation discrepancy range from the omission of orbital decay effects in producing the UAH MSU data sets, or the problems of no-modern analogs in the CLIMAP reconstruction of ice age ocean temperatures.
In other fields, these kinds of issues arise in unacknowledged laboratory effects or instrument calibration errors. Examples abound, most recently for instance, the supposed ‘observation’ of ‘faster-than-light’ neutrinos.
2. Model Error
There are of course many model errors. These range from the inability to resolve sub-grid features of the topography, approximations made for computational efficiency, the necessarily incomplete physical scope of the models and inevitable coding bugs. Sometimes model-observation discrepancies can be easily traced to such issues. However, more often, model output is a function of multiple aspects of a simulation, and so even if the model is undoubtedly biased (a good example is the persistent ‘double ITCZ’ bias in simulations of tropical rainfall) it can be hard to associate this with a specific conceptual or coding error. The most useful comparisons are then those that allow for the most direct assessment of the cause of any discrepancy.”Process-based” diagnostics – where comparisons are made for specific processes, rather than specific fields, are becoming very useful in this respect.
When a comparison is being made in a specific experiment though, there are a few additional considerations. Any particular simulation (and hence diagnostic from it) arises as a result from a collection of multiple assumptions – in the model physics itself, the forcings of the simulation (such as the history of aerosols in a 20th Century experiment), and the initial conditions used in the simulation. Each potential source of the mismatch needs to be independently examined.
3. Flawed Comparisons
Even with a near-perfect model and accurate observations, model-observation comparisons can show big discrepancies because the diagnostics being compared while similar in both cases, actually end up be subtly (and perhaps importantly) biased. This can be as simple as assuming an estimate of the global mean surface temperature anomaly is truly global when it in fact has large gaps in regions that are behaving anomalously. This can be dealt with by masking the model fields prior to averaging, but it isn’t always done. Other examples have involved assuming the MSU-TMT record can be compared to temperatures at a specific height in the model, instead of using the full weighting profile. Yet another might be comparing satellite retrievals of low clouds with the model averages, but forgetting that satellites can’t see low clouds if they are hiding behind upper level ones. In paleo-climate, simple transfer functions of proxies like isotopes can often be complicated by other influences on the proxy (e.g. Werner et al, 2000). It is therefore incumbent on the modellers to try and produce diagnostics that are commensurate with what the observations actually represent.
Flaws in comparisons can be more conceptual as well – for instance comparing the ensemble mean of a set of model runs to the single realisation of the real world. Or comparing a single run with its own weather to a short term observation. These are not wrong so much as potentially misleading – since it is obvious why there is going to be a discrepancy, albeit one that doesn’t have much implications for our understanding.
Implications
The implications of any specific discrepancy therefore aren’t immediately obvious (for those who like their philosophy a little more academic, this is basically a rephrasing of the Quine/Duhem position on scientific underdetermination). Since any actual model prediction depends on a collection of hypotheses together, as do the ‘observation’ and the comparison, there are multiple chances for errors to creep in. It takes work to figure out where though.
The alternative ‘Popperian’ view – well encapsulated by Richard Feynman:
… we compare the result of the computation to nature, with experiment or experience, compare it directly with observation, to see if it works. If it disagrees with experiment it is wrong.
actually doesn’t work except in the purest of circumstances (and I’m not even sure I can think of a clean example). A recent obvious counter-example in physics was the fact that the ‘faster-than-light’ neutrino experiment has not falsified special relativity – despite Feynman’s dictum.
But does this exposition help in any current issues related to climate science? I think it does – mainly because it forces one to think about the other ancillary hypotheses are. For three particular mismatches – sea ice loss rates being much too low in CMIP3, tropical MSU-TMT rising too fast in CMIP5, or the ensemble mean global mean temperatures diverging from HadCRUT4 – it is likely that there are multiple sources of these mismatches across all three categories described above. The sea ice loss rate seems to be very sensitive to model resolution and has improved in CMIP5 – implicating aspects of the model structure as the main source of the problem. MSU-TMT trends have a lot of structural uncertainty in the observations (note the differences in trends between the UAH and RSS products). And global mean temperature trends are quite sensitive to observational products, masking, forcings in the models, and initial condition sensitivity.
Working out what is responsible for what is, as they say, an “active research question”.
Update: From the comments:
“our earth is a globe
whose surface we probe
no map can replace her
but just try to trace her”
– Steve Waterman, The World of Maps
References
- M. Werner, U. Mikolajewicz, M. Heimann, and G. Hoffmann, "Borehole versus isotope temperatures on Greenland: Seasonality does matter", Geophysical Research Letters, vol. 27, pp. 723-726, 2000. http://dx.doi.org/10.1029/1999GL006075
Gavin-
You say:
I’m not sure I understand the basis for this claim. Cost benefit analysis is a standard requirement whenever a commitment of resources of this magnitude is contemplated. There is simply no other way for governments to determine how to best allocate the resources at their disposal. The fact that numerous peer reviewed studies have attempted to quantify impacts (e.g. Yohe et al. [2007]) demonstrates that even if you do not understand the need to conduct this type of analysis, others do.
[Response: Ha. I’m actually well aware of the literature on cost-benefit studies and that most (even from Nordhaus) suggest that action is warranted. But the point I am making is somewhat different; namely, I do not think that cost-benefit analyses are anything but a rough guide to different pathways. Damage functions in these analyses are simplistic and mostly untested for the magnitudes of change projected, their ability to balance economic costs and social costs involve ethical judgements that are both opaque and not universal. The other point is that government decisions are made all the time with huge consequences with no such modeling being done – was the TARP bail-out assessed for it’s impact on the African poor? Is the EU CAP? etc. So while I am all for policy-specific impacts modeling, I recognise it’s limitations. – gavin]
The confidence comes from the fact that third world countries have consistently voiced their objections to certain mitigation efforts such as carbon taxes on grounds that they, not the industrial world, would bear the brunt of the burden.
[Response: Where does this come from? Most actions being contemplated are related to western use of fossil fuels for western consumption of transport and electricity. Impacts of a carbon price in the US say, are not going to be predominantly felt in Somalia. – gavin]
I agree that energy poverty is a symptom of the institutional issues. However, you cannot deny that anything that makes energy more expensive will have severely negative impacts on the third world. If local solar is indeed cheaper than fossil fuels, there is no need for carbon taxes to make fossil fuels more expensive–market forces will ensure the switch all by themselves.
[Response: Not true. The biggest handicap in rural poverty is lack of access to capital – even if new projects are enormously beneficial on even a standard cost/benefit analysis, they still do not get done. The idea that market forces automatically do the best thing in these cases is a fallacy. – gavin]
I never claimed that climate must be balanced directly and exclusively against the global poor.
[Response: Good. – gavin]
Tamino’s on a roll. Referring to the 15-year span from 1992 – 2006 when the rate of surface warming was greater than the long-term trend,
“Hardly-earned riches”. I am so stealing that.
Bradley McKinley #148
We can quantify past impacts based on the sea level feedback (cryosphere response). This tells us that with the current climate forcing we established at least a Pliocene climate response, which was triggered by anthropogenic induced energy imbalance. And that is a very bad thing, because paleoclimate data tells us that the Pliocene regime 5 mil years ago had 15 meter higher sea level. Further do we know from proxy records (corals) that the slow ice sheet feedback includes non-linear episodes, when ice sheets disintegrate and abruply cause SLR.
James Hansen explains Earth energy imbalance
Link
bjchip:
That’s what I’ve gathered from KX13 and discussions of it (especially John Nielsen-Gammon’s), and I’ve been using it against a couple of “15-year pause” parrots elsewhere. I’m eager to read RC’s post on it, even if I find out I’ve got it backwards 8^}.
re: 151 and Cost Benefit Analysis
I remember the famous 2% Doctrine of the Bush era, i.e. if there were a 2% chance of WMDs, we had to intervene.
And then there’s Luke 19:22.
Gavin wrote: “… there are large parts of rural Africa where local solar (for instance) is already cheaper than extending the grid to supply fossil energy.”
Bradley McKinley: “If local solar is indeed cheaper than fossil fuels, there is no need for carbon taxes to make fossil fuels more expensive – market forces will ensure the switch all by themselves.”
Carbon taxes have been proposed in developed countries as a mechanism to internalize the environmental and public health costs of burning fossil fuels, which are currently externalized and foisted off on the public. As such, carbon taxes ARE a “market force”. They simply “correct” the market by forcing it to reflect the true cost of fossil fuels. Failure to do that amounts to a massive public subsidy to fossil fuels, which is why they appear to be (but actually are not) “cheaper” than renewable energy.
None of which has anything to do with poor people in rural Africa that Gavin wrote about — people who use little or no fossil fuels, and who would therefore not pay any carbon taxes, even if someone were to impose such taxes in rural Africa, which no one has suggested doing.
The reality is that many of those people, and millions of others like them throughout the developing world who have NO access to electricity, will NEVER have access to fossil-fuel-fired electricity because no one is ever going to build the centralized power plants and the grid to deliver electricity to them. The ONLY way they will ever have electricity is with distributed renewable energy, principally photovoltaics — which are vastly less expensive than building a whole new grid-based electricity generation and distribution infrastructure.
And in fact, although it doesn’t get a lot of media attention, there is an ongoing revolution in rural solar electrification in the developing world, where low-cost systems as simple as a few solar panels and batteries are providing power for electric lights, refrigeration, medical equipment, radios, cell phones, satellite TV and Internet access, and more, to entire villages.
The Truth About Global Warming – Science & Distortion – Stephen Schneider
Arguments about impact on the poor due to CO2 mitigation ring hollow, as usual, because of the choices we make on a daily basis to blow off massive amounts of hard cash on the useless and the pointless .
Shiny for us equals starvation for others but it’s a choice we make every single day. The expensive but useless layer of chrome applied to the plastic grille of an automobile is an opportunity cost. Can you eat hair gel? How many are fed with the $23 billion spent in the US per year on cosmetics? How does a 32″ television taste and what vitamins does it provide, compared to a 70″ set? What’s the nutritional value of $80/month for ESPN Sports Center?
Imagining just a little bit of restraint in our self-indulgence makes the costs of dealing with CO2 mitigation appear rather smaller.
Meanwhile, money doesn’t vanish; the person now operating the machinery to produce aerosol cheese could be employed doing something useful, such as installing solar energy collection apparatus.
Strip away the moralizing about hurting the poor with CO2 mitigation and it’s true that the argument here is still actually about the money vector. The trouble is, the desired outcome is antithetical to what the moralizers are talking about.
Re- Comment by Doug Bostrom — 25 Sep 2013 @ 3:18 PM
What you say is so inconvenient.
Steve
Another mismatch with an interesting possible explanation:
http://onlinelibrary.wiley.com/doi/10.1002/2013EO390007/abstract
Research Spotlight
Gravity waves could explain powerful thermospheric cooling
Colin Schultz
online: 24 SEP 2013
DOI: 10.1002/2013EO390007
Eos, Transactions American Geophysical Union
Volume 94, Issue 39, page 348, 24 September 2013
(This is about the Thermosphere — very thin up there, not many molecules involved — so I doubt a geoengineering control knob would be possible here.)
Very hollow indeed. Especially because we probably won’t live to see any negative consequences of our current use of fossil fuels – only our children and grandchildren will.
Having said that though, at least it’s rational. To say “let’s just continue to do what is best for us (the well off, not the poor …) and leave any negative consequences for our children and grandchildren to solve” is a perfectly rational point of view. Cold-hearted maybe, but rational.
The problem is not philosophical or even scientific it’s political, the graphs of the models were extensively used to form government policy they were widely circulated as hard predictions
Politicians in Australia and Germany have to take into account the brickbats thrown at them , this above discussion is ,for them ,vacuous in the extreme.
[Response: I strongly doubt that policy was made on the assumption that models are perfect (evidence?). Uncertainties have been discussed in respect to climate predictions since the beginning of the policy debate. – gavin]
In my comments I never talked about global warming and it’s effects. I talked only about the topic of this post, which is: the mismatch betweem model results and observations, and it’s implication for model uncertainty (since the mismatch cannot be attributed to observation errors). I also stated that the wide spread of model results further increases uncertainty.
Gavin implicitly agrees about model uncertainty and states that climate change is also proven by other lines of eivdence – like paleoclimatology. Fine.
Then Gavin says:
“The odd thing is that people seem to think if there is some huge unknown in climate that models don’t capture that this makes things better somehow. ”
Uncertainty is what it is. It’s not a matter of “better” or “worse”. (“Better” for what?)
There is a high degree of uncertainty in model results. We should be able to agree (or dissagree) on this, based on the model facts, without reference to our general opinion about global warming.
Jacob says:”Uncertainty is what it is. It’s not a matter of “better” or “worse”. (“Better” for what?)”
Uncertainty IS worse, because when you plan for the worst, and you think the worst is worse than it really is, you spend more effort than needed.
Jacob,
“There is a high degree of uncertainty…”
So you assert. Yet other than being alarmed by the word ‘mismatch’, I don’t see you backing that up or putting it into some proportional context — which is why I, for one, am starting to doubt that you have any idea about what level of uncertainty best characterizes what is known and how it is taken into account in research and literature reviews.
FUD: Fear, UNCERTAINTY, Doubt.
A well known set of rhetorical tactics used by fake AGW skeptics to paralyze sensible discussions, mischaracterize scientists, and generally poison the political atmosphere.
Perhaps if you asked some good, straightforward, specific questions and did a little less lecturing and maneuvering, the discussion might advance more smoothly.
Retrograde Orbit wrote: “we probably won’t live to see any negative consequences of our current use of fossil fuels – only our children and grandchildren will”
What in the world are you talking about?
We are already seeing negative consequences of our use of fossil fuels, on a massively destructive and costly scale, all over the world.
Which is not to say that “our children and grandchildren” will not experience far worse consequences.
#60 “[Response: Nice thought. If such principles can be found, they might indeed be useful. However, I am not optimistic – the specifics of the small scale physics (aerosol indirect effects on clouds, sea ice formation, soil hydrology etc.) are so heterogeneous that I don’t see how you can do without calculating the details. The main conservation principles (of energy, mass, momentum etc) are already incorporated, but beyond that I am not aware of anything of this sort. – gavin]”
Well, something must be there, because main conservation principles are unable to account for the observed extremely precise match between all sky hemispheric albedoes, while clear sky albedoes are vastly different. It must be an emergent phenomenon fuelled by phenomena on all scales. Whenever one runs into a symmetry of this kind, it suggests the possibility of simplification in theory, which is not utilized yet.
Journal of Climate, Volume 26, Issue 2 (January 2013)
The Observed Hemispheric Symmetry in Reflected Shortwave Irradiance
Aiko Voigt, Bjorn Stevens, Jürgen Bader and Thorsten Mauritsen