RealClimate logo


On mismatches between models and observations

Filed under: — gavin @ 13 September 2013

It is a truism that all models are wrong. Just as no map can capture the real landscape and no portrait the true self, numerical models by necessity have to contain approximations to the complexity of the real world and so can never be perfect replications of reality. Similarly, any specific observations are only partial reflections of what is actually happening and have multiple sources of error. It is therefore to be expected that there will be discrepancies between models and observations. However, why these arise and what one should conclude from them are interesting and more subtle than most people realise. Indeed, such discrepancies are the classic way we learn something new – and it often isn’t what people first thought of.

The first thing to note is that any climate model-observation mismatch can have multiple (non-exclusive) causes which (simply put) are:

  1. The observations are in error
  2. The models are in error
  3. The comparison is flawed

In climate science there have been multiple examples of each possibility and multiple ways in which each set of errors has arisen, and so we’ll take them in turn.

1. Observational Error

These errors can be straight-up mistakes in transcription, instrument failure, or data corruption etc., but these are generally easy to spot and so I won’t dwell on this class of error. More subtly, most of the “observations” that we compare climate models to are actually syntheses of large amounts of raw observations. These data products are not just a function of the raw observations, but also of the assumptions and the “model” (usually statistical) that go into building the synthesis. These assumptions can relate to space or time interpolation, corrections for non-climate related factors, or inversions of the raw data to get the relevant climate variable. Examples of these kinds of errors being responsible for a climate model/observation discrepancy range from the omission of orbital decay effects in producing the UAH MSU data sets, or the problems of no-modern analogs in the CLIMAP reconstruction of ice age ocean temperatures.

In other fields, these kinds of issues arise in unacknowledged laboratory effects or instrument calibration errors. Examples abound, most recently for instance, the supposed ‘observation’ of ‘faster-than-light’ neutrinos.

2. Model Error

There are of course many model errors. These range from the inability to resolve sub-grid features of the topography, approximations made for computational efficiency, the necessarily incomplete physical scope of the models and inevitable coding bugs. Sometimes model-observation discrepancies can be easily traced to such issues. However, more often, model output is a function of multiple aspects of a simulation, and so even if the model is undoubtedly biased (a good example is the persistent ‘double ITCZ’ bias in simulations of tropical rainfall) it can be hard to associate this with a specific conceptual or coding error. The most useful comparisons are then those that allow for the most direct assessment of the cause of any discrepancy.”Process-based” diagnostics – where comparisons are made for specific processes, rather than specific fields, are becoming very useful in this respect.

When a comparison is being made in a specific experiment though, there are a few additional considerations. Any particular simulation (and hence diagnostic from it) arises as a result from a collection of multiple assumptions – in the model physics itself, the forcings of the simulation (such as the history of aerosols in a 20th Century experiment), and the initial conditions used in the simulation. Each potential source of the mismatch needs to be independently examined.

3. Flawed Comparisons

Even with a near-perfect model and accurate observations, model-observation comparisons can show big discrepancies because the diagnostics being compared while similar in both cases, actually end up be subtly (and perhaps importantly) biased. This can be as simple as assuming an estimate of the global mean surface temperature anomaly is truly global when it in fact has large gaps in regions that are behaving anomalously. This can be dealt with by masking the model fields prior to averaging, but it isn’t always done. Other examples have involved assuming the MSU-TMT record can be compared to temperatures at a specific height in the model, instead of using the full weighting profile. Yet another might be comparing satellite retrievals of low clouds with the model averages, but forgetting that satellites can’t see low clouds if they are hiding behind upper level ones. In paleo-climate, simple transfer functions of proxies like isotopes can often be complicated by other influences on the proxy (e.g. Werner et al, 2000). It is therefore incumbent on the modellers to try and produce diagnostics that are commensurate with what the observations actually represent.

Flaws in comparisons can be more conceptual as well – for instance comparing the ensemble mean of a set of model runs to the single realisation of the real world. Or comparing a single run with its own weather to a short term observation. These are not wrong so much as potentially misleading – since it is obvious why there is going to be a discrepancy, albeit one that doesn’t have much implications for our understanding.

Implications

The implications of any specific discrepancy therefore aren’t immediately obvious (for those who like their philosophy a little more academic, this is basically a rephrasing of the Quine/Duhem position on scientific underdetermination). Since any actual model prediction depends on a collection of hypotheses together, as do the ‘observation’ and the comparison, there are multiple chances for errors to creep in. It takes work to figure out where though.

The alternative ‘Popperian’ view – well encapsulated by Richard Feynman:

… we compare the result of the computation to nature, with experiment or experience, compare it directly with observation, to see if it works. If it disagrees with experiment it is wrong.

actually doesn’t work except in the purest of circumstances (and I’m not even sure I can think of a clean example). A recent obvious counter-example in physics was the fact that the ‘faster-than-light’ neutrino experiment has not falsified special relativity – despite Feynman’s dictum.

But does this exposition help in any current issues related to climate science? I think it does – mainly because it forces one to think about the other ancillary hypotheses are. For three particular mismatches – sea ice loss rates being much too low in CMIP3, tropical MSU-TMT rising too fast in CMIP5, or the ensemble mean global mean temperatures diverging from HadCRUT4 – it is likely that there are multiple sources of these mismatches across all three categories described above. The sea ice loss rate seems to be very sensitive to model resolution and has improved in CMIP5 – implicating aspects of the model structure as the main source of the problem. MSU-TMT trends have a lot of structural uncertainty in the observations (note the differences in trends between the UAH and RSS products). And global mean temperature trends are quite sensitive to observational products, masking, forcings in the models, and initial condition sensitivity.

Working out what is responsible for what is, as they say, an “active research question”.

Update: From the comments:

“our earth is a globe
whose surface we probe
no map can replace her
but just try to trace her”
– Steve Waterman, The World of Maps


References

  1. M. Werner, U. Mikolajewicz, M. Heimann, and G. Hoffmann, "Borehole versus isotope temperatures on Greenland: Seasonality does matter", Geophysical Research Letters, vol. 27, pp. 723-726, 2000. http://dx.doi.org/10.1029/1999GL006075

167 Responses to “On mismatches between models and observations”

  1. 151
    Bradley McKinley says:

    Gavin-

    You say:

    You are setting an special (and probably impossible) bar for this issue that is never set for any other decision. Are the consequences of current economic policies (which have a much bigger impact on the poor) being worked out at the ‘level of precision’ you are demanding? The answer is obviously not.

    I’m not sure I understand the basis for this claim. Cost benefit analysis is a standard requirement whenever a commitment of resources of this magnitude is contemplated. There is simply no other way for governments to determine how to best allocate the resources at their disposal. The fact that numerous peer reviewed studies have attempted to quantify impacts (e.g. Yohe et al. [2007]) demonstrates that even if you do not understand the need to conduct this type of analysis, others do.

    [Response: Ha. I’m actually well aware of the literature on cost-benefit studies and that most (even from Nordhaus) suggest that action is warranted. But the point I am making is somewhat different; namely, I do not think that cost-benefit analyses are anything but a rough guide to different pathways. Damage functions in these analyses are simplistic and mostly untested for the magnitudes of change projected, their ability to balance economic costs and social costs involve ethical judgements that are both opaque and not universal. The other point is that government decisions are made all the time with huge consequences with no such modeling being done – was the TARP bail-out assessed for it’s impact on the African poor? Is the EU CAP? etc. So while I am all for policy-specific impacts modeling, I recognise it’s limitations. – gavin]

    Additionally, I am unclear as to where your confidence in exactly what impact climate policies will have on the poor comes from. It is easy to imagine policies that price carbon and use the funds to reduce regressive taxation that is a huge burden.

    The confidence comes from the fact that third world countries have consistently voiced their objections to certain mitigation efforts such as carbon taxes on grounds that they, not the industrial world, would bear the brunt of the burden.

    [Response: Where does this come from? Most actions being contemplated are related to western use of fossil fuels for western consumption of transport and electricity. Impacts of a carbon price in the US say, are not going to be predominantly felt in Somalia. – gavin]

    Similarly, the poverty that you rightly condemn in Africa and other areas is far more associated the very endemic institutional issues than it is with energy poverty. That is a symptom, not a (primary) cause. But even there, there are large parts of rural Africa where local solar (for instance) is already cheaper than extending the grid to supply fossil energy.

    I agree that energy poverty is a symptom of the institutional issues. However, you cannot deny that anything that makes energy more expensive will have severely negative impacts on the third world. If local solar is indeed cheaper than fossil fuels, there is no need for carbon taxes to make fossil fuels more expensive–market forces will ensure the switch all by themselves.

    [Response: Not true. The biggest handicap in rural poverty is lack of access to capital – even if new projects are enormously beneficial on even a standard cost/benefit analysis, they still do not get done. The idea that market forces automatically do the best thing in these cases is a fallacy. – gavin]

    The idea that climate must be balanced directly and exclusively against the global poor is a false dichotomy. And none of this has anything to do with models.

    I never claimed that climate must be balanced directly and exclusively against the global poor.

    [Response: Good. – gavin]

  2. 152
    Mal Adapted says:

    Tamino’s on a roll. Referring to the 15-year span from 1992 – 2006 when the rate of surface warming was greater than the long-term trend,

    Does this mean that global warming is wrong? That the computer models are utter junk? That this whole climate science thing is just a hoax, a nefarious scheme to cheat us all out of tax dollars in order to support the lifestyle of gaudy luxury that we all know scientists wallow in? (Science: money for nothin’ and your chicks for free…)

    How did these evil denizens of global warming react? Did they use that result to push world government based on soshalism [re-spelled to get past the spam filter; he spells it correctly], so that they could destroy our economy by taxing the super-rich out of some of their hardly-earned riches? Did they run screaming through the streets yelling about how we’re all going to suffer spontaneous combustion by the year 2100?

    “Hardly-earned riches”. I am so stealing that.

  3. 153
    prokaryotes says:

    Bradley McKinley #148

    …it is not enough to simply say “we know that increasing CO2 levels will make the world warmer and that is a bad thing”, you need to be able to specify how much warmer and quantify the impacts with some level of precision.

    We can quantify past impacts based on the sea level feedback (cryosphere response). This tells us that with the current climate forcing we established at least a Pliocene climate response, which was triggered by anthropogenic induced energy imbalance. And that is a very bad thing, because paleoclimate data tells us that the Pliocene regime 5 mil years ago had 15 meter higher sea level. Further do we know from proxy records (corals) that the slow ice sheet feedback includes non-linear episodes, when ice sheets disintegrate and abruply cause SLR.

    James Hansen explains Earth energy imbalance

    Earth’s energy balance
    In response to a positive radiative forcing F (see Appendix A), such as characterizes the present-day anthropogenic perturbation (Forsteret al., 2007), the planet must increase its net energy loss to space in order to re-establish energy balance (with net energy loss being the difference between the outgoing long-wave (LW) radiation and net incoming shortwave (SW) radiation at the top-of-atmosphere (TOA)). Assuming that this increased energy loss is proportional to the surface temperature change T, we can write F = λT + Q (1) where λ is the climate feedback parameter.

    Complete restoration of the planetary energy balance (and thus full adjustment of the surface temperature) does not occur instantaneously due to the inherent inertia of the system, which lies mainly in the slow response times of the oceans and cryosphere. Therefore, prior to achieving a new equilibrium state, there will be an imbalance, Q, between radiative forcing and climate response. This imbalance represents the net heat flux into the system, with nearly all of this heat flux at present going into the ocean (Levitus et al., 2005).

    Link

  4. 154
    Mal Adapted says:

    bjchip:

    Have I understood correctly that the models do not predict the ENSO phenomena but simply place it randomly, and that when it is placed correctly the model works THAT well?

    That’s what I’ve gathered from KX13 and discussions of it (especially John Nielsen-Gammon’s), and I’ve been using it against a couple of “15-year pause” parrots elsewhere. I’m eager to read RC’s post on it, even if I find out I’ve got it backwards 8^}.

  5. 155
    Jeffrey Davis says:

    re: 151 and Cost Benefit Analysis

    I remember the famous 2% Doctrine of the Bush era, i.e. if there were a 2% chance of WMDs, we had to intervene.

    And then there’s Luke 19:22.

  6. 156
    SecularAnimist says:

    Gavin wrote: “… there are large parts of rural Africa where local solar (for instance) is already cheaper than extending the grid to supply fossil energy.”

    Bradley McKinley: “If local solar is indeed cheaper than fossil fuels, there is no need for carbon taxes to make fossil fuels more expensive – market forces will ensure the switch all by themselves.”

    Carbon taxes have been proposed in developed countries as a mechanism to internalize the environmental and public health costs of burning fossil fuels, which are currently externalized and foisted off on the public. As such, carbon taxes ARE a “market force”. They simply “correct” the market by forcing it to reflect the true cost of fossil fuels. Failure to do that amounts to a massive public subsidy to fossil fuels, which is why they appear to be (but actually are not) “cheaper” than renewable energy.

    None of which has anything to do with poor people in rural Africa that Gavin wrote about — people who use little or no fossil fuels, and who would therefore not pay any carbon taxes, even if someone were to impose such taxes in rural Africa, which no one has suggested doing.

    The reality is that many of those people, and millions of others like them throughout the developing world who have NO access to electricity, will NEVER have access to fossil-fuel-fired electricity because no one is ever going to build the centralized power plants and the grid to deliver electricity to them. The ONLY way they will ever have electricity is with distributed renewable energy, principally photovoltaics — which are vastly less expensive than building a whole new grid-based electricity generation and distribution infrastructure.

    And in fact, although it doesn’t get a lot of media attention, there is an ongoing revolution in rural solar electrification in the developing world, where low-cost systems as simple as a few solar panels and batteries are providing power for electric lights, refrigeration, medical equipment, radios, cell phones, satellite TV and Internet access, and more, to entire villages.

  7. 157
  8. 158
    Doug Bostrom says:

    Arguments about impact on the poor due to CO2 mitigation ring hollow, as usual, because of the choices we make on a daily basis to blow off massive amounts of hard cash on the useless and the pointless .

    Shiny for us equals starvation for others but it’s a choice we make every single day. The expensive but useless layer of chrome applied to the plastic grille of an automobile is an opportunity cost. Can you eat hair gel? How many are fed with the $23 billion spent in the US per year on cosmetics? How does a 32″ television taste and what vitamins does it provide, compared to a 70″ set? What’s the nutritional value of $80/month for ESPN Sports Center?

    Imagining just a little bit of restraint in our self-indulgence makes the costs of dealing with CO2 mitigation appear rather smaller.

    Meanwhile, money doesn’t vanish; the person now operating the machinery to produce aerosol cheese could be employed doing something useful, such as installing solar energy collection apparatus.

    Strip away the moralizing about hurting the poor with CO2 mitigation and it’s true that the argument here is still actually about the money vector. The trouble is, the desired outcome is antithetical to what the moralizers are talking about.

  9. 159
    Steve Fish says:

    Re- Comment by Doug Bostrom — 25 Sep 2013 @ 3:18 PM

    What you say is so inconvenient.

    Steve

  10. 160
    Hank Roberts says:

    Another mismatch with an interesting possible explanation:
    http://onlinelibrary.wiley.com/doi/10.1002/2013EO390007/abstract

    Research Spotlight
    Gravity waves could explain powerful thermospheric cooling
    Colin Schultz
    online: 24 SEP 2013
    DOI: 10.1002/2013EO390007
    Eos, Transactions American Geophysical Union
    Volume 94, Issue 39, page 348, 24 September 2013

    For the past few decades the upper reaches of Earth’s atmosphere have been cooling much faster than researchers anticipated. While the rising atmospheric concentration of carbon dioxide is heating the air near the ground, that same increase is expected to cool the thermosphere—the atmospheric band that stretches from around 80 kilometers altitude to the exosphere at 500 kilometers—by emitting heat into space. However, while carbon dioxide should theoretically cool the thermosphere by around 2 kelvins per decade, the observed cooling was around 10 times this rate. Building on recent theoretical and modeling work, Oliver et al. lay out a mechanism that could explain the observed cooling.

    (This is about the Thermosphere — very thin up there, not many molecules involved — so I doubt a geoengineering control knob would be possible here.)

  11. 161
    Retrograde Orbit says:

    Very hollow indeed. Especially because we probably won’t live to see any negative consequences of our current use of fossil fuels – only our children and grandchildren will.
    Having said that though, at least it’s rational. To say “let’s just continue to do what is best for us (the well off, not the poor …) and leave any negative consequences for our children and grandchildren to solve” is a perfectly rational point of view. Cold-hearted maybe, but rational.

  12. 162
    Jeannick says:

    The problem is not philosophical or even scientific it’s political, the graphs of the models were extensively used to form government policy they were widely circulated as hard predictions

    Politicians in Australia and Germany have to take into account the brickbats thrown at them , this above discussion is ,for them ,vacuous in the extreme.

    [Response: I strongly doubt that policy was made on the assumption that models are perfect (evidence?). Uncertainties have been discussed in respect to climate predictions since the beginning of the policy debate. – gavin]

  13. 163
    Jacob says:

    In my comments I never talked about global warming and it’s effects. I talked only about the topic of this post, which is: the mismatch betweem model results and observations, and it’s implication for model uncertainty (since the mismatch cannot be attributed to observation errors). I also stated that the wide spread of model results further increases uncertainty.

    Gavin implicitly agrees about model uncertainty and states that climate change is also proven by other lines of eivdence – like paleoclimatology. Fine.

    Then Gavin says:
    “The odd thing is that people seem to think if there is some huge unknown in climate that models don’t capture that this makes things better somehow. ”

    Uncertainty is what it is. It’s not a matter of “better” or “worse”. (“Better” for what?)
    There is a high degree of uncertainty in model results. We should be able to agree (or dissagree) on this, based on the model facts, without reference to our general opinion about global warming.

  14. 164
    t_p_hamilton says:

    Jacob says:”Uncertainty is what it is. It’s not a matter of “better” or “worse”. (“Better” for what?)”

    Uncertainty IS worse, because when you plan for the worst, and you think the worst is worse than it really is, you spend more effort than needed.

  15. 165
    Radge Havers says:

    Jacob,

    “There is a high degree of uncertainty…”

    So you assert. Yet other than being alarmed by the word ‘mismatch’, I don’t see you backing that up or putting it into some proportional context — which is why I, for one, am starting to doubt that you have any idea about what level of uncertainty best characterizes what is known and how it is taken into account in research and literature reviews.

    FUD: Fear, UNCERTAINTY, Doubt.
    A well known set of rhetorical tactics used by fake AGW skeptics to paralyze sensible discussions, mischaracterize scientists, and generally poison the political atmosphere.

    Perhaps if you asked some good, straightforward, specific questions and did a little less lecturing and maneuvering, the discussion might advance more smoothly.

  16. 166
    SecularAnimist says:

    Retrograde Orbit wrote: “we probably won’t live to see any negative consequences of our current use of fossil fuels – only our children and grandchildren will”

    What in the world are you talking about?

    We are already seeing negative consequences of our use of fossil fuels, on a massively destructive and costly scale, all over the world.

    Which is not to say that “our children and grandchildren” will not experience far worse consequences.

  17. 167

    #60 “[Response: Nice thought. If such principles can be found, they might indeed be useful. However, I am not optimistic – the specifics of the small scale physics (aerosol indirect effects on clouds, sea ice formation, soil hydrology etc.) are so heterogeneous that I don’t see how you can do without calculating the details. The main conservation principles (of energy, mass, momentum etc) are already incorporated, but beyond that I am not aware of anything of this sort. – gavin]”

    Well, something must be there, because main conservation principles are unable to account for the observed extremely precise match between all sky hemispheric albedoes, while clear sky albedoes are vastly different. It must be an emergent phenomenon fuelled by phenomena on all scales. Whenever one runs into a symmetry of this kind, it suggests the possibility of simplification in theory, which is not utilized yet.

    Journal of Climate, Volume 26, Issue 2 (January 2013)
    The Observed Hemispheric Symmetry in Reflected Shortwave Irradiance
    Aiko Voigt, Bjorn Stevens, Jürgen Bader and Thorsten Mauritsen

    “Climate models generally do not reproduce the observed hemispheric symmetry, which the authors interpret as further evidence that the symmetry is nontrivial.”


Switch to our mobile site