RealClimate logo


Uncertainty, noise and the art of model-data comparison

Filed under: — gavin @ 11 January 2008 - (Español) (English)

There is a chinese translation available here.


624 Responses to “Uncertainty, noise and the art of model-data comparison”

  1. 601

    Thank you, Timothy Chase. I understand (now) the search facility on this site and will employ it in the future.

    I’m new to this stuff and have several colleagues (interestingly, all of conservative tendency) who keep suggesting that AGW is on a shaky foundation. So be patient with my naive questions (not that you are otherwise).

    Burgy (www.burgy.50megs.com)

  2. 602
    Dave Occam says:

    I have been tracking the measured annual global mean surface temperature versus the IPCC trend predictions, 1995 and 2000.
    While the IPCC made no annual predictions, I see no reason one can not construct annual bounds based on their trendline predictions to facilitate gauging the performance of their predictions to date.

    To do this I extended their prediction, uppper and lower bounds, by 1.28 std. deviations (90% confidence limit) to account for natural interannual variation. The variance was calculated as twice the variance in the residuals of a linear regression on the annual surface temperature data: 1970-2000. This is to account for the natural variability one can expect between any year of interest and an arbitray reference year: 1990 in this case.

    A link to the graph:

    http://i161.photobucket.com/albums/t231/Occam_bucket/IPCCTempPredictions.jpg

  3. 603
    David B. Benson says:

    Dave Occam (602) — Thank you!

  4. 604
    MikeN says:

    Aren’t you lumping all models together, than saying a short trend doesn’t invalidate model X, therefore all models are still plausible?

    For example, I would think a small downward trend or flatline over an 8 year period, should make the forecasts of larger warming like 6C less likely than the 2C scenario.

  5. 605
    Martin Vermeer says:

    Dave Occam #602, is that right? A two-sided 90% confidence interval is +/- 1.645 sigma for the normal distribution.

  6. 606
    David B. Benson says:

    MikeN (604) — The major uncertainty is human behavior; are we or are we not going to add lots of extra carbon dioxide to the atmosphere?

  7. 607
    Mark says:

    “Aren’t you lumping all models together, than saying a short trend doesn’t invalidate model X, therefore all models are still plausible?”

    Nope, a short trend doesn’t invalidate model X therefore it doesn’t prove model X is wrong.

    There is no flatline or downward trend.

    It was going up before 1998.

    It has gone up after 1998.

    And when in the 80s the AGW theory was just beginning and well into the 90s, it was all “you haven’t got enough data to say there’s even been any warming”.

    That was 80 years of data.

    Now 8 years is enough???

  8. 608
    Dave Occam says:

    MarkN (604),

    The 90% confidence limit does not apply to the probability of annual temperatures falling within the two bounds but rather the confidence limits of the best and worst case scenarios only. You can’t put a confidence limit on the aggregate because none were supplied by the TAR or SAR for their trendline prediction.

    1.281 std devs leaves 10% (one tail) of the population outside the limit of interest if temperatures are tracking one of the envelope trendline bounds. Only one tail is relevant for each bound. If for example temperatures are running close to the lower bound then they are testing the slowest temperature rise model/scenario(s); the upper range of natural variability would fall well within the upper bounds. So while in this example you might have 10% of the years falling below my lower confidence limit it’s highly unlikely you will have any above the upper limit.

  9. 609
    Dave Occam says:

    Martin Vermeer (605),

    Unfortunately, I thought quantifying the confidence limit might cause some confusion.

    The 90% confidence limit does not apply to the probability of annual temperatures falling within the two bounds but rather the confidence limits apply to the best and worst case scenarios only. You can’t put a confidence limit on the aggregate because none were supplied by the TAR or SAR for their trendline prediction.

    1.281 std devs leaves 10% (one tail) of the population outside the limit of interest if temperatures are tracking one of the envelope trendline bounds. Only one tail is relevant for each bound. If for example temperatures are running close to the lower bound then they are testing the slowest temperature rise model/scenario(s); the upper range of natural variability would fall well within the upper bounds. So while in this example you might have 10% of the years falling below my lower confidence limit it’s highly unlikely you will have any above the upper limit.

  10. 610

    Dave OK, thanks.

  11. 611
    MikeN says:

    Dave, so were you agreeing with me that a 10 year flatline, if it happens, would have to lower the confidence ranges for temperature increase by 2100? Don’t want get confused by terminology. I’m saying that the 6.4C upper range temperature is less valid given a flatline. Not debunked or invalidated, but less likely than it already was, since a flatline is more likely under a 2.4C scenario than a 6.4C scenario.

  12. 612
    Mark says:

    MikeN #611. Not if you find the reason for the flatline is something that isn’t going to continue.

    Without that, the most likely candidate is that the errors will get WIDER.

    And note: it isn’t flatlining.

    But even if it were, if that reason turned out to be that the ocean chemistry was changing into a new mode and about to exhaust all the stored methane in it, the temperature by 2100 would be higher than predicted.

    If it were, and that reason were that the sun is reducing its output and will for the next thousand years, then the temperature by 2100 would be lower than predicted.

    But since either could be happening, if it WERE flatlining (which it isn’t), we don’t know why and so our uncertainty would be higher. We’d have, in the words of Rummy, a known unknown. And each of those increases the uncertainty, not the trend.

  13. 613
    MikeN says:

    Well going off the post at the top which shows various trends over short periods as part of general ‘weather’ variability.
    I’m assuming a flatline comes from this and not some external forcing not in the models.

    In that case, then I think basic math would suggest that the lower end of warming is more likely than the higher end. A flatline should be more likely in a 2C model than a 6C model correct?

  14. 614

    Hi MikeN. :-)

    This question has come up in another discussion, and both Mike and I would love to get a more informed third opinion. If I may rephrase the question I think Mike is asking…

    What does the frequency of “lulls” tell us, if anything, about the climate sensitivity? Sensitivity estimates for climate range from about 1.5C to 4.5C per 2xCO2, or (equivalently) from about 0.4 to 1.2C per unit forcing (W/m^2).

    Should different sensitivities result in a different frequency of “lulls”? Can we use information about the range of variation in the short term 8-year slope to help constrain the sensitivity estimates?

  15. 615
    Mark says:

    You’d have to know how the chaotic system reacts.

    Short answer: no.

    Long answer, theoretically and in general.

    An example is to take the variations in temperature and go “how much of that is matched out by solar variation”. you then subtract a variation at that frequency until you get a line that is more straight than before.

    And do that with all other variables.

    Though that gives more the attribution rather than the sensitivity. And is subject to a lot of error.

    But that’s not looking at the lulls either.

  16. 616
    Dave Occam says:

    “Dave, so were you agreeing with me that a 10 year flatline, if it happens, would have to lower the confidence ranges for temperature increase by 2100?”

    Are you addressing me, MikeN? I accidentally double posted – I meant to direct my post to Martin Vermeer.

    In case you were asking my opinion: Statistically speaking, I would not look at a select 10 year period, a selection already biases the result, but rather the full record from the date of the prediction/s. I would consider the prediction suspect if we got a couple years outside my bounds over the next decade (exempting years of a major episodic event).

    I think if temperatures hug the lower bound over the next few years it would make the upper 2030 bound an unlikely occurrence, but not necessarily the upper 2100 bound. Some models and scenarios produce very slow increases in temperatures in the first couple decades but accelerate more in the later years. For example scenario A2 of the 6 SRES scenarios (Fig 9.15 of my reference) produces the smallest increase in temperature in 2030 but the second highest in 2100 – for all models.

    So far the IPCC predictions are holding their own. Hypothetically speaking, if they should require modification in the future to levels outside current bounds then I don’t think we can say at this point if it is because of inaccurate models, forcing sensitivities, initial conditions or natural changes that were not anticipated. So we couldn’t say much about 2100 temperatures till that was resolved.

  17. 617
    MikeN says:

    >I think if temperatures hug the lower bound over the next few years it would make the upper 2030 bound an unlikely occurrence, but not necessarily the upper 2100 bound.

    OK, not necessarily unlikely, but wouldn’t it then make the lower 2100 bound more likely than the upper 2100 bound?

    Duae has hypothesized that a high positive feedback model that produces a 6C warming is as likely to produce lulls as a low feedback model that produces warming of 2C. Any opinion on this?

    By the way, I don’t wish to compare models of different carbon scenarios, but rather models with different feedback variables.

  18. 618
    Dave Occam says:

    Re 617: “By the way, I don’t wish to compare models of different carbon scenarios, but rather models with different feedback variables.”

    But the IPCC prediction envelope encompasses both; when the authors determined the bounds of this envelope they had both in mind. The bounds of their prediction were based on expert judgment, not statistical computation.

    If you want to tease out more information than what they summarized you need to address specific models and/or specific scenarios – but then how do you choose – statistically speaking? And then adding confidence limits for a chosen model is problematic – for experts not to mention lay readers.

    Re 617: “OK, not necessarily unlikely, but wouldn’t it then make the lower 2100 bound more likely than the upper 2100 bound?”

    Short answer, I am not qualified to say, but IMO I still think not necessarily – despite the intuitive attractiveness of your assumption. It all depends on how you determine likelihood and how well the causes of the slower than expected warming are understood in the future of your hypothetical scenario.

    But we are just splitting hairs. By the time we reach 2030 they will have greater understanding of climate, better data and models and be able to narrow the range of plausible outcomes. However there will always be some natural variation that is beyond prediction. At this point in time I don’t see that it matters in terms of decisions that have to be made today regarding policy around GHG emissions that affect resource spending prior to 2030.

  19. 619
    Mark says:

    “but wouldn’t it then make the lower 2100 bound more likely than the upper 2100 bound?”

    It doesn’t MAKE the upper bound less likely than before.

    You can make a ***guess*** that it should, but unless you know why your model and reality are disagreeing, you don’t ***know*** that your assumption will hold.

  20. 620
    MikeN says:

    That’s the thing. It isn’t one model, but a range of model possibilities.

  21. 621
    Dave Occam says:

    Re 620: “That’s the thing. It isn’t one model, but a range of model possibilities.”

    Right. As I understand it, capturing the results from a range of models is a proxy for capturing all the uncertainties in modeling. This might seem like they are sandbagging by playing it so safe, but I believe the models are using a nominal value for climate sensitivity to various forcings. If they were to choose only one model then it would make sense to capture the uncertainty in the sensitivities by running them with a range of values. Probably how it will be done in the future, after they get confidence in more evolved models and when they can do a better job of independently bounding the sensitivities.

  22. 622
    Dave Occam says:

    Re: #602

    I have updated the data and added near term statistical projections to the plots; i.e. projections free of any climate specific assumption.

    http://i161.photobucket.com/albums/t231/Occam_bucket/IPCCStatTempPrediction.jpg

  23. 623
    Dave Occam says:

    Captured in these four plots we see that the global annual mean surface temperature, through 2008, is tracking within International Panel on Climate Change projections.

    [IMG]http://i161.photobucket.com/albums/t231/Occam_bucket/IPCCTemperatureProjection.png[/IMG]

    The top chart shows actual mean temperatures relative to 1990 and annual bounds derived from IPCC’s multi-decadal trend bounds – values on left scale. These bounds capture uncertainty in the model, emissions scenario, and inter-annual noise. Also a linear trend-line is plotted from the measured temperature points and extrapolated out to 2030 so we can see the projected 40 year change – right scale, based on data through 2008.

    The next two charts are the associated multi-decadal temperature trend projections published by the IPCC in the Third Assessment Report, based on climate models run in the late 1990s. As it turned out, per the last chart in the lower right (published in the Copenhagen Climate Report – 2009) emissions have so far closely followed SRES A1F1. I have circled the point in the IPCC chart where model ECHAM4/OPYC predicts the 40 yr temperature change given emission scenario A1F1. You can see it (0.72C) falls right on top of the actual trend to date.

    Of course 19 years of data do not definitively define a 40 year trend, but we now have enough data to get a meaningful indication of how these 1990s models are tracking reality. So far actual temperatures are closely following median model projections.

    It is disturbing to note that should these models continue to hold true and we continue on the emissions path of scenario A1F1 we are facing a global temperature increase of around 4.5C or 8.1F before the end of the century.

  24. 624

Switch to our mobile site