{"id":21529,"date":"2018-07-04T10:48:09","date_gmt":"2018-07-04T15:48:09","guid":{"rendered":"http:\/\/www.realclimate.org\/?p=21529"},"modified":"2018-07-04T10:48:09","modified_gmt":"2018-07-04T15:48:09","slug":"model-independence-day","status":"publish","type":"post","link":"https:\/\/www.realclimate.org\/index.php\/archives\/2018\/07\/model-independence-day\/","title":{"rendered":"Model Independence Day"},"content":{"rendered":"
\n

\nWe hold these truths to be self-evident, that all models are created equal, that they are endowed by their Creators with certain unalienable Rights, that among these are a DOI, Runability and Inclusion in the CMIP ensemble mean.<\/i>\n<\/p><\/blockquote>\n

Well, not quite. But it is Independence Day in the US, and coincidentally there is a new discussion paper (Abramowitz et al)<\/a><\/span> (direct link<\/a>) posted on model independence just posted at Earth System Dynamics. <\/p>\n

<\/p>\n

What does anyone mean by model independence<\/em>? In the international coordinated efforts to assess climate model skill (such as the Coupled Model Intercomparison Project<\/a>), multiple groups from around the world submit their model results from specified experiments to a joint archive. The basic idea is that if different models from different groups agree on a result, then that result is likely to be robust based on the (shared) fundamental understanding of the climate system despite the structural uncertainty in modeling the climate. But there are two very obvious ways in which this ideal is not met in practice.<\/p>\n

First, if the models are actually the same, then it’s totally unsurprising that a result might be common between them. One of the two models would be redundant and add nothing to our knowledge of structural uncertainties. <\/p>\n

Second, the models might well be totally independent in formulation, history and usage, but the two models share a common, but fallacious, assumption about the real world. Then a common result might reflect that shared error, and not reflect anything about the real world at all. <\/p>\n

These two issues are also closely tied to the problem of model selection. Given an ensemble of models, that have varied levels of skill across any number of metrics, is there a subset or weighting of models that could be expected to give the most skillful predictions? And if so, how would you demonstrate that?<\/p>\n

These problems have been considered (within the climate realm) since the beginnings of the “MIP” process in the 1990s, but they are (perhaps surprisingly) very tough to deal with. <\/p>\n

Ensemble Skill<\/b><\/p>\n

One of the most interesting things about the MIP ensembles is that the mean of all the models generally has higher skill than any individual model. This is illustrated in the graphic from Reichler and Kim, (2008)<\/a><\/span>. Each dot is a model, with the ensemble mean in black, and an RMS score (across a range of metrics) increasing left to right, so that the most skillful models or means are those furthest to the left.<\/p>\n

<\/p>\n

But as Reto Knutti and colleagues have showed, the increase in skill of the ensemble mean doesn’t keep increasing as you add more models. After you’ve averaged 10 or 15 models, the skill no longer improves. This is not what you would expect if every model result was an unbiased independent estimate of the true climate. But since the models are neither unbiased, nor independent, the fact that there is any increase in skill after averaging is more surprising!<\/p>\n