Journal list menu
Model averaging in ecology: a review of Bayesian, information-theoretic, and tactical approaches for predictive inference
Corresponding Author
Carsten F. Dormann
Biometry and Environmental System Analysis, University of Freiburg, Tennenbacher Str. 4, 79106 Freiburg, Germany
E-mail: [email protected]Search for more papers by this authorJustin M. Calabrese
Conservation Ecology Center, Smithsonian Conservation Biology Institute, 1500 Remount Road, Front Royal, Virginia, 22630 USA
Search for more papers by this authorGurutzeta Guillera-Arroita
School of BioSciences, University of Melbourne, Royal Parade, Parkville, Melbourne, Victoria, 3052 Australia
Search for more papers by this authorEleni Matechou
School of Mathematics, Statistics and Actuarial Science, University of Kent, Parkwood Road, Canterbury, CT2 7FS UK
Search for more papers by this authorVolker Bahn
Department of Biological Sciences, Wright State University, 3640 Colonel Glenn Hwy., Dayton, Ohio, 45435 USA
Search for more papers by this authorKamil Bartoń
Institute of Nature Conservation, Polish Academy of Sciences, al. A. Mickiewicza 33, 31-120 Kraków, Poland
Search for more papers by this authorColin M. Beale
Department of Biology, University of York, Wentworth Way, York, YO10 5DD UK
Search for more papers by this authorSimone Ciuti
Biometry and Environmental System Analysis, University of Freiburg, Tennenbacher Str. 4, 79106 Freiburg, Germany
Laboratory of Wildlife Ecology and Behaviour, School of Biology and Environmental Science, University College Dublin, Belfield D4, Dublin, Ireland
Search for more papers by this authorJane Elith
School of BioSciences, University of Melbourne, Royal Parade, Parkville, Melbourne, Victoria, 3052 Australia
Search for more papers by this authorKatharina Gerstner
Computational Landscape Ecology, Helmholtz Centre for Environmental Research-UFZ, Permoser Str. 15, 04318 Leipzig, Germany
German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig, Deutscher Platz 5E, 04103 Leipzig, Germany
Search for more papers by this authorJérôme Guelat
Swiss Ornithological Institute, Seerose 1, 6204 Sempach, Switzerland
Search for more papers by this authorPetr Keil
German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig, Deutscher Platz 5E, 04103 Leipzig, Germany
Search for more papers by this authorJosé J. Lahoz-Monfort
School of BioSciences, University of Melbourne, Royal Parade, Parkville, Melbourne, Victoria, 3052 Australia
Search for more papers by this authorLaura J. Pollock
Univ. Grenoble Alpes, CNRS, Univ. Savoie Mont Blanc, Laboratoire d'Ecologie Alpine (LECA), Grenoble, 38000 France
Search for more papers by this authorBjörn Reineking
University Grenoble Alpes, Irstea, UR LESSEM, F-38402 St-Martin-d'Hères, Grenoble, France
Biogeographical Modelling, Bayreuth Center of Ecology and Environmental Research BayCEER, University of Bayreuth, Dr. Hans-Frisch-Straße 1-3, 95448 Bayreuth, Germany
Search for more papers by this authorDavid R. Roberts
Biometry and Environmental System Analysis, University of Freiburg, Tennenbacher Str. 4, 79106 Freiburg, Germany
Department of Geography, University of Calgary, 2500 University Dr. NW, Calgary, Alberta, T2N 1N4 Canada
Search for more papers by this authorBoris Schröder
Landscape Ecology and Environmental Systems Analysis, Institute of Geoecology, Technische Universität Braunschweig, Langer Kamp 19c, 38106 Braunschweig, Germany
Berlin-Brandenburg Institute of Advanced Biodiversity Research (BBIB), Altensteinstr. 34, 14195 Berlin, Germany
Search for more papers by this authorWilfried Thuiller
Univ. Grenoble Alpes, CNRS, Univ. Savoie Mont Blanc, Laboratoire d'Ecologie Alpine (LECA), Grenoble, 38000 France
Search for more papers by this authorDavid I. Warton
School of Mathematics and Statistics, Evolution and Ecology Research Centre, University of New South Wales, Sydney, New South Wales, 2052 Australia
Search for more papers by this authorBrendan A. Wintle
School of BioSciences, University of Melbourne, Royal Parade, Parkville, Melbourne, Victoria, 3052 Australia
Search for more papers by this authorSimon N. Wood
School of Mathematics, Bristol University, Tyndall Avenue, Bristol, BS8 1TW UK
Search for more papers by this authorRafael O. Wüest
Univ. Grenoble Alpes, CNRS, Univ. Savoie Mont Blanc, Laboratoire d'Ecologie Alpine (LECA), Grenoble, 38000 France
Swiss Federal Institute for Forest, Snow and Landscape Research WSL, Zürcherstrasse 111, 8903 Birmensdorf, Switzerland
Search for more papers by this authorFlorian Hartig
Biometry and Environmental System Analysis, University of Freiburg, Tennenbacher Str. 4, 79106 Freiburg, Germany
Theoretical Ecology, University of Regensburg, Universitätsstr. 31, 93053 Regensburg, Germany
Search for more papers by this authorCorresponding Author
Carsten F. Dormann
Biometry and Environmental System Analysis, University of Freiburg, Tennenbacher Str. 4, 79106 Freiburg, Germany
E-mail: [email protected]Search for more papers by this authorJustin M. Calabrese
Conservation Ecology Center, Smithsonian Conservation Biology Institute, 1500 Remount Road, Front Royal, Virginia, 22630 USA
Search for more papers by this authorGurutzeta Guillera-Arroita
School of BioSciences, University of Melbourne, Royal Parade, Parkville, Melbourne, Victoria, 3052 Australia
Search for more papers by this authorEleni Matechou
School of Mathematics, Statistics and Actuarial Science, University of Kent, Parkwood Road, Canterbury, CT2 7FS UK
Search for more papers by this authorVolker Bahn
Department of Biological Sciences, Wright State University, 3640 Colonel Glenn Hwy., Dayton, Ohio, 45435 USA
Search for more papers by this authorKamil Bartoń
Institute of Nature Conservation, Polish Academy of Sciences, al. A. Mickiewicza 33, 31-120 Kraków, Poland
Search for more papers by this authorColin M. Beale
Department of Biology, University of York, Wentworth Way, York, YO10 5DD UK
Search for more papers by this authorSimone Ciuti
Biometry and Environmental System Analysis, University of Freiburg, Tennenbacher Str. 4, 79106 Freiburg, Germany
Laboratory of Wildlife Ecology and Behaviour, School of Biology and Environmental Science, University College Dublin, Belfield D4, Dublin, Ireland
Search for more papers by this authorJane Elith
School of BioSciences, University of Melbourne, Royal Parade, Parkville, Melbourne, Victoria, 3052 Australia
Search for more papers by this authorKatharina Gerstner
Computational Landscape Ecology, Helmholtz Centre for Environmental Research-UFZ, Permoser Str. 15, 04318 Leipzig, Germany
German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig, Deutscher Platz 5E, 04103 Leipzig, Germany
Search for more papers by this authorJérôme Guelat
Swiss Ornithological Institute, Seerose 1, 6204 Sempach, Switzerland
Search for more papers by this authorPetr Keil
German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig, Deutscher Platz 5E, 04103 Leipzig, Germany
Search for more papers by this authorJosé J. Lahoz-Monfort
School of BioSciences, University of Melbourne, Royal Parade, Parkville, Melbourne, Victoria, 3052 Australia
Search for more papers by this authorLaura J. Pollock
Univ. Grenoble Alpes, CNRS, Univ. Savoie Mont Blanc, Laboratoire d'Ecologie Alpine (LECA), Grenoble, 38000 France
Search for more papers by this authorBjörn Reineking
University Grenoble Alpes, Irstea, UR LESSEM, F-38402 St-Martin-d'Hères, Grenoble, France
Biogeographical Modelling, Bayreuth Center of Ecology and Environmental Research BayCEER, University of Bayreuth, Dr. Hans-Frisch-Straße 1-3, 95448 Bayreuth, Germany
Search for more papers by this authorDavid R. Roberts
Biometry and Environmental System Analysis, University of Freiburg, Tennenbacher Str. 4, 79106 Freiburg, Germany
Department of Geography, University of Calgary, 2500 University Dr. NW, Calgary, Alberta, T2N 1N4 Canada
Search for more papers by this authorBoris Schröder
Landscape Ecology and Environmental Systems Analysis, Institute of Geoecology, Technische Universität Braunschweig, Langer Kamp 19c, 38106 Braunschweig, Germany
Berlin-Brandenburg Institute of Advanced Biodiversity Research (BBIB), Altensteinstr. 34, 14195 Berlin, Germany
Search for more papers by this authorWilfried Thuiller
Univ. Grenoble Alpes, CNRS, Univ. Savoie Mont Blanc, Laboratoire d'Ecologie Alpine (LECA), Grenoble, 38000 France
Search for more papers by this authorDavid I. Warton
School of Mathematics and Statistics, Evolution and Ecology Research Centre, University of New South Wales, Sydney, New South Wales, 2052 Australia
Search for more papers by this authorBrendan A. Wintle
School of BioSciences, University of Melbourne, Royal Parade, Parkville, Melbourne, Victoria, 3052 Australia
Search for more papers by this authorSimon N. Wood
School of Mathematics, Bristol University, Tyndall Avenue, Bristol, BS8 1TW UK
Search for more papers by this authorRafael O. Wüest
Univ. Grenoble Alpes, CNRS, Univ. Savoie Mont Blanc, Laboratoire d'Ecologie Alpine (LECA), Grenoble, 38000 France
Swiss Federal Institute for Forest, Snow and Landscape Research WSL, Zürcherstrasse 111, 8903 Birmensdorf, Switzerland
Search for more papers by this authorFlorian Hartig
Biometry and Environmental System Analysis, University of Freiburg, Tennenbacher Str. 4, 79106 Freiburg, Germany
Theoretical Ecology, University of Regensburg, Universitätsstr. 31, 93053 Regensburg, Germany
Search for more papers by this authorAbstract
In ecology, the true causal structure for a given problem is often not known, and several plausible models and thus model predictions exist. It has been claimed that using weighted averages of these models can reduce prediction error, as well as better reflect model selection uncertainty. These claims, however, are often demonstrated by isolated examples. Analysts must better understand under which conditions model averaging can improve predictions and their uncertainty estimates. Moreover, a large range of different model averaging methods exists, raising the question of how they differ in their behaviour and performance. Here, we review the mathematical foundations of model averaging along with the diversity of approaches available. We explain that the error in model-averaged predictions depends on each model's predictive bias and variance, as well as the covariance in predictions between models, and uncertainty about model weights. We show that model averaging is particularly useful if the predictive error of contributing model predictions is dominated by variance, and if the covariance between models is low. For noisy data, which predominate in ecology, these conditions will often be met. Many different methods to derive averaging weights exist, from Bayesian over information-theoretical to cross-validation optimized and resampling approaches. A general recommendation is difficult, because the performance of methods is often context dependent. Importantly, estimating weights creates some additional uncertainty. As a result, estimated model weights may not always outperform arbitrary fixed weights, such as equal weights for all models. When averaging a set of models with many inadequate models, however, estimating model weights will typically be superior to equal weights. We also investigate the quality of the confidence intervals calculated for model-averaged predictions, showing that they differ greatly in behaviour and seldom manage to achieve nominal coverage. Our overall recommendations stress the importance of non-parametric methods such as cross-validation for a reliable uncertainty quantification of model-averaged predictions.
Supporting Information
Filename | Description |
---|---|
ecm1309-sup-0001-AppendixS1.pdfPDF document, 758.2 KB | |
ecm1309-sup-0002-DataS1.zipZip archive, 2.1 MB | |
ecm1309-sup-0003-MetadataS1.pdfPDF document, 66.8 KB |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
Literature Cited
- Akaike, H. 1973. Information theory as an extension of the maximum likelihood principle. Pages 267–281 in B. Petrov and F. Csaki, editors. 2nd International Symposium on Information theory. Akademiai Kiado, Budapest, Hungary.
- Armstrong, J. S. 2001. Combining forecasts. Pages 417–439 in J. S. Armstrong, editor. Principles of forecasting: a handbook for researchers and practitioners. Springer, New York, New York, USA.
- Augustin, N., W. Sauerbrei, and M. Schumacher. 2005. The practical utility of incorporating model selection uncertainty into prognostic models for survival data. Statistical Modelling 5: 95–118.
- Banner, K. M., and M. D. Higgs. 2017. Considerations for assessing model averaging of regression coefficients. Ecological Applications 28: 78–93.
- Bates, J. M., and C. W. J. Granger. 1969. The combination of forecasts. Journal of the Operational Research Society 20: 451–468.
- Bauer, P., A. Thorpe, and G. Brunet. 2015. The quiet revolution of numerical weather prediction. Nature 525: 47–55.
- Berger, J. O., and L. R. Pericchi. 1996. The intrinisic Bayes factor for model selection and prediction. Journal of the American Statistical Association 91: 109–122.
- Bolker, B. M. 2008. Ecological models and data in R. Princeton University Press, Princeton, New Jersey, USA.
10.2307/j.ctvcm4g37 Google Scholar
- Bollen, K. A., S. Ray, J. Zavisca, and J. J. Harden. 2012. A comparison of Bayes factor approximation methods including two new methods. Sociological Methods & Research 41: 294–324.
- Brandon, J. R., and P. R. Wade. 2006. Assessment of the Bering-Chukchi-Beaufort seas stock of bowhead whales using Bayesian model averaging. Journal of Cetacean Research Management 8: 225–239.
- Breiner, F. T., A. Guisan, A. Bergamini, and M. P. Nobis. 2015. Overcoming limitations of modelling rare species by using ensembles of small models. Methods in Ecology and Evolution 6: 1210–1218.
- Brodziak, J., and C. M. Legault. 2005. Model averaging to estimate rebuilding targets for overfished stocks. Canadian Journal of Fisheries and Aquatic Sciences 62: 544–562.
- Brodziak, J., and K. Piner. 2010. Model averaging and probable status of North Pacificstriped marlin, Tetrapturus audax. Canadian Journal of Fisheries and Aquatic Sciences 67: 793–805.
- Brook, B. W., and C. J. A. Bradshaw. 2006. Strength of evidence for density dependence in abundance time series of 1198 species. Ecology 87: 1445–1451.
- Buckland, S. T., K. P. Burnham, and N. H. Augustin. 1997. Model selection: an integral part of inference. Biometrics 53: 603–618.
- Burnham, K. P. and D. R. Anderson. 2002. Model selection and multi-model inference: a practical information-theoretical approach. Second edition. Springer, Berlin, Germany.
- Burnham, K. P., D. R. Anderson, and K. P. Huyvaert. 2011. AIC model selection and multimodel inference in behavioral ecology: some background, observations, and comparisons. Behavioral Ecology and Sociobiology 65: 23–35.
- Butler, A., R. M. Doherty, and G. Marion. 2009. Model averaging to combine simulations of future global vegetation carbon stocks. Environmetrics 20: 791–811.
- Cade, B. 2015. Model averaging and muddled multimodal inferences. Ecology 96: 2370–2382.
- Cariveau, D. P., N. M. Williams, F. E. Benjamin, and R. Winfree. 2013. Response diversity to land use occurs but does not consistently stabilise ecosystem services provided by native pollinators. Ecology Letters 16: 903–911.
- Chamberlin, T. C. 1890. The method of multiple working hypotheses. Science 15: 92–96.
- Chen, X., G. Zou, and X. Zhang. 2012. Frequentist model averaging for linear mixed-effects models. Frontiers of Mathematics in China 8: 497–515.
- Chickering, D. M., and D. Heckerman. 1997. Efficient approximations for the marginal likelihood of Bayesian networks with hidden variables. Machine Learning 29: 181–212.
- Claeskens, G., and N. L. Hjort. 2008. Model selection and model averaging. Cambridge University Press, Cambridge, UK.
10.1017/CBO9780511790485 Google Scholar
- Claeskens, G., J. R. Magnus, A. L. Vasnev, and W. Wang. 2016. The forecast combination puzzle: A simple theoretical explanation. International Journal of Forecasting 32: 754–762.
- Clarke, B. 2003. Comparing Bayes model averaging and stacking when model approximation error cannot be ignored. Journal of Machine Learning Research 4: 683–712.
- Clemen, R. 1989. Combining forecasts: a review and annotated bibliography. International Journal of Forecasting 5: 559–581.
- Congdon, P. 2007. Model weights for model choice and averaging. Statistical Methodology 4: 143–157.
10.1016/j.stamet.2006.05.002 Google Scholar
- Corani, G., and A. Mignatti. 2015. Robust Bayesian model averaging for the analysis of presence–absence data. Environmental and Ecological Statistics 22: 513–534.
- Crimmins, S. M., S. Z. Dobrowski, and A. R. Mynsberge. 2013. Evaluating ensemble forecasts of plant species distributions under climate change. Ecological Modelling 266: 126–130.
- Crisci, C., R. Terra, J. Pablo, B. Ghattas, M. Bidegain, G. Goyenola, J. José, G. Méndez, and N. Mazzeo. 2017. Multi-model approach to predict phytoplankton biomass and composition dynamics in a eutrophic shallow lake governed by extreme meteorological events. Ecological Modelling 360: 80–93.
- Crossman, N. D., and D. A. Bass. 2008. Application of common predictive habitat techniques for post-border weed risk management. Diversity and Distributions 14: 213–224.
- Dai, Q., and Y. Wang. 2011. Effect of road on the distribution of amphibians in wetland area – test with model-averaged prediction. Polish Journal of Ecology 59: 813–821.
- Dietze, M. C. 2017. Ecological forecasting. Princeton University Press, Princeton, New Jersey, USA.
10.1515/9781400885459 Google Scholar
- Diks, C. G. H., and J. A. Vrugt. 2010. Comparison of point forecast accuracy of model averaging methods in hydrologic applications. Stochastic Environmental Research and Risk Assessment 24: 809–820.
- Diniz-Filho, J. A. F., L. Mauricio Bini, T. Fernando Rangel, R. D. Loyola, C. Hof, D. Nogués-Bravo, M. B. Araújo, L. M. Bini, and T. F. L. V. B. Rangel. 2009. Partitioning and mapping uncertainties in ensembles of forecasts of species turnover under climate change. Ecography 32: 897–906.
- Dormann, C. F., et al. 2008. Prediction uncertainty of environmental change effects on temperate european biodiversity. Ecology Letters 11: 235–244.
- Draper, D. 1995. Assessment and propagation of model uncertainty. Journal of the Royal Statistical Society B 57: 45–97.
- Dwyer, J. F., R. E. Harness, and K. Donohue. 2014. Predictive model of avian electrocution risk on overhead power lines. Conservation Biology 28: 159–168.
- Edeling, W. N., P. Cinnella, and R. P. Dwight. 2014. Predictive RANS simulations via Bayesian model-scenario averaging. Journal of Computational Physics 275: 65–91.
- Elith, J., J. R. Leathwick, and T. Hastie. 2008. A working guide to boosted regression trees. Journal of Animal Ecology 77: 802–813.
- Ellison, A. M. 2004. Bayesian inference in ecology. Ecology Letters 7: 509–520.
- Engler, R., L. T. Waser, N. E. Zimmermann, M. Schaub, S. Berdos, C. Ginzler, and A. Psomas. 2013. Combining ensemble modeling and remote sensing for mapping individual tree species at high spatial resolution. Forest Ecology and Management 310: 64–73.
- Fletcher, D., and P. W. Dillingham. 2011. Model-averaged confidence intervals for factorial experiments. Computational Statistics and Data Analysis 55: 3041–3048.
- Fletcher, D., and D. Turek. 2012. Model-averaged profile likelihood intervals. Journal of Agricultural Biological and Environmental Statistics 17: 38–51.
- Forester, B. R., E. G. DeChaine, and A. G. Bunn. 2013. Integrating ensemble species distribution modelling and statistical phylogeography to inform projections of climate change impacts on species distributions. Diversity and Distributions 19: 1480–1495.
- Freckleton, R. P. 2011. Dealing with collinearity in behavioural and ecological data: model averaging and the problems of measurement error. Behavioral Ecology and Sociobiology 65: 91–101.
- Garcia, R. A., N. D. Burgess, M. Cabeza, C. Rahbek, and M. B. Araújo. 2012. Exploring consensus in 21st century projections of climatically suitable areas for African vertebrates. Global Change Biology 18: 1253–1269.
- Garthwaite, P. H., and E. Mubwandarikwa. 2010. Selection of weights for weighted model averaging. Australian & New Zealand Journal of Statistics 52: 363–382.
- Geisser, S. 1993. Predictive inference. CRC Press, Boca Raton, Florida, USA.
10.1007/978-1-4899-4467-2 Google Scholar
- Gelman, A., J. Hwang, and A. Vehtari. 2014. Understanding predictive information criteria for Bayesian models. Statistics and Computing 24: 997–1016.
- Ghosh, D., and Z. Yuan. 2009. An improved model averaging scheme for logistic regression. Journal of Multivariate Analysis 100: 1670–1681.
- Gibbons, J. M., G. M. Cox, A. T. A. Wood, J. Craigon, S. J. Ramsden, D. Tarsitano, and N. M. J. Crout. 2008. Applying Bayesian model averaging to mechanistic models: An example and comparison of methods. Environmental Modelling & Software 23: 973–985.
- Gibbs, J. W. 1902. Elementary principles in statistical mechanics. Charles Scribner's Sons, New York, New York, USA.
- Giorgi, F., and L. O. Mearns. 2002. Calculation of average, uncertainty range, and reliability of regional climate changes from AOGCM simulations via the “reliability ensemble averaging” (REA) method. Journal of Climate 15: 1141–1158.
- Gneiting, T., A. E. Raftery, A. H. Westveld, and T. Goldman. 2005. Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Monthly Weather Review 133: 1098–1118.
- Graefe, A., J. S. Armstrong, R. J. Jones, and A. G. Cuzan. 2014. Combining forecasts: An application to elections. International Journal of Forecasting 30: 43–54.
- Graefe, A., H. Küchenhoff, V. Stierle and B. Riedl. 2015. Limitations of Ensemble Bayesian Model Averaging for forecasting social science problems weights. International Journal of Forecasting 31: 943–951.
- Granger, C. W. J., and R. Ramanathan. 1984. Improved methods of combining forecasts. Journal of Forecasting 3: 194–204.
- Green, P. J. P. 1995. Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82: 711–732.
- Grenouillet, G., L. Buisson, N. Casajus, and S. Lek. 2010. Ensemble modelling of species distribution: the effects of geographical and environmental ranges. Ecography 34: 9–17.
- Grueber, C. E., S. Nakagawa, R. J. Laws, and I. G. Jamieson. 2011. Multimodel inference in ecology and evolution: challenges and solutions. Journal of Evolutionary Biology 24: 699–711.
- Hannemann, H., K. J. Willis, and M. Macias-Fauria. 2015. The devil is in the detail: unstable response functions in species distribution models challenge bulk ensemble. Global Ecology and Biogeography 25: 26–35.
- Hansen, B. E. 2007. Least squares model averaging. Econometrica 75: 1175–1189.
- Hansen, B. E., and J. S. Racine. 2012. Jackknife model averaging. Journal of Econometrics 167: 38–46.
- Hanson, P. J., et al. 2004. Oak forest carbon and water simulations: Model intercomparisons and evaluations against independent data. Ecological Monographs 74: 443–489.
- Harrell, F. E. 2001. Regression modeling strategies - with applications to linear models, logistic regression, and survival analysis. Springer, New York, New York, USA.
10.1007/978-1-4757-3462-1 Google Scholar
- Hartig, F., J. M. Calabrese, B. Reineking, T. Wiegand, and A. Huth. 2011. Statistical inference for stochastic simulation models - theory and application. Ecology Letters 14: 816–827.
- Hastie, T., R. J. Tibshirani and J. H. Friedman. 2009. The elements of statistical learning: data mining, inference, and prediction. Second edition. Springer, Berlin, Germany.
- Hauenstein, S., C. F. Dormann and S. N. Wood. 2017. Computing AIC for black-box models using Generalised Degrees of Freedom: a comparison with cross-validation. Communications in Statistics-Simulation and Computation: 1–17. https://doi.org/10.1080/03610918.2017.1315728.
- Hill, S. L., G. M. Watters, A. E. Punt, M. K. McAllister, C. L. Qéré, and J. Turner. 2007. Model uncertainty in the ecosystem approach to fisheries. Fish and Fisheries 8: 315–336.
- Hobbs, N. T., and R. Hilborn. 2006. Alternatives to statistical hypothesis testing in ecology: a guide to self teaching. Ecological Applications 16: 5–19.
- Hoeting, J., D. Madigan, A. Raftery, and C. Volinsky. 1999. Bayesian model averaging: a tutorial. Statistical Science 14: 382–401.
- Hollowed, A. B., N. A. Bond, T. K. Wilderbuer, W. T. Stockhausen, Z. T. A'mar, R. J. Beamish, J. E. Overland and M. J. Schirripa. 2009. A framework for modelling fish and shellfish responses to future climate change. ICES Journal of Marine Science, 66: 1584–1594.
- Hooten, M. B., and N. T. Hobbs. 2015. A guide to Bayesian model selection for ecologists. Ecological Monographs 85: 3–28.
- Jiao, Y., K. Reid, and E. Smith. 2009. Model selection uncertainty and Bayesian model averaging in fisheries recruitment modeling. Pages 505–524 in R. J. Beamish and B. J. Rothschild, editors. The future of fisheries science in North America. Springer, New York, New York, USA.
- Johnson, C., and N. Bowler. 2009. On the reliability and calibration of ensemble forecasts. Monthly Weather Review 137: 1717–1720.
- Johnson, J. B., and K. S. Omland. 2004. Model selection in ecology and evolution. Trends in Ecology and Evolution 19: 101–108.
- Kabaila, P., A. H. Welsh, and W. Abeysekera. 2015. Model-averaged confidence intervals. Scandinavian Journal of Statistics 43: 35–48.
- Kass, R. E., and A. E. Raftery. 1995. Bayes factors. Journal of the American Statistical Association 90: 773–795.
- Katsanevakis, S. 2006. Modelling fish growth: Model selection, multi-model inference and model selection uncertainty. Fisheries Research 81: 229–235.
- Katsanevakis, S., and C. D. Maravelias. 2008. Modelling fish growth: multi-model inference as a better alternative to a priori using von Bertalanffy equation. Fish and Fisheries 9: 178–187.
- King, R., S. P. Brooks, C. Mazzetta, S. N. Freeman and B. J. T. Morgan. 2008. Identifying and diagnosing population declines: A Bayesian assessment of lapwings in the UK. Journal of the Royal Statistical Society C, 57: 609–632.
- Knutti, R., R. Furrer, C. Tebaldi, J. Cermak, and G. A. Meehl. 2010. Challenges in combining projections from multiple climate models. Journal of Climate 23: 2739–2758.
- Krishnamurti, T. N., C. M. Kishtawal, T. E. LaRow, D. R. Bachiochi, Z. Zhang, C. E. Williford, S. Gadgil, and S. Surendran. 1999. Improved weather and seasonal climate forecasts from multimodel superensemble. Science 285: 1548–1550.
- Kuhn, M., and K. Johnson. 2013. Applied predictive modeling. Springer, Berlin, Germany.
10.1007/978-1-4614-6849-3 Google Scholar
- Lauzeral, C., G. Grenouillet, and S. Brosse. 2015. The iterative ensemble modelling approach increases the accuracy of fish distribution models. Ecography 38: 213–220.
- Le Lay, G., R. Engler, E. Franc, and A. Guisan. 2010. Prospective sampling based on model ensembles improves the detection of rare species. Ecography 33: 1015–1027.
- Liang, H., G. Zou, A. T. K. Wan, and X. Zhang. 2011. Optimal weight choice for frequentist model average estimators. Journal of the American Statistical Association 106: 1053–1066.
- Link, W. A., and R. J. Barker. 2006. Model weights and the foundations of multimodel inference. Ecology 87: 2626–2635.
- Lukacs, P. M., K. P. Burnham, and D. R. Anderson. 2010. Model selection bias and Freedman's paradox. Annals of the Institute of Statistical Mathematics 62: 117–125.
- Madigan, D., and A. E. Raftery. 1994. Model selection and accounting for model uncertainty in graphical models using Occam's window. Journal of the American Statistical Association 89: 1535–1546.
- Marmion, M., J. Hjort, W. Thuiller, and M. Luoto. 2009a. Statistical consensus methods for improving predictive geomorphology maps. Computers & Geosciences 35: 615–625.
- Marmion, M., M. Parviainen, M. Luoto, R. K. Heikkinen, and W. Thuiller. 2009b. Evaluation of consensus methods in predictive species distribution modelling. Diversity and Distributions 15: 59–69.
- Meller, L., M. Cabeza, S. Pironon, M. Barbet-Massin, L. Maiorano, D. Georges, and W. Thuiller. 2014. Ensemble distribution models in conservation prioritization: from consensus predictions to consensus reserve networks. Diversity and Distributions 20: 309–321.
- Montgomery, J. M., F. M. Hollenbach, and M. D. Ward. 2012. Improving predictions using Ensemble Bayesian Model Averaging. Political Analysis 20: 271–291.
- Mouquet, N., et al. 2015. Predictive ecology in a changing world. Journal of Applied Ecology 52: 1293–1310.
- Nagelkerke, N. J. D. 1991. A note on a general definition of the coefficient of determination. Biometrika 78: 691–692.
- Nakagawa, S., and R. P. Freckleton. 2011. Model averaging, missing data and multiple imputation: a case study for behavioural ecology. Behavioral Ecology and Sociobiology 65: 91–101.
- Nakagawa, S., and H. Schielzeth. 2013. A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods in Ecology and Evolution 4: 133–142.
- Namata, H., M. Aerts, C. Faes, and P. Teunis. 2008. Model averaging in microbial risk assessment using fractional polynomials. Risk Analysis 28: 891–905.
- Newbold, P., and C. W. J. Granger. 1974. Experience with forecasting univariate time series and the combination of forecasts. Journal of the Royal Statistical Society A 131: 131–165.
- Nguefack-Tsague, G. 2014. On optimal weighting scheme in model averaging. American Journal of Applied Mathematics and Statistics 2: 150–156.
10.12691/ajams-2-3-9 Google Scholar
- O'Hagan, A. 1995. Fractional Bayes factors for model comparison. Journal of the Royal Statistical Society B 57: 99–138.
10.1111/j.2517-6161.1995.tb02017.x Google Scholar
- Ordonez, A., and J. W. Williams. 2013. Climatic and biotic velocities for woody taxa distributions over the last 16 000 years in eastern North America. Ecology Letters 16: 773–781.
- Pan, W. 2001. Akaike's information criterion in generalized estimating equations. Biometrics 57: 120–125.
- Picard, N., M. Henry, F. Mortier, C. Trotta, and L. Saint-André. 2012. Using Bayesian model averaging to predict tree aboveground biomass in tropical moist forests. Forest Science 58: 15–23.
- Posada, D., and T. R. Buckley. 2004. Model selection and model averaging in phylogenetics: advantages of Akaike information criterion and Bayesian approaches over likelihood ratio tests. Systematic Biology 53: 793–808.
- Potempski, S., and S. Galmarini. 2009. Est modus in rebus: analytical properties of multi-model ensembles. Atmospheric Chemistry and Physics 9: 9471–9489.
- Raftery, A. E., D. Madigan, and J. A. Hoeting. 1997. Bayesian model averaging for linear regression models. Journal of the American Statistical Association 92: 179–191.
- Raftery, A. E., T. Gneiting, F. Balabdaoui, and M. Polakowski. 2005. Using Bayesian model averaging to calibrate forecast ensembles. Monthly Weather Review 133: 1155–1174.
- Rapacciuolo, G., D. B. Roy, S. Gillings, R. Fox, K. Walker, and A. Purvis. 2012. Climatic associations of British species distributions show good transferability in time but low predictive accuracy for range change. PLoS ONE 7: e40212.
- Richards, S. A. 2005. Testing ecological theory using the information-theoretic approach: examples and cautionary results. Ecology 86: 2805–2814.
- Richards, S. A., M. J. Whittingham, and P. A. Stephens. 2011. Model selection and model averaging in behavioural ecology: the utility of the IT-AIC framework. Behavioral Ecology and Sociobiology 65: 77–89.
- Roberts, D. R., et al. 2017. Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40: 913–929.
- Romero, D., J. Olivero, J. C. Brito, and R. Real. 2016. Comparison of approaches to combine species distribution models based on different sets of predictors. Ecography 39: 561–571.
- Rougier, J. 2016. Ensemble averaging and mean squared error. Journal of Climate 29: 8865–8870.
- Rovai, A. S., et al. 2015. Scaling mangrove aboveground biomass from site-level to continental-scale. Global Ecology and Biogeography 25: 286–298.
- Schomaker, M. 2012. Shrinkage averaging estimation. Statistical Papers 53: 1015–1034.
- Schomaker, M., A. T. K. Wan, and C. Heumann. 2010. Frequentist model averaging with missing observations. Computational Statistics and Data Analysis 54: 3336–3347.
- Schwartz, G. E. 1978. Estimating the dimension of a model. Annals of Statistics 6: 461–464.
- Shen, X., and H.-C. Huang. 2006. Optimal model assessment, selection, and combination. Journal of American Statistical Association 101: 554–568.
- Shmueli, G. 2010. To explain or to predict? Statistical Science 25: 289–310.
- Smith, R. L., C. Tebaldi, D. W. Nychka, and L. O. Mearns. 2009. Bayesian modeling of uncertainty in ensembles of climate models. Journal of the American Statistical Association 104: 97–116.
- Smith, A. B., M. J. Santos, M. S. Koo, K. M. C. Rowe, K. C. Rowe, J. L. Patton, J. D. Perrine, S. R. Beissinger, and C. Moritz. 2013. Evaluation of species distribution models by resampling of sites surveyed a century ago by Joseph Grinnell. Ecography 36: 1017–1031.
- Smyth, P. and D. Wolpert. 1998. An evaluation of linearly combining density estimators via stacking. Technical Report No. 98-25. Information and Computer Science Department, University of California, Irvine, California, USA.
- Solomon, S., D. Qin, M. Manning, M. Marquis, K. Averyt, M. Tignor, H. L. MillerJr. and Z. Chen, editors. 2007. Climate change 2007: The physical science basis. Cambridge University Press, Cambridge, UK.
- Stanley, T. R., and K. P. Burnham. 1998. Information-theoretic model selection and model averaging for closed-population capture-recapture studies. Biometrical Journal 40: 475–494.
- Stock, J., and M. Watson. 2001. A comparison of linear and nonlinear univariate models for forecasting macroeconomic time series. Pages 1–44 in R. Engle and H. White, editors. Cointegration, causality, and forecasting: a festschrift in honour of Clive Granger. Oxford University Press, Oxford, UK.
- Stone, M.. 1977. An asymptotic equivalence of choice of model by cross-validation and Akaike's criterion. Journal of the Royal Statistical Society B, 39: 44–47.
- Symonds, M. R. E., and A. Moussalli. 2011. A brief guide to model selection, multimodel inference and model averaging in behavioural ecology using Akaike's information criterion. Behavioral Ecology and Sociobiology 65: 13–21.
- Thibaut, L. M., and S. R. Connolly. 2013. Understanding diversity-stability relationships: towards a unified model of portfolio effects. Ecology Letters 16: 140–150.
- Thomson, M. C., F. J. Doblas-Reyes, S. J. Mason, R. Hagedorn, S. J. Connor, T. Phindela, A. P. Morse, and T. N. Palmer. 2006. Malaria early warnings based on seasonal climate forecasts from multi-model ensembles. Nature 439: 576–579.
- Thuiller, W. 2004. Patterns and uncertainties of species’ range shifts under climate change. Global Change Biology 10: 2020–2027.
- Timmermann, A. G. 2006. Forecast combinations. Pages 135–196 in G. Elliott, C. Granger, and A. Timmermann, editors. Handbook of economic forecasting. Elsevier, Dordrecht, The Netherlands.
- Ting, K. M., and I. H. Witten. 1999. Issues in stacked generalization. Journal of Artificial Intelligence Research 10: 271–289.
- Toni, T., D. Welch, N. Strelkowa, A. Ipsen, and M. P. H. Stumpf. 2009. Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems. Journal of the Royal Society Interface 6: 187–202.
- Trolle, D., J. A. Elliott, W. M. Mooij, J. H. Janse, K. Bolding, D. P. Hamilton, and E. Jeppesen. 2014. Advancing projections of phytoplankton responses to climate change through ensemble modelling. Environmental Modelling & Software 61: 371–379.
- Turek, D., and D. Fletcher. 2012. Model-averaged Wald confidence intervals. Computational Statistics & Data Analysis 56: 2809–2815.
- van der Laan, M. J., S. Dudoit and S. Keles. 2004. Asymptotic optimality of likelihood-based cross-validation. Statistical Applications in Genetics and Molecular Biology, 3: Article 4.
- van Oijen, M., et al. 2013. Bayesian calibration, comparison and averaging of six forest models, using data from Scots pine stands across Europe. Forest Ecology and Management 289: 255–268.
- Wang, Y.-P., C. M. Trudinger, and I. G. Enting. 2009. A review of applications of model–data fusion to studies of terrestrial carbon ƒuxes at different scales. Agricultural and Forest Meteorology 149: 1829–1842.
- Wasserman, L. 2000. Bayesian model selection and model averaging. Journal of Mathematical Psychology 44: 92–107.
- Watanabe, S. 2010. Asymptotic equivalence of Bayes cross validation and Widely Applicable Information Criterion in singular learning theory. Journal of Machine Learning Research 11: 3571–3594.
- Watanabe, S. 2013. A widely applicable Bayesian Information Criterion. Journal of Machine Learning Research 14: 867–897.
- Weinberg, M. D. 2012. Computing the Bayes factor from a Markov chain Monte Carlo simulation of the posterior distribution. Bayesian Analysis 7: 737–770.
- Wintle, B. A., M. A. McCarthy, C. T. Volinsky, and R. P. Kavanagh. 2003. The use of Bayesian model averaging to better represent uncertainty in ecological models. Conservation Biology 17: 1579–1590.
- Wolpert, D. H. 1992. Stacked generalization. Neural Networks 5: 241–259.
- Wood, S. N. 2015. Core statistics. Cambridge University Press, Cambridge, UK.
10.1017/CBO9781107741973 Google Scholar
- Yang, Y. 2001. Adaptive regression by mixing. Journal of the American Statistical Association 96: 574–588.
- Yao, Y., A. Vehtari, D. Simpson and A. Gelman. 2017. Using stacking to average Bayesian predictive distributions. arXiv, (stat.ME):1704.02030v3.
- Ye, J. 1998. On measuring and correcting the effects of data mining and model selection. Journal of the American Statistical Association 93: 120–131.
- Yuan, Z., and D. Ghosh. 2008. Combining multiple biomarker models in logistic regression. Biometrics 64: 431–439.
- Yuan, Z., and Y. H. Yang. 2005. Combining linear regression models: When and how? Journal of the American Statistical Association 100: 1202–1214.
- Zhang, X., R. Srinivasan, and D. Bosch. 2009. Calibration and uncertainty analysis of the SWAT model using Genetic Algorithms and Bayesian Model Averaging. Journal of Hydrology 374: 307–317.
- Zhang, X., A. T. K. Wan, and G. Zou. 2013. Model averaging by jackknife criterion in models with dependent data. Journal of Econometrics 174: 82–94.
- Zhao, K., D. Valle, S. Popescu, X. Zhang, and B. Mallick. 2013. Hyperspectral remote sensing of plant biochemistry using Bayesian model averaging with variable and band selection. Remote Sensing of Environment 132: 102–119.