Forum—P Values and Model Selection
In defense of P values
Corresponding Editor: A. M. Ellison. For reprints of this Forum, see footnote 1, p. 609.
Abstract
Statistical hypothesis testing has been widely criticized by ecologists in recent years. I review some of the more persistent criticisms of P values and argue that most stem from misunderstandings or incorrect interpretations, rather than from intrinsic shortcomings of the P value. I show that P values are intimately linked to confidence intervals and to differences in Akaike's information criterion (ΔAIC), two metrics that have been advocated as replacements for the P value. The choice of a threshold value of ΔAIC that breaks ties among competing models is as arbitrary as the choice of the probability of a Type I error in hypothesis testing, and several other criticisms of the P value apply equally to ΔAIC. Since P values, confidence intervals, and ΔAIC are based on the same statistical information, all have their places in modern statistical practice. The choice of which to use should be stylistic, dictated by details of the application rather than by dogmatic, a priori considerations.
Citing Literature
Number of times cited: 87
- Sebastian T. Meyer, Robert Ptacnik, Helmut Hillebrand, Holger Bessler, Nina Buchmann, Anne Ebeling, Nico Eisenhauer, Christof Engels, Markus Fischer, Stefan Halle, Alexandra-Maria Klein, Yvonne Oelmann, Christiane Roscher, Tanja Rottstock, Christoph Scherber, Stefan Scheu, Bernhard Schmid, Ernst-Detlef Schulze, Vicky M. Temperton, Teja Tscharntke, Winfried Voigt, Alexandra Weigelt, Wolfgang Wilcke and Wolfgang W. Weisser, Biodiversity–multifunctionality relationships depend on identity and number of measured functions, Nature Ecology & Evolution, 2, 1, (44), (2018).
- J. H. Pagella, R. W. Mayes, F. J. Pérez-Barbería and E. R. Ørskov, The development of an intraruminal nylon bag technique using non-fistulated animals to assess the rumen degradability of dietary plant materials, animal, 12, 01, (54), (2018).
- Ishan Ajmera, Jing Shi, Jitender Giri, Ping Wu, Dov J. Stekel, Chungui Lu and T. Charlie Hodgman, Regulatory feedback response mechanisms to phosphate starvation in rice, npj Systems Biology and Applications, 4, 1, (2018).
- G Yang, W N Brandt, F Vito, C-T J Chen, J R Trump, B Luo, M Y Sun, Y Q Xue, A M Koekemoer, D P Schneider, C Vignali and J-X Wang, Linking black hole growth with host galaxies: the accretion–stellar mass relation and its cosmic evolution, Monthly Notices of the Royal Astronomical Society, 475, 2, (1887), (2018).
- F.J. Pérez-Barbería and D.M. Walker, DYNAMICS OF SOCIAL BEHAVIOUR AT PARTURITION IN A GREGARIOUS UNGULATE, Behavioural Processes, (2018).
- Jan Graffelman and Bruce S. Weir, On the testing of Hardy‐Weinberg proportions and equality of allele frequencies in males and females at biallelic genetic markers, Genetic Epidemiology, 42, 1, (34-48), (2017).
- Adam J. Becker, Diane V. Roeder, Michael S. Husak and Michael T. Murphy, Annual survival and breeding dispersal of a migratory passerine, the Scissor‐tailed Flycatcher, Journal of Field Ornithology, 89, 1, (22-36), (2018).
- William D. Hintz, Brian M. Mattes, Matthew S. Schuler, Devin K. Jones, Aaron B. Stoler, Lovisa Lind and Rick A. Relyea, Salinization triggers a trophic cascade in experimental freshwater communities with varying food‐chain length, Ecological Applications, 27, 3, (833-844), (2017).
- Joseph P. Ceradini and Anna D. Chalfoun, When perception reflects reality: Non‐native grass invasion alters small mammal risk landscapes and survival, Ecology and Evolution, 7, 6, (1823-1835), (2017).
- Marion Barbé, Nicole J. Fenton, Yves Bergeron and Kerry Woods, Boreal bryophyte response to natural fire edge creation, Journal of Vegetation Science, 28, 5, (915-927), (2017).
- Anthony P. Porreca, William D. Hintz, David P. Coulter and James E. Garvey, Subtle physiological and morphological differences explain ecological success of sympatric congeners, Ecosphere, 8, 10, (2017).
- Daria Corcos, Diego J. Inclán, Pierfilippo Cerretti, Maurizio Mei, Filippo Di Giovanni, Daniele Birtele, Paolo Rosa, Alessio De Biase, Paolo Audisio, Lorenzo Marini, Raphael Didham and Philip Barton, Environmental heterogeneity effects on predator and parasitoid insects vary across spatial scales and seasons: a multi‐taxon approach, Insect Conservation and Diversity, 10, 6, (462-471), (2017).
- F.J. Pérez-Barbería, Scaling methane emissions in ruminants and global estimates in wild populations, Science of The Total Environment, 579, (1572), (2017).
- C. F. Manara, L. Testi, G. J. Herczeg, I. Pascucci, J. M. Alcalá, A. Natta, S. Antoniucci, D. Fedele, G. D. Mulders, T. Henning, S. Mohanty, T. Prusti and E. Rigliaco, X-shooter study of accretion in Chamaeleon I, Astronomy & Astrophysics, 604, (A127), (2017).
- Bernhard Schmid, Martin Baruffol, Zhiheng Wang and Pascal A. Niklaus, A guide to analyzing biodiversity experiments, Journal of Plant Ecology, 10, 1, (91), (2017).
- Haili Yu, Nianpeng He, Qiufeng Wang, Jianxing Zhu, Yang Gao, Yunhai Zhang, Yanlong Jia and Guirui Yu, Development of atmospheric acid deposition in China from the 1990s to the 2010s, Environmental Pollution, 231, (182), (2017).
- Michael G. Smircich, David L. Strayer and Eric T. Schultz, Zebra mussel (Dreissena polymorpha) affects the feeding ecology of early stage striped bass (Morone saxatilis) in the Hudson River estuary, Environmental Biology of Fishes, 100, 4, (395), (2017).
- Valentin Amrhein, Fränzi Korner-Nievergelt and Tobias Roth, The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research, PeerJ, 5, (e3544), (2017).
- Alireza Ermagun and David Levinson, “Transit makes you short”: On health impact assessment of transportation and the built environment, Journal of Transport & Health, 4, (373), (2017).
- John Antonakis, On doing better science: From thrill of discovery to policy implications, The Leadership Quarterly, 10.1016/j.leaqua.2017.01.006, 28, 1, (5-21), (2017).
- Michel Duhalde, Harold Levrel and Olivier Guyader, Is the choice of conservation measures influenced by the targeted natural habitats? The case of French coastal Natura 2000 sites, Ocean & Coastal Management, 142, (15), (2017).
- Rosalinda Gonzalez, Jason Dunham, Scott Lightcap and Jeff McEnroe, Large Wood and Instream Habitat for Juvenile Coho Salmon and Larval Lampreys in a Pacific Northwest Stream, North American Journal of Fisheries Management, 37, 4, (683-699), (2017).
- Gregory Francis, Equivalent statistics and data interpretation, Behavior Research Methods, 49, 4, (1524), (2017).
- François Guillemette, Martin-Michel Gauthier and Rock Ouimet, Partitioning risks of tree mortality by modes of death in managed and unmanaged northern hardwoods and mixedwoods, The Forestry Chronicle, 10.5558/tfc2017-033, 93, 03, (246-258), (2017).
- Wei Wu, Hailong Huang, Patrick Biber and Matthew Bethel, Litter Decomposition ofSpartina alternifloraandJuncus roemerianus: Implications of Climate Change in Salt Marshes, Journal of Coastal Research, 332, (372), (2017).
- Michael L. Schummer, John M. Coluccy, Michael Mitchell and Lena Van Den Elsen, Long‐term trends in weather severity indices for dabbling ducks in eastern North America, Wildlife Society Bulletin, 41, 4, (615-623), (2017).
- A.B. Stoler, B.M. Mattes, W.D. Hintz, D.K. Jones, L. Lind, M.S. Schuler and R.A. Relyea, Effects of a common insecticide on wetland communities with varying quality of leaf litter inputs, Environmental Pollution, 226, (452), (2017).
- Timothy L. Lash, The Harm Done to Reproducibility by the Culture of Null Hypothesis Significance Testing, American Journal of Epidemiology, 186, 6, (627), (2017).
- Hanna K. Nuuttila, Chiara M. Bertelli, Anouska Mendzil and Nessa Dearle, Seasonal and diel patterns in cetacean use and foraging at a potential marine renewable energy site, Marine Pollution Bulletin, (2017).
- László Zsolt Garamszegi and Pierre de Villemereuil, Perturbations on the uniform distribution of p-values can lead to misleading inferences from null-hypothesis testing, Trends in Neuroscience and Education, 8-9, (18), (2017).
- Derek P. Crane and John M. Farrell, Trends in body condition of smallmouth bass and northern pike (1982–2013) following multiple ecological perturbations in the St. Lawrence River, Canadian Journal of Fisheries and Aquatic Sciences, 74, 8, (1158), (2017).
- Joseph P. Ceradini and Anna D. Chalfoun, Species’ traits help predict small mammal responses to habitat homogenization by an invasive grass, Ecological Applications, 27, 5, (1451-1465), (2017).
- Zachary J. Kuzniar, Robert W. Van Kirk and Eric B. Snyder, Seasonal effects of macrophyte growth on rainbow trout habitat availability and selection in a low‐gradient, groundwater‐dominated river, Ecology of Freshwater Fish, 26, 4, (653-665), (2016).
- Ken Aho, Dewayne Derryberry, Teri Peterson and Robert B. O'Hara, A graphical framework for model selection criteria and significance tests: refutation, confirmation and ecology, Methods in Ecology and Evolution, 8, 1, (47-56), (2016).
- Nwaiwu Ogueri and I. Ibekwe Vincent, Prevalence and risk of heterotrophic microorganisms in a carbonated soft drink factory, African Journal of Microbiology Research, 11, 6, (245), (2017).
- Heather T. Root, Jake Verschuyl, Thomas Stokely, Paul Hammond, Melissa A. Scherr and Matthew G. Betts, Plant diversity enhances moth diversity in an intensive forest management experiment, Ecological Applications, 27, 1, (134-142), (2017).
- Haili Yu, Nianpeng He, Qiufeng Wang, Jianxing Zhu, Li Xu, Zhilin Zhu and Guirui Yu, Wet acid deposition in Chinese natural and agricultural ecosystems: Evidence from national‐scale monitoring, Journal of Geophysical Research: Atmospheres, 121, 18, (10,995-11,005), (2016).
- V. Deblauwe, V. Droissart, R. Bose, B. Sonké, A. Blach‐Overgaard, J.‐C. Svenning, J. J. Wieringa, B. R. Ramesh, T. Stévart and T. L. P. Couvreur, Remotely sensed temperature and precipitation data improve species distribution modelling in the tropics, Global Ecology and Biogeography, 25, 4, (443-454), (2016).
- Mark J. Brewer, Adam Butler, Susan L. Cooksley and Robert Freckleton, The relative performance of AIC, AICC and BIC in the presence of unobserved heterogeneity, Methods in Ecology and Evolution, 7, 6, (679-692), (2016).
- Pavel Dodonov, José Carlos Morante-Filho, Eduardo Mariano-Neto, Eliana Cazetta, Edyla Ribeiro de Andrade, Larissa Rocha-Santos, Igor Inforzato, Francisco Sanches Gomes and Deborah Faria, Forest loss increases insect herbivory levels in human-altered landscapes, Acta Oecologica, 77, (136), (2016).
- Peter M.C. Harrison, Jason Jiří Musil and Daniel Müllensiefen, Modelling Melodic Discrimination Tests: Descriptive and Explanatory Approaches, Journal of New Music Research, 45, 3, (265), (2016).
- Felipe Martello, Fernando Andriolli, Thamyrys Bezerra de Souza, Pavel Dodonov and Milton Cezar Ribeiro, Edge and land use effects on dung beetles (Coleoptera: Scarabaeidae: Scarabaeinae) in Brazilian cerrado vegetation, Journal of Insect Conservation, 20, 6, (957), (2016).
- Amalia M. Harrington and Kevin A. Hovel, Patterns of shelter use and their effects on the relative survival of subadult California spiny lobster (Panulirus interruptus), Marine and Freshwater Research, 67, 8, (1153), (2016).
- Thees F Spreckelsen and Mariska Van Der Horst, Is Banning Significance Testing the Best Way to Improve Applied Social Science Research? – Questions on , Sociological Research Online, 21, 3, (1), (2016).
- Timothy H. Parker, Wolfgang Forstmeier, Julia Koricheva, Fiona Fidler, Jarrod D. Hadfield, Yung En Chee, Clint D. Kelly, Jessica Gurevitch and Shinichi Nakagawa, Transparency in Ecology and Evolution: Real Problems, Real Solutions, Trends in Ecology & Evolution, 31, 9, (711), (2016).
- Paola C. López-Duarte, F. Joel Fodrie, Olaf P. Jensen, Andrew Whitehead, Fernando Galvez, Benjamin Dubansky, Kenneth W. Able and Heather M. Patterson, Is Exposure to Macondo Oil Reflected in the Otolith Chemistry of Marsh-Resident Fish?, PLOS ONE, 11, 9, (e0162699), (2016).
- G. Berberich, A. Grumpe, M. Berberich, D. Klimetzek and C. Wöhler, Are red wood ants ( Formica rufa -group) tectonic indicators? A statistical approach, Ecological Indicators, 61, (968), (2016).
- Robert Courtney, Sally Browning, Tobin Northfield, Jamie Seymour and Robert E. Steele, Thermal and Osmotic Tolerance of ‘Irukandji’ Polyps: Cubozoa; Carukia barnesi, PLOS ONE, 11, 7, (e0159380), (2016). 2016 IEEE International Conference on Mechatronics and Automation, (2016).Peng Liu, Weiquan Huang and Fengzhao Sun174410.1109/ICMA.2016.7558827
- S. L. Cox, P. I. Miller, C. B. Embling, K. L. Scales, A. W. J. Bicknell, P. J. Hosegood, G. Morgan, S. N. Ingram and S. C. Votier, Seabird diving behaviour reveals the functional significance of shelf-sea fronts as foraging hotspots, Royal Society Open Science, 3, 9, (160317), (2016).
- Shulin Zhang, Ostap Okhrin, Qian M. Zhou and Peter X.-K. Song, Goodness-of-fit test for specification of semiparametric copula dependence models, Journal of Econometrics, 193, 1, (215), (2016).
- Sander Greenland, Stephen J. Senn, Kenneth J. Rothman, John B. Carlin, Charles Poole, Steven N. Goodman and Douglas G. Altman, Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations, European Journal of Epidemiology, 31, 4, (337), (2016).
- Marion Barbé, Nicole J. Fenton, Yves Bergeron and Peter Vesk, So close and yet so far away: long‐distance dispersal events govern bryophyte metacommunity reassembly, Journal of Ecology, 104, 6, (1707-1719), (2016).
- Nathan P. Lemoine, Ava Hoffman, Andrew J. Felton, Lauren Baur, Francis Chaves, Jesse Gray, Qiang Yu and Melinda D. Smith, Underappreciated problems of low replication in ecological field studies, Ecology, 97, 10, (2554-2561), (2016).
- Brian S. Cade, Model averaging and muddled multimodel inferences, Ecology, 96, 9, (2370-2382), (2015).
- Peter Groenendijk, Peter Sleen, Mart Vlam, Sarayudh Bunyavejchewin, Frans Bongers and Pieter A. Zuidema, No evidence for consistent long‐term growth stimulation of 13 tropical tree species: results from tree‐ring analysis, Global Change Biology, 21, 10, (3762-3776), (2015).
- John Fieberg and Douglas H. Johnson, MMI: Multimodel inference or models with management implications?, The Journal of Wildlife Management, 79, 5, (708-718), (2015).
- Mark S. Lindberg, Joshua H. Schmidt and Johann Walker, History of multimodel inference via model selection in wildlife science, The Journal of Wildlife Management, 79, 5, (704-707), (2015).
- M. V. Eitzel, J. Battles, R. York and P. de Valpine, Can't see the trees for the forest: complex factors influence tree survival in a temperate second growth forest, Ecosphere, 6, 11, (1-17), (2015).
- Matthias De Beenhouwer, Maarten Van Geel, Tobias Ceulemans, Diriba Muleta, Bart Lievens and Olivier Honnay, Changing soil characteristics alter the arbuscular mycorrhizal fungi communities of Arabica coffee (Coffea arabica) in Ethiopia across a management intensity gradient, Soil Biology and Biochemistry, 91, (133), (2015).
- Jakob Bro-Jørgensen and Joshua Beeston, Multimodal signalling in an antelope: fluctuating facemasks and knee-clicks reveal the social status of eland bulls, Animal Behaviour, 102, (231), (2015).
- F. J. Pérez-Barbería, J. Carranza, C. Sánchez-Prieto and Alistair Robert Evans, Wear Fast, Die Young: More Worn Teeth and Shorter Lives in Iberian Compared to Scottish Red Deer, PLOS ONE, 10, 8, (e0134788), (2015).
- Steven Hecht Orzack, J. William Stubblefield, Viatcheslav R. Akmaev, Pere Colls, Santiago Munné, Thomas Scholl, David Steinsaltz and James E. Zuckerman, The human sex ratio from conception to birth, Proceedings of the National Academy of Sciences, 10.1073/pnas.1416546112, 112, 16, (E2102-E2111), (2015).
- Daniel R. Uden, Craig R. Allen, David G. Angeler, Lucía Corral and Kent A. Fricke, Adaptive invasive species distribution models: a framework for modeling incipient invasions, Biological Invasions, 17, 10, (2831), (2015).
- Derek P. Crane and John M. Farrell, Muskellunge egg incubation habitat in the upper Niagara River, Journal of Great Lakes Research, 41, 2, (448), (2015).
- L. V. Madden, D. A. Shah and P. D. Esker, Does the P Value Have a Future in Plant Pathology?, Phytopathology, 105, 11, (1400), (2015).
- Elodie Allié, Raphaël Pélissier, Julien Engel, Pascal Petronelli, Vincent Freycon, Vincent Deblauwe, Laure Soucémarianadin, Jean Weigel, Christopher Baraloto and Sebastien Lavergne, Pervasive Local-Scale Tree-Soil Habitat Association in a Tropical Forest Community, PLOS ONE, 10, 11, (e0141488), (2015).
- Meike Hiermes, Marion Mehlis, Ingolf P. Rick and Theo C. M. Bakker, Habitat-dependent olfactory discrimination in three-spined sticklebacks (Gasterosteus aculeatus), Animal Cognition, 18, 4, (839), (2015).
- J. Pauwels, B. Taminiau, G.P.J. Janssens, M. De Beenhouwer, L. Delhalle, G. Daube and F. Coopman, Cecal drop reflects the chickens' cecal microbiome, fecal drop does not, Journal of Microbiological Methods, 117, (164), (2015).
- Lewis G Halsey, Douglas Curran-Everett, Sarah L Vowler and Gordon B Drummond, The fickle P value generates irreproducible results, Nature Methods, 12, 3, (179), (2015).
- F.J. Pérez-Barbería, S.L. Ramsay, R.J. Hooper, E. Pérez-Fernández, A.H.J. Robertson, A. Aldezabal, P. Goddard and I.J. Gordon, The influence of habitat on body size and tooth wear in Scottish red deer (Cervus elaphus), Canadian Journal of Zoology, 93, 1, (61), (2015).
- Marie-Jeanne Holveck, Anne-Laure Gauthier and Caroline M. Nieberding, Dense, small and male-biased cages exacerbate male–male competition and reduce female choosiness in Bicyclus anynana, Animal Behaviour, 104, (229), (2015).
- Claire S. Teitelbaum, William F. Fagan, Chris H. Fleming, Gunnar Dressler, Justin M. Calabrese, Peter Leimgruber, Thomas Mueller and Marco Festa‐Bianchet, How far to go? Determinants of migration distance in land mammals, Ecology Letters, 18, 6, (545-552), (2015).
- Jean‐David Moore and Martin Ouellet, Questioning the use of an amphibian colour morph as an indicator of climate change, Global Change Biology, 21, 2, (566-571), (2014).
- Xoaquín Moreira, Luis Abdala‐Roberts, Yan B. Linhart, Kailen A. Mooney and Akiko Satake, Effects of climate on reproductive investment in a masting species: assessment of climatic predictors and underlying mechanisms, Journal of Ecology, 103, 5, (1317-1324), (2015).
- Milan Chytrý, Alessandro Chiarucci, Valério D. Pillar and Meelis Pärtel, Transfer of scientific knowledge to practitioners: Do we need a reform of the journal policy?, Applied Vegetation Science, 17, 3, (609-610), (2014).
- Aris Spanos, Recurring controversies about P values and confidence intervals revisited, Ecology, 95, 3, (645-651), (2014).
- Michael Lavine, Comment on Murtaugh, Ecology, 95, 3, (642-645), (2014).
- Jarrett J. Barber and Kiona Ogle, To P or not to P?, Ecology, 95, 3, (621-626), (2014).
- Ken Aho, DeWayne Derryberry and Teri Peterson, Model selection for ecologists: the worldviews of AIC and BIC, Ecology, 95, 3, (631-636), (2014).
- Paul A. Murtaugh, Rejoinder, Ecology, 95, 3, (651-653), (2014).
- K. P. Burnham and D. R. Anderson, P values are only an index to evidence: 20th‐ vs. 21st‐century statistical science, Ecology, 95, 3, (627-630), (2014).
- John Stanton-Geddes, Cintia Gomes de Freitas and Cristian de Sales Dambros, In defense of P values: comment on the statistical methods actually used by ecologists, Ecology, 95, 3, (637-642), (2014).
- Perry de Valpine, The common sense of P values, Ecology, 95, 3, (617-621), (2014).
- Richard J Smith, The continuing misuse of null hypothesis significance testing in biological anthropology, American Journal of Physical Anthropology, , (2018).
- Alisha D. Davidson, Chad L. Hewitt, Donna R. Kashian and Andrew R. Mahon, Understanding Acceptable Level of Risk: Incorporating the Economic Cost of Under-Managing Invasive Species, PLOS ONE, 10.1371/journal.pone.0141958, 10, 11, (e0141958), (2015).
- Kyungeun Jang and Young Min Baek, How to effectively design public health interventions: Implications from the interaction effects between socioeconomic status and health locus of control beliefs on healthy dietary behaviours among US adults, Health & Social Care in the Community, , (2018).
