Toward rigorous use of expert knowledge in ecological research

,


INTRODUCTION
This paper addresses a number of core issues related to using expert knowledge in ecological research: the diversity of expert knowledge, its expressions and utility to research, and the rigor of the methods in its use.The intent is to help researchers increase their awareness of this v www.esajournals.orglargely untapped knowledge source and its potential uses, as well as to provide a rigorous methodological framework for extracting and applying expert knowledge.
Expert knowledge of ecology and natural resource management can include scientific and non-scientific knowledge (e.g., local ecological knowledge: Olson and Folke 2001; traditional ecological knowledge: Nakasuk et al. 1999).Here we focus on the technical expertise of knowledgeable practitioners.The use of this technical expert knowledge is becoming increasingly popular in a range of scientific disciplines.The appearance of several texts in the past decade suggests increasing interest in the methods for eliciting, evaluating, and applying expert knowledge.For example, the handbook of Ericsson et al. (2006) provides a comprehensive overview of the elicitation and application of expert knowledge.This work is complemented by texts that include generic discussions of elicitation methods and uncertainty (Ayyub 2001, O'Hagan et al. 2006), as well as applications in specific fields such as conservation ecology (Burgman 2005) and landscape ecology (Perera et al. 2012).In ecology as a whole, the use of expert knowledge has increased during the last 30 years.A search of the Web of Science for the terms expert knowledge, expert opinion, and expert judgment in the categories ecology, environmental sciences, environmental studies, forestry, plant sciences, soil science and zoology, illustrates this rise.Even when adjusting for the increased volume of publications in general, the number of articles that use these terms increased by a factor 200 from the early 1980s to the late 2000s.During the 2000s alone, their use increased by almost 40% (Fig. 1).
Typically, experienced scientists are considered experts.However, besides of these scientists, practicing ecologists and natural resource managers can accumulate a wealth of knowledge and become experts.In contrast to formal scientific knowledge, much expert knowledge is informal and undocumented, remaining hidden until it is expressed for a specific application (Boiral 2002).Moreover, experts tend to express their knowledge linguistically different from what is typical for empirical evidence, which makes it difficult to assess its variability, uncertainty, or accuracy (e.g., Johnson and Gillingham 2004).The informal and inexact nature of expert knowledge and how it is incorporated into ecological studies creates challenges to ascertaining transparency and repeatability in methods used to collect expert Fig. 1.Increase in the use of the terms expert knowledge, expert opinion, and expert judgment in the ecological literature from 1982 to 2011.Use is measured as the number of articles that mention expert knowledge in the categories ecology, environmental sciences, environmental studies, forestry, plant sciences, soil science and zoology per 10,000 articles; and is adjusted for the increasing number of publications during this 30 year period.v www.esajournals.orgknowledge (Johnson andGillingham 2004, Drew andPerera 2011).And even though the knowledge of experts is often used to inform scientific studies and to develop policies and practices, these contributions are seldom fully documented or acknowledged.Ultimately, these factors can lead to distrust of expert knowledge among researchers and decision-makers in resource management (Huntington 2000).
To address this distrust, standards for the elicitation and use of expert knowledge should be as rigorous as those that apply to the collection and use of empirical data (Davis and Ruddle 2010).Therefore the elicitation of expert knowledge must be a deliberate, well-planned component of the study (O'Leary et al. 2009), which includes careful consideration of the method for identifying and selecting experts.The process of knowledge elicitation should be systematic and meticulous to reduce inaccuracy and imprecision, and to increase reliability.Once the knowledge is elicited, it should be tested and validated to quantify its level of uncertainty.Finally, if integration with empirical data is a study goal, a valid method has to be chosen for this purpose.All these steps should be clearly documented in reports and publications, to make it possible to scrutinize the methods and to reproduce and validate the work.
The intent of this paper is to increase awareness within the ecological research community around various issues in the use of expert knowledge.We hope it will provide for a better understanding of the diversity of experts, their expertise, and their contributions to ecological research and help in deciding when and how to use expert knowledge.Furthermore, we offer a range of different approaches and methods to elicit and apply expert knowledge, which are likely to increase the rigor and quality of ecological research.We caution that this paper is not a comprehensive review of the use of expert knowledge in ecology.Instead, based on our experience, conceptual insights, and evidence from the literature, we provide a synthetic overview of experts, expert knowledge, and associated methodology that will serve as a primer for researchers in ecology.
Our exploration begins with an overview of expert knowledge-the spectrum of experts and their expertise and the different forms of its expressions-followed by an investigation of its roles in ecological research.Subsequently, we present a methodological framework, with details of different knowledge elicitation techniques.Finally, we discuss the sources of bias and uncertainty in expert knowledge, and offer methods to reduce or quantify those, as well as considerations for validating elicited knowledge.

DIVERSITY OF EXPERTS AND EXPERT KNOWLEDGE
We focus here on expert practitioners, because they represent the most abundant and so far untapped wealth of latent knowledge available to researchers in ecology.These expert practitioners are practitioners who gained expertise in a sub-discipline of ecology or natural resource management (e.g., population dynamics, specieshabitat relationships, agricultural systems or natural disturbance dynamics) through training and years of experience in applying their practical, technical or scientific knowledge to solve questions of natural resource management.Such practitioners become experts through deliberate practice (sensu Ericsson et al. 2006): After completing their formal or informal training, they intentionally practice for many years during which they receive regular, unambiguous feedback on their judgments.This allows experts to improve their knowledge, increase their reasoning skills and develop a special ability to solve domain-specific problems (Ericsson et al. 2006).Nevertheless, by definition, expert knowledge is subjective in the sense that it is based on the personal experiences of individual experts (Fazey et al. 2005).By expert knowledge, we mean the experts' personal beliefs that are in agreement with facts.However, we acknowledge that it is at times difficult to differentiate knowledge from personal opinion and judgment.Therefore, we use the term expert knowledge inclusively and mean it to encompass expert opinion and expert judgment.

A continuum of experts and expertise
In principle, experts in ecology and natural resource management can include anyone with substantive knowledge and experience relevant to a particular problem in these domains (Burgman et al. 2011a).Following this argument, the term expert can include hunters and gatherers utilizing local ecological knowledge gleaned from their own experiences (Gilchrist et al. 2005), societal elders who possess traditional ecological knowledge passed down through generations (e.g., Nakasuk et al. 1999, Santomauro et al. 2012), as well as scientists who conduct research and publish their knowledge formally (Ericsson et al. 2006).Clearly, a large spectrum of different types of experts exists.
Along with this spectrum of experts comes a diversity of expertise.Different levels of expertise have been acknowledged before, such as levels ranging from novice to expert in the Dreyfus model of skill acquisition (Dreyfus and Dreyfus 1986) or from amateur naturalist to scientifically trained local expert along the K (knowledge) gradient of Elbroch et al. (2011).However, we go beyond distinguishing these broad levels by investigating the variation of expertise among experts in more detail.Instead of viewing this expertise as varying between a few levels, we propose that expertise exists on a continuum, along which the characteristic traits of experts' knowledge and its utility for research differ.Characteristic traits include breadth of perspective, degree of critical assessment, and awareness of limitations of knowledge.In turn, these characteristic knowledge traits determine the utility of the expert knowledge for research and decision-making (Fig. 2).At one end of the continuum, experts emphasize direct experience in immediate surroundings (Murray et al. 2009), tend to be abductive (Bentley 1989) and stress continuity while acknowledging local variation (Lauer and Aswani 2010).Their knowledge can be useful for informing local observations (Murray et al. 2009), providing explanations for events and processes in their immediate context (e.g., Kovacs 2000) and generating ideas for testable, causal relationships (Bart 2006).At the other end of the continuum, experts use general features and patterns (Chase and Simon 1973), can adapt their knowledge to new situations (Fazey et al. 2005) and are mindful of their own uncertainty (Fazey et al. 2005).Their knowledge helps to generate hypotheses about ecological events and processes (Chalmers and Fabricius 2007), synthesize information to identify knowledge gaps (Morgan et al. 2001) and provide ecological insights in support of wise natural resources decision-making (Theobald et al. 2005).Of course, not all experts are found at the extreme ends of this continuum, many of them will be situated somewhere in the middle.In many cases it may be difficult or even impossible to assign specific degrees of expertise to individual practitioners and to distinguish traits of expert knowledge.Nevertheless, we believe that it is useful to think of experts in the terms described above, because it helps us to be cognizant of the diversity of expertise and to clarify the different functions that expert knowledge can have in ecological research.

Forms of knowledge expression
Experts can express their knowledge in different forms.These forms range along a gradient of increasing contextualization of knowledge (i.e., increasing integration and value assessment of individual pieces of knowledge; Fig. 3).At the one end of the gradient, experts can express specific knowledge about the presence of individual objects or occurrence of events, but a comparison among them does not necessarily occur.Knowledge expressions toward the middle of the gradient go beyond an indication of the existence of objects or occurrence of events and allow comparisons on a qualitative or quantitative level (Martin et al. 2005).At the other end of the gradient, experts can express synoptic knowledge that evaluates objects or events and the different forms of relationships among them that can form the basis of decision-making (Cook et al. 2010).
It is interesting to note that the gradient of increasing knowledge contextualization outlined above is fairly consistent with concepts employed in educational research, notably Bloom's taxonomy in the cognitive domain (Bloom 1956).Bloom stated that the acquisition of knowledge is a hierarchical process that starts with the collection of facts, proceeds towards the application of learned content to solve problems and finally arrives at the synthesis and evaluation of learned content (Bloom 1956).Similarly, our model would predict that some experts may have specific knowledge without having synoptic knowledge, but that all experts that have synoptic knowledge also must have specific knowledge.Of course, the way in which experts express themselves depends to some extent on the knowledge traits described above.For example, if concrete, local experiences are the focus of expert knowledge, then maybe the only possible knowledge expression is as an existential statement (i.e., the presence/absence of an object or not/occurrence of an event).However, if a broad perspective and logical thinking characterize the demands on expert knowledge in a given Fig. 3. Expression of expert knowledge in different forms ranging from specific to synoptic.The form of knowledge expressions depends on the degree of contextualization of the knowledge through integration and assessment of individual pieces of knowledge.Specific forms of knowledge expression are typically suited for contributions of a certain nature.v www.esajournals.orgcontext, then knowledge may be better expressed as an existential statement as well as an evaluative statement (i.e., comparing several possible decisions).

Utility of expert knowledge
For many ecologists the concept of expert knowledge is synonymous with an expert system, which attempts to model the decisionmaking abilities of a human expert for purposes such as natural resource management (e.g., Coulson 1987, Rykiel 1989) or environmental conservation (Burgman 2005).However, the value of expert knowledge in ecological research and applications has many facets.Expert knowledge can contribute to all stages of research (Fig. 4).For example, experts can conceive ideas that are grounded in practical experience; they can assist in generating contextually relevant hypotheses; they can deliver information for the parameterization of models used for system prediction; they can evaluate modeling results as a form of model validation; they can support extension activities by enabling the transfer of knowledge to non-expert users (Perera et al. 2012).If the main role of expert knowledge is the supply of information in support of ecological research, three functions of expert knowledge can be distinguished; it can serve as source of qualitative information, quantitative information, or provide insights and syntheses.
Expert knowledge as qualitative information.-Whenpublished knowledge is not readily available, as with exploratory studies or when rapid solutions are needed, experts can inform ecologists qualitatively (Table 1).For example, when ecologists conceptualize a study objective and refine specific research questions, they may acquire broad expert knowledge to support information from literature reviews or past research.In such cases, experts must draw upon the full breadth of their empirical and theoretical knowledge rather than specific observations.Such consultations are rarely recorded in the literature and therefore elude formal recognition, beyond the occasional mention of ''pers.comm.''Examples of qualitative expert knowledge include explorations of knowledge gaps or scoping of research questions (Table 2).However, qualitative expert advice is documented more often when it is used to explore broad aspects of knowledge.For example, Sutherland et al. (2009) surveyed experts to select the most important research questions that would, if answered, have the greatest effect on conservation practices of global biological diversity.Similarly, Wilson et al.Identifying priority areas for conservation management in the Cape Floristic Region, South Africa (Cowling et al. 2003) Predicting movement corridors for black bear in Alberta, Canada (Clevenger et al. 2002) Parameterizing a succession model in boreal Ontario, Canada (Drescher and Perera 2010a) Defining ecoregions using Bayesian mixture models in Queensland, Australia (Williams et al.

2012) Supplement
Predicting the species distribution of brush-tailed rock wallaby in eastern Australia (Murray et al. 2009) Identifying behavioral change in response to human presence for 26 bird species in Scotland, UK (Whitfield et al. 2008) Strengthening model predictions in a Bayesian framework (Martin et al. 2005)

Insights and decisions
Predicting effects of climate change on polar bear abundance and distribution in the Arctic (O'Neill et al. 2008) Informing strategic and tactical land management decisions for the recovery of woodland caribou (McNay 2012) Predicting movement corridors for black bears (Clevenger et al. 2002) Deriving hypotheses of reed grass invasion in salt marshes in New Jersey, USA (Bart 2006) v www.esajournals.org(2010) worked with experts to identify knowledge gaps in the understanding of climate change effects on coral reef fish.
Expert judgments are used when responses decision-making must occur quickly, as in the study by Cowling et al. (2003), where expert views were sought as part of the process of selecting areas for conservation reserves in Africa.Similarly, Grech and Marsh (2007) used experts to identify and rank human-related factors influencing Dugongs and their habitats in Australia.Some conservation management decisions in Australian reserves are based on experiential evidence when resources to collect and use empirical evidence are limited (Cook et al. 2010).Increasingly, qualitative elicitation is used to define the contextual framework of a study within which quantitative elicitation is performed (Fazey et al. 2006).
Expert knowledge as quantitative information.-Ecologistsfrequently encounter situations where empirical data are absent or limited (Table 1).For example, data may be inaccessible because the questions involve future events, the data may not be at the appropriate scale or level of detail or they may contain spatial or temporal gaps.Alternatively, collection of new data may be precluded by a lack of resources or by ethical concerns about negative effects of the observational methods on the study objects (e.g., Dugger et al. 2006).Though improved observational methods or statistical techniques may be used to solve some of these problems, expert knowledge can offer an efficient, alternative source of data (Drew and Perera 2011).In these instances experts use a combination of field observations, formal knowledge and mental models to generate quantitative information (Fazey et al. 2006).The use of expert knowledge as quantitative information is controversial.Experts are drawing upon personal observations, training and logic to estimate values, ranges, or the moments of distributions.This information is typically elicited through direct or indirect probabilistic statements of belief (Burgman 2005, Martin et al. 2011).
Expert knowledge is commonly used as a surrogate when empirical data are unavailable (Table 2).For example, ecologists have used expert knowledge to provide the quantitative basis for predicting the location and quality of wildlife habitat (e.g., Johnson and Gillingham (2004): caribou habitat in British Columbia, Canada; Smith et al. (2007): habitat of the ground dwelling Julia Creek dunnart in Queensland, Australia), species occurrences (e.g., Yamada et al. (2003): sambar deer in Victoria, Australia; Rothlisberger et al. (2009): fish stocks in the Great Lakes, North America), and species behavior (e.g., Hurley et al. (2009): vehicle collisions with moose in British Columbia, Canada; Pullinger and Johnson (2010): caribou movement corridors in British Columbia, Canada).
Expert knowledge can be used to complement empirical data (Table 2).This approach was recommended by Cowling et al. (2003) in the design of conservation areas in South Africa.In such cases, some knowledge gaps are filled with empirical data while other gaps are filled with expert knowledge.In another example, Drescher and Perera (2010a) parameterized a quantitative model of forest succession in Ontario, Canada, using empirical data as well as expert knowledge.By filling gaps in empirical data of forest succession with expert knowledge, the total space of available knowledge becomes wider.This approach may also be useful when exploring extreme events and processes beyond their normal range, as suggested by Franklin et al. (2008) for exotic species invasions or natural catastrophes.
Occasionally, expert knowledge can be a supplement to empirical data (Table 2).In these cases, research objectives are addressed through the application of empirical data, but expert knowledge enhances or strengthens the findings and conclusions.For example, Murray et al. (2009) combined expert knowledge with field data to construct a species distribution model for the brush-tailed rock-wallaby in Eastern Australia to improve the predictive capacity of the model.Whitfield et al. (2008) also used expert knowledge to supplement limited empirical information on estimates of the behavioral changes in 26 bird species in Scotland in response to human presence.Bayesian approaches are increasingly common, in which a prior parameter distribution based on expert knowledge may be updated with empirical data to arrive at an updated, posterior parameter distribution (Low Choy et al. 2009).
Expert knowledge as syntheses and decisions.-Expertinsight is an advanced form of expert knowledge, which can be expressed as syntheses or decisions (Fig. 4, Table 1), and can aid ecologists in scientific research as well as in the practical application of knowledge (Table 2).Fazey et al. (2006) summarize the unique and complementary value of experiential and experimental knowledge and promote the value of an experts' ability to synthesize ideas across these diverse information sets.Aspinall (2010) highlighted the utility of expert insight for solving complex problems when there is no simple agreement among experts and when decisions have profound and far-reaching consequences.Expert knowledge supports predictions about the occurrence of future events, particularly in new, changing, or poorly defined systems (e.g., Maddock and Samways 2000), or about the consequences of a specific decision or action (e.g., Gregory et al. 2006).The use of expert insight could become more prevalent as a result of the increasing concern about the uncertain consequences of climate change and possible adaptation measures.For example, O'Neill et al. ( 2008) provided experts with maps of predicted future sea ice extent and asked them to describe the effects of changing climate on the distribution and abundance of polar bears in the Arctic.Prutsch et al. (2010) used expert judgments to develop guiding principles for climate change adaptation in Europe.
When ecologists conceptualize a study objective and refine specific research questions, experts may supplement knowledge gathered from reviews of literature.The expert sources are rarely recorded in the literature and therefore elude formal recognition.However, expert insight is an important source of research hypotheses.For example, Drescher and Perera (2010b) used expert knowledge to formulate testable hypotheses of boreal forest succession in Canada and were able to test these with empirical data.In some cases, experts may perform as both domain experts and as decision-makers.In a study by Kennedy et al. (2006), biologists defined parameters but also expressed preference for management alternatives that maximize benefit to ecological systems.Such situations require that clear distinctions are made between these dual expert functions and great care should be taken to elicit knowledge separately from the decisionmaking process (Failing et al. 2004).

RIGOROUS METHODS FOR ELICITING AND USING EXPERT KNOWLEDGE
Resistance to the collection and application of expert knowledge is often based on philosophical grounds.Some practitioners and scientists believe that expert knowledge is non-testable, burdened with bias, onerous to collect and generally associated with less rigorous methods than empirical data (Dennis 1996, Seoane et al. 2005, Knol et al. 2010).We believe that rejection of expert knowledge by many ecologists is the result of their lack of exposure to the extensive theory and methods of the social sciences.This leads to the perception that the use of expert knowledge leads to soft science with less value than empirically-based studies.Unfortunately, examples of poor practice in the application of expert knowledge in ecology add to its mixed reputation (Brooks 1997, Johnson andGillingham 2004).Insufficient methodological consideration can lead to biased, uncertain or inaccurate results and improper generalizations.Poor practices threaten the validity of results from individual studies (Johnson and Gillingham 2004) and lessen the broad acceptance and utility of expert knowledge for understanding ecological processes and guiding decision making (Cook et al. 2010).
Given the objections to the use of expert knowledge, it is especially important that researchers apply rigorous designs when implementing expert-based studies or decisionmaking.Rigorous methods are repeatable and transparent to both study participants and to the users of study results.A rigorous method should also incorporate a measure of uncertainty for the elicited expert knowledge and should entail an assessment of the internal or external validity of findings.In the subsequent sections of this paper, we discuss appropriate and rigorous practices for working with experts and expert knowledge.We present a broad perspective on methods for (1) identifying, selecting, recruiting and retaining experts, (2) eliciting expert knowledge, (3) dealing with bias, uncertainty and aggregation of expert knowledge and (4) evaluating expert knowledge (Fig. 5).For more complete coverage of these topics we refer readers to the vast information available from the social sciences and increasingly the natural sciences (e.g.,

Identifying, selecting, recruiting and retaining experts
Gathering and applying expert knowledge should not be viewed as an inexpensive or easy alternative to collecting empirical data.Identifying experts is challenging in practice, especially because true experts are few and their availability may be limited.Furthermore, there is no consensus about how best to select, motivate and retain experts (e.g., Iglesias and Kothmann 1998).Below we provide major considerations to help with this challenge.
Identifying experts.-Acrucial step in using expert knowledge is deciding the type of expertise required, depending on the research question.Experts must be matched carefully to the research question because different experts harbor different kinds of expertise.This statement may seem self-evident, but efficient matching goes beyond simply recruiting an expert whose expertise is in the domain of inquiry.Fig. 5.A rigorously planned expert knowledge study consists of many steps.The main phases are the engagement of experts, the elicitation of expert knowledge and the assessment of the knowledge.Throughout the elicitation process measures can be taken to reduce knowledge biases and to aggregate knowledge from multiple experts.Effective use of experts in all phases relies on the careful matching of the type of expert with the required type of knowledge and the intended function of this knowledge.v www.esajournals.orgEfficient matching must be sensitive to aspects such as the required breadth of expertise, the contextualization of knowledge and awareness of uncertainties and limitations.Addressing these aspects a priori is essential to ensure a good fit between experts and the research question, and thus effective use of expert knowledge.
The starting point for an expert knowledge study is to identify an appropriate pool of potential participants (Murray et al. 2009).For this, careful preparation is required to define what expertise is needed and how an individual's degree of expertise will be determined.Although there is general guidance in the social sciences and risk analysis literature on expert identification and selection, the identification of experts for applications in ecology, conservation biology or resource management is an area of ongoing research (Burgman et al. 2011a, Martin et al. 2012).In general, there is no consistent definition of expert or standard method for identifying experts across projects (Sherry 2002, Knol et al. 2010).Appropriate identification of experts is dependent on a clear understanding of project scope and objectives.Where synthesis of knowledge for management or conservation decisionmaking is the goal, persons with well-contextualized, synoptic knowledge in that area are most desirable.If the mechanism of some ecological process is the focus of the work, then more specific domains of expertise and persons with very specialized knowledge would be required.
When assessing a pool of potential experts, the relevance of any individuals' experience must be judged in relation to the spatial and temporal scope of the study.Depending on the geographic specificity of the context, a research team may seek experts with knowledge of a particular region or area (Elbroch et al. 2011), particularly when expert knowledge may not transcend sitespecific processes (Doswald et al. 2007, Hurley et al. 2009).For many studies, a range of expertise may be needed to address different phases of the work.Broad knowledge may be necessary in early study phases to address uncertainty in system structure and function, while at a later stage the parameterization of a model may require the specialist knowledge.Toward the end of the study, well-contextualized and crossdisciplinary knowledge may be necessary to interpret the results and form policy recommen-dations.For example, parameterizing a model that predicts the distribution of a species at risk may require a different set of experts than those needed to develop adequate policy recommendations to conserve the identified habitat (Johnson and Gillingham 2004, Gregory et al. 2012).
The selection of experts can strongly influence whether a project reaches a successful outcome and garners wide acceptance.Developing a set of rigorous and defensible methods to identify and sample experts is analogous to defining a study population when collecting empirical data, but numerous past studies have failed to define the expert (e.g., Petit et al. 2003, McNay et al. 2006, Van der Lee et al. 2006).Ideally, a written set of criteria and interview questions to assess expertise should be developed a priori to the start of the study.Criteria for the identification of experts often centre on their experience with the study subject and objectives.For academic experts, experience may be indicated by the number of years of study of a particular species or ecological process or by authorship of relevant manuscripts.Scholarly databases can reduce the need for peer nomination and word of mouth referencing.Despite this search power, research teams still may not identify a sufficient number of experts to address the full range of knowledge areas (Kappel et al. 2011).For practitioners, experience may be demonstrated by years of relevant experience.Professional certifications can also indicate expertise and these are provided by some agencies (e.g., Ecological Society of America, The Wildlife Society).Some of these agencies maintain directories of certified members that may be accessible to identify experts.Researchers could also employ alternative indices of expertise such as involvement with expert panels or committees.For example, O'Neill et al. ( 2008) considered membership on the International Union for the Conservation of Nature (IUCN) Polar Bear Specialist Group as the qualifying criterion to participate in a survey of polar bear population dynamics under hypothetical climate futures.
Selecting experts.-If a large enough pool of qualified individuals is available, experts may be selected by random or stratified-random (e.g., for even geographic or agency representation) strategies, but non-random selection is more common.Sociological research (e.g., Palys 2008) employs several methods for selecting experts, few of which have appeared in the ecological or conservation literature.To date, most experts are selected by researchers based on convenience and availability, though breadth of representation is usually the stated goal (USEPA 2011).However, not all experts are academics or employees of management agencies and direct selection of these other experts may be difficult.In such a situation a peer nomination process could be employed.Chain referral sampling requires that researchers identify an initial expert and that this person then nominates additional experts (Lewis-Beck et al. 2004).To an extent, this sampling scheme may conflate the processes of expert identification and expert selection.And although easy to apply, it can lead to selection bias or underestimated knowledge variance because of peer nomination from specific groups of likeminded people (i.e., population clustering).Also, Burgman et al. (2011b) found that peer status is not always effective for indicating expertise for prediction and estimation tasks; instead they suggest a testing system for identifying qualified experts.
In the case of objective, factual knowledge, expertise in the relevant knowledge domain should be the main selection criterion.However, especially in a resource allocation context, the expert pool may be compromised because potential participants may be polarized by a social or political debate that is central to their expert contributions.In such cases, selection should be sensitive to the potential for motivational biases or strong and obvious contextual differences among experts that may influence their judgment.In such a situation, stratification of the expert pool is a critical step in the selection process.Axes of diversity that may contribute to effective group judgments include styles of reasoning, political persuasion, employment sector, gender, age (see Tetlock 2006) and race and ethnicity (Yancey et al. 2006).In many situations, experts are both stakeholders and experts, and naı ¨ve assumptions of expert objectivity may create substantial biases (Wintle and Cleeland 2012).The social context of stakeholders (i.e., people who may be affected by or may affect a decision) may be mapped on axes representing influence, interest or other relevant factors (e.g., Gilmour et al. 2011).These maps may be used to guide expert selection to ensure that the background and context cover the full space of influence and interest.Selection requirements, such as breadth of diversity and experience among experts could be important for political purposes, but also for statistical reasons, especially if expert knowledge is used to substitute for or complement empirical data.
Recruiting experts.-Motivatingexperts to participate requires effective communication and reward systems.Guidelines and reviews focused on improving citizen stakeholder participation in natural resource management (e.g., Reed 2008) and the literature on knowledge management (e.g., Holsapple 2003) and professional development (e.g., Evetts 2003) offer some valuable, general insights.When involving experts, researchers should ensure that candidates are approached early in the study, made aware of the expectations relative to time and effort commitment, effectively oriented about the elicitation process, fully informed about the required knowledge and promptly supplied with feedback and results.Multiple factors determine an individual's motivation, satisfaction, and success as an expert participant.However, expert motivation can often be enhanced by focusing less on their potential contribution to the study and more on what the study can contribute to them or their agency (i.e., appealing to their altruism towards their profession or community).Potential benefits from participation in an expert-based study may include early or privileged access to data products (e.g., annotated literature review or map data), an opportunity to broaden and deepen their knowledge of the topic, training in study methods, more effective collaboration due to knowledge of recent and ongoing studies, expanded professional networks for career advancement, an example of service to the profession or community, and improved knowledge in support of effective use of the study results.
Retaining experts.-Retention of experts may be as challenging a task as recruitment.The attrition or 'drop-out' rate of participants can be reduced by effective communication of study objectives and methods, as well as screening of experts.Where elicitation processes are lengthy or have multiple stages, commitment can be maintained or enhanced through periodic feedback, project summaries, or honoraria (Sherry 2002).A con-sistent expert pool will enhance the quality of information, maintain a sufficient sample size, and result in a more efficient elicitation process.Generally it is important to consult experts judiciously throughout the process; although expert knowledge may be essential at several stages of a study, it is unlikely that all experts need to be consulted through group workshops at every stage.
Both stakeholders and experts experience consultation fatigue, especially from high commitment and low satisfaction with study outcomes (Reed 2008).Fatigue may corrupt expert knowledge or make it unavailable, and erode the pool of available people.Some experts may feel anonymous and de-identified supporting the need for recognition, feedback, and reinforcement that the elicitation process has value.For many experts, time is limited, the elicitation and evaluation process is demanding and, in general, people tire of estimation (Valverde 2001).Many systems are complex; experts may be expected to estimate tens or even hundreds of parameters.There may be more than one plausible judgment, requiring several alternative models or submodels.Experts may also be lost from the study because people move and fail to leave contact details.Simple failure to locate expert participants can be a significant cause of attrition, but can be limited by effective tracking systems (Ribisl et al. 1996).
Attrition may damage the quality of group assessments because the losses may not be balanced with respect to the stratification criteria, and the attributes of those lost may differ in fundamental ways from those who persist (Ribisl et al. 1996, Robinson et al. 2007).However, in general, reasons for attrition are complex and successful retention relies on a suite of strategies.Some successful strategies include maintaining regular contact, scheduling of tasks and reminders, reinforcing study benefits and community involvement, avoiding unnecessary inputs and repetition (Robinson et al. 2007), and shaping interactions to be sensitive to cultural differences (Yancey et al. 2006).Retention of experts may involve providing enjoyable environs, reducing competing demands for time and enhancing positive perceptions about the impact of the work.Retention may also be increased by providing financial rewards, though this does not seem to be commonplace in ecological research, and if provided appears to be modest.Instead, travel and accommodation costs of expert participants may be reimbursed.This practice likely differs from the remuneration of experts by industry and government, though exact data for these sectors are difficult to come by.As an example, information published by the Canadian Nuclear Safety Commission shows reimbursement of scientific services at a rate of approximately $10,000 per month (CNSC 2013), suggesting the possibility of substantial financial reward for expert advice.
Web-based or other remote interactions typically are less demanding of time, but it is more difficult to maintain the focus of participants.Typically electronic communication is less enjoyable than face-to-face meetings, and it is more difficult to communicate the value of the work and engender a sense of enthusiasm for the outcomes (e.g., Jessup and Tansik 1991).Most people enjoy interactions and value their own contributions more if procedures are equitable, and if they feel heard and respected.Unfortunately, people treat each other differently based on personal characteristics and social power hierarchies.To avoid some individuals dominating the group setting, anonymous contributions can be used such as in electronic forums.However, there is evidence that anonymous, computer mediated interactions do not increase equality in communication between individuals of differing status (Christopherson 2007).Alternatively, dominance effects could be avoided by eliciting knowledge separately from individual experts.In volunteer-based systems generally, as well as in expert groups, financial reward does not improve individual or group performance much (Yancey et al. 2006).The most persistent contributions and the greatest effort in participation are derived from 'internal' motivators such as peer recognition, enhancement of status, career prospects, prospects of enjoyable interactions, participation in a positive environment and making a difference (Osterloh and Frey 2002, Benabou and Tirole 2006, Al-Ubaydli and Lee 2011, Shaw et al. 2011)

Eliciting expert knowledge
The elicitation of expert knowledge is arguably the most exciting part of working with experts.
v www.esajournals.orgComparable to empirical data collection, it can bring the researchers in close contact with the experts to elicit the knowledge.The elicitation requires effective communication between the researchers and experts and the implementation of well-designed plans that guide expert interaction and the expression of their knowledge.
Communicating elicitation objectives and encoding schemes.-Researchersshould develop rigorous methods specific to their study objectives, which will help them communicate their study to the experts efficiently and to capture expert knowledge using appropriate encoding schemes (Knol et al. 2010).During the elicitation process, the researchers work in-person or remotely with the experts to explain key definitions; refine and explain research questions; explain and adapt the methods for collecting, encoding, and validating the relevant expert knowledge.A poor understanding of the process and methods by researchers or experts can result in the collection of knowledge that is biased or inaccurate and ultimately fails to address the study objectives (O'Hagan et al. 2006).It is therefore important to invest sufficient forethought and time into developing and testing the elicitation process.
The encoding process translates expert knowledge into explicit terms that can be analyzed or used in models.Some elicitation processes directly integrate coding as a component of the questioning process, while other studies, such as the ones that aim at expert-based decisions or evaluation, may not require any encoding.Researchers in the ecological sciences are actively developing software tools to assist with encoding and verifying expert knowledge (James et al. 2010).Several methods for eliciting and encoding expert knowledge exist that vary by: (1) the degree of interaction and consensus building among experts, (2) the structural frameworks for enforcing consistent and explicit documentation rules, and (3) the techniques that ensure logical parameterization and allow confirmation whether basic assumptions are satisfied.Advice on good practices for conducting elicitation (Cooke 1991, Low Choy et al. 2009, Kuhnert et al. 2010, Burgman et al. 2011b) and methods for encoding is available.A general guiding principle is that elicitation and coding must be well understood by all study participants and well documented for subsequent application of the study results.
Interaction among experts.-Thelevel of interaction and knowledge sharing among study participants during elicitation can influence study results.The concept of 'group think' and the role of dominant personalities within groups are well understood (MacDougall and Baum 1997).However, it is possible that a shared or consensus perspective can develop when experts are asked to openly discuss knowledge provided by others and in some contexts this is the desired outcome.For example, experts may be asked to identify the scope of a project, define project objectives, or develop management decisions or conservation actions based on their experience and institutional perspective (Orsi et al. 2011).In such cases participant buy-in and collaborative action are the primary objectives and success of the process depends on effective interaction and communication (Gregory et al. 2012).
When knowledge serves as a surrogate for observations, elicitation should be designed to represent the central view of experts as well as the variation among experts (Murray et al. 2009).In large projects with many experts, group settings are often necessary to allow the research team to efficiently interact with participants.If carefully planned and well facilitated, group elicitations can save significant time and money.However, through the course of a study, it is likely that different study components may require different elicitation methods.For example, a study may commence with a group elicitation to reach consensus on the scope, objectives, and ecological variables for consideration, but might be followed by individual surveys to elicit subject area information to meet specific data needs.
Elicitation approaches.-Allelicitation methods need to manage interactions among experts, to accommodate human factors and the psychology of group deliberations.The Delphi approach is commonly used to develop group judgments related to questions of conservation or management.In this method, an expert group completes a survey instrument, the results are collated and circulated among experts, and then the experts are asked to revise their answers based on the group's central perspective.While some implementations use several revision rounds to reach a consensus (e.g., Linstone and Turoff 1975), consensus is not an essential element.Typically, expert anonymity is maintained through written interaction, but more dynamic interaction can be achieved through modified Delphi methods with facilitated discussion (Sherry 2002, Burgman et al. 2011b).
Generally, group performance may be enhanced by maximizing the diversity of groups (Page 2007), and managing their interactions to avoid the most pervasive psychological causes of bias, particularly dominance effects, overconfidence and group-think.Although Delphi processes are well established in the ecological literature (Kangas and Leskinen 2005), applications vary widely and in some circumstances, insufficient care is taken to manage group dynamics, document dissention or communicate uncertainty.Nevertheless, if properly implemented, Delphi techniques can be used to assemble estimates of parameters together with their uncertainties, avoiding group-think, anchoring and other elements that can mar expert deliberations.They can be combined with other techniques that test expert knowledge and weight judgments appropriately (Cooke 1991, Aspinall 2010, Burgman et al. 2011b).
A range of other elicitation techniques can explicitly encode knowledge while maintaining the identity of individuals, including measures of uncertainty across experts.Each has strengths and weaknesses.For example, the analytical hierarchy process has often been used to solve applied ecological problems (e.g., Clevenger et al. 2002, Doswald et al. 2007, Hurley et al. 2009) and allows formalization of the decision-making process, including relative scaling of parameters and checking of the logical consistency of values.Using this method, experts order a set of hypothesized categorical variables, which they may have collectively adjusted based on their expert knowledge.Ordering occurs by pairwise ranking of each possible variable combination (Saaty 1977).The order of variables indicates the strength of their influence, which can be employed in various forms (e.g., geographic information systems) to represent expert knowledge (Eastman et al. 1995).Uncertainty can be indicated by comparing variable rankings among experts and consistency ratios can be calculated that reveal illogical rankings of variables.However, this method is impractical when considering a large number of variables (i.e., many pairwise comparisons) and the use of ordinal scores can make conversion to more general measures, such as probabilities difficult.
Recent advances in the application of Bayesian methods have led to considerable innovation in expert knowledge elicitation and aggregation methods (O'Hagan et al. 2006, Albert et al. 2012).Using Bayesian methods, empirical data and expert knowledge are formally integrated in a general statistical framework.Typically, expert knowledge is used to construct prior parameter probability distributions, while in parallel the parameters are also independently estimated based on empirical observation.The prior probabilities and the empirical estimates are then combined to generate a posterior parameter probability distribution that represents expert knowledge as modified by the data.When empirical data are absent, but likely to be collected in the future, Bayesian belief networks can be useful to represent expert knowledge as hypotheses (e.g., Drew and Collazo 2011).Uncertainty and sensitivity analyses of the belief networks can then guide sampling designs for the collection of empirical observations to update prior probabilities into data-informed posterior probabilities.
Bayesian methods require specialized software for developing the mathematical relationships (McNay et al. 2006, James et al. 2010) and while these methods lead to sophisticated integrations of knowledge and data, they are not without controversy.For example, Dennis (1996) criticized Bayesian studies for not being open to falsification, for confusing conclusions with decisions and for opening science up to the influence of subjective opinions.The application of Bayesian methods does not negate the need for structured protocols to engage with experts and elicit their estimates.Sophisticated statistical methods cannot replace the need for careful consideration of the evidence.Fortunately, though ad hoc methods can be found (McNay et al. 2006), some practitioners of Bayesian approaches have demonstrated much innovation in developing effective and rigorous methods for elicitation and encoding (Low Choy et al. 2009), which reduce many potential biases (Kadane and Wolfson 1998).For example, Murray et al. (2009) provided experts with graphical tools to explore and test their knowledge of habitat factors v www.esajournals.orginfluencing the distribution of brush-tailed rockwallaby in Australia.While the accuracy of expert knowledge often remains unknown, recent efforts have focused on developing standalone tools that formalize the generation of priors and simultaneously allow experts to explore their own logic and assumptions (Low Choy et al. 2011).

Bias, uncertainty and aggregation of expert knowledge
Biases are inherent in people and experts are no exception.A variety of biases can be distinguished and approaches are available to minimize them.While bias can be affected by sample size, a small number of experts does not necessarily lead to larger bias and a consideration of an appropriate sample size is important.Multiple experts almost certainly will not fully agree on a topic.This may make it necessary to aggregate individual knowledge statements with a suitable method, if a single knowledge statement is desired.Additionally, expert disagreement is one of the sources of uncertainty.Identifying these sources and quantifying uncertainty are necessary steps in determining the reliability of the elicited knowledge.
Expert knowledge bias.-Avariety of biases can affect the accuracy of expert knowledge, (Kangas andLeskinen 2005, Kuhnert et al. 2010).If recognized, measures are available to limit the effects of bias (e.g., Kuhnert 2011).A sampling bias can occur when experts that are included in the elicitation are not fully representative of the entire population of experts (Henry 2009).This bias may occur because experts share some characteristics with so-called 'hard-to-reach' and 'hidden populations' (Singer 1999), which are difficult to access because of their (un)intentional concealment.Similarly, experts may be difficult to recognize and approach unless they organize in interest groups or publicize products of their expertise that are searchable in some manner.Even if some experts organize in this way, others will not and those that are most visible may be so for reasons other than their expertise.Selecting from such groups will likely be affected by a selfselection bias, leading to an expert sample that may not be representative of the entire expert population (Mitchell and Jolley 2010).If the type and extent of the sampling bias is known, an uneven weighting approach can limit its effect on the study outcomes (Aspinall 2010).
Alternatively, various forms of non-probability sampling may counter sampling bias.Meyer and Booker (1991) recommended chain referral (see Identifying, selecting, recruiting and retaining experts: Selecting experts), which is also referred to as 'snowballing' (Goodman 1961, Streeton et al. 2004).The selection bias that can result from population clustering can be mitigated to some extent by beginning the snowballing process at many independent starting points to ensure that relatively isolated subtrees of experts are not overlooked (Erickson 1979).Network graphics provide a visual representation of the different waves of snowball nominations and a cluster analysis of expert nominations can provide an indication of any sub-clustering resulting from the reputation of expertise (Christopoulos 2007).Recently, methods have been developed to mathematically compensate for non-random, social network biases (Heckathorn 1997, Salganik andHeckathorn 2004).This work has shown that a variation of snowball sampling, respondentdriven sampling, is able to produce asymptotically unbiased results.
Additional biases often occur after a group of experts has been recruited, during the elicitation phase of the study (Tversky and Kahneman 1974).Some of the most important types of biases are motivational, behavioural and cognitive.Cleaves (1994) and Meyer and Booker (1991) provide overviews of these biases and strategies to minimize their effects.Important motivational biases include groupthink (Janis 1971) and impression management.In these cases, experts succumb to the social influence and expectations exerted by other expert group members or by the researcher.Avoiding value-laden questions and encouraging anonymous knowledge expressions can help minimize these biases.Misrepresentation and misinterpretation are important behavioural biases.These biases can occur when knowledge transfer is hampered by disconnects between the knowledge frames of the researcher and the expert (Brugnach et al. 2008) or when the knowledge encoding does not accurately capture the expert knowledge statement or its characteristics (Sage 1987).Applying objective, value-free coding schemes and involving experts in the development of the coding scheme can limit these biases.Finally, anchoring, overconfidence and inconsistency are frequent cognitive biases.The unifying feature of these biases is a deficient evaluation of evidence; causes include a human preference for familiarity (Moore and Miles 1991), insufficient (use of ) feedback (Fischer and Budescu 2005) and limited processing capacity (Halford et al. 2007).It may be possible to control these biases by providing continuous and immediate feedback to experts about their use of available evidence (Knol et al. 2010).
Sample size.-Because the likelihood of sampling bias increases with decreasing sample size, the number of experts that constitutes an appropriate sample size is also a consideration.
A key question is how many experts are necessary to provide sufficient breadth of knowledge to capture the parameters of interest and associated uncertainty.The answer to this question depends on the study objectives and the type of elicitation process.Sutherland et al. (2013) involved 388 participants in a study to generate fundamentally important questions in ecology.For group-based methods, the available guidance suggests that effective group dynamics and profitable discussions can occur with up to 12 (Cooke and Probst 2006) or even 15 participants (Aspinall 2010).However, no formal guidance is available for ecological and environmental studies, which often require sampling of knowledge that addresses broad spatial and temporal dimensions.Some studies have relied on as few as one (Seoane et al. 2005) or two experts (Clevenger et al. 2002) and it is common to use less than ten participants (e.g., Smith and Wilkinson 2002, Pollock et al. 2007, Murray et al. 2009, Pullinger and Johnson 2010).Especially in cases when exceptionally local knowledge is required, only very few individuals may posses this knowledge and sample size may be irrelevant as long as one knowledgeable expert is involved (Bart 2010).
Given that the knowledge characteristics of experts are unknown before they are sampled, a calculation of necessary sample size before initiating sampling may be impossible.Adaptive interim analysis is a statistically defensible approach to sample size adjustment in the course of an ongoing study (Bauer and Ko ¨hne 1994).The main value of this method is that it allows adjustments of sample size or study design without having to sacrifice data collected in a pilot study (Neuha ¨user 2001).An alternative approach to adaptive interim analysis is sequential sampling (Krebs 1989).Using this approach, data are collected and sample size increased until a decision can be made about rejecting or not rejecting a hypothesis.Sequential sampling is similar to an approach often employed in the social sciences, which is based on the concept of data saturation.Data saturation is reached when sampling further participants does not add any new information, at which point sampling is ended (Guest et al. 2006).Given the difficulties of recruiting large numbers of experts, it is preferable not to have to discard any expert knowledge collected in a pilot study.Consequently, approaches such as adaptive interim analysis and sequential sampling may be attractive alternatives to conventional sampling strategies.
Expert knowledge uncertainty.-Knowledge is always imperfect.The various characteristics of the object to be known and the means we use to uncover and communicate the related knowledge contribute to uncertainty.Quantifying this uncertainty is a critical step in the elicitation of expert knowledge.Several types of uncertainty are distinguished (Morgan and Henrion 1992), as is the case for epistemic uncertainty and aleatory uncertainty (e.g., Hora 1996).Epistemic uncertainty is due to a lack of empirical knowledge and can be reduced by collecting more information.Aleatory uncertainty is due to the inherent stochasticity of the system and cannot be totally eliminated.In most cases within-expert uncertainty will comprise both epistemic and aleatory uncertainty.
Linguistic uncertainty involves imprecise communication and includes vague, underspecified and ambiguous terms and constructs (Elith et al. 2002, Regan et al. 2002, McBride and Burgman 2011).Where structured elicitation is used, epistemic and linguistic uncertainty can be evaluated and minimized.For example, epistemic uncertainty may be assessed through selfevaluation by experts, where they provide the outer limits of some estimate of a probability of occurrence, credibility intervals, or a qualitative estimate of their confidence in an estimate (Sutherland 2006, Van der Lee et al. 2006, Drescher et al. 2008, Murray et al. 2009).Linguistic uncertainty is addressed most easily through the design of a transparent elicitation process and the application of appropriate treatments when examples arise (Regan et al. 2002).Experts should understand key definitions including methods of quantification.The researchers can pretest the elicitation process with non-participants or solicit a review by an elicitation expert.Open and continuous communication and feedback between researchers and experts can identify and correct unclear language and ensure that knowledge collection remains consistent with the objectives (Carey and Burgman 2008).Although there is no direct way of reducing aleatory uncertainty, it is possible to capture this uncertainty as part of the uncertainty model and include it explicitly in the study results (e.g., Parry 1996).Monte-Carlo and other simulation approaches are useful for representing the uncertainty in expert knowledge as applied to prediction (Burgman et al. 2001, Johnson andGillingham 2004) and in some cases, expert knowledge can be used post hoc to assess the uncertainty inherent to an existing ecological model (Van der Lee et al. 2006).
It is generally advisable to use more than one expert in an expert-based study (see Sample size).However, these experts will not perfectly agree with each other and may even contradict one another, which can be referred to as source conflict (Smithson 1999).Sometimes this amongexpert uncertainty is acknowledged (Iglesias and Kothmann 1998, Drescher et al. 2008, Hurley et al. 2009), though most expert knowledge studies do not partition the different sources of uncertainty and only report aggregated expert knowledge.Nevertheless, among-expert disagreement can provide valuable information about the characteristics of the studied system (Uusitalo et al. 2005), research needs (Morgan et al. 2001) and about the nature of expert knowledge itself (Iglesias andKothmann 1998, Drescher et al. 2008).
Aggregating expert knowledge.-Inmost expert knowledge studies, knowledge from multiple experts is aggregated to reduce uncertainty and bias.While conceptual problems about the meaning of group judgment persist (Garthwaite et al. 2005), in practice, knowledge aggregation often has to occur for practical reasons and can be achieved in various ways (Clemen and Winkler 1999).However, only by maintaining discrete expert perspectives is it possible to fully understand uncertainty in expert knowledge and potentially partition that uncertainty based on parameters such as geography and institutional background (Doswald et al. 2007, Hurley et al. 2009, Murray et al. 2009, Kappel et al. 2011), type of ecological process (Drescher and Perera 2010a) or degree of expertise (Fazey et al. 2006, Czembor and Vesk 2009, Hurley et al. 2009).This is not to say that group-based approaches for eliciting knowledge are necessarily ineffective or always generate biased information.Burgman et al. (2011b) reported that an expert's consistency of knowledge with reality increased substantially when participants were allowed to consider other expert's knowledge or discuss the questions within a structured process.Furthermore, not all group elicitations need to seek consensusindividual knowledge can also be collected within group settings.Regardless, reporting and using exclusively aggregated expert knowledge might create an undue impression of certainty despite possible knowledge uncertainty.This false impression of certainty should be avoided (Czembor and Vesk 2009).
If aggregation procedures are employed, researchers should strive to produce a consensus result (measure of central tendency) as well as a measure of uncertainty (measure of variability).Aggregation approaches can be classified as behavioural approaches and mathematical approaches.Behavioural approaches aim to create a consensus among experts, before any mathematical processing of the various knowledge statements.Commonly used techniques include the Delphi method (e.g., Linstone and Turoff 1975), the Nominal Group technique (e.g., Delbecq et al. 1975) and Cooke's method (Cooke 1991).The techniques are similar in that they usually consist of multiple rounds of elicitation interspersed with evaluations of the knowledge statements.They largely differ in whether knowledge statements are anonymous or discussed face-to-face and whether an initial group discussion occurs.Regardless, these techniques are susceptible to the motivational biases noted earlier (see Expert knowledge bias).Mathematical approaches are based on two different theories: probability theory or fuzzy set theory.
The approaches based on probability theory are split into those employing a frequentist or a Bayesian perspective (Clemen and Winkler 1999).Bayesian approaches include new techniques that apply hierarchical models that account for a range of sources of information as well as potential dependence among expert judgments (Albert et al. 2012).Using a Bayesian perspective, expert knowledge can be used to define a prior distribution that can be updated with other information to arrive at a posterior distribution (Low Choy et al. 2009).Applying a frequentist perspective, the most common method to aggregate expert knowledge is pooling several statements through averaging (Scholz and Hansmann 2007).This averaging can be unweighted, implying equal expertise for all experts, or weighted, in which the unequal weights represent different levels of expert credibility (Aspinall 2010).Fuzzy set theory (Zadeh 1965) can also be used as the basis for aggregating expert knowledge statements.For example, Delgado et al. (1998) used fuzzy relations to fuse numerical and linguistic values and represent the preference of the majority of experts in their study.
While some authors suggest that behavioural and mathematical approaches may perform equally well (Clemen and Winkler 1999), others argue that the choice of the best approach depends on the type of knowledge to be elicited (Rowe 1992): numeric estimation may be best approached with mathematical techniques, while procedural or complex knowledge may be better approached with behavioural methods.In practice, knowledge aggregation often will encompass both approaches (Clemen and Winkler 1999).

Expert knowledge validation
The final step in the elicitation process is an assessment of knowledge validity, which is closely linked to knowledge accuracy.The accuracy of expert predictions can be low, can vary widely among experts and may not necessarily be an improvement over that of novices (McBride et al. 2012).Although validation is a critical step for knowledge elicitation, few studies have addressed it.Approaches that improve the rigor of expert judgments include widening the set of experiences and skills that define an expert, employing structured methods to engage with experts, and making experts more accountable through empirical testing of their performance and through training (Cooke 1991, Bolger andWright 2011).Despite these steps, accuracy and validity of expert knowledge must not simply be assumed, but an assessment of absolute or relative validity of expert knowledge should be an important consideration for all studies.This is especially the case where expert knowledge serves as a surrogate for empirical observations, for example in the context of management or conservation decision-making.
Validation provides some confidence that expert knowledge accurately represents the ecological process of interest.In some cases, predictions from an expert-based model can be validated by comparison to empirical data where the empirical data is assumed to be reasonably precise and unbiased.For example, Anado ´n et al. ( 2009) compared the estimates of the abundance of tortoise provided by local experts to empirical counts collected using linear-transect surveys.In some cases, expert knowledge can be evaluated as hypothesis by comparison to experimental outcomes (Bart 2006).In other cases, predictions from expert-based models are validated by comparison to parallel model outcomes generated using empirical data (Clevenger et al. 2002, Pullinger and Johnson 2010, Drescher and Perera 2010a, b, Iglecia et al. 2012).However, many applications of expert knowledge are a response to a lack of independent empirical data and in these cases researchers must seek alternative means to assess validity.
In the absence of independent empirical data, validation may default to a test of plausibility (e.g., face validity sensu Pitchforth and Mengersen 2013).For example, if an elicited knowledge statement does not conform to expectations based on any fundamental rules or laws, then this statement does not pass a very basic test of plausibility.Tests such as this can be integral parts of the knowledge elicitation as in the study by Fischer et al. (2012), who used this approach to ensure consistency between expected confidence limits and observed ranges.Another option is to compare the results of similar studies across multiple areas.In this case, plausibility is affirmed when study results are similar among regions.Such comparisons do not test the predictive accuracy of the expert knowledge, but do validate the usefulness of the study's outcomes for application to other locations v www.esajournals.org(Kappel et al. 2011).If other studies are not available for comparison, it might be possible to exclude some experts initially and then to provide an independent expert peer review of the final product; this approach, however, assumes that the second set of experts is reasonably precise and unbiased, which in itself is uncertain.In cases where future outcomes are uncertain or unknown, expert-based guidance can be tested through monitoring or active adaptive management experiments.

CONCLUSIONS
The use of expert knowledge for the purposes of ecological research has a long history and has gained momentum during the last decades.However, it is not uncommon for the scientific community to be wary of or to disregard this practice completely.This mistrust is partially based on an insufficient understanding of the range of expertise and the breadth of its utility to the research process.Lack of methodological rigor in some research studies may have also contributed to such reservations.However, the vast amount of knowledge held by experts, especially the practitioners, is too valuable for researchers to continue to ignore.As we showed here, the spectrum of experts and potential utility of their knowledge is broad and diverse.Most weaknesses inherent to expert knowledge are with its usage, which can be mitigated by judicious and explicit steps in the process: choice of experts, eliciting their knowledge, explicit assessment of uncertainties, and validating results.Fortunately, researchers now have at their disposal advanced methods that help elicit, analyze, validate and apply expert knowledge, providing ready access to an underutilized resource that could also be relatively inexpensive and time-efficient.Only when practiced appropriately, can the use of expert knowledge in research be equally valid as the use of empirical data.The responsibility for this task, i.e., using expert knowledge rigorously, lies with the researchers.

Fig. 2 .
Fig. 2. Variation in the characteristics of expert knowledge in terms of knowledge traits and utility for research.The characteristics vary along a continuum from local focus to global outlook that is based on the scale of the experiences and the degree of conscious reflection on these experiences.

Fig. 4 .
Fig. 4. Expert knowledge has many facets and can contribute to all stages of the research process and natural resource decision-making.

Table 1 .
Types of expert knowledge contributions with common reasons for their use and general examples.

Table 2 .
Types of expert knowledge contributions with example applications.