Expert consensus-building for developing guidelines: lessons learned from a dengue economics workshop


Unificación de las opiniones de los expertos para la formulación de directrices: enseñanzas extraídas de un taller sobre los aspectos económicos del dengue



Dagna Constenla; Noah Lefcourt; Cristina Garcia

Department of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America. Send correspondence to: Dagna Constenla, email: dconsten@jhsph.edu




A workshop with 20 experts of diverse backgrounds from five countries in the Americas was convened for two-and-a-half days in March 2012 to discuss and develop a standardized methodology for assessing the economic cost of dengue. This article discusses a number of factors that contributed to the workshop's success, including: engaging the experts at various stages of the process; convening a multidisciplinary group to reduce expert bias and provide a more comprehensive and integrated approach; facilitating guided small- and large-group discussions; developing effective cross-cultural collectivism, trust, communication, and empathy across the expert panel; establishing clear lines of responsibilities within each group of experts; breaking down the complex issues into smaller and simpler ideas; providing ample background materials in multiple languages prior to the workshop. Challenges and areas for improvement are also covered.

Key words: consensus development conferences as topic; dengue, economics; guidelines as topic.


Se convocó un taller con 20 expertos provenientes de diversos ámbitos de cinco países de las Américas con el propósito de analizar y elaborar un método normalizado de evaluación del costo económico del dengue; la duración del taller fue de dos días y medio en marzo del 2012. En el presente artículo se analizan diversos factores que contribuyeron al éxito del taller, entre ellos la vinculación de los expertos en diversas.

Plabras clave: conferencias de consenso como asunto; dengue, economía; guías como asunto.



Guidelines for measuring the cost of disease are not well established. There are important reasons for this, the principal ones being the paucity of studies on disease costing-especially those that permit scientific generalization-and the heterogeneity of evidence generated by the available studies.

In Europe and North America, there are some countries with healthcare systems in which formal processes have been established around economic evaluation guidelines. The 1996 publication of the conclusions of a working group (the Washing Panel) funded by the United States Public Health Service marked an attempt to standardize the methodological approach to economic evaluation in healthcare (1). On-going efforts by governments and healthcare systems to harmonize approaches to economic evaluation, however, have made little progress since the publication of this working group.

In Latin America and the Caribbean, the situation is no different. With governments and healthcare systems continuing to take an independent line on evaluation, the scientific community is left in a difficult position, especially regarding the international aspect of its research activities. Problems result when economic evaluations are compared and there is little commonality among the underlying theories of economic costs, with limited data systematically collected for this purpose. These challenges are compounded by the fact that there is little consensus on which guidelines to adopt. Studies are needed that employ a standard methodology to measure the cost of disease in Latin America and the Caribbean. Such studies are one way to inform policymakers on the economic consequences of disease.

In this paper, we look at our experience with developing standards for costing dengue through consensus building in Latin America and the Caribbean, and describe the process of conducting a workshop in a multicultural setting (Figure 1). We conclude with recommendations on how to address conceptual and practical issues of expert consensus-building. The intension of this paper was not to provide guidelines for estimating dengue costs, but rather to describe a way of improving the process of developing guidelines. The guidelines that were developed as part of this consensus-building exercise are discussed elsewhere (2).




A workshop with 20+ experts of diverse backgrounds from five countries in the Latin America and the Caribbean was convened for two-and-a-half days in March 2012 to discuss and develop a standardized methodology for assessing the economic cost of dengue. The approach was to acquaint the workshop participants with existing dengue-costing methodologies prior to the workshop; to have each expert share the methodology they were currently using; to compare and contrast the methodologies; to prepare strategic options for harmonizing differences among the methodologies; and to produce a set of dengue-costing guidelines for Latin America and the Caribbean. The workshop combined self-supported material, small group discussion, and large group discussions to achieve its objective.

Expert panel

The panel was composed of the following: (i) experts from multidisciplinary backgrounds (health economists, epidemiologists, clinicians, dengue experts, program managers, and policymakers); (ii) experts representing low- and middle-income dengue-endemic countries in Latin America and the Caribbean; and (iii) experts affiliated with various institutions (universities, ministries of health, and private organizations). This multidisciplinary group helped to ensure adequate discussion of the evidence when building consensus around costing guidelines. The main challenge was to find a multidisciplinary group that would represent various countries, methodologies, and expertise.

Pre-workshop materials

While the structure of the workshop would be new to most of its participants, the content was drawn from existing sources and adapted to the specific needs of the workshop. This required the writing of linking material and the development of some new background material, including a literature review and analysis of expert survey responses.

Literature review. A literature review was conducted to provide a critical overview of the issues related to dengue economics research and to form a background with which to address the question of cost analysis of dengue in the Latin America and Caribbean context. To achieve these goals, several objectives were set: (i) to summarize what is known about the cost of dengue; (ii) to identify appropriate methods used to assess the costs of dengue; and (iii) to identify and recognize gaps in and pitfalls of previously published research in this area.

A total of 28 papers were identified during the search. The studies reviewed indicated great variation in cost estimates for dengue within and across countries due to differences in dengue classification, definition of cost categories, sampling, data sources, discount rates, different health care financing systems, delivery systems, and conversion into U.S. dollar values. Findings of the literature were not of sufficient quantity and quality. The current evidence suggests that the cost of dengue is substantial due to cost of hospital care and lost earnings. Notwithstanding these shortcomings, we found a relatively strong mix of methodologies, evenly spread across the two geographic areas. Further information about the literature review is discussed elsewhere (3).

Expert survey. In addition to the literature review, a survey was sent to the experts prior to the workshop to collect information that would be important to the workshop's focus and discussion. The survey was designed to identify areas in a cost of dengue analysis considered "high priority" by workshop participants; to identify gaps between cost of dengue methodology and practice; and to assist in determining the agenda and small group exercises.

A total of 17 questions listing specific areas of cost evaluation were included and experts were asked to rate each as "a very high priority," "somewhat of a priority," "a low priority," or "not a priority at all." Several categories of cost of dengue studies were addressed including: cost analysis economic concepts and/or issues; criteria for defining scope of a cost analysis of dengue; measures/indicators for dengue costing; age groups for dengue costing; sources of information; cost components for dengue costing; cost components for costing a dengue outbreak; economic approaches for costing dengue. The focus of the expert survey was specifically on methodological areas where there is little development or agreement. The criteria used to assess overall agreement on priority ranking were: High agreement: 10 checks per box; Moderate agreement: 7 - 9 checks per box; and Low agreement: < 6 checks per box. A total of 14 experts responded to the survey-11 health economists and three epidemiologists.

Handbook. This information and the evidence generated from the literature review were used to develop a pre-workshop handbook that included questions for the experts to address during small group discussions/exercises, excerpts on the scope of work, findings of the literature review (3), an assessment of the expert survey responses (2), and an analysis of current country guidelines (2). This handbook, available in English, Portuguese, and Spanish, was distributed to the experts several weeks prior to the workshop to get them acquainted with the workshop materials and desired outcomes. The expert panel was asked to review the information and provide feedback on the handbook prior to the workshop. A revised handbook was produced based on expert feedback; it was distributed on the first day of the workshop.


The expert groups were divided by language primarily to avoid issues of translation during small group discussions. The primary language during the workshop was Spanish since it was the language most commonly spoken among the selected experts; English and Portuguese were used as secondary languages.

In addition, participants had the option of wearing a headset to listen to United Nations-certified interpreters who provided simultaneous translation in three languages.


In order to allow fruitful discussion of the methodology, a timetable was set not to exceed 8 hours of discussion per day with suitable breaks. To avoid interruptions and distractions and for easy access, the workshop was not held at any of the participants' normal workplace.

Workshop proceedings

The workshop was divided into eight sessions. The objective of Day 1 was to frame the issues relevant to estimating the costs of dengue. The workshop opened with introductions and an overview of dengue, which included a discussion of the epidemiology of dengue and current vector-control strategies. This was followed by a discussion on the types of economic evaluations with a focus on cost analysis and, more specifically, reasons for estimating costs of dengue, uses of these costs, and challenges faced when costing dengue, an overview of current dengue economics research, and an assessment of the quality of the current evidence, followed by a group discussion.

The objective of Day 2 was to find an optimal (acceptable) compromise on various components of a cost analysis through a series of group discussions. The results of the pre-workshop expert survey designed to identify areas in a cost of dengue analysis considered to be of high priority by workshop participants were presented. This was followed by a session where country experts had an opportunity to share their experiences in designing dengue economic research. For the remainder of the day, workshop participants were divided in four teams to discuss specific methodological issues of the various phases of a cost analysis: planning phase, data collection phase, and analysis and presentation phase. At the end of each group discussion workshop participants came together to discuss issues raised, reach agreement on the various study components, and propose steps to address the remaining issues and uncertainties in the methodology. Each team was asked to work independently through a series of small group exercises that helped to establish quality standards for the conduct of cost analysis.

During the third and final day of the workshop, experts began drafting the guidelines and identified the general steps that were needed to develop the regional guidelines. The first half of the day was spent discussing how to communicate results from a cost analysis of dengue. The latter part of the day was spent putting the guidelines together. A Regional Steering Committee on Estimating Costs of Dengue was proposed during the day to address the issue of the guidelines continually evolving with future applications and studies.

Participants worked in groups of four or more, facilitated by the organizing technical staff. There were four groups: Group 1 (English only); Groups 2 and 3 (Spanish only); Group 4 (Portuguese only). Each group was assigned a small group exercise packet (included in the handbook) corresponding to specific topics from each subject area, and was asked to work independently through a series of small group exercises that would help to establish quality standards for conducting cost studies. Each small group exercise was divided into four methodological areas related that had been identified from the literature review and expert survey: study design; data components and data collection; analysis and presentation of results; and dissemination of cost information. The exercises helped guide the small group discussions toward a common theme that was later debated in the larger group discussions. The small group exercises are described elsewhere (2).

The larger group discussed issues raised during the breakouts in order to reach agreement on the various study components and to propose steps that would need to be taken for costing dengue in specific situations. A summary of a new standard methodology was then prepared. Workshop participants then reviewed the draft guidelines and made any additional recommendations.

The end product generated from this workshop, a set of costing guidelines, was the result of a collaborative effort, drawing on the direct input of international experts attending the workshop. The combined experience of the international experts was imperative. Further information about the costing guidelines is available elsewhere (2).



Consensus building

Using a combination of self-supported materials and small and large group discussions, the expert panel reached consensus on a number of conceptual issues relevant to costing dengue and developing dengue-costing guidelines. After the experts had a chance to work in their groups, there were large groups to help with consensus building. The success of these large groups depended greatly on the strength of the questions being asked, and the ability of experts to reach agreement on the various issues. For some questions, the experts arrived quickly at a consensus. The response to some of the questions, however, was not as straightforward, and it took longer for the experts to agree on specific methodological areas. This led to some confusion and delay in the process of gaining consensus, in part caused by what some experts pointed out were translation errors.

Another deterrent was the length of the small group exercise and the number of questions per exercise. Due to the complexity of the issues discussed and the choice of wording for some of the questions, there were delays in reaching consensus in a number of areas. Many of the experts felt that too much time was spent discussing specific methodological areas, which caused some groups to skim over questions they did not understand or on which they did not agree.

Pre-workshop materials

Several documents were included in the pre-workshop handbook. These included: a synopsis of each country's experience with existing costing studies, including design and implementation plans; an overview of the findings of the literature review that considered peer-reviewed studies on the economics of dengue and dengue prevention and control strategies; available guidelines on costing; the results of the expert survey; and the small group exercises.

One area for potential improvement is the expert survey. Its results were not always as clear as we hoped, possibly due to the lack of an ordered ranking system. Most of the areas of costing dengue were listed as "very high priority." While the experts may have truly felt all of the areas where of great importance, in order to better guide the discussion, more emphasis should be put on developing an ordered ranking system to better establish priorities.

There are many other factors that may have influenced the experts' responses to the survey, including translation issues, length of the survey, cultural differences, and knowledge of the subject area. The survey was translated from English into Spanish and Portuguese, which may have changed the meaning of some of the statements that the experts were evaluating. Unfortunately, delays in developing the survey, and later translating it, did not allow time for a more thorough review of the translated versions. Subsequently, the survey was distributed later than expected. While the experts did have time to review the survey and answer the questions, additional time would have improved the outcome of the survey responses. Finally, the experts' knowledge of the subject may have had a large impact on the survey responses. The same survey was used for all participants; therefore the epidemiologists answered questions about economics, a topic with which they may not have been familiar, and vice versa.

Country vs. regional context

Another difficulty arose when experts had trouble thinking outside of their country context. The overriding goal of the guidelines was to increase comparability by standardizing dengue costing across countries. Without thinking of the geographic area as a whole, the experts tended to make recommendations that were relevant only to their own country. This may be because many of the issues discussed were felt to be specific to their country. This made it harder to generalize and to think in terms of Latin America and the Caribbean as a whole. In order to assist the experts in thinking about the area as a whole, in the future they could be asked to answer questions from the perspective of a country other than their own. This may help the participants to be more open and apt to think in terms of the entire geographic area.

Language and logistics

The logistics of the workshop were somewhat complicated by equipment malfunctioning and the location of the interpreters, which resulted in occasional obstruction of the conversation. Some participants heard the echo of the interpreters over the speaker.

The quality of the actual simultaneous interpretation was another issue. Presenters tended to speak quickly and the interpreters were not fully comfortable with the technical language, leading to frequent pauses and skipping of critical information. Although we shared all workshop materials ahead of time to allow the translators to review them, last minute changes to the materials, particularly the slide presentations, meant that the interpreters had to lean out of their audio boxes to read from the slides. Additionally, the insufficient use of microphones during the large group discussions made proper simultaneous translation challenging at those times. Despite these challenges, simultaneous interpretation of the workshop was overall good, helping those not fluent in the official language to listen to and contribute to group discussions.

Balanced expertise

Another challenge was the underrepresentation of some areas of expertise. Initially, the experts were grouped by language to help facilitate small group discussions. Unfortunately this created an imbalance of expertise, with some groups being represented solely by epidemiologists, and others, by economists. Ideally, it would have been more effective to have an even split in expertise, with at least one area expert in each group. This would have allowed for all problems to be analyzed from multiple perspectives. To get a better balance of expertise, the small groups were rebalanced on the third day. While this was successful, it presented certain challenges (i.e., group members not being familiar with their new group's exercise questions). It is important to consider an equitable representation of expertise to ensure that problems are tackled from multiple perspectives.

A related issue was the concentration of the countries around a basic expertise area, which may have marked the direction of the workshop discussion. In the case of Brazil, the epidemiologists heavily influenced discussion. Colombia had a fairly balanced representation of public health experts and economists. Similarly, Mexico had a good representation of public health experts, health care managers, and policy experts with some health economics background. However, two of the countries were not as well represented: both Panama and Jamaica had only one public health expert with a background in epidemiology who attended the workshop.



Key to the workshop's success was the pre-workshop materials, which included: a synopsis of each country's experience with existing costing studies, including design and implementation plans; a review of the literature, including a discussion of the similarities and differences between the costing methodologies; an assessment of existing, publicly available guidelines; and a survey to be completed by each expert in order to identify areas of relevant and high-priority economics research among all participants.

Regarding the size of the expert panel, having 20+ experts in attendance ensured adequate discussion of the evidence (or lack thereof) and guarded against individual biases. Too few experts might have limited adequate discussion and too many would have made group dynamics and effectiveness difficult. When faced with the challenge of trading off full representation against the requirement of having a functional group, we felt that we obtained an optimal group size.

Other factors that contributed to its success were: involving experts at the various stages of the consensus building exercise; convening a multidisciplinary group of experts to reduce expert bias and provide a more comprehensive and integrated approach to developing costing guidelines; organizing small and large group discussions to allow for fruitful discussions; developing effective cross-cultural collectivism, trust, communication, and empathy across the expert panel; establishing clear lines of responsibilities within each group of experts; breaking down the complex issues into smaller, simpler ideas; convening various groups of experts involved in previous, ongoing, or future cost of dengue studies and building up a network of researchers with common interests; providing ample background materials to the expert panel several weeks prior to the workshop; and, making all background materials available in all relevant languages.



Among the lessons learned was how important it is to state clearly, from the start, what the objectives and scope of the workshop will be. It is also important to encourage experts to think about the geographic area as a whole, to ensure everyone understands the benefits of collaboration, to develop effective cross-cultural collectivism, to establish clear lines of responsibilities within each group of experts, to break down the complex issues into simpler ideas, and to ensure that each area of expertise is represented in each small group.

Other lessons learned were related to the simple fact that the discussion among the expert panel had to be simultaneously translated into two other languages, and all the materials likewise had to be prepared in three languages. Though it is quite common for multiple languages to be used at an international meeting, initially the main challenge is to decide on which would be the workshop's official language. Once the dominant language is identified, the participants should be notified and given options for interpretation. Stick to the official language throughout the workshop and have all documents translated and checked for accuracy ahead of time.

Test the layout of the room prior to the meeting to avoid equipment malfunction; optimize the location of the interpreters (back of the room or a separate room to ensure they will not be competing with the presenter); test equipment (including batteries). The speakers should be periodically reminded to slow down to allow the interpreters to keep up.

With regards to cultural issues, make sure to provide additional pre-workshop materials, as needed, to help smooth over the cultural differences. Doing so will facilitate meaningful conversation, without restricting the flow of knowledge and ideas. To better ensure an understanding of local context, include data on each country's healthcare system in the pre-workshop materials. When working with experts from multiple disciplines, consider providing materials that will get everyone up to speed on a variety of relevant areas, e.g., basic epidemiology of disease, basic management practices, basic economics concepts.

In terms of timing, allow more time for extended discussion of complex issues; guide the discussion, but be conscious of time allotments for individual topics-ensure balanced discussion of all relevant topics.

If ranking priorities in a survey, have the experts rank cost-components against each other, rather than individually. This will generate better results regarding the relative importance of the costing methodologies.

In addition, develop and distribute the surveys as early as possible to allow time for the experts to delve more deeply into the "why" behind their answers. It may also be beneficial to tailor a number of survey questions to the participants' areas of expertise. This would allow the experts to rank only the areas with which they are most familiar.

It is also important to pilot the surveys and make adjustments that clarify and validate the instrument. Allow time to ensure the accuracy of the translated versions of the surveys and other workshop materials.

When reaching consensus on technical areas, it is useful to have a balanced representation of expertise to help guide the discussion. In addition to the scientific evidence and the opinions of experts, consider resource implications and feasibility issues, such as time, skills, and expertise needed to carry out the recommendations. Last, but not least, when working with such a diverse group, especially one that will have multiple language needs, plan to provide strong leadership during the workshop.

Consensus-building for the purpose of developing costing guidelines is a complex process that requires innovative approaches to tackling the myriad of potential factors involved. We expect that the lessons learned from this workshop will help ensure success in future consensus-building.



1. Gold MR, Siegel JE, Russell LB, Weinstein MC, eds. Cost-effectiveness in health and medicine. Oxford University Press: New York; 1996.         

2. Constenla D, Lefcourt N, Garcia C. Expert panel on developing guidelines for the estimation of costs of dengue in the Latin American and Caribbean region. In: Constenla et al. Proceedings of the workshop on developing regional guidelines of costs of dengue with a focus on the Latin America and Caribbean region. Available from: http://www.denguevaccines.org/sites/default/files/Proceedings%20Guidelines%20Workshop.pdf Accessed 14 August 2013.         

3. Constenla D, Lefcourt N, Garcia C. Assessing the economics of dengue: results from a systematic review (abstract). In: Proceeding of the Third Pan American Dengue Research Network Meeting. Cartagena, Columbia; 2012.         



Manuscript received on 1 October 2012.
Revised version accepted for publication on 17 June 2013.

Organización Panamericana de la Salud Washington - Washington - United States
E-mail: contacto_rpsp@paho.org