ORIGINAL RESEARCH INVESTIGACIÓN ORIGINAL
Developing a research agenda for promoting physical activity in Brazil through environmental and policy change
Diseño de un programa de investigación para promocionar la actividad física en el Brasil mediante cambios ambientales y de políticas
Rodrigo S. ReisI,*; Cheryl M. KellyII; Diana C. ParraIII; Mauro BarrosIV; Grace GomesV; Deborah MaltaVI; Thomas SchmidVII; Ross C. BrownsonIII
IPhysical Activity and Quality of Life Research Group, School of Health and Biosciences, Pontifícia Universidade Católica do Paraná, Curitiba, Brazil
IIDepartment of Health Sciences, Beth-El College of Nursing and Health Sciences, University of Colorado, Colorado Springs, Colorado, United States of America
IIIPrevention Research Center in St. Louis, Brown School, Washington University in St. Louis, St. Louis, Missouri, United States of America
IVUniversidade do Estado de Pernambuco, Recife, Brazil
VUniversidade Estadual Paulista, Rio Claro, Brazil
VICoordenação de Agravos e Doenças Não-Transmissíveis, Ministry of Health, Brasilia, Brazil
VIIDivision of Nutrition and Physical Activity, Physical Activity and Health Branch, Centers for Disease Control and Prevention, Atlanta, Georgia, United States of America
OBJECTIVE: To identify the highest priorities for research on environmental and policy changes for promoting physical activity (PA) in Brazil; to uncover any gaps between researchers' and practitioners' priorities; and to consider which tools, methods, collaborative strategies, and actions could be useful to moving a research agenda forward.
METHODS: This was a mixed-methods study (qualitative and quantitative) conducted by Project GUIA (Guide for Useful Interventions for Activity in Brazil and Latin America) in February 2010-January 2011. A total of240 individuals in the PA field (186 practitioners and 54 researchers) were asked to generate research ideas; 82 participants provided 266 original statements from which 52 topics emerged. Participants rated topics by "importance" and "feasibility;" a separate convenience sample of 21 individuals categorized them. Cluster analysis and multidimensional scaling were used to create concept maps and pattern matches.
RESULTS: Five distinct clusters emerged from the concept mapping, of which "effectiveness and innovation in PA interventions" was rated most important by both practitioners and researchers. Pattern matching showed a divergence between the groups, especially regarding feasibility, where there was no consensus.
CONCLUSIONS: The study results provided the basis for a research agenda to advance the understanding of environmental and policy influences on PA promotion in Brazil and Latin America. These results should stimulate future research and, ultimately, contribute to the evidence-base of successful PA strategies in Latin America.
Key words: Concept formation; physical fitness; exercise; health research agenda; health policy; environment and public health; Brazil.
OBJETIVO: Identificar las máximas prioridades en la investigación sobre cambios ambientales y de políticas para promover la actividad física en el Brasil; descubrir las posibles disparidades en las prioridades de los investigadores y las de los profesionales; y evaluar qué herramientas, métodos, estrategias colaborativas y acciones podrían ser útiles para el avance de un programa de investigación.
MÉTODOS: Fue un estudio de metodología mixta (cualitativa y cuantitativa) llevado a cabo por el Proyecto GUIA (Guía para intervenciones útiles de actividades físicas en Brasil y Latinoamérica) entre febrero de 2010 y enero de 2011. Se pidió a 240 personas del campo de la actividad física (186 profesionales y 54 investigadores) que generaran ideas para la investigación; 82 de los participantes aportaron 266 afirmaciones originales de las que surgieron 52 temas. Los participantes puntuaron los temas en función de su "importancia" y "viabilidad" y otra muestra de conveniencia formada por 21 personas los clasificó. Se usó el análisis por conglomerados y el escalonamiento pluridimensional para crear mapas conceptuales y concordancias de patrones.
RESULTADOS: Con la elaboración de mapas conceptuales surgieron cinco conglomerados diferenciados y, de ellos, el que tanto los profesionales como los investigadores consideraron más importante fue "eficacia e innovación en las intervenciones de actividad física". La concordancia de patrones indicó una divergencia entre los grupos, especialmente con respecto a la viabilidad, sobre la que no hubo consenso.
CONCLUSIONES: Los resultados del estudio sirvieron de base para un programa de investigación con el que avanzar en la comprensión de la influencia que el ambiente y las políticas ejercen sobre la promoción de la actividad física en el Brasil y en América Latina. Estos resultados deben fomentar la futura investigación y, en último término, aumentar la base de evidencia sobre estrategias fructíferas de actividad física en América Latina.
Palabras clave: Formación de concepto; acondicionamiento físico; ejercicio; agenda de investigación en salud; política de salud; medio ambiente y salud pública; Brasil.
Over the last decades, growing and consistent evidence has shown the health benefits of a physically active lifestyle (1, 2); however, physical inactivity levels are increasing in many countries around the world (3, 4). There are numerous interventions that are delivered in community settings that have been shown to be effective (5-7), even though there remain some mixed findings for large-scale, multicomponent, community interventions (8). Despite the fact that researchers and practitioners are faced with inconsistent evidence, the communication gap between them must be bridged in order to advance a common research and practice-based agenda (9). There is also a need for evaluating and publishing evaluations of current interventions as a complement to evidence from systematic reviews and to enhance external validity (10).
A growing area of interest involves environmental and policy interventions that encourage active lifestyles and increase levels of physical activity (PA) (11, 12). Evidence on the potential effectiveness of these types of interventions is rapidly growing (13). This context has led many organizations to advocate for more and improved interventions that promote PA through community-based policy and environmental changes (14).
The adoption of a broader evidence base is dependent on a scientific approach (i.e., effectiveness), in addition to incorporating community experience into the intervention design (i.e., efficacy studies) (15). While practitioners are often more oriented to such needs because they are exposed to more "real world situations," researchers focus their attention on filling scientific gaps. This potential mismatch may undermine actions toward effective and sustainable environmental and policy modifications to promote PA.
Alternately, development of a common research agenda is likely to produce new data on the effectiveness of novel interventions and expand knowledge of proven interventions as they are applied to a variety of real world and policy settings (16-18). To date, such an agenda has not been proposed by any country in Latin America, where the gap between practice and research is likely more evident (19). While the United States and other high income countries have already advanced an understanding of the role that environmental and policy interventions play in promoting PA, this is a relatively new area in Latin America (20). Nonetheless, practitioners here are already adopting PA interventions using environment and policy as major components (20).
This study's objectives were threefold: To identify the highest priorities for research on physical activity (PA) in Brazil; to uncover any gaps between researchers' and practitioners' priorities; and to consider which tools, methods, collaborative strategies, and action steps would be useful to moving a research agenda forward. The results of this study should be useful for stimulating future research to identify environmental and policy strategies that will effectively promote PA in Brazil, and will ultimately contribute to the evidence-base of successful PA strategies in Latin America.
MATERIALS AND METHODS
This mixed-methods study (qualitative and quantitative) was conducted by Project GUIA (Guide for Useful Interventions for Activity in Brazil and Latin America) (21, 22). Six researchers from Project GUIA comprised the core scientific team, which oversaw all aspects of the project. All six had prior experience conducting PA research, implementing interventions, and working with practitioners.
The project utilized concept mapping, a tool well-suited to situations where different groups (e.g., researchers and practitioners) work together (23) to develop, organize, and prioritize ideas. Although originally developed for use by the social sciences and education, concept mapping is now successfully applied to public health topics, such as tobacco use (24), HIV/AIDS (25), and PA (16), where it has led to an improved understanding of the barriers and opportunities for policy implementation.
This study followed the customary six steps of concept mapping, briefly outlined in Figure 1. A more detailed description of the process of concept mapping and of each step can be found elsewhere (23).
After identifying a focus prompt for this study ("One research topic that will best inform policy or environmental approaches for PA promotion is . . ."), two samples of researchers and practitioners were obtained. A list of 82 researchers was created in February 2010 using a database from the National Council for Scientific and Technological Development (CNPq) linked to the Ministry of Science and Technology of Brazil (26). Inclusion criteria included experience with conducting PA promotion/intervention studies, evaluating PA programs, and/or studying policies and environments for PA promotion. After checking for complete contact information, 54 researchers were invited to participate in the brainstorming process.
The list of practitioners was obtained from two sources. First, all the PA programs continuously supported by the Ministry of Health (MoH) over the last 5 years were identified. After checking for complete information of local coordinators, 132 practitioners were invited to participate in the brainstorming process. Second, PA program coordinators from the Serviço Social da Indûstria (SESI) were identified and 54 additional practitioners (two from each of the 27 states in Brazil) were invited, for a total of 186 practitioners.
In March 2010, a total of 240 individuals-53 researchers and 186 practitioners-were invited to participate in the study. They were asked to generate research ideas in response to the focus prompt via a secure Internet site. Because participants submitted their ideas anonymously, the average number of responses submitted per respondent could not be calculated. A total of 82 participants provided 266 original statements. Response rates were 18.8% (n = 35) and 81.4% (n = 44) for practitioners and researchers, respectively (3 were unknown).The core scientific group checked these statements for overlap and appropriateness. As a result, 52 topics were extracted from the original 266 statements (Table 1).
In April-May 2010, in a separate process (Figure 1, Step 3), 177 individuals (down from the original 240 due to email errors detected after the first contact) were invited to complete two online surveys. Using the same secure Internet site that was used in the brainstorming phase (Figure 1, Step 2), participants rated each of the 52 research topics on importance (Survey 1) and feasibility of implementation within the next 5 years (Survey 2), relative to the other ideas. The possible ratings ranged from 1 (relatively unimportant/not feasible) - 10 (extremely important/feasible). The importance scale was completed by 18 practitioners and 19 researchers, for a total of 37 participants (21% response rate). The feasibility scale was completed by 15 practitioners and 18 researchers, or 33 participants (19% response rate).
Additionally, a convenience sample was drawn from a list of individuals who had participated in prior MoH, CNPq, and/or Project GUIA research projects, but had not as yet participated in the present study. These practitioners/ researchers, who were selected for their familiarity with the PA field (n = 21), engaged in a face-to-face meeting in January 2011. Each was asked to independently sort the research ideas into categories based on similarities among the themes. They were told to create their own categories and place each statement in only one category, and that the sorting process should result in more than one category, but fewer than the total number of ideas.
Data from the sorted and rated ideas were entered into Concept Systems Global® (Concept Systems Incorporated, Ithaca, New York, United States) (23). Cluster analysis and multidimensional scaling (MDS) were performed to produce a visual representation of the data. The ideas were grouped or partitioned on a map, per the MDS, with clusters of ideas that had been similarly categorized by the participants appearing in contiguous areas, and those unrelated, farther apart.
The final "concept map" presented an arrangement of these idea-clusters. A standardized, systematic process identified the most useful number of clusters by considering the range of issues represented, the purpose, and intended uses of the resulting map and the observed coherence of clusters at different levels (27). Each cluster was named by the core group of 21 participants according to the set of ideas it held.
The concept map was followed by pattern matching, which created a series of graphs that ranked the map's ideaclusters by importance and/or feasibility (based on the average rating of the statements in each cluster). Pattern matching determined if, and to what degree, practitioners and researchers rated the same clusters as most important or most feasible, and also, whether those rated most important were also considered to be most feasible. Pattern matching also allowed a comparison of researchers' perceptions of feasibility to practitioners' perceptions of importance.
To interpret the concept map and pattern match, a convenience sample was organized into five workgroups in different parts of the country. The workgroups were organized to gain qualitative feedback on concept mapping results, but the sampling was not designed to identify subgroup differences (e.g., by the country's geographic areas, researchers vs. practitioners, etc.). Three of the workgroups were composed of researchers, predominantly faculty and doctoral (Ph.D.) students with backgrounds in PA and/or public health, in Rio Grande do Sul's Parana, Paraiba, and Pernambuco areas. The other two groups were composed of practitioners; one comprised predominantly PA instructors and SESI managers, and the other, public health system staff who worked or had experience on PA programs supported by the MoH. Group size was 10-18 individuals and the meetings lasted 1-2 hours (mean = 1.8 hours). Each group received a summary of the study methods and a description of the idea-clusters that had been identified.
After reviewing the materials, the workgroups were asked to discuss the following topics, ones that had been successfully used previously in a similar study (16): (i) the roles of various disciplines and professions in addressing research gaps; (ii) the tools and methods useful to addressing the issues; and (iii) the actions and actors that might help to move the agenda forward.
Protection of human participants
Human subjects approval was obtained from the institutional review boards of Washington University (St. Louis, Missouri, United States) and St. Louis University (St. Louis, Missouri, United States).
Of the 37 individuals who rated the importance of statements (Figure 1, Step 3), 51% were researchers and 49% practitioners; and of the 33 who rated feasibility, 55% were researchers and 45% practitioners. Concept mapping resulted in a map with five distinct clusters (Figure 2) composed of 9-26 statements each. In such a map, smaller clusters suggest a tighter grouping of statements in that domain (i.e., more agreement among those rating the statements) and greater distance between clusters indicates greater conceptual difference. Cluster layers signify the overall importance of the statements within the cluster, with more layers suggesting higher importance. The statements given the highest priority within each cluster are shown in Table 2.
PA priorities by importance
Pattern matching showed that there was relatively low overall concordance between researchers and practitioners on importance (r = 0.20) (figure available upon request). Though they agreed on the most important topics, i.e., both ranking "effectiveness and innovation in PA interventions" as highest and "evaluation and impact of PA policies" as second (researchers) and third (practitioners), they disagreed on the remaining clusters, e.g., "promoting PA through urban environment, active commuting, and social networks" was ranked third by researchers, but last by practitioners.
PA priorities by feasibility
Researchers and practitioners showed even greater divergence regarding feasibility (r = -0.76) (figure available upon request). "Promoting PA through urban environment, active commuting, and social networks" was rated most feasible by researchers and least feasible by practitioners. The second most feasible cluster among researchers was "Individual and environmental PA correlates," which was rated fourth by practitioners.
Researcher versus practitioner perception
To better understand the gap between researcher perception of feasibility and practitioner perception of importance, a third pattern match compared the two (Figure 3). The rationale for this analysis was that practitioners, being closer to the community, are likely to better understand its needs, which could be defined as importance; while researchers, who are closer to the research process, may be better equipped to rate feasibility. Based on the rating process described earlier, the range in values for importance was 8.96-8.42, and for feasibility, 7.70-7.43, both on a scale of 1-10. These analyses showed a negative association between importance and feasibility (r = -0.94).
In the interpretation phase, five groups were asked to review the results and present ideas to implement the agenda by addressing methods and tools, transdisciplinary approaches, and actions that could be useful for this effort (Figure 1, Steps 5 and 6). Table 3 summarizes the discussions conducted with practitioners and researchers and shows the methods/tools grouped into three areas: qualitative approaches, observation techniques, and quasi-experimental or longitudinal designs.
When asked how various disciplines and professions might work together to address these research areas, two ideas were identified: first, the need to integrate different levels of decision-making (i.e., municipal, state, and federal), as well as recognize and engage diverse disciplines; and second, dissemination of results through different channels (e.g., media) and strategies (e.g., short summaries friendly to policymakers).
Using concept mapping and input from a sample of practitioners and researchers in Brazil, this study identified five research clusters related to promoting PA through environmental and policy interventions. In general, however, there was low agreement between the study's practitioners and researchers regarding the importance and feasibility of the clusters identified. Agreement on the importance was highest for the cluster labeled "health and economic benefits of PA," and lowest for "evaluation and impact of PA policies."
Regarding feasibility, opinions were even more divergent, with a negative correlation between researchers' and practitioners' scores. The comparison between scores for practitioner's importance rating and researcher's feasibility rating also yielded a negative correlation. A similar study conducted in the United States found higher agreement between practitioners and researchers, (16) which highlights the importance of context when defining and establishing research and practice priorities.
Differences between practitioners and researchers ratings are not completely unexpected and several hypotheses could help understand these findings. PA has only recently emerged as a priority for the public health system in Brazil (28); despite this fact, research on PA, as well as implementation of interventions, has been growing rapidly in the country (29). For instance, the number of cities that implemented community PA programs supported by the MoH increased from a few dozen to more than 1 000 over a 5-year period (30). Consequently, the vast majority of practitioners working in PA within the public health system (e.g., nurses, social workers, and physicians) are dealing with a field in which they are not formally trained.
Additionally, research on PA in Brazil has increased in both quantity and quality during recent years (21, 31). This rapid growth in the field of PA may have resulted in gaps in knowledge and practice between practitioners and researchers. While practitioners tend to respond more quickly to the community's needs because they are dealing with its issues on a daily basis, researchers respond to other demands, such as scientific gaps and funding opportunities. There may also be a communication gap that would otherwise link practitioners' needs with scientists' research priorities.
The discussion of tools and methods for PA promotion that took place in Step 5 (Figure 1) revealed several common patterns: qualitative approaches, systematic observation methods, and quasi-experimental or longitudinal designs. Community audits and direct observation of recreational park users (32, 33) were mentioned as useful tools because of their low cost; plus, it was thought that they could effectively address financial barriers in low resource areas (34). Moreover, tools such as these could form part of an advocacy strategy, engaging and empowering citizens by training local leaders or community groups to evaluate their own community programs.
All five workgroups suggested the use of qualitative methods integrated with more traditional approaches (e.g., focus groups and large-scale surveys). This combination was considered important to identifying people's preferences and any possible barriers to environmental and policy changes, information thought to be unattainable through telephone interviews or other traditional methods alone.
Finally, all groups mentioned the lack of strong experimental evidence regarding the effects of environmental and policy changes on PA, particularly in the local context. This response concurs with current evidence (6, 7). A particular concern was the lack of capacity for conducting PA intervention evaluations. Given as much, future funding in Brazil should focus on using these methods and tools, as well as on building skills for conducting research with quasi-experimental designs and carrying out program evaluations.
When addressing the question related to disciplines and professions working together, the workgroups provided responses mainly in two areas. First, there was consensus on the need to integrate decision-making levels through task forces and common agendas. This proposal was mentioned as key to optimizing resources and prioritizing efforts. Second, they agreed on the need to engage many diverse areas, such as architecture, medicine, nursing, physical education, and psychology, to promote PA through coordinated environmental and policy changes. For example, a specific concern referred to the lack of participation by physical education professionals in the decision-making process in Brazil. Future funding might then require researchers and practitioners to involve a variety of partners before granting awards.
The discussion on how to advance the agenda revolved around dissemination, translation, and applicability of results through different strategies and to other contexts. Dissemination was considered essential as a way to persuade decisionmakers to adopt effective environmental and policy strategies. Various useful channels for dissemination were identified, including policy briefings, mass media, and social networks.
Some limitations of this study should be considered. First, due to the process of reducing the number of ideas to a more manageable number of themes, some specificity may have been lost. Second, the importance and feasibility ratings are simple measures when considering the relative complexity of the concepts investigated. As a consequence, the results are limited to the experience and understanding of the participants on these concepts. Third, a convenience sample of practitioners and researchers limited to MoHfunded localities and SESI was used, which limits the variability and generalization of findings. Fourth, the response rates were low, yet comparable, to similar studies in the area (16). As reported in similar studies (16, 24), practitioners showed a lower response rate, which also may have limited the variety of information gathered from this group. However, despite the low response rate, the study had good variability with participation from nearly all the states in Brazil and a broad range of expertise among the participants. The selection of participants came from reliable sources for both researchers (CNPq) and practitioners (MoH and SESI) and all were experienced in PA intervention and promotion.
This study shows that, according to importance ratings, the highest priority issues for PA research in Brazil are "effectiveness and innovation in PA interventions" and "evaluation and impact of PA policies," while in terms of feasibility, researchers and practitioners did not agree on any single issue. The methods and tools suggested for moving the PA agenda forward are qualitative approaches, observation techniques, and quasi-experimental or longitudinal designs.
Finally, integrating all decisionmaking levels and ensuring that efforts engage diverse disciplines were suggested as collaborative strategies that could promote the PA agenda. A need for disseminating the study results through different channels and strategies was also reported.
These results parallel ongoing efforts by research agencies and groups of practitioners in Brazil. Nonetheless, advancing areas such as "evaluation and impact of policies" and "cost effectiveness studies" as indicated by importance ratings, may contribute to improving PA interventions in Brazil and in other parts of Latin America. The authors and their team are working closely with other Brazilian researchers and practitioners to implement this agenda and to guide future calls for research proposals in Brazil.
These study findings are highly relevant to other areas of Latin America that face similar challenges. The methods used in this study can be adapted and used for developing PA research agendas in other countries or geographic areas.
Acknowledgments. Funding for this study was provided by the Prevention Research Centers Program at the Centers for Disease Control and Prevention (Atlanta, Georgia, United States [contract U48/DP001903, Applying Evidence-Physical Activity Recommendations in Brazil]). The findings and conclusions in this article are those of the author(s) and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
The authors wish to thank the members of Project GUIA and the concept mapping participants for their valuable contributions.
1. O'Donovan G, Blazevich AJ, Boreham C, Cooper AR, Crank H, Ekelund U, et al. The ABC of physical activity for health: a consensus statement from the British Association of Sport and Exercise Sciences. J Sports Sci. 2010;28(6):573-91.
2. Warburton DE, Nicol CW, Bredin SS. Health benefits of physical activity: the evidence. CMAJ. 2006;174(6):801-9.
3. Dumith SC, Hallal PC, Reis RS, Kohl 3rd HW. Worldwide prevalence of physical inactivity and its association with human development index in 76 countries. Prev Med. 2011;53(1-2):24-8.
4. World Health Organization. Global recommendations on physical activity for health. Geneva: WHO; 2010. Pp. 58.
5. Kahn EB, Ramsey LT, Brownson RC, Heath GW, Howze EH, Powell KE, et al. The effectiveness of interventions to increase physical activity. A systematic review. Am J Prev Med. 2002;22(4 suppl):73-107.
6. Hoehner CM, Soares J, Perez DP, Ribeiro IC, Joshu CE, Pratt M, et al. Physical activity interventions in Latin America: a systematic review. Am J Prev Med. 2OO8;340): 224-33.e4.
7. Conn VS, Hafdahl AR, Mehr DR. Interventions to increase physical activity among healthy adults: meta-analysis of outcomes. Am J Public Health. 2011;101(4):751-8.
8. Baker PR, Francis DP, Soares J, Weightman AL, Foster C. Community wide interventions for increasing physical activity. Cochrane Database Syst Rev. 2011(4):CD008366.
9. DiPietro L. Physical activity and public health: the challenges ahead. J Phys Act Health. 2012;9(1):3-4.
10. Green LW. Making research relevant: if it is an evidence-based practice, where's the practice-based evidence? Fam Pract. 2008;25(suppl 1):i20-4.
11. Sallis JF, Cervero RB, Ascher W, Henderson KA, Kraft MK, Kerr J. An ecological approach to creating active living communities. Annu Rev Public Health. 2006;27:297-322.
12. Stokols D, Allen J, Bellingham RL. The social ecology of health promotion: implications for research and practice. Am J Health Promot. 1996;10(4):247-51.
13. McCormack GR, Shiell A. In search of causality: a systematic review of the relationship between the built environment and physical activity among adults. Int J Behav Nutr Phys Act. 2011;8(1):125.
14. World Heath Organization. The World Health Report 2003: shaping the future. Geneva: WHO; 2003.
15. O'Neall MA, Brownson RC. Teaching evidence-based public health to public health practitioners. Ann Epidemiol. 2005;15(7):540-4.
16. Brownson RC, Kelly CM, Eyler AA, Carnoske C, Grost L, Handy SL, et al. Environmental and policy approaches for promoting physical activity in the United States: a research agenda. J Phys Act Health. 2008;5(4):488-503.
17. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof 2006;26(1):13-24.
18. Wang S, Moss JR, Hiller JE. Applicability and transferability of interventions in evidence-based public health. Health Promot Int. 2006;21(1):76-83.
19. Guimarães JA. Medical and biomedical research in Brazil. A comparison of Brazilian and international scientific performance. Ciência e saúde coletiva. 2004;9(2):303-27.
20. Sallis JF. Environmental and policy research on physical activity is going global. Res Exerc Epidemiol. 2011;13(2):111-7.
21. Pratt M, Brownson RC, Ramos LR, Malta DC, Hallal PC, Reis RS, et al. Project GUIA: A model for understanding and promoting physical activity in Brazil and Latin America. J Phys Act Health. 2010;7(suppl 2):S131-4.
22. Brownson RC, Parra DC, Dauti M, Harris JK, Hallal PC, Hoehner C, et al. Assembling the puzzle for promoting physical activity in Brazil: a social network analysis. J Phys Act Health. 2010;7(suppl 2):S242-52.
23. Trochim WMK. An introduction to concept mapping for planning and evaluation. Eval Program Plann. 1989;12(1):1-16.
24. Trochim WM, Stillman FA, Clark PI, Schmitt CL. Development of a model of the tobacco industry's interference with tobacco control programmes. Tob Control. 2003;12(2):140-7.
25. Kagan JM, Kane M, Quinlan KM, Rosas S, Trochim WM. Developing a conceptual framework for an evaluation system for the NIAID HIV/AIDS clinical trials networks. Health Res Policy Syst. 2009;7:12.
26. Conselho Nacional de Desenvolvimento Científico e Tecnológico. Available from: http://efomento.cnpq.br/efomento/distribuicaoGeografica/distribuicaoGeografica.do?metodo=apresentar. Accessed 20 July 2012.
27. Southern D, Batterham R, Appleby N, Young D, Dunt D, Guibert R. The concept mapping method. An alternative to focus group inquiry in general practice. Aust Fam Physician. 1999;28(Suppl 1):S35-40.
28. Malta DC, Castro AM, Gosh CS, Cruz DKA, Bressan A, Morais-Neto OL, et al. A política nacional de promoção da saúde e a agenda da atividade física no contexto do SUS. Epidemiol Serv Saúde. 2009;18(1).
29. Knuth AG, Malta DC, Cruz DKAd, Freitas PCd, Lopes MP, Janaina F, et al. Rede nacional de atividade Física do Ministério da Saúde: resultados e estratégias avaliativas. Rev Bras Ativ Fís Saúde. 2010;15(4):229-33.
30. Knuth AG, Malta DC, Cruz DK, Castro AM, Fagundes J, Sardinha LM, et al. Description of the countrywide physical activity network coordinated by the Brazilian Ministry of Health: 2005-2008. J Phys Act Health. 2010;7 (suppl 2): S253-8.
31. Hallal PC, Dumith Sde C, Bastos JP, Reichert FF, Siqueira FV, Azevedo MR. Evolução da pesquisa epidemiológica em atividade física no Brasil: revisao sistematica. Rev Saude Publica. 2007;41(3):453-60.
32. Hoehner CM, Ivy A, Ramirez LK, Handy S, Brownson RC. Active neighborhood checklist: a user-friendly and reliable tool for assessing activity friendliness. Am J Health Promot. 2007;21(6):534-7.
33. McKenzie TL, Cohen DA, Sehgal A, Williamson S, Golinelli D. System for observing play recreation in communities: Reliability and feasibility measures. J Phys Act Health. 2006;3(Suppl 1):s208-s22.
34. Hino AAF, Reis RS, Ribeiro IC, Parra DC, Brownson RC, Fermino RC. Using observational methods to evaluate public open spaces and physical activity in Brazil. J Phys Act Health. 2010;7(Suppl 2):S146-54.
Manuscript received on 14 September 2011.
Revised version accepted for publication on 2 May 2012.
* Send correspondence to Rodrigo Siqueira Reis, email: email@example.com