Quality dimensions in health evaluation: manager's conceptions
Dimensiones de la calidad en la evaluación en salud: concepciones de gestores
Maria Lúcia Magalhães BosiI; Ricardo José Soares PontesI; Suziana Martins de VasconcelosII
IDepartamento de Saúde Comunitária. Faculdade de Medicina. Universidade Federal do Ceará (UFC). Fortaleza, CE, Brasil
IIPrograma de Pós-Graduação em Saúde Coletiva. UFC. Fortaleza, CE, Brasil
OBJECTIVE: To understand manager's perceptions and experiences in regards to qualitative evaluations in basic health care.
METHODOLOGICAL PROCEDURES: A qualitative study, based on the critical interpretive approach, was performed in 2006, in the city of Fortaleza, Northeastern Brazil. The sample consisted of the group responsible for planning basic health care at the state level. In order to obtain the empirical data, the focus group technique was utilized.
ANALYSIS OF RESULTS: Two central themes emerged concerning the perceptions about quality and the dimensions of quality employed in health evaluations, which were revealed in distinct ways. The concepts of quality evaluation and qualitative evaluation did not appear clearly understood, confusing qualitative evaluation with formal quality evaluations. Likewise, the inherent multidimensionality of quality was not recognized. Despite the criticism expressed by the participants regarding the improper quantification of certain dimensions, the necessary technical skills and understanding were not observed for the approach to include the distinct dimensions of quality in the evaluation process.
CONCLUSIONS: The conceptions of managers responsible for the planning of basic health care at the state level revealed an important disassociation from the premises of qualitative evaluation, especially those evaluations oriented by the fourth generation approach. Therefore, the model adopted by these actors for the evaluation of program and service quality did not consider their multidimensionality.
Descriptors: Health Management. Health Manpower. Health Services Evaluation. Personnel Management. Qualitative Research.
OBJETIVO: Comprender concepciones y experiencias de gestores con relación a la evaluación cualitativa en la atención básica en salud.
PROCEDIMIENTOS METODOLÓGICOS: Estudio cualitativo, fundamentado en la vertiente crítico-interpretativa, realizado en 2006 en la ciudad de Fortaleza, en noreste de Brasil. La muestra teórica fue compuesta por el grupo responsable por la planificación de la atención básica en nivel estatal. Para la obtención del material empírico, se utilizó la técnica de grupo focal.
ANÁLISIS DE LOS RESULTADOS: Emergieron dos temas centrales: concepciones de calidad y dimensiones de la calidad en la praxis de la evaluación en salud, desdoblándose en aspectos distintos. Los conceptos calidad y evaluación cualitativa no se mostraron claramente demarcados, confundiéndose la evaluación cualitativa con la evaluación de la calidad formal. Del mismo modo, no se reconoce la multidimensionalidad inherente a la calidad. A pesar de las críticas de los participantes a la cuantificación indebida de ciertas dimensiones, no se observó claridad y dominio técnico con relación al abordaje a utilizar para abarcar las distintas dimensiones de la calidad en procesos evaluativos.
CONCLUSIONES: Las concepciones de los gestores responsables por la planificación de la atención básica, en el espacio estudiado, revelan un importante distanciamiento de las premisas de la evaluación cualitativa, sobretodo en aquella orientada por el enfoque de cuarta generación. Por lo tanto, el modelo adoptado por esos actores en la evaluación de la calidad de los programas y servicios no contempla su multidimensionalidad.
Descriptores: Gestión en Salud. Recursos Humanos en Salud. Evaluación de Servicios de Salud. Administración de Personal. Investigación Cualitativa.
The Family Health Strategy (Estratégia Saúde da Família, ESF) introduces an intervention in basic health care to restructure the service model established in the country. The basic health care structure, by means of the ESF, has a good chance of realizing the directives of the National Health System (Sistema Único de Saúde, SUS) in the organization of services and health practices. This makes it possible for the universal, democratic and participative principles of the SUS1 to be integrated into the everyday experiences of health system clients. However, regardless of its reach and potential, the ESF sill does not have systematic mechanisms for monitoring and evaluation throughout the period of its implementation and expansion.
In this context and in accordance with the policy of the Ministry of Health, defined in Portaria 588, from 7 April of 2004,ª the need to structure a policy for monitoring and evaluating basic care actions arose in the state of Ceará, Northeastern Brazil. Such an initiative is tied to the planning process and the intersectoral performance of the State Department of Health of Ceará (SESA-CE), considering the existence of inadequacies in the health financing for secondary and tertiary level of care, which indicates the need to improve quality in basic care services.
In view of the accelerated growth of the ESF, the necessity emerged for the implementation of evaluation processes, especially concerning the quality of care in the scope of this strategy. According to some authors,12 such a concept demands a rigorous consideration of what is known as qualitative evaluation, in order to identify interfaces and distinctions between both the qualitative and quantitative domains. When considering subjectivity as inherent in the production of care, the complex nature of the problems involved in health care increases the scope of the quality evaluation. Thus, a qualitative approach is necessary to understand the significance of the actions through the perspective of those involved.3
Therefore, in regards to the use of qualitative and quantitative instruments in health evaluation, even though we do not consider such approaches oppositional or "alternative", we assume that the approaches are ontologically different: the quantitative approach reveals "objective" aspects of the evaluated phenomenon, while the qualitative approach deepens understanding.2
Quality evaluations and qualitative evaluations should be differentiated, since the terms are often used as synonyms. Qualitative evaluation involves the analysis (understanding) of dimensions that are not captured by quantitative indicators and language, focusing on the subjective aspects that permeate the processes. Quality evaluation, which encompasses qualitative analysis, also includes dimensions of quality that allows for measurement and objectivity.4
When we mention the multidimensionality of the term quality, we refer on one hand to the fact that the term quality has distinct meanings (of an objective or subjective nature). On the other hand, quality varies according with group interests or the social actors, in relation to their interests or positions, as well as their relation to the program or service in question. In the first case, it corresponds to what some authors describe as intrinsic multidimensionality and in the second case it is described as extrinsic multidimensionality.12
In regards to qualitative evaluation, the emergence of an evaluation strategy called fourth generation evaluation, as described by Guba & Lincoln, should be re-emphasized.6 In this approach, evaluation, besides including a vision of the context, is also participative. In other words, it privileges not only the validation of the results by the various segments, but also the incorporation of different actors (and interests) involved in the processes evaluated, attempting in this way to ensure above all else the perspectives of the clients, who are almost always excluded in the process of formulating, agreeing upon and executing the actions.11
Considering that the production of health systems results from the interaction between clients, health professionals, providers and the entire context that involves the relationship between them, the perspective of the client constitutes a central element in the evaluation process. Together with the participants involved in the evaluative process, the evaluator becomes a mediator in regards to the specific aspects evaluated.8,10
In addition, a good evaluation is one that the users of the evaluation, including managers, professionals and also clients for whom the practices are destined, are identified and involved in the process and one that the objectives of the evaluation consider the utility and goals of the evaluative process.9 According to Patton,9 these aspects characterize qualitative evaluation as a tool in the process of managing the quality of care offered to the population.
Despite the conceptual elements highlighted and which have been forwarded in the more recent literature on health evaluation, the quantitative approach predominates in evaluative practices. Such a focus is based on principles of positive science that confirm the predominance of measurements in detriment to the understanding of processes, which is done in special circumstances in the context of the Brazilian health system.
According to Tanaka & Melo11 (p. 123) "the formal rigor adopted through the presumptions of the scientific method lead to the understanding of evaluation as being an intervention able to be exclusively done by specialists". This "world view" makes evaluation (based on either a quantitative or qualitative basis) to be, on one hand, seen as a "threatening process", connected to the idea of control from beyond the actors and, on the other hand, a specialized activity, attributed to experts in the field, who in general are not involved in the contexts evaluated.
Another place where difficulties are encountered in performing evaluations of health services is in the area of theoretical methods and specialized techniques for this end.11 The difficulty in operationalizing qualitative evaluation within health services is related, also, to the precarious training of professionals, especially of managers, in the field of social research processes.
The objective of the present study was to understand the perceptions and experiences of managers in relation to quality evaluation and qualitative evaluation in basic care.
The study was carried out in the municipality of Fortaleza, state of Ceará, in 2006, was part of a broader investigation concerning the evaluation and monitoring practices of the group responsible for the planning of basic care at the state level, during the period from 2003 to 2006.
When trying to understand quality in evaluation, within a field characterized by specificity, it is necessary to recognize the polemical characterization of quality.12 Vuori13 (p.19) argues that the "concept of quality in health has many facets and different authors may utilize distinct meanings for this term. In general, the term (...) denotes a large spectrum of desirable characteristics".
The complexity of the concept of quality challenges us to "interpret the interpretation" of quality in the evaluation performed by actors at the central level. It therefore is concerned with interpreting quality in regards to the empirical experiences of the participants of the study.
Five professionals of the State Department of Health who worked at the Núcleo de Apoio à Organização da Atenção Primária (NUORG - Center of Support to the Organization of Primary Care) were selected as participants of the study. Despite not specifying schooling criteria, all participants had a higher-level degree in the health area. Four had extensive experience in the field of basic health care, having developed programs in the area of immunization, management and coordination of basic health units in health districts, in the micro-region and in the Family Health Program (Programa Saúde da Família, PSF) in the interior of Ceará state.
For the collection of interviews, the focus group technique was used in order to facilitate group interaction.5,7 The technique was conducted by project coordinators, using an interview guide with key questions on group perceptions about dimensions of quality used in health evaluation, as well as the discrimination between quality and qualitative evaluation in basic care, and more specifically, in the ESF. Questions were progressively revealed in consonance with the principle of "non-directivity".
After this step, cross-sectional analyses of the transcripts were performed, as well as the identification of meaning units expressed in the various themes in the discursive material, which constituted the analytical focus, grouping them into categories present in the discourses. In accordance with the dialectic nature of the critical perspective used in interpretive activity, this subjective work was related to its material aspect in the context studied.
The validation of the content by the participants was seen as a part of the process of constructing and analyzing data. However, due to the absence of various professionals, the content of the evaluation was presented in a workshop and strongly relied on the group that participated in the actions.
The study followed Resolution 196 of the national Health Council and followed the protocol for inter-regional research, as one of its approaches, approved by the Research Ethics Committee in health of the Faculdade do Paraná (FEPAR) under number 2771/2004.
ANALYSIS OF RESULTS AND DISCUSSION
The findings presented here were developed through the processing and interpretation of information. The findings were organized around two main analytical categories - perceptions of quality and qualitative evaluations in health evaluation - which, in turn, presented themselves in two distinct dimensions.
Perceptions of quality
We observed that in the perceptions of participants, the recognition of the distinct dimensions of quality was unclear. Nonetheless, participants frequently criticized the objectivity or the quantification of certain dimensions, understood by the group as something that would require other indicators or tools that the group acknowledged not knowing. We also noticed a slight recognition of the subjective dimensions of quality, especially in reference to relationships, experiences and interaction between professionals and clients:
"[...] quality is the client entering (...) having a connection with the community, being attended by name, giving privacy to the patient, closing the door, asking what he has, interacting with the patient, prescribing and explaining the mechanisms of action of the medications and the service can offer the medicine. That is quality [...]" (Informant 1).
Uchimura & Bosi12 point out that the terms quality and quantity do not imply a dichotomy. For example, in its objective meaning, quality is measured as the number or the formal qualification of professionals that are part of the Family Health Teams. Nonetheless, this generalization and measurement are not possible when we focus on the subjective dimension, since this concerns experiences, emotions, feelings, expressing singularities that are not possible to be expressed numerically.
Despite these perceptions, the approach utilized by the group to operationalize the quality evaluation of the health service does not consider its multidimensionality. The term 'qualitative' is imprecise and in the interviews it oscillated between objective and subjective domains, often being juxtaposed with formal quality. We observed that the use of the term quality by the group was mostly used in reference to the tool of Methodology and Improvement of Quality in Basic Health Care (Metodologia de Melhoria da Qualidade em Atenção Primária à Saúde, Proquali).b Developed by the group in partnership with international institutions, this methodology was adopted by the NUORG in some evaluation practices, and the group understood these as qualitative evaluation.
Although the purpose of this article was not to analyze this tool, we think that it is appropriate to at least mention that this technology exclusively uses structured questionnaires without the necessary refinement to distinguish between objective aspects and those whose nature cannot be quantified. Nonetheless, Proquali has given special significance to Ceará in the area of health evaluations because of the fact it represents a methodology directed to the evaluation of quality in basic care. Despite its limitations, it is an innovative proposal that appears to have stimulated discussion about quality, even before it had highlighted in the agenda of the SUS and been expressed in instruments distributed at the national level. Nonetheless, regardless of Proquali's view, the operationalized evaluation of quality has been approximating the benchmark established in the tool Evaluation for the Improvement of Quality of the Family Health Strategy (Avaliação para a Melhoria da Qualidade da Estratégia Saúde da Família),c considered as a way to confer objectivity to quality evaluations of the components of health care. The informants refer to this tool, used to evaluate the quality of the PSF, as follows:
"The tool is systematic, each section in discussed in four ways: one the secretary of health responds and discusses, the other the PSF coordinator, the other the unit coordinator, the other the entire team of Family Health Program, and the fifth tool is only the higher-level team" (Informant 3).
In summary, the concept of quality is not clearly established. In group discussion the identification of what the literature describes as intrinsic and extrinsic multidimensionality inherent to the concept was not observed.12 In addition, despite the recognition of the subjective aspects inherent to health care, the concepts of quality ended up mixed with the objective dimension, giving it a perspective of traditional evaluation focused on normative aspects, with this type of evaluation considered as "qualitative evaluation".
Qualitative evaluation in health evaluation
Although the studied group considers the participation of the different actors in the health service as an important factor, a disconnect between what was said and what effectively happens was observed when they responded that the view of clients was not incorporated in the few quality evaluations experienced by the group:
"The team does the evaluation in the health unit with professionals of a higher level; all are involved; health workers, nurse assistants, servers, dentists, nurses, doctors, everybody involved." (Informant 1).
Quality evaluation, in a broad sense, includes qualitative evaluation, and must include the different views of the distinct participants in health services and actions, with an indispensible actor in this process being the client of the service or program. Regarding this, Serapioni10 (p. 209) points out that "the product in health services is the result of direct contact between professionals and patients and interactions that constitute this relation".
In justifying their adherence to Proquali, the participants indicated the following as essential points: the context that the participating actors in this process are incorporated; the collective construction of the evaluation tool and discussion between the subjects involved. Besides, they also considered the creation of a multidisciplinary evaluation team at the local level and the reorganization of the team at the central level in order to accompany and develop the evaluative practices, as well as the technicians (in the case of the central level of the SESA) that performed the evaluation. We noticed that the distinction between quality evaluation and qualitative evaluation is not clear to the group, as defined here. There was not sufficient theoretical and methodological understanding or expertise to evaluate the quality of programs and health services in their distinct dimensions.
Despite this, the group revealed discussions already occurring and there is evidence of an opportunity to advance the understanding of quality evaluations of programs and services beyond what the traditional models allow:
"(...)we can not say that it is complete, we must take the jump of quality. Ceará went ahead with the health worker, went ahead with the Family Health Program, and [what about] the quality of this Program?" (Informant 5)
In focusing on the materiality of these services and the possibility of making the teams effective at a local level, as well as promoting the necessary involvement of actors, the discussions showed less optimism, pointing out new obstacles for the incorporation of a broader evaluative focus emphasizing the discontinuity in management and its impact.
Another aspect is that, contrary to what is observed in relation to the formal dimensions connected to the allocation of resources, the performance of quality evaluations has been optional. Tanaka & Melo11 warn about the relationship of evaluating specific programs with external financers.
Given the limits of this exposition, it is not possible to examine questions inherent to economic neoliberalism and its impacts on the economy, on education and on health. There is at least room to refer to its connection with the history of evaluation and to point out the presence of this logic in health policies. This is extremely critical and dissonant for the execution of policies based on the principles and values of the SUS, when evaluation is done without including the perspective of actors involved in the production of care. Even the evaluation of quality in its formal dimension becomes 'optional', expressing a value scale and a conception of what really matters to control. This culture appeared as an obstacle in the discussion of the interviewees, as illustrated below:
"the participation of the municipality in the process is optional (...) while the evaluation of the PSF organized by the Ministry of Health is verticalized and has a cut in resources as a penalty (...) the question is obligatory and does not pass through a process of sensitization and cooperation of the managers. (...) principally if they are in the management of the system" (Informant 1)
The group perceptions concerning the concept of quality were not clear, and the inherent polemical nature of the concept was not recognized. Despite being mentioned, program quality is mainly expressed in an objective dimension, using quantification, or in other words traditional evaluation, often considering this type of evaluation as qualitative evaluation. Consequently, even though the process and subjective nature of evaluation are recognized and valued, the method to address these questions remains unclear. Beyond the simple lack of understanding regarding the methodological basis for evaluation processes, such lack of knowledge has consequences for the construction of broader evaluations that triangulate objective information with the participants' subjectivity, integrating clients and structure, processes and results.
The analysis of the findings suggests the predominance of quantitative approaches in evaluation practices, resulting in analyses through an objective lens based on statistical measures and inferences. The fundamental question about a strategy should focus on its ability to provide insight concerning the nature of the problem we want to evaluate. Considering the complexity of the object of health, this methodological reduction has apparently limited the extent that evaluations can help formulate high value objectives to intervene upon and improve the health system, as an effective tool for decision-making and quality management.
According to their guidelines, evaluations are done without including the perspective of the actors or clients of the health services. Also, what apparently matters is the efficiency of a specific program, without consideration of other problems that end up revealing effects that are maybe invisible in the routine data but that have costs to the system. This situation reinforces the necessity of using complementary methodologies and interdisciplinary dialogue in the design of evaluations in order to minimize the large dissonance between the objects evaluated and the methods adopted.
Given the challenges presented by the complex field of basic care, the informants recognize, in an incipient manner, the limits and insufficiency of the traditional approach. Besides this, the study verified acceptance from the participants for new perspectives and methodologies. They also recognized the gaps in their professional training, holding qualitative evaluation as a highly specialized activity. This view has been creating, on one hand, the impression this is a complex practice and, on the other, the perception the activity is a long ways from being performed by themselves.
1. Andrade LOM, Barreto ICHC. SUS passo a passo: história regulamentação financiamento políticas nacionais. São Paulo: Hucitec; 2007.
2. Bosi MLM, Mercado-Martinez FJ. Notas para um debate. In: Mercado-Martinez, FJ, Bosi MLM, organizadores. Pesquisa qualitativa de serviços de saúde. Petrópolis: Vozes; 2004. p.23-71.
3. Bosi MLM, Mercado FJ, organizadores. Avaliação qualitativa de programas de saúde: enfoques emergentes. Petrópolis: Vozes; 2006.
4. Bosi MLM, Uchimura KY. Avaliação da qualidade ou avaliação qualitativa do cuidado em saúde? Rev Saude Publica. 2007;41(1):150-3. DOI:10.1590/S0034-89102007000100020
5. Carlini-Cotrin B. Potencialidades da técnica qualitativa grupo focal em investigações sobre abuso de substâncias. Rev Saude Publica 1996;30(3):285-93. DOI:10.1590/S0034-89101996000300013
6. Guba E, Lincoln Y. Fourth generation evaluation. Newsbury Park: Sage; 1989.
7. Morgan DL, Kruger RA. The focus group kit. Thousands Oaks: Sage; 1998.
8. Patton MQ. How to use qualitative methods in evaluation. Newsbury: Sage; 1987.
9. Patton MQ. Utilization-focused evaluation. The New Century Text: Sage; 1997.
10. Serapioni M. Métodos qualitativos e quantitativos na pesquisa social em saúde: algumas estratégias para a integração. Cienc Saude Coletiva. 2000;5(1):187-92. DOI:10.1590/S1413-81232000000100016
11. Tanaka OY, Melo C. Reflexões sobre a avaliação em serviços de saúde e a adoção das abordagens qualitativa e quantitativa. In: Bosi MLM, Mercado-Martinez FJ, organizadores. Pesquisa qualitativa de serviços de saúde. Petrópolis: Vozes; 2004. p.121-137.
12. Uchimura KY, Bosi MLM. Qualidade e subjetividade na avaliação de programas e serviços em saúde. Cad Saude Publica. 2002;18(6):1561-9. DOI:10.1590/S0102-311X2002000600009
13. Vuori H. A qualidade da saúde. Divulg Saude Debate. 1991;3:17-25. Correspondence: Received: 3/19/2009 The authors declare that there are no conflicts of interest.
Maria Lúcia Magalhães Bosi
R. Prof. Costa Mendes, 1608 - 5º andar
21943-570 Fortaleza, CE, Brasil
Study financed by the Ministry of Health, through the Expansion and Consolidation Project of the Family Health Strategy (Projeto de Expansão e Consolidação da Estratégia Saúde da Família, PROESFIII), part of the State Plan for the Strenghtening of Monitoring and Evaluation Actions in Basic Care of Ceará State (Plano Estadual para o Fortalecimento das Ações de Monitoramento e Avaliação da Atenção Básica do Estado do Ceará).
a Ministério da Saúde. Portaria 588, de 7 de abril de 2004. Aprova Convocatória Pública nº 01/2004 para que Unidades Federadas (UF) apresentem Planos Estaduais para o Fortalecimento das Ações de Monitoramento e Avaliação da Atenção Básica no âmbito do Projeto de Expansão e Consolidação do Saúde da Família - PROESF. Diario Oficial Uniao. 7 de abril 2004;Seção 1:87.
b Secretaria da Saúde do Estado. Metodologia de melhoria da qualidade da atenção à saúde: instrumento de melhoria do desempenho. Fortaleza; 2005.
c Secretaria de Atenção à Saúde, Ministério da Saúde. Avaliação para a melhoria da qualidade da estratégia saúde da família. Brasília: Ministério da Saúde; 2006.
The authors declare that there are no conflicts of interest.