Added therapeutic value of new drugs approved in Brazil from 2004 to 2016

Valor terapêutico acrescentado de novos medicamentos aprovados no Brasil de 2004 a 2016

Valor terapéutico añadido de los nuevos medicamentos aprobados en Brasil desde 2004 a 2016

Rogério Hoefler Teresa Leonardo Alves Hubertus G. Leufkens Janeth de Oliveira Silva Naves About the authors

Abstract:

This study aimed to assess the level of therapeutic innovation of new drugs approved in Brazil over 13 years and whether they met public health needs. Comparative descriptive analysis of therapeutic value assessments performed by the Brazilian Chamber of Drug Market Regulation (CMED) and the French drug bulletin Prescrire for new drugs licensed in Brazil, from January 1st 2004 to December 31st 2016. The extent to which new drugs met public health needs was examined by: checking inclusions into government-funded drug lists and/or clinical guidelines; comparing Anatomical Therapeutic Chemical Classification (ATC) codes and drug indications with the list of conditions contributing the most to the national disease burden; and assessing new medicines aimed to treat neglected diseases. 253 new drugs were approved. Antineoplastics, immunossupressants, antidiabetics and antivirals were the most frequent. Thirty-three (14%) out of 236 drugs assessed by the Brazilian chamber and sixteen (8.2%) out of 195 assessed by the French bulletin Prescrire were considered innovative. Thirty-six drugs (14.2%) were selected for coverage by the Brazilian Unified National Health System (SUS), seven of which were therapeutically innovative, and none were aimed to treat neglected disease. About 1/3 of the drugs approved aimed to treat conditions among the top contributors to Brazil’s disease burden. Few therapeutically innovative drugs entered the Brazilian market, from which only a small proportion was approved to be covered by the SUS. Our findings suggest a divergence between public health needs, research & development (R&D) and drug licensing procedures.

Keywords:
Drug Evaluation; Drug Industry; Products Registration; Diffusion of Innovations; Health Technology Assessment

Resumo:

O objetivo foi avaliar o nível de inovação terapêutica de novos medicamentos aprovados no Brasil ao longo de 13 anos e se eles atendem a necessidades de saúde pública. Foi feita uma análise comparativa descritiva da avaliação de valor terapêutico realizada pela Câmara de Regulação do Mercado de Medicamentos (CMED) e pelo boletim de medicamentos francês Prescrire para novos medicamentos licenciados no Brasil entre 1º de janeiro de 2004 e 31 de dezembro de 2016. Examinamos em que medida os novos medicamentos atendem a necessidade de saúde pública por meio de: checagem da inclusão em listas de medicamentos financiados pelo governo e/ou diretrizes clínicas; comparação de códigos da Classificação Anatômica Terapêutica Química (ATC, em inglês) e indicações de medicamentos com a lista de condições que mais contribuem para a carga de doença nacional; e avaliação de se os novos medicamentos tinham por objetivo tratar doenças negligenciadas. Foram aprovados 253 novos medicamentos. Antineoplásicos, imunossupressores, antidiabéticos e antivirais foram os mais frequentes. Trinta e três (14%) dos 236 medicamentos avaliados pela Câmara brasileira e 16 (8,2%) dos 195 avaliados pelo boletim francês Prescrire foram considerados inovadores. Trinta e seis medicamentos (14,2%) foram selecionados para cobertura no Sistema Único de Saúde (SUS), sete dos quais eram inovadores do ponto de vista terapêutico e nenhum dos quais tinha por objetivo tratar uma doença negligenciada. Em torno de 1/3 dos medicamentos aprovados tinha por objetivo o tratamento de doenças que figuram entre as principais contribuidoras da carga de doença no Brasil. Poucos medicamentos inovadores do ponto de vista terapêutico entraram no mercado brasileiro, dos quais apenas uma pequena proporção foi aprovada para ser coberta pelo SUS. Nossos resultados sugerem uma divergência entre necessidades de saúde pública, pesquisa e desenvolvimento (P&D) e procedimentos de licenciamento de medicamentos.

Palavra-chave:
Avaliação de Medicamentos; Indústria Farmacêutica; Registro de Produtos; Difusão de Inovações; Avaliação de Tecnologias de Saúde

Resumen:

El objetivo fue evaluar el nivel de innovación terapéutica de los nuevos medicamentos aprobados en Brasil durante 13 años y si cumplen con las necesidades sanitarias. Llevamos a cabo un análisis comparativo descriptivo acerca del valor terapéutico presente en las evaluaciones realizadas por la Cámara de Regulación del Mercado de Medicamentos (CMED) y la revista francesa Prescrire sobre los nuevos medicamentos autorizados en Brasil, desde el 1º de enero 2004 hasta el 31de diciembre de 2016. Su alcance, es decir, hasta qué punto los nuevos medicamentos cumplían con las necesidades de salud pública se comprobaron revisando las inclusiones en listas de medicamentos subvencionados por el gobierno y/o directrices clínicas; comparando los códigos de la Classificación Anatómicos Terapéuticos Químicos (ATC por sus siglas en inglés) y las indicaciones de los medicamentos respecto a la lista de enfermedades que contribuían a la mayor carga de morbilidad nacional; y asesorando si los nuevos medicamentos tenían como objetivo tratar enfermedades desatendidas. Se aprobaron 253 nuevos medicamentos. Los antineoplásicos, inmunosupresores, antidiabéticos y antivirales fueron los más frecuentes. Treinta y tres (14%), aparte de los 236 medicamentos evaluados por la Cámara Brasileña, y 16 (8,2%), aparte de los 195 evaluados por la revista francesa Prescrire, se consideraron innovadores. Treinta y seis medicamentos (14,2%) se seleccionaron para que tuvieran cobertura por el Sistema Único de Salud (SUS), siete de ellos eran terapéuticamente innovadores, y ninguno tenía como meta tratar enfermedades desatendidas. Alrededor de 1/3 de las medicinas aprobadas tenían como meta tratar problemas de salud entre las enfermedades con mayor carga de morbilidad en Brasil. Pocos medicamentos terapéuticamente innovadores accedieron al mercado brasileño y de éstos sólo una pequeña parte fueron aprobados para que fueran cubiertos por el SUS. Nuestros resultados sugieren una divergencia entre las necesidades públicas de salud, investigación & desarrollo (I&D) y los procedimientos para la autorización de medicamentos.

Palabras-clave:
Evaluación de Medicamentos; Industria Farmacéutica; Registro de Productos; Difución de Innovaciones; Evaluación de Tecnologías de Salud

Introduction

No consensus was reached regarding defining what constitutes pharmaceutical innovation 11. Morgan S, Lopert R, Greyson D. Toward a definition of pharmaceutical innovation. Open Med 2008; 2:E4-7.. In fact, neither the attributes of an innovative product nor the criteria to be considered when assessing its innovation have been clearly established 22. Soto Álvarez JS. Grado de innovación de los nuevos medicamentos: propuesta de criterios que deben tenerse en cuenta para su valoración. Med Clin (Barc) 2009; 132:481-3.. In general, new medicines are commonly believed to be therapeutically innovative and offer better health outcomes than older drugs 33. Davis C, Abraham J. Rethinking innovation accounting in pharmaceutical regulation: a case study in the deconstruction of therapeutic advance and therapeutic breakthrough. Sci Technol Human Values 2011; 36:791-815.,44. Vitry AI, Shin NH, Vitre P. Assessment of the therapeutic value of new medicines marketed in Australia. J Pharm Policy Pract 2013; 6:2.,55. Caetano R. Avaliação de tecnologias em saúde: contribuições para a assistência farmacêutica. In: Osorio-de-Castro CGS, Luiza VL, Castilho SR, Oliveira MA, Jaramillo NM, organizadores. Assistência farmacêutica: gestão e prática para profissionais da saúde. Rio de Janeiro: Editora Fiocruz; 2014. p. 221-38., although that is not confirmed in the clinical practice 66. van Luijn JCF, Gribnau FWJ, Leufkens HGM. Superior efficacy of new medicines? Eur J Clin Pharmacol 2010; 66:445-8.. It is currently accepted that a new compound must have a clinically relevant advantage when compared head-to-head with existing established therapies to be considered innovative 77. Puntmann I, Schmacke N, Melander A, Lindberg G, Muhlbauer B. EVITA: a tool for the early evaluation of pharmaceutical innovations with regard to therapeutic advantage. BMC Clin Pharmacol 2010; 10:5.,88. Center for Drug Evaluation and Research, U.S. Food and Drug Administration. Novel drugs 2015 summary. Silver Spring: U.S. Food and Drug Administration; 2016.; such as showing better population indicators for morbidity, mortality and quality of life. This notion of therapeutic advance can be useful to recognize and reward medicine manufacturers that develop products with a high therapeutic value and, therefore, to encourage and sustain the innovation 22. Soto Álvarez JS. Grado de innovación de los nuevos medicamentos: propuesta de criterios que deben tenerse en cuenta para su valoración. Med Clin (Barc) 2009; 132:481-3.,99. Aronson JK, Ferner RE, Hughes DA. Defining rewardable innovation in drug therapy. Nat Rev Drug Discov 2012; 11:253-4. that meets patients’ needs 1010. Kaplan W, Wirtz VJ, Mantel-Teeuwisse A, Stolk P, Duthey B, Laing R. Priority medicines for Europe and the world: 2013 update. Geneva: World Health Organization; 2013.,1111. Nathan C. Aligning pharmaceutical innovation with medical need. Nat Med 2007; 13:304-8..

The list of advantages of innovative drugs when compared with available therapeutic options are 22. Soto Álvarez JS. Grado de innovación de los nuevos medicamentos: propuesta de criterios que deben tenerse en cuenta para su valoración. Med Clin (Barc) 2009; 132:481-3.: greater efficacy/effectiveness and safety; improved patients’ quality of life and satisfaction; reduction in treatment costs; better therapeutic outcomes in patient subgroups; or enabling treatment of otherwise unmet medical needs. The decision to introduce the new drug into the clinical practice must consider these aspects as well as users’ and providers’ interests 22. Soto Álvarez JS. Grado de innovación de los nuevos medicamentos: propuesta de criterios que deben tenerse en cuenta para su valoración. Med Clin (Barc) 2009; 132:481-3.,1212. Eichler HG, Thomson A, Eichler I, Schneeweiss S. Assessing the relative efficacy of new drugs: an emerging opportunity. Nat Rev Drug Discov 2015; 14:443-4.. Several methods have been proposed to probe and define the added therapeutic value of new medicines 77. Puntmann I, Schmacke N, Melander A, Lindberg G, Muhlbauer B. EVITA: a tool for the early evaluation of pharmaceutical innovations with regard to therapeutic advantage. BMC Clin Pharmacol 2010; 10:5.,1313. Motola D, De Ponti F, Poluzzi E, Martini N, Rossi P, Silvani MC, et al. An update on the first decade of the European centralized procedure: how many innovative drugs? Br J Clin Pharmacol 2006; 62:610-6.,1414. New products and new indications in 2016: a system that favours imitation over the pursuit of real progress. Prescrire Int 2017; 26:136-9.,1515. Centro Andaluz de Documentación e Información de Medicamentos. Nuevos medicamentos: qué son las fichas. http://www.cadime.es/es/que_son_las_fichas_de_novedad_terapeutica.cfm (accessed on 26/Feb/2018).
http://www.cadime.es/es/que_son_las_fich...
,1616. Kesselheim AS, Wang B, Avorn J. Defining "innovativeness" in drug development: a systematic review. Clin Pharmacol Ther 2013; 94:336-48.,1717. Die information für ärzte und apotheker neutral, unabhängig und anzeigenfrei Berlin. https://www.arznei-telegramm.de/01index.php3 (accessed on 26/Feb/2018).
https://www.arznei-telegramm.de/01index....
. Generally, therapeutic advance is identified when a the superiority of a drug is shown in methodologically robust studies, using active comparators and hard clinically relevant outcomes 1616. Kesselheim AS, Wang B, Avorn J. Defining "innovativeness" in drug development: a systematic review. Clin Pharmacol Ther 2013; 94:336-48.,1818. Naci H, Ioannidis JP. How good is "evidence" from clinical studies of drug effects and why might such evidence fail in the prediction of the clinical utility of drugs? Annu Rev Pharmacol Toxicol 2015; 55:169-89..

Added therapeutic value assessors generally agree that therapeutic advance is rare, despite the alleged increase in research and development costs 1616. Kesselheim AS, Wang B, Avorn J. Defining "innovativeness" in drug development: a systematic review. Clin Pharmacol Ther 2013; 94:336-48.. A recent report by Public Citizen in the US showed despite very high profits of more than 100 billion USD per year, the 20 largest pharmaceutical corporations only reported spending half that amount on research and development (R&D) for new medicines 1919. Claypool R. Pharmaceutical industry profits exceed industry's self-reported research and development. https://www.citizen.org/sites/default/files/pharma-profits-and-r-and-d-report.pdf (accessed on 26/Feb/2018).
https://www.citizen.org/sites/default/fi...
. Researchers have called on governments to define policies to align R&D and real health needs 1010. Kaplan W, Wirtz VJ, Mantel-Teeuwisse A, Stolk P, Duthey B, Laing R. Priority medicines for Europe and the world: 2013 update. Geneva: World Health Organization; 2013.,1111. Nathan C. Aligning pharmaceutical innovation with medical need. Nat Med 2007; 13:304-8..

The assessment of added therapeutic value can guide clinical decisions by healthcare professionals, thus benefiting patients 22. Soto Álvarez JS. Grado de innovación de los nuevos medicamentos: propuesta de criterios que deben tenerse en cuenta para su valoración. Med Clin (Barc) 2009; 132:481-3.,1515. Centro Andaluz de Documentación e Información de Medicamentos. Nuevos medicamentos: qué son las fichas. http://www.cadime.es/es/que_son_las_fichas_de_novedad_terapeutica.cfm (accessed on 26/Feb/2018).
http://www.cadime.es/es/que_son_las_fich...
; and it drives more effective and efficient decisions in health systems. This is particularly important in poorer settings in which drug selection enables the allocation of resources - and therefore access - of those medicines that benefit the population the most 2020. Wirtz VJ, Hogerzeil HV, Gray AL, Bigdeli M, de Joncheere CP, Ewen MA, et al. Essential medicines for universal health coverage. Lancet 2017; 389:403-76..

Difficulties in accessing health services and medicines remain a global social problem of great concern 2020. Wirtz VJ, Hogerzeil HV, Gray AL, Bigdeli M, de Joncheere CP, Ewen MA, et al. Essential medicines for universal health coverage. Lancet 2017; 389:403-76.. Brazil is no exception, even though it was the 8th largest pharmaceutical market in the world during 2016, with sales volumes amounting to approximately 28 billion USD 2121. Associação da Indústria Farmacêutica de Pesquisa. Guia 2017 Interfarma. https://www.interfarma.org.br/guia/guia-2017/dados-do-setor/ (accessed on 26/Feb/2018).
https://www.interfarma.org.br/guia/guia-...
.

A policy to improve the access to medicines through price-control measures 2222. Câmara de Regulação do Mercado de Medicamentos. Resolução nº 2, de 5 de março de 2004. Diário Oficial da União 2004; 6 mai., based on incentive mechanisms to increase the sector’s offer and competitiveness, was established in Brazil in 2004. This was the first Brazilian policy enshrining systematic health technology assessment (HTA) as a component of price-setting procedures and being applied to all new drugs approved by the Brazilian health authority - Brazilian Health Regulatory Agency (Anvisa). The maximum price to be borne depends on the added therapeutic value of the drug 2222. Câmara de Regulação do Mercado de Medicamentos. Resolução nº 2, de 5 de março de 2004. Diário Oficial da União 2004; 6 mai.,2323. Lemgruber A. Avaliação e regulação de preços de medicamentos novos: a experiência do Brasil. In: Organização Pan-Americana da Saúde, organizador. O acesso aos medicamentos de alto custo nas Américas: contexto, desafios e perspectivas. Brasília: Organização Pan-Americana da Saúde, Ministério da Saúde, Ministério das Relações Exteriores; 2009. p. 55-7..

A cross-sectional study 2424. Bonfim JRA. O registro de produtos farmacêuticos novos: critérios para a promoção do uso racional de fármacos no Sistema Único de Saúde [Dissertação de Mestrado]. São Paulo: Programa de Pós-graduação em Ciências, Coordenadoria de Controle de Doenças, Secretaria de Estado da Saúde de São Paulo; 2006. of new pharmaceutical products registered in Brazil between 1999 and 2004 concluded that many of the medicines approved were me-too drugs driven by market demands. Another descriptive study analyzed new medicines entering the Brazilian market between 2000 and 2004 and reported that only one third were innovative products and none was indicated to treat infectious diseases prevalent in developing countries 2525. Vidotti CCF, Castro LLC. Fármacos novos e necessidades do Sistema Único de Saúde no Brasil. Espaç Saúde 2009; 10:7-11..

Bearing in mind the need to improve the access to and the rational use of medicines, our study aimed to assess the added therapeutic value of all new medicines registered in Brazil since the implementation of the HTA policy and to investigate their alignment with national public health needs.

Methods

Study design

We conducted a comparative descriptive analysis of therapeutic value assessments for new medicines receiving marketing authorization in Brazil, from January 1st 2004 to December 31st 2016, and examined their alignment with local therapeutic needs.

Data collection

New medicines approved by Anvisa - from January 1st 2004 to December 31st 2016 - were identified under the codes 175, 1,458, 10,464 or 1,528, which represented respectively: entry for a new medicine, electronic entry for a new medicine, entry for a new biological product. Even though vaccines are biological products, they were excluded from our study as they are identified as non-innovative by the Brazilian criteria 2222. Câmara de Regulação do Mercado de Medicamentos. Resolução nº 2, de 5 de março de 2004. Diário Oficial da União 2004; 6 mai.,2626. Gerência de Avaliação Econômica de Novas Tecnologias, Agência Nacional de Vigilância Sanitária. Efeitos da Resolução CMED nº 02/04 no processo de análise de preços de novos medicamentos. Brasília: Agência Nacional de Vigilância Sanitária; 2013..

The following general data were collected for each medicine: name and country of origin of manufacturer; composition; Anatomical Therapeutic Chemical Classification (ATC, 5th level) 2727. WHO International Working Group for Drug Statistics Methodology; WHO Collaborating Centre for Drug Statistics Methodology. ATC/DDD index 2017. https://www.whocc.no/atc_ddd_index/ (accessed on 26/Feb/2018).
https://www.whocc.no/atc_ddd_index/...
; date of the first authorization in Brazil; and indications approved.

Assessing the added therapeutic value of new medicines

The Brazilian Chamber of Drug Market Regulation (CMED) is an interministerial Brazilian body responsible for setting the prices of new medicines. It evaluates and classifies products into one of the six categories divided into two groups: new molecules (categories I and II) and new formulations (categories III, IV, V and VI). A new medicine is considered innovative (category I) when it contains a molecule (active ingredient) under national patent and offers a proven treatment gain when compared with available treatment options for that same indication. That treatment gain is translated into greater efficacy, or similar efficacy with significant reduction in adverse effects, or similar efficacy with a significant reduction in the overall treatment costs. A medicine is classified as category II i.e. as non-innovative if it has a new molecule (active ingredient) without patent in Brazil or if it does not bring any treatment gain 2222. Câmara de Regulação do Mercado de Medicamentos. Resolução nº 2, de 5 de março de 2004. Diário Oficial da União 2004; 6 mai.,2626. Gerência de Avaliação Econômica de Novas Tecnologias, Agência Nacional de Vigilância Sanitária. Efeitos da Resolução CMED nº 02/04 no processo de análise de preços de novos medicamentos. Brasília: Agência Nacional de Vigilância Sanitária; 2013..

The independent drug bulletin Prescrire evaluates new drugs or new indications approved in France, according to their efficacy, safety and convenience. The Prescrire bulletin and its English edition Prescrire International are fully financed by subscriptions and do not accept advertising or external sponsorship. Prescrire classifies new drugs or new indications into added therapeutic value categories as follows 2828. Prescrire's ratings system: new drugs and indications, at a glance. http://english.prescrire.org/en/81/168/53692/0/NewsDetails.aspx (accessed on 02/Apr/2019).
http://english.prescrire.org/en/81/168/5...
:

  1. Bravo: the product represents a major therapeutic advance in an area in which previously no treatment was available;

  2. A real advance: the product is an important therapeutic innovation but has certain limitations;

  3. Offers an advantage: the product has some value but does not fundamentally change the current therapeutic practice;

  4. Possibly helpful: the product has minimal additional value and should not change prescribing habits except in rare circumstances;

  5. Nothing new: the product may be a new substance but it is superfluous because it does not add to the clinical possibilities offered by previous products available;

  6. Not acceptable: product without evident benefit but with potential or real disadvantages;

  7. Judgement reserved: the editors postpone their rating until better data and a more thorough evaluation of the drug are available.

New medicines were dichotomously classified, as to their added therapeutic value, when compared with available therapies for the same indication, into “therapeutic innovation” or “no therapeutic innovation”. To do so, we used the assessments provided by the CMED and Prescrire. The first rating - CMED - stems from an official body of the Brazilian government responsible for implementing the national drug pricing policy and is the only systematic assessment of drug innovativeness available in Brazil based on clear criteria. The second rating is conducted by Prescrire, which is an independent and globally acknowledged drug bulletin that has published systematic drug reviews for more than three decades. Both entities have assessed most of the medicines included in our sample. The comparison of both ratings aims to contrast a governmental and an independent assessment. In a similar analysis in Canada, the Prescrire criteria were divided into two broad categories 2929. Lexchin J. Health Canada's use of its priority review process for new drugs: a cohort study. BMJ Open 2015; 5:e006816.. We have also applied this categorization, considering all the medicines rated within the first three Prescrire’s categories (bravo, real advance, offers an advantage) as “therapeutic innovation” and those belonging to the remaining categories (possibly useful, nothing new and not acceptable) as “no therapeutic innovation”. Medicines that Prescrire judged to have insufficient evidence to rate for therapeutic advantage are included under the category judgment reserved 2929. Lexchin J. Health Canada's use of its priority review process for new drugs: a cohort study. BMJ Open 2015; 5:e006816..

Adoption of drugs in national listings of the Brazilian Unified National Health System (SUS)

In Brazil, the National Committee for Technology Incorporation (CONITEC), created in 2011, is the resposible for governmental health technology assessment decisions 3030. Brasil. Lei nº 12.401, de 28 de abril de 2011. Altera a Lei nº 8.080, de 19 de setembro de 1990, para dispor sobre a assistência terapêutica e a incorporação de tecnologia em saúde no âmbito do Sistema Único de Saúde - SUS. Diário Oficial da União 2011; 29 abr.. CONITEC’s reviews are based on scientific evidence, considering aspects such as efficacy, accuracy, effectiveness and safety, as well as the comparative economic evaluation of the benefits and costs of new technologies versus existing ones. The data from the Committee were retrieved from its public website to assess whether new medicines authorized by Anvisa had been adopted or excluded from the national coverage lists as well as from clinical guidelines.

Alignment with national health needs

All new drugs approved during our study period were classified by the ATC code and their approved indications were compared with the conditions contributing the most to the Brazilian disease burden, as measured in disability-adjusted life years (DALYs) 3131. Leite IC, Valente JG, Schramm JM, Daumas RP, Rodrigues RN, Santos MF, et al. Burden of disease in Brazil and its regions, 2008. Cad Saúde Pública 2015; 31:1551-64.. A conservative approach was adopted when attributing indications to the various conditions, i.e. only allocating approved indications and the ATC code into specific conditions that matched that indication.

Data analysis

Descriptive statistics are reported with all variables presented as absolute numbers and proportions. The kappa index 3232. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33:159-74. was calculated to determine the level of agreement between CMED and the grouped Prescrire therapeutic value ratings in our study sample. We used Epi Info (https://www.cdc.gov/epiinfo/index.html) version 7.1.4.0 and IBM SPSS (https://www.ibm.com/) version 24 for data analysis.

Results

From January 1st 2004 to December 31st 2016, 268 new pharmaceutical products were approved by Anvisa. From these, 253 were considered for analysis, after excluding 15 vaccines (Figure 1). Antineoplastics (L01: n = 44; 17.4%), immunosuppressants (L04: n = 18; 7.1%), systemic antivirals (J05: n = 17; 6.7%) and antidiabetics (A10: n = 16; 6.3%) were the most frequent therapeutic classes.

Figure 1
New medicines authorized in Brazil from January 2004 to December 2016.

The therapeutic value of 248 of these medicines was evaluated by at least one of the two institutions, and 183 by both (Figure 1). Among the drugs assessed by Prescrire, 16 out of 195 (8.2%) were considered therapeutic innovations, as per the grouped Prescrire criteria. Thirty-three out of 236 drugs assessed by Brazilian CMED were rated as therapeutic innovations (Table 1). Five drugs - laronidase, nivolumab, pasireotide, sofosbuvir and sunitinib - were considered by both evaluators to be therapeutic innovations. As shown in Table 2, the therapeutic classes with the most innovative drugs were: antineoplastic agents (n = 19); systemic antivirals (n = 7) and immunosupressants (n = 3). Eleven (4.3%) drugs were considered innovative by Prescrire but not by the Brazilian Chamber, whereas another 24 (9.5%) drugs were considered innovative by the CMED and non-innovative by Prescrire. Overall, the level of agreement between both ratings was weak [kappa = 0.123 (95%CI: 0.014; 0.260; p = 0.077)].

Table 1
Rating of the added therapeutic value of newly registered medicines in Brazil as per three criteria from January 2004 to December 2016.

Table 2
Distribution of therapeutic innovations by Anatomical Therapeutic Chemical Classification (ATC) class: newly registered medicines in Brazil from 2004 to 2016 that were considered therapeutic innovations by either Brazilian Chamber of Drug Market Regulation (CMED) or Prescrire and their distribution by ATC class (n = 44).

Forty-four (17.4%) out of 253 drugs in our sample were rated as therapeutic innovations by at least one of the criteria. From these, seven were adopted in the SUS coverage listings, which means that they are to be made freely available to Brazilian patients (Table 3) (Supplementary Material, Table S1: http://cadernos.ensp.fiocruz.br/site/public_site/arquivo/suppl-e00070018_6597.pdf). None of these drugs was included in the basic pharmaceutical care package nor aimed to treat a neglected disease. Of the 29 drugs adopted by the SUS rated as “no therapeutic innovation”, 8 were only assessed by one of the criteria (Supplementary Material, Table S2: http://cadernos.ensp.fiocruz.br/site/public_site/arquivo/suppl-e00070018_6597.pdf).

Table 3
Adoption of new medicines approved in Brazil from January 2004 to December 2016 in coverage listings (Brazilian Unified National Health System - SUS).

Table 4 shows the distribution of indications approved in relation to the burden of disease. Sixty-three (30.9%) out of 204 non-therapeutically innovative drugs and 9 (20.4%) out of the 44 therapeutically innovative drugs aimed to treat a condition within the top 15 contributors to the national disease burden.

Table 4
Drugs approved by Brazilian Health Regulatory Agency (Anvisa) from January 2004 to December 2016 distributed into the 15 conditions contributing the most to the Brazilian disease burden (31) and categorized by their therapeutic innovation rating.

Discussion

This study has shown that relatively few new medicines approved in Brazil from 2004 to 2016 were considered therapeutic innovations and adopted in national drug listings. Most (82%) were non-innovative medicines.

Despite their low added therapeutic value ratings, more than 11% of these non-innovative drugs were included in government-funded drug listings. One-third of all new drugs approved during the study period aimed to treat one of the 15 conditions contributing the most to the Brazilian disease burden.

These are worrying findings from a public health perspective. First, they suggest many medicines approved in Brazil over the last 13 years had low utility levels 44. Vitry AI, Shin NH, Vitre P. Assessment of the therapeutic value of new medicines marketed in Australia. J Pharm Policy Pract 2013; 6:2.. Second, they indicate a poor allocation of resources by allowing public money to be spent on new, often expensive, non-innovative drugs. Although this is not different from more industrialized settings 3333. Organisation for Economic Co-operation and Development. Pharmaceutical pricing policies in a global market. Paris: Organisation for Economic Co-operation and Development; 2008. (OECD Health Policy Studies)., it is problematic for Brazil, where limited resources are available to ensure public coverage 3434. Inter-American Development Bank. Breve 16: ethics of health resource allocation in the Brazilian publicly financed health care system. Washington DC: Inter-American Development Bank; 2016.. Finally, most new drugs entering the market target specific niches and chronic conditions rather than other indications more relevant to public health, such as neglected diseases. Taken together, these findings are contrary to the principles of rational medicine use 3535. Organisation for Economic Co-operation and Development. Focus on health spending. Paris: Organisation for Economic Co-operation and Development; 2015. (OECD Health Statistics)..

Our results corroborate those of previous studies in which no more than 10% of newly approved drugs were considered therapeutic innovations 44. Vitry AI, Shin NH, Vitre P. Assessment of the therapeutic value of new medicines marketed in Australia. J Pharm Policy Pract 2013; 6:2.,66. van Luijn JCF, Gribnau FWJ, Leufkens HGM. Superior efficacy of new medicines? Eur J Clin Pharmacol 2010; 66:445-8.,1414. New products and new indications in 2016: a system that favours imitation over the pursuit of real progress. Prescrire Int 2017; 26:136-9.,2424. Bonfim JRA. O registro de produtos farmacêuticos novos: critérios para a promoção do uso racional de fármacos no Sistema Único de Saúde [Dissertação de Mestrado]. São Paulo: Programa de Pós-graduação em Ciências, Coordenadoria de Controle de Doenças, Secretaria de Estado da Saúde de São Paulo; 2006.,2929. Lexchin J. Health Canada's use of its priority review process for new drugs: a cohort study. BMJ Open 2015; 5:e006816..

One could presume that the partial overlap between the indications approved and the national disease burden confirms that the available therapeutic arsenal is sufficient to treat the conditions contributing the most to DALYs. Yet, this mismatch shows a focus on new drug approvals for specific indications, such as oncology. This might be explained by the fact that market pressures do not necessarily mirror public health needs, especially for health conditions that mainly occur in low to medium income countries. Notwithstanding the importance medicines have in society as a treatment modality and their contribution to health care costs, the production of medicines is the domain of a few large multinational companies, which opt to focus on specific market niches and chronic conditions 3636. Silva HF, Oliveira EA. Panorama da produção de medicamentos no Brasil. In: Osorio-de-Castro CGS, Luiza VL, Castilho SR, Oliveira MA, Jaramillo NM, organizadores. Assistência farmacêutica: gestão e prática para profissionais da saúde. Rio de Janeiro: Editora Fiocruz; 2014. p. 197-205.. Four out of the five medicines considered therapeutically innovative by both CMED and the grouped Prescrire criteria have orphan drug product denomination either in the Europe or in the USA. Within our sample, antineoplastics, immunosupressants, systemic antivirals and antidiabetics were the most frequent therapeutic classes. In an analysis of clinical trials carried out in Brazil 3737. Silva RE, Novaes MRC, Pastor EM, Barragan E, Amato AA. Trends in research involving human beings in Brazil. Rev Panam Salud Pública 2015; 37:118-24., diabetes type 2, breast cancer and bronchial or lung cancer were among the conditions most frequently studied. While research in neglected diseases is a high priority for Brazil, it does not yield returns on investment for the pharmaceutical industry 3737. Silva RE, Novaes MRC, Pastor EM, Barragan E, Amato AA. Trends in research involving human beings in Brazil. Rev Panam Salud Pública 2015; 37:118-24.. Nevertheless, there are two ongoing initiatives by multinational companies regarding neglected diseases in collaboration with the Institute of Drug Technology Farmanguinhos: a product development for a pediatric praziquantel formulation to treat schistosomiasis (now at phase 2); and a cooperative R&D agreement to develop a dengue virus vaccine (at preclinical stage) 3838. International Federation of Pharmaceutical Manufacturers & Associations. Doing our part: innovating to fight neglected tropical diseases. Geneva: International Federation of Pharmaceutical Manufacturers & Associations; 2017..

Our study has shown that only 36 (14.2%) of the new medicines registered during these 13 years were listed for public coverage under the SUS, and that 21 out of the 36 medicines included in these listings were considered non-innovative by both CMED and Prescrire. Such a low number of therapeutic innovations stresses the importance of strengthening pharmacy and therapeutics committees 3939. Tyler LS, Cole SW, May JR, Millares M, Valentino MA, Vermeulen Jr. LC, et al. ASHP guidelines on the pharmacy and therapeutics committee and the formulary system. Am J Health Syst Pharm 2008; 65:1272-83.,4040. Holloway K, Green T. Drug and therapeutics committees: a practical guide. Geneva: World Health Organization; 2003.,4141. Marques DC, Zucchi P. Comissões farmacoterapêuticas no Brasil: aquém das diretrizes internacionais. Rev Panam Salud Pública 2006; 19:58-63.. While they have been established in many health facilities in Brazil, their roles are somewhat limited due to financial and human resources constraints 4141. Marques DC, Zucchi P. Comissões farmacoterapêuticas no Brasil: aquém das diretrizes internacionais. Rev Panam Salud Pública 2006; 19:58-63.. Many new technologies receive poor health technology assessments. About 40% of all the HTA recommendations around the world are negative and so are those of CONITEC 4242. Petramale CA. Nota de esclarecimento à Interfarma. http://conitec.gov.br/images/pdf/Esclarecimentos_Interfarma.pdf (accessed on 26/Feb/2018).
http://conitec.gov.br/images/pdf/Esclare...
. Suggestions were made to improve Anvisa procedures with a focus on added therapeutic value and systematic disclosure of assessment results. These measures would benefit medicine users, health professionals and managers 4343. Gava CM, Bermudez JAZ, Pepe VLE, Reis ALA. Novos medicamentos registrados no Brasil: podem ser considerados como avanço terapêutico? Ciênc Saúde Colet 2010; 15:3403-12..

We found very weak agreement between CMED and Prescrire’s evaluations of therapeutic innovativeness. This can be partly attributed to differences in the organizations, as well as their rationale and criteria. For instance, the Brazilian Chamber considers patent protection as a prerequisite for therapeutic innovation, whereas this is not the case for Prescrire. Intellectual property policies aim to ensure a financial return on investment by introducing a patent protection (and subsequent monopoly) to encourage technology development. Yet a drug that has a patent does not necessarily represent a therapeutic advance, as clinical outcomes are not a key criterion for patent attribution. Nonetheless, both criteria examine the available scientific evidence on the efficacy, safety and effectiveness of a drug. Our results mirror those of a Canadian study that compared drug assessments conducted by the Patented Medicine Prices Review Board (PMPRB) and Prescrire2929. Lexchin J. Health Canada's use of its priority review process for new drugs: a cohort study. BMJ Open 2015; 5:e006816.. In addition, notwithstanding Prescrire’s scientific standards, one could argue that its reviews are led by healthcare professionals living in another social, health and economic context than that of Brazil, and that, therefore, their ratings might not be transferable to other settings. Likewise, the political context in Brazil and changes in the government and at Anvisa might also have affected the CMED ratings. Moreover, assessing the therapeutic innovativeness of a drug compared with existing treatment options depends partly on other available treatments and can therefore vary by setting and over time. Nonetheless, the clarity and validity of the criteria adopted, their evidence-based approach and extensive peer review are all factors that minimize that likelihood.

Some caution is warranted when interpreting these assessments considering the absence of benchmark for the evaluation of the therapeutic value of new medicines. Many of the data used is available at the time of approval and is limited, as new evidence is likely to emerge at a later stage once the medicine has been marketed and used by a larger patient population 44. Vitry AI, Shin NH, Vitre P. Assessment of the therapeutic value of new medicines marketed in Australia. J Pharm Policy Pract 2013; 6:2..

The CMED’s under patent requirement as a prerequisite for therapeutic innovation is highly questionable. Although the existence of a patent presumes demonstrated progress over previous knowledge 1616. Kesselheim AS, Wang B, Avorn J. Defining "innovativeness" in drug development: a systematic review. Clin Pharmacol Ther 2013; 94:336-48., patents are frequently granted based on technical aspects unrelated to a drug’s efficacy or its therapeutic benefits, and consequently play a limited role when ascertaining the quality of pharmaceutical innovation 1616. Kesselheim AS, Wang B, Avorn J. Defining "innovativeness" in drug development: a systematic review. Clin Pharmacol Ther 2013; 94:336-48.,4444. Hemphill CS, Sampat BN. Evergreening, patent challenges, and effective market life in pharmaceuticals. J Health Econ 2012; 31:327-39..

The data in this study were collected in parallel to and cross-referenced from different sources, most of which publicly available, thus contributing to the reliability of our findings. Additional studies would be needed to explore the divergence between therapeutic value assessment criteria and to promote greater harmonization and reproducibility.

This discussion on drug innovation cannot be separated from an analysis of the R&D trends both nationally and globally, as these reflect the interests and priorities of public and private research funders. Some authors have advocated that, to foster future innovation, the current criteria for drug approval should be changed to introduce clear demonstration of added therapeutic value as a requirement to obtain a marketing authorization 44. Vitry AI, Shin NH, Vitre P. Assessment of the therapeutic value of new medicines marketed in Australia. J Pharm Policy Pract 2013; 6:2.,4545. Wieseler B. How, and on whose behalf, should the "added therapeutic value" of a new drug be assessed? Paris: La Revue Prescrire; 2011.,4646. International Society of Drug Bulletins. ISDB declaration on therapeutic advance in the use of medicines. Paris: International Society of Drug Bulletins; 2001.. A multi-stakeholder debate in Brazil put forward a dual role for the State: to stimulate and redirect research and development towards therapeutically innovative medicines treating unmet medical needs, and to reward value based on results 4747. Sandroni M, Tagliati C. Relatório de resultados e desdobramentos do 9º Encontro Nacional de Inovação em Fármacos e Medicamentos. http://www.ipd-farma.org.br/uploads/paginas/file/2015/Relatorio%20Resultados%20e%20Desdobramentos%209o%20ENIFarMed_v05.pdf (accessed on 26/Feb/2018).
http://www.ipd-farma.org.br/uploads/pagi...
. While industry stakeholders in Brazil agree that innovation should bring therapeutic gains and real benefits for patients, they have called for early price-setting discussions, before investments are made in R&D, claiming that this would encourage national innovation 4747. Sandroni M, Tagliati C. Relatório de resultados e desdobramentos do 9º Encontro Nacional de Inovação em Fármacos e Medicamentos. http://www.ipd-farma.org.br/uploads/paginas/file/2015/Relatorio%20Resultados%20e%20Desdobramentos%209o%20ENIFarMed_v05.pdf (accessed on 26/Feb/2018).
http://www.ipd-farma.org.br/uploads/pagi...
. Ultimately, clinical research priorities should be based on local epidemiological data, with value placed on studies that examine important health aspects and respond to current and future gaps in services, thus protecting systems from becoming reliant on a few multinational pharmaceutical companies that dominate the sector 4848. World Health Organization. Research for universal health coverage. Geneva: World Health Organization; 2013.. Undoubtedly, this approach would benefit Brazil, a major emerging economy that still faces many public health challenges while striving to provide universal health coverage.

Acknowlegments

We would like to thank Aukje Mantel-Teeuwisse, Barbara Mintzes and Joel Lexchin for reviewing this manuscript. We are also grateful to the Executive Secretary of the Brazilian Chamber of Drug Market Regulation (CMED) and Prescrire for granting access to their data.

References

  • 1
    Morgan S, Lopert R, Greyson D. Toward a definition of pharmaceutical innovation. Open Med 2008; 2:E4-7.
  • 2
    Soto Álvarez JS. Grado de innovación de los nuevos medicamentos: propuesta de criterios que deben tenerse en cuenta para su valoración. Med Clin (Barc) 2009; 132:481-3.
  • 3
    Davis C, Abraham J. Rethinking innovation accounting in pharmaceutical regulation: a case study in the deconstruction of therapeutic advance and therapeutic breakthrough. Sci Technol Human Values 2011; 36:791-815.
  • 4
    Vitry AI, Shin NH, Vitre P. Assessment of the therapeutic value of new medicines marketed in Australia. J Pharm Policy Pract 2013; 6:2.
  • 5
    Caetano R. Avaliação de tecnologias em saúde: contribuições para a assistência farmacêutica. In: Osorio-de-Castro CGS, Luiza VL, Castilho SR, Oliveira MA, Jaramillo NM, organizadores. Assistência farmacêutica: gestão e prática para profissionais da saúde. Rio de Janeiro: Editora Fiocruz; 2014. p. 221-38.
  • 6
    van Luijn JCF, Gribnau FWJ, Leufkens HGM. Superior efficacy of new medicines? Eur J Clin Pharmacol 2010; 66:445-8.
  • 7
    Puntmann I, Schmacke N, Melander A, Lindberg G, Muhlbauer B. EVITA: a tool for the early evaluation of pharmaceutical innovations with regard to therapeutic advantage. BMC Clin Pharmacol 2010; 10:5.
  • 8
    Center for Drug Evaluation and Research, U.S. Food and Drug Administration. Novel drugs 2015 summary. Silver Spring: U.S. Food and Drug Administration; 2016.
  • 9
    Aronson JK, Ferner RE, Hughes DA. Defining rewardable innovation in drug therapy. Nat Rev Drug Discov 2012; 11:253-4.
  • 10
    Kaplan W, Wirtz VJ, Mantel-Teeuwisse A, Stolk P, Duthey B, Laing R. Priority medicines for Europe and the world: 2013 update. Geneva: World Health Organization; 2013.
  • 11
    Nathan C. Aligning pharmaceutical innovation with medical need. Nat Med 2007; 13:304-8.
  • 12
    Eichler HG, Thomson A, Eichler I, Schneeweiss S. Assessing the relative efficacy of new drugs: an emerging opportunity. Nat Rev Drug Discov 2015; 14:443-4.
  • 13
    Motola D, De Ponti F, Poluzzi E, Martini N, Rossi P, Silvani MC, et al. An update on the first decade of the European centralized procedure: how many innovative drugs? Br J Clin Pharmacol 2006; 62:610-6.
  • 14
    New products and new indications in 2016: a system that favours imitation over the pursuit of real progress. Prescrire Int 2017; 26:136-9.
  • 15
    Centro Andaluz de Documentación e Información de Medicamentos. Nuevos medicamentos: qué son las fichas. http://www.cadime.es/es/que_son_las_fichas_de_novedad_terapeutica.cfm (accessed on 26/Feb/2018).
    » http://www.cadime.es/es/que_son_las_fichas_de_novedad_terapeutica.cfm
  • 16
    Kesselheim AS, Wang B, Avorn J. Defining "innovativeness" in drug development: a systematic review. Clin Pharmacol Ther 2013; 94:336-48.
  • 17
    Die information für ärzte und apotheker neutral, unabhängig und anzeigenfrei Berlin. https://www.arznei-telegramm.de/01index.php3 (accessed on 26/Feb/2018).
    » https://www.arznei-telegramm.de/01index.php3
  • 18
    Naci H, Ioannidis JP. How good is "evidence" from clinical studies of drug effects and why might such evidence fail in the prediction of the clinical utility of drugs? Annu Rev Pharmacol Toxicol 2015; 55:169-89.
  • 19
    Claypool R. Pharmaceutical industry profits exceed industry's self-reported research and development. https://www.citizen.org/sites/default/files/pharma-profits-and-r-and-d-report.pdf (accessed on 26/Feb/2018).
    » https://www.citizen.org/sites/default/files/pharma-profits-and-r-and-d-report.pdf
  • 20
    Wirtz VJ, Hogerzeil HV, Gray AL, Bigdeli M, de Joncheere CP, Ewen MA, et al. Essential medicines for universal health coverage. Lancet 2017; 389:403-76.
  • 21
    Associação da Indústria Farmacêutica de Pesquisa. Guia 2017 Interfarma. https://www.interfarma.org.br/guia/guia-2017/dados-do-setor/ (accessed on 26/Feb/2018).
    » https://www.interfarma.org.br/guia/guia-2017/dados-do-setor/
  • 22
    Câmara de Regulação do Mercado de Medicamentos. Resolução nº 2, de 5 de março de 2004. Diário Oficial da União 2004; 6 mai.
  • 23
    Lemgruber A. Avaliação e regulação de preços de medicamentos novos: a experiência do Brasil. In: Organização Pan-Americana da Saúde, organizador. O acesso aos medicamentos de alto custo nas Américas: contexto, desafios e perspectivas. Brasília: Organização Pan-Americana da Saúde, Ministério da Saúde, Ministério das Relações Exteriores; 2009. p. 55-7.
  • 24
    Bonfim JRA. O registro de produtos farmacêuticos novos: critérios para a promoção do uso racional de fármacos no Sistema Único de Saúde [Dissertação de Mestrado]. São Paulo: Programa de Pós-graduação em Ciências, Coordenadoria de Controle de Doenças, Secretaria de Estado da Saúde de São Paulo; 2006.
  • 25
    Vidotti CCF, Castro LLC. Fármacos novos e necessidades do Sistema Único de Saúde no Brasil. Espaç Saúde 2009; 10:7-11.
  • 26
    Gerência de Avaliação Econômica de Novas Tecnologias, Agência Nacional de Vigilância Sanitária. Efeitos da Resolução CMED nº 02/04 no processo de análise de preços de novos medicamentos. Brasília: Agência Nacional de Vigilância Sanitária; 2013.
  • 27
    WHO International Working Group for Drug Statistics Methodology; WHO Collaborating Centre for Drug Statistics Methodology. ATC/DDD index 2017. https://www.whocc.no/atc_ddd_index/ (accessed on 26/Feb/2018).
    » https://www.whocc.no/atc_ddd_index/
  • 28
    Prescrire's ratings system: new drugs and indications, at a glance. http://english.prescrire.org/en/81/168/53692/0/NewsDetails.aspx (accessed on 02/Apr/2019).
    » http://english.prescrire.org/en/81/168/53692/0/NewsDetails.aspx
  • 29
    Lexchin J. Health Canada's use of its priority review process for new drugs: a cohort study. BMJ Open 2015; 5:e006816.
  • 30
    Brasil. Lei nº 12.401, de 28 de abril de 2011. Altera a Lei nº 8.080, de 19 de setembro de 1990, para dispor sobre a assistência terapêutica e a incorporação de tecnologia em saúde no âmbito do Sistema Único de Saúde - SUS. Diário Oficial da União 2011; 29 abr.
  • 31
    Leite IC, Valente JG, Schramm JM, Daumas RP, Rodrigues RN, Santos MF, et al. Burden of disease in Brazil and its regions, 2008. Cad Saúde Pública 2015; 31:1551-64.
  • 32
    Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33:159-74.
  • 33
    Organisation for Economic Co-operation and Development. Pharmaceutical pricing policies in a global market. Paris: Organisation for Economic Co-operation and Development; 2008. (OECD Health Policy Studies).
  • 34
    Inter-American Development Bank. Breve 16: ethics of health resource allocation in the Brazilian publicly financed health care system. Washington DC: Inter-American Development Bank; 2016.
  • 35
    Organisation for Economic Co-operation and Development. Focus on health spending. Paris: Organisation for Economic Co-operation and Development; 2015. (OECD Health Statistics).
  • 36
    Silva HF, Oliveira EA. Panorama da produção de medicamentos no Brasil. In: Osorio-de-Castro CGS, Luiza VL, Castilho SR, Oliveira MA, Jaramillo NM, organizadores. Assistência farmacêutica: gestão e prática para profissionais da saúde. Rio de Janeiro: Editora Fiocruz; 2014. p. 197-205.
  • 37
    Silva RE, Novaes MRC, Pastor EM, Barragan E, Amato AA. Trends in research involving human beings in Brazil. Rev Panam Salud Pública 2015; 37:118-24.
  • 38
    International Federation of Pharmaceutical Manufacturers & Associations. Doing our part: innovating to fight neglected tropical diseases. Geneva: International Federation of Pharmaceutical Manufacturers & Associations; 2017.
  • 39
    Tyler LS, Cole SW, May JR, Millares M, Valentino MA, Vermeulen Jr. LC, et al. ASHP guidelines on the pharmacy and therapeutics committee and the formulary system. Am J Health Syst Pharm 2008; 65:1272-83.
  • 40
    Holloway K, Green T. Drug and therapeutics committees: a practical guide. Geneva: World Health Organization; 2003.
  • 41
    Marques DC, Zucchi P. Comissões farmacoterapêuticas no Brasil: aquém das diretrizes internacionais. Rev Panam Salud Pública 2006; 19:58-63.
  • 42
    Petramale CA. Nota de esclarecimento à Interfarma. http://conitec.gov.br/images/pdf/Esclarecimentos_Interfarma.pdf (accessed on 26/Feb/2018).
    » http://conitec.gov.br/images/pdf/Esclarecimentos_Interfarma.pdf
  • 43
    Gava CM, Bermudez JAZ, Pepe VLE, Reis ALA. Novos medicamentos registrados no Brasil: podem ser considerados como avanço terapêutico? Ciênc Saúde Colet 2010; 15:3403-12.
  • 44
    Hemphill CS, Sampat BN. Evergreening, patent challenges, and effective market life in pharmaceuticals. J Health Econ 2012; 31:327-39.
  • 45
    Wieseler B. How, and on whose behalf, should the "added therapeutic value" of a new drug be assessed? Paris: La Revue Prescrire; 2011.
  • 46
    International Society of Drug Bulletins. ISDB declaration on therapeutic advance in the use of medicines. Paris: International Society of Drug Bulletins; 2001.
  • 47
    Sandroni M, Tagliati C. Relatório de resultados e desdobramentos do 9º Encontro Nacional de Inovação em Fármacos e Medicamentos. http://www.ipd-farma.org.br/uploads/paginas/file/2015/Relatorio%20Resultados%20e%20Desdobramentos%209o%20ENIFarMed_v05.pdf (accessed on 26/Feb/2018).
    » http://www.ipd-farma.org.br/uploads/paginas/file/2015/Relatorio%20Resultados%20e%20Desdobramentos%209o%20ENIFarMed_v05.pdf
  • 48
    World Health Organization. Research for universal health coverage. Geneva: World Health Organization; 2013.

Additional informations

  • Additional informations

    ORCID: Rogério Hoefler (0000-0003-3851-7833); Teresa Leonardo Alves (0000-0003-4967-1916); Hubertus G. Leufkens (0000-0002-7175-1499); Janeth de Oliveira Silva Naves (0000-0003-4793-2836).

Publication Dates

  • Publication in this collection
    23 May 2019
  • Date of issue
    2019

History

  • Received
    07 Apr 2018
  • Reviewed
    24 Oct 2018
  • Accepted
    05 Nov 2018
Escola Nacional de Saúde Pública Sergio Arouca, Fundação Oswaldo Cruz Rio de Janeiro - RJ - Brazil
E-mail: cadernos@ensp.fiocruz.br