On-line version ISSN 1680-5348
Print version ISSN 1020-4989
Rev Panam Salud Publica vol.26 n.2 Washington Aug. 2009
INVESTIGACIÓN ORIGINAL ORIGINAL RESEARCH
Evaluación de la calidad de los datos de vacunación generados por el sistema nacional de registro individual en Uruguay, 2006
Olivier RonveauxI; Fernando ArrietaII; Sergio CurtoIIHilda LauraniII; M. Carolina Danovaro-HollidayIII
IExpanded Programme of Immunization, World Health Organization, Geneva, Switzerland. Send correspondence and reprint requests to: Olivier Ronveaux, Technical Officer, Department of Immunization, Vaccines and Biologicals, World Health Organization, 20 Avenue Appia, 1211 Geneva 27, Switzerland; telephone: +41 22 7911079; e-mail: firstname.lastname@example.org
IIComisión Honoraria para la Lucha Antituberculosa y Enfermedades Prevalentes, Montevideo, Uruguay
IIIPan American Health Organization, Washington, DC, United States of America
ABSTRACTOBJECTIVE: The nominal registration system of Uruguay's national immunization program (NIP) tracks administered vaccines on a paper form filled out after each vaccination and collated into a national database, thus allowing for individual follow-up. This study performed a comprehensive assessment of the quality of Uruguay's immunization data in November 2006 to evaluate the validity of the information and to confirm the high national immunization coverage reported by the program.
METHODS: The research team analyzed the concordance of the operational-level numerators (infant immunization data from 18 public and private vaccination centers in six country departments) with department- and national-level data, and compared the national-level (NIP) infant denominators with other official sources. A standardized questionnaire was used to evaluate system performance at the operational (vaccination center), department, and national level. Rapid house-to-house monitoring was conducted to generate additional coverage estimates.
RESULTS: Numerator accuracy throughout the data flow was 100%, and national-level denominators appeared to be exhaustive. Overall system performance was excellent (proper archiving and recording of form data, sufficient supply of forms, timely flow of information, adequate defaulter tracing practices and computer system security). The main weaknesses were the degree of data analysis and feedback to peripheral levels. House-to-house monitoring showed high overall immunization coverage (97%).
CONCLUSION: Uruguay's NIP registration system produces remarkably reliable information, ensuring valid measurement of immunization coverage. In addition, by allowing for monitoring of each child's current vaccination status, it facilitates management of interventions designed to reduce vaccination default and thus helps achieve the country's high level of coverage.
Key words: Immunization; health surveys; information systems; data collection; registries; Uruguay.
OBJETIVOS: El sistema de registro nominal del programa nacional de vacunación (PNV) de Uruguay registra las vacunas administradas en un modelo en papel que se llena después de cada vacunación y se introduce en una base de datos nacional, lo que permite el seguimiento individual. En este estudio, realizado en noviembre de 2006, se hizo un análisis integral de la calidad de los datos de vacunación en Uruguay para evaluar la validez de la información y confirmar la alta cobertura nacional de vacunación informada por el programa.
MÉTODOS: Se analizó la concordancia de los numeradores operacionales (datos de vacunación infantil de 18 centros de vacunación públicos y privados de seis departamentos del país) con los datos departamentales y nacionales, y se compararon los denominadores del PNV infantil a nivel nacional con los de otras fuentes oficiales. Se empleó un cuestionario estandarizado para evaluar el desempeño del sistema en los niveles operacional (centros de vacunación), departamental y nacional. Se realizó un monitoreo rápido de casa en casa para obtener estimados adicionales de la cobertura.
RESULTADOS: La precisión de los numeradores en todo el flujo de datos fue de 100% y los denominadores a nivel nacional al parecer eran exhaustivos. El desempeño general del sistema fue excelente (archivo y registro adecuados de los datos, suministro suficiente de modelos, flujo oportuno de información, práctica adecuada de rastreo de fallas y sistema apropiado de seguridad informática). Las principales debilidades fueron el grado de análisis de los datos y la retroalimentación a las instancias periféricas. El monitoreo de casa en casa mostró una alta cobertura general de vacunación (97%).
CONCLUSIONES: El sistema uruguayo de registro del PNV genera una información notablemente confiable, lo que asegura la validez de la medición de la cobertura de vacunación. Además, al permitir el monitoreo del estado de vacunación actual de cada niño, facilita la aplicación y el control de intervenciones diseñadas para reducir las omisiones en la vacunación y ayudar así a alcanzar un alto nivel de cobertura en el país.
Palabras clave: Vacunación; encuestas epidemiológicas; sistemas de información; recolección de datos; sistema de registros; Uruguay.
In November 2006, the Pan American Health Organization (PAHO) Immunization Unit conducted an international evaluation of Uruguay's national immunization program (NIP) (1). This type of external evaluation is part of PAHO's policy to provide an opportunity for member states to improve their immunization programs. At the request of Uruguay's Ministry of Public Health (MOH), a specific evaluation of the quality of immunization data at the operational (vaccination center), country department, and national level was also conducted.
Uruguay borders Argentina and Brazil in eastern South America, comprising 176 220 square kilometers divided into 19 country departments (state-like regions). It has an estimated population of 3.3 million and an average of about 48000 births per year (national census 2004). The country's gross national income (per capita) is US$ 4360 (2). The immunization registration system in Uruguay collects data on childhood vaccinations by requiring the submission of a paper form filled out after each administered vaccine and collated into a centralized (national) database, and thus allows for tracking and follow-up of each child's vaccination status countrywide. This nominative system is the oldest in the Americas and was therefore a particularly interesting target for an extensive evaluation.
In 2002 and 2006, PAHO's Technical Advisory Group (TAG) on Vaccine-preventable Diseases recommended that countries in the Americas region periodically evaluate the accuracy, consistency, completeness, and timeliness of the data produced by their immunization monitoring systems (3). Uruguay has been reporting high immunization coverage for many years. For example, coverage for the third dose of the diphtheria-tetanus-pertussis vaccine (DTP3) in 2004, 2005, and 2006 was 94%, 96%, and 95%, respectively, and coverage for the first dose of the measles-mumps-rubella vaccine (MMR1) has been 94% or higher since 2001 (4). This study used the methodology of the World Health Organization (WHO) data quality self-assessment (DQS) tool (5, 6) to assess the quality of Uruguay's immunization information (particularly the validity of the numerators and denominators used to produce the estimated rates of coverage reported by the NIP) and to determine the strengths and shortcomings of the NIP registration system.
MATERIALS AND METHODS
NIP registration system
Uruguay's national immunization program is a model of public-private inter-institutional cooperation centered around the joint operations of the Ministry of Health, the Honorary Committee to Fight Tuberculosis and other Prevalent Diseases (Comisión Honoraria para la Lucha Antituberculosa y Enfermedades Prevalentes, CHLA-EP), and vaccination facilities countrywide (operating under the CHLA-EP and country department authorities) (Figure 1). The CHLA-EP, which is under the direction of the MOH and receives technical advice from the National Advisory Committee on Immunization, oversees NIP operations, including training and supervision at the public and private facilities that provide vaccination services. It comprises 23 peripheral CHLA-EP offices in Uruguay's 19 departments reporting to the central office in the capital city of Montevideo. Under national law, all vaccinations are free and mandatory, even those provided in the private sector.
The CHLA-EP implemented Uruguay's current immunization registration system in 1987. The system is based on the use of a single paper form to record data on each vaccination administered. Vaccination centers nationwide (public and private) fill out the form for each child they immunize (listing all vaccines administered in a given session) and send the original (hard copy) to the department-level CHLA-EP office (keeping a paper copy on file). Each vaccination form includes the following information:
Child: full name, sex, date of birth, maternity ward in which he/she was born, and national identification number.
Mother: full name, street address (including neighborhood and police district), and phone number. An alternate address and phone number are also requested.
Vaccination center: center identification code.
Vaccine: biological agent(s) administered within the 0-12-month schedule (Bacillus Calmette-Guérin (BCG) vaccine against tuberculosis at birth; four doses of poliomyelitis and DTP-hepatitis B-Haemophilus influenzae type b (Hib) (pentavalent) vaccines; first dose of MMR vaccine; and varicella vaccine).
The registration system does not record the following vaccinations: tetanus-diphtheria (Td) (for adults), hepatitis B (for adolescents), and influenza (for annual campaigns). Since 2005, the NIP vaccination forms have also tracked the second (school entry) dose of MMR for all children in Uruguay living outside Montevideo.
For children born in the department where they received the vaccination, the data from the single paper vaccination forms are entered into the department-level database as a one-line record (per child). For children born outside the department where they received the vaccination, the single paper forms are sent each month to their department of birth, where the corresponding data are entered into the department-level database and the child's original record is updated to reflect any changes in residency (Figure 2). Every one or two weeks, all department-level offices submit an electronic backup of their databases to the national office, and all data are collated into the central CHLA-EP electronic database ("NIP database"). No other report (e.g., monthly or other regular summary of vaccine doses administered) is sent to the national office by any of the entities involved in vaccination administration/registration (i.e., the department-level CHLA-EP offices and public and private vaccination centers), which are not required to meet a monthly or annual immunization target.
The full content of the NIP database, which was programmed in COBOL, is only accessible at the national CHLA-EP office in Montevideo, which manages defaulters (children identified in the NIP database as not having received the required vaccinations according to the prescribed schedule) and sends out monthly reminders alerting health centers in their current department of residency about the missed vaccinations.
The denominators used to calculate national immunization coverage are derived from the registration system itself. In Uruguay, 99.5% of deliveries are attended by trained personnel (7), and the first vaccination (BCG for newborns) is administered as part of the national health system's maternity services. Along with registration in the NIP system, BCG vaccination is a necessary condition for the newborn's discharge. Therefore, almost 100% of newborns are registered in the NIP database and receive the BCG vaccine. Interestingly, the vaccination form is filled out even if the child does not receive the BCG vaccine (for medical contraindications), allowing for tracking of all newborns by the NIP database. Immunization coverage is measured by registered birth cohort (rather than by year using an annual fixed target, the standard parameter for coverage analysis in most countries).
The survey was carried out in six of Uruguay's 19 departments (Artigas, Canelones, Colonia, Maldonado, Montevideo, and Rivera). The sample was obtained using WHO's DQS methodology for convenience sampling (5). To ensure the evaluation findings had the highest possible impact, departments with the lowest immunization coverage rates (<90% in at least one of the previous three years) and those with geographical vulnerability (i.e., border areas with relatively high rates of migration) were selected for study. In addition, to capture a cross-section of the population, one public and one private vaccination center within each of the six departments were randomly selected for study, along with the department-level CHLA-EP office. Six teams of two completed training in data collection techniques and participated in a field test (6 November 2006) before conducting the survey (7-8 November 2006). Three core elements of the NIP registration system were assessed:
1. System performance (quality): All levels (vaccination center, department, and national) were studied to assess analysis and feedback practices and determine if vaccination forms were provided in adequate quantity, filled out completely and accurately, properly archived, and forwarded throughout the system on time.
2. Numerator accuracy/concordance (throughout data flow) (Figure 2): At the department level, the data collection teams identified the first 50 newborns registered in the NIP system and vaccinated for BCG and MMR (first dose) in August 2006, based on the original vaccination forms, and verified that the children were included in the NIP database. This process was repeated for the second MMR dose (in August 2006).
At the vaccination center level, the data collection teams verified that a copy of the vaccination forms for the first 10 children born August 2005 and earlier and vaccinated at the center with the first dose of MMR (according to the NIP database) could be retrieved from the center's records.
For the studies at both the department and vaccination center level, sample sizes were determined based on the estimated number of forms one data collection team (consisting of two individuals) could review in one day.
3. Denominator accuracy: The 2004 and 2005 birth cohorts at the national level (i.e., the number of registered births in the NIP database) were compared with three other sources: country projections from the 2004 census, the national civil registry, and the registration system for the early congeni-tal hypothyroidism detection program (measuring thyroid-stimulating hormone or TSH). Two-sample capture-recapture analysis was also performed (8) to ascertain the proportion of children tracked by the NIP database. Record matching was based on first name, last name, and date of birth. Drawing from the 2005 birth cohort, the study team 1) randomly selected NIP database records for 50 children from each of the six surveyed departments (for a total of 300 records) and searched for the corresponding data in the national civil registry and 2) randomly selected national civil registry records for nine children from each of the six surveyed departments (for a total of 54 records) and searched for the corresponding data in the NIP database. The size of the sample drawn from the national civil registry was limited to accommodate logistical issues arising from that system's paper-only format.
Immunization coverage was validated using the rapid house-to-house monitoring methodology known as RCM (rapid coverage monitoring) (9)-non-representative assessment of coverage in a specific community using a small convenience sample. The RCM was conducted in areas presumed to have low immunization coverage, according to the department-level CHLA-EP administrator. Each of the six data collection teams performed RCM to determine vaccination status for first-dose MMR among 20 children aged 24 to 59 months and second-dose MMR among 20 children aged 5 to 9 years. Vaccination status was determined based on direct observation of the survey households' child immunization cards (issued to parents for recording their child's vaccinations) or, if no card was available, historical recall (child caretakers' recollection of past immunizations). The accuracy and completeness of the child immunization cards were also evaluated.
Archiving and form supply. At all vaccination centers, the data collection teams had access to the original vaccination forms, which were filed in chronological order. None of the centers were missing original vaccination forms at the time of the survey (7-8 November 2006), and no department-level CHLA-EP office or vaccination center experienced a shortage of vaccination forms or immunization cards during that calendar year.
Data timeliness. For children immunized in their department of birth, the reported frequency for sending the completed (paper) vaccination forms from the vaccination centers to the departmental level was either weekly or biweekly (with half of the centers following the first schedule, and half following the second). These data transfer frequencies were verified in 17 out of the 18 vaccination centers visited (with one center recording a 27-day turnaround time for submitting its most recent batch of forms at the time of the study). For children immunized outside their department of birth, the reported frequency for sending completed forms to the departmental level was once per month. According to the department-level CHLA-EP offices, the subsequent transfer of the forms to the national level occurred on either a weekly or biweekly basis (with the same 50/50 distribution of centers). This frequency of data transfer was confirmed by the form submission dates recorded throughout the system.
Completeness of vaccination forms. The two fields of the vaccination form in which information was most commonly missing, particularly for vaccines other than BCG, were "mother's telephone number" and "alternate telephone number" (Table 1).
Analysis and feedback at national level. Data analyses are produced by the NIP database software program in the form of standardized reports or other outputs, and no other specific analyses that could serve to guide the program are conducted. For instance, there is no routine analysis of vaccination timeliness (e.g., to determine if the third dose of pentavalent vaccine or the first dose of MMR, which should be administered before ages 1 and 2, respectively, are provided on time), and no written (bulletin-type) feedback on coverage or other types of data is sent to NIP department-level administrative entities. The backup of the NIP database is performed daily and is well documented. Major system operations can be tracked for quality control, including the number of children vaccinated versus vaccine requests, and defaulter tracing.
There was a 100% match between the number of completed paper vaccination forms stored at the department level (submitted by the vaccination centers) and the number of children listed as vaccinated in the electronic department-level database (collated into and retrieved from the NIP database) for both the first and second MMR doses (Table 2).
Concordance between the number of copies of completed paper vaccination forms filed at the vaccination centers and the NIP database was also 100% (Table 3).
Although vaccination forms are filled out for children born outside the country but currently residing in Uruguay, the resulting data are not entered into the NIP database. Therefore, the information on these individuals is not quantified, and no reminders are generated by the NIP system regarding their immunization schedule.
All sources examined to determine the accuracy of the NIP denominator (the national civil registry, 2004 national census projections, and the hypothyroidism detection program registry) showed very similar figures for the 2004 and 2005 birth cohorts (Table 4) in terms of order of magnitude. One interesting finding that emerged in comparing the official data was that the NIP denominator was higher (by 2.5%) than that of the civil registry.
In the capture-recapture analysis, 272 (90.6%) of the 300 records selected from the NIP database were found in the national civil registry, and all (100%) of the 54 records selected from the national civil registry were found in the NIP database. Based on both samples, the accuracy of the denominator provided by the NIP database could be estimated at 100% (95% confidence interval (CI): 98.8-100%) of the actual number of births in Uruguay.
Rapid coverage monitoring
The six rounds of RCM (one for each department surveyed) validated the high coverage levels reported by the country (Table 5) and indicated a high level of completeness and accuracy in form completion (with all 193 immunization cards examined during the RCM reported as properly filled in).
Valid immunization coverage data is a prerequisite for appropriate decision-making and immunization program guidance and is also crucial for maintaining credibility with program financers. Improving the quality of the reported data is a huge challenge for immunization programs (10) and a major priority of WHO's Global Framework for Immunization Monitoring and Surveillance (11), because improving immunization coverage measurement seems to improve the coverage itself (12).
This study used the DQS methodology (5) for its comprehensive assessment of immunization data quality in Uruguay. Country departments with the weakest immunization coverage were selected for study in an attempt to gather data for the assessment and support local public health interventions. Due to this selection bias, caution must be taken when generalizing the study findings to the entire country. To address this limitation, the study used a relatively large sample, encompassing one-third of the departments in the country and including both urban and rural populations. Furthermore, within each department, vaccination facilities were selected randomly. The RCM focused on the two-dose MMR vaccine (using it as a proxy for all vaccines) because its estimated coverage rate is similar to those of Uruguay's other required vaccines, and its two-dose delivery schedule falls between the four dose schedule required for DTP and the single-dose required for BCG and varicella. However, because the data flow is similar for all vaccination forms, and there are no particular incentives for achieving specific vaccine targets, the authors strongly believe the immunization results obtained are representative of the other antigens as well, and that the picture provided by the current study reflects the immunization situation nationwide.
This study demonstrates that Uruguay's national individual immunization registration system provides accurate information, validating the high immunization coverage reported by the country based on a denominator even more complete than the one from the national civil registry and on numerators whose accuracy was reflected in perfect matches system-wide in quality checks conducted at all levels of data flow.
The main advantages of Uruguay's NIP registration system include the following six elements:
1. The use of a cohort-based denominator versus an annual fixed target. The majority of immunization registration systems still use denominators generated from census data at the central level (made available at the beginning of each year) and require regular reporting of aggregated immunization data from peripheral levels. However, annual denominators from census projections may be imprecise, because the census data, the fertility/mortality indicators, and/or the migration data (particularly at the lower geographical level) may be outdated.
The denominator provided by the NIP database takes all children born in the country into account and thus provides a more accurate estimate than other official sources (e.g. the national civil registry).
2. Collection of information for every infant, allowing for adequate individual follow-up, and defaulter tracing, a key element in ensuring high immunization coverage levels (12). The close monitoring of defaulters can also improve the timeliness of vaccinations, a factor that will become particularly important with the introduction of some of the newer vaccines, such the one against rotavirus (13). The only shortcoming of the Uruguayan NIP registry identified by this study is that children born outside the country are not included, an aspect of the system that the authors recommend be rectified.
3. Its simplicity, stemming from the use of a paper vaccination form, and a streamlined flow of information. The evaluation showed that 1) the system was well understood by all those involved in the data collection process and 2) its performance was excellent. The data are complete (although the level of completion of some of the form fields could be improved) and are transmitted throughout the system on time.
4. The centralized management of immunization coverage as well as vaccination defaulters, facilitated by the lack of pre-set coverage targets for vaccination centers and other department-level administrative entities, which eliminates problems that may occur in decentralized systems, such as data inflation (as a result of institutional pressure to meet annual targets) and data loss (when children are vaccinated outside their area of residence). However, these advantages needs to be weighed against the usefulness of setting local targets for immunization delivery, which may be beneficial in some settings by encouraging vaccination facilities to monitor their activities and providing an incentive for improved performance.
5. The provision of information on entire birth cohorts, which allows for specific analyses (e.g., vaccination timeliness (14), vaccination status by sex, and vaccination status by residential area) and helps pinpoint low immunization coverage. This, in turn, helps identify high-risk groups (by cohort or residence) and thus strengthens interventions to control vaccine-preventable diseases. A correct denominator also allows for accurate vaccine management and forecasting, a definite advantage considering the new vaccines on the horizon and the increased pressure they will exert on the vaccine management system. Due to the wide availability of the information collected, the range of analyses conducted could be expanded, which would help decentralize decision-making.
6. The security of the system and the good design and reliability of the database software. The system and software also provide some degree of built-in flexibility, allowing for the adjustments that would be necessary to accommodate new requirements and changes (e.g., the introduction of new antigens in the national schedule, integration with TSH results, etc.) However, its flexibility has limits. For example, the database is programmed in COBOL, a now-obsolete programming language. This means that incorporating new types of analysis into the system requires the assistance of a specialized programmer. Redeveloping the database with a new, higher-performing language would not only address the above-mentioned challenges but would also increase the functionality of the software (e.g., extending its reach to peripheral levels, and enabling the development of linkages with geographic information and other systems). In addition, decentralized computer technology and new tools for data entry and transmission (which were either unavailable or underutilized when the system was originally designed) are now available. These tools include personal data assistants (PDAs), for paperless entry; barcode printing and scanning, for assigning and tracking information; fingerprint scanning, for streamlining patient identification (although this technology has yet to be adapted for use in infants (15)); and improved coverage of low-cost Internet access (16).
The running costs of the Uruguayan system at the vaccine-delivery level are likely to be quite low, as the paper vaccination form is the only item that is filled out and kept on file. This type of system ultimately requires less paper and time than other systems in which the health worker has to record daily tally sheets and registries and compile a monthly aggregate report. In addition, the cost of form distribution is reduced by consolidating form shipments with those of vaccines and immunization supplies. Program expenses incurred at departmental and national offices, which include the cost of the human resources needed to carry out electronic data entry and quality control, and purchase and maintenance costs for the requisite hardware and software, are likely comparable with those of other immunization systems, based on aggregated administrative reports in the region.
Key factors of success
A key factor in the reliability of Uruguay's national immunization information is the registration system's ability to capture and track almost all infants. To maintain this system strength, special attention must be focused on ensuring the inclusion of individuals marginalized by ethnic, regional, or socioeconomic factors, and those born outside the country. As nearly all babies in Uruguay are delivered in a hospital, the country has an effective means of facilitating enrollment in the immunization registry, resulting in high levels of registration. While civil registries also provide a good entry point for immunization enrollment, they must be accurate and up-to-date in order to be effective conduits into the immunization registry. In countries where the coverage in these types of registries is less than ideal, the use of incentives (e.g., linking immunization to eligibility for school entry or various social insurance schemes) could be considered. Other factors crucial to the success of Uruguay's NIP registration system include ownership of the system by all stakeholders and timely data transfer. This latter criterion has not been a challenge in Uruguay due to its relatively small size, which allows for rapid data flow throughout the system.
The results of this study indicate the information from Uruguay's immunization registration system is remarkably reliable. Use of the DQS methodology allowed the authors to prove the system's effectiveness by validating the data collected at each different level (vaccination center, department, and national). The numerator data used by the national and department-level registries and the vaccination centers were identical; the national system denominators were exhaustive; and registry users system-wide demonstrated a uniform, clear understanding of the program. This study also shows that nominal immunization registries can be used successfully to trace and reduce the number of vaccination defaulters by ensuring each child completes a full immunization schedule. Achieving full immunization coverage is essential not only for the health of individuals but also for reducing the burden of vaccine-preventable diseases in the population as a whole. In this context, Uruguay's immunization registration system should serve as a model for implementation in other countries.
Acknowledgments. The authors wish to express their gratitude to Uruguay's Ministry of Health and to Roberto Calvo, director general of the General Directorate of the Civil Registry (Dirección General del Registro de Estado Civil, DGREC). They also sincerely acknowledge the excellent work of the personnel who provide immunizations in Uruguay, and express their profound appreciation to Béatrice Carpano and Simona Zipursky for providing editorial assistance.
Conflict of interest statement. The evaluation of the immunization program was financed by PAHO. The authors have no conflicts of interest to declare.
1. Pan American Health Organization. Summary of EPI evaluation in Uruguay, 2006. Immun Newsl. 2007;29(2):16. [ Links ]
2. World Bank. World development indicators. Washington: World Bank; 2007. [ Links ]
3. Pan American Health Organization. Protecting the health of the Americas: moving from child to family immunization. Final report. XVII Meeting of the Technical Advisory Group on Vaccine-preventable Diseases, Guatemala City, Guatemala, 25-27, July 2006. Washington: PAHO; 2007. [ Links ]
4. World Health Organization [Internet]. Geneva: WHO; c2008 [updated 2008 Dec 16]. Immunization surveillance, assessment and monitoring. WHO-UNICEF estimates of DTP3 coverage; [about 5 screens]. Available from: http://www.who.int/immunization_ monitoring/en/globalsummary/timeseries/tswucoveragedtp3.htm. [ Links ]
5. World Health Organization. The immunization data quality self-assessment (DQS) tool. Geneva: WHO; 2005. (WHO/IVB/05.04). [ Links ]
6. Machado Cruz V, Salas Peraza D, Cervantes M, Esquevel G, García M, Hernández O, et al. Immunization data quality self-assessment, Costa Rica, November 2005 [abstract]. In: Protecting the health of the Americas: moving from child to family immunization. Final report. XVII Meeting of the Technical Advisory Group on Vaccine-preventable Diseases, Guatemala City, Guatemala, 25-27 July 2006. Washington: PAHO; 2007. p. 69. [ Links ]
7. Pan American Health Organization, Health Analysis and Statistics Unit. Regional core health data initiative: technical health information system. Washington: PAHO; 2007. [ Links ]
8. Laska EM. The use of capture-recapture methods in public health. Bull World Health Organ. 2002;80(11):845. Epub 2002 Dec 3. [ Links ]
9. Dietz V, Venczel L, Izurieta H, Stroh G, Zell ER, Monterroso E, et al. Assessing and monitoring vaccination coverage levels: lessons from the Americas. Rev Panam Salud Publica. 2004;16(6):432-42. [ Links ]
10. Ronveaux O, Rickert D, Hadler S, Groom H, Lloyd J, Bchir A, et al. The immunization data quality audit: verifying the quality and consistency of immunization monitoring systems. Bull World Health Organ. 2005;83(7):503-10. [ Links ]
11. Dabbagh A, Eggers R, Cochi S, Dietz V, Stre-bel P, Cherian T. A new global framework for immunization monitoring and surveillance. Bull World Health Organ. 2007;85(12):904-5. [ Links ]
12. Papiana M, Rodewald L. For better immunisation coverage, measure coverage better. Lancet. 2006;367(9515):965-6. [ Links ]
13. de Oliveira LH, Danovaro-Holliday MC, Matus CR, Andrus JK. Rotavirus vaccine introduction in the Americas: progress and lessons learned. Expert Rev Vaccines. 2008; 7(3):345-53. [ Links ]
14. Dayan GH, Shaw KM, Baughman AL, Orellana LC, Forlenza R, Ellis A, et al. Assessment of delay in age-appropriate vaccination using survival analysis. Am J Epidemiol. 2006;163(6): 561-70. [ Links ]
15. SonLa Study Group. Using a fingerprint recognition system in a vaccine trial to avoid misclassification. Bull World Health Organ. 2007;85(1):64-7. [ Links ]
16. Gudnason T, Briem H. An interactive central database of vaccinations in Iceland. Euro Surveill [Internet]. 2008;13(2):[about 2 p.]. Available from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=8010. [ Links ]
Manuscript received on 26 August 2008.
Revised version accepted for publication on 14 February 2009.