On-line version ISSN 1680-5348
Print version ISSN 1020-4989
Rev Panam Salud Publica vol.16 n.6 Washington Dec. 2004
CURRENT TOPICS TEMAS DE ACTUALIDAD
Estimación y monitoreo de los niveles de cobertura de la vacunación: lecciones aprendidas en las Américas
Vance DietzI, *; Linda VenczelII; Héctor IzurietaIII; George StrohIV; Elizabeth R. ZellV; Edgar MonterrosoVI; Gina TambiniVII
IPan American Health Organization, Immunization Unit, Washington, D.C., United States of America (currently assigned to the Centers for Disease Control and Prevention, National Immunization Program, Global Immunization Division, Atlanta, Georgia, United States of America)
IIPan American Health Organization, Immunization Unit, La Paz, Bolivia (currently assigned to the Centers for Disease Control and Prevention, National Immunization Program, Global Immunization Division, Atlanta, Georgia, United States of America)
IIIPan American Health Organization, Immunization Unit, Washington, D.C., United States of America (currently with the United States Food and Drug Administration, Rockville, Maryland, United States of America)
IVPrivate consultant, Donnelly, Idaho, United States of America
VUnited States of America, Centers for Disease Control and Prevention, Atlanta, Georgia, United States of America
VIPan American Health Organization, Immunization Unit, Mexico City, Mexico (currently assigned to the Centers for Disease Control and Prevention, Global AIDS Program, Guatemala City, Guatemala)
VIIPan American Health Organization, Family and Community Health Area, Washington, D.C., United States of America
Key words: mass immunization, immunization programs, health care evaluation mechanisms, health surveys, population surveillance, Americas.
Según lo establecido por la Organización Panamericana de la Salud (OPS), conseguir una alta cobertura de vacunación es una meta esencial para la Región de las Américas. Es indispensable lograr niveles de cobertura de 95% o mayores para poder alcanzar los objetivos de la OPS de eliminar el sarampión y la rubéola, controlar las enfermedades prevenibles mediante la vacunación, y hacer perdurar la eliminación de la poliomielitis en territorio americano. Para poder alcanzar esos niveles, es imprescindible que las estadísticas de vacunación sean fiables y que las autoridades sanita rias midan y monitoreen los niveles de cobertura a lo largo del tiempo. Los métodos elegidos por los directores de los programas de vacunación para calcular la cobertura dependerán de la información que haga falta. En general, los directores del Programa Ampliado de Inmunización (PAI) necesitarán información acerca de la cobertura para poder: 1) determinar la verdadera cobertura en los niveles nacional y local, 2) determinar cuán adecuada es la cobertura en una zona determinada, 3) monitorear las tendencias a lo largo del tiempo, y 4) monitorear las actividades de vacunación mientras se están llevando a cabo. Para lograr lo primero determinar cuáles son los niveles verdaderos de cobertura, los administradores tienen dos opciones: a) valerse de los datos acerca de las dosis administradas (es decir, el número de dosis de la vacuna que se ha administrado, dividido por la población que debió recibir una dosis) o b) llevar a cabo una encuesta para determinar la cobertura. Para lograr lo segundo saber si la cobertura en una zona determinada es adecuada (por ej., mayor de 90%), se puede realizar un muestreo por lotes para garantizar la calidad (MLGC). El MLGC es una metodología de encuesta basada en el uso de muestras pequeñas que permite determinar si la cobertura en una zona determinada es adecuada o no, pero no sirve para estimar el nivel de cobertura. Para el tercer propósito monitorear las tendencias a lo largo del tiempo, se pueden usar los datos correspondientes al número de dosis administradas. Para lograr el cuarto propósito determinar si procede vacunar o llevar a cabo una campaña de vacunación u otra actividad afín, la "herramienta de monitoreo rápido" creada por la OPS es una magnífica solución. Cada uno de estos métodos posee ventajas y desventajas. Los datos sobre el número de dosis administradas, más la herramienta de monitoreo rápido, deben usarse para lograr las metas de vacunación en las Américas. Ambos métodos son los preferidos actualmente por la OPS para estimar la cobertura de vacunación.
The Pan American Health Organization (PAHO) has made the achievement of high vaccination coverage levels a key goal for the Region of the Americas. Achieving coverage levels of 95% or higher at the national and local levels is central to PAHO's objectives of eliminating measles and rubella, controlling vaccine-preventable diseases, and maintaining polio eradication in the Americas (13). Besides the national coverage levels, PAHO currently recommends that countries utilize two other measures or indicators of coverage for each vaccine: (1) the number and proportion of municipalities (i.e., districts) with a vaccine-specific coverage level of > 95%, and (2) the number and proportion of children in a given age group who live in a municipality with > 95% coverage.1 These three measures of coverage enable national programs to identify high-risk municipalities and to ensure equity in the provision of immunization services (4).
Implicit in adopting coverage goals is the ability to measure and monitor vaccination coverage levels. Countries in the Americans annually report coverage levels to PAHO that are based on the administrative method (doses administered), which is a method that PAHO recommends (5). The method involves dividing the number of doses of a particular vaccine that were given (the numerator) by the number of persons who should have received that vaccine (the denominator). Using population figures for the Western Hemisphere, PAHO then calculates Regional coverage levels for all the vaccines in its Expanded Program on Immunization (EPI).
Some countries employ coverage surveys utilizing various methodologies to estimate coverage. PAHO encourages the use of a "rapid house-to-house monitoring tool" to assess the quality of vaccination activities in a given area (6). In addition, serologic surveys can be conducted to measure the immunologic protection that a community has against vaccine-preventable diseases.
Each coverage assessment methodology has advantages and disadvantages. Therefore, national program managers must be aware of the strengths and limitations of each methodology as well as when a particular methodology is indicated for use. In this piece we will review currently available options for assessing coverage. We will also provide criteria to assist EPI managers in selecting the most appropriate method for assessing coverage, depending on their particular situation or need, and for directing vaccination activities.
For this article we conducted a review of the methods that countries in the Americas currently use to assess vaccination information and coverage levels. We also reviewed the methods that PAHO recommends. In addition, we performed a review of general methodological issues for sampling and of the advantages and disadvantages of the most common vaccination coverage survey methodologies.
BACKGROUND ON SAMPLING ISSUES
Prior to discussing how to assess coverage, a general background on basic sampling issues is warranted. One can estimate coverage either with a complete ascertainment (i.e., census) of the individuals in question or through a sample of the population. (The doses administered method is essentially a census (a complete ascertainment) of all children in a health center's catchment area.) In a complete ascertainment the vaccination status of all individuals in the population under investigation is reviewed and tabulated, thus providing a coverage level for that population. However, in most situations it is not feasible to interview or assess all children or all geographic areas. A sample of children is therefore taken, in a "sample survey." For the results of a sample survey to be representative of the population, all children in the population must have a known probability of being selected for the sample.
Several factors can affect the accuracy or precision of the result of a survey, including response rates and sampling and nonsampling error (7). Response rate refers to the proportion of children targeted for assessment who participated or responded during the survey and were actually included in the calculation of coverage. Generally, a response rate of at least 80% is considered acceptable.
Sampling error is the error in the coverage estimate obtained in a sample survey. The sampling error is dependent on the sample design and sample size, and it can be quite different for different survey methods. Sampling error results from the fact that not all children in the area are included in the survey, that is, that only a sample are included. Sampling error is measurable, and the amount of sampling error that is acceptable can be predetermined and incorporated in the sample size estimate. Statistical methods are available to estimate the sampling error, and the point estimate (e.g., coverage level) is often expressed with the corresponding 95% confidence interval.
In addition to the sampling error, nonsampling errors may reduce the accuracy of the vaccination information obtained either in a sample survey or in a complete ascertainment (79). Three important types of nonsampling error are discussed in the following paragraphs: selection error, nonresponse error, and measurement error.
Selection error occurs when not all the members of the target population are represented in the sample frame. For example, failing to include indigenous persons or rural residents in a sample for an area that has those populations would result in selection error.
Nonresponse error occurs when those children who are selected to be included in the survey are either not all located or have parents who deny them permission to participate. The result is that the children who are surveyed are not representative of the entire population.
The third type of nonsampling error is measurement error. It is associated with incorrect vaccination information and occurs when parents do not have a child's vaccination card, the parents provide incorrect information, or the interviewer misreads or misinterprets the information on the vaccination card. In general, the measurement error is similar for the various coverage survey methods.
In addition to the various types of error discussed above that can affect the accuracy of survey coverage results, another key issue in assessing coverage is the accuracy of a child's vaccination information, that is, whether the dates that a child received individual vaccines are correct. That accuracy is dependent on such factors as good record-keeping by health center officials, whether the information is obtained from parental recall or a written vaccination record for the child, and whether the health facility has accurate population data to calculate a coverage level.
THE INFORMATION REQUIRED AND THE APPROPRIATE METHODS TO ASSESS COVERAGE
To determine which method or methods should be used to assess coverage, one must identify the information required. In general, EPI managers will be confronted with one or more of four situations in which information on coverage is needed: (1) to determine an actual coverage level, (2) to determine if an area has adequate coverage, (3) to monitor trends over time, and (4) to monitor vaccination activities as they are being conducted. We will deal with each of these four situations in the following subsections.
Determining the actual coverage level
To determine the actual coverage level, one can either utilize data on the doses administered or conduct a coverage survey. Both of these approaches are described below.
Doses administered method for assessing coverage levels. The use of doses administered data is a type of complete ascertainment (census) of the population who use that health facility for receiving vaccines. The vaccination information for all children in the population in question, generally a health center catchment area, is recorded and monitored. In order to calculate a coverage estimate for that specific population, the number of children who received a specific vaccine dose is divided by the best population estimate for the number of children who should have received the vaccine. Also calculated are the number and proportion of municipalities with 95% coverage and the number and proportion of children living in municipalities with 95% coverage.
The doses administered method has both advantages and disadvantages for assessing coverage (Table 1). The method is relatively inexpensive and timely. The method also forces program managers to emphasize data management and the quality of both local immunization information and population estimates. The method can monitor trends over time and can determine coverage at all levels, including the national, state, and district/local.
The accuracy of the doses administered methodology is heavily dependent on good local record-keeping practices and on accurate population estimates. Problems can and do occur in obtaining accurate data for both the numerator and the denominator used in calculating coverage. For example, not recording doses given can result in underestimates of coverage. Children may be vaccinated during a campaign and the dose not recorded. Children vaccinated in the private sector may not be included in coverage estimates made by national EPI programs. Children outside the program's recommended age for vaccination can be vaccinated and then incorrectly included in the numerator. Children living in one municipality may be vaccinated in a neighboring municipality without the knowledge of the municipality of residence. This would result in an underestimate of coverage in the child's municipality of residence and a potential overestimate of coverage in the municipality where the child was vaccinated.
Population estimates (that is, the denominator) are often outdated or not realistic, as many countries do not regularly perform a national census. Migration from rural to urban areas results in an overestimate of population figures in rural areas, with an inflated denominator leading to an underestimate of true coverage in the rural areas. That migration would also produce an underestimated population figure in urban areas, with a denominator that is too low, resulting in an overestimation of coverage.
In addition to monitoring doses administered data for national, state, or local levels, PAHO's two other indicators of coverage (that is, the number and proportion of municipalities and of children with a given coverage level), provide valuable coverage information on the local level, but they also have potential biases. For example, municipalities with differing populations are given the same weight when one calculates the proportion of municipalities with 95% coverage.
Situations could arise in which a country could have many municipalities with small populations with low coverage and only a few municipalities with very large populations with very high coverage. If this were the case, the number and proportion of municipalities with high coverage would be low, while national coverage (an average of coverage of all municipalities together) could be high. This is often the case in countries with dispersed, hard-to-reach populations.
When one wants to calculate the number and proportion of children in the country who live in municipalities with high coverage, municipalities can be unaware that their children have been vaccinated in other municipalities or by the private sector. That is, there is very high coverage in the country (with the vast majority of the children having been vaccinated), but only a few municipalities report high coverage due to the fact that the municipality of residence is unaware that its children have been vaccinated in another municipality or by private sector health care providers.
Surveys commonly used in EPI programs to assess coverage. Various surveys are commonly used in EPI programs to assess vaccination coverage levels. These surveys differ from the doses administered method in that while the doses administered method is a type of ascertainment, surveys are not. Instead, surveys sample a portion of the population.
Below we present concise descriptions of the sample surveys most commonly used to determine immunization coverage levels. In addition, there are numerous published sources (10) and unpublished ones2,3 on the pros and cons of different survey methodologies, and they should be consulted for more detail.
The first type of surveys discussed are random sample surveys. These are classified as simple, stratified, and systematic sampling.
With a simple random sample (SRS), the children who are to be surveyed are randomly chosen from a list of all the children in that particular population. That is, each child has a known probability of being selected and evaluated. Simple random sampling may be used in a stratified sampling plan, in which the population under investigation is divided into different subgroups or strata and an SRS of children is selected from each stratum. Strata may be defined by any factor that may be associated with coverage, such as health facility catchment area, socioeconomic status, or rural versus urban. Stratification ensures that the sample selected is more representative of the population in terms of the variable stratified than if a SRS survey were done for the entire population. Stratified sampling allows for evaluation and comparison of coverage levels among the different strata. This is important when it is believed that coverage may differ by strata.
With systematic sampling, every nth child (e.g., every fourth or ninth one) is selected from a list. The first child is selected at random between the first child on the list and the nth child on the list. This selection process assumes that all children are listed and that the results approximate those of an SRS.
As mentioned earlier, each survey method has its advantages and disadvantages. SRS surveys are representative of the area surveyed but require a complete list of the population to be surveyed in order to select the sample. In addition, subgroups of persons with characteristics that may influence vaccination status, such as ethnic or transient populations, may be missed or underrepresented. However, the impact of this depends on the proportion of the total population that these subgroups represent. If the population to be sampled is widely dispersed geographically, the logistics of implementing a simple random sample survey are costly and can be time-consuming.
Systematic sampling assumes that complete lists of all children are available and that there is no periodicity (clustering of children) by important characteristics in the list of children that is used to select the sample of children. For example, if most unvaccinated children are grouped together on the list, they may or may not be chosen in the systematic sampling, depending on whether the grouping is located on the list before or after the nth interval, thus resulting in biased coverage. Even with this limitation, systematic sampling usually provides as representative a sample as simple random sampling. In stratified surveys the sample sizes needed in each stratum in order to obtain estimates with adequate precision may increase the overall sample size so much that the survey becomes prohibitively expensive.
Cluster surveys are another type of survey often used. In cluster surveys, specific areas or populations are grouped. These groupings, called clusters, could be schools, city blocks, neighborhoods, or villages. Within each of the clusters selected to be surveyed, a random sample of households is selected, and the vaccination status of children in these households is reviewed. These surveys are generally referred to as household cluster surveys. Many countries in the Americas have conducted large national surveys using household cluster methodologies, generally with the support of outside donors. Two of these household cluster surveys, the Multiple Indicator Cluster Surveys (MICS) and the Demographic and Health Surveys (DHS), are funded primarily through the United Nations Children's Fund (UNICEF) and the United States Agency for International Development (USAID), respectively (11, 12). These surveys have many similarities in the methodology and content of their questionnaires. Importantly, these national surveys are used to collect other health-related information.
A specific type of cluster survey used in EPI programs throughout the world to assess coverage is the 30-cluster EPI survey, often referred to as the EPI cluster survey or simply the EPI survey (13, 14). The method uses practical and simplified approaches to develop the sampling frame and to then select and survey clusters of children in the field.
The identification of clusters is the first stage of this two-stage design. An area is divided into clusters, and 30 clusters are selected, with a probability proportional to the size of their population. In each of the 30 clusters, an initial household is randomly selected in the field, and neighboring households are visited until seven children have been surveyed. The households visited to find the seven children constitute a cluster. The results of EPI cluster surveys approximate those of a two-stage cluster survey that assesses all children in each cluster. The EPI cluster survey is designed such that estimates have a precision of ± 10 percentage points with 95% confidence, that is, 95 of 100 surveys performed would include the true coverage within ± 10 percentage points.
Several variations of the EPI cluster survey have been proposed. The modified methodology of Kok (15) uses schools instead of geographic units. Other modifications include increasing the sample size by increasing the number of clusters or the number of children in each cluster (16).
The EPI cluster survey methodology has confidence intervals of ± 10 percentage points when coverage is 50%, but the confidence interval becomes narrower as coverage becomes greater or smaller than 50%. This increase in precision may not be sufficient to monitor program goals of 95% coverage. To detect a significant improvement from one survey to a subsequent survey, considerable increases in coverage relative to the previous rate are needed due to the confidence levels obtained with this method. For example, to detect a significant increase in coverage from a baseline of 85%, coverage would need to have increased to 94% (13). Therefore, EPI cluster surveys may not be useful to monitor change in areas with high coverage levels.
It is assumed that the process used in an EPI cluster sampling to select seven children approximates an SRS. However, if that assumption is not true, it would bias the estimate. This approximation is probably more accurate in areas with a homogeneous distribution of vaccinated persons and unvaccinated persons, such as areas with a very high level or a very low level of coverage. In settings with a grouping of undervaccinated individuals in the cluster, it may not approximate the results of an SRS. If undervaccinated children are grouped, they could be missed in the sample and not included in the survey. This would result in an overestimate of true coverage (10).
A fourth type of survey used to assess coverage is the retrospective school-enterer survey (RSES). Although not widely used in the Americas, RSESs sample children who are entering kindergarten or first grade ("school enterers") and retrospectively determine their vaccination status when they were 1 or 2 years of age (17). Therefore, the coverage information obtained is generally three to four years out of date. The survey has a two-stage cluster survey design, in which schools are randomly selected, with probability proportional to the estimated number of school enterers. At each school, children are randomly selected and their vaccination status reviewed. Although the RSES method is simple to implement, it assumes that the vast majority of children attend school and that schools have vaccination records for all the children.
Another type of survey uses the telephone to contact parents. In its National Immunization Survey (NIS), the National Immunization Program of the United States uses a random-digit-dialed telephone survey (that is, telephone numbers are randomly selected and dialed to locate eligible children) to measure vaccination coverage estimates for its 50 states and selected urban areas for children 1935 months of age (18). The NIS is conducted in conjunction with a provider record check study to improve the accuracy of the vaccination information. That is, the providers of surveyed children are contacted and the vaccination status of the child is verified. The data obtained are weighted to account for sampling design and to reduce biases from nonresponse (persons not contacted) and from not being able to contact parents of children who do not have a telephone. This survey allows the United States to have estimates for all of the states and for selected urban areas.
All the surveys commonly used in EPI programs to assess coverage that we described above provide actual coverage estimates (the percentage of children vaccinated), often with a great deal of certainty. Each method has its advantages and disadvantages. Some are specific to that method, while other advantages and disadvantages are common to all sample surveys. For example, when stratification is employed, estimates of coverage with adequate precision can be determined for each stratum, such as rural or urban. Sample surveys, that is, surveys that use a sample of a population to estimate coverage, are especially valuable when little or no administrative data or other information on coverage are available for an area, to set baseline levels, for large areas where it is difficult to document "pockets of need," and to determine the urgency of corrective activities. Sample surveys may be indicated in settings where an area has poor record-keeping practices and the information on doses administered coverage is unreliable or unavailable. Sample surveys can also be useful where many children are vaccinated in the private sector but not reported for inclusion in coverage calculations. In these ways, surveys can validate coverage calculated by the doses administered method. By expanding surveys to include questions for parents on such things as knowledge, attitudes, and practices as well as missed opportunities for vaccination, managers can obtain valuable information on other factors related to the vaccination process. In addition, governments often need coverage estimates for donor agencies. Sample surveys can also be useful to measure the potential for disease transmission in areas at high risk due to low coverage, the presence of groups in large periurban areas that are considered to be at risk for measles transmission, and other factors.
However, in general, surveys are time-consuming and costly and require technical expertise, and they may produce results too late for directing interventions to correct immediate problems. In addition, unless estimates are obtained for the local level or for strata, sample surveys may not provide sufficient detail to design interventions for subgroups in particular need.
An important issue in conducting a survey, regardless of its type, is the use of vaccination information obtained when parents do not have their children's vaccination records and so must try to recall that information from memory. Program managers must develop guidelines on how to interpret and use vaccination information obtained from recall. If surveys that are conducted sequentially, such as annually, apply different approaches or rules, the comparability of their results is then limited.
Determining whether an area has a minimum coverage level, by using lot quality assurance sampling
The second major purpose for which EPI managers will need information on coverage is for determining the adequacy of coverage in a given area. One methodology that is appropriate for that purpose is lot quality assurance sampling (LQAS). LQAS is a survey methodology that allows a decision to be made with a prespecified probability that a coverage level is above or below a preselected level. That is, the result is given as pass or fail, accept or reject, or good or bad (19, 20). With LQAS, the sample sizes tend to be small, so the results do not provide an adequately precise point estimate of coverage. LQAS is essentially a stratified random sample from a population divided into strata or "lots" (lots are generally geographic areas such as neighborhoods). A simple random sample is selected from each lot and is used to determine whether each lot passes or fails in terms of meeting the preselected level of coverage. LQAS was originally designed for use in industry to rapidly determine if a production lot should be rejected based on finding a predetermined number of defective products in the sample chosen. The actual proportion of "defectives" in each lot (for vaccination programs, the number of children who are not vaccinated) is usually not determined nor is it needed by local managers.
LQAS is useful since it requires smaller sample sizes, is often less expensive than other types of sample surveys in the same geographic area, and can be performed rapidly. The method is ideal for situations in which a program manager needs to determine if an area has or has not achieved a given coverage level. In addition, and as opposed to the EPI 30-cluster sample, the sampled lots (i.e., strata) can be combined. An overall coverage level can then be determined by using weighting procedures based on lot populations.
The main disadvantage with LQAS is that it does not provide a precise coverage estimate for individual lots. In addition, in rural areas it may be expensive and time-consuming to implement since the members of each lot are actually surveyed and the distances between them may be great (21).
Monitoring trends in vaccination coverage
The third situation where coverage information is often needed is in monitoring trends in vaccination coverage over time. This is critical in EPI programs, and it should be performed at all administrative levels (that is, national, state, local, etc.) and on a regular basis. The methods described above can be used to accomplish this goal. In general, the use of doses administered data is preferable. It is simpler and less expensive than surveys, and it should be incorporated as part of good record-keeping practices in EPI programs. Surveys provide the necessary information, but they are too costly and time-consuming to be used frequently in a routine manner. However, if national trends are monitored at infrequent intervals, such as every five years, then surveys could be considered. LQAS is suitable if the question poised is only whether coverage is being maintained at a predetermined level, e.g., > 95%.
Determining if vaccination activities have been successful or if further activities are immediately needed
The fourth situation where coverage information is often needed is to monitor vaccination activities as they are being conducted. The completeness of the vaccination activities at the local level or for a health center catchment area can be rapidly assessed during regular visits to the community using a method developed by PAHO that is known as rapid house-to-house monitoring (6). This monitoring approach involves visiting a limited number of homes with children aged 1 to 4 years and asking for evidence of vaccination. A total of four nonadjacent blocks in a clinic's catchment area are chosen based on having difficult access to a health facility, being a zone underserved by health services, or being areas with a high proportion of recent immigrants. Starting in the southeast corner of the first of the four blocks chosen, the team moves from one door to the nearest one until 5 households with children aged 1 to 4 years provide information about the children's vaccination status. If 5 eligible homes are not found in the block selected, the team moves to the adjacent block and continues until a total of 5 eligible homes are found. The same procedure should be followed for the remaining three blocks until 20 households with children aged 1 to 4 years have provided information on their children's vaccination status. That is, a total of 20 children will be located from 20 different households.
In properly vaccinated communities almost all children in any neighborhood will be up to date for vaccination, and the results of this rapid house-to-house monitoring are very likely to show that. Although the monitoring ends when a total of 20 homes with eligible children have been visited, the finding of a total of 3 households with at least one unimmunized child each (even before 20 homes are visited) is reason enough to stop monitoring and to recommend revaccination of that neighborhood, since it would be very unlikely to find such an scenario in a well-vaccinated neighborhood (Figure 1).
Rapid house-to-house monitoring assesses the completeness of vaccination in a nonprobabilistic manner. Therefore, some managers may want to utilize a more systematic approach. A model, referred to as systematic rapid house-to-house monitoring, has been developed based on LQAS. This method uses a two-stage approach (Figure 2), in which the second stage is conducted similarly to the routine rapid monitoring methodology. It is based on the assumption that there is a 90% probability that a community with a 95% coverage level will be correctly identified as such. In addition, there is a 90% probability that a community with a coverage level of 85% or less will be correctly identified as not having acceptable coverage.
Both types of rapid house-to-house monitoring are simple, are completed quickly and inexpensively, and can provide a quick impression of the completeness of vaccination. It is ideal for use during routine supervision or to assess how well vaccination efforts are implemented during a vaccination campaign. Importantly, it forces clinic staff to visit their communities and see firsthand the impact of their services.
Rapid house-to-house monitoring (i.e., nonprobabilistic monitoring) is a supervisory tool and is not statistically valid, nor can the results be generalized to the community. The results cannot be interpreted as a coverage level. In addition, if the local managers choose the areas to be visited, they will likely choose only areas with high vaccination coverage levels, thus providing a false sense of security. To avoid this problem, it is preferable that a supervisor or team leader unrelated to the management of the local vaccination program choose the blocks to be visited. For areas with acknowledged low coverage, rapid house-to-house monitoring is less useful since one is already aware that coverage is low. Disadvantages for the systematic rapid house-to-house monitoring are similar to those discussed for LQAS.
Although the experiences and issues presented in this piece relate to managing vaccination information and assessing coverage in the Region of the Americas, they are applicable to other areas of the world as well. Problems in calculating coverage are not limited to the Americas. An assessment of the accuracy of coverage estimates based on administrative data in 45 countries showed the problem to be widespread and not isolated or in only one part of the world (22). A recent study demonstrated that coverage estimates can also be biased and may not represent true population immunity due to vaccine doses being inappropriately administered. For example, if the time interval between the first and second dose is too short, the second dose may not be protective (23).
Prior to embarking on activities to assess coverage levels, managers should be clear about what information is needed (Figure 3) as well as the advantages and limitations of each method (Table 1). First, managers must decide what information is needed. Thus, they must decide if (1) an actual coverage level is needed, (2) it is acceptable to know only whether an area is performing well in rais- ing coverage, (3) trends need to be monitored, or (4) an area's ongoing vaccination activities have been acceptable. Monitoring accurate doses administered data can be used in all situations. However, and as is discussed above, situations may arise when a survey is needed to determine coverage, for example, to validate doses administered information, to assess the impact of an intervention, or for donor needs. Once the decision has been made to conduct a coverage survey, managers must then choose the survey methodology to be used. The choice should take into account the logistics and resources needed to apply a specific methodology.
If managers can accept knowing only whether an area's coverage is acceptable or not acceptable and do not need an actual coverage level estimate, then LQAS is an appropriate methodology. If managers wish to monitor trends, then use of doses administered data is sufficient. If managers need to know if vaccination activities are adequate, such as after a campaign, the rapid house-to-house monitoring tool is acceptable. For achieving EPI coverage goals and to monitor vaccination activities, the use of accurate doses administered data and population estimates with PAHO's rapid house-to-house monitoring tool will provide the needed information.
Regardless of the methodology utilized in assessing coverage, the data for the calculations must be accurate. Ensuring the completeness of reporting and ensuring that data (including population estimates) are of good quality are fundamental steps. To make certain that there are good record-keeping practices in all health centers, managers must frequently evaluate and improve the accuracy of both the numerator (the doses given) and the denominator (the population). Population denominators can be verified with figures extrapolated from other information such as the number of births during the year, local population survey counts, and the number of doses of BCG vaccine and the first doses of diphtheria-tetanus-pertussis vaccine (DTP1) or polio vaccine (polio-1).
Several countries have begun efforts to address the quality of vaccination information and the accuracy of both the numerator and denominator used to calculate coverage estimates. For example, current supervisory efforts in Bolivia include comparing the numbers of doses recorded on vaccination cards to the number of vaccine doses that clinic reports indicate have been given. The number of doses recorded in the clinic reports is compared to the number recorded in reports from the clinic to larger administrative areas such as the district level. Then the number reported by the district level is compared to the number that the national level reports.4 If the error rate, that is, the discrepancy among reports of vaccines having been given at the clinic, district, and national level, is 10% or less, the accuracy of the data is considered acceptable for programmatic needs. Similarly, in southern Mexico, local managers assess the accuracy of population data by comparing the number of doses of DPT1 and BCG with different population denominators, including local population estimates obtained from census data, national malaria programs, and local prenatal programs operated by the Ministry of Health. The data are accepted as reliable if there is a concordance of at least 90% between population figures from different sources or between levels of coverage calculated with denominators from all sources.5 Both PAHO and WHO have developed paper instruments to use in the field during supervisory visits to assess the quality of vaccination data.
There are other examples of country-based efforts to validate the accuracy of vaccination information. For instance, in Brazil, dropout levels between the number of first doses of DTP and the number of measles vaccines given are compared with coverage levels. This provides a valuable "reality check" on the potential accuracy of local coverage data since high dropout rates (e.g., > 5%) cannot be associated with high coverage (5). In addition, reports of municipalities with coverage of more than 100%, which is impossible, should be viewed skeptically and evaluated critically. Such reports represent errors with either the denominator (e.g., the population is underestimated) or the numerator (e.g., including the vaccination of nonresidents).
Lastly, an additional important point to consider when calculating coverage figures is how to calculate coverage when vaccination campaigns are conducted and whether doses administered in campaigns should be included in calculations for national (or local) coverage levels. Calculating coverage for routine vaccination services provides information on EPI program performance and is crucial for program management. However, incorporation of campaign doses into coverage calculations provides information on protection against disease in the community, at least for that year. Thus, coverage should be calculated and monitored both with and without doses administered during a campaign.
Achieving high coverage levels is a key objective of national EPI programs. Methods are needed to monitor the progress towards achieving high coverage as defined by the three indicators or measures of vaccination coverage currently used in the Americas (that is, coverage levels, the number and proportion of municipalities with high coverage, and the number and proportion of children living in municipalities with high coverage). These three measures should be analyzed together, rather than being interpreted in isolation. Managers must understand that national data represent an average and can hide wide variations at the local level. Therefore, managers must place priority on two key tasks: ensuring data of high quality and monitoring local coverage levels in order to maintain or improve vaccination services.
1. Pan American Health Organization. XXIV Pan American Sanitary Conference: document CSP24/14: Expanded Program on Immunization. Washington, D.C.: PAHO; 1994. [ Links ]
2. Pan American Health Organization. XV Meeting of the Technical Advisory Group (TAG) on Vaccine-preventable Diseases: final report, conclusions and recommendations. Washington, D.C.: PAHO; 2002. [ Links ]
3. de Quadros CA, Andrus JA, Olive JM, Macedo CG de, Henderson DA. Polio eradication from the Western Hemisphere. Annual Rev Public Health. 1992;12:23952. [ Links ]
4. Pan American Health Organization. 54th Session of the Regional Committee: document CSP26/8: report of the 26th Pan American Sanitary Conference. Washington, D.C.: PAHO; 2002. [ Links ]
5. Pan American Health Organization. The importance of ensuring high quality and accurate data in EPI programs. EPI Newsl. 2003;25(1):24. [ Links ]
6. Izurieta H, Venczel L, Dietz V, Tambini G, Barrezueta O, Carrasco P, et al. Monitoring measles eradication in the Region of the Americas: critical activities and tools. J Infect Dis. 2003;187 Suppl 1:S1339. [ Links ]
7. Lohr SL. Sampling: design and analysis. Pacific Grove: Duxbury Press; 1999. [ Links ]
8. Willis G, Brittingham A, Lee L, Tourangeau R, Ching P. Response errors in surveys of children's immunizations. Washington, D.C.: Superintendent of Documents, U.S. Government Printing Office; 1999. (Vital and Health Statistics, Series 6, No. 8). [ Links ]
9. Zell E, Ezzati-Rice T, Massey J, Brick JM. Response errors associated with household reports of immunizations: analysis of subgroup differences. In: American Statistical Association. Proceedings of the Section on Survey Research Methods. Chicago: ASA; 1996. Pp. 3416. [ Links ]
10. Lemeshow S, Robinson D. Surveys to measure programme coverage and impact: a review of the methodology used by the Expanded Programme on Immunization. World Health Stat Q. 1985;38:6575. [ Links ]
11. Le TN, Verma VK. An analysis of sample designs and sampling errors of the Demographic and Health Surveys. Calverton: Macro International Inc.; 1997. (DHS Analytical Reports No. 3). [ Links ]
13. Henderson RH, Sundaresan T. Cluster sampling to assess immunization coverage: a review of experience with a simplified sampling method. Bull World Health Organ. 1982;60(2): 25360. [ Links ]
14. World Health Organization. The EPI coverage survey. Geneva: WHO; 1991. (WHO/EPI/MLM/91.10). [ Links ]
15. Kok PW. Cluster sampling for immunization coverage. Soc Sci Med. 1986;22(7):7813. [ Links ]
16. Serfling RE, Sherman IL Attribute sampling methods for local health departments with special reference to immunization surveys. First edition. Washington, D.C.: U.S. Public Health Service; 1965. [ Links ]
17. Zell E, Dietz V, Stevenson J, Cochi S, Bruce R. Low vaccination levels of US preschool and school-aged children: retrospective assessments of vaccination coverage, 19911992. JAMA. 1994;271:8404. [ Links ]
18. Zell E, Ezzati-Rice T, Battaglia M, Wright A. National Immunization Survey: the methodology of a vaccination surveillance system. Public Health Rep. 2000;115:6577. [ Links ]
19. Lanata CF, Stroh G, Black RE, Gonzales H. An evaluation of lot quality assurance sampling to monitor and improve immunization coverage. Int J Epidemiol. 1990;19(4):108690. [ Links ]
20. World Health Organization. Monitoring immunization services using the lot quality technique. Geneva: WHO; 1996. (WHO/VRD/TRAM/ 96.01). [ Links ]
21. Robertson SE, Anker M, Roisin A, Macklai N, Engstrom K, LaForce M. The lot quality technique: a global review of applications in the assessment of health services and disease surveillance. World Health Stat Q. 1997;50:199209. [ Links ]
22. Murray CJ, Shengelia B, Gupta N, Moussavi S, Tandon A, Thieren M. Validity of reported vaccination coverage in 45 countries. Lancet. 2003; 362(9389):10227. [ Links ]
23. Stokley S, Maurice E, Smith P, Klevens R. Evaluation of invalid vaccine doses. Am J Prev Med. 2004;26 (1):3440. [ Links ]
* Send correspondence to: Vance Dietz, Global Immunization Division, National Immunization Program, Centers for Disease Control and Prevention, 1600 Clifton Rd., MS E05, Atlanta, Georgia 30333, United States of America; telephone: 404-639-6225; fax: 404-639-8572; e-mail: email@example.com
1 Dietz V, Venczel L, Tambini G, Monterroso E. Assessment of the quality of vaccination information in the Americas. Presented at the XIV meeting of the PAHO Technical Advisory Group on Vaccine Preventable Diseases, Washington, D.C., 22 November 2002.
2 Lo EKC. Sampling methods and sample size. WHO/IMR Regional Centre In-Service Course on Research Designs and Methodologies, Kuala Lumpur, April 1983.
3 Lemeshow S, Stroh G. Sampling techniques for evaluating health parameters in developing countries. Paper prepared for the Board on Science and Technology for International Development, National Research Council, National Academy Press, Washington, D.C., 1988.
4 Quiroga R. Activities for the control of the quality of vaccination information in Bolivia. Presented at the Pan American Health Organization's Annual Sub-Regional EPI Manager's Meeting for the Countries of the Southern Cone, Buenos Aires, Argentina, 29 August 2002.
5 Monterroso E. Assessment of the quality of vaccination information, Mexico. Presented at the Pan American Health Organization's Annual Sub-Regional EPI Manager's Meeting for the Countries of Central America, Mexico, and the Latin Caribbean, San Jose, Costa Rica, 89 July 2002.