Validity and reproducibility of a food frequency questionnaire to assess dietary intake of women living in Mexico City*
Objective. To assess the reproducibility and validity of a 116 item semi-quantitative food frequency questionnaire (FFQ), designed to assess the relation between dietary intake and chronic diseases. Material and methods. To test the reproducibility of the FFQ questionnaire, the FFQ was administered twice to 134 women residing in Mexico City at an interval of approximately one year; to assess the validity we compared results obtained by the FFQs with those obtained by four 4-day 24-hour recalls at three month intervals. Validity and reproducibility were evaluated using regression analysis and Pearson and intraclass correlation coefficients of log-e and calorie-adjusted nutrient scores. Results. Mean values for intake of most nutrients assessed by the two food frequency questionnaires were similar. However, means for the 24-hr recall were significantly lower. Intraclass correlation coefficients for nutrient intakes, assessed by questionnaires, administered one year apart, ranged from 0.38 for cholesterol to 0.54 for crude fiber. Correlation coefficients between energy-adjusted nutrient intakes, measured by diet recalls, and the first FFQ ranged from 0.12 for polyunsaturated fatty acids to 0.67 for saturated fatty acids. Regression coefficients between 24-hr recall and FFQ,s were all significant were significant for all nutrients, except for polyunsaturated fat, folic acid, vitamin E and Zinc. Conclusions. These data indicate that this semi-quantitative FFQ is reproducible and provides a useful estimate by which to categorize individuals by level of past nutrient intake. However, its application outside Mexico City or in different age and gender populations will require additional modifications and validation efforts.
KEY WORDS: questionnaire/validity; diet; women; epidemiology; Mexico
Objetivo. Evaluar la reproducibilidad y validez de un cuestionario semicuantitativo de frecuencia de consumo de alimentos, para ser utilizado en estudios epidemiológicos sobre enfermedades crónicas. Material y métodos. Para evaluar la reproducibilidad del cuestionario de frecuencia de consumo se comparó el resultado obtenido al aplicar la encuesta a 134 mujeres en dos tiempos separados por 12 meses. Para evaluar la validez del cuestionario, comparamos los resultados obtenidos a partir de su aplicación, con el promedio de 16 recordatorios de 24-horas, los cuales se obtuvieron en el transcurso de 12 meses. Para estimar la validez utilizamos modelos de regresión y correlación de Pearson. Para evaluar la reproducibilidad estimamos la correlacion intraclase entre los cuestionarios de frecuencia de consumo. Resultados. Las ingestas promedio diarias estimadas por los cuestionarios de frecuencia de consumo fueron similares. Sin embargo, las medias estimadas mediante los recordatorios de 24 horas fueron significativamente menores. Las correlaciones intraclase entre los cuestionarios de frecuencia de consumo aplicados 12 meses aparte, variaron entre 0.38 para colesterol y 0.54 para fibra cruda. Las correlaciones ajustadas por calorías entre el primer cuestionario y la media de los recordatorios variaron entre 0.12 para ácidos grasos polinsaturados y 0.67 para ácidos grasos saturados. En el análisis de regresión observamos asociaciones estadísticamente significativas para casi todos los nutrientes, excepto para grasas polinsaturadas, ácido folico, vitamina E y zinc. Conclusiones. Los resultados indican que el cuestionario es útil para evaluar la ingesta dietética. Sin embargo, su utilización en poblaciones con diferente estructura de edad y género o ubicación geográfica requerirá de acciones adicionales para garantizar su validez y reproducibilidad.
PALABRAS CLAVE: cuestionario/validez; dieta; mujeres; epidemiología; México
Exposure assessment is a critical issue in epidemiological studies. The value of any research finding depends largely on the validity of the information used to measure exposure. For studies relating the effect of food consumption and chronic disorders, food frequency questionnaires (FFQs) have been widely used because they allow the evaluation of individual dietary intake for large populations at low cost. FFQs are designed to measure dietary intake over an extended period of time, such as months or years. The underlying assumption of the FFQ approach is that the measurement of long term dietary exposures is more relevant to chronic diseases. Therefore, in this context it is better to rely on less precise estimates of usual dietary intake than on rather precise estimates of short-term intake, which may not adequately reflect exposure.1 Consequently, it is expected that the information derived from FFQs will not reflect current dietary intake as accurately, but instead will be more valid to rank individuals by levels of their past nutrient intake.2 FFQs are based on a list of foods and questions regarding how often these foods are eaten. Therefore, in order to provide a reasonable estimation of intake, the food items included in the questionnaire need to be selected carefully, using methods that would yield a list of foods that will reflect the food consumption patterns and choices of the population under study.
We therefore conducted a study to assess the validity and reproducibility of an FFQ to be used in future epidemiological studies, to assess nutritional risk factors of chronic diseases among women who live in Mexico City.
Material and methods
Development of the food frequency questionnaire
The FFQ was developed using the methodology proposed by Willett et al.2 We identified food items by tabulating the results of a dietary survey carried out in 1983 by the National Institute of Nutrition on a random sample of 240 low to medium income families in Mexico City.3 Data collected in this survey included both 24-hour dietary recalls and home visits to weigh and measure the food items actually consumed. Using, stepwise linear regression, the foods which were the best predictors of each nutrient of interest were identified. We then compiled all the foods identified using this methodology. In a second step we identified important foods that may have not been reported by the population who participated in the survey carried out by the National Institute of Nutrition. For this purpose, a group of Mexican dietitians and nutritionists were invited to participate and were asked to identify relevant food items that may have not been identified with the stepwise linear regression approach. A final format that included 85 food items was then pilot-tested using a convenient sample of women of medium to low socioeconomic status, living in Mexico City. The final instrument was composed of a matrix listing 116 food items and 10 frequencies of consumption. The consumption frequencies were: 6 or more per day, 4-5 per day, 2-3 per day, 1 per day, 5-6 per week, 2-4 per week, 1 per week, 2-3 per month, 1 per month or less and never. (Copy of the questionnaire is abailable from author M. Hernandez-Avila).
Estimation of nutrient scores
For each food item, the nutrient content per average unit (specified serving size: slice, glass, or natural unit) was compiled. Nutritional composition of each food included in the questionnaire was derived from the US Department of Agriculture (USDA) food composition tables4 and, when necessary, complemented by the nutrient data base developed by the National Institute of Nutrition.5 Daily dietary nutrient intakes for study participants were estimated as follows: for each food a nutrient score was calculated using the food nutrient content taken from the food table, adjusted to the specified portion size of the questionnaire. We then multiplied this score by the weight corresponding to the frequency of use. The weights used were: 6 for reported frequencies of 6 or more per day; 4.5 for 4-5 per day; 2.5 for 2-3 per day; 1 for 1 per day; 0.8 for 5-5 per week; 0.43 for 2-4 per week, 0.08 for 2-3 per month and 0.016 for 1 per month or less. Then the product of frequencies times the nutrient scores were summed, producing scores for overall food items, to obtain a total nutrient score.
Portion sizes were estimated by "natural" units or standard quantities where possible. To account for the seasonal availability of some foods, the questionnaire had a section on the frequency of consumption of these foods during the season (e. g. mangos, mammey).
The data from the 16 24-hour dietary recalls were coded in a specifically-made data entry program. We calculated the specific nutrient content of the reported portion size and estimated a daily nutrient intake for each day.
We randomly selected an age-stratified sample of 527 women residing in Tlalpan, the southern district of Mexico City. Among them, 211 agreed to participate. A final sample of 134 women completed the study. The main reason for non-participation was the inability to comply with the complete study protocol. All study participants signed an informed consent and completed the study. Over a one-year period, we obtained four 4-day 24-hour recalls, corresponding to 16 days of diet recall in batches of 4 days, representing all seasons in Mexico. Interviewers visited each participating women at her home and recorded the type and quantity of food (using local cooking utensils) they had consumed during the previous 24-h. We arranged visitation of each subject on different days of the week (including Saturdays and Sundays) to account for differences in daily dietary habits. At baseline, women completed an FFQ applied by a trained dietitian and provided blood samples three and nine months later. At the end of the study, the women completed a second FFQ. Results from the blood sample analyses are described else-where.**
All items consumed, by frequency and quantity, in either the FFQ or the 24-hour recalls were coded and transformed to daily nutrients with the use of a food- composition database, developed to accommodate the characteristics of the Mexican diet. We used food composition tables provided by the National Institute of Nutrition5 and completed these tables with nutrition composition tables from the USDA4 and other published sources and personal communications from laboratories and manufacturers.
For the calculation of correlations, we used logarithmic transformation for all nutrient values and blood levels to reduce skewedness. We calculated energy-adjusted nutrients as the residual of each nutrient regressed on caloric intake.6 We calculated intraclass correlation coefficients to assess the reproducibility of the FFQ.7 To assess the validity of nutrient values, derived from the first and second FFQ, we used regression analysis and Pearson correlation coefficients and compared them with the mean of the 16 24-hour recalls. To take into account within-person variation, we used within- and between-person components of variation for the 24-hour recalls to "deattenuate" the Pearson correlation coefficients.8 All analyses were conducted using STATA statistical software.9
Mean daily intakes of nutrients estimated by the FFQ and the average of 16 24-hour recalls for the 134 women included in the validation analysis, are presented in Table I. The questionnaire measurements of total energy intake were higher than the intake assessed by diet recalls. Intake of most nutrients was slightly higher when measured by the FFQs compared with diet recalls, particularly for retinol. However the absolute values of nutrient intake, estimated by the second FFQ, were quite comparable to the estimate using diet recalls, with the questionnaire estimates showing a higher standard deviation. Eighty one percent of the mean values for macro and micronutrients, apart from vitamins, measured by questionnaire were within 15 percent of the 24-hr recall values. For vitamins, 82% of mean intake values measured by questionnaire were within 25 percent of the diet record values.
To obtain a measure of the existing within-subject variability, we computed intraclass correlations for unadjusted and energy-adjusted daily nutrients using the average intake, measured by each of the 4-day diet recalls. Alpha-carotene was the nutrient with the lowest correlation coefficient (intraclass r= 0.09) followed by beta-carotene (intraclass r= 0.12), whereas the highest intraclass correlation was observed for carbohydrate (r= 0.34) and calcium (r= 0.31). Energy-adjusted coefficients were similar. The reproducibility of the questionnaire was assessed by means of estimating correlation between nutrient scores, measured with the same instrument twice (Table II). Pearson correlation coefficients between unadjusted nutrient intake from the FFQ, spaced one-year apart, ranged from 0.43 for Alpha-carotene to 0.60 for monounsaturated fatty acids. Adjustment of the nutrient intake for total energy intake tended to decrease these correlations slightly, especially for total proteins, cholesterol, Vitamin B1 and polyunsaturated fatty acids (Table II).
To assess the validity of the questionnaire, we calculated the unadjusted and energy-adjusted Pearson correlation coefficients between the average of the 16 24-hour recalls and the first and second administration of the FFQ (Table II). Crude unadjusted correlation between the first questionnaire and diet recalls ranged from 0.13 for Folic acid to 0.52 for magnesium. Average correlation coefficients of 0.40 or higher were noted for total caloric intake, carbohydrate, protein, total fat, animal fat, saturated, monosaturated, and polyunsaturated fatty acids, fiber, thiamin, riboflavin, vitamin B6, vitamin C, vitamin D, calcium, magnesium, and potassium. Adjustment for total energy intake modified these estimates only slightly, except for polyunsaturated fatty acids, for which a substantial decrease was observed. There was no clear difference between the correlations of the 24-hour recalls with the first and second FFQ. However correlations between the second FFQ and the mean of the 24-hr recalls was higher.
The "deattenuated" coefficients represent the correlation between nutrient intake, estimated from the FFQ and the average intake of the four 4-day diet recalls after taking into account the effect of within-person variation in the diet record data.8 These coefficients ranged from 0.65 for saturated fatty acids to 0.12 for polyunsaturated fatty acids for the first FFQ and from 0.63 for total fat to 0.21 for polyunsaturated fatty acids for the second FFQ.
Calorie adjusted regression coefficients between the FFQís and the mean of the 24-hr recalls are presented in table III. Most nutrients showed a significant association. The coefficients between the first FFQ and the 24-hr recalls, ranged from 0.147 to 0.55 for Vitamin B6 and Carotenois, respectively. For the second FFQ, the regression coeffcientes ranged between 0.056 for Vitamin E and 0.581 for retinol.
We developed and evaluated the performance of a 116-item FFQ by comparing daily nutrient intake, obtained from this instrument, with those derived from four 4-day 24-hour recalls. Correlations assessed by dietary records, after correcting for within-person variation, averaged 0.45 for the macronutrients and 0.41 for the micronutrients. Energy-adjusted intraclass correlations were not systematically higher than unadjusted values, as observed in studies conducted in Spain10 and Greece.11 This could be due to the fact that although energy adjustment removes nutrient variation due to variation in energy intake, it also reduces the range of between-person variance, which may decrease the correlation.2 Reported correlations between the nutrient intakes estimated by FFQ and the "gold standard" estimates (obtained by dietary records or repeated 24-hour recalls) vary between studies.1-2,10-17 Although differences in study populations and in the magnitude of the within and between-person variation in diets among these populations preclude a valid comparison of results across studies,1 our estimates are, in general, similar to those reported in other studies among women.2,10,16 The study that reported higher correlation coefficients10 was conducted among knowledgeable professionals such as nurses or motivated volunteers.11,17 Our validation study included women from the general population, of low to medium income with an average of 7 years of formal education. Although trained interviewers applied the FFQ and it is a simple form to complete, this is not the case for the 24-hour recall methodology. To obtain complete information regarding food consumption during the day prior to the interview, we had to rely on the diligence of the participants to accurately recall foods and portion sizes. It is likely that errors in this procedure may have attenuated the observed correlations in our study.
Correction of energy-adjusted correlations improved the estimates because of the high within-person variation in the nutrient intakes in our study population, related in part to the change in the availability of food across the seasons (in particular for fruits and vegetables). This is corroborated by the low intraclass correlation between nutrient intake scores measured by the four 4-day diet recalls, compared to that reported in other studies.2,10,13,16 For micronutrients the intraclass correlation was on average 0.19, and for macronutrients 0.27. Correlation coefficients between diet recalls and FFQs were only slightly higher for the first FFQ and there is no clear explanation for this observation. If we consider that a correlation of 0.40 between the FFQ and the dietary recalls is indicative of good agreement, our questionnaire provides reasonable estimates of intake for 52% of the nutrients evaluated, including carbohydrate, animal protein, animal fat, saturated and monounsaturated fat, fiber, carotenoids, vitamin C, calcium, iron, magnesium, potassium, and phosphorous. Nutrients like polyunsaturated fat, cholesterol, vitamin E and folic acid were not well estimated by the FFQ, although the reproducibility data for these nutrients was not bad. We can not exclude errors due to incomplete nutrient databases or large within person variation, especially for cholesterol and vitamin E. However, the current format of the questionnaire is not recommended for studies interested in evaluating this nutrients.
Instead of asking participants about the portion size consumed for each food, we chose a pre-determined portion size for each food item in the questionnaire. For example the questionnaire asks how often a glass of milk is consumed. We chose this approach because most of the variation in food intake is determined by the frequency of consumption rather than by the portion size.18-20 Several investigators have reported that questions regarding portion sizes add little to the information collected. Other investigators have reported that the use of food models did not significantly improve the correlations between FFQs and dietary records.19 However, there are no data published for urban population in Mexico City regarding portion size, therefore errors in estimation of the portion size may be an additional source of random error in our study.
A source of misclassification that may explain some of the low correlations found, for example for cholesterol, may be that the Mexican diet includes a variety of composite dishes and the FFQ provided only a few options of such dishes. This could have also affected the correlation observed between the FFQ and the diet recalls. However, given that we asked for most individual foods, it is likely that these dishes were reported by participants and therefore were considered in the FFQ. Finally, one of the major limitations we encountered in this study was the lack of adequate nutrient composition in databases for Mexican foods. As previously mentioned, we used food composition tables provided by the National Institute of Nutrition5 and completed these tables with nutrition composition tables from the US.4 However for some tropical fruits and vegetables, we were not able to obtain information for all nutrients. For example pumpkin flower (flor de calabaza) was frequently consumed, and we were unable to obtain its food composition. This error would be similar for both methods and would generate a subestimation of the real intakes, and would result in a subestimation of the concordance between methods.
We estimate that this study provides the first steps for the development of a FFQ questionnaire that could be used in different studies to provide comparable data. A true validation procedure of a dietary assessment method is essentially impossible, as there is no absolute gold standard for measuring dietary intake, which limits the ability of our results to be generalized. To obtain a complete list of foods, we tabulated a previous survey, obtained qualitative information from experts in the field, and pilot tested the questionnaire. We believe that the food list of which the questionnaire is composed provides an adequate representation of the foods regularly consumed in Mexico City. Our results may also be influenced by the incompleteness of food composition tables. To help rectify this problem, we used an international source and complemented the information, when needed, with published food composition tables, specific for Mexican foods. Our questionnaire was designed to assess long-term exposure to different nutrients in order to study their potential role as risk factors for chronic diseases. Our results suggest that our FFQ questionnaire work relatively well, for women of medium to low socioeconomic status, living in Mexico City. The questionnaire validated in this study provides a cost-effective method to obtain information regarding dietary intake. However, its use in other geographical areas or populations with a different gender or age structure will require additional development and validation efforts.
1. Willett W. Nutritional epidemiology. New York: Oxford University Press, 1990.
2. Willett W, Sampson L, Stampfer MJ, Rosner B, Bain C, Witschi J, Hennekens CH, Speizer FE. Reproducibility and validity of a semiquanntitative food frequency questionnaire. Am J Epidemiol 1985; 122:51-65.
3. Hernandez M, Aguirre J, Serrano L. Alimentacion de obreros y sus familias. Mexico: División de Nutrición de Comunidad, Instituto Nacional de la Nutricion Salvador Zubirán. Publicacion L-61, 1983.
4. US Department of Agriculture. Composition of Foods raw, processed, and prepared. Agricultural Hand Book No. 8. Washington D.C.: U.S. Government Printing Offices, 1963-1997
5. Chávez M, Hernández M, Roldán JA, ed. Valor nutritivo de los alimentos de mayor consumo en México. Mexico: Comision Nacional de Alimentacion y Instituto Nacional de la Nutrition, 1992.
6. Willett W, Stampfer MJ. Total energy intake: implications for epidemiologic analyses. Am J Epidemiol 1986; 124:17-22.
7. Zar JH. Biostatistical Analysis. Englewood Cliffs, New Jersey: Prentice Hall, 1996.
8. Rosner B, Willett WC. Interval estimates for correlation coefficients corrected for within-person variation: implications for study design and hypothesis testing. Am J Epidemiol 1988; 127:377-386.
9. StataCorp 1995. Stata Software: Release 4.0 College Station TX: Stata Corporation.
10. Martín-Moreno JM, Boyle P, Gorgojo L, Maisonneuve P, Fernandez-Rodriguez JC, Salnini S, et al. W. Development and validation of a food frequency questionnaire in Spain. Int J Epidemiol 1993; 22:512-519.
11.Gnardellis C, Trichopoulou A, Katsouyanni K, Polychronopoulos E, Rimm EB, Trichopoulos D. Reproducibility and valididty of an extensive semiquantitative food frequency questionnaire among Greek school teachers. Epidemiology 1995; 6:74-77.
12. Lee-Han H, McGuire V, Boyd NF. A review of the methods used by studies of dietary measurements. J Clinical Epidemiol 1989; 42:269-279.
13. Rimm EB, Giovanucci EL, Stampfer MJ, Colditz GA, Litin LB, Willett WC. Reproducibility and validity of an expanded self-administered semiquantitative food frequency questionnaire among male health professionals. Am J Epidemiol 1992; 135:1114-11126.
14. Tjonneland A, Overvad K, Haraldsdottir J, Bang S, Ewertz M, Jensen OM. Validation of semiquantitative food frequency questionnaire developed in Denmark. Int J Epidemiol 1991; 20:906-912.
15. Pietinen P, Hartman AM, Haapa E, Rasanen L, Haapakoski J, Palmgren J et al. Reproducibility and validity of dietary assessment instruments. I. A self-administerd food use questionniare with portion size booklet. Am J Epidemiol 1988; 128:655-66.
16. Jain M, Howe GR, Rohan T. Dietary assessment in epidemiology: comparison of a food frequency and a diet history questionniare with a 7-day food record. Am J Epidemiol 1996; 143:953-60.
17. Hunter D, Sampson L, Stampfer MJ, Coldiztz GA, Rosner B, Willett WC. Variability in portion sizes of commonly consumed foods among a population of US women. Am J epidemiol 1988; 127:906-12.
18. Samet JM, Humble CG, Skiper BE. Alternatives in the collection and analysis of food frequnecy interview data. Am J Epidemiol 1984; 120: 572-581.
19. Hernández-Avila M, Martin C, Hunter D. Influence of additional portion size data on the reproducibility and validity of a semi-quantitative food frequency questionnaire. Am J Epidemiol 1988:128:891.
20. Willett W. Nutritional epidemiolgy. In: Rothman K and Greenland S, ed. Modern epidemiology. 2nd. edition. Lippincott-Raven, Philadelphia PA 1998.
* This paper was presented in the II International Conference on Nutritional Survey Methods. Boston, Mass. January 22-24, 1995.
** Romieu Y, Parra S, Hernández-Avila J, Madrigal H, Willett W, Hernández-Avila M. Questionnaire assessment of antioxidants retinol intakes in Mexican women. Unpublished paper.
(1) Instituto Nacional de Salud Pública, Cuernavaca, México.
(2) Centro Panamericano de Ecología Humana y Salud, México.
(3) Instituto Nacional de la Nutrición Salvador Zubirán, México.
(4) Department of Epidemiology and Nutrition, Harvard School of Public Health, Boston MA, USA.
(5) Channing Laboratory, Department of Medicine, Brigham and Women´s hospital and Harvard Medical School, Boston MA, USA.
Fecha de recibido: 31 de octubre de 1997 · Fecha de aprobado: 7 de abril de 1998
Correspondece should be addressed to Dr. Mauricio Hernandez, Instituto Nacional de Salud Publica, 655 Av. Universidad, Col. Sta. Maria Ahuacatitlan, Cuernavaca, Morelos, Mexico. Tel: 52-73-17-53-91; Fax: 52-73-11-11-48. Supported by a Research Grant of the National Institute of Health (Small grants for epidemiology), by the Ministry of Health of Mexico and the National Center for Environment from the Centers for Disease Control and Prevention, Atlanta, GA, USA.