Quality assessment of the Website for Eating Disorders: a systematic review of a pending challenge
Avaliação da qualidade do Website de Transtornos da Alimentação: uma revisão sistemática de um desafio pendente
Rocío Guardiola-Wanden-BergheI; Javier Sanz-ValeroI; Carmina Wanden-BergheII
IDepartment of Community Nursing, Preventive Medicine and Public Health and History of Science, University of Alicante. Campus de Sant Vicent del Raspeig. Apdo. Correos 99. 03080 Alicante Spain. javier.sanz@ua.es
IIDepartments of Physiology, Pharmacology, and Toxicology, University Cardenal Herrera-CEU
ABSTRACT
A Systematic review of articles from peer-reviewed journals was conducted, studying the complete texts in all cases. The key words selected were: "Eating Disorders" and "Internet." Descriptive analysis was performed on the variables selected, and Fisher's Exact test was used to analyze the presence of a linear relationship between qualitative variables. Of the 7 studies reviewed, 5 (71.42%) focused on content criteria and 2 (28.57%) examined both documentary and content criteria. Documentary quality was evaluated using two assessment instruments developed by two of the authors, comprising a set of 20 different variables, 5 of which were common to both studies (25% observed percentage agreement). Fisher's Exact test revealed significant differences between the two questionnaires (p = 0.049). Studies assessing Website content used 30 different topics to classify the subject matter, of which 6 were common to 2 studies. Although generic instruments for assessing the quality of Websites devoted to eating disorders were found, no validated questionnaire was found. This research did not locate any checklist or simple tool which could be used by consumers of health information available on the Web.
Key words: Eating Disorders, Internet, Quality control, Consumer health information
RESUMO
Revisão sistemática de artigos provenientes de periódicos com revisão por pares, estudando os textos completos em todos os casos. Os Descritores selecionados foram:"Transtornos Alimentares" e "Internet". A análise descritiva foi realizada para as variáveis selecionadas, e o teste exato de Fischer foi usado para analisar a presença de uma relação linear entre variáveis qualitativas. Dos 7 estudos revistos, 5 (71.42%) enfocaram critérios de conteúdo e 2 (28.57%) examinaram tanto critérios documentais e de conteúdo. A qualidade documental foi estudada usando-se dois instrumentos de avaliação desenvolvidos por dois dos autores, compreendendo um conjunto de 20 variáveis diferentes, 5 das quais eram comuns a ambos estudos (25% de percentagem de concordância observada). O teste exato de Fisher revelou significantes diferenças entre os dois questionários (p = 0.049). Apesar de terem sido encontrados instrumentos genéricos para avaliar a qualidade de sites dedicados a transtornos alimentares, nenhum questionário validado foi observado. A pesquisa nem mesmo localizou qualquer checklist ou simples ferramenta que poderia ser utilizada por consumidores de informação em saúde disponível na rede.
Palavras-chave: Transtornos da alimentação, Internet, Controle de qualidade, Informação de saúde ao consumidor
]]> Introduction
The Internet has become the single most important information search tool available, with libraries and specialists being consulted ever more infrequently on specific questions about certain subjects. Instead, questions are addressed through the use of key words on search engines, usually Google, which in recent years has gained ground over other search engines1. One of the purposes for which search engines are used is to find information on health-related issues. In a survey carried out in the year 2000, it was revealed that of the 53% of people who had access to the Internet in their homes, more than half used that access to obtain health information2.
There is no doubt that the use made of the Internet by patients who wish to satisfy their information needs will continue to grow as Internet access becomes more widely available. This increase, combined with the total lack of control over the Internet, could lead to problems related to the appropriateness of the information available. Moreover, given the dynamics of the Web, it would be difficult to implement quality standards which were clear, helpful and easy to use. Thus, due to its characteristics and constant changes, operating on a global scale and storing an excessive quantity of data, the Internet can have a negative influence on consumers of health information3.
The issue is not lack of information, but rather an excess of incomplete, unconnected and imprecise messages. Given this situation, access to quality, reliable information on the Internet is one of today's greatest challenges. How, therefore, to assess quality? And who should assess it? The unquestionable importance of measuring the quality of sites providing health-related information has led, in recent years, to the development of numerous instruments and scales for this purpose4. The problem, however, is that the answers to the questions raised above are still unresolved due to the lack of standardization and consensus on the part of the scientific community. Thus, one of the most current concerns among Public Health professionals is the quality of such information available on the World Wide Web5.
Continuing in the same vein, there are Websites in existence on the Internet providing information on anxieties or illnesses which primarily concern the adolesent population, such as those related to eating disorders, and some of these may have immediate negative effects, especially among young women6. Given that adolesents have still not completely formed their personality, making them more vulnerable to these potential negative effects, it is of the utmost importance to verify the quality of the information provided by such Websites.
Accordingly, the objetive of the present research was to study how the documentary and content quality of Websites concerning eating disorders has been assessed, analyzing the results and conclusions reported in the documents reviewed.
Method
This research comprised an observational, descriptive and cross-sectional study of articles obtained through a literature search, carried out using systematic check techniques, on assessment of documentary and content quality assessment of eating disorder Websites.
Literature search
]]> Following a study of the U.S. National Library of Medicine Thesaurus (Medical Subject Heading Terms, MeSH) and that of the American Psychological Association (Psychological Index Terms), the Descriptors selected as most suitable were: Eating Disorders (including the dependent Descriptors Anorexia Nervosa and Bulimia Nervosa) and Internet. The resulting search equation was defined creating a Boolean union with the connector "AND". Descriptors relating to Quality were not used as it was felt that these would unduly restrict the search.Furthermore, given that the Descriptor Eating Disorders includes other eating disorders which were not the the subject of this study, referents to Coprophagia, Female Athlete Triad Syndrome, Pica, Food Aversion or Binge Eating Disorder, were eliminated using the connector "NOT". The final equation was adapted to each of the bibliographical databases consulted.
The literature search was carried out from the first available date of each database consulted, to May 2009 (time of the final search). The search was limited to humans, and neither Subheadings nor Tags were used.
In addition, a secondary search was carried out in order to reduce potential publication bias. For this, the bibliographical list of articles selected in the main search was examined with the aim of identifying studies which had not been detected by the computerized search.
The following databases were consulted: Medline (via PubMed), Embase, The Cochrane Library, Institute for Scientific Information (ISI) Web of Science, Latin American and Caribbean Health Sciences Literature (Lilacs), Psychology Information (PsycINFO) and Documentación en Psicología (Psicodoc).
Google Scholar was also used to identify relevant documents. Additionally, as a secondary search, the bibliographies given in the selected articles were reviewed in order to identify studies not found by the primary search (gray literature).
Article selection
Final selection of articles was carried out according to compliance with inclusion and exclusion criteria.
Inclusion criteria were: the documents should be original articles published in peer-reviewed journals. Only articles where the complete text was available for retrieval were included.
Exclusion criteria comprised: studies which did not contain at least one instrument, whether validated or not, for measuring the documentary quality of eating disorder Websites, or for analyzing their content, were excluded. Also excluded were those which did not make specific reference to Anorexia Nervosa or Bulimia Nervosa, according to eating disorders as defined in the Diagnostic and Statistical Manual of Mental Disorders DSM IV-TR7.
]]> Documentary quality: informetric variables used in the reviewed studies.Content quality: thematic classification of the contents listed in the studies selected.
Studies were selected by the authors Rocio Guardiola-Wanden-Berghe and Javier Sanz-Valero. Divergence in article selection was resolved by Carmina Wanden-Berghe.
Statistical analysis
A descriptive study was carried out based on a calculation of the absolute and relative frequency (percentages) of variables, the most relevant of which were represented in tables. Information quality was monitored using double-entry tables, and errors were corrected by reference to the originals. Fisher's Exact test was used to analyze the presence of a linear relationship between qualitative variables. In all cases, p < 0.05 was considered statistically significant. Statistical analyses were carried out using SPSS 15.0 software for Windows (SPSS, Inc., Chicago, IL, USA).
Special research characteristics
Although it is preferable to base a systematic review on potential studies over an appropriate monitoring period, it was decided to include all research on documental or content quality of eating disorder Websites, irrespective of methodological design, with the condition that an instrument (questionaire or check list) had been used to measure or verify documentary or content quality.
Results
A total of 289 articles were found, of which 67 (23.18%) appeared on Medline, 22 (7.61%) on PsycINFO, 136 (47.06%) on Embase, 11 (3.81%) on the Cochrane Library, 50 (17.30%) on ISI Web of Knowledge and 3 (1.04%) on Psicodoc; no research was retrieved from Lilacs. 99 (34.26%) were repeated articles, leaving a total of 190 studies suitable for this research, of which 113 (39.10%) were original articles available in their complete version and published in peer-reviewed journals. The pertinent articles found in bibliographical lists had previously been retrieved during the database search.
]]> An analysis of scientific production at the present time produced the following statistics: mean age of the articles was 3.98 ± 0.18 (IC95% 3.63 - 4.34), with a minimum of 0 and a maximum of 12 years. Obsolesence of the articles analyzed, measured using the Median citation age (Burton and Kebler's Half-Life) was 3 years, and the Price's Index, 61.05% (percentage of articles less than 5 years old).Of the foregoing articles, 7 were selected, with the following designs: 2 (28.57%) were retrospective, 2 (28.57%) were prospective and 3 (42.85%) were cross-sectional (Table 1). It was observed that no study presented a validated questionnaire for assessing eating disorder Website quality; 3 (2.5%) contained a checklist, compiled ad hoc, of the documentary or content quality studied1,8,9. One contained ad hoc checklists1, and a clinical guide to eating disorder treatments compiled by the American Psychological Association (APA) together with the Level of Evidence scale drawn up by the Australian National Health and Medical Research Council. Four articles (3.53%) analyzed the conceptual contents of the Websites using qualitative techniques8,10-12 and two (1.76%) used descriptive statistical measurements to analyze the content of the Websites studied8,13.
One study14 developed a questionnaire to measure perceived impact of visits and characteristics of visitors to Pro-eating disorder Websites, but this was not included in the review as it did not specifically analyze documentary or content quality of these Websites. Another article15, excluded for the same reasons, reported a qualitative analysis of concealed identity, a common phenomenon on these Websites.
Website assessment
Winzelberg12 analyzed the content of Websites using a qualitative-quantitative technique. Based on this study, Eichhorn9 also looked at content, but only from a quantitative perspective. Mulveen and Hepworth11 and Brotsky and Giles10 also focused on content, applying a qualitative design. Wasemann and Grunwald13 used an assessment instrument of their own design to study Website content. Norris et al.8 carried out a qualitative analysis of pro-Ana Websites (Websites with pro-anorexia content) based on interviews, focus groups and earlier research into other topics. At the same time, they designed a non-validated instrument to conduct a quantitative analysis of the different documentary characteristics of such Websites. Murphy et al.1 studied both the content and documentary quality using quantitative analysis. Their documentary criteria comprised an ad hoc checklist and the Silberg Score. For content, they applied the clinical guidelines compiled by the American Psychological Association (APA), the Level of Evidence scale drawn up by the Australian National Health and Medical Research Council, and various non-validated instruments of their own design.
The review did not find any specific questionnaire for determining the quality, whether content or documentary, of eating disorder Websites. However, the research conducted by Murphy et al.1 described quality using statistical analysis.
Questionnaires and checklists: description and use
With regard to the assessment techniques employed in the research studied (Table 2), it was observed that 3 studies (42.85%) made use of passive observation to carry out an analysis of Website content9,11,12, 1 (14.28%) employed concealed participant observation10 and 3 studies (42.85%) measured quality using their own assessment instruments, designed ad hoc1,8,13. Only 1 of the 7 articles (14.28%) employed more than one instrument for assessing quality1.
Of the 7 studies reviewed, 5 (71.42%) focused on content criteria9-13, whilst 2 (28.57%) looked at both content and documentary criteria1,8.
To assess documentary quality, two assessment instruments designed for that purpose by the authors were used, having a total of 20 different variables; 5 of these (25%) were common to both studies (observed agreement percentage, 25%)1,8. These were: purpose, scope/clarity, owner, warning and website changes/up-dates (Table 3). Fisher's Exact test revealed significant differences between the two questionnaires (p = 0.049).
]]> In contrast, the studies assessing Website content used 30 different topics to classify subject matter, of which, 6 were common to two studies9,12 (Table 4).The American Psychological Association (APA) Guideline Score was not used to measure content quality, as it is comprises clinical guidelines to eating disorder treatment. Neither was the Australian National Health and Medical Research Council's Level of Evidence Score used, as this constitutes a hierarchical scale of scientific evidence1.
Discussion
The analysis of bibliometric indicators of the currency/obsolescence of the articles reviewed revealed that the subject selected is very contemporary, with 3 out of every 5 articles having been published in the last 5 years16.
Despite this currency, the data obtained in this review concerning approaches to assessing eating disorder Websites do not clarify research into content quality, or approachable of documentary structure. Moreover, although the instruments used to measure quality and the methodologies employed are useful to the scientific community, they are difficult to understand, and thus apply, for the general public. This fact was reported previously by Bernstam et al.4, who observed that, despite the existence of a range of Website quality assessment instruments, very few of these could be used by consumers of the health services, probably because they were designed for research purposes. Nonetheless, possible adaptations and/or simplifications could be used to assess the quality of such sites17.
Assessing Website quality presents a complex field for various reasons. Firstly, and as is the case for certain aspects of psychology, it is not possible to measure quality as if it were a a weight or a length; quality is a construct, and defining a construct is not the same as measuring it18. Secondly, quality is defined on the basis of user expectations, implying a large subjective component19. Consequently, assessment represents a systematic process of information gathering based on certain criteria and references, in order to form a value judgement concerning a specific situation and take decisions accordingly20. This definition summarizes an effective and efficient approach (effective, if the process is correct, and efficient if it enables comparisons to be made).
In practice, two means of assessing Website resource quality can be identified21:
a) Through the application of a formal, scientific, rigorous method which enables objective measurement using various parameters to identify whether the resources fulfill or attain minimum levels of quality, particularly in relation to the information contained.
b) Through the application of an inductive-deductive process resulting from compliance with a series of indicators, and through which the user can form a completely subjective opinion or judgement about the quality of the resource, based on its helpfulness and the use he or she intends to make of said resource.
]]> Clearly, assessement Website content or documentary criteria, or comparison to a Gold Standard, is a very specific task reserved for professionals, a situation which has also been observed in other areas of the health sciences22. At the same time, it is known that users do not generally judge the quality of health information obtained from the Web18, or they use relatively invalid criteria in order to judge the veracity or currency of the information23. Consequently, the eEurope 2002 study24 concluded that there was a need to exchange experiences and information on how to apply quality criteria in order to assist health information Website users when consulting such information. Many other studies have also emphasized the need to help citizens obtain reliable information from the Internet25, especially when such information relates to topics of interest to adolescents26.To recapitulate, and returning to the original aim of this research concerning quality assessment of eating disorder Websites, it can be concluded that:
Although no validated questionnaire was found, there do exist generic instruments for assessing eating disorder Website quality which, due to their complexity, are only appropriate for use by experts. Likewise, qualitative assessment methods were located, but these are also only suitable for use by qualified people.
Not a single checklist or simple instrument was found which could have been of service to consumers of health information on the Internet.
Lastly, the authors believe that it would be of interest if future research were to focus on developing instruments, techniques or simple checklists which would enable the general public to classify Webites, in general terms, as "good" or "bad" as regards quality of the information. These tools should be user-friendly and reliable: easy to use (no more than 10 items), self-directed, trustworthy and highly accessible.
Collaborations
R Guardiola-Wanden-Berghe was responsible by conception and study design, acquisition of data, data interpretation, manuscript an final approval; J Sanz-Valero was responsible by conception and study design, acquisition of data, data analysis, data interpretation, manuscript, statistical analysis and final approval; C Wanden-Berghe was responsible by conception and study design, data analysis, manuscript, statistical analysis and final approval.
References
]]>1. Murphy R, Frost S, Webster P, Schmidt U. An evaluation of web-based information. Int J Eat Disord 2004; 35(2):145-154. [ Links ]
2. Mollyann B, Flournoy RE, Altman DE, Blendon RJ, Benson JM, Rosenbaum MD. Health information, The Internet, and the Digital Divide. Health Aff (Millwood) 2000; 19(6):255-265. [ Links ]
3. Mathur S, Shanti N, Brkaric M, Sood V, Kubeck J, Paulino C, Merola AA. Surfing for Scoliosis: The quality of information available on the Internet. Spine (Phila Pa 1976) 2005; 30(23):2695-2700. [ Links ]
4. Bernstam EV, Shelton DM, Walji M, Meric-Benrstam F. Instruments to assess the quality of health information on the World Wide Web: What can our patients actually use? J Med Internet Res 2005; 74(1): 13-19. [ Links ]
5. Eysenbach G, Powell J, Kuss O, Eun-Ryoung S. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA 2002; 287(20):2691-2700. [ Links ]
]]>6. Bardone-Cone AM, Cass KM. What does viewing a pro-Anorexia Website do? An experimental examination of Website exposure and moderating effects. Int J Eat Disord 2007; 40(6):537-548. [ Links ]
7. Walsh T, Garfinkel P, Halmo KA, Mitchell J, Wilson GT. Eating disorders. In: American Psyciatric Association (APA). DSM-IV-TR Manual diagnóstico y estadístico de los trastornos mentales. Washington DC: APA; 2001. [ Links ]
8. Norris ML, Boydell KM, Pinhas L, Katzman DK. Ana and the Internet: A review of pro-anorexia Websites. Int J Eat Disord 2006; 39(6):443-447. [ Links ]
9. Eichhorn KC. Soliciting and providing social support over the Internet: An Investigation of online Eating Disorder support groups. Journal of Computer-Mediated Communication 2008; 14(1):67-78. [ Links ]
10. Brotsky SR, Giles D. Inside the "pro-ana" community: A covert online participant observation. Eat Disord 2007; 15(2):93-109. [ Links ]
]]>11. Mulveen R, Hepworth J. An interpretative phenomenological analysis of participation in a pro-Anorexia Internet site and its relationship with disordered eating. J Health Psychol 2006; 11(2):283-296. [ Links ]
12. Winzelberg A. The analysis of an electronic support group for individuals with Eating Disorders. Comput Human Behav 1997; 13(3):393-407. [ Links ]
13. Wesemann D, Grunwald M. Online discussion groups for Bulimia Nervosa: an inductive approach to Internet-based communication between patients. Int J Eat Disord 2008; 41(6):527-534. [ Links ]
14. Csipke E, Horne O. Pro-eating disorder Websites: users' opinions. Eur Eat Disord Rev 2007; 15(3):196-206. [ Links ]
15. Gavin J, Rodham K, Poyer H. The presentation of "pro-Anorexia" in online group interactions. Qual Health Res 2008; 18(3):325-333. [ Links ]
]]>16. Culebras-Fernández J, García de Lorenzo A, Wanden-Berghe C, David Castiel L, Sanz-Valero J. Careful! Your bibliographic references may be examined. Nutr Hosp 2008; 23(2):85-88. [ Links ]
17. Guardiola Wanden-Berghe R, Sanz-Valero J, Wanden-Berghe C. Eating Disorders Blogs: Testing the Quality of Information on the Internet. Eating Disorders 2010; 18(2):148-152. [ Links ]
18. Jiménez Pernett J, García Gutiérrez JF, Bermúdez Tamayo C, Silva Castro MM, Tuneu Valls L. Assessment of Websites with information on medicines. Aten Primaria 2009; 41(7):360-366. [ Links ]
19. Ayuso García MD, Martínez Navarro V. Evaluation of sources and quality of digital resources. An Doc 2006; 9:17-42. [ Links ]
20. González López I. Assessment: strategies to improve the quality in communication processes. Comunicar - Rev Cient Comun Educ 2006; 27:199-203. [ Links ]
]]>21. Salvador Oliván JA, Angós Ullate JM. To evaluate the quality of Web resources or just filter? Doc Cienc Inf 2001; 24:105-126. [ Links ]
22. Breckons M, Jones R, Morris J, Richardson J. What do evaluation instruments tell us about the quality of complementary medicine information on the Internet?. J Med Internet Res 2008; 10(1):e3. [ Links ]
23. Zeng X, Parmanto B. Evaluation of web accessibility of consumer health information websites. AMIA Annu Symp Proc. 2003; 743-747. [ Links ]
24. Commission of the European Communities. eEurope 2002: Quality criteria for health related Websites. J Med Internet Res 2002; 4(3):e15. [ Links ]
25. Rice RE. Influences, usage, and outcomes of Internet health information searching: multivariate results from the Pew surveys. Int J Med Inform 2006; 75(1):8-28. [ Links ]
]]>26. Hansen DL, Derry HA, Resnick PJ, Richardson CR. Adolescents searching for health information on the Internet: an observational study. J Med Internet Res 2003; 5(4):e25. [ Links ]
Artigo apresentado em 09/06/2011
Aprovado em 10/09/2011
Versão final apresentada em 30/09/2011
