Him ORIGINAL CONTRIBUTION Health Information on the Internet Accessibility, Quality, and Readability in English and Spanish Leo S. Morales, MD, PhD Jeffrey L Algazy, MD, MPH Richard L. Kravitz, MD, MSPH Michael S, Broder, MD David E. Kanouse, PhD Jorge A. Munoz, MS, MPhil Juan-Antonio Puyol Hannah Yan, Tizabeth A. McG HE INTERNET 1S AN INCREAS- ingly important source of health-related information for consumers, One recent sur- vey estimated that more than 60 mi liom US residents went online in search, of health information in the past year-' The online population is becoming more representative of the larger US population in terms of race, age, in- come, and educational attainment? Among those who use the Internet, more than 70% report the health infor mation they find influences decision about treatment! The ability to obtain accurate medi- cal information quickly, conveniently, and privately online presents to con sumers an opportunity for bet informed decision making and greater participation in care.’ Litle is known, however, about whether the available material is sufficiently complete and ac- curate to stupport consumer decision See also Patient Page. 2612 JAMA stay 2150, 2001 Context Despite the substantial amount of health-related information available on the internet, litle is known about the accessibility, quality, and reading grade level of that health information, Objective To evaluate health information on breast cancer, depression, obesity, and childhood asthma available through English- and Spanish-language search engines and Web sites. Design and Setting Three unique studies were performed from July 2000 through December 2000. Accessibilty of 14 search engines was assessed using a structured search experiment. Quality of 25 health Web sites and content provided by 1 search engine was evaluated by 34 physicians using structured implicit review (interaterre- labiity >0.90). The reading grade level of text selected for structured implicit review was established using the Fry Readability Graph method. ‘Main Outcome Measures. For the accessibility study, proportion of links leading to relevant content; for quality, coverage and accuracy of Key clinical elements; and grade level reading formulas Results Less than one quarter of the search engine's fist pages of links led to rel- evant content (20% of English and 12% of Spanish). On average, 45% of the clinical elements on Englsh- and 22% on Spanish-language Web sites were more than mini mally covered and completely accurate and 24% ofthe clinical elements on English and 53% on Spanish-language Web sites were not covered at all. All English and 86% of Spanish Web sites required high school level or greater reading abilty Conclusion Accessing health information using search engines and simple search terms isnot efficient. Coverage of key information on Engish- and Spanish-language Web sites, is poor and inconsistent, although the accuracy of the information provided's generally good. High reading levels are required to comprehend Web-based health information JAMA, 200%2852612-2621 Even online materialsare compre hensive and accurate, the ability of users to apply these assessment tools depends on their ability to loca’ and understand those materials. The making. Several studies of single medi- cal conditions have suggested deficien- cies in the quality of Web-based health {information Several organizations have devel- 5, No 20 Rept) oped criteria to guide and evaluate health-related Website content (eg, HON Code, American Medical Association, In- (ernet HealthCare Coalition, Hi-Ethies, MedCertain)," but these criteria have not been systematically applied toa broad set of Web pages and conditions. Pur- thermore, because many of these sys- ems rely on voluntary self-assessments by Web page developers, the reliability and validity of many of these evalua- tions is unknown. Internet has the potential to eliminate barriers in access to information for ‘Author Afations: RAND Heath (Os Bend fet, Moraes Agar, Broder, Kanouse, Lara Wat lens and McGiy, Hr Puyel ana M6 Yan) and RAND Graduate School (Mr ox), Santa Monica, Catt Robert Wood lbson Cones Scholars ro gram, Unversty of Catia Los Anges Os Ber (Snare Agar Unversity of Calfoma, Los Ange le Shoalf Meine Ds Motes, Bode and at) pd Unversy of Calflomi, Dav Snel of Med fe (Drea ‘Conespnding Author and Reprints: Gretchen Be. land, MD, 1760 Man St, Me, Santa Mose, CA 90407-7138 (e-mal:belanddrand og). (©2001 American Medical Association, All rights reserved.patients, but only online material ean be read and understood by many dif ferent types of users"? Preliminary data from the 2000 US Census indicate that the popilation is becoming increasingly diverse. Since 1900, the US Hispanic population has grown from 22.3 million to 35.3 mil lion, making Hispanies the largest mie nority group in the United States Among immigrant Hispanics, more than 98% report speaking primaeily Spanish at home.** While accesible and high-quality health information om the Internet simportant for English speak cst could be even more useful for Spanish speakers, who face greater bar- Hers to traditional sources of medical ceare and information.**"* We are un- aware of any studies that have evalu- ated Spanish-language materials We conducted a large cross- sectional study to describe and evalu- ate health information on the Internet in English and Spanish, We evaluated information that we found using search, engines and by visiting health-related Web sites on 4 medical conditions: breast cancer, childhood asthma, depression, and obesity. We asked 4 main questions: What are consumers likely to find when they search online about these conditions? How compre- hensive isthe information? How accu- rate is i? At what grade reading level is the material presented? METHODS: ‘We conducted 3 studies to assess the ac- cessibility of relevant content; the qual- lty of health information; and the read- ing grade level of text. Each study used. different methods 1o assess the same 4 conditions. Conditions were selected by project staff based on prevalence, clinical significance, and diversity of the affected populations." Bach study was conducted independently in English and Spanish, Study 1: Accessibility of Relevant Content Selecting Search Engines. Search en- ines are designed to help people lo- cate information on the Internet, To as- (©2001 American Medical Association, All rights reserved, HEALTH INFORMATION ON THE INTERNET sess how well search engines perform this function, we selected 10 English- language and 4 Spanish-language search engines, Three ofthe English-language and 2 of the Spanish-language search e gines were chosen based on popularity (defined as the number of unique vis tors per month as reported by Media Metrix, Incin June 2000"). The remain- ing 9 search engines were selected be- cause they featured unique methods of ranking Web sites.”” Examples of rank- ing methods included ranking by loca Lion and frequency of key words within asite; ranking by the number of times a site is linked to by another site; rank- ing by payment from sites; and ranking by human editing. ‘Conducting Standardized Searches, Trained searchers entered the 4 search terms ( asthma,” “depression,” and “obesity”) {nto each of the 14 search engines. All links on the first electronie page for cach search engine were then counted and classified. Links were classified as east cancer,” “childhood relevant if the search term or 1 of 30 to 40 related key terms per condition (eg, tamoxifen, inhaler, gastric bypass surgery, St John's wort) was present fm the link itself or the surrounding Searchers then followed a sample of relevant inks to determine whether they led to relevant content, One sample was the first 5 relevant links on the search sults page. All remaining links were en rmerated and divided into 5 strata of equal size; relevant link from each stratum was selected randomly. Searchers clicked on selected relevant links until they reached a Web page with content (de- fined as when 50% of the space oceu- pled contained text that was not primar ily an index of the site). If the first relevant link led to a content page, the page was saved for further analysis. [the first link led to more links, the searcher randomly selected a relevant link from the fist 15 relevant links on that Web page. searchers had not reached acon- {ent page after 10 cycles, the search was discontinued, Characterizing Content. Using a standardized form, trained coders first classified Web page content by rel- cevance. Web pages were coded as rel- ‘evant if they contained any materials lnted to the 4 search terms or the 30 to 40 key terms related to each condi- lion, Coders then assessed the rel- ‘evant pages fr the presence of promo- tional content (defined as material designed to encourage site Visitors to purchase products oF services or par- licipate in research programs spon- sored by the site). Explicit advertise- ments were classified separately from promotional material and had to be lo- cated ina banner or sidebar on the Web page Study 2: Quality of Health Information on the Internet Selecting Web Sites, Eightcen unique English-language health Web sites (6 general health, 12 condition-specifi) and 7 unique Spanish-language Web sites (3 general health, 4 condition- specific) were selected for this study (TABLE 1). We selected 6 English- language general health Web sites that were ranked highly in2 widely used Ine temet industry reports, Cyber Dia- logue and PC Data Online for Septem- ‘ber 2000.** Content provided by one ‘ofthe most popular search engines was also inclided."* Condition-specifie En- glish-language Web sites and all Span- ish-language Websites were selected by project stall o represent prominent x amples of condition-specilic Web sites from commercial, government, and nonprofit educational organizations Project staff limited Web sites to those nol requiring subscriptions or pay. Developing Condition-Related Top- fes and Questions. Panels oft 4 nae Uionally recognized clinical experts and represenativesfrom patient advocacy of- snizaions identified 5 to key clinical topicareas foreach condition (26 in al). Panelists were reerited for their cini- cal or scientific experience, familiarity with national guidelines, current re- search, oF national reputations in the medical conditions of interest. No pat clisthad consulted for, or had any finan- ial involvement with, any e-health Web (Reprinted) JAMA, May 23/30, 2001 Vol 295, No, 20 2613HEALTH INFORMATION ON THE INTERNET site, Panelists were asked to identify top- the Web. The panels then wrote 36 con- les that were relevant to patients, their sumer-oriented questions relating to the families, orlaypersons seeking informa- 26 topics. tion on the study conditions. Panelists For example, the topic “breast can- also considered whether it was reason- cer screening” was characterized by the able to expect tofind thisinformationon following questions: “No one in my fam Selected English and Spanish-Language Web Stes" TAL Aa Engiah-Language Web Sites Popular genera heatht ‘Alneath.com vw atheath com Breast cancer, chiahood asthma, depressor, obesty ‘as Teamh Wath ‘wan cbehealiwalehcom Graal cancer, chidhood salina, depressor, obesty Bimoapcom ww aRGOp-CoM Breas cancer, cidhood ath, depressor, obesty inaeath Wau hsineathcon Breast canes, cnidhood salina, depressor, obesty Onna aw onteathcom Grea cancer, chdhood aha, depressor, obesty Wane wan webna om Grea cancer, chdhood aha, depresson, obesty Conair spester "hmercon Reademy of ww aaaaiong Chidhood asthe ‘ergy, Aston, nd inimunobg ‘Rnercan Gancer Sosy ww saree? 97 Breas eaeer Imerean Obesity Twn obest og ‘best Sszociton ieath om Depression Tan reas cancer Depresson cong Twa depression com —Depressan iyasima wave myasthma.com —Chidhoed asia Nafgnal Hear Ling, and _www.nhbinh gow ‘Chidhood asim Blood state ‘ajonat netute of WRT AGaY Dapressan Monta Heath ‘ajonat obra of WRC GST ‘best eaione best Onna ww obesTyonina rom Obey resin Twn onectniccom reas cancer ‘Seen enone "Yahoo! ww yahoo.com Breast cancer, depression, obesity “Sparish-Language Web Sites General heath ‘Gracaedocter www gracasdoctorcom Breast cancer, chidhood asthma, depressor, obesty Er waa cor Grea cancer, chdhood aha, depressor, obesty Sane ‘Wau saudltnacom Breast canes, chidhood salina, depresson, obesty Conan specter ‘Cancemet wawcancemetgov Breas cancer Tena Fess wawccentopeso.com Obes ‘atonal netuter of Heath —www.ngov ‘Chishood arin ‘New York Onine Access ——ww-noaf-healnorg Depression Pees Ege acer by JAW ae RAND a ay 7, 2801 Cop othe Web st cee 2614 JAMA stay 2150, 2001 ily has had breast cancer. Do I stil need breast exams and mammograms? When should I start having regular mammo- ‘grams? Do I need one every year?” A ‘complete lst ofall condition-relatedtop- ies and questions is located in ONLINE TABLE 1 (available at http://www jama com). Development of Clinical Elements Toenhance the consistency ofthe struc- tured implicit review, the 4 clinical pan- elseach developed series of | 08 cli cal elements for each of the questions based on evidence-based guidelines and materials from selected literature re- for the topic of breast cancer screening, 4 clinical el ments were developed. These included. the following: women older than 50 years should have mammograms every 1 102 years; early detection of breast cancer ini- views." For exam! proves outcomes; most breast cancers oc- ‘eur in women without a family history fof the disease; and a lack of consensus exists about the need for or appropriate interval of mammography in women from age 40 to 49 years. A total of 100 clinical elements were developed (On- line Table 1). Retrieving Health Information, Four abstractors (2 monolingual in En- lish, 2 bilingual in English and Span- ish) independently reviewed each Web site (spending a maximum of 90 min- utes persite using high-speed Internet connections) on October 18-30, 2000, and November 6-13, 2000, to retrieve content related to the questions (On- line Table 1). Abstractors did not r ceive any of the condition-relatedclini- wgeach sieved cal elements prior to conduct search, On average, 65% of |Web pages were common between ab- stractors. Search results were saved us- Ing a software application (Catch- TheWeb, Math Strategies, Greensboro, NC) that enabled project researchers to and manage accurately save, abstra Web pages for later use. rieved materials were stripped of identifying information, printed, and as- sembled into notebooks. Each now book contained the materials retrieved from a single search on a Website (¢ 1 condition persite). The 78 unique En- 5, No 20 Reps) (©2001 American Medical Association. All rights reservedslish-Language notebooks averaged 250 printed pages (range, 21-547 printed pages). The 32 unique Spanish- language notebooks averaged 68 printed pages (range, 8-366 printed pages). Ato- tal of 21711 printed pages (2660 Web pages, defined by the programmer's end- (of page mark) were abstracted across + conditions: 19529 printed pages (2262 |Web pages) from English-language and 2182 printed pages (398 Web pages) Irom Spanish-language Web sites. Evaluating the Web Sites. Thitt four physicians (30 monolingual in E lish, 4 bilingual in English and Span- ish) from around the United States wer recruited to evaluate the abstractor- retrieved material. All reviewers were board eligible or board certified in fam- ily medicine, general surgery, internal medicine (including allergy and immu- nology, hematology and oncology. in fectious diseases, pulmonary and criti- calcare),orpediatries. No reviewer rated more than 5 notebooks for any condi- lion or evaluated materials from the same Web site twice, Forty English- language (51%) and 14 Spanish- language (44%) randomly selected note- books were evaluated by 2 reviewers. Each Web site underwent 2 to 4 re- views per condition, Four standardized rating forms we developed that listed the condition- related topics, questions, and clinical elements (eg, 1 condition perform). R viewers were asked to rate the level of coverage foreach clinical element as not addressed, minimally addressed, or more than minimally addressed. Not al- dressed meant there was no reference to the issue on any page of the note- book. Minimally addressed meant the clinical element was mentioned at least briefly. For example, for breast cancer screening, if mammography was men- tioned as a way to identify early breast ceancer, but no mention was made of who should have mammograms, how often they should be done, or their util ity in reducing breast cancer mortal- this was considered minimal cov- erage. More than minimally addressed meant that most of the clinical ele- ments were mentioned and the level of (©2001 American Medical Association, All rights reserved, HEALTH INFORMATION ON THE INTERNET explanation was more than cursory. For example, coverage was considered more than minimal ifa Web site mentioned that screening mammography was the best way for breast cancer to be di tected early in women older than 50 years, or that breast cancer may be de- tected earlier by mammography than physical examination, or if a detailed discussion of the pros and cons of mam- ography and the appropriate ages for screening were provided. Reviewers also rated the accuracy of content for each clinical element that was at least minimally addressed: mostly incorrect, mostly correct, and completely correct Alter rating Web site materials on. coverage and accuracy, reviewers were asked to list instances of conflicting in- formation found during their review. These conflicts were not limited to the set of clinical elements for which cov- erage and accuracy were evaluated. Six categories of conflicting information, were identified: (1) treatments; (2) dh agnosis; (3) definitions; (4) adverse ef- fects; (5) etiology and risk factors; and (6) incidence and prevalence. Two project physicians (RIK andJ.1A.)in- dependently rated whether the ex- amples of conflicting information were ‘minor, significant, or potentially dan- gerous, Examples that were identified as significant or potentially dangerous by both physicians were included in the final analysis, Analytic Methods We used Stata statistical software (ver- sion 6.0; Stata Corporation, College Sta- tion, Tex). The unit of analysis was the link (specific URL [uniform resource lo- cator]) forthe study of search engine ef- ficiency, the standardized rating form for the study of quality. and the Website for the study of grade reading level Rating forms contained multiple rat- ings (corresponding to clinical el ments) of coverage and accuracy us- ing the 3-point ordinal scales mentioned previously. For purposes of analysis, summary measures were computed by averaging across elements within a given rating form, All analyses were conducted sepa rately for English- and Spanish- language search engines and Websites. All statistical tests were 2-sided and ‘were assessed for significance atthe 03 level. Measures were tested for vari tion by condition, search engine, and site, as applicable. A 2-stage test pro- cedure was used Lo examine variation in each outcome by these indepen- dent categorical variables. First, an om- nibus or overall test of the association, was performed. Ifthe omnibus teste lablished that variation in the out ccome of interest was statistically sig- nificant for the categorical variable (condition, search engine, site), a s Hes of 2-sample follow-up tests wer performed comparing the outcome at each level of the categorical variable ‘with the overall distribution ofthe out- The omnibus tests used were I-way analysis of variance, the Kruskal-Wallis rank-sum tes, and the x* tes of homo- geneity for measures that were nor- mally, ordinally, and nominally distib- uted, respectively. Two-sample ¢ tests, Wilcoxon rank-sum tests, and x*tests of homogeneity were the corresponding fl- low-up tests, Interrater Reliability Inthe search engine study, interrater te liability of the judgments by searchers and coders was high for both classi tion of links and content («=0.80).” In the Web site study, 2 measures of interrater reliability of Web site review= cers were computed. standard me: sure of reliability, computed as the cor- relation in ratings between reviewers examining identical notebooks of ma- terial retrieved from the same Web site, was caleulated, To assess the sensitiv= lty of reviewer ratings to variation in the retrieved material (eg, the material re trieved by abstractor I vsabstractor 2 on, the same Web site for the same condi tion), a second, more stringent me: sure of reliability was computed as the correlation in ratings between review- cersexamining diferent notebooks of ma- terial from the same Web site and con- dition. We computed 16 interrater (Reprinted) JANA, sy 2/30, 2001 Vol 295, No, 20 2615HEALTH INFORMATION ON THE INTERNET reliabilities by the standard ruleand 16 and end of the material abstracted from — topic. Thus, when following apparentl by the stringent rule for each language: each Web site were selected. For each relevant links, relevant content was den- or every combination ofthe #condi-100-word sample, the number of sen- tified 59% of the time (Figure 1 and Table tions and the 4 assessments (any cov- tences and syllables were counted. The 2). There was significant vari erage, more than minimal coverage, FRG calculates an estimated grade level__ likelihood of reaching relevant content completely correct, and the combina- as a function of the average number of from potentially relevant links by search. tion of more than minimal coverageand sentences and average number of syl- engine (range, 35%-88%, P<.001, Table complete correctness). Thirty reviews lables for each source document. The 2). Onein 5 (20%) links on the frst page ‘were included ineach calculation of n- FRG accounts forthe fact that Spanish of search results led to relevant content terrater reliability on English-language documents tend to have more syllables (Table 2). There was no significant varia- Web sites and 12 reviews were in- per word than English documents ofthe on among search engines in the prob- jon inthe cluded in each calculation of interrater same reading level abulty that first-page inks would lead to reliability on Spanish-language Web relevant content sites, The standard interrater reliability RESULTS Results for Spanish-language search ‘vas0.90 orgreaterforallconditionsand Efficiency of Searches engines were shailar. The fist page of teases, averaging 0.96 for both En- ad Type of Information results returned 1685 links, 206 (18%) slish- and Spanish-language sites. The FIGURE Land TABLE2 summarizetheex- of which were relevant. Among the 151 second measure of interrater reliabilt jence that someone secking informa- selected relevant links, 101 (67%) led t0 averaged 0.77 for English- and 0.60 for tion would havewhen using asearch en- content and 95 (94%) of those pages Spanish-language Web sites gine. The first page of earch resulls from — contained relevant content. Overall, 63% allEnglishlanguagescarchengines listed of relevant links le to relevant content Study 3: Reading Grade Levels 3735 links, 1205 (34%) of which were (Figure 1 and Table 2). There was sig- ‘of Web Sites relevant. The proportion ofthese links nificant variation in the likelihood of To determine reading grade level, we that were relevant varied significantly by _ reaching relevant content by search en- used the Fry Readability Graph (FRG) search engine (P<.001). Among 389 _ gine (range, 49%-78%, P<.001, Table) method, which has been validated in sampled relevant links, 288 (7436) se- ‘Twelve percent of all links on the frst both English and Spanish." Three lected led to a content page within 10 _ page of search results led trelevant con- sample passagesof text exactly 100 words clicks, and230(79%) ofthose pagescon- tent, with no significant variation by in length from the beginning, middle, tained content relevant to the search search engine (Table 2) Figure 1. Pow of Seach Engine Stacy io sobseg — A semi ss0. 08) Tom be Span o8 Be] > eevat gin 1209 24%) gen 288 content Pane Teta ne Spann 206) ant Toone 4tsimpe seach terme [>| Eatin 739 Paar ees gen 2479 panen 1309 2616 JAMA stay 2130, 2001 5, No 20 Reps) (©2001 American Medical Association. All rights reservedsix percent (n=120) of the rel. evant English-language content pages contained explicit advertisements and 44% (n=101) contained other promo- tional material The presence of adver Usements and promotional materials on relevant Spanish-language content pages was 36% (n=34) and 21% (n=20), respectively ‘Quality of Health Information Coverage of Topics. Coverage is re- ported as the mean proportion of clini- cal elements across sites with no cov- ‘erage; minimal coverage: minimal coverage of the clinical ele- ments for each condition, Among En- slish-language sites, the mean percent age of clinical elements that wer covered varied significantly across con- ditions: 16% for breast cancer, 27% for childhood asthma, 20% for depres- sion, and 35% for obesity (TABLE 3) Topies that were not covered most of- ten included alternatives to standard. medical and surgical treatments for breast cancer (28%), symptoms sug- gestive of poorly controlled asthma (48%), evaluation of depression (33%), and safety and effectiveness of dietary supplements used for obesity (61%). ‘On Spanish-language Web sites, the ean percentage of clinical elements ceiving no coverage also varied signili- cantly across conditions: 40% for breast cancer, 33% for childhood asthma, 61% for depression, and 69% for obesity (Table 3). Topics that were not cov- cered most often included alternatives tostandard medical and surgical treat ments for breast cancer (90%), ex- pected benefits and possible adverse ef- fects of asthma therapies (44%), evaluation of depression (84%), safety and effectiveness of dietary supple- ments (100%), and types of popular di- s for obesity (100%). Accuracy of Information, On En- alish-language Web sites, the mean per- centage of covered clinical elements for which the text was completely correct was 91% for breast cancer, 84% for childhood asthma, 75% for depres- sion, and 86% for obesity. In Spanish, the mean percentages were 96% for sad more than (©2001 American Medical Association, All rights reserved, HEALTH INFORMATION ON THE INTERNET ‘Table 2 senrch Engine Characterstcr and Evteency m Reaching Relevant content No of inks levant Proportion of ‘on ist Results ing” Links Leading Search Page é Mechel . ato Relevant Engine Gondiions) Gontent, Content So" Engish-Languag over 3795 1285 @4) 89 20 Raweta 282 Taz 60), SF 78 ake Ti 209 B5IF 58 7 Dreet tie ate Ta 3h 76 Ect Be Sree 80 76 ‘Google 83 STB Ta 2 See ae 135 3 Br 7 jeer 35s Tze 30 7 eracaner ee 15.40 70 2 Tort Ea 35 23 3a 2 Yahoo Te BEET) oF B ‘Sparish Language Engin overt 1685 206 (8) 63 2 ‘Guepara 253 Ea oe 7 Te Respond 353 105.02 Ta 7 Yahoo Espanct Tat BSS 5 Ey TE 70 breast cancer, 53% for childhood asthma, 63% for depression, and 68% for obesity ‘On English-Language sites, the mean percentages of covered clinical ele- ments rated as mostly incorrect were 0% for breast cancer, 3% for childhood asthma, 3% for depression, and 3% for obesity. In Spanish, the mean propor- Lions were 0% for breast cancer, 4% for childhood asthma, 18% for depres sion, and 0% for obesity. As an ex- ample, one depression site stated that omega-3 fatty acid deficiencies cause major depressive disorders. One child hood asthma site describes cock- roaches as the leading cause of asthma among children, ‘Combined Measure of Coverage and Accuracy. In English, the mean per- centage of clinical elements receiving ‘more than minimal coverage that were completely accurate was 63% for breast cancer, 36% for childhood asthma, 44% for depression, and 37% for obesity. For breast cancer, depression, and obe- sity, there was significant variation among English-language Web sites (Fable 3). Two sites performed statis- Lically better than average: www.On- colink.com for breast cancer and www.nimh.gov for depression (for both, P=.02). No Web site was statis- Lically better than the condition ave age for childhood asthma and obesi ‘On Spanish-language Web sites, the corresponding proportions receiving more than minimal coverage that wei completely accurate were 39% for breast ‘cancer, 23% for childhood asthma, 12% for depression, and 15% for obesity There was significant variation among Websites for breast cancer and depres- sion (P<.05), but no Web site was statistically better than the condition average For a comprehensive summary of coverage and accuracy of elements of condition-related topics for the 4 con- ditions, see ONLINE TABLE 2 Conflicting Information, Overall, just, cover half of English-language Web site reviews revealed 1 or more conflicts in the information provided (TARLE +; Spanish reviewers noted no conflicts) Conflicts involved treatment (present in 35% of reviews), diagnosis (13%), definitions (7%), adverse effects (5%), (Reprinted) JAMA, May 2/30, 2001 Vol 285, No, 20 2647HEALTH INFORMATION ON THE INTERNET ‘Table 3. Comparison of Coverage ana Accuracy of Selected Condition Related Topi for Conaiuons “Average for10 English-Language “Average foe 4 Spanish Langan eb Stos por Condon "Web Stes par Conditon Coverage, % More Than "Coverage, % ra Conectness, 5% Minimal Corectness, More | —TeCUPS® coverage and More Condition Related Tran Mosty ‘Completely ‘Than Mosty iy ‘Topics NoneMinimal Minmal "Not" Mosty Completely Comect NoneMinimal Minmal Not’ Mostly Completely Const, Breast Cancer overaig 117 eto ot ee ee 39 Fikasuementand 10 —12——TB 0 11a 78 0 3 lee of amorien forrskregucton ‘Seeenng a a LY a a Ener oT 3 ot ¥ .—2—30 7 3 pabablbreast Taare Wa 5 a 0 or Bernatves Ts 2a a a a 10 ‘lander eupeal na mesial treatments Chichoed hati vera aya) ast gt asp st uot 23 Symptons a 3 a a Poor conrated 18 —25 wo 2-3 7 % ehma Theapes and savers TT a8 a8 3 ete ial nanagenetal 215 Tar wa 7 ‘avers aetima REDE 1 = I 7 7 zr Ehotgy z 13 ae a a5 2 Evpectatone Fan 2 =o 0 a 1@ therapy Depression vera aor ss ge 8 ai ete tS te 128 Syptons a i 1a Tea 41 z 5 an! Ea i z Inidepressnt 1817 87 3 —22—T a 7 adestons Foe ofcounseing 308 a1 573 a 7 esaaeston —T 2 7 1345 0 5 Evauaton ass 3a es 7 Ehotay Ts a = ‘best overaig 8b aoe st as Sree 158 oe 18 Danner 7 co 1 —Te 7 F a7 ingeatons for woghtoss ath rte 2a a zr Fikendomalea a9 a8 388 710 T—0 7 popular dels Prysealachaty 2057.8 BT a a prevention Nafesons soe 05 5 7 ar z 9 T 7 for weight bse Sage E27 a a a z Seley and 321 10 1 7 7 hectveness olde Supplements Nansen a 2 a 7 —10 7: onions {Weighting af cincal elements equal, rater than welghtng al tplcsequaty. See Onne Table 2 iupies ccsie hat anoemet waotatare na Co orbs seed wos 2618 JAMA. ay 2720, 2001 205, No 20 Reps) (©2001 American Medical Association. All rights reserved Downloaded From: https:/HEALTH INFORMATION ON THE INTERNET Tresence of Confcting information on Engieh-canguage Wee ster ‘of Reviews With Any Conflicting Information by Category “of Reviows Noting ‘Adverse Etiology and Incidence ‘Any Gontict Treatment Diagnosis __Defritin _‘Eifects__lskFactors and Prevalence ‘imeteal condone 3 % 13 7 = 5 7 Breas cancer a 2 T 7 3 3 ‘hadhood aetna o 0 a 0 7. 7 Depression oT 7 7 7 7 iw besa F 77 z 7 7 greeny ener ieee er pa an ar ne ean conaten PUSS) Syme moreno Mp man orne msn cons [= 63) «tiology and risk factors (59), and inci search engines led toa Web page with = ==S Net Cte tee aes dence and prevalence (4%). Asan ex- relevant content ‘Wea Stes by Language ample, childhood asthmasitestated at We examined 2 dimensions of Web fone point that inhaled steroids do not site quality: whether key consumer pun opae nie stunt growth and later stated that in- questions were covered and whether the I Eogerrgoge oes) haled steroids do stunt growth. Materi-_ information was accurate. Although we als on depression were the most likely found thousands of pages of material cause [a to have conflicts on treatment, whereas related to key questions, there were sub-—* =t I+ breast cancer materials were the most stantial gaps in the availability of key Ukely to contain contlits on diagnosis information. Onlyhalf ofthe topics that cage (Table 4, P<.001), the expert panels thought wereimpor- £ tant forconsumers werecovered more = Reading Grade Level than minimally. This deficiency was 22 For English-language Websites, theav- particularly striking across Spanish- m3 erage reading level was collegiate mean language sites, where more than halfof [SD] grade, 13.2[2.1)) and ranged from the condition-related topies were not wpa loth grade to graduate school level addressed. (FIGURE 2). For the Spanish-language Our results suggest that consumers ote eo oe Websites, the average reading level was using the Internet may have a dificult at 10th grade (mean [SD] grade, 9.9 time finding complete and accurate in- Raging gage ewh were eemined ung ey [2.5)) and ranged from grades 7 to 13. formation on a health problem, If Reasbiy Graph method. (Figure 2). The mean grade reading people are relying on the Internet to level for the English-Language Websites make treatment decisions, including, ‘was significantly higher than forSpan- whether toseek cate, deficienciesin in- almost 48960 the population—and more Ish-language Web sites (P<.003). formation could negatively influence than 75% of current welfare recipients consumer decisions. For example, less have low or very low reading skill.” COMMENT than half of the Spanish-language ma- Thus, even if wider access to computer Toour knowledge, thisis the firststudy terials explained that mastectomy and technologies narrowed the digital di- to examine English- and Spanish- lumpectomy plus radiation are equiva- vide, the online health information cur- language health information on the In- lent treatments for early-stage breast rently available would be difficult for ternet across multiple conditions. We cancer. many people to understand. found that search enginesare only mod- The reading level of most Web-based This study has some important limi- crately efficient in locating informa- material isquite high. Allofthe English- tations. First, the Internet isa moving tar- tion ona particular health topic. More language sites had material that re- get, and we were able to take only asnap- than halfof consumers who use the In- quired atleasta lOth-grade readinglevel, shot of its performance. Changes in ternet report that they spend about a and more than half of the sites pre- content over time are not represented. half hour looking for health informa- sented material at the college level. Al- However, without dedicated attention, lion, so efficiency is an important as- though 1 Spanish-language site pre- it seems unlikely that the variability in pect of performance (Carolyn Gratzer, sented material at theelementary school performance, gaps in availability of i Cyber Dialogue, oral communication, _level,all others required atleastaninth-_ formation, and high reading levels will October 13, 2000). Overall, Lin S links grade reading level. According to the change dramatically. Second, we looked. identified by 10 English-language and 1902 National Adult Literacy Survey,92._atasmall set ofsearch engines, Web: Lin 8 links from 4 Spanish-language million adults in the United States— and conditions, and hence cannot draw (©2001 American Medical Association, All rights reserved. (Repent) JANA, sy 280, 2001 Vol 285, No, 20 2619HEALTH INFORMATION ON THE INTERNET more general conclusions. However, be- cause we included the most popular search engines and Web sites, the results are likel periences. Their variability in perfor- mance suggests that the likelihood of finding the information one needs, on the topic of one's choice, will depend on where one starts. Third, we studied the performance of search engines using very simple search terms, Had more sophis- licated search strategies been used, our Tindings might have b Fourth, our research was not a natural experiment (eg, using actual consum- cers to search for information and test- ing their knowledgeafter such a search), so we cannot draw conclusions about what people actually encounter when they search for information, or about thow well they are able to interpret the information they find. Filth, the neces to reflect common ex en different saty inclusion of medical terms in ana- lyzed text may be partially responsible for high estimated grade reading levels, although we assessed Flesch-Kincaid scores" in the same passages with and without the medical terminology in- cluded, and when medical terminology ‘was removed, the grade level declined by only 0.3 grade levels on average (range, 0.1-08) Sixth, because the Internet and many Web sites make avalable a large ‘volume of material, itis possible tht our searchers missed information that was available on a site. For that reason, we hhad 2searchers look for information on cach site, and they found different ma- terial, But the conclustons that review= cers reached about sites were quite con- sistent, even when the retrieved material they evaluated was different. Further more, oursearchers were skilled, trained for the task, and devoted more time 10 Finding information than people report spending on average. Thus, four search- cers could not find the information, prob- ably most consumers also would have difficulty doing so. (Our results suggest several ways to rake Web-based information more use- ful. First, variation among search en- fines suggests that is possible to im- prove search efficiency, perhaps by improving the methods for indexing 22620. JAMA. ay 2920, 2001Vl 205, No 20 Reps) ‘Downloaded From: https:/jamanetwork.com/ on 07/27/2021 Web pages. Second, the lack of eritical information for each of the 4 condi- Lions suggests that Web site developers should focus on providing more com- plete information. Third, Web site de- velopers need to enstre that the infor mation isaccurate and fie from conflict, Although accuracy levels were gener- ally high, the presence of conflicting in- formation males it possible that people will be more confused than enlight- ened, Fourth, some mechanism for rou- tinely rating Web sites for coverage and accuracy may be useful. Comprehen- sive assessments ofthe type conducted for this study are highly labor inten- sive, but simpler methods also may be effective. Fifth, information on the Web needs to be made more readable if the Internet is to serve asa “leveler” across different socioeconomic backgrounds. The Internet has the potential to be 1 powerlul resource for meeting some of the publics health information needs. Ideally, consumers would be able t0 learn much of what they need to know from high-quality Web sites, so that the limited time they have with their phy- siclans could be used more efficiently However, this requires that Web sites present well-organized and accurate in- formation in a way that is understand- able. Research is needed on how the publics use of the Internet facilitates, complements, or complicates patient- physician communication and on how patients and health professionals can make better use of this resource. ‘Autor Contin: Sut cncptand sig Be {an Esot, Moles, ears, Alga. Broder a tous, Pye Lara tuna Yang Meco, ‘Acguistion of dt: Beran, orale, Agazy, fas, aye Yang ‘nai an tpt of dt: Ban, Eo. dies Rvs, Rey Kruse tae Min Disting of ne manip Brand Ein Mores Kees gay, Sader anu Pye Wats Yan teva Cite eon fhe manu fr inert i {tects contens Bran Eo, Morse Ke, ‘rss oder tance, hana, as Mya. ‘Sia eet’ hot, Mores, Nur. Obie nding Stan ore, lay, Ninn ‘amish, rata sopparbr {aa oe Alga, Kanoue, anos, Pye La Yang. cy ‘Sd pene: Send, ot, rode, Kanouse, Byal ang Meciyn. Funaing/Support Suppor for ths sacha ‘ned foe alterna Hes Founéaon 2 FAN (Gr Berard), and the Robert Wood Josan Foundston Cine Scholars Program Drs Beandand ‘Agazp (Ontne Tables are posted on the JAMA Web ste at Iitpy/jama ama-assn.org/ssues/v285n20/2B5 Jjeca278 ni ‘elnowedgment: We hank Dan Durazo for his ad ‘ein seeing spans language search engines and Wied ste and An F Marron uaty iste "eto, and Sam Karp, ce nfrmaton feof the alterna Hest foundation forthe plc ‘pens on the manascigt We ao tank 34 phy ‘ean who reviewed the Web ste mate RERRENGS 4. FoxS, Rane L. The Online Healt Care Reval tian Ho the Web Hels Americans Tae Better Care ‘of Themgeves. Washington, DC: Pew Chartable Sr 3000 2. Fox’, Horgan J Lenhart A ta Mor ole, foing rare: 16 miton newcomers gain Internet {aries of 20002 ome, minnie ac ‘es win modest nome: contin o sug on Tne Washington, OC: Pew Ilene & Amaian ile Project, Feary 2001, 3 Calder) ot te nemet change ou heath Sst? Heath a7 (ihc) 200018 188-156 a beri PK, verikaD) Stererg DR. Ber Stein Evaluating the source ana content of tho. Pc ifermation on he nee: he ease fetal lane! synarome,J one Joint Surg Any. 2000/3: 1540-198 5. Berra 5, Collady Gl, rene MLV Baker {valuation of cance infomation on he Inteme. (Cancer 200 86581-290, 6 DavionK. The gully of etary information on ‘heWar Wide We. Ci Petr Qua Heath Cre sores 2 Gan KN, Chrtensen H, Cuaity of eb based Infomation on trestmant of depression coe sc ona suve. 84 2000 3211811-1515, A inpcastreP, Pando, cselaN Bort M Re Tit of heath ner re pusicon he Word ie Web systematic suey of adie on managing {everinchlten st ome Bi 1997314 875 107 5 Tang Quay etaluaon of ether nr ‘ation the Word Wide Web. Am] Orthos Den {otacal Orthop, 2000,11849 40. clung Hi, Munay HO, Hellinger LA Thee {eral s acute fr caret ptt normaion Pe fiat 1998;1017-8 {Hi Soot LC, Moneta Gl, Edwards JM. Vases Gey and the internet: 2 poor source of pallet ented formation / Vase Surg 1998;3088-51 ‘2: Heat onthe Net foundation HON code on con uct (HONcode) for medial and eth web tes. ‘Aalable a hp s/wmehon chHONCode!. Ac (eta ane 13, 2000. 53. sentach G Vane G, Lampe K, Cos, ley D. Quality Management. Cerfieation and Rating of Heath Information on the Net with MegcERTAR using a medPIC/RDE IML metals Sucre ar mplemertigeHeath ets and cating {ns bly J Med Ite! Res 2000.23/spp1 2) Algo avaiable at: p/w ior 203 ‘ip fdex tm Acessed Novebet 182000 1 one | Guard Peeler FM, Renn Rip pea H Crea for sing the quality ean Formato onthe nteret May 4, 1999. vata at hited trek org/aesrtera Mme (sie August 11, 200, 45. Rippent RSA Eheath code of eis. ed Intra es. 2000229 Aso aval a ta ‘i org/2000/268. cessed le 13, 200, 46 Tip ortath Conumers Finding Cty Heth Information onthe Intent astngton, DC: ne rneteathesre Coston 2000, Aso aval tp Uw neshcoation of. Accessed December 13, ono. (©2001 American Medical Association, All rights reserved.47, Winker MA anagin A Ci-Lum Beta. Guide Ine formedeai na heat ifomation ste on the Intemet. AMA, 2000283-1600-1606, Also Wa fe at tp fama mason oglu v283n2 ‘uljco0034 tre Acesed Apel 23,2001 4a, Heath intemet etc ethea! panopes foro {eingnteratheath ence to cnsurers May 200, ‘alaleat Hip wu ets co /panpesindex ‘Bop, Accessed Ap 28, 2001 4B. fam P eng TR, Decting MU, Maxfld A. Pub- Ise eteta er evaluating heath eateg Web tes: even. AM, 1999-31847. 30 Baur, Gearing Proposed framewerk ton Dove he gusty of heath Web ses revew. Med en ed Septet 26, 2000 £35 21 Edger. Dssernating heath information in Gevlopng cours: ero of te Inteet sovod757-800, 2a Hilton Fike hunger, the this”: patent, ‘Burnl, andthe inet ance 1996 352upel Sosuas, 23. US Census Breau. Poputon by race and Hi Panic or Latino ng fr the Ute Stes: 1990 and S000 Ae 22001 Avs anit: vy cess “Zov/populatin/cent000/phe-"1/ab04 pt. Ae- tse Ap 30,2001, 24. Lopez DE Language and assimilation. in: Waldinger, Bzorgmete Mes fic Los Ange Tes New ork HY: Russel Sage Foundation 1995: as 25, Gneerg€ Accesso heath cave for Hpac. Thin 99 sense a 26, Maybery RM. Mil Of. Racl and ethic ‘irences maces to mee cte. Med Care Ree ev 2000;57Gupp 1108-15. 27 json D8, Fontaine KR, Manson , Steves ‘anal 8 Annual death tute tocestyn the United Sates JAMA, 1999 289°1930-198, 28 Mokdad AH, SerdusinK Det Ws, Sownan BA ‘Mark, Kopan J. The spied of th obey ep derncinine United tates, 1991-1998 JAMA 1999; dancin 522. 28. Maine Div, Homa DM, Petowst CA, ea Strvlance for asthma United States, 19601995. (©2001 American Medical Association, All rights reserved, ‘Downloaded From: https:/jamanetwork.com/ on 07/27/2021 HEALTH INFORMATION ON THE INTERNET san a cary 8.38 se ten Bret ies eran evict rise Ne Be ite eseebeiee ee et poses Bo ta dp Ernie ds! Niece cele rome Seeker ee EE ream hina Pei be tn (pi Seo Fe ei ane cn 2S Seer kets ee ae nk cette a a Bleek eiiesy (ities ane eel $ave mnt Pr dere th un a ce Beira am aca Brel It noma Sip aut Garret feraorce hia Bee cetera, Seater ESN, watt nh Ant espe aces fox ngtand: Cochrane brary, Update Software; 18bieue A. Aoercan Paci Association Diagnosicand ‘Stebel anual of ent! Disorder, Fourth a tion Washington DC: American Psychic Asoc. tion: 198 12, cia Met, nc. ntat rags Jane 2000. ‘Nalablst tp 7us medametc ca /hame pc (esse September 5, 2000 {3 Iteret com Cop. Search Engine Watch Search tngin aliances cha. Avatabe at ht) ww seathengaewatch.com, Accessed une 10,000. internet com Car, Search Engine Watch, How Search engines rank Web pages. Avalable at Hp ‘hor Searcengineaeh com. ACcesse ne 10, ono 45. Cyber Dialogue The rumberofheath Web tes: Plenty of optons, ately few cokes. Cyber es Heath rend tepert No. 17.2000. Avalable at Fit fi eyberdaogue com. Accessed Deer er 10,2000, 46, PC Data. Top twenty helhcare ses fr Sep ‘erber 2000, Who'on ie? WebMD moves upin Farr, Sepiember 2000, Avalble stp peta com. Accessed October 12 2000. £7 Caveta|ASsesing agreement on dessfcaton {tic te kappa tate Cemputatonal Lingus, 199622249.058. 8. FE The eadabilty raph validated at primary ltl The Reasing Teacher 1969,22538-56, 4, Fry Frys readabuty graph darifeations, va- Idi, and extension to level 17! Reading. 197725 panos 50. Gillam 8, Pena S, Moun L. The Fy aaph ap eto Spans eadaily The Reazng Teacher Ja fry t380e26-430, 5, Kinch hngetut Akins Kattad A A raacyin Amara. Washington, BC: Natonal Ce ter fo Education satis 1983 52. Kincaid, Pshbume RP Rogers RL, Cisom Devivatian of New Readebilty Formulas (Auto mated Readings Fog Count and Flesch ead Ing Ease Formula foray Enid Personne. Me is, Tea: Naval Stato, 1975. (Reprinted) JANA, ay 280, 2001 285, No, 20 2624‘Online Table ‘Condition-Related Topics, Consumer Questions, and Elements “Condltion-Related Topics “Consumer Questions Elements of Condition Related Topics Assessment of breast cancer Te ard une of amoxaen forrisereduchon| Breast Cancer ‘re there ary medications Ican ake to oduce mi rekof geting treat cancer? ik faces for breast cancer ince a fami or personalise of boast ‘cancer, ear menarche, prograncyhstory, anda story of react bBopses bathe art un foaning <6 yar) tamosten reduces breast cancer ok inhghvisk wore ‘Soaring for reat cancor THe one hry Tay fas ha breast Cancer Got sti need breast fa ard emmograre? Aner shoud Tat Revng regular mammograms? Bo raed one every yar? Tleman >50 years should have mammograms aay TO ary dotacton of reat cancer mproves cutcomee ‘Moet breast cancers occur in omen wiht amy istry of the ‘seoase There fock of conceneus abou the need or appropri eral of Tammograpty in woman fom age #040 are Eanaton ofa pasos ‘reas mass Thavea ump my boast What ‘should be done to check ti? ‘Naw breast lumps shoud be brought othe attention oa pysioan Mammography an ulrasoncerapy ere useuin valu ums ‘Anogatve mammogram recut doesnot elmnate the need fo thr ‘eruaton ‘Apatastant oneysti anf ld breast mass palpated by a Bryson shou be bled Tainan nekaing pian ‘reament and ayo) flea tat for ttestment of advanced Tihave sage or Woeast cance ‘ohich sete eaten ‘mastectomy or mipecomy pus radiation? Wier can get Information spout beast cancer ‘eal a? Tsteciomy and impeciomy pus Tadion ae SquNala Weaimars or eaiy-slage reat cancer Patent preferences shoud be considered fn treatment decions of ‘asiectomy vs umpactomy pls radon ‘east econstction = avaiso for omen whe have mastectomy Creal a re svalbl for women ih actanced cancer: some infomation about fining cial als gion amas oaanaard ‘medics and surgeal treatment or bear Wat alate araios Buch as facupancur, het, of FRomecpathy) can ho me fight brent cancer? “Rtas rapes to Weat boast cancer have ganeraly nol bast subjected to igorous scenic susie ‘Atomave rerapes should na be used as a subst for proven ‘Siecle testers Your physician shoud be informed of ry alleratve ratments you are ‘ing. neu hare, supplemenis, and over-the-counter products ‘Symptoms of ilchood thm Chichood Asthma What ara the carmen symptom of ‘asta nchidren? ‘Reh with astra can experience the ellowing symptoms: cough ‘ahaezing, chest tightness, shornees of breath Ste breathing, ‘ran “asthma aac [proneunced or prelongad presancs of hess ‘smmptoms) These sjmploms can be worse at righ, biggred by excise, ‘rvrorrretal late, changes h wer rl ness, can occur Spontancoual atest Chien th asthma can have rtrmitent symptoms (wice a week or esl or persistent symptoms (mors than ios a wee Chicen ith intertent Symptoms may havea severe exacerbation ‘Srnpione agpeeieot Ucontold chidhood stm TThave been (6d bya physica rat ‘my chid has asthma, Hehe nae Cttcuty breathing tight and gos he inalr every dy. Does this mean that my oh astra lerot wal contaled? ‘Chiron ith ntertent symptoms ay eympIOMS twee a Wook o Bae lr right amptoms tice amanth ofa) are considered “convll™ Chin vith parson symptoms (ay symptoms need to use a eecus madicaton mare than ties a week or akg wp wth Symptoms durng the right more than vice @ mony ae "not cine’ Therapie modal and Tisocitadaovore ‘ects ‘What should do about my ons asta, eopectaly ft not Wo ‘contol? ve thre ary Medicines or speci equpmont {hata physisan ean peserbe? 1th have any acerse| ects! “Bonchaalaior mediation fig, abuso] open The aways Breanna, asap) thay are used as cuck rel or “resca" medabons for patents whose symptoms arentermitlent beled cortcostrode og bacomsthasone,furisoide,tiancinlone) and ‘crmalm are 2 kreds of iled mecicatons tal redce nfammation Inte arwaye: thoy reused as long form esments for pars \nhose symptoms ae persistent or uncontaled | spacer device limprove dalvery of haled medications tothe lungs ‘uch devices ae requted for young chisren and are strongly ‘recommended ler ola chicen and adolescents Peak fon martin i a Ustd way for palants to recognize ery signs ‘ol worsening athe Oral sterode a tlc or shor-tor exacerbations but have sgnfcant ‘adver fects ovr tha long term bled streids,taean usu oes, do ntact chickens growth; unconiald sthma can retard a chs growth ‘Ateraive rapes for athna (eg ata ramadles and chopra ‘maniulatr) have net boon show Tobe facve ‘niledkotiones 26 ara drug cass that might be Useful as an ack-on to Inhaled sede oro proven exercise symptom chien ade than (years: tho sally and eteacy ofthese drogen chisten younger than 6 ears has not been demonstated ‘Downloaded From: https:/jamanetwork.com/ on 07/27/2021 (© 2001 Amarican Medal Asociton, Al hs reserved