Abstract
Introduction
The Internet provides great opportunities for patient healthcare education, but poses risks that inaccurate, outdated, or harmful information will be disseminated. Osteoporosis is a topic of great interest to patients, many of whom use the Internet to obtain medical information. The aim of this study was to develop and evaluate measurement tools to determine the quality of osteoporosis websites for patients.
Methods
Quality indicators in the categories of content, credibility, navigability, currency, and readability were incorporated into separate evaluation tools for healthcare professionals and for patients. Websites were selected from an Internet search. Interobserver reliability and validity were assessed, and a sample of osteoporosis websites was evaluated by an osteoporosis nurse educator and compared to patient evaluations.
Results
For the quality indicators, there was 79% agreement between the osteoporosis nurse educators, 88% agreement between the physician osteoporosis experts, and 71% agreement comparing the osteoporosis nurse educators to the physician osteoporosis experts. Quality scores for evaluated websites ranged from 18–96 (maximum possible=100), with a mean of 66. Websites with Uniform Resource Locator (URL) suffix .com scored significantly lower compared to those with .gov (P<0.05), .edu (P<0.01), and .org (P<0.01). Healthcare professionals and patients were in agreement on the quality of the highest-rated websites, with less agreement for lower-rated websites.
Conclusions
In summary, a tool for measuring the quality of medical websites was developed and evaluated. Significant variability in osteoporosis-website quality was observed. Higher-quality scores were associated with a higher level of search engine match and specific URL suffixes. A validated tool for evaluating medical websites may have value in assisting patients to select high-quality osteoporosis educational information on the Internet, and may encourage website developers to improve the quality of information that is provided.
Similar content being viewed by others
References
Ryan M, Smith KM (2003) Webographies: a patient’s guide to health information on the Internet. Am J Health Syst Pharm 60:876–877, 881
Krane D Healthcare News (2005) Number of “cyberchondriacs”- US adults who go online for health information- increases to estimated 117 million. http://www.harrisinteractive.com/news/newsletters/healthnews/HI_HealthCareNews2005Vol5_Iss08.pdf. Cited 21 Aug 2005
Eysenbach G, Powell J, Kuss O, Sa ER (2002) Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 287:2691–2700
Jain T, Barbieri RL (2005) Website quality assessment: mistaking apples for oranges. Fertil Steril 83:545–547
Huang JY, Discepola F, Al-Fozan H, Tulandi T (2005) Quality of fertility clinic websites. Fertil Steril 83:538–544
RAND Health (2001) Proceed with caution: a report on the quality of health information on the Internet. http://www.chcf.org/documents/consumer/ProceedWithCautionCompleteStudy.pdf. Cited 7 Sept 2005
Benigeri M, Pluye P (2003) Shortcomings of health information on the Internet. Health Promot Int 18:381–386
Walji M, Sagaram S, Sagaram D, Meric-Bernstam F, Johnson C, Mirza NQ, Bernstam EV (2004) Efficacy of quality criteria to identify potentially harmful information: a cross-sectional survey of complementary and alternative medicine web sites. J Med Internet Res 6:e21
Matthews SC, Camacho A, Mills PJ, Dimsdale JE (2003) The internet for medical information about cancer: help or hindrance? Psychosomatics 44:100–103
Green CJ, Kazanjian A, Helmer D (2004) Informing, advising, or persuading? An assessment of bone mineral density testing information from consumer health websites. Int J Technol Assess Health Care 20:156–166
URAC (2005) Health web site check-up service and accreditation program. http://www.urac.org/prog_accred_HWS_po.asp?navid=accreditation&pagename=prog_accred_HWS. Cited 7 Sept 2005
Internet Healthcare Coalition (2005) eHealth Code of Ethics. http://www.ihealthcoalition.org/ethics/ethics.html. Cited 21 Aug 2005
Winker M, Flanagin A, Chi-Lum B, White J, Andrews K, Kennett R, DeAngelis C, Musacchio R (2005) Guidelines for medical and health information sites on the Internet. http://www.ama-assn.org/ama/pub/category/1905.html. Cited 27 Aug 2005
Health On the Net Foundation (2005) http://www.hon.ch/. Cited 21 Aug 2005
Risk A, Dzenowagis J (2001) Review of internet health information quality initiatives. J Med Internet Res 3:E28
Gagliardi A, Jadad AR (2002) Examination of instruments used to rate quality of health information on the internet: chronicle of a voyage with an unclear destination. BMJ 324:569–573
Jadad AR, Gagliardi A (1998) Rating health information on the Internet: navigating to knowledge or to Babel? JAMA 279:611–614
Sullivan D (2005) Nielsen NetRatings search engine ratings. http://searchenginewatch.com/reports/article.php/2156451.Cited 21 Aug 2005
National Osteoporosis Foundation (2002) America’s bone health: the state of osteoporosis and low bone mass in our nation. National Osteoporosis Foundation, Washington, DC
Greenberg L, D’Andrea G, Lorence D (2003) Setting the public agenda for online health search, a white paper and action agenda. http://www.urac.org/documents/URAC-CWWHealthSearchWhitePaper1203.pdf. Cited 7 Sept 2005
Ambre J, Guard R, Perveiler F, Renner J, Rippen H (1997) Criteria for assessing the quality of health information on the Internet. http://hitiweb.mitretek.org/docs/criteria.pdf. Cited 7 Sept 2005
Ansani NT, Vogt M, Henderson BA, McKaveney TP, Weber RJ, Smith RB, Burda M, Kwoh CK, Osial TA, Starz T (2005) Quality of arthritis information on the Internet. Am J Health Syst Pharm 62:1184–1189
Howitt A, Clement S, de LS, Thiru K, Goodwin D, Wells S (2002) An evaluation of general practice websites in the UK. Fam Pract 19:547–556
Flesch RF (1951) How to test readability. Harper, New York
Kincaid J, Fishburne R, Rogers R, Chissom B (1975) Derivation of new readability formulas: automated reliability index, fog count, and Flesch reading ease formula for Navy enlisted personnel (branch report 8-75). US Navy, Millington, TN
Doak CC, Doak LG, Root JH (1996) Teaching patients with low literacy skills. J.B. Lippincott, Philadelphia
D’Alessandro DM, Kingsley P, Johnson-West J (2001) The readability of pediatric patient education materials on the World Wide Web. Arch Pediatr Adolesc Med 155:807–812
Williams DM, Counselman FL, Caggiano CD (1996) Emergency department discharge instructions and patient literacy: a problem of disparity. Am J Emerg Med 14:19–22
Lee PP (1999) Why literacy matters. Links between reading ability and health. Arch Ophthalmol 117:100–103
Kenkel PJ (2002) Boon for consumers? Hospital quality regulator and big business health purchasing coop join forces. Manag Care Q 10:51
O’Leary DS (1987) Leaders speak out on the quality of healthcare delivery. Interview by Richard L. Clarke. Healthc Financ Manage 41:38–40, 44
Watts NB (2002) Bone quality: getting closer to a definition. J Bone Miner Res 17(7):1148–1150
Bernstam EV, Shelton DM, Walji M, Meric-Bernstam F (2005) Instruments to assess the quality of health information on the World Wide Web: what can our patients actually use? Int J Med Inform 74:13–19
Fricke M, Fallis D, Jones M, Luszko GM (2005) Consumer health information on the Internet about carpal tunnel syndrome: indicators of accuracy. Am J Med 118:168–174
Fraquelli M, Conte D, Camma C, Casazza G, Di BD, Rebulla P, Prati D (2004) Quality-related variables at hepatological websites. Dig Liver Dis 36:533–538
Butler L, Foster NE (2003) Back pain online: a cross-sectional survey of the quality of web-based information on low back pain. Spine 28:395–401
Author information
Authors and Affiliations
Corresponding author
Additional information
Prior abstract presentation: Preliminary data for this study were presented as a plenary poster at the 2005 ISCD Annual Meeting in New Orleans, LA, USA, and published as an abstract in Lewiecki EM, Rudolph LA, Kiebzak GM (2005) Evaluation of osteoporosis website quality. J Clin Densitom 8:243
Appendices
Appendix 1
Observer Initials
Website
Instructions: The main page to be evaluated is the “linked” page (LP) generated from the Google search. If LP is not the primary education page (PEP), then PEP must be evaluated as well. In addition to viewing LP and/or PEP, browse home page and other web pages enough to determine whether there are any obvious problems or bias. Each item identifies whether evaluation is for LP, PEP, or entire website (WS). Record score of “0” or “1” in every highlighted box.
Healthcare Website Assessment Tool (HWAT) 3.0 | ||||
Present or absent | Weighting (9,6,4) | Score | ||
(0 or 1) | ||||
Content | ||||
WS | 1. Originator (person or organization) of website identified | 9 | ||
LP | 2. Subject of website clearly stated in title or description | 9 | ||
WS | 3. Unbiased- website clearly distinguishes sale of products/services from educational information | 9 | ||
WS | 4. Website includes definition, consequences, prevention, treatment of osteoporosis (note: linked websites are excluded from analysis) | 6 | ||
Credibility | ||||
LP/PEP | 5. Author/institution providing information clearly stated | 9 | ||
WS | 6. “Seal of Approval” (HONcode, URAC, eCode of ethics, or equivalent) | 6 | ||
LP/PEP | 7. Credible source (university, professional society, government agency, non-profit foundation) or published references are provided | 9 | ||
Navigability | ||||
WS | 8. Functioning intrawebsite links | 9 | ||
WS | 9. Functioning interwebsite links | 9 | ||
LP | 10. Fully printable (margins not cut off) | 9 | ||
WS | 11. Email link for feedback to website provider | 6 | ||
Currency | ||||
LP | 12. Revision or copyright in last 12 months (score 0 if not provided) | 6 | ||
Readability | ||||
PEP | 13. Flesch-Kincaid Grade Level (score 0 if >8, 1 if 8 or less)=________ | 4 | ||
Total Score | 100 |
Appendix 2
If you are age 50 or older and have access to the Internet, you are invited to participate in a study to express your opinions on websites providing osteoporosis educational information for patients. By logging on to ten different websites and completing five multiple choice questions for each, your results will help us to determine whether these websites provide you with helpful and accurate information. The websites may vary widely in their quality. Please be totally honest in your answers. This is not a test. There are no right or wrong answers. After you have completed all the questionnaires, return this page with all forms to:
[address]
You will earn $25 for completing questionnaires for 10 web sites. The check will be written to you or donated in your name to [name of nonprofit osteoporosis foundation], according to your preference.
Please provide the following information about yourself:
-
Sex: Female □ Male □
-
Age:
-
Highest level of education:
-
Less than high school □
-
High school □
-
Degree beyond high school □
-
-
Computer skills
-
Beginner □
-
Intermediate □
-
Advanced □
-
Indicate preference for payment of $25 by checking one of the following:
-
□ Mail check to me at this address-
-
□ Donate $25 in my name to [name of nonprofit osteoporosis foundation]
Website: [URL of website]
Circle the answer to each of the following questions for this website.
-
1.
This website offers information about osteoporosis that is helpful to me.
-
a.
Agree
-
b.
Disagree
-
c.
Not sure
-
a.
-
2.
I believe the information provided on this website is scientifically correct.
-
a.
Agree
-
b.
Disagree
-
c.
Not sure
-
a.
-
3.
It is easy to find information I need on this website.
-
a.
Agree
-
b.
Disagree
-
c.
Not sure
-
a.
-
4.
The information on this website is up-to-date.
-
a.
Agree
-
b.
Disagree
-
c.
Not sure
-
a.
-
5.
The information on this website is easy to understand.
-
a.
Agree
-
b.
Disagree
-
c.
Not sure
-
a.
Appendix 3
Google search matching number | Qualifying order | URL | Website title | Weighted score |
---|---|---|---|---|
1 | 1 | National Osteoporosis Foundation | 90 | |
3 | 2 | Osteoporosis Society of Canada | 96 | |
5 | 3 | NIH ORBD-NRC - Osteoporosis and Related Bone Diseases | 96 | |
7 | 4 | National Osteoporosis Society | 90 | |
9 | 5 | International Osteoporosis Foundation | 87 | |
12 | 6 | MedlinePlus: Osteoporosis | 96 | |
16 | 7 | FORE | 84 | |
17 | 8 | The Osteoporosis Center | 51 | |
20 | 9 | Osteoporosis Australia | 51 | |
24 | 10 | Boning Up on Osteoporosis | 81 | |
26 | 11 | Imaginis - Osteoporosis | 48 | |
27 | 12 | OBGYN.net - Osteoporosis | 66 | |
30 | 13 | Medinfo: Osteoporosis | 84 | |
31 | 14 | Osteoporosis Centre | 42 | |
32 | 15 | CHR:YOUR HEALTH | 42 | |
33 | 16 | Stop Osteoporosis!!! | 48 | |
38 | 17 | Osteoporosis | 69 | |
40 | 18 | http://dir.yahoo.com/Health/Diseases_and_Conditions/ Osteoporosis/ | Yahoo! Directory Osteoporosis | 51 |
42 | 19 | Osteoporosis/Bone Loss | 63 | |
43 | 20 | Osteoporosis Education Project | 51 | |
47 | 21 | medicalresources.org | 96 | |
50 | 22 | Osteoporosis and bone physiology | 87 | |
53 | 23 | http://orthopedics.about.com/health/orthopedics/ blosteoporosis.htm | Osteoporosis | 33 |
54 | 24 | Osteoporosis Resource Centers | 66 | |
55 | 25 | Arthritis Research Campaign: Osteoporosis | 66 | |
57 | 26 | The Physician and Sportsmedicine | 78 | |
58 | 27 | http://health.discovery.com/diseasesandcond/ encyclopedia/1666.html | Discovery Health: osteoporosis | 51 |
61 | 28 | Osteoporosis Topic Page | 96 | |
72 | 29 | Osteoporosis Treatment | 30 | |
74 | 30 | Creating Health- Osteoporosis | 81 | |
75 | 31 | Arthritis Insight- Osteoporosis Information | 72 | |
76 | 32 | http://www.arthritis.ca/types%20of%20arthritis/ osteoporosis/default.asp?s=1 | Osteoporosis | 87 |
77 | 33 | Prevention of Osteoporosis | 75 | |
80 | 34 | Osteogenesis Imperfecta Foundation: Osteoporosis | 81 | |
81 | 35 | http://www.thebestisyet.net/cgi-bin/cgiwrap/ pando19/start.cgi/viewfrom16.htm | Diagnosis: Osteoporosis | 45 |
82 | 36 | http://www.merck.com/mrkshared/mmanu75al/ section5/chapter57/57a.jsp | The Merck Manual | 84 |
86 | 37 | http://www.project-aware.org/Heal75th/Osteo/osteo-what.shtml | Osteoporosis, osteopenia, bone density | 75 |
90 | 38 | Osteoporosis | 75 | |
94 | 39 | Osteoporosis Update | 75 | |
96 | 40 | Calcium & Osteoporosis | 27 | |
97 | 41 | Osteoporosis | 72 | |
99 | 42 | http://www.healingwithnutrition.com/odisease/ osteoporosis/osteoporosis.html | Osteoporosis | 57 |
102 | 43 | Healthology Osteoporosis | 48 | |
103 | 44 | Welcome to Actonel | 51 | |
105 | 45 | MJA: Osteoporosis | 63 | |
108 | 46 | Osteoporosis- WomensHealthChannel | 96 | |
112 | 47 | Osteoporosis | 78 | |
113 | 48 | Medicare.gov- Osteoporosis Details | 51 | |
116 | 49 | Nutrition and Osteoporosis | 78 | |
120 | 50 | SCOI Presents Osteoporosis | 60 | |
121 | 51 | Osteoporosis on the Internet | 24 | |
122 | 52 | Research and recipes | 18 | |
124 | 53 | http://www.oznet.ksu.edu/humannutrition/ nutlink/pages/BONES.HTM | Osteoporosis | 81 |
126 | 54 | Osteoporosis | 69 | |
127 | 55 | Postmenopausal Osteoporosis | 18 | |
129 | 56 | http://www.spine-health.com/topics/cd/ osteoporosis/osteopor02.html | What you need to know | 75 |
130 | 57 | AAPM&R | 51 | |
132 | 58 | Exercise and Osteoporosis | 33 | |
133 | 59 | http://www.healthfinder.gov/Scripts/ SearchContext.asp?topic=614 | Healthfinder | 81 |
134 | 60 | Colorado HealthSite | 78 | |
135 | 61 | http://www.intelihealth.com/IH/ihtIH/WSIHW000/9339/9564.html | InteliHealth: osteoporosis | 90 |
137 | 62 | Osteoporosis | 60 | |
138 | 63 | Osteoporosis | 81 | |
139 | 64 | Osteoporosis | 60 | |
140 | 65 | Texas Dept. of Health | 84 | |
141 | 66 | http://www.healthatoz.com/healthatoz/Atoz/ dc/caz/bone/poro/osteoCFindex.html | Osteoporosis | 66 |
143 | 67 | Osteoporosis | 57 | |
146 | 68 | Osteoporosis Center | 84 | |
147 | 69 | Osteoporosis | 66 | |
148 | 70 | http://www.healthservices.gov.bc.ca/msp/ protoguides/gps/bone/bonepg.html | Bone Density Measurement and Osteoporosis | 57 |
149 | 71 | aceZones.com | 24 | |
153 | 72 | New Scientist | 24 | |
154 | 73 | Osteoporosis- Wikipedia | 63 | |
161 | 74 | Truth About Osteoporosis | 48 | |
163 | 75 | Menopause Online | 33 | |
174 | 76 | http://www.dundee.ac.uk/medicine/tayendoweb/ images/osteoporosis.htm | Osteoporosis | 60 |
178 | 77 | Osteoporosis Cure | 28 | |
179 | 78 | The Ridgeway Surgery: osteoporosis | 42 | |
181 | 79 | VegSource.com | 27 | |
183 | 80 | Osteoporosis | 48 | |
184 | 81 | Post-menopausal Osteoporosis | 87 | |
185 | 82 | AllRefer Health - Osteoporosis | 69 | |
197 | 83 | http://www.sutterhealth.org/ health/healthinfo/index.cfm?page=article&sgml_id=hw131419 | Health Information | 72 |
198 | 84 | MEdIC - Health Explorer - Osteoporosis | 69 | |
200 | 85 | Osteoporosis | 24 | |
201 | 86 | http://www.roche-diagnostics.com/health_kiosk/ framesets/3.html | Roche Health Kiosk | 48 |
202 | 87 | http://nursing.villanova.edu/womenwithdisabilities/ other/osteoJ.htm | Osteoporosis | 57 |
204 | 88 | Tourism of India | 24 | |
205 | 89 | HealthCentral | 48 | |
206 | 90 | Osteoporosis | 66 | |
207 | 91 | http://www.state.vt.us/health/_hs/epidemiology/ osteoporosis/osteoporosis.htm | Osteoporosis | 84 |
208 | 92 | WebHealthCentre.com | 66 | |
218 | 93 | http://www.gemedicalsystems.com/rad/bonedens/osteo/osteo.html | Bone Densitometry - Osteoporosis | 45 |
219 | 94 | http://www.ostex.com/ Osteoporosis/Osteoporosis_Related_Links.asp | Osteoporosis Related Links | 27 |
224 | 95 | Clinical Preventive Services | 72 | |
225 | 96 | The Body: SFAF BETA | 69 | |
226 | 97 | Osteoporosis | 81 | |
227 | 98 | http://www.mhcs.health.nsw.gov.au/health-public-affairs/ mhcs/publications/4925.html | How strong are your bones? | 90 |
228 | 99 | Osteoporosis | 58 | |
229 | 100 | Osteoporosis | 69 |
Rights and permissions
About this article
Cite this article
Lewiecki, E.M., Rudolph, L.A., Kiebzak, G.M. et al. Assessment of osteoporosis-website quality. Osteoporos Int 17, 741–752 (2006). https://doi.org/10.1007/s00198-005-0042-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00198-005-0042-5