Journal of Public Health DOI: 10.


Vol. 26, No. 1, pp. 101–104 Printed in Great Britain

Cochrane update Evidence-based public health: the importance of finding ‘difficult to locate’ public health and health promotion intervention studies for systematic reviews
Faline Howes, Jodie Doyle, Nicki Jackson and Elizabeth Waters

Systematic reviews are used to assess and summarize research, and therefore are instrumental to decision-making in public health and health promotion policy and practice. The validity of the results of a systematic review is highly dependent on the data collection methods used.1 This includes a systematic search to locate all relevant studies, which is often a difficult and challenging task. A great deal of public health and health promotion research has been conducted, but only some of it is available in the public domain, and overall the literature is widely dispersed.2 This paper aims to highlight some of the issues surrounding publication bias, including database bias, language bias and grey literature bias, and to illustrate these issues with public health examples where available. Publication bias is defined here as the publication or non-publication of research findings based on the nature and direction of results.3–6 Methodology to prevent or minimize publication bias is described, including an outline of the role of the Cochrane Health Promotion and Public Health Field (the HP&PH Field) in improving access to ‘difficult to locate’ health promotion and public health intervention studies and effectiveness reviews.

avoidable suffering of individuals (whether healthy populations or patients4,8) and a waste of limited resources.4 Within health promotion, concurrent reviews have often overlapped in the questions they address and the primary studies they examine.9 Non-publication of trial results has been described as scientific misconduct,5 unethical8 and as breaking the contract that investigators make with trial participants, funding agencies and ethics boards.10

Downloaded from at University of Portland on May 24, 2011

Types of publication bias
Poor access to, and therefore the non-inclusion of, studies or journals that are not indexed in databases, studies in nonEnglish language publications and those from the unpublished (grey) literature can lead to bias being introduced into systematic reviews. This will occur if the results in the ‘difficult to locate’ sources are systematically different from those found in the easily accessible sources.4,11

Database (indexing) bias
Studies that are published in non-indexed journals are to some extent hidden from reviewers.6 This is of particular importance for the dissemination of findings from research in less developed

What are the consequences of publication bias in public health and health promotion reviews?
Reviews have been described as an efficient way of becoming familiar with state-of-the-art research and practice in specific topics in public health.7 However, publication bias imposes limitations on the conclusions that can be drawn from reviews if the search strategy employed fails to identify an unbiased sample of relevant studies. This may lead to inappropriate or inadequate decisions being made regarding policy, practice and future research. The consequences of such decisions include the

1 Cochrane Collaboration Health Promotion and Public Health Field, Carlton South, Vic. 3053, Australia. 2 Centre for Community Child Health, University of Melbourne, Royal Children’s Hospital, Murdoch Children’s Research Institute, Parkville, Vic. 3052, Australia.

Faline Howes,1 Research officer Jodie Doyle,1 Field Coordinator Nicki Jackson,1 Training and Support officer Elizabeth Waters,1,2 Associate Professor Address correspondence to Associate Professor E. Waters. E-mail:

Journal of Public Health 26(1) © Faculty of Public Health 2004; all rights reserved.

Downloaded from jpubhealth.1. as approximately only 2 per cent of journals originating in these countries (e. unpublished trials are particularly prevalent in oncology. rheumatology and orthopaedics reviews.6 The quality of primary studies and reviews in public health and health promotion has already been identified as an issue.4 These can be broadly defined as retrospective registers (see below).18 Unpublished trials often have not undergone peer review. However. 35 (25 per cent) adopted a narrative synthesis. summarized their results narratively.4 Assistance in this area exists in the form of trial or studies registers.controlled-trials.htm). www.19 made efforts to control bias at entry. It is encouraging that journals. regardless of outcome. and trials in complementary medicine are frequently difficult to locate. at least in part.15. In a review of reviews of health promotion in schools.1 As relevant data may be incomplete or missing. their methodological quality can be difficult to Thus reviewers are likely to have difficulty in accessing many more journals and intervention studies.oxfordjournals. the number of reviews using this technique is even higher. as a large proportion of reviews lend themselves better to a narrative technique. the quality of the reviews was found to vary.102 JOURNAL OF PUBLIC HEALTH countries.20 As the validity of the results of systematic reviews is also dependent on the methodological quality of the component studies. For example. The Core Public Health Journals list consists of 560 journals13 (86 of which do not appear on the Lamar Soutter Library list). including the Other issues to consider Publication bias in narrative systematic reviews The problem of publication and publication-related biases has been examined more often in reviews containing a meta- . reviewers need to make a concerted effort to locate all relevant studies and update their reviews when new studies become available.12 Thirty-eight per cent (n ϭ 235) of these are not indexed in MEDLINE. and thus inclusion decisions are problematic.6 thorough quality assessments must be undertaken by reviewers. and the overall quality of studies did not improve over time. 2011 Grey literature bias Grey literature is generally assumed to include literature that has not been published. has limited distribution or is not available through conventional channels. There is a lack of methods that can be used to detect such biases in narrative systematic reviews. it is unlikely that they cover the full complement of non-English language or developing country public health and health promotion journals. Ninety-one per cent (361/398) of the health promotion effectiveness reviews identified by Peersman et al. it is believed that because of the way the lists were compiled.17 The finding that trials that are difficult to locate are often of lower quality raises the possibility that rather than preventing bias through extensive searching.14 Whether this also applies to health promotion and public health is unquantified and is likely to be topic dependent.16 and therefore whether their inclusion or exclusion will (statistically) significantly affect the results of meta-analyses. and prospective registers (such as Current Controlled Trials. randomized controlled trials in cancer research. reviewers and editors can all play a role in reducing publication bias. For example. Language bias English is the predominant language in contemporary medical research.10 Methods available to prevent or minimize publication bias Investigators.g. The Lamar Soutter Library list compiled by the University of Massachusetts Medical School consists of 614 journals.9 Methodological quality of ‘difficult to locate’ studies Debate continues in the literature regarding the methodological quality of trials published in non-English language activities/reviews. analysis. Some of the non-indexed journals may not include intervention studies or reviews. explain this. an examination of the fate of doctoral theses from the School of Hygiene and Public Health. it may also reflect reduced accessibility to non-English language studies.17 For example. India) are indexed in MEDLINE. Of the 142 health promotion and public healthrelated Cochrane reviews posted on the HP&PH Field’s website (at September 2003.6 What proportion of health promotion and public health journals are indexed in MEDLINE? Consider the following two lists of public health journals.vichealth. trials published in languages other than English and trials not indexed in MEDLINE are important to locate and consider for inclusion in psychiatry.20 Only one-third of the public health-related randomized controlled trials identified by Fahey et al.3 The importance of unpublished trials also appears to vary across medical specialties. editors could make a concerted effort to include studies based on methodological criteria only4 and to increase the breadth of study origin. Outside of Cochrane reviews.4 This is of particular relevance to public health and health promotion. Investigators need to be encouraged to publish their studies. bias could be introduced by including trials of low methodological quality.6 A recent publication concluded that bias against publishing non-significant results is an issue even for large (sample size over 200). after entry and at outcome assessment. at University of Portland on May 24. with less than half the reviews gaining a quality score of 10 or more out of 16. found that onethird of graduates had not published from their thesis. Johns Hopkins University.15 Although factors such as the logistical and cost barriers of translation16 may. phase III.6 Gregoire et al. Forty-four per cent (n ϭ 246) of these are not indexed in MEDLINE. for Equally. (1995) determined that 26 out of 36 (72 per cent) meta-analyses restricted their searches to English only.

mouthrinses. Health care financing systems for increasing utilization of tobacco dependence treatment. Competitions and incentives for smoking cessation. Ultimately. the identification of these ‘difficult to locate’ studies will depend on a collaborative effort across the breadth of health promotion and public healthrelated disciplines. http://www. Although many Cochrane Review Groups undertake handsearching to varying degrees. It is then that the impact of including or excluding these studies in systematic reviews can be evaluated. in part. retrospective trial registries compiled by Cochrane Review Groups. Helmets for preventing injury in motorcycle riders. controlled clinical trials (quasi-randomized controlled trials) and effectiveness reviews in journals that are not indexed in electronic databases including seminal non-English language journals. This is generated from the comprehensive searches the EPPI Centre conducts whilst undertaking its own systematic reviews. and readily accessible journals may be of lower quality than studies that are easier to access. The trials identified by the Cochrane Collaboration’s handsearching activities form part of an international register of controlled trials: the Cochrane Central Register of Controlled Trials (CENTRAL).21 The role of the HP&PH Field in assisting reviewers to locate relevant studies Identifying sources of unpublished studies can be time consuming and involve considerable cost and effort.22 The generation and maintenance of these registries is achieved. Currently this register (known as TRoPHI. Smoking cessation for the secondary prevention of coronary heart disease. by the handsearching of published and unpublished sources to ensure the identification of as many relevant studies as possible. language or publication status. potentially there is a myriad of studies relevant to public health and health promotion that are being missed. Fluoride mouthrinses for preventing dental caries in children and home. Impact of tobacco advertising and promotion on increasing adolescent smoking behaviours. The HP&PH Field believes that increasing access to all relevant studies and reviews will not only assist researchers interested in undertaking a systematic review but will also add to the information base available to decision-makers and assist in the prevention of duplication of effort. Family and carer smoking control programmes for reducing children’s exposure to environmental tobacco smoke. Lactational amenorrhea for family planning. gels or varnishes) for preventing dental caries in children and adolescents.ioe. Downloaded from jpubhealth.eppi.EVIDENCE-BASED PUBLIC HEALTH 103 Journal of Public Health. readers and reviewers must take due care to assess the quality of the study or review. are taking steps to improve access to studies from developing countries. The Cochrane Health Promotion and Public Health Field wishes to expand its activities to identify studies to be included in the HP&PH Field’s specialized register. Topical fluoride (toothpastes. The HP&PH Field is developing a strategy to assist reviewers in identifying all relevant public health and health promotion-related intervention studies.oxfordjournals. A handsearching strategy is currently being prepared to undertake these activities. Fields and Networks. and conference at University of Portland on May 24.aspx) consists of the studies collected and coded by the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI Centre) in the United Kingdom. Interventions for improving communication with children and adolescents about a family member’s cancer. New reviews ● ● ● ● ● ● ● ● ● ● ● Post-license driver education for the prevention of road traffic crashes. 2011 Conclusion Further research is required to determine the quantity and quality of public health and health promotion intervention studies that exist in non-indexed English and non-English language journals and in the grey literature. The HP&PH Field is particularly interested in locating randomized controlled trials. Screening programmes for the early detection and prevention of oral cancer. Below are new health promotion and public health oriented reviews and protocols from Issues 3 and 4. 2003. Over 1700 journals have been or are being handsearched by individuals within the Cochrane Collaboration. Interventions for promoting use of booster seats for children aged 4–8 travelling in cars. Accessibility to all relevant studies. As studies identified outside of widely used databases New protocols ● ● ● ● ● ● ● ● Community-based interventions for the prevention of burns and scalds in children.17 Cochrane reviewers are assisted in this process by the development of topic-specific. 2003 of The Cochrane Library was released in October. Modification of the home environment for the reduction of injuries. Pharmaceutical policies: effects on rational drug use. Heroin maintenance for chronic heroin dependants. . is important in improving the quality and relevance of systematic reviews. Interventions for smokeless tobacco use cessation. Update of Cochrane protocols and reviews of relevance Issue 4. Printed educational materials: effects on professional practice and health care outcomes. which will inevitably include some journals covering health promotion and public health intervention regardless of results.

Clarke M. Health promoting schools and health promotion in schools: two systematic reviews.library. Grey literature in meta-analysis of randomised trials of health care interventions (Cochrane methodology review). Thorogood M. 267: 374–378. Meinert C. 2003. Issue 2. 21 Vetter N. 2003.oxfordjournals. Tugwell P. University of Massachusetts Medical School. McDonald S. Gillespie at University of Portland on May 24. 2001: 34. Learning from research. Vaccines for measles. Lefebvre C. Scherer R. mumps and rubella in (26 August 2003. Hyde C. 4 Song F. Eastwood AJ. Oliver S. Ba’Pham. Lancet 1996. Evidence-based public health. 53: 964–972. Handsearching versus electronic searching to identify reports of randomised trials (Cochrane methodology review). Langenberg P. 22 Hopewell S. Sutton AJ. Hlth Technol Assess 1999. Issue 2. Oakley A. 5 Scherer RW. 350: 326–329. Leet T. In: The Cochrane Library. 7 Brownson R. LeLorier J. Fundam Clin Pharmacol 1995.] http://www. Research publication in developing countries (editorial). Klassen T. 2003. Milne R.yale. Oxford: Update software. Oxford: Update software. Jadad A. et al. et al. 356: 1228. et al. Pintilie M. 14 Egger M.104 ● ● ● ● ● ● ● ● ● ● ● ● ● ● JOURNAL OF PUBLIC HEALTH School-based education programmes for the prevention of child sexual abuse. Hlth Technol Assess 2000. Institute of Education. 12 The Lamar Soutter Library. Using research for effective health promotion.umassmed. The ‘WHO safe communities’ model for the prevention of injuries in whole populations. and the Australian Commonwealth Government. In: Oliver S. 2003: 46. Sterne J. Juni P. [On-line. Factors associated with failure to publish large randomised trials presented at an oncology meeting. Home visits during pregnancy and postpartum for women with an alcohol and/or drug problem. Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. eds. Interventions for the prevention of relationship and dating violence in adolescents and young people. [On-line. 3(22). Oxford: Oxford University Press. Br Med J 1994. Publ Hlth Med 2003. Moher D. Zellwegger-Zahner T. Baker E. In: The Cochrane Library. Acknowledgements The Cochrane Health Promotion and Public Health Field base in London: EPPI Centre. Tannock I. Australia acknowledges the generous support of the Victorian Health Promotion Foundation (VicHealth). Population-based interventions for the prevention of fallrelated injuries in older people. Egger M. Low glycaemic index diets for coronary heart disease. Accepted on 31 October 2003 . 9 Peersman G. 309: 1286–1291. Influence of comprehensive versus partial information on consumers’ screening choices. 25: 189. University of London. Full publication of results initially presented in abstracts (Cochrane methodology review). 347: 363–367. JAMA 2003. Sowdon A. 16 Moher D. Bartlett C. The type and quality of randomized controlled trials (RCTs) published in UK public health journals. 13 Core Public Health Journal Project (ver 0. Department of Health and Ageing. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Lancet 1997. SSRI. Family-based programmes for preventing smoking by children and adolescents. Harden A. Domestic violence screening and intervention programmes for adults with dental or facial injury. date last accessed). 2003. Language bias in randomised controlled trials published in English and German. 18 Moher D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analysis? Lancet 2000. Effectiveness reviews in health promotion. In: The Cochrane Library. Oakley A. Registering clinical trials. Stage-based interventions for smoking cessation. 290: 495–501. 2003. Stewart-Brown S. 48: 159–163. 6 Egger M. Oxford: Update software. Ba’ Pham. Factors influencing publication of research results. Peersman G. Information and support interventions for carers of people with dementia. Derderian F. Fortin P.] http://www. 17 Hopewell S. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias? J Clin Epidemiol 1995. Systematic reviews: identifying relevant studies for systematic reviews. Schneider M. 7(1). Influence of comprehensive versus partial information on consumers’ screening choices. Gilbody S. 2011 15 Gregoire G. Min Y. 10 Krzyzanowska M. Downloaded from jpubhealth. 2 Peersman G. Rennie D. 20 Lister-Sharpe D. Issue 2. Scherer R. Oxford: Oxford University (4 July 2003. Holenstein F. 2003. 19 Fahey T. Clarke M. Hlth Technol Assess 2003. 290: 516–522. 9: 384–386. 4(10). Publication and related biases. JAMA 1992. What contributions do languages other than English make on the results of meta-analyses? J Clin Epidemiol 2000. 8 Dickersin K. References 1 Dickersin K. date last accessed). JAMA 2003. Duley L. 1999. 11 McAuley L. Lefebvre C. Chapman S. 3 Dickersin K.9). Nutritional interventions for reducing morbidity and mortality in HIV infected individuals.