You are on page 1of 15

Review article

Health policy-makers’ perceptions of their use
of evidence: a systematic review
Simon Innv×r, Gunn Vist, Mari Trommald, Andrew Oxman
Health Services Research Unit, National Institute of Public Health, Oslo, Norway

Objectives: The empirical basis for theories and common wisdom regarding how to improve appropriate use of
research evidence in policy decisions is unclear. One source of empirical evidence is interview studies with policymakers. The aim of this systematic review was to summarise the evidence from interview studies of facilitators of,
and barriers to, the use of research evidence by health policy-makers.
Methods: We searched multiple databases, including Medline, Embase, SocioŽ le, PsychLit, PAIS, IBSS, IPSA and
HealthStar in June 2000, hand-searched key journals and personally contacted investigators. We included interview
studies with health policy-makers that covered their perceptions of the use of research evidence in health policy
decisions at a national, regional or organisational level. Two reviewers independently assessed the relevance of
retrieved articles, described the methods of included studies and extracted data that were summarised in tables and
analysed qualitatively.
Results: We identiŽ ed 24 studies that met our inclusion criteria. These studies included a total of 2041 interviews
with health policy-makers. Assessments of the use of evidence were largely descriptive and qualitative, focusing on
hypothetical scenarios or retrospective perceptions of the use of evidence in relation to speciŽ c cases. Perceived
facilitators of, and barriers to, the use of evidence varied. The most commonly reported facilitators were personal
contact (13/24), timely relevance (13/24), and the inclusion of summaries with policy recommendations (11/24).
The most commonly reported barriers were absence of personal contact (11/24), lack of timeliness or relevance of
research (9/24), mutual mistrust (8/24) and power and budget struggles (7/24).
Conclusions: Interview studies with health policy-makers provide only limited support for commonly held beliefs
about facilitators of, and barriers to, their use of evidence, and raise questions about commonsense proposals for
improving the use of research for policy decisions. Two-way personal communication, the most common suggestion,
may improve the appropriate use of research evidence, but it might also promote selective (inappropriate) use of
research evidence.
Journal of Health Services Research & Policy Vol 7 No 4, 2002: 239–244

Introduction
Weiss and Weiss argue that, ‘both decision-makers and
social scientists behave as though social science research
makes a genuine contribution to public policy’.1 At the
same time, many researchers are sceptical about the
extent to which research is used and many policy-makers
are sceptical about the usefulness of research. As
Lindblom and Cohen suggest: ‘
in public policy
making, many suppliers and users of social research are
dissatisŽ ed, the former because they are not listened to,
the latter because they do not hear much they want to
listen to’.2
Several theories have been put forward to explain the
role of research in policy-making and common wisdom
Simon Innvær PhD, Researcher, Gunn E Vist PhD, Researcher, Mari
Trommald MD, Researcher, Andrew D Oxman MD, Head of Section,
Health Services Research Unit, National Institute of P ublic Health,
P ostboks 4404, Nydalen, 0403 Oslo, Norway.
Correspondence to: SI.

# The Royal Society of Medicine Press Ltd 2002

about how to improve the appropriate use of research is
not hard to Ž nd, although empirical evidence to
support it is difŽ cult to Ž nd.3 One source of such
evidence is the perceptions of those involved in policymaking. The objective of this systematic review was to
summarise the evidence from interview studies with
health policy decision-makers that cover their use of
research evidence in health policy decisions. We were
particularly interested in identifying facilitators of, and
barriers to, the use of evidence by health policy decisionmakers as a basis for developing strategies to improve
the appropriate use of research.
We found three previous non-systematic reviews that
addressed policy-makers’ use of evidence.4–6 The Ž rst
evaluated evidence regarding research utilisation in
relationship to the ‘two-communities’ metaphor and six
models of research utilisation.3 The second review
included 27 empirical studies with data relevant to the
question of how to improve the utilisation of organisational research.5 The third review was similar to the
second, but less comprehensive.6 In contrast to these

J Health Serv Res Policy Vol 7 No 4 October 2002

239

1994 (vols 3 and 4) were hand-searched. beginning with relevant articles that we had in our Ž les and searching for related articles. available from the authors). Burkina Faso (n ˆ 1). the type of interviews used. Two people extracted data from the included studies using a data collection form. Twenty-four studies reported in 26 articles met the inclusion criteria. the use of research. select and synthesise relevant research. Twenty-one of the 24 studies examined the role of evidence in actual decision-making processes. The following databases were searched in June 2000 using combinations of index terms and text words derived from relevant articles that we had previously identiŽ ed: Medline from 1966. Staff members who appeared to have less responsibility for decisions were also included in several studies. We only included studies of health policy-makers responsible for decisions on behalf of a large organisation or jurisdiction. ISI Web of Science from 1981. Additional articles were identiŽ ed by hand-searching journals (n ˆ 2).catchword. Generally the participants were poorly described. Face-to-face interviews. IPSA (International Political Science Abstracts) from 1989. are included in a list of excluded studies (46 studies. Most of the studies focused on one or a few cases. PAIS from 1972. Studies that appeared relevant. . Utilization was handsearched. 2 and 4). . If others were also interviewed – most often researchers – decision-makers had to be explicitly deŽ ned as a subgroup within the study. Mexico (n ˆ 1). all the included papers were in English. with widely varying policy questions and types of evidence. All studies measured perceptions of use or hypothetical . tracking references in articles and books (n ˆ 7) and searching in other electronic databases. Due to lack of availability. South Africa (n ˆ 1) and Sweden (n ˆ 1). the UK (n ˆ 3). When disagreements could not be resolved by discussion. The retrieved articles were assessed for inclusion by at least two people. Although non-English language papers were not excluded from our search. Studies of clinical decision-making for individual patients were excluded. Table 1 (characteristics): the study’s objective. how the use of evidence was measured or assessed. The studies had to address decision-makers’ use of research evidence in health policy decisions or on a broader range of policy decisions. The extracted data were summarised in three tables with the following information: . HealthStar from 1992. a third person assessed the article. Results More than 3000 references were found. 2 and 3 are included only in the online version of the journal. but most of the studies interviewed people with apparent responsibility for making important policy decisions. which was hand-searched for 1998–2000. following which 240 J Health Serv Res Policy Vol 7 No 4 October 2002 one person reviewed all of the citations and abstracts that were found and the reference lists of all of the retrieved articles. but which were not. IBSS (International Bibliography of the Social Sciences) from 1981. both the USA and the UK (n ˆ 1). such as people in ‘upper level positions’ and ‘senior managers’. Tables 1. the types of research evidence and the types of policy decisions that were considered. how the determinants of use were measured or assessed. and barriers to. telephone interviews and postal questionnaires were used with open-ended. Table 3 (main results): identiŽ ed the facilitators of.Review article Health policy-makers’ perceptions of their use of evidence earlier reviews. Search strategy The study designs varied. keywords and abstracts. Australia (n ˆ 1). EMBASE from 1980. Data collection and analysis Articles were retrieved if it appeared likely that they contained empirical data from interviews or if they otherwise appeared highly relevant. Diffusion. The studies were undertaken in the USA (n ˆ 10). personal communication (n ˆ 3). 1980 (vols 1. We have also focused the review on interview studies involving health policy decision-makers and their perceptions of the enablers of. semi-structured and structured interviews (Table 1). Seventy studies appeared relevant on the basis of title.com/rpsv/catchword/rsm/ 13558196/v7n4/s8/p239. Canada (n ˆ 3). the use of research evidence for policy decisions. The Netherlands (n ˆ 1). and the NHS National Research Register. Knowledge and Policy was hand-searched for 1995–1997 when it continued as Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization. characteristics of the participants. we have used a systematic approach to identify. response rate. Methods Search strategy We searched PubMed using an iterative strategy. Table 2 (methods): the sampling frame and strategy. only 1979. if these included health policy decisions. 1981–1984. and barriers to. Three people reviewed a sample of 200 citations and abstracts. SocioŽ le (Sociological Abstracts) from 1963. 1985 (vols 3 and 4). Knowledge: Creation. Pakistan (n ˆ 1). Selection criteria We included interview studies and surveys with health policy decision-makers. PsychLIT from 1987. The other three studies examined hypothetical decisions. 1987– 1993. and may be viewed free of charge at http://www. Relevant articles from these searches were identiŽ ed and new searches for related articles were performed.

34 Most studies reported how many decision-makers were included and their positions. The other 14 studies did not adequately describe the methods that were used. including personal two-way communication.17. last column).34 i. but the relation between the included policy-makers and the system from which they were drawn was rarely described. may not be easy to establish – for example. Political instability or high turnover of policy-making staff (5/24).e. However. . It can inform researchers about what the decision-makers consider timely. Based on the Ž ndings of these studies. Research that included effectiveness data (3/24). Six of the 24 studies rated variables that appeared to determine the use of evidence (Table 2). . Research that included a summary with clear recommendations (11/24). only three responded and no additional references were identiŽ ed through these contacts. is available from the authors. the use of research evidence. Nonetheless.17.18. Power and budget struggles (7/24). Timeliness and relevance of the research (13/24). Personal communication with investigators in the Ž eld yielded only three additional studies and. Community pressure or client demand for research (4/24). Study methods The included studies were all limited with respect to the generalisability of their results. we may have missed relevant studies. The limitations of the included studies with respect to sampling and generalisability are not surprising in light of the fact that most of the studies were qualitative and were not necessarily intended to include representative samples. Every included study was read and appraised by at least two of the authors.18. Lack of timeliness or relevance of research (9/24). given the diversity of contexts in which health policy decision-making occurs. Three of the 24 included studies met all of our methodological criteria:17. Absence of personal contact between researchers and policy-makers (11/24). No factor was mentioned in more than 13 of the 24 studies as a facilitator or barrier. Mutual mistrust. Three studies interviewed a representative sample of health policy decision-makers drawn from a clearly described sampling frame. It is unclear which barriers and facilitators the decisionsmakers found most important and which barriers and facilitators were considered. the use of evidence. . inadequate descriptions of the participants and contexts for many of the included studies make it difŽ cult to interpret the results. including perceived political naivety of scientists and scientiŽ c naivety of policy-makers (8/24). Because much research in the social sciences is poorly indexed in electronic databases. the frequently identiŽ ed facilitators. . This can reduce mutual mistrust and promote a better understanding of policy-making by researchers and research by policy-makers. Discussion The strengths of this review include an extensive and systematic literature search.18.9.Health policy-makers’ perceptions of their use of evidence use rather than actual use of research evidence. Review article The most commonly mentioned barriers to the use of research evidence in policy-making were: . even when similar factors were grouped together. explicit inclusion criteria and a systematic and transparent approach to collecting and presenting data from the included studies. Another source of variation is the fact that most studies focused J Health Serv Res Policy Vol 7 No 4 October 2002 241 . range 16–479). including the exact words used in each of the included studies. . Moreover. The most commonly mentioned facilitators of the use of research evidence in policy-making were: .35 Facilitators and barriers Many of the studies used open-ended questions. The respondents and investigators used different words to describe facilitators of. Seven studies partially met our criteria (Table 2. This variation can in part be explained by the fact that some of the studies focused on speciŽ c factors that were identiŽ ed a priori and therefore may have excluded other factors. Five of the included studies did not report how many policymakers were interviewed. The limitations of our review largely re ect the limitations of the literature we reviewed.14. it is not possible to obtain a generalisable sample. . . several common factors were described across studies. Poor quality of research (6/24). Measurement or assessment of the use and determinants of use of evidence was mostly descriptive. The remaining 19 studies interviewed a total of 2041 policy-makers (median 58. . relevant questions and policy-makers about how to obtain valid answers to these questions. However. personal twoway communication between researchers and decisionmakers should be used to facilitate the use of research. Research that conŽ rmed current policy or endorsed self-interest (6/24). Personal contact between researchers and policymakers (13/24). Good quality research (6/24). clearly described the sampling frame and the relationship of the included policy-makers to this. . A detailed version of this table. had a response rate of at least 60%. and barriers to. although we attempted to contact the authors of all of the included studies. and clearly described how the use of evidence and the determinants of how evidence was used were measured or assessed (Table 2). . These are summarised in Table 3. or facilitators of. . Most studies included a list of what the investigators perceived to be barriers to. because of political instability or high turnover of policymaking staff.

If what is required for research to be used is that researchers do what the policy-maker wants them to do. since these will always require some political judgement.5 The different deŽ nitions of ‘use’ contribute to the difŽ culty of interpreting the results of the studies included in this review. or focused and simple. reliable and unbiased. since the most commonly identiŽ ed facilitator of the use of evidence was personal contact between researchers and decision-makers. involving use ‘to legitimate and sustain predetermined positions’. namely to be objective.Review article Health policy-makers’ perceptions of their use of evidence on speciŽ c cases and factors were considered only in relationship to those cases. see themselves as responsible. Twoway communication between the two camps can facilitate a mutual understanding of a policy question and the kind of knowledge that is needed.36 This is often referred to as the ‘two-communities thesis’ or the ‘two cultures’. The other perspective focuses on the concept of the ‘use of research’. relevant or good quality research. Different types of policy questions: vague and complex. the included studies do not clarify whether decision-makers use only the research they want to use. Additionally. The most frequent categorisation of different types of use in this review is direct (‘instrumental’ or ‘engineering’).12 The study that found 40% direct use was in a context where 94% of all the research used . For example. action-oriented and pragmatic. the results should directly affect the solution without much adjustment. indifferent to evidence and new ideas. Different types of decision-makers: upper. Direct use of evidence refers to speciŽ c use of research results. Different issues: adoption versus implementation. Two theoretical perspectives were dominant in the literature. or decision versus action. For example. if research results are relevant for a solution. on the other hand. What is `use’ of evidence? Along with the ‘two-communities’ thesis. The ‘two-communities’ thesis sheds light on an important possible paradox inherent in the results: the factors that facilitate use of research may not necessarily be factors that researchers should seek to enhance. it is more likely that international support would be considered as a facilitator. . P ersonal twoway communication may also be a necessary precondition for other facilitators. and the relationship between the natural sciences and the humanities. they see scientists as naive. In a developing country study. jargon-ridden and irresponsible in relationship to practical realities.38 Selective use is strategic. middle and lower level. in a study focusing on the use of an economic evaluation. It is further supported by the most commonly mentioned barriers to the use of evidence. One study found that 40% of the use of evidence was direct. Many of the studies have addressed different kinds of use and found that health policy decision-makers refer to use in all three of the above ways. This requires that researchers and decision-makers agree on which questions can be answered on the basis of research evidence and which require political judgement. Decision-makers. Caplan et al found that social scientists see themselves as rational.9 they see decisionmakers as action. absence of personal contact and mutual mistrust between decision-makers and researchers. The degree to which evidence is used directly. The results of this review support the ‘two communities thesis’.9 Another study found only 7% direct use. A health policy decision-maker who deŽ nes use as direct use is likely to report less use of research evidence than decision-makers who also include selective or enlightening use of evidence. in an enlightening way or selectively may vary in relation to: . it would be more likely that decision-makers would consider alliances with interest groups as a facilitator or a barrier. In a study focusing on an issue on which interest groups have a strong position. this review is mostly based on information from one side of the two camps: the decision- 242 J Health Serv Res Policy Vol 7 No 4 October 2002 makers. objective and open to new ideas.9. Enlightening use of evidence refers to research that helps to ‘establish new goals and bench marks of the attainable’37 and helps to ‘enrich and deepen understanding of the complexity of problems and the unintended consequences of action’. not for researchers to assume the role of policy-makers. The degree to which evidence was directly used varied across studies. . it is more likely that Ž nancial constraints would be considered as a barrier. It is not surprising that decision-makers Ž nd it easier to use evidence that they have had an opportunity to in uence through two-way communication with the researchers. This does not imply that research alone can answer policy questions. The Ž rst perspective draws analogies between the relationship of researchers and policy-makers. Decision-makers may use evidence that supports their own ideology or their own political programme. On the other hand. without personal two-way communication it may be difŽ cult for researchers to understand what decision-makers regard as timely. then research may fail to fulŽ l one of its most important functions. It addresses how the word ‘use’ may have fundamentally different meanings. the question of what is meant by the concept of the ‘use’ of evidence is the most commonly discussed theoretical issue in the literature on knowledge utilisation. The `two-communities thesis’ The ‘two-communities thesis’ postulates the existence of two camps that lack the ability to take into account the realities or perspectives of one another. deŽ ned as the ‘primary source in the policy formulation process’. problems with communication between researchers and policy-makers are large and difŽ cult to overcome. From this perspective. The aim of two-way communication is to help ensure that research appropriately informs judgements for which policy-makers are accountable. selective (‘symbolic’ or ‘legitimating’) and enlightening (‘conceptual’) use of evidence. It indicates that.2.and interest-oriented.

they may not always be able to get what they want. relevant and of high quality. Review article The methods that were used in the included studies and their diversity limit the extent to which any Ž rm conclusion can be drawn from this review. these strategies will often not be effective. supported by anecdotes. 28: 140–146 9. Personal contact may also facilitate selective use of research. Some studies suggested document analysis as a way to check whether these perceptions correspond to what was done. If decisionmakers and researchers start to talk together about the barriers and facilitators found in this review. 27: 591–622 6. use of evidence will be facilitated by timely and relevant research that gives decision-makers the answers that they want. MI: The University of Michigan. the policy-makers who were interviewed and the contexts in which they worked. particularly for research that is not commissioned. one of the main conclusions of this review is that there is. Diffusion. Advice about how to improve the use of evidence by policy-makers is typically based on personal experience. The utilization of social science information by policy makers. Social scientists and decision makers look at the usefulness of mental health research. ensure that their research is perceived as timely. EUR-Assess Project Subgroup Report on Dissemination and Impact. American Psychologist 1973. Cochet C et al.19. which values both relevance and validity. Weiss JA. Weiss CH. Roberts J. 1979 3. 1996: 165–181 2. Lindblom CE. argue that the results of their research are relevant to current policy and demands from the community. Morrison A. given the diversity of the included studies. Taylor SH. Beyer JM. Decision-makers may interpret this as no more than a question of whether they consider research to be of value. Usable knowledge: social science and social problem solving. Research that conŽ rms current policy or supports community pressure may facilitate selective use.15. In: Lorion RP et al. 30: 569–577 7. Journal of Applied Behavioral Science 1982. CT: Yale University Press. Factors in uencing the success of applied research.21. appropriate and cooperative way of working together. Conclusions From the results of this review of 24 studies that met our inclusion criteria.12. At the same time. A third study. A minimal set of conditions necessary for the utilization of social science knowledge in policy formulation at the national level. eds. 18: 49–67 8. since researchers should. International Journal of Technology Assessment in Health Care 1997. Washington. DC: American Psychological Association. at best. They should avoid getting involved in power and budget struggles and be aware of the high turnover of policy-making staff. 1: 515–536 5. Glaser EM. Psychology and public policy: balancing public service and professional need. References 1. Caplan N. close to programme operations. Banta HD. and it is to be hoped will. a more re ective. The two-communities metaphor and models of knowledge use: an explanatory case study. If the results from the interview studies with decisionmakers are taken literally. insist on using methods that protect against bias. found that local administrators. based on 2041 interviews with policymakers. focus on commissioned research and clearly deŽ ne what is meant by ‘use’ of research. Of course. Johnson B. The utilization process: a conceptual framework and synthesis of empirical Ž ndings.Health policy-makers’ perceptions of their use of evidence had been commissioned. The studies included in this review address decisionmakers’ perceptions of their use of evidence. Cohen DK. reported more direct use of evidence than federal-level decision-makers. Trice HM. include effectiveness data. Knowledge: Creation. Granados A. However.33 but none of the included studies did this systematically. but not a basis for any strong recommendations. 1975: 1–63 10. if the aim is to increase appropriate (direct or enlightening) use of research rather than selective use. it is striking that some factors were identiŽ ed as frequently as they were. One study found that 60% of decision-makers ‘used’ evidence in this way. the two theories that are commonly used in these studies provide useful insights for understanding the perceptions summarised in this review and other opinions about the use of evidence by policy-makers. Nelson CE. Jonsson E. Stambaugh RJ. Caplan N. Wertheimer B. The use of social science knowledge in policy decisions at the national level: a report to respondents. American Behavioral Scientist 1987. The results of this review provide a context in which to consider such advice. may evolve. provide decision-makers with a brief summary of their research with clear policy recommendations. van de Vall M. may be difŽ cult to elicit in interviews because it puts decisionmakers in a bad light. but there are limited empirical data to support these or other theories. Bonair A.9 Self-reported selective use of research. Similarly. Using social J Health Serv Res Policy Vol 7 No 4 October 2002 243 . This suggests that expecting direct use frequently may be unreasonable. on the other hand. Bero L. as well as being difŽ cult to implement. Ann Arbor. researchers who wish to increase the use of the results of their research should: have personal and close two-way communication with decision-makers. They provide a menu of factors to consider. New Haven. Acknowledgement We thank Janette Boynton for help with literature searches. although it is commonly perceived as a problem by researchers and others. The study with only 7% direct use did not cover examples of commissioned research. Bolas C. In: Weiss CH. only limited support for any of the many opinions put forward in the literature on the use of research evidence by policy-makers. Future research should combine interviews with document analysis. Using social policy research for reducing social problems: an empirical analysis of structure and functions. Dunn WN. Maederer CM. ed. In fact.39 Enlightening use of evidence is more difŽ cult to assess than direct use. Utilization 1980. not included in this review. Administrative Science Quarterly 1982. 13: 220–286 4.

Journal of Management in Medicine 1993. 75: 11–34 Harries U. eds. Merton RK. Hohenadel J. 54: 461–468 Trostle J. Toronto: Lexington Books. Geneva: The Council of Health Research and Development (COHRED) Working Group on Research to Action and Policy. Langer A. Burns A. Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization 1997. 2000: 3–37 30. In: Weiss CH. DeMartini JR. 14. Gerhardus A. Jacobs M. How do researchers in uence decision-makers? Case studies of Mexican policies. Aaron HJ. Administration in Social Work 1983. Health policy-makers’ perceptions of their use of evidence research in public policy making. MI: University of Michigan Press. Geneva: The Council of Health Research and Development (COHRED) Working Group on Research to Action and Policy. Bronfman M. Fox DM. 18: 49–67 36. The role of research in child health policy and programs in Pakistan. 1974 . Evidence-based policymaking in the NHS: exploring the interface between research and the commissioning process. Darling H. ed. better outcomes? The use of health technology assessment and clinical effectiveness data in health care purchasing decisions in the United Kingdom and the United States. Hamilton: McMaster University. experience. Palfrey TR. Knowledge: Creation. 19. Experimental foundations of political science. Breanan NJ et al. Mason J. Rich RF. Lessons in research to action and policy: case studies from seven different countries. The utility of evaluation research for administrative decision-making. Ann Arbor. Lavis JN. In search for impact: an analysis of the utilization of federal health evaluation research. van de Vall M. Factors in uencing the use of health evaluation research in congress. Stoddard GL. Los Angeles: Center for the Study of Evaluation. Grimes PS. Higgins A. Philosophy of Science 1949. Better information. 21. Hurley J. How are policy makers using evidence? Models of research utilisation and local NHS policy making. Stoddard GL. Hamlyn L et al. 1998: 1–46 28. 15: 507–532 Ibbotson SL. 1978 39. Simon J. Truth test and utility test: decision makers’ frames of reference for social science research. Lavis JN. 7: 77–87 35. Elliott H. DiNitto DM. 17. Using social policy research for reducing social problems: an empirical analysis of structure and functions. Molyneaux-Smith L. Explaining use of information in public policymaking. 2000: 54–66 32. New York: Milbank Memorial Fund. 2000: 00–03. Evaluation and decision-making: the title vii experience. Hilderbrand M. 1977: 183–198 Patton MQ. 7: 77–87 Sunesson S. Farrant MSR. 1959 37. The role of applied social science in the formation of policy: a research memorandum. Administration in Social Work 1983. Diffusion. 10: 22–55 Oh CH. 1977: 141–164 Weiss CH. The use of economic evaluation in health care: Australian decision makers’ perceptions. In: Kinder DD. Johnson PJ. Washington. Evaluation of the Effective Health Care Bulletin. Hamilton: McMaster Institute of Environment and Health. American Sociological Review 1980. Ross SE. 1993: 185– 208 McNeece CA.I. 1997021. Bucuvalas MJ. Bolas C. DC: Brookings Institution. Utilization 1989. Geneva: The Council of Health Research and Development (COHRED) Working Group on Research to Action and Policy. Barriers to employment-related healthy public policy. Eyles J. 45: 302–313 Weiss JA. 2000: 1–44 34. 25. The role of health services research in Canadian provincial policy-making. Seligman R.E. 13. 2000: 19–27 31. 16: 161–181 38. 31: 103–110 Oh CH. Explaining the impact of policy information on policy-making. Fortess EE. London: OfŽ ce For Public Management. Coping with complexity: An experimental study of public policy decision-making. 16. Charlwood P. Stoddart G. Fitzgibbon C. DiNitto DM. 22. Kosecoff J. 2000: 77–85 27. 12. Making resource shifts supportive of the broad determinants of health: The P. Sanou A. 24. Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization 1996. Health Policy and Planning 1999. 1–28 33. 7: 48–57 Soumerai SB. Knowledge 1993. Popay J. Nilsson K.Review article 11. Ross-Degnan D. 21: 29–36 Elliott H. Determinants of change in Medicaid pharmaceutical cost sharing: does evidence affect policy? Milbank Quarterly 1997. Politics and the professors: the great society in perspective. 20. Toronto: McMaster University Centre for Health Economics and Policy Analysis. University of California. Evaluation Review 1991. An initial evaluation of Effective Health Care bulletins as instruments of effective dissemination. CHSRF Report. 10: 140–155 Florio E. The two cultures and the scientiŽc revolution. 15. Kielmann K. Journal of Public Health Medicine 1999. Woodward C et al. Johnson PJ. The use of information by policymakers at the local community level. Research to action and policy: combating vitamin a deŽ ciencies in South Africa. Explaining research utilization: beyond ‘functions’. Pranger T. Alkin MC. The utility of evaluation research for administrative decision-making. Health Policy 1995. Journal of Applied Behavioral Science 1982. GreenŽeld L. 18. Using social research in public policy making. Snow CP. Sheldon TA. McNeece CA. 14: 103–114 Ross J. Cambridge: Cambridge University Press. 15: 106–123 Boyer JF. 23. Journal of Epidemiology and Community Health 2000. Lavis J. Long AF. McMullan C. 2000: 1–26 29. Rudat K. Langbein LI. Walser BL. Moodley J. 9: 3–35 244 J Health Serv Res Policy Vol 7 No 4 October 2002 26. Toronto: Lexington Books. Guthrie KM. Hyder A.

143) Each decision-maker read two abstracts (two-pagers) out of 50 for actual social science research reports** (pp. but published April 1980. 5) Policy-makers’ perceptions Personal interviews. 140) `Most of the examples respondents offered to illustrate knowledge applications really involved the application of secondary source information’ (p. 144). 141) Principal investigator. with different questions for the decision-maker and the evaluator (p. (pp. what Personal interviews (p. N ˆ not given Health** (p. social welfare and public health (p. interest in social indicators. 303). 165. the decision-maker (government level) and the evaluator (p. 1). 168. 303) 302) Open-ended interviews. v. year conducted Objective (#year published). 187) *Revised version of paper presented in 1979 (p. ix) `To examine the nature and degree of utilization of federal evaluation research’ (p. information retrieval practice. 49) Study design The Netherlands 1969^19717* Country. five projects rated high for the success for ten administrators and persons `success’** (p. candid enough to be replicable. N ˆ 155 Three key informants from each of the 20 studies were interviewed: a project officer. N ˆ 204 (p. factors that influence use (p. 54 institute directors. regional and urban planning. both of the usefulness of social open-ended and structured sciences research (pp. 141). based on 11 factors identified by the authors (pp. attitudes about use. 50)** What was done. 67 deputy assistant directors. 167. 15% of the use of knowledge was on health or welfare problems (p. other Five projects rated low and Perceived use helped and what hindered staff members. and that the findings were disseminated adequately (p. vii). 171. alcoholism. vii) and USA 1973^197410 Semi-structured surveys (p. 8) Perceived and self-defined use (p.e. 143) USA 19738* Researchers and policymakers that were responsible for incorporating the research results (p. methods and measurement procedures. viii): 31 deputy under secretaries. loan forgiveness and training workshop (pp. Reprinted from American Psychologist 1981. N ˆ not stated Types of evidence (from which decisions were made) USA 1973^19749 (p. openended and structured (pp. made between 1971 and 1974. 140^141) **Success defined as project that was directed to the original objectives. v) USA 19802* and USA 198012 USA 1977#11 Personal interviews. knowledge of sources. degree of confidence on findings. 141) The conditions and functions influencing utilisation of social policy research (p. 94% were either funded by the government. perceived or actual)/types of policy Social science research utilisation and policy formation (p. 168. 166. Health training. v. alcohol and drug abuse** (pp. laboratory proficiency. `relevant to the issues your office deals with’) on mental health. N ˆ 60 (p. 142^143) 575 reported instances of use. where 385 involved primary research sources (p. embodied in a clear and cogent report. conducted by the government or both (p. 143) Upper-level decisionmakers in the executive branch of the US federal government (pp. 142^143) Hypothetical decisions (i. 143). two-pagers) and are therefore difficult to compare on the effect variables continued *The interviews were done in either 1967 or 1968 (pp.244a J Health Serv Res Policy Vol 7 No 4 October 2002 51 federal. 52 state (from ten states) and 52 local official decision-makers in upper-level positions (pp. 303) Perceived (self-reported) use with specific follow-up questions when use was reported (p. Awareness of research. neighbourhood health centres. 50) Participants/level of policy-making 120 research projects on Perceived (retrospective) `client-oriented’ social decisions (p. reference Table 1 Characteristics of included studies Review article Health policy-makers’ perceptions of their use of evidence . ix) *Not stated.e. evaluation of worth and objectivity. 52 agency personnel. 50). 50) research within industrial and labour relations. Based on the same research project as USA 198012 (but partly on other parts of the data) **The decision-makers were given different `empirical’ interventions (i. 140) research projects under the most likely to be utilisers National Institute of Mental (p. health services delivery systems. 49) **40 out of the 120 projects were on social welfare and public health Types of decisions Comments (hypothetical. 303) 20 national health care evaluations (case studies) with some systematic data collection. 8).

143^144) To determine what type of information decisionmakers use. year conducted Objective (#year published). 110). 51) Structured telephone interviews (p. N ˆ 172 (p. 515). 81) Each decision-maker received a long memorandum describing a problem. 507. interactive. reference Table 1 continued Two hypothetical decisions: one on the state’s Department of Health and Rehabilitative services and one on a methadone maintenance programme in New York City (pp. 50)* confidence to act on (pp. political/ conflict. 51. 50^51) and directors of planning/ commissioning in UK district health authorities or commissioning consortia (p. The six different versions of the cases varied in information complexity. 112) *Reprinted from the Journal of Policy Analysis and Management 2: 66^87 (p. 109) What factors influence the use of evaluation research (on health and human services) in congress (p. 185) **The experiment was based on six different versions of each of the two cases. 515) Sources of information used Perceived use (p. 514) E¡ective Health Care Perceived use of information *Respondents were questioned Bulletin No. conceptual (it changed the way of thinking) and persuasive (used to convince others. N ˆ 90 (p.244b To determine some of the factors that influence administrators’ use of evaluative data in decisionmaking. had on the quality of the decision process (p. enlightenment. 189^190. one study they were familiar Evaluation studies from the with (p. N ˆ 42 Study design Country. 141. 112) in community health care planning. semi-structured interviews (p. Each decision-maker had only one decision to make at only one level of complexity (pp. 152) Conducted studies on programme effectiveness or impact during the last 12month period (p. presenting evidence useful in analysing the problem. 152) Local planning committees and government employees in two rural communities in the Pacific Northwest (p. MIT. local and national politicians in 15 Swedish cities and communes (p. 188) Participants/level of policy-making Congress members and their staff (p. N ˆ 100 Social workers. and amount of information available. and tactical (pp. 1991. 52^54) Each decision-maker chose Perceived use (p. 513)** Personal. 52) information in the bulletin (pp. 512). semi-structured and open-ended questionnaires combined with a controlled experiment. 516) **The interviews lasted 15^40 minutes (p. 50). use of the E¡ective Health postal questionnaires directors of public health Care Bulletin (pp. 508)* USA 1983#14 J Health Serv Res Policy Vol 7 No 4 October 2002 UK 199118 USA 1991#17 USA 199016 Sweden 1984^198515 (p. 80^81) continued *Use is defined here as both instrumental (direct implementation). management directors. decisions (p. Stanford and the Washington Public Affairs Center (pp. p. 187. 152). 1. perceived or actual)/types of policy Documents that the Perceived/retrospective respondents referred to. and posed a decision to be made based on real people and events (p. 110) Use and reasons for use/not Structured and open-ended District general managers. 79^80) Florida and North Carolina (p. and to examine the types of decisions (p. 152) Structured and open-ended Directors in community postal questionnaires mental health centres in (pp. and how ideology and interests influence the use of information (p. N ˆ 132 (p. reported) use (pp.e. 79). Consequences of complexity were examined by effects of information load and number of decision alternatives (pp. on and hypothetical decisions. 51^54) (pp. 193. 112) Interviews (p. 194)** To observe the impact that decision complexity. 188). 152) Structured. N ˆ 27 (p. and the assessment of the strategic planning process (p. 188) Types of evidence (from which decisions were made) Review article Health policy-makers’ perceptions of their use of evidence . 79) Different ways of using social research: instrumental. 187) USA 198213* Experienced public and private sector managers from six mid-career programmes at Harvard. 152) These documents were also analysed by the authors (p. 193) Types of decisions Comments (hypothetical. 193) Perceived (i. 188^189). 515) Department of Health and Human Services that deal with national health programmes (pp. Decision had to be made from available information within 1 hour 45 minutes (p. 187. about a bulletin that had been screening for osteoporosis change of opinion and previously posted to them to prevent fractures (p.

77). 31). the planning commission and provincial health authorities (p. 1) (p.244c `To identify factors which facilitate or impede evidence-based policymaking’ (p. 16) Key informants* in states that changed cost-sharing policies for prescription reimbursement limits and co-payments (p. 29) USA 1993^199419 Participants/level of policy-making Open-ended in-depth interviews (p. 1) development of our policy. 14) and semistructured telephone interviews (p.e. directors or policy analysts. although this has been going on for some time’) (p. knew that there was at least cholera. 25) Types of evidence (from which decisions were made) Review article Health policy-makers’ perceptions of their use of evidence .g. 13). breastfeeding and polio (pp. 77) Structured interviews (p. N ˆ 16 (p. GPs topic guide) (p. 105) continued *The date is not clearly stated. service provision and financing) (pp. but `infotype’ refer to programme evaluation. 103) Research publications and comments (p. New South Wales State. open-ended telephone interviews (p. N ˆ 28 Open-ended (`in-depth’) `Researchers and policyinterviews with a common makers from different interview protocol (p. 77) The impact that the E¡ective Telephone interviews. pharmacy consultants and legislators (p. state and local policy-makers at different levels dealing with finance and services (p. radiology. N ˆ 34 (p. 30) Use of research Perceived** use of publications `where we evidence concerning AIDS. N ˆ 479 (p. 105) Federal. 17) `Perceived use of economic evaluations’ (p. 29) `The relationship between health research (and researchers) and health policies (and policymakers) in four vertical health programmes in Mexico’ (p. N ˆ 67 Study design Country. 25ff) *Informants included Medicaid drug programme administrators. statistical data and policy analysis (p. 104) General perceptions. 10) No specific report. 25) Perceived/retrospective decisions (p. 34) Not stated Use of economic evaluations for pricing on drugs. `it influenced the (p. family planning some interaction between and immunisation in researchers and decision. but are not relevant in this review *Open-ended interviews were used to obtain basic background information about the policy process. 2).21 Mexico 1994^199522 USA 1996#24 and USA 1997#25 J Health Serv Res Policy Vol 7 No 4 October 2002 UK 199827* Pakistan 199826 Australia 1995#23 Questionnaires (p. perceived or actual)/types of policy Social research (p. 29. N ˆ 34 Decision-makers from government. telephone or face-toface) (p. maternal mortality. 77. year conducted Objective (#year published). 42 in the Perceived decisions. 29 and and researchers in nine response to a letter) different case studies at a local level in the NHS (pp. 105) Health authorities** (p. demonstration project. 15) Types of decisions Comments (hypothetical.14). 77) Senior managers in the Federal Department of Health. 79) Two specific bulletins: June Hypothetical decisions 1997 and October 1997 (e. transplants and scanning (p. since there are no actual decisions mentioned Only 2^3 pages out of 21 were relevant (p. reference Table 1 continued Perceived/retrospective decisions (p. the Health Department (p. N ˆ inadequately described In-depth interviews (using a Lead policy-makers. 104). The structured interviews built on the open-ended interviews (p.103). 16)* Interviews. screening. open-ended and structured (not stated if postal. 16). Closest focus on child health problems such as co-trimoxazole. 106)* institutions and levels of responsibility’ (p. Healthcare Bulletin has structure not mentioned inside the NHS organisation (p.governmental programmes makers’ (p. 14) The role of research information in policy development (p. 16^17) The role that research plays in experiences in linking research with policy (p. 104)* Decision-makers’ awareness and use of economic evaluations in health care (p. 33) *Personal communication gave additional information on this issue **It is difficult to discern whether the decisions were perceived or hypothetical. Housing and Community Services. vitamin A. 103) To determine whether the process of (social science) information utilisation varies across different areas (i. the central ministry of health. Policy project report) on the interface between R&D and commissioning at the health authority level (p. 105) (p. but it is 1998 or 1999 **Trusts and GPs are included in the study. 1) UK 1994^199520.

Canada 200033 Canada 200032* South Africa 200031 Burkina Faso 199930 Canada 1998^199929 UK and USA 1998#28* Country. MIT. general practitioners. 4) Services and various sectors (p. R&D. N ˆ not stated the vitamin A research^ policy^action connection (p. 5) To understand whether. all sectors and management levels (p. five health regions. 13). 4. 11) in two Canadian provinces research plays a role in (Ontario and provincial policy-making Saskatchewan) (p.Key informants holding sectoral resource allocation ended interviews and focus different positions within the in the human services groups (pp. NHS. seven public health physicians. N ˆ 58 (p. 18) (p. N ˆ not given decision-making about `Shared Care’* for childhood illness (pp. research and development. 3) Types of decisions Comments (hypothetical. 55) To assess the influence of Structured. 55). v. N ˆ 55 (p. 13) Perceived use (p. programme delivery arrangements and programme content (pp. ii). 7. 5. 5) Actual and perceived decisions within the fields of health services jurisdiction. financial arrangements. 13^16. semi-structured Policy-makers in the health information about the health and open-ended telephone and employment sectors of consequences of interviews (pp. and executive insecurity on policy-making directors of ten Canadian (p. 20) Perceived decisions on cross-sectoral resource allocations (p. 13. 21) Not stated Perceived/retrospective usefulness of past research information about the health consequences of unemployment and insecurity on policy-making (p. 4) assessing health technology and clinical effectiveness (pp. 20) (p. 5) *It is not stated what kind of evidence was used. 5). 19^ 20) Barriers against and Semi-structured interviews Key informants that were mechanisms for improving (p. but references to direct use of evidence in decision reports. three consultants. National Health Service. 55) relevant role-players (p.244d J Health Serv Res Policy Vol 7 No 4 October 2002 Participants/level of policy-making UK: 14 non-medical health authority purchasers. the system in Prince Edward Appendix 1) Department of Social Island (p. nine GPs. 22). 55) Perceived use of the provided scientific information in decisionmaking (p. N ˆ 48 (p. nine private sector purchasers. 20) *Stated 1999 (p. 5) non-governmental organisations (p. 18^19) *Stated date for working paper is March 2000 *`Shared care’ is about improved contact and collaboration between health centre staff and mothers (p. 11) Types of evidence (from which decisions were made) GPs. implies that evidence was used (pp. 4). reporting on actual use of research. perceived or actual)/types of policy Access to and use of health Perceived use (p. semiKey informants within a how. reference Table 1 continued Review article Health policy-makers’ perceptions of their use of evidence . Both personal. 1) Study design Not stated* Not stated Not stated Workshops and research papers that the authors had provided Ministry of Health officials with during previous years (p. all regions. 9) all three levels of Canadian unemployment and government. 1) technology assessment/ clinical effectiveness information (pp. USA: 13 public officials from 11 states. Massachusetts Institute of Technology. 4) A case study of crossSemi-structured and open. 9) A case study on how Semi-structured interviews Ministry of Health officials research played a role in (p. i) How UK and US purchasers Semi-structured telephone value and use research interviews (p. N ˆ 22 (p. 8. year conducted Objective (#year published). and under what structured interviews and variety of key health policies conditions health services surveys (pp.

79)* SF: not stated SS: not stated Measurement of determinants of use A quantitative 4-point score weighted according to Two-way ANOVA (p. 143) SS: snowballed. viii) The Netherlands 1969^19717 USA 19738 J Health Serv Res Policy Vol 7 No 4 October 2002 SF: complete (p. 51) SS: random sample (p. 79) Volunteered (p. 145) Not adequately described *Complete refers to the 27 representatives in the two planning committees in the two communities. 51^54) Complete Qualitative Frequencies and rated importance (as very. and if there were only one director in each Community Mental Health Center Partial *The 132 decision-makers were randomly distributed to two of 50 case studies and six different levels (p. 143) USA 19802 and USA SF: partial (p. . 149) Descriptive listing of ratings (pp. Descriptive listing of 33. or not adequately described) and comments Descriptive listing of weighted ratings (pp. 82) Listing of frequencies (pp. 167) 198012 SS: not stated USA 197711* USA 1973^19749 and USA 1973^ 197410 (p. 66) three decision levels. 194) Partial *Both a 4-point (p.244e SF: not stated SS: not stated SF: not stated SS: not stated SF: partial (p. 144). 169ff. 25. `. enlightenment. 196^198) (pp. Subsequent reports not found Not adequately described Not adequately described Summary of description of methods (complete. 187^188) SS: random sample* (pp. partly snowballed (p. . moderately or not important) (p. 303) 90% (p. 519^522) Qualitative and descriptive (p. reference Table 2 Study methods Review article Health policy-makers’ perceptions of their use of evidence . 51) Descriptive No statistical technique Measurement of use of evidence 95% (p. 187^188). 51) UK 199118 60% (p. with summed score ranging from 0 to 18 (p. 36) and qualitatively described (pp. as either: instrumental. 79)* SS: not stated (p. 517^523) and multiple regression (pp. partly random (p. 27) Descriptive No statistical technique SF: partial (p. 35. 303). 513) USA 199117 100% (p. or tactical (p. 146^150) Quantitative listing and ratings (pp. 36) and MCA (multivariate regression) analysis (p. 33. political/conflict. 513^514) SS: random sample (p. partial. SF: not stated (p. 304) Quantitative listing of the decision quality of Listing of frequencies evidence ^ use on six different levels of complexity (pp. 188) Quantitative rated descriptions (pp. 514) SF: complete (pp. 110) SS: complete* (p. 145.110) Not stated 65% (p. 173. 112) Qualitative and descriptive (pp. 518. 195^208) Both quantitative descriptions. 170^172. 1^4) ratings (pp. 28. 28. 82^86) continued Not adequately described The authors mainly discuss and demonstrate different ways of utilising research. 167) Not stated Both quantitatively rated and weighted (pp. 51) 73% (p. 174) and 25 variables analysed using factor analysis to develop five dimensions** (p. rated and weighted and qualitative* (pp. viii) SS: not mentioned. 24) Not stated Not stated Response rate USA 199016 Sweden 1984^ 198515 USA 198314 USA 198213 SF: partial (p. No further generalisation can be made on the basis of this sample Complete **Weighted on 3-point and 5-point scales *SF and SS complete if the study’s focus considered only directors in Florida and North Carolina. 35. 523^527) Percentages (pp. 51^54) Quantitatively descriptive. 168) and a 5-point weighted rating scale (pp. vi). vii) Sample frame (SF)a and sample strategy (SS)b Study. interactive. rated and weighted** (pp. The qualitative rating comes from listing of most mentioned characteristics of usefulness **The five dimensions are listed in Table 3 under facilitators Partial *The interviews appear to have been made between 1973 and 1976 Partial It is stated that `this report represent only a first summary of some of the highlights and more important findings’ (p. 110) SF: Partial (pp. using research ``instrumentally’’ or for ``enlightenment’’ purposes meant different things in different settings and different organizational positions’ (p.

16) 100% (all those Qualitative and descriptive (p. 11^14. 18) Qualitative Qualitative Qualitative Qualitative Measurement of determinants of use Partial *Called `a purposive sample of Canadian policy-makers’ (p. 30^31) SS: combined random and snowballed (pp. 35^37) Descriptive Qualitative descriptions and quantitative listings (pp. any mention. 26) Qualitative Qualitative Qualitative Qualitative Qualitative Qualitative Quantitative regression and a path model (p. 29) Not adequately described Summary of description of methods (complete. 80.e. 104) Partial.15) SS: not stated SF: partial (pp. partial.e. 12^22) Qualitative and descriptive (i. a Canada 200033 Canada 200032 South Africa 200031 Canada 1998^199929 Burkina Faso 199930 UK and USA 199828 UK 199827 Pakistan 199826 USA 199624 and USA 199725 Sample frame (SF)a and sample strategy (SS)b Study. subjective view on key variables) (pp. 77) SS: not stated SF: partial (pp. but with a good description of SS and SF Not adequately described Only Medicaid programmes with specific cost-containment policies in the last several years (p. 16. i. 465) SF: not stated* SS: not stated* SF: partial SS: not stated* SF: partial (pp. general perceptions and impact) Descriptive (pp. subjective view on key variables of policy change. 23) Not stated Not stated Not stated Not stated Not stated Not stated Not stated Both quantitative and qualitative descriptions of barriers (pp. 78. reference Table 2 continued Review article Health policy-makers’ perceptions of their use of evidence . 26) Both quantitative and qualitative descriptions of facilitators (pp. partial ˆ incomplete information. 105^108) Not stated Response rate Qualitative Frequencies (p.e. 104. 16^17) Not adequately described Not adequately described Not adequately described Not adequately described Not adequately described *Purposive sample (personal communications) Not adequately described *Called `purposive sampling frame’ (p. or not adequately described) and comments b Sample frame: complete ˆ the sample was described in relation to all the decision-makers in the context of the study.21 Mexico 1994^199522 Australia 199523 J Health Serv Res Policy Vol 7 No 4 October 2002 Measurement of use of evidence 88% (22/25) (p. 18^30. no ranking) (p. 22) Qualitative and descriptive (pp. 16) SS: not stated* SF: not stated SS: not stated SF: not stated SS: not stated SF: partial SS: not stated* SF: not stated SS: snowballed (p. 1^2) SS: not stated (pp. 31ff) approached for interview agreed to the interview) Not stated Qualitative and descriptive (i. 465) SS: snowballed (pp. Sample strategy: complete sample. 82) Quantitative regression and a path model (p. 461. or not stated. ii. 1^2) SF: partial (p. not stated ˆ no description. 106^108) Not stated Quantitative and descriptive (pp. 4) SS: not stated SF: partial (p. 8) Not adequately described Not adequately described Partial *A qualitative validation of the interviews is described (pp. 461. random sample. snowballed sample.244f SF: partial (p. 18) 42% (p. 32^34) Qualitative and descriptive (p. 30^31) SF: partial (p. 18) Qualitative and descriptive (i. ii) USA 1993^199419 UK 1994^199520. subjective view on awareness.

reference Table 3 Main results Poor quality of research (p. 149) Research that includes a summary with clear recommendations (p. 155. 27) Personal contact between researchers and decision-makers (p. 142. 519. 32^34) Timeliness and relevance of the research (pp. 32^34) Personal contact between researchers and decision-makers (p. 114) Research that confirms current policy or endorses self-interest (p. 149. 158) Timeliness and relevance of the research (p. 33) Mutual mistrust between policy-makers and researchers Absence of personal contact between policy-makers and researchers Political instability or high turnover of policy-making staff (pp. vii) USA 19738 The Netherlands 1969^1971 Study. 82. 106) Political instability or high turnover of policy-making staff (pp. 33) Lack of timeliness or relevance of research (p. 519) Research that includes a summary with clear recommendations (p. 33. 108) Mexico 1994^199522 USA 199624 Australia 199523 UK 1994^199520. 106) Lack of timeliness or relevance of research (p. 27) Power and budget struggles (pp. 142) Timeliness and relevance of the research (p. 108) Good quality research (p. 145) Mutual mistrust between policy-makers and researchers (p. 107^111) continued Not stated Lack of timeliness or relevance of research Absence of personal contact between policy-makers and researchers (pp. 107) Timeliness and relevance of the research (p. 143) Research that confirms current policy or endorses self-interest (p. 520) Research that includes a summary with clear recommendations (p. 115) Poor quality of research (pp. 60) Community pressure or client demand for research (p. 143) Personal contact between researchers and decision-makers (pp. 36) Timeliness and relevance of the research (p. 29) Personal contact between researchers and decision-makers (pp. 149) Research that includes a summary with clear recommendations (p. 146. 141) Community pressure or client demand for research (p. 152) Political instability or high turnover of policy-making staff (pp. 106^108. 159^160) Absence of personal contact between policy-makers and researchers (pp. 27) Timeliness and relevance of the research (p. 141. 109^113) Personal contact between researchers and decision-makers (p. 55) Good quality research (p. 31^32) Poor quality of research (p. 21) Personal contact between researchers and decision-makers (p.21 UK 199118 USA 1993^199419 USA 199117 USA 199016 Sweden 1984^198515 USA 198213 USA 198314 Good quality research Research that confirms current policy or endorses self-interest Timeliness and relevance of the research Research that includes a summary with clear recommendations None of the facilitators were mentioned in the other studies Community pressure or client demand for research (pp. 148) Timeliness and relevance of the research (p. 118) Good quality research (p. 155. 82) Research that confirms current policy or endorses self-interest (pp. 54) Research that includes a summary with clear recommendations (p. 108) Research that includes a summary with clear recommendations (p. 28) Absence of personal contact between policy-makers and researchers (p. 111. 54) Research that confirms current policy or endorses self-interest (p. 84) Inclusion of effectiveness data (p. 119^122) Power and budget struggles (p. 519) Personal contact between researchers and decision-makers (p. 143) Barriers to use of research identi¢ed in three or more studies Review article Health policy-makers’ perceptions of their use of evidence . 145) Facilitators to use of research identi¢ed in three or more studies USA 19802 and USA 198012 USA 197711 USA 1973^19749 and USA 1973^197410 (p. 33) Personal contact between researchers and decision-makers (pp. 106) Absence of personal contact between policy-makers and researchers (p. 65) Personal contact between researchers and decision-makers (p. 13^14. 150) None of the barriers were mentioned in the other studies Not stated Power and budget struggles (pp. 158) Absence of personal contact between policy-makers and researchers Mutual mistrust between policy-makers and researchers Power and budget struggles Mutual mistrust between policy-makers and researchers (p.244g 7 J Health Serv Res Policy Vol 7 No 4 October 2002 Personal contact between researchers and decision-makers Timeliness and relevance of the research Research that includes a summary with clear recommendations Research that confirms current policy or endorses self-interest Community pressure or client demand for research Inclusion of effectiveness data (pp. 143) Lack of timeliness or relevance of research (pp. 33) Good quality research (p. 519) Timeliness and relevance of the research (pp. 28) Absence of personal contact between policy-makers and researchers (pp.

21) Absence of personal contact between policy-makers and researchers (p. 82) Lack of timeliness or relevance of research (pp. 62) Timeliness and relevance of the research (p. 23) Mutual mistrust between policy-makers and researchers (p. 33^34) Political instability or high turnover of policy-making staff (p. 82) Poor quality of research (p. 32) UK 199827 Pakistan 1998 26 Study. 80. 82) Power and budget struggles (p. 61) Lack of timeliness or relevance of research (p. 61) Power and budget struggles Mutual mistrust between policy-makers and researchers Poor quality of research (p. 78. 12) Barriers to use of research identi¢ed in three or more studies Review article Health policy-makers’ perceptions of their use of evidence J Health Serv Res Policy Vol 7 No 4 October 2002 . 80) Lack of timeliness or relevance of research (pp. 8) Research that includes a summary with clear recommendations (p. 22) Mutual mistrust between policy-makers and researchers (p. 13) Personal contact between researchers and decision-makers Good quality research Research that includes a summary with clear recommendations Timeliness and relevance of the research Inclusion of effectiveness data (pp. 12^17) Absence of personal contact between policy-makers and researchers (pp. 62) Not stated Personal contact between researchers and decision-makers (p. 17^21) Personal contact between researchers and decision-makers (p. 26) Not stated Absence of personal contact between policy-makers and researchers Poor quality of research Lack of timeliness or relevance of research (pp.244h Not stated Research that includes a summary with clear recommendations (p. 63) Absence of personal contact between policy-makers and researchers (p. 61) Political instability or high turnover of policy-making staff (p. 5. 8. reference Table 3 continued Power and budget struggles Lack of timeliness or relevance of research (pp. i) Burkina Faso 199930 South Africa 200031 Canada 200032 Canada 200033 UK and USA 199828 Canada 1998^199929 Not stated Facilitators to use of research identi¢ed in three or more studies Timeliness and relevance of the research (p. 3. 80) Mutual mistrust between policy-makers and researchers (p.