Review article

Health policy-makers’ perceptions of their use
of evidence: a systematic review
Simon Innv×r, Gunn Vist, Mari Trommald, Andrew Oxman
Health Services Research Unit, National Institute of Public Health, Oslo, Norway

Objectives: The empirical basis for theories and common wisdom regarding how to improve appropriate use of
research evidence in policy decisions is unclear. One source of empirical evidence is interview studies with policymakers. The aim of this systematic review was to summarise the evidence from interview studies of facilitators of,
and barriers to, the use of research evidence by health policy-makers.
Methods: We searched multiple databases, including Medline, Embase, SocioŽ le, PsychLit, PAIS, IBSS, IPSA and
HealthStar in June 2000, hand-searched key journals and personally contacted investigators. We included interview
studies with health policy-makers that covered their perceptions of the use of research evidence in health policy
decisions at a national, regional or organisational level. Two reviewers independently assessed the relevance of
retrieved articles, described the methods of included studies and extracted data that were summarised in tables and
analysed qualitatively.
Results: We identiŽ ed 24 studies that met our inclusion criteria. These studies included a total of 2041 interviews
with health policy-makers. Assessments of the use of evidence were largely descriptive and qualitative, focusing on
hypothetical scenarios or retrospective perceptions of the use of evidence in relation to speciŽ c cases. Perceived
facilitators of, and barriers to, the use of evidence varied. The most commonly reported facilitators were personal
contact (13/24), timely relevance (13/24), and the inclusion of summaries with policy recommendations (11/24).
The most commonly reported barriers were absence of personal contact (11/24), lack of timeliness or relevance of
research (9/24), mutual mistrust (8/24) and power and budget struggles (7/24).
Conclusions: Interview studies with health policy-makers provide only limited support for commonly held beliefs
about facilitators of, and barriers to, their use of evidence, and raise questions about commonsense proposals for
improving the use of research for policy decisions. Two-way personal communication, the most common suggestion,
may improve the appropriate use of research evidence, but it might also promote selective (inappropriate) use of
research evidence.
Journal of Health Services Research & Policy Vol 7 No 4, 2002: 239–244

Introduction
Weiss and Weiss argue that, ‘both decision-makers and
social scientists behave as though social science research
makes a genuine contribution to public policy’.1 At the
same time, many researchers are sceptical about the
extent to which research is used and many policy-makers
are sceptical about the usefulness of research. As
Lindblom and Cohen suggest: ‘
in public policy
making, many suppliers and users of social research are
dissatisŽ ed, the former because they are not listened to,
the latter because they do not hear much they want to
listen to’.2
Several theories have been put forward to explain the
role of research in policy-making and common wisdom
Simon Innvær PhD, Researcher, Gunn E Vist PhD, Researcher, Mari
Trommald MD, Researcher, Andrew D Oxman MD, Head of Section,
Health Services Research Unit, National Institute of P ublic Health,
P ostboks 4404, Nydalen, 0403 Oslo, Norway.
Correspondence to: SI.

# The Royal Society of Medicine Press Ltd 2002

about how to improve the appropriate use of research is
not hard to Ž nd, although empirical evidence to
support it is difŽ cult to Ž nd.3 One source of such
evidence is the perceptions of those involved in policymaking. The objective of this systematic review was to
summarise the evidence from interview studies with
health policy decision-makers that cover their use of
research evidence in health policy decisions. We were
particularly interested in identifying facilitators of, and
barriers to, the use of evidence by health policy decisionmakers as a basis for developing strategies to improve
the appropriate use of research.
We found three previous non-systematic reviews that
addressed policy-makers’ use of evidence.4–6 The Ž rst
evaluated evidence regarding research utilisation in
relationship to the ‘two-communities’ metaphor and six
models of research utilisation.3 The second review
included 27 empirical studies with data relevant to the
question of how to improve the utilisation of organisational research.5 The third review was similar to the
second, but less comprehensive.6 In contrast to these

J Health Serv Res Policy Vol 7 No 4 October 2002

239

Due to lack of availability. The following databases were searched in June 2000 using combinations of index terms and text words derived from relevant articles that we had previously identiŽ ed: Medline from 1966. We have also focused the review on interview studies involving health policy decision-makers and their perceptions of the enablers of. All studies measured perceptions of use or hypothetical . Table 3 (main results): identiŽ ed the facilitators of. tracking references in articles and books (n ˆ 7) and searching in other electronic databases. Diffusion. Studies of clinical decision-making for individual patients were excluded. The other three studies examined hypothetical decisions.catchword. . 1980 (vols 1. both the USA and the UK (n ˆ 1). keywords and abstracts. When disagreements could not be resolved by discussion. 2 and 3 are included only in the online version of the journal. HealthStar from 1992. If others were also interviewed – most often researchers – decision-makers had to be explicitly deŽ ned as a subgroup within the study. are included in a list of excluded studies (46 studies. Table 2 (methods): the sampling frame and strategy. characteristics of the participants. Utilization was handsearched.Review article Health policy-makers’ perceptions of their use of evidence earlier reviews. 1987– 1993. Knowledge: Creation. Canada (n ˆ 3). Face-to-face interviews. PAIS from 1972. PsychLIT from 1987. South Africa (n ˆ 1) and Sweden (n ˆ 1). 2 and 4). Mexico (n ˆ 1). Although non-English language papers were not excluded from our search. 1985 (vols 3 and 4). Twenty-four studies reported in 26 articles met the inclusion criteria. beginning with relevant articles that we had in our Ž les and searching for related articles. Additional articles were identiŽ ed by hand-searching journals (n ˆ 2). following which 240 J Health Serv Res Policy Vol 7 No 4 October 2002 one person reviewed all of the citations and abstracts that were found and the reference lists of all of the retrieved articles. SocioŽ le (Sociological Abstracts) from 1963. The studies had to address decision-makers’ use of research evidence in health policy decisions or on a broader range of policy decisions. The studies were undertaken in the USA (n ˆ 10). Data collection and analysis Articles were retrieved if it appeared likely that they contained empirical data from interviews or if they otherwise appeared highly relevant. Table 1 (characteristics): the study’s objective. EMBASE from 1980. Seventy studies appeared relevant on the basis of title. semi-structured and structured interviews (Table 1). IPSA (International Political Science Abstracts) from 1989. the use of research evidence for policy decisions. Relevant articles from these searches were identiŽ ed and new searches for related articles were performed. response rate. such as people in ‘upper level positions’ and ‘senior managers’. Tables 1. all the included papers were in English. and may be viewed free of charge at http://www. only 1979. Studies that appeared relevant. Knowledge and Policy was hand-searched for 1995–1997 when it continued as Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization. Australia (n ˆ 1). 1981–1984. how the use of evidence was measured or assessed. select and synthesise relevant research. Twenty-one of the 24 studies examined the role of evidence in actual decision-making processes. Burkina Faso (n ˆ 1). Search strategy The study designs varied. ISI Web of Science from 1981. and barriers to.com/rpsv/catchword/rsm/ 13558196/v7n4/s8/p239. but which were not. Staff members who appeared to have less responsibility for decisions were also included in several studies. IBSS (International Bibliography of the Social Sciences) from 1981. We only included studies of health policy-makers responsible for decisions on behalf of a large organisation or jurisdiction. which was hand-searched for 1998–2000. Pakistan (n ˆ 1). Results More than 3000 references were found. if these included health policy decisions. The extracted data were summarised in three tables with the following information: . the type of interviews used. Selection criteria We included interview studies and surveys with health policy decision-makers. available from the authors). Three people reviewed a sample of 200 citations and abstracts. The Netherlands (n ˆ 1). . The retrieved articles were assessed for inclusion by at least two people. how the determinants of use were measured or assessed. Methods Search strategy We searched PubMed using an iterative strategy. the UK (n ˆ 3). and barriers to. and the NHS National Research Register. with widely varying policy questions and types of evidence. telephone interviews and postal questionnaires were used with open-ended. Two people extracted data from the included studies using a data collection form. Generally the participants were poorly described. a third person assessed the article. the use of research. but most of the studies interviewed people with apparent responsibility for making important policy decisions. we have used a systematic approach to identify. personal communication (n ˆ 3). the types of research evidence and the types of policy decisions that were considered. 1994 (vols 3 and 4) were hand-searched. Most of the studies focused on one or a few cases.

The respondents and investigators used different words to describe facilitators of. . given the diversity of contexts in which health policy decision-making occurs.34 Most studies reported how many decision-makers were included and their positions. Power and budget struggles (7/24). . . Personal communication with investigators in the Ž eld yielded only three additional studies and.Health policy-makers’ perceptions of their use of evidence use rather than actual use of research evidence. relevant questions and policy-makers about how to obtain valid answers to these questions. the use of evidence.18. inadequate descriptions of the participants and contexts for many of the included studies make it difŽ cult to interpret the results. Moreover.14.18. . Six of the 24 studies rated variables that appeared to determine the use of evidence (Table 2). Because much research in the social sciences is poorly indexed in electronic databases. including personal two-way communication.18. This variation can in part be explained by the fact that some of the studies focused on speciŽ c factors that were identiŽ ed a priori and therefore may have excluded other factors. last column). Community pressure or client demand for research (4/24). Lack of timeliness or relevance of research (9/24). or facilitators of. Another source of variation is the fact that most studies focused J Health Serv Res Policy Vol 7 No 4 October 2002 241 . Political instability or high turnover of policy-making staff (5/24). clearly described the sampling frame and the relationship of the included policy-makers to this. personal twoway communication between researchers and decisionmakers should be used to facilitate the use of research.34 i. However. and barriers to. It is unclear which barriers and facilitators the decisionsmakers found most important and which barriers and facilitators were considered. . The limitations of the included studies with respect to sampling and generalisability are not surprising in light of the fact that most of the studies were qualitative and were not necessarily intended to include representative samples. including the exact words used in each of the included studies.17. Good quality research (6/24). . Mutual mistrust. The limitations of our review largely re ect the limitations of the literature we reviewed. is available from the authors. the use of research evidence. it is not possible to obtain a generalisable sample. A detailed version of this table. although we attempted to contact the authors of all of the included studies. Research that conŽ rmed current policy or endorsed self-interest (6/24).9. Based on the Ž ndings of these studies. and clearly described how the use of evidence and the determinants of how evidence was used were measured or assessed (Table 2). The most commonly mentioned facilitators of the use of research evidence in policy-making were: . These are summarised in Table 3. Research that included a summary with clear recommendations (11/24). .e. Poor quality of research (6/24). Timeliness and relevance of the research (13/24). the frequently identiŽ ed facilitators. Study methods The included studies were all limited with respect to the generalisability of their results. had a response rate of at least 60%. because of political instability or high turnover of policymaking staff. Three studies interviewed a representative sample of health policy decision-makers drawn from a clearly described sampling frame. we may have missed relevant studies. Seven studies partially met our criteria (Table 2. several common factors were described across studies. including perceived political naivety of scientists and scientiŽ c naivety of policy-makers (8/24). but the relation between the included policy-makers and the system from which they were drawn was rarely described.17. . Discussion The strengths of this review include an extensive and systematic literature search. Five of the included studies did not report how many policymakers were interviewed. range 16–479). Most studies included a list of what the investigators perceived to be barriers to. even when similar factors were grouped together. explicit inclusion criteria and a systematic and transparent approach to collecting and presenting data from the included studies. The other 14 studies did not adequately describe the methods that were used. Research that included effectiveness data (3/24). Review article The most commonly mentioned barriers to the use of research evidence in policy-making were: . This can reduce mutual mistrust and promote a better understanding of policy-making by researchers and research by policy-makers. It can inform researchers about what the decision-makers consider timely. . Personal contact between researchers and policymakers (13/24). No factor was mentioned in more than 13 of the 24 studies as a facilitator or barrier. Measurement or assessment of the use and determinants of use of evidence was mostly descriptive. . Three of the 24 included studies met all of our methodological criteria:17.35 Facilitators and barriers Many of the studies used open-ended questions. only three responded and no additional references were identiŽ ed through these contacts. Absence of personal contact between researchers and policy-makers (11/24). . The remaining 19 studies interviewed a total of 2041 policy-makers (median 58. However. may not be easy to establish – for example. Every included study was read and appraised by at least two of the authors. Nonetheless.

9 they see decisionmakers as action. Direct use of evidence refers to speciŽ c use of research results. The ‘two-communities’ thesis sheds light on an important possible paradox inherent in the results: the factors that facilitate use of research may not necessarily be factors that researchers should seek to enhance. objective and open to new ideas. The results of this review support the ‘two communities thesis’. One study found that 40% of the use of evidence was direct.9 Another study found only 7% direct use.5 The different deŽ nitions of ‘use’ contribute to the difŽ culty of interpreting the results of the studies included in this review. P ersonal twoway communication may also be a necessary precondition for other facilitators. It is not surprising that decision-makers Ž nd it easier to use evidence that they have had an opportunity to in uence through two-way communication with the researchers. It is further supported by the most commonly mentioned barriers to the use of evidence. A health policy decision-maker who deŽ nes use as direct use is likely to report less use of research evidence than decision-makers who also include selective or enlightening use of evidence. deŽ ned as the ‘primary source in the policy formulation process’. From this perspective. If what is required for research to be used is that researchers do what the policy-maker wants them to do. jargon-ridden and irresponsible in relationship to practical realities. since these will always require some political judgement. this review is mostly based on information from one side of the two camps: the decision- 242 J Health Serv Res Policy Vol 7 No 4 October 2002 makers. . since the most commonly identiŽ ed facilitator of the use of evidence was personal contact between researchers and decision-makers. Different types of decision-makers: upper. Decision-makers may use evidence that supports their own ideology or their own political programme. Twoway communication between the two camps can facilitate a mutual understanding of a policy question and the kind of knowledge that is needed.36 This is often referred to as the ‘two-communities thesis’ or the ‘two cultures’. the included studies do not clarify whether decision-makers use only the research they want to use. in an enlightening way or selectively may vary in relation to: . relevant or good quality research. namely to be objective. it is more likely that Ž nancial constraints would be considered as a barrier. see themselves as responsible. It indicates that. This does not imply that research alone can answer policy questions. On the other hand. action-oriented and pragmatic. Decision-makers. The most frequent categorisation of different types of use in this review is direct (‘instrumental’ or ‘engineering’). absence of personal contact and mutual mistrust between decision-makers and researchers. it is more likely that international support would be considered as a facilitator. or focused and simple.38 Selective use is strategic. reliable and unbiased.and interest-oriented. selective (‘symbolic’ or ‘legitimating’) and enlightening (‘conceptual’) use of evidence. . the results should directly affect the solution without much adjustment. The degree to which evidence was directly used varied across studies. middle and lower level. without personal two-way communication it may be difŽ cult for researchers to understand what decision-makers regard as timely. Different issues: adoption versus implementation. or decision versus action. indifferent to evidence and new ideas. For example. This requires that researchers and decision-makers agree on which questions can be answered on the basis of research evidence and which require political judgement. not for researchers to assume the role of policy-makers. on the other hand. they see scientists as naive. Many of the studies have addressed different kinds of use and found that health policy decision-makers refer to use in all three of the above ways. and the relationship between the natural sciences and the humanities. involving use ‘to legitimate and sustain predetermined positions’. The aim of two-way communication is to help ensure that research appropriately informs judgements for which policy-makers are accountable. What is `use’ of evidence? Along with the ‘two-communities’ thesis. then research may fail to fulŽ l one of its most important functions. Enlightening use of evidence refers to research that helps to ‘establish new goals and bench marks of the attainable’37 and helps to ‘enrich and deepen understanding of the complexity of problems and the unintended consequences of action’. The `two-communities thesis’ The ‘two-communities thesis’ postulates the existence of two camps that lack the ability to take into account the realities or perspectives of one another. the question of what is meant by the concept of the ‘use’ of evidence is the most commonly discussed theoretical issue in the literature on knowledge utilisation. In a study focusing on an issue on which interest groups have a strong position.9.12 The study that found 40% direct use was in a context where 94% of all the research used . Two theoretical perspectives were dominant in the literature. The other perspective focuses on the concept of the ‘use of research’. problems with communication between researchers and policy-makers are large and difŽ cult to overcome.2. The degree to which evidence is used directly. in a study focusing on the use of an economic evaluation. Additionally. The Ž rst perspective draws analogies between the relationship of researchers and policy-makers. For example. In a developing country study.Review article Health policy-makers’ perceptions of their use of evidence on speciŽ c cases and factors were considered only in relationship to those cases. It addresses how the word ‘use’ may have fundamentally different meanings. Caplan et al found that social scientists see themselves as rational. Different types of policy questions: vague and complex. it would be more likely that decision-makers would consider alliances with interest groups as a facilitator or a barrier. if research results are relevant for a solution.

relevant and of high quality. Administrative Science Quarterly 1982. The utilization process: a conceptual framework and synthesis of empirical Ž ndings. Taylor SH. at best. researchers who wish to increase the use of the results of their research should: have personal and close two-way communication with decision-makers. The utilization of social science information by policy makers. van de Vall M. Research that conŽ rms current policy or supports community pressure may facilitate selective use. A third study. Cochet C et al. Lindblom CE.21. American Psychologist 1973. one of the main conclusions of this review is that there is. Factors in uencing the success of applied research.9 Self-reported selective use of research. Nelson CE. Ann Arbor. Stambaugh RJ. American Behavioral Scientist 1987. ed. 30: 569–577 7. insist on using methods that protect against bias.Health policy-makers’ perceptions of their use of evidence had been commissioned. Advice about how to improve the use of evidence by policy-makers is typically based on personal experience. If the results from the interview studies with decisionmakers are taken literally. A minimal set of conditions necessary for the utilization of social science knowledge in policy formulation at the national level. not included in this review.33 but none of the included studies did this systematically. Jonsson E. Social scientists and decision makers look at the usefulness of mental health research. The results of this review provide a context in which to consider such advice. Wertheimer B. the two theories that are commonly used in these studies provide useful insights for understanding the perceptions summarised in this review and other opinions about the use of evidence by policy-makers.39 Enlightening use of evidence is more difŽ cult to assess than direct use. which values both relevance and validity. The two-communities metaphor and models of knowledge use: an explanatory case study. Journal of Applied Behavioral Science 1982. Decision-makers may interpret this as no more than a question of whether they consider research to be of value. although it is commonly perceived as a problem by researchers and others. In: Lorion RP et al. if the aim is to increase appropriate (direct or enlightening) use of research rather than selective use. may evolve. 1979 3. References 1. eds. only limited support for any of the many opinions put forward in the literature on the use of research evidence by policy-makers. 1996: 165–181 2. based on 2041 interviews with policymakers. Dunn WN. Some studies suggested document analysis as a way to check whether these perceptions correspond to what was done. 1975: 1–63 10. Maederer CM. CT: Yale University Press. Conclusions From the results of this review of 24 studies that met our inclusion criteria. Cohen DK. One study found that 60% of decision-makers ‘used’ evidence in this way. as well as being difŽ cult to implement. At the same time. the policy-makers who were interviewed and the contexts in which they worked. Glaser EM. it is striking that some factors were identiŽ ed as frequently as they were. Using social J Health Serv Res Policy Vol 7 No 4 October 2002 243 . Granados A. Bero L. 1: 515–536 5. Caplan N. close to programme operations. Weiss JA. Future research should combine interviews with document analysis. International Journal of Technology Assessment in Health Care 1997. Johnson B. 27: 591–622 6. The studies included in this review address decisionmakers’ perceptions of their use of evidence. Trice HM. Using social policy research for reducing social problems: an empirical analysis of structure and functions. 13: 220–286 4. They should avoid getting involved in power and budget struggles and be aware of the high turnover of policy-making staff. and it is to be hoped will. Beyer JM. but not a basis for any strong recommendations. 18: 49–67 8. argue that the results of their research are relevant to current policy and demands from the community. particularly for research that is not commissioned. EUR-Assess Project Subgroup Report on Dissemination and Impact. This suggests that expecting direct use frequently may be unreasonable.12. Bolas C. Acknowledgement We thank Janette Boynton for help with literature searches. Of course. found that local administrators. Usable knowledge: social science and social problem solving. since researchers should. If decisionmakers and researchers start to talk together about the barriers and facilitators found in this review. The study with only 7% direct use did not cover examples of commissioned research. Personal contact may also facilitate selective use of research. use of evidence will be facilitated by timely and relevant research that gives decision-makers the answers that they want. Banta HD. Review article The methods that were used in the included studies and their diversity limit the extent to which any Ž rm conclusion can be drawn from this review. provide decision-makers with a brief summary of their research with clear policy recommendations. ensure that their research is perceived as timely. MI: The University of Michigan.19. these strategies will often not be effective. reported more direct use of evidence than federal-level decision-makers. They provide a menu of factors to consider. 28: 140–146 9. Roberts J. focus on commissioned research and clearly deŽ ne what is meant by ‘use’ of research. they may not always be able to get what they want. Weiss CH. include effectiveness data. New Haven. may be difŽ cult to elicit in interviews because it puts decisionmakers in a bad light. Psychology and public policy: balancing public service and professional need. In: Weiss CH. Washington. Knowledge: Creation. appropriate and cooperative way of working together. In fact. given the diversity of the included studies. Diffusion. Similarly. The use of social science knowledge in policy decisions at the national level: a report to respondents. Caplan N. Morrison A. DC: American Psychological Association. Bonair A. a more re ective.15. on the other hand. However. but there are limited empirical data to support these or other theories. Utilization 1980. supported by anecdotes.

Journal of Epidemiology and Community Health 2000. Guthrie KM. MI: University of Michigan Press. Hamilton: McMaster Institute of Environment and Health. Snow CP. Hohenadel J. eds. Explaining research utilization: beyond ‘functions’. 18: 49–67 36. Elliott H. Barriers to employment-related healthy public policy. Ann Arbor. Ross SE. Coping with complexity: An experimental study of public policy decision-making. Stoddard GL. Determinants of change in Medicaid pharmaceutical cost sharing: does evidence affect policy? Milbank Quarterly 1997. 2000: 00–03. Langer A. In: Weiss CH. Lavis JN. Gerhardus A. Evaluation of the Effective Health Care Bulletin. Ross-Degnan D. 1978 39. 54: 461–468 Trostle J. How are policy makers using evidence? Models of research utilisation and local NHS policy making. 16: 161–181 38. Toronto: Lexington Books. Stoddart G. Toronto: Lexington Books. Fox DM. Stoddard GL. 1977: 141–164 Weiss CH. Nilsson K. Knowledge: Creation. Grimes PS. DiNitto DM. Hilderbrand M. experience. Higgins A. Geneva: The Council of Health Research and Development (COHRED) Working Group on Research to Action and Policy. Diffusion. Palfrey TR. Better information. Lavis J. Truth test and utility test: decision makers’ frames of reference for social science research. 10: 140–155 Florio E. Walser BL. 2000: 77–85 27. Farrant MSR. DeMartini JR. In: Kinder DD.Review article 11. 9: 3–35 244 J Health Serv Res Policy Vol 7 No 4 October 2002 26. Charlwood P. 7: 77–87 35. 1998: 1–46 28. Bronfman M. Rudat K. Kielmann K. 2000: 1–44 34. 1993: 185– 208 McNeece CA. 45: 302–313 Weiss JA. American Sociological Review 1980. 2000: 1–26 29. 1997021. 15: 507–532 Ibbotson SL. Knowledge 1993. Evaluation Review 1991. How do researchers in uence decision-makers? Case studies of Mexican policies. Long AF. The use of economic evaluation in health care: Australian decision makers’ perceptions. 19. better outcomes? The use of health technology assessment and clinical effectiveness data in health care purchasing decisions in the United Kingdom and the United States. Lessons in research to action and policy: case studies from seven different countries. The use of information by policymakers at the local community level. Lavis JN. Evaluation and decision-making: the title vii experience. 14. Journal of Public Health Medicine 1999. 18. Johnson PJ. Explaining use of information in public policymaking. Woodward C et al. Langbein LI. Los Angeles: Center for the Study of Evaluation. 1977: 183–198 Patton MQ. 20. Politics and the professors: the great society in perspective. The utility of evaluation research for administrative decision-making. Seligman R. 13. 21: 29–36 Elliott H. 24. Cambridge: Cambridge University Press. Health Policy and Planning 1999. 1959 37. 25. Research to action and policy: combating vitamin a deŽ ciencies in South Africa. Evidence-based policymaking in the NHS: exploring the interface between research and the commissioning process. Administration in Social Work 1983. Eyles J. Geneva: The Council of Health Research and Development (COHRED) Working Group on Research to Action and Policy. McNeece CA. Fitzgibbon C. 31: 103–110 Oh CH. Bucuvalas MJ. Jacobs M. Making resource shifts supportive of the broad determinants of health: The P. 7: 77–87 Sunesson S. Toronto: McMaster University Centre for Health Economics and Policy Analysis. McMullan C. Factors in uencing the use of health evaluation research in congress.I. Philosophy of Science 1949. GreenŽeld L. New York: Milbank Memorial Fund. Hurley J. Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization 1997. Breanan NJ et al. The two cultures and the scientiŽc revolution. Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization 1996. Explaining the impact of policy information on policy-making. 23. 15. van de Vall M. Merton RK. Alkin MC. 7: 48–57 Soumerai SB. Washington. Experimental foundations of political science. 1974 . 15: 106–123 Boyer JF. Journal of Management in Medicine 1993. 22. Utilization 1989. 2000: 19–27 31. 10: 22–55 Oh CH. 2000: 54–66 32. ed. 21. Aaron HJ. 16.E. Administration in Social Work 1983. DiNitto DM. London: OfŽ ce For Public Management. 2000: 3–37 30. Bolas C. The role of health services research in Canadian provincial policy-making. 17. Simon J. Hyder A. Burns A. Mason J. 75: 11–34 Harries U. Using social research in public policy making. Health Policy 1995. The role of applied social science in the formation of policy: a research memorandum. The utility of evaluation research for administrative decision-making. Kosecoff J. Darling H. Health policy-makers’ perceptions of their use of evidence research in public policy making. CHSRF Report. In search for impact: an analysis of the utilization of federal health evaluation research. An initial evaluation of Effective Health Care bulletins as instruments of effective dissemination. The role of research in child health policy and programs in Pakistan. Journal of Applied Behavioral Science 1982. DC: Brookings Institution. Molyneaux-Smith L. 14: 103–114 Ross J. 12. Sheldon TA. Fortess EE. Pranger T. Popay J. Hamilton: McMaster University. Sanou A. Using social policy research for reducing social problems: an empirical analysis of structure and functions. Geneva: The Council of Health Research and Development (COHRED) Working Group on Research to Action and Policy. Moodley J. Rich RF. 1–28 33. Johnson PJ. Hamlyn L et al. University of California.

evaluation of worth and objectivity. the decision-maker (government level) and the evaluator (p. Reprinted from American Psychologist 1981. `relevant to the issues your office deals with’) on mental health. social welfare and public health (p. alcohol and drug abuse** (pp. 67 deputy assistant directors. N ˆ 204 (p. 142^143) 575 reported instances of use. Health training. N ˆ not stated Types of evidence (from which decisions were made) USA 1973^19749 (p. knowledge of sources. factors that influence use (p. attitudes about use. perceived or actual)/types of policy Social science research utilisation and policy formation (p. Awareness of research. 143) Upper-level decisionmakers in the executive branch of the US federal government (pp. alcoholism. 143). degree of confidence on findings. vii) and USA 1973^197410 Semi-structured surveys (p. 50) research within industrial and labour relations. 15% of the use of knowledge was on health or welfare problems (p. but published April 1980. loan forgiveness and training workshop (pp.244a J Health Serv Res Policy Vol 7 No 4 October 2002 51 federal. 50)** What was done. viii): 31 deputy under secretaries. N ˆ not given Health** (p. reference Table 1 Characteristics of included studies Review article Health policy-makers’ perceptions of their use of evidence . candid enough to be replicable. 187) *Revised version of paper presented in 1979 (p. 8) Perceived and self-defined use (p. five projects rated high for the success for ten administrators and persons `success’** (p. 141). neighbourhood health centres. what Personal interviews (p. 167. conducted by the government or both (p. where 385 involved primary research sources (p. N ˆ 60 (p. 303) 302) Open-ended interviews.e. ix) *Not stated. embodied in a clear and cogent report. 94% were either funded by the government. 49) **40 out of the 120 projects were on social welfare and public health Types of decisions Comments (hypothetical. v) USA 19802* and USA 198012 USA 1977#11 Personal interviews. 166. 8). based on 11 factors identified by the authors (pp. 303). 141) The conditions and functions influencing utilisation of social policy research (p.e. 168. vii). two-pagers) and are therefore difficult to compare on the effect variables continued *The interviews were done in either 1967 or 1968 (pp. N ˆ 155 Three key informants from each of the 20 studies were interviewed: a project officer. 50) Participants/level of policy-making 120 research projects on Perceived (retrospective) `client-oriented’ social decisions (p. both of the usefulness of social open-ended and structured sciences research (pp. methods and measurement procedures. health services delivery systems. information retrieval practice. 303) 20 national health care evaluations (case studies) with some systematic data collection. 140^141) **Success defined as project that was directed to the original objectives. interest in social indicators. with different questions for the decision-maker and the evaluator (p. 140) `Most of the examples respondents offered to illustrate knowledge applications really involved the application of secondary source information’ (p. 140) research projects under the most likely to be utilisers National Institute of Mental (p. 50). 165. 142^143) Hypothetical decisions (i. 143) Each decision-maker read two abstracts (two-pagers) out of 50 for actual social science research reports** (pp. 54 institute directors. 141) Principal investigator. 303) Perceived (self-reported) use with specific follow-up questions when use was reported (p. and that the findings were disseminated adequately (p. 168. Based on the same research project as USA 198012 (but partly on other parts of the data) **The decision-makers were given different `empirical’ interventions (i. 52 state (from ten states) and 52 local official decision-makers in upper-level positions (pp. 144). v. 1). ix) `To examine the nature and degree of utilization of federal evaluation research’ (p. 52 agency personnel. laboratory proficiency. made between 1971 and 1974. 171. regional and urban planning. 143) USA 19738* Researchers and policymakers that were responsible for incorporating the research results (p. other Five projects rated low and Perceived use helped and what hindered staff members. 5) Policy-makers’ perceptions Personal interviews. v. year conducted Objective (#year published). 49) Study design The Netherlands 1969^19717* Country. (pp. openended and structured (pp.

513)** Personal. 189^190. 50). 188^189). 143^144) To determine what type of information decisionmakers use. conceptual (it changed the way of thinking) and persuasive (used to convince others. enlightenment. N ˆ 172 (p. political/ conflict. 194)** To observe the impact that decision complexity. 188). Decision had to be made from available information within 1 hour 45 minutes (p. 193) Perceived (i. 187. 112) in community health care planning. 110). The six different versions of the cases varied in information complexity. MIT. N ˆ 100 Social workers. N ˆ 27 (p. one study they were familiar Evaluation studies from the with (p. and amount of information available. N ˆ 132 (p. and how ideology and interests influence the use of information (p. 193. 512). 152) Local planning committees and government employees in two rural communities in the Pacific Northwest (p. about a bulletin that had been screening for osteoporosis change of opinion and previously posted to them to prevent fractures (p. use of the E¡ective Health postal questionnaires directors of public health Care Bulletin (pp. 51. 515) Sources of information used Perceived use (p. 515).e. decisions (p. 152) These documents were also analysed by the authors (p. 516) **The interviews lasted 15^40 minutes (p. 52^54) Each decision-maker chose Perceived use (p. 188) Types of evidence (from which decisions were made) Review article Health policy-makers’ perceptions of their use of evidence .244b To determine some of the factors that influence administrators’ use of evaluative data in decisionmaking. reference Table 1 continued Two hypothetical decisions: one on the state’s Department of Health and Rehabilitative services and one on a methadone maintenance programme in New York City (pp. 187) USA 198213* Experienced public and private sector managers from six mid-career programmes at Harvard. 187. 50^51) and directors of planning/ commissioning in UK district health authorities or commissioning consortia (p. 152) Structured and open-ended Directors in community postal questionnaires mental health centres in (pp. 50)* confidence to act on (pp. 188) Participants/level of policy-making Congress members and their staff (p. semi-structured and open-ended questionnaires combined with a controlled experiment. 193) Types of decisions Comments (hypothetical. 112) *Reprinted from the Journal of Policy Analysis and Management 2: 66^87 (p. 514) E¡ective Health Care Perceived use of information *Respondents were questioned Bulletin No. N ˆ 42 Study design Country. 152) Structured. reported) use (pp. 110) Use and reasons for use/not Structured and open-ended District general managers. 79^80) Florida and North Carolina (p. 152). Consequences of complexity were examined by effects of information load and number of decision alternatives (pp. Each decision-maker had only one decision to make at only one level of complexity (pp. 51) Structured telephone interviews (p. 152) Conducted studies on programme effectiveness or impact during the last 12month period (p. perceived or actual)/types of policy Documents that the Perceived/retrospective respondents referred to. p. 507. 1991. semi-structured interviews (p. and to examine the types of decisions (p. 79) Different ways of using social research: instrumental. 81) Each decision-maker received a long memorandum describing a problem. management directors. had on the quality of the decision process (p. N ˆ 90 (p. and the assessment of the strategic planning process (p. 141. 109) What factors influence the use of evaluation research (on health and human services) in congress (p. 79). presenting evidence useful in analysing the problem. 80^81) continued *Use is defined here as both instrumental (direct implementation). 515) Department of Health and Human Services that deal with national health programmes (pp. 51^54) (pp. local and national politicians in 15 Swedish cities and communes (p. Stanford and the Washington Public Affairs Center (pp. 1. year conducted Objective (#year published). and posed a decision to be made based on real people and events (p. 52) information in the bulletin (pp. and tactical (pp. 112) Interviews (p. 185) **The experiment was based on six different versions of each of the two cases. on and hypothetical decisions. interactive. 508)* USA 1983#14 J Health Serv Res Policy Vol 7 No 4 October 2002 UK 199118 USA 1991#17 USA 199016 Sweden 1984^198515 (p.

pharmacy consultants and legislators (p.g. 103) Research publications and comments (p. 79) Two specific bulletins: June Hypothetical decisions 1997 and October 1997 (e. 42 in the Perceived decisions. 104)* Decision-makers’ awareness and use of economic evaluations in health care (p. although this has been going on for some time’) (p. the central ministry of health. N ˆ 34 Decision-makers from government. 1) (p. 1) UK 1994^199520. family planning some interaction between and immunisation in researchers and decision. screening. open-ended and structured (not stated if postal. statistical data and policy analysis (p.244c `To identify factors which facilitate or impede evidence-based policymaking’ (p. 77) Structured interviews (p. 104). reference Table 1 continued Perceived/retrospective decisions (p.governmental programmes makers’ (p. vitamin A. N ˆ 16 (p. since there are no actual decisions mentioned Only 2^3 pages out of 21 were relevant (p. 77) Senior managers in the Federal Department of Health.e. 29. New South Wales State. GPs topic guide) (p. transplants and scanning (p. 77) The impact that the E¡ective Telephone interviews. 106)* institutions and levels of responsibility’ (p. the Health Department (p. 29) `The relationship between health research (and researchers) and health policies (and policymakers) in four vertical health programmes in Mexico’ (p. 2). `it influenced the (p. 14) The role of research information in policy development (p. Housing and Community Services. 17) `Perceived use of economic evaluations’ (p. breastfeeding and polio (pp. 25ff) *Informants included Medicaid drug programme administrators.103). 13). The structured interviews built on the open-ended interviews (p. maternal mortality. directors or policy analysts.14). N ˆ 34 (p. 105) Health authorities** (p. knew that there was at least cholera. 105) Federal. the planning commission and provincial health authorities (p.21 Mexico 1994^199522 USA 1996#24 and USA 1997#25 J Health Serv Res Policy Vol 7 No 4 October 2002 UK 199827* Pakistan 199826 Australia 1995#23 Questionnaires (p. but it is 1998 or 1999 **Trusts and GPs are included in the study. 16)* Interviews. radiology. 25) Perceived/retrospective decisions (p. 25) Types of evidence (from which decisions were made) Review article Health policy-makers’ perceptions of their use of evidence . 29 and and researchers in nine response to a letter) different case studies at a local level in the NHS (pp. 105) (p. open-ended telephone interviews (p. 16^17) The role that research plays in experiences in linking research with policy (p. telephone or face-toface) (p. but are not relevant in this review *Open-ended interviews were used to obtain basic background information about the policy process. 10) No specific report. Policy project report) on the interface between R&D and commissioning at the health authority level (p. 105) continued *The date is not clearly stated. 104) General perceptions. demonstration project. year conducted Objective (#year published). 14) and semistructured telephone interviews (p. state and local policy-makers at different levels dealing with finance and services (p. 16). 103) To determine whether the process of (social science) information utilisation varies across different areas (i. service provision and financing) (pp. 34) Not stated Use of economic evaluations for pricing on drugs. N ˆ inadequately described In-depth interviews (using a Lead policy-makers. 31). Closest focus on child health problems such as co-trimoxazole. 77. 1) development of our policy. 30) Use of research Perceived** use of publications `where we evidence concerning AIDS. 29) USA 1993^199419 Participants/level of policy-making Open-ended in-depth interviews (p. 33) *Personal communication gave additional information on this issue **It is difficult to discern whether the decisions were perceived or hypothetical. 16) Key informants* in states that changed cost-sharing policies for prescription reimbursement limits and co-payments (p. 77). Healthcare Bulletin has structure not mentioned inside the NHS organisation (p. 15) Types of decisions Comments (hypothetical. N ˆ 67 Study design Country. N ˆ 28 Open-ended (`in-depth’) `Researchers and policyinterviews with a common makers from different interview protocol (p. perceived or actual)/types of policy Social research (p. N ˆ 479 (p. but `infotype’ refer to programme evaluation.

reporting on actual use of research. 8. 20) Perceived decisions on cross-sectoral resource allocations (p. 13) Perceived use (p. USA: 13 public officials from 11 states. N ˆ 48 (p. 1) Study design Not stated* Not stated Not stated Workshops and research papers that the authors had provided Ministry of Health officials with during previous years (p. semi-structured Policy-makers in the health information about the health and open-ended telephone and employment sectors of consequences of interviews (pp. i) How UK and US purchasers Semi-structured telephone value and use research interviews (p. 11) Types of evidence (from which decisions were made) GPs. 18) (p. 13^16. and under what structured interviews and variety of key health policies conditions health services surveys (pp. 4. research and development. 18^19) *Stated date for working paper is March 2000 *`Shared care’ is about improved contact and collaboration between health centre staff and mothers (p. perceived or actual)/types of policy Access to and use of health Perceived use (p. 19^ 20) Barriers against and Semi-structured interviews Key informants that were mechanisms for improving (p. seven public health physicians. five health regions. 13. MIT. 22). 9) all three levels of Canadian unemployment and government. National Health Service. 5) *It is not stated what kind of evidence was used. semiKey informants within a how. 4) assessing health technology and clinical effectiveness (pp. 5) To understand whether. 4). 4) Services and various sectors (p. implies that evidence was used (pp. R&D. reference Table 1 continued Review article Health policy-makers’ perceptions of their use of evidence . 55) To assess the influence of Structured. N ˆ not given decision-making about `Shared Care’* for childhood illness (pp. ii). 13). N ˆ not stated the vitamin A research^ policy^action connection (p. financial arrangements. programme delivery arrangements and programme content (pp. 9) A case study on how Semi-structured interviews Ministry of Health officials research played a role in (p. the system in Prince Edward Appendix 1) Department of Social Island (p. 21) Not stated Perceived/retrospective usefulness of past research information about the health consequences of unemployment and insecurity on policy-making (p. 3) Types of decisions Comments (hypothetical. but references to direct use of evidence in decision reports. three consultants. all regions. 5) Actual and perceived decisions within the fields of health services jurisdiction. 4) A case study of crossSemi-structured and open. NHS. 5) non-governmental organisations (p. Both personal. all sectors and management levels (p. N ˆ 22 (p. 55) Perceived use of the provided scientific information in decisionmaking (p. nine GPs. general practitioners. 1) technology assessment/ clinical effectiveness information (pp. 55). N ˆ 55 (p. 20) *Stated 1999 (p. Canada 200033 Canada 200032* South Africa 200031 Burkina Faso 199930 Canada 1998^199929 UK and USA 1998#28* Country. nine private sector purchasers. 5).Key informants holding sectoral resource allocation ended interviews and focus different positions within the in the human services groups (pp. 11) in two Canadian provinces research plays a role in (Ontario and provincial policy-making Saskatchewan) (p. 7. 55) relevant role-players (p. 5. year conducted Objective (#year published). N ˆ 58 (p. 20) (p. and executive insecurity on policy-making directors of ten Canadian (p. Massachusetts Institute of Technology. v.244d J Health Serv Res Policy Vol 7 No 4 October 2002 Participants/level of policy-making UK: 14 non-medical health authority purchasers.

with summed score ranging from 0 to 18 (p. 187^188). `. 28. 145) Not adequately described *Complete refers to the 27 representatives in the two planning committees in the two communities. 51) UK 199118 60% (p. 35. 518. reference Table 2 Study methods Review article Health policy-makers’ perceptions of their use of evidence . 187^188) SS: random sample* (pp. 195^208) Both quantitative descriptions. vi). 145. No further generalisation can be made on the basis of this sample Complete **Weighted on 3-point and 5-point scales *SF and SS complete if the study’s focus considered only directors in Florida and North Carolina. 110) SF: Partial (pp. 143) USA 19802 and USA SF: partial (p. 514) SF: complete (pp. partly snowballed (p. The qualitative rating comes from listing of most mentioned characteristics of usefulness **The five dimensions are listed in Table 3 under facilitators Partial *The interviews appear to have been made between 1973 and 1976 Partial It is stated that `this report represent only a first summary of some of the highlights and more important findings’ (p. 169ff. . 51) 73% (p. 51^54) Quantitatively descriptive. 35. 517^523) and multiple regression (pp. 170^172. 513^514) SS: random sample (p. . 519^522) Qualitative and descriptive (p. 82) Listing of frequencies (pp. 51) Descriptive No statistical technique Measurement of use of evidence 95% (p. 66) three decision levels. as either: instrumental. 146^150) Quantitative listing and ratings (pp. rated and weighted and qualitative* (pp. enlightenment. 167) Not stated Both quantitatively rated and weighted (pp. 79)* SS: not stated (p. 33. 194) Partial *Both a 4-point (p. 82^86) continued Not adequately described The authors mainly discuss and demonstrate different ways of utilising research. 27) Descriptive No statistical technique SF: partial (p. 25. 523^527) Percentages (pp. 79) Volunteered (p. 36) and qualitatively described (pp. moderately or not important) (p. viii) The Netherlands 1969^19717 USA 19738 J Health Serv Res Policy Vol 7 No 4 October 2002 SF: complete (p. partial. SF: not stated (p. 149) Descriptive listing of ratings (pp. 143) SS: snowballed. 1^4) ratings (pp. using research ``instrumentally’’ or for ``enlightenment’’ purposes meant different things in different settings and different organizational positions’ (p. 168) and a 5-point weighted rating scale (pp. 304) Quantitative listing of the decision quality of Listing of frequencies evidence ^ use on six different levels of complexity (pp. and if there were only one director in each Community Mental Health Center Partial *The 132 decision-makers were randomly distributed to two of 50 case studies and six different levels (p. 303) 90% (p. political/conflict. 196^198) (pp. 112) Qualitative and descriptive (pp. 51) SS: random sample (p. 174) and 25 variables analysed using factor analysis to develop five dimensions** (p. interactive. vii) Sample frame (SF)a and sample strategy (SS)b Study. 110) SS: complete* (p. 513) USA 199117 100% (p. 173. 303).110) Not stated 65% (p. 79)* SF: not stated SS: not stated Measurement of determinants of use A quantitative 4-point score weighted according to Two-way ANOVA (p. 144). or tactical (p. partly random (p. 28. 188) Quantitative rated descriptions (pp. viii) SS: not mentioned. or not adequately described) and comments Descriptive listing of weighted ratings (pp. 36) and MCA (multivariate regression) analysis (p. 51^54) Complete Qualitative Frequencies and rated importance (as very. 24) Not stated Not stated Response rate USA 199016 Sweden 1984^ 198515 USA 198314 USA 198213 SF: partial (p. 167) 198012 SS: not stated USA 197711* USA 1973^19749 and USA 1973^ 197410 (p.244e SF: not stated SS: not stated SF: not stated SS: not stated SF: partial (p. Subsequent reports not found Not adequately described Not adequately described Summary of description of methods (complete. rated and weighted** (pp. Descriptive listing of 33.

18) Qualitative Qualitative Qualitative Qualitative Measurement of determinants of use Partial *Called `a purposive sample of Canadian policy-makers’ (p.244f SF: partial (p. 22) Qualitative and descriptive (pp. partial ˆ incomplete information. 11^14. subjective view on key variables of policy change. 16^17) Not adequately described Not adequately described Not adequately described Not adequately described Not adequately described *Purposive sample (personal communications) Not adequately described *Called `purposive sampling frame’ (p. ii. 8) Not adequately described Not adequately described Partial *A qualitative validation of the interviews is described (pp. but with a good description of SS and SF Not adequately described Only Medicaid programmes with specific cost-containment policies in the last several years (p. general perceptions and impact) Descriptive (pp. 82) Quantitative regression and a path model (p. 12^22) Qualitative and descriptive (i. 26) Both quantitative and qualitative descriptions of facilitators (pp. 77) SS: not stated SF: partial (pp. subjective view on key variables) (pp. 104. 26) Qualitative Qualitative Qualitative Qualitative Qualitative Qualitative Quantitative regression and a path model (p. ii) USA 1993^199419 UK 1994^199520. 105^108) Not stated Response rate Qualitative Frequencies (p. 465) SS: snowballed (pp. 1^2) SS: not stated (pp. 16) SS: not stated* SF: not stated SS: not stated SF: not stated SS: not stated SF: partial SS: not stated* SF: not stated SS: snowballed (p. 80. reference Table 2 continued Review article Health policy-makers’ perceptions of their use of evidence . or not stated. 4) SS: not stated SF: partial (p.e. 35^37) Descriptive Qualitative descriptions and quantitative listings (pp. 78. 16) 100% (all those Qualitative and descriptive (p.21 Mexico 1994^199522 Australia 199523 J Health Serv Res Policy Vol 7 No 4 October 2002 Measurement of use of evidence 88% (22/25) (p. 29) Not adequately described Summary of description of methods (complete. 106^108) Not stated Quantitative and descriptive (pp.e. 18) 42% (p. Sample strategy: complete sample. 32^34) Qualitative and descriptive (p. 30^31) SF: partial (p. 18^30. 31ff) approached for interview agreed to the interview) Not stated Qualitative and descriptive (i. 461. or not adequately described) and comments b Sample frame: complete ˆ the sample was described in relation to all the decision-makers in the context of the study. 16.15) SS: not stated SF: partial (pp. 465) SF: not stated* SS: not stated* SF: partial SS: not stated* SF: partial (pp. a Canada 200033 Canada 200032 South Africa 200031 Canada 1998^199929 Burkina Faso 199930 UK and USA 199828 UK 199827 Pakistan 199826 USA 199624 and USA 199725 Sample frame (SF)a and sample strategy (SS)b Study. 18) Qualitative and descriptive (i. 1^2) SF: partial (p. any mention. subjective view on awareness. 461. partial. 104) Partial. i. not stated ˆ no description. 23) Not stated Not stated Not stated Not stated Not stated Not stated Not stated Both quantitative and qualitative descriptions of barriers (pp.e. random sample. snowballed sample. 30^31) SS: combined random and snowballed (pp. no ranking) (p.

155. 106) Absence of personal contact between policy-makers and researchers (p. 31^32) Poor quality of research (p. 33) Personal contact between researchers and decision-makers (pp. 33) Mutual mistrust between policy-makers and researchers Absence of personal contact between policy-makers and researchers Political instability or high turnover of policy-making staff (pp.21 UK 199118 USA 1993^199419 USA 199117 USA 199016 Sweden 1984^198515 USA 198213 USA 198314 Good quality research Research that confirms current policy or endorses self-interest Timeliness and relevance of the research Research that includes a summary with clear recommendations None of the facilitators were mentioned in the other studies Community pressure or client demand for research (pp. 32^34) Timeliness and relevance of the research (pp. 111. 519) Personal contact between researchers and decision-makers (p. 145) Facilitators to use of research identi¢ed in three or more studies USA 19802 and USA 198012 USA 197711 USA 1973^19749 and USA 1973^197410 (p. 27) Personal contact between researchers and decision-makers (p. 155. 143) Lack of timeliness or relevance of research (pp. 33. 146. 21) Personal contact between researchers and decision-makers (p. 520) Research that includes a summary with clear recommendations (p. 143) Personal contact between researchers and decision-makers (pp. 158) Timeliness and relevance of the research (p. 54) Research that confirms current policy or endorses self-interest (p. 519. 119^122) Power and budget struggles (p. 149. 82. 27) Timeliness and relevance of the research (p. 82) Research that confirms current policy or endorses self-interest (pp. 28) Absence of personal contact between policy-makers and researchers (pp. vii) USA 19738 The Netherlands 1969^1971 Study. 143) Research that confirms current policy or endorses self-interest (p. 150) None of the barriers were mentioned in the other studies Not stated Power and budget struggles (pp. 109^113) Personal contact between researchers and decision-makers (p. 65) Personal contact between researchers and decision-makers (p. 118) Good quality research (p. 33) Lack of timeliness or relevance of research (p. 158) Absence of personal contact between policy-makers and researchers Mutual mistrust between policy-makers and researchers Power and budget struggles Mutual mistrust between policy-makers and researchers (p. 106^108. 108) Good quality research (p. 32^34) Personal contact between researchers and decision-makers (p. 107^111) continued Not stated Lack of timeliness or relevance of research Absence of personal contact between policy-makers and researchers (pp. 13^14. 108) Research that includes a summary with clear recommendations (p. 36) Timeliness and relevance of the research (p. 142. 55) Good quality research (p. reference Table 3 Main results Poor quality of research (p. 159^160) Absence of personal contact between policy-makers and researchers (pp. 108) Mexico 1994^199522 USA 199624 Australia 199523 UK 1994^199520. 27) Power and budget struggles (pp.244g 7 J Health Serv Res Policy Vol 7 No 4 October 2002 Personal contact between researchers and decision-makers Timeliness and relevance of the research Research that includes a summary with clear recommendations Research that confirms current policy or endorses self-interest Community pressure or client demand for research Inclusion of effectiveness data (pp. 107) Timeliness and relevance of the research (p. 115) Poor quality of research (pp. 84) Inclusion of effectiveness data (p. 114) Research that confirms current policy or endorses self-interest (p. 29) Personal contact between researchers and decision-makers (pp. 143) Barriers to use of research identi¢ed in three or more studies Review article Health policy-makers’ perceptions of their use of evidence . 142) Timeliness and relevance of the research (p. 149) Research that includes a summary with clear recommendations (p. 54) Research that includes a summary with clear recommendations (p. 28) Absence of personal contact between policy-makers and researchers (p. 149) Research that includes a summary with clear recommendations (p. 141) Community pressure or client demand for research (p. 145) Mutual mistrust between policy-makers and researchers (p. 141. 60) Community pressure or client demand for research (p. 106) Lack of timeliness or relevance of research (p. 519) Research that includes a summary with clear recommendations (p. 106) Political instability or high turnover of policy-making staff (pp. 33) Good quality research (p. 152) Political instability or high turnover of policy-making staff (pp. 519) Timeliness and relevance of the research (pp. 148) Timeliness and relevance of the research (p.

82) Poor quality of research (p. 82) Power and budget struggles (p. 5. 3. reference Table 3 continued Power and budget struggles Lack of timeliness or relevance of research (pp. 80. 21) Absence of personal contact between policy-makers and researchers (p. 78.244h Not stated Research that includes a summary with clear recommendations (p. 80) Mutual mistrust between policy-makers and researchers (p. 17^21) Personal contact between researchers and decision-makers (p. i) Burkina Faso 199930 South Africa 200031 Canada 200032 Canada 200033 UK and USA 199828 Canada 1998^199929 Not stated Facilitators to use of research identi¢ed in three or more studies Timeliness and relevance of the research (p. 62) Timeliness and relevance of the research (p. 8) Research that includes a summary with clear recommendations (p. 61) Power and budget struggles Mutual mistrust between policy-makers and researchers Poor quality of research (p. 23) Mutual mistrust between policy-makers and researchers (p. 12^17) Absence of personal contact between policy-makers and researchers (pp. 61) Political instability or high turnover of policy-making staff (p. 8. 33^34) Political instability or high turnover of policy-making staff (p. 62) Not stated Personal contact between researchers and decision-makers (p. 63) Absence of personal contact between policy-makers and researchers (p. 26) Not stated Absence of personal contact between policy-makers and researchers Poor quality of research Lack of timeliness or relevance of research (pp. 32) UK 199827 Pakistan 1998 26 Study. 22) Mutual mistrust between policy-makers and researchers (p. 13) Personal contact between researchers and decision-makers Good quality research Research that includes a summary with clear recommendations Timeliness and relevance of the research Inclusion of effectiveness data (pp. 82) Lack of timeliness or relevance of research (pp. 12) Barriers to use of research identi¢ed in three or more studies Review article Health policy-makers’ perceptions of their use of evidence J Health Serv Res Policy Vol 7 No 4 October 2002 . 61) Lack of timeliness or relevance of research (p. 80) Lack of timeliness or relevance of research (pp.