Available online at www.sciencedirect.com
International Journal of Drug Policy 20 (2009) 70–75
Research paper
How do drug policy makers access research evidence? Alison Ritter a,b,∗ a
Drug Policy Modelling Program, National Drug and Alcohol Research Centre, University of New South Wales, Sydney, 2052, Australia b Regulatory Institutions Network, The Australian National University Australia
Received 2 May 2007; received in revised form 22 November 2007; accepted 23 November 2007
Abstract Background: Policy decisions are informed by a number of factors: politics, ideology and values, perceived public opinion, and pragmatic constraints such as funding. Research evidence is also used to inform decision-making but must compete with these other inputs. Understanding how policy makers access research evidence may assist in encouraging greater use of this evidence. This study examined the sources of research evidence that Australian government drug policy makers accessed when faced with their most recent decision-making opportunity. Method: Drug policy makers across health and police government portfolios were interviewed (n = 31) and asked to report on the sources of research evidence used in their most recent decision-making. Results: Nine sources were reported, the most frequent of which were seeking advice from an expert and consulting technical reports. Accessing the internet, using statistical data and consulting policy makers in other jurisdictions were used in about half the cases. The least frequently used sources were academic literature, relying on internal expertise, policy documents and employing a consultant. Conclusion: There is a tension between the type of information source most suited to policy makers – simple, single-message, summative and accessible – and the types of information produced and valued by researchers—largely academic publications that are nuanced and complex. Researchers need to consider the sources that policy makers use if they wish their research to be utilised as one part of policy making. © 2007 Elsevier B.V. All rights reserved. Keywords: Policy making; Research utilisation; Sources for research
Introduction Good policy is presumed to be based on a solid evidencebase. The ways in which research can provide such a solid evidence-base are contested, both in terms of the types of evidence that are valued and in terms of the ways in which evidence is taken up in the policy making process. The ways in which research evidence informs policy decision-making, especially in the social sciences, has been a source of consternation and examination for many years (Callahan & Jennings, 1983; Hanney, Gonzalez-Block, Buxton, & Kogan, 2003; Nutley, Walter, & Davies, 2003). Weiss has studied the application of research evidence in policy decision-making and identified a number of different ways in which research is used, including instrumentally for direct problem-solving; for political or strategic purposes; ∗
Tel.: +61 2 9385 0236; fax: +61 2 9385 0222. E-mail address:
[email protected].
0955-3959/$ – see front matter © 2007 Elsevier B.V. All rights reserved. doi:10.1016/j.drugpo.2007.11.017
and more generally to advance the knowledge base, what Weiss termed ‘enlightenment’ (Weiss, 1979; Weiss & Weiss, 1981). Although both researchers and policy makers are committed to improving the use of research evidence in policy, there appear to be any number of barriers. One of the inherent barriers is that research findings, especially in the social sciences, are often equivocal, with marginal or uncertain impact (Coleman, 1991; Gregrich, 2003). There is a large quantity of research information that is often highly specific (Anderson et al., 2005). The translation of specific hypothesis-testing research into general policy principles is a complex and highly skilled task (Brownson, Royer, Ewing, & McBride, 2006; Lavis et al., 2005). Rapid change within the policy environment also means that research can become less relevant than originally anticipated (Weiss & Weiss, 1981). From the individual researcher’s perspective, one significant barrier is the dissonance between what is rewarded within the academic community and what is valued by pol-
A. Ritter / International Journal of Drug Policy 20 (2009) 70–75
icy makers (Denniss, 2006; Weiss & Weiss, 1981). Research career structures are geared towards and reward academic publication, rather than policy-oriented publication; spending time writing submissions for parliamentary enquiries or providing media briefings is not generally recognized as valuable work within the academy. Barriers are also multiple from the perspective of the policy maker (Brownson et al., 2006; Gregrich, 2003; Secker, 1993; Weiss & Weiss, 1981). Policy makers do not necessarily have ready access to research results, in terms of where they are published, the absence of simple summaries and the lack of translation into policy-accessible language. Research does not necessarily address the most pressing questions for policy-makers. The timing of the release of research results does not often take into consideration decision-making cycles such as budget and legislative processes. In addition to these various barriers within the research and policy-making environments, consideration must also be given to the very nature of policy making. The technical/rational approach describes the conventional image of policy making, and comprises a series of cyclical steps to policy making: identify the problem, identify the causes, develop options, analyse options, select intervention, implement intervention, and evaluate (Bridgman & Davis, 2000; Edwards, 2005). Technical/rational policy making has research as a direct input. This type of policy making, however, is relatively infrequent. A more common approach is that of incrementalism (Lindblom, 1959, 1979). Incrementalism sees typical policy making as a process of incremental, small adjustments to existing policies, based on ‘successive limited comparisons’. There are also substantial influences upon policy other than simple decision-making driven by research evidence. These are the politics, power and pressure groups and opportunistic policy windows that can drive decisionmaking. Models such as the Advocacy Coalition Framework (Sabatier, 1988), the Iterative model (Weiss, 1983), and the Multiple Streams model of policy making (Kingdon, 2003) accommodate the diverse influences of politics, power and interest groups. In addition, as noted for example by Room (2005) and Fitzgerald (2005), drug policy carries great symbolic value and in this context, evidence can be largely irrelevant. In light of the multiple processes associated with policymaking, it becomes clear that research evidence is but one contributory factor in policy-making. Research evidence competes with political processes, power and interest groups, and symbolism (Hanney et al., 2003; Weiss, 1983). It is in this light that the work reported herein was conducted. Given that research evidence is but one input to policy making, it seems important to maximise the chances that research evidence will at least be available and accessible such that the likelihood of policy being framed by research evidence is heightened. The study reported in this paper sought to better understand how policy makers access research evidence when faced with a decision-making opportunity.
71
Method Drug policy units in Australia are located in both health and police bureaucracies; and at state and federal levels of Government, representing a total of eight jurisdictions. The senior bureaucrats in health and police for each jurisdiction were approached to be interviewed. All agreed. Other team members were also interviewed based on advice from the jurisdictional seniors themselves. The interviews covered three domains: priority areas for drug research and policy for Australia, difficulties associated with policy decision-making and the use of evidence; and third, the sources they used to obtain research evidence. This paper focuses on the last of the three domains (the results for the priority areas can be found in Ritter, 2007). A total of 39 people were interviewed, in 22 separate interviews. Of these, 31 people (in 18 interviews) were asked about the sources of research evidence. More health (n = 22) than police (n = 9) respondents were interviewed. The 31 respondents were asked to recall the most recent policy issue that they had considered, preferably where a decision was required. They were not asked to disclose the content to the interviewer but to use it to reflect on what sources they accessed to assist them in their decision-making. For most respondents, this prompt sufficed and produced an immediate list of approaches or sources that they had used. In addition, respondents were asked what sources they considered to be most helpful to them. Extensive notes were taken during the interviews and then transcribed immediately afterwards. The responses were then read through and clustered into nine categories. Once the categories had been determined, the transcripts were re-read and the responses coded against the nine categories. The project was approved by the University of New South Wales Ethics Committee. Interviews were conducted between September and November, 2006.
Results Context of policy decision-making Many policy makers commented on the context in which policy decisions are made. In the vast majority of circumstances, policy decisions need to be made very rapidly—often within hours or a day. For example: “We need to make a quick judgement” “Within four hours we need a policy position” A distinction was drawn between this most common scenario – where a rapid decision is required – and the alternate scenario where there is a considerable amount of time and planning involved in decision-making. In cases of the latter scenario, policy makers engage in a comprehensive approach to reviewing and synthesising the research evidence. Because this scenario is infrequent, the responses reported in this paper
72
A. Ritter / International Journal of Drug Policy 20 (2009) 70–75
Table 1 Reported resources accessed by policy-makers (N = 31, 18 group interviews) Source for research evidence Consult an expert Consult technical reports, monographs and bulletins Access the internet Use statistical data Consult policy makers in other jurisdictions Use academic literature Use internal expertise Use government policy documents Employ a consultant
N 14 14 8 8 7 5 4 2 2
pertain to rapid decision-making. This context substantially limits the extent to which policy makers can review and synthesise research evidence. Respondents noted that they have an excessive amount of information to deal with in a very limited time. As stated by one, they are “skimming the surface” and cannot conduct a “thorough analysis”. The resources they access need to be readily to hand, in the office already or available through the touch of a keyboard. Many respondents also commented on the regular appearance of large volumes of information and research evidence. The opportunity to read and digest information as it comes to hand is very limited. For example: “I read 1–2% of what comes across the desk”. This means that any new reports or resources are left unread until such time as they may be recalled to be of use in a decision-making context. Sources of research evidence Nine key sources were mentioned by policy makers (Table 1). The vast majority of policy makers reported consulting an external expert during their most recent decision-making process. Some relied on this method exclusively. Policy makers noted that they had a small group who they would consult (an “inner circle”). Trust was a key theme—the expert needed not only to be trustworthy both in terms of being a credible source of knowledge, but also in terms of being known as appropriately discrete. Respondents noted that it is often the same small group of people that they call upon, as reflected in the comments about the importance of a trustworthy inner circle. As noted by respondents, the advantages of this source were both the speed with which information could be obtained (one telephone call), and the ability of an expert to synthesise the research information quickly and concisely and provide an opinion. It was not stated explicitly that consulting an expert obviated the need to seek research evidence from academic sources (as the expert would have that to hand) but it was implied that the skills of the expert were in integrating and interpreting the sometimes contradictory academic research evidence.
It was noted that the experts do need to have relevant expert knowledge. There appears to be a tension between having a small group of trusted experts versus consulting the most knowledgeable expert in any one particular situation. For example: “[I] talk to people who know something, and to those who don’t but have strong opinions”. This comment also serves to highlight that policy makers are not just reviewing the research evidence when making decisions but also the political viability, degree of community support and other non-evidentiary aspects to decision-making. Equal to consulting an expert was the use of technical reports, monographs and bulletins. All the policy units had an internal ‘library’ of reports and monographs that they consulted when required to make a decision. As noted earlier, these resources are rarely read when they originally come to hand, and when they are used in policy decision making, they are still likely to only receive a cursory examination. For example: “A policy maker only reads the executive summary” The third most frequently mentioned source used by policy makers during their last decision-making opportunity was the internet, notably “Google” and clearinghouses of drugrelated information. Almost half of the respondents used the internet in their last policy decision. It was noteworthy that generic search engines such as Google were frequently mentioned. More specific specialist websites were also mentioned; for example, the websites of national research centres and national clearinghouses. It did seem that people had their ‘favourites’. For example, one respondent who had worked in the UK continued to use the UK Home Office website on a regular basis. The extensive use of Google probably reflects the ease of access and the experience of finding useful resources through that means. However, one respondent cast the use of Google in a different light: “I use Google. It keeps it centred on the real world, I need to know this to see if it [a policy position] will fly or how to counter what will be said”. In this sense it is a resource that politicians and the general public can access and where they may source their information from. Hence for this policy maker it was important to know what information existed in the public domain. The next most commonly mentioned source was statistical data contained in reports that were held by the policy unit, or which were available in raw form. About half the respondents mentioned using statistical data. Specific data sources mentioned included the Australian Bureau of Statistics demographic data (Australian Bureau of Statistics, 2006), the National Drug Strategy Household Survey (Australian
A. Ritter / International Journal of Drug Policy 20 (2009) 70–75
Institute of Health & Welfare, 2005), the Illicit Drug Reporting System (an annual survey of injecting drug users O’Brien et al., 2007), the Australian Crime Commission report (Australian Crime Commission, 2006), the Australian Secondary School Students Survey (White & Hayman, 2006), and various police and treatment datasets. One ready source for policy makers was to contact their counterparts in other jurisdictions and ask how they approached the problem or issue. This was mentioned in 40% of the interviews. For example: “[Ask] colleagues from other jurisdictions–other people have faced similar problems” Academic literature was used in 28% of cases. Respondents referred to a number of reasons for not accessing the academic literature. Most obviously, not all policy units have access to libraries or online journals so this is a fundamentally limiting factor. In addition, not all are trained in how to use academic search engines (such as Medline or PsychInfo). Lastly, respondents mentioned that the academic literature was not only nuanced and complex but also frequently contradictory. “When searching for research on the fly, I don’t use Medline or systematic reviews because they can contradict each other” Less commonly mentioned was reliance upon one’s own judgement and experience. For example: “[We have] expertise within the policy unit in government–inhouse capacity” Most of the respondents had been in their positions for many years, hence they were a very well-informed group of policy makers. Policy officers are on many national committees that debate substantive policy issues and this was described as a significant mechanism for bureaucrats to gain information and knowledge about a diverse set of issues over time. Police also mentioned using their own confidential intelligence sources. In two interviews, the use of existing policy documents to facilitate decision-making was mentioned. For example: “I use the NDS [National Drug Strategies] and stay within these, always refer back to these source documents for any response, because I can rely on approved phrases and words” The need to repeat simple statistics frequently was noted. Finally, in two interviews, the use of a consultant was noted in the last policy decision. There were differences between the health and police drug policy makers. No police mentioned the use of academic literature or a consultant. Police were most likely to use
73
technical reports, in-house data and to consult other jurisdictions. Health drug policy makers were most likely to consult an expert, use the internet, use technical reports and access academic literature. What types of research information are most helpful? Respondents were asked about the types of research information most helpful to them. All respondents stated that they preferred summative information, in the form of bulletins, dot points or single page summaries. This summative information was most useful when it considered multiple studies or sources. Importantly, one respondent pointed out that even though the summary bulletins were most valuable, a references list was crucial for those wanting more information and to substantiate the bulletin content. A compendium of the latest advice on ‘hot topics’ was also raised by more than one respondent. Structured around key questions that might arise, and regularly updated, these ‘hot topic’ bulletins would serve to summarise current and pertinent research. Whilst the above two points reflect the need for policy makers to have simple and concise information, there was also a call by respondents for greater analytic capacity. Policy makers want to find ways of using research that does not ignore the complexity. As stated pejoratively by one: “At present we use one report, dummy it down, and ignore all other data” Thus, one key challenge is to synthesise and collapse often diverse pieces of information and not forgo the sophistication of the research but still produce useable data for a policy maker. Finally, respondents commented on the quality of the information supplied by researchers. The sources need to be authoritative, credible, thorough and; “. . .not be hijacked by any one interest group”.
Discussion There are many inputs to decision-making – politics, values, opinion – along with research evidence. Research evidence must compete with these others inputs. As such, ensuring the evidence is available and accessible is one way to increase the likelihood of its use within the decision process. Policy makers operate in a highly pressured environment, where decision making most frequently occurs rapidly. The use of research evidence within the policy process is potentially constrained by this time pressure. A key feature of the sources of research evidence accessed by drug policy makers in their most recent decision was the degree to which they were accessible. All the most frequently used sources were highly accessible.
74
A. Ritter / International Journal of Drug Policy 20 (2009) 70–75
Policy makers generally articulated a preference for simple, uncomplicated information and indeed, this is reflected in their choice of sources. Researchers generally produce more complex and nuanced work. It is encouraging that respondents in this study were aware of this distinction and that at least some are actively seeking ways to better integrate the complexity of an issue into policy decision-making. One theme that emerged was the capacity for greater sophistication and analysis of divergent evidence-bases. One of the frustrations expressed by policy makers is that when they do have time to consult multiple sources, the sources often contradict one another, especially in the academic literature. This may go some way to explaining why telephone contact with an expert is so highly utilised. Policy makers are seeking sources that assist in the distillation of diverse research evidence—making it digestible and useable. The most common source of research evidence was through contacting an expert. The relationship between individual researchers and policy makers has been highlighted in the literature (for example Lavis et al., 2005; Secker, 1993). The most influential opportunities are when one is contacted for advice; being available and responsive will ensure that the evidence reaches the policy maker. On the other hand, caution may be required for a researcher who is contacted for advice about a topic not within his/her specialist domain. The temptation to be responsive should be balanced with acknowledgement of the limits of one’s expertise. For those researchers not within an “inner circle” a secondary route for the input of evidence is provision of their research results to the experts who will be contacted. The tension between academic reward and policy relevance is highlighted in these results. Policy makers used technical reports and generic internet search engines. Research evidence freely available via the internet is likely to enhance the chances that it will be used to inform policy. Researchers are thus in an invidious position in relation to deciding whether to invest time in writing technical reports as well as papers for peer review journals. These are not necessarily mutually exclusive (for example, a researcher can send a copy of the published journal article to the policy unit) but this does not attend to the apparent difficulties for policy makers with academic literature. At the same time, the demand for authoritative and rigorous evidence means that policy makers need researchers to publish in academic circles, which is directly linked to being regarded and consulted as an ‘expert’. In light of the finding of differences between police and health respondents, further research could examine in greater detail the extent of difference between policy makers in their access to and use of research evidence. Different policy issues may also be more or less amenable to research evidence. Ultimately, drug researchers are interested in the ways in which their work can inform and influence government decision-making. Previous research has highlighted the multiple factors at play in policy decision-making (Hanney et al., 2003; Room, 2005; Weiss, 1983). It appears that research evi-
dence is regarded as an important input by Australian policy makers, as demonstrated by their accessing of it in their most recent decision. This does not, however, tell us the extent to which that evidence then formed the basis for the decision. The extent to which Australian drug policy makers are similar to those in other countries is unknown. Finally and most importantly further work exploring how decisions are actually made in terms of the interactions between research evidence, power and politics will be informative.
Acknowledgements This work forms part of the Drug Policy Modelling Program (DPMP), funded by the Colonial Foundation Trust and the NMHRC. Thanks go to the senior policy makers who took time out of their busy schedules to be interviewed, and to the two anonymous reviewers who provided helpful comments.
References Anderson, L. M., Brownson, R. C., Fullilove, M. T., Teutsch, S. M., Novick, L. F., Fielding, J., et al. (2005). Evidence-based public health policy and practice: Promises and limits. American Journal of Preventive Medicine, 28(5 Suppl. 1), 226–230. Australian Bureau of Statistics. (2006). Australian Demographic Statistics—December quarter 2006. Canberra: ABS. Australian Crime Commission. (2006). Australian Crime Commission (ACC) Annual Report 2005–2006). Canberra: ACC. Australian Institute of Health and Welfare. (2005). 2004 National Drug Strategy Household Survey: Detailed Findings. Drug Statistics Series No. 16. Canberra: AIHW. Bridgman, P., & Davis, G. (2000). Australian policy handbook (2nd ed.). St, Leonards, NSW: Allen & Unwin. Brownson, R. C., Royer, C., Ewing, R., & McBride, T. D. (2006). Researchers and policymakers: Travelers in parallel universes. American Journal of Preventive Medicine, 30(2), 164–172. Bulmer, M. (1986). Social science and social policy. London: Allen & Unwin. Callahan, D., & Jennings, B. (1983). Ethics, the social sciences, and policy analysis. NY: Plenum Press. Coleman, D. (1991). Policy research—who needs it? Governance: An International Journal of Policy and Administration, 4(4), 420–455. Denniss, R. (2006). Policy research and organisational demands. In H. K. Colebatch (Ed.), Beyond the policy cycle (pp. 228–239). Sydney: Allen & Unwin. Edwards, M. (2005). Social science research and public policy: Narrowing the divide. Australian Journal of Public Administration, 64(1), 68–74. Fitzgerald, J. L. (2005). The Australian National Council on Drugs (ANCD) and governance in the Australian drug policy arena. Contemporary Drug Problems, 32, 259–293. Gregrich, R. J. (2003). A note to researchers: Communicating science to policy makers and practitioners. Journal of Substance Abuse Treatment, 25(3), 233–237. Hanney, S., Gonzalez-Block, M., Buxton, M., & Kogan, M. (2003). The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Research Policy and Systems, 1(2). Kingdon, T. (2003). Agendas, alternatives, and public policy (2nd ed.). New York: Longman. Lavis, J., Davies, H. T. O., Oxman, A. D., Denis, J.-L., Golden-Biddle, K., & Ferlie, E. (2005). Towards systematic reviews that inform health care
A. Ritter / International Journal of Drug Policy 20 (2009) 70–75 management and policy-making. Journal of Health Services Research & Policy, 10(Suppl. 1), S35–S48. Lindblom, C. E. (1959). The science of ‘muddling through’. Public Administration Review, 19, 79–88. Lindblom, C. E. (1979). Still muddling, not yet through. Public Administration Review, 39(26), 517–526. Nutley, S., Walter, I., & Davies, H. T. O. (2003). From knowing to doing: a framework for understanding the evidence-into-practice agenda. Evaluation, 9(2), 125–148. O’Brien, S., Black, E., Degenhardt, L., Roxburgh, A., Campbell, G., de Graaff, B., et al. (2007). Australian Drug Trends 2006: Findings from the Illicit Drug Reporting System (IDRS). NDARC Monograph No. 60. Sydney: National Drug and Alcohol Research Centre. Ritter, A. (2007). Priority areas in illicit drug policy: Perspectives of policy makers. Monograph No. 15: DPMP Monograph Series. Sydney: National Drug and Alcohol Research Centre. Room, R. (2005). Symbolism and rationality in the politics of psychoactive substances. In B. Lindgren & M. Grossman (Eds.), Substance use: Indi-
75
vidual behaviour, social interactions, markets and politics (pp. 331–346). Amsterdam: Elsevier. Sabatier, P. A. (1988). An advocacy coalition framework of policy change and the role of policy-oriented learning therein. Policy Sciences, 21, 129–168. Secker, A. (1993). The policy-research interface: An insider’s view. Addiction, 88(Suppl.), 115S–120S. Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review, 39, 426–431. Weiss, C. H. (1983). Ideology, interests and information: the basis of policy positions. In D. Callahan & B. Jennings (Eds.), Ethics, social sciences and policy analysis. NY: Plenum Press. Weiss, J. A., & Weiss, C. H. (1981). Social scientists and decision makers look at the usefulness of mental health research. American Psychologist, 36(8), 837–847. White, V., & Hayman, J. (2006). Australian secondary school students’ use of over-the-counter and illicit substances in 2005. Canberra: Drug Strategy Branch, Australian Government Department of Health and Ageing.
