What is the name for the measure of a drugs effectiveness?

Foreword | Seizing drugs and arresting those who import, manufacture, grow and/or distribute these drugs is often viewed as the most important purpose of drug law enforcement. This view is certainly strong in popular media depictions of organised drug criminals. Unfortunately, the reality is perhaps far less entertaining or straightforward, although just as, if not more, important. While there is no doubt that a key role of drug law enforcement is to remove drugs and high-risk offenders from the community, the most critical factor is what this actually achieves in the longer term. That is, a community that is less burdened by the impact of drugs, such as crime, illness, injury and death.

Increasingly, there is both internal and external pressure on drug law enforcement to demonstrate not just how much work they do (the seizures and arrests), but how well they do it (the community impacts)—something that has so far proven very difficult. This paper outlines the nature of these challenges and summarises findings from a national project that shows a practical and effective way forward in measuring the impacts of drug law enforcement.

Adam TomisonDirector

Measuring the impact of drug law enforcement (DLE) practice on illicit drug markets is a notoriously difficult task. Conventional approaches to assessing DLE performance focus on the use of drug seizure and arrest data. However, these data say more about the extent to which police engage in certain types of activities and allocate resources than they do about DLE effectiveness because offences relating to illicit drugs are far more likely to be detected by law enforcement agencies than reported to them. As such, the more effort and resources DLE invest in detecting illicit drugs, the more likely it is that drugs will be seized. On the one hand, DLE can potentially claim success for not seizing any drugs—that is, based on the absence of seizures and arrests, it could be argued that there is no drug problem. Conversely, a lack of seizures and arrests could lay police open to substantial criticism for failing to address the drug problem.

Aside from this, traditional measures say little about the complexities of DLE work and the broader impacts of law enforcement effort. For example, they cannot provide an assessment of the full impact of DLE in producing something of value for communities, such as making communities feel safer and more secure, which is something that Australian DLE personnel view as an important outcome of their work (Willis, Homel & Gray 2006).

The volume of crime is but one measure that can be considered in the broader assessment of the quality of work done by law enforcement. A range of appropriate measures that captures the complexities of law enforcement work can:

• permit a more rigorous assessment of the broader range of outcomes that law enforcement actually produce for their communities (and so help law enforcement agencies demonstrate impacts in real terms);
• inform communities of the depth and breadth of work in which modern law enforcement is engaged;
• form the basis upon which both operational and long-term strategic decision making can be made; and
• assist agencies to justify expending and seeking resources.

Supply and use of illicit drugs are, by their largely clandestine nature, a hidden phenomenon that can only be monitored through use of indirect indicators linked to observable consequences, such as crime, drug-related illness, injury and death. The use of these types of multidisciplinary indicators to monitor and measure law enforcement performance is gaining increasing acceptance here in Australia and elsewhere as there is greater recognition that arrest and seizure data alone are unsatisfactory when interpreted in isolation from other factors (Castle 2008; Kilmer & Hoorens 2010; Osnick Milligan & Fridell 2006; Rossi 2001; Weatherburn 2000).

This paper summarises key findings, both conceptual and practical, from the second stage of a major national project that sought to test the feasibility of a model performance measurement framework for Australian DLE agencies and to provide advice on its national implementation (Willis, Homel & Anderson 2010). As such, it does not provide an overview of the effectiveness of DLE in Australia. The first project stage involved development and preliminary testing of the initial performance measurement framework. Trends & Issues no. 332 (Homel & Willis 2006) includes a summary of the project’s rationale and development and so this is not repeated in this paper in detail. Both project stages were undertaken by the Australian Institute of Criminology (AIC) on behalf of the National Drug Law Enforcement Research Fund.

The framework: Conceptual issues

The rationale behind the use of the framework as a DLE performance measurement tool is that the framework’s measures address a combination of supply and demand market issues. The framework’s premise is that DLE impacts on both of these. In theory, illicit drug supply is reduced through action such as controls on drug production and distribution, seizures and the arrest (and ultimately incarceration) of those involved in the importation, production and distribution of illicit drugs (for a detailed summary of supply-side controls see Willis, Homel & Gray 2006). In essence, the aim of supply-side DLE is to disrupt the supply or availability of illicit drugs, thereby increasing the costs and risks associated with drug importation and distribution. The aim of demand-side DLE is to reduce the level of demand for illicit drugs within the general community. Demand-side DLE is primarily directed at the drug user. The rationale behind demand-side DLE is that, even if DLE agencies are unable to increase the financial cost of illicit drug use or restrict its availability, they can increase the non-monetary costs associated with its use. So, as the level of inconvenience, time, risk or cost of trying to find a drug seller increases, more drug purchasers are tempted to leave the illicit drug market (for example by entering treatment) while those who remain tend to use illicit drugs less frequently (Weatherburn et al. 2000). This then has clear flow-on effects in terms of reducing public harms.

The core components of the model performance measurement framework developed by this project are built on this premise and address the following four high-level outcome areas for DLE:

• reducing drug crime and drug-related crime;
• reducing organised crime;
• improving public health; and
• improving public amenity.

Each of the four outcomes is interrelated and as such, they are not discrete areas. In other words, activities specifically targeting a reduction in drug and drug-related crime could also influence the other three high-level outcomes. Each outcome area is underpinned by a series of appropriate measures and indicators (see Table 3 for a full list of these) that seek to:

• better account for the benefits from the estimated $1.7b expenditure on DLE in Australia (Collins & Lapsley 2008);
• improve the type and range of performance measures and indicators currently used by DLE; and
• provide DLE agencies, from national through to local levels, with a consistent and systematic means of assessing and reporting performance.

Although the high-level outcomes and measures are essentially prescriptive, the indicators can be tailored to suit different jurisdictional monitoring and reporting needs. For example, where the framework suggests the use of an incidence-based count (such as the number of cannabis traffic/supply arrests), an offender-based count can be substituted (such as the number of offenders processed for cannabis traffic/supply offences). In this way, the basic intent and integrity of the core framework elements are maintained, while also allowing for local data recording conventions.

Since development of the initial framework in 2006, a considerable amount of work on improving law enforcement performance measurement has been undertaken in Australia and overseas. In contrast to Australia, where developments have focused primarily on DLE performance measurement, overseas work has concentrated on performance measurement in law enforcement more generally. Despite this difference, both approaches advocate the importance of developing measurement systems that:

• capture information beyond traditional law enforcement performance measurement boundaries (such as from the public health sector);
• use multiple indicators for monitoring performance, rather than relying on single indicators; and
• emphasise that measurement should move beyond simple police activity measures to focus on future perspectives (such as achieving strategic goals).

It was these principles that guided the development of the DLE performance measurement framework.

Feasibility of the framework: Some practical considerations

Key stakeholder input

The second stage of this project had three overall objectives:

• refining the clarity and precision of the performance measures;
• identifying the training and development needed for implementing various stages of the framework; and
• developing a long-term business case for sustained implementation of the framework.

To re-establish the framework’s goals and relevance to contemporary DLE efforts and priorities, the framework was reviewed and refined at a national planning and development workshop convened by the AIC. Representatives at the workshop included key DLE practitioners, as well as experts from other relevant fields, such as the health sector. The validity and relevance of the framework was also further explored during stakeholder consultations as part of project fieldwork.

Field testing the framework

Following the planning and development workshop and stakeholder consultations, in-depth testing of the model performance measurement framework was undertaken in four field locations. These test sites were located within Australian Customs and Border Protection Service (ACBPS), Victoria Police, South Australia Police and at Tasmania Police. Field findings demonstrated that the framework can be applied at national and state/territory DLE levels—this is both in terms of its practical application (ie availability and applicability of the indicator data) as well as its use as a performance measurement tool. Indeed, each of the jurisdictions that participated in the field study already used many of the framework measures, including drug-related public health measures, although usually informally and inconsistently. In addition, where regular drug market monitoring and review did take place within jurisdictions, it occurred within existing agency accountability processes and structures (for instance, in unit and operational command review/COMPSTAT processes). This was an important finding as it supports a key recommendation from the first project stage that the framework be embedded in existing accountability processes to facilitate its uptake.

Indicator data strengths and limitations

Fieldwork findings supported application of the model framework within agencies with a national DLE focus through to those with a local DLE focus. However, it is clear that there are data limitations in some jurisdictions that mean that certain DLE agencies are better positioned than others to use the framework for regular performance monitoring, measurement and reporting. For instance, all of the field agencies used a range of information from sources external to their agencies (including drug-related public health data). However, with one exception, this information did not form part of any formal performance measurement processes and there were no explicit arrangements with third parties to regularly capture this information. This latter point is probably the biggest potential obstacle for full implementation of the framework.

Victoria Police’s drug market analytical tool (their Drug Attribution Model or DAM) provides a useful case study for how to operationally formalise the capture and reporting of a broad range of indicator data that is similar to the framework described here.

With the exception of one jurisdictional health agency, the AIC was able to obtain de-identified quarterly or annual data from each of the health agencies in the field locations via no more than an exchange of correspondence at the executive level. Although the AIC’s requirements were for a once only research project, it demonstrates that health agencies can be receptive to law enforcement agencies accessing these data, providing there is full disclosure as to precisely why, and in what way, the data will (and will not) be used. This finding was supported by the views of the health representatives at the project’s national workshop, where they indicated that there was need for frank data exchange so that both sectors could effectively monitor changes within the illicit drug environment, including new and emerging issues.

Another aspect of the framework’s feasibility rested heavily on the ability to collect and interpret the identified indicator data. Most data identified for the framework were obtained from law enforcement and health agencies’ administrative collections, which are largely designed to monitor agency output. Common limitations across these data collections include limitations in scope (ie limited to narrowly prescribed criteria), changes in collection methods over time, a lack of stringent quality checks for missing or incorrectly entered data and having long lags between an incident being reported and recorded (see Willis, Anderson & Davis 2010 for a more detailed overview of the limitations).

Survey data used (such as from the Drug Use Monitoring in Australia (DUMA) and Illicit Drug Reporting System (IDRS) collections) were able to fill some knowledge gaps. However, they were also restricted at times by factors such as small sample sizes (thus affecting the ability to monitor trends effectively), the collection of data focusing almost exclusively on metropolitan locations (so do not necessarily reflect the nature of regional and rural drug markets) and that they may represent specific sub-groups (eg police detainees) rather than the broader population (Willis, Homel & Gray 2006).

Neither the administrative nor survey data collections alone are able to measure the more complex issues of performance success and failures. This, coupled with the limitations described, underscore the importance of using multiple appropriate measures and indicator data to minimise the risk of erroneously identifying drug market trends.

Use of data collection plans

To obtain a clear and informed assessment of the most appropriate data sources to be used, the AIC developed and tested a ‘data collection plan’ (Table 1). The data collection plan needed to consider factors such as:

• information requirements;
• information sources;
• data collection processes;
• data collection costs;
• data protection/security;
• data quality; and
• implementing a trial run of the process (Roberts 2006).

The data collection plan used during fieldwork ensured that each data source was able to provide enough information for each indicator source. Use of this method reduced the risk of including unrealistic or difficult to measure indicators.

Source: Willis, Homel & Anderson 2010

The framework as an effective performance measurement tool

In the absence of direct measures of DLE effectiveness, the suite of measures and indicators outlined in the framework provide a broad platform upon which the impact of drug seizures and arrests can be systematically assessed over time. To illustrate this general point, two scenarios are provided in Table 2 relating to two hypothetical drug markets. The arrows describe how each market scenario is trending.

Under the first scenario (drug market A), DLE effort (ie seizures and arrests) is stable but public harms are increasing (ie there is increasing drug purity, availability, drug-related deaths and hospitalisations, and there is an increasing concern among the community about drugs). This might suggest that DLE is not performing well and needs to take remedial action by redirecting appropriate resources. Under the second scenario (drug market B), DLE effort is increasing and public harms are decreasing, although community concern is constant. This might suggest that DLE is performing well, although it may also suggest that DLE needs to review current resourcing priorities and re-deploy surplus capacity to other areas of greater need.

These hypothetical examples are simplistic and used here to briefly illustrate how the framework can be applied and how DLE data can be interpreted within a broader context to assess performance and inform strategic decision making. As such, they do not include the full range of suggested framework measures, nor do they cover the range of complexities and other contexts (such as the impact of environmental factors such as policy changes, police resource commitments, underlying changes in different drugs markets and so on) that may influence an interpretation of the indicator data trends and any conclusions made. In reality, consideration of the breadth of contextual factors is crucial in any assessment of DLE activity.

Furthermore, in reality, not all indicator data move in an expected direction and some can be quite perverse for no discernable reason; however, what is important is the overall pattern. That is, if most of the indicator data point to a market change (or lack thereof) then it increases the reliability of inferences made about that data. It is this long-run trend information that is important and which is more meaningful and indicative of substantive market change. This is why it is necessary to monitor and compare data over the long term.

The framework reflects what is considered, on the basis of detailed testing at the four sites, a reasonable suite of measures and indicators upon which to assess DLE performance at this stage. Although outside the scope of this summary paper, and as noted above, issues to do with indicator data strengths and limitations (including availability, completeness, geographic coverage, lag problems and so forth) are outlined in detail in Willis, Anderson and Davis 2010.

There are no pretensions that the framework is perfect—other measures and indicators could be considered for inclusion and probably will in the future should the framework be widely adopted. For example, it would be possible to combine the price and purity indicator data to form a single metric that monitors changes in price-adjusted purity levels. Similarly, inclusion of a metric covering police assets confiscation may be useful in the broader assessment of police impact on organised crime. However, the most critical point here is that in the absence of other more direct measures of DLE effort, the framework provides a sound basis for a more effective and systematic means of monitoring and reporting DLE performance than is currently undertaken by most DLE agencies.

Implementing the framework: Important considerations

Even a viable performance measurement framework may be difficult to implement. For this reason, one of the key outputs for the second project stage was the development of a comprehensive implementation plan, including the development of resources to assist in the national roll-out of the framework.

Project fieldwork experiences, a review of the literature and consultation with DLE professionals assisted in the development of this implementation plan and a series of key implementation issues were identified. These were:

• actually recognising the need for a new DLE performance measurement framework (including the limitations of traditional measures of DLE and the benefits of more robust measures);
• determining who is responsible for DLE performance measurement;
• developing major steps to developing a sound measurement framework;
• determining a realistic timeframe for national implementation of the framework;
• identifying and recognising data limitations;
• resourcing the framework;
• considering key change management issues;
• considering a future evaluation of the framework; and
• identifying key challenges for national implementation.

However, there were also some general lessons learned from this work that need to be considered if national implementation of the framework is to be undertaken. These include the requirement that:

• An effective performance measurement system must be based on a sincere desire to use timely and accurate data to improve performance. Where there is concern about communicating unfavourable performance data (‘we can’t show them that, the numbers look really bad!’), implementation of any performance measurement system will fail. As such, measurement systems designed to focus on performance improvements (as opposed to instruments of control) are much more easily accepted than systems designed exclusively for accountability purposes, particularly in hierarchical command and control organisations such as law enforcement, although ultimately they serve the same function (O’Connell & Straub 2007; Roberts 2006).
• Performance measurement must take place for the purpose of supporting management decisions and not just counting for accounting’s sake. If performance data are not used for managing performance then the system merely becomes a costly and elaborate exercise in accounting (O’Connell & Straub 2007).
• Superimposing an entirely new measurement system steeped in management jargon almost guarantees that the system will not be actively supported. In any case, to be effective, the system needs to accurately reflect the culture and practices of the organisation into which it is being introduced, even if it is part of an overall organisational change process.
• Organisational change can often be perceived as difficult and/or threatening to staff. This may be true particularly for organisations such as law enforcement agencies that have established and highly structured chains of command and practices. To this end, it is important to involve staff from management through to operational staff right from the start in defining operational goals and in developing appropriate performance measurement systems, including participation in analysing and reporting results. This helps to ‘buy-in’ staff support (O’Connell & Straub 2007; Roberts 2006).
• Success in DLE performance will mean that goals and targets will probably change. Therefore, the measurement system must be able to readily change and adapt to new priorities and outcomes (O’Connell & Straub 2007).
• Ensuring that a performance measurement system is built into strategic planning processes helps to establish accountability for the measures and ensures that they are both reported and used for performance improvement (Osnick Milligan & Fridell 2006; Roberts 2006).
• A communication strategy is essential for the dissemination of results within and across agencies. Broad communication of results allows DLE agencies to be perceived both internally and externally (including the public) to have open, accountable and honest processes (PSU 2006; Roberts 2006).

Conclusion

The implementation trial demonstrated that the DLE performance measurement framework is a viable mechanism for improving the capacity of DLE agencies across Australia to better account for their effectiveness as well as a useful tool that contributes to Australia’s National Drug Strategy’s goals of reducing drug-related harm, demand and supply. In addition:

• The framework has the potential to provide a way for the wider community to better understand the nature of the impacts that DLE can have upon issues such as drug-related public health and public amenity. It also highlights the need to shift focus from viewing simple police performance measures (such as number of seizures and arrests) as being adequate indicators in DLE performance measurement.
• The trial implementation of the framework demonstrates that it is a viable method for generating jurisdictionally and locally responsive reporting and accountability systems. Furthermore, it contains sufficiently robust and compatible set of core components with the potential to form the basis for an ongoing national reporting system.
• The implementation trial demonstrates that the significant, but currently fragmented and less than optimally systematic efforts that are going into enhancing DLE performance measurement can be strengthened. Furthermore, this can be achieved without imposing an entirely new regime of data collection and reporting on already stretched management structures.
• The framework provides a vehicle for building new, more systematic processes from within existing structures and procedures. In this way, the framework has been shown to be an evolutionary and easily accepted way to generate a sustainable performance management and accountability procedure.

Taken as a whole, the project findings suggest that a number of things should occur if a decision were made to undertake national implementation of the framework. These are as follows:

• The outcomes of the project could be brought to the attention of senior law enforcement executives through mechanisms such as the Australasian and South-Western Pacific Police Commissioners Conference, as well as to the Ministerial Council on Drug Strategy and the Intergovernmental Committee on Drugs with a view to considering a plan for a national implementation program for the DLE performance framework. While the companion report A Plan for National Implementation of the Drug Law Enforcement Performance Measurement Framework (Willis, Anderson & Homel 2010) broadly outlines a suggested approach and timeframe for national implementation of the framework, the precise processes and supporting mechanisms would need to be further developed within jurisdictions and based on local requirements.
• The national implementation plan should be conceived of as a series of jurisdictionally specific implementation strategies with an over-arching national reporting agenda. In other words, implementation should be the responsibility of specific state, territory and national DLE agencies, with a coordinated reporting mechanism capturing those core elements that would constitute the most important common elements. Application of the framework itself is considered to be largely cost-neutral because many of the indicator data are already collected by law enforcement agencies. However, it is recognised that further work may need to be undertaken in some jurisdictions in relation to administrative arrangements, IT systems and training should the framework be implemented nationally.
• To facilitate the framework’s national implementation, future detailed research could be undertaken to determine precisely how implementation of the framework could be staged and what processes would be employed to report against the framework at the national level.
• Should a decision be taken to undertake national implementation of the framework, it should be evaluated at an agreed point in the future to assess whether and how well the framework is being adopted by jurisdictional law enforcement agencies.
• Consideration should be given to commissioning an appropriate agency to provide the necessary ongoing technical assistance to ensure continuity, sustainability and quality assurance for the national performance measurement program.

Acknowledgements

This project was undertaken with a grant from the National Drug Law Enforcement Research Fund, which is funded by the Australian Government Department of Health and Ageing. The full report, Developing the Capacity and Skills for National Implementation of a Drug Law Enforcement Measurement Framework is available at http://www.ndlerf.gov.au/pub/Monograph_34.pdf

References

• Castle A 2008. Measuring the impact of law enforcement on organized crime. Trends in Organised Crime 11: 135–156
• Collins D & Lapsley H 2008. The costs of tobacco, alcohol and illicit drug abuse to Australian society in 2004/05. Australia: Australian Government Department of Health and Ageing
• Homel P & Willis K 2006. A framework for measuring the performance of drug law enforcement. Trends & Issues in Crime and Criminal Justice no. 332. Canberra: Australian Institute of Criminology. http://www.aic.gov.au/publications/current series/tandi/321-340/tandi332.aspx  
• Kilmer B & Hoorens S 2010. Understanding illicit drug markets, supply-reduction efforts, and drug-related crime in the European Union. Santa Monica: RAND Corporation. http://wwwcgi.rand.org/pubs/technical_reports/TR755/  
• O’Connell P & Straub F 2007. Performance-based management for police organisations. Long Grove: Waveland Press Inc
• Osnick Milligan S & Fridell L 2006. Implementing an agency-level performance measurement system: a guide for law enforcement executives. Washington DC: Police Executive Research Forum
• Police Standards Unit (PSU) (UK) 2006. Police performance management: Practical guidance for police authorities. http://www.community-safety.info/43.html  
• Roberts DJ 2006. Law enforcement tech guide for creating performance measures that work: A guide for executives and managers. Washington DC: Office of Community Oriented Policing Services. http://www.search.org/files/pdf/PMTechGuide.pdf  
• Rossi C 2001. Using available data with dynamic models to estimate the extent of problem drug use, in European Monitoring Centre for Drugs and Drug Addiction (ed), Modelling drug use: Methods to quantify and understand hidden processes. Monograph no.6. Lisbon: European Monitoring Centre for Drugs and Drug Addiction
• Weatherburn D 2000. Performance indicators for drug law enforcement. Contemporary Issues in Crime and Justice no.48. Bureau of Crime Statistics and Research, Sydney. http://www.bocsar.nsw.gov.au/lawlink/bocsar/ll_bocsar.nsf/vwFiles/cjb48.pdf/$file/cjb48.pdf  
• Weatherburn D, Topp L, Midford R & Allsopp S 2000. Drug crime prevention and mitigation: A literature review and research agenda. Sydney: New South Wales Bureau of Crime Statistics and Research
• Willis K, Anderson J & Davis B 2010. Implementing a drug law enforcement (DLE) performance measurement framework in Australia: Companion document to NDLERF Monograph Series no. 34, ‘Developing the capacity and skills for national implementation of a drug law enforcement performance measurement framework’. Hobart: National Drug Law Enforcement Research Fund. http://www.ndlerf.gov.au/pub/DLE_Perf_Mgt_Fieldwork_Findings.pdf  
• Willis K, Anderson J & Homel P 2010. A plan for national implementation of the drug law enforcement performance measurement framework: Companion document to NDLERF Monograph Series no. 34, ‘Developing the capacity and skills for national implementation of a drug law enforcement performance measurement framework’. Hobart: National Drug Law Enforcement Research Fund. http://www.ndlerf.gov.au/pub/DLE_Perf_Mgt_Implementation.pdf  
• Willis K, Homel P & Anderson J 2010. Developing the capacity and skills for national implementation of a drug law enforcement measurement framework. National Drug Law Enforcement Research Fund Monograph no. 34. Hobart: National Drug Law Enforcement Research Fund. http://www.ndlerf.gov.au/pub/Monograph_34.pdf  
• Willis K, Homel P & Gray K 2006. Developing and implementing a performance measurement framework for drug law enforcement in Australia. National Drug Law Enforcement Research Fund Monograph no. 18. Payneham South Australia: National Drug Law Enforcement Research Fund. http://www.ndlerf.gov.au/pub/Monograph_18.pdf

About the authors

Dr Katie Willis is a Senior Research Analyst, Jessica Anderson is a Research Analyst andPeter Homel is Program Manager Crime Reduction and Review