Blackwell Science, LtdOxford, UKADDAddiction0965-2140 2005 Society for the Study of Addiction 100 Miscellaneous

ICAP Report on alcohol education David Foxcroft

FOR DEBATE

International Center for Alcohol Policies (ICAP)s latest report on alcohol education: a awed peer review process
David Foxcroft
School of Health and Social Care, Oxford Brookes University, Oxford, UK

Correspondence to: David Foxcroft School of Health and Social Care Oxford Brookes University Marston Campus Jack Straws Lane Oxford OX3 OFL UK Tel: + 44 (0) 1865 485283 Fax: + 44 (0) 1865 485297 E-mail: david.foxcroft@brookes.ac.uk Submitted 8 February 2005; initial review completed 23 February 2005; nal version accepted 15 March 2005.

FOR DEBATE
With regard to the International Center for Alcohol Policies (ICAP) report number 16, entitled Alcohol Education and its Effectiveness [1], I must express my disappointment and more generally highlight probable deciencies in the peer review process for evidence-based reports from reputable organizations. In November 2004 ICAP sent me a copy of the draft report and asked if I would be prepared to offer some comments. ICAP did not ask explicitly for an academic peer review of the draft report, but as the report was written in an academic style, contained many references to academic journal papers and also discussed effectiveness of prevention approaches, I decided to offer my comments as if I were undertaking a peer review of an academic paper. Before I provided my comments I also declared to ICAP a potential conict of interest. A Cochrane Review of alcohol misuse prevention for young people, on which I was lead author and which was published in Addiction in 2003, highlighted the potential effectiveness of the Strengthening Families Programme (SFP1014) and researchers in my Department have subsequently been awarded funding to adapt and test the SFP1014 for use in the United Kingdom. Some of this funding is from the alcohol industry.
2005 Society for the Study of Addiction

The comments I made were generally critical of the substance of the ICAP report, pointing out a number of weaknesses and inconsistencies in the draft and suggesting a number of improvements. I drew particular attention to the inappropriate use of evidence. Yet the nal report hardly differs from the draft, although there are some mainly cosmetic changes. Of course, other reviewers may not have pointed out similar concerns and I could simply be guilty of arrogance, but when many signicant comments are disregarded alongside the expectation that the organization that produced the report should uphold the highest scientic standards, that causes me concern. I will not list all the specic comments I made as they were substantial and ran to four typed sides of A4 paper. However, I feel I should point out several areas of concern that were not addressed in the nal report. The report states that life skills training approaches can be effective and that such approaches work in many different countries. My comments to ICAP pointed out that there is no good evidence to support life skills prevention approaches for alcohol, and that the cited studies of life skills are beset by methodological shortcomings, inappropriate statistical analysis, poor reporting and occasionally statistically signicant ndings that are of
Addiction, 100, 10661068

doi:10.1111/j.1360-0443.2005.01144.x

ICAP Report on alcohol education

1067

trivial importance. My comments also pointed out that this latter issue, of statistical signicance as opposed to effectiveness, was problematic in two other prevention approaches highlighted in the ICAP report, SHAHRP [2] and Northland [3]. While I agreed that the SHAHRP study was well designed and the results interesting and therefore, on the face of it, shows some promise, there are also some issues with this study that need to be claried before rm conclusions can be drawn. First, the ndings as reported in several journal articles contain several reporting inconsistencies in which non-signicant effects are reported as signicant in the conclusion and summary sections. Secondly, few of the directly measured behaviours showed important changes, and it is only when a post-hoc multiplicative combination of outcome measures is compared between intervention and control groups that an apparently important effect emerges. Moreover, the misuse of combined end-points is a fairly common error [4], and the implication of the researchers combining their endpoint measures in this study needs to be fully understood. Finally, an important issue which runs through all evaluation research is how to interpret the importance of the ndings. Although results are presented frequently as statistically signicant, this is not the same as importance in terms of effectiveness. In the SHARHP study relative risk reductions are reported which increase the perception of importance but often hide the realabsolute impact. I also pointed out that while Project Northland evidence was derived from a sophisticated statistical analysis, with statistically signicant results, I was concerned that the results fall into the category of studies that show statistically signicant but trivially important effects. If the effect sizes shown in the growth curve modelling analyses are examined, the growth rate differences between intervention and control communities is of very questionable importance. I commented to ICAP that I was happy to be persuaded otherwise, but at the moment I could see no real benet, in terms of alcohol misuse prevention effectiveness, of the Northland approach. The ICAP report also highlights social norming theory evidence but as this was not included in our Cochrane Systematic Review of alcohol misuse prevention I pointed out that I was not familiar with the quality of this evidence. I suspect that this evidence is not from controlled studies, so the potential for confounding (i.e. other factors that might account for the results) is high. Therefore I suggested that the report should be more tentative about this evidence. In fact, overall the report reads like a fairly traditional, or narrative, literature review undertaken by someone with knowledge in the eld. The problem, however, with such narrative reviews is that there is good evidence from
2005 Society for the Study of Addiction

the work of Mulrow and others [5,6] that they are an unreliable source of information, with conclusions made that are not based on rigorous and critical review methods and that are often inconsistent with best evidence. In the ICAP report the nal conclusions are even somewhat inconsistent with the preceding sections. One conclusion is that targeted interventions are more effective than broad-based measures, but there is no evidence in the body of the report that directly addresses or substantiates this conclusion. Another conclusion is that combined approaches have been shown to be more effective than single approaches. Again, there is no evidence in the body of the report that directly addresses or substantiates this conclusion. The points above highlight specic concerns with the quality of this particular ICAP report that were not addressed through the usual academic peer review process. Surely, to command respect and trust, to earn and maintain credibility in the eld, the peer review process used by any organization should be independent, authoritative and inuential. Those expectations have not been met in the present case. Moreover, ICAP reports are unattributed and that must detract from their authority. I do not understand why ICAP does not state who authored a particular report and, more honestly, cast such reports as reecting the opinions of the authors rather than the potentially misleading assumption that such reports are comprehensive, authoritative, critical and state-of-theart reviews of the available evidence. In the process of independent peer review there are three parties: the author(s), the reviewers, and the editor(s). It is the editors job to ensure that the author(s) take the suggestions of the reviewers into account, and indeed most journals allow the reviewer the option to rereview the paper to ensure that their suggestions have been acted upon. Here, partly because the issue of authorship is not clear and the publication emerges from the organization itself, the roles of author and editor seem blurred, with no one in fact seeming to take on the responsibility of ensuring that the nal report has taken account of the reviewers comments. There is a danger that such reports are seen to uphold the highest scientic standards, that the process of peer review is similar to that of reputable journals, when in fact the process is very different. Perhaps it is time for all organizations which issue apparently authoritative but unattributed reports, with the aim of guiding and inuencing policy, to be transparent about how such reports are authored and quality controlled. Reassuringly, in response to my concerns about the process, I am pleased to say that ICAP have recommended a number of signicant changes to the way ICAP reports are prepared and peer reviewed in the future. Perhaps other organizations should follow their example.
Addiction, 100, 10661068

1068

David Foxcroft

References
1. International Center for Alcohol Policies (ICAP) (2004) Alcohol Education and its Effectiveness. ICAP Report no. 16. ICAP: Washington, USA. 2. McBride, N., Farringdon, F., Midford, R., Meuleners, L. & Phillips, M. (2003) Harm minimization in school drug education: nal results of the School Health and Alcohol Harm Reduction Project (SHAHRP). Addiction, 99, 278291. 3. Perry, C. L., Williams, C. L., Komro, K. A., VeblenMortensen, S., Stigler, M. H. & Munson, K. A. (2002) Project

Northland: long-term outcomes of community action to reduce adolescent alcohol use. Health Education Research, 17, 117132. 4. Julian, D. G. (2003) What is right and what is wrong about evidence-based medicine? Journal of Cardiovasular Electrophysiology, 14, S2S5. 5. Mulrow, C. D. (1987) The medical review article: state of the science. Annals of Internal Medicine, 106, 485488. 6. Oxman, A. D. & Guyatt, G. H. (1988) Guidelines for reading literature reviews. Canadian Medical Association Journal, 138, 697703.

2005 Society for the Study of Addiction

Addiction, 100, 10661068