You are on page 1of 9

GACS Review Sub-Group on the Agency-funded systematic review of nutrient content of organic

and non-organic foods

Final response, agreed by GACS

Background
GACS agreed at its 5th meeting to establish a Sub-Group that would:
(i) consider the issues raised by the Soil Association (SA) and Organic Trade Board (OTB) on the
Agency-funded systematic review of nutrient content of organic and non-organic foods and its
wider implications for how the Agency commissions and communicates science in contentious
areas.
(ii) draft a response for approval by the full GACS.

The Sub-Group comprised three GACS members: Professor Janet Bainbridge, Dr Ian Brown and
Mrs Pamela Goldberg. It produced a Scope of Work (published separately) that sets out its terms
of reference, ways of working and the key issues to be considered by the Sub-Group.

Summary of approach
The Group considered that the concerns raised by the SA/OTB fall into two broad areas:
(i) how the review itself was conducted by the research team, and
(ii) how the Agency managed the wider engagement and communication around the
commissioning, definition, conduct and dissemination of results of the study and its
implications.

The Group did not take a view on the subject of the systematic review itself, namely whether
there are differences between the nutrient content of organic and non-organic foods, or between
their associated health benefits.

General points
The review addressed a very complex area of research. One of the principal findings of the review
was to highlight the highly variable nature and generally poor quality of the published research in
this area. This presents difficulties for anyone attempting a systematic, evidence-based
assessment.

The Group originally decided to consider the definition of the question including the decision to
focus on nutrient content and associated health effects. The SA/OTB did not raise this specific
issue in their communication to GACS, but it has been a feature of subsequent wider commentary
on the review and it relates to GACS previous discussions on definition of research questions.
However, the Group concluded that it would focus in its report to GACS only on the specific
questions raised by SA/OTB.

1
Summary of findings/recommendations
Overall the review Sub-Group did not concur with the majority of the concerns raised by the
SA/OTB. It considers that the review itself followed good practice, including publication of the
study protocol online at an early stage in the project, inviting comments. The Group noted that
the work had undergone extensive and rigorous peer review, first by external expert appraisers
convened to consider the report to the Agency, and again before publication of the work by a high
quality journal.

The Sub-Group makes two recommendations to the Agency:

Recommendation 1
The Agency should have a clear policy on release of underpinning data not already in the public
domain, which sets out clearly what underpinning data will be released and where, expected
timetables, and the nature and reasons for any exceptions. The default assumption should be that
underpinning data would be released wherever possible. (In this context ‘underpinning data’
should be taken to mean the data arising from a project that form the basis of findings and
conclusions and which would enable workings to be followed and analysis to be reproduced, or
new analyses to be conducted; it does not mean every printout or piece of information generated
in the process of the research.) Research specifications should include clear plans for release of
underpinning data, where appropriate.

In making this recommendation, the Sub-Group notes that the researchers published the
underpinning data from the 55 studies that had met the selection criteria and had formed the
basis for the results reported in the American Journal of Clinical Nutrition, on the journal’s website
on the day that the first article was published in print.

Recommendation 2
It is inevitable that the Agency will sometimes only be able to give stakeholders short notice of
impending release of results depending on the notice given by the journal/publication. However,
the Agency should ensure where possible that interested parties know in advance that they will be
given short notice of impending release of results.

The Group notes that responsibility for nutrition now lies with the Department of Health in
England (and with the Welsh Assembly Government in Wales), and that those Departments rather
than the Agency would consider the need for any further research in this area. GACS does not
have any role in advising other Departments. The Sub-Group notes that is good practice to look at
new evidence in any area of evidence–based policy, as it is published, and to update reviews and
conclusions with new data that meet appropriate quality and relevance criteria.

2
Conduct of the study

1. The validity of the published review protocol and how it was applied in the study,
particularly the criteria for exclusion and their application.
The SA/OTB argue that this resulted in the exclusion of studies that were robust, thorough and
recent, and inappropriate weight being given to some types of study, and too little to other
types. Specific aspects include:
Whether the review followed best practice
The approach to potential confounding factors (including geographical location, growing
season, harvest time, processing methods)
Correction of data for dry matter content
The approach to information concerning ‘organic’ status in selection/exclusion of studies

The Group felt the issues raised here are a matter of scientific method.

Overall, the Group felt that the protocol was robust, and had been published and consulted on in
line with good practice. Criteria for exclusion were reasonable, and were formulated in ways that
were reasonably inclusive with regard to the question that was being investigated. The Group
considered that there was no choice but to have made a clear definition of organic production
method a necessary criterion for inclusion in the review, or there would be no assurance that
papers really were comparing organic with conventional crops. The Group noted that the review
team had applied this criterion inclusively.

The SA/OTB identified 28 papers which it felt should have been included. The Group noted that
the research team had published a list of the papers that had passed this initial stage but were
excluded from the analysis along with a reason for their exclusion. The Group was satisfied that
the reasons for exclusion had been explained satisfactorily by the research team. Four scientific
papers were identified by the SA/OTB that had not been identified in the systematic review. On
the basis of the abstracts of these 4 papers it is not possible to judge whether they would have
been included or excluded, or whether they would contribute any relevant information to the
review, although it is not obvious from the abstracts that they are reporting comparisons of
nutrient content in organic and in non-organic foods. The Group noted that it is never possible to
guarantee that a systematic review will identify 100% of potentially relevant papers, and felt that
the research team had made thorough and reasonable efforts to identify as many papers as
possible. Over 52,000 papers were identified initially and of these 55 were selected as within the
criteria and formed the basis of the review findings.

The researchers did have to adapt their approaches in light of the nature of the data passing the
inclusion criteria, but had done so with reference to published guidelines and to a well-qualified
expert panel.

3
The Sub-Group noted that the heterogeneous nature of the studies and their data presented
challenges in handling of variation in data reporting and data structure among different studies.
The Group felt that the researchers had employed reasonable, straightforward approaches to
these issues where a consistent approach was possible, for example in calculating robust standard
errors for data from papers that reported multiple outcomes for the same or similar samples, to
account for potential clustering.

The Group acknowledged the importance of confounding factors in work in this area, and that the
review authors discussed the importance of these factors in the first AJCN paper and had
produced an intuitive map at the start of the review process of the many factors that could
potentially influence the nutrient content of foodstuffs.

In relation to potential sources of confounding and variability, the Group is satisfied that the
researchers conducted the systematic review in an appropriate manner, acknowledging that the
researchers had to work from the published data and that the available papers had employed
differing approaches to handling and reporting these issues. The review grouped papers into
three types of study designs each of which exerts a different level of control over the various
factors that could potentially influence nutritional composition. None of the studies provided data
on all potentially relevant factors identified in the intuitive map, so true “like-for-like” comparisons
were not possible. However, the Group is satisfied that the review based its analysis on
comparisons that were as similar with regard to other factors as the available data allowed.

The Sub-Group noted that the treatment of dry matter/fresh matter results could be an issue in
studies of these sorts, but felt that it would not be feasible to apply a ‘correction’ for this across all
the studies that could be done in a consistent, systematic way, and which would not require
assumptions and introduce further uncertainties.

2. The range of external expertise brought to bear in the review.


The SA/OTB argue that the range of expertise was too narrow (nutrition and statistics).

The Sub-Group felt that the expertise in the review team and its expert panel had been extensive
and appropriate.

The Group noted that the full extent of this expertise was not immediately apparent from the
details provided in the protocol, but there had been the opportunity to raise concerns over the
expertise at the time of the publication of the protocol when names had been made public.

The Agency also included a wide and appropriate range of expertise in its peer review of the final
reports from the review (including expertise in nutrition and nutrition science, animal science and
health, soil sciences, and an independent statistician with experience of systematic reviews). This

4
number of peer reviewers is greater than that normally used on Agency projects, which is
appropriate in this case.

3. The extent of consultation with organic businesses and other stakeholders by the study
researchers before the study was launched, at any point during the study, and after
publication.
The SA/OTB argue that engagement with organic businesses and other stakeholders was
limited and that this affected the robustness and inclusiveness of the process.

The Group felt that the bulk of attention to engagement would happen before the review – in
defining the question and approach – and after it was completed, in relation to the results and
their implications; this falls principally to the Agency and is dealt with separately under 7 below.

The Group felt the review group itself had followed good practice in publishing and inviting
comments on the protocol. In addition, members of the review team had met and otherwise
engaged with key stakeholders during the review; this included meeting with Lord Melchett and
email and telephone correspondence with the leader of the QLIF project.

The Group also felt Dr Dangour had engaged extensively on outcomes and implications of the
review at and following publication.

4. Statistical analysis and how they were presented, particularly with regard to statistical
significance of differences between organic and non-organic foods.
The SA/OTB question the appropriateness of the tests applied and the assessment and
description of statistical significance.

The Group’s view is that there was appropriate involvement of statisticians in the project. The
handling of statistical significance is clear and follows widely accepted approaches.

The review team had acknowledged openly that the nature of the data surviving the application of
the exclusion was such that their proposed meta-analysis was not possible. They took expert
advice and in the absence of a single standard approach adopted an approach that was
straightforward and clear. The Group notes that the review team did conduct reanalysis when an
alternative approach was suggested and that this had resulted in similar conclusions.

5
5. Distinction between scientific and personal perspectives in presentation of the results by the
study leader.
The SA/OTB argue that the project leader did not properly make this distinction, particularly
with regard to whether the highlighting of ‘important’ findings was justified by the results.

The Sub-Group felt in the reports and statements it had seen, the review authors had been clear in
setting out the findings of the review, including highlighting the poor nature of the data available,
and the need for more, better quality studies to provide better evidence on these questions. The
published papers explain clearly what is meant by ‘no important differences’.

Agency management and communication of the study

6. Timing of commissioning of the review, with regard to the timescale of other relevant
studies underway at the time and their expected publications, particularly the EU-funded
project QLIF (http://www.qlif.org/).
The SA/OTB believe the exclusion of data from this project seriously affected the quality and
relevance of the review conclusions. They also state that the Agency has not explained the
reasons for its decision not to defer the review to allow results from this study to be considered.

As a general comment, the Group noted that it was inevitable that any systematic review would
have to have a cut-off date and that studies might be missed. It was good practice to keep
emerging publications under review and to consider renewing systematic reviews, as important
new evidence emerges that meets appropriate quality and relevance criteria.

The Group noted that the Agency had been under pressure to review its position on the relative
nutritional benefits of organic versus conventional food in light of new research findings and
commentary on these, including from the Soil Association.

The Agency’s estimate at the time was that waiting for the results of QLIF to be reported fully
could have meant deferring the review until 2010, with reporting a year later. There was already,
reportedly, a body of new evidence to consider, aside from QLIF. The Group notes that as things
stand now, the QLIF work is still not published.

The Group concluded that the Agency’s decision to commission the review at the time that it did
was reasonable in light of these pressures.

The Group notes the SA/OTB’s comment that the Agency has not explained to the SA/OTB the
reasons for its decision not to delay the review, and the Agency’s comment that there is no
written record of such a request. This is not something the Group can really take a view on but
the Group notes that there was extensive contact between the Agency and the Soil Association
before, during and after the review.

6
One of the researchers at Newcastle University, Dr Kirsten Brandt, offered to make available to the
Group details of some work from the QLIF project which had been presented at a conference, and
a paper has been accepted for publication in the journal Critical Reviews in Plant Science. The
researchers provided the conference presentation, and offered to make available, in confidence,
the pre-publication article. The Group considered the slide presentation, which summarised
different reviews in this area and the approaches taken to some of the selection/exclusion criteria.
It concluded that it would not be appropriate to consider the unpublished work in the context of
its report to GACS but that the paper, when published, should be considered to see if it might be
relevant to a future update of the systematic review.

The Group noted that responsibility for nutrition has passed to the Department of Health in
England (and to the Welsh Assembly Government in Wales) and that these Departments would
need to consider the need for any further research in this area. GACS does not have a role in
advising other Departments. However, it is good practice to look at new evidence in any area of
evidence–based policy, as it is published, and to consider the need to update reviews and
conclusions with any new data that meet appropriate quality and relevance criteria.

7. Extent of consultation with organic businesses by the Agency, before the study was
launched, and at any point during the study, and after publication (in itself and in contrast
with the approach for other stakeholders).
The SA/OTB argue that:
engagement with organic businesses and other stakeholders was not sufficient; that this
affected the robustness and inclusiveness of the process, and made it more confrontational.
engagement with organic businesses in this case was not at the same level as the Agency
routinely affords to non-organic businesses

The Group noted that engagement with organic businesses had taken place before and during the
study, and at publication, including extensive contact with the SA. There had been a
representative of the organic business in the peer review panel that had assessed proposals for
the review.

The period of notice of the findings given to stakeholders before publication was short. This was
constrained by the fact that the Agency’s release of the results would coincide with publication in
AJCN, so as not to prejudice publication in that journal, but AJCN could give very little notice of its
publication date.

The Group felt that publication in a respected peer–reviewed journal was good practice in this
case and the resulting impacts on publication were justified. The release of information just prior
to publication was considered to have been as open as it could have been in the circumstances,
and it was noted that all parties knew that it was the intention to publish work in a peer-reviewed
journal.

7
The period of notice was not markedly different to that offered by other organisations in similar
circumstances. The Group did not conduct a detailed review of Agency’s practices in other cases
but it noted that a similar situation had arisen in the case of the Southampton food colours study
where the Agency had encouraged Southampton University to publish their study in a peer
reviewed journal, and the results were of significant interest to sectors of industry (in this case
predominantly non-organic businesses). Here the Agency did not publish the results in advance
because of the usual concern that once they were published by the Agency, most quality journals
would not be interested in publishing them. The Agency had no control over the timing of the
publication since this is a journal decision. The industry was informed that the results were
coming for some months before publication but they did not know what the results said. Once the
Agency knew the publication date (Thursday 7 September 2007) stakeholder meetings with
industry, media and NGOs/consumer organisations were arranged on the preceding Tuesday and
Wednesday.

The Group felt that it would inevitable that the Agency would need to give stakeholders short
notice of impending release of results. However, the Agency should consider ensuring interested
parties know in advance that they will be given short notice of impending release of results.

8. Timing of publication of underpinning data from the study


The SA/OTB argue that these data should have been made available sooner.

The Agency published the project reports on 29th July 2009 at the same time as the online
publication of the first article in the American Journal of Clinical Nutrition.

The researchers published the underpinning data from the 55 studies that had met the selection
criteria and had formed the basis of the results reported in the American Journal of Clinical
Nutrition, on the journal’s website on the day that the first article was published in print (1st
September 2009).

The Agency published a further, wider set of underpinning data at a later stage in May 2010, which
included studies which had passed the preliminary sift stage but had not met the selection criteria
for inclusion in the final analysis. It appears that this second, wider set of data was published in
response to requests, rather than proactively as part of the research plan for the review.

It is good practice to make underpinning data from research available to others; the gold standard
is to release underpinning data at the same time as the main results and conclusions, although this
may not always be possible. The Group acknowledged that it may not be a straightforward task to
prepare the underpinning data from a systematic review into a format that will be clear and
helpful to others.

8
The Group recommends that the Agency should have a clear policy on release of underpinning
data, which sets out clearly what data will be released and where, expected timetables, and the
nature and reasons for any exceptions. The default assumption should be that underpinning data
would be released wherever possible. (In this context ‘underpinning data’ should be taken to
mean the data arising from a project that form the basis of findings and conclusions and which
would enable workings to be followed and analysis to be reproduced, or new analyses to be
conducted; it does not mean every printout or piece of information generated in the process of
the research.) Research specifications should include clear plans for release of underpinning data,
where appropriate.

In this case, the most important underpinning data were those relating to the 55 studies that had
been the basis for the reported analyses.