5/13/13

6 Ways Scientific Studies Can Trick You

Published on Alternet (http://www.alternet.org)

Home > 6 Ways Scientific Studies Can Trick You

AlterNet [1] / By Jill Richardson [2]

6 Ways Scientific Studies Can Trick You

May 9, 2013 | Ever see a headline boasting of an outrageous conclusion that some new scientific study found? These headlines pop up regularly, and they are a boon for publications that get lots of eyeballs reading their articles about the shocking new findings. Factory farming is actually good for the environment! Labeling genetically engineered foods will raise your grocery bills! Wow, really? No way! These too-good-to-be-true or too-bad-to-be-true headlines are accurate, in that there was a study and it did come to those conclusions. But how accurate was the study? As it turns out, scientific studies with phony findings are not as uncommon as they should be. And far too often, bad journalism results in the uncritical reporting of these phony findings. If you ever read about a study finding that all-cupcake diets are the key to longevity and good health, read the study to see whether the cupcakes tested were made from spinach and wheat germ. Here are some favorite tactics used to design a study to get the findings youre looking for. 1. Start With a Wrong Assumption If you live in California, you may recall hearing how labeling genetically engineered (GE) foods would increase your grocery bill. When a ballot initiative to label GE foods was first announced, voters overwhelmingly approved it. But by the 2012 election, it narrowly lost. A study [3] proving that GE food labeling would make food costs in California skyrocket may be why voters had such a sudden change of heart. How was that conclusion reached? The study authors partially funded by the No on 37 campaign began with a wrong assumption [4] . American consumers are just like European consumers, they figured. And, just like in Europe, when GE foods must be labeled, most food manufacturers will instantly remove all GE ingredients from their products. Because GE ingredients like corn and soybeans are present in almost all processed foods, reformulating every food sold in California to remove them would be massively disruptive to the food industry. In fact, it would raise food prices! But their assumption is wrong. European [5] s are willing to pay more for non-GE food, but most Americans aren't. So why would food manufacturers reformulate their products, resulting in higher prices, if they know most Americans wouldn't pay for it? They wouldnt. The average voter in California never heard these details. They just heard that their food prices were going up unless they voted no on Prop 37. So they did. 2. Throw Out the Data You Dont Like In 2012, researchers in the Netherlands published a study [6] finding that organic agriculture yielded only 80 percent as much as conventional agriculture. Wow, is that true? Well, if it is, this study certainly doesnt prove it. The study doesnt actually prove much of anythingbecause the researchers disregarded any data they did not like. Michael Hansen, a senior scientist at Consumers Union and a formidable agriculture expert, read the study and reflected that, When you actually look at the paper, you'll see it's incredibly biased in favor of conventional ag, but in a very technical way. The study authors essentially picked and chose which data to include, excluding any organic method that did not meet the very strictest definition of the term organic, and throwing out any data from conventional systems that included unrepresentative yield levels. Details provided on that say: Yield data for industrialized countries were considered unrepresentative if conventional yields appeared to be far below the regional average, unless this was caused by factors that can also occur in real farming situations, such as pests, diseases or droughts. For developing countries unrepresentative implied conventional yield levels that seemed to be far below yields achieved under best farmers management.
www.alternet.org/print/food/6-ways-scientific-studies-can-trick-you 1/3

5/13/13

6 Ways Scientific Studies Can Trick You

In other words, wrote Hansen, who is well-traveled and very familiar with agriculture in the Global South, for developing countries, rather than compare organic to what local farmers that use some chemicals find, they dismiss all but the yields achieved using lots of chemicals under the best conditions which is not what farmers actually face in developing countries. Finally, Hansen concluded, look at their main hypothesis: 'Our hypothesis was that the closer conventional agriculture gets to the potential or water-limited yield, the larger the yield gap between organic and conventional systems will be.' In other words, they're not interested in the conditions that farmers face in the real world in developing countries. 3. Oops, We Cant Detect the Chemical in Question Whenever you hear that there is none of a chemical in something, its time to ask: Whats the detection limit? Whats the smallest amount of the chemical they were looking for? Take, for example, early studies of the pesticide Imidacloprid by its manufacturer, Bayer. A recent European report [7] tells what happened when beekeepers began alleging that the pesticide was the cause of mass bee die-offs. Bees consume nectar and pollen, so Bayers first step was checking to see whether any of its pesticide was present in nectar or pollen. In 1993, Bayer set the detection limit at 10 parts per billion (ppb). When it tested for its pesticide in nectar or pollen of treated crops, it couldnt find any or it couldnt quantify the amount detected. Clearly, this pesticide wasnt harming any bees, because the bees werent exposed to it in nectar or pollen! Six years later, in 1999, a study of sunflowers found 3.3 ppb of the pesticide in pollen and 1.9 ppb in nectar amounts far below the previous 10 ppb detection limit! And, in 2001, scientists found that chronic exposure to 0.1 ppb of the pesticide kills a bee in 10 days. How much money did Bayer earn from pesticide sales while it stalled the science for six years with its high detection limit? 4. Findings That Arent Statistically Significant GMOs Cause Tumors in Rats screamed the headlines after a French scientist named Seralini published findings based on a two-year feeding study using Monsantos Roundup Ready corn and its pesticide Roundup. Oh my god, this is terrible! Americans have been eating the variety of Roundup Ready corn in question since 2001! Well all get cancer! Okay, take a step back. Were the findings statistically significant? No, they werent. Consumers Union scientist Michael Hansen points out that the study only used 10 rats of each sex for each group tested. After publication, the studys author also noted [8] , the sample size of their treatment groups was too small to allow them to draw conclusions with regard to long-term carcinogenicity and mortality. Oops. That said, in this case, Hansen points out that the data suggests that there might be something there. For the most part, the control rats were healthier than the rats that were fed Roundup or Roundup Ready corn. According to Hansen, if the findings were entirely random, one would expect that the number of control group rats afflicted with each morbidity would sometimes be more and sometimes be less than the number of sick rats in the treatment groups. Hed like to see further study using a statistically significant sample size. 5. Design the Study to Get the Results You Want When the EPA allowed the commercialization of Bayers pesticide Clothianidin, they required the company to conduct a study proving that the pesticide would not harm bees. So Bayer performed a study [9] . It placed four beehives in the middle of 2.5 acres of treated canola. Bees, of course, go as far as several miles from their hive to forage so no doubt the bees feasted on plenty of pesticide-free nectar and pollen during the study. Sothe scientists found that their pesticide caused no harm to the bees. Beekeepers were outraged enough when they discovered the inadequacy of the study, but a revelation [10] that came out two years later made matters even worse. In the U.S., clothianidin is used on corn and canola. Canola is a minor crop in the U.S., whereas corn is the most commonly grown crop weve got. Clothianidin-treated corn has about 10 times the levels of pesticide in its pollen as treated canola. Perhaps thats why Bayer chose to perform its study on canola. 6. All of the Above For a Bad Science Sampler, check out the studies used to justify the safety of the genetically engineered AquaAdvantage salmon [11] . You name it, they did it [12] . Again, Michael Hansen easily poked holes in their work until the study resembled Swiss cheese. They culled young fish with the worst deformities, thus excluding them from the data. They used sample sizes as low as six fish. And they used a detection limit too high to detect any growth hormone in the muscle and skin of any fish. (The GE salmon are engineered to produce extra growth hormone, so they might have looked a little harder to find it if they wanted to show there was no difference between GE salmon and non-GE salmon.) As you can see, just because a study finds a conclusion, that doesnt mean its correct. Poorly designed studies and statistically insignificant results happen all the time. Now, if only journalists did a little bit more digging about the validity of outrageous studies findings before they report on them.
www.alternet.org/print/food/6-ways-scientific-studies-can-trick-you 2/3

5/13/13

6 Ways Scientific Studies Can Trick You

See more stories tagged with: science [13] , food [14] , gmo [15]
Source URL: http://www.alternet.org/food/6-ways-scientific-studies-can-trick-you Links: [1] http://www.alternet.org [2] http://www.alternet.org/authors/jill-richardson-0 [3] http://www.noprop37.com/files/Alston-Sumner-Prop-37-review.pdf [4] http://www.sourcewatch.org/index.php/Prop_37#Industry_Funded_Study_Claims_Prop_37_Will_Raise_Food_Prices [5] http://www.agbioforum.org/v7n12/v7n12a13-mccluskey.pdf [6] http://www.wageningenur.nl/upload_mm/9/a/9/4eff5e4c-0058-4fac-9441288b83a105d4_The%20crop%20yield%20gap%20between%20organic%20and%20conventional%20agriculture%20De%20Ponti%20et%20al%202012.pdf [7] http://www.eea.europa.eu/publications/late-lessons-2 [8] http://www.efsa.europa.eu/en/press/news/121128.htm [9] http://www.beeculture.com/storycms/index.cfm?cat=Story&recordID=714 [10] http://www.alternet.org/story/154039/have_bees_become_canaries_in_the_coal_mine_why_massive_bee_dieoffs_may_be_a_warning_about_our_own_health [11] http://www.sourcewatch.org/index.php/Aquadvantage_salmon [12] http://www.sourcewatch.org/index.php/Concerns_About_Science_Justifying_the_Safety_of_AquAdvantage_Salmon [13] http://www.alternet.org/tags/science-0 [14] http://www.alternet.org/tags/food-0 [15] http://www.alternet.org/tags/gmo [16] http://www.alternet.org/%2Bnew_src%2B

www.alternet.org/print/food/6-ways-scientific-studies-can-trick-you

3/3