Module Eight Lesson Three Notes Name_______________________

THE VALIDITY OF A STATISTICAL STUDY OR EXPERIMENT

Now that you have learned some statistical techniques, you are going to put your new knowledge
into practice.  In this lesson, you will read two articles. As you are reading the articles, think
about what you have learned: 
1- Was this from a simple random sample?
2- How did they design the study?
3- Did they make correct inferences to the population? 
4- Can you evaluate the claim the study is making?

Validity of a Study
Vocabulary Review
In this lesson, we are going to be given a real-life study or article, and you are going to have to make
the decision as to whether or not the conclusions that were made were valid or invalid.

Statistical study – A study where data is collected and the statistics (means, proportions etc.) are
examined.

This is where a survey is usually sent out and data is collected. Just remember you must take an
SRS in order to ensure that you have a mini population! If you do not have an SRS, you cannot
make inferences about the population.

Experiment - an investigator imposes a change or treatments on one or more group(s), often

called treatment group(s).

What’s important about an experiment

With an experiment, we must make sure the treatment was randomly applied. If not,
establish cause and effect!

New Vocabulary- News can be so misleading!

Validity - validity tells a researcher that the conclusions of an experiment were correct and
justified through sound, statistical reasoning, and experimentation.

Invalid – some part of the statistical process was not done correctly, and the results cannot be
applied to the larger population.

Don’t take the article for face value or a line that is highlighted. Read the background
information!
What’s Important with Articles/Websites
Statistics and Websites can mislead you very easily! You need to make sure that you are able to
understand how to understand when data is misrepresented.

What to look for:

1) Source of the data – to make sure the source is valid and reliable. Simply because the source
has the word science in it, we aren’t guaranteed that it is accurate.
2) Check and see how the data was collected. Was it random?
3) Is it an opinion piece or was actual data gathered?
4) Wording of the question – did the question lead the audience in a certain direction? For
example: How strongly do you agree that the drinking age should be lowered to 18?
5) A leading psychologist stated that too much TV leads to violence. Do you agree, somewhat
agree, or disagree with this statement?

Kids and Screen Time: What Does the Research Say?

By Juana Summers
August 28, 2014

Kids are spending more time than ever in front of screens, and it may be inhibiting their ability to
recognize emotions, according to new research out of the University of California, Los Angeles.

● The study, published in the journal Computers in Human Behavior, found that sixth
graders who went five days without exposure to technology were significantly better at
reading human emotions than kids who had regular access to phones, televisions, and
computers.
● The UCLA researchers studied two groups of sixth graders from a Southern California
public school.
● One group was sent to the Pali Institute, an outdoor education camp in Running Springs,
California where the kids had no access to electronic devices.
● For the other group, it was life as usual.
● At the beginning and end of the five-day study period, both groups of kids were shown
images of nearly 50 faces and asked to identify the feelings being modeled.
● Researchers found that the students who went to camp scored significantly higher when
it came to reading facial emotions or other nonverbal cues than the students who
continued to have access to their media devices

2
Question 1 - What is the source of the data?

The source of data is its origin or where it was published.

 Is it reliable?
 Is it in a trusted journal?
 Is it an opinion blog?

The study, published in the journal Computers in Human Behavior, found that sixth graders who
went five days without exposure to technology were significantly better at reading human
emotions than kids who had regular access to phones, televisions, and computers.

Question 2 - Describe the design of the study.

What did experiment look like?

Group 1 was sent to Pali Institute where they had no access to electronic devices.

Group 2 was life as usual.

At the beginning and end of the day, both groups of kids were shown images of nearly 50
faces and asked to identify the feeling being modeled.

Was there any random assignment to treatments?

NO!

Question 3 - After analyzing the graph, evaluate the claim that the “kids who had been to camp …
improved significantly more.”

Control Group: Number of Errors Before Experiment - 12.24

Number of Errors After Experiment – 9.81
Difference in errors: 12.24 - 9.81 = 2.43
Control Group: Number of Errors Before Experiment – 14.02
Number of Errors After Experiment – 9.41
Difference in errors: 14.02 – 9.41 = 4.61

4.61 VS. 2.43

The camp DID make a difference when it came to making less errors after the experiment vs. only
2.43 errors. That is nearly double.

3
 What was the conclusion?
The UCLA researchers studied two groups of sixth graders from a Southern California public
school. One group was sent to the Pali Institute, an outdoor education camp in Running Springs,
Calif., where the kids had no access to electronic devices. For the other group, it was life as usual.

Find anything in here that might not make for good inferences?
Take a look at the sample!
It was not a simple random sample.
What does that mean? That means that most likely we cannot make inferences beyond the sixth
graders that were part of the study.
Why Teenagers Need More Sleep
By C. Price
A study was done by Christine Price in the Journal of NCVPS mathematics that stated, “Teenagers
Grades are Decreasing Due to Lack of Sleep.”
● The study was done with 150 teenagers from NCVPS.
● A survey was sent out to all 150 students and each student had to respond with their grade
in math and the amount of sleep they got each night.
● Students who got 7 - 8 hours sleep a night were grouped and students who got less than 7
hours of sleep were grouped. The grades for each of the two groups were analyzed.
● The students’ math grades for those who got 7 - 8 hours of sleep had an average of 88% in
Math III.
● The students who had less than 7 hours of sleep had an average of 84% in Math III.
● The study concluded that sleep affects all teenagers’ grades.

Do you agree with the conclusion? Why or why not?

1) Is the source reliable?

 Journal of NCVPS mathematics
 The journal is for a small institution.
 No information is provided about the researcher.
 Do they have a background in mathematics or research?
2) Is the sample random?
 It was sent out to 150 students
 It does not state that 150 students were a random sample.
 Therefore, it is not a random sample.
3) Can you make inferences to a larger population?
 Was there a random sample?

4
  No!
 No random sampling implies that the results only apply the group of students in the study.

The students’ make grades for those who got 7 – 8 hours of sleep had an average of 88% in Math III.
The students who had less than 7 hours of sleep had an average of 84% in Math III. The study
concluded that sleep affects all teenagers’ grades.

Do you agree with the conclusion? Why or why not?

Look at the difference in the averages.
4% is not a huge difference in averages to say that less sleep really causes you to have lower
grades.


7 – 8 hours of sleep Less than 7 hours of sleep

88% 84%

Module Eight Lesson Three Notes One - YOU TRY

Scenario 1
Technology in the Classroom
● A survey of 6,000 middle school and high school teachers found that using technology
inside and outside the classroom helped their students in many ways.
● Researchers concluded that there was a difference between teachers in the comfort
level of technology use in the classroom.
● One of the most notable findings of the study was that students are using their mobile
devices in the classroom to succeed.
● 75% of the teachers surveyed stated that their students used some sort of mobile device
to complete their homework.
● This could entail looking up an assignment, submitting an assignment or conducting
research on a laptop.
● This study was done over the course of three months from January 2010 to March 2010.
● The 6,000 teachers surveyed were from Puerto Rico, the U.S. Virgin Islands the United
States.
● The survey was posted online and any teacher with access to the website could respond
to the survey.

1) Is the source reliable?

 No source is listed.
 There are 6,000 participants in the study.

5
  Since no source is listed, we may not find the conclusions of the study valid.
2) Is the sample random?
 The survey was posted online and any teacher with access to the website could respond to
the survey.
 Therefore, the sample was not random because the only people that answered were
teachers with access to the website. Anyone with access to the website was left out. The
survey was not sent out to a random group of teachers, and all were asked to respond.
 Since it was not a random sample, we cannot make inference to the larger population.
3) Why might we need to be careful about this survey based on this line?
 Notice how it states that the survey was posted online.
 What does this tell you?
 The teachers that had access to the internet were more likely to respond to the study!
 What about the teachers that did not have access to this online survey? All of their
answers were not included as part of the study!

Based on the previous slide, what do you think about the conclusion listed below?
One of the most notable findings of the study was that students are using their mobile devices in
the classroom to succeed.
 75% of the teachers surveyed said that their students used some sort of mobile device to
complete their homework. This could entail looking up an assignment, submitting an
assignment or conducting research on a top.
4) Do you think this conclusion is valid? If it is invalid, is it an over or underestimate of the data?
 the conclusion might be an overestimate for the amount of students that use a mobile
device in the classroom.
 Why?
 Because if the teachers are using technology to answer the questions, then most likely
they are from wealthier districts where more students are likely to use technology at
home.
This study was done over the course of three months from January 2010 to March 2010.

5) Based on the dates of the survey do you think it is still valid today?
 This study was done a while ago for some studies it might hold valid over time.
 However, technology is constantly changing, and more and more students and teachers
are using technology in the classroom.
 It might be best to see if we could conduct this study again with teachers using a simple
random sample.

6
 Scenario 2
“The Value, Usefulness, and Cost of College”
By Wendy Wang and Kim Parker - Pew Research Center
P A study done by the Pew Research Center
examined the differences between men and
women and their thoughts on the value of a
higher education.
P The graph to the right displays men’s and
women’s responses as to who they believe paid
for their college education.

What conclusions can you make from the graph in regard to the percentage of men and women
that believe they paid for college?
P It appears as if more women believe their parents paid for college (40%) compared to only
29% of men who believe their parents paid for college.
P It looks as if more men believed that they paid for college on their own (29%) as compared
to only 19% of women who believe that they paid for college on their own.
Did one pay more than the other?
P Both men and women have about equal beliefs that both student loans and scholarships
paid for a portion of their college.
Module Eight Lesson Three Notes Two
Misleading Graphs

Have you ever played the game three truths and a lie? Guess what? Statistical graphs are kind of like playing the
game three truths and a lie. You can have one set of data and present it in four different ways. Three ways will
distort the data to make the reader believe what you want them to believe and the fourth one will display the data
correctly. You know what that means? The best way to mislead someone using statistics is with graphs. In this
section we will look at how graphs can be manipulated so that they can display data incorrectly. That means by
the end of this lesson no one will be able to fool you!

HOW TO LIE WITH GRAPHS

7
 P Graphs can be manipulated very easily to display data inaccurately.
P Remember what we talked about in previous lessons? You can’t believe every statistic you
hear.
P What was the sample?
P How was the data collected?
P In this lesson let’s talk about how graphs can be used to accurately represent information.
P This is so important because you will see graphs on the news, social media, in well trusted
magazines, and online.

The moral of the story is the following:

Don't’ believe what you see until you use your statistical knowledge to evaluate the graph!

How exactly can statisticians get away with creating a graph that is a misrepresentation of the
data?

1. They exaggerate the vertical axis. This means they make the distance between the values
larger than it should be.

The graph is always start at 0 on the y-axis to show an accurate height of the graph.
Don’t start the graph at 10, 20, 30 etc.

2. They make a pie chart that is larger than 100%. Remember those pie charts from your
previous courses?
They represent 100% of the data so the pie wedges should never add up to more than 100%!
3. They create graphs that are made up of pictures that are different sizes.

 Statisticians will create graphs with different size pictures.

 They will often create one picture larger than another, even if the data does say that it is
in fact SMALLER!

EXAGGERATING THE VERTICAL AXIS

In a recent magazine article, you read that Apple cell phone users were significantly more
satisfied than Android cell phone users.

The graph that accompanied the article is shown below.

Before you go out and switch out your Android phone for an Apple cell phone
you remember something your statistics teacher in high school said: “The easiest way to lie with
statistics is with a graph.”
You examine the graph and find something is misleading.

8
What is it?

Ways the graph is misleading:

1) the vertical axis does not start at 0.

2) the vertical axis is exaggerated. Notice that it only goes up by 2% each time.
3) there is only a 4% difference in the satisfaction rating between Android and Apple cell phone
users.
That is NOT a huge difference.
4) The bar for the Apple cell phone user is a lot wider than the bat for the Android cell phone.
That would lead the reader of the article to believe that more Apple cell phone users are
satisfied.
5) One other way to mislead with graphs is not to label the vertical axis with numbers.
6) This gives the readers no idea how much data is in each category.

Using Pictures as a Misrepresentation

In a recent Facebook community post, a neighbor was concerned about the number of increasing
dead frogs on the side of the road. She put together this graph to show neighbors how bad the
problem really was becoming.

The statistician in you told all your neighbors that while frog deaths may be increasing the graph
your neighbor made is not necessarily a good representation of the number of dead frogs seen on
the side of the road.

Why??

1) Notice that the second frog is MUCH larger than the first frog!
2) If you want a true representation of the data, all your pictures must be the same size.

9
 3) The larger frog makes you think WOW, frog deaths really have really multiplied because
the picture is so much larger!

Examine the vertical or y-axis.

 If you look at the y-axis there seems to be about 15 frog deaths in 2015 and only about 35 frog
deaths in 2018.
 While that may be a large difference, the second frog is much more than two times as large!

Your school recently did a survey to determine their favorite actors.

Favorite Actor Percentage

Christian Bale 41%

Ben Affleck 23%

George 20%
Clooney

Justin Bieber 16%

Based on the percentages and the graph what do you notice about Ben Affleck?

 notice that Ben Affleck is at 23% which is only slightly larger than both George Clooney
(20%) and Justin Bieber (16%)
 in order to make Ben Affleck look like he is more popular than he actually is, he is moved to
the front of the graph.
 Also notice the 3D feature here. The 3d feature exaggerates Ben Affleck even more.
 The placement makes him look like he is pretty close in popularity to Christian Bale when in
reality he is far behind.

Check the percentages.

10
 Always check to make sure the percentages add up to 100%!
 In this case, they do but often they will make pie charts where they are more than 100%.
 Check out the percentage of Christian Bale – it states that he was favored by 41% however
the “piece” of the pie is more than 60%!

11