# School of Information University of Michigan

Sampling and experimental design

Exercise 4.63 (to help out with problem 1 on PS3)
 Pr(domestic abuse) = 1/3, or maybe 1/10  Sample of 15 women; 4 have been abused  If p=1/3, what is Pr(X>=4)?  If p=1/10, what is pr(X>=4)?  Given evidence from the sample, which abuse rate

seems more plausible?  Note: this is a preview of thinking about sampling distributions

Exercise 4.63 (to help out with problem 1 on PS3)
 p = 0.33 or 0.1  n = 15 women, x=4  If p=1/3, what is Pr(X>=4)?  > pbinom(3,15,0.33,lower.tail=F)  [1] 0.7828694  If p=1/10, what is pr(X>=4)?  > pbinom(3,15,0.1,lower.tail=F)  [1] 0.05555563  Given evidence from the sample, which abuse rate

seems more plausible?

Types of statistical studies  Surveys  Experiments  Observational .

Surveys  Subjects fill out questionnaires  Choices:  Random sample  Same characteristics as the population  list the whole population  draw random numbers to select a subset (sampling without replacement)  Quota sampling (stratified sampling)  Every subset of the population has an specified chance of being selected .

Sampling frame and target population target population sampling frame sampled population nonresponse ineligible .

errors in the survey process target population coverage error sampling frame sampling error sample nonresponse error respondents .

definitions of non-sampling errors  selection bias – sample frame does not correspond to target population  nonresponse bias – respondents with a certain characteristic are more likely to not fill out a survey  (e. other ethnicities)  self-selection – if survey is mailed out or available to a wide audience online.g. those who fill it out may have a bias  e. US Census undersamples blacks vs.g NRA survey  ABC’s online survey of ‘addiction to the internet’ .

Kinsey reports  studies of human sexuality  widely used to support the claim that 10% of the male population is homosexual (also an example of a persistent statistic. that was never explicitly made by Kinsey)  criticism:  over-representation of some groups in the sample:  25% were.  non-response bias . prison inmates  5% were male prostitutes. or had been.

didn’t know  Year later.definitions of non-sampling errors (cont’d)  question effects – differently posed questions can yield measurably different results  e. 1993  Roper poll: “Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?”  22% . “Poll finds 1 out of 3 Americans Open to Doubt There is a Holocaust”.didn’t know . Los Angeles Times.possible  8% .g.possible  12% . April 20. second Roper poll: “Does it seem possible to you that the Nazi extermination of the Jews never happened. or do you feel certain that it happened?  1% .

definitions of non-sampling errors (cont’d)  survey format  in person interview. phone. mail. online  sensitive questions are especially affected  position of a question on the survey  survey length  placement of instructions  interviewer effects  behavioral considerations  subjects may want to give the “right” answer .

0  difficulty in sampling: what is a family? .Family size sampling  How many children are there in your family including you?  data for this class  national average # kids/per family for families having kids ~ 20 years ago: 2.

Telephone survey example target population: inhabitants of a town sampling frame: people listed in the phonebook nonresponse: people who could not be reached or declined to take the survey sampled population ineligible: people who have moved away undersampled: households without a landline .

telephone survey sampling strategy  Call a random telephone #  Ask to speak to the person in the household whose birthday falls next  What kind of sampling bias occurs with this strategy?  people in large households underrepresented  young people underrepresented .

Phone book experiment  sampling by last name  is it really random?  what are the sources of non-randomness?  two people with the same last name sharing the same phone #  one person having multiple phone lines  combination of the two .

worst statistic ever (from damned lies and statistics)  "Every year since 1950."  from a 1995 journal article  how do we know this statistic is bad? . the number of American children gunned down has doubled.

exponential growth  assume the # of children gunned down in 1950 is 1  the number of children gunned down in  1951 : 2  1952 :  1953 :  1960 :  1980 :  2000 : .

# of children gunned down over time (assuming doubling every year since 1950) 500 # kids gunned down 0 1950 100 200 300 400 1952 1954 year 1956 1958 .

# of children gunned down over first 20 years (assuming doubling every year since 1950) 0 e+00 1 e+05 2 e+05 3 e+05 4 e+05 5 e+05 1950 # kids gunned down 1955 1960 year 1965 .

# kids gunned down (45 years of exponential growth) # kids gunned down 1 e+00 1950 1 e+04 1 e+08 1 e+12 1960 1970 year 1980 1990 2000 .

"[1]  How does the difference in wording change things?  Are there phenomena that truly are exponential? .what the study actually said  The CDF's The State of America's Children Yearbook--1994 does state: "The number of American children killed each year by guns has doubled since 1950.

world population and production source: federal reserve bank of Minneapolis .

.

the growth of the internet .

growth of wikipedia .

population projection bias: are we most likely to be alive during the population peak? .

a bit of philosophy: anthropic principle  Why are the parameters of the universe fine tuned exactly the way they are (out of all the possible parameters)  How did everything balance out exactly right to allow for life to exist?  (weak) anthropic principle.org/wiki/Anthropic_principle . a truism:  conditions that are observed in the universe must allow the observer to exist  http://en.wikipedia.

Experimentation  Control one variable – measure effect of another  Randomization  taste tester tastes soda in random order  survey questions appear in random order  Blindness  subject does not know which treatment they are receiving (placebo vs. new medication. or diet vs. ESP research .g. regular)  double-blind: experimenter making observation does not know which treatment the subject received  e.

UN Survey  Please complete the survey individually without consulting with your neighbors .

 the experimenter determines which experimental units receive which treatments  when carefully designed can be used to prove causation  e. “I planned to give myself an ulcer. then treat myself. to prove that H.Experimentation vs.  Marshall then took the unusual step of using himself as guinea pig and drank a solution containing the newly-discovered bacteria. pylori can be a pathogen in normal people. observation  In an experiment. but the resulting stomach inflammation was clearly surrounded by the distinctive curved bacteria. In all the patients with duodenal ulcer and 80 percent of patients with gastric ulcer. the spiral organism H.” he explained in one interview. . He did not develop an ulcer.g. stomach ulcer Nobel prize  Barry Marshall and Robin Warren conducted a 100-patient study with 65 suffering from gastritis directly due to the presence of the bacteria. pylori was present.

g. smoking causes lung cancer . observation  In an observational study.Experimentation vs.  we compare units that happen to have received each of the treatments  useful for identifying possible causes but cannot reliably establish causation  Useful when it would be impractical or unethical to conduct and experiment  e.

observational studies  claims made by test-prep organizations  students taking a test-prep class increase their SAT scores by 100 on average  is this proof that test-prep classes are the cause of the improvement?  What are possible lurking variables? Before Control group Treated group After .

freakonomics  which studies were observational?  which studies were experiments?  how did the studies minimize sampling bias? .

freakonomics: studies  bagels  crime stats  parenting  school choice  cheating teachers  sumo wrestlers  real estate agents .

we have to consider  population  sampling frame  non-response  bias resulting from survey/experimental design and sampling .summary  3 types of studies  surveys  observational  experiments  in all of them.