Professional Documents
Culture Documents
Ultimate
Guide to
Effective
Data
Collection
The
Ultimate
Guide to
Effective
Data Collection
Chapters:
Introduction Why Data Quality is Crucial
Chapter 7
The MECE Framework: Mutually Exclusive,
Collectively Exhaustive Questions
2. Piloting a Survey
4. Census Survey
Introduction: “In God we trust.
Why Data All others must
Deming describes the importance that huge losses — have been stalwarts in using
organisations, funders, and people in data to make the smallest decisions.
general give to data today. Data has become
fundamental in nearly every aspect of life. In addition, the development and policy spaces
have seen the use of data to drive decision
Businesses and corporates use data to making and increase impact. Non-profit and
make better decisions, increase profits, government organisations are using data to
grow revenues and improve efficiency. inform decisions, such as how much money to
Organizations such as hedge funds, stock invest in a particular project or how to improve
brokers and investment banks — where a split impact per dollar spent.
second delay in decision making can lead to
With the growing importance placed on data, • How should you write the questions in your
surveys have become an indispensable tool survey?
for every organization, from billion-dollar tech • Who should you survey?
companies to rural nonprofits.
This ebook is designed to take you through
Creating a survey seems simple — just ask these questions and help you design a survey
a few simple questions, and you’ll get back that will give you high-quality data. Chapter
data to solve your every problem. However, 1 will help you think through the purpose,
designing a survey correctly takes time and outcomes and indicators of your survey.
knowledge. A poorly-designed survey will Chapters 2 and 3 will help you determine
lead to useless data, wasting your time and what data collection method you should use,
money. As computer scientists say, “Garbage as well as whether you need a qualitative or
in, garbage out.” quantitative survey. Chapters 4-8 address
writing the questions in your survey — what
Case Study you want to ask and best practices around
how to ask it. Lastly, Chapter 9 covers all
In 1936, the Literary Digest polled 2.4 million aspects of sampling your population —
people on the upcoming U.S. presidential sampling methods, best practices, and a
election. After conducting one of the largest quick sample size formula.
and most expensive polls in history, the
Chapter 1:
Literary Digest predicted that Alfred Landon
would win the election 57% to 43% against
the incumbent Franklin D. Roosevelt. At the
same time, George Gallup polled around
50,000 people and predicted a win for
Roosevelt. Survey Design:
The actual results of the election were 62%
for Roosevelt against 38% for Landon. The
Literary Digest poll’s prediction had an error of
Creating Your
19%, the largest error in the history of major
public opinion polls in the U.S. Research
The explanation for this error — survey
design. Though Gallup surveyed only 2%
of the people that the Literary Digest did,
Question,
Gallup’s data was far more accurate because
he designed the survey and sampled the
population more effectively.
Outcomes
Designing a survey involves
several considerations:
and Indicators
• What is the purpose of your survey? What
data are you looking to collect? The most important part of your survey is
• How can you best collect that data? What determining your purpose – why are you
sort of survey and research methodology conducting this survey and what do you want
should you use? to learn?
Setting your research question, outcomes and 2. Why am I collecting this information?
indicators clearly makes writing the rest of
your survey far simpler. Moreover, it ensures To measure the impact of my NGO’s
that everyone in your organization is on the programme so I can compare it to my other
same page about your survey. programmes and communicate its impact to
my funders.
This chapter will show how to build your
research question, outcomes and indicators 3. Who can I collect data from?
through an exploration of two case studies.
I can collect data from the students and
Determining Your Research teachers where the NGO works.
Question
Case Study #2
Before you start collecting data, it is important
to figure out your research question. As part Your NGO works with women’s SHGs from a
of this process, you also should think broadly district in Jharkhand. Your model for change
about who you can survey. is two fold – you directly impact the SHGs you
work with by helping them fundraise, which in
To fully formulate your research question, you turn empowers women who participate in the
should be able to answer three questions: SHGs.
You run an education NGO, which works on a 3. Who can I collect data from?
teacher training program across 1,500 schools
in Bihar and Uttar Pradesh. As part of your I can collect data from the SHGs that my NGO
program, your team trains teachers on how to works with and the families of the members
improve your students’ reading skills. of the SHGs. I can also collect data on SHG
funding from my program officers.
1. What is the research question behind
collecting data on the impact of the Tool Tip
programme? The second question might seem
unnecessary. After all, it doesn’t directly go
How has my NGO improved the teaching skills into your research question. However, it is an
of the teachers we work with, and how has essential part of the survey design process.
this improved student reading skills? If you cannot fully answer why you are
conducting your survey, you are not ready to
start your survey.
Once you can answer these three questions Outcomes
for your own survey, you have figured out your
research question and target population. • Change in the funds raised by the SHGs
before and after they entered my programme
Determining Your Outcomes • Empowerment of women in the district
Your NGO works with women’s SHGs from a Once you know what data you want to
district in Jharkhand. Your model for change collect, it is important to figure out which data
is two fold – you directly impact the SHGs you collection method you will use. Each method
work with by helping them fundraise, which in has its own advantages, disadvantages and
turn empowers women who participate in the use cases.
SHGs.
Outcomes
Observation
• Change in the funds raised by the SHGs
before and after they entered my programme Seeing is believing, they say. Making direct
• Empowerment of women in the district observations, when the situation allows for it,
is a very quick and effective way of collecting
Indicators data with minimal intrusion. Establishing the
right mechanism for making the observation is
• Measure funds raised by SHGs each month, all you need.
starting when they enter your programme
• Take qualitative feedback from women on Advantages
whether they feel empowered, each month
• Measure women’s likehood to make • Non-responsive sample subjects are a non-
household decisions, each month issue when you are simply making direct
observations.
After creating your research question, • This mode does not require a very extensive
outcomes and indicators, you will know and well-tailored training regime for the survey
exactly what data you need to collect. From workforce.
there, it is much easier to design the rest of • It is not as time-consuming as the other
your survey. models that we will discuss below.
• Infrastructure requirement and preparation finer nuances, leaving the responses open to
time are minimal. interpretation. Interviews and Focus Group
Sessions, as we will see later, are instrumental
Disadvantages in overcoming this shortfall of questionnaires.
• Response rates can be quite low. Choosing
• Heavy reliance on experts who must know the right question types can help to optimize
what to observe and how to interpret the response rates, but very little can be done to
observations once the data collection is done. encourage the respondents without directly
• Possibility of missing out on the complete conversing with them.
picture due to the lack of direct interaction
with sample subjects. Use Case
Questionnaires Interviews
Questionnaires, as we consider them here, are Conducting interviews can help you overcome
stand-alone instruments for data collection most of the shortfalls of the previous two data
that are administered to the sample subjects collection methods that we have discussed
either through mail, phone or online. They here by allowing you to build a deeper
have long been one of the most popular data understanding of the thinking behind the
collection methods. respondents’ answers.
Advantages Advantages
• Questionnaires give the researchers • Interviews help the researchers uncover rich,
an opportunity to carefully structure and deep insights and learn information that they
formulate the data collection plan with may have missed otherwise.
precision. • The presence of an interviewer can give
• Respondents can take these questionnaires the respondents additional comfort while
at a convenient time and think about the answering the questionnaire and ensure
answers at their own pace. correct interpretation of the questions.
• The reach is theoretically limitless. The • The physical presence of a persistent, well-
questionnaire can reach every corner of the trained interviewer can significantly improve
globe if the medium allows for it. the response rate.
Disadvantages Disadvantages
Chapter 3:
together at the same time can encourage
them to engage in a healthy discussion and
may help researchers uncover information that
they may not have envisaged.
• It helps the researchers corroborate the facts
instantly; any inaccurate response will most
likely be countered by other members of the
Qualitative vs.
focus group.
• It gives the researchers a chance to view
both sides of the coin and build a balanced
Quantitative
perspective on the matter.
Disadvantages
Research
• Finding groups of people who are relevant
to the survey and persuading them to come
together for the session at the same time can Before you formulate your questionnaire, it is
be a difficult task. important to consider what type of information
• The presence of excessively loud members you’d like to collect — qualitative or
in the focus group can subdue the opinions of quantitative. Both qualitative and quantitative
those who are less vocal. research have their places in data collection.
• The members of a focus group can often fall
Quantitative Research investigates the “why” and “how” of your
research question.
Quantitative research (derived from the word
“quantity”) describes research that produces Examples of Qualitative Questions
countable or numerical results.
Do you like your commute to and from work?
Examples of Quantitative Questions Why?
How do you generally get to and from work?
How long does it take you to travel to work? Why is the metro your favorite form of
transportation?
□□ 0-20 minutes Is there anything else you’d like to tell us
□□ 21-40 minutes about your commute?
□□ 41-60 minutes
□□ Over 1 hour When to Use Qualitative and
Quantitative Research
What forms of transportation do you use while
traveling to and from work? Please select all Qualitative research is often used as
that apply. exploratory research. It is helpful to provide
insights into the problem you want to
□□ Personal car or taxi research more, or it helps to identify ideas and
□□ Auto hypothesis for future quantitative research.
□□ Rickshaw Qualitative research also is useful in learning
□□ Bicycle more about the “why” and “how” behind your
□□ Metro question.
□□ Other
Quantitative research is a great way to
Would you move to a new location just to generate numerical data, create usable
decrease your commute time? statistics, and generalize results or uncover
patterns from a larger population.
□□ Yes
□□ No To figure out whether you should use
□□ Not applicable qualitative research, quantitative research, or a
mix of the two, look at your research question,
Rate each form of transportation on a outcomes and indicators. (If you don’t have
scale of 1-5. these, go back to Chapter 1!)
(1 is strongly dislike, 2 is dislike,
3 is neutral, 4 is like, 5 is strongly like) Examples
□□ Personal car or taxi
□□ Auto Research question: Are the children in my
□□ Rickshaw classrooms improving?
□□ Bicycle
□□ Metro Quantitative data:
• Ask participants to rank their Now that you are aware of the different
independence on a quantitative scale elements of a questionnaire, the next step is
before and after the program to think about the various types of questions
• Ask participants if they feel more that you would want to ask in a given
independent (Yes or No question) questionnaire. The below questions can help
• Ask participants how likely they are to you decide which questions you should ask.
engage in measures of independence
(i.e. standing up to their husband, taking 1. What Kind of Information Do
more control over household finances) on You Need?
a quantitative scale before and after the
program Different categories of information include:
personal background (name, religion, age,
Qualitative data: caste, gender, etc.), education information,
health information, government schemes
• Ask participants how they feel after subscription, etc.
completing the program
• Ask participants about whether they think For example, say that we are looking to
they are likely to engage in measures measure the change in students’ learning
of independence (i.e. standing up to outcomes. We could decide that we need
their husband, taking more control over some personal details of the students (age
household finances) before and after the and gender) as well as learning levels of the
program students, classroom activities of the teachers,
• Ask participants’ friends, husbands, and/ and some school-level information. We would
or children about the participants’ behavior not need details on the religion or caste of the
before and after the program students, personal details on the teachers, or
information about the students’ families.
As the previous examples show, many
research questions can be answered using Tool Tip
both quantitative and qualitative research. To To arrive at the different sets of information,
decide which is right for you, think about your put the outcome in the centre of a paper and
research question, what questions you need write all the things that can possibly impact
to answer, and the type of data that you are that outcome. Talk to your program officers
hoping to collect. and field staff about it.
2. What Information Can Be information that will help you with your analysis.
Easily Collected?
4. Did You Include the 5
Personal information can be easily collected Key Questions (Introduction,
but BMI, height, weight, etc. might be difficult Identifiers, Consent, Open-Ended
to collect. It is easy to ask someone their Fields and Validations)?
weight, but the accuracy of this data is often
low. Measuring people’s weight with a scale is Always have the following questions in your
far more accurate, but it is also more difficult. questionnaire.
Choose parameters that are useful and can be • Introduction: the right introduction to the
collected effortlessly. survey can set the tone of the survey and
is often helpful in making the respondent
For example, say that you want to judge understand why the survey is crucial and
a teacher’s classroom skills. You might be how it will help her/him in return.
tempted to capture a lot of information about • Identifiers: name, age, father’s name and
the classroom — you can probably sit in the location (for example).
classroom and capture classroom activities • Consent: most surveys in India require
for an hour. Or you could simply do a 5-minute organizations to seek the beneficiary’s
observation to learn about what happens on a consent. It’s a good ethical practice.
typical day. You need to balance the effort in • Open-ended fields: ask for any information
collecting additional information and the value that might not be captured by the specific
of that information. question types.
• Validations: information like GPS and time
taken to validate whether the questionnaire
Tool Tip was filled correctly.
To arrive at the final data points, think of
the following things: how difficult will it By leveraging smartphone-based tools
be to collect that information, how would for data collection, you will be able to
respondents react to a particular question, automatically capture GPS location and the
and how quickly can you collect a particular average time taken for surveys. This will be
piece of information? helpful in creating a check to ensure your field
surveyors are collecting accurate data from
the ground.
3. What Information Is Actually For instance, if you are collecting data from
Useful for the Organization? households in a village, then possibly the
GPS coordinates of each survey response
It is tempting to collect all information that should be a minimum of 15 metres apart.
you can. But it is important to only collect Similarly, if the average time taken for a survey
information that is useful for the organization. is 20 minutes and one of your surveyors is
submitting responses in under 5 minutes, this
For example, say that you want to judge a could indicate issues with his data quality and
teacher’s classroom skills through a 5-minute validity.
observation. It would be easy to simultaneously
collect other information on the school or Example
students. However, don’t collect information
just because you can collect it! Only collect Say that you want to measure the
improvement in student learning outcomes at • Add an open-ended field for surveyor
a given school. Go through the four questions comments
above to create the most important questions • Add GPS and time stamp to validate the
in your survey. information from the field
Chapter 5:
Question 1: What kind of information do
you need?
• Reading level of the kids: use the ASER Choosing a question for your survey is not
battery and sample a few kids from a enough. It is essential to choose the correct
classroom. Information on all the kids is question type. A good question asked in the
not necessary. wrong way will not give you good data.
• Teacher effort aimed at reading: observe
classrooms for 5-10 minutes while the There are 8 main question types that you can
teachers are teaching reading skills. use for surveys. Below are descriptions of
• Background information: for students, only each question type and when to use each.
collect age, gender, class, and name; for
teachers, collect only experience, classes Text
and subjects taught.
• School information: collect information on This is the most open-ended type of question.
number of students, teachers, teacher- One can type in anything. It is ideal to use this
pupil ratio, as well as average fees. Any type of question to collect a person’s name or
other information that is irrelevant to the to collect qualitative information such as “Any
analysis should not be collected. other feedback”.
Question 4: Did you include the 5 key Example: What is your name?
questions?
Dichotomous
• Add introduction (Yes/No, True/False)
• Add relevant identifiers for students
(if not already covered in background Dichotomous questions seek a binary
information) response to a question.
• Add consent
Example: Do you love to read? exclusive but collectively exhaustive. (See
A. Yes Chapter 7 for more information.)
B. No
Tabular/Roster
Numerical
These are used to capture the same sets
This is used to capture numbers. Numerical of information about multiple entities. For
questions should only be used to collect example, personal background about a
specific numbers that cannot be confined into household can be captured using a table.
certain ranges.
Chapter 6:
Remember, the “length” of your questionnaire
is not determined by the number of questions,
but by the time taken to answer them.
The second format of asking the same
For example, rather than asking “Do you While asking sensitive questions, it also
think India is on a downward trajectory?”, you helps to use the right words. In the previous
should ask “What trajectory is India on? A. example, “copying” and “cheating” were not
Downward B. Upward C. Other”. used because they will make the question
negative. Using more neutral words like
4. Avoid Negative Questions “sharing” makes people more likely to answer
honestly.
One of the best ways to be objective is by
avoiding negative questions. Most positive 6. Don’t Ask for Too Much Detail
questions are more direct than negative
questions. It is important to have the right amount of
detail. Don’t dig deeper than needed, and do
For example, rather than asking “What are take only superficial information.
the reasons India is not growing?”, ask “What
are the factors affecting India’s growth?” This For example, if you want to know about the
reduces bias and makes your question more sources of energy at home, don’t ask about
objective. all the appliances used. Only ask whether
specific energy sources (electricity, cowdung,
5. Be Careful while Asking gas, etc.) are being used or not. Make sure to
Sensitive Questions cover all the major sources of energy.
Chapter 7:
Don’t ask any critical or sensitive information
directly. People are often unwilling to share
sensitive information with a third party.
Chapter 8: • Yes
• No
• I feel the same
Open-Ended • Yes
• No
• Strongly dislike
• Dislike
• Neutral
Every question on a survey will be either an • Like
closed or open-ended question. This means • Strongly like
that closed and open-ended questions are
at the core of your survey design. It is crucial In contrast, open-ended questions do not
to know the difference between closed and have a defined set of responses. This means
open-ended questions and when to use each. that the set of possible responses is infinite,
so respondents can provide any answer they
The Difference Between Closed like. Open-ended questions are generally in
and Open-Ended Questions the form of narrative or text questions.
Closed-ended questions have a defined, Here are the previous examples, reworded as
closed set of responses. This means that open-ended questions:
respondents only have a limited number
of options for their answer to the question. How old are you?
Closed-ended questions come in a multitude
of forms, but they are often in the form of How do you feel today
multiple choice (single or multi-select), scale compared to yesterday?
or dichotomous questions.
Is it possible that you might • What could be improved on the website?
be pregnant?
Using the answers to these questions would
What do you think about this help to write the final survey. For example, if
health program? many people said during preliminary research
that they don’t like the colors on the website,
These are open-ended questions because you could include a section on your final
the answers to these questions are not pre- survey where respondents rank different color
determined, like they were previously. For palettes.
example, in the first question, the respondent
is not limited to 6 age brackets; they can In addition, you can use preliminary research
answer with any number, or even ages like “8 to improve the closed-ended questions in your
years and 3 months”. final survey. Most surveys use closed-ended
questions, but writing closed-ended questions
When to Use Open-Ended requires knowing the possible set of answers
Questions to your questions. Often, you don’t know
this before you start your survey. Preliminary
In general, open-ended questions are useful research with open-ended questions is helpful
for qualitative research, learning more to learning the set of answers for future
information about a topic, and surveys with closed-ended questions.
small sample sizes. There are four main cases
when open-ended questions should be used. For example, imagine that you want to learn
more about why people are not attending your
1. Preliminary Research meetings. It would be easy to analyse the
results of a closed-ended question like:
It is often helpful to conduct preliminary
research to learn more about your problem Why did you miss the last meeting?
before conducting your final survey. Open-
ended questions are a key component of • It was too early for me to attend
preliminary research, since you generally • It was too late for me to attend
won’t know the answers that you’ll receive. • It was too far from my house
Instead, you are looking to gain information • Other
that you likely don’t know.
However, preliminary research would be a
For example, imagine that you want to great way to learn the full set of possible
improve your website. Before you can write answers. For example, you could use an
an effective survey, it would be useful to get open-ended question (e.g. “Why did you miss
people’s general thoughts on your website. the last meeting?”) to get more information
This will help you write a more targeted final on why people miss meetings. Then, once
survey. you understand the most common reasons,
you can write a much better closed-ended
Your preliminary research could use open- question in your final survey.
ended questions like:
2. Expert Interviews
• What do you think about this website?
• What are your favorite parts of the website? Experts usually know more about a subject
• What are your least favorite parts of the than you will, so it is useful to use open-ended
website? questions to get as much information as
possible. Limiting experts to a pre-determined 1,000 people about their internet usage. If you
set of responses with closed-ended questions ask the open-ended question “Tell me about
will be less productive than giving them the your internet usage?”, you will end up with
freedom to demonstrate their knowledge and 1000 unique responses that cannot easily be
talk at length. analysed or reported. Instead, if you use a
closed-ended questions like the one below,
3. Surveys with a Small Sample Size you will be able to better understand and
report the results.
For a large number of respondents, it can
be difficult to read and analyse the answers On average, how many hours do you use the
to open-ended questions. Open-ended internet per week?
questions can often lead to responses of
several sentences or paragraphs. Comparing • 0-5 hours
these answers across dozens or hundreds of • 6-10 hours
respondents is extremely difficult and time- • 11-15 hours
consuming. • 16-20 hours
• Greater than 20 hours
However, this becomes much easier if you’re
conducting a survey with a small sample size With a closed-ended question, you can easily
(e.g. under 20 respondents). For small sample analyse the data and report a clear result like
sizes, open-ended questions are a great way “63% of respondents use the internet less
to solicit more detailed information in a way than 5 hours per week”.
that is still analyzable.
Chapter 9:
you’d like to tell me?” or “Is there anything
that I’ve missed?” — will allow respondents to
share extra information, opinions, or concerns.
Giving respondents the freedom to include
additional information or comments is also a
good way to show respect for the time and
effort they took in completing your survey.
Sampling Your
When to use Closed-Ended
Questions
Population
Closed-ended questions should be used for
easier analysis and reporting of the data you
are collecting. Ideally, a survey should gather data on every
single person in the target population. For
For example, imagine that you are polling example, a survey about learning outcomes
at a small school could track the test scores must be as diverse as the population itself and
of every student. Collecting data on everyone sensitive to local differences.
in the target population is the best case
scenario, since it ensures that everybody 3. Transparency
who matters to the survey is represented
accurately. There are several constraints that dictate
the size and structure of the population. It
However, this is only possible if the population is imperative that researchers discuss these
is small enough and the researchers have limitations and maintain transparency about
sufficient resources to reach out to everyone. the procedures followed while selecting the
This often is not the case, so researchers sample, so that the results of the survey are
have to identify a subset of the population to seen with the right perspective.
survey.
Choosing Your
How you choose this subset of the target Sampling Technique
population is crucial to the quality of your
data. The group must be carefully identified Probability Sampling
and representative of the larger population,
else your data will not be useful for drawing For probability sampling techniques, each
inferences. person in the population has a defined,
non-zero probability of being included in
If done right, survey sampling can save the sample. Probability sampling provides
time and money while allowing you to draw the most valid or credible results because it
interferences about a large group of people. reflects the characteristics of the population
from which they are selected. There are
3 Things to Keep in Mind While three probability sampling methods: random
Choosing a Sample Population sampling, systematic sampling and stratified
sampling.
1. Consistency
Random Sampling
It is important that researchers understand
the population on a case-by-case basis and When: There is a very large population and
test the sample for consistency before going it is difficult to identify every member of the
ahead with the survey. This is especially population.
critical for surveys that track changes across
time and space. If your sample is consistent, How: The entire process of sampling is
you can be confident that any change in done in a single step, where each subject is
the data reflects real change across the selected independently of the other people in
population, rather than change across atypical the sample.
individuals in the population.
Pros: In this technique, each member of the
2. Diversity population has an equal chance of being
selected for the sample.
Ensuring diversity of the sample is a tall order,
as reaching some portions of the population Cons: When there is a very large population,
and convincing them to participate in the it is often difficult to identify every member
survey can be difficult. However, for a sample of the population so the pool of subjects can
to truly represent the population, the sample become biased. For example, dialing numbers
from a phone book may not be entirely Sunday customers. They can choose every
random since the numbers would correspond 10th customer entering the supermarket and
to a localized region. conduct the study on this sample.
Minimizing Sampling Error For example, imagine that you want to learn
how attendees feel about your program, and
There is one easy way to minimize sampling you use the attendance sheets from two
error – increase the sample population size. weeks ago to create a target population.
The more respondents you have, the more However, unknown to you, a new group of
accurate your survey will be. However, it people started attending your program one
isn’t always possible to increase the sample week ago. The results of your survey will be
population because of financial restrictions. misleading, since they do not include one of
the key segments of your target population.
Avoiding three common errors will help to
minimize sampling error without increasing An easy way to avoid sample frame error is
your sample size. to take plenty of time to study your target
population. Be sure that nothing has changed
1. Avoid Population about it recently, and be sure that you have
Specification Errors accounted for all types of people in your
sample.
A population specification error occurs when
a critical segment of the population is not 3. Avoid Non-Response Error
included in the sample. This is the result of a
knowledge problem or gap. The results of a It is normal for some targeted respondents
survey with a population specification error to not respond to a survey. However, this can
will shed some light into your issue, but they become a problem if the non-respondents
cannot provide the full picture. generally hold a view that is different from the
respondents. As a result, the final data will
For example, imagine that you want to learn be skewed toward the opinions of those who
more about household decision making, so responded.
you survey men about the decisions in their
household. This would be correct if only For example, imagine that you want to survey
men make household decisions, but often housewives about their free time, and you
women and children also have influence over do this by calling them on the phone during
decisions. By only surveying men, you will the day. The women who do not respond are
miss out on part of the picture. the ones who have less free time (since they
don’t have enough time to pick up the phone).
An easy way to avoid population specification Meanwhile, the women who respond are the
errors is to learn how similar surveys sampled ones who have more free time. The results
of the data will report higher free time among 95%, then you find that 68% of women take
housewives than is actually true. iron pills. You can be 95% confident that the
true number of women who take iron pills is
An easy way to avoid non-response error is to between 66% and 70%.
conduct follow-up surveys or contact those
who did respond through alternative means. The confidence level corresponds to a given
In addition, keeping the questionnaire short z-score:
will help encourage people to respond the first
time.
Confidence level Z-score
Calculating Sample Size 1 90% 1.645
2 95% 1.96
Determining the size of your sample
population is one of the most difficult 3 99% 2.576
decisions to make in your survey. A larger
sample can yield more accurate results — but
the more responses you collect, the more Sample Size Formula for a
expensive it gets. Population of Unknown Size
1. Margin of error The sample size formula when you don’t know
(also known as confidence interval): the size of your large population is
For example, imagine that you set a margin You can calculate a more accurate sample
of error of 2% and a confidence level of
size if you know the size of the population that m is the margin of error, and z is the z-score.
you are surveying. (Note that if you have a margin of error of 5%,
m = 0.05, z=1.96)
For example, if you are studying the learning
outcomes for a school with 200 students, It is usually safe to use a margin of error of 5%
your population size is 200. If you are studying and a confidence level of 95%. The sample
women in Gujarat, the population size is the size formula with those figures (margin of error
total number of women in Gujarat. of 5% and confidence level of 95%) is
Tool Tip
Don’t know which sample size formula to use?
The last formula (for a sample of known size
where n is the sample size you should use, with margin of error of 5% and confidence
p is the size of the population being surveyed, interval of 95%) is your safest bet.
References
1. Introduction - Case Study
Details on polling during the 1936 U.S. Presidential Election from
Dennis DeTurck, “Case Study 1: The 1936 Literary Digest Poll”, University of Pennsylvania
(https://www.math.upenn.edu/~deturck/m170/wk4/lecture/case1.html)
4. Select the mode of your survey Words mean different things to different
people and taking care of some of the finer
The mode that you employ to administer the nuances involved in formulating an effective
survey depends on the sample type and size. question can go a long way. Don’t leave any
Use the mediums that are the most effective scope for ambiguity. Be clear about what
in reaching your target population. The time you need and get your questions proofread
span of the research is also an important by somebody who is not familiar with your
factor that impacts this decision. Leveraging study before sending out the survey. Brevity
technology to maximize the extent and depth is essential. Respondents are more likely to
of your reach might come in very handy in respond positively to questions that are con-
such situations. SocialCops has had great cise and able to hold their attention. Avoid
success in using mobile application and low- unnecessary jargon; the language should be
cost smartphones to collect data at the grass- as simple and generic as possible. And finally,
roots level. the answer choices must be well defined.
Example
If you ask respondents to rank their level of
satisfaction on a scale from 1 to 5 but fail to
explain whether 1 is very satisfied or 5 is very
satisfied, their responses will be of little value.
4. Pilot the survey in the correct Don’t just focus on the questions themselves;
medium be sure to also look at the data that is being
collected. Sometimes, issues in the collected
Ensure that you pilot the survey in the data can show problems in your survey. Use
medium it will roll out in. Do not do any of the your data to make a data-driven decision on
following: what’s working and what’s not working.
Pilot a paper survey if you will actually collect
data on a mobile app. 8. Pilot all aspects of the survey
Pilot an English survey if you will actually
collect data with a Hindi survey. Pilot the question flow, the order of the
Pilot a small survey of 10 questions if you questions, question types, and even the help
will collect data with a long survey of 100 text to ensure that the survey is clear to both
questions. your surveyors and respondents. To pilot
question flow and order, you should observe
5. Integrate the pilot with your how comfortable surveyors and respondents
training are as they go through the survey. Check
whether the surveyors or respondents get
Usually, organizations conduct surveyor confused or give the same information twice.
training before data collection starts. Use the If this happens, it might be helpful to change
pilot as part of this training. The pilot will give the order of the questions or add help text.
the surveyors a chance to test out the survey
in the field, ask questions about the survey, 9. Collect feedback from everyone
and figure out if they have any issues.
Since piloting is an iterative process, it
6. Question the survey questions is important to ensure that you include
viewpoints from all stakeholders. Get
The most important part of the format is to feedback from supervisors, surveyors,
understand what questions are you asking observers, respondents, and any other
and what information you are getting. people in the pilot, and make sure everyone
One of the problems to look for during your contributes during feedback sessions. In
pilot is compound questions — a question addition, you can observe different people to
that asks two or more questions at the same compile your own feedback.
time. For example, “How many children do Consider and balance all these viewpoints
you have and which one is the youngest?” is carefully. It is important to make sure that you
a compound question. It actually is asking two don’t get biased by any single opinion.
separate questions: “How many children do
you have?” and “Name the child which is the 10. Never re-pilot the same version of
youngest.” Be sure that each question on the a survey
survey only asks for one piece of information.
In addition, the pilot will show which phrases At the end of each day, incorporate all the
feedback into your survey. Then you can Night session: Assemble the day’s feedback,
use this updated survey for the next day of analyze what feedback to include, and update
piloting. By immediately updating your survey, the survey accordingly.
you won’t waste time on the same feedback.
That’s how you pilot like a boss!
11. Be thoughtful about what Happy data collection.
changes to make
SocialCops recently worked with Tata Trusts We used Census village lists to decide which
as the data intelligence partner to conduct two villages that we should survey. However, we
census surveys in Chandrapur (Maharashtra) were sometimes left searching for villages that
and Noamundi (Jharkhand). Chandrapur and didn’t exist.
Noamundi are some of the most backward Though it is standard practice to trust Census
areas in their states. The areas have difficult records, try asking local coordinators for a list
terrain with little or no access to phone of villages.
connectivity and electricity.
The large-scale census surveys collected data 4. Piloting — a never-ending story
for over 60,000 households within 2 months
using tablets. The data was then used to Each geography is unique with its own set
help in micro-planning activities in the area. of demographic, geographic, social, and
SocialCops’ challenge was to train local economic problems. It’s important to tweak
people to collect this data while overcoming each survey for its specific geography. Even
operational challenges that arose throughout after consulting experts and doing two rounds
the survey. of piloting, surveyors were still calling our
team for changes in the survey.
These are some of our major With Collect, we can edit surveys in the field
learnings from these two and sync those changes on all other devices.
surveys. This helped us tweak our surveys on the go.
When surveying in tribal areas of Maharashtra Any large-scale census survey usually involves
and Jharkhand, we often couldn’t find cellular a train-the-trainers model. This means that
connection, let alone internet, for hours. we train people who then train the surveyors,
Sometimes we had to travel up to 8 hours to rather than us training the surveyors directly.
reach areas with cell or internet connectivity. This indirect method can lead to knowledge
We used our Collect app to collect data leaks as information is transferred from person
offline throughout Chandrapur and Noamundi. to person.
Collected data was synced to our servers at Closely monitor surveyors’ training, since that
the end of the week. training can make or break a survey. A good
training can reduce survey bias, while a bad
2. Gender neutrality — the magic training leads to incorrect data.
wand that eases social norms
6. Google Play Services — the jugaad
It’s interesting how we had to anticipate social method
dynamics since they could intervene with our
work. In one village, our field manager caused It can be difficult to get enough bandwidth to
download a phone app while working in rural • 10 am – 6 pm: Monitor surveyors and
India. However, all the tablets used in our collected data
census surveys needed Google Play Services • 6 – 9 pm: Collect all tablets from the field
(used for updating the phone’s location, so that the daily surveying schedule is not
scanning barcodes, and more), which required disturbed
46 MBs to download. How could we do this • 9 – 10 pm: Travel to the district headquarters
for 350 tablets? • 10 pm – 1 am: Install updated APKs on 350
Instead of trying to directly download the file tablets
to the tablets, we used our own jugaad. We • 1 – 7 am: Rotate charging tablets and sleep
first downloaded Google Play Services to
our computer, then we transferred the file to Make sure that your field team is dedicated
all 350 tablets over USB. If we later found a and hard working. They might have to spend
tablet without Google Play Services, we used sleepless nights to run the survey smoothly.
ShareIt to share Google Play Services over
Bluetooth.
Even basic functions like downloading apps
might not work in certain areas. Be ready to
do some extra work.
Gaurav Jha
Content Creator, Data for Impact
Richa Verma
Resident Entrepreneur
Sahaj Talwar
Graphic Designer
Collect
Create Mobile Surveys and Collect
Data From Any Android Device
Collect is a smartphone-based data collection tool to capture
data from the field, monitor progress of projects, and make
quick decisions based on real-time, accurate data.
TRY
COLLECT
About SocialCops
Our world’s most important decisions are Our work takes the big data revolution to
crippled by an astonishing lack of data. No where it matters the most – to use in decisions
map in the world can tell women the safest that affect human health, well being, safety,
route home or tell the police the best route livelihoods, and more.
to patrol. No survey tracks parameters like
teacher attendance or school infrastructure in We partner with leaders and organizations
real time. National-level healthcare decisions around the world — from governments and
affecting millions are made based on a sample philanthropies to corporates and nonprofits
survey of 100 people. — to make any data-driven decision easier,
faster, and more effective. Our work helps
SocialCops is a data intelligence company build a world with smarter cities, happier
focused on powering the world’s most citizens, better policies, and happier citizens.
important decisions through data.
Partners
We are currently working with over 150 organizations across 6 countries to drive large-scale de-
cisions and confront the world’s most critical problems. Our partners include the Government of
India, World Bank, United Nations, Tata Trusts, Gates Foundation, Unilever, and BASF.
Recognition
The work we have done at the intersection of data, technology and people has won us accolades
globally – including Fortune India 40 Under 40 2015 and 2016, Forbes India 30 Under 30 2015,
Forbes Asia 30 Under 30 2015, the United Nations World Youth Summit Award, Global Social
Entrepreneurship Competition, IBM/IEEE Smart Planet Challenge and grants from Microsoft, IBM
and Salesforce.
Thank You For Reading
CLICK HERE