RESEARCH METHODOLOGY

“Methodology” implies more than simply the methods you intend to use to collect data. It is often
necessary to include a consideration of the concepts and theories which underlie the methods

To understand the use of statistics, one needs to know a little bit about experimental design or how a
researcher conducts investigations. A little knowledge about methodology will provide us with a place
to hang our statistics. In other words, statistics are not numbers that just appear out of nowhere.
Rather, the numbers (data) are generated out of research. Statistics are merely a tool to help us answer
research questions. As such, an understanding of methodology will facilitate our understanding of
basic statistics.

Validity

A key concept relevant to a discussion of research methodology is that of validity. When an individual
asks, "Is this study valid?", they are questioning the validity of at least one aspect of the study. There
are four types of validity that can be discussed in relation to research and statistics. Thus, when
discussing the validity of a study, one must be specific as to which type of validity is under
discussion. Therefore, the answer to the question asked above might be that the study is valid in
relation to one type of validity but invalid in relation to another type of validity.

Each of the four types of validity will be briefly defined and described below. Be aware that this
represents a cursory discussion of the concept of validity. Each type of validity has many threats
which can pose a problem in a research study. Examples, but not an exhaustive discussion, of threats
to each validity will be provided. For a comprehensive discussion of the four types of validity, the
threats associated with each type of validity, and additional validity issues see Cook and Campbell
(1979).

Statistical Conclusion Validity: Unfortunately, without a background in basic statistics, this type of

validity is difficult to understand. According to Cook and Campbell (1979), "statistical conclusion
validity refers to inferences about whether it is reasonable to presume co-variation given a specified
alpha level and the obtained variances (p. 41)." Essentially, the question that is being asked is - "Are
the variables under study related?" or "Is variable A correlated (does it co-vary) with Variable B?". If
a study has good statistical conclusion validity, we should be relatively certain that the answer to these
questions is "yes". Examples of issues or problems that would threaten statistical conclusion validity
would be random heterogeneity of the research subjects (the subjects represent a diverse group - this
increases statistical error) and small sample size (more difficult to find meaningful relationships with
a small number of subjects).

Internal Validity: Once it has been determined that the two variables (A & B) are related, the next
issue to be determined is one of causality. Does A cause B? If a study is lacking internal validity, one
cannot make cause and effect statements based on the research; the study would be descriptive but not
causal. There are many potential threats to internal validity. For example, if a study has a pre-test, an
experimental treatment, and a follow-up post-test, history is a threat to internal validity. If a difference
is found between the pre-test and post-test, it might be due to the experimental treatment but it might
also be due to any other event that subjects experienced between the two times of testing (for
example, a historical event, a change in weather, etc.).

Construct Validity: One is examining the issue of construct validity when one is asking the
questions "Am I really measuring the construct that I want to study?" or "Is my study confounded

1
(Am I confusing constructs)?” For example, if I want to know a particular drug (Variable A) will be
effective for treating depression (Variable B) , I will need at least one measure of depression. If that
measure does not truly reflect depression levels but rather anxiety levels (Confounding Variable X),
than my study will be lacking construct validity. Thus, good construct validity means that we will be
relatively sure that Construct A is related to Construct B and that this is possibly a causal relationship.
Examples of other threats to construct validity include subject’s apprehension about being evaluated,
hypothesis guessing on the part of subjects, and bias introduced in a study by expectancies on the part
of the experimenter.

External Validity: External validity addresses the issue of being able to generalize the results of your
study to other times, places, and persons. For example, if you conduct a study looking at heart disease
in men, can these results be generalized to women? Therefore, one needs to ask the following
questions to determine if a threat to the external validity exists: "Would I find these same results with
a difference sample?", "Would I get these same results if I conducted my study in a different setting?",
and "Would I get these same results if I had conducted this study in the past or if I redo this study in
the future?" If I cannot answer "yes" to each of these questions, then the external validity of my study
is threatened.

Populations and Samples

When conducting research, one must often use a sample of the population as opposed to using the
entire population. Before we go further into the reasons why, let us first discuss what differentiates
between a population and a sample.

A population can be defined as any set of persons/subjects having a common observable

characteristic. For example, all individuals who reside in the United States make up a population.
Also, all pregnant women make up a population. The characteristics of a population are called a
parameter. A statistic can be defined as any subset of the population. The characteristics of a sample
are called a statistic.

Why Sample?

This brings us to the question of why sample. Why should we not use the population as the focus of
study. There are at least four major reasons to sample.

First, it is usually too costly to test the entire population. The United States government spends
millions of dollars to conduct the U.S. Census every ten years. While the U.S. government may have
that kind of money, most researchers do not.

The second reason to sample is that it may be impossible to test the entire population. For example, let
us say that we wanted to test the 5-HIAA (a serotonergic metabolite) levels in the cerebrospinal fluid
(CSF) of depressed individuals. There are far too many individuals who do not make it into the mental
health system to even be identified as depressed, let alone to test their CSF.

The third reason to sample is that testing the entire population often produces error. Thus, sampling
may be more accurate. Perhaps an example will help clarify this point. Say researchers wanted to
examine the effectiveness of a new drug on Alzheimer's disease. One dependent variable that could be
used is an Activities of Daily Living Checklist. In other words, it is a measure of functioning o a day
to day basis. In this experiment, it would make sense to have as few of people rating the patients as
possible. If one individual rates the entire sample, there will be some measure of consistency from one
patient to the next. If many raters are used, this introduces a source of error. These raters may all use a

2
slightly different criterion for judging Activities of Daily Living. Thus, as in this example, it would be
problematic to study an entire population.

The final reason to sample is that testing may be destructive. It makes no sense to lesion the lateral
hypothalamus of all rats to determine if it has an effect on food intake. We can get that information
from operating on a small sample of rats. Also, you probably would not want to buy a car that had the
door slammed five hundred thousand time or had been crash tested. Rather, you probably would want
to purchase the car that did not make it into either of those samples.

Types of Sampling Procedures

As stated above, a sample consists of a subset of the population. Any member of the defined
population can be included in a sample. A theoretical list (an actual list may not exist) of individuals
or elements who make up a population is called a sampling frame. There are five major sampling
procedures.

The first sampling procedure is convenience. Volunteers, members of a class, individuals in the
hospital with the specific diagnosis being studied are examples of often used convenience samples.
This is by far the most often used sample procedure. It is also by far the most biases sampling
procedure as it is not random (not everyone in the population has an equal chance of being selected to
participate in the study). Thus, individuals who volunteer to participate in an exercise study may be
different that individuals who do not volunteer.

Another form of sampling is the simple random sample. In this method, all subject or elements have
an equal probability of being selected. There are two major ways of conducting a random sample. The
first is to consult a random number table, and the second is to have the computer select a random
sample.

A systematic sample is conducted by randomly selecting a first case on a list of the population and
then preceding every nth case until your sample is selected. This is particularly useful if your list of
the population is long. For example, if your list was the phone book, it would be easiest to start at
perhaps the 17th person, and then select every 50th person from that point on.

Stratified sampling makes up the fourth sampling strategy. In a stratified sample, we sample either
proportionately or equally to represent various strata or subpopulations. For example if our strata were
states we would make sure and sample from each of the fifty states. If our strata were religious
affiliation, stratified sampling would ensure sampling from every religious block or grouping. If our
strata were gender, we would sample both men and women.

Cluster sampling makes up the final sampling procedure. In cluster sampling we take a random
sample of strata and then survey every member of the group. For example, if our strata were
individuals’ schools in the St. Louis Public School System, we would randomly select perhaps 20
schools and then test all of the students within those schools.

Sampling Problems

There are several potential sampling problems. When designing a study, a sampling procedure is also
developed including the potential sampling frame. Several problems may exist within the sampling
frame. First, there may be missing elements - individuals who should be on your list but for some

3
reason are not on the list. For example, if my population consists of all individuals living in a
particular city and I use the phone directory as my sampling frame or list, I will miss individuals with
unlisted numbers or who cannot afford a phone.

Foreign elements make up my second sampling problem. Elements which should not be included in
my population and sample appear on my sampling list. Thus, if I were to use property records to
create my list of individuals living within a particular city, landlords who live elsewhere would be
foreign elements. In this case, renters would be missing elements.

Duplicates represent the third sampling problem. These are elements who appear more than once on
the sampling frame. For example, if I am a researcher studying patient satisfaction with emergency
room care, I may potentially include the same patient more than once in my study. If the patients are
completing a patient satisfaction questionnaire, I need to make sure that patients are aware that if they
have completed the questionnaire previously, they should not complete it again. If they complete it
more than once, their second set of data represents a duplicate

Choosing qualitative or quantitative research methodologies

Your research will dictate the kinds of research methodologies you use to underpin your work and
methods you use in order to collect data. If you wish to collect quantitative data you are probably
measuring variables and verifying existing theories or hypotheses or questioning them. Data is often
used to generate new hypotheses based on the results of data collected about different variables. One’s
colleagues are often much happier about the ability to verify quantitative data as many people feel
safe only with numbers and statistics.

However, often collections of statistics and number crunching are not the answer to understanding
meanings, beliefs and experience, which are better understood through qualitative data. And
quantitative data, it must be remembered, are also collected in accordance with certain research
vehicles and underlying research questions. Even the production of numbers is guided by the kinds of
questions asked of the subjects, so is essentially subjective, although it appears less so than qualitative
research data.

Qualitative research

This is carried out when we wish to understand meanings, look at, describe and understand
experience, ideas, beliefs and values, intangibles such as these. Example: an area of study that would
benefit from qualitative research would be that of students’ learning styles and approaches to study,
which are described and understood subjectively by students.

Using quantitative and qualitative research methods together

This is a common approach and helps you to 'triangulate' ie to back up one set of findings from one
method of data collection underpinned by one methodology, with another very different method
underpinned by another methodology - for example, you might give out a questionnaire (normally
quantitative) to gather statistical data about responses, and then back this up and research in more
depth by interviewing (normally qualitative) selected members of your questionnaire sample.

Research methods in brief:

Look at the very brief outlines of different methods below. Consider which you intend using and
whether you could also find it more useful to combine the quantitative with the qualitative.

4
 a) Qualitative research methods
Interviews

Interviews enable face to face discussion with human subjects. If you are going to use interviews you
will have to decide whether you will take notes (distracting), tape the interview (accurate but time
consuming) rely on your memory (foolish) or write in their answers (can lead to closed questioning
for time’s sake). If you decide to interview you will need to draw up an interview schedule of
questions which can be either closed or open questions, or a mixture of these. Closed questions tend
to be used for asking for and receiving answers about fixed facts such as name, numbers, and so on.
They do not require speculation and they tend to produce short answers. With closed questions you
could even give your interviewees a small selection of possible answers from which to choose. If you
do this you will be able to manage the data and quantify the responses quite easily. The Household
Survey and Census ask closed questions, and often market researchers who stop you in the street do
too. You might ask them to indicate how true for them a certain statement was felt to be, and this too
can provide both a closed response, and one which can be quantified (30% of those asked said they
never ate rice, while 45% said they did so regularly at least once a week... and so on).

The problem with closed questions is that they limit the response the interviewee can give and do not
enable them to think deeply or test their real feelings or values.

If you ask open questions such as ‘what do you think about the increase in traffic?’ you could elicit an
almost endless number of responses. This would give you a very good idea of the variety of ideas and
feelings people have, it would enable them to think and talk for longer and so show their feelings and
views more fully. But it is very difficult to quantify these results. You will find that you will need to
read all the comments through and to categorise them after you have received them, or merely report
them in their diversity and make general statements, or pick out particular comments if they seem to
fit your purpose. If you decide to use interviews:

 Identify your sample.

 Draw up a set of questions that seem appropriate to what you need to find out.

 Do start with some basic closed questions (name etc.).

 Don't ask leading questions.

 Try them out with a colleague.

 Pilot them, and then refine the questions so that they are genuinely engaged with your
research object.

 Contact your interviewees and ask permission, explain the interview and its use.

 Carry out interviews and keep notes/tape.

 Transcribe.

 Thematically analyse results and relate these findings to others from your other research
methods.

b) Quantitative research methods:

Questionnaires

5
Questionnaires often seem a logical and easy option as a way of collecting information from people.
They are actually rather difficult to design and because of the frequency of their use in all contexts in
the modern world, the response rate is nearly always going to be a problem (low) unless you have
ways of making people complete them and hand them in on the spot (and this of course limits your
sample, how long the questionnaire can be and the kinds of questions asked). As with interviews, you
can decide to use closed or open questions, and can also offer respondents multiple choice questions
from which to choose the statement which most nearly describes their response to a statement or item.
Their layout is an art form in itself because in poorly laid out questionnaires respondents tend, for
example, to repeat their ticking of boxes in the same pattern. If given a choice of response on a scale
1-5, they will usually opt for the middle point, and often tend to miss out subsections to questions.
You need to take expert advice in setting up a questionnaire, ensure that all the information about the
respondents which you need is included and filled in, and ensure that you actually get them returned.
Expecting people to pay to return postal questionnaires is sheer folly, and drawing up a really lengthy
questionnaire will also inhibit response rates. You will need to ensure that questions are clear, and that
you have reliable ways of collecting and managing the data. Setting up a questionnaire that can be
read by an optical mark reader is an excellent idea if you wish to collect large numbers of responses
and analyse them statistically rather than reading each questionnaire and entering data manually.

You would find it useful to consult the range of full and excellent research books available. These will
deal in much greater depth with the reasons for, processes of holding, and processes of analysing data
from the variety of research methods available to you.

 Developing and using a questionnaire - some tips

 Identify your research questions

 Identify your sample

 Draw up a list of appropriate questions and try them out with a colleague

 Pilot them

 Ensure questions are well laid out and it is clear how to 'score them' (tick, circle, delete)

 Ensure questions are not leading and confusing

 Code up the questionnaire so you can analyse it afterwards

 Gain permission to use questionnaires from your sample

 Ensure they put their names or numbers on so you can identify them but keep real names
confidential

 Hand them out/post them with reply paid envelopes

 Ensure you collect in as many as possible

 Follow up if you get a small return

 Analyse statistically if possible and / or thematically