Doctoral Comprehensive Exam Research Question

You are employed as a school psychologist in a district that has a broad range of
family income levels. The district just started a Vocational Transition Program for
juniors in High School. The program will be scheduled for one period once a week for
the entire school year. You have been asked to evaluate the effectiveness of this
program.

Describe how you would evaluate the effectiveness of this curriculum. Provide
your answer using sentences and paragraphs using the following headings:

Introduction including the purpose,

Research Questions to be answered,
Research/Evaluation Design to be used and why this design was chosen,
Sample and Population to be used,
Variables and Measures to be used and why these measures were chosen,
Procedures you will use,
Data Analysis proposed and why this method of data analysis was chosen,
Ethical Considerations, e.g. the need for Parental/Guardian permission,
Treatment appropriateness, etc.,
Strengths of your research/ program evaluation plan, and
Weaknesses of your research/ program evaluation plan.

Comment from the EDSP faculty

The doctoral program in School Psychology at IUP prepares students to work in a

number of settings ranging from serving as practitioners in schools to teaching at
the university level. Since the majority of our students choose to continue as
practitioners, our emphasis in the area of research is largely on applied research.
Even if our graduates continue to work in the schools, however, we do expect that
they will have sufficient competence to assist in school-based research and
evaluation even in areas outside the normal domains of a school psychologist.
Additionally this is a skill emphasized in the 2010 NASP Standards, reflecting the
importance of this competency regardless of employment location (e.g., academia,
public or private practice).

It should not matter whether you have a significant level of expertise in what is
being evaluated as long as you have a good understanding of how research design,
sampling, measurement, and statistics are connected in an applied setting and you
have sufficient understanding of how you can work with others to develop practical
solutions to evaluation problems and applied research questions.

In the sample response to the question posed, we provide you with some notes that
should assist you in preparing a response to any other similar question. In
addition, a reading list is provided that will help you prepare for this portion of
your comprehensive exam.

1
 Sample Response Notes
Introduction
As a school psychologist, I have been The introduction should primarily
asked to assist the district in set the stage for what will
evaluating a newly implemented follow. It should be reasonably
Vocational Transition Program for brief, but should make it clear
juniors in the high school. In order that you understand the purpose
for me to provide adequate assistance of the task.
in this project, it will be important
for me to work with a team of
district personnel to identify the
criteria for success as well as ways
to collect data regarding the level
of program implementation (fidelity)
and the level to which the various
criteria for success have been met.
Any approach to this evaluation will
need to be realistic and manageable
and also must meet the needs of the
district.

2
 Sample Response
Research Questions
Basically, the primary impetus for my
involvement is to assist in the
evaluation of the newly implemented
Vocational Transition Program (VTP)
for high school juniors. However,
many more specific questions could be
asked and my job would be to help the
district decide which questions are
of most importance and then to help
in designing an evaluation model that
addresses these questions.
One strategy I could use in the
beginning would be to create an
evaluation committee that would help
oversee the process. An early task
would be to help the district
identify exactly what constitutes
“effective.” That is, what are the
criteria the district wants to use in
determining whether the program is
effective? Identifying these
criteria is important because it will
be directly related to the measures I
will have to collect and,
potentially, the design of the study
as well as the statistical procedures
to be employed.
Notes
In any program evaluation there
are several ways to approach the
basic research question or
questions to be addressed. There
is one overarching point,
however, and that is the basic
question of whether the program
was effective. This is
complicated by the different ways
in which effectiveness can be
measured and you, for this
question, need to 1) make it
clear that you understand these
differences and 2) pick two or
three upon which you will build
the rest of your response to this
comprehensive exam question.

For example, I would need to know

what “successful vocational
transition” means to the district.
Does it mean that current juniors who
take the program have a particular
set of skills? Does it mean that the
students who take the program will be
able to get employment during the
summer between their junior and
senior years? Are the goals more
long-range and related to finding
work after graduation?

Another set of issues that would need

to be addressed have to do with the
audience for the program. Are there
certain sub-groups for whom this
program is particularly designed? Is
it intended for special populations
or all juniors? Is it expected to be
particularly effective for boys more
than girls?

3
In addition to these basic questions,
I would need to know whether there
are other areas of particular
interest or concern. For example, is
the district interested in knowing
how satisfied the teachers are in
delivering the program? Is cost of
the curriculum and its implementation
to be monitored and evaluated? Are
there issues of fidelity that need to
be evaluated or is fidelity simply a
factor that needs to be considered in
designing the evaluation?

Finally, it would be important to

know who the district views to be the
stakeholders. Are we to assess the
satisfaction by students, teachers,
parents, community personnel, agency
personnel, or others?

Once I have had the opportunity to

hold these discussions and involve
the appropriate persons, I would be
able to formulate specific questions
around which the program evaluation
should be designed. For the sake of
this exam, I will identify the
following three questions as coming
from the above process:

1. Do juniors who participate in

the program acquire skills X,
Y, and Z?
2. Are teachers who teach the
curriculum satisfied with the
manner with which it is
delivered?
3. Are parents of the children who
participate in the program
satisfied?

Sample Response Notes

Research/Evaluation Design

As stated above, I have selected The first decision I’d have to

three specific research questions for make in answering this question
the purpose of this examination and is whether to approach the design
it is these three questions that will of this evaluation from a

4
guide the remainder of my response.
Understanding the three questions is
important to selecting an appropriate
design methodology. The first
question states, “Do juniors who
participate in the program acquire
skills X, Y, and Z?” This question
asks whether the program is effective
in helping the students achieve the
desired skill outcomes of X, Y, and
Z. To answer this question, it would
be important for the design of this
evaluation design to include some
pre-test data so that it can be
determined whether after the program
occurred, there were any changes in
these skills. However, it will also
be necessary to consider whether
maturation alone might have caused
these changes in skills, so it may be
necessary to consider a control group
to compare against.
quantitative or a qualitative
approach. A mixed-method
approach is also possible and
might be considered. Each of
these is legitimate and a case
can be made for each. My
preference in answering is use a
quantitative approach for the
skills of X, Y and Z; with a
pseudo qualitative, though still
quantitative, approach for the
questions dealing with
satisfaction by teachers and
parents.
Ultimately, the choice of the
method for the quantitative
aspect was made to help minimize
some of the threats to internal
validity.

The second question states, “Are

teachers who teach the curriculum
satisfied with the manner with which
it is delivered?” and the third
question states, “Are parents of the
children who participate in the
program satisfied?” Both of these
questions might best be answered
through a survey of teachers and
parents.

Therefore, I would suggest an

evaluation model that would be
considered a combination of causal-
comparative (ex post facto) for
collecting student data and
descriptive for collecting parent and
teacher data. The general model for
the student data might be:

O1 X O2

O1 C O2

This would include identifying two

groups of students, X (treatment –
those participating in the
curriculum) and C (control – those
NOT participating in the curriculum).
Prior to the start of the program,
students in both groups would be pre-
tested on the three dependent
variables of X, Y, and Z

5
(Observations at time 1 - O1) and
after the treatment, the groups would
again be tested on the three
dependent variables of X, Y, and Z
(Observations at time 2 – O2).

In addition, I would develop a survey

form for collecting satisfaction data
from the teachers who participated in
the delivery of the program and a
second survey form for collecting
satisfaction data from the parents of
the students who participated in the
program.

If possible, I would like to use a

delivery method such as Qualtrics for
my teachers (as I am confident that
they all have access to computers in
the school) and I would probably send
a mailed survey to the parents,
though it may be possible to collect
the data in person at some point if
the parents are all at the school at
the same time.

Sample Response Notes

Sample and Population

Since this is a program evaluation, As with other aspects of the

the primary population of interest
can best be described as future
students and teachers in this
district who may take (students) or
deliver (teachers) this Vocational
Transition Program. In addition, one
of the research questions involves
the satisfaction of parents, so
parents, also, need to be
stakeholders. In other words, the
district has implemented a program
that it would like to continue to use
into the future. Therefore, the
population would include those
persons who may be involved in this
curriculum within the district. One
might argue that, from a research
perspective, a broader population
could include other students,
parents, and teachers (district
personnel) who are similar to those
in the district I am employed. If I
were to set this up as a research
answer to this question, there
are a number of approaches that
could be taken. One approach
might be to consider choose a
sample by selecting a random
number of students participating
in the curriculum, another might
be to select all students
participating in the curriculum –
the decision as to what the
population might be could include
future students, teachers, and
parents in the district or it
could be other students, parents
and teachers similar to those in
my district.

6
project, per se, I might be
interested in making generalizations
to a larger population beyond my own
district. However, because I am
primarily interested in the
effectiveness of this curriculum in
my own district, I will define the
population accordingly. One positive
outcome of defining my population in
this manner is that issues of
external validity are less of a
concern since, barring any major
shifts in the demographics of my
district, my sample should be quite
representative of this population.

The sample, then, would be the 11th

graders who are enrolled in the VTP,
the parents of these students, and
the teachers who are responsible for
delivering the program.

Sample Response Notes

Variables and Measures

For the purpose of this response, I
will define the three dependent
variables X, Y and Z as:
X: Holland Self-Directed Search
(HSDS) which identifies personality
traits that align to different career
choices.
Y: Career Occupational Preference
System (COPS) Interest Inventory
which is completed by the student and
identifies broad occupational /
vocational areas that align with
their reported interests.  specifically identify in your
Z: Career Aptitude Placement Survey
(CAPS) which is an assessment of
reading, writing, mathematics,
science, social studies, etc.
knowledge and skill to see if the
student has the pre-requisite skills
that would predict success (provided
additional training) in the
identified broad interest areas.
The satisfaction of the teachers in
developing the curriculum and the
There are basically five (5)
dependent variables in this
evaluation. These include the
skills (X, Y, and Z) as well as
the attitude of teachers and the
attitude of the parents.
If the skills (X, Y, and Z) are
skills/behaviors that are
measured by some standardized
instrument, then the response
might include how you’d go about
identifying those instruments.
Alternatively, if you
response what those skills are
and if you know how those skills
might be measured, your response
could include that level of
detail.
Similarly, if attitudinal
instruments for measuring
satisfaction by teachers and
parents already exist, you can
cite those. If they do not
exist, you could discuss in your
answer how you would go about

7
satisfaction of the parents would be
measured by surveys I would create.
I would identify a group of teachers
who are not directly involved in the
delivery of the program I am
evaluating in my own district – but
who are familiar with such programs,
in general, from another district or
building in my district. I would
then create sample items and
distribute them to these teachers to
seek their input into the content
validity of the items. I would also
distribute the test instruments to a
group of volunteers to determine
reliability using an internal
reliability method such as split half
or one of the Kuder-Richardson
formulas.
creating such instruments and how
you would determine the validity
and reliability of the
instruments.
Just a note about the instruments
– it may well be that these
particular instruments and the
construct they purport to assess
might not measure traits that
would be expected to change over
the course of time this
evaluation would be conducted.
They are listed here only as
examples of how a response could
be crafted.

Since this is a program evaluation,

and not a research project for
publication, I would most likely not
be as concerned about the reliability
of the instruments as I would be
about the utility of the data in
answering the original evaluation
questions.

Sample Response Notes

Procedures
For this response, I identified three The response to this section is
possible questions to be addressed in somewhat dependent upon the way
this evaluation. These are: in which earlier portions of the
1. Do juniors who participate in question was answered.
the program acquire skills X, Therefore, the procedures need to
Y, and Z? be reasonably specific to the
2. Are teachers who teach the three evaluation questions
curriculum satisfied with the identified above.
manner with which it is
delivered?
3. Are parents of the children who
participate in the program
satisfied?

Consequently, the design or

procedures to be employed need to
allow for me to answer these three
questions.

The first question asks whether the

program was successful in helping
students acquire three skills (X, Y,
and Z). To investigate this, I would

8
recommend that the students be given
a pre-test and a post-test for each
of the three skills.

The remaining two questions deal with

participant (teachers and parents)
satisfaction. I would work with the
personnel in the school to develop
survey instruments and collect data
from participants in that manner.
These surveys would be collected
either in paper/pencil, electronic,
or face-to-face (focus group) formats
– depending on what happens to work
best for the participants.

In terms of designs, the first

question would use what is typically
called an ex post facto non-
experimental design. The response to
the second and third questions would
utilize what is typically called
descriptive or survey research.

In addition to collecting the data on

the three outcome questions, I would
make sure that an appropriate measure
of program fidelity was utilized in
order to document that the program
being evaluated was, in fact,
implemented properly. This
additional implementation fidelity
data would also be useful in
documenting any difficulties that may
have been encountered in delivering
the program.

Sample Response Notes

Data Analysis
The statistical analysis would be
dependent on the specific evaluation
questions and the type of data
collected. The first evaluation
question for which pre and post data
were collected for the three skill
areas (X, Y and Z) would probably be
a paired t-test. The analysis for
the remaining two evaluation
questions would most likely be simple
descriptive statistics such as means
and standard deviations.
The response to the section on
data analysis is very highly
dependent upon the type of data
collected. As such, there are
many different approaches to this
response and it is important to
carefully respond to this in light
of those sections.

9
 Sample Response Notes
Ethical Considerations
As a School Psychologist I must take No notes are provided for this
very seriously my obligations portion of the response as it is
regarding ethical practice(s). assumed that ethical
considerations is an area that
For the purpose of this question, I every student in our program
am going to assume that all the data understands.
collected are not normally and
routinely collected by the district
and that the instruments employed
were selected specifically for the
purpose of this evaluation.
Consequently, I cannot claim that I
am simply using routinely collected
archival data. On the other hand, I
am also not conducting a research
project, per se. Rather, I am
conducting an evaluation and schools
do have the right to evaluate their
effectiveness.

In this particular case I am also not

withholding a valuable treatment from
those in need of that treatment in
order to evaluate it. Therefore, I
do not have to worry about that
aspect of my ethical role.

Given these, I would still advise the

district to inform the parents and
teachers that an evaluation was to be
conducted. I would make it clear
that all participants knew and
understood that THEY were not being
evaluated, that data would be
anonymous (when possible) and
confidential for all, and that
participation was voluntary.

Increasingly, and especially among

larger school districts, districts
convene an institutional review board
(IRB) which approves internal (and
external) requests to conduct
research. Assuming my district has
such a committee, I would seek
approval from this committee before
approaching prospective participants
and gathering any data to be used in
the program evaluation.

10
 Sample Response Notes
Strengths
I think one of the strengths of this Strengths typically would be ways
approach to the evaluation is that it in which internal and external
tries to build “ownership” on the part validity are enhanced.
of the district and does not simply
employ me, as the School Psychologist,
to come in and collect data. The
identification of the evaluation
questions and the design of the
evaluation, itself, would be planned
with the appropriate school personnel.

A second strength would be that part

of the design is to monitor program
fidelity. Without some measure of
program fidelity, it would not be
possible to conclude whether the
program was effective.

Sample Response Notes

Weaknesses

As with any kind of program Weaknesses would typically be

evaluation, it is possible that those things that limit internal
students and other participants try to or external validity.
do better simply because they are
being evaluated and know they are
being evaluated. This would be a
threat to the internal validity of the
evaluation. Program satisfaction by
parents and teachers is very
subjective and may not be very
reliable.

11