Evaluation of Determining Instructional Purposes

Far West Laboratories

Educational Research and Development
Submitted to:
Florence W. Larson
Chief Program Officer
Far West Laboratories

Picture courtesy of Quest Garden

Submitted by:
Angie Kruzich
President and Evaluation Team Leader
Kruzich Evaluation Services
October 20, 2013

Introduction
Kruzich Evaluation Services (KES) proposes to evaluate Far West Laboratory's (FWL)
Determining Instruction Purposes (DIP) training program. This evaluation would seek
information about the units being used to train school administrators and graduate
students within the DIP program. In addition, the evaluation will help to determine
whether FWL should continue to invest and/or expand the use of the training units. The
Request for Proposal (RFP) also clarifies that the materials have not yet been used in a
real training course, yet. KES believes that a Decision Making Evaluation Model will be
the best model to answer these inquiries for FWL.

Description of Program
KES has inventoried the materials being used in the DIP program which include a
handbook, three main training units and more detailed teaching modules within each
unit. The purpose of the DIP training is to help those enrolled plan effective school
programming.
The coordinator's handbook contains procedures and other detailed information that
helps one plan a series of training sessions. It is mentioned in the RFP, that one does
not need any prior instruction or knowledge of this training course in order to be the
managing coordinator.
The training units themselves each contain four to six detailed modules. The three units
in the DIP program are:
Unit 1: Setting Goals
Unit 2: Analyzing Problems
Unit 3: Deriving Objectives
Each module within a unit contains several types of teaching materials. This includes:
Instructional Objectives
Reading Material
Individual and/or Groups Activities
Feedback Techniques
The RFP also states that any one unit or combination of units can be used together or
independently of one another. For example, a coordinator could just use Unit 2:
Analyzing Problems for a smaller training course.

2|E d T e c h 5 0 5 : E v a l u a t i o n P r o j e c t

Evaluation Method
Purpose
The leadership at Far West Laboratories would like the results of this evaluation to
determine whether it is sensible to continue producing and marketing the DIP training
units. Will this program be a good investment of FWL money? Likewise, these results
will also be helpful to school administrators who may or may not be initially interested in
purchasing the DIP training.

Audience
This particular evaluation will provide information to three main groups.
1. The Far West Laboratory leadership staff will be the primary group interested in
the outcomes of this evaluation. FWL primary objectives are to use this
evaluation to help make decisions about further marketing and production of the
DIP training units.
2. In addition, the Training Coordinators hired to conduct the DIP training sessions
for the purposes of this evaluation will be provided with the final evaluation.
Feedback to and from the trainers can determine if any necessary adjustments to
the DIP units should take place before marketing and distribution.
3. Finally, the trainees or future prospective clients, such as school administrators
and graduate students, would be interested in these evaluation results.
Evaluation recommendations, a satisfactory report or both will help anyone
interested in purchasing the DIP units make an informed decision. Questions
about the DIP units could also be answered with the information obtained
through evaluation.

Information-Collection
In order to truly determine the effectiveness of the DIP program, this evaluation must
obtain information from more realistic scenarios. Thus far, the DIP program has not
been simulated or used in any real training situations. Therefore, the main focus of the
evaluation process should include real-life training situations. KES proposes that three
training sessions are conducting using the full DIP unit curriculum in each. For each
session, one coordinator would ideally have 10-15 trainees over the course of eight total
days (2 days/wk). Coordinators would travel to trainees. Only from true training courses
and invested participants can many questions be answered to assist FWL in their quest
for answers. Data that would be of interest to analyze would include:
Who is interested in participating in this program?
3|E d T e c h 5 0 5 : E v a l u a t i o n P r o j e c t

How many participants are interested?

How many schools are interested?
How many school districts are interested?
How much are participants, administrators and/or school districts willing to pay?
Did participants feel the information was valuable in the trainings?
Which topic(s) were most helpful? Which topic(s) were least helpful?
Would participants apply their knowledge from the program?
Did participants leave the training understanding how to apply the information?
Was the training powerful enough that participants followed through by applying
learned information from the training?
Would participants recommend the DIP training to others?
How did participants apply the training within their own schools?
What are the costs to provide the trainings and distribute materials?

Data Collection Instruments and Sources

In order to answer the above questions, several different techniques would be used to
collect data and information. One collection method would be to conduct surveys of the
coordinators, and participating trainees during one of three training sessions. This can
collect data to assist answering questions related to interest to purchase and how well
the DIP units succeeded during the trainings. The surveys can be administered right
after the DIP trainings but then a follow up survey can also be conducted several
months or even a year later to see how the trainings were applied by the trainees.
Another technique to collect information about the DIP units would involve interviewing
the coordinators, school administrators and graduate students. The interviews would
take place during the trainings and be conducted by the evaluation team leader.
Interviews can go a little further than a survey in that the interviewer can ask the
interviewee to expand on their thoughts and ideas.
Additionally, interviews would take place with participating school district central offices,
FWL accounts payable and receivable personnel and FWL marketing department
personnel. During these interviews, data can be collected to help analyze the marketing
and sales of the DIP units. What cost is acceptable such that the units are purchased by
clients and FWL covers their expenses.
Finally pre-testing and post-testing would be conducted. This is more concrete data
collection to show if there was growth in knowledge by the trainees. This data would be
analyzed and included in the final evaluation report as would all interview and survey
data.

4|E d T e c h 5 0 5 : E v a l u a t i o n P r o j e c t

Task Schedule
The timeline involved to complete the evaluation from start to finish would incorporate
the following tasks, dates and personnel.
Task

Team Member

1. Submit evaluation proposal to FWL

2. Await evaluation acceptance from FWL
3. Advertise program through the United States
4. Simultaneously, create surveys and interview questions
5. Meet with and update FWL
5. Hire three coordinators for the training sessions
6. Begin setting up training session dates and times
7. Conduct multiple training sessions
8. During training sessions, conduct interviews
9. As training sessions come to a close, distribute surveys
10. Analyze survey and interview results
11. Write final evaluation report
12. Present final report and recommendations to FWL

Kruzich
Kruzich
FWL
Jones & Rice
Kruzich & Jones
Kruzich
Kruzich
New Hires
Kruzich
Kruzich
Jones & Rice
Kruzich & Jones
Kruzich & Jones

Closing
Date
Oct 2013
October
November
November
December
Nov - Jan
Dec - Jan
Feb - Apr
Feb - Apr
Mar - Apr
May
May - Jun
July 1, 2014

KES Project Personnel

Angie Kruzich, Ph.D., KES President - Dr. Kruzich, also KES's Evaluation Team
Leader, provides leadership, mentorship and technical advising for the company and is
the key consultant who works directly with clients. With specialties in social policy,
program evaluation and statistical analysis, she adds her wealth of experience to all
aspects of research, including problem definition, project design, survey construction,
focus group moderation, data analysis, and reporting.
Andrea Jones, Ph.D., KES Research Analyst - Andrea brings extensive experience in
online survey methodology, quantitative analysis and reporting to KES. As a part of her
role, Andrea consults with clients on questionnaire design, guides projects through data
collection and oversees the analysis and writing of research reports.
William Rice, Ph.D., Washington State University Education Department - William
Rice is frequently used as a subject matter expect (SME) by KES. For this particular
evaluation, he would be called in to consult and assist with this evaluation regarding the
DIP units. William has 20 years of experience in the education department at WSU and
specializes in development of teacher training courses.

5|E d T e c h 5 0 5 : E v a l u a t i o n P r o j e c t

Budget
Budget expenses would include:
KES evaluation team costs
$20,000
o This portion of the budget will cover costs for:
Cost for Kruzich and Jones
Meeting Times for KES with FWL
Cost for SME Consultant
All Surveys and Interview Questions Written
All Data Analyzed
The Full Evaluation Report Write Up
Editing and Rewrite Time Involved
Presentation Time of Final Evaluation Report
Evaluation Team Leader Travel Expenses
o Travel two nights at a time
o Visit each coordinator twice maximum
o Conduct Surveys and Interviews during trip
o Hotel Est. $150/night
o Food Allowance $100/day
o Plane Fare $550/trip
o 6 trips: $1050/trip
$ 6,300
Hire 3 Coordinators to conduct 1 full DIP training session each
o Salary
8 days each
6 hours/day
$200/day
$ 4,800
o Travel Expenses: will travel to the trainees
Hotel, Food, Plane
$12,600
o Pre-Training of Coordinators
$ 4,800
DIP Training Materials
provided by FWL
o free to participants during evaluation process
o Provide Snack Food to participants
$ 900
On the Road Training Costs
o Photocopying
o Other Miscellaneous

$
$

300
300

Total Cost = $50,000

6|E d T e c h 5 0 5 : E v a l u a t i o n P r o j e c t