Integrated Improvement Drive 1

Individual Project

Review of a Social Education Program

Submitted by

Tineka Jagan (320084996)

EDID6504- Program Evaluation & Course Assessment Methods

Ms. Camille Bremmor

 Integrated Improvement Drive 2

Table of Contents

Goal Oriented Model versus Naturalistic Model……………………………………………Page 3-4

The Comparison …………………………………………………..Page 5-6

Discussion & Justification ……………………………………………….Page 7

Selection ………………………………………………………….Page 8

Practical Evaluation Process …………………………………….Page 9-10

Steps to assure validity of using Naturalistic Evaluation model----------------------------11-12

 Integrated Improvement Drive 3

Phase 3

Goal Oriented Model versus Naturalistic Model

Both the Goal Oriented Model and the Naturalistic Evaluation Model can be applied to the new program proposed.

The goal-oriented evaluation model focuses on defining goals in measurable terms and measuring their degree of

attainment. The Naturalistic Model, on the other hand, focuses on audience concerns and issues to base its

evaluation. Whilst the goal-oriented model focuses on achieving stipulated desired outcomes at the beginning of

the program and measures success based on such. The naturalistic evaluation method is continuous, interactive and

qualitative.

If the goal oriented model is used in the proposed program, the focus for the program will be determined in

advance and the main audience and aim will be the goals set by program administers. Goals and objectives will be

determined based on stakeholders/managers/funders requirements for the program and the will be designed to meet

those expectations. This model has many advantages as the program will be well organized to achieve set

outcomes. Also, requirements of stakeholders will be known and therefore meeting success will be perceivable.

Another benefit of this model which is employed in the program would be the fact that goals are set by experts in

the field and those setting the goal are fully aware of what the program hopes to achieve. This creates unison of

purpose thereby leading to the better execution of the program.

However according to (Marsh, 1978) ‘the evaluation researcher who elects to use a goal oriented evaluation

strategy, must be aware of the limitations or biases introduced by this method. Specifically, this approach may lead

the researcher to neglect informal organizational goals, unintended outcome, or organizational dynamics that

influence the achievement of program goals’. Whilst a goal – oriented model of evaluation may provide many

benefits, such a model also has its limitations. The downfall of such models, can lead to stunt growth in program as

participant’s views and other external factors are not taken into consideration, thereby undermining the

effectiveness and growth of the program. In a goal oriented model, the focus will be one directional thereby

reducing creativity and flexibility for growth and enhancement.

 Integrated Improvement Drive 4

The naturalistic model if employed creates a bit of ambiguity for the program as information is dependent on

various sources continuously throughout the program. The advantages of this method, however, will be a more

qualitative descriptive of the program, creating a more applicable program based on the needs of participants, and

flexibility for unintended outcomes. The naturalistic model presents a broader view of the benefits and gaps of the

program based on the needs of its audience. The shortfall of the naturalistic model is that it is nondirective, there is

no clear focus for measuring the program and so large amounts of data will have to be analyzed and synthesized to

determine assessment result and program focus.

 Integrated Improvement Drive 5

The comparison

Clearly, there are some major differences between the goal oriented and naturalistic models which can be

highlighted; however, this is not to say that there are in fact no similarities between the two as it should be noted

that they each serve particular purpose when considering the most appropriate approach for evaluation (Esposito

and Freda, 2015). In this regards, the two models can be considered to be similar in the following ways:

- The purpose is to evaluate the effectiveness of a program: while the methods may differ, the underlying

purpose is still the same; this is to provide an understanding about whether a program was successful.

- - procedural outcomes are not neglected: while it may be assumed that the goal oriented model only focuses

on the finish line, it, quite like the naturalistic model, has the ability to also evaluate how the program is

being conducted and whether intermediate targets are being met as expected. It is needless to say that there

also vast differences between the two models, some of these differences have been highlighted as follows:

- - data collection: one of the more striking differences regards the quantities and qualitative nature through

which data collection processes are performed by the goal oriented and naturalistic models respectively,

which the goal oriented model focuses on statistical analysis, the naturalistic model has a deeper view on

individual experiences,; this can lead to a more detailed understanding about the why behind the process

and outcome of program. Furthermore, this infers that the techniques for data collection will differ as well

as methods for sampling, data analysis and presentation of findings.

- - subjectivity and objectivity: while the naturalistic model places heavy emphasis on personal reflection, the

goal, the goal oriented model seeks to evaluate the collective experience in a procedural manner which

dictates the use of reliable statistical methods.

- -reliability and validity: when considering research methods, particularly for the purpose of assessment and

evaluation it is always vital to ensure that one can provide and defend evidence of reliability and validity.

This speaks directly to how trustworthy the nature of the result are, how well they can be generalized and

taken at face value.( Rossi, Lipsey, & Freeman, 2004).

 Integrated Improvement Drive 6

- Clearly, the goal oriented model provides more scope for ensuring these two constructs are clearly defined

and achieved.

- Context; the use of the naturalistic evaluation make direct consideration for the context of the program;

contextual factors such as setting/environment and interaction can invariably have an effect (whether

positive or negative) on the outcome of the program altogether. In contrast, the goal oriented model places

no specific emphasis on contextual factors and how they affect outcomes.

 Integrated Improvement Drive 7

Discussions and Justification

The integrated improvement drive is one where it is hoped to see the improvement in the quality of behavior

among children. Based on the aims, goals, desired outcomes, it is deemed critical to ensure that there is not

only quantitative measure to assess whether the foreseeable changes have taken place (because this may only

take place and be better evaluated in a long term manner) but also there is a need for a qualitative process to

ensure that the views, experiences, insights and perception of the participants are taken into consideration. This

is an effort to provide more detailed information about the effectiveness of the program.
 Integrated Improvement Drive 8

Selection

A mixed method of evaluation is proposed with the adoption of the use of both goal oriented model as well as

the naturalistic model. This will allow for the use of both quantitative and qualitative methods which will reap

data representative of both nature of research. A mixed method have been seen to be more advanced as it

allows the researcher to enjoy the benefits of both a quantitative and qualitative design which produces results

of a statistical nature and explanatory depth ( Tucker-Brown,2012 and Wisdom& Creswell,2013).

 Integrated Improvement Drive 9

Practical Evaluation Process

During the implementation of the goal oriented model, a trend of goal setting, feedback and monitoring which

is used to present a more systematic analysis (Marsh, 1978 and Shaw & Gaynor, 1982) (See Figure 1.) this

stipulate that goal oriented model will be used to help ensure that strict, measurable goals are defined, that there

is a consistent monitoring throughout the program for assessing the use of checklist and task sheets and finally

to provide feedback about the overall program.

Figure 1.

The Goal oriented Approach to Evaluation and Case Study from Drug Abuse Treatment

Figure 1 Goal setting and monitoring steps

Diagram showing the practical application of the goal oriented approach according to Marsh (1978).
 Integrated Improvement Drive 10

To evaluate this initiative with the use of the goal oriented model, the naturalistic model will be implemented to

allow participants to provide feedback. Feedback will be done through the use of individual and focus group

interview, case studies and journals. Each person that participate will be asked to keep a journal during the program

which will be used to document their thoughts and experiences as they progress. This journal will also be used to

monitor their progress from beginning to the end of the program and will also assist participants in setting

individual goals to be attained even after completion to ensure continued personal development and to assure

program success.
 Integrated Improvement Drive 11

Steps to Assure Validity of using Naturalistic Evaluation Model

The concept of validity and reliability are key in any nature of research which seeks to ensure rigour and scientific

evidence (Rossi, Lipsey, & Freeman, 2004). Emphasis is placed on ensuring methods of evaluation can meet

standards which prescribe that one’s measurement is valid, meaning that it measures the construct that it wants and

that it is reliable, meaning that it has consistency in measurement regardless of the situation ( Stemberg, &

Sternberg, 2010).

The naturalistic evaluation model may be characterized into three phases, familiarization, action and synthesis

(Rubin, 1982). These phases were used as an aid to pilot the valid assurance of the naturalistic model to evaluate

the Integrated Improvement Drive. During the familiarization phase persons involved in the programme are given

the opportunity to become accustom to its objectives. Wolf (1979) implied that this is the foundation phase in

which the investigation begins and an overview of the setting and related literature is recognized. After analyzing

the objectives and the anticipated outcome of the programme, evaluator will be able to determine the assurance of

the selected evaluation model. Essential decision would then be made at this stage that will impact the effective

operation of the programme. Decisions such as the roles of the project managers, the frequency of workshops, the

availability of funds and the methodologies that may utilized to train persons.

Field (2013).

While reliabity may be a bit easier to assess through for example a test-retest method; validity often requires more

rigour to ensure that the instrument (s) used really are ‘fit for purpose’ (Rossi, Lipsey, & Freeman, 2004).

Therefore, two (2) measures which would be implemented to assure validity are:

1. Expert evaluation: which speaks to the use of experts in the field of program evaluation as well as behavior

management and psychology would be beneficial to review the detailed nature of the instrument to be used

for the evaluation purpose.

 Integrated Improvement Drive 12

2. Participation Evaluation: stakeholders should have the ability to have a say in whether any questions asked

for the purposes of feedback are effectively and efficiently designed.

Based on these two reviews conducted, changes can be made as a result of appropriate recommendations to

ensure that validity is achieved.

 Integrated Improvement Drive 13

References

Council, I.o. (2002). Community Programs to Improve Youth Development. Washington DC:

Esposito, G. and Freda, M.F. (2015). Evaluating Training Context Competence of Use:

Field, A. (2013). Discovering Statistics Using IBM SPSS Statistics. (4th Ed.). SAGE Publications Inc.

https://www.nap.edu/read/10022/chapter/2

Marsh, J. C. (1978). The Goal-Oriented Approach to Evaluation: Critique and Case Study from Drug Abuse

Treatment. Journal of Evaluation and Program Planning. Vol. 1. Pp. 41-49. Pergamon Press Limited

Production and Unproductive Models of Use. Journal of Evaluation and Program Planning, Vol.50, pp.77-87.

Rossi, P.., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A Systematic Approach. (7th Ed.). Thousand

Oaks, CA: SAGE.

Rubin, B.M. (1982). Naturalistic Evaluation: Its Tenets and Application. Studies in Art Education. Vol 24. No. pp.

57-62. National Art Education Association. Retrieved from https://www.jstor.org/stable/1319804

The National Academics Press. Retrieved from

Tucker-Brown, A. (2012). CDC Coffee Break: Using Mixed Methods in Program Evaluation. Retrieved from:

https://www.cdc.gov/dhdsp/pubs/docs/cb July 2012 pdf