Braxton County Program Evaluation

FRIT 7237 Final Project

Spring 2019

Written By:

Bryandra Bell

Shawn Benton

Michelle Wynn
 I. Background:

Addressed Needs​: “Lack of sufficient preparedness of middle school math teachers.”

Stakeholders:

● Great Southern University personnel

● Braxton County Superintendent of Schools

● Assistant Superintendent for Student Achievement

● Director of Early Academics

● Braxton County Director of Professional Learning

○ Information Need: ​There was a need for the PBL project and collaboration in

planning and executing the PBL project.

Participants​: Braxton County Teachers (6th-8th grade)

Objectives:​

● Objective 1:​ create PBL modules for mathematics based on examples from local

businesses and industries.

● Objective 2:​ create PBL modules, which address MGSE.

● Objective 3:​ integrate available instructional technology into PBL experiences.

● Objective 4:​ implement and evaluate the learning modules created.

​Activities:​

● 5-day summer workshop,

● Electronic consultation (GSU faculty and teacher participants)

● 2 post-workshop meetings

Location and Longevity:​ Braxton County, June 2019-January 2020

Resources for Implementation​: L

​ ocal businesses, GSU faculty, Email and Wikis
 Measurable outcomes:​ Deepen teachers’ knowledge and preparation in mathematics and

increase student achievement in mathematics.

Conceptual Model Narrative

This conceptual
model illustrates
the implementation
of the grant project
which leads to
desired short-term
and long-term
outcomes.

II. Implementation Evaluation

Evaluation Questions Data Sources

P1: Were the initial experience and follow up activities implemented Surveys and Observations
as planned?

P2: What is the quality of the initial program activities? Surveys and Observations

P3: Who are the program participants and how were they recruited? GA Milestone Data, Surveys
and Observations

P4: What is the quality of follow up and support activities? Surveys, Observations,
Self-Evaluation Rubric
 Evaluation Design Narrative

Methodological Approach:​
Mixed Methods
(Quantitative/Qualitative)
Sampling​: Volunteer teachers
from the school district
Comparison Group​: Groups
without PBL
Data from spring 2018/2019 GA milestone data (quantitative data) as
well as observations, self-evaluation rubric and survey data
(qualitative data) would be used to make comparisons and draw
conclusions. Sampling of classes from the district will be used which
will include classes with similar teaching styles, class sizes and
materials. Other classes with similar teaching styles, class size and
materials will be chosen to represent groups that do not incorporate
PBL in order to make comparisons with those that do use PBL.

Collect Data Narrative

Surveys and Surveys and observations will be used to determine if the initial experience and
Observations follow up activities were implemented as planned, determine the quality of the
initial program activities, as well as decide who are the program participants and
how they were recruited. Surveys, observations along with the self-evaluation
rubric will be used to assess the quality of follow up and support activities.

Analyze Data Narrative

Georgia Milestone Data The data will be analyzed by reviewing the

responses of the surveys and self-evaluation as
Surveys and Self-Evaluation-Review the responses well as review the observed behaviors and
implemented activities. Data will also be
Observations-Review the observed behaviors and compared to the comparison group during the
activities summative evaluation.

Provide Information to Audiences Narrative

Formal Report to include:
● Background research
● Evaluation Questions
● Procedures
● Analysis of the data
● Findings from the Evaluation
● Conclusion and recommendation
for further evaluation
A formal report will be used to present the information
to audiences (stakeholders) in order to help members
have a clear understanding of the evaluation and make
decisions about the continuation of this type of project
for creating learning experiences in math using problem
based learning. The report will also document any
assumptions made as well as possible risks during the
data collection plan and schedule.
III. Summative Evaluation

According to the ​The 2010 User-Friendly Handbook for Project​ ​Evaluation,​ a

summative evaluation uses the data gathered from a project to assess its effectiveness and

longevity (pg. 10). A summative evaluation of the ​ “Standards for Mathematics and

Problem-based Learning: A Grant Funded Project” will be conducted to determine the success

of its implementation, and to analyze the outcomes of the project. The following questions will

be addressed: 1) To what extent were teachers able to develop PBL modules that were

connected to local business and industries, aligned with MGSE, and incorporated appropriate

uses of technology? (Objectives 1, 2, and 3) 2) To what extent were teachers able to implement

and evaluate those modules? (Objective 4) Quantitative and qualitative data sources will be used

to assess the outcomes of the program. They include: GA Milestone data, pre and post surveys,

self-evaluation rubrics, and observations. Milestone data will be compared between classes that

participated in implementing the new mathematics modules and control classes that did not

participate in the grant to implement the curriculum. This comparison will allow evaluators to

see the effect the curriculum made on student achievement. Pre and post survey data as well as

observation data will allow evaluators to see how efficiently teachers implemented the

knowledge they gained from the professional development within their classrooms. The results

from the evaluation will be used to ascertain the efficiency of the project’s success regarding

goals and objectives, strategies and activities, and other associated processes. Upon the

completion of the module implementation, teachers will take part in a self-evaluation using a

rubric provided to them. This data will be compared with the results from the teacher group

interviews. A final meeting will be held with all participants from Braxton County Schools and
 Great Southern University to review the results from the data collected and discuss the project’s

success. The projected impact of the PBL modules are to create deeper understanding of GA

state standards through real world experiences, appropriate use of technology, as well as the

long-term expectation of an increase in student achievement in mathematics.

Projected Problems to Consider Solutions

Teacher evaluators not properly 1) All evaluators receive training during summer 2019
trained 2) Evaluator support available-ongoing

Observation length not equal 1) All observations equal in length (minimum 1 hour)
2) All observations equal in number

Students with special needs not 1) Students with special needs are included in focus and
considered control groups (accommodations and modifications
followed)

Evaluation Questions Objectives Indicators Data Sources

O1) To what extent Objectives 1, 2, Modules draw on Surveys (pre

were teachers able to and 3) Create PBL content and processes and post),
develop PBL modules Modules based on from local businesses teacher
that were connected to local businesses and industries, module observations,
local business and and industries, content, tasks, and observe local
industries, aligned with addressing MGSE, assessments are aligned businesses/ind
MGSE, and and integrate with appropriate MGSE, ustries using
incorporated appropriate technology into modules contain math skills in
uses of technology? PBL experiences activities that effectively real-life
(Objectives 1, 2, and 3) use technology situations

O2) To what extent Objective 4) Record of Self-evaluation

were teachers able to Implement and implementation rubric
implement and evaluate evaluate PBL
those modules? Modules Record of Teacher
(Objective 4) self-evaluation observations

Focus and
control groups
IV. Data Collection Schedule and Summary Table

Data Set Date of Instruments Data collected by:

Collection already
developed?

Georgia Milestones Data Spring 2018 Yes GaDoe

Collected (School and State)

Teacher Survey of Need October 2018 Yes Great Southern University

Meetings with Braxton County October 2018 Yes Great Southern University
Personnel

Teacher Pre-Grant Survey June 2019 Yes Great Southern University

Local Business Observations June 25th Yes Teacher Participants

Classroom Observations August 2019 - Yes Great Southern University

December 2019

Self-Evaluations December 2019 Yes Great Southern University

Teacher Interviews January 2020 Yes Great Southern University

V. Project Timeline

Event Date

Collect milestone data. Spring 2018

Teacher Survey of Need October 2018

Meetings with Braxton County Personnel October 2018

Collect milestone data. Spring 2019

Teacher Pre-Grant Survey June 2019

Summer 5-day workshop June 2019

● Introductions and New Standards June 24, 2019

● Visits to local businesses/industries for identification of Mathematics in June 25, 2019

context.

● Problem-based learning and Instructional Design; Identification/review June 26, 2019

 of Mathematics in context

● Instructional Technology, Idea Sharing, Problem-based Learning June 27, 2019

module creation.

● Problem-based Learning Module creation, Planning for implementation June 28, 2019
and Evaluation.

Electronic Consultations & Post Workshop Meetings July 2019

Phase 1: Refinement of Problem-based learning modules. July 1-19, 2019

Mid-implementation Meeting September 2019

Phase 2: Refinement of Problem-based learning modules through electronic Fall 2019

communication.

Teachers conduct self-evaluation of module implementation. Fall 2019

Classroom Observations Take Place August 2019 -

December 2019

Post Grant Survey December 2019

Data Analysis Conducted January 2020

Project Debrief Meeting in Braxton County January 2020

(Teacher group interviews)

Final Report Provided to Braxton County January 31, 2020

VI. Additional Resources:

Click here to access the Teacher Self-Evaluation Rubric.

Click here to access the Summative Rubric

VII. References:

Frechtling-Westat, J. et al. (2010). ​The 2010 User-Friendly Handbook​ for Project Evaluation.
Arlington, VA: National Science Foundation.