DyKnow Training Evaluation

Prepared for Culver Educational Foundation By Culver Technology Support Committee Phillip Cook, Director October 9, 2012

Table of Contents
Executive Summary ........................................................................................................................... 3 Introduction...................................................................................................................................... 5 Evaluation Framework ...................................................................................................................... 6
Level 1: Reaction...................................................................................................................................................................................... 6 Level 2: Learning ..................................................................................................................................................................................... 6 Level 3: Behavior ..................................................................................................................................................................................... 6 Level 4: Results......................................................................................................................................................................................... 7

Conclusion ........................................................................................................................................ 8 Appendix .......................................................................................................................................... 9

Evaluation Plan......................................................................................................................................................................................... 9 Gantt Chart ............................................................................................................................................................................................... 11 Level 1 (Reaction) Instrument ......................................................................................................................................................... 19 Level 2 (Learning) Instruments ...................................................................................................................................................... 21 DyKnow Training Scenario .................................................................................................................................................................... 21 Peer Evaluation Checksheet .................................................................................................................................................................. 23 Level 3 (Behavior) Instrument ........................................................................................................................................................ 24 Level 4 (Results) Instruments .......................................................................................................................................................... 25 Faculty & Staff Self-Reflection .............................................................................................................................................................. 25 Student Experience Questionnaire ..................................................................................................................................................... 27

Executive Summary
As a result of past discussions amongst faculty at Culver Academies, the Instructional Technology department at Culver has installed a server to run DyKnow Vision and Monitor software and purchased licenses for student, faculty and staff machines at Culver, beginning in the spring semester of the 2012-2013 school year. All faculty and staff at Culver are trained on proper use of DyKnow Monitor, with faculty receiving additional training in the use of DyKnow Vision over the course of a two-day training. All staff attends the first day of training, focused on DyKnow Monitor. The primary goal of the training is to ensure that faculty and staff are able to appropriately implement DyKnow software within the academic activities that occur on campus. Training is designed to be interactive, with both sessions focused on using the software within the context of a classroom setting. This evaluation is designed to determine the effectiveness of training, as well as gauge the impact of DyKnow software upon the academic community at Culver. Evaluation planning began in August of 2012, headed by the Director of the Technology Support Committee on campus at Culver. The Technology Support Committee (TSC) is comprised of fourteen faculty and staff and is charged with the design and implementation of technology related trainings and evaluation on campus at Culver. In early August, four members of the TSC received training from DyKnow representatives; those members then trained the remaining members of the TSC in the use of DyKnow through designated TSC meetings in September and into early October. As a result of the initial trainings within the TSC, the following three objectives were identified: faculty and staff competency in the use of DyKnow Monitor to manage classroom and related academic settings, the use of DyKnow Vision by faculty to create digital collaborative situations within the classroom, and the ability to create DyKnow Vision compatible presentations for use within the academic classroom. Based upon these objectives, competencies of management, design and implementation were developed. Utilizing Kirkpatricks evaluation framework, instruments were created to evaluate the effectiveness of the training program in regards to these competencies related to the first four levels of the framework: reaction, learning, behavior and results. Level one (reaction) survey gauges the effectiveness of the training in regards to training materials, training methodology, instructor implementation and setting. Level two (learning) evaluates comprehension of basic terminology and skills, and consists of a real world situation which is peer evaluated using a supplied check sheet. Level three (behavior)

involves an observation of the use of DyKnow within a classroom or related academic setting by a member of the TSC. Level four (results) consists of a written self-reflection as well as implementation and usage data collected via student experience questionnaires administered at the end of term 3 and term 4 on campus.

Introduction
DyKnow Vision and Monitor software provides a digital means for faculty and staff to collaborate and monitor academic activities on campus at Culver Academies. The DyKnow training program was developed based upon the need for a diverse group of employees to be able to utilize appropriate feature sets during academic class and study periods. DyKnow Monitor provides a much desired means to observe student laptop screens remotely, and allows for the enabling or disabling of specific applications and websites. DyKnow Monitor is viewed as an additional tool to ensure that students are using their school-issued laptops appropriately and are making good use of instructional time. In addition to the observational benefits that DyKnow Monitor provides, DyKnow Vision allows for the creation of digital collaborative learning environments, both face to face or across campus. Faculty can easily present class material remotely to students, and students can work collaboratively on projects using the team-centered features offered within the DyKnow Vision software package. Overall, the ability to create managed collaborative digital learning environments is very attractive, and makes good use of the schools existing 1:1 laptop program. Of primary interest regarding this evaluation is to determine the effectiveness of training, increased fluency in terminology and skills related to the implementation of appropriate use of DyKnow Monitor and Vision software within the academic classrooms at Culver Academies. As mentioned previously, Kirkpatricks evaluation framework serves as the basis for the preparation of each of the four evaluations developed. These levels and corresponding instruments are described in the evaluation framework that follows.

Evaluation Framework
Level 1: Reaction
A Level 1 reaction survey was designed in order to evaluate faculty and staff response to training in regards to learning environment, relevance, delivery, and overall training experience. The survey uses a four point Likert scale. Faculty and staff will be administered the survey immediately following the conclusion of each of the two day training sessions. The level 1 survey can be found in the appendix. The survey provides timely feedback regarding the two-day training session. Specifically, evaluation questions focus on classroom environment, utility of training, and effectiveness of the instructor and instruction related materials. This data is important to ensure that training is identified as useful within the classroom and similar academic settings and was clearly communicated within an effective learning environment. Results from the level 1 instrument will be reported using mean scores as well as standard deviations. Positive scores will indicate faculty and staff feel prepared and able to use DyKnow Monitor and Vision appropriately on campus at Culver.

Level 2: Learning
DyKnow Monitor training is scheduled for Saturday, November 24th, 2012. The learning evaluation is administered during the final hour of the training. Participants are assigned a peer reviewer and given one of two related tasks involving the use of DyKnow, as well as a scoring check sheet. The peer reviewer will provide the scored check sheets to the training instructor. Level 2 instrument scores will be reported using mean scores as well as standard deviations. Mean scores on individual check sheet components are useful in determining topics in need of follow-up with future trainings and can help to identify areas in need of greater coverage and illustration during training.

Level 3: Behavior
The measurement of behavior occurs via academic observations through the submission of an observation sheet. Individual faculty and staff are each observed within their respective academic setting (either classroom or within the dorms & barracks on campus) by a member of

the Technology Support Committee. The observing TSC member records an overview of the observation and uses an observation sheet to mark observed uses of DyKnow Monitor and/or DyKnow Vision (see the appendix for observation sheet). Each TSC member will observe trained faculty and staff on two separate occasions, with each trained faculty and staff member being observed a minimum of two times. All observations will occur after training during terms 3 and 4 in the spring 2013 semester. Observational data within context is ideal for showing the extent to which DyKnow software usage is being implemented on campus at Culver. Results from the observations are compiled at the end of term 4 of the 2012-2013 school year and analyzed for frequency of observed usage, broken down by item from the observation sheet. The results from the behavior instrument serve to identify departments that may require follow-up regarding implementation of DyKnow as well as identifying departments on campus that are making exceptional use of DyKnow.

Level 4: Results
Results are determined via two methods: self-reflection and student feedback. The selfreflection (located in the appendix) is administered to faculty and staff during June week (beginning June 10, 2013) at the conclusion of the 2012-2013 school year, and consists of prompts related to the change of classroom or academic environment as the result of the use of DyKnow Monitor and Vision, with responses to prompts consisting of rated values and written comments. In addition to the self-reflection, students will provide feedback in regards to the impact that DyKnow Monitor and Vision software has had within their academic courses and within academic study times on campus during the spring semester. Questions addressing impact are inserted into the existing Student Evaluation Questionnaire (SEQ) that is administered to students at the end of term 3 and term 4(attached for reference within the appendix). TSC members will collate results from the faculty and staff self-reflections, and summarize common themes expressed within the written portions of the instrument. Rated prompts will have data collected and analyzed, with mean values and standard deviations presented. Results will also be broken down by department to identify regions of campus that experienced the most significant results. This analysis is particularly useful for future training planning for the 20132014 school year, where training can be focused within departments that have had lesser results

during the initial rollout of the software. The results data will also be shared with the Head of Schools, the Director of Instructional Technology, and the Co-chairs of the Academic Instructional Technology Department, all of whom are stakeholders in the funding and decision to bring DyKnow to campus at Culver.

Conclusion
In education, it is always a key concern to maximize instructional time. DyKnow Monitor and Vision software provide a means by which faculty and staff can minimize distractions during academic time, and as a result increase learning on campus. The proper training of faculty and staff is critical for ensuring that the software is used appropriately and has maximum impact upon the learning environment at Culver. Being able to conduct appropriately designed Monitor sessions and create interactive collaborative Vision sessions are key skills that are a focus of the DyKnow training program. DyKnow software is not cheap, and the initial investment in software licenses and server equipment must be justified through increased on-task behavior of students during academic times. Building a community that leverages DyKnow software for good and enhancing the technology skills of an already talented faculty and staff serves to improve not only the overall quality of education of students at Culver, but also opens the possibility for future thought leadership in the thoughtful implementation of technology programs amongst accredited collegepreparatory boarding schools.

Appendix
Evaluation Plan
Project Scope Identification Purpose Levels of Evaluation Objectives that will be measured Evaluation Plan DyKnow Training Evaluation Evaluation of two-day training program administered to faculty and staff at Culver Academies DyKnow Training Evaluation Plan, October 9th, 2012 Goals Determine the effectiveness of DyKnow training Levels 1-4: Reaction, Learning, Behavior & Results 1. Teach faculty & staff proper protocol in the setup of DyKnow Monitor sessions and plans. 2. Teach faculty & staff skills related to facilitating DyKnow Monitor sessions during academic time. 3. Teach faculty skills and techniques in using DyKnow Vision to create digital collaborative sessions. Process Date Due 8/9/12 10/30/12 11/24/12; 12/1/12 11/24/12; 12/1/12 11/24/12; 12/1/12 5/31/12 5/31/12 6/1/12 6/7/12 6/7/12 6/10/12 6/11/12 6/14/12 Evaluation Budget N/A; Evaluation completed in-house Who TSC TSC TSC Phil Peer evaluators TSC TSC Phil, Megan, Oliver, Lauren All faculty & staff TSC TSC TSC Phil

Item Create evaluation plan Develop instruments Observe DyKnow training Administer and collect level 1 data Administer and collect level 2 data Observe DyKnow usage by faculty & staff within academic setting Collect level 3 data Analyze data from levels 1-3 Administer and collect SEQ data Administer and collect level 4 data from faculty & staff Analyze data from SEQ and level 4 Prepare final report Present results to Head of Schools, Director of IT and AITD

Data Item Reaction to training Learning Behavior Results

When After training After training After training during terms 3 and 4 June week, 2013

Data Collection How Collected Survey Peer evaluation Observation sheet Self-reflection; Student SEQ survey Data Analysis How to control for other factors

From Whom Faculty & Staff Faculty & Staff Faculty & Staff Faculty, Staff & Students

By Whom TSC Members Peer evaluators TSC Members TSC Members

Target Measures Reaction

Trainers leave room Learning Mean scores; SD Testing scenarios similar to training scenarios Behavior Observation; Observation via frequency more than one TSC member Results Frequency; Mean Self-reflection scores; SD and SEQ data weighted equally Report of Findings What results are reported To whom 1. Reaction results Head of Schools, Director of IT & AITD 2. Learning results Head of Schools, Director of IT & AITD 3. Behavior results Head of Schools, Director of IT & AITD 4. Self-reflection & SEQ results Head of Schools, Director of IT & AITD

Methods of performing calculations Mean scores; SD

Who will perform analysis? Phil Phil, Megan, Oliver, Lauren

Technology Support Committee Technology Support Committee

How communicated Written report Written report Written report Written report

10

Gantt Chart

12

13

14

15

16

17

18

Level 1 (Reaction) Instrument

Level 1 DyKnow Training Survey

Instructions: Thinking about the training that you just completed, please indicate to what degree you agree with each statement using the following scale: 1= Strongly Disagree 2= Disagree 3= Agree 4= Strongly Agree

Please provide comments along with your rating to help the TSC improve this training in the future. Learning Environment 1. The training environment helped me to learn 2. There were no major distractions that interfered with my learning Comments: 1 2 3 4 1 2 3 4

Relevance 3. The training will be helpful for my success using DyKnow in the future. 4. I will be able to immediately use what I learned. Comments:

1 2 3 4 1 2 3 4

Delivery 5. I was well engaged with what was going on during the training. 6. The activities and exercises aided in my learning. 7. I was given adequate opportunity to demonstrate what I was learning. Comments:

1 2 3 4 1 2 3 4 1 2 3 4

Overall 8. The training met my expectations. 9. I am clear on how to apply what I learned on the job. 10. I would recommend this training to my colleagues. Comments:

1 2 3 4 1 2 3 4 1 2 3 4

20

Level 2 (Learning) Instruments

DyKnow Training Scenario DyKnow Monitor Training Peer Evaluation Scenarios Instructions: Select a partner within your training group to serve as a peer evaluator. Next, select one of the two scenarios, and respond to the prompts. Scenario A You are an instructor in a humanities class and you have designed an activity where students use online resources to research the life of Antoine Laviosier. To ensure students remain on task, you would like them to be able to access the internet, and use only Microsoft Word or Microsoft OneNote to record their research findings. 1. Identify and record the settings within DyKnow Monitor that you would utilize to accomplish this task in the space below. 2. Create an appropriate plan with these settings within DyKnow Monitor. 3. Demonstrate proper use of the plan to your peer evaluator. Scenario B You are a dorm supervisor in Beason in charge of facilitating academic study time from 8pm to 10pm. Lately there have been issues with students using their computers for non-academic tasks during academic study time. Specifically, the use of Skype, twitter and facebook have been causing significant distractions. 1. Identify and record the settings within DyKnow Monitor that you would utilize to minimize distractions during academic study time. 2. Create an appropriate plan with these settings within DyKnow Monitor. 3. Demonstrate proper use of the plan to your peer evaluator.

21

DyKnow Vision Training Peer Evaluation Scenarios Instructions: Select a partner within your training group to serve as a peer evaluator. Next, select one of the two scenarios, and respond to the prompts. Scenario A You are an instructor in a science class and you would like to conduct a DyKnow Vision collaborative session where you will present a series of slides illustrating the changing models of the atom. You have created the PowerPoint already, but need to convert the presentation to be able to use it with DyKnow. 1. Within Microsoft PowerPoint, create a DyKnow compatible version of the supplied PowerPoint file. 2. Demonstrate how you would import the file into DyKnow to your peer evaluator. 3. Demonstrate proper use of Vision, using the converted PowerPoint to your peer evaluator. Scenario B You are a dorm supervisor in Argonne and several of the girls from the dormitory have requested a collaborative session using DyKnow Vision to discuss dormitory issues. 1. Identify and record the settings within DyKnow Vision that you would utilize to create a collaborative session. 2. Create an appropriate session with these settings within DyKnow. 3. Demonstrate proper use of the session to your peer evaluator.

22

Peer Evaluation Checksheet DyKnow Training Peer Evaluation Checksheet Name of person being evaluated:_________________________________________ Name of evaluator:______________________________________ Training: Monitor Vision Scenario:

DyKnow Monitor Faculty or staff member being evaluated utilized: Start/Stop Monitor Allow Applications Block Applications Allow URL Block URL Attention Send File Remote Control Shutdown/Restart Poll Status Monitor class view Monitor individual view

DyKnow Vision Faculty or staff member being evaluated utilized: Chat Group session Presentation Quiz inserted within presentation Status Request file Collect file Return file Share screen Presenter control Presentation view

23

Level 3 (Behavior) Instrument

DyKnow Training Observation Sheet Name of person being observed:______________________________________ Department:_______________________________ Observer:____________________________________ Instructions: Record the frequency of DyKnow skills and features that are used during your observation using tick marks. DyKnow Monitor Faculty or staff member being observed utilized: Start/Stop Monitor Allow Applications Block Applications Allow URL Block URL Attention Send File Remote Control Shutdown/Restart Poll Status Monitor class view Monitor individual view

DyKnow Vision Faculty or staff member being observed utilized: Chat Group session Presentation Quiz inserted within presentation Status Request file Collect file Return file Share screen Presenter control Presentation view

Level 4 (Results) Instruments

Faculty & Staff Self-Reflection DyKnow Training Self-Reflection Name (optional):______________________________________ Instructions: Reflecting back upon your use of DyKnow Monitor and Vision over the past two terms, respond to the following questions. Your responses will help the TSC to assess the impact that DyKnow training has had upon . What is your current position at Culver? a. Faculty member b. Staff member Which department are you a member of at Culver? a. Fine Arts b. Humanities c. Science d. Math e. Modern & Classical Languages f. Student Life g. Other:______________________________ 1. I use DyKnow Monitor during academic settings on campus at Culver. a. Always b. Frequently c. Sometimes d. Occasionally e. Never 2. I use DyKnow Vision during academic settings on campus at Culver. a. Always b. Frequently c. Sometimes d. Occasionally e. Never 3. The use of DyKnow Monitor has resulted in fewer technology related distractions. a. Strongly Agree b. Agree c. Disagree d. Strongly Disagree e. I do not use DyKnow Monitor

25

4. The use of DyKnow Vision has resulted in greater classroom collaboration. a. Strongly Agree b. Agree c. Disagree d. Strongly Disagree e. I do not use DyKnow Vision 5. As you reflect upon your use of DyKnow Monitor, how have you observed the use of Monitor affect the learning environment at Culver?

6. As you reflect upon your use of DyKnow Vision, how have you observed the use of Vision affect the learning environment at Culver?

26

Student Experience Questionnaire Excerpt of DyKnow prompts from the Student Experience Questionnaire Section E: DyKnow Software Instructions: The questions that follow relate the to use of DyKnow by your instructor as well as counselors and other staff during academic time on campus at Culver. Read each statement carefully, and then indicate to what degree you agree with each statement using the following scale: 1= Strongly Disagree 2= Disagree 3= Agree 4= Strongly Agree

1. Many faculty and staff are using DyKnow on campus during academic time. 2. Faculty and staff use DyKnow in helpful ways. 3. DyKnow helps me to stay focus on learning within the classroom. 4. DyKnow helps me to stay focused on learning during academic study time. 5. My study behaviors have changed as a result of DyKnow usage on campus. 6. DyKnow has improved the quality of instruction on campus at Culver. 7. I am using technology more frequently in class as a result of DyKnow.

27