Running Head: Water Cycle Report

An Effectiveness Test on the Water Cycle Activity

Charles Wilcox
California State University, Monterey Bay

IST 622 Evaluation and Assessment

Dr. Bude Su
July 24, 2015

Water Cycle Report: Wilcox

Table of Contents

Introduction ..
Methodology
Activity
Learners .
Assessment ...
Results .
Entry
Instruction ..
Data .
Interpretation of Questions
Usability Test Results .
Outcomes
Recommendations .
Summary and Conclusions ..
Appendix ..

Page
3
4
4
4
5
6
6
7
7
11
12
14
15
15
17

Introduction
The purpose of this report is to discuss the outcomes of an effectiveness test of a
learning activity designed for fifth grade elementary school students. The focus of the
activity is a fifth grade science lesson on the water cycle. As part of the regular
curriculum in California fifth grade classrooms, students learn about the continuous
movement of Earths water as it changes physical states during evaporation,
precipitation and runoff. Students study the basic molecular process of how the

Water Cycle Report: Wilcox

physical states of matter, such as water, changes from liquid to gas to solid. In addition,
students begin to study weather patterns and how the water cycle causes different
weather patterns, with the understanding that warm air rises, cooler air sinks and
causes wind patterns to emerge.
The water cycle activity was designed as a 20-minute computer-based learning
module with built in formative and summative assessments (see appendix for link and
examples). Built using Adobe captivate, the water cycle module contains interactive
text, video, images, animations, as well as both formative assessments as the learner
progresses and a comprehensive summative assessment upon completion of the
learning module. Using different media in the learning module was purposely created
so as to attract different learning styles.
This paper focuses on the results of an effectiveness test given to random
individuals in a completely online setting and a usability test given to 15 fifth grade
students, ages 11-12, on May 8th, 2015.

Methodology
Activity
As described in the introduction, the water cycle activity was designed to teach
fifth grade students about the water cycle as part of regular grade level curriculum in the
state of California. Participants in this effectiveness test were asked to take a pretest
prior to embarking on the activity, in order to determine prior knowledge of the subject

Water Cycle Report: Wilcox

material. Upon completion of the water cycle activity, participants were asked to
complete a post test in order to draw conclusions about the effectiveness of the learning
module and whether or not the participants gained new knowledge by completing the
water cycle learning module.
Learners
The intended learners for the effectiveness test were originally limited to grade
level students, ranging in age from 10 to 12 years of age. This effectiveness test was
conducted during the summer recess of 2015. Initial participants were recruited using
social media, however the response level was insufficient to conduct a statistical
analysis. The search for participants was expanded to a Craigslist forum in which the
researcher requested anonymous participation in the activity for the purpose of data
collection. This Craigslist post was placed in all major metropolitan areas for California
(see appendix). The majority of the posts were flagged and removed, consequently
there was no data collection from the Craigslist post. The search for participants was
then expanded to an email appeal, which was sent to the parents of former students of
the researcher for this report. Again, there was insufficient response to conduct
statistical analysis. Last, students in the MIST cohort 10 program were asked to
participate. This final appeal, coupled with previous attempts at recruiting participants,
eventually yielded a total sample size of seven individuals, age range unknown.
The original intention of the effectiveness test was to determine if grade level
students gained new knowledge after participating in the water cycle activity. Since the
participants did not necessarily fall into the age level category, this portion of the
assessment had to be abandoned, and the assessment became an overall

Water Cycle Report: Wilcox

effectiveness test for all participants, regardless of age level. Given this outcome, much
of the demographic questions at the end of the posttest were abandoned, but the
usability questions remained and were analyzed separately from the data used to
measure the effectiveness of the water cycle activity.
In addition to the usability questions on the posttest provided for this report, there
are results from a previous usability test conducted during the 2014/15 school year with
grade level students. The results from this usability test will be discussed further, along
with the results of the effectiveness test.
Assessment
In order to obtain meaningful results, the pre and posttest for the water cycle
activity were identical, with the exception of the final questions on the posttest, which
requested responses as to the usability of the learning module. Only the responses to
the first eleven questions on each assessment were used in the data analysis. The
results of the individual scores on the pretest and posttest were statistically analyzed to
determine if the participants were able to increase their score after completing the water
cycle activity. Two t-tests were completed based on the results, one test with a 95%
confidence interval and another with a 90% confidence interval.
In addition to the pretest/posttest results, the tests were analyzed to determine
which of the questions were missed most, as well as which were answered correctly
most often. The purpose of this analysis was to outline potential weaknesses in the
water cycle learning module, and to draw conclusions about the wording of specific
questions. The response frequency for each question on both the pretest and the
posttest are listed in the next section of this report.

Water Cycle Report: Wilcox

Results
Entry
As previously stated, the initial and intended entry conditions were set up for fifth
grade elementary school children with an age range between 10 -12 years. The
pretest, water cycle activity, and posttest were conducted asynchronously online,
without observation. Participants were made aware of the effectiveness assessment in
an online format, therefore it is concluded that the participants were familiar with
navigating through a computer simulation.
The observed entry conditions were different than the intended entry conditions
in that the age range and grade level of the participants did not fall into the intended
category. As a result, the age level of the participants in the interpretation section of this
report had to be abandoned and the data was analyzed strictly for pretest and posttest
results.

Instruction
Instruction for the effectiveness test was provided on the web link for the activity
(see appendix figures 2 and 3). The participant was to click on the link that would take
them to the pretest (appendix figures 4, 5, and 6) and then return to the web page with
the water cycle activity. Upon completion of the water cycle activity, the participants
returned to the home page to complete the posttest. The website with the activity and
assessments were designed so each page would open in a new tab, the purpose of

Water Cycle Report: Wilcox

which to facilitate navigation back and forth between the home website and the
individual activity the participant was engaged in at any given time.

Data
The data for the effectiveness test was collected using a Google form in which
the responses for each question for each respondent were recorded onto a
spreadsheet. Two separate spreadsheets were used, one for the pretest and one for
the posttest. These responses were uploaded to one excel spreadsheet and then
statistically analyzed. The first analysis conducted was a t-test paired two sample for
means. This test was done at the 95% confidence level. The research hypothesis for
this test was to determine that participants scores had improved on a posttest taken
after completing the learning activity on the water cycle (H 1.>H0). The null hypothesis in
the analysis would determine that there is not statistical proof that the participants had
gained an improved score on the posttest (H1=H0). The results of the t-test for the 95%
confidence level is listed below in figure 1.

95% confidence

Water Cycle Report: Wilcox

Fig. 1

Given that the absolute value of the t statistic (1.84) in the one-tailed t-test is lower than
the critical value of 1.94, and given that the probability of the alternative hypothesis is
true is less than the confidence value of 95%, the null hypothesis for this test was not
rejected. At the 95% confidence level, it was not concluded that the participants fared
better on the posttest after completing the water cycle activity.
Another t-test was conducted using the same null and alternative hypotheses at
the 90% confidence level. The results are displayed below in figure 2.
90% Confidence:

Fig. 2

Water Cycle Report: Wilcox

The t-test conducted at the 90% confidence level yielded different results than
the t-test conducted at the 95% confidence level. The t-statistic for the 90% confidence
level yielded an absolute value of 2.295, with a critical value of 1.439. Since the tstatistic at the 90% confidence level is greater than the critical value, and given that the
probability that the research hypothesis is true is greater than the probability of the 90%
confidence level, the alternative hypothesis cannot be rejected. At the 90% confidence
level, the data shows that there was a correlation between participating in the water
cycle activity and performing better on the posttest.
When comparing the mean scores of the pretest and posttest, the data shows
improvement in the overall score (see fig. 3). However, when the overall mean scores
are analyzed at the 95% confidence interval that there is a direct correlation between
the water cycle activity and an increase in the test scores, it cannot be concluded that
the increase in the score had to do with the water cycle activity. Yet if the analyst is
willing to accept a 90% confidence interval, then the data does appear to show that
there is a direct correlation between the water cycle activity and in improvement in test
scores. Thus, when mean scores alone are used to determine the effectiveness of the
activity, it is possible to misinterpret the results, depending on the level of confidence
the researcher is willing to accept.

Water Cycle Report: Wilcox

10

80.00%
70.00%
60.00%
50.00%
40.00%
30.00%
20.00%
10.00%
0.00%
pretest
post test

Fig. 3

Results of Pre/Post Tests

120.00%
100.00%
80.00%
60.00%
40.00%
20.00%
0.00%

3
pretest

post test
Fig. 4

Figure 4 above shows the pretest and posttest results for each of the seven
participants. Overall test scores improved in five of the seven participants, one
participant scored lower on the posttest than on the pretest, and one participant scored
the same. An interpretation of each question and how each participant individually
responded follows.

Water Cycle Report: Wilcox

11

Interpretation of questions
The chart in figure 5 shows the individual question results for each response in
both the pretest and the posttest. The results were organized to check for questions
that each participant may have missed on both tests. The purpose of this analysis was
to use the information to review both the wording of the questions, as well as the section
of the water cycle activity that refers to the material that was tested in order to evaluate
and check for clarity. The data collected in the figure below was then compared to the
usability test results obtained on May 8th, 2015. The usability test was completed by
grade level participants immediately following a classroom session in which each
student completed the same water cycle activity. Based on the results of the usability
test, response patterns to the pretest and posttest, as well as the responses to the
functionality questions asked in the posttest, this report will draw conclusions as to the
usability and effectiveness of the water cycle activity.

Fig. 5

Water Cycle Report: Wilcox

12

Question Response Freqency:

120.00%
100.00%
80.00%

Pretest

60.00%

Posttest

40.00%
20.00%
0.00%
1

10

11

Fig. 6

Usability Test Results

The usability test of the water cycle activity (appendix figures 7-10) was
conducted on May 8, 2015 in a regular classroom setting with fifth grade students from
Fort Bragg, CA. The ages of the students averaged 11 years at the time the test was
conducted. Students were each provided with a Dell Latitude E6410 school-issued
laptop computer, equipped with wifi. The classroom was equipped with a wifi booster to
facilitate internet speed.
Students were instructed to log onto the website where they would find the water
cycle activity. They were asked to complete the activity in its entirety, and then
complete a usability survey. The questions on the survey focused on functionality and
navigability of the water cycle activity. It was the goal of the usability survey to compile
meaningful information about how easily the students were able to actually go through
the activity without finding functional glitches that could hamper learning and potentially
increase their frustration level.
The information that was obtained in the usability test showed that there were
some functional issues with the water cycle activity that didnt allow participants to
navigate through certain sections easily. Examples of responses are as follows:

Water Cycle Report: Wilcox

13

Multiple voices speaking at one time.

Drag and drop words didnt drop in the correct location.
Undo button on sea breeze activity only works one time.
Had to wait a long time before the next slide played.

In contrast to the functionality issue examples above, of the 15 respondents, 11

found no issues with functionality. Ratings for the questions that required a score
are as follows: (Rating scale is 1-5, 5 the most favorable, 1 the most unfavorable).

Question 1: The water cycle activity was easy to use on my computer

o Overall score: 4.4
Question 2: The activity didn't work as it should have. There were technical

problems.
o Overall score: 4
Question 4: In the Water Cycle activity, I found that the different pieces of

information were ordered in a way that made sense.

o Overall score: 2
Question 5: I couldn't figure out how to advance to the next slide in the

activity.
o Overall score: 1.3
Question 7: The volume was at the appropriate level
o Overall score: 3.1

For questions 1 and 2, the relative scores create a contradiction in that question
1 asks the students to rate how easy it was to navigate through the activity, and
question 2 asks students if they found technical problems. Both scores are relatively
high, which would indicate that the learners found the navigation easy, but discovered
many technical issues. Further research into this will be required before an adequate
conclusion may be drawn, but it may be considered that students found technical issues
that did not cause navigability problems. In addition, the research may also suggest
that the wording of some of the questions on the usability test might have been
confusing for the intended audience. It should also be taken into account that the

Water Cycle Report: Wilcox

14

usability test was provided during a 45 minute block of time in a regular classroom
setting, where students were each provided with their own laptop computer with
headphones and wifi connectivity. Given that the water cycle activity was designed to
be completed in 20 minutes or less, all of the students in the usability test were unable
to complete the entire activity in the 45 minute time frame.
Outcomes
The intended outcome of the effectiveness test would have demonstrated that at
the 95% confidence level, learners would have demonstrated conclusive improvement
upon completion of the water cycle activity. The data did not show this to be true, so it
cannot be concluded with 95% confidence that the water cycle activity provided
sufficient instruction for learners to have improved their scores on the posttest.
However, when analyzed for 90% confidence, the results did show an improvement in
scores.
In addition to the t-test results, the interpretation of the usability test must be
taken into account because there is anecdotal information provided by a separate
sample of participants in which there may have been usability errors in the water cycle
activity which could have caused lower posttest scores if the participants were unable to
access some of the content during the activity.
Recommendations
Based upon the conclusions of both the effectiveness test and the usability test,
the design of the water cycle activity should be improved upon so as to achieve the
desired results that the statistical analysis demonstrated to be missing. The activity

Water Cycle Report: Wilcox

15

should be pared down, or broken up into two to three different learning modules, each
with their own assessment in order to achieve better test results.
In addition, given the high percentage of participants in the effectiveness test
who missed questions 9 and 11, the wording of the two questions must be taken into
account in addition to the content in the water cycle activity in which the questions refer
to. The high percentage of participants who missed these two questions both on the
pretest and the posttest indicate two or more issues; namely the section of the water
cycle activity that these questions referred to were confusing or not accessible due to
technical glitches, or the questions themselves were worded poorly. Further review of
the section in the water cycle activity that refers to sea and land breezes should occur
during the redesign, in addition to clarification of the test questions.

Summary and Conclusions

When analyzing the pretest and posttest data for the mean test scores alone, it is
possible to construe that the water cycle activity helped the participants to improve their
scores. Empirically, there is data to suggest that this is the case. However, when the
data is analyzed at the 95% confidence level, there does not appear to be sufficient
evidence to conclude that learning took place. Based on this evidence alone, further
improvements must be made to the water cycle activity, and/or the pretest and posttest
must be further analyzed for their wording. Taking the information from the
effectiveness test alone does not provide sufficient information to make specific
changes that would be effective enough to demonstrate improvements on posttest
scores in the future. Therefore, based upon empirical and anecdotal information

Water Cycle Report: Wilcox

16

provided in the usability test conducted on May 8, 2015, further conclusions may be
drawn with which to improve upon the water cycle activity.
Further conclusions and recommendations to improve the water cycle activity:

Break the water cycle activity into two to three different learning activities.
o Organizing the different learning objectives into separate activities will
reduce learner fatigue and confusion.
Review the water cycle activity for technical glitches.
o A sufficient number of participants expressed an inability to navigate
through the activity based on technical issues.
o Technical glitches do not measure learning.
Review pretest/posttest questions for confusing verbiage.
o Do not create confusing questions based on information in the water cycle
activity that is difficult to master.

Appendix
Water Cycle Activity Access Point:
http://itcdland.csumb.edu/~csargent/ist526/C_Sargent_IST526_FinalProject/html5/Earth
's%20Water%20Cycle%20IST526%20Final%20Project%20v5_final
%20html5/WaterCyclev5.html

Appendix figure 1:

Water Cycle Report: Wilcox

Appendix figure 2:

17

Water Cycle Report: Wilcox

Appendix figure 3:

18

Water Cycle Report: Wilcox

Appendix figure 4:

Appendix figure 5:

19

Water Cycle Report: Wilcox

Appendix figure 6:

Appendix figure 7:

20

Water Cycle Report: Wilcox

Appendix figure 8:

Appendix figure 9:

21

Water Cycle Report: Wilcox

Appendix figure 10:

22