ETEC 500 65A Assignment 2A: Final Submission William Tam

April 11th, 2020

Research Analysis and Critique
I Get to Use an iPod in School? Using Technology Based Advance Organizers to
Support the Academic Success of English Learners

Purpose of the Study

The purpose was clearly stated by Billings and Mathison. This study was to look
at the effect of Technology-Based Advance Organizers (TBAOs), in forms of individual
handheld mobile devices, on the academic performance of two English-only
assessments and the level of engagement of 240 fourth graders in a science class at
School in the Park (SITP), a museum school collaboration (Billings and Mathison 2012).

The Connection with Existing Literature

Schools have become more diverse ethnically and linguistically. The numbers of
English Learners (ELs) are rapidly increasing, with some states in America experiencing
more than 400% growth in ELs since 1998 (National Clearinghouse for English
Language Acquisition (NCELA) 2001). According to the latest Nation's Report Card
released by the National Center for Education Statistics, it illustrates that the
achievement gap between minority students and their white peers remains large,
especially amongst Hispanic students (Hemphill and Vanneman 2010). Language ability
especially played an important role in this achievement gap.
In Science, where complex academic language is demanded, ELs are faced with
challenges when expected to simultaneously process large amount of scientific
information—verbal and written, while trying to comprehend basic instructions in English
(Lee and Fradd 1996; Moje et al. 2001) This difficulty to understand limits their
comprehension and motivation. Thus, this theoretical framework of the study is
supported by learner choice (e.g. providing primary language support in content areas)
and learning access (e.g. using an educational tool such as advance organizers to
support Els’ comprehension and retention).

This research aimed to investigate the effect of TBAOs, such as handheld mobile
devices (HMDs) displayed in the choice of learner’s native language or English, or by
DVDs in only English, on the academic performance on English-only assessments and
student’s level of engagement in hands-on activities. The justification of the need for this
study was clear and compelling. It includes findings from previous researches that were
related to the purpose of this study. This study applies to my practice as a school
teacher in a public-school district that is dominated by English Language Learners.

Constructs

The most significant constructs of this study were the academic performance, or
mean scores on English-only assessments and the construct of motivation in hands-on
activities of the learners. The operational definition of “performance” used the mean
scores to depict how well the students performed on the Starburst and Mineral
assessment, including levels of engagement, on a scale of 10 and 18 respectively.

1
ETEC 500 65A Assignment 2A: Final Submission William Tam
April 11th, 2020
The Research Approach, Variables Used

This research was a mixed-method of both quantitative and qualitative aspects.

Scores and statistics were collected from pre- and post-tests and Starburst and Mineral
Sort Activities (quantitative), and focus group interviews were given at the end of the
study (qualitative). This research was a quasi-experimental in the fact that two
treatments or experimental groups (HMDs and DVD) were formed, to compare with a
control group of participants (no treatment). The participants in the groups were not
randomly assigned to groups.

This study was problem-based as the researchers were concerned about the
achievement gap between native English speakers and English learners, and wanted to
find out whether students’ academic performance and engagement would be affected
when TBAOs (intervention) were applied. This study was also inferential, evidenced by
the level of significance, permitting a conclusion about the larger population.

The independent variables used were HMDs and DVDs (treatment groups), and
no TBAOs (the control group). The dependent variables used were the scores on the
performance-based assessments, including levels of engagement (Starburst activity
and Mineral sort activity) out of 10 and 18 points, respectively. Also, there was a pre-
test and a posttest of 10 multiple choice questions on key concepts and comprehension
of vocabulary. There were no attribute variables.

Research Design

The primary research design is best described as a mixed-method, quasi-

experimental pretest posttest design.

Control Design

Matching was applied as a control design in this study to avoid some extraneous
variables. All three groups (HMDs, DVDs, and control) were exposed to the same
instructions and versions of the assessments at STIP. However, blinding may have
been used but was not mentioned. This would fail to control rater bias. In addition, even
though English Learners were divided more or less equally amongst the three groups,
selection bias was presented as the participants were not randomly assigned. Thus, the
validity of extrapolating the result to a larger population is weakened. The researchers
also used multiple regression to assess the effect of TBAOs by controlling the influence
of some extreme individual ability differences). Confounding was noticed when students
in the HMD group were able to choose between Spanish and English on their podcasts.
Doing so confounded the usage of TBAOs with language choice.

Sampling

The sampling was purposeful to the research question as the researchers

wanted to investigate the effect of TBAOs on learning, particularly of English Learners.

2
ETEC 500 65A Assignment 2A: Final Submission William Tam
April 11th, 2020
The participants were chosen from an inner-city neighborhood in San Diego’s Balboa
Park (Billings and Mathison 2012). However, details of the sampling method or of the
participants (language proficient levels, racial backgrounds) were not described fully.
Informed consent was also not addressed in this case.

The sample of the study includes 240 fourth-grade students among which were
156 Limited English Proficient (LEP)—students whose primary language is not English.
There were 120 students in the control group (divided among 4 subgroups), and a total
of 120 students in the treatment groups (60 students in the HMD group, and 60 students
in the DVD group.) These numbers fit within the rule of thumb of sampling. Note that
only 229 students were assessed in the second mineral sorting activity, indicating a
4.5% of mortality rate. This loss of subjects is considered minor, and should not
represent a serious bias.

Reliability and Validity

The reliability of measures, particularly the Starburst activity and mineral sort
activity assessments, were not addressed.

There are many flaws that threatens the internal validity of the results. One of the
questions that is raised is how do we know whether the positive effect is due to the
treatment of TBAOs, or other extraneous events? The researchers failed to address
how they attempted to neutralize this unwanted influence in the study. Students could
have purchased a self-study guide, or taken lessons outside of the class, hence the
improvement in scores. Some may have been more interested in certain scientific topics
that affected the engagement level in the activities.
Hawthorne effect also may have influenced the effect as participants with TBAOs may
be motivated by the novelty of having a technological device or being able to watch
DVDs before and after class.

Alternative Hypotheses

One possible alternative hypothesis is that the frequency of viewing on

instructional materials may explain the treatment effect. That is, if participants could
watch the instructional materials more often, then they would do better on their
assessments. However, this was not further explained and supported by evidence in the
current study. It was mentioned as a plan for further investigation, though.

Another alternative hypothesis is that learning language (English or native

language) on TBAOs could have an effect on student’s academic success and
engagement level, rather than the mode of learning. Again, researchers did not study
more in-depth to neutralize this possible explanation.

Data Analysis

3
ETEC 500 65A Assignment 2A: Final Submission William Tam
April 11th, 2020
The study used a two-way analysis of variance to determine the effect of using a
TBAO for both ELs and fluent English proficient students (FEP) on two performance
measures. A repeated measures analysis was also used to investigate the effect
between the treatment and control groups on the pretest and posttest.

They noticed that there was a significant main effect (p<0.05) where students in
the HMD group scored significantly higher than students in the DVD or control group.
While there was no main effect for language status (ELs vs FEPs) as p>0.05. There
was a significant interaction effect which indicated the effect of language status was
greater in the HMD group as English Learners scored higher in the HMD, but not in the
DVD or control group. Effect was not apparent between the groups in the pre- and
posttest scores. There was also a significant interaction effect between group
assignment, language status, and total score of both assessments, indicating that ELs
performed better in the HMD group.

As for student engagement, there was no main nor interaction effect between
TBAO type and language status. In contrast, there was a main effect between TBAO
type and engagement level in individual and group work, where HMD group scored
significantly higher than the DVD and the control groups.

Furthermore, student focus groups provided more evidence that the intervention
of TBAO helped the students become “more efficient and experienced learners” by
developing schema, concept reviews, and vocabulary reinforcement. Students in the
HMD group demonstrated high levels of knowledge retention and increased excitement
about the learning the material. Better focus and higher level of motivation of students in
the HMD group were revealed by interviews with the SITP director and classroom
teachers.

Descriptive Clarity

The study was clear in the sense that information about the participants was
informed (they were chosen for their English proficiency). However, the questions of the
focus group interviews were not revealed in the study, giving rise to biases (question-
order bias and researcher bias).

Major Findings, Implications, and Key Conclusions

Billings and Mathison concluded that there was a significantly positive effect of
TBAOs when viewed on an HMD on both academic and engagement indicators: HMD
group (M = 7.7, SD = 2.2), DVD (M = 6.5, SD = 2.7), control group ( M = 6.4, SD = 2.7);
p<0.05. This effect was even more significant, as shown in Table 3, for English
Learners, who scored higher than Fluent English proficient participants when using
HMDs to view the advance organizer podcasts (Billings and Mathison 2012).These
findings were surprising that while the advance organizers were helpful, the type of
TBAO is equally important.

4
ETEC 500 65A Assignment 2A: Final Submission William Tam
April 11th, 2020
The strength of the study lies in the fact that it had a clear purpose in context and
that the study was succinct and easy to read. However, it holds several limitations which
suggests further research and clarification are needed. First, language choice (Spanish
or English) must be expanded in the format of TBAOs for students who are not
proficient in either one. Also, the researchers were unable to definitively identify why
students excelled significantly via only the mode of HMDs.

More questions and issues are asked of whether other devices, such as iPads or
social media, would offer the same success as HMDs? Does frequency of viewing on
HMDs affect the result, hence challenging the validity of the findings? Whether student
choice around the choice of language on TBAOs would impact student’s academic
performance and level of engagement?

While this quasi-experimental study holds a lot of promises in its mixed method
investigations on the relationship between using TBAOs and academic performances
and engagement, there are still many unknowns and voids that need to be clarified and
researched further in order to improve the internal validity and eliminate extraneous
influences.

References:
Billings, E., & Mathison, C. (2012). I Get to Use an iPod in School? Using Technology
Based Advance Organizers to Support the Academic Success of English
Learners. Journal of Science Education and Technology, 21(4), 494-503.
Retrieved April 4, 2020, from www.jstor.org/stable/41674477

Hemphill FC, Vanneman A (2010) Achievement gaps: how Hispanic and White students
in public schools perform in mathematics and reading on the National
Assessment of Educational Progress (NCES 2011-459). National Center for
Education Statistics, Institute of Education Sciences, US Department of
Education. Washington, DC. Retrieved 14 July 2011 from http://nces.ed.
gov/pubsearch/pubsinfo.asp?pubid=20 1 1 459

Lee O, Fradd SH (1996) Interactional patterns of linguistically diverse students and

teachers: insights for promoting science learning. Linguist Educ 8(3):269-297

Moje EB, Collazo T, Carrillo R, Marx RW (2001) "Maestro, what is 'quality'?": language,
literacy, and discourse in project-based science. J Res Sci Teach 38(4):469-498

National Clearinghouse for English Language Acquisition (NCELA) (2011) The growing
numbers of English learner students 1998/99-2008/09. Retrieved 21 Oct 2011
from http://www. ncela.gwu.edu/files/uploads/9/growingLEP

5
ETEC 500 65A Assignment 2A: Final Submission William Tam
April 11th, 2020

Reflection:

I have learned so much after reading others’ analyses as they provided me with
examples to follow for future examinations of research studies. As for the feedback I
received from my peers, it was helpful to improve my draft. Lots of them focused on
improving the quality of my writing, namely grammar and word choices. The pieces of
advice were valuable as it reminded me that it’s important to proofread my own work a
few more times before submitting. Regarding the feedback to improve the quality of my
critique, a comment was suggested for further elaboration which I believe will make my
critique more insightful. Thank you for providing me this opportunity to reflect and learn
more, Kisha.