You are on page 1of 10

Go Boundless?

: A comparison of efficiency, efficacy, and user experience of Boundless versus traditional e-textbooks
Zachary A. Rosner, Ph.D. and Nancy Tsai, M.Ed.

Please address correspondence to: Healy Jones (press@boundless.com) Boundless 207 South St. Boston, MA 02111

Introduction
Boundless is an online educational platform that curates open-source resources to create alternatives to textbooks assigned to undergraduate students. In addition, these textbook alternatives are supplemented with study aids, including summaries of key concepts, highlighting tools, interactive flashcards, and quizzes. Given the fact that Boundless is lower-cost (free) and replaces any major psychology textbook, Boundless could be the superior study option, if it is as good or better than traditional textbooks. A survey of metacognitive learning strategies found that college students often re-read textbooks rather than employ active learning processes, despite the limited benefit of this strategy (Karpicke et al., 2009). The study aids offered by Boundless provide a direct way for students to actively engage with the information they need to learn, consolidating the information for later recollection. For example, the quizzes allow for self-testing, known to be one of the most effective learning strategies (Roediger III & Karpicke, 2006a; Agarwal et al., 2008), especially for scientific text (Karpicke & Blunt, 2011). In addition, the flash cards may help students determine when they have sufficiently learned the information (Roediger & Karpicke, 2006b). Not only does Boundless offer these invaluable learning tools, the content within Boundless is presented in a manner that may take less time to read and understand than traditional textbook content. This preliminary study compared the efficacy of studying with Boundless to studying with a typical paid eTextbook for Introductory Psychology. Participants studied psychology content on the Boundless platform or within the eBook Psychology (Schacter et al., 2012). This study was conducted entirely online. After a short retention interval, participants took a multiple-choice test that assessed their memory for the previously studied information. Specifically, this study assessed efficiency, effectiveness, and user experience. Efficiency was measured by the duration of time each participant studied the psychology material before feeling sufficiently prepared to take a test. Effectiveness was defined as long-term memory retention, measured by performance on that multiple-choice test. Responses to Likert-scale and open-ended questions gauged user experience.

Go Boundless?: A comparison of efficiency, efficacy, and user experience of Boundless versus traditional e-textbooks by Zachary A. Rosner, Ph.D. and Nancy Tsai, M.Ed.

Section I: Methods
Participants

Forty undergraduate students at American universities participated in this study. An online crowdsourcing site, Amazon MechanicalTurk, was employed to recruit all participants. An initial demographic survey screened hundreds of subjects for age, undergraduate status, and cognitive effort. Qualified participants were then selected and invited to participate in the study via Qualtrics.

Study Material

The experiment utilized a between-subjects design. All participants learned about topics from the foundations of psychology, reading information from either Boundless or Psychology. Topics included functionalism, William James, Mary Calkins, and Margaret Washburn. Regardless of condition, all participants took the same test, which consisted of 10 multiple-choice questions.

Go Boundless?: A comparison of efficiency, efficacy, and user experience of Boundless versus traditional e-textbooks by Zachary A. Rosner, Ph.D. and Nancy Tsai, M.Ed.

Section II: Procedure


The study consisted of a study phase, followed by a retention interval, ending in a test phase.

Study Phase
Participants were assigned to one of two conditions - Boundless or Paid eTextBook. This study material was the only aspect of the experiment that differed among participants, as they all then received identical instructions, survey questions and test questions. After registration, participants were given an overview of the study. They were told that they would read information from an introductory psychology book regarding the foundations of psychology. As guidance, participants were provided with a study aid specifying 4 key topics on which they would be subsequently tested. Participants were instructed to study carefully, yet quickly, and to move past the study phase only when they felt they learned the material sufficiently enough take the test (recommended study time no greater than 20-30 minutes). Depending on the assigned condition, subjects were directed to an external site that displayed the study material. No influence was provided as to whether participants should or should not use available study aids specific to the learning platforms. We recorded the amount of time students took to study to examine learning efficiency.

Retention Interval

We presented a short 2-minute demographic survey after the study phase as a distractor task to prevent rehearsal and clear short-term memory, eliminating any recency effects.

Test Phase
To assess retention of the study information, participants in all conditions performed a 4-answer forced choice recognition task (multiple choice test) consisting of 10 questions. Questions were designed to be unbiased and easily answerable regardless of study condition.

Post-test Survey

Participants in both conditions were presented a series of questions regarding their experiences using their assigned platforms and other study preferences. Questions were presented via 7-point Likert scale as well as open-ended formats, gauging ease of navigation, ease of comprehension, clarity, enjoyment, and elements of each condition participants found most and least useful.

Demographic Information
Additionally, participants submitted demographic information, including age, sex, year in school, major, GPA, university affiliation, native language, and school participation and enjoyment.

Go Boundless?: A comparison of efficiency, efficacy, and user experience of Boundless versus traditional e-textbooks by Zachary A. Rosner, Ph.D. and Nancy Tsai, M.Ed.

Section III: Results


Students in the Boundless condition required less time to study material

Participants in the Boundless condition spent significantly less time studying than did students in the Paid eTextbook condition (t(38) = 2.17, p = 0.04). Students in the Boundless condition took an average of 7.6 minutes to study material, whereas students in Paid eTextbook condition took an average of 13.1 minutes (Figure 1). This indicates that it took less time for students in the Boundless condition than the paid eTextbook condition to locate important content and effectively learn information, thereby demonstrating more efficient encoding. What is most notable about the Boundless platform is that the most relevant academic material is condensed into a good short read. Presenting content in a more concise, tothe-point manner allows students to more quickly go through study material, spending more time learning and less time searching. Students in the paid eTextbook condition suggested that a search function would greatly improve the use of that platform as it would help them locate key information. The greatest value of Boundless is the presentation of condensed content.

Figure 1.

Go Boundless?: A comparison of efficiency, efficacy, and user experience of Boundless versus traditional e-textbooks by Zachary A. Rosner, Ph.D. and Nancy Tsai, M.Ed.

Students in the Boundless condition demonstrated superior memory retention


Students using the Boundless platform answered more questions correctly during the post-study test than did students using the Paid eTextbook (t(38) = 2.69, p = 0.01), demonstrating superior retention of the study material. Students in the Boundless condition accurately answered 80% of the recognition test questions, whereas students in the Paid eTextbook answered 66% correctly (Figure 1).

According to the overwhelmingly positive feedback regarding the summary and bullet points presented on the Boundless platform, it seems likely that this organization helps students focus on the most relevant information thereby allowing them to spend their time studying key information. Students in the Boundless condition liked that the main key elements were pointed out at the very beginning of each page. It seems that the key features of the Boundless platform, namely: search, bullet points, summary, and concise text, all help students study more effectively.

Students in the Boundless condition found Boundless more userfriendly


As predicted, compared with the Paid eTextbook,, users rated Boundless as: having clearer study information; being easier to read; being easier to locate information; being more enjoyable to study. As shown in Figure 2 (next page), on a scale from 1-7, Boundless received a significantly higher average rating for all user experience factors measured. Boundless was rated higher for clarity of information, ease of reading, ease of locating pertinent information, and overall study enjoyment (t(38) = 3.84, p < 0.001; t(38) = 5.18, p < 0.001; t(38) = 3.24, p < 0.01; t(38) = 3.32, p < 0.01). Clarity of information Boundless: 5.1 Paid eTextbook: 3.4 A junior from Florida State University liked how straightforward all of the information was. [Boundless] was clear and easy to understand. Ease of reading Boundless: 5.2 Paid eTextbook: 2.75 Locating pertinent information Boundless: 5 Paid eTextbook: 3.35 When students were asked what the best aspect was about the way in which the study materials were presented, 12 out of 20 students voluntarily noted the utility of the bullet points explaining, I liked the bullet points at the top; After I read the paragraph, I felt the bullets helped to summarize the information for me and listed all the most important facts.
Go Boundless?: A comparison of efficiency, efficacy, and user experience of Boundless versus traditional e-textbooks by Zachary A. Rosner, Ph.D. and Nancy Tsai, M.Ed.

Enjoyment while studying Boundless: 4.4 Paid eTextbook: 2.6 A junior from Southwestern Oklahoma State University liked the ability to use flashcards or take a quiz.

Figure 2.

Go Boundless?: A comparison of efficiency, efficacy, and user experience of Boundless versus traditional e-textbooks by Zachary A. Rosner, Ph.D. and Nancy Tsai, M.Ed.

Section IV: Conclusions and Future Directions


This study examined the effectiveness of Boundless as compared to a typical paid eTextbook for Introductory Psychology. When learning with Boundless, participants spent less time studying (Figure 1), achieved a higher score on a subsequent memory test (Figure 1), and rated Boundless as: clearer; easier to read; easier to locate information; and more enjoyable to study (Figure 2). These findings indicate that studying with Boundless results in superior encoding efficiency, memory retention, and user experience. These results are likely due to several aspects of the Boundless platform. First, the information is presented more clearly and concisely, allowing participants both to more easily read the relevant information, and also to read all information in fewer words. Next, the organization helps students locate information more readily, and the structure likely helps to consolidate the encoded information. Finally, the study aids such as flashcards and quizzes provide a strong mnemonic benefit beyond simply reading the text, and also help the students assess their own understanding of the material. In total, these characteristics of Boundless lead to a superior learning experience. While all differences were significant, some issues should be addressed from this initial study to conclusively demonstrate the superiority of Boundless and satisfy the rigor of an academic audience. Future studies should increase power. Options include increasing the sample size from 20 to 100 participants and increasing the breadth and depth of study information and test question. These strategies would reduce variance and increase effect size. In addition, a within-subjects rather than between-subjects design should be employed to reduce any random biases that may be inherent in the study groups. Finally, the study information for the exact same Boundless and Paid eTextbook versions should be compared. The currently presented study, along with a potential study that addresses the formerly mentioned issues, would truly demonstrate the superiority of Boundless to other study platforms, and give students and professors the evidence, and confidence, needed to use Boundless in the classroom.

Go Boundless?: A comparison of efficiency, efficacy, and user experience of Boundless versus traditional e-textbooks by Zachary A. Rosner, Ph.D. and Nancy Tsai, M.Ed.

References
Agarwal, P. K., Karpicke, J. D., Kang, S. H. K., Roediger III, H. L., & McDermott, K. B. (2008). Examining the testing effect with open- and closed-book tests. Applied Cognitive Psychology, 22(7), 861876. Karpicke, J. D., & Blunt, J. R. (2011). Retrieval practice produces more learning than elaborative studying with concept mapping. Science, 331(6018), 772775. Karpicke, J. D., Butler, A. C., & Roediger III, H. L. (2009). Metacognitive strategies in student learning: Do students practise retrieval when they study on their own? Memory, 17(4), 471479. Roediger III, H. L., & Karpicke, J. D. (2006a). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249255. Roediger III, H. L., & Karpicke, J. D. (2006b). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1(3), 181 210.

Go Boundless?: A comparison of efficiency, efficacy, and user experience of Boundless versus traditional e-textbooks by Zachary A. Rosner, Ph.D. and Nancy Tsai, M.Ed.

About the Authors


Dr. Zachary Alexander Rosner earned his Ph.D. at U.C. Berkeley while studying the cognitive neuroscience of human learning and memory. His research investigates effective learning strategies from cognitive, cross-cultural, and neuroimaging perspectives. With over eight years of teaching experience, he brings his passion in the science of human learning into the classroom and continues to seek ways to apply his research in meaningful, educationally relevant ways. Nancy Tsai received her Ed.M in Mind, Brain, and Education from Harvard University and B.A. in Cognitive Neuroscience from U.C. Berkeley. Her work aims to illuminate the inner workings of human learning on the behavioral, cognitive, and neurological level. With over five years of cognitive neuroscience and psychology research, she will continue her studies of human learning through doctoral research fall 2013. Her work is applied to the development of innovative, interactive, and effective learning tools and has been published in Journal of Neuroscience.

Go Boundless?: A comparison of efficiency, efficacy, and user experience of Boundless versus traditional e-textbooks by Zachary A. Rosner, Ph.D. and Nancy Tsai, M.Ed.

10