Small to Big Before Massive: Scaling up Participatory Learning Analytics

Daniel T. Hickey
Indiana University Bloomington IN, 47401 01-812-856-2344

Tara Alana Kelley

Indiana University Bloomington IN, 47401 01-239-233-5035

Xinyi Shen
Beijing Normal University Beijing 100875, China 86-159-0100-1713

dthickey@indiana.edu ABSTRACT

tmcilmoi@indiana.edu

shenxinyi1128@gmail.com

2. RESEARCH CONTEXT
This particular approach to personalized learning is rooted in prior design-based studies of educational multimedia [14, 16], educational videogames [1] and English language instruction [13]. This prior research resulted in a general set of design principles and local theories [4] for enacting them. Starring in 2008, these ideas were used to design and refine two online graduate level courses: Learning & Cognition in Schools and Assessment in Education. Both courses served students with very diverse experiences & ambitions; both courses brought with them detailed expectations for disciplinary content coverage. These refinements resulted in a set of participatory design principles for fostering productive forms of individual and social engagement in disciplinary knowledge [5] while also consistently impacting individual understanding (as assessed with classroom performance assessments) and aggregated achievement (as measured with conventional tests). The courses were organized around wikifolios that every learner could see and comment on. Both courses were organized around texts that were challenging for less-experienced learners. The first author taught each course online at least once per year using the Sakai CMS. These features and analytics (a) were reasonably efficient with up to 30 students, (b) supported extensive levels of individual and shared disciplinary engagement, (c) generated enduring understanding of targeted course concepts, and (d) resulted in significant and substantial gains in student achievement [15]. The Assessment course was scaled up to a 12-week open course using Google Course Builder (Version 1.4) starting September 2013. The assignments, interactions, assessments and analytics were all revised to be manageable with up to 500 participants. Ultimately, 460 registered for the course, including 8 who also enrolled for official university credit and agreed to complete all optional assignments. Some participants held doctoral degrees in education while others had no prior coursework in education. In addition to preparing for the BOOC, more scaling took place across the BOOC as the features and analytics were streamlined and/or automated. Some streamlining involved making more efficient use of the instructors time as the weekly analytics and feedback were handed off to the teaching assistant (TA, the 2nd author) and in some cases to an intern (the 3rd author). Other streamlining involved moving from the cumbersome manual examination of wikifolio pages to downloading spreadsheets to automated algorithms. Other scaling (not discussed here) is continuing as entire features are redesigned (and sometimes reconceptualized) to function more autonomously. Generally speaking, the courses were designed and refined in order to align immediate real-time feedback as individual wikifolios were

This case study describes how course features and individual & social learning analytics were scaled up to support participatory learning. An existing online course was turned into a big open online course (BOOC) offered to hundreds. Compared to typical open courses, relatively high levels of persistence, individual & social engagement, and achievement were obtained. These results suggest that innovative learning analytics might best be scaled (a) incrementally, (b) using design-based research methods, (c) focusing on engagement in consequential & contextual knowledge, (d) using emerging situative assessment theories.

Categories and Subject Descriptors

K.3.1 [Computers in Education]: Computer Uses in Education collaborative learning, distance learning

General Terms
Algorithms, Measurement, Performance, Design.

Keywords
Personalized learning, learning analytics, assessment, social learning analysis, analytic approaches, analytic approaches.

1. INTRODUCTION AND CONTEXT

This paper introduces an approach to personalized learning [22] that is rooted in situative views of cognition [3, 7], connectivist views of learning [21], and participatory views of online culture [18]. The features and learning analytics that define this approach were first theorized and then refined informally and intuitively in smaller online courses. They were then scaled up to function more systematically and automatically in a big open online course (BOOC). Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org.
LAK '14, March 24 - 28 2014, Indianapolis, IN, USA ACM 978-1-4503-2664-3/14/03$15.00. http://dx.doi.org/10.1145/2567574.2567626

posted, close feedback across the entire set of wikifolios, proximal reflections on wikifolios, and distal assessment of achievement and post-hoc examinations of engagement.

to the next revealed an average change of 140 words, with spikes at the transition to Part Two and to Part Three

3.2 Assign Networking Groups

The formation of affinity groups around common interests is a crucial aspect of participatory learning [6]. Interaction within and between networking groups helps learners see how the disciplinary knowledge of the course takes on different meaning in different contexts. In the small course, students were intuitively assigned to groups using the information they provided in the first assignment. Reassignments and group mergers were handled informally and case-by-case. In the BOOC, assignment to a larger number of larger groups was done systematically by downloading the information provided at registration to a spreadsheet. Participants were then assigned to the networking groups by adding identifiers to their names and using that information to sort the display of names on the participant page.

3. LEARNING FEATURES & ANALYTICS

Thirteen personalized learning features and analytics were scaled. Each is described as it was first refined informally and intuitively in the small course. The paper then describes how each was scaled up in the BOOC and summarizes the associated engagement.

3.1 Define Personalized Learning Contexts.

Situative perspectives on cognition suggest that the context in which learning occurs is equally as important as the disciplinary knowledge that is learned [7, 10]. Thus a core participatory design principle is that learners should define a personalized context to anchor disciplinary learning. Throughout the learning experience, learners should revisit and refine their characterization of their context, co-constructing knowledge of their experience alongside the disciplinary knowledge of the course. A more specific principle focuses directly on aspects of disciplinary knowledge that are contextual (take meaning from contexts of use) and consequential (have consequences from practice); a related assumption is that the discourse that follows readily will generate factual, procedural, and conceptual knowledge. In the Assessment courses, learners first define a curricular aim that represents a particular domain and their actual or aspirational role in the educational system (teacher, administrator, researcher, etc.). Ideally, personalization should reveal bottlenecks [19] that impede learning. In assessment, this is understanding the distinction between practices of teaching and the processes of learning. The public individual feedback (described below) provides contextualized guidance that is ideal for helping newcomers appreciate this seemingly-nuanced distinction. In the small class, students defined their personalized learning contexts in their first graded wikifolio assignment. After introducing themselves on their wikifolio home page, they added a subpage where they defined their curricular aim and their educational role, interests, and goals. They were instructed to refer to a relevant section in the text, the instructors example, and samples from previous semesters, and to refer to their classmates examples and instructor feedback once the first few were posted. BOOC participants defined their personalized contexts in two stages. First, registrants were instructed to draft an initial curricular aim and describe their role while registering. This introduced the personalized approach from the outset, and presumably discouraged registrants who were not serious about taking the course and/or disliked the approach. While 631 completed a pre-reregistration form that only required an email address, 460 completed the registration, and 358 completed the initial curricular aim. Eight participants paid full tuition for the course in order to earn three graduate credits and were required to complete all parts of all assignments. Registrants aim and role were automatically inserted into the wikifolio for the first assignment, which had them elaborate on the aim and role, describe their goals, and locate a relevant educational standard. Participants were encouraged to examine others work and individual and social feedback (described below) was provided. Each wikifolio assignment instructed participants to use their expanding knowledge of assessment to further refine the description of their context and curricular aim. To emphasize this aspect of learning, they were encouraged to return to this introductory wikifolio element after completing the other parts. Post-hoc analysis of the change of this element from one wikifolio

3.3. Identify Secondary/Emergent Groups

As participatory learning unfolds, new cross-cutting groups should emerge. In the small class, this was supported informally as students got to know each other, and primarily concerned academic domains. Thus, mathematics teachers might seek out administrators and researchers with interest with mathematics. In the BOOC, the third assignment invited participants to revisit their profiles and extend their name to add self-defined identifiers to afford secondary groups and project more distinct identities. Fiftynine percent of those who posted the third wikifolio did so. Most extensions were disciplinary, and some resulted in new crosscutting groups (e.g., mathematics) that fostered additional affinity. A few non-disciplinary extensions (e.g., unemployed) seemed to foster additional affinity without distracting.

3.4 Post Course Artifacts Publically

Meaningful artifacts are central to participatory learning [18]. Artifacts that are public and persistent play a crucial role in the interest-driven social networks that provide much of the inspiration for participatory learning [17]. This is the logic behind the wikifolio assignments introduced above. Students in the small class viewed a streaming video on making wikifolios, accessed guidelines for each, and posted a single continuous page. In the BOOC, guidelines were contained in section headers that could be shown or hidden in edit and view modes. In edit mode a WYSIWYG text-editing window appeared in each section. This design helped frame the wikifolios as stand-alone documents (rather than class assignments). One new feature allowed BOOC participants to choose to Save as Draft or Post for Review. Another new feature posted a link next to each participants name at the course homepage. This simplified the process of locating classmates whose wikifolios were ready for review and comments, and facilitated some of the other analytics.

3.5. Rank Relative Relevance

The prior research confirmed that having learners rank targeted elements of disciplinary knowledge in terms of relevance to their particular context was a productive course feature. The contextual and consequential interaction and discourse that it affords lends itself nicely to disciplinary interaction among students and instructors [15]. For example, sometimes students appeared to rank an idea as least-relevant because they did not understand its relationship to their context. This was an inviting context for others to point to unexplored relevance.

In the small course, students were instructed to summarize the ideas in order of relevance and include rationale for the ordering. In the BOOC, participants rearranged text boxes containing summaries of the ideas (and sometimes links to additional information) and provided a rationale in the text box below. This simplified the process and generated data for social feedback.

Significantly, 33% of comments made reference to context (either the wiki author or the commenter and sometimes both). Wikifolios received an average of 2.9 comments and the average number was relatively constant across the course. Post hoc analysis of one third of the wikifolios revealed that 65% included an initial question and 55% of those questions got a response. Coding of comments was streamlined by a subsystem that output comment text and context, allowing faster examination of the impact of new or modified assignment features on engagement.

3.6. Accessing Personalized Content.

Consistent with connectivist views of learning [4], learners locate new course content that is relevant to their personalized context. In the small online course, some assignments instructed students to (a) use search engines and a classwiki to search for outside resources, (b) include links to those resources in their wikifolio, and (c) add annotated links into the classwiki. Informal analytics organized the classwiki, helped struggling students locate resources, and provided individual and social feedback. This was all done informally and intuitively. This support was laborious and limited by the instructors available time. In the BOOC, this activity was introduced gradually. The assignments in the first part of the course were completed using the well-structured guidelines in the textbook. The assignments in the second part of the course had participants search for relevant resources and post links to their wikifolio. Assignments in the third part had participants rank the relative relevance of a range of provided resources and then search for new resources to share with their peers. These new resources were automatically compiled on a page where they could be readily located.

3.9 Instructor and Peer Endorsement

Artifact-oriented education calls for some form of accountability for the artifact quality and associated interactions. Yet, even with a highly structured rubric, it would be overwhelming to formally grade such extensive artifacts and interactions. More importantly, doing so is likely to undermine the disciplinary engagement that occurs when creating and discussing artifacts. In the small course, a core innovation emerged around the notion of consequential engagement [9]. Each wikifolio assignment included three reflections to be completed a week after the original deadline (presumably before beginning the next wikifolio). These included consequential engagement (what will you do differently in your context as a consequence of learning this knowledge?), critical engagement (how suitable was your context for learning this knowledge?), and collaborative engagement (who elses work and whose comments helped you learn this knowledge?). Each wikifolio was awarded full points (5 out of 100) as long as a complete draft was posted on time and the reflections were coherent. Analyses of reflections in the small class (and other contexts) confirmed that (a) it was difficult for students to write coherent reflections without having (or getting) a reasonable understanding of the targeted concepts and (b) the public nature of the artifacts discouraged students from gaming the system by reflecting on incomplete assignments [15]. This artifact accountability strategy was too laborious for the BOOC. In response, the reflections were deemed optional and artifact accountability was accomplished via peer endorsement. Once a wikifolio is posted, peers (but not the author) can click a button to endorse it as Complete (required elements) or Complete (including optional elements). Participants were instructed to endorse at least three wikifolios as part of their peer interaction. There was no limit to the number of endorsements and the names of each endorser were displayed on the wikifolio. Participants whose wikifolios were complete but unendorsed were invited to request TA endorsement if necessary. The BOOC peer accountability strategy for artifacts was remarkably successful. Wikifolios earned an average of 5.1 endorsements, increasing from 3.6 in the first week to 5.7 on the last week. Formal review of all endorsed wikifolios revealed that just two percent were incomplete. The exception was the instruction to post a question to peers, which perhaps one third of the wikifolios lacked.

3.7. Public Individualized Feedback

Most interaction in these courses occurs publically and persistently within threaded comments at the bottom of each wikifolio. In the small course, the LMS notified the instructor that the initial wikifolios are posted; the instructor immediately provided relatively extensive feedback. This included more nuanced concepts that were too advanced to be included in the assignment (where they would overwhelm the less experienced students). The instructor immediately sent a message to the class thanking those students for posting early and encouraging other students to examine the examples and the feedback. While this practice was time-sensitive and time-consuming for the instructor, comments from students around that feedback and in the collaboration reflections (described below) confirmed that students read and used the early feedback. A similar strategy was employed in the BOOC, except that the TA helped identify the best early posters from the larger number that posted on the first day, and helped draft the more routine feedback, allowing the instructor to focus on inserting the advanced ideas and opinions.

3.8. Peer Commenting and Discussion

While artifact comments anchor interaction to contexts, strategies are needed to elevate illustrative interactions involving a few learners (and instructors and TAs) so that they are apparent to all learners. In the small class, each assignment instructed students to post at least one question about their own wikifolio as a comment to their peers and the instructor. Students were also instructed to comment on three or more classmates wikifolios. In addition to the early feedback described above, the instructor was able to informally track most of the conversation and highlight the most productive interactions in weekly summaries. In the most recent class, wikifolios attracted an average of six comments. Coding revealed that 89% of the comments concerned educational assessment and that 78% concerned the weeks assignment.

3.10. Peer Promotion

Another core participatory design principle is that disciplinary engagement should be recognized and rewarded, but that should occur outside any formal evaluation or accountability system. This is because the summative function of formal evaluation undermines engagement by encouraging students to game the system and the community. Students in the small course were instructed to post one (and only one) stamp of approval each week via a comment starting with a distinctive string (&&&) that

warranted their claim that the artifact, exchange, or comment was exemplary. The wikifolio homepage indicated which student earned the most such stamps each week. In the BOOC, participants simply clicked a box to promote a wikifolios, which then asked them to add a warrant. The names of promoters and their warrants were then displayed on the wikifolio. Participant in each group with most promotions each week were acknowledged in the weekly feedback. The group members with the most promotions for each of the three units and for the entire course were awarded leader versions of the digital badges describe below. Participation in peer promotion averaged 67%, increasing from 51% in the first week to 77% in in the last week.

3.11. Weekly Individual & Social Feedback

As detailed above, much of the interaction each week considered the relative relevance of that weeks ideas from the specific perspective of individual peers and the more general perspective of different groups. Doing so prepared both individuals and groups to learn more by seeing the eventual rankings of relevance of those ideas for the various networking groups and revisiting exemplary considerations of those relationships. For example, when learning about validity, that fact the teachers, administrators, and researchers respectively found content validity, criterion validity, and construct validity most relevant clearly reified important (but initially nuanced) distinctions that learners first encountered during the assignment. In the small class, the instructor drafted a weekly summary while reviewing and commenting on student assignments. This information and links to good examples were then posted as an announcement. The course evaluations confirmed that students liked this feedback and enjoyed seeing themselves and classmates recognized. But the process was time-sensitive and time-consuming for the instructor. Some students complained that the feedback arrived too late and that the process was biased to reward early posters. In the BOOC, all of the relevance rankings were accessed via a spreadsheet, which supported additional tagging, coding, and counting; examples were selected starting with the wikifolios that received the most promotions. While still requiring approximately five hours per week, this was entirely managed by the TA.

Because of the workload, the open-ended items were dropped in the BOOC, which included three 20-item unit exams and a 30item comprehensive final. While item-level feedback was not provided, participants were allowed to take the exams twice in three hours and take the final twice in four hours. Participants were required to complete the exams and final to earn digital badges, but the original 80% criteria was relaxed. Participants were able to choose whether they included their exam performance on their digital badges, and exam scores were factored into final course grades for the for-credit students. Average scores for non-credit participants were 84%, 76%, 78%, and 75%, while average scores for the for-credit students were 88%, 83%, 78%, and 82%.

3.13. Web-Enabled Digital Badges

In contrast to conventional credentials, open digital badges can contain specific claims about learning and achievement, along with detailed evidence (and links to additional evidence) supporting those claims. Open badges can be readily accumulated in external backpacks and shared over social networks. In the most recent small class, the instructor experimented with awarding open badges in order to prepare for the BOOC. The badges were issued directly to the Mozilla backpack and summarized the assignments the student had completed and indicated that they had passed the exam. In the BOOC, badges were automatically generated and could contain details regarding the assignments completed, number of comments, endorsements, promotions, and exam performance. Significantly earners were able to select which evidence to be included, and whether to include links to their completed wikifolios (minus peer comments for privacy). Participants could earn one badges for each of three course parts; participants who earned all three and completed the final exam were issued the Assessment Expert badge. A Leader version of each of the four badges was issued to the participant in each group who earned the most promotions; a customizable Expert badge which displayed the earners self-define area of assessment expertise was offered to participants who turned their weekly wikifolios into a comprehensive paper. In the end of course survey, 41% of completers reported sharing their badges, mostly via Facebook and/or email. While seven of the participants submitted a paper, only four of the papers met the criteria for the custom badge.

3.12 Appropriate Accountability

A situative focus on productive forms of disciplinary engagement reframes traditional notions of assessment [6, 8, 11, 13]. This treats classroom assessments as peculiar forms of disciplinary discourse that are primarily useful for improving curriculum and of little value for directly advancing knowledge; standardsoriented achievement tests become bizarre discourse that is primarily useful for evaluating complete courses and seeing broad improvement over time, but nearly useless for evaluating individuals or directly advancing individual learning. The small class took timed exams with both open-ended items and multiple-choice items. The open-ended items were curriculumoriented; each asked students to consider the relevance of a randomly selected chapter implication to the small class context in five minutes; the multiple choice items were randomly selected from the textbook item bank and students were given two minutes per item. Item-level feedback was only provided for the openended items and the midterm and final exams were worth just 30 of 100 points. Average scores have averaged around 90%. Individual exam scores typically mirrored engagement and were generally where weaker students ended up with lower grades.

4. CONCLUSIONS
In light of the low completion and engagement rates reported for university-sponsored MOOCs [2], the BOOC appeared quite promising. Of the 160 participants who completed the first assignment, 60 (37%) completed the course. In contrast to the declining engagement among completers in most MOOC, engagement levels across the BOOC units was stable (comments) or increased (endorsements and promotions). BOOC completers reported working an average of 7.5 hours per week, with a high of 30 hours. Open-ended comments were overwhelmingly positive; one participant who completed his MS and PhD at top-ranked universities deemed the BOOC best graduate-level education course he had ever taken. Reassuringly, there were virtually no meltdowns like those at one widely cited failure to foster more participatory personalized learning at a massive scale [25]. The success of this course supports four suggestions for scaling up participatory learning. The first suggestion is that scaling should be done incrementally. With around 100 participants completing weekly assignments, it is possible to gradually streamline the learning analytics by handing them off to a TA and making use

spreadsheets and the other resources. With thousands of learners, this useful iterative refinement and the insights that followed would have been impossible. While this brief paper does not allow elaboration, efforts are already underway to further streamline most BOOC features and analytics to allow their use in automated and/or massive contexts. The second suggestion is that efforts to scale should consider design-based research methods [4]. The specific effort described here was part of a larger ongoing effort to refine and exemplify a set of more general participatory design principles. In translating these more general principles into specific features and analytics, more specific design principles are emerging alongside knowledge of the contextual factors that help make them possible. Together, these general principles, specific principles, specific features, and contextual factors offer useful knowledge for broader effort and the efforts of others The third suggestion is that the design principles used to scale participatory learning embrace emerging situative theories of assessment. [8, 11, 12] This provides a coherent framework for aligning informal assessment during activities, semi-formal assessment in reflections, and formal assessment in exams. The final suggestion is that the design and refinements of interactive features and analytics focus on contextual and consequential knowledge. Doing so appears to be a scalable way to foster the natural forms of disciplinary engagement [5] that that can also foster learning of the factual, procedural, and conceptual knowledge that courses are typically accountable for.

[8]

[9] [10]

[11]

[12]

[13]

[14]

5. ACKNOWLEDGEMENTS
This research was supported by a gift from Google to Indiana University. Garrett Poortinga and Thomas Smith contributed directly to many of the features and learning analytics described in this paper. Rebecca Itow contributed to key aspects of the design research and the writing of this manuscript and Retno Hendryanti supported the instruction described here. [15]

6. REFERENCES
[1] Barab, S, Zuiker, S., Warren, S, Hickey, D., IngramGoble, A, Kwon, E., Kouper, I, & Herring, S., 2007. Situationally embodied curriculum: Relating formalisms and contexts. Science Education, 91, 750-782. Brinton, G. C., Chiang, M., Jain, S., Lam, H, Liu, Z., & Wong, F. M. F. (2013). Learning about social learning in MOOCs: From a statistical analysis to a generative model. Cornell. http://arxiv.org/abs/1312.2159. Brown, J. S., & Adler, R. P., 2006.. Open education, the long tail, and learning 2.0. EDUCAUSE Review, 43,1, 16-20. Cobb, P., Confrey, J., Lehrer, R., & Schauble, L., 2003. Design experiments in educational research. Educational researcher, 32, 1, 9-13. Engle, R. A., & Conant, F. R..2002. Guiding principles for fostering productive disciplinary engagement: Explaining an emergent argument in a community of learners classroom. Cognition and Instruction, 20 3, 399-483. Gee, J. P., 2004. Situated language and learning: A critique of traditional schooling. New York: Psychology Press. Greeno, J. G., 1998 The situativity of knowing, learning, and research." American Psychologist 53, 1, 526

[16]

[17] [18] [19]

[2]

[3] [4] [5]

[20] [21] [22]

[6] [7]

Greeno, J. G., and Gresalfi, M. S., 2008. Opportunities to learn in practice and identity." In Moss, P. (Ed) Assessment, equity, and opportunity to learn. New York: Teachers College Press: 170-199. Gresalfi, M., et al., 2009 Virtual worlds, conceptual understanding, and me: Designing for consequential engagement. On the Horizon 17, 1, 21-34. Hickey, D. T., 2003. Engaged participation vs. marginal non-participation: A stridently sociocultural model of achievement motivation. Elementary School Journal, 103, 4, 401-429. Hickey, D. T.. 2012. A gentle critique of formative assessment and a participatory alternative. In P. Noyce & D. T. Hickey (Eds.), New frontiers in formative assessment (pp. 207-222). Cambridge, MA: Harvard Education Press. Hickey, D. T., Ingram-Goble, A., and Jameson, E., 2007. Designing assessments and assessing designs in virtual educational environments. Journal of Science Education Technology, 18, 187-208. Hickey, D. T., McWilliams, J. T., & Honeyford, M. A., 2011. Reading Moby-Dick in a participatory culture: Organizing assessment for engagement in new media. Journal of Educ. Computing Research, 44 (4), 247-273 Hickey, D. T., Tassoobshirazi, G., Cross, D., 2012. Assessment as learning. Enhancing discourse, understanding, and achievement in innovative science curricula. Journal of Research in Science Teaching, 49, 1240-1270. Hickey, D. T., & Rehak, A., 2013. Wikifolios and participatory assessment for engagement, understanding, and achievement in online courses. Journal of Educational Media and Hypermedia, 22, 4, 229-263. Hickey, D. T., & Zuiker, S. J., 2012. Multi-level assessment for discourse, understanding, and achievement in innovative learning contexts. The Journal of the Learning Sciences, 22, 4, 1-65. Ito, M., et al., 2010 Hanging out, messing around, and geeking out: Kids living and learning with new media. Cambridge MA: MIT Press. Jenkins, H., 2009. Confronting the challenges of participatory culture: Media education for the 21st century. Cambridge MA: The MIT Press. Middendorf, J, and Pace, D. 2004. Decoding the disciplines: A model for helping participants learn disciplinary ways of thinking. New Directions for Teaching and Learning, 98, 1-12. Oremus, W. 2013 Online class on how to teach online goes laughably awry. Slate (February 5, 2013), 1. Siemens, G., 2005. Connectivism: A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning, 2, 1, 3-10. Woolf, B. P. 2010. A roadmap for education technology. Computing Research Association, Washington, DC.