This action might not be possible to undo. Are you sure you want to continue?
MIT 77 Massachusetts Ave., 5-122 +1-617-253-8604
Glenda S. Stump
MIT 77 Massachusetts Ave., 5-122 +1-617-253-0602
The virtual classrooms of open online courses include students from a vast array of individual, social, economic, and educational contexts. Detailed data were collected for the first course MIT ran on the edX platform, including student behavior, performance, and background information. In this paper, we estimate the systematic differences in average performance, distribution of performance, and performance conditional on behaviors for countries with different characteristics (e.g., language, income). K.3.1 [COMPUTERS AND EDUCATION]: Computer Uses in Education – Distance learning.
lump all students together or compare national systems as a whole6. Global indicators further show that the availability and use of computers in schools differ drastically between countries7. In addition, studies within the US, Brazil, and other countries note that computer use, access, and utility vary significantly by SES8,9. Numerous studies have also demonstrated that there is a strong correlation between national income (as measured, for instance, by GDP per capita) and overall educational achievement10,11.
Categories and Subject Descriptors General Terms
Measurement, Performance, Economics.
MOOCs, comparative education Massive Open Online Courses (MOOCs) are offered as avenues of access for students around the world. They are touted in particular as opportunities for students in countries with lowperforming education systems1,2. However, researchers and policymakers who work with developing countries have criticized the capacity for MOOCs to deliver learning opportunities. They note that MOOCs presume a stable internet connection and bandwidth to access video links3, that they cater to students who already have the wherewithal to succeed in formal educational institutions4, and that the perspective of developing country leaders is lacking5. In this paper, we investigate country-level differences in performance and the distribution of achievement for those students who participated in the first edX course.
Our sample comprises students who participated in the first edX MOOC, “Circuits and Electronics,” which was offered in the spring of 2012. The data we analyze come from the clickstream logs of students’ interactions with the site. From these, we estimate the country from which students access the site as well as the time of each interaction and the website component accessed. We observe students from 194 countries with a diversity of backgrounds and prior experiences12; in nearly every country, we see overwhelming heterogeneity in behaviors and performance. We demonstrate possible systematic differences between students from different countries in the entire 6.002x population, and we provide a focused look at eleven countries of interest: USA, UK, Canada, Colombia, Brazil, Mexico, India, Pakistan, Spain, France, and Greece. These countries all demonstrate high numbers of participating students. In addition, they represent diverse national educational systems, linguistic and cultural spaces, and different engineering education and online learning usage profiles. We provide results from these 11 countries individually and in four groups: (1) English-speaking high-income countries, in this case with notable providers of MOOCs or online universities; (2) middle-income countries in Latin America; (3) low-middle income countries in South Asia; and (4) non-Englishspeaking high-income countries in Western Europe.
2. CONCEPTUAL FRAMEWORK
We address the issue of national variation in performance by asking the question, “Are there significant differences in the overall performance of students from different countries?” More specifically, we ask: (a) Are there significant differences in mean performance for all students, students who attempted assessments, and certificate earners?; (b) Are there significant differences in the range and distribution of performance for these students?; (c) and, controlling for students’ behaviors, what national characteristics are significant? We ask these questions for the whole sample but then focus our attention on four groups of countries with high participation—English-speaking high-income countries, Hispanoand Lusophone middle-income countries, low-middle income countries in South Asia, and non-English high-income countries. We draw on literature from numerous disciplines for our work. Comparative education research emphasizes that, even in internationally standardized assessments, vastly different school cultures, pedagogies, and structural factors inhibit inferences that
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. LAK '14, Mar 24-28 2014, Indianapolis, IN, USA ACM 978-1-4503-2664-3/14/03. http://dx.doi.org/10.1145/2567574.2567602
We first estimate the overall variation in achievement that is attributable to group membership using a two-level randomeffects model, where a student’s group (country) is assigned by the country from which s/he accessed the site most frequently. We then conduct ANOVA of student achievement and proportion of certificate earners by region and by the four country groups of interest. Next, we apply a robust test of the equality of variances to investigate the distribution of performance and completion overall and between our four country groups of interest, which we illustrate graphically. Last, we show that differences persist in a predictive model; policymakers should note systematic differences in the profiles of students in varied contexts.
5. MEAN DIFFERENCES
We confirm that there is a great deal of heterogeneity within each national group of students in 6.002x. We calculate the intraclass correlation (ICC, given in Table 1) for the full set of students, the full set of students who have scored more than 0 points, and those students in our 11 countries of interest. Table 1. Variance in achievement attributable to group Population All students All scoring students N 108,008 29,639 Percentage of variance shared within country (ICC) 2.65% 4.28%
All students in sample 73,583 1.96% All scoring in sample 20,361 2.52% Despite this heterogeneity within countries, we also observe significant differences in overall achievement and in the proportion of students who receive certificates between countries, whether or not we exclude students who do not score any points (F = 5.52, p <0.001 for all countries; F = 90.58, p<0.001 for 11 countries of interest). The significant differences in mean performance and spread of performance are notable in the patterns we can see emerging from variability between the four country groups of interest given in Table 2. Table 2. Differences in mean, distribution of achievement Country group (1) (2) (3) (4) N 37,573 12,690 17,352 5,968 Mean grade (of 100) 6.07 7.29 7.51 13.84 SD 19.99 21.36 21.47 29.84 Skew 3.67 3.23 3.22 2.03 Kurt -osis 15.25 12.19 12.26 5.42 % cert. (of total N) 5.41% 6.29% 6.05% 13.74%
Table 3. Select coefficients from multi-level model Predictor GDP per capita English-speak. country Internet penetration Hours on homework (ex.) FE coeff. 0.00 -4.89 -0.05 2.39 Std. Err. 0.00 1.76 0.19 0.22 p 0.32 0.01 0.62 <0.01 Rand. coeff. stand. dev. NA NA NA 0.50
6. DISTRIBUTIONAL DIFFERENCES 6.1 Differences in Distributional Profile
Policymakers who look to MOOCs as a tool for increasing access must understand that the students who access these courses in each respective context are diverse. National contexts may suggest factors such as language as important, but we must understand that even signing up for the course may preclude less English-proficient students from participating. Providing initial information about the course in other languages and other earlier interventions may be more promising in supporting diverse students to learn through MOOCs. MOOC providers, policymakers, and researchers should identify pathways to reach students who would not otherwise be able to access this course for MOOCs to truly increase access to educational opportunities.
Funding was provided by NSF Grant, DRL-1258448. We also acknowledge the support of MIT’s RELATE group and edX.
As Table 2 shows, there appear to be systematic differences between country groups in the mean, variance, skew, and kurtosis of achievement in 6.002x as well as in the performance of those students who score any points and those who earn a certificate. We use a robust test of the equality of variances (Levene’s test) and confirm that this systematic difference is statistically significant (W0=661.98, p<0.001 for all students in the 11 countries of interest; W0 = 149.51, p<0.001 for scoring students). Figure 1 shows the notable differences in profiles for these four groups, whether we look at scoring students or certificate earners.
 Chakrabarti, R. & Ghosh, A. (2013). Are MOOCs the Answer?. The Hindu, August 4, 2013.  Koller, D. (2012). Daphne Koller: What we're learning from online education. TED Talks, August 2012.  Mitchell, A. (2013). The Underlying Inequality of MOOCs. eLearning Africa, August 27, 2013.  Malesic, J. (2013). A Catholic Case Against MOOCs. The Chronicle of Higher Education, September 16, 2013.  Trucano, M. (2013). Missing Perspectives on MOOCs-Views from developing countries. EduTech Blog, April 19, 2013.  Rotberg, I. (2006). Assessment around the world. Educational Leadership, 64(3): 58-63.  World Economic Forum. (2008). Global Competitiveness Report. Geneva, Switzerland.  Vigdor, J.L. & Ladd, H.F. (2010). Scaling the Digital Divide: Home Computer Technology and Student Achievement. NBER Working Paper No. 16078.  Wainer et al. (2008). Too much computer and Internet use is bad for your grades, especially if you are young and poor: Results from the 2001 Brazilian SAEB  Gamoran, A. & Long, D.A. (2008). Equality of educational opportunity: A 40 year perspective. In Education and equity volume 1: international perspectives on theory and policy, edited by Richard Teese, Stephen Lamb, and Marie DuruBellats, chapter 1. New York: Springer Press, 2008.  Chudgar, A. & Luschei, T.F. (2009). National income, income inequality and the importance of schools: A hierarchical cross-national comparison. American Education Research Journal, 46(3): 626-658.  DeBoer, J., Stump, G.S., Seaton, D., and Breslow, L. (2013). Diversity in MOOC Students’ Backgrounds and Behaviors in Relationship to Performance in 6.002x. LINC’13.
6.2 Predictive Model
Finally, we estimate a multi-level model wherein students are nested in countries. We include country characteristics such as GDP per capita and a dummy variable for the country being predominately English-speaking, and we control for individual behaviors such as time spent on online homework problems or lecture videos. Achievement is largely related to individual behaviors. However, contrary to established comparative research10, overall achievement has no significant relation with GDP per capita. Instead, higher achievement is related, counterintuitively, to being in a non-English speaking country. This result persists even when high-income, high-scoring OECD countries such as France are excluded. This suggests that the profile of students in English-speaking countries such as group (1) are more diverse, while those in non-English speaking countries group (2) come from an already-high achieving group of students who know about and can engage with a college-level class entirely in English. Further, students from group (4) show a larger, flatter spread of scores with a notably higher mean.
Figure 1. Overall performance of scoring students and certificate earners by country group
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue listening from where you left off, or restart the preview.