See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/315468590

Task Assignment of Peer Grading in MOOCs

Conference Paper  in  Lecture Notes in Computer Science · March 2017

DOI: 10.1007/978-3-319-55705-2_28

CITATION READS

1 270

3 authors:

Yong Han Wenjun Wu

Hangzhou Dianzi University Beihang University (BUAA)
10 PUBLICATIONS   25 CITATIONS    147 PUBLICATIONS   1,422 CITATIONS   

SEE PROFILE SEE PROFILE

Yanjun Pu
Beihang University (BUAA)
12 PUBLICATIONS   14 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

the National Key Research and Development Program of China View project

intelligent microservice governance View project

All content following this page was uploaded by Yong Han on 10 July 2022.

The user has requested enhancement of the downloaded file.

 Task Assignment of Peer Grading in MOOCs

Yong Han, Wenjun Wu, Yanjun Pu

State Key Laboratory of Software Development Environment,
School of Computer Science, Beihang University, China
{hanyong, wwj, puyanjun}@nlsde.buaa.edu.cn

Abstract. In a massive online course with hundreds of thousands of students, it

is unfeasible to provide an accurate and fast evaluation for each submission.
Currently the researchers have proposed the algorithms called peer grading for
the richly-structured assignments. These algorithms can deliver fairly accurate
evaluations through aggregation of peer grading results, but not improve the
effectiveness of allocating submissions. Allocating submissions to peers is an
important step before the process of peer grading. In this paper, being inspired
from the Longest Processing Time (LPT) algorithm that is often used in the
parallel system, we propose a Modified Longest Processing Time (MLPT), which
can improve the allocation efficiency. The dataset used in this paper consists of
two parts, one part is collected from our MOOCs platform, and the other one is
manually generated as the simulation dataset. We have shown the experimental
results to validate the effectiveness of MLPT based on the two type datasets.
Keywords: Task Assignment, Crowdsourcing, LPT, peer grading

1 Introduction

Currently MOOCs learning is becoming a popular way to acquire knowledge besides

the traditional classrooms. A typical MOOC course contains videos, exercises, online
test, peer grading and forums. Such an online course often has thousands of active
learners in every study session [9]. When students submit their homework to the course,
there are a lot of submissions of each assignment for course staffs to grade. Given the
scale of the submissions, if the staffs have to examine all the submissions, they will be
overwhelmed by the grading workload so that they can’t have sufficient time and
energy to focus on other things that will be helpful for students.
Researchers have proposed the methods of peer grading which are similar to crowd
sourcing. In the process of peer grading, students are asked to grade their submissions.
Therefore, the students are both submitters and the graders, which allows us to exploit
this unique feature to solve this peer grading problem. In general, there are two basic
problems in peer grading: (1) one is how to assign the submissions to the graders
considering the difference of knowledge level between graders, (2) the other is how to
aggregate to peer grades to generate a final fair score for each student. In this paper we
mainly focus on the first problem.
In most MOOCs platforms, the systems adopt a simple random assignment approach.
Their peer grading servers record the submitted time of an incoming submission and
store it in queue. Then the distribution mechanism chooses an idle grader randomly and
assign it to the grader without considering the grader’s knowledge level. Such a random
assignment approach often results in biased grading scores for students because the
assignment plan may put graders with a high level of knowledge mastery in the same
group to evaluate the same homework assignment. Previous research papers have
shown that the reliability of every grader is related to his knowledge skills. The students
with good knowledge mastery tend to deliver reliable and accurate grading scores.
Therefore, in the uneven distribution plan, some submissions receive very accurate
evaluations while the others receive with very inaccurate evaluations. The problem can
be avoided if the peer grading server makes grading task allocation with the awareness
of the knowledge level of each participating student. The task allocation algorithm
should be to distribute students with different knowledge levels evenly to grading
groups so that the difference among the average knowledge level in each group could
be minimized.
Considering the unique characteristic of peer grading, we choose the method
Longest Process Time(LPT) [5] as a heuristic algorithm for grading task assignment.
The algorithm is used in parallel systems to deal with task scheduling. It is a static task
allocation strategy, so it requires all the tasks are assigned before the processors begin
to deal with the tasks. Assigning tasks in peer grading has its unique characteristics
compared to other task scheduling problems. A homework submission in the
submission set may be assigned to its author if the distribution mechanism is not
restricted and this is not allowed in practice. Furthermore, one submission should be
graded by several graders and conversely one grader should also grade the same number
of submissions.
Based on the algorithm LPT, we propose our algorithm Modified LPT (MLPT) to
solve the allocating submissions in peer grading. Most researchers on peer grading
focus on developing statistical models to infer accurate scores based on peer grading
results. Few work consider the impact of the peer grading task allocation. Our main
contribution is the introduction of student knowledge mastery into the design of task
allocation algorithm. By developing the student performance evaluation model, our
peer grading system can accurately estimate the student knowledge level. And based
on the, we redefine the LPT algorithm to adapt the peer grading process to the
distribution of student knowledge level.
The rest of the paper is organized as follows: Section 2 compares our work with
other related efforts. Section 3 presents the overview of our peer grading framework
and describes the MLPT algorithm in detail. Section 4 and Section 5 presents our
experimental datasets and analysis results. Discussion and Future work is given in
Section 6.

2 Related Work

In this paper, before allocating tasks, we detailed analysis the student online
performance of learning for accurately estimating the knowledge level of student. We
have referred to the thought task assignment in Crowdsourcing and proposed an
effective method by compare the previous models. We first review some existing
researches for allocating tasks. Allocating tasks is a long historical issue and appears in
many fields such as distributing conference papers or in the parallel systems and so on.
It is also one of the core problems of the crowdsourcing.
In the mode of Crowdsourcing, a task is decomposed into multiple small tasks [8]
which is assigned to massive online workers. Because of the existing of human error
and fraudulent workers, the confident of the collection results which typically contains
a lot of noise is usually very low [2, 3]. In order to ensure the reliability of the results,
the measure of redundant allocation is typically taken, namely, assigning the same task
to multiple workers, and aggregating the multiple results to generate a final one. Here
the task allocation is to find a balance between the reliability of the results and the
redundancy of the task allocation.
Umair and Edward [15] proposed a task allocation solution based on the human
ability. This method first models the abilities of a worker according to historical data.
Similarly, it models the types of workers’ abilities that the task required. Then through
adaptation, the algorithm allocates the task to the most suitable workers thus obtaining
the optimal results with the minimal redundancy. Finally, they adjust the worker model
by comparing the completing results to the aggregating result.
Hyun and Matthew [16] had solved the problem of data sparsity by decomposing
matrix based on the research of Umair and Edward, increasing the universality of the
method. In addition to model the abilities of the workers based on the historical data,
other researchers also considered the demographic factors, education background, and
especially the social information to determine the level of the workers’ abilities which
can allocate the task to the worker accurately.
Furthermore, because the most Crowdsourcing tasks are paid tasks, the capital
budget is constraint [4], David, Sewoong and Devavrat [17] inspired by the algorithm
of propagation of confidence and the algorithm of low rank matrix approximation
designed the minimum cost allocation strategy. The results in the strategy make the cost
of the spending the least if it meets certain prerequisites of reliability.
Yan Yan et al. [18] proposed an online adaptive task assignment algorithm by
ensuring the confidence of the results. The algorithm allocates the subsequent
redundancy and the workers according to the required of the task by analyzing the
completion of the task time to time and the confidence of the results to make the final
results highly reliability.
Piech et al. [6] exploited the confidence estimated to determine what point in the
grading process is confident about each submission’s score. They simulated grading
taking place in rounds. For each round they ran model using the corresponding subset
of grades and counted the number of submissions for which they were over 90%
confident that they predicted grades were within 10pp of the students’ true grade.
In our work, there is no need to classify the students according to the types. The
primary information about students is their historical learning data, thus we exploit
some models compute the student knowledge level based on this data. By extracting
student features, we using the method of logistic regression to fit the features and this
part accounts for the proportion of our work. We also define a hyper-parameter as the
redundancy value, which requires us to constantly experiment.
We have done experiments over different redundancy values and different number
of students. The results can be seen in section 5. Our another attempt is using the
examination scores as the knowledge level, but it does not generate a good effect, the
reasons may be that the score is a monotonous factor.
Fig. 1. Peer Grading Framework.

3 Methods

In this section, we present the overview of the entire peer grading framework and
introduce the design of grading task assignment in detail. Figure 2 illustrates the basic
framework of our peer grading process. It consists of three major components: the
student performance evaluation model, the peer grading task allocator and the score
aggregation model. The student performance evaluation model is responsible for
computing each student’s knowledge level by analyzing each student’s behavior in the
watching course videos and the performance exercises after the videos. The outcome
from the model presents the prior knowledge level of students for the peer grading
process to determine the arrangement of peer grading groups. The peer grading process
includes two steps: grading task allocation and score aggregation model. In this paper,
we introduce the MPLT algorithm to perform task allocation in order to minimize the
difference among the students in each grading group. Based on the task assignment
plan, students can evaluate other peers’ submissions and give scores using their own
judgement. As each submission in general receives multiple scores from its graders, the
score aggregation model needs to infer the accurate and unbiased score for the
submission from the scores recommended by peer graders.
At the beginning of the section, we describe the problem of assignment allocation in
peer grading. And then we present the algorithm of LPT that is used in the field of task
scheduling, and how we can extend the idea LPT for the purpose of peer grading and
introduce the new algorithm MLPT. At the end of the section, we briefly discuss our
research work on student performance evaluation model.

3.1 The Problem of Submission Allocation in Peer Grading

In the process of peer grading, each student plays two roles: a grader and a submitter.
Suppose that there are ℕ students participating peer grading, and there are also ℕ
submissions from these students. For convenience, we suppose that each student needs
to grade ℳ submissions and each submission is graded by ℳ graders, typically the
value of ℳ is between 3 and 6. The description of assignments allocation in peer
grading is shown in Figure 2. Specifically, the students are divided into ℕ groups:
each group contains ℳ students, and each student is included in one group, so each
group corresponds to one submission [1, 8].
The process of improving the allocating submissions can be expressed as the
following problem:
Define the student collection V = {𝑣1 , 𝑣2 , … , 𝑣𝑛 } and the submission collection
U = {𝑢1 , 𝑢2 , … , 𝑢𝑛 }.
N students v1 v2 ……
...
vn

v1 v8 v6 v4
N groups
v3 …… v9

N submissions u1 u2 …… un

Fig. 2. The process of task assignment in peer grading.

The degree of a student 𝑣𝑖 mastering the knowledge is denoted by w = w(𝑣𝑖 )

where w(𝑣𝑖 ) lies in the interval (0, 1) and the parameter i falls in the range (0, n) if
student 𝑣𝑖 submits the submission 𝑢𝑖 . w(𝑣𝑖 ) that is generated by the student
evaluation model, is regarded as the prior level of student knowledge mastery.
The problem is that given the parameter m how to divide the student collection
𝐺1 , 𝐺2 , … , 𝐺𝑛 , and make it satisfy the following conditions:
1. The grader collection 𝐺𝑖 = {𝑣𝑖1 , 𝑣𝑖2 , … , 𝑣𝑖𝑚 } only contains m students.
2. For any student 𝑣𝑖 , there are m groups that include her.
3. 𝑣𝑖 ∈ 𝐺𝑖 , it means that all of the students grade the submission 𝑢𝑖 , so the
submitter 𝑣𝑖 could be contained in the grading group 𝐺𝑖 .
4. The variance of the students’ comprehensive knowledge level must be the
minimal, namely min{𝑣𝑎𝑟(𝑤(𝐺1 ), 𝑤(𝐺2 ), … , 𝑤(𝐺𝑛 ))}, for the purpose that
the grading level of each group is as close as possible in where w(𝐺𝑖 ) =
∑𝑣𝑗∈𝐺𝑖 𝑤(𝑣𝑗 ).

3.2 The Algorithm of LPT

The algorithm LPT is a basic method to solve the task scheduling problem in parallel
computing system. The tasks scheduling problem in a parallel system can be described
Table 1. The process of the Algorithm of LPT.

Algorithm: the LPT algorithm of task scheduling

1. Sort the tasks in the descend order by the costing time, make the result
as 𝐭(𝑱𝟏 ) ≥ 𝐭(𝑱𝟐 ) ≥ ⋯ ≥ 𝐭(𝑱𝒏 ).
2. for 𝐢 = 𝟏, 𝟐 … , 𝐩:
Assign the load 𝑳𝒊 of machine 𝑴𝒊 as 0, namely 𝑳𝒊 ← 𝟎.
Assign the task collection 𝐉(𝐢) of the machine 𝑴𝒊 as
empty, namely 𝐉(𝐢) ← 𝛟.
3. for 𝐤 = 𝟏, 𝟐 … , 𝐧:
Select the current machine with the smallest load, recorded
as 𝑴𝒊 .
Add the current task 𝑱𝒌 to the task collection of the
machine 𝐉(𝐢), namely
𝐉(𝐢) ← 𝐉(𝐢) ∪ 𝑱𝒌 .
Update the load of machine 𝑴𝒊 , namely 𝑳𝒊 ← 𝑳𝒊 + 𝒕(𝑱𝒌 ).

as the optimization process of minimizing the difference of processing duration among

all the machines. Given the tasks J = {𝑗1 , 𝑗2 , … , 𝑗𝑛 } , each of which has a processing
time 𝑡(𝑗𝑖 ). Define T = {𝑡(𝑗1 ), 𝑡(𝑗2 ), … , 𝑡(𝑗𝑛 )}. Assume the tasks are allocated to the
machines M = {𝑚1 , 𝑚2 , … , 𝑚𝑝 } with the same model. We assume that the tasks are
independent and their arrivals occur at the same time point.
The task scheduling has been proved to be a NP-complete problem [7, 14], namely
the complexity of finding the optimal solution increases as the number of tasks and the
number of machines in the exponential way. To find the acceptable solution in practice,
it is necessary to employ the heuristic algorithm to compute an approximate solution.
The LPT algorithm adopts the greedy strategy for exploring approximate solutions
according to the static task assignment policy. At the initial moment, all of the tasks are
allocated to the machines, and during the executions of the tasks, the task schedule must
remain until the end. The LPT algorithm arranges the pending tasks in a descending
order and always assigns the tasks to the machines as soon as they become available.
The detail process of the algorithm is described as follows.
The process of the algorithm LPT is simple and clear, and the time complexity can
reach O(n log 𝑛) if the step of finding the earliest free machine adopts the minimum
heap algorithm [10, 12]. Furthermore, the algorithm also has the optimal approximate
result. It can be proved that if 𝐹 ∗ (𝐽) denotes the optimal costing time when assigning
the task collection J to machine collection M, F(J) denotes the costing time adopting
the algorithm LPT, so we can obtain:
𝐹(𝐽) 4 1
≤ −
𝐹 ∗ (𝐽) 3 3𝑚
From the approximation relation we can draw that LPT is an efficient algorithm. The
detailed process of proving the approximation relation can be found in references [9].

3.3 The Algorithm of MLPT

Similarly, there are many similarities between assigning submissions and task
scheduling in parallel systems. We introduce a MLPT algorithm by redefining the
process of LPT towards the needs of grading task assignment. Because for the problem
of assigning submissions, assigning submissions to the graders can be viewed in
another angle as assigning the graders to the submissions, so the student collection V =
{𝑣1 , 𝑣2 , … , 𝑣𝑛 } is considered as the task collection J = {𝑗1 , 𝑗2 , … , 𝑗𝑛 } and the
knowledge level w(𝑣𝑖 ) of each student is the processing time t(𝑗𝑖 ) , then the
submission collection U = {𝑢1 , 𝑢2 , … , 𝑢𝑛 } is considered as the machine collection
M = {𝑚1 , 𝑚2 , … , 𝑚𝑝 }. According to these above assumptions, assigning the graders to
submissions in peer grading is equivalent to allocating the tasks to machines in the
scheduling process, the MLPT algorithm can ensure the sum of the knowledge levels
of the graders in each group is close to any of the others.

Table 2. The Algorithm of MLPT.

Algorithm: the algorithm of MLPT allocating submissions

1. Sort the graders by the level of knowledge in descending order, namely w(𝑣1 ) ≥
𝑤(𝑣2 ) ≥ ⋯ ≥ 𝑤(𝑣𝑛 )
2. For i = 1, 2, … , n:
The submission collection U(i) graded by grader 𝑣𝑖 is assigned as empty,
Namely, U(i) ← ϕ.
3. For j = 1, 2, … , n:
The comprehensive grading level 𝐿𝑗 corresponding to the submission 𝑢𝑗 is
assigned to 0, namely 𝐿𝑗 ← 0.
The student group V(j) corresponding to the submission 𝑢𝑗 is assigned to 0,
namely V(j) ← 0.
4. For i = 1, 2, … , n:
For k = 1, 2, … , m:
Select the submission with the smallest comprehensive grading level from
submission collection {𝑢𝑗 |𝑗 ≠ 𝑖}, recorded as 𝑢𝑗 .
Insert the current grader 𝑣𝑖 to the submissions collection 𝑢𝑗 , namely V(j) ←
V(j) ∪ 𝑣𝑖 .
Update the comprehensive grading level 𝐿𝑗 of the submission 𝑢𝑗 , namely 𝐿𝑗 ←
𝐿𝑗 + 𝑤(𝑣𝑖 ).
Insert the submission 𝑢𝑗 to the collection U(i) of graded by grader 𝑣𝑖 , namely
U(i) ← U(i) ∪ 𝑢𝑗 .

Moreover, there are two conditions in assigning submissions compared to task

scheduling. One condition is that the redundant grading parameter m guarantees that
the number of different submissions for each grader is the same [13]. The other is that
the grader could not evaluate his own submission.

3.4 Student Performance Evaluation Model

We establish a two-stage model to assess student mastery level of each knowledge

skill, in other words, to estimate the probability that a student has learned the specific
knowledge of a chapter. The first stage is the behavior-based student engagement model,
which first analyze student video-watching activities within a chapter and extract
interpretive quantities to predict the probability that a student has mastered the
knowledge of that certain chapter. This model extract eight features including the total
video playing time, the number of video played, the number of pauses, the number of
sessions, the number of requests, the number of rewinds, the number of slow play rate
setting, the number of fast forwards. The logistic regression method is used to fit these
features and predict the engagement level of every student.
The second stage is the knowledge tracing model to infer the level of student
knowledge mastery. The knowledge tracing model, popularized in Intelligent Tutoring
System (ITS), leverages the quiz response sequence of every student to assess their
learning outcomes. Our work adopts the knowledge tracing model and ameliorates it
by combining the prediction results obtained in the behavior-based student engagement
model. Essentially, it is based on a 2-state dynamic Bayesian network where student
responses are the observed variable and student knowledge is the latent one. The model
takes student response sequences and uses them to estimate the student’s level of
knowledge. The initial state of the model can be trained by the results obtained in the
behavior-based student engagement model.

4 Datasets

The dataset using in our paper is collective from the course of computer network
experiment abbreviated as CNE) of our MOOCs platform of Beihang Xuetang
(http://mooc.buaa.edu.cn) The MOOC course CNE includes three assignments for the
Sophomore undergraduate who had not studied this course before and we only use
assignment 1 and assignments 2. The course required that the students had to upload
the submissions before the end of each week.
Specially, the session course was open in the year of 2015 and active for a 11-week
semester. The course contains 10 chapters where there are several video units per
chapter and a final examination. After each video unit, there are multiple choice
problems as quiz for students. The number of the problems ranges from 5 to 13 in each
unit. Also, the course contains 4 open-ended design problems that demand the students
to design and implement the experimental programs independently. The score of these
open-ended problems are generated through the peer grading mechanism in our MOOC
platform. There are approximately 600 active students each semester since the course
CNE went online and roughly 500 students to take part in the final exams. The number
of the records collected by the system is 2 million, which logs every user interaction
with the courseware including scanning and clicking, video interaction, posting
questions and comments in the course forum.
After removing the incomplete items in the original dataset, we choose 210 students
including 210 items, and in our setting each student grade 4 submissions and each
submission must be graded by 4 graders. In order to prove the advantage of the MLPT
algorithm, we also produced two simulation datasets imitating the real data. In the
simulation datasets, the number of items is 200 and we valued the redundancy peer
grading number as 3, 4 and 5 in our experiments. The major difference between the real
data and the simulation data is that the prior is generated from the normal distribution
and evenly distribution, respectively.
5 Experimental Results

100.00%
the decline of variance of groups

98.00%

96.00%

94.00%

92.00%

90.00%

88.00%
100 200 300 400 500 600
m=3 m=4 m=5 students

Fig. 3. The degree of declining of variance from MLPT algorithm to the random algorithm.

This section mainly verifies the algorithm of MLPT from both the real dataset and
simulation dataset. For each dataset, we compute the variance of all the groups using
the sum of each group as one item and we have run 10 times for the computing an
average value. In both table 3 and table 4, the simulation dataset 1 is generated from an
evenly distribution and the simulation dataset 2 is generated from a random assignment
distribution.

Table 3. The comparison between random assignments and MLPT based on the simulation
datasets.

Random Distributions MLPT

Simulation dataset 1 0.381 0.009
Simulation dataset 2 0.266 0.017

Table 4. The comparison between random assignments and MLPT based on the real datasets.

Random Assignment MLPT

Assignment 1 0.355 0.053
Assignment 2 0.376 0.046

Table 3 shows the results from the simulation datasets, the dataset 1 and dataset 2,
which are generated from the evenly distribution and normal distribution, respectively.
The MLPT algorithm reduces the variances of the knowledge master level among the
groups from 0.381 to the 0.009 on the simulation dataset 1 and from 0.266 to 0.017 on
the simulation dataset 2.
Similarly, Table 4 is the results generated from the real datasets. The value of the
variance is 0.053 after using MLPT in Assignment 1, and the optimization percentage
is 85.0%. The optimization percentage is 87.7% and the MLPT value is decreased from
0.376 down to 0.046 in Assignment 2. Note that the MLPT optimization results on the
simulation datasets seem better than the results on the real dataset. The reason may be
that the prior inferred from the performance of quiz doesn’t completely associated with
the actual knowledge level of students in the open-ended problems.

Fig. 4. The effectiveness of task assignment on RMSE of peer grading.

Through simulation, we can explore different settings such as the various numbers
of students and the different values of the parameter m to compare the performance
between the MLPT algorithm and the random algorithm. In the simulation dataset, we
have verified a series of datasets with the number of students varying from 100 to 600
and the value of parameter m ranging from 3 from 5. And in each simulation, we
compute the decline in the group variance caused by the MLPT algorithm over the
Random algorithm. Figure 3 plots the results generated from the Random and MLPT
algorithms. It demonstrates that the MLPT algorithm can reduce the variance of all the
groups significantly in comparison with the Random algorithm, especially when there
are not many students and the parameter m is relatively small. Figure 4 shows the
comparison results generated from the MLPT and Random algorithm based on the real
datasets in the process of peer grading. The vertical axis indicates the Root Mean
Square Error (RMSE), which is used to compute the accurate of grading scores. In the
research of Fei Mi et al. [11], researchers proposed new score aggregation methods
based on probabilistic graph models to infer accurate scores of student submissions.
These methods outperform the method using the mean aggregation strategy. It is
necessary to investigate whether the MPLPT performance has any impact on different
aggregation methods. The result in Figure 4(a) shows the difference of RMSE between
the MLPT algorithm and the RMSE algorithm based on the mean aggregation strategy
and Figure 4(b) shows the results based on the probabilistic graph model. No matter
which aggregation strategy is adopted, the MLPT algorithm can effectively improve
the overall accuracy of peer grading scores.

6 Discussion and Future Work

Peer grading in MOOC platforms need an effective framework for grading tasks
allocation and score aggregation in order to deliver accurate feedbacks of homework
evaluation to online students. Our paper presents a complete peer grading framework
and introduces a new task scheduling algorithm MLPT for the task assignment problem
of peer grading. This framework can infer the knowledge mastery level of every student
using a two-stage data analysis pipeline with the behavior-based engagement model
and knowledge tracing-based student performance evaluation model Our task
allocation algorithm regards the assessment outcomes of each student in the quizzes of
the MOOC courses as the priors and utilizes the priors to build peer grading groups. It
aims at minimizing the capability difference among grading groups and generating a
fair and accurate evaluation for each student submission. The experimental results from
both simulation datasets and real course datasets demonstrates that the MLPT algorithm
can outperform the conventional random strategy.
There are a number of issues to be addressed in future work. In this paper, we assume
that the redundancy number of peer grading tasks for every submission remains
constant. However, in practice, students may fail to complete their grading tasks in time,
thus resulting in less redundant grading results for some submissions. It would be
interesting to investigate whether this problem affects the performance of the schedule
algorithm. Another issue is whether the errors in the student evaluation model has any
impact on the MLPT algorithm. Given the probability nature of the knowledge tracing
model, it is unavoidable that it may give a biased estimation of knowledge levels of
some students when there are noises in the data of user interactions and question
answering. One of the feasible solutions is to have staffs to cross-examine some peer
grading results and correct the potential errors.

7 Acknowledgments

This work was supported in part by grant from State Key Laboratory of Software
Development Environment (Funding No. SKLSDE-2015ZX-03) and NSFC (Grant No.
61532004).

References

1. Fonteles A S, Bouveret S, Gensel J.: Heuristics for Task Recommendation in

Spatiotemporal Crowdsourcing Systems[C]//Proceedings of the 13th International
Conference on Advances in Mobile Computing and Multimedia. ACM, 1-5 (2015).
2. Cheng J, Teevan J, Bernstein M S.: Measuring Crowdsourcing Effort with Error-Time
Curves[C]//Proceedings of the 33rd Annual ACM Conference on Human Factors in
Computing Systems. ACM, 1365-1374 (2015).
3. Qiu C, Squicciarini A C, Carminati B, et al.: Crowdselect: Increasing accuracy of
crowdsourcing tasks through behavior prediction and user selection[C]//Proceedings of the
25th ACM International on Conference on Information and Knowledge Management. ACM,
539-548 (2016).
4. Howe J.: The rise of crowdsourcing[J]. Wired magazine, 14(6): 1-4 (2006).
5. Wiki for Multiprocessor Scheduling Information,
https://en.wikipedia.org/wiki/Multiprocessor_scheduling.
 6. Piech C, Huang J, Chen Z, et al.: Tuned models of peer assessment in MOOCs[J]. arXiv
preprint arXiv:1307.2579 (2013).
7. Coffman Jr E G, Sethi R.: A generalized bound on LPT sequencing[C]//Proceedings of the
1976 ACM SIGMETRICS conference on Computer performance modeling measurement
and evaluation. ACM, 306-310 (1976).
8. Alfarrarjeh A, Emrich T, Shahabi C.: Scalable spatial crowdsourcing: A study of distributed
algorithms[C]//2015 16th IEEE International Conference on Mobile Data Management.
IEEE, 1: 134-144 (2015).
9. Baneres D, Caballé S, Clarisó R.: Towards a Learning Analytics Support for Intelligent
Tutoring Systems on MOOC Platforms[C]//Complex, Intelligent, and Software Intensive
Systems (CISIS), 2016 10th International Conference on. IEEE, 103-110 (2016).
10. Gonzalez T, Ibarra O H, Sahni S.: Bounds for LPT schedules on uniform processors[J].
SIAM Journal on Computing, 6(1): 155-166 (1977).
11. Mi F, Yeung D Y. Probabilistic Graphical Models for Boosting Cardinal and Ordinal Peer
Grading in MOOCs[C]//AAAI. 454-460 (2015).
12. Massabò I, Paletta G, Ruiz-Torres A J.: A note on longest processing time algorithms for
the two uniform parallel machine makespan minimization problem[J]. Journal of
Scheduling, 19(2): 207-211 (2016).
13. Gardner K, Zbarsky S, Harchol-Balter M, et al.: The Power of d Choices for Redundancy[J].
Acm Sigmetrics Performance Evaluation Review, 44(1):409-410 (2016).
14. Feier M C, Lemnaru C, Potolea R.: Solving NP-Complete Problems on the CUDA
Architecture Using Genetic Algorithms[C]// International Symposium on Parallel and
Distributed Computing, Ispdc 2011, Cluj-Napoca, Romania, July. DBLP, 278-281 (2011).
15. Ul Hassan U, Curry E. Efficient task assignment for spatial crowdsourcing[J]. Expert
Systems with Applications An International Journal, 58(C):36-56 (2016).
16. Jung H J, Lease M.: Crowdsourced Task Routing via Matrix Factorization[J]. Eprint Arxiv
(2013).
17. Karger D R, Oh S, Shah D.: Budget-optimal crowdsourcing using low-rank matrix
approximations[C]//Communication, Control, and Computing (Allerton), 2011 49th Annual
Allerton Conference on. IEEE, 284-291 (2011).
18. Yan Y, Fung G M, Rosales R, et al.: Active learning from crowds[C]//Proceedings of the
28th international conference on machine learning (ICML-11). 1161-1168 (2011)

View publication stats