See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/281554739

Review of Computerized Evaluation Tools in Education

Article · December 2012

CITATIONS READS
3 898

2 authors, including:

Mandalika Syamala Devi

Panjab University
34 PUBLICATIONS   216 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Disaster Management View project

Multiagent Integrated scheme for Intrusion Detection View project

All content following this page was uploaded by Himani Mittal on 20 August 2019.

The user has requested enhancement of the downloaded file.

I J A I C R, 4(2), 2012, pp. 111-117

Review of Computerized Evaluation Tools in Education

M. Syamala Devi1 and Himani Mittal2
1
Professor, Department of Computer Science and Applications, Panjab University,Chandigarh, India
2
Research Scholar, Department of Computer Science and Applications, Panjab University, Chandigarh

Recent advancements in the field of e-learning and virtual learning have changed the face of education. An important part
of learning process is assessment of student learning through examinations. The Examination process involves conduct of
examinations, evaluation of the answers and result declaration. This paper reviews various computer-based, computer-
assisted and web-based tools for evaluation of the Subjective, Objective and Practical Examinations. For subjective
examinations, statistical and mathematical tools available for evaluation of short-answer, long-answer and essay-length
answer questions are reviewed. The scope of evaluation done by these tools is limited to style and keyword presence. For
objective examinations, a number of automated tools for multiple-choice question evaluation are available. For practical
examinations, the scope of evaluation tools available is limited to computer science education. These tools are used for
evaluation of software programs in assignments or competitions. The paper also includes a discussion on the reviewed
work. The subjective tools use a bag-of-words approach. The objective tools focus on Multiple Choice Questions. The use of
practical tools can be extended to formal examinations also. Finally, it is concluded that though there are many tools
available for objective evaluation, but there is a need to develop tools for evaluating subjective and practical examinations
satisfactorily. The scope for further work to enhance the performance of evaluation tools is also included.
Keywords: Evaluation tools, Subjective Evaluation, Statistical Methods, Latent Semantic Analysis, Practical Assignments.

1. INTRODUCTION
Teaching and learning process is undergoing a transition. Virtual learning and E-learning platforms have changed
the face of classroom teaching. Examinations are an important part of learning process which require to be
computerized. In traditional examination system, the student submits the answers written on answer-book.
These answer-books are given to evaluator for grading. Grading is time consuming and depends on availability
of evaluator. Then the results are compiled. The use of computerized tools reduces the limitations of manual
process. The online examination terminals transfer the answers submitted by students, electronically to the
centralized database and thus avoiding the physical movement of answer books. The advantages of intelligent
software tools are manifold. They are fast and avoid human errors of omission and totaling mistakes. They
ensure uniformity of marking scheme because they have the same inference mechanism for checking all the
answers. Automation can ensure speedy result declaration.
The student’s performance is evaluated with the help of Objective, Practical and Subjective examinations
as per the need of the course. Subjective Examinations include short-answer, long-answer and essay-length
answer questions. The answers are evaluated on the basis of a number of parameters like correctness, presence
of keywords and style of writing. The scope of evaluation done by available tools is limited to presence of
keywords and style of writing. They use a bag-of-words approach and cannot evaluate factual correctness. The
objective examinations can be any of the following variety like multiple-choice questions (MCQ), multiple-
answer questions, fill-in-the-blanks, match-the-following and one-word answers. For evaluation of Objective
examination, a range of computerized evaluation tools are available that have good performance. However,
these tools primarily focus on MCQ questions. Practical Examinations for computer science education include
project work, assignments and programming experiments. The available tools focus only on the evaluation of
programming assignments and competitions.
The paper is organized as follows. The review of related work in Subjective, Objective and Practical
evaluation tools is given in section 2. The discussion on available reviewed work is included in section 3.
Conclusions and scope for future work is mentioned in section 4.
*Corresponding author: syamala@pu.ac.in; research.himani@gmail.com
112 M. Syamala Devi and Himani Mittal

2. REVIEW OF RELATED WORK

2.1. Subjective Evaluation

In 1982, Macdonald, Frase, Ginrich, and Keenan [1] developed Writer’s Workbench programs, which analyze
English prose and suggest improvements. However, the focus was on correction of grammar. In 1994, Project
Essay Grader (PEG) [2], was developed for automated subjective evaluation. It performs the evaluation based
on style analysis of the answer and does not take content into account. Its agreement with human graders is
87%. It measures features like essay length, word length and vocabulary used etc. In 2001, Shermis, Mzumara,
Olson and Harrington [3] enhanced PEG for the World Wide Web. PEG cannot evaluate technical validity of
answers. It was used at primary and secondary levels of education for English subject essay evaluation.
Electronic Essay Rater (E-Rater) was developed by Burstein, Kukich, Wolff, Chi, & Chodorow, 1998 [4]
and later enhanced in 2006 [5]. It used MSNLP for parsing the text and extracting text features. Then weight-
age is assigned to these features. Whenever a new essay is to be evaluated, its features are compared to already
graded essays. It is successfully used for AWA test in GMAT with agreement rates between human expert and
system consistently between 84%. However, it is not suitable for technical answers and is like an extension of
PEG. Conceptual Rater (C-rater) (Burstein et al. 2000 [6] and Sukkarieh et al. 2009 [7]) is a Natural language
based prototype aimed at the evaluation of short answers related to content-based questions. In 2011, Sukkarieh
[8] discussed that max-entropy technique is used in C-rater. C-Rater achieved over 80% agreement with the
score assigned by a human-grader.
In 1999, Foltz et al. [9] applied mathematical technique called Latent Semantic Analysis (LSA) to
computerized evaluation in a tool called Intelligent Essay Assessor (IEA). In this method a matrix is made with
keywords to be searched as rows and documents as columns. The frequency of each word in each document is
recorded. Then Singular value decomposition is done on this matrix. The correlation between human and IEA
score is 80%. This technique is not capable of evaluating short-answer questions and technical answers. In
1999, Hofmann [10] developed Probabilistic latent semantic analysis (PLSA). This method is an improvement
over LSA, as it has strong statistical foundation in aspect modelling. It calculates doument to word joint
probability using estimation maximization algorithm.
In 2001, Callear, Jerrams-Smith and Soh [11] presented a survey of major systems for the automated
evaluation and proposed a new technique called Automated Text Marker (ATM) prototype. The two main
components of ATM are the syntax and semantics analyzers. ATM is written in Prolog. The standard answer
and students response are broken down into basic concepts using conceptual dependencies; and then the two
are compared. Bayesian Essay Test Scoring System (BETSY) (2002) [12] is a program that classifies text
based on trained material. The models used for text classification are Multivariate Bernoulli Model (MBM)
and the Multinomial Model. An accuracy of over 80% was achieved.
In 2003, Blei et al. [13] suggested a generalization of PLSA, by using mixture model and dirichlet allocation
distribution for calculating co-occurrence probability of words. This technique is known as Latent Dirichlet
Allocation (LDA). In 2003, Rose, Roque, Bhembe, and VanLehn [14], proposed a tool called CarmelTC. The
hybrid CarmelTC approach uses decision trees and Naive Bayes text classifier statistical technique. The success
rate of this CamelTC was 90% which was comparable to that of LSA. In 2005, P´erez, Gliozzo, Strapparava,
Alfonseca, Rodr´ıguez and Magnini [15] have tested the hypothesis that combining different knowledge sources
and algorithms is a viable strategy for an automatic evaluation of students’ free-text answers. The combination
schema for the techniques Bleu (machine translation effectiveness algorithm) and LSA (essay clustering
technique) was found effective. The mean correlation to the human’s scores has reached 50%.
In 2008, Kakkonen, Myller, Sutinen and Timonen [16], Automatic Essay Assessor (AEA) is a system that
utilizes information retrieval techniques such as LSA, PLSA and LDA for automatic essay grading. They
performed a series of experiments using LSA, PLSA and LDA for document comparisons in AEA comparing
the applicability of LSA, PLSA, and LDA to essay grading with empirical data. It was found that using LSA
yielded slightly more accurate grading than PLSA and LDA.
Review of Computerized Evaluation tools in Education 113

In 2008, Li Bin, Lu Jun, Yao Jian-Min, Zhu Qiao-Ming [17], the K-Nearest Neighbor (KNN) algorithm for
text classification is applied. It makes use of training documents which have known categories and finds closest
neighbors of the new sample document among all. These neighbors enable to find the category of new document.
The experiments show accuracy above 76%. In 2010, Islam and Hoque [18] proposed a system that makes use
of Generalised latent semantic analysis (GLSA) technique for evaluation. In GLSA n-gram by document matrix
is created instead of a word by document matrix of LSA. It has 89% of accuracy which show that the system is
very closer to human grader. In 2010, Cutrone and Chang [19] in their research paper proposed a short answer
evaluation method using Natural language processing (NLP) techniques. It reduces the standard answer and
student answer into its canonical form and compares them. It uses NLP techniques like parsing, stemming,
stop-word removal, morpho-syntactic-variation handling, etc. The system in its current format is capable of
processing answers containing a single sentence that is free of grammar and spelling mistakes.

2.2. Objective Evaluation

In research work done by Tinoco, Fox, Ehrich, Fuks (1996) [20] and Tinoco, Fox , Barnetter (1997) [21],
developed a web-based tool called QuizIt. This software was capable of evaluating students on the basis of
Multiple-choice-questions. It uses static web pages for presenting the questions prepared and entered into the
system by human instructor. An important improvement over this was generation of questions using question
bank as done in Cecil tool developed by Gardner, 2001 [22]. WebAssign [23] is a web-based system that
enables the authoring, delivery and marking of MCQ-based coursework. Question banks are available for the
assessment in various disciplines. In 2007, Xingbao Li and Yunming Wu [24] proposed an automation of
examination system. The key features are automatic and manual both type of paper setting using question
banks, and paper correction automatic for objective and manual for subjective. In 2009, Li Jun [25] propose an
architecture of online examination system, and set up a distributed cross-platform examination system based
on web service and COM components.
In Moe Moe Aye and Mie Mie Thet Thwin, 2008 [26], proposes a mobile agent based solution for examination
system. It makes use of distributed architecture.The paper setting , conduct and evaluation are kept on different
servers and mobile agents running between each for synchronization and carrying out various tasks. There are
four agents: Answer agent, question agent, prepare agent and fetch agent. All the agents are mobile agent. In
Gawali and Meshram, 2009 [27], again agent based model was developed for examination system. It was for
conducting objective exams. The agents developed for online examination system belongs to goal-based type
category of agents. There is a main agent responsible for conducting the examination, static agents for client,
server and database and mobile agents running between the different machines carrying data.

2.3. Practical Evaluation

Jackson and Usher, 1997, developed a computer based evaluation tool [28] called ASSYST. This tool evaluates
computer science practical assignments. This tool helps the examiner in evaluating the programming assignments
of students. It could evaluate C and ADA language assignments. It assessed computer programs on four
parameters: correctness, style, efficiency and complexity. RoboProf (Daly, 1999 [29]) is an on-line teaching
system based on the World Wide Web technology. It is structured as a formative, self-paced course-book which
presents the students with information on programming topics and then automatically assesses student exercises
that cover these topics. RoboProf teaches syntax and structure of C++ programming language to students.
BOSS [30] is an online submission system, which provides features for course management and automated
testing. It could evaluate C, C++ and Java programs. It makes use of JUnit for testing.
Mooshak tool was developed in 2003 by Leal and Silva [31]. Mooshak is a online programming contest
managing software. It helps in conduct, answer evaluation and result declaration and feedback. The languages
supported by default are C, Pascal, C++ and Java. In 2009, García-Mateos and Fernández-Alemán [32] and
Montoya-Dato, Fernández-Alemán, García-Mateos [33], used Mooshak for Algorithms and Data Structures
course. Douce et al. in 2005 [34], developed Automated System for Assessment of Programming (ASAP)
project which fits into the e-learning framework as an evaluation tool. Mandal, Mandal and Reade discussed in
114 M. Syamala Devi and Himani Mittal

there paper published in 2006 [35] the architecture of an Automatic Program Evaluation System. This system
can handle only C programs. Their approach is to perform White Box test, instead of, Black box or Grey box
testing.
SQLify (2007) [36] is to assist students learn SQL. It facilitated writing test queries against databases,
receive immediate feedback which is more informative than what can be offered by DBMS. In 2010, Zhang
and Haifeng Ke [37] also proposed a design for SQL Paperless Examination System. In 2008, Farrow and King
[38] used an online programming examination to assess undergraduates who had learned Java for two terms. It
uses BLUEJ and JUnit software. In 2010, Bronius Skupas [39] wrote a paper on feedback improvement in
automatic program evaluation systems. He says that some programming errors should not influence the final
score heavily. Black box testing alone cannot ensure this. A typical approach is to use semi-automatic systems
for programming exam evaluation. It proposes two grading steps. The first step is responsible for student program
output format checking. The second step relates to output correctness. In 2011, Fernández Alemán [40] proposed
an automated assessment tool for programming courses. It extends the functionality of Mooshak by adding a
external static marker.
In Zhenming, Liang and Guohua, 2003 [41] and Zhang, Zhuang, Zhenming and Guohua, 2006 [42], tool
was developed to measure operating skills of the students and their proficiency in using software like Ms-
Word, Ms-Excel, Ms-Powerpoint and internet familiarity.
An overview of features required by practical evaluation tools is given in [43]. For Evaluation of GUI-
based student programs English [44] proposed a tool called JEWL for Java GUI programs.

3. DISCUSSION ON REVIEWED WORK

3.1. Subjective Evaluation

For subjective evaluation, primarily statistical and mathematical techniques are used. Several statistical techniques
used are PLSA, LDA, BLEU, BETSY, KNN, E-rater and C-rater. In LSA and GLSA techniques, mathematical
modeling of student answers is done to evaluate them. These statistical and mathematical methods classify,
cluster, index and entail the student answers by extracting essay features to calculate quantities like probability,
entropy and latent variables. The above methods do not utilize fact based evaluation techniques. These methods
extract the conceptual meaning and use bag-of-words approach. Several hybrid techniques were used in [14,
15, 16]. However, there success has been limited to style check and keyword matching. These techniques are
unable to distinguish between “boy stepped on spider” and “spider stepped on the boy” as pointed out in [19].
These tools are not used for evaluating technical correctness of answers. Many evaluation tools that make use
of statistical methods require training data of pre-graded essays.
The recent shift in natural language processing is towards semantic meaning based evaluation. Some tools
that extract the semantic meaning are syntactic and semantic analyzers in ATM and canonical form reduction in
[19]. However, these tools evaluate only short answers or single sentence answers.

3.2. Objective Evaluation

For objective evaluation, matching algorithms are used for marking the student answers against the standard
answer key. Though ample tools are available for MCQ based tests, there is still scope for development of tools
for fill-in-the-blanks. These questions can be answered using synonyms and polysemy words, current tools do
not take this fact into account.

3.3. Practical Evaluation

The available practical evaluation tools are used for different purposes. Assyst, Roboprof, Boss and SQLify are
used for assignment evaluation and classroom teaching. Mooshak is used for conducting online programming
contests and evaluating contestants programs. The only tool that is used for conducting and evaluating
Review of Computerized Evaluation tools in Education 115

programming examinations is discussed in [38]. The operating skills are evaluated in [41, 42]. Programming
languages evaluated by these tools are C, C++, Fortran, Pascal and Java (All tools do not evaluate all languages).
The technique used in all the tools for evaluation is black box testing. [39, 45] emphasize the use of white-box
testing in evaluation tools.
The practical evaluation tools are built using different technologies. Assyst is a collection of Linux shell
scipts. Boss is developed using Java. Mooshak is implemented using Tcl/Tk. [41,42] are designed using DCOM.

4. CONCLUSIONS AND SCOPE FOR FURTHER WORK

4.1. Conclusions
Using computer technology to conduct and evaluate examinations will benefit both students and the universities,
by saving time, effort, ensuring minimal error and universal accessibility of the system. For the Objective type
of examinations, many Universities and institutes use computerized tools for the evaluation. However, there is
enough scope for research in the evaluation of Practical and Subjective examinations as the existing computerized
tools do not focus on correctness of answers.

4.2. Scope for Further Work

In subjective evaluation tools, several enhancements can be done. The tools need to extract factual content of
the answers. These tools can be designed to handle variation of english language (morpho-syntactic variations).
Discourse analysis can be done to evaluate multi-sentence answers. These tools should allow flexibility in
marking by assigning weightage to individual keywords, depending on its importance in the answer. Lastly, the
tools should be able to justify the scores assigned (like an expert system).
The Practical Evaluation tools need to compare the space and time complexity of different implementations.
The tools should not completely reject a code that has incorrect syntax. Some weightage should be given by
doing step marking. The tools have to provide for evaluation of Graphical User Interface (GUI) programs.
Black box testing needs to be complemented with white box testing as some algorithm cannot be evaluated
completely using black box testing alone (e.g. algorithm to generate all permutations of word “peace”). Lastly,
the student programs should be checked for malware uploading.
The above changes can make the computerized evaluation successful. The new generation evaluation tools
should give ample feedback to the students and justify the scores. These tools can be developed using multi-
agent technology (MAS). The software agents can be designed to assist and share the workload of a human
examiner.

References
[1] Macdonald, N. H., Frase, L. T., Gingrich, P. S., Keenan, S. A., (1982), The Writer’s Workbench: Computer Aids for Text
Analysis, IEEE Transactions on Communications, 30(1), 105-110.
[2] Page, E. B., (1994), Computer Grading of Student Prose, Using Modern Concepts and Software, The Journal of
Experimental Education, 62(2), 127-142.
[3] Shermis, M. D., Mzumara, H. R., Olson, J., Harrington, S., (2001), On-line Grading of Student Essays: PEG goes on the
World Wide Web, Assessment & Evaluation in Higher Education, 26(3), 247-259.
[4] Burstein, J., Kukich, K., Wolff, S., Chi Lu, Chodorow, M., Braden-Harder, L., and Harris, M. D., (1998), Automated
Scoring Using A Hybrid Feature Identification Technique, Proceedings of the Annual Meeting of the Association of
Computational Linguistics and International Conference on Computational Linguistics, 1, 206-210. Doi: 10.315/
980451.980879.
[5] Attali, Y., Burstein, J., (2006), Automated Essay Scoring With E-rater? V.2.0, The Journal of Technology, Learning and
Assessment, 4(3), e-journal.
[6] Burstein, J., Marcu, D., (2000), Towards Using Text Summarization for Essay-Based Feedback, in Le 7e Conference
Traitement Automatique des Langues Naturelles TALN’2000.
116 M. Syamala Devi and Himani Mittal

[7] Sukkarieh, J. Z., Blackmore, J., (2009), c-rater: Automatic Content Scoring for Short Constructed Responses, Proceedings
of the Twenty-Second International FLAIRS Conference, AAAI Press, pp. 290-295.
[8] Sukkarieh, J. Z., (2011), “Using a MaxEnt Classifier for the Automatic Content Scoring of Free-Text Responses”, American
Institute of Physics Conference Proceedings, 1305(1), 41.
[9] Foltz, P. W., Laham, D., & Landauer, T. K. (1999), Automated Essay Scoring: Applications to Educational Technology. In
B. Collis and R. Oliver (Eds.), Proceedings of EDMedia’99, Charlottesville, VA: Association of Computing in Education,
(pp. 939-944).
[10] Hofmann, T., (1999), Probabilistic Latent Semantic Indexing Proceedings of the 22nd International Conference on Research
and Development in Information Retrieval (SIGIR’99).
[11] Callear, D., Jerrams-Smith, J., Soh, V., (2001), Bridging Gaps in Computerised Assessment of Texts, Proceedings of the
IEEE International Conference on Advanced Learning Techniques (ICALT’01), IEEE, 139-149.
[12] Rudner, L. M., Liang, T., (2002), Automated Essay Scoring Using Bayes’ Theorem, The Journal of Technology, Learning,
and Assessment, 1(2), e-journal.
[13] Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003), Latent Dirichlet Allocation. Journal of Machine Learning Research, 3(5),
993-1022.
[14] Rose, C. P., Roque, A., Bhembe, D., VanLehn, K., (2003), A Hybrid Approach to Content Analysis for Automatic Essay
Grading, Proceedings of the 2003 Conference of the North American Chapter of Association of Computational Linguistics
on Human Language Technology, 2, 88-90. Doi: 10.3115/1073483.1073513.
[15] Perez, D., Gliozzo, A., Strapparava, C., Alfonseca, E., Rodrýguez, P., Magnini, B., (2005), Automatic Assessment of
Students’ Free-text Answers Underpinned by the Combination of a BLEU-inspired Algorithm and Latent Semantic Analysis,
in Proceedings of 18th International Florida Artificial Intelligence Research Society Conference, Flairs, AAAI Press.
[16] Kakkonen, T., Myller, N., Sutinen, E., Timonen, J., (2008), Comparison of Dimension Reduction Methods for Automated
Essay Grading, Educational Technology & Society, 11(3), 275-288.
[17] Li Bin, Lu Jun, Yao Jian-Min, Zhu Qiao-Ming, (2008), Automated Essay Scoring Using the KNN Algorithm, International
Conference on Computer Science and Software Engineering, IEEE, Vol. 1, pp 735-738. DOI 10.1109/CSSE.2008.623
[18] Islam M. , Hoque A.S.M.L., (2010), Automated Essay Scoring Using Generalized Latent Semantic Analysis, Proceedings
of 13th International Conference on Computer and Information Technology, pp. 358-363.
[19] Cutrone, L., Chang, M., (2010), Automarking: Automatic Assessment of Open Questions, 10th IEEE International
Conference on Advanced Learning Technologies, IEEE, 143-147.
[20] Tinoco, L. C., Fox, E., Ehrich, R., Fuks, H., (1996), QUIZIT: An Interactive Quiz System for WWW-based Instruction, 7 th
Proceedings of the Symposium of Educational Technology.
[21] Tinoco, L. C., Fox, E. A., Barnette, N. D., (1997), Online Evaluation in WWW-based Courseware, Proceedings of the 28 th
SIGCSE technical Symposium on Computer Science Education, 29(1), 194-198. Doi:101145/268085.268156.
[22] Gardner, L., (2001), Computer Supported Learning. A Large-Scale, Web-based Learning and Assessment System to Support
Flexible Education, Proceedings of the 34th Hawaii International Conference on System Sciences, Vol 1.
[23] http://www.webassign.net/
[24] Xingbao Li, Yunming Wu, (2007), Design and Development of Online Examination and Evaluation System based on B/
S Structure, Proceedings of International Conference on Wireless Communications, Networking and Mobile Computing,
IEEE, 6223-6225.
[25] Li Jun, (2009), Design of Online Examination System Based on Web Service and COM, The 1st International Conference
on Information Science and Engineering, pp. 3276-3279.
[26] Moe Moe Aye and Mie Mie Thet Thwin, (2008), “Mobile Agent Based Online Examination System”, Proceedings of
International Conference on Electrical/electronics, Computer, Telecommunication and Information Technology Association,
1, 193-196.
[27] Gawali, R. D., Meshram, B. B., (2009), “Agent-Based Autonomous Examination Systems”, International Conference on
Intelligent Agent and Multi-agent Systems, IEEE, pp 1-7.
[28] Jackson, D., Usher, M., (1997), “Grading Student Programs using ASSYST”, In Proceedings of the 28 th SIGCSE Technical
Symposium, pp. 335-339. dx.doi.org/doi:10.1145/268084.268210.
 Review of Computerized Evaluation tools in Education 117

[29] Daly C., (1999), RoboProf and an Introductory Computer Programming Course, Proceedings of the 4 th Annual SIGCSE/
SIGCUE on Innovation and Technology in Computer Science Education June 27 -30, 1999, Krakow Poland, Pages 155-
158. DOI : 10.1145/384267.305904.
[30] http://www.boss.org.uk/
[31] Leal, J.P., Silva, F., (2003), Mooshak: AWeb-based Multi-site Programming Contest System, Journal of Software- Practice
and Experience, John Wiley and Sons, 33(6), 567-581.DOI:10.1002/spe.522.
[32] García-Mateos, G., Fernández-Alemán, J., L., (2009), A Course on Algorithms and Data Structures Using On-line Judging,
Proceedings of the 14th annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education
41(3), 45-49. Doi: 10.1145/1505496.1562897.
[33] Montoya-Dato, F. J., Fernández-Alemán, J. L., and García-Mateos, G., (2009), “An Experience on Ada Programming
using On-line Judging,” in Proc. 14th International Conference on Relaible Software Technologies, pp. 75-89.Doi: 10.1007/
978-3-642-01924-16.
[34] Douce et al., (2005), A Technical Perspective on ASAP - Automated System for Assessment of Programming. in Proceedings
of the 9th Computer Assisted Assessment Conference.
[35] Mandal, A. K., Mandal, C., Reade, C. M. P., (2006), Architecture of an Automatic Program Evaluation System, In CSIE
Proceedings.
[36] Raadt. M., Dekeyser, S., Tien Yu Lee, (2007), A Student Employing Peer Review and Enhanced Computer Assisted
Assessment of Querying Skills, Informatics in Education, 2007, 6(1), 163-178.
[37] Gaoyan Zhang, Haifeng Ke, (2010), SQL Paperless Examination System Design, 2010 Second International Conference
on Computer Modeling and Simulation, IEEE, 3, 475-478. DOI 10.1109/ICCMS.2010.468.
[38] Farrow, M., King, P. J. B., (2008), Experiences with Online Programming Examinations, IEEE Transactions on Educatio,
51(2).
[39] Skupas, B., (2010), Feedback Improvement in Automatic Program Evaluation Systems, Informatics in Education, 9(2),
229-237.
[40] Fernández Alemán, J. L., (2011), Automated Assessment in a Programming Tools Course, IEEE Transactions on Education,
54(4), 576-581.
[41] Yuan Zhenming, Zhang Liang, Zhan Guohua, (2003), “A Novel Web-Based Online Exmination System For Computer
Science Education”, IEEE, Proceedings of International Conference on Frontiers in Education.
[42] Liang Zhang, Yue-ting Zhuang, Zhen-ming Yuan, Guo-hua Zhan, (2006), “A Web-based Examination and Evaluation
System for Computer Education”, IEEE, Proceedings of the Sixth International Conference on Advanced Learning
Technologies.
[43] Hollingsworth, J. (1960), Automatic Graders for Programming Classes. Commun. ACM 3, 10, 528-529. DOI: 10.1145/
367415.367422.
[44] ENGLISH, J. (2004), Automated Assessment of GUI Programs using JEWL. Proceedings of the 9th Annual SIGCSE
Conference on Innovation and Technology in Computer Science Education. 131-141.
[45] FORISEK, M., (2006), On the Suitability of Programming Tasks for Automated Evaluation, Informatics in Education,
2006, 5(1), 63-76 63.

View publication stats