2021 3rd International Conference on Advances in Computing, Communication Control and Networking (ICACCCN)
AI Based Proctoring
2021 3rd International Conference on Advances in Computing, Communication Control and Networking (ICAC3N) | 978-1-6654-3811-7/21/$31.00 ©2021 IEEE | DOI: 10.1109/ICAC3N53548.2021.9725547
Shakti Priya Saurav
Computer Science and Engineering Computer Science and Engineering
Dronacharya Group Of Institutions Dronacharya Group Of Institutions
Greater Noida, Uttar Pradesh Greater Noida, Uttar Pradesh
hello@shaktisaurav.com topranaypandey@gmail.com
Bipin Pandey
Computer Science and Engineering
Dronacharya Group Of Institutions
Greater Noida, Uttar Pradesh
hodcse@gnindia.dronacharya.info
Abstract --- From the past year, Online Examination has
become popular in all the educational fields due to covid19 and
its flexibility. However, the institutions and community confront
a big difficulty in terms of proctoring methods, as it is extremely
difficult to administer cheating-free online exams. Here we
present techniques and tools through which the proctor need not
to be present throughout the exam. This is based on neural
networks and machine learning. Our AI-based model will be able
to detect any unfair means in an examination. Our experiments
proved that the proposed system is better than the existing ones.
Keywords --- Neural Network, Convolutional Neural Networks
(CNN), Deep Learning, Proctoring, Covid19
I. INTRODUCTION
Exams are a crucial part of every educational programme,
whether it be online or offline. Cheating is more likely to occur
in online tests, thus detecting and preventing it is critical.
AI-based proctored Online examinations are gaining more
importance due to their flexibility, security, and comfort. This
is not only limited to college and school examinations but can
also help online certification to be more valid. Instead of
conduction examination in offline mode with the use of our AI
proctored online examination, we can save much human effort
and time while also giving comfort to the examinee. According
to the UNESCO Educational Disruption and Response to
covid19 pandemic, many of the governments are closing down
educational institutions and are significantly shifting to online
activity and remote working impacting over 85% of the world’s
student population. Technological advancements can have
remote exams as an alternative to in-person proctoring.
Moreover, whether AI-Based Proctoring is equal in function to
physical proctoring is questionable, but this is the future of the
examination system as Academics have shifted to online mode
and their number will be increasing post covid19 also. This
presents a significant difficulty, not just in terms of learning but
also in terms of testing.
Conducting examinations without committing any errors is
a huge challenge that must be overcome. A faculty member can
physically supervise students with all of their senses during
offline proctored tests at the centres. They can readily detect
student movement or any sound in the room, ensuring a smooth
examination. Because the teacher is not physically present at
the location, online tests limit oversight. The AI-based
proctoring system should allow for easy movement and sound
detection on all students at the same time.
ISBN: 978-1-6654-3811-7/21/$31.00 ©2021 IEEE
Authorized licensed use limited to: United Arab Emirates University. Downloaded on April 27,2022 at 07:54:24 UTC from IEEE Xplore. Restrictions apply.
Pranay Pandey Shubham Kr Sharma
Computer Science and Engineering
Dronacharya Group Of Institutions
Greater Noida, Uttar Pradesh
amshubhamsharma@gmail.com
Rajat Kumar
Department of Computer Science and Engineering
Dronacharya Group Of Institutions
Greater Noida, Uttar Pradesh
rajat.kumar@gnindia.dronacharya.info
While our AI-based Proctoring System invites, many
detection mechanisms, like Face detection, Noise detection,
Gaze Tracking, mouth movement detection, mobile or other
device detection, head pose detection to find where the
examinee is looking, change of tabs detection, together with
these all can facilitate the fairness of examination and add
credibility and integrity to it.
II. LITERATURE REVIEW
In [1] published by Asep Hadian S. G and Yoanes Bandung,
followed a unique approach. A very large data-set of images is
used to train and identify the user in low light and general
scenarios. It was not designed for first time setup for an online
examination.
The main focus of Aiman Kiun's presentation [2] was on
fraud discovery in doing the video recordings during exams by
using CNN, with image classification models based on the use
of Rectified activation units (RAU).
In [3] by Sanjana Yadav and Archana Singh, where they
used computer vision for information extraction for object
detection. Image is checked with a matching algorithm using
the methods such as re-scaling, filtration and binarization.
Chamfer distance transformation.
In [4] by Swathi Prathish, He proposed four major channels
of detection, System usage analysis, Video analysis, Audio
analysis, Inference system. This evaluates multiple faces
detection.
In [5] by N.L Clarke and P. Dowland, presents a model to
facilitate remote and electronic proctoring during examinations
of students. They used translucent recognition to give a non-
disruptive and persistent authentication of student’s identity.
In [6] by Vahid Kazemi and Josephine Sullivan, proposed
Face Arrangement for a solitary picture showing how utilizing
fitting priors misusing the design of picture information assists
with effective component choice. Moreover, they examinee the
accurate predictions on the quantity of available training data.
In [7] by A.T. Awaghade et.al, proposed that all the
contributions to measure and gauge the assortment of occasions,
practices and examples ordinarily connected with cheating.
III. PROPOSED METHODS
In this case, we propose developing a comprehensive system
with numerous detection mechanisms capable of detecting any
unethical behavior.
Before the start of Online Exam
The examinee will have to verify himself before the test. At
this stage, the system will collect all the facial data of the
610
 2021 3rd International Conference on Advances in Computing, Communication Control and Networking (ICACCCN)
examinee. Then the system will extract all the data and use it examination has constrained resource like web-cam so the
for the proctoring. TensorFlow lite was most suitable for this situation.

• The TensorFlow model gives ~8 FPS and the landmark

Exam Phase
prediction step takes around 0.06 seconds
The examinee takes the exam while our technology • Dlib gives ~12 FPS and the landmark prediction step
monitors it in real time for unfair means detection. To capture takes around 0.006 seconds
the examinee's audio-visual signals, our system will only
require two sensors (webcam and microphone). Dlib looks like better option but the overall result with
Five components will process the audio-video streams to accuracy was far better with Tensor Flow.
extract middle-level information. These traits are then
combined to create our main classifier. Part-dependent features
are mean and (SD) standard deviation within a pane, and B. Eye Tracking (v2)
features based on component correlation are among the features We used a facial key-points detector to detect eyes in real-
generated. time. For this we used a already trained network in dlib library
which is capable of detecting 68 key points. It takes a
rectangular object of the dlib as input which is the coordinates
Vision Based Functionality used
of a face.
• Gaze tracking.
• Find if the candidate opens his mouth.
• Find any instances of mobile or other devices.
• Head pose estimation.
• Face detection

A. User Verification & Facial Landmarks (v1)

A proctoring system should be able to continuously verify
if the examinee is the same person who he claims to be. There
are various methods for continuous user verification, we used
facial recognition in our system.

Figure 3. Dlib facial keypoints. [9]

C. Mouth Opening Detection (v3)

We proposed this system to check if the examinee opens
his/her mouth to say something during the exam. This was

Figure 1. Metric for OpenCV DNN
We compared different face detection models. OpenCV's
DNN module provides best result (Figure 1). This module
employs the ResNet-10 architecture and is based on the Single
Shot Multibox detector. (Figure 2)
similar to eye detection. Again we used the same dlib facial
key-points and for this task examinee is required to sit straight
and the distance between the key-points (5 outer pairs and 3
inner pairs) is checked for 100 frames. If the examinee opens
his mouth the distances of the points increases and if the
increase in distance is more than a fixed value for at least three
outer pairs and two inner pairs then vector is generated.
D. Mobile & Device Detection (v4)
We used Tensorflow, OpenCV, and wget-python. We used
instance segmentation which can distinguishing between
different labels and also able to distinguish between multiple
objects of that label. We used [8] YOLOv3 already trained
model to classify 80 objects. It uses Darknet-53 which has 53
convolutional layers and is 1.5 times faster than previous
versions.

Figure 2. SSD.

At starting we used dlib’s facial landmarks model but the

expected result was not obtained so we used facial landmark
detection based on convolution neural network. This model
was built using Tensor Flow. As our system for online

611

Authorized licensed use limited to: United Arab Emirates University. Downloaded on April 27,2022 at 07:54:24 UTC from IEEE Xplore. Restrictions apply.
 2021 3rd International Conference on Advances in Computing, Communication Control and Networking (ICACCCN)
Figure 4. YOLOv3 runs much faster than other detection methods with a
comparable performance.
E. Head Pose Detection (v5)
This is proposed to find where the user is looking. This can
be very beneficial to detect if user is trying to cheat by looking
at some additional display or devices.
We tried implementing this using OpenCV and
TensorFlow. We used a Caffe (Convolutional Architecture for
Fast Feature Extracting) model of OpenCV’s DNN module as
it provided the most accurate results (Figure 5).
The first step was to detect the face and for this we used the
same facial landmark detection based on convolution neural
network. We took 3D coordinates of the facial landmarks and
tried to estimate the rational and translational vectors at the
center tip(nose).
Figure 5. Caffe Model Accuracy.
After getting the required vectors we projected those 3D
points on a 2D surface that is our image.
F. Active Window Detection and Tab Switching (v6)
The cheating from the internet from the same device is one
of the most frequent unfair means among e-learners. is is
proposed to find where the user is looking at. For this we used
JavaScript as we wanted this to be a system without
612
Authorized licensed use limited to: United Arab Emirates University. Downloaded on April 27,2022 at 07:54:24 UTC from IEEE Xplore. Restrictions apply.
downloading any additional software. Using JS we achieved
this (Figure 6) and store a count of this practice and a timer along
with that. Doing this will generate the notification for examinee
and with repeat attempt the exam will be auto submitted or the
examinee will be disqualified.
G. Audio (v7)
Audio is one of the main sources to monitor during an
online examination. That’s why we added this to our AI Based
Proctoring system. We took the audio of the microphone and
converted to text using Google Speech Recognition. Further
using NLTK, we remove the stop-words from the text file.
Question Paper was also processed in the same way and then
common words are found out and reported.
IV. UNFAIR MEANS DETECTION
After the output of all basic components (v1,v2,v3,v4,v5,v6,v7)
where all the vectors have same sampling rate, i.e; one element
per frame, We now at this stage try to further classify the
output as Unfair Means or Fair Means. The vectors ( v 1 & v6 )
are used directly to provide a unfair means decision, and other
five vectors be utilized for extracting features, which will be
used for classifier to make continuous decisions on unfair
means. While v7 output is used to check whether the same
words are used by the examinee or not. Depending on that it
can determine this as unfair means.
Feature Extraction
Because unfair means occur over a period of time, features
must be defined according to the temporal window. For the
purpose of feature extraction, we specify this with a set time in
seconds. We create a variety of segments. We can convert this
labelling to the label of each segment if we manually label fair
vs unfair for all gathered frames at each second. That is, the
majority of per-second result labels within a segment decide the
binary outcome of a segment.
Figure 6. Unfair Means Detection.
 2021 3rd International Conference on Advances in Computing, Communication Control and Networking (ICACCCN)
V. CONCLUSION
The purpose of this study is to describe an AI-based system for
online exam proctoring in order to maintain academic integrity.
The system is simple and convenient to use from the
perspective of the test taker, as it only requires two inexpensive
cameras and a microphone. Using the captured films and audio,
we extract low-level information from seven essential
components: user authentication, eye tracking, speech
recognition, and so on. These data are then analyzed in a
temporal window to get decision-making traits, which are then
used to detect cheating. detection.
REFERENCES
[1] “A Design of Continuous User Verification for Online Exam Proctoring
on M-Learning”, Hadian S. G. Asep; Yoanes Bandung, 2019
International Conference on Electrical Engineering and Informatics
(ICEEI), 9-10 July 2019.
[2] “Fraud detection in video recordings of exams using Convolutional
Neural Networks”, Aiman Kuin, University of Amsterdam, June 20,
2018.
613
Authorized licensed use limited to: United Arab Emirates University. Downloaded on April 27,2022 at 07:54:24 UTC from IEEE Xplore. Restrictions apply.
[3] “An Image Matching and Object Recognition System using Webcam
Robot”, Sanjana Yadav; Archana Singh, 2016 Fourth International
Conference on Parallel, Distributed and Grid Computing (PDGC), 22-24
Dec. 2016.
[4] “An Intelligent System For Online Exam Monitoring”, Swathi Prathish,
Athi Narayanan S, Kamal Bijlani, 2016 International Conference on
Information Science (ICIS), 12-13 Aug. 2016
[5] “e-Invigilator: A biometric-based supervision system for e-Assessments”,
N.L Clarke, P. Dowland, S.M. Furnell International Conference on
Information Society (iSociety 2013), 24-26 June 2013.
[6] “One Millisecond Face Alignment with an Ensemble of Regression
Trees”, Vahid Kazemi,Josephine Sullivan, 2014 IEEE Conference on
Computer Vision and Pattern Recognition,23-28 June 2014.
[7] “Online Exam Proctoring System”, A.T. Awaghade, D. A. Bombe, T. R.
Deshmukh, K. D. Takawane, International Journal of Advance
Engineering and Research Development (IJAERD) “E.T.C.W”, January
-2017, e-ISSN: 2348 - 4470, print-ISSN: 2348-6406.
[8] “YOLO3 Real-Time Object Detection” by Joseph Redmon, Ali Farhadi
published by Cornell University on Apr 8, 2018
[9] “Facial landmarks with dlib, OpenCV, and Python” by Adrian
Rosebrock on Apr 3, 2017