Page 1 of 8 - Cover Page Submission ID trn:oid:::1:2790395308

Sudam Rohanadeera
Integration of AI-based Teaching Assistant BOT in Academic
Courses to Facilitate Learning among Large Student Groups
Quick Submit

Quick Submit

USJP - 2018

Document Details

Submission ID

trn:oid:::1:2790395308 6 Pages

Submission Date 1,697 Words

Dec 22, 2023, 6:20 AM GMT+5:30

9,927 Characters

Download Date

Dec 22, 2023, 6:22 AM GMT+5:30

File Name

Extended_Abstract-ver2.docx

File Size

22.8 KB

Page 1 of 8 - Cover Page Submission ID trn:oid:::1:2790395308

 Page 2 of 8 - AI Writing Overview Submission ID trn:oid:::1:2790395308

How much of this submission has been generated by AI?

*18%
Caution: Percentage may not indicate academic misconduct. Review required.

It is essential to understand the limitations of AI detection before making decisions

about a student's work. We encourage you to learn more about Turnitin's AI detection
capabilities before using the tool.
of qualifying text in this submission has been determined to be
generated by AI.

* Low scores have a higher likelihood of false positives.

Frequently Asked Questions

What does the percentage mean?

The percentage shown in the AI writing detection indicator and in the AI writing report is the amount of qualifying text within the
submission that Turnitin's AI writing detection model determines was generated by AI.

Our testing has found that there is a higher incidence of false positives when the percentage is less than 20. In order to reduce the
likelihood of misinterpretation, the AI indicator will display an asterisk for percentages less than 20 to call attention to the fact that
the score is less reliable.

However, the final decision on whether any misconduct has occurred rests with the reviewer/instructor. They should use the
percentage as a means to start a formative conversation with their student and/or use it to examine the submitted assignment in
greater detail according to their school's policies.

How does Turnitin's indicator address false positives?

Our model only processes qualifying text in the form of long-form writing. Long-form writing means individual sentences contained in paragraphs that make up a
longer piece of written work, such as an essay, a dissertation, or an article, etc. Qualifying text that has been determined to be AI-generated will be highlighted blue
on the submission text.

Non-qualifying text, such as bullet points, annotated bibliographies, etc., will not be processed and can create disparity between the submission highlights and the
percentage shown.

What does 'qualifying text' mean?

Sometimes false positives (incorrectly flagging human-written text as AI-generated), can include lists without a lot of structural variation, text that literally repeats
itself, or text that has been paraphrased without developing new ideas. If our indicator shows a higher amount of AI writing in such text, we advise you to take that
into consideration when looking at the percentage indicated.

In a longer document with a mix of authentic writing and AI generated text, it can be difficult to exactly determine where the AI writing begins and original writing
ends, but our model should give you a reliable guide to start conversations with the submitting student.

Disclaimer
Our AI writing assessment is designed to help educators identify text that might be prepared by a generative AI tool. Our AI writing assessment may not always be accurate (it may misidentify
both human and AI-generated text) so it should not be used as the sole basis for adverse actions against a student. It takes further scrutiny and human judgment in conjunction with an
organization's application of its specific academic policies to determine whether any academic misconduct has occurred.

Page 2 of 8 - AI Writing Overview Submission ID trn:oid:::1:2790395308

 Page 3 of 8 - AI Writing Submission Submission ID trn:oid:::1:2790395308

Integration of AI-based Teaching Assistant BOT in Academic Courses to Facilitate Learning

among Large Student Groups

Page 3 of 8 - AI Writing Submission Submission ID trn:oid:::1:2790395308

 Page 4 of 8 - AI Writing Submission Submission ID trn:oid:::1:2790395308

Abstract- Integration of AI-based Teaching Assistant BOT in Academic

Courses to Facilitate Learning among Large Student Groups
S. L Rohanadeera, K. P. Hewagamage
When expanding the intake size of a higher education institution, maintaining educational
quality requires proportional increases in both physical and human resources. However,
augmenting human resources, particularly competent staff, poses greater challenges than
expanding physical infrastructure. Insufficient academic staff, including teaching assistants,
directly impacts student interaction and engagement, leading to a decline in educational quality.
To address this, our project introduces an AI-based Teaching Assistant BOT designed to
facilitate student learning through conversational interactions, mitigating the impact of limited
human resources. The deployment of such TA BOTs is anticipated to alleviate the need for an
excessive number of teaching assistants in high-enrollment academic courses.
Students commonly encounter difficulties comprehending new subjects, necessitating one-on-
one support to prevent loss of motivation and disengagement from the course. In large courses,
the traditional ratio of one teaching assistant to 15-25 students limits their ability to provide
personalized assistance promptly. The TA BOT serves as a real-time online alternative,
offering immediate support tailored to individual student needs.
Recent advancements in Conversational AI tools, exemplified by Chat GPT from OpenAI and
Bard from Google, have gained popularity for online subject exploration. However, these tools
rely on publicly available knowledge on the web, providing answers in a context-independent
manner and often yielding different responses to the same question. Hence, these tools could
provide limited assistance to students since they do not depend on the course content, syllabi,
or references in the subject when answering questions in a particular educational context. In
contrast, the TA BOT leverages a Large Language Model (LLM), designed to infer course
resources as primary references, ensuring contextually relevant responses. By aligning with
course content, syllabi, and references, the TA BOT minimizes inaccuracies and guides students
along the correct learning path.
Development of the TA BOT utilized 'gpt-3.5-turbo,' a potent LLM from OpenAI, fine-tuned
for conversation using the using course resources of a Software Engineering course. Employing
the Retrieval Augmented Generation (RAG) methodology, supported by the LlamaIndex
framework, we created a comprehensive dataset using the reference book of a Software
Engineering course. The TA BOT, adopting the persona of a Teaching Assistant, delivers
contextual interactions based on subject matter questions while prohibiting academic offences
or malpractices.
Even in cases where the fine-tuned knowledge lacks an answer, the TA BOT can provide
responses beyond the dataset, showcasing its versatility. Preliminary tests confirm the TA
BOT's ability to furnish precise, high-quality answers to Software Engineering-related queries.
Positive student feedback underscores its potential to enhance the educational experience. The
tool is publicly accessible for experimentation at http://chat.ucsc.cmb.ac.lk, offering a
promising solution to sustain educational quality amidst resource constraints and escalating
student demands in higher education institutes.

Page 4 of 8 - AI Writing Submission Submission ID trn:oid:::1:2790395308

 Page 5 of 8 - AI Writing Submission Submission ID trn:oid:::1:2790395308

Keywords: - AI, Retrieval Augmented Generation, TA BOT, Teaching Assistant BOT,

Teaching Assistants

Page 5 of 8 - AI Writing Submission Submission ID trn:oid:::1:2790395308

 Page 6 of 8 - AI Writing Submission Submission ID trn:oid:::1:2790395308

Introduction
The introduction of artificial intelligence (AI) into academic settings is a sign of a new age in
education brought about by the rapid growth of technology. This extended abstract investigates
how using an AI-based Teaching Assistant BOT in academic courses can be transformative in
addressing the issues brought on by growing student populations. The traditional strategy of
increasing physical and human resources—particularly in hiring and maintaining qualified
teaching assistants—faces challenges as higher education institutions struggle to accommodate
increasing numbers of students.
The main issue is how a lack of academic personnel affects student participation and interaction,
which has a direct effect on the standard of education in an education institute. Our research
offers a technology based approach, know as, the AI-based Teaching Assistant BOT. With the
goal of fostering conversational exchanges between students, this intelligent BOT offers
personalized real-time support and advice to meet each student's real learning needs. By
applying this approach, we hope to reduce the burden on the requirement of human teaching
assistants and reduce the deterioration in the quality of education when taking academic courses
by a large number of students.
In large courses, students frequently struggle to understand new material, and the individualized
help is necessary to keep them from becoming disengaged in the course. Providing one-to-one
support is not practical due to number of students and human teaching assistants in the course.
At the same time, human teaching assistants cannot facilitate students on demand and they may
be available during the normal working hours. On other hand, TA-BOT which considers the
main resources as the course materials, syllabi, and references, could become more effective
facilitator providing the interaction at anytime when the student follows the course. We believe
that this approach will have a drastic impact on the student engagement in the course. In this
project, we have considered the subject, Software Engineering, when developing the teaching
assistant BOT to evaluate the conversation interaction when the student follows the course.
During the last two years, several Conversational AI tools have gained popularity to provide
assistance when learning a new subject, since students face the difficulties to understand context
based on their prior knowledge about the subject. Teaching Assistant BOT has been developed
using a Large Language Model (LLM) and it is fine-tuned for specific academic contexts of
learning a subject. Unlike common tools to facilitate students, TA BOT aligns with course
materials to minimize the misunderstanding to guide the students in the correct learning path.
This extended abstract presents the developing model of TA-BOT considering the teaching
materials as the core reference on top of the Large Language Model. Authors also discuss the
outcome of evaluation during the preliminary testing, followed by the discussion and
conclusions of the outcome. Hence, this research presented here contributes to the ongoing
discourse on AI in education and showcases the potential of technology to maintain the quality
education in a large course.

Page 6 of 8 - AI Writing Submission Submission ID trn:oid:::1:2790395308

 Page 7 of 8 - AI Writing Submission Submission ID trn:oid:::1:2790395308

Materials and Methods

Pre-processing of data: The main objective of data pre-processing is to transform raw data
into understandable and usable for further processing. This involves noise reduction, treating
duplicates, etc.
The contents of the first nine chapters of the course book ‘Software Engineering: Seventh
Edition - Ian Sommerville’ are the of our project. After carefully investigating the data we found
some noisy data which includes sections Further Reading, Website and Exercises; between
every chapter.
To remove the noise, we manually split the document with the aid of an online tool for pdf
splitting by setting the page ranges that does not includes the noisy content.
OpenAI Finetuning dataset generation: A typical OpenAI finetuning dataset which consists
(but not limited to) a set questions and answers formally known as ‘query’ and ‘response’. To
generate this dataset, we employed a methodology called Retrieval Augmented Generation
(RAG) aided by the open-source framework Llmaindex (Anon., n.d.).
Fine Tuning of an OpenAI LLM: Fine tuning of a LLMs, can make it perform better for
specific applications (Anon., n.d.). the process of fine-tuning played a pivotal role in optimizing
the performance of the AI-based Teaching Assistant BOT. Leveraging OpenAI's 'gpt-3.5-turbo,'
which had been pre-trained on extensive text data, we employed fine-tuning to enhance its
effectiveness in an educational context. This approach surpassed the limitations of traditional
prompting, allowing for higher quality results by training the model on a multitude of examples
that couldn't fit within a conventional prompt. The advantages of fine-tuning extended beyond
improved performance, encompassing token savings due to shorter prompts and lower-latency
requests. In the process of developing TA-BOT, training data were prepared using course
materials and basic student interaction in order to train the model and carry out the initial
assessment. We follow the iterative approach to optimize the model which is the basis of this
educational application. Our initial assessment demonstrates that revised model performed
better to provide user friendly interaction.

Page 7 of 8 - AI Writing Submission Submission ID trn:oid:::1:2790395308

 Page 8 of 8 - AI Writing Submission Submission ID trn:oid:::1:2790395308

Results, Discussion and Conclusion/s

Results
The first version of TA BOT was developed using 'gpt-3.5-turbo,' as the base LLM. It was able
to provide responses to student questions during the interaction similar to subject matter expert
in the Software Engineering to build the confidence of students. We experimented the tool
hosting it at http://chat.ucsc.cmb.ac.lk and requesting students of UCSC who are following
software engineering courses to provide the feedback. We requested the students to interact
with TA BOT considering real problems that they face when studying the subject. We invited
all students to provide the feedback using online form (xxxx) to identify both positive and
negative sides of TA BOT. Interesting, majority of feedback given by the students who are
following the software engineering courses, are positive and they were very happy to use this
type of tool within the learning management system. Hence, this encouraged us to further
develop the tool to make it more effective facilitating tool for the students.

Discussion
The success of TA BOT depends on the methodology that we followed to fine tune it based on
with course content, syllabi, and references to provide effective interaction. This highlights the
importance of developing the Generative AI tools for education considering the context of a
particular subject matter. They could provide better answers than common tools such as
ChatGPT, Bard, since responses could be directly linked to the course materials. Providing the
tailored support to students will improve the quality of an academic course. Since this tool is
designed considering one subject domain, it is too early to justify whether we can follow similar
approach for all subjects.

Conclusion/s
Our main objective is to provide better interaction with respect to subject matter when students
encounters main difficulties. Students feel TA BOT is a subject matter expert and it could help
better than human teacher who usually takes more time to reply a student queries in the LMS.
The scalability of providing such a service to large number of students depends on the software
architecture of TA BOT and this tool has not evaluated large number of concurrence
interactions. Hence, we need further refinements and continuous adaptation will be required to
address current limitations and ensure the success of the Teaching Assistant BOT. We believe
that this study will opens dialog with multi-disciplinary community the importance of AI in
education tools to provide the quality education.

Page 8 of 8 - AI Writing Submission Submission ID trn:oid:::1:2790395308