2021 4th International Seminar on Research of Information Technology and Intelligent Systems (ISRITI)

Implementation of Chatbot on University Website

Using RASA Framework
Lia Fauzia
Cryptographic Engineering
Politeknik Siber dan Sandi Negara
Bogor, Indonesia
lia.fauzia @student.poltekssn.ac.id
2021 4th International Seminar on Research of Information Technology and Intelligent Systems (ISRITI) | 978-1-6654-0151-7/21/$31.00 ©2021 IEEE | DOI: 10.1109/ISRITI54043.2021.9702821
Raden Budiarto Hadiprakoso
Cryptographic Engineering
Politeknik Siber dan Sandi Negara
Bogor, Indonesia
raden.budiarto@poltekssn.ac.id
Girinoto
Cryptographic Engineering
Politeknik Siber dan Sandi Negara
Bogor, Indonesia
girinoto@poltekssn.ac.id

Abstract— Chatbots are increasingly being utilized to help
human performance to boost communication ease and provide
better and faster services. A chatbot was built to make it easier
to give better and more conveniently available information
services 24 hours a day, seven days a week. The suggested
chatbot design is constructed using the Rasa framework and is
based on the Dual Intent and Entity Transformer (DIET). DIET
is a multi-tasking transformer architecture that is both
advanced and lightweight. The chatbot will be implemented on
the “Politeknik Siber dan Sandi Negara” website, focusing on
addressing questions about new student admittance. The
chatbot is built with Docker and put as a Chat Widget on the
website. The dataset utilized to train the algorithm combines
past chat data and data from university social media. The built-
in chatbot is evaluated using accuracy metrics and F1 scores.
Model evaluation metrics and functionality tests are used in the
evaluation. Testing with evaluation results in a precision value
of 0.99. a recall value of 1.0, and an F1 score of 0.99. While the
functioning test percentage is 90.625 %.
Keywords—chatbot, rasa framework, deep learning, DIET,
transformer model
I. INTRODUCTION
Many web-based services are available in today's world,
including e-commerce, entertainment, virtual help, and many
others. Currently, there is a significant increase in the world
of online services, where everything is now connected to the
web. The ease of finding information is very much needed at
this time. Various institutions and agencies certainly want to
provide convenience in finding information for people
interested in it. Several forms of customer service are
accessible, such as live chat assistance and phone services.
However, all such human-to-human support services require
time to respond to consumers' queries. As the number of
clients grows, so does the waiting time, resulting in low client
satisfaction [1].
The definition of natural and instinctive connection modes
is an essential objective in the science of Human-Computer
Interaction. Specifically, several efforts have been devoted to
developing frameworks for communicating with the customer
in a specific language. Computer-based chatbots are becoming
increasingly well-known as an intuitive and effective open
framework between humans and computers. A chatbot is an
artificial material designed to mimic an intelligent
conversation with human accomplices using natural language
[2]. Currently, chatbots are used by many online users to get
access to information or learning bases, as well as to conduct
non-specific dialogues. Since chatbots imitate real humans,
Artificial Intelligence (AI) techniques are used to create them.
One of the techniques in AI is deep learning which imitates
the human brain [3].
Services in the form of contact persons are frequently
included to support the convenience of providing information,
particularly regarding selecting new student admissions so
that that prospective student can ask questions regarding the
desired information, either via telephone numbers or chat
services such as live chat. Artificial intelligence in the form of
chatbots has primarily supplanted the existence of contact
personnel nowadays. If an administrator runs this service, it is
ineffective if many questions are entered at the same time.
Furthermore, the chatbot service has various advantages, such
as not having an administrator and delivering nonstop service
24 hours a day, seven days a week. With this service, the
chatbot will always offer answers in real-time and will fulfill
the availability aspect.
There have been a number of research conducted on
chatbot creation. The researcher [4] [5] uses Artificial
Intelligence Markup Language (AIML) and Latent Semantic
Analysis to create a chatbot that delivers a question and
answer service in the study. AIML will be used to respond to
template-based and general questions such as greetings and
general questions. Using the RASA framework, researchers
[6] created a university chatbot. They employ the LSTM
mode, which is already supported by the RASA framework.
One disadvantage of LSTM is that they demand more
hardware resources to train and run than attention-based
models. Because LSTMs primarily use sequential processing
over time, long-term information must pass through all cells
sequentially before reaching the current processing cell.
Attention model, one of the deep learning models, has
been widely employed in natural language processing. By
generating a coding representation of what significant material
is, attention may automatically learn what is considered
relevant [7]. Integrating attention into neural architectures has
been shown to increase performance significantly [8]. The
transformer is the first transduction model entirely based on
attention, with multi-headed self-attention replacing the most
often used repeating layer in encoder-decoder architecture.
Transformers can be trained substantially quicker than designs
based on recurrent or convolutional layers for NLP task [9].
Unlike the previous paper, in this paper we introduce a
chatbot with an attention-based architecture. The chatbot will
be created with Dual Intent and Entity Transformer (DIET), a
transformer-based model classifier [7]. DIET transformers are
present in user utterances as tokens to aid with intended
categorization and entity extraction. DIET advances the
sophistication of complex multi-domain NLU datasets and

978-1-6654-0151-7/21/$31.00
rized licensed ©2021
use limited to: SVKM¿s NMIMS Mukesh IEEE
Patel 373
School of Technology Management & Engineering. Downloaded on April 26,2022 at 15:21:16 UTC from IEEE Xplore. Restrictions a
 achieves equally high DIET is not only substantially faster to
train and outperforms large-scale pre-trained language
models.
The proposed chatbot is developed using the RASA
framework. RASA framework is made up of two parts: Rasa
Core and Rasa NLU [10]. RASA Core is a conversation
engine that decides what to do next. Meanwhile, RASA NLU
is a component of an open-source flavor that supports NLU,
or natural language understanding, as well as intent
categorization and entity extraction [11]. RASA NLU, an
open-source natural language processing module, is
frequently used as a tool to develop a conversational system
[12]. RASA NLU comprises loosely connected modules that
mix a variety of natural language processing and machine
learning libraries into a unified API. The chatbot will be
focused on answering questions regarding the selection of new
student admissions in ‘Politeknik Siber dan Sandi Negara’
(PoltekSSN) called ‘Seleksi Penerimaan Mahasiswa Baru’
(SPMB).
II. METHODOLOGY
This study adopts a research flow regarding chatbots with
adjustments in several parts. There are 4 stages passed in this
research, that are data collection, analysis and design, build
the chatbot, implementation on the web, and testing.
A. Analysis and Design
The analysis and design stages are carried out before the
chatbot creation stage with the aim of making it easier to
create chatbots.
1) Requirement Analysis
Requirement analysis is carried out to dig up
information and describe the data that will be used in
making chatbots. In this process, data is grouped from the
results of data collection that has been done previously.
Data grouping is intended to simplify the process of
entering data on intents, utter chatbot response templates,
stories and actions.
2) Chatbot Design
In making software, it is necessary to have a good
design, so that the software that is built has good
documentation to produce a system that has the needs that
are in accordance with what the user wants.
B. Build The Chatbot
At this stage, a chatbot is made on a framework sense. This
stage uses a chatbot workflow diagram.
1) Simulate Data
This step begins by simulating data based on general
interaction dialogues and grouping each sentence into an
intent topic, utter chatbot response template, and stories.
Intent topics submitted as chatbot features must define the
entities there.
2) Arrange Data to Stories
At this stage set the intent, utter and action on a story
stream. The creation of stories will be placed on
stories.yml.
3) Train NLU and Core
Data collection for training will be taken from the live
chat service installed on the previous live chat website and
additional data to be more extensive and provide better
374
rized licensed use limited to: SVKM¿s NMIMS Mukesh Patel School of Technology Management & Engineering. Downloaded on April 26,2022 at 15:21:16 UTC from IEEE Xplore. Restrictions a
performance. Additional training data will be obtained
from other platforms such as Social media-owned by
PoltekSSN. After all the data is ready, then train the data
using the RASA Trainer.
4) Evaluate Model
Measure and evaluate the model made with precision
score, recall score and F1-score. Evaluation of this model
will be carried out in the terminal with the command
provided by rasa. Then an assessment will be made on the
intent whether the incoming questions will be categorized
in the appropriate intent, so that later they will get the
appropriate answer. This experiment will get the value of
precision, recall, and the F1 score directly.
C. Implementation
This stage is carried out after the chatbot creation stage is
complete. Implementation on the PoltekSSN web will be done
using Docker. Docker is a container virtualization technology
similar to a lightweight virtual machine (VM) [13]. In addition
to using Docker, the implementation will use Heroku. Heroku
is a container-based Platform as a Service (PaaS) cloud that
can be used to deploy, manage, and scale modern applications.
Then the URL address obtained from Heroku will be attached
to a display that can be directly implemented on the website.
D. Testing
This stage analyzes the results of the implementation and
testing of the chatbot application. Tests will be carried out on
the model and functional testing of the chatbot using a Likert
scale based. This Likert scale will have a scale of 4 by using
questions in the form of a checklist. The calculation of this
test is to calculate the average answer based on the scoring of
each respondent's answer.
∑(𝑎 × 𝑏)
× 100% (1)
𝑐×𝑑
Information:
𝑎 = total respondents chose b
𝑏 = respondent's choice score
𝑐 = total respondents
𝑑 = highest score
The overall ideality percentage can be calculated using:
∑𝑥
× 100% (2)
𝑦 ×𝑐 × 𝑑
Information:
𝑥 = total score of respondent's choice
𝑦 = total questions
𝑐 = total respondents
𝑑 = highest score
III. IMPLEMENTATION
A. Build The Chatbot
The following are the steps taken to create an SPMB
chatbot using the Rasa Framework.
1) Simulate Data
The data that has been obtained in data collection will
be categorized into intent topics based on the SPMB
registration information. In general, intent will be divided
into greeting, general, conditional, selection, and random
 categories. The intent will be entered in the nlu.yml and 14 utter_syarat_gigi
domain.yml files. The nlu.yml intent will contain data for 15 utter_syarat_berat
16 utter_syarat_mata
questions that will be trained later. Furthermore, based on 17 utter__random_militer
the incoming data, it will be divided into several more 18 utter_iambot
detailed categories as shown in Table I. 19 utter_noworries

TABLE I. Explanation of Intent Fuction 2) Arrange Data to Stories

Intent Function Amount The next step is to create a conversation flow or
of data
Sapa Categorize conversations in the form of stories. In Stories.yml several conversations flow that
26 might occur in this SPMB chatbot are created. The flow
greetings
goodbye Categorize conversations in the form of
10 of the conversation will be replied with an answer from
saying goodbye utter. In addition to preparing stories, test_stories are also
pendaftar Categorize conversations during SPMB
an_waktu registration
150 prepared, which are similar to stories but filled with
umum_bi Categorize conversations Poltekssn entrance sentences that users might give to bots.
20
aya fees
umum_as Categorizing dorm conversations 3) Train NLU and Core
5
rama
Train or data training performed on the NLU, and
umum_te Categorize exam venue conversations
mpat_tes
16 Core of Rasa is affected by the configuration of the
umum_ju Categorize conversations of majors that can pipeline used. In this SPMB chatbot, the configuration is
45
rusan enter Poltekssn used as shown in Fig 1.
umum_in Categorize the conversation address where
133
formasi the exam
umum_k Categorize conversations quota received
13
uota
umum_ni Categorize minimal entry value
12
lai conversations
umum_k Categorize conversations about contactable
17
ontak contacts
syarat_w Categorize conversations of needs when you
ajib_pem want to enter Poltekssn 18
inatan
syarat_ti Categorize conversations about height
21
nggi requirements
Syarat_gi Categorize conversations about dental
7
gi requirements
Syarat_b Categorize conversations about weight
6
erat requirements
Syarat_m Categorize conversations about eye
19
ata requirements
Random_ Categorize conversations about whether
8
militer Poltekssn is semi-military
iambot Categorize conversations about bots 7
thankyou Categorize conversation thank you 15
random_l Categorize conversations about links
ink_berm 7
asalah

To answer questions for each category of intent, an

action is created as to determine the conversation action.
This action is defined in domain.yml with the name utter
along with the intent. Each utter contains an answer
template for intent questions contained in domain.yml.
Fig 1. Pipeline chatbot SPMB
The list of actions on this SPMB chatbot is in Table II.
There are several components in the pipeline, the first
TABLE II. List of Utter of which are:
Number Utter Name 1. The tokenizer used in the SPMB chatbot pipeline is
1 utter_sapa WhitespaceTokenizer. WhitespaceTokenizer will
2 utter_goodbye
generate a token for each space-separated sequence
3 utter_pendaftaran_waktu
4 utter_umum_biaya of characters.
5 utter_umum_asrama 2. The feautizer in this pipeline is used by
6 utter_umum_tempat_tes
7 utter_umum_jurusan
Regexfeaturizer, LexicalSyntacticFeaturizer, and
8 utter_umum_informasi CountVectorsFeaturizer.
9 utter_umum_kuota 3. Then the next component is pre-trained subword
10 utter_umum_persyaratan
11 utter_umum_kontak embeddings, this component uses BPEmb (Byte-
12 utter_syarat_wajib_peminatan Pair Encoding) using Indonesian.
13 utter_syarat_tinggi

375
rized licensed use limited to: SVKM¿s NMIMS Mukesh Patel School of Technology Management & Engineering. Downloaded on April 26,2022 at 15:21:16 UTC from IEEE Xplore. Restrictions a
 4. For the intent classifier component, DIETclassifier
is used. DIET (Dual Intent and Entity Transformer).
5. Fallback Classifier

Some components on policies:

1. Memoization Policy
2. TEDPolicy
3. MappingPolicy
4. RulePolicy

4) Evaluate Model
An evaluation of the model on the chatbot that has
been made is carried out. Rasa itself facilitates users to
carry out thorough validation and testing using test stories
that have been created by users. Test stories are similar to
stories in training data, but with messages from the user
inserted into them. The command used to perform a
performance test is to use "taste test".
Rasa test will test the trained model using the
previously created test stories. The following below in Fig
Fig 3. Story Confusion Matrix
2, Fig 3, and Fig 4 are the results of the evaluation using
a sense test on the model. Fig 3 is a story confusion matrix table. Similar to the
confusion matrix intent, the difference is that it's not the
intent that's being tested but from the story. The vertical
axis shows the true table, and the horizontal axis shows
the predicted table. For the story confusion matrix table,
all the data tested are appropriate.

Fig 2. Intent Confusion Matrix

Fig 2 is the intent confusion matrix table, the vertical
axis shows the true table and the horizontal axis shows the
predicted table. True table shows the original data that is
in the training data and for the predicted table is the data
predicted by the model. If the same true table and
predicted table meet at the point where they intersect, it
indicates that the model predicts the data according to the
original data entered. Intent classification prediction
works well, there is only one misclassification when
classifying intent, the model predicts the data to enter in
umum_pendaftaran_waktu.
Fig 4. Intent Prediction Confidence Distribution
The following is an image of the intent prediction
confidence distribution table. The numbers below
indicate the number of samples held. The number in the
correct section shows a number up to 500 because the
number of words is close to that number. For the wrong
part, because the wrong value is little or no, the scale
shown is small until the value is below 1. While the
numbers 0.49 to 1 indicate the percentage value of
predictive confidence that the model predicts correctly.
The bar itself is around 1 because the average obtained is

376
rized licensed use limited to: SVKM¿s NMIMS Mukesh Patel School of Technology Management & Engineering. Downloaded on April 26,2022 at 15:21:16 UTC from IEEE Xplore. Restrictions a
 close to 1.0. The following in Fig 5 is the result of the Chatbot-Widget template, in constant.js which can then be
intent error that occurred. accessed when opening index.html.

Fig 5. intent_error.json
The sentence “saya ingin mendaftar masuk” is meant
to go on umum_informasi intent but is predicted to go on
umum_pendaftaran_waktu intent. This will have an
impact on the confidence value of the intent prediction so
that it gives an answer that is not in accordance with what
was intended by the person asking. The following in
Table is the test result value of precision, recall, and F1
score of the SPMB chatbot intent.
Fig 6. Chatbot Implementation Result
TABLE III Test Result
Intent Result IV. TESTING
Precision Recall F1-score A functionality test on the chatbot was conducted on 10
Sapa 1.0 1.0 1.0 people at the same time to find out whether the chatbot can
goodbye 1.0 1.0 1.0
pendaftaran_wa 0.993288590 1.0 0.996632996
function properly. A URL link is given for respondents so
ktu 6040269 6329966 that respondents can test the chatbot in the form of a widget.
umum_biaya 1.0 1.0 1.0 A questionnaire is given which contains 8 questions with
umum_asrama 1.0 1.0 1.0 answer choices strongly agree, agree, quite agree and
umum_tempat_t 1.0 1.0 1.0 disagree. Each answer has a value, strongly agree is worth 4,
es
umum_jurusan 1.0 1.0 1.0 agree is worth 3, quite agree is worth 2 and disagree is worth
umum_informas 1.0 1.0 0.995983935 1. From the responses that have been calculated based on the
i 7429718 questionnaire that has been filled in and the total percentage
umum_kuota 1.0 1.0 1.0 result is 90.625%. The following in Fig 7 is a graph of the
umum_persyara 1.0 1.0 1.0 results of the functionality tests that have been carried out.
tan
umum_kontak 1.0 1.0 1.0
syarat_wajib_pe 1.0 1.0 1.0
minatan
syarat_tinggi 1.0 1.0 1.0
Syarat_gigi 1.0 1.0 1.0
Syarat_berat 1.0 1.0 1.0
Syarat_mata 1.0 1.0 1.0
Random_militer 1.0 1.0 1.0
iambot 1.0 1.0 1.0
thankyou 1.0 1.0 1.0

The F1-score value is obtained from calculating the

accuracy of a test which is influenced by the precision and
recall values. The highest value of the F1-score is 1.0 and
the lowest value is 0. From testing on the chatbot model
that has been made, almost all intents get the highest
score. Two intents whose values do not reach 1.0 are
umum_pendataran_waktu and umum_informasi. This
value does not reach 1 because it is influenced by errors
that occur as listed in Fig 5.
B. Implementation on The Web
The Rasa Chatbot is implemented using Docker and
Heroku. Heroku is used because it is quite easy to use and can
be used for free. After the Rasa project is completed, a docker
file is provided. After that, Heroku is deployed. After the
deployment is complete, the URL of the project will be
obtained Furthermore, this URL will be entered in the
Fig 7. Chatbot Functionality Test Results Graph
Question 1 contains the question whether the chatbot appears
on the page accessed by the respondent. There are
suggestions from respondents regarding the welcome writing
which is rather small so that it does not attract attention.
Question 2 contains the question whether the chatbot dialog
box appears when the icon is pressed.
Question 3 contains the question whether the respondent can
enter questions on the chatbot.
Question 4 contains the question whether the 'loading' sign
appears when the chatbot is processing the question.
Question 5 contains questions the chatbot can provide
answers after being given input.
Question 6 contains questions Chatbot provides appropriate
answers to questions given by respondents. Respondents

377
rized licensed use limited to: SVKM¿s NMIMS Mukesh Patel School of Technology Management & Engineering. Downloaded on April 26,2022 at 15:21:16 UTC from IEEE Xplore. Restrictions a
 several times got answers that were not in accordance with
the questions given. The chatbot issues some default answers
to a given question. This default answer is due to a fallback
policy. In the Fallback policy, there is a confidence limit
setting for an intent. If the incoming question is below the set
confidence value, the default answer will appear instead of
utter. In addition to setting the confidence value, respondents
received answers that were not appropriate due to the limited
topics from the chatbot which only focused on SPMB.
Respondents gave suggestions to expand the scope of topics
owned by the chatbot.
Question 7 contains the question of whether the use of
chatbots is easily understood by respondents.
Question 8 contains the question of whether the chatbot is
easy to use.
V. CONCLUSION
SPMB Chatbot with Deep learning Techniques is made
using the Rasa Framework. Chatbots are made by collecting
data, analyzing, and designing, creating chatbots, and
implementing them on the web. Based on the tests that have
been carried out on the chatbot, the chatbot performance is
generated in the values of precision, recall and F1-score.
Testing is done automatically using the features of Rasa. The
average value of precision is 0.9964. The average value of
recall is 1.0. The average value of the F1 score is 0.9961.
REFERENCES
[1] N. Deepika, M. M. Bala, and R. Kumar, “Design and implementation
of intelligent virtual laboratory using RASA framework,” Mater.
Today Proc., Feb. 2021, doi: 10.1016/j.matpr.2021.01.226.
378
rized licensed use limited to: SVKM¿s NMIMS Mukesh Patel School of Technology Management & Engineering. Downloaded on April 26,2022 at 15:21:16 UTC from IEEE Xplore. Restrictions a
[2] L. Zhou, J. Gao, D. Li, and H.-Y. Shum, “The Design and
Implementation of XiaoIce, an Empathetic Social Chatbot,” Comput.
Linguist., vol. 46, no. 1, pp. 53–93, Mar. 2020, doi:
10.1162/coli_a_00368.
[3] A. Tiha, “Intelligent Chatbot using Deep Learning Artist
Identification from Artworks with Deep Learning [Python, Keras,
CNN, Deep Learning] View project Big Data Application for Large
Scale US Stock Market Data Analysis [JAVA, Apache Spark, Maven]
View project,” 2018, doi: 10.13140/RG.2.2.14006.75841.
[4] S. Roller et al., “Recipes for building an open-domain chatbot,”
arxiv.org, Apr. 2020.
[5] M. Chung, E. Ko, H. Joung, and S. J. Kim, “Chatbot e-service and
customer satisfaction regarding luxury brands,” J. Bus. Res., vol. 117,
pp. 587–595, Sep. 2020, doi: 10.1016/j.jbusres.2018.10.004.
[6] Y. Windiatmoko, R. Rahmadi, and A. F. Hidayatullah, “Developing
Facebook Chatbot Based on Deep Learning Using RASA Framework
for University Enquiries,” IOP Conf. Ser. Mater. Sci. Eng., vol. 1077,
no. 1, p. 012060, Feb. 2021, doi: 10.1088/1757-899X/1077/1/012060.
[7] A. Elcholiqi and A. Musdholifah, “Chatbot in Bahasa Indonesia using
NLP to Provide Banking Information,” IJCCS (Indonesian J. Comput.
Cybern. Syst., vol. 14, no. 1, p. 91, 2020, doi: 10.22146/ijccs.41289.
[8] U. Gnewuch, S. Morana, M. T. P. Adam, and A. Maedche, “Faster is
not always better: Understanding the effect of dynamic response
delays in human-chatbot interaction,” 2018.
[9] Rakesh Kumar Sharma, “An Analytical Study and Review of open
source Chatbot framework, Rasa,” Int. J. Eng. Res., vol. V9, no. 06,
Jun. 2020, doi: 10.17577/IJERTV9IS060723.
[10] A. Singh, K. Ramasubramanian, and S. Shivam, “Introduction to
Microsoft Bot, RASA, and Google Dialogflow,” in Building an
Enterprise Chatbot, Berkeley, CA: Apress, 2019, pp. 281–302.
[11] S. Meshram, N. Naik, V. Megha, … T. M.-… A. C. on, and U. 2021,
“College Enquiry Chatbot using Rasa Framework,”
ieeexplore.ieee.org.
[12] J. Shabbir, M. U. Arshad, and W. Shahzad, “NUBOT: Embedded
Knowledge Graph with RASA Framework for Generating Semantic
Intents Responses in Roman Urdu,” arxiv.org, Feb. 2021.
[13] A. M. Rahman, A. Al Mamun, and A. Islam, “Programming
challenges of chatbot: Current and future prospective,” in 2017 IEEE
Region 10 Humanitarian Technology Conference (R10-HTC), Dec.
2017, pp. 75–78, doi: 10.1109/R10-HTC.2017.8288910.