RiskGuru Chatbot
Madhavi Aghera Mentor Name: Amaresh Chandra Singh
Enrollment ID: 202211049
MTech Major Project - II Report
Mode: Off Campus Company Address: 1401 to 1410, 14 th Floor, B Block,
DAIICT Westgate by True Value, S.G. Highway,
Gandhinagar, Gujarat, India
202211049@daiict.ac.in
On Campus Supervisor: Prosenjit Kundu
Assistant Professor
DAIICT
Gandhinagar, Gujarat, India
prosenjit kundu@daiict.ac.in
Abstract—This project is a segment of the RiskGuru product,
which comprises several AI-driven modules. Among these mod-
ules is a Chatbot, designed to respond to user queries. Chatbots
play a pivotal role in modern software solutions by providing in-
stant and personalized assistance to users. They serve as efficient
virtual assistants, capable of addressing inquiries, guiding users
through processes, and even executing tasks autonomously. By
leveraging natural language processing and artificial intelligence,
chatbots enhance user experience, streamline communication,
and improve efficiency within software ecosystems. In essence,
chatbots are indispensable components of software solutions,
driving engagement, productivity, and innovation in various
industries.
The focus of this project is on managing policy documents,
enabling the Chatbot to address queries across various sections
of policies. The project consists of various stages - factual,
explanatory & generative. To achieve this, we are exploring
different types of LLM models and their effectiveness with
insurance domain. We’re also generating and refining a variety
of Prompt & Responses to achieve this task.
Engaging in this project provides me with an opportunity to
delve into the domain of insurance and its associated documents.
The domain’s complexity necessitates expert guidance for com-
prehension. Therefore, RiskGuru aims to assist users in reviewing
their documents and selecting the most suitable policies.
Index Terms—Annotations, Prompt Engineering, Large Lan-
guage Models - LLM, Prompt & Responses (P&R), Quantization
I. I NTRODUCTION
Injala is the leading disruptor in the insurance industry,
offering a range of risk management software solutions, in-
cluding Wrapportal, Kinetic, Asuretify, Anzenn, Prequaligy,
and RiskGuru. Collectively, these software solutions form a
complete insurance technology ecosystem known as the Risk
Management Operating System (RMIS 360). RMIS 360 offers
automation, rapid verification, document review, comparison,
and other functionalities.
RiskGuru serves as an AI Advisor, offering intelligent anal-
ysis for diverse insurance documents. Its AI-driven function-
Manager - AI/ML Team
Company Name: Injala Pvt. Ltd.
Ahmedabad, Gujarat, India
Amaresh.Singh@injala.com
On Campus Co-Supervisor: Bakul Gohel
Assistant Professor
DAIICT
Gandhinagar, Gujarat, India
bakul gohel@daiict.ac.in
alities ensure consistency, accuracy, thoroughness, and speed,
effectively preventing costly errors and omissions in policy
coverage. By providing optimal match coverages and uncover-
ing hard terms through dissecting form wording and coverage
intent, RiskGuru delivers unparalleled value. Additionally, it
extracts key policy details, ensuring quick results.
Our project is structured into three distinct stages: factual,
explanatory, and generative. Leveraging the capabilities of
LLM models and efficient prompt engineering techniques, we
aim to facilitate the extraction of key policy details.
II. O BJECTIVE
RiskGuru Chatbot allows users to access necessary infor-
mation from policies without navigating through lengthy and
complex documents, enabling quicker decision-making and
facilitating the purchase of policies that align best with their
needs. Moreover, this approach aids users in comprehending
the coverage, limits, and any other conditions associated with
the policy.
III. L EARNING O UTCOMES
Despite already possessing knowledge in Python, Machine
Learning, Deep Learning, Natural Language Processing, and
other fundamental concepts, the scheduled sessions were ini-
tially planned to span a total of six months to cover all
these topics comprehensively. However, upon our request, they
accommodated our needs by agreeing to assess us through
assignments and provided modules to work on. Due to their
coordination and support, I’ve been able to learn the following
topics within a relatively short period:
A. Domain Knowledge
1) Types of Policies:
• General Liability: Commercial general liability (CGL)
insurance is a policy type that offers coverage to busi-
nesses for injury and property damage resulting from
 the business’s operations or accidents that occur on the
business premises. However, it does not cover intentional
damages or accidents involving automobiles, aircraft, or
watercraft.
• Auto Liability: Auto liability insurance provides cov-
erage for businesses that use vehicles as part of their
operations. It proves beneficial in the event of crises
related to automobiles.
• Workers Compensation: Workers’ compensation is a
type of employer insurance coverage that provides ben-
efits to workers who sustain injuries or become disabled
as a result of their job duties.
• Umbrella and Excess: An umbrella liability policy
supplements underlying liability insurance policies by
extending coverage beyond their limits. Once the un-
derlying coverage is exhausted due to claim payments,
the umbrella policy can step in to cover the remaining
balance.
2) Policy Sections:
The policy contains sections like Declaration, Rating, Cov-
erage, Form list, Endorsement & Notice.
• Declaration: The declaration page provides compre-
hensive information such as carrier details, names and
addresses of the insured, insurer, and producer. It also
encompasses essential policy details like the policy num-
ber, effective date, retroactive date, coverage list, limits,
premium breakdown, and any fees included in the policy.
• Rating: Rating refers to the process of determining the
premium amount for an insurance policy based on the
insured’s risk profile. Insurers use various factors, such
as the insured’s age, location, type of property, and claims
history, to calculate the premium rate. The rating process
helps insurers ensure that premiums accurately reflect the
level of risk associated with insuring the policyholder.
• Form list: A formlist is a document that outlines the
various forms and endorsements attached to an insurance
policy. It serves as a checklist to ensure that all neces-
sary forms and endorsements are included and properly
executed.
• Endorsement: An endorsement is a modification or
addition to an insurance policy that changes the terms
or coverage provided by the policy. Endorsements can
be used to add or remove coverage, change coverage
limits, or modify policy conditions. They are typically
attached to the policy as separate documents and are
signed by both the insurer and the insured to indicate
mutual agreement to the changes.
• Notice: A notice in insurance refers to formal com-
munication provided by the insurer to the insured or
other parties involved in the insurance policy. Notices
may include important information about changes to the
policy, updates on claims processing, renewal reminders,
cancellation notices, or other relevant notifications.
3) ACORD 25 | Certificate of Insurance (COI):
The Certificate of Insurance (COI), also known interchange-
ably as an ACORD form, is a concise document that sum-
marizes an insurance policy and confirms that a business
holds insurance coverage. An ACORD Certificate of Insurance
includes various key components to communicate important
information. COI contains details of producer, insured, in-
surer, certificate number, coverages, type of insurance, policy
number, policy effective & expiration date, policy limits, de-
scription of operations/ vehicles/ locations, additional insured,
signature and cancellation notice.
B. Concepts
1) Prompt Engineering:
Hallucination in Large Language Models (LLMs) describes
the occurrence when the model generates text that lacks factual
basis or coherence with the provided context. Instead of
producing accurate responses, the model generates fictional,
misleading, or nonsensical content, presenting false knowledge
as if it were accurate. This behavior involves the model
generating misleading or incorrect information.
Prompt engineering aims to define the role and scenario
for the model to guide its generation process, preventing
it from producing unbounded hallucinations. This approach
helps ensure that the model generates responses that are
grounded in the provided context and aligned with the intended
purpose, minimizing the risk of producing nonsensical or
misleading content [1]. There are some well-known types of
prompt engineering.
• Retrieval-Augmented Generation (RAG): RAG is a
framework that combines retrieval-based and generation-
based approaches to improve the quality and relevance of
the generated text based on domain knowledge. It incor-
porates a retrieval mechanism to fetch relevant passages
or documents from a knowledge source (e.g., Wikipedia)
based on the input query or prompt. It uses a language
model (e.g., GPT) to generate responses conditioned on
both the input query and the retrieved passages, ensur-
ing that the generated text is contextually relevant and
informative.
RAG is useful for tasks requiring contextually rich re-
sponses, such as question answering, summarization, and
dialogue systems.
• Chain of Thoughts (COT): It is a few-shot learning
technique. COAT is a prompting technique that involves
generating a sequence of interconnected prompts to guide
the generation of coherent and contextually relevant text.
Instead of providing a single prompt, which may limit
the scope or coherence of the generated output, Chain
of Thoughts prompts the model with a series of related
questions or statements, each building upon the previous
one. This approach encourages the model to maintain a
coherent narrative or logical progression throughout the
generated text, resulting in more structured and meaning-
ful outputs.
 Chain of Thoughts is particularly useful in tasks requiring
the generation of complex or multi-faceted responses,
such as storytelling, essay writing, or problem-solving.
By breaking down the prompt into a series of intercon-
nected thoughts, this technique helps guide the model’s
generation process and ensure consistency and coherence
in the output.
• Reason + Act (ReACT): It is also a few-shot learning
technique. ReACT is similar to the RAG framework
specifically tailored for conversational applications, such
as chatbots and virtual assistants. It has much more
reasoning capacity. It gains additional information to
arrive at a response by getting knowledge from both
Private Public - External knowledge bases.
• Directional Stimulus Prompting (DSP): DSP aims to
influence the output of language models by providing
directional prompts that guide the generation process
towards specific topics, sentiments, or styles. Prompts are
carefully crafted to evoke a particular response from the
model. They often include keywords, cues, or instructions
designed to steer the model’s output in a desired direction.
DSP prompts may vary in complexity and specificity,
depending on the desired outcome. They can range from
simple directives to more nuanced instructions tailored to
the task.
DSP is commonly used in tasks such as content genera-
tion, sentiment modification, or topic manipulation, where
guiding the model’s output towards a specific direction is
crucial.
2) Parameter Efficient Fine Tuning (PEFT):
Parameter Efficient Fine Tuning (PEFT) is a strategy utilized
to enhance the performance of large language models (LLMs)
on particular tasks while mitigating the computational burden
associated with their massive parameter count [2]. Despite
LLMs’ effectiveness in natural language processing (NLP)
tasks, their extensive parameters make training and storage
computationally expensive. Fine-tuning is a common tech-
nique to adapt these models for specific tasks, requires training
all these parameters, which can be very costly. PEFT addresses
this issue by focusing on a subset of parameters for training
while freezing the reamining, thereby significantly reducing
computational resources while maintaining performance.There
are concepts which works along with PEFT.
• Quantization: The main concept is to change the preci-
sion of the parameters in a pre-trained model so it can fit
into less powerful hardware [3]. It converts the parameters
from high precision to low precision. Most parameters are
converted to a simpler form to save memory, but a small
part that’s crucial for adjusting the model during training
stays in its original form because making them simpler
might affect how well the model learns. Post Training
Quantization (PTQ) and Quantization Aware Training
(QAT) are two modes of training with quantization. There
are various quantization techniques. Below are some
widely used peft techniques that uses quantization:
– LORA - Low rank adaption is a technique that uses
SVD (singular value decomposition) to decompose a
high rank matrices in to a combination of low rank
matrices. Instead of adjusting all the model’s param-
eters, LORA introduces small, specialized matrices
that capture the essence of the new task [4].
– QLORA is an extension of LORA, which introduces
4-bit NormalFloat (NF4), Double Quantization and
Paged Optimizers [5].
C. Non-tech values:
In addition to technical expertise, I’ve learned the impor-
tance of certain non-tech skills crucial for my career growth.
• Sharpening soft skills like communication and teamwork,
which are essential for success in any workplace.
• I’ve also realized the value of keeping code organized,
checking its coverage, and documenting it well, ensuring
that projects stay manageable and understandable.
• Lastly, effective task management is key for meeting
deadlines and achieving goals efficiently.
IV. T OOLS
A. LabelImg
LabelImg is an user-friendly tool used for annotating im-
ages, providing a graphical interface for this task. Developed
in Python, it utilizes the Qt framework for its visual interface.
Annotations done by LabelImg are saved as XML files in
PASCAL VOC format, the format used by ImageNet. It also
supports YOLO format [6].
B. YEDDA
YEDDA, formerly known as SUTDAnnotator, is a versatile
tool designed for annotating various forms of text, including
multiple languages such as English and Chinese, as well as
symbols and emojis. It is compatible across all major operating
systems, including Windows, Linux, and MacOS [7].
C. Postman
Postman is a powerful API testing and development tool
that simplifies the process of working with APIs. It offers a
user-friendly interface for sending requests, testing responses,
and debugging APIs. With features like automated testing and
collaboration capabilities, Postman streamlines API develop-
ment workflows and enhances productivity.
D. Amazon Textract
Amazon Textract is a machine learning service provided by
Amazon Web Services (AWS) that enables users to extract
text and data from scanned documents, PDFs, and images.
Leveraging advanced machine learning algorithms, Textract
can accurately detect and extract printed text, handwriting,
tables, and forms from a wide range of document types.
We’ve utilized this service with policy and accord documents
to provide context in generating prompts and responses for
training large language models (LLMs).
E. Azure DevOps
Azure DevOps is a cloud platform from Microsoft that
streamlines the entire software development lifecycle. It in-
cludes a range of services including version control, build
automation, continuous integration and delivery (CI/CD), agile
planning, and project management. With Azure DevOps, teams
can collaborate more efficiently, automate repetitive tasks,
track progress, and deliver high-quality software faster.
V. C ONTRIBUTIONS
A. Understanding Policy Document & Helping team restruc-
turing the policy pipeline
As mentioned, policy documents are complex and often
reuires the guidance of domain experts for comprehension.
Our deliberate efforts involved dissecting the primary sections
of the policy to compile a list of classes crucial for training
the classifier module. In the RiskGuru pipeline provided in the
figure below, policy section classification is a pivotal point that
determines from which section the model should look after to
address specific queries.
Fig. 1. RiskGuru Policy Pipeline
B. Data Segregation
As mentioned earlier, while the classification module con-
sists of a list of classes, there is currently a lack of readily
available data. However, the company owns a vast array of
policies, from which we aim to extract our requirements.
Our strategy involves converting all policies into images and
systematically organizing them into folders corresponding to
respective classes, including Declaration, Rating, Form List,
and various Endorsement sections.
C. Annotation and P&R Generation
Data annotation and prompt & response generation are
critical aspects of training models. Data annotation involves
labeling datasets with relevant information, providing super-
vision for machine learning algorithms. This annotated data
is essential for training accurate models. Prompt & response
generation involves crafting input prompts and generating
corresponding responses, which is crucial for developing con-
versational AI systems like chatbots. These processes are
essential for creating high-quality datasets and ensuring that
models effectively understand and generate meaningful natural
language text.
Following the data segregation process, the next step in-
volves preparing prompts to feed into the model. This includes
generating prompts based on the respective section folders and
aligning them with the chat template designed for a particular
model. The template includes questions and answers with their
corresponding context, ensuring the model receives structured
input for effective training and comprehension.
D. RAG Module
RAG (Retrieval-Augmented Generation) with LLM (Large
Language Models) holds significant importance in natural
language processing tasks, particularly in the domain of ques-
tion answering and information retrieval. RAG enhances the
capabilities of LLMs by integrating a retrieval component that
accesses external knowledge sources. This enables LLMs to
generate responses based not only on learned patterns from
training data but also on relevant information retrieved from
external sources. By leveraging this combination of generation
and retrieval, RAG with LLMs can produce more accurate, co-
herent, and contextually relevant responses, thereby improving
the overall performance and effectiveness of natural language
understanding and generation tasks.
After testing the RAG module on a pre-trained LLM, we
observed satisfactory results at some stage but observed that it
encounters difficulties in understanding the structure of policy
documents. Consequently, we made the strategic decision to
first train the LLM model specifically for policy data. By doing
so, we aim to enhance the model’s comprehension of policy
document structures and nuances. Subsequently, we plan to
integrate the RAG module with this tailored LLM model,
leveraging its augmented capabilities to provide more accurate
and contextually relevant results. This approach is expected to
improve performance in generating responses to policy-related
queries.
E. LLM module for Declaration section
Assigned with the task of fine-tuning LLM models for the
declaration section of policies, I am tasked with selecting the
most suitable LLM model for this purpose and integrating
it with the declaration section. Due to the unavailability
of sufficient data, there is a need to generate prompts and
responses for our designated section. Currently, this module is
in progress, and I have completed the training pipeline while
focusing on developing the inferencing pipeline and generating
more prompts to get better training results.
VI. C ONCLUSION
Along with the classifier module, fine-tuned models trained
on section-wise prompts, and the RAG module, the RiskGuru
chatbot will be able to deliver improved responses for policy
documents.
Currently, our focus is solely on handling factual data. In
the subsequent phase, we aim to delve into the explanatory
and generative aspects of the chatbot. Moving forward, the
project’s scope extends to comparing policy documents and
suggesting the most suitable policy to users based on the
coverages offered. This expanded functionality is intended to
enrich the user experience and offer invaluable assistance in
selecting the right insurance policies.
ACKNOWLEDGMENT
I want to thank my manager, colleagues, and mentors for
their invaluable support and guidance throughout my learning
journey. Working with such an exceptional team and under
the guidance of a supportive manager has been truly fortunate.
Their mentorship helped me grow professionally and fostered
a friendly and nurturing work environment. The expertise and
experience gained from collaborating with my colleagues will
undoubtedly hold great significance for me in the future.
R EFERENCES
[1] https://paperswithcode.com/task/prompt-engineering
[2] https://huggingface.co/blog/peft
[3] https://www.tensorops.ai/post/what-are-quantized-llms
[4] ”LoRA: Low-Rank Adaptation of Large Language Models”, by Edward
J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li,
Shean Wang, Lu Wang, Weizhu Chen, arXiv preprint arXiv:2106.09685.
2021 Jun 17. (https://arxiv.org/abs/2106.09685)
[5] ”QLoRA: Efficient Finetuning of Quantized LLMs”, by Tim Dettmers,
Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer, Advances in Neural
Information Processing Systems, 36. (https://arxiv.org/abs/2305.14314)
[6] https://github.com/HumanSignal/labelImg
[7] https://github.com/jiesutd/YEDDA