MINI PROJECT REPORT

A MINI PROJECT REPORT ON

“Stock Sentiment Market Prediction”

Project ID: (CSE-AI-06/2022-23)

For the Degree of

Bachelor of Technology in Department of Computer Science & Engineering – Artificial

Intelligence

SUBMITTED BY:

Harshraj Ushire (PRN No. 2064191913014)

Chinmay Sonsurkar (PRN No. 2064191913012)

UNDER THE GUIDANCE OF

Dr. Ashwini Shinde

Department Of Computer Science & Engineering – Artificial Intelligence

PCET- NMVPM’s,
Nutan College of Engineering and Research, Talegaon, Pune
“Vision of the Institute”
“To be a leading technological institute ensuring quality education giving rise to innovative
technocrats, professionals and global citizens.”

2022-2023
 Nutan Maharashtra Vidya Prasarak Mandal’s
NUTAN COLLEGE OF ENGINEERING AND RESEARCH
Under Administrative Support of Pimpri Chinchwad
Education Trust

CERTIFICATE
This is to certify that the Mini Project Report entitled “Stock Sentiment Market Prediction”,
which is being submitted by, Harshraj Ushire, Chinmay Sonsurkar as partial fulfillment
for the Third Year Degree of Bachelor of Technology (Computer Science & Engineering –
Artificial Intelligence) of DBATU, Lonere

This is bonafide work carried under my supervision and guidance.

Place: Pune

Date:

Dr. Ashwini Shinde Dr. Ashwini Shinde

Project Guide Project I/C

Prof. Priyanka Vyas Dr. Aparna Pande

Head of Department Principal

External Examiner

SEAL
If project is sponsored then attach sponsorship letter/certificate here
 ACKNOWLEDGEMENT

The satisfaction & euphoria that accompany the successful completion of any task
would be incomplete without the mention of people who made it possible . So I acknowledge
all those whose guidance and encouragement served as a beacon light & crowned my efforts
with success.

I have immense pleasure in expressing thanks to the principal Dr. Aparna Pande for
providing all the facilities for the successful completion of the project.
With due respect, I thank my H.O.D. Prof. Priyanka Vyas Department of Computer Science
& Engineering – Artificial Intelligence, for her motivating support, keen interest which kept
my spirits alive all through.

I would like to express thanks to my guide Dr. Ashwini Shinde, Department of

Computer Science & Engineering – Artificial Intelligence who has guided me throughout the
completion of this project.

Finally I would like to thank all the teaching and non-teaching staff and all my friends
who have rendered their support in the completion of this report.

Name and Signature of student:

Sr.no Name Sign

1 Harshraj Ushire

2 Chinmay Sonsurkar
 ABSTRACT

Stock market prediction is a challenging task due to the complex nature of financial markets, which are
influenced by numerous factors including investor sentiment, economic indicators, and news events. In this
project, we propose a sentiment-driven approach to stock market prediction by leveraging news analysis and
the LangChain framework. The goal is to develop a robust model that can analyze news sentiment from various
sources, such as social media platforms and financial news websites, and generate accurate predictions of stock
market trends..
To achieve this, we collected a comprehensive dataset comprising stock market data, news articles, and social
media posts related to a selected set of stocks. The collected data was preprocessed to remove noise, normalize
text, and eliminate hyperlinks. The LangChain framework was then employed as an interface to facilitate
seamless communication between the user and the language model, allowing for efficient data processing and
sentiment analysis.
The results of our experiments demonstrate that sentiment analysis, when combined with news analysis and
the LangChain framework, can provide valuable insights into stock market trends. The sentiment-driven
prediction model exhibited promising accuracy in predicting bullish or bearish market movements. This project
contributes to the growing field of sentiment analysis for stock market prediction and highlights the potential
of leveraging news sentiment in making informed investment decisions.

Keywords: stock market prediction, cloud, big data, machine learning, regression.
 INDEX

Sr. Page
Content
No. No.

1 Introduction 1
1.1 Related Work 1
1.2 Literature Review 2
1.3 Problem Statement 6
1.4 Objectives 6
2 Methodology 7
2.1 Data Collection 7
2.2 Preprocessing 7
2.3 Sentiment Analysis 7
2.4 Feature Extraction 7
2.5 Historical Data Analysis 8
2.6 Predictive Modelling 8
2.7 Model Evaluation 8
2.8 Deployment And Monitoring 8

Conclusion
References
 LIST OF FIGURES

Fig. Page
Figure Name
No. No.
Data Collection and Preprocessing
2.1 7
1. INTRODUCTION

Predicting stock market trends has been a longstanding challenge for analysts and researchers alike. The
dynamic and volatile nature of financial markets, coupled with the influence of various factors such as
economic indicators, investor sentiment, and news events, makes accurate prediction a complex task.
Traditional approaches relying solely on historical data or technical indicators often fall short in capturing
the nuanced dynamics of the market. In recent years, sentiment analysis has emerged as a promising
technique for enhancing stock market prediction by leveraging the power of natural language processing and
machine learning.
The sentiment analysis approach considers the impact of news sentiment on stock market movements. It
recognizes that financial news articles, social media posts, and other textual data contain valuable
information that can influence investor behavior and, subsequently, stock prices. By analyzing the sentiment
expressed in these sources, it becomes possible to gauge market sentiment and make predictions based on
the collective sentiment of investors. In this project, we aim to develop a sentiment-driven stock market
prediction model by leveraging news analysis and the LangChain framework. The LangChain framework
provides an interface that connects language models, such as ChatGPT, to external data sources and
facilitates seamless communication and data processing. By integrating the LangChain framework into our
project, we can harness the power of language models to analyze news sentiment and generate accurate
predictions of stock market trends.

Our approach involves implementing a sentiment analysis model using deep learning techniques. This model
analyzes the preprocessed data and assigns sentiment scores to each news article or social media post. The
sentiment scores are then mapped to time series data using advanced time series analysis methods, allowing
us to generate visual representations of bullish or bearish market trends.

1.1 RELATED WORK

Recent studies have shown a strong link between news articles and stock price movements, indicating
the significance of sentiment analysis in stock prediction beyond traditional methods.
Chen et al. (2015) explored the use of sentiment analysis and topic modeling to predict stock market
trends. They developed a model that combined sentiment analysis of financial news articles with topic
modeling techniques to capture the influence of different topics on stock prices. Their results
demonstrated the efficacy of sentiment analysis in improving stock market prediction accuracy.
Liu et al. (2017) proposed a sentiment analysis framework that integrated social media sentiment with
historical stock data. They collected tweets and financial news articles and developed a sentiment
analysis model to generate sentiment scores. Their model was then combined with historical stock data
to predict market trends. The study showed that sentiment analysis provided valuable insights for stock
market prediction. Zhang et al. (2018) employed deep learning techniques for sentiment analysis
and stock market prediction. They built a deep neural network model that integrated sentiment analysis
with historical stock data. Their results indicated that the deep learning approach outperformed
traditional methods and provided more accurate predictions of stock market movements.
1.2 LITERATURE REVIEW

Predicting stock market trends has been a longstanding challenge for analysts and researchers alike.
The dynamic and volatile nature of financial markets, coupled with the influence of various factors
such as economic indicators, investor sentiment, and news events, makes accurate prediction a
complex task. Traditional approaches relying solely on historical data or technical indicators often fall
short in capturing the nuanced dynamics of the market.
In recent years, sentiment analysis has emerged as a promising technique for enhancing stock market
prediction by leveraging the power of natural language processing and machine learning.

The sentiment analysis approach considers the impact of news sentiment on stock market
movements. It recognizes that financial news articles, social media posts, and other textual data contain
valuable information that can influence investor behavior and, subsequently, stock prices. By analyzing
the sentiment expressed in these sources, it becomes possible to gauge market sentiment and make
predictions based on the collective sentiment of investors. In this project, we aim to develop a
sentiment-driven stock market prediction model by leveraging news analysis and the LangChain
framework. The LangChain framework provides an interface that connects language models, such as
ChatGPT, to external data sources and facilitates seamless communication and data processing. By
integrating the LangChain framework into our project, we can harness the power of language models to
analyze news sentiment and generate accurate predictions of stock market trends.

Recent studies have shown a strong link between news articles and stock price movements,
indicating the significance of sentiment analysis in stock prediction beyond traditional methods.

Chen et al. (2015) explored the use of sentiment analysis and topic modeling to predict stock market
trends. They developed a model that combined sentiment analysis of financial news articles with topic
modeling techniques to capture the influence of different topics on stock prices. Their results
demonstrated the efficacy of sentiment analysis in improving stock market prediction accuracy.

Liu et al. (2017) proposed a sentiment analysis framework that integrated social media sentiment
with historical stock data. They collected tweets and financial news articles and developed a
sentiment analysis model to generate sentiment scores. Their model was then combined with
historical stock data to predict market trends. The study showed that sentiment analysis provided
valuable insights for stock market prediction.

Zhang et al. (2018) employed deep learning techniques for sentiment analysis and stock market
prediction. They built a deep neural network model that integrated sentiment analysis with historical
stock data. Their results indicated that the deep learning approach outperformed traditional methods
and provided more accurate predictions of stock market movements

1
1.2 LITERATURE REVIEW

The 1st Pneumatic Fluidic muscles-based Exoskeleton Suit in the U.A.E [5] -

Authors: -Abdulla Almomani, Faisal Miqdadi, Mustafa Hassanin, Mustafa

Samy, and Mohammed Awadallah.
The publishers Prototyped a fully functional wearable exoskeleton arm that moves
in sync with the operator’s motion within 3 DOFs. With the help of this device the user will no
longer feel any fatigue in carrying heavy loads for long time periods. Recyclable, light and
enduring materials were used in this project in order to fulfil safety and environmental
concerns. Pneumatic muscles are placed cloning the human anatomy to provide absolute
synchronization with ordinary human motion. Biceps, triceps and deltoids were assisted by
the device and limited according to their allowable angles of motion. The exoskeleton arm can
specifically be used in physiotherapeutic treatment and to aid people with difficulties in
mobility and also in the heavy industry. This project was sponsored by Festo. Festo is a
worldwide leading supplier of pneumatic and electrical automation technology.

Upper extremity robotics exoskeleton application, structure and actuation [1] -

Authors: - Kwok-Hong Chay, Jer-Vui Lee, Yea-Dat Chuah and Yu-Zheng Chong.
Robotic exoskeleton is getting important to human in many aspects such as power
assist, muscle training, regain motor function and rehabilitation. The research and development
towards these functions are expected to be combined and integrated with the human intelligent
and machine power, eventually becoming another generation of robot which will enhance the
machine intelligent and human power. This paper reviews the upper extremity exoskeleton with
different functions, actuators and degree of freedom (DOF). Among the functions,
rehabilitation and power assist have been highlighted while pneumatic actuator, pneumatic
muscle, motor and hydraulic actuator are presented under the categories of actuator. In addition,
the structure of exoskeleton is separated by its DOF in terms of shoulder, elbow, wrist and
hand.

Design of a powered full-body exoskeleton for physical assistance of elderly people [2] -

Authors: - Simon Christensen, Sajid Rafique and Shaoping Bai.

The development of full-body exoskeletons has been limited due to design

complexities, mechanical integration intricacies, and heavier weight, among others.
Consequently, very few full-body powered exoskeletons were developed to address these
challenges, in spite of increasing demand for physical assistance at full-body level. This article
presents an overall design and development of a powered full-body exoskeleton called “FB-
AXO.” Primarily, FB-AXO consists of two main subsystems, a lower-body and an

2
upper-body subsystem connected together through waist and spine modules. FBAXO is
developed for the support of weaker ageing adults so that they can continue functioning their
daily activities. At the onset of the project, a set of functional and design requirements has been
formulated with an extensive end-user involvement and then used in realizing the FB- AXO.
The final FB-AXO design comprises of 27 degrees of freedom, of which 10 are active and 17
are passive, having a total system weight of 25 kg. Overall, the article elaborates
comprehensively the design, construction, and preliminary testing of FB-AXO. The work
effectively addresses design challenges including kinematic compatibility and modularity with
innovative solutions. The details of the mechanics, sensors, and electronics of the two
subsystems along with specifics of human-exoskeleton interfaces and ranges of motion are also
provided. The FB-AXO exoskeleton effectively demonstrated to assist full-body motionssuch
as normal walking, standing, bending as well as executing lifting and carrying tasks to meet
the daily living demands of older users.

Development of an armored upper limb exoskeleton [3] -

Authors: - Sntiago López-Méndez, Hader Vladimir Martínez-Tejada, Marco Fidel

Valencia-García.

Personal safety is a critical aspect of daily life, but also in the military. Active soldiers
often have to carry heavy gear during missions, which puts pressure on their backs. Therefore,
the military must come up with new technologies that allow both protection and movement. In
this paper, it is explaining the development of an armored upper limb exoskeleton with three
degrees of freedom. To ensure portability, it is used battery-fed DC actuators. The system was
encased in a metal matrix that doubles up as a protective plate. The exoskeleton, the control
system, the actuators, and the plate are integrated so that they offer protection while supporting
the flexion and extension of the upper limb.

Design of an Exoskeleton ARM with Pneumatic Muscle Actuation [6] -

Authors: - Shahaan Khatri, Sumit Thaploo, Astha Singh and Mr. Lokesh M. Giripunje.

For centuries now, humans have developed machines for tasks which are too labour
intensive for species cannot do. So, creative imagination and subtle engineering has led to the
development of the powered exoskeleton. It is a device which can be worn over the human
body. A powered exoskeleton enables a human to perform tasks which are beyond the physical
prowess by amplifying the muscular movements. We have outlined the process of developing
an exoskeleton arm which increases the load lifting capacity of a human. The PAM has a thin-
walled, rubber bladder placed inside an axially stiff but radically compliant braided sleeve. As
the rubber bladder expands due to an increase in pressure, the diameter of the combined sleeve
and bladder assembly easily changes in the radial direction and the PAMshortens in the axial
direction. As the consequence of this interaction, a large contraction

3
force produced can perform external work at rapid rate. However, non-linearity exists as the
pressure changes in the bladder because its area expands proportionally to the square of the
diameter. Also as the outer sheath material moves, its length is dependent on trigonometric
relationships involving the outer sheath material, which are non-linear.

A Review on Design of Upper Limb Exoskeletons [7] -

Authors: - Muhammad Ahsan Gull, Shaoping Bai and Thomas Bak.

Exoskeleton robotics has ushered in a new era of modern neuromuscularrehabilitation

engineering and assistive technology research. The technology promises to improve the upper-
limb functionalities required for performing activities of daily living. The exoskeleton
technology is evolving quickly but still needs interdisciplinary research to solve technical
challenges, e.g., kinematic compatibility and development of effective human–robot
interaction. In this paper, the recent development in upper-limb exoskeletons is reviewed.
The key challenges involved in the development of assistive exoskeletons are highlighted by
comparing available solutions. This paper provides a general classification, comparisons, and
overview of the mechatronic designs of upper-limb exoskeletons. In addition, a brief overview
of the control modalities for upper-limb exoskeletons is also presented in this paper. A
discussion on the future directions of research is included.

Analysis of Exoskeleton Introduction in Industrial Reality: Main Issues and EAWS Risk
Assessment [16] -

Authors: - Stefania Spada, Lidia Ghibaudo, Silvia Gilotta & Laura Gastaldi

Exoskeletons are part of the technological and organizational innovation sought by

the fourth industrial revolution to support and re-launch the manufacturing area. In the
present study, we described the experimental protocol designed to test the usability and
acceptance of an upper limb’s passive exoskeleton. In total, 42 workers from FCA plants
volunteered to participate in the research study. The testing campaign included static and
dynamic tests aimed at evaluating the potential benefit of the exoskeleton (lessen muscle strain,
higher comfort rating and dexterity) vs. possible restrictions to movements and work- device
interactions in tasks resembling work activities. Open questions remain on how to assess the
biomechanical workload risk, especially in the design phase, for which holistic methods like
EAWS are needed.

4
 Exoskeletons for industrial application and their potential effects on physical work load
[17] -

Authors: - Michiel P. de Looze, Tim Bosch, Frank Krause, Konrad S. Stadler

This review shows an wide interests in passive and active exoskeleton for industrial
purpose, but most developments are at early stage of technology development with many
concepts not tested beyond the laboratory. Passive industrial exoskeleton is aimed at supporting
or unloading the lower back region and appear to be quite successful herein for both dynamic
lifting or static holding activities. Some concerns have been raised regarding the potentially
negative effects associated with increasing leg muscle activity, high levels of discomfort and
muscle deconditioning. The potential effect in reducing physical loads seems to be even higher
for active exoskeleton. Both lower body, trunk and upper body regions could benefit from large
reduction in loading. Exoskeleton thus have the potential to considerably reduce the underlying
factors associated with developing work-relatedmusculoskeletal injuries. The true impact on
potentially reducing injury prevalence, however still needs to be determined, as until now
significant technical challenges and a lack of specific safety standards stands in the way of
large-scale implementation in workplaces.

A Brief review on upper extremity robotic exoskeleton systems [19] -

Authors: - Ruwan Chandra Gopura, Kazuo Kiguchi, Sanjaya Vipula Bandara

The robotic exoskeleton system, especially the exoskeleton system used for power
assist is a brand-new type of man-machine intelligent system which fully combines the human
intelligence and machine power. Advancement of this technology is a milestone of researchers’
effort to embed the intelligence to the robots since the exoskeleton robot is a man-machine
intelligence system. With the advances in recent technology in many fields including
mechanical engineering, electronic engineering, biomedical engineering, and artificial
intelligence the robotic exoskeleton technology has acquired a rapid development in recent
years. In this paper a brief review of upper extremity exoskeleton systems is presented. The
concept of an exoskeleton system was identified as an extension of the exoskeleton in biology.
The challenges for an upper extremity robotic exoskeleton were identified. Special efforts have
to be made when developing exoskeletons for human shoulder, wrist, and thumb since human
shoulder, wrist and thumb motions are biomechanically intricate. Upper extremity exoskeleton
robots can be classified according to the applied segment of the upper extremity, the DOF, the
actuators used in the system, power transmission methods, purposeof the robotic system
and/or the control methods.

5
 1.3 PROBLEM STATEMENT

Develop a stock sentiment market prediction system that accurately predicts the future direction of
stock prices based on sentiment analysis of news articles and social media data. The system should
leverage machine learning techniques to analyze large volumes of textual data and provide insights into
market sentiment, enabling investors to make informed trading decisions. The goal is to build a robust
and reliable model that outperforms traditional methods of stock market analysis and delivers
actionable predictions with high accuracy and precision.

1.4 OBJECTIVES

• The primary objective is to analyze the sentiment of market participants towards specific stocks or the
broader market.
• Another objective is to use the gathered sentiment data to predict the future movements of stock prices.
• Stock sentiment market prediction can also be used to assess the risk associated with specific stocks or
the overall market.
• Stock sentiment market prediction can serve as an early warning system for potential market
downturns or specific stock price fluctuations.
• The objective is to develop real-time monitoring systems that capture and analyze sentiment data as it
emerges. This enables investors to stay updated with the latest sentiment trends and make timely
investment decisions.

6
2. METHODOLOGY

2.1 Data Collection: Gather relevant data from various sources such as financial news websites, social
media platforms, company filings, analyst reports, and other financial information providers. This data
will serve as the foundation for sentiment analysis.

2.2 Preprocessing: Clean and preprocess the collected data to remove noise, irrelevant information, and
duplicate entries. This may involve removing stop words, punctuation, and special characters, as well as
normalizing the text (e.g., converting to lowercase).

Fig 2.1 : Data Collection And Preprocessing

2.3 Sentiment Analysis: Apply sentiment analysis techniques to the preprocessed data to determine the
sentiment associated with each piece of information. There are several approaches to sentiment analysis,
including:
• Lexicon-based methods: Use sentiment lexicons or dictionaries that assign sentiment scores to
words. By aggregating the scores of words in a document or text, an overall sentiment polarity
(positive, negative, neutral) can be determined.
• Machine learning-based methods: Train machine learning models, such as Naive Bayes, Support
Vector Machines (SVM), or Recurrent Neural Networks (RNNs), using labeled data to classify
sentiment. Labeled data consists of examples with known sentiment (positive, negative, neutral)
that are used to train the model.
• Natural Language Processing (NLP) techniques: Utilize NLP techniques like part-of-speech
tagging, named entity recognition, and sentiment analysis libraries (e.g., NLTK, spaCy) to
extract sentiment-related features and analyze the sentiment contextually.

2.4 Feature Extraction: Extract relevant features from the sentiment analysis results, such as sentiment
scores, sentiment intensity, sentiment trends, or sentiment-based indicators. These features will be used
as inputs for the predictive modeling stage.

2.5 Historical Data Analysis: Analyze historical stock price data and correlate it with the extracted
sentiment features. Identify patterns, trends, and relationships between sentiment and stock price
movements. This analysis can help identify the most significant sentiment indicators and their impact
on stock prices.
7
2.6 Predictive Modeling: Develop predictive models to forecast future stock price movements based
on the extracted sentiment features. Common modeling techniques include:
• Regression models: Utilize regression techniques (e.g., linear regression, logistic regression) to
predict stock price movements based on sentiment scores or sentiment-related indicators.
• Time series analysis: Apply time series forecasting models (e.g., Autoregressive Integrated
Moving Average, ARIMA) to capture the temporal dependencies and predict stock prices using
sentiment time series data.
• Machine learning algorithms: Employ machine learning algorithms, such as Random Forests,
Gradient Boosting Machines (GBM), or Long Short-Term Memory (LSTM) networks, to build
models that incorporate sentiment features and historical stock price data for prediction.

2.7 Model Evaluation: Assess the performance of the predictive models using appropriate evaluation
metrics (e.g., mean squared error, accuracy, precision, recall). Validate the models using historical data
that was not used during the model development phase.

2.8 Deployment and Monitoring: Deploy the predictive model in a real-time or near-real-time
environment to generate sentiment-based stock market predictions. Continuously monitor the
performance of the model and retrain/update it as new data becomes available.

8
3. CONCLUSION
In this project, we focused on utilizing sentiment analysis to predict stock market trends. We
implemented various models, including ARIMA, Facebook Prophet, and RNN LSTM, and conducted
extensive experiments to evaluate their performance. The results obtained from our experiments
provide valuable insights into the effectiveness of sentiment analysis in stock market prediction.

We observed that the ARIMA model, with its ability to capture short-term forecasting patterns, showed
promising results in predicting stock prices. By optimizing the model parameters through grid search
and selecting the best-fit model based on the Akaike Information Criteria (AIC), we achieved accurate
predictions for different time periods.

The Facebook Prophet model, designed to capture seasonality in stock prices, demonstrated its
effectiveness in predicting future stock price trends. By enabling yearly and daily seasonality and
carefully managing the changepoint prior scale parameter, we achieved robust and reliable predictions.

The RNN LSTM models, implemented with different approaches, showcased their ability to handle
arbitrary intervals of information and incorporate textual sentiment analysis. By experimenting with
various model configurations, we fine-tuned the parameters to achieve improved stock price prediction
accuracy. Incorporating sentiment polarity derived from textual data provided additional insights and
enhanced the predictive power of the models.

Overall, our project demonstrates the potential of sentiment analysis in stock market prediction. By
analyzing sentiment from various sources and combining it with traditional quantitative approaches,
we can gain valuable insights into market trends and make informed investment decisions.

However, it is important to note that stock market prediction is a challenging task influenced by
numerous factors, including market volatility, economic conditions, and unexpected events. While
sentiment analysis provides valuable inputs, it is not a standalone solution for accurate stock market
forecasting. Therefore, our models should be used as complementary tools alongside other fundamental
and technical analysis methods.

In conclusion, our project highlights the significance of sentiment analysis in stock market
prediction and provides valuable insights into the performance of different models. Further
research and enhancements in data collection, preprocessing techniques, and model
architectures can potentially improve the accuracy and reliability of sentiment-based stock
market prediction systems.

9
 REFERENCES

[1]. Gambhir, M., & Gupta, V. (2017). Recent automatic text summarization techniques:
A survey. Artificial Intelligence Review, 47(1), 1–66
[2]. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of
deep bidirectional transformers for language understanding. In Proceedings of the 2019
Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies (Vol. 1, pp. 4171–4186)
[3]. Liu, Y., Wang, L., Li, Z., Zhao, M., Deng, H., & Chen, H. (2020). Fine-tuning BERT
for extractive summarization. In Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics (pp. 1641–1650).
[4]. Cao, Z., Zhang, W., Li, F., Zhang, W., & Xu, C. (2020). Beyond Rouge-1: An
evaluation metric for text summarization. In Proceedings of the 43rd International ACM
SIGIR Conference on Research and Development in Information Retrieval (pp. 1607–
1616).
[5]. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of
deep bidirectional transformers for language understanding. In Proceedings of the 2019
Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies (Vol. 1, pp. 4171–4186).
[6]. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., & Gao, J. (2020). Unified
language model pre-training for natural language understand.

10
11