Assessing the Stock Market for new investors

Assessing the Stock Market for New Investors

A PROJECT REPORT
for
ARTIFICIAL INTELLIGENCE (ITE2010)
in
B.Tech. (IT)
by
Binayak Bishnu (20BIT0155)
4th Semester, 2nd Year

Under the Guidance of

Prof. CHIRANJI LAL CHOWDHARY
Associate Professor, SITE

School of Information Technology and Engineering

May-June, 2022
Assessing the Stock Market for new investors

DECLARATION BY THE CANDIDATE

I hereby declare that the project report entitled “Assessing the stock market for
new investors” submitted by me to Vellore Institute of Technology University,
Vellore in partial fulfilment of the requirement for the award of the course
Artificial Intelligence (ITE2010) is a record of bonafide project work carried out
by me under the guidance of Prof. Chiranji Lal Chowdhary. I further declare
that the work reported in this project has not been submitted and will not be
submitted, either in part or in full, for the award of any other course.

Place: Vellore Signature: Binayak Bishnu

Date: 21 April 2022

Assessing the Stock Market for new investors

School of Information Technology & Engineering [SITE]

CERTIFICATE

This is to certify that the project report entitled “Assessing the stock market for
new investors” submitted by Binayak Bishnu (20BIT0155) to Vellore Institute
of Technology University, Vellore in partial fulfilment of the requirement for the
award of the course Artificial Intelligence (ITE2010) is a record of bonafide
work carried out by them under my guidance.

Prof. Chiranji Lal Chowdhary

GUIDE
Asso. Professor, SITE
Assessing the Stock Market for new investors

Assessing the Stock Market for new investors

Binayak Bishnu1
1
Department of Information Technology, VIT University, Vellore, Tamil Nadu, India

Abstract
New investors are often overwhelmed when it comes to investing in the stock market and prefer an
automated system to guide them and help them make informed decisions. Neural networks play an
important role in developing machine learning algorithms that deal with sequential data like stock
market data. The goal was to determine an effective and reliable prediction system to predict the stock
market prices for multiple steps. This study aims to compare various predicting algorithms to
determine the best fit for the given context. The best fit is decided by how close the prediction is to
the real value and for how many steps can it maintain acceptable predictions. The study found that the
Long Short-Term Memory (LSTM) model matches these requirements. It is a modified form of
Recurrent Neural Networks (RNN) and overcomes various challenges like the vanishing gradient
problem. The literature review contributed toward determining various paths to develop such systems.
The implementation done as part of this study showed the effectiveness of the LSTM model.
Currently, it works on a sample dataset of one stock but can be scaled up to make a more
comprehensive and deployable model. The LSTM is fairly new and complex. The aim is to develop it
further to make it easier to implement while retaining efficiency and reliability. The simplicity of
implementation will also support further expansion.

Keywords – Stock prediction, Neural networks, LSTM

I. INTRODUCTION
Stock market prediction systems have been in high demand in recent times after artificial intelligence
and machine learning models became popular even in non-technical fields and workspaces. An efficient
predictor can not only help businesses and individuals to make informed decisions but also act as a guide for
those just starting. RNN models are quite effective in predicting sequential time series data due to their recursive
nature but cannot overcome the vanishing gradient problem. LSTM takes over and introduces a multi-level
recursive system with three gates to control the flow of information. It also decides which data to keep and
remove to be used for the next steps of prediction.

II. PRELIMINARIES
Preliminary concepts include the various neural networks namely, Convolution Neural Network, Recurrent
Neural Network and Deep Neural Network. It also includes knowledge of Python and Jupyter notebooks for the
implementation section.

III. LITERATURE SURVEY

Various algorithms and research papers have been explored, reviewed and compared. The following table
illustrates the findings.

Note: The following table contains the condensed findings from multiple papers. A list of references is
attached in the appendix.
Assessing the Stock Market for new investors

Name of technique Properties Advantages of Disadvantages of using this

techniques technique

Deep predictor Predicts the price movement
using candlestick of a given (k + 1)th day, by
charts taking the price trend of the
past k-days as input.
Input charts are easy to More towards qualitative
read. analysis which is limited to
giving the direction only.
RNN is better when
working with
sequential data.

The dynamic Best out of multiple models Works better than a Very large size. Ignores
predictor selection selected based on input. single model. Reduces feature dependency.
algorithm overfitting.

DNNs for Creates non-linear Reduces overfitting. Might lead to over-analysis

classification relationships between the concerning the scope of this
prediction independent and dependent Good balance of speed research.
variables. and capacity.

RNN prediction Retains memory of what it Design specifically for Training the model is slow
models has already processed sequential datasets. and complex.

CNN prediction Designed to map image data Not very dependent on Not ideal for sequential data
models to an output variable. pre-processing of
model or data
Assessing the Stock Market for new investors

IV. PROPOSED ALGORITHM WITH EXPLANATION

Long Short-Term Memory (LSTM) Model
LSTM is a modified form of recurrent neural network (RNN) architecture with feedback mechanisms.
It can process an entire sequence of data. LSTM model works best on making predictions on time series data
since important events in a time series have undetermined lags between one another.

Architecture

The dataset has been collected from the Kaggle dataset (HP stocks). The data is first passed through
some preprocessing to get an idea about how the data looks and check for any outliers since they disrupt the
training process for the model. This is done primarily using matplotlib.

The process data set (Final dataset) is then split into train and test sets. The training set is used to build
the model while the testing set is used to test the model created in every iteration of its building process. If there
are discrepancies, the test model guides the LSTM process to rebuild the model for better accuracy.

While the testing is going on, the model created in each iteration is evaluated and evaluation data is
also created to show how the model building builds and rebuilds itself.
Assessing the Stock Market for new investors

Module Description

1. Data collection and preparation

Data collected from Kaggle is visualized using matplotlib to check for outliers which
negatively affect the model building process.

The data is then prepared by first dividing it into train and test subsets. The data is then
normalized. Normalization is used to make all numerical columns into having the same scale. This is
necessary when combining columns for modelling. There are two methods:

1. Min-Max Scaling – Subtract the minimum value from the highest value and divide by
the range. Each new column has a minimum value of 0 and a maximum value of 1.
2. Standardization Scaling – Subtracting the mean of each observation and then dividing
by the standard deviation. Making mean=0, SD=1

Among these, min-max scaling has been used.

2. Training and testing the model

Assessing the Stock Market for new investors

3. Generating insights and results

After the LSTM is run, the prediction and the evaluation data are ready.

The output of the LSTM prediction of each iteration is plotted on a graph and compared to real values
to show the precision at each step.

The best iteration is then selected based on it having the least Mean Squared Error (MSE). This is then
plotted against the real values to show its precision.

V. EXPERIMENTS RESULTS

1. Collected data displayed

The dataset collected from Kaggle is loaded into the python project. Into the df variable
Assessing the Stock Market for new investors

2. Data visualized

The graph of the data is plotted to spot outliers.

3. Code divided into modules

Modularized the whole code for better comprehension.

Assessing the Stock Market for new investors

4. LSTM model running

The LSTM is running and printing the Mean Squared Error (MSE) of each step and the loss too.

5. Predictions from all cycles plotted together and compared to real data

As mentioned in the module description, results or all iterations are shown against the real value to
show the precision generated at each step.
Assessing the Stock Market for new investors

6. Best cycle chosen and plotted

The best iteration is chosen (smallest MSE) and plotted against the real values to show the final (and
the best) prediction of the LSTM model.

VI. COMPARATIVE STUDY

1. RNN vs LSTM

RNN has nodes which are joined by directed or undirected graphs. They are quite efficient in
predicting time series data. However, they have a major drawback.

RNN has the vanishing gradient problem wherein the gradients either become negligible or explode
to infinity during back-tracking. This is partly because they use numbers with finite precision.

On the other hand, LSTM allows the gradients to flow without making any changes to them. They
have multiple layers and more than one gate to handle the flow of data within the model.

2. Averaging vs LSTM

There are two averaging methods. However, both can be used only for one-step prediction, that is,
only the next value after the last value can be effectively predicted by these methods. The Standard
Averaging method predicts the data based on the average of N data points while the exponent moving
average gives more weightage to newer data to calculate the future data.
Assessing the Stock Market for new investors

VII. CONCLUSION AND FUTURE WORK

This presentation has introduced and given context to the topic of this research work. The literature survey
done has revealed various paths that can be taken to develop an effective and reliable model for stock prediction.
This work also involves a small implementation of the same using a sample stock price dataset which can then
be scaled up into a more comprehensive and deployable model.

The proposed architecture uses LSTM which is a fairly new and complex concept used for sequential data
prediction. The objective is to make the model easy to implement to support further expansion and work on this
while also retaining its efficiency and reliability. The concept of a bidirectional can be introduced where an
additional layer of LSTM is used and the outputs of both are combined to form a final output.
Assessing the Stock Market for new investors

REFERENCES
1. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0252404
DPP: Deep predictor for price movement from candlestick charts
Chih-Chieh Hung ,Ying-Ju Chen

2. https://www.sciencedirect.com/science/article/abs/pii/S0957417421011982
Dynamic Predictor selection algorithm for predicting stock market movement
Shuting Dong, Jiaxin Wang, Hongze Luo, Haodang Wang, Fang-Xiang Wu

3. https://ijcsit.com/docs/Volume%202/vol2issue3/ijcsit2011020322.pdf
Filter versus Wrapper Feature Subset Selection in Large Dimensionality Microarray: A Review Binita
Kumari#1, Tripti Swarnkar*2

4. https://www.sciencedirect.com/science/article/pii/S1877050920307237
Optimizing LSTM for time series prediction in the Indian stock market
AnitaYadav, C K Jha, AditiSharan

5. https://content.iospress.com/articles/algorithmic-finance/af176
Classification-based financial markets prediction using deep neural networks
Dixon, Matthew, Klabjan, Diego, Bang, Jin Hoon

6. https://www.sciencedirect.com/science/article/pii/S1877050918307828
NSE Stock Market Prediction Using Deep-Learning Models
Hiransha M, Gopalakrishnan E.A., Vijay Krishna Menon, SomanK.P.

7. https://neptune.ai/blog/predicting-stock-prices-using-machine-learning
Predicting Stock Prices Using Machine Learning
Author Katherine (Yi) Li
8. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5167535/
Structured prediction models for RNN based sequence labelling in clinical text
Abhyuday N Jagannatha1 and Hong Yu1,2

9. https://ieeexplore.ieee.org/abstract/document/7952591
Sequence segmentation using joint RNN and structured prediction models
Yossi Adi; Joseph Keshet; Emily Cibelli; Matthew Goldrick

10. https://www.sciencedirect.com/science/article/pii/S1877050916311619
Prediction Models for the Indian Stock Market
AparnaNayak, M. M. Manohara Pai, Radhika M. Pai

11. https://machinelearningmastery.com/gentle-introduction-long-short-term-memory-networks-experts/
A Gentle Introduction to Long Short-Term Memory Networks by the Experts
Jason Brownlee

12. https://www.mdpi.com/1999-4893/14/8/251
Comparative Analysis of Recurrent Neural Networks in Stock Price Prediction for Different Frequency
Domains
Polash Dey, Emam Hossain, Md. Ishtiaque, Mohammed Armanuzzaman Chowdhury, Md. Shariful
Alam, Mohammad Shahadat Hossain, Karl Andersson

13. https://www.researchgate.net/publication/220355039_The_Vanishing_Gradient_Problem_During_Lear
ning_Recurrent_Neural_Nets_and_Problem_Solutions
Assessing the Stock Market for new investors

The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions
Sepp Hochreiter

14. https://www.mdpi.com/2079-9292/10/21/2717
Stock Market Prediction Using Machine Learning Techniques: A Decade Survey on Methodologies,
Recent Developments, and Future Directions
Nusrat Rouf, Majid Bashir Malik, Tasleem Arif, Sparsh Sharma, Saurabh Singh, Satyabrata Aich, Hee-
Cheol Kim

15. https://www.ijeat.org/wp-content/uploads/papers/v8i4/D6321048419.pdf
Stock Market Prediction Using Machine Learning Algorithms
K. Hiba Sadia, Aditya Sharma, Adarrsh Paul, SarmisthaPadhi, Saurav Sanyal

16. https://iq.opengenus.org/disadvantages-of-rnn/
ITE2010-AI REVIEW 2 PRESENTATION

Assessing the Stock

Market for new
investors
A look at how ML models can predict and analyse the market
for investors who have just started out

Binayak Bishnu 20BIT0155

Discussion
Review 1 work

Final Review work

points Proposed algorithm

Key topics covered

Architecture and Module description
in this presentation
Comparative study

Other challenges

Conclusion

Future directions

Binayak Bishnu
References
Motivation

New investors are often overwhelmed when it comes to investing in the stock market and
prefer an automated system to guide them and help them make informed decisions.

Binayak Bishnu
Synopsis

The research deals with searching for and implementing

an efficient algorithm to analyse the stock market.

Various algorithm and learning models have been explore

to decide on the best fit(s) for the purpose of this research

Binayak Bishnu
Introduction 40

30

Stock market prediction systems have been in

high demand in recent times after artificial
20
intelligence and machine learning models became
popular even in non-technical fields and
workspaces. 10
An efficient predictor can not only help businesses
and individuals to make informed decisions but
also act as a guide for those just starting out. 0
Qtr1 Qtr2 Qtr3 Qtr4

Binayak Bishnu
Preliminaries
Useful pre-existing concepts

Movement prediction ANN neuron

target = 1, Cd > μ+α*σ
0, otherwise
for dth day
μ -> mean of price
σ -> SD of price about μ
α -> user-defined
Binayak Bishnu
LSTM Networks
Long Short Term Memory
A = Σ(XiWi) + b
networks
Output = F(A)
Well-suited to classifying,
processing and making
predictions based on time
series data.
 Literature
Review
Binayak Bishnu
 Technique Property Advantage Disadvantage

Predicts the price movement of

Input charts are easy to read. More towards qualitative
Deep predictor using a given (k + 1)-th day, by taking
RNN is better when working analysis which is limited to
candlestick charts price trend of the past k-days as
with sequential data. giving the direction only.
input.

Dynamic predictor selection Best out of multiple models Works better than single model. Very large size. Ignores feature
algorithm selected based on input. Reduces overfitting. dependency.

Creates non-linear relationships
DNNs for classification
between the independent and
prediction
dependent variables.
Reduces overfitting. Might lead to over analysis with
Good balance of speed and respect to the scope of this
capacity. research.

Retains a memory of what it Design specifically for Slow and complex training
RNN prediction models
has already processed sequential datasets. procedures.

Designed to map image data to Not very dependent on pre- A lot of data required for proper
CNN prediction models
an output variable. processing of model or data training

Binayak Bishnu
 Final review
Work
Binayak Bishnu
Proposed Algorithm
LONG SHORT-TERM MEMORY MODEL (LSTM)

LSTM is an artificial recurrent neural network (RNN) architecture. It has feedback

mechanisms too. It can process an entire sequence of data.

LSTM networks are well-suited to classifying, processing and making predictions

based on time series data since there can be lags of unknown duration between
important events in a time series.
Hence standard RNNs fail to learn in the presence of time lags greater than
5 – 10 discrete time steps between relevant input events and target
signals. The vanishing error problem casts doubt on whether standard
RNNs can indeed exhibit significant practical advantages over time
window-based feedforward networks. A recent model, “Long Short-Term
Memory” (LSTM), is not affected by this problem. LSTM can learn to bridge
minimal time lags in excess of 1000 discrete time steps by enforcing
constant error flow through “constant error carrousels” (CECs) within
special units, called cells.

FELIX A. GERS

Professor of Computer Science at Beuth University of Applied Sciences Berlin

Architecture and Module
Description
Architecture
Dataset Pre-processing Final dataset

From kaggle Using matplotlib and seaborn Ready for modelling

Train Building model to train

Using LSTM
Rebuilding based
on the result

Test Testing model

Evaluation data
1. Data collection and preparation

Data collected from

Kaggle database
Visualizing the trend
in the data

Using matplotlib and seaborn

Why normalize?
Make all numerical columns into a
common scale Dividing data into train
Necessary when combining them for and test subsets
modelling

Standardization scaling

Min-max scaling Normalising the data

With min-max scaling

2. Training and testing

Datagenerator for Defining the

training hyperparameters
Calculate loss and
A custom made python class created Variables assigned values
before running the mode
define optimizer

Calculations and
prediction

Plotting the predictions Running LSTM

3. Generating insights and results

LSTM outputs generated

Best cycle selected

to display predictions

Other cycles also

plotted for comparison
Implementation
Gathering and
plotting data
Running LSTM
Output and evaluation
Comparative Analysis
Part 1
[Vanilla] RNN vs LSTM

Nodes are joined by directed and/or LSTM is a more sophisticated RNN

undirected graphs
Contain gates that regulate the
Prediction problems flow of information through the
Speech Recognition unit better
Generating Image Descriptions
Text Summarization Solves the vanishing gradient
Call Center Analysis problem in RNN
Face detection
OCR - Image Recognition

Predicting future data based on
average of observed data
Expanding to more than one step
leads to incorrect predictions
Prediction restricted to one step
(next day/session)
1. Standard averaging
2. Exponential moving average
Part 2
Averaging vs LSTM
Arbitrary number of prediction
steps possible in LSTM
Contains gates that regulate the
flow of information through the
unit better
Solves the vanishing gradient
problem in RNN
Standard Averaging
P(t+1) = (ΣPi)/N
P(t+1) = price next day or session
Pi = observed prices
N total number of days

Exponential moving average

P(t+1) = γ*EMA(t-1) + (1-γ)Pt
EMA = Exponential Moving Average
γ = % contribution of recent value
Other Challenges
Additional SELECTION BIAS

challenges Arbitrary stocks used

Must consider the entire

and set/domain from where stock

is picked
PRE-PROCESSING
ERROR
drawbacks Standardisation applied
to both train and test
LOOK AHEAD BIAS Train-set parameters should be
taken to apply on the test-set
Data based on
assumptions
The data used was not
released during the period of
time being analysed
Conclusion
 This presentation has introduced and given context to the topic of this
research work.
The literature survey done has revealed various paths that can be taken
to develop an effective and reliable model for stock prediction.
This work also involves a small implementation of the same using a
sample stock price dataset which can then be scaled up into a more
comprehensive and deployable model.

Binayak Bishnu
Future Directions
 The proposed architecture uses LSTM which is a fairly new and complex
concept used for sequential data prediction.
The objective is to make the model easy to implement to support further
expansion and work on this while also retaining its efficiency and reliability.

The concept of a bidirectional can be introduced where an additional layer

of LSTM is used and the output of both are combined together to form a
final output.

Binayak Bishnu
References
https://journals.plos.org/plosone/article?
id=10.1371/journal.pone.0252404
https://www.sciencedirect.com/science/article/abs/pii
/S0957417421011982
https://ijcsit.com/docs/Volume%202/vol2issue3/ijcsit2
011020322.pdf
https://www.sciencedirect.com/science/article/pii/S18
77050920307237
https://content.iospress.com/articles/algorithmic-
finance/af176
https://www.sciencedirect.com/science/article/pii/S18
77050918307828
https://neptune.ai/blog/predicting-stock-prices-using-
machine-learning
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5167535/
https://ieeexplore.ieee.org/abstract/document/7952591
https://tinyurl.com/Model4IndiaStock
https://machinelearningmastery.com/gentle-introduction-
long-short-term-memory-networks-experts/
Comparative Analysis of Recurrent Neural Networks in
Stock Price Prediction for Different Frequency Domains
The vanishing gradient problem during learning recurrent
neural nets and problem solutions
Stock Market Prediction Using Machine Learning
Techniques: A Decade Survey on Methodologies, Recent
Developments, and Future Directions
https://www.ijeat.org/wp-
content/uploads/papers/v8i4/D6321048419.pdf
Do you have any questions?

Binayak Bishnu