Commodity and Forex trade automation using Deep
Reinforcement Learning
Usha B A
Professor and Head,
Department of ISE,
BMSIT&M
Bengaluru, INDIA
ushaba@bmsit.in
Abstract— Machine learning is an application of artificial
intelligence based on the theory that machines can learn from
data, discern patterns and make decisions with negligible
human intervention. With today's world being inundated by
data, machine learning is very relevant due to the amount of
learning potential. Machine learning caters to a variety of
applications including image recognition, speech recognition,
weather prediction, portfolio optimization and so on. The
Forex Exchange is a market that allows traders and investors
to buy, sell and exchange currencies of various nations. It is
regarded as the largest financial market with over 5 trillion
American dollars in daily trades, which is larger than the
equity and futures markets combined. The Commodity market
is a market that allows buying, selling and exchanging of raw
materials or primary products. Using the concept of machine
learning, this project aims to develop and introduce an agent to
automate the trade of a given commodity or currency in a
simulated market with the objectives of maximizing returns
and minimizing losses for the trader. The model learns from
trends in historical market data and is capable of buying,
selling or holding a trade at a given instance. The model is
validated by running the agent on unseen market data of a
later period and the returns generated are analyzed.
Keywords— machine learning, deep learning, reinforcement
learning, commodity trading, forex trading
I. INTRODUCTION
Machine learning has been the cornerstone of solutions to
most problems in computer science over the last few
decades. It has made its way into most modern applications
of information technology and with that, a rather central,
although usually latent, part of our lives. With the colossal
amounts of data made available to researchers and the public
in general, there is a good reason to believe that effective and
innovative data analysis will become even more prevalent as
a necessary integrant for technological advancement [1].
Machine Learning can be widely classified into supervised
learning, unsupervised learning and reinforcement learning.
Most of the existing machine learning systems use
supervised learning or deep learning to predict prices of
currencies and commodities based on historical market data
– making it a regression problem. This essentially aids
human traders to make informed decisions and take
calculated risks. Since the goal of trading currencies and
commodities is to make a profit and not merely give
predictions, our experiment, instead of just giving out
predictions, uses deep reinforcement learning to make
trading decisions without human intervention [2].
Reinforcement learning uses intelligent bots or agents that
978-1-7281-0418-8/19/$31.00©2019 IEEE
Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on March 12,2024 at 12:06:30 UTC from IEEE Xplore. Restrictions apply.
Manjunath T N Thrivikram Mudunuri
Professor, Department of ISE,
Department of ISE, BMSIT&M
BMSIT&M Bengaluru, INDIA
Bengaluru, INDIA tmudunuri@outlook.com
manju.tn@bmsit.in
learn from an environment by interacting with it over a
period of time. This is accomplished by receiving feedback
from the environment. The agent or bot is rewarded and/or
penalized based on the actions taken by it. The agent uses
this feedback to decide its next course of action on the
environment with the goal of maximizing its reward for
every action.
The union of deep learning and reinforcement learning
serves well to address the profit optimization problem in
forex and commodity trading. The associated neural
network sequential model is involved in action prediction
and the reinforcement learning agent is used to make
decisions to maximize reward. Both working in tandem help
optimize profit generation. The action space of trading is
simplified to include either buy, hold or sell. This makes
trading an ideal application of deep reinforcement learning
[3]. Our agent is trained using historical market data to
observe price trends of currencies and commodities. The
agent is then run on an unseen period of forex and
commodity market data in order to maximize profit
generation. The return on investment is calculated as a
performance measure of the agent. The goal is to produce a
model that maximizes returns while also minimizing losses
in a bear market.
II. LITERATURE SURVEY
Typical Reinforcement learning problems encompass
solutions to learning what to do—how to map situations to
actions—with the objective of maximizing a numerical
reward signal. They are essentially closed-loop problems as
the learning system’s actions influence its later inputs. Most
importantly, the learner is not informed of the recommended
actions, as in many forms of machine learning, but instead
must discover by itself which actions yield the most reward
by trying them all out. In the most interesting and
challenging cases, actions may affect not only the immediate
reward but also the next situation and, through that, all
subsequent rewards which is what makes reinforcement
learning critical in most applications [4].
Supervised learning is the type of machine learning that
is currently studied in the most current research. It involves
learning from a training data set of labelled examples
provided by a competent external overseer. The labelled
examples from the training dataset consist of descriptions
and specifications — the label of the ideal action the system
must take in a given situation. The goal of supervised
learning is to develop a system that is capable of
extrapolating and generalizing its responses to make accurate
27
 predictions for situations not presented in the training set.
Despite its relevance and significance in most applications, it
is inadequate for learning from interaction with an
environment. Reinforcement learning is different from
supervised learning in that it learns from interaction and
would complement the supervised learning system [4]. This
reiterates that supervised learning, by itself, is not the best
approach to a profit optimization problem in forex and
commodity trading.
Google's DeepMind had published a paper in 2013
presenting a technique of combining deep learning and
reinforcement learning to play games with a discrete action
space. This has achieved tremendous results overcoming the
drawbacks of each type of learning by supplementing each
other [5].
Forex markets are the largest markets in the world and
commodity markets deal in primary products and raw
materials making them relatively attractive to an amateur
investor who may not be comfortable in buying equity stock
[6]. In this paper, we have attempted to solve the problem of
automated profit generation and loss minimization in the
forex and commodity markets using the deep reinforcement
learning.
III. PROBLEM DEFINITION
This paper tries to solve the problem of automated profit
generation in the forex and commodity markets. Profit is
defined as the amount of money gained minus the amount of
money spent.
Investment is the total money spent on the purchase of
currency or commodity. The returns is the amount of money
obtained on sale of the purchased currencies or commodities.
For a given currency or commodity  the purchase cost
and the sale price may be given as  and  respectively.
Hence profit  per unit currency or commodity is given as:
=–
Return on investment is a financial metric of profitability
that is widely used in the industry and academia to measure
the return or gain from an investment. Return on investment
or ROI is a simple ratio of the gain from an investment
relative to the total expense of making the investment
including trading fee, taxes and so on. It is very useful in the
evaluation of the potential return from a stand-alone
investment [7]. Return on Investment  is used as a
performance measure of the obtained results.  may be
expressed as a percentage as:
 = 100 * ( – ) 
IV. METHODOLOGY
This project paper is organized as follows:
A. Data Collection
B. Data Preprocessing and Formatting
C. Agent Training
D. Agent Testing
A. Data Collection
The datasets are detailed in Table I:
Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on March 12,2024 at 12:06:30 UTC from IEEE Xplore. Restrictions apply.
TABLE I. DATASETS
Category Dataset
Gold (XAU/USD)
Commodities Crude Oil (OPEC)
Silver (XAG/USD)
United States Dollar $ (USD/INR)
Forex
Euro € (EUR/INR)
Historical prices of Gold and Silver are obtained from
investing.com as XAU/USD and XAG/USD trading pairs. It
denotes the price of 1 Troy Ounce of pure Gold and 1 Troy
Ounce of pure Silver in United States Dollars.
Historical crude prices are obtained from the
Organization of the Petroleum Exporting Countries in
United States Dollars. The price reflects the cost of crude oil
per barrel on a given day.
Historical prices of USD and EUR are obtained from
investing.com as USD/INR and EUR/INR trading pairs. It
denotes the price of 1.00 United States Dollar and 1.00 Euro
in Indian Rupees.
The obtained data is stored as separate relations before
any further processing.
B. Data Preparation and Formatting
All the datasets are reduced to two fields namely date and
price. The obtained datasets are first checked for null values
and missing values in any of the fields. Rows with null
values and missing values are removed. Since the data is
presented as time-series values, the date field is treated as the
index and hence converted into date objects for easy
manipulation.
()
The dataset of historical prices of currencies and
commodities is not entirely continuous due to trading
holidays across various markets. For convenience and easy
comparison of results across markets, multiple inner joins are
performed to obtain a coherent dataset with prices of all the
commodities and currencies for all the given days in a
period. i.e. only prices of currencies and commodities traded
on the same corresponding day are included in the dataset to
ensure accurate plotting of graphs with date as one of the
axes. This exercise also allows analysis of the relation
between price changes of various items.
Prices of commodities and currencies in USD are
() multiplied with the value of the USD/INR trading pair on the
corresponding date to obtain the price of the items in Indian
Rupees.
Prices of Gold and Silver are divided by 31.1035 to
present the rates as cost per gram as they are initially present
in Troy Ounce unit. This ensures granularity in trade
execution as prices of Gold and Silver in Troy Ounce may be
significantly high for a given initial investment amount.
Oil prices are divided by 158.987 to present the values as
cost per liter as they are initially present in cost per oil barrel
unit. This ensures granularity in trade execution as prices of
28
 oil per oil barrel unit may be significantly high for a given V. RESULTS
initial investment amount. The absolute return, initial investment and return on
The dataset is then divided into a test dataset and a train investment are obtained for multiple runs of the agent on the
dataset based on a selected fixed date value. The train dataset market data and is presented in Table III:
consists of values indexed lower than the fixed date value
and the test dataset consists of values indexed higher than the
fixed date value. This exercise translates to the prediction of TABLE III. PERFORMANCE
future prices based on historical or past market data.
Metrics
C. Agent Training Market Run
The training process involves the definition of the neural Return a. Investment a. Market ROI
network’s layers and tuning of the hyperparameters to
achieve optimal weights. The hyperparameters are specified Gold I -1289.26 133034.25 -46.91% -0.97%
in Table II. In a time-series problem, since each trade affects Gold II 10900.47 187977.64 -46.91% 5.8%
the next trade, care must be taken to achieve the right
parameters. This is a crucial activity and could possibly Commodity Oil I 53.29 2335.40 -1.27% 2.28%
determine the long-term outcome of the agent. The window
size is set to 10 to prevent overfitting. Oil II 180.83 1879.22 -1.27% 9.62%

The mean squared error is used as the loss function for Silver 53.29 2335.40 -3.47% 2.28%
the experiment. It measures the average of the squares of the
errors. The optimizer used is Adam optimizer. The learning EUR 71.62 3512.21 3.95% 2.04%
Forex
rate is 10−3. The training is carried out for 800 episodes in USD -7.47 2436.28 -5.33% -0.31%
mini batches of 32.
a.
Values in Indian Rupees ₹
TABLE II. HYPERPARAMTERS OF THE NEURAL NETWORK

Hyperparameter Value

Window size 10

Batch size 32

Learning rate 10-3

Episodes 800

Gamma 0.90

Epsilon 1.0

Epsilon Min 0.02

Epsilon Decay 0.990

Train Set Range 2010-01-01 - 2017-12-31 Fig. 1. Gold I Trades in 2018

Test Set Range 2017-01-01 - 2018-11-30

D. Agent Testing
The trained agent is tested on unseen market data of a
later period obtained from the same sources and allowed to
execute trades in a simulated environment. The action space
consists of buy, sell and hold. The absolute return, initial
investment and return on investment is computed to measure
the performance of the trained agent.
The return on investment (expressed as percentage)
achieved by the agent is compared to the actual return from
the market. With the cost of an item on the first day of the
trading period represented by  and the cost of the same item
on the last day of the trading period represented by , the
actual market return  is computed as (expressed as
percentage): Fig. 2. Gold II Trades in 2018

=– ()

29

Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on March 12,2024 at 12:06:30 UTC from IEEE Xplore. Restrictions apply.
 Fig. 3. Oil I Trades in 2018 Fig. 6. Euro Trades in 2017

Fig. 4. Oil II Trades in 2018 Fig. 7. US Dollar Trades in 2017

The market returns are calculated as change in the price

of the commodity or currency between the first and last date
of the given trading period. This reflects the return generated
upon purchase of the item on the first day of the trading
period and sale of the same item on the last day of the trading
period.
The losses incurred without the use of the model are
significantly high in bear markets. The returns generated
without use of the model are noticeably lower than the
returns generated by the model. It is therefore inferred that
the agent is able to consistently generate higher returns and
minimize losses in bear markets.
Fig. 5. Silver Trades in 2017 VI. CONCLUSION
This experiment proposes a deep reinforcement learning
method to solve the problem of automated trade execution
for profit generation. Deep reinforcement learning has thus
proven successful in the development of an automated
trading system. This agent does not use any financial concept
to derive insights or boost performance.
On comparison of the market return and the agent’s
return on investment, it is observed that the agent is able to
produce a positive profit and outperform the market on

30

Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on March 12,2024 at 12:06:30 UTC from IEEE Xplore. Restrictions apply.
 multiple occasions. The agent is also especially effective
during bear runs to minimize losses.
The major limitation of the presented approach is the
agent’s ability to produce a profit only in the short term.
Despite the availability of ample historical data, the model is
unable to capture long term trends due to short term state
representation. The agent, however, is able to successfully
detect short term trends and execute trades accordingly.
Future work could focus on the agent’s ability to run for
longer periods taking long-term returns into consideration.
This would aid most long-term investors who do not have the
time or expertise to constantly monitor the markets. A one-
time setup could possibly continue to generate income for an
investor with minimal human intervention or expertise.
ACKNOWLEDGMENT
We express our gratitude to the Department of
Information Science and Engineering, B.M.S. Institute of
Technology and Management for providing the necessary
infrastructure, guidance, and motivation to undertake this
experiment.
Authorized licensed use limited to: LAHORE UNIV OF MANAGEMENT SCIENCES. Downloaded on March 12,2024 at 12:06:30 UTC from IEEE Xplore. Restrictions apply.
REFERENCES
[1] A. J. Smola and S. Vishwanathan, Introduction to Machine Learning,
Cambridge University Press, 2008.
[2] Z. Jiang and J. Liang, "Cryptocurrency Portfolio Management with
Deep Reinforcement Learning," in Intelligent Systems Conference,
London, 2017.
[3] H. v. Hasselt, A. Guez and D. Silver, "Deep Reinforcement Learning
with Double Q-Learning," in The AAAI Conference On Artificial
Intelligence, Phoenix, 2016.
[4] R. S. Sutton and A. G. Barto, Reinforcement Learning:, Cambridge,
Massachusetts, London, England: The MIT Press, 2015.
[5] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D.
Wierstra and M. Riedmiller, "Playing Atari with Deep Reinforcement
Learning," DeepMind Technologies, NIPS Deep Learning Workshop,
2013.
[6] J. Chen, "Forex - FX," Investopedia , 2018. [Online]. Available:
https://www.investopedia.com/terms/f/forex.asp.
[7] A. Beattie, "FYI on ROI: A Guide to Calculating Return on
Investment," Investopedia, 3 December 2018. [Online]. Available:
https://www.investopedia.com/articles/basics/10/guide-to-calculating-
roi.asp.
31