Available online at www.sciencedirect.

com

Borsa _Istanbul Review

_
Borsa Istanbul Review 19-4 (2019) 283e287
http://www.elsevier.com/journals/borsa-istanbul-review/2214-8450

Review

Big data in finance: Evidence and challenges

Avanidhar Subrahmanyam
Anderson School, UCLA, Los Angeles, CA, 90095-1481, USA
Received 25 June 2019; accepted 26 July 2019
Available online 8 August 2019

Abstract

I review the literature on the use of big data sets in finance. While big data and machine learning are exciting fields, there is a danger that we
get carried away by the novelty of the topics but pay less attention to the reliability of the academic evidence. Following my review, I argue that
big data-based evidence should be held to the same academic standards as the rest of the finance academic literature if we are to provide useful
advice to finance practitioners.
_
Copyright © 2019, Borsa Istanbul Anonim Şirketi. Production and hosting by Elsevier B.V. This is an open access article under the CC BY-NC-
ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

JEL classification: G12; G14; G40

1. Introduction of the datasets without ensuring that these datasets actually

Of late, the availability of big data has spurred interest
amongst finance academics. These datasets promise to bring in
new data to bear on the field of investment management, and
consequently they have attracted much attention. While big
datasets do hold a lot of promise, there are caveats associated
with their usage that need to be mentioned if we are to make
material progress in understanding equity markets and honing
in on reliable signals which can improve the efficacy of money
managers.
For a start I would like to categorize big data into three parts.
First, there is human generated data via social media, and on
various online forums, which can be analyzed using standard
tools for text analysis. Second, there is process generated data,
via the act of purchases and sales, such as credit card receipts,
supermarket scanners and so on. Finally, there is machine
generated data, such as tracking GPS movements of delivery
trucks, and satellite-based images of parking lots.
New datasets like the ones delineated above naturally
excite academics, but it is important to note that there is the
potential that academics can get carried away with the novelty
E-mail address: asubrahm@anderson.ucla.edu.
_
Peer review under responsibility of Borsa Istanbul Anonim Şirketi.
add value. As an example, take the case of GPS movements of
delivery trucks. While these can in principle be used to better
predict company revenues, analysts already disseminate sales
forecasts (pretty much for free). It would be good to ascertain
that before paying for such expensive datasets, that they
actually add value over available forecasts. Similarly, consider
measuring the number of cars in parking lots of Walmart or the
queues at McDonald's. Again, while these can in principle be
used to measure sales at these companies, one should first
ascertain that simple time series forecasting models do not
already predict sales in a reliable fashion. That is tried and
trusted techniques should not take a back seat to novelty.
Big data is of interest to authors as well as editors; authors
want to further their career while editors want to get a high
citation count for their journal articles. The availability bias
(Tversky & Kahneman, 1973) exacerbates the appeal of novel
data. These aspects have led to a flurry of work using novel
data sets at the major finance journals. However, while these
exciting new studies analyze different aspects of big data, I
maintain that we have not made much headway in ensuring the
value-added nature of big datasets. Many of these studies use
limited samples, and/or use inadequate controls, that leave us
wondering whether studies are picking up unique financial
phenomena. In the next section, to make this point, I review

https://doi.org/10.1016/j.bir.2019.07.007
_
2214-8450/Copyright © 2019, Borsa Istanbul Anonim Şirketi. Production and hosting by Elsevier B.V. This is an open access article under the CC BY-NC-ND
license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
284
A. Subrahmanyam / Borsa Istanbul Review 19-4 (2019) 283e287
some of the major pieces of research on the usage of big data
in finance.
2. The research findings
One of the very first examples of big data applied to finance
is Tetlock (2007). This paper considers whether the textual
content of a popular newspaper column “Abreast of the
Market” in the Wall Street Journal predicts market returns.
Tetlock finds that the impact of the column on the Dow Jones
Industrial Average is first negative, followed by a partial
reversal. This suggests that the market overreacts to extremely
negative news. The analysis spans a fifteen year period starting
from 1984. Yet, there is not much attempt to control for
massive negative news that could affect markets in a lumpy
way. Is it news events or the actual sentiment that predicts
returns? Text analysis has also been applied to predict com-
panies' fundamentals such as earnings (Tetlock, Saar-
Tsechansky, and Macskassy, 2008).
Both the preceding papers use the nonproprietary Harvard
General Inquirer data to classify words into those conveying
positive and negative connotations. Antweiler and Frank
(2004) use naïve Bayes algorithms to perform computational
linguistic analyses of internet bulletin board posts. They find
that greater disagreement in messages and increased message
postings are both associated with increased volatility, though
the directional effect on returns of message board posts is
small. Their analysis uses a single year of data. Loughran and
McDonald (2011) argue that the HGI lists and naïve Bayes
approaches do not work effectively for financial applications
and come up with a new list of positive and negative words
specifically applicable to finance. They find that these words
bear significant relations to trading volume, volatility, earnings
surprises, as well as incidences of corporate fraud.
In other work on textual data, Hoberg and Phillips (2010)
use a text-based analysis of product descriptions in 10K
statements to argue that firms with similar product descriptions
are more likely to conduct mergers. Jiang, Lee, Martin, and
Zhou (2019) construct an index of managerial sentiment
using textual analyses of corporate disclosures and show that
this index strongly predicts aggregate market returns. Li
(2010) uses a machine learning approach instead of a dictio-
nary. Specifically, he trains a machine on a manually classified
set of managerial disclosures, and then uses the learnt machine
to show that managerial disclosures predict accounting fun-
damentals. He also shows that the negative impact of accruals
on future stock returns (Sloan, 1996) only occurs when man-
agers do not guide the investing public via their disclosures
about impending bad news.
Coval and Shumway (2001) point a directional microphone
into a futures pit for trading Treasury Bond futures and show
that ambient sound level increases point to an increased level
of volatility. This is a creative attempt to ascertain how non-
financial variables impact the stock market. It indicates that
other forms of online chatter (number of posts, number of
visits to companies' websites) could predict returns. Along
these lines Mayew and Venkatachalam (2012) use conference
_
call audio files to measure positive and negative dimensions of
a manager's affective or emotional state using commercially
available software. They find that emotional states predict the
direction of future accounting fundamentals.
Another recent example of big data analysis is by Huang
(2018), who considers product quality ratings by Amazon
customers over a 12 year period starting from 2004. This type
of data is not easily available to investors and could potentially
affect returns with a lag. Huang finds that indeed, such
customer ratings predict returns to the tune of a 1% per month
return differential between firms falling in the highest/lowest
ratings deciles. The paper is unique in bringing subjective
quality perceptions into objective return predictions, and, as
such, should attract much attention. However, the paper does
not control for analysts' forecasts of sales and earnings, and
associated revisions, thus leaving open the possibility that
forecasts by other agents dominate quality perceptions.
Tirunillai and Tellis (2012) consider the impact of the volume
(number) of opinions and the actual ratings on stock returns at
the daily level. Using a vector autoregression framework, they
find that the volume (but not the number) of posts predict a
daily company's abnormal returns a few days ahead.
Chen, De, Hu, and Hwang (2014) consider opinions posted
on a popular investment blog “Seeking Alpha.” They perform
text analysis on the posted opinions using the list constructed
by Loughran and McDonald (2011) and show that the pro-
portion of negative words in the posts on a given stock are
associated with lower returns, both contemporaneously, and in
the future. The total sample period is about nine years. It re-
mains to be seen if these results can be extended to other
investor blogs.
Cohen and Frazzini (2008) address the issue that events
that affect a firm's customer firms impact the original firm
with a lag. Does keeping track of customer firms allow
abnormal returns to the original firms? Using 25 years of data,
they find that indeed, there is an extra 1.5% per month return
premium for going long (short) on stocks with customer re-
turn in top (bottom) 20%. While this paper does not directly
use big data, it does imply that keeping track of news stories
of “downstream” firms affects profitability of trading “up-
stream firms.” This forms a promising line of thinking.
Nonetheless, the paper does not use other proxies for agents'
expectations such as analysts' forecasts, leaving open the
possibility that customer prospects could be incorporated into
analysts' expectations.
Chordia, Green, and Kottimukkalur (2018) address the
issue that there is a huge developing market for obtaining news
feeds split seconds before others. Does getting such macro
news seconds ahead of its release predict returns? Using
intradaily data, they find that the returns to trading on news
around macroeconomic announcements (such as inflation,
GDP, unemployment, etc.) are fairly small amongst various
combinations of entry/exit times, attain a maximum of 8bp for
entering within 0.1 s and exiting by 5 s. Such modest levels of
profit imply that the returns to attaining information seconds
before others are fairly modest, indicating a good level of
market efficiency at the intraday horizon. However, as
 _
A. Subrahmanyam / Borsa Istanbul Review 19-4 (2019) 283e287
technology improves and information becomes available to
trade upon microseconds sooner, it remains to be seen whether
the profits continue to remain at low levels or increase.
Hvidkjaer (2008) carefully calculates order imbalances on
the New York Stock Exchange by processing billions of data
points, separately for small and large investors (categorized by
trade size). Using 23 years of data, he documents that trade
imbalances of small investors are negatively related to future
stock returns in the cross-section over the subsequent one to
twelve months. While the exact rationale for this finding re-
mains to be explored, the result is consistent with the notion
that small investors overreact to information, and the reversal
of their sentiment may cause stock return predictability. Using
intradaily transactions data, Gao, Han, Li, and Zhou (2018)
show that for the S&P 500 ETF, the first half hour of returns
predicts the last half hour of returns. They attribute this finding
to late informed trading at the open and close (the open due to
the need to act quickly on information and the close to act on it
before the information perishes overnight).
Chordia and Subrahmanyam (2004) show that individual
stock order flow predicts returns at the daily horizon as well.
Heston, Korajczyk, and Sadka (2010) use comprehensive
intradaily data to show that returns during a particular half-
hour period during a day predict returns at the same half-
hour interval over about 40 trading days. This is an
intriguing finding which lacks a convincing explanation at
this point.
Using data from millions of retail investors' trades obtained
from brokerage firms, Kelley and Tetlock (2013) show that
aggregate net buying of retail investors predicts returns posi-
tively at the monthly level with no reversals, suggesting that
aggregate retail buying contains valuable information. In a
followup paper Kelley and Tetlock (2016) show that aggregate
short sales of retail investors negatively predict returns. The
data used by Kelley and Tetlock form an appealing way to
measure the aggregation of diverse retail opinions and to see if
the aggregate opinion is predictive of stock returns. Using data
on a stock ranking game played on a website, Da, Huang, and
Jin (2018) show that aggregate rankings of stocks are nega-
tively associated with future returns; i.e., the higher the
ranking, the lower the subsequent return. That is, rankings
appear to overprice stocks. This result does not dovetail with
the work of Kelley and Tetlock, who seem to suggest that
retail investors, on aggregate, appear to possess fundamental
information about securities.
Neely, Rapach, Tu, and Zhou (2014) show that simple
forms of technical analysis, such as moving average rules,
strongly forecast stock returns. These rules entail a buy
signal when a short-term moving average crosses a long-term
one from below, and vice versa. The profitability of simple
rules is intriguing and deserves further analysis to discern the
psychological biases that permit the abnormal returns from
these rules.
Da, Engelberg, and Gao (2011) argue that interest in a
company can be conveyed via searches for the company on
search engine. So, does internet search volume predict returns?
Using Google Trends to measure such search volume, they
285
find that a one standard deviation cross-sectional increase in
search volume increases returns in the subsequent week by 18
basis points. After one week, the impact of search volume tails
off rapidly. They also find that the highest search volume IPOs
earn a 7% more first day return than the lowest ones. The
paper is based on a relatively short sample of five years,
leaving open the issue of whether the results are robust to
longer time periods, including more recent years. Da,
Engelberg, and Gao (2014) show that investors' searches of
terms such as “recession” and “unemployment” predicts short-
term reversals and increases in market volatility. This study is
also based on a short sample of seven years. Once again, this
study does not seem to incorporate analysts' expectations, nor
does it control for retail trades as in Kelley and Tetlock (2013,
2016). It also seems to contradict the wisdom in Da, Huang,
and Jin (2018) that naïve investors overprice stocks.
3. Future research directions
There are a large number of exciting datasets being made
available for commercial usage. For example, firms like
RSMetrics provide detailed analyses of data from satellites,
drones and planes to estimate retail traffic, which can be used
to synthesize buy/sell equity signals. Firms like Eagle Alpha
provide data on emailed customer receipts to again estimate
company revenues. Ravenpack analyzes media coverage
which is translated into a sentiment score for each stock.
Firms like iSentient provide Twitter sentiment data on com-
panies. Firms like Estimize gather crowd-sourced earnings
and revenue forecasts via their websites. Naturally these data
sources are attractive. There is sound reason to believe that
the data they disseminate are not easily available from other
sources and as such, could serve as a reliable source of equity
return predictability. Such data have not yet found their way
to academic desks, but analyses as to whether they actually
add value over a long list of return predictors would be
extremely valuable.
Machine learning is also a promising area for academic
work. Non-OLS estimators (such as Lasso) form a promising
forecasting technique. Lasso achieves a parsimonious predic-
tion by forcing the sum of OLS coefficients to be less than a
pre-specified value, thus potentially setting some OLS co-
efficients to zero and thereby excluding variables that poten-
tially worsen the quality of forecasts. Similarly, random forest
algorithms are trained on random subsamples of a dataset to
produce superior forecasts. They form a promising way to
generate equity buy/sell signals. Artificial neural networks
allow pattern recognition within stock market data in complex
ways. As an example, there are multiple signals available
about equities on any given day (e.g., social media sentiment,
analysts' forecasts of earnings and revenues, credit card
transactions for the companies' products) and how these
interact is not easy to ascertain. The neural network can
analyze complex interactions between these variables and
produce a summary forecast for price movements. The
advantage of such networks is that they learn on a daily basis
where their forecasting was in error, thus providing real time
286 _
A. Subrahmanyam / Borsa Istanbul Review 19-4 (2019) 283e287
learning and thus potentially superior forecasts. I Know First is
an example of a company that uses artificial neural networks
to predict equity returns.
Machine learning holds particular promise in forecasting
stock returns for the reason that while there are a large number
of documented anomalies in the finance literature (Harvey,
Liu, & Zhu, 2016) the nature of the forecasting relation be-
tween the variables and stock returns can change over time. As
an example, consider that past returns forecast future returns
(Jegadeesh & Titman, 1993) and so does Twitter sentiment,
but over time, the increased usage of past returns in funds'
trading strategies may weaken the role of past returns but as
more people start using social media, the role of Twitter
sentiment in predicting returns may actually increase. Machine
learning techniques can easily pick up such relations whereas
standard OLS techniques may not be able to do so. How such
techniques improve forecasting is of tremendous interest to
academics, and large sample of evidence on this matter is yet
to come by. The availability of packages such R and Python to
program the preceding techniques has greatly simplified ap-
plications. Doubtless as more data and computing power be-
comes available, academics will spend more time on this issue.
Yet another application of machine learning is to improve
upon volatility forecasts. It is easy to see how techniques such
as random forest algorithms can be trained and used to fore-
cast future volatility. For example, simple ARCH-type fore-
casts (Engle, 1982) can be improved by considering other
signals such as social media coverage and the extent of high
frequency trading in individual firms. Studies such as Fan, Li,
and Yu (2012) have already made progress in using high fre-
quency data for estimating volatility.
In analyzing the preceding “alternative approaches” to eq-
uity return and volatility forecasting it is important to reiterate
that the obvious caveat applies. Specifically, we should ensure
that value being added relative to previously known predictors
and forecasting techniques such as OLS is truly adding value.
We should endeavor to ensure that simple forecasts do not
dominate those obtained from more complex techniques.
4. The challenges
The research using big data suffers from the problem of
novelty: when confronted with an analysis using novel data,
academics are likely to recommend publication using a
relatively low bar in terms of the reliability of the results.
This is because novelty garners citations, which helps the
journal. Unfortunately, reliable results that control for pre-
vious characteristics that predict returns are relevant for
ascertaining if the research has truly unearthed a new pre-
dictor. The research I have reviewed, while creative and
insightful, does not always control for existing proxies for
investor expectations (such as analysts' opinions) casting
doubt on the robustness of the results. Further, the research is
sometimes contradictory; thus Da, Huang, and Jin (2018)
suggest that one should trade in a contrarian manner to
retail sentiment, while Kelley and Tetlock (2013) suggest that
retail investors' net buying relates positively to future returns.
It is fair to say that much work needs to be done in replicating
the results and ensuring that they are robust to a variety of
time periods and controls.
What are the practical implications of the research? They lie
in exercising caution before using big data in financial appli-
cations. For example, practitioners are frequently approached
by vendors marketing new data sets for the purposes of
implementing equity strategies. In succumbing to vendors, a
frequent problem is the same as that afflicting academia, the
practitioner is likely to be swayed by the novelty of the data
set alone, which is dangerous. As such, I recommend the
following checklist for both academics and practitioners. First,
at least five years of historical analyses should be required
showing the data actually “works” in terms of forecasting
returns. Second, preferably, the data should be divided into two
subperiods with validity checked in each period. There should
also be a sound economic hypothesis behind why the data set
works. Questions should be asked on what economic force or
behavioral bias precludes the information from being incor-
porated into current prices in a timely way. Finally, papers such
as Grossman and Stiglitz (1980) suggest that extra rents can
only be earned on genuine information not available to all. So a
pertinent question is that of how many people have access to
the data. If too many do have access, then the signals generated
will be easily negated by crowding.
Conflict of interest
The author has nothing to disclose and no conflict of in-
terest with regard to the contents of this article.
References
Antweiler, W., & Frank, M. Z. (2004). Is all that talk just noise? The infor-
mation content of internet stock message boards. The Journal of Finance,
59(3), 1259e1294.
Chen, H., De, P., Hu, Y. J., & Hwang, B. H. (2014). Wisdom of crowds: The
value of stock opinions transmitted through social media. Review of
Financial Studies, 27(5), 1367e1403.
Chordia, T., Green, T. C., & Kottimukkalur, B. (2018). Rent seeking by low-
latency traders: Evidence from trading on macroeconomic announcements.
Review of Financial Studies, 31(12), 4650e4687.
Chordia, T., & Subrahmanyam, A. (2004). Order imbalance and individual
stock returns: Theory and evidence. Journal of Financial Economics,
72(3), 485e518.
Cohen, L., & Frazzini, A. (2008). Economic links and predictable returns. The
Journal of Finance, 63(4), 1977e2011.
Coval, J. D., & Shumway, T. (2001). Is sound just noise? The Journal of
Finance, 56(5), 1887e1910.
Da, Z., Engelberg, J., & Gao, P. (2011). In search of attention. The Journal of
Finance, 66(5), 1461e1499.
Da, Z., Engelberg, J., & Gao, P. (2014). The sum of all FEARS: Investor
sentiment and asset prices. Review of Financial Studies, 28(1), 1e32.
Da, Z., Huang, X., & Jing, L. (2018). Extrapolative beliefs in the cross-
section: What can we learn from the crowds?. working paper. University
of Notre Dame.
Engle, R. F. (1982). Autoregressive conditional heteroscedasticity with esti-
mates of the variance of United Kingdom inflation. Econometrica: Journal
of the Econometric Society, 987e1007.
 _
A. Subrahmanyam / Borsa Istanbul Review 19-4 (2019) 283e287
Fan, J., Li, Y., & Yu, K. (2012). Vast volatility matrix estimation using high-
frequency data for portfolio selection. Journal of the American Statistical
Association, 107(497), 412e428.
Gao, L., Han, Y., Li, S. Z., & Zhou, G. (2018). Market intraday momentum.
Journal of Financial Economics, 129(2), 394e414.
Grossman, S. J., & Stiglitz, J. E. (1980). On the impossibility of informa-
tionally efficient markets. The American Economic Review, 70(3),
393e408.
Harvey, C. R., Liu, Y., & Zhu, H. (2016). … and the cross-section of expected
returns. Review of Financial Studies, 29(1), 5e68.
Heston, S. L., Korajczyk, R. A., & Sadka, R. (2010). Intraday patterns in the
cross-section of stock returns. The Journal of Finance, 65(4), 1369e1407.
Hoberg, G., & Phillips, G. (2010). Product market synergies and competition
in mergers and acquisitions: A text-based analysis. Review of Financial
Studies, 23(10), 3773e3811.
Huang, J. (2018). The customer knows best: The investment value of consumer
opinions. Journal of Financial Economics, 128(1), 164e182.
Hvidkjaer, S. (2008). Small trades and the cross-section of stock returns.
Review of Financial Studies, 21(3), 1123e1151.
Jegadeesh, N., & Titman, S. (1993). Returns to buying winners and selling
losers: Implications for stock market efficiency. The Journal of Finance,
48(1), 65e91.
Jiang, F., Lee, J., Martin, X., & Zhou, G. (2019). Manager sentiment and stock
returns. Journal of Financial Economics, 132(1), 126e149.
Kelley, E. K., & Tetlock, P. C. (2013). How wise are crowds? Insights from
retail orders and stock returns. The Journal of Finance, 68(3), 1229e1265.
287
Kelley, E. K., & Tetlock, P. C. (2016). Retail short selling and stock prices.
Review of Financial Studies, 30(3), 801e834.
Li, F. (2010). The information content of forward-looking statements in
corporate filingsda naïve Bayesian machine learning approach. Journal of
Accounting Research, 48(5), 1049e1102.
Loughran, T., & McDonald, B. (2011). When is a liability not a liability?
Textual analysis, dictionaries, and 10-Ks. The Journal of Finance, 66(1),
35e65.
Mayew, W. J., & Venkatachalam, M. (2012). The power of voice: Managerial
affective states and future firm performance. The Journal of Finance,
67(1), 1e43.
Neely, C. J., Rapach, D. E., Tu, J., & Zhou, G. (2014). Forecasting the equity
risk premium: The role of technical indicators. Management Science,
60(7), 1772e1791.
Sloan, R. G. (1996). Do stock prices fully reflect information in accruals and
cash flows about future earnings? The Accounting Review, 289e315.
Tetlock, P. C. (2007). Giving content to investor sentiment: The role of media
in the stock market. The Journal of Finance, 62(3), 1139e1168.
Tetlock, P. C., Saar-Tsechansky, M., & Macskassy, S. (2008). More than
words: Quantifying language to measure firms' fundamentals. The Journal
of Finance, 63(3), 1437e1467.
Tirunillai, S., & Tellis, G. J. (2012). Does chatter really matter? Dynamics of
user-generated content and stock performance. Marketing Science, 31(2),
198e215.
Tversky, A., & Kahneman, D. (1973). Availability: A heuristic for judging
frequency and probability. Cognitive Psychology, 5(2), 207e232.