BIG DATA GETS EVEN BIGGER

What is Big Data?

Big data is data that contains greater variety arriving in increasing volumes and with even higher
velocity.
(Gartner’s definition, circa 2001)

Big data is also data but with huge size. Those data sets are so large and complex that none of the
traditional data management tools are able to store it or process it efficiently. Although the
concept of big data itself is relatively new. But the origins of large data sets get go back to the
1960s and ’70s. Around 2005, that developed an open source frame work created specifically to
store and analyze big data sets, which is named “Hadoop”. Development of this open source
frame works essential for the growth of big data because they make big data easier to work with
and cheaper to store. And also cloud computing has expanded big data possibilities even future.

Big data examples: -The New York Stock Exchange generates about one terabyte of new
Trade data per day.
-The statistic shows that 500 terabytes of new data get ingested in to the
databases of social media sites, Facebook every day.
There are 3 types of big data,
1.Structured – Data that can be stored, accessed and processed in the form of fixed format.
EX: - An employee table in a database
2.Unstructured – Data with unknown form. Unstructured data is a heterogeneous data source
containing a combination of simple text files, images, videos etc.
EX: - The output returned by ‘Google Search’
3.Semi-structured – These types of data can contain both the forms of data.
EX: - XML file
Big data in field of finance
Big data in finance refers to the petabytes of structured and unstructured data that can be used to
anticipate customer behaviors and create strategies for banks and financial institutions. Finance
industry generates lots of data. Structured data is information managed within an organization in
order to prove decision making insights. Unstructured data exists in multiple sources in
increasing volumes and offers significant analytical opportunities.
Big data technology has become an integral part of the financial services industry. Financial firm
can leverage big data to,
-Generate new revenue streams through data driven offers, such as personalized
recommendations.
-Become more efficient in order to computer with FinTech firms, which use cutting edge
technology to provide customers with banking and financial services.
-Provide better services to customers, such strengthened security.
The finance sector has been an intensively data driven industry for many years, with financial
institutes having managed large quantities of customer data and using data analytics in areas such
as capital market trading. However, there is an increase in prevalence of data which falls into the
domain of big data.
In 2001, Gartner’s Doug Laney first presented what become known as the ‘3V’s of Big Data’ to
describe some characteristics that make big data different from other data processing. Those are,
increasing Volume (amount of data), Velocity (speed of data in and out), Variety (range of data
types and sources). Later in 2012 Gartner update those as High volume, High velocity and High
variety.
FIGURE 1
VOLUME is considered to reach big data levels at many Terabytes or even Petabytes of data.
The financial industry produces a huge volume of quotes, market data and historical trade data.
The New York Stock Exchange alone writes more than a Terabyte per day.

VELOCITY suits big data when the speed of data storage or processing is on the order of 105
transactions per second or more. Generating data at this speed is no challenge for the financial
markets. Moreover, the faster systems can process trade data, the faster they can manage trading.

VARIETY implies that big data algorithms do well with various format and data sources. In
corporate banking, institutions work with reference data, trade and market data, request from
clients and many other sources.

How big data is changing the finance industry

The finance industry is a highly competitive space. In the past few years, big data in finance has
led to significant technological innovations that has enabled convenient, personalized and secure
solutions for the industry. As a result, big data analytics has managed to transform entire
financial service sector. The financial services industry has always been at the forefront of
technical innovation. The availability of new datasets has provided a powerful way to understand
behavior and offers new directions for the financial industry to be predictive. Big data
application in financial services goes beyond predicting share prices. Big data changing the
finance industry in different ways,

Personalization has been an emerging trend in a number of different industries with the
financial industry among them. New banks disrupting the market by a more modern sensibility to
traditional banking methods. A common thread that runs through all of these banks in their use
of personalization to improve customer engagement. Understanding users necessary to offer
them personalized services is best achieved with a data driven approach. Big data enables banks
and financial institutions to much more accurately define and profile their userbase, which in
turn make it possible to target services and messaging at them much more precisely.
Security – Fraud is a big problem in the financial industry. Its now relatively easy for anyone
who wants to hide money to do so under layer of shell corporations. Similarly, by funneling
money through multiple transactions, it can become all but impossible to trace or identify the
original sources of lots of money. Big data analytics enables financial services watchmen to
identify patterns of behavior that would be vary difficult for a person to spot. These algorithms
can pick out suspicious transaction patterns that warrant further investigation.
Investing – Automated investing is becoming increasingly common. There are now a variety of
ways that new and experienced traders can automatically invest their funds. Big data analysis has
enabled the development of reliable trading algorithms that are capable of reliably making
money for their owners.

Applications of big data in finance

Financial firms have capability to leverage big data for use cases such as generating new revenue
streams through data driven offers, delivering personalized recommendations to customers,
creating more efficiency to drive competitive advantages and providing strengthened security
and better services to customers. Most of the financial firms already doing big data right and
getting instant results.
1.Increasing revenue and customer satisfaction
Some world wide companies have been able to apply big data solutions to evolve analytics
policies that predict client’s payment behaviors. By getting insight in to the behaviors of their
client’s a company can shorten payment delay and generate more cash while improving customer
satisfaction.
2.Streamlined workflow and reliable system processing
3.Analyze financial performance and control growth
4.Speeding up manual processes
Data integration solutions have the ability to scale up as business requirements change. Access to
a complete picture of all transactions every day, enables credit card companies to automate
manual processes, save IT staff work hours and offer insights in to the daily transactions of
customers.
5.Improved path to purchase
Legacy tools no longer offer the solutions needed for large, contrast data and often have limited
flexibility in the number of servers can utilize. Cloud based data management tools have helped
companies to get data from several web services into data warehouses for utilization by various
departments such as, finance, marketing, business intelligence, market intelligence and reporting.
Cloud strategies like these improve the path to purchase for customers, enable daily metrics and
execution forecasts as well as ad hoc data analysis.
RESEARCH FINDINGS

One of the very first examples of big data applied to finance is Tetlock (2007). Tetlock finds that
the impact of the column on the Dow Jones Industrial Average is first negative, followed by a
partial reversal. This suggests that the market overreacts to extremely negative news. The
analysis spans a fifteen-year period starting from 1984. Yet, there is not much attempt to control
for massive negative news that could affect markets in a lumpy way. Text analysis has also been
applied to predict companies' fundamentals such as earnings (Tetlock, saar-Tsechanky, and
Macskassy,2008).

Another recent example of big data analysis is by Huang (2012), who considers product quality
ratings by Amazon customers over a 12-year period starting from 2004. This type of data is not
easily available to investors and could potentially affect returns with a lag. Huang finds that
indeed, such customer ratings predict returns to the tune of a 1% per month return differential
between firms falling in the highest/lowest ratings deciles. The paper is unique in bringing
subjective quality perceptions into objective return predictions, and, as such, should attract much
attention. However, the paper does not control for analysts' forecasts of sales and earnings, and
associated revisions, thus leaving open the possibility that forecasts by other agents dominate
quality perceptions. Tirunillai and Tellis (2012) consider the impact of the volume (number) of
opinions and the actual ratings on stock returns at the daily level. Using a vector autoregression
framework, they find that the volume (but not the number) of posts predict a daily company's
abnormal returns a few days ahead.
Chordia and Subrahmanyam (2004) show that individual stock order flow predicts returns at the
daily horizon as well. Heston, Korajczyk, and Sadka (2010) use comprehensive intradaily data to
show that returns during a particular half-hour period during a day predict returns at the same
half-hour interval over about 40 trading days. This is an intriguing finding which lacks a
convincing explanation at this point.

https://www.talend.com/resources/big-data-finance/
https://www.sciencedirect.com/science/article/pii/S2214845019302650
https://www.infoq.com/articles/big-data-in-finance/
https://www.oracle.com/big-data/guide/what-is-big-data.html#link8