International Conference on Innovations in Power and Advanced Computing Technologies [i-PACT2017]
A SURVEY ON BIG DATA ANALYTICS USING
SOCIAL MEDIA DATA
P. Victer Paul, K. Monica, M. Trishanka
Department of Information Technology
Sri Manakula Vinayagar Engineering College, Pondicherry, India
victerpaul@gmail.com, monicaranirajan@gmail.com, trishanka.manoj@gmail.com
Abstract— Analytics is very important in all fields in order to
make decisions over certain facts. Social media analytics is the
process of collecting information from various social media
platforms, websites and blogs. This analytics is done to make
effective business conclusions. The usage of social media has
become the latest trend in today’s world. Social data analytics is
not about just collecting likes and comments shared by
individuals but it has become the platform for many trademarks
to bring out promotion. Applications such as marketing, elections
widely used social data to make predictive decisions. Some of the
approaches followed are forming hypothesis, getting deep into
the data, mapping events etc. This analytics can also be done in
applications such as business, Change in amendments,
Education, Demonetization etc. The challenges faced are metrics
formed by social media should reach the right people,
unstructured data being difficult to process. This paper discusses
about the model, theme, performance evaluation, advantages and
disadvantages under literature survey.
Keywords— social media; insights; datasets; analysis ;
challenges (key words)
I. INTRODUCTION
Big data is nothing but huge amount of data which cannot
be processed using traditional methods. The data to be
processed may be structured, unstructured or semi structured.
These data come from various sources such as sensor data,
transactional data, cameras, microphones and lot more of
resources. On these large data sets analytics is performed in
order to obtain undiscovered insights. Based on facts,
everyday petabytes of data are being generated and now it has
come up to zeta data. All social media platforms in which
billions of users are connected generate huge data which can
be of any type such as text, images, audio, video, gif, etc.
Big data is highly used in many applications because each
and every field now generates tones of data which need to be
processed at one point of time. Let us discuss the various
applications in which Big data is being used. The foremost is
the Banking in which Securities Exchange Commission is
being used to monitor various activities. At present, they are
using the network analytics and natural language process as
trend in the markets. Retail traders, big banks and many others
are using analytics for making decisions. Moving on to the
next application, it is the media. Big data is widely used in
media, since it has now reached the users through many hand
978-1-5090-5682-8 /17/$31.00 ©2017 IEEE
held devices. Here analysis is done on the consumer insights,
social media content and understanding pattern. Real time
examples of application using Big data are Spotify, Amazon
Prime etc. Next widely used sector is Healthcare. This sector
has access to huge sets of the data as they handle much
number of patients and their respective data. So here analytics
is being used to link the health related data of the patients in
order to track them at respective intervals. One such
application is Obamacare which has made use of Big data in
multiple ways. Then comes the education, the staffs and the
students are now using Big data to learn various analysis tools
and other technologies related to it. Similarly Manufacturing,
Government, Insurance, Retail, Transportation and Energy
have made great use of Big data.
The term “Characteristics of Big data” reminds about the
3V’s which is Volume, Variety and Velocity. But now it has
grown up to 6 V’s. The first is volume. Big data in general
refers to huge volumes. The data is being generated by
machines, systems and user interactions which are seen on
social media. This makes the analysis process a little tedious.
Next is the variety, refers to the different types of data that we
come across. It may be structure, unstructured or semi
structured. The data is usually stored in spread sheets and then
moved to databases, but now data comes through various
formats such as emails, audios, videos, Gif’s. Next is the
velocity. It deals with the rate at which the data is being
produced. Data flows become too heavy to be handled. Next is
veracity, which refers to the abnormal data. Data being
generated need not always be clean hence we need to work on
those data to filter out the necessary ones. Processing must be
done in order to avoid collecting “Dirty data”. Next is the
validity which refers to the correct data. Data must always be
accurate during analysis only then it can help in making the
right choices. The next is the volatility refers to the duration of
the data being valid. When data becomes irrelevant it must not
be used for analysis. Hence these are the characteristics of Big
data.
The main issue that the organization face is to collect the
accurate data and to make the best use of it. After collecting
process, the organization must find the right technology to
work with in order to bring out the best insights. Then comes
the access towards the data. Most organizations fail to find the
right platform to work with the data generated. A survey has
mentioned that 55% of the Big data projects are never
1
 International Conference on Innovations in Power and Advanced Computing Technologies [i-PACT2017]
completed, the problem is that the software is not easy to
handle. Not only this, it also requires internal resources to be
maintained. There is also a shortage of Data Scientist and
Developers who must have Domain knowledge so that they
find the right objectives. The final concern is the Security that
is the data protection.
II. LITERATURE SURVEY
This section deals with the study in Social media analytics and
Demonetization.
A. Can Twitter save lives? A Broad scale study on Visual
Social Media Analytics for Public Safety
Theme: According to [1], Dennis Thom et al. presents that
nowadays the social media is being used for commercial
purpose rather than safety. The paper gives a clear
understanding on the crisis intelligence field study via Twitter
data during the flood on German 2013. Scatter Blogs systems
was introduced to implement other techniques. The second
phase sketched out a system based on the feedback about
Scatter Blogs and a comparison is made between both the
systems.
Model: In [1], a new system was introduced known as
Scatter Blogs. It is a visual social media analytics solution.
This was designed for analyst so that they could make
decisions using social media. This is an alternative because
not every software with required version is obtained.
Performance Evaluation: Here visual analytics is being
used with situation assessment as a crucial element. Scatter
Blogs system provided the experts with variety of tools and
techniques to be implemented which are in current research.
They made a complete redesign of the system and named it as
Event Digest. The existing techniques were lead line, a topic
modeling to show sudden burst segments on visualization. To
find peaks in Twitter a custom designed algorithm called
Twitter info is used. LDA topic modeling and Natural
language processing were used on the visualization process.
Thus both field and user study received feedbacks on the
newly introduced system called Scatter Blogs [8-10].
Advantage:
Here visual analytics is being used with situation assessment
as a crucial element and the positive review are as follows
ƒ It is positively commented on the social and system
monitoring usefulness.
ƒ A tool to harvest the public’s opinion and responses.
ƒ Gathering information using this tool was fast and
tremendous than reporting channels.
Disadvantage:
ƒ There were difficulties in understanding the current active
filters and modules.
ƒ It is good to have more images and videos rather than
word based content.
ƒ It is expected by the experts that automation techniques
must be complemented by human understanding and
knowledge.
B. Big Data Analytics in Healthcare
Theme: Based on the field of healthcare Big Data
analytics and its benefits with the methodologies used and its
outcomes and the challenges faced are discussed here
according to Wullianallur Raghupathi et al. [2].
Model: Big data can avoid inefficiency in the areas of
Clinical operation, public health, research and development,
pre-adjudication fraud analysis, genomic analytics, evidence
based medicine, patient profile monitoring and device and
remote profile monitoring. Different sources and data types
that are included are machine to machine data, web and social
media data, biometric data, big transaction data and human
generated data.
Performance Evaluation: For analytics to be done these
data has to be pooled. Initially the data sources are identified;
data are collected and transformed for analytics. Then the
analytics platform/tool is selected as from Hadoop, IBM
BigInsights, and Cloudera. Next apply various big data
analytics techniques to data [11-12].
Advantage:
ƒ The outcome of the models are validated and presented to
the stakeholders for action.
ƒ The big data analytics in healthcare provides a
sophisticated technology to observe insights from medical
records and various information storage areas to obtain
valuable conclusions.
Disadvantage:
ƒ Although the analytics in health care has to be user
friendly and menu driven with active availability of
abundant analytics models, algorithms and methods in a
pull down type of menu for large scale maintenance.
ƒ Moreover, the adminstrative problems as of standards and
partnerhip have to be considered.
C. Big Social Data Analytics for Public Health: Facebook
Engagement and Performance
Theme: In referred paper [3], analysis of Facebook data
using unsupervised learning indicates recent tendency of user
engagement in public health has been increased as observed
by Nadia Straton et al.
Model: Data from 153 public health organization are
gathered from Facebook wall using Social Data Analytics
Tool (SODATO). Clustering is used to discover pattern and
interesting correlations among large datasets.
Performance Evaluation: From several algorithms, K-
Means algorithm is used here for clustering. By comparing the
clusters we get the clear idea on the popular and less popular
posts. Analysis can also be done based on some category
2
 International Conference on Innovations in Power and Advanced Computing Technologies [i-PACT2017]
types, as to understand the feature of each category
individually K-Means algorithm is suitable. Using this K-
Means clustering and frequency analysis the result shows the
posts with low engagement cluster as those posts have very
low user engagement. It is found that form 2013 onwards there
is increase in share of high and medium clusters. Thus
increase in average span of user engagement gives the post to
acquire great interest as the fame of health care post got
increased in modern era. From this analysis, the photo and link
type posts gain more attention as neutral and immense
engagement. Thus the chief share of 50% of intermediate and
50% of high engagement posts are found during the period of
10-16 and less during evening times.
Advantage:
ƒ K-Means algorithm makes the clustering task of
unsupervised machine learning simpler.
ƒ Clustering process is easy to configure and provides more
options to the users.
ƒ There is limited single point of failure.
Disadvantage:
ƒ However, month, season, day is non decisive features for
placing the posts in specific or certain cluster.
ƒ Therefore, to get more efficiency in communication for
public health the organization must avoid status updates
and think of better visualization and providing posts with
photos and link. It is noticed that the posts that are active
and available attract more user.
ƒ The future work includes the analysis of documented
content of the public health data using supervised
machine learning methods for the discipline specific
models as the disease specific models, inner natures and
so on.
D. SoLoMo analytics for Telco Big Data monetization
Theme: The reference [4] has observed that the world can
be accessed using the mobile internet. This has helped widely
in Business. The algorithms and technologies used in this
paper are used to find out the insights with the help of social,
location and mobile data of the individual.
Model: In the paper referred by H. Cao et al., there are
many platforms used each for a different purpose. IBM
Infosphere Big Insights, IBM Infosphere streams, IBM
Netezza Data Warehouse are the tools used to address the big
data challenges. This system obtains the telecom data from
various profiling such as social network, location based and
mobile usage profiling.
Performance Evaluation: The paper clearly explains that
all the users across the world are socially connected through
various mediums. Useful insights have been explored from the
social, location and mobile usage of the users. IBM Infosphere
streams component is used to stream data in real time. It takes
data such as call Detail Record (CDR) which is completely
about the telecom transactions. These data are finally stored in
IBM Netezza Data Warehouse. The next component IBM
Infosphere BigInsights component helps to extract the social
individual neighborhood and key location. Next, the telco
analytics data warehouse where the huge data sets are stored
so that clustering and statistics are performed. This supports
in-database Map Reduce as well. SoLoMo insights include
visualization, social network analysis and various profiling.
Social profiling is based on Key Performance Indicators.
Location Profiling is based on low, middle and high level
features which includes various factors. Mobile profiling
analyzes the call detail record which is usually structured. This
also includes Modeling web pages using Open directory
Project (ODP) and URL pattern analyzer under various
categories.
Advantage:
ƒ This analysis was useful in finding like minded
communities using points of interest.
ƒ This helps to indicate the interaction between subscribers.
ƒ Netezza provides native support for Map Reduce.
Disadvantage:
ƒ IBM Infosphere component does not support social
network analysis.
ƒ The data obtained through call detail record is not as
precise as Global Positioning System(GPS)
E. Impact of Demonetization on Indian Economy
Theme: India has the highest rate of currencies. 87% of the
currencies are rotating in the form of Rs.500 and Rs.1000.
These currencies becomes the source of income either by
working under organizations or through illegal enterprise.
This paper [5] by Dr. Pratap Singh et al. presents the impacts
after the process of demonetization occurred in India.
Model: In the paper demonetization is the act of doing
away with the certain currency notes. The process of
restricting the currency notes led to the decision of getting rid
of Corruption. This move may improve the economy system
being followed in India and to lower the cash being circulated.
It also could be a step in making the nation digitalized.
Performance Evaluation: The paper refers to the data
which is secondary. Secondary data refers to the data which
has been collected from various sources such as survey,
census. These data are being collected by various government
officials and other departments who take survey for research
as well as analysis purpose. Usually the demonetization
process would have been started earlier, only after almost all
the process has been over, the official date will be announced.
The process of demonetization is not new to our country. It
has happened before with very high currency notes such as
Rs.10000, but these cash weren’t on high circulation among
the citizens thus it did not affect the country as now. The
recent denomination that occurred on the restricted currency
was 87% in circulation among the people. Hence it had
brought adverse effects that people couldn’t not manage.
Analysis on demonetization helps other researchers who wish
3
 International Conference on Innovations in Power and Advanced Computing Technologies [i-PACT2017]
to analyze the process and also for government institutions and
officials.
Advantage:
ƒ This analysis would be useful for the government
institutions the next time they plan to perform the same
process.
ƒ It highly helped in bringing out the black money.
Disadvantage:
ƒ The recent denomination which occurred has affected the
honest normal tax payers who work for daily wages
ƒ The government did not plan about the new currency that
are to be circulated which led the people standing in
queues before ATM.
III. OBSERVATION
It has been observed that each paper has used different
techniques which have their own pros and corns. The
techniques overall used are Scatter Blogs, SoLoMo, IBM Big
Insights each used for a purpose such as analysis, storage,
streaming real time data etc. We have decide to take the
positive features of all the algorithms, tools and techniques
used in the above discussed papers. The social Media analytics
is the main resource to perform analysis. It is actually the
process of developing informative tools to collect and
summarize the data. The usage of social media has been
evolving more as the days pass by. It has been used for many
positive as well as negative usage. It all depends upon the
attitude of the users. Facebook, YouTube and Twitter are in
the first, third and tenth position on account of the usage. In
general the analysis process comprises of three stages Capture,
Understand and Present. Topic modeling is also one of the
important techniques being used in above papers. Hence
Social media analytics is highly used in the analysis that is
performed on demonetization.
IV. FUTURISTIC SOLUTION
According the literature study that we did on the paper,
there were many techniques for analysis that were implied on
social data. This was especially to gather insights and make
efficient decisions so that it becomes useful for the users who
wish to do research based on the problem. The first step
process is to collect data from the social networks such as
Twitter, Facebook. The data can be easily retrieved through
the API (Application Programming Interface) of the respective
platform. Sometimes it is requested to pay in order to retrieve
the data as they might keep their resources secure. The next
step processes it to perform analysis on the retrieved data. This
process can also make use of R studio and flume platform to
store the data and perform analysis. The outcome is derived in
any of the formats such as Graph, line or pie chart. The
analytics process can be used in applications such as
predicting elections, sports and demonetization etc. It stands
as a support for all the researchers who wish to analyze the
impacts and effects.
V. CONCLUSION
This paper presents about the introduction, characteristics,
applications and limitations of Big data. Then about the
introduction about social data analytics which is the source for
performing analysis. The literature study includes the
proposed work, used tools and techniques in each respective
paper. It also contains the overall observations with the
futuristic solutions where it has been discussed about the steps
involved in analysis and the applications. The conclusion and
the reference for the above surveyed paper are also included.
REFERENCES
[1] Dennis Thom, Robert Kruger, Thomas Ertl, “Can Twitter Save Lives? A
Broad-scale Study on Visual Social Media Analytics for Public Safety”,
IEEE Transactions on Visualization and Computer Graphics, 2015.
[2] Wullianallur Raghupathi and Viju Raghupathi, “Big data Analytics in
Healthcare: promise and potential”, Raghupathi and Raghupathi Health
Information Science and Systems, 2014.
[3] Nadiya Straton, Kjeld Hansen, Raghava Rao Mukkamala, Abid Hussain,
Tor-Morten Grønli Henning Langberg and Ravi Vatrapu, “Big Social
Data Analytics for Public Health: Facebook Engagement and
Performance”, IEEE 18th International Conference on e-Health
Networking, Applications and Services (Healthcom), 2016.
[4] H. Cao, W.S. Dong, L.S. Liu, C.Y. Ma, W.H. Qian, J.W. Shi, C.H. Tian,
Y. Wang, D. Konopnicki, M. Shmueli-Scheuer, D. Cohen, N. Modani,
H. Lamba, A. Dwivedi, A. A. Nanavati, M. Kumar, “SoLoMo analytics
for telco Big Data monetization”, IBM J. RES. & DEV. VOL. 58 NO.
5/6 Paper 9 September/November, 2014.
[5] Dr. Partap Singh, Virender Singh, “Impact of Demonetization on Indian
Economy”, 3rd International Conference on Recent Innovations in
Science, Technology, Management and Environment, 18th December,
2016
[6] Weiguo Fan, Michel D. Gordan, “Unveiling the Power of social Media
Analytics”
[7] Dr. Sunitha V Gangier, Ranganatha B, “Demonetization and its impact
on Social Development”, INDIAN JOURNAL OF APPLIED
RESEARCH, January 2017
[8] P. Victer Paul, T. Vengattaraman, P. Dhavachelvan, “Improving
efficiency of Peer Network Applications by formulating Distributed
Spanning Tree”, Third International Conference on Emerging Trends in
Engineering & Technology (ICETET-2010),IEEE, India, May 2010. pp.
813-818.
[9] P. Victer Paul, R. Baskaran, P. Dhavachelvan, “A Novel Population
Initialization Technique for Genetic Algorithm”, IEEE International
Conference on Circuit, Power and Computing Technologies (ICCPCT),
March 2013, India, pp 1235 - 1238. ISBN: 978-1-4673-4921-5.
[10] N. Saravanan, R. Baskaran, M. Shanmugam, M.S. SaleemBasha and P.
Victer Paul, "An Effective Model for QoS Assessment in Data Caching
in MANET Environments", International Journal of Wireless and
Mobile Computing, Inderscience, Vol.6, No.5, 2013, pp.515-527. ISSN:
1741-1092.
[11] P. Victer Paul, A. Ramalingam, R. Baskaran, P. Dhavachelvan,
K.Vivekanandan, R. Subramanian and V.S.K.Venkatachalapathy,
"Performance Analyses on Population Seeding Techniques for Genetic
Algorithms", International Journal of Engineering and Technology
(IJET), Vol 5, No 3, Jun-Jul 2013, pp. 2993-3000. ISSN: 0975-4024.
[12] P. Victer Paul, N. Saravanan, S.K.V. Jayakumar, P. Dhavachelvan and
R. Baskaran, “QoS enhancements for global replication management in
peer to peer networks”, Future Generation Computer Systems, Elsevier,
Volume 28, Issue 3, March 2012, Pages 573–582. ISSN: 0167-739X.