2016 IEEE International Conference on Big Data (Big Data)
Nowcasting with Social Media Data
David L. Kimmey and Jin S. Yoo
Department of Computer Science
Indiana University-Purdue University Fort Wayne
Indiana, USA
Email: kimmd01@ipfw.edu, yooj@ipfw.edu
Extended Abstract —Social media is a source of real time
data that individuals create and voluntarily share on major
social media generators such as Facebook, Twitter, Google,
Yahoo, and Instagram. Nowcasting uses real-time data and
is defined as the prediction of the present, the very near
future, and the very recent past. Nowcasting with social media
assists businesses and government agencies in understanding
public opinion and trends, and in creating timely forecasts
of economic indicators. The example includes geo-economic
events: how the public’s sentiment affects the stock mar-
ket [1], predicting political alignment: management of political
strategy [4], and social questions: popular events associated
with increased public sentiment [8]. Nowcasting and social
media are also being used together to discover unusual social
events such as demonstrations and spontaneous festivals, and
natural disasters such as earthquakes and storms [6], and to
forecast early detection of disease activity - thereby allowing
rapid disease response, which reduces the public impact of
disease [2].
Research has shown high correlations between social media
data and government reports of 76.7% up to 90% [3]. There
are advantages to using nowcasting to predict report values. For
example, it is not trivial to forecast turning points in time series
data, however a forecast using nowcasting has turning points
with higher correlation to the actual data than a traditional
time series model [2]. Many government and business report
values do not use real-time data values (i.e., the reports value
describes past results from the previous week, month, or
quarter), however, to make timely decisions, governments and
businesses need to forecast, in real-time, trends and events
which may affect their operations. Nowcasting provides a real-
time data prediction solution for real-time decisions.
The research question is: how does users obtain clean, rele-
vant, and timely social media data to complete the nowcasting
process for their specific reporting needs? One possible ap-
proach is to query real-time data from social media and to now-
cast a trend or event; however, the defining and detection of
events is a non-trivial task and has long been a research topic.
Furthermore, nowcasting with social media gives challenges
due to the characteristics of big data such as heterogeneous,
autonomous, complex, and evolving associations. For example,
Twitters tweet content is usually overwhelmed with “babble”
(i.e., about 40% of tweets queried or data mined will not
include the queried keyword) [3]. Twitters broadcasted tweet
locations are autonomous and have evolving association that
are dynamically changing and increasing in a real-time nature.
Popular methods to find real-time social media events are
frequency bursts method [9], deviation (/anomaly) detection
method [5], and probabilistic soft logic method [7]. However
978-1-4673-9005-7/16/$31.00 ©2016 IEEE
their algorithms are not scalable to handle big social media
data.
This work proposes a scalable parallel and distributed
algorithm for social media event detection, which finds a
user specified event within social media utilizing the prob-
abilistic soft logic method. The proposed method identifies
the relationships between the geospatial and causal model
variables within the data. Generated probabilistic soft logic
rules are then aggregated by a given rule weight threshold to
identify interesting events users specified. Our method also
provides three advantages over query approaches: the user
specified events corpus has less noise; the clearer recognition
of the user specified events approximate current location, and
the identification of popular keyword themes from a user’s
specified event within social media. For the evaluation of
the proposed approach, we used Twitter data to nowcast a
corporate or government agency report, and to compare and
contrast the nowcast results with a traditional time series
model. The experiment of predicting the CDCs influenza like
illness report shows nowcasting with social media outperforms
the traditional time series model by as much as 16%, up to
20%, in terms of statistical error. Furthermore, the parallel
event detection algorithm with MapReduce improved the al-
gorithms running time by 65%.
R EFERENCES
[1] J. Bollen, H. Mao, and X. Zeng. Twitter mood predicts the stock market
. Journal of Computational Science, 2(1):1–8, 2011.
[2] H. Choi and H. Varian. Predicting the present with Google Trends.
Economic Record, 88:2–9, 2012.
[3] V. R. Kamala and L. MayGladence. An optimal approach for social
data analysis in Big Data. In Proc. of International Conference on
Computation of Power, Energy Information and Communication, pages
192–199, 2011.
[4] S. Kumar, F. Morstatter, and H. Liu. Twitter Data Analytics. Springer,
2013.
[5] R. Lee, S. Wakamiya, and K. Sumiya. Discovery of unusual regional so-
cial activities using geo-tagged microblogs. World Wide Web, 14(4):321–
349, July 2011.
[6] T. Sakaki, M. Okazaki, and Y. Matsuo. Earthquake shakes twitter users:
Real-time event detection by social sensors. In Proceedings of the
International Conference on World Wide Web, pages 851–860, 2010.
[7] R. D. Santos, S. Shah, F. Chen, A. Boedihardjo, C. Lu, and N. Ramakrish-
nan. Forecasting location-based events with spatio-temporal storytelling.
In Proceedings of the ACM SIGSPATIAL International Workshop on
Location-Based Social Networks, pages 13–22, 2014.
[8] M. Thelwall, K. Buckley, and G. Paltoglou. Sentiment in Twitter events.
Journal of the American Society for Information Science and Technology,
62(2):206–418, 2011.
[9] Y. Yang, T. Pierce, and J. Carbonell. A study of retrospective and on-
line event detection. In Proceedings of the International ACM SIGIR
Conference on Research and Development in Information Retrieval,
pages 28–36, 1998.
4004