Advanced Engineering Informatics 42 (2019) 100945

Contents lists available at ScienceDirect

Advanced Engineering Informatics

journal homepage: www.elsevier.com/locate/aei

Review article

State of the art in big data applications in microgrid: A review T

Karim Moharm
Alexandria University, Egypt

ARTICLE INFO ABSTRACT

Keywords: The prospering Big data era is emerging in the power grid. Multiple world-wide studies are emphasizing the big
Big data data applications in the microgrid due to the huge amount of produced data. Big data analytics can impact the
Microgrid design and applications towards safer, better, more profitable, and effective power grid. This paper presents the
recognition and challenges of the big data and the microgrid. The construction of big data analytics is in-
troduced. The data sources, big data opportunities, and enhancement areas in the microgrid like stability im-
provement, asset management, renewable energy prediction, and decision-making support are summarized.
Diverse case studies are presented including different planning, operation control, decision making, load fore-
casting, data attacks detection, and maintenance aspects of the microgrid. Finally, the open challenges of big
data in the microgrid are discussed.

1. Introduction selling opportunities and quick experimentation. In manufacturing, big

Over the past years, data size is dramatically increasing supported
by new technologies like Internet of Things, spread sensors, and new
communication technologies. Massive amounts of data are being gen-
erated in structured, unstructured, and semi-structured forms as shown
in Fig. 1. The traditional techniques for data processing is not efficient
nor sufficient for processing this huge amount of data which requiring
more powerful computational analysis strategies. Big Data analytics
and storage offer scalable solutions and new methods to handle that
huge complex amounts of data.
Big data found its applications in different fields [1] such as business
[2] where it could help analyzing of the customers preferences and
interactions to achieve better goods recommendation and reduces costs
[3]. In social life, big data can be processed to have more insights about
public needs [3] such as processing data from Facebook, Twitter, and
LinkedIn to comprehend the user interaction and behaviors. In
healthcare field [4], data sources are mostly unstructured and can be
Electronic Medical Records (EMR), X-rays, CT scans, documents,
claims, machinery, etc. Big data help in profiling patients, illness pre-
diction, and support doctors. In government and defense field, big ci-
tizens data can be analyzed for crime detection, attacks analysis, and
visualization. In insurance field, big data extracts customer behaviors,
social media, and call records for developing predictions, better cus-
tomer satisfaction, and profits maximization. In finance field, big data
adoption reveals new analytical trade analysis, invisible market
chances, and client analysis because of the growing financal datasets
[5]. In automotive industry [6], big data finds its applications for more
data help risk analysis, identify new investment areas, and planning.
Supported by those achievements in different perspectives, micro-
grid big data analytics offers interesting applications. Big data can en-
hance reliable, efficient, and cost-effective operation through the high
computational power. The data analytics application can lead to better
energy forecasting, smart meter analytics, asset management and ana-
lytics, more reliable grid operation, customer segmentation, more
profitable energy trading, customer service data analytics, energy effi-
ciency, end-user engagement and marketing, bad data detection, energy
theft detection, load management and profiling, burden forecasting,
and outage management [7].
The microgrid concept combines the micro source distributed gen-
eration (DG), loads, control, and storage elements operating together in
islanded or grid connected modes [8]. It can be seen by the utility
network as a controllable unit. DG makes use of micro turbines, fuel
cells, internal combustion engines, solar energy, wind, hydro power,
biomass, ocean, bio fuels, and hydrogen into electrical and heat energy
which are easier to use and consume. Distributed storage systems like
batteries, flywheels, and super capacitors. Mirogrid supports the usage
of renewable energy in the power grid. The requested electrical power
is increasing fast. Simultaneously, conventional energy resources used
for electricity generation are exhausting. Also, environmental effects
caused by the conventional power stations are causing global problems.
The interest of renewable and environment friendly energy sources is
increasing to achieve energy savings and best performance. Renewable
energy solutions are assured to present reliable solution to energy
challenge although the oscillatory electrical power and imbalance can

E-mail address: karim.i.moharm@gmail.com.

https://doi.org/10.1016/j.aei.2019.100945
Received 31 December 2018; Received in revised form 1 May 2019; Accepted 7 June 2019
1474-0346/ © 2019 Published by Elsevier Ltd.
K. Moharm
Fig. 1. Big data content format.
happen if inconsistency between load and renewable generation hap-
pens. Moreover, disturbances in the grid.
Vast employment of uncontrolled and uncertainty intermittent
nature of the renewable energy triggers the need for new real time and
data-based energy management system in the microgrid control system
to avoid volatility of the whole grid. Huge data sources are pervading
the microgrid such as data exported from PV measurements, advanced
measuring units, smart meters, SCADA, weather reporting, environ-
mental conditions, and sensors data. Smart control involves high
communication data rate to improve power quality, reliability, effi-
ciency, and regulation of the whole power grid. Data levels are of
Petabytes level to be analyzed within small reasonable which going
beyond the traditional computational methods. The microgrid re-
presents high volume of data and there is still primitive transition to-
wards using new big data technologies [9].
The electrical networks are generating big data. For example, only
40 Phasor Measurement Units (PMUs) can generate about 5.6 Terabytes
per year. 1 Million smart meters can generate 2920 Terabytes when
data is collected each 15 min [10]. IBM analyzed 4 heterogeneous Pe-
tabytes data like environmental data (weather, wind, etc.) to optimally
select the wind turbine best location [11]. The data in the microgrid are
increasing more and more in the streaming volume and speed with
different formats and structures. E-Sketch project [12] showed the
importance of analyzing the big data of the grid at streaming at second
level leads is more appreciated. Power disturbance economic loss in US
is between 119$ billions to 188& billions which shows the emerging
need for big data processing in the microgrids [13].
Big data analytics provides the microgrid with the appropriate
powerful computational methods. Big Data platforms like Hadoop and
Spark are able to process this data within the minimal time to improve
operation efficiency through fast data collection, grid adaptation to the
user, microgrid management for best operation. It can help for main-
tenance decisions in appropriate time and prevent failures in the power
grid network. A simple conceptual view is shown in Fig. 2.
The new era of big data analytics is providing much more ad-
vantages than the traditional used techniques in the power grid in-
dustry. It shall provide the parallel real time speedy processing to the
data to knowledge extraction and discovering value underneath of the
Fig. 2. Conceptual view.
2
Advanced Engineering Informatics 42 (2019) 100945
collected data in a decentralized computing platform where analysis is
moved near the data [14]. The cost of deploying big data in the mi-
crogrid market is being studied. Moreover, the big data processing
centers can be fed by renewable energy for complete economic and
environment friendly environment. Big data applications in microgrid
field are of great value, impact and importance. In this paper, we
summarize the sources of big data in the microgird, platform constric-
tion of handling the big data in the microgrid, and summarize case
studies.
The rest of the paper is organized as follows. The big data Vs in
microgrid is discussed in Section 2. Section 3 summarizes the data
sources that formulates the big data problem in the grid. Section 4
presents the widely used big data environments for processing. Section
5 introduces the layered representation of big data problem solving.
Opportunities offered by adopting big data solution in the power grid
are studied in Section 6. Challenges of proceeding with big data in
microgrid and open points are discussed in Section 7. Finally, the
conclusion is presented in Section 8.
2. Big data problem in the microgrid
Microgrid includes advanced technologies, control, sensors, and
data transmission for the sake of better services in the power grid. Data
infers important important to different stake holders such as distribu-
tion, power dispatch, power generation and transmission [15]. Data
gathered by the microgrid are increasing in size and complexity. Data
becomes a big data problem when the traditional data processing
methods cannot handle that amount of data (volume), data growth
(velocity), or format (variety). Big Data does not only infer the huge
amount but also the variety and velocity of the data [16]. Even, big data
can be characterized by the following 7V’s [17] shown in Fig. 3:
• Value: Big data processing results in the value extracted from the
microgrid data. The two way communication between the grid and
the customers reveals opportunities such as electrical energy
transfer with dynamic pricing between loads and generation, better
paying rates for the consumers, better profit for the utility, and more
savings for the clients could be achieved.
• Volume: The tremendous data in the microgrid is generated by a
plethora of sources such as consumers data, sensors, equipment
data, real time acquisition data, operation data, weather, cloud
storage, SCADA, metering units, weather prediction, assets mon-
itoring, finance, energy market, social media, power quality data,
signals waveforms, etc. [14]. Weather prediction problem only re-
quires 18 billion calculation [18].
• Variety: Different kinds of data and communication protocols are
available in the microgrid such as remote sensing data for renewable
Fig. 3. Microgrid big data 7Vs.
K. Moharm
energy integration coming from satellite, historical data in cloud,
etc. Microgrid data is a joint among structured data, unstructured
data, and semi structured data. Structured data is modeled through
tables and rational data bases and examples are data generated by
smart meters and sensors [19]. It can be used for representing
consumption and market. Semi-structured data model through
JSON, and XML. It can be used for several sensors data and one
example is weather data representation. Social media data, human
interaction, and visual data are examples of unstructured data [14].
• Velocity: The fast communication technologies inside the microgrid
is increasing the speed of data generation and processing. Data in
the microgrid can be in batches like planning data, online data like
SCADA, and it can be streaming data like PMU, and Cyber Security
data [14].
• Validity: Data should be valid for its intended use.
• Veracity: Data communicating between microgrid elements can be
noisy, certain and reliable and maybe not. Observations data shall
be checked against the reasonable change. For example, checking if
temperature can change by this value within this time range or this
outlier observation shall be neglected.
• Volatility: Microgrid data is not directly neglected or removed after
processing, contrariwise the traditional methods.
3. Big data collection in microgrid
Sources of big data in microgrid can be:
3.1. Equipment related data
Equipment data results from monitoring and management are so
spreading and can include data from tranformers, switch gears, trans-
mission lines, ccables, substations, and circuit breakers [20,21].
Transformer data includes data about oil pressure, discharge,
ground current, nominal data, manufacturer, capacity, connection,
cooling system, release data, transformer model, fault recorders, de-
scription, fault source, transformer dielectric loss, polarization index,
resistance, insulation data, moisture content, load data, short circuit
current, tap-change data, and tripped situations [22]. Switch gear data
such as discharge, moisture content, and discharge info. Transmission
lines data such as voltage sag, electrical signal waveforms, and leaning
of the tower [22]. Power cables data such as temperature, discharge,
and circulation current [22]. Substation data such as name, ID, type,
location, and pollution status. Circuit breaker parameters such as
nominal voltage, current, capacity, connection, model, manufacturer,
cooling system, release data, operation data, fault recorders, fault
source, fault action, substation name, tripping details, SF6 inspection
reports, insulation resistance, control mechanisms, equipment produc-
tion data, and number of circuit breakers.
Equipment monitoring data represents a big data source.
Monitoring data can include energy management system such as
equipment voltage, current, power, frequency, etc. Equipment real-time
monitoring data can represent transformer healthy operation, ventila-
tion, lubricant or oil status. Also, voltage and current waveforms are
stored for power quality calculation such as the total harmonic distor-
tion and flicker index [23]. Distribution relay responses with 40 bytes
for Modbus [24] standard protocol for equipment monitoring [25].
Remote Telemetry Units (RTU) used for monitoring of renewable en-
ergy [26]. Equipment diagnostics data can be data about previous
faults, location, trip times, voltage dips records, transients, faults de-
scriptions, previous disturbances, and digital protection data.
3.2. Distributed generation data
Generation datasets such as that represents the satisfaction of the
generated power to the consumption. The data represents if the gen-
eration power is sufficient, more or less than the load needed power.
3
Advanced Engineering Informatics 42 (2019) 100945
Each renewable micro energy source has its own data. Biogas energy
is dependent on the anaerobic digestion of biomass. Data source for
biogas energy simulation is available using BSM2 tool [27]. Wind tur-
bine generation includes data about amount of generated energy,
monitoring, and control depends on different sensors data like em-
ploying temperature sensor PT100 for external air temperature,
gearbox temperature, casting temperature, and enclosure temperature
[28]. Wind speed is measured using Anemometer and direction is de-
termined using wind vane. Other sensors are used in wind turbines
operation like humidity sensors, accelerometers for tower movement
measurement, electrical voltage and current measurement, and rota-
tional speed measuring (for example, hall effect sensors).
Concentrated solar plant (CSP) generates data like tilt angles, lo-
cation, orientation of the collector, mirrors dimensions, generated
power, current limit, pump head specifications, ultrasonic signals to
detect possible damage, and temperature status signals [29]. Con-
centrated solar plant must have data about market demand, meteor-
ology information given by Anemometer, Pluviometer, Hygrometer,
Thermometer, Barometer, Pyrometer, Cloud cover, and Radiance index
[29].
SCADA data provides online real-time data, measurement flows
data, set points, control commands, grid assessment, and grid variables.
3.3. Storage battery data
Battery management system includes data about its internal states,
terminal voltage, State of Health (SoH), State of Discharge (SoD), fault
detection data, temperature, voltage, and current [30].
3.4. Load-side data
Load management data sources such as the smart meters, DHC
communication between appliances in the smart home [31], customers
life patterns, consumer questionnaire [32], cooling/heating, water
status, gas consumption, and local building data such as rooms tem-
perature and attendance in building. The load-side is an extensive
source of big data. For example, 1 Million Smart meters produces more
than 1.82 Tb of data [33,34]. Big data can reveal the load type from the
energy consumption data such as consumption patterns. Consumers
also may share the obtained loads information [35].
Load generated data such as the electric vehicle data also is a big
data source. The data encompasses all data like vehicle location, faults,
pedal readings, motor drive, velocity, tire stress, etc. Beijing Electric
Vehicles Monitoring and Service Center [36] reserves the real time data
(online) and stored data.
3.5. Environmental data
Numerical Weather prediction data represents high volume suspi-
cious data streaming in different velocities in different structures (the
standard or time stamp) [37,18]. Weather prediction equations are
highly complex and nonlinear.
Solar and wind data have different sources. Satellite data is pro-
vided by National Aeronautics and Space Administration (NASA)
globally [38]. Environmental data can be provided by some local or-
ganizations provide data like Renewable Energies Organization of Iran
(SUNA) and Iran Meteorological Organization (IRMO) [39].
3.6. Sensors
The spread sensors for maintenance or operation are a viable source
of data. Sensors generated data may be expressed in key-value pairs.
(sensor ID, time step, consumption, etc.). Metering infrastructures in
New York State generates more than 127 terabytes per day [14].
K. Moharm Advanced Engineering Informatics 42 (2019) 100945

3.6.1. PMU
Phasor measurement units (PMUs) and fault disturbance recorders.
More than 2500 phasor measurement units in China [40,41]. PMU logs
the voltage, frequency, current, and phase angles with time stamp. It
communicates with the GPS.
3.6.2. Wireless network connections
Wireless Networked Sensors [42] are a main source of big data in
the microgrid. One example [43] is for PV modules inspection. This
data can be the current, power, temperature, serial number, voltage,
and sun intensity per each PV module. All data sets are considered as
the highest irradiance does not strictly mean the highest output power.
At the highest irradiance, temperature is high and affects the output
power negatively. Data can be transferred using Wi-Fi or Zigbee [44].
3.6.3. Cloud storage
The cloud can serve as a storage for the data collected by smart
homes, sensors, meters, etc. The cloud also supports the security,
availability, fault-tolerant storage [45]. The big data can make use of
the vast cloud storage.
3.6.4. Internet of Things
IoT is a source of big data [46]. IoT represents networked sensors
embedded in devices.
3.6.5. Specialized monitoring sensors
Specialized monitoring sensors such as anemometer, illumination,
ceilometer, acceleration, stresses, strain, speed and power data given by
blades, gearbox of wind turbines, bearing sensors [47], frequency dis-
turbance recorders (FDR), customer software for pricing, outage man-
agement systems and wide-area monitoring systems (WAMS) [48].
3.7. Off-domain data sources
The off-domain data sources that not directly related to the power-
industry but affects it are shown in Table 1.
3.8. Public datasets
There are several public datasets that represents big data in the
microgrid as shown in Table 2.
4. Big data environments in microgrid
Some general big data tools are:
• Apache Hadoop [54,55]: was developed by Dough Cutting and Mike
Cafarella in 2005. Most popular big data platform built on the top of
Hadoop Distributed File System (HDFS) [56]. HDFS is used for
storage through dividing the data files into 64 MB partitions called
chunks using like master-slave architecture. Master node (called
NameNode) is responsible for locating the data and retrieve data
from NameNodes (slave nodes) when requested. It includes Ma-
pReduce [57] for distributed data processing and running the ap-
propriate jobs on data. Disadvantages of Hadoop is somewhat heavy
resource usage at operation [58]. Hadoop is recommended for batch
processing of microgrid big data that analyses the data without high
caring on response time [59]. Through Hadoop, Java scripts can be
used for programmers while other users can access big data using
Hive [60], Pig [61], and Avro [62]. Hadoop consists of Hadoop
Common libraries and modules, HDFS, Apache YARN [63] for re-
source and task management and scheduling, MapReduce. Hadoop
offers high reliability as single or even multiple failure will not block
Hadoop operation, scalability to thousand of nodes is easy, high
performance and fast operation, and open source with no cost [64].
It is an ecosystem and not single application and has a number of
components most of them are created by Apache Software Foun-
dation and open source. Main components of Hadoop ecosystem are
Hive, Drill, Pig, Cascading, Giraph, and Mahout.
• MapReduce [65]: An effective distributed processing approach in-
troduced by Google. It performs two steps of operation. The map
task its input dataset as key/value pairs. The task does some scalar
transformations. Outputs of map are grouped by the keys, sorted,
and divided to multiple smaller tasks. Reduce task inputs are list of
keys to be handled. It does set operations on values arrays for each
key and can optionally combine matching keys from map operation
and it collects the results.
• Apache Zeppelin [66,67]: A web based interface for interactive data
visualization, analysis, and understanding.
• Apache Zookeeper [68]: distributed configuration service and syn-
chronization. It can manage HBase and Hadoop clusters.
• Apache Chukwa [69]: built on HDFS and MapReduce used for
monitoring of distributed systems. It has tools for displaying and
analyzing data results.
• Apache SAMOA [70]: (Scalable Advanced Massive Online Analysis)
is a distributed streaming machine learning framework for data
mining and machine learning tasks.
• Apache Mahout [71]: provides machine learning
• Apache RHadoop [72] for data mining and visualization.
• Apache Hive [60]: a data warehouse used for data storage on top of
Hadoop MapReduce. Its main components are Hcatalog and
WebHcat. Data can be written easily in Hcatalog as it is a table while
HTTP interface is available through WebHcat that enables running
Hadoop MapReduce, Pig, and Hive.
• Apache Pig [73]: programmed using PigLatin and run into MapRe-
duce tasks on Hadoop for parallel data flow system for datasets
analysis.
• Apache Flume [74]: streaming data acquisition tool to HDFS.
• Apache HBase [75]: Data actually get stored in HDFS. NoSQL data
storage. Hadoop Data base used when the HDFS not enough at cases
when data is updated by more than one application [76].
• Cassandra [77]: open source distribution database management

Table 1
Off-domain data sources in microgrid.
Data source Summary

Human Data such as the customers feedback, customer calls at power outage [49], population, behaviors, public security data, and consuming behaviors.
Human generated social media data sources like emails, Facebook, Twitter, etc.
Off-Domain Generation Data such as the traffic condition data
Off-Domain Monitoring Data such as the environmental conditions and its effect on the equipment such as the temperature and rain, animal migration data for outage
expectation and management, and Seismic data [49]
Enterprise data Data about the power generation from the enterprise
Video on-site video recording
Financial data Data during operation and management including business reports, prices, sales, and market conditions
Records electrical energy consumption per population, spare renewable energy, education rate, land price, emissions, grid connectivity length to the area,
water consumption

4
K. Moharm Advanced Engineering Informatics 42 (2019) 100945

Table 2
Public available datasets.
Dataset Description

Sinogreenergy represents solar generation data sets [50]

EnergyPlus other data sets sources can be provided by [51]
NREL [52] provides geospatial data and provides large datasets about energy data resources like biomass, geothermal, marine, wave power, wind, solar, and Hydrogen
data for U.S. and some other areas.
OpenAMIData Structured example of the 5 min energy data contains time stamp, date, time, value, and correctness of the data can be found in the online repository [53]

system and used to store high volume datasets. It supports cloud and historical data based on Hadoop and Storm with open source
storage. It is used at Netflix, Twitter, etc. stack integration [96]. The solution is deployed in [97] for energy
• MongoDB [78]: used for storage not based on Hadoop. efficiency big data applications.
• Apache Sqoop [79]: (SQL to Hadoop) used to transfer structured • PS CLEMENTINE PRO [98] provides a solution for big data analysis
data to Hadoop. Apache Sqoop or Apache Tajo can be used to in- environment and mining that used by a study in [99] for high re-
tegrate the smart meters data with the big data processing. Smart liability operation and revealing correlation between elements
meters data can be saved in relational data bases like PostgreSQL based on equipment and monitoring data.
[80,76]. • Oxdata H2O [100] is a scalable machine learning platform for big
• Apache Tajo [81]: parallel querying for big data warehouse. data and can be standalone or run over Hadoop or Spark. It is used
• Apache Spark [82,83]: For analytics, scalable and fast data proces- for energy consumption prediction [101].
sing due to its built-in memory computing framework. Spark can run
as a standalone, on Hadoop, on cloud or on Mesos. It can be used for Big data and cloud computing are very popular research topics.
real time data processing and batch processing [59] and can be in- While the high volume data is the main motive for big data and cloud
tegrated with R for data mining. Spark consists of [84]: computing concepts, they are not the same. Big data methods can work
– Spark key component: for task scheduling, memory and fault without cloud computing technologies but combining big data and
handling. cloud computing enables more economic and efficient analysis. Big data
– Spark SQL: programmed by Java, Python, or Scala for processing refers to have large volume datasets and dealing with them. Cloud
data like Hive tables, Parquet, and Jason. computing refers to network services, platform for getting data, ac-
– Spark streaming component: for handling data in memory that cessing, and computing.
makes data processing much more faster than Hadoop which uses
drive. 5. Big data construction in microgrid
– MLib: a library includes machine learning algorithms like re-
gression and clustering. The construction of big data platform in the microgrid can be de-
– GraphX: a component for visualization. scribed in layered architecture [102] shown in Fig. 4:
• Apache Flink [85,86]: Flink can be used also for real time data
processing and batch processing. Flink is scalable, fault tolerant and • Database Collection Layer: It gathers the data from the sources like
fast. Flink is used for machine learning applications because it is sensors, load data, generation data, equipment data, etc. presented
coming with its FlinkML machine learning library [87]. To store in Section 3. It also transmits the data via wire or wireless
data, Flink has ready integration with HDFS.
• Apache Storm [88,89]: The recommended tool for real time pro-
cessing of big data in the microgrid. Other tools for real time pro-
cessing are S4 [90], StreamBase [91] and Splunk [92]. Storm offers
open source, fault tolerant, parallel processing, and simple pro-
gramming. Storm consists of some definitions like Stream, spout,
bolt, and topology. Storm creates a graph for computation called
Topology. Stream is the core of Storm which is unlimited sequence
of tuples where a tuple is a named list of values (integars, decimals,
etc.). The tuples is read from external sources and injected in the
topology by the Spout. A spout is a logical node acting as the source
of streams. Bolts are logical units that process input streams and
emit the output streams.
• Apache Kafka [93]: fault tolerant that keeps the data if an error
occurred. Kafka can be easily integrated with Storm for data in-
tegration. It can be used for storing streaming data.

Different from general big data ecosystems, some platforms have

been proposed. Generally in the microgrid, the big data initiatives are
using Spark over Hadoop due to easiness to access more machine
learning algorithms and iterative or streaming real time processing
[94]. Better for microgrid data processing is streaming data processing
more than batch data because of continuous real time pricing or sta-
bility analysis.
The customized big data processing platforms:

• SuperDoop environment provided by Ingenia [95] is a complete Fig. 4. Big data processing and analytics platform in the microgrid.
solution for big data storage, scalable parallel processing in real time

5
K. Moharm
communication for the analytics.
• Data preprocessing: The layer includes subsystems for data interfa-
cing using tools like SQOOP for interaction with HDFS and tradi-
tional data storage, FTP to data exchanging, and MQTT for data-
bases handling. The output of this layer is data transformation,
assessment, and outlier detection and correction [103].
• Data Management: Management of the big data whether un-
structured or structured using Hive, Hbase, Impala. Also, MongoDB
represents lately outstanding solution as it has compatibility with
the MapReduce [104].
• Data Analytics Layer: apply big data tools on the data to mine and
make use of the data. Tools of this layer can be Spark, MLib [105],
GraphX [106], Python, MATLAB. The purpose of this layer is to
achieve analysis or risks, decision support, correlation [107].
• Application Layer: Reports the final results of the mining technol-
ogies and the sense of the data. The report may be visualized, esti-
mation, grid and equipment expectations.
6. Big data applications in the microgrid
Big Data analytics offers enhanced business decisions to have new
customers, recognize the potential competitors, eradicate the electricity
deception, more efficient markets based on data and not only experi-
ence, reliable operation, and distinguish the needs of the customers
[108].
Big data also finds its applications in different elements of microgrid
like green and renewable energies (wind, solar, marine, etc.) [109],
control, load management and operation. Big Data enhances the op-
timal operation of the microgrid by making use of the data collected for
better decision making. Big data applications can be [110]:
• Eradicate electrical energy leakage and fraud detection [102].
• Identification of the electrical equipment connected to the microgrid
based on big consumption data patterns classification [45].
• Enhance reliable, sustainable, safe, effective operation of the power
grid through the computational analysis and prediction provided by
real time big data mining.
• Improve integration of DG in the power grid, fault detection and
power quality [111].
• Big data can support very high sampling rate data streaming of
electrical signals like the current, voltage, etc. This can enhance
power quality analysis of the microgrid [112].
• Enhance deployment of complex control algorithms to achieve
better and efficient power grid operation.
• Support decision makers through highlighting the areas to be im-
proved.
• Implementation of dynamic pricing algorithms. Consumers pattern
analytics will enable analysis of the millions of users to visualize or
analysis of the rate that will be assigned [14]. EDF [113] already
estimated 35 million of smart meters in the French network for the
real time pricing for lower bills. Data generation estimation is about
120 terabytes per year.
• Load forecasting data generated by big data analytics for every
consumer every time step can be used to improve utility planning,
determine unit price, decrease operating costs, increase the quality,
and help reducing power loss [14,].
• Assets monitoring and predictive fault analysis will allow predictive
maintenance.
• Value extraction can be detected. Big data can help selection of
relevant and non-relevant factors in decision making and future
planning.
• Enable backup solutions for the power grid enterprises [115].
• Enable dynamic pricing strategies.
• Peak load management.
• Demand response management and scheduling of temperature
controlled loads like heat pump days ahead [116].
6
Advanced Engineering Informatics 42 (2019) 100945
• Value and knowledge discovery and visualization about customers,
power plants operation and control, and markets after mining the
big data in the microgrids [84].
• Protection of data hacking attacks [117].
• Real time monitoring of the microgrid and discover correlations
[118].
• Optimal operation considering operation and maintenance cost.
The applications are discussed according to:
6.1. Feasibility studies
Big Data can effectively participate in feasibility studies for different
improvements in the elements inside grid. For example, big data sup-
ports discovery in battery materials innovations [119]. Another ex-
ample, it can enhance the electric vehicle industry, being part of the
microgrid. Bryan et al. [120] presented a big data-based approach for
analyzing the extension of the electric vehicle range issue impacted by
increasing the power limit of the battery to allow more regenerative
braking power on the plug-in hybrid electric vehicle range. Data re-
corded by different driving styles of Ford employees and could found
the slightly impact of the increase on the range.
6.2. Planning & decision making
Big data would improve the urban planning [108] and the microgrid
planning as well based on support the transition to renewable sources of
generation. The proper planning is achieved as big data can combine
the sales, operational data, management, etc. to achieve the optimal
planning. The energy market combines different sources of energy like
fuel, nuclear, renewable energy which includes biomass, solar, hydro,
geothermal and wind energies. Making a policy future plan for the
emphasis of the future energy source involves big data. Big data sup-
ports policy makers to have energy sources selectivity. Chalvatzis et al.
[121] used big data approaches to help the UK decision makers to select
their future sources of energy plan taking into consideration the tech-
nical and environmental terms. The results based on the detailed ana-
lysis indicated investment should have tendency to mix all renewable
energy sources except biomass, even being environmentally friendly, in
UK. The studies are summarized in the following areas:
6.2.1. Micro generation optimum plant design
Big data has its applications in the planning to design most efficient
residential power plant. Huang and Chang [122] provided an approach
to build rooftop power plant based on data about roof material and
height, inverter specifications, solar power, number of Maximum Power
Point Tracking (MPPT) units and environment data like azimuth angles
gathered by Sinogreenergy [50] to recommend best efficient build solar
power plan in Taiwan based on best connection type (series, parallel,
series-parallel), altitude angle and module voltage. The system uses
Self-Adaptive Harmony Search (SAHS) [123] for distinguishing relevant
elements and selecting their weights. Association rules, SVM, and K-
means classifiers are employed to identify optimal models.
Vetas Wind Turbines in Denmark and IBM use big data processing
for optimal location for wind turbines based on data from weather, tidal
expectations, geographical, and satellite [115].
Big data and cloud computing perspectives can manage con-
centrated solar plant and monitoring based on big data sources like
solar data collected, data stored, meteorology, market demand, and
equipment conditions [29].
6.2.2. Wave energy integration support
BigDataOcean project proposes maritime big data for interest of
organization and scientists for maritime sector enhancements and em-
powering development. The project value chain starts from filtering of
measurements of wave, weather, marine and equipment and supporting
K. Moharm
site planning, assessment of wave energy potential, and grid impact
analysis [124,125].
6.2.3. Operator decision making support
Big data application in microgird can help operators for better de-
cision making under circumstances like hurricane and the storms. [22]
provided an Hadoop based platform, called Disaster-Mitigating Dis-
patch Platform (DMDP), for disaster prevention and immediate re-
sponse. The purpose of this platform is risk identification, visualization
(like through warnings and emergent locations and repairing centers)
and emergency response decision making like vehicles and transpor-
tation distribution at the weak points and during the disasters. Zoo-
Keeper is used for cluster co-ordination. Data stored in HBase from
weather, environmental, historical, equipment status, etc. is mined to
provide preventive procedures to weak areas, identify major problems
with corrective actions in case of power outage due to disasters, identify
weak areas in the grid, self enhancement and efficiency calculation of
measures done.
6.2.4. Optimal renewable energy integration
[35] presented a solution to dispatch between the renewable
sources and the conventional grid. The renewable energy is supportive
when the conventional grid not able to provide the full electricity
consumption. The methodologies are presented and applied on town
had 2700 residents.
Ifaei et al. [126] analyzed Iran’s case study about load and climate
big data for renewable energy management using ANOVA and ANN.
Related to RES prospect, the study provided clustering Iran into 5
clusters. The big data analysis is used for selection of optimal design
and areas of microgrid implementation and making action plan prior-
itized in Iran.
6.3. Power prediction
Cloud based platform with big data find application in microgrid
planning through matching between the supply and the load. [127]
proposes Cassandra cloud storage of historic weather, consumers pro-
files, generation data, and load data. The power generation is predicted
based on the data. Processing the data is performed using HDFS for data
retrieval and then MapReduce for processing and prediction.
Grolinger et al. [101] presented an energy consumption prediction
strategy based on local learning with Support Vector Regression (SVR)
of big data [128]. The study compared SVR, local SVR, and H2O deep
learning results for energy consumption forecasting with superior ac-
curacy gained by Local SVR.
Daki et al. [129] proposed big data based solution for energy fore-
casting system for matching the production with the consumption in a
Moroccan engineering school. Data sources include semi-structured
weather data, unstructured Kafka streaming real time sensors data, and
structured operational data (planning and equipment) from rational
data bases using HTTP protocol. Data stored in HBase column-oriented.
MapReduce for data processing to manage distributed computing
system. Zookeeper is a mandatory component for co-ordination for
HBase. Machine learning algorithms are applied for prediction of the
electrical consumption. The studies can generally categorized as fol-
lows:
6.3.1. Intermittent sources forecasting
Big data can improve forecasting of the output power of the inter-
mittent sources like the wind turbines. [130] used game theoretical big
data analytics for short term prediction of the output power of the wind
energy in a microgrid in Hebei in China.
[104] built big data based Solar Photovoltaic Power Forecasting
System (SPPFS) to forecast the photovoltaic energy accurately based on
real-time neural network processing of meteorological data.
7
Advanced Engineering Informatics 42 (2019) 100945
6.3.2. Weather forecasting
For the optimal operation of the renewable energy like the solar and
wind, estimates of the output power. The weather forecasting may add
strengths to the planning of the power grid days or hours ahead [37]
and would enhance the scheduling of the sources in the grid. SUN4-
CAST [18] represents a model for big data-based approach for weather
prediction.
6.3.3. Burden forecasting
Old methods for long period forecasting the load like regression,
wavelet analysis, SVM, and ANN are prone to disadvantages like apt to
uncertainties conditions like the environmental conditions and climate,
so detailed analysis is requested. Effective burden forecasting enhances
the stability of the microgird and improves the power dispatch between
sources. Big data supports detailed analysis for the environmental
conditions and electricity consumers behavioral data with the historical
analysis of the old data. Hu et al. [131] presented a framework based on
Spark to predict the load to be considered for dispatching and planning.
[96] used SuperDoop to predict consumption model based on energy
data using Lambda architecture for big data analytics considering data
acquisition challenge and speed batch processing [132]. SuperDoop
based analysis with machine learning allows identification of con-
sumption for each home based on meters and historical data.
[133] studied short term load forecasting based on Hadoop big data
analytics of smart meter data.
[134] studied short term load forecasting based on improved SVM
based on Hadoop for faster computations and better prediction based
on load data.
[97] studied application of big data analytics of consumption da-
tasets to forecast future energy behavior and recommending system for
smart home consumers. Big communication data between appliances,
device active or inactive status, cloud stored home data, historical data,
and energy profiles of users are analyzed using ANN deployed on Su-
perDoop to batch and real-time processing, excavate consumption
pattern, generate cheaper recommendations, and energy predictions.
Zhu [135] presented a data mining technique based on Hadoop
HDFS and MapReduce platform with cloud technology for prediction of
the yearly electrical power consumption using electrical power system
data.
6.4. Electrical equipment assessment
Big data provides an opportunity for the monitoring of the on-line
fault detection of equipment working in different environmental con-
ditions. The approach in [131] shows the framework can handle mas-
sive data from different monitoring sensors for health assessment of the
equipment to reduce the blackout and power outage cases.
6.5. Control
Big data analytics can determine the risk based on analysis in the
power grid [136].
6.5.1. Monitoring, reliability, management, cost reduction and maintenance
The management and maintenance of the DG is a challenging topic
to preserve healthy operation of those sources. Big data techniques can
monitor the power system and evaluate its status for decision making
and visualization [137].
Hu et al. [43] presented an experimental approach for PV power
plant operation based on wireless sensor networks presented in 3.6.2.
Using Hadoop for the big data collection of all sensors and historical
data, then merging and semi-supervised Support Vector Machine
techniques (SVM) [138] is used to distinguish the failed modules in real
time. The presented system can also anticipate the output power using
the irradiance data and uses Password-based Encrypted Group Key
Agreement protocol [139] to protect the data transmission.
K. Moharm
Liu [107] et al. presented a big data platform for stability en-
hancement for inspection of fault conditions of transmission lines.
Based on the analysis, decisions can be addressed for prevention and
fast response when faults occur. The data analytics is performed using
Spark, Scala, R, Python and some computational tools like NumPy. The
results are mapped in 3D using Cesium. The correlations between
transients and transmission equipment faults are addressed using big
data to enhance operator’s knowledge about the relationship between
the transmission weather, line trips, and voltage dips to take protection
procedures.
[58] proposed a new big data architecture to recommend reduction
of energy consumption in industrial and mining application and pro-
vide insights and data visualizing for management and decision makers.
The proposed system makes use of MongoDB [140] for document sto-
rage to store multi-structured data integrated with standalone Apache
Spark. Pyhton [141,142] is used for analytics and is integrated with
Python using APIs. Applying the big data analytics on emissions, energy
consumption and costs real time data of mining and industrial com-
panies, it provided identification about wastage areas and areas to
improve to have more energy reduction and pathways to plan com-
pany’s road maps.
[143] presented a proof of concept big data analytics platform in
Alakhawayn University campus microgird using OpenStack [144] for
high performance scalable cloud computing [145] and integrating with
Hadoop and MapReduce for storage and analytics. Running electrical
power consumption and storage status data are analyzed by big data
platform to decision making purposes and decide either to store the
energy or transmit in the campus. For efficient energy management,
each water heater is connected using with an actuator controlled by the
analytics of the big data gathered from temperature and humidity
sensors in the rooms connected using Zigbee [44] protocol through
Raspberry PI controller [146].
[76] piloted a renewable energy management system called Na-
tional Virtual Power Plant (NVPP) based on big data for excess energy
storage, exchanging, load shifting or load reducing. The system pro-
vides future energy demand prediction, policy and economy simula-
tions, automated demand response and monitoring based commis-
sioning services using 2 clusters, one using Spark and the other is using
Hadoop. The system uses Hbase is used for storage, Hadoop big data
analytics of the billing and power metering data. The system uses Sqoop
for gathering structured data, MapReduce for processing, Zookeeper
and Apache Chukwa for ease of synchronization and searching. The
machine algorithm is implemented using Mahout and RHadoop for
mining and visualization.
Liu et al. [147] implemented an energy monitoring system in Tun-
ghai University that gathers the sensors and the smart meters data using
Java programs and then stores it into HDFS and Hive for real-time
Spark big data processing for the historical data (using Sqoop) and the
real time data.
[94] provided Spark big data processing based on hardware and
software implementation to get the stability estimation and monitoring
using PMUs data. The system used Apache Kafka for streaming data,
Cassandra for data storage, and Spark for streaming data processing.
While [148] based environment for big data analytics of millions of
PMUs data for analysis and locating disturbances.
[149] used parallel de-trended fluctuation analysis for fast detection
of PMU measurements based on MapReduce processing of the PMUs
data from openPDC [150]. MapReduce is dividing the PMUs data into
smaller units and then processing the smaller units to calculate the
fluctuation while Reduce stage to compare the fluctuation to the
threshold to identify abnormal situation.
[151] proposed a big data based real time monitoring system for
power loss in the transmission lines based on Spark streaming analysis
of big data collected from spatial data and monitoring data.
8
Advanced Engineering Informatics 42 (2019) 100945
6.5.2. Dispatch
Big data supports new methods for grid scheduling and dispatch
process [152].
6.5.3. Energy savings and optimization
[153] provided an energy saving optimization to reduce substations
energy consumption based on HDFS storage, YARN for scheduling, and
Hadoop big data analysis of the substation variables data like the cur-
rent, voltage, line status, control equipment, primary and secondary
equipment status, environment, etc. The system also provides the vi-
sualization of the consumption data based on the analysis of energy
saving strategies.
6.5.4. Fault diagnosis
[47] supposed a big data-based algorithm driven by Hadoop and
Spark that provides support for maintenance of the vast amount of wind
turbines per wind farm and detection of the abnormal conditions. The
approach used HDFS for wind turbines historical data, Apache Kafka for
read time data acquisition and Apache Spark for data processing. De-
cision trees perform fault identification rules.
[154] provided health monitoring and failure analysis of wind
turbines based on fuzzy-C means algorithm implemented by Hadoop
HBase and MapReduce platform using cloud storage and computing of
gathered sensors and SCADA data.
[28] proposed big data solution based on Hadoop for FFT analysis of
wind turbine for anomaly detection.
6.5.5. Maintenance
Maintenance of wind turbines affects the planning and dispatching
of the microgrid. Maintenance of the wind turbines is a field can get
support from big data [155] because each wind turbine contains 30
sensors and multiple business reports and descriptions. [28].
6.5.6. Storage systems
Mismatch between generation and time varying load may result in
disturbances in the grid. Storage system is one of the microgrid ele-
ments to enhance reliable operation of the microgrid and generation-
load stabilization. There is a possibility of terminal voltage fault due to
several reasons like mechanical weariness and mechanical failure inside
the battery. Big data can support building fault recognition and de-
tection of the exceptional faults in the battery systems depending on
statistical methods and machine learning [30].
Undersized batteries in the microgrid leads to possible power
outages and unreliable operation and higher sizes of unneeded batteries
in the microgrid is high cost. Big data simulation can be used to opti-
mally design storage systems to operate with intermittent PV genera-
tion and isolated PV based microgrids [156].
6.5.7. Stability
Microgrids suffers from lack of inertia that sustain the stability of
the grid. The big data analytics can be used to improve the stability of
microgrid. [40] proposed a big data approach for stability improvement
at microgrid islanding operation when for example a failure happen in a
transmission line. The approach is based on monitoring to detect the
sudden islanding based on Spatial-Temporal approach. After detection
of outage, optimally PSO approach is employed to enhance dynamic
response simultaneously with occurrence of disturbances in most effi-
cient operation.
[157] proposed intelligent stability analysis system using Hadoop
processing on historical data, SCADA, EMS, WAMS, etc. for addressing
the stability and risk management with the aid of advanced data mining
techniques like ANN, Deep Reinforcement learning, and Automatic
supervised learning.
6.5.8. Energy efficiency support
[158] proposed a big data-based approach for energy loss detection
K. Moharm
through analysis of irregular electrical consumption patterns inside
buildings based on MongoDB data storage for historical, operational
(lighting, temperature, ventilation, etc.), environmental, and energy
usage data.
6.5.9. Power quality assessment
[159] proposed real-time transient response Naive Bayes classifi-
cation [160] based on Hadoop-MapReduce platform big data processing
on historical and streaming user data to identify the status of the
whether regular, small disturbance, potential risk or defect mode.
[111] proposed a Hadoop based system for harmonics analysis in
active distribution systems for THD and harmonics estimation.
6.5.10. Power-cut management
Weather and environmental conditions usually cause power grid
cut. Big data can analyze the historical data, PMUs, sensors readings
and Geographic Information Systems along with advanced measure-
ment units, the environmental conditions, and customer interaction
with emails or calls for better prediction and localization of faults,
sensitive area proactive measures, and detailed analysis of protection
schemes reaction [49].
6.5.11. Intensive power flow analysis calculation
Big data superior computational analysis can be used to solve the
complex power flow analysis. Spark and its interfacing with machine
learning can solve the increased complexity of matrix algebra Newton-
Raphson equations [161] for power flow analysis [162]. Newton-
Raphson method includes the calculation of the power mismatch, par-
tial derivatives Jacobian matrix of active and reactive power with re-
spect to the phase angle and voltage magnitude, and the unknown
vector of voltage and phase angle at each iteration. Spark can help in
calculation of the computational effort like the matrix inverse and
iterations. Spark deployment lead to six times increase in calculation.
6.6. Customer loads
[163] proposed applying big data on SCADA systems steady state
readings combined with WAMS dynamic data for real time modeling
the loads.
Big data can provide customer analysis for variety of applications
like pricing [164] and optimizing power delivery efficiency. [32] pro-
posed a system for 800,000 power consuming data, consumer analysis,
social, economic, and geographical data in Shanghai city. Hadoop
platform is employed after data cleaning. The data mining can lead to
improve planning and load side management.
6.6.1. Smart building control
Big data and IoT can control the smart building appliances. [165]
proposed an integrated IoT based big data analytics system with virtual
streaming sensors transferred from sensors using TCP (Transmission
Control Protocol) [166] to Flume agent and stored in HDFS and pro-
cessing using Spark for visualization and auto tuning of oxygen pump
control, illuminance adjusting and fire alarming in the building.
6.6.2. Load adaptation to supply
[167] provided a big data real time processing using Apache Storm
to reduce load at the peaks by disconnecting selected loads and identify
if the building is empty to be disconnected and the savings. A Storm
streaming data spout is gathered from customer location through mo-
bile app and the other spout represents device measurements. The da-
tabase contains info about all appliances inside the home, family size,
etc. Data is processed (Storm bolts) through retrieving house ID and
location, checking the family is inside the house based on their location
and home location, customer permission to disconnect his devices, and
calculate the potential savings.
9
Advanced Engineering Informatics 42 (2019) 100945
6.7. Marketing enhancement
Big data analytics supports customers segmentation and do statistics
for the market. [168] proposed an initial prototype for cloud based
Hadoop platform for doing energy consumption statistics. Processed
data in < key, pair > values where the key is the serial number of the
customer and the value is the energy feeding source values whether
wind, hydro power, thermal, etc. Reduce task to collect the number of
customer per each energy source for statistics purposes.
6.8. Detection of data attacks
As data grows in the power grid, it is prone to hacking attacks. For
example, SCADA systems nodes communicates packets data of high
importance. The packets include multiple attributes like the setpoints,
gain, control commands, etc. Big data supports classification of those
attacks for proper action. By K-means clustering for analysis of SCADA
communication packets history, Paramkusem and Aygun [117] pro-
vided a big data analysis of communication packets (read and write
packets) in the SCADA nodes based on Hadoop and random forest
classification [169] built on Mahout for detection and classification of
hacking attacks.
7. Challenges & future prospects
Big Data in the microgrid offers a lot of achievements, simulta-
neously, it offers open points to be proceeded in future work. Some
challenges are in the fields of:
• Real time parallel processing as some big data like Hadoop not able
to operate with the streaming data [170] given the persistent need
for very hard real time processing as the transient control in the
microgrid for extreme conditions like faults requires real-time re-
action. Even if the latency, processing congestion, and how com-
plicated were the algorithms, the reaction time may be less than
milliseconds [171].
• Data Collection: Data collected may be introduced to some in-
accurate, incomplete and unreliable at some times. For example, not
all microgrid sources and elements would have techniques for real
time data transmission or cases when some sensors are missing in
some units. More investment can be directed to operate new sensors
technologies and enhanced data quality and automation of all data
entries. The data collection requires also support by the government.
• Data Privacy: Data can be infiltrate, so it must be protected espe-
cially for some personal data like usage, and load information. Law
can enforce data privacy for the big data in the microgrid using
legalization strict laws for data ownership [19]. Customers must
trust their private data for its intended use only.
• Data security: Data should be secure from hacking activities. This
data is highly considered in prediction and analysis, so it must have
secure data transmission and highly encrypted [172]. Missing a
platform that incorporates data security mechanisms with big data
analytics in the microgrid [173].
• Data storage: The expansion in the amount of data, representation,
and format requires expansion in the data storage management tools
and technologies.
• Data visualization enhancements: The waveforms of electrical net-
work can be visualized easily. However, for the complicated re-
lationships and correlations required for decision making or getting
value behind the data needs to be enhanced given the multi-source
data, different standard, and variable formatting [171].
• Data Processing: Historical data shall be merged with the real time
instantaneous gathered data related to operation and electricity
usage and consumption and proceeded with in very short time.
• Data opening: records the data and enable public to have good
quality data to enable companies and developers find ways to
K. Moharm
improve the grid.
• Existence of reliable databases for power data collected from dif-
ferent sources.
• Cost effective big data solutions for the whole solution including
data center, tools, storage, control algorithms, warehouses, etc. The
current power grid stakeholders cannot easily accept cost of the
massive sensors and the new infrastructure [174].
• Integrated holistic approach study of big data solution incorporates
cloud with sufficient scalable specialized functionality in the mi-
crogrid.
• Perform online learning and make use of its continuous feedback
correction and its superior precise prediction than the statistical
machine learning approaches [175,176].
• Unsuccessful applicable obligatory data standards in the microgrid
and lack of big data analysts for the whole microgrid data sources
and processing. [172]: Several sensors and communication meth-
odologies are producing data operating in the microgrid. The need
for a standard communication and protocols describes the speed,
data transfer, and secure data transmission.
•
Data integration [172,177]: The challenge for integrating different
sources like social data, sensors, reports, etc. together as there is no
standardized environment today.
•
Big data techniques are viable and effective for only huge data sets.
Little data sets will not have the effectiveness of technologies like
Hadoop [178].
•
Combination of cloud computing and big data processing provide
new perspectives for analysis and incorporation of IoT data for
management of the microgrid [179,180]. Big data and cloud com-
puting are complementary as big data is an application of cloud
computing while cloud computing provide tools for big data pro-
cessing [115].
•
Using Starfish for Hadoop based big data processing for optimizing
the configurations to get the maximum Hadoop power and flex-
ibility like number of reducers, memory settings, etc. [181]. Hadoop
has over 190 configurations parameters having high performance
effect and need to be investigated. Few researches [182] consider
tuning of the Hadoop parameters in power grid applications and
show more speedy performance for PMUs data processing. Other
advances can be achieved with the optimal configuration.
• Big data sources integration
• Uncertainty about the big data integration in the current microgrid
systems [183].
• The need for very effective preprocessing and data quality en-
hancement algorithms. For example, the failure of a critical sensor
must be discovered and cleaned [183].
• Missing innovative case-studies that transform big data into opera-
tional intelligence [184].
•
Imperfect research on big data system architecture analytics design
in the microgrid [184]
• Missing return on investment studies and researches about impact of
applying costly big data analytics in the power grid.
•
The necessity of intelligent selective real-time processing algorithms
for big gathered data as the blackout may occur within multiple
seconds [173].
•
Skills leakage for big data experts in the energy sector [185].
• Lack of familiarity of end user with energy data [185].
8. Conclusion
In this paper, we demonstrated the Big Data relevant opportunities
and studies in the microgrid applications using the presented layered
analytics framework. The data sources, storage, analytics, are pre-
sented. Several opportunities are presented like the optimal design of
power plants, grid stability enhancement, optimal dispatching, assets
maintenance, and fault diagnosis. Nevertheless, there are still open
challenges like the data security and privacy. Big Data will impact
10
Advanced Engineering Informatics 42 (2019) 100945
soundly the microgrid creating it more efficient, reliable, and profitable
microgrid.
Declaration of Competing Interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influ-
ence the work reported in this paper.
Appendix A. Supplementary material
Supplementary data associated with this article can be found, in the
online version, at https://doi.org/10.1016/j.aei.2019.100945.
References
[1] S. John Walker, Big data: A revolution that will transform how we live, work, and
think, 2014.
[2] A. Vera-Baquero, R. Colomo-Palacios, O. Molloy, Towards a process to guide big
data based decision support systems for business processes, Procedia Technol. 16
(2014) 11–21.
[3] K. Venkatram, M.A. Geetha, Review on big data & analytics–concepts, philosophy,
process and applications, Cybernet. Inform. Technol. 17 (2) (2017) 3–27.
[4] M.R. Bendre, V.R. Thool, Analytics, challenges and applications in big data en-
vironment: a survey, J. Manage. Analyt. 3 (3) (2016) 206–239.
[5] B. Fang, P. Zhang, Big data in finance, Big Data Concepts, Theories, and
Applications, Springer, 2016, pp. 391–412.
[6] X. Ge, J. Jackson, The big data application strategy for cost reduction in auto-
motive industry, SAE Int. J. Commer. Veh. 7 (2014-01-2410) (2014) 588–598.
[7] Y. Wang, Q. Chen, T. Hong, C. Kang, Review of smart meter data analytics: ap-
plications, methodologies, and challenges, IEEE Trans. Smart Grid (2018).
[8] X. Liu, B. Su, Microgrids-an integration of renewable energy technologies,
Electricity Distribution, 2008. CICED 2008. China International Conference on,
IEEE, 2008, pp. 1–7.
[9] H. Jiang, K. Wang, Y. Wang, M. Gao, Y. Zhang, Energy big data: a survey, IEEE
Access 4 (2016) 3844–3861.
[10] S. Sagiroglu, R. Terzi, Y. Canbay, I. Colak, Big data issues in smart grid systems,
Renewable Energy Research and Applications (ICRERA), 2016 IEEE International
Conference on, IEEE, 2016, pp. 1007–1012.
[11] Y. Song, G. Zhou, Y. Zhu, Present status and challenges of big data processing in
smart grid, Power Syst. Technol. 37 (4) (2013) 927–935.
[12] Z. Huang, H. Luo, D. Skoda, T. Zhu, Y. Gu, E-sketch: Gathering large-scale energy
consumption data based on consumption patterns, Big Data (Big Data), 2014 IEEE
International Conference on, IEEE, 2014, pp. 656–665.
[13] J. Zhu, E. Zhuang, J. Fu, J. Baranowski, A. Ford, J. Shen, A framework-based
approach to utility big data analytics, IEEE Trans. Power Syst. 31 (3) (2016)
2455–2462.
[14] H. Akhavan-Hejazi, H. Mohsenian-Rad, Power systems big data analytics: an as-
sessment of paradigm shift barriers and prospects, Energy Rep. 4 (2018) 91–100.
[15] C. Kang, Y. Wang, Y. Xue, G. Mu, R. Liao, Big data analytics in china’s electric
power industry: modern information, communication technologies, and millions
of smart meters, IEEE Power Energ. Mag. 16 (3) (2018) 54–65.
[16] R. Lu, H. Zhu, X. Liu, J.K. Liu, J. Shao, Toward efficient and privacy-preserving
computing in big data era, IEEE Network 28 (4) (2014) 46–50.
[17] M.F. Uddin, N. Gupta, et al., Seven v’s of big data understanding big data to extract
value, American Society for Engineering Education (ASEE Zone 1), 2014 Zone 1
Conference of the, IEEE, 2014, pp. 1–5.
[18] S.E. Haupt, B. Kosović, Variable generation power forecasting as a big data pro-
blem, IEEE Trans. Sustain. Energy 8 (2) (2017) 725–732.
[19] N. Amaro, J.M. Pina, Big data in power systems leveraging grid optimization and
wave energy integration, Engineering, Technology and Innovation (ICE/ITMC),
2017 International Conference on, IEEE, 2017, pp. 1046–1054.
[20] K. Ye, Y. Cao, F. Xiao, J. Bai, F. Ma, Y. Hu, Research on unified information model
for big data analysis of power grid equipment monitoring, Computer and
Communications (ICCC), 2017 3rd IEEE International Conference on, IEEE, 2017,
pp. 2334–2337.
[21] Y. Guo, S. Feng, K. Li, W. Mo, Y. Liu, Y. Wang, Big data processing and analysis
platform for condition monitoring of electric power system, Control (CONTROL),
2016 UKACC 11th International Conference on, IEEE, 2016, pp. 1–6.
[22] Y. Wang, M. Deng, Y. Bao, H. Zhang, J. Chen, J. Qian, C. Guo, Power system
disaster-mitigating dispatch platform based on big data, Power System Technology
(POWERCON), 2014 International Conference on, IEEE, 2014, pp. 1014–1019.
[23] F.C. Trindade, L.F. Ochoa, W. Freitas, Data analytics in smart distribution net-
works: applications and challenges, Innovative Smart Grid Technologies-Asia
(ISGT-Asia), 2016 IEEE, IEEE, 2016, pp. 574–579.
[24] I. Modbus, Modbus messaging on tcp, IP Implementation Guide v1. 0a, North
Grafton, Massachusetts (www.modbus.org/specs.php), 2004.
[25] M. Aiello, G.A. Pagani, The smart grid’s data generating potentials, Computer
Science and Information Systems (FedCSIS), 2014 Federated Conference on, IEEE,
2014, pp. 9–16.
[26] G. Suciu, A. Vulpe, A. Martian, S. Halunga, D.N. Vizireanu, Big data processing for
K. Moharm
renewable energy telemetry using a decentralized cloud m2m system, Wireless
Pers. Commun. 87 (3) (2016) 1113–1128.
[27] I. Nopens, L. Benedetti, U. Jeppsson, M.-N. Pons, J. Alex, J.B. Copp, K.V. Gernaey,
C. Rosen, J.-P. Steyer, P.A. Vanrolleghem, Benchmark simulation model no 2: fi-
nalisation of plant layout and default control strategy, Water Sci. Technol. 62 (9)
(2010) 1967–1974.
[28] D. Ferguson, V. Catterson, Big data techniques for wind turbine condition mon-
itoring, European Wind Energy Association Annual Event (EWEA 2014), 2014.
[29] A.A. Jiménez, C.Q. Gómez, F.P.G. Márquez, Concentrated solar plants manage-
ment: Big data and neural network, Renewable Energies, Springer, 2018, pp.
63–81.
[30] Y. Zhao, P. Liu, Z. Wang, L. Zhang, J. Hong, Fault and defect diagnosis of battery
for electric vehicles based on big data analysis methods, Appl. Energy 207 (2017)
354–362.
[31] I.G. Alonso, O.Á. Fres, A.A. Fernández, P.G. del Torno, J.M. Maestre,
M.A.G. Fuente, Towards a new open communication standard between homes and
service robots, the dhcompliant case, Robot. Auton. Syst. 60 (6) (2012) 889–900.
[32] H. Qu, P. Ling, L. Wu, Electricity consumption analysis and applications based on
smart grid big data, Ubiquitous Intelligence and Computing and 2015 IEEE 12th
Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on
Scalable Computing and Communications and Its Associated Workshops (UIC-
ATC-ScalCom), 2015 IEEE 12th Intl Conf on, IEEE, 2015, pp. 923–928.
[33] K. Zhou, C. Fu, S. Yang, Big data driven smart energy management: from big data
to big insights, Renew. Sustain. Energy Rev. 56 (2016) 215–225.
[34] T. Dunne, Big data, analytics, and energy consumption, 2012.
[35] A. Nasiakou, M. Alamaniotis, L.H. Tsoukala, Power distribution network parti-
tioning in big data environment using k-means and fuzzy logic, 2016.
[36] E. Aarseth, Electric vehicle service center and method for exchanging and charging
vehicle batteries, Dec. 7 1999, US Patent 5,998,963.
[37] S.E. Haupt, B. Kosovic, Big data and machine learning for applied weather fore-
casts: forecasting solar power for utility operations, Computational Intelligence,
2015 IEEE Symposium Series on, IEEE, 2015, pp. 496–501.
[38] Nasa, https://www.nasa.gov/ (accessed: 2018-11-16).
[39] Irimo, http://irimo.ir/far/eng (accessed: 2018-11-16).
[40] H. Jiang, Y. Li, Y. Zhang, J.J. Zhang, D.W. Gao, E. Muljadi, Y. Gu, Big data-based
approach to detect, locate, and enhance the stability of an unplanned microgrid
islanding, J Energy Eng 143 (5) (2017) 04017045.
[41] B. Yang, J. Yamazaki, N. Saito, Y. Kokai, D. Xie, Big data analytic empowered grid
applications-is pmu a big data issue? European Energy Market (EEM), 2015 12th
International Conference on the, IEEE, 2015, pp. 1–4.
[42] M.J. Prieto, A.M. Pernía, F. Nuño, J. Díaz, P.J. Villegas, Development of a wireless
sensor network for individual monitoring of panels in a photovoltaic plant, Sensors
14 (2) (2014) 2379–2396.
[43] T. Hu, M. Zheng, J. Tan, L. Zhu, W. Miao, Intelligent photovoltaic monitoring
based on solar irradiance big data and wireless sensor networks, Ad Hoc Netw. 35
(2015) 127–136.
[44] P. Kinney et al., Zigbee technology: wireless control that simply works, in:
Communications design conference, vol. 2, 2003, pp. 1–7.
[45] M. Rodriguez, I. González, E. Zalama, Identification of electrical devices applying
big data and machine learning techniques to power consumption data,
International Technology Robotics Applications, Springer, 2014, pp. 37–46.
[46] M. Chen, S. Mao, Y. Liu, Big data: a survey, Mob. Netw. Appl. 19 (2) (2014)
171–209.
[47] I. Abdallah, V. Dertimanis, H. Mylonas, K. Tatsis, E. Chatzi, N. Dervilis, K. Worden,
E. Maguire, Fault diagnosis of wind turbine structures using decision tree learning
algorithms with big data, Safety and Reliability-Safe Societies in a Changing
World, CRC Press, 2018, pp. 3053–3061.
[48] S.M. Bhuiyan, J.F. Khan, G.V. Murphy, Big data analysis of the electric power pmu
data from smart grid, SoutheastCon, 2017, IEEE, 2017, pp. 1–5.
[49] P.-C. Chen, T. Dokic, M. Kezunovic, The use of big data for outage management in
distribution systems, International Conference on Electricity Distribution (CIRED)
Workshop, 2014.
[50] Sinogreenergy, https://www.Sinogreenergy.com/ (accessed: 2018-11-16).
[51] Energyplus, https://EnergyPlus.net/ (accessed: 2018-11-16).
[52] Nrel, https://www.nrel.gov/ (accessed: 2018-11-16).
[53] Open data, https://open-enernoc-data.s3.amazonaws.com/anon/index.html (ac-
cessed: 2018-11-16).
[54] Hadoop, https://hadoop.apache.org/ (accessed: 2018-11-16).
[55] T. White, Hadoop: The Definitive Guide, O’Reilly Media, Inc., 2012.
[56] K. Shvachko, H. Kuang, S. Radia, R. Chansler, The hadoop distributed file system,
Mass Storage Systems and Technologies (MSST), 2010 IEEE 26th Symposium on,
IEEE, 2010, pp. 1–10.
[57] J. Dean, S. Ghemawat, Mapreduce: simplified data processing on large clusters,
Commun. ACM 51 (1) (2008) 107–113.
[58] J. Herman, H. Herman, M.J. Mathews, J.C. Vosloo, Using big data for insights into
sustainable energy consumption in industrial and mining sectors, J. Clean. Prod.
197 (2018) 1352–1364.
[59] H. Daki, A. El Hannani, A. Aqqal, A. Haidine, A. Dahbi, Big data management in
smart grid: concepts, requirements and implementation, J. Big Data 4 (1)
(2017) 13.
[60] Y. Huai, A. Chauhan, A. Gates, G. Hagleitner, E.N. Hanson, O. O’Malley, J. Pandey,
Y. Yuan, R. Lee, X. Zhang, ”Major technical advancements in apache hive,
Proceedings of the 2014 ACM SIGMOD International Conference on Management
of Data, ACM, 2014, pp. 1235–1246.
[61] S. Kamburugamuve, G. Fox, D. Leake, J. Qiu, Survey of apache big data stack,
Indiana University, Tech. Rep., 2013.
11
Advanced Engineering Informatics 42 (2019) 100945
[62] D. Cutting, Data interoperability with apache avro, Cloudera Blog, 2011.
[63] V.K. Vavilapalli, A.C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans,
T. Graves, J. Lowe, H. Shah, S. Seth, et al., Apache hadoop yarn: yet another
resource negotiator, Proceedings of the 4th Annual Symposium on Cloud
Computing, ACM, 2013, p. 5.
[64] J. Tan, The design and implementation of big data platform for telecom operators,
International Conference on Industrial IoT Technologies and Applications,
Springer, 2016, pp. 3–11.
[65] M. Bhandarkar, Mapreduce programming with apache hadoop, Parallel &
Distributed Processing (IPDPS), 2010 IEEE International Symposium on, IEEE,
2010, p. 1.
[66] Apache zeppelin, https://zeppelin.apache.org/ (accessed: 2018-11-16).
[67] A. MadhaviLatha, G.V. Kumar, Streaming data analysis using apache cassandra
and zeppelin, IJISET-Int. J. Innov. Sci. Eng. Technol. 3 (10) (2016).
[68] S. Haloi, Apache ZooKeeper Essentials, Packt Publishing Ltd, 2015.
[69] A. Rabkin, R. Katz, Chukwa: A system for reliable large-scale log collection, in:
Proceedings of LISA’10: 24th Large Installation System Administration
Conference, 2010, p. 163.
[70] Apache samoa, https://samoa.incubator.apache.org/ (accessed: 2018-11-16).
[71] G. Ingersoll, Introducing Apache Mahout, Scalable, Commercial-friendly Machine
Learning for Building Intelligent Applications, IBM, 2009.
[72] L. Cai, X. Guan, P. Chi, L. Chen, J. Luo, Big data visualization collaborative fil-
tering algorithm based on rhadoop, Int. J. Distrib. Sens. Netw. 11 (10) (2015)
271253.
[73] Apachepig, https://pig.apache.org/ (accessed: 2018-11-16).
[74] Apacheflume, https://flume.apache.org/ (accessed: 2018-11-16).
[75] A.H. Team, Apache hbase reference guide, Apache, version, vol. 2, no. 0, 2016.
[76] J. Choi, M. Kim, J. Yoon, Implementation of the big data management system for
demand side energy management, Computer and Information Technology;
Ubiquitous Computing and Communications; Dependable, Autonomic and Secure
Computing; Pervasive Intelligence and Computing (CIT/IUCC/DASC/PICOM),
2015 IEEE International Conference on, IEEE, 2015, pp. 1515–1520.
[77] A. Lakshman, P. Malik, Cassandra: a decentralized structured storage system, ACM
SIGOPS Oper. Syst. Rev. 44 (2) (2010) 35–40.
[78] K. Banker, MongoDB in Action, Manning Publications Co., 2011.
[79] K. Ting, J.J. Cecho, Apache Sqoop Cookbook: Unlocking Hadoop for Your
Relational Database, O’Reilly Media, Inc., 2013.
[80] B. Momjian, PostgreSQL: Introduction and Concepts vol. 192, Addison-Wesley,
New York, 2001.
[81] J. Jung, Sql-on-hadoop with apache tajo, and application case of sk telecom, 2013.
[82] Spark, https://spark.apache.org/ (accessed: 2018-11-16).
[83] M. Zaharia, R.S. Xin, P. Wendell, T. Das, M. Armbrust, A. Dave, X. Meng, J. Rosen,
S. Venkataraman, M.J. Franklin, et al., Apache spark: a unified engine for big data
processing, Commun. ACM 59 (11) (2016) 56–65.
[84] Y. Gu, H. Jiang, Y. Zhang, J.J. Zhang, T. Gao, E. Muljadi, Knowledge discovery for
smart grid operation, control, and situation awareness-a big data visualization
platform, North American Power Symposium (NAPS), 2016, IEEE, 2016, pp. 1–6.
[85] Flink, https://flink.apache.org/ (accessed: 2018-11-16).
[86] P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, K. Tzoumas, Apache
flink: stream and batch processing in a single engine, Bull. IEEE Comput. Soc.
Tech. Committee Data Eng. 36 (4) (2015).
[87] S. Aridhi, E.M. Nguifo, Big graph mining: Frameworks and techniques, Big Data
Res. 6 (2016) 1–10.
[88] Storm, http://storm.apache.org/ (accessed: 2018-11-16).
[89] M.H. Iqbal, T.R. Soomro, Big data analysis: apache storm perspective, Int. J.
Comput. Trends Technol. 19 (1) (2015) 9–14.
[90] F. Xhafa, V. Naranjo, S. Caballé, Processing and analytics of big data streams with
yahoo! s4, Advanced Information Networking and Applications (AINA), 2015 IEEE
29th International Conference on, IEEE, 2015, pp. 263–270.
[91] Streambase, https://www.tibco.com/products/tibco-streambase (accessed: 2018-
11-16).
[92] P. Zadrozny, R. Kodali, Big Data Analytics Using Splunk: Deriving Operational
Intelligence from Social Media, Machine Data, Existing Data Warehouses, and
Other Real-time Streaming Sources, Apress, 2013.
[93] Kafka, https://kafka.apache.org/ (accessed: 2018-11-16).
[94] R. Shyam, B.G. HB, S. Kumar, P. Poornachandran, K. Soman, Apache spark a big
data analytics platform for smart grid, Procedia Technol. 21 (2015) 171–178.
[95] Ingenia company, https://www.ingenia.es/ (accessed: 2018-11-16).
[96] A. Cortés, A. Téllez, M. Gallardo, J. Peralta, Big data technology to exploit climate
information/consumption models and to predict future behaviours, International
Technology Robotics Applications, Springer, 2014, pp. 25–36.
[97] I. González Alonso, M. Rodríguez Fernández, A holistic approach to energy effi-
ciency systems through consumption management and big data analytics, Int. J.
Adv. Softw. 6 (2013).
[98] Ps clementine, http://www.psclementinepro.pl/index_en.html (accessed: 2018-
11-16).
[99] L. Guo, H. Yan, Y. Hao, Y. Chen, Comprehensive analysis and evaluation of big
data for main transformer equipment based on pca and apriority, IOP Conference
Series: Earth and Environmental Science, vol. 108, no. 5, IOP Publishing, 2018, p.
052026.
[100] H2o, https://www.h2o.ai/ (accessed: 2018-11-16).
[101] K. Grolinger, M.A. Capretz, L. Seewald, Energy consumption prediction with big
data: Balancing prediction accuracy and computational resources, Big Data
(BigData Congress), 2016 IEEE International Congress on, IEEE, 2016, pp.
157–164.
[102] M. Ruiguang, W. Haiyan, Z. Quanming, L. Yuan, Technical research on the electric
K. Moharm
power big data platform of smart grid, MATEC Web of Conferences, vol. 139, EDP
Sciences, 2017, p. 00217.
[103] J. Neto, P. de Andrade, J. Vilanueva, F. Santos, Big data analytics of smart grids
using artificial intelligence for the outliers correction at demand measurements,
2018 3rd International Symposium on Instrumentation Systems, Circuits and
Transducers (INSCIT), IEEE, 2018, pp. 1–6.
[104] J. Wang, Y. Chen, R. Hua, P. Wang, J. Fu, A distributed big data storage and data
mining framework for solar-generated electricity quantity forecasting, Photonics
and Optoelectronics Meetings (POEM) 2011: Optoelectronic Devices and
Integration, vol. 8333, International Society for Optics and Photonics, 2012, p.
83330S.
[105] X. Meng, J. Bradley, B. Yavuz, E. Sparks, S. Venkataraman, D. Liu, J. Freeman,
D. Tsai, M. Amde, S. Owen, et al., Mllib: machine learning in apache spark, J.
Mach. Learn. Res. 17 (1) (2016) 1235–1241.
[106] R.S. Xin, J.E. Gonzalez, M.J. Franklin, I. Stoica, Graphx: a resilient distributed
graph system on spark, First International Workshop on Graph Data Management
Experiences and Systems, ACM, 2013, p. 2.
[107] Y. Liu, Y. Guo, Z. Yang, J. Hu, G. Lu, Y. Wang, Power system transmission line
tripping analysis using a big data platform with 3d visualization, Computational
Intelligence (SSCI), 2017 IEEE Symposium Series on, IEEE, 2017, pp. 1–8.
[108] Y.X. Sun, Q.Y. Yan, The application mode of energy big data and its enlightenment
for power grid enterprises, Advanced Materials Research, vol. 1008, Trans Tech
Publ, 2014, pp. 1452–1455.
[109] J. Tian, L. Huang, Big data analysis and simulation of distributed marine green
energy resources grid-connected system, Polish Marit. Res. 24 (s1) (2017)
182–191.
[110] D. Liu, G. Li, R. Fan, G. Guo, Research about big data platform of electrical power
system, International Conference on Industrial IoT Technologies and Applications,
Springer, 2016, pp. 36–43.
[111] Z. Cao, J. Lin, C. Wan, Y. Song, G. Taylor, M. Li, Hadoop-based framework for big
data analysis of synchronised harmonics in active distribution network, IET Gener.
Transm. Distrib. 11 (16) (2017) 3930–3937.
[112] N. Mollaei, S.H. Mousavi, Application of a hadoop-based distributed system for
offline processing of power quality disturbances, Int. J. Power Electron. Drive Syst.
(IJPEDS) 8 (2) (2017) 695–704.
[113] lectricité de france, www.edf.fr (accessed: 2018–11-16).
[114] S.A. Jees, V. Gomathi, Load forecasting for smart grid using non-linear model in
hadoop distributed file system, Cluster Comput. (2018) 1–13.
[115] J. Zhan, J. Huang, L. Niu, X. Peng, D. Deng, S. Cheng, “Study of the key tech-
nologies of electric power big data and its application prospects in smart grid,
Power and Energy Engineering Conference (APPEEC), 2014 IEEE PES Asia-Pacific,
IEEE, 2014, pp. 1–4.
[116] L. Chang, X. Wang, M. Mao, Forecast of schedulable capacity for thermostatically
controlled loads with big data analysis, Power Electronics for Distributed
Generation Systems (PEDG), 2017 IEEE 8th International Symposium on, IEEE,
2017, pp. 1–6.
[117] K.M. Paramkusem, R.S. Aygun, Classifying categories of scada attacks in a big data
framework, Ann. Data Sci. (2018) 1–28.
[118] M. Kezunovic, L. Xie, S. Grijalva, The role of big data in improving power system
operation and protection, Bulk Power System Dynamics and Control-IX
Optimization, Security and Control of the Emerging Power Grid (IREP), 2013 IREP
Symposium, IEEE, 2013, pp. 1–9.
[119] X. Qu, A. Jain, N.N. Rajput, L. Cheng, Y. Zhang, S.P. Ong, M. Brafman, E. Maginn,
L.A. Curtiss, K.A. Persson, The electrolyte genome project: a big data approach in
battery materials discovery, Comput. Mater. Sci. 103 (2015) 56–67.
[120] S. Bryan, M. Guido, D. Ostrowski, N.K. Ahmed, Big data analysis of battery charge
power limit impact on electric vehicle driving range while considering driving
behavior, SAE Technical Paper, Tech. Rep., 2017.
[121] K.J. Chalvatzis, H. Malekpoor, N. Mishra, F. Lettice, S. Choudhary, Sustainable
resource allocation for power generation: the role of big data in enabling inter-
industry architectural innovation, Technol. Forecast. Soc. Chang. (2018).
[122] Y.-F. Huang, S.-H. Chang, Mining optimum models of generating solar power
based on big data analysis, Sol. Energy 155 (2017) 224–232.
[123] C.-M. Wang, Y.-F. Huang, Self-adaptive harmony search algorithm for optimiza-
tion, Expert Syst. Appl. 37 (4) (2010) 2826–2837.
[124] R.F.-W.-U.B.U. Germany, Bigdataocean project profile, Framework 6 (2017) 1.
[125] Bigdataocean project, http://www.bigdataocean.eu/site/ (accessed: 2018-11-16).
[126] P. Ifaei, A. Farid, C. Yoo, An optimal renewable energy management strategy with
and without hydropower using a factor weighted multi-criteria decision making
analysis and nation-wide big data-case study in Iran, Energy 158 (2018) 357–372.
[127] M. Mayilvaganan, M. Sabitha, A cloud-based architecture for big-data analytics in
smart grid: a proposal, Computational Intelligence and Computing Research
(ICCIC), 2013 IEEE International Conference on, IEEE, 2013, pp. 1–4.
[128] A.J. Smola, B. Schölkopf, A tutorial on support vector regression, Stat. Comput. 14
(3) (2004) 199–222.
[129] H. Daki, A. El Hannani, H. Ouahmane, Hbase-based storage system for electrical
consumption forecasting in a moroccan engineering school, Optimization and
Applications (ICOA), 2018 4th International Conference on, IEEE, 2018, pp. 1–6.
[130] Z. Zhou, F. Xiong, B. Huang, C. Xu, R. Jiao, B. Liao, Z. Yin, J. Li, Game-theoretical
energy management for energy internet with big data-based renewable power
forecasting, IEEE Access 5 (2017) 5731–5746.
[131] B. Hu, C. Pang, L. Wang, H. Chu, C. Mao, ”Big data management and application
research in power load forecasting and power transmission and transformation
equipment evaluation, Journal of Physics: Conference Series, vol. 1069, IOP
Publishing, 2018, p. 012084 no. 1.
[132] M. Kiran, P. Murphy, I. Monga, J. Dugan, S.S. Baveja, Lambda architecture for
12
Advanced Engineering Informatics 42 (2019) 100945
cost-effective batch and speed big data processing, Big Data (Big Data), 2015 IEEE
International Conference on, IEEE, 2015, pp. 2785–2792.
[133] P. Zhang, X. Wu, X. Wang, S. Bi, Short-term load forecasting based on big data
technologies, CSEE J. Power Energy Syst. 1 (3) (2015) 59–67.
[134] H. Zhao, Z. Tang, W. Shi, Z. Wang, Study of short-term load forecasting in big data
environment, Control And Decision Conference (CCDC), 2017 29th Chinese, IEEE,
2017, pp. 6673–6678.
[135] J. Zhu, Research on data mining of electric power system based on hadoop cloud
computing platform, Int. J. Comput. Appl. (2017) 1–7.
[136] B. Schmidt, P. Flannery, M. DeSantis, Real-time predictive analytics, big data &
energy market efficiency: key to efficient markets and lower prices for consumers,
Appl. Mech. Mater. 704 (2014).
[137] Y. Guo, Z. Yang, S. Feng, J. Hu, Complex power system status monitoring and
evaluation using big data platform and machine learning algorithms: a review and
a case study, Complexity 2018 (2018).
[138] K.P. Bennett, A. Demiriz, Semi-supervised support vector machines, in: Advances
in Neural Information processing systems, 1999, pp. 368–374.
[139] R. Dutta, R. Barua, Password-based encrypted group key agreement, IJ Netw.
Secur. 3 (1) (2006) 23–34.
[140] K. Chodorow, MongoDB: The Definitive Guide: Powerful and Scalable Data
Storage, O’Reilly Media Inc, 2013.
[141] T.E. Oliphant, Python for scientific computing, Comput. Sci. Eng. 9 (3) (2007).
[142] Python, https://www.python.org/ (accessed: 2018-11-16).
[143] M.R. Abid, R. Lghoul, D. Benhaddou, Ict for renewable energy integration into
smart buildings: Iot and big data approach, AFRICON, 2017 IEEE, IEEE, 2017, pp.
856–861.
[144] K. Pepple, Deploying Openstack, O’Reilly Media, Inc., 2011.
[145] M. Armbrust, A. Fox, R. Griffith, A.D. Joseph, R. Katz, A. Konwinski, G. Lee,
D. Patterson, A. Rabkin, I. Stoica, et al., A view of cloud computing, Commun.
ACM 53 (4) (2010) 50–58.
[146] E. Upton, G. Halfacree, Raspberry Pi User Guide, John Wiley & Sons, 2014.
[147] R.-H. Liu, C.-F. Kuo, C.-T. Yang, S.-T. Chen, J.-C. Liu, On construction of an energy
monitoring service using big data technology for smart campus, 2016 7th
International Conference on Cloud Computing and Big Data (CCBD), IEEE, 2016,
pp. 81–86.
[148] N. Matloff, The Art of R Programming: A Tour of Statistical Software Design, No
Starch Press, 2011.
[149] M. Khan, M. Li, P. Ashton, G. Taylor, J. Liu, Big data analytics on pmu measure-
ments, 2014.
[150] G.P. Alliance, Openpdc, 2011. Available on-line: http://openpdc.codeplex.com.
[151] Q.-H. Huang, J. Huang, X.-Z. Wang, C.-Q. Huang, X.-C. Heng, Big data based
service platform and its typical applications of electric power industries, DEStech
Transactions on Environment, Energy and Earth Sciences, no. epee, 2017.
[152] A. Minghao, G. Xianjun, W. Xiaohui, L. Zhihong, C. Naishi, P. Tianjiao, X. Zhiheng,
Y. Fei, A big data analysis based new method for power grid dispatch and control
training simulation, Electricity Distribution (CICED), 2016 China International
Conference on, IEEE, 2016, pp. 1–4.
[153] J. Mu, Y. Pei, W. Li, J. Xu, C. Guan, J. Dong, H. Zhang, L. Pan, Research on energy
saving optimization strategy of substation operation based on big data technology,
2018 Chinese Control And Decision Conference (CCDC), IEEE, 2018, pp.
3567–3571.
[154] H. Wang, S. Zhao, H. Zhao, Y. Yue, Research on data processing for condition
monitoring of wind turbine based on hadoop platform, Mechatronics and
Automation (ICMA), 2017 IEEE International Conference on, IEEE, 2017, pp.
322–326.
[155] E.G. Nabati, K.D. Thoben, Big data analytics in the maintenance of off-shore wind
turbines: a study on data characteristics, Dynamics in Logistics, Springer, 2017,
pp. 131–140.
[156] Q. Yuan, K. Zhou, W. Lu, J. Yao, Big data driven optimal sizing of stand-alone
photovoltaic energy systems, 2018 13th IEEE Conference on Industrial Electronics
and Applications (ICIEA), IEEE, 2018, pp. 679–684.
[157] W. Hu, L. Zheng, X. Liu, P. Zhang, X. Xu, C. Wang, “Power grid’s intelligent sta-
bility analysis based on big data technology, Power and Energy Engineering
Conference (APPEEC), 2016 IEEE PES Asia-Pacific, IEEE, 2016, pp. 623–627.
[158] J.W. Kim, An automatic detection of anomalous energy consumption by lever-
aging bems big data analytics, Int. J. Appl. Eng. Res. 12 (14) (2017) 4345–4349.
[159] H. Zhiwei, G. Tian, Z. Huaving, H. Xu, C. Junwei, H. Ziheng, Y. Senjing,
Z. Zhengguo, Transient power quality assessment based on big data analysis,
Electricity Distribution (CICED), 2014 China International Conference on, IEEE,
2014, pp. 1308–1312.
[160] A.Y. Ng, M.I. Jordan, On discriminative vs. generative classifiers: a comparison of
logistic regression and naive bayes, Adv. Neural Inform. Process. Syst. (2002)
841–848.
[161] J.J. Grainger, W.D. Stevenson, Power System Analysis vol. 67, McGraw-Hill, New
York, 1994.
[162] D. Šutić, E. Varga, Apache spark as distributed middleware for power system
analysis, Telecommunication Forum (TELFOR), 2017 25th, IEEE, 2017, pp. 1–4.
[163] R.C. Yuan, H. Yan, X.M. Zhou, F.C. Di, L.X. Li, Application and architecture of
power dispatching & distribution system using big data technology, Advanced
Materials Research, vol. 1070, Trans Tech Publ, 2015, pp. 1425–1429.
[164] J. Yang, J. Zhao, F. Luo, F. Wen, Z.Y. Dong, Decision-making for electricity re-
tailers: a brief survey, IEEE Trans. Smart Grid 9 (5) (2018) 4140–4153.
[165] M.R. Bashir, A.Q. Gill, ”Towards an iot big data analytics framework: smart
buildings systems, High Performance Computing and Communications; IEEE 14th
International Conference on Smart City; IEEE 2nd International Conference on
Data Science and Systems (HPCC/SmartCity/DSS), 2016 IEEE 18th International
K. Moharm
Conference on, IEEE, 2016, pp. 1325–1332.
[166] J. Postel, Transmission control protocol, Tech. Rep., 1981.
[167] I. Kovačević, A. Erdeljan, S. Vukmirović, N. Dalčeković, J. Stankovski, Combining
real-time processing streams to enable demand response in smart grids, Networks,
Computers and Communications (ISNCC), 2017 International Symposium on,
IEEE, 2017, pp. 1–6.
[168] S. Yu, W. Maomao, M. Lin, Design and realization of the smart grid marketing
system architecture based on hadoop, Control Engineering and Communication
Technology (ICCECT), 2012 International Conference on, IEEE, 2012, pp.
500–503.
[169] M. Liu, M. Wang, J. Wang, D. Li, Comparison of random forest, support vector
machine and back propagation neural network for electronic tongue data classi-
fication: application to the recognition of orange beverage and chinese vinegar,
Sensors Actuat. B: Chem. 177 (2013) 970–980.
[170] Z. Zhao, Q. Ma, A real-time processing system for massive traffic sensor data,
Connected Vehicles and Expo (ICCVE), 2012 International Conference on, IEEE,
2012, pp. 142–147.
[171] C. Tu, X. He, Z. Shuai, F. Jiang, Big data issues in smart grid–a review, Renew.
Sustain. Energy Rev. 79 (2017) 1099–1107.
[172] V. Potdar, A. Chandan, S. Batool, N. Patel, Big energy data management for smart
grids-issues, challenges and recent developments, Smart Cities, Springer, 2018, pp.
177–205.
[173] J. Hu, A.V. Vasilakos, Energy big data analytics and security: challenges and op-
portunities, IEEE Trans. Smart Grid 7 (5) (2016) 2423–2436.
[174] T.-H. Dang-Ha, R. Olsson, H. Wang, The role of big data on smart grid transition,
2015 IEEE International Conference on Smart City/SocialCom/SustainCom
(SmartCity), IEEE, 2015, pp. 33–39.
[175] P.D. Diamantoulakis, V.M. Kapinas, G.K. Karagiannidis, Big data analytics for
13
Advanced Engineering Informatics 42 (2019) 100945
dynamic energy management in smart grids, Big Data Res. 2 (3) (2015) 94–101.
[176] B. Wang, S. Huang, J. Qiu, Y. Liu, G. Wang, Parallel online sequential extreme
learning machine based on mapreduce, Neurocomputing 149 (2015) 224–232.
[177] C. Zhu, H. Zhou, V.C. Leung, K. Wang, Y. Zhang, L.T. Yang, Toward big data in
green city, IEEE Commun. Mag. 55 (11) (2017) 14–18.
[178] L. Lu, H. Dong, C. Yang, L. Wan, A novel mass data processing framework based on
hadoop for electrical power monitoring system, Power and Energy Engineering
Conference (APPEEC), 2012 Asia-Pacific, IEEE, 2012, pp. 1–4.
[179] N. Chen, C. Wang, P. Han, J. Zhang, K. Wang, E. Dai, W. Kang, F. Yang, B. Sun,
G. Guo, ”Research about solutions to the bottleneck of big data processing in
power system, International Conference on Industrial IoT Technologies and
Applications, Springer, 2016, pp. 44–51.
[180] A. Ghosal, S. Halder, Building intelligent systems for smart cities: Issues, chal-
lenges and approaches, Smart Cities, Springer, 2018, pp. 107–125.
[181] H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F.B. Cetin, S. Babu, Starfish: a
self-tuning system for big data analytics, Cidr 11 (2011) (2011) 261–272.
[182] M. Khan, Z. Huang, M. Li, G.A. Taylor, P.M. Ashton, M. Khan, Optimizing hadoop
performance for big data analytics in smart grid, Math. Problems Eng. 2017
(2017).
[183] C.S. Lai, L.L. Lai, Application of big data in smart grid, Systems, Man, and
Cybernetics (SMC), 2015 IEEE International Conference on, IEEE, 2015, pp.
665–670.
[184] N. Yu, S. Shah, R. Johnson, R. Sherick, M. Hong, K. Loparo, Big data analytics in
power distribution systems, Innovative Smart Grid Technologies Conference
(ISGT), 2015 IEEE Power & Energy Society, IEEE, 2015, pp. 1–5.
[185] S. Rusitschka, E. Curry, Big data in the energy and transport sectors, New Horizons
for a Data-Driven Economy, Springer, 2016, pp. 225–244.