Big Data (Analytics) in power systems

By: Fatima Hassan

Email: F_995@outlook.com

1|P age
Abstract:
Power system has faced different challenges among the years, technologies and innovations
within last few years helped with growing the whole system and improving the methodology of
each concept. Starting from producing large amount of data which called concept of Big Data to
how to manage storing those data’s and the availability in real-time needed for further
operations within electrical power system.

2|P age
Table of Contents

Abstract:................................................................................................................................................................ 2

I. Introduction: ................................................................................................................................................. 4

II. Big data: ........................................................................................................................................................ 5

What is big data: ........................................................................................................................................... 5

Characteristics: ............................................................................................................................................. 5

Tools and types: ............................................................................................................................................ 6

Big Data Analytics: ........................................................................................................................................ 8

Importance of big data: ................................................................................................................................ 8

The future: .................................................................................................................................................. 10

III. Power System ......................................................................................................................................... 11

IV. Big Data (Analytics) in power system: .................................................................................................... 12

Big data technologies for smart grid:.......................................................................................................... 12

Big data Analytics in smart grid: ................................................................................................................. 15

Future issues and challenges: ..................................................................................................................... 17

V. Conclusion:.................................................................................................................................................. 19

References .......................................................................................................................................................... 20

3|P age
 I. Introduction:
“Big data is not a fad. We are just at the beginning of a revolution that will touch every business
and every life on this planet.” [1]

Due to the rapid growth of the world and daily new innovations, keeping up is mandatory to
ensure reserving the data for generations to come, this is how the concept of big data or -
information explosion as Oxford dictionary- came.

With all the details around us, unseen data transferred between smart phones, cars, even
houses, and human interactions itself. Requiring complicated analysis and massive storage,
where the collected data are on exponential growth due to the world development. All the new
methods required for this amount and volume of data is a new challenge to face it. But a new
opportunity as well to this century.

The great progress for IT team with collecting data, provides new vision and future for
engineers to ensure the ability of growing the capacity of data receiving.

Figure 1 - A day of data [2]

4|P age
 II. Big data:
What is big data:
To describe what is it exactly, think of one single smartphone user how many data can be
generate of (texting, phone calls, emails, photos, videos, ...etc.) approximately 50 exabytes.
Now imagine this number is multiplied by 5 billion smartphone users, that is a lot to process
only by smartphones, think of machines and mother nature that process data as well. This
amount of data is quite large to process by one normal computer. so, this giant amount of data
we refer to it by big data.

It is a technology evolution, to ensure that each user got the right information at the right
needed time from all the available data that keep growing from a long time till now.

The challenge is not how to deal with big massive data, but how to manage and control the
diverse data with diverse information in addition to real time access.

Characteristics:
As big data refer to massive amount representation of data set (Volume), the speed of
generating this information (Velocity), and more branched out to include structure, semi
structure, and unstructured (Variety).

It is most commonly based on 3V’s model as per analysts at Gartner [3]

- Volume:
The amount of data generated and represented
each day, and it increased exponentially by the
information analytics.
- Velocity:
Represent where data is captured, shared and
generated in the real time.
- Variety:
Diversity of the type and source of data
managed by system and this leads to sort the
data in structured manner and links the
relationship of each.
Figure 2 - The 3 V's of big data

5|P age
Due to importance of the concept how crucial has been added to 3 fundamental V’s, and it
might increase more depend on the need to each company, group, …etc.

- Validity or Veracity:
Is to assure of data quality, and authenticity. As working with huge amount of data
results kind of less quality.
- Value:
Adding value to each user. Establish each data platform from company it might be
without real value.

Figure 3 - 5 V's of big data

Tools and types:

As big data requires a lot of information on daily bases to grow as well big data tools always
improving and developing. Tools such as (Hadoop, Cassandra, Spark, Kafka, Hive, Pig, …etc.).
uses based on organization’s requirements to provide many different solutions.

Most popular and established one is Apache Hadoop, which is an open source framework for
saving and processing / analyzing data.

Another one is Apache Spark, which can store a big part of data processing in a memory and
disk. Hence it is much faster. Other advantage it can run on one single local machine.

In addition, it can work with Hadoop by (Hadoop Distributed File System).

6|P age
One more is Apache Kafka, that allows to publish and subscribe to real time data. This help with
bringing reliability to messaging system.

Figure 4 - Big Data tools

There are three types of Big Data, Structure, Semi Structure, and Un Structure:

- Structured data:
Its fixed format and handled by machines. Consists of information already managed by
the organization in database.
- Un-structured data:
Is unorganized information, no specific format. Can be gathered from anywhere such as
data from social media sources.
- Semi-structured data:
Contain both the forms of data. As an example sensor’s data entered by the developer
and web server logs.

7|P age
Big Data Analytics:
Big Data Analytic refers to collecting, organizing, and analyzing of different information to
achieve the purpose. Mainly focusing on solving new problems or old problems in a new better
way. Here are types of Big Data Analytics:

- Descriptive Analytics
(What is happening?), first stage of data analytics that creates a history for a data. Help
to uncover pattens that offer perception and provides prospects and trends.
- Diagnostic Analytics:
(Why did it happen?), it looks and search for the root cause of the problem. This uses to
identify and understand the cause of events and behaviors.
- Predictive Analytics:
(What is likely to happen?), uses many techniques like artificial intelligence and data
mining to investigate and make scenarios of what will happen.
- Prescriptive Analytics:
(What should be done?), provide historical data and predictive analytics to find the right
action and the best solution to take.

Importance of big data:

While the concept of big data is new, the act of collecting and storing massive amount of
information is ages ago. Its not the amount of stored data what is important. It what’s
companies might do with that data that matters.

Big data can help the organization to come up and create a whole new growth. Each
organization uses data in its own way, the more efficiently uses data the possibility to achieve
and growth is high.

The ability to operate big data in an efficient way brings many benefits for different sectors
such as, health, education, industry, and much more. How exactly it is important?

- Understand market conditions:

By analyzing big data company can get better understanding of current market
situation. For example, can improve the outcome by knowing what’s on trend and
produce it more.
- Cost saving:

8|P age
 In business large amount of data to be stored, some kind of big data tools like (cloud-
based analytics, and Hadoop) can help in identifying more efficient way of doing
business and bring coast advantages as well.
- Time reduction:
Due to high speed of big data tools can easily check and find new sources of data and
this led to immediately analyzing data and making new quick discussions based on past
learning.
- Customer care:
In a business wise customer behavior is very important to trigger loyalty, where any
business asset is customers. It allows to observe the various of customer related
patterns and trends.

And much more reasons why big data is important to our life. Here is an example in figure
bellow, a study by Business Application Research Center (BARC) how some of biggest company
of the world utilizing big data analytics.

Figure 5 - BARC Study on utilizing big data [6]

9|P age
The future:
Nowadays everything is heading to next level of development and future to come, controlling
everything with one device such as IoT concept (Internet of Things). Once everything starting to
use IoT the possibility of using big data will be giant. Not only the amount of data that will
increase the analytics techniques will variety as well. Here is a report of global big data market
forecast 2019-2027 -bellow in the figure-.

Figure 6 - Global big data forecast 2019-2027 [7]

10 | P a g e
 III. Power System
Electrical Power System is a network consist of three phases (Generation, Transmission,
Distribution). Uses one kind of energy to convert it into Electrical Energy. Most of electricity
generated in UAE using natural gas. In addition, UAE is developing to achieve strategic
objectives of the Dubai Integrated Energy Strategy 2030 to diversity energy resources and
improve efficiency of electricity and water usage. [4]

Figure 7 - power system structure [8]

11 | P a g e
 IV. Big Data (Analytics) in power system:
Big data technologies for smart grid:

- Data sources:
Varity of data is based on how the values are extracted, as we have Operational Data
related to electrical data of the grid, which represent real and reactive power flows, voltage,
…etc. Non-Operational Data not related to grid power, but it refers to main data, which is
on power quality and reliability, …etc. Meter Usage Data other type of data related to
consumer on power usage and demand values as average usage, peak and time of the day,
…etc. Event Message Data which is related to smart grid devices as fault detection, voltage
loss, …etc. finally Metadata which is related to explain and design all other types of data
from several sources as example sensors, devices, mobile data, substations, …etc.

- Data integration
To ensure data integration, several technologies and approach used in latest
communication technology and advanced operations methods are to improve smart grid
reliability, efficiency and performance. Such as:

12 | P a g e
 ▪ Common Information Models (CIM) is critical specially in failure or success of data
management in energy management systems in term of time, coast, and data
integration. Which helps to exchange data with technical grid infrastructure.
▪ Enterprise Service Bus (EBS) reduces coast and time in terms of monitoring and
management. Which is achieve great approach to manage communication between
different kind of systems as GIS, CIS, OMS, …etc.
▪ Messaging which responsible on communication systems based on exchanging
messages include data and some information.
▪ Service Oriented Architecture (SOA) makes data integration flexible and easier by
using single approach software communicate together. Which solve the problem of
how to maintain such amount of systems provided to the user.

- Data storage
Data storage works as critical role in smart grid, as collecting data from many sources and
delivering it to analytics tools. Storage system need to be developed to meet big data
requirements.

▪ Distributed File System (DFS): permits multiple operators on multiple devices to

share files and storage resources. And allows every user to get a copy of stored data.
as an example, Google GFS, HDFS, …etc.
▪ NoSQl databases: which is the new approach to overcome the limitations of SQL
databases. Its represent three styles: 1. Key-value solutions as Dynamo and

13 | P a g e
 Voldermort, 2. Column-oriented solutions as Cassandra and HBase, 3. Documents
databases solutions as MongoDB and CouchDB.

- Data analytics
Smart grid does collect data from many sources and stored it in huge quantity of dataset
that should be easier for analyzing. It is essential role to make the grid more efficient and
intelligent:

▪ Single analytics: which is based on single processing.

▪ Event analytics: focused of events.
▪ State analytics: provide a vision about the state of the grid.
▪ Engineering operation analytics: manage grid’s operating side.
▪ Customer analytics: process customer data.
Several models can combine the previous analytics such as: 1. Descriptive. 2. Diagnostic. 3.
Predictive. And 4. Prescriptive. As explained before, but from power system perspective where
each model describes an operation side of the grid.

Descriptive model is used in describe customers behaviors. Diagnostic model understands their
behavior and analyze it. Predictive model is to predict customers decision in the future. Finally,
prescriptive model high level of analytics in smart grid to affect marketing and decision making.

14 | P a g e
Two ways to process big data, first is batch processing which process data without high
requirements on time. Second, is stream processing which is used in real-time applications.

- Data visualization
Based on different high dimension visualization, 2D and 3D is used by the system. But due
to massive amount of data required data presenting such as 3D power map, scatter
diagram, …etc.

- Data transmission
Due to importance of data transmission’s role, maintaining is required for high bandwidth
capacity, speed, data security and privacy, …etc.

To describe data transmission it is basically communication system, starting by access

network technologies containing PLC, ZigBee, WIFI…, then area network technologies IP,
IMPLS …, at the end backbone network technology which focus on fiber optics, microwave
link, …etc.

Big data Analytics in smart grid:

▪ Distribution Automation (DA):
Mainly focused on operation and system reliability at the distribution level. A successful
DA has the capability to define and isolate the faults in distribution system with
reducing time and improving customer happiness. Where due to the concept of DA a
massive data has been gathered from SCADA or AMI. As an example to avoid or reduce
the outages with the data related to Pole Mounted-Auto Recloser (PMAR), it is
protection device installed on the OHL Over Head Lines.

15 | P a g e
 ▪ Electric device state monitoring:
A single failure in power transformers may cause huge outage in power system.
Therefore, management of life cycle of power transformer is extremely important.
Where the existing methods focused on limited state parameters, while the potential
risk problem and health condition can be predicted with the help of Proportional Hazard
Model (PHM) which developed to process and classify lifecycle data.

▪ Power quality monitoring

It refers to magnitude, frequency, voltage, and currant in power system and extremely
related to safe operation of power grid and customers satisfaction. Deep learning
successfully helped with classification of power quality timeline. The image file of three
phase are processed for classification by deep learning techniques.

▪ Renewable energy forecasting

Environmentally friendly energy such renewable energy source as wind and
photovoltaic energies are next generation’s source of power grid. However, the
discontinuous is always considered as an obstacle for large utilization. To deal and
maintaining such challenge neural network is used. Available information in historical
records are used to classify the days into categories. Then neural network is trained to

16 | P a g e
 get the forecasting results data of wind energy. Later by using vector regression method
to predict the wind speed and timeline.

▪ Non-technical loss detection

According to the published survey by Northeast Group. LLC. Approximately the lost
caused by electricity theft reached more than $89.3 billion in the world every year.
Therefore, an approach by (decision trees) DT and (Support Vector Machines) SVM is
proposed on 2016. DT trained with heavy data collection from number of users,
appliances, study weather conditions to expect the electricity consumption by customer
within a specific time. Then the computed consumption is fed to SVM which is already
has been trained with historical data to determine whether customer behavior is normal
or fraud.

Future issues and challenges:

▪ Collaboration
A huge field deals with different branches and on continuous based needs several
professionals to collaborate with each, it is important to issue an extensive big data
infrastructure with specialist that have accessibility to arrange and distribute
information.

▪ Cyber-Security
Is a main issue facing this generation to deal with security issues as availability, privacy,
integrity, auditability, authentication, authorization, confidentiality, nondeducibility,
…etc.

Cyber-security plays significant role, where there is a lack of appropriate IoT

infrastructure to protect confidential data of several components against the threats.

▪ Big data management

The development in storage and analyzing system is never decreases with high level of
entered data, till here developing and improving data evaluation principle is highly
required to select and store the valuable data and neglect the others.

17 | P a g e
 ▪ Real-time big data intelligence
Massive amount of available data in operation makes the process for such data is
challenging as well while the real time operations/responses of monitoring and
analyzing real time big data energy demand are required.
▪ Data quality
Due to massive data resources available, databases include data with all the
characteristics as incompleteness, inconsistency, and inaccuracy.

V. Case study:
17 years ago, 2003, Aug 14th a huge blackout hit Northeast America, which caused more
than 40 million people in 8 state and 10 million people in Canada to lose their electricity up
to days.

After investigation found that software bug in alarm’s system of Ohio, which collapsed and
failed to redirect power from an overloaded power line. Then the wire got heated and
dropped down into a tree near to Cleveland, which tripped a circuit and caused power to
redirect to other line. Hence it got overloaded and set off a line of failures that resulted a
huge blackout in history.

Later more than decade, electrical grid still having some kind of failures, but due to new
data monitoring system it has the potential to transform the grid by providing real-time
data and solving any issues caused by the weather.

Which is the concept of smart grid, a system connected to a bunch of sensors to secure
two-way communications and analytics.

Self-Healing smart grid:

Like a human body, smart grid can be thought of self-healing. It has the huge ability to
identify and solve problems.

As GTM Research global utility data analytics market of 20$ billion between 2013-2020. The
investment includes sensors, hence US has installed over 1000 sensors all over the country,
funded by the Recovery Act Smart Grid Investments.

18 | P a g e
 In 2012 during Hurricane, having PMUs (Phasor Measurement Units) installed all over, did
reduced the storm’s impact. Where the physical damage was done, but the sensor did stop
it before spreading to other near places, to prevent repeating the history of 2003.

Smart grid does not only monitor the grid in real-time it also reducing theft energy and
knowing when to rely on renewable energy, which accounted for 13% of US electricity in
2014. It also will be able to integrate with smart building and smart home technologies.

In 2009, US utilities had 194 Petabytes of stored data, to make it clear if we will compare it
to entire digital collection library of Congress its just 3 Petabytes.

The question is how much of all stored data you really used? and with all that massive
amount of data security is essential. However, if the stored data is cleaned it will remain
secure. [5]

VI. Conclusion:
The future approach for big data is increasing wisely, starting from monitoring to analyzing then
will start to act smartly and make decisions.

It can be applied by combining all knowledges from Artificial intelligence to IoT and of-course
concept of big data we can head to next step. Letting the machines apply self-learning concept
and start making decisions for further steps. In addition, predicting to predicting the future will
prevent faults and increase efficiency. This is how the future journey begins from yesterday.

19 | P a g e
References

[1] B. Marr, "Forbes," 30 SEP 2015. [Online]. Available:

https://www.forbes.com/sites/bernardmarr/2015/09/30/big-data-20-mind-boggling-facts-everyone-
must-read/#1353ee2117b1.

[2] J. Desjardins, 17 Apr 2019. [Online]. Available: https://www.weforum.org/agenda/2019/04/how-much-

data-is-generated-each-day-cf4bddf29f/.

[3] "Gartner Glossary," [Online]. Available: https://www.gartner.com/en/information-

technology/glossary/big-data.

[4] DEWA, "Fact sheet DEWA," [Online]. Available:

https://www.dewa.gov.ae/~/media/Files/Consultants%20and%20Contractors/Press%20office/DEWA_F
ACT.ashx.

[5] T. Lewis, "Business insider," Oct 2015. [Online]. Available: https://www.businessinsider.com/smart-

electrical-grid-with-big-data-2015-10.

[6] S. Durcevic, 13 SEP 2018. [Online]. Available: https://www.datapine.com/blog/benefits-of-business-

intelligence-and-business-analytics/.

[7] [Online]. Available: https://www.inkwoodresearch.com/reports/big-data-market/.

[8] "circuit globe," [Online]. Available: https://circuitglobe.com/power-system.html.

[9] Yuanjun Gueo, Kang Li, Wenxiong Mo, "IEEE Explore," 02 SEP 2016. [Online]. Available:
https://ieeexplore-ieee-org.ezproxy.rit.edu/stamp/stamp.jsp?tp=&arnumber=7737581.

[10 N. A. a. J. M. Pina, "Big Data in Power System," Portugal.

]

[11 R. A. a. Y. Zhou, "scinse direct," [Online]. Available:

] https://www.sciencedirect.com/book/9780128119686/big-data-application-in-power-systems#book-
info.

20 | P a g e