Data Processing: 

It is defined as Collection, manipulation, and processing of collected data for

the required use. It is a task of converting data from a given form to a much more usable and
desired form i.e. making it more meaningful and informative. Using Machine Learning algorithms,
mathematical modelling and statistical knowledge, this entire process can be automated. This might
seem to be simple but when it comes to really big organizations like Twitter, Facebook,
Administrative bodies like Parliament, UNESCO and health sector organisations, this entire process
needs to be performed in a very structured manner.

So, the steps to perform are as follows:

Data Cleaning:  Data cleaning is the process of fixing or removing incorrect, corrupted,
incorrectly formatted, duplicate, or incomplete data within a dataset. It is one of the important parts
of machine learning. It plays a significant part in building a model. Data Cleaning is one of those
things that everyone does but no one really talks about. It surely isn’t the fanciest part of machine
learning and at the same time, there aren’t any hidden tricks or secrets to uncover. However, proper
data cleaning can make or break your project.

Steps involved in Data Cleaning –

 
What is data preprocessing?

Data preprocessing, a component of data preparation, describes any type of processing performed on rawdata to

prepare it for another data processing procedure. It has traditionally been an important preliminary step for

the data mining process. More recently, data preprocessing techniques have been adapted for training machine

learning models and AI models and for running inferences against them.

Data preprocessing transforms the data into a format that is more easily and effectively processed in data mining,

machine learning and other data science tasks. The techniques are generally used at the earliest stages of

the machine learning and AI development pipeline to ensure accurate results.

There are several different tools and methods used for preprocessing data, including the following:

 sampling, which selects a representative subset from a large population of data;

 transformation, which manipulates raw data to produce a single input;

 denoising, which removes noise from data;

 imputation, which synthesizes statistically relevant data for missing values;

 normalization, which organizes data for more efficient access; and

 feature extraction, which pulls out a relevant feature subset that is significant in a particular context.

What is Data Integration?

Data integration is the process of bringing data from disparate sources together to provide users with a
unified view. The premise of data integration is to make data more freely available and easier to consume and
process by systems and users. Data integration done right can reduce IT costs, free-up resources,
improve data quality, and foster innovation all without sweeping changes to existing applications or data
structures. And though IT organizations have always had to integrate, the payoff for doing so has potentially
never been as great as it is right now.

Companies with mature data integration capabilities have significant advantages over their competition, which
includes:

 Increased operational efficiency by reducing the need to manually transform and combine data sets
 Better data quality through automated data transformations that apply business rules to data
 More valuable insight development through a holistic view of data that can be more easily analyzed
DATA REDUCTION:-
Data reduction is a technique used in data mining to reduce the size of a dataset while still preserving the
most important information. This can be beneficial in situations where the dataset is too large to be processed
efficiently, or where the dataset contains a large amount of irrelevant or redundant information.

There are several different data reduction techniques that can be used in data mining, including:

1. Data Sampling: This technique involves selecting a subset of the data to work with, rather than
using the entire dataset. This can be useful for reducing the size of a dataset while still
preserving the overall trends and patterns in the data.
2. Dimensionality Reduction: This technique involves reducing the number of features in the
dataset, either by removing features that are not relevant or by combining multiple features into a
single feature.
3. Data Compression: This technique involves using techniques such as lossy or lossless
compression to reduce the size of a dataset.
4. Data Discretization: This technique involves converting continuous data into discrete data by
partitioning the range of possible values into intervals or bins.
5. Feature Selection: This technique involves selecting a subset of features from the dataset that
are most relevant to the task at hand.
6. It’s important to note that data reduction can have a trade-off between the accuracy and the size
of the data. The more data is reduced, the less accurate the model will be and the less
generalizable it will be.

DATA DECENTRALIZATION:-
Decentralization is known as the distribution of functions among several units. It is an interconnected
system where no single entity has complete authority. It is the architecture in which the workloads, both
hardware, and software, are distributed among several workstations.
The functions are distributed among several machines in a decentralized system instead of relying on a single
server. They have multiple central owners. The owners can store the resources so that each user can have
access. The system can be imagined in a graphical manner. Each user’s machine can be visualized as nodes
that are connected to one another. Each node has a copy of another node’s data and the multiple owners
have copies of all the nodes as well so as to reduce the access time. So whenever an update or change is
made in a node’s data, the changes are reflected in the copies as well. Let us illustrate with the help of
examples: Bitcoin is the latest technology and a prime example of a decentralized system. It is a blockchain
where no central authority exists. Anyone and everyone can become a part of the network get involved in
transactions and take part in voting. The decision is taken on the majority of votes. Dogecoin is a
cryptocurrency that is decentralized and peer-to-peer that allows us to do transactions. 
We must have a clear concept of decentralization, centralization, and distributed networks. In a centralized
network, there is a central network authority who takes the decisions. In a decentralized system, there are
multiple owners. Distributed systems are a further extension of decentralization. Here there is no concept of
owners. All users are owners and all have equal rights.
Importance of Decentralization
Decentralization is very important because of the following reasons:

 Optimization of Resources: Each user does not have to have all resources. The decentralized
setup allows the user to share his burden with others at a lower level.
 Greater output: Since all users have the same authority, therefore each and every user work with
greater efficiency so as to enhance the maximum productivity.
 Flexibility: Users can share their own views as there are no restrictions imposed by any central
authority. They also have the flexibility to change their decisions.