Data-driven control

Fundamentally, a control system consists of a system to be controlled and a controller.

Except in few cases, control system design generally starts with modelling of the system.
Designing a control strategy consists of characterizing the system in terms of linear
equations and then subsequently solving them using different methods. Since most of
the real world systems are non-linear in nature, a lot of research has already been done
on linearization techniques. However, such techniques are limited to specific classes of
non-linear equations. Beside this, it is not always possible to write down a model for the
system (e.g. neuroscience, turbulence, climate, finance etc.). Hence, the controller is
constrained by the knowledge of the system.

Considering the above limitations, the Machine Learning (ML) comes to rescue in
two ways:

1. Characterizing a system for later use with model based control, or

2. Characterizing a control law that interacts with the system (model-free control).

Following ML methods are mainly used in model-free control-

1. Reinforcement Learning (RL)

RL is an important artificial intelligence technique used in many applications such as self-

driving cars, automated robots, online games etc. It is known as partially supervised
method. It works on the framework of Markov decision process (MDP). In RL, an agent
interacts with an environment taking different actions (a) in different states (s). For each
action, the agent is either punished or rewarded. The agent attempts to maximize the
reward and continues to learn from the experiences in this way. Developing a quality
function Q is an important aspect of RL which captures the expected total future reward
an agent in state s can receive by executing a certain action a. With experience, the
agent learns and refines this Q value for making better decisions.

Control related limitations of RL include lack of stability, transparency,

interpretability and vulnerability to rapid and unforeseen changes in the environment.

2. Extremum Seeking Control (ESC)

ESC is an adaptive control strategy. It doesn’t require any model and has guaranteed
convergence and stability under a set of well-defined conditions. ESC can track minima
or maxima of a function despite the local disturbances. In ESC a sinusoidal perturbation
is additively injected into the actuation signal to determine the gradient of the objective
function and subsequently to find out the maxima or minima of the objective function.

3. Genetic Algorithms (GA)

This technique is based on the biological principle of natural selection. It is frequently

used for parameter tuning of a controller. In GA, a population comprised of many system
realizations with different parameter values are propagated to future generations
through a set of genetic rules. Next, a number of realizations with different parameter
values, called individuals, are initialized in a population and their performance is
evaluated and compared on a well defined task. Successful individuals with a lower cost
have a higher probability of being selected to advance to the next generation following
operations such as Replication, Crossover, Mutation etc.

4. Iterative Learning Control (ILC)

This method was initially developed for industrial robots where repetition of a task was
involved. ILC don't require one to know the system equations, it refines and optimizes
the process after each cycle.