Comparative Study of PMSM Control Using

Reinforcement Learning and PID Control.

Adil Najem1 , Ahmed Moutabir2, Mohamed Rafik3, Abderrahmane Ouchatti4
124
G.E.I.T.I.I.L laboratory, Hassan 2nd University, Aïn chok Faculty of Sciences Casablanca, Morocco
3
ENSET Mohammedia, EEIS Laboratory, Hassan II University of Casablanca, Morocco
1
adilnajem@gmail.com ,2 a.moutabir65@gmail.com, 3rafik@enset-media.ac.ma, 4ouchatti_a@yahoo.fr

Abstract— The use of reinforcement learning for process control
does not require knowledge of its mathematical model. This paper
focuses on the control of a permanent magnet synchronous motor
(PMSM) based on the Field Oriented Control strategy (FOC). The
objective is to compare the performances of the classical PID
control with that using reinforcement learning (RL). The RL
algorithm used is Double Delay Deterministic Policy Gradient
(TD3).
First, the general principle of vector control of a PMSM motor is
described. Then, the control using reinforcement learning is
analyzed and compared to PID control. The performances to be
compared are accuracy, dynamic response and the ability to
control torque and speed.
Finally, the simulation models have been developed and tested in
the MATLAB / SIMULINK. Simulated results are displayed to
validate the effectiveness of the proposed strategies.
Keywords— PMSM, FOC, reinforcement learning, TD3,
Simulink.
I. INTRODUCTION
Permanent magnet synchronous motors (PMSM) are widely
used in electric drive systems. They have the advantages of
being simple in structure, efficient and easy to maintain.
Despite all these advantages and because of their non-linear
behavior due to motor and load dynamics, their control is
difficult [1-2].
Many strategies have been developed to control and regulate
the parameters of the PMSM.
Unlike classical approaches, reinforcement learning does
not need to define and know the mathematical model of the
system to be controlled, it simply sends actions to the controlled
environment to maximize rewards.
There are several modern RL algorithms. We can mention:
Deep Q learning (DQN), Deep Deterministic Policy Gradient
(DDPG), Double Delay Deterministic Policy Gradient (TD3).
For our case, we will use the TD3 algorithm.
The paper is organized as follows: the PMSM model and the
FOC principle are presented in Section II, Section III describes
the reinforcement learning, Section IV contains the PID
controller and Section V presents the methods and results of
numerical simulations of the PMSM control signal correction
based on the FOC strategy using the PID controller and the
reinforcement learning agent. Finally, Section VI presents the
conclusion.
II. PMSM MODELING AND FOC STRATEGY
DESCRIPTION
A. PMSM Model
The diagram in Fig.1 shows the different phases of PMSM
modeling.

In addition to the conventional PI based control method,

artificial intelligence control methods have been applied. In this
context, the use of fuzzy logic [3], artificial neural network [4-
6] and neuro-fuzzy are mentioned [7].

The improvement of the static and dynamic performance of

the PMSM and the insensitivity to parameter uncertainties and Fig. 1. PMSM Model Diagram
disturbances is the objective.

The application of RL reinforcement learning to the control The model of the PMSM is defined by the following
of physical systems is a desired approach, where RL agents electrical (1) and mechanical (2) equations:
train themselves using the input and output signals of the
system to find a suitable control policy before implementation
in a real application.

979-8-3503-9836-6/23/$31.00 ©2023 IEEE

 0 0 0 0
0 0 0 0

0 0
1
0 1

Where:
• is the stator resistor.
• and , and , and are respectively d-q axis
stator voltages, currents and inductances.
• , are respectively the rotor inertia and viscous friction
coefficient.
• is the electrical angular velocity of rotor.
• , , and are respectively the electromagnetic
torque, the load torque, the Flux induced by the
permanent magnets of the rotor in the stator phases and
the mechanic speed.

Fig. 2. FOC schematic diagram

is expressed by the following relationship:
3
"# ∗ ∗ % 3
2 III. REINFORCEMENT LEARNING DESCRIPTION
Where: and are the d-q axis flux.
A. Description
Reinforcement learning is a method for rewarding desired
We have: behaviour and/or punishing undesired behaviour.
& 4 This learning method has been adopted in the field of
5 artificial intelligence to guide unsupervised machine learning
using rewards and punishments [9-10].

The expression of becomes: Fig.3shows the general structure of reinforcement learning

3
"# % 6
2

B. Field Oriented Control Principle

Field Oriented Control (FOC), also known as vector control,

which aims to control and adjust the speed of a PMSM to track
a reference value. FOC provides robust control over all speed
ranges [8].

A schematic diagram of the vector control of a PMSM is

shown in Fig. 2:
Fig. 3. Block diagram for a RL structure
 The agent has two input signals which are Observation and
Reward. It is the main actor in reinforcement learning, it makes
decisions and takes optimal actions under certain circumstances.
It interacts with the environment according to input and output
signals and has a policy that helps it to provide the best action
at an appropriate time.
The role of the learning algorithm is to update the parameters
associated with the policy which is based on maximizing the
cumulative reward.
The reward is the value that will guide the agent in its choice
and show it how good or bad the action sent was. It is therefore
defined as a measure of the success of the output signal Action.
The reward can be a mathematical equation combining
criteria, such as energy efficiency, error between the set point
and the actual value... During the learning process the algorithm
updates the policy so that the agent can maximize the rewards.
The observation characterizes the process by a set of
predefined signals. Observations are the signals measured by
sensors or cameras and which are visible to the agent.
The action is represented by the control quantities of the
controlled process.
The environment represents the real or virtual (simulated) Fig. 5. Reinforcement Learning TD3 Agent block
physical system and is external to the agent.
For the reward function:
B. Structure
• The error speed is defined by
*+ (7)
• The reward of speed:
10 0 . *+ . 0.1
-
,
1
(8)

• The reward of d-q axis currents:

0 0.2*10 (9)

2 0.2*10 (10)
• The reward of action:
3 0.1456,
0
(11)
Finally, the reward is defined by:
, 0 2 3 (12)

IV. PMSM PID CONTROLLER

We start with the use of a PID controller shown in Fig. 6.
The calculation of the coefficients requires the use of the
Fig. 4. MATLAB/SIMULINK Reinforcement Learning TD3
transfer function and the PMSM values given by the
manufacturer.
The algorithm is defined as follows: The Ziegler-Nichols method used in practice does not
require the transfer function. After selecting the setpoint, the
controller is set to proportional action, the other actions are
zero. By gradually increasing the proportional coefficient until
oscillation (pumping) is achieved, the controller coefficients are The simulation is in Matlab/Simulink 2021a to compare the
subsequently adjusted. two approaches RL and Ziegler-Nichols, with a load torque of
2N.m and a reference speed of 750rpm.
This type of adjustment is difficult for our system.

Fig. 7. PMSM speed and his reference variation

Fig. 6. MATLAB/SIMULINK PID Controller Speed
V. CONTROL DESIGN AND SIMULATIONS
There are three operational control objectives:
(i). Regulating the output speed to a desired value
(ii). Guarantee the static and dynamic performances of the
PMSM against parameter uncertainties and
disturbances.
(iii). Ensure the robustness of the system.
Figure 7 shows the velocity results of two approaches RL
and PID. Both speeds follow the same reference with zero
accuracy, but the transient of the two approaches is different,
for the Ziegler-Nichols approach there is an overshoot, and a
response time (time settling) is 71ms, for the reinforcement
learning there is no overshoot, a response time (time settling) is
86ms. The PID control that it gains in response time.
RL after learning, in case of disturbance, it corrects in real
time the system thanks to the agent and takes into account the
. non-linearity and the uncertainties of the engine elements. For
a non-linear system, the calculation of the PID coefficients is
for a given operating point.

The PMSM control system has been simulated by using

Matlab software [11] with the set of parameters values depicted
in Table I:
TABLE I. PMSM PARAMETERS

Fig. 8. q axis current variation

Parameters Values
Stator Resistor 0,58
Inductance 1,4. 1062 9
Inductance 2,8. 1062 9

0,02;<=0
Combined inertia PMSM-Load

0,014 >=?/AB
Combined viscous friction
PMSM-Load

0,2CD
Flux induced by the permanent Fig. 9. d axis current variation
magnets
After training, reinforcement learning has zero accuracy, the
Pole pairs number EF 4 dynamic overshoots of the Id and Iq currents have decreased by
54% compared to the classical PID method.
The reinforcement learning kept the machine decoupled
(Id=0).

Fig. 10. Electromagnetic torque variation
The PID controller has a poor dynamic response, therefore
the electromagnetic torque instability increases.
The RL reduces the dynamic response by 74%, in effect
reducing the torque instability.
Fig. 11. Stator currents variations of PID controller
Fig. 12. Stator currents variations of RL
At the start time, the stator currents in the reinforcement
learning approach are lower compared to the currents in the PID
control case.
VI. CONCLUSION
This paper proposes a comparative study between two
PMSM control methods based on the vector control strategy.
These are the classical PID strategy and a new method based on
artificial intelligence algorithms. TD3 reinforcement learning
was used and simulated and compared with the PID results.
The performance of the PID was improved by using a
reinforcement learning technique. Thus, comparative results are
presented for the case where the reinforcement learning agent
is correctly trained and provides correction signals to be added
to the control signals. Numerical simulations are used to
demonstrate the superiority of the control system using
reinforcement learning.
REFERENCES
[1] B. Wu and M. Narimani «Control of Synchronous Motor Drives,
in High-Power Converters and AC Drives » , Wiley-IEEE Press,
2017, pp.353-391.
[2] S. Sakunthala, R. Kiranmayi, and P. N. Mandadi, « A Review on
Speed Control of Permanent Magnet Synchronous Motor Drive
Using Different Control Techniques » International Conference
on Power,Energy, Control and Transmission Systems
(ICPECTS), Chennai, China , 2018, pp. 97-102.
[3] Y.Zhou and al « Fuzzy Control For Treadmill PMSM Speed
System » IEEE Xplorer September ,14 2020. doi :
10.1109/ICARM49381.2020.9195339.
[4] M, Nicola and al « Sensorless Control of PMSM Using DTC Strategy
Based onMultiple ANN and Load Torque Observer» IEEE Xplorer
August ,14 2020. doi : 10.1109/SIELA49118.2020.9167120.
[5] M, Nicola and al « Sensorless Control of PMSM Using FOC Strategy
Based onMultiple ANN and Load Torque Observer» IEEE Xplorer
August ,14 2020. doi : 10.1109/DAS49615.2020.9108914.
[6] M, Hanke and al « Comparison of Artificial Neural Network and Least
Squares Prediction Models for Finite Control Set Model Predictive
Control of a PMSM » IEEE Xplorer August ,14 2020. doi :
10.1049/icp.2021.1122.
[7] C, Elmas and al, « A neuro-fuzzy controller for speed control of
a permanent magnet synchronous motor drive » Expert Systems
with Applications Volume 34, Issue 1, January 2008, Pages 657-
664 https://doi.org/10.1016/j.eswa.2006.10.002
[8] H. Celik and T. Yigit, “Field-Oriented Control of the PMSM
with 2- DOF PI Controller Tuned by Using PSO,” International
Conference on Artificial Intelligence and Data Processing
(IDAP), Malatya, Turkey, 2018, pp. 1-4.
[9] Z. Song, J. Yang, X. Mei, T. Tao, and M. Xu, « Deep
reinforcement learning for permanent magnet synchronous motor
speed control systems » in Neural Computing and Applications,
vol. 33, September 2020, pp. 5409-5418.
[10] T. Schindler, L. Foss, and A. Dietz, « Comparison of
Reinforcement Learning Algorithms for Speed Ripple Reduction
of Permanent Magnet Synchronous Motor » IKMT 2019 -
Innovative small Drives and Micro-Motor Systems; 12.
ETG/GMM-Symposium, Wuerzburg, Germany, 2019, pp. 1-6.
[11] Reinforcement Learning Toolbox™ User's Guide, Matlab and
Simulink, MathWorks, Natick, MA, USA, 2020.
[12] H. Celik and T. Yigit, “Field-Oriented Control of the PMSM with
2- DOF PI Controller Tuned by Using PSO,” International
Conference on Artificial Intelligence and Data Processing
(IDAP), Malatya, Turkey, 2018, pp. 1-4.