Q-Learning based quadcopter control

Introduction:

Self-driving vehicles, including yourself a more sophisticated system will allow you to feel the
influence of the environment, and a route plan to your destination, as well as steering wheel
control and speed, has been rapidly developing over the past few years. There are hundreds of
companies around the world that are well advanced to be that autonomous vehicle technology is
ready in 2020, particularly Toyota, Google and Tesla. At the moment, the obstacle to the
academic world and industry wanting to prepare and test an intelligent control algorithm is the
massive cost of a full-scale vehicle, not to mention the cost of building a test site for safety,
environmental monitoring, and testing a self-driving vehicle. For example, the University of
Michigan will spend $ 10 million to develop an entire 32-acre mock-up of the city of Mcity in
order to serve as a plasterer for its intelligent vehicle. The solution to this problem would be to
develop a small vehicle that can be used as a test platform for intelligent transport technologies,
without the costs and risks associated with full vehicle maintenance. Photo. 1 shows a small-
sized and full-scale vehicle, a platform for developing and testing an intelligent control
algorithm.

For tracking and adaptive cruise control, intelligent vehicles and proportional integration
differentiation (PID) controls the control loop mechanism, using feedback that is widely used in
industrial control systems and various other applications that require constant signal reception. It
has been a classic controller type since the mid-20th century, and it will continue to be most
commonly used in industrial control systems, due to its amazing efficiency, ease of
implementation, and wide application. However, it is very difficult to adjust the processing
parameters of a conventional PID controller, and the results achieved in severe conditions require
adequate processing, which can lead to a large waste of labor, material resources and equipment.
Several other CLOSED options, including expert PID control and fuzzy PID control and neural
network PID control, etc., Although they are said to work better, wide-open eyes, good, rich
expert knowledge, expert, full path name, please table and fuzzy control solutions for fuzzy PID
control and fine-tuning neural network parameters neural network PID-need to be controlled, and
this may interfere with the widely used self-management, vehicles, detected and adaptive cruise-
control. However, in complex high-order systems, large teams and strong relationships are
harmonious and when parameters change, as well as traditional control theory, which is based on
mathematical models, is still immature, and some methods are complex and cannot be directly
applied for use in industrial applications. In this context, it is important to create "a model-free
intelligent algorithm used to achieve end-to-end education, intelligent management, while at the
same time taking into account industrial requirements in its simplicity and reliability.

Problem Formulation:

In this study, we found that three types of behavior control need to be learned, i.e.: (a) tracking
control corrected quickly, and (b) monitoring tracking gradually at a constant rate and (c)
monitoring tracking at a constant rate.

adaptive cruise speed. For more details, the first task, which requires small vehicles moving in
the center line of the test track and at the same time, constantly keeps the speed. The second task
you can control the difficulty, force the vehicle, find out the maximum speed that allows, the
vehicle moves along the center line without going beyond the boundaries. The main goal of this
work is to enable the vehicle to keep them in the center line, and at the same time learn to slow
down or speed up self-adaptively depending on different road conditions. Unlike all other control
methods, the aim of this study is to find practical approaches that allow a vehicle in a real-world
environment, in real time, to control its behavior and at the same time adapt to a changing
situation. Therefore, it is necessary to develop an algorithm that is balanced in terms of time,
efficiency, and accuracy. However, to implement an adaptive cruise control system, namely, a
set of symmetrical control measures consisting of steering angle and speed must be optimized at
the same time. Shows the flow control of a small-scale intelligent vehicle these sensors use the
controller to regulate the main steering wheel and experience the engine cycle to hold while the
vehicle is moving the center line along the required vehicle speed.

Training is the core technology used in most artificial intelligence programs. Compared to
supervised learning and unsupervised engine training, in-room learning, continuous use of the
trial-and-error mechanism, and "development and research" is a balance, but correctly noted
example. Thanks to understand, real-time communication about the state of the environment (for
example, monitoring the signal, about its actions), settings are continuously adjusted to achieve
optimal consistent decision making in any given task. Below are the basic concepts, RL-based
trekking, and an adaptive cruise control algorithm.

System Sate:

This control algorithm is directly defined by the system states. In this study, the deviation from
the roofline centerline, vehicle yaw angle, and vehicle speed are selected to form

The three-dimensional state space, i.e., s(T) = (F(T), Φ (t), b (T)) T, where f (t) represents the
offset of the center line of time t, α(T) and (t) and the yaw angle and vehicle speed, time t,
respectively. Photo. 8 describes three states defined by America sum, power representing a
number

of turning angles states sum pulse width pulse modulation (PWM) collection vehicle speed
states, number, and at the top of the figure is a set of offsets of the center roof line. It should be
noted that only in a limited number of countries selected in this study, we there aim to: (a) reduce
learning and training; and (b) show that the generalizability of RL methods.
Control Action:

Decision-making during the management of the servo motor, which decides whether increased
the angle of rotation of the vehicle's engine determines speed of the vehicle based on
management tasks

actions, denoted as A(t) = (µ(t), u(t)), where t is the index of a temporary step. A(t) must be
discretized in there. We select the angle of the servomotor and the period of operation of the
motor, as a check for the implementation of the RL-based algorithm, that is, the entire location of
the action, in case A = A1, A2, ... unit, here n is the level of the example and multiply the
number of actions, controls and the number for high-speed action. In this study, for the first two
problems, we consider n to be the case of 9, and the main question, n, n is set to 27 with three-
speed activity control, additional.

Algorithm of Q:

Initialize Q (s, a) and M (s, a) for all s ∈ S and a ∈ A(s)

Do forever :

S ← current (nonterminal) state

A ← ε − greedy (S, Q)

Execute action A; observe resultant reward, R, and state, SJ

Q (S, A) ← Q (S, A) + α [R + γmaxaQ (SJ, a) − Q (S, A)]

Model (S, A) ← R, SJ (assuming deterministic environment)

Repeat n times:

S ← random previously observed state

A ← random action previously taken in S

R, SJ ← Model (S, A)
Application of Q-Learning:

 Robotics for industrial automation.

 Business strategy planning
 Machine learning and data processing
 It helps you to create training systems that provide custom instruction and materials
according to the requirement of students.
 Aircraft control and robot motion control
 Online Web Systems Auto-configuration
 News Recommendations
 Network Traffic Signal Control
 Game-playing:
 Deep Q-Learning  to play Snake
 DDQN (Double Q-Learning) for Atari
 Health

Advantage:

 Q-learning directly learns the optimal policy.

 Q-learning will ignore possible penalties from exploratory moves. If there is risk of a
large negative reward close to the optimal path, Q-learning will tend to trigger that
reward whilst exploring.

Conclusion:

Although Q-learning is experiencing difficulties in the ongoing space but with local development
and artificial intelligence, Q-in-depth learning of the neural network such as the Deep Q-network
improves the Q-learning application in the ongoing space. Q-learning has a high sample per unit
and may suffer from changing problems as a result of this which ends up being a problem when
training neural networks with Q-learning. Q-learning is obviously one of the most widely used
ways to strengthen students. The importance of enhanced learning is enhanced by the inclusion
of non-personal intelligence in almost every aspect of computing. Q-learning will continue to
drive innovation and the development of smart systems.

References:

1. Paden, B.; Cˇ áp, M.; Yong, S.Z.; Yershov, D.; Frazzoli, E. A Survey of Motion Planning
and Control Techniques for Self-Driving Urban Vehicles. IEEE Trans. Intell. Veh. 2016, 1,
33–55. [CrossRef]
2. Broggi, A.; Cerri, P.; Debattisti, S.; Laghi, M.C.; Medici, P.; Molinari, D.; Panciroli, M.;
Prioletti, A. PROUD—Public Road Urban Driverless-Car Test. IEEE Trans. Intell.
Transp. Syst. 2015, 16, 3508–3519. [CrossRef]
3. Li, L.; Huang, W.; Liu, Y.; Zheng, N.; Wang, F. Intelligence Testing for Autonomous
Vehicles: A New Approach. IEEE Trans. Intell. Veh. 2016, 1, 158–166. [CrossRef]
4. Xu, Z.; Wang, M.; Zhang, F.; Jin, S.; Zhang, J.; Zhao, X. Patavtt: A hardware-in-the-loop
scaled platform for testing autonomous vehicle trajectory tracking. J. Adv. Transp. 2017,
1–11. [CrossRef]
5. From the Lab to the Street: Solving the Challenge of Accelerating Automated Vehicle
Testing. Available online: http://www.hitachi.com/rev/archive/2018/r2018_01/trends2/index.html/
(accessed on 1 September 2019).
6. Ruz, M.L.; Garrido, J.; Vazquez, F.; Morilla, F. Interactive Tuning Tool of Proportional-
Integral Controllers for First Order Plus Time Delay Processes. Symmetry 2018, 10, 569.
[CrossRef]
7. Liu, X.; Shi, Y.; Xu, J. Parameters Tuning Approach for Proportion Integration
Differentiation Controller of Magnetorheological Fluids Brake Based on Improved Fruit
Fly Optimization Algorithm. Symmetry 2017, 9, 109. [CrossRef]
8. Chee, F.; Fernando, T.L.; Savkin, A.V.; Heeden, V.V. Expert PID Control System for
Blood Glucose Control in Critically Ill Patients. IEEE Trans. Inf. Technol. Biomed. 2003,
7, 419–425. [CrossRef]
9. Savran, A. A multivariable predictive fuzzy PID control system. Appl. Soft Comput.
2013, 13, 2658–2667. [CrossRef]
10. Lopez_Franco, C.; Gomez-Avila, J.; Alanis, A.Y.; Arana-Daniel, N.; Villaseñor, C.
Visual Servoing for an Autonomous Hexarotor Using a Neural Network Based PID
Controller. Sensors 2017, 17, 1865. [CrossRef]
11. Moriyama, K.; Nakase, K.; Mutoh, A.; Inuzuka, N. The Resilience of Cooperation in a
Dilemma Game Played by Reinforcement Learning Agents. In Proceedings of the IEEE
International Conference on Agents (ICA), Beijing, China, 6–9 July 2017.
12. Meng, Q.; Tholley, I.; Chung, P.W.H. Robots learn to dance through interaction with
humans. Neural Comput. Appl. 2014, 24, 117–124. [CrossRef]
13. Zhang, Z.; Zheng, L.; Li, N.; Wang, W.; Zhong, S.; Hu, K. Minimizing mean weighted
tardiness in unrelated parallel machine scheduling with reinforcement learning. Comput.
Oper. Res. 2012, 39, 1315–1324. [CrossRef]
14. Iwata, K. An Information-Theoretic Analysis of Return Maximization in Reinforcement
Learning. Neural Netw.
15. 2011, 24, 1074–1081. [CrossRef] [PubMed]