20I1 3rd International Conference on Advanced Computer Control (ICACC 2011)

In-situ Optimal Control of Nutrient Solution for Soilless Cultivation

2
Feng Chen\ Haidong He y ongning Tang
Department of Automation School ofInformation Technology
University of Science and Technology of China Illinois State University
Hefei, China Chicago, USA
1 chenfeng@ustc.edu.cn ytang@iIIinois.edu
2haidongee@gmail.com

Abstract-Soilless cultivation is one of main production modes vegetable growth and nutrient solution. Therefore, it is a
in precision agriculture. Nutrient solution for soilless challenge to achieve optimal control of nutrient solution!
cultivation primarily consists of macroelements such as The purpose of this work is to find an in-situ optimal
2 2 control policy for nutrient solution under natural climate
N03 - , H2PO� , K+ , Ca + , Mg + , and SO;- . In
environment. Research on optimal control for nutrient
vegetable-nutrient solution system, there are complicated
solution has been received considerable attention in recent
correlations between greenhouse vegetable growth and
two decades [3]. Several efforts have been done to use
nutrient solution, thus it is a challenge to achieve optimal
nutrient solution as efficient as possible for soilless
control of nutrient solution. In this work, based on Q-Iearning,
cultivation. In 1999, Savvas et al. first proposed an algorithm
we first propose an in-situ optimal control method of nutrient
solution compositions for greenhouse vegetable. Instead of
to perform replenishment and reuse of the drain nutrient
modeling the correlations between greenhouse vegetable
solution in close hydroponic systems on the basis of
growth and nutrient solution, this method searches for optimal dispensing as many nutrients and water in it as were needed
control policy through systematic interaction with the to maintain a target electrical conductivity in the irrigation
environment. The effect of nutrient solution compositions on solution. To achieve this in each watering application,
photosynthetic rate of greenhouse vegetable is experimentally dilution ratios of the concentrated fertilizer solutions are
investigated, and on this basis reward function is designed. The automatically adjusted based on this algorithm, in relation to
experimental results show our method is effective and practical. the volume and the electrical conductivity of the drain
solution [4]. Lykas et al. presented a method to control
Keywords-nutrient solution; in-suit; optimal control; Q­ recirculating nutrient solutions in hydroponic crops made use
learning of correction algorithms to manage soilless cultures by
measurement of electrical conductivity (EC) and pH [5].
I. INTRODUCTION Based on the least square fit with neural BP network, Yang
et al. developed a method to control pH and EC of nutrient
Soilless cultivation is one of main production modes for
solution [6]. Zhang et al. studied an unlimited growth
greenhouse agriculture. The control system of closed cycle
cultivation mode for greenhouse tomato grows under deep
for nutrient solution is a key technique in solilless cultivation,
flow technique condition, By researching on the control
which involves growing greenhouse vegetables by collecting
technique of the nutrient solution root system
left over nutrient solution, blending it with fresh nutrient
microenvironment, a control system is developed to control
solution, and recirculating in subsequent irrigation [1]. The
pH, EC and temperature of nutrient solution [7].
nutrient solutions can be drained back down and re-utilized
All the aforementioned studies fulfill the control of
to avoid the waste of nutrients and to reduce the
nutrient solution only rely on pH and electronic conductivity
environmental impact, thus fulfilling recent directives for
(EC) values. Because pH represents the total acidity and
environmental protection.
basicity of solution and EC values represents the total
Nutrient solution for soilless cultivation primarily
activity of ions, there is no the function relation between pH,
consists of the macroelements such as N03 -, H2PO� , EC and the macroelements of nutrient solution [8], such
2 Lf 2 2 . methods can not achieve optimal control of nutrient solution
K+ , Ca + , 1vig + , and S04 - . Optimal control of compositions.
nutrient solution for soilless cultivation can minimize water Based on Q-leaming, we first propose an in-situ optimal
and fertilizer losses and gain maximum the yield and high control algorithm for nutrient solution compositions. Instead
quality of vegetables for greenhouse culture [2]. of modeling the complex correlations between greenhouse
Greenhouse vegetable is influenced by not only its vegetable growth and its nutrient environment, this algorithm
surrounding climate but the nutrient solution. Vegetable finds optimal control policy of nutrient solution
growth and its nutrient environment form a dynamic and compositions through systematic interaction with the
complex system under greenhouse environment. Because the environment. The correlations between the nutrient solution
growth process of greenhouse vegetable has the properties compositions and the photosynthesis rate of greenhouse
such as large time delay and uncertainty, it is extremely vegetable are experimentally investigated. On this basis, the
difficult to model the correlations between greenhouse

978-1-4244-8810-0 /11/$26.00 mOll IEEE 412

 2011 3rd International Conference on Advanced Computer Control (ICACC 2011)
reward function is designed. The experimental results
demonstrate this algorithm is effective and feasible.
II. Q -LEARNING
Reinforcement learning is a type of unsupervised
learning and consistent with the behaviorism theory
presented by Brooks [9]. Recently, reinforcement learning
has been the focus of machine learning and applied
successfully in many fields [lO-13]. Q-learning is one of the
most important reinforcement learning algorithms [14]. In
reinforcement learning module, an agent acts on environment
and receives reinforcement signal which is either punishment
or reward induced by environment state transition. The
learning task for an agent is to search for an optimal policy:
state-action sequences, which is an action selection rule
corresponding to a given state. The basic model of
reinforcement learning is shown as Fig.l.
State s
Rewardr
Actiolla
Figure I. The basic model of reinforcement learning
In Q-learning, an environment is regarded as a Markov
process with finite states. Q-learning optimizes directly a Q­
function that can be iteratively computed without
environment model. Q-function, defined as formula I, is a
discounted cumulative reward given that an agent starts in
state St ' applies action at once, and follows a policy
thereafter.
Where, r (0 < r < 1) is discounted rate, � is reward
received when the environment makes a transition from state
St to St+l . Due to the above properties, Q-Iearning is
suitable for achieving the optimal control of nutrient solution
compositions, which offers optimal nutrient environment for
greenhouse vegetable.
III. IN-SITU OPTIMAL CONTROL OF NUTRIENT SOLUTION
A. The Correlations Between Photosynthetic Rate and
Nutrient Solution Compositions
In order to fulfill in-situ optimal control of nutrient
solution compositions, it is prerequisite to select proper
evaluation indicator for growing status of greenhouse
vegetable. Photosynthesis is the most important physiology
processes of greenhouse vegetable. It is vital to survival of
all biology. Photosynthetic rate is the indicator of
photosynthesis. There is the significant positive correlation
between vegetable yield and its photosynthetic rate.
413
Photosynthetic rate can represent real-time status of
greenhouse vegetable growing. The current sensor
techniques make it possible to on-line measure
photosynthetic rate of greenhouse vegetable. Therefore, we
choose photosynthetic rate as evaluation indicator of real­
time growing status of greenhouse vegetable.
The correlations between nutrient solution compositions
and photosynthetic rate of the greenhouse vegetable are
explored. Through experimental investigation for two years,
we have the findings in the following:
• In a given greenhouse climate, proper composition
concentrations of nutrient solution can increase
photosynthetic rate of greenhouse vegetable. Too high
or too low composition concentration of nutrient
solution will lead to decrease the photosynthetic rate
(shown as Fig.2)
25
00
1�
�
.� 15
m
.c
{
c
10
'"'
OL-�____�____-L____-L____�____��
100 150 200 250 :nJ 350
k concentration(mgll)
Figure 2. The effect of k concentration on photosynthetic rate of tomato
plant
• Nutrient solution is confected according to the definite
proportion of nutrient compositions can promote
photosynthetic rate of greenhouse vegetable.
Therefore, based on the variation of photosynthetic rate
of greenhouse vegetable, an in-situ optimal control
mechanism is investigated in this work.
B. In-situ Optimal Control ofNutrient Solution
For practical application, we study in-situ optimal control
method of nutrient solution compositions without
considering the control of greenhouse climate. Observing
that greenhouse climate changes in a finite range and is
almost cyclical, therefore, global optimal control on nutrient
solution could be decomposed into multi-step local optimal
control to achieve daytime nutrient solution optimal control.
For greenhouse vegetable growth process has the
characteristics such as large time delay and uncertainty,
greenhouse vegetable status S can be defined as
S = APHO X APAR , where APHO and APAR denote
the variations of photosynthetic rate and photosynthetic
active radiation, respectively. Thus a discrete state sequence
 2011 3rd International Conference on Advanced Computer Control (ICACC 2011)

S1' S2' ..., S;, ..., Sn is adopted to represent greenhouse basic unit. The basic unit is determined according to the
following formula:
vegetable growing process, where i =1 denotes initial time
and i = n is terminal time in one iteration of the nutrient I"JX = Ymax - Ymin (4)
solution control. The related definition is given in the a
following: Here, Ymax and Ymin express the maximum and
Definition: in-situ optimal control of nutrient solution for minimum of the nutrient solution compositions, a is
greenhouse vegetable is defined as a quadri-tuple constant and generally set to 5 or 10. Search space can be
)
(S, A,P, R , where, S is a finite set of the growing states significantly reduced with state segmentation and action
definition.
of greenhouse vegetable;
Furthermore, in order to avoid trapping into local
A = AN X Ap X AK X 4: X AM X As is finite set of optimum, Boltzmann distribution is introduced to select the
executable joint action, where A N' Ap, AK, Ac' AM and joint actions, which can achieve effective tradeoff between
exploration and exploitation during the search process.
As are the subsets of joint action set A, which are used to
IV. ALGORITHM DESCRIPTION
2 2
respectively control N03-, H2PO�, K+, Ca +, Mg +,
Let a be learning rate, r be discounted rate, where
and SO;- in nutrient solution, P is transition probability of a, rE (0,1) , a policy f is a function from states to
actions: f: S � a, and f(x) denotes the actions to be
state; R : S X A � R is a numerical reward function. The
control system aims at finding a policy that can obtain
optimal discounted cumulative rewards for any discrete state conducted in state x. For a fixed policy f, let VI (x) be
the value function for policy f , which is desired rewards
sequences in the conditions of unknown P and R.
Reward function is real-time photosynthetic rate, which
is defined as PHO /PHOMAX (PHOMAX is possible given that the process begins in state x and follows policy
maximal photosynthetic rate). f thereafter. The control algorithm for nutrient solution is
As the states and actions are continuous values, it leads to described as follows:
extremely large search space. Considering greenhouse Q+- initialized actions' value
vegetable growth characteristics of delay time and k= 0 Ilk is iterative index
imprecision, MHO and MAR can be segmented in
Order S e is goal state
term of formula (2) and (3) to decrease the search space.
1: Repeat
pho1 , -1.0-:;;�PHO < -0.5
S +- current state
pho2 , - 0.5-:;; �PHO < 0 If S = se then goto 1 endif
MHO = pho3 , �PHO=O (2) Select an action to perform III term of Boltzmann
distribution:
eQ(s,a)IT
pho4 , 0.0 < �PHO < 0.5
phos, 0.5-:;; �PHO < 1.0 p(a,s) =
>Q(s,b)IT
beLA
pal], - 50-:;; �PAR < -30 II Where T is a temperature parameter used to adjust
degree of randomization for choice action
par2 , -30-:;; �PAR < -10 If the concentration of composition i :2: the maximum
MAR = pa1) , -10-:;;�PAR-:;;10 (3) concentration of composition i then aj = null endif

par4 , 10 < �PAR < 30 Perform action a and update Q(s, a) function:
pars' 30-:;; �PAR < 50 Q(s,a)� (1- a)Q(s,a)+ a[r + }'V (y)]
Where, the units of MHO and MAR are II V(y) Q(y,j(y)) , for each YES : f(Y)f- a
= that
2
f.111101 / m S . Q(y,a) maxbeA Q(y,b)
=

Considering both the possible variation range of nutrient Revise policy value Vj(s) maxbeA Q(s,b)
=

solution compositions in sample interval and measurement

precision of the related sensors, each sub-action of joint T = flkTa liTo is initial temperature
action can be defined as {null,I"JX,21"JX}. Where, null k=k+l
denotes empty action, I"JX denote that one basic unit of Sf-yo
nutrient solution composition is added, and 21"JX is two

414
 2011 3rd International Conference on Advanced Computer Control (ICACC 2011)
V. EXPERIMENT AND DISCUSSION
For the purpose of comparison, we built an experimental
environment consisting of two hydroponics channels where
the unlimited growth tomatoes are cultivated. The nutrient
solution is prepared according to the nutrient formula in Ref.
[ IS]. The nutrient solutions for greenhouse tomato are
controlled respectively using the conventional method and
this proposed algorithm. The conventional method adopts
PID controller to conduct the control of nutrient solution
according to the setpoints of pH and EC of nutrient solution
[ IS, 16]. The control systems run only in daytime to avoid
expensive costs. The photosynthetic rate of tomato plant is
measured on line using the device Li-6400. The sample
interval is set to five minutes. When our algorithm is used to
implement the control of nutrient solution, temperature
parameter T is set to 100, fJ is 0.95, learning rate a is 0.1.
After over three months of experiments, the results are
shown as Fig.3
25
-- The proposed algorithm
....... The conventional method
20
0:
Q.
�
'"
� 10
>
�
�
� [3]
°0�� 10�-�=-�3�0--�40��50�-6=0--�7�0--�BO��90��100
time (day)
Figure 3. The comparison of the proposed algorithm with the
conventional method (PR denotes photosynthetic rate)
Form Fig.3, it can seen that the daily average
photosynthetic rate using our algorithm is slightly less than
that using the conventional method only in initial period. It is
due to the exploration of this proposed algorithm to search
for optimal control policy. When this proposed algorithm is
used to control nutrient solution compositions, the daily
average photosynthetic rate of tomato plant increases with
iterations. During whole experimental process, daily average
2 [8)
photosynthetic rate of our algorithm is 16.26 f1Jnol / m S .
Accordingly, daily average photosynthetic rate of the
2 [9]
conventional method is 15.77 f1Jnol / m s. Our method is
significantly superior to the conventional method.
VI. CONCLUSION
Greenhouse vegetable and its nutrient environment form
a dynamic and complex system. Due to difficulty for
modeling the correlations between greenhouse vegetable
growing and nutrient solution compositions, it is a challenge
to achieve optimal control of nutrient solution. Based on
415
reinforcement learning, a model-free algorithm is presented
for in-situ optimal control of nutrient solution. According to
vegetable growth characteristics such as large time delay and
uncertainty, the states can be segmented to decrease the state
space. Through the experimental investigation, we find that
there are the significant correlations between photosynthetic
rate of greenhouse vegetable and nutrient solution
compositions. Reward function is designed based on
photosynthetic rate, which can represent the real-time
growing status of greenhouse vegetable. This proposed
algorithm is obviously superior to the conventional method.
This work offers an effective means for optimal control of
complex nutrient solution system.
ACKNOWLEDGMENT
This work is supported financially by National Science
Fund of China under Granted No. 60775014, the Science and
Technology Key Project of "llth-five-year plans" of Anhui
Province, which are grateful.
REFERENCES
[I] Francesco Montesano, Angelo Parente, Pietro Santamaria, "Closed
cycle subirrigation with low concentration nutrient solution can be
used for soilless tomato production in saline conditions," Scientia
Horticulturae,vol. 124, no. 3,pp. 338-344, April 2010.
[2] Harmanto, V.M. Salokhe, M.S. Babel, H.J. Tantau, "Water
requirement of drip irrigated tomatoes grown in greenhouse in
tropical environment," Agricultural Water Management, vol. 71, pp.
225-242,2005
Margarita Darder, Andres Valera, Ernesto Nieto, et aI., "Multisensor
device based on Case-Based Reasoning (CBR) for monitoring
nutrient solutions in fertigation," Sensors and Actuators B: Chemical,
vol. 135,no. 2,pp. 530-536,January 2009.
[4] D. Savvas, G. Manos, "Automated Composition Control of Nutrient
Solution in Closed Soilless Culture Systems, Journal of Agricultural
Engineering Research, " vol. 73,no. I,pp. 29-33,May 1999.
[5) C Lykas, N. Katsoulas, P. Giaglaras, C Kittas, "Electrical
conductivity and pH prediction in a recirculated nutrient solution of a
greenhouse soilless rose crop," J Plant NutL, vol. 29, pp. 1585-1599,
2006.
[6) Yang chenzhong, Huang yinchun, Zheng weihong, "Research of
Hydroponics Nutrient - Solution Control Technology," Proceedings
of the 5 World Congress on Intelligent Control and Automation, June
15-19,2004, Hangzhou,P.R China,pp. 642-645.
[7) Zhang Shuge, Song Weitang, Huang Zhidong, "Study on
Environmental Control Technology of Some Unlimited Growth
Tomato in Greenhouse by OFT," Journal of China Agricultural
University, vol. 4, pp. 34-38,2002.
Feng Chen, Dali Wei, Yongning Tang, "Virtual ion selective
electrode for on-line measurement of nutrient solution components,"
IEEE sensors Journal (accepted).
R A Brooks, "Intelligence Without Representation," Artificial
Intelligence Journal,vol. 47,pp.139-159,1991.
[10) AG. Barto, "Reinforcement learning in the real world," IEEE
International Joint Conference on Neural Networks,July 2004, vol. 3,
pp. 25-29.
[II) 0 Jangmin, L. B. Jongwoo, Byoung-Tak Zhang, et aI., "Adaptive
stock trading with dynamic asset allocation using reinforcement
learning," Information Sciences,vol. 176,pp. 2121-2147,2006.
[12) K. Macek, 1. Petrovic, N. Perie, "A reinforcement learning approach
to obstacle avoidance of mobile robots," 7th International Workshop
on Advanced Motion Control,July 2002,pp.462 - 466.
 2011 3rd International Conference on Advanced Computer Control (ICACC 2011)

[13] Steven O. Whitehead, Long-li Lin, "Reinforcement learning of non­
Markov decision processes," Artificial Intelligent, vo1.73, pp. 271-
306,1995.
[14] C l. C H. Watkins, P.Oayan. "Technical notes: Q-Iearning,"
Machine Learning, vol. 82,pp. 39-46,1992.
[IS] Weijie liang, Wei Liu, New technique for soilless cultivation of
vegetable,Beijing lindun Press,I998
[16] F. Lafont, 1.F. Balmat, "Optimized fuzzy control of a greenhouse,"
Fuzzy Sets and Systems,vol. 128,pp. 47-59,2002

416