Biomedical Signal Processing and Control 78 (2022) 103987

Contents lists available at ScienceDirect

Biomedical Signal Processing and Control

journal homepage: www.elsevier.com/locate/bspc

Design and intelligent control of mock circulation system to reproduce

patient-specific physiological indexes
Te Li , Heng Li , Wenbo Cui , Nan Xie , Xu Li , Yongqing Wang *
Key Laboratory for Precision & Non-traditional Machining of Ministry of Education, Dalian University of Technology, Dalian 116024, China

A R T I C L E I N F O A B S T R A C T

Keywords: Mock circulatory system (MCS) is a practicable solution to the cardiovascular device evaluation and pathological
Mock circulatory system study prior to in vivo testing. In order to enhance the patient-specific imitative ability of the MCS in vitro tests, a
Deep reinforce-ment learning personalized MCS that simulates aortic pressure and mean flow for a patient-specific condition is developed. The
Intelligent control
designed MCS can be fully controlled automatically, which is consisted of left ventricle, aortic compliance
Personalized simula-tion
Cardiovascular device evaluation
chamber, systemic vascular resistance and reservoir. A soft actor-critic based intelligent control method for
personalized in vitro simulation is proposed, which is strongly robust to the MCS complexity and uncertainty.
The designed MCS with the proposed control method can simulate aortic pressure and flow of a personalized
patient with high accuracy and fast adjusting speed. The MCS control experiments are conducted with several
groups of typical cardiovascular patients’ physiological indexes. The results show that the maximum deviations
of aortic systolic and diastolic pressures from the target values are less than 2 mmHg, and the maximum devi­
ation of aortic flow is less than 0.2 L/min. Therefore, the developed MCS is potential to the personalized in vitro
evaluation of cardiovascular devices such as ventricular assist devices and heart valves.

1. Introduction
In recent decades, cardiovascular diseases have become a major
cause of death, which is a nonnegligible public health problem in the
world [1]. Cardiovascular devices (CVDs) such as ventricular assist de­
vices, intra-aortic balloon pumps, heart valves and others have achieved
rapid progression and provided important support for the treatment of
cardiovascular diseases in recent years. In particular, ventricular assist
devices (VADs) have become an important clinical option for the
treatment of heart failure (HF) due to the severe shortage of donor hearts
and advances in technology [2].
Mock circulatory system is an essential in vitro evaluation platform
in the design and development of CVDs. The MCS simulates the structure
and function of the human circulatory system, allowing rapid and
repeatable adjustment of relevant parameters to reproduce the physio­
logical variables associated with real blood circulation, such as arterial
pressure and flow. MCS provides an evaluative environment similar to
that of the human body for CVDs [3].
Many kinds of MCSs have been developed, including early non-
pulsatile MCSs and multi-parameter adjustable pulsatile MCSs [4–10].
These MCSs can simulate the healthy and pathological conditions to help
evaluate VADs [11], test heart valves [12], etc. Takatani et al., [6]
designed an MCS with a passively filling ventricle, which changed the
ventricular elasticity by adjusting the volume ratio of air to fluid. Allaire
et al., [10] developed an MCS with pneumatically driven ventricles for
left ventricular assist device (LVAD) evaluation, which can simulate a
wide range of physiological conditions such as rest, exercise and heart
failure. Ochsner et al., [13] presented a hybrid mock circulation which
combined parts of physical system and numerical model. More complex
MCSs have been developed in order to more closely resemble the real
human blood circulation system [14–16]. Gregory et al., [17] designed
an advanced mock circulation loop with systemic, pulmonary, cerebral
and coronary circulations with autoregulatory responses.
Most MCSs generally can only simulate a few fixed physiological
conditions and provide limited assessment environments for CVDs.
However, it’s also very important for MCS to simulate patient-specific
conditions. If personalized in vitro simulation can accurately simulate
the physiological characteristics of the specific patient, it can be a
valuable tool for hemodynamic studies of patient pathology and CVDs
development. It also allows for personalized in vitro evaluation of CVDs
for use in patients prior to surgery, reducing post-surgical adverse

* Corresponding author at: Key Laboratory for Precision & Non-traditional Machining of Ministry of Education, Dalian University of Technology, Dalian 116024,
China.
E-mail addresses: teli@dlut.edu.cn (T. Li), liheng0424@mail.dlut.edu.cn (H. Li), 1297691410@mail.dlut.edu.cn (W. Cui), xiedanan@mail.dlut.edu.cn (N. Xie),
imlixu@dlut.edu.cn (X. Li), yqwang@dlut.edu.cn (Y. Wang).

https://doi.org/10.1016/j.bspc.2022.103987
Received 12 December 2021; Received in revised form 5 May 2022; Accepted 11 July 2022
Available online 15 July 2022
1746-8094/© 2022 Published by Elsevier Ltd.
T. Li et al. Biomedical Signal Processing and Control 78 (2022) 103987

Nomenclature PPV Pneumatic electric proportional valve

C0 Initial compliance
MCS Mock circulatory system Ctarget Target compliance
CVDs Cardiovascular devices Hnow Current level height in AOC
VADs Ventricular assist devices Htarget Target level height
HF Heart failure SAC Soft actor-critic
LVAD Left ventricular assist device LVP Left ventricular pressure
MIMO Multi-input and multi-output LAP Left atrial pressure
DRL Deep reinforcement learning AOP Aortic pressure
LV Left ventricle AOPsys Aortic systolic pressure
AOC Aortic compliance chamber AOPdia Aortic diastolic pressure
SVR Systemic vascular resistance Qmean Mean aortic flow
Res Reservoir AOPtsys Target value of aortic systolic pressure
AV Aortic valve AOPtdia Target value of aortic diastolic pressure
MV Mitral valve t
Qmean Target value of mean aortic flow
HR Heart rate m
AOPsys Measured value of aortic systolic pressure
Pne Compressed air pressure
m
Vacu Vacuum rate AOPdia Measured value of aortic diastolic pressure
C Aortic compliance chamber compliance m
Qmean Measured value of mean aortic flow
Rs Systemic vascular resistance values TOT TOT plus 1 when S2 is less than 4 once
ACS Apical cannulation site S2 Variance between measured values and target values
P Pressure sensor HEN Hypertension
Q Flow sensor HON Hypotension
L Liquid level sensor
SV Solenoid valve

effects.
Some attempts have been made to allow the MCS to simulate patient-
specific features. Cestari et al., [18] presented a novel automated pe­
diatric simulator, which can rapidly adjust vascular resistance, aortic
compliance and left ventricular preload. After entering arbitrary target
values, the system can quickly and accurately simulate aortic and left
atrial pressures, providing a wide range of in vitro testing environments
for the assessment of pediatric LVAD. This MCS was used for infants
under 1 year of age. The aortic flow could not be set arbitrarily and was
determined by the formula.
Balabani et al., [19] designed an in vitro platform that could repro­
duce patient-specific vascular conditions. The platform included a new
adjustable three-element Windkessel vascular simulators and a
computer-controlled pulsatile pump. This platform allowed personal­
ized simulation of left ventricular, aortic pressures and flow in clinical
patients by combining mathematical models and computer routines. The
maximum pressure deviation is 5 mmHg. Systemic resistance and
compliance need to be individually configured (for example, resistance
can be adjusted by 3D printing porous materials). The time required to
generate physical manifestations from patient-specific data varies from
at least 4 days to about 2–3 weeks.
To simulate the patient’s physiological conditions, it requires good
coordination of the relevant component parameters of the MCS to ach­
ieve target values. So, MCS is a complicated MIMO (multi-input and
multi-output) nonlinear system. Previous control methods [25] for a
single variable are difficult to apply. Personalized MCS requires faster
and more accurate reproduction of patient-specific features on a wide
scale. Therefore, it is necessary to study the intelligent control method to
reproduce patient-specific physiological indexes.
This study presents a personalized MCS for simulating patient-
specific aortic pressure and mean flow. A personalized in vitro simula­
tion control method based on deep reinforcement learning (DRL) is
proposed to address the complexity and uncertainty of the MIMO MCS
control system. After inputting patient-specific target values, the MCS
can quickly and accurately simulate aortic pressure and mean flow.
2. Methods
2.1. Design of mock circulation system
A soft pulsatile pump is designed to simulate the ventricular function
for more realistic simulation of the natural heart. In addition, automatic
controllability of critical parts of the whole system is considered in the
design in order to achieve controllable precise and fast regulation. The
fully automated MCS is developed to simulate the systemic circulation,
including left ventricle (LV), aortic compliance chamber (AOC), sys­
temic vascular resistance (SVR), reservoir (Res), aortic valve (AV) and
mitral valve (MV). A complete schematic and photograph of the com­
ponents of the MCS are shown in Fig. 1. Heart rate (HR), the driving
compressed pressure (Pne), the vacuum rate (Vacu), aortic compliance
chamber compliance (C), and systemic vascular resistance values (Rs)
can all be adjusted automatically. The water or glycerol mixture (60 %
or 40 % mass fraction) is adopted as the working fluid.
The LV is simulated by a silicone diaphragm encased in a transparent
acrylic sealed air chamber. The silicone diaphragm is semi-elliptical,
with a volume of 220 mL and a location (ACS) for an external ventric­
ular assist device at the bottom. Check valves are connected at the left
ventricular inlet and outlet to simulate the MV and AV, respectively.
Compressed and vacuum air are alternately introduced into the air
chamber to simulate the systolic and diastolic functions of the ventricle,
with SV3 and SV4 controlling the diastolic and systolic times, respec­
tively. The compressed air is adjusted by a pneumatic electric propor­
tional valve (VPPM-6F-L-1-F-0L10H-V1N, Festo). The vacuum generator
(ZL112A-K15GZ, SMC) is connected to the vacuum chamber to generate
vacuum air. The rate of vacuuming is determined by the pressure of the
air supplied to the vacuum generator. A pressure sensor (CYYZ11-H-67-
A1-17-C-G, Star Sensor) is connected to the top of the LV to measure the
left ventricular pressure (LVP).
The AOC is made of transparent acrylic with an internal diameter of
120 mm and a volume of 2.5 L. A liquid level sensor (PT124B, Zhaohui)
measures the level of the AOC in real time, and a pressure sensor is
connected to the bottom of the AOC to measure the aortic pressure.
Vascular compliance is defined as the amount of change in vascular

2
T. Li et al.
Fig. 1. The mock circulation system including (a) schematic of MCS and (b)
photograph of the completed MCS. LV, left ventricle; AOC, aortic compliance
chamber; SVR, systemic vascular resistance; Res, reservoir; AV, aortic valve;
MV, mitral valve; ACS, apical cannulation site; P, pressure sensor; Q, flow
sensor; L, liquid level sensor; SV, solenoid valve; PPV, pneumatic electric pro­
portional valve. Arrows indicate direction of flow.
volume when the intravascular blood pressure is changed by one unit.
The compliance of the AOC can be expressed by the following equation
[10]:
C=
dVfluid Vair Vtank − Stank Hfluid
= = (1)
dPfluid Pair Pfluid − ρgHfluid
where C is the compliance of the AOC, generally at 1 ~ 2 mL/mmHg,
Vfluid is the volume of fluid in the AOC, Pfluid is the absolute pressure of
fluid in the AOC, Vair is the volume of air sealed in the AOC, Pair is the
absolute pressure of gas sealed in the AOC, Vtank is the volume of the
AOC, Stank is the internal bottom area of the AOC, Hfluid is the height of
fluid in the AOC, ρ is the fluid density, and g is the acceleration of
gravity.
From Eq. (1), it is known that the AOC compliance is determined by
the volume of sealed air inside the AOC. To achieve the automatic
adjustment of compliance, the volume of air in the AOC is adjusted by
connecting compressed gas or air. The compliance adjustment method of
the AOC is shown in Fig. 2.
Initially calculate the C0 of the current AOC and compare C0 with
Ctarget. If C0 is greater than Ctarget, SV1 is opened to eject air to reach the
target level Htarget. If C0 is less than Ctarget, SV2 is firstly opened to inject
compressed air until the measured level is below the target level by 5
mm, and then air is ejected to reach the target level. When Ctarget is
3
Biomedical Signal Processing and Control 78 (2022) 103987
START
Calculate C0
NO Open SV2,
C0 > Ctarget ? Read Hnow
air injection
YES
Open SV1, NO
Close SV2 Hnow < Htarget - 5 ?
air ejection
YES
Read Hnow
Delay 0.05s
NO
Hnow > Htarget ? Delay 0.05s
YES
END
Fig. 2. Automated algorithm for adjusting aortic compliance. C0, initial
compliance; Ctarget, target compliance; Hnow, current level height in AOC; Htarget,
target level height. SV1, solenoid valve 1; SV2, solenoid valve 2.
known, the target level can be obtained by the following equation:
Vtank − Ctarget Pair
Htarget = (2)
Stank
This MCS simplifies the pulmonary circulation, the veins of the sys­
temic circulation and the left atrium into a single Res. The Res is an open
transparent acrylic container that functions to store fluid with a volume
of 5 L. A pressure sensor measures left atrial pressure at the bottom of the
Res.
SVR is simulated with a regulated electric ball valve (KLD20T, KLD)
that is linearly controlled for opening magnitude with a 0–10 V electrical
signal. A flow sensor (HI4360, Pointer Technology) is installed between
the AOC and the Res to measure aortic flow.
The diameter of the aorta is 25 mm. To reduce resistance, the LV is
directly connected to the AOC with a translucent silicone tube with inner
diameter of 25 mm. The rest of the MCS is connected with a silicone tube
of 8 mm inner diameter.
The PLC (NX1P2, OMRON) communicating with the PC through
TCP/IP, can receive signals from all sensors and control all actuators.
TCP/IP communication time is 5 ms. The ports of the PLC are refreshed
in 2 ms. The PC acquires sensor data every 6 ms. The accuracy of the
pressure sensor is ± 0.75 mmHg. The accuracy of the flow sensor is ±
0.15 L/min. The response time of the pneumatic electric proportional
valve is 250 ms, the response time of the solenoid valve is 100 ms, and
the response time of the electric ball valve is 18◦ /s (total 90◦ ).
2.2. Personalized simulation control based on soft actor-critic algorithm
Simulating patient-specific aortic pressure and flow by adjusting
multiple parameters on the MCS is a complex MIMO nonlinear system
control problem. Real physical systems are difficult to model accurately
and the control laws for multiple targets are often complex. Therefore, a
model-free and data-driven control approach for MCS is proposed here
based on DRL theory.
T. Li et al. Biomedical Signal Processing and Control 78 (2022) 103987

Fig. 3. (a): Schematic diagram of the SAC controller working. (b): Schematic diagram of patient-specific personalized simulation. AOPsys, aortic systolic pressure;
AOPdia, aortic diastolic pressure; Qmean, mean aortic flow; Δ, the difference; TOT, TOT plus 1 when S2 is less than 4 once. The S2 is shown in Eq. (6).

DRL has superior performance in the control of model-free nonlinear
systems, and has found great applications in robot control [21], auton­
omous driving [22], and gaming [23] etc. Soft actor-critic (SAC) is a
deep reinforcement learning algorithm with randomized strategies [20].
Compared with the traditional deterministic policy algorithm (DDPG),
the SAC algorithm with randomized policy has stronger exploration
capability and more advantages in multi-objective complex control.
Therefore, SAC is used as the controller of the system.
The control problem of the MCS personalized simulation can be
formulated as a Markov decision process. At moment t, the agent gets the
state st from the MCS and chooses an action at based on its policy π. The
environment returns the reward rt of the current step after performing
the action. The agent transitions to the next state st+1. The agent can
continuously interact with the MCS to update the policy. In order to
facilitate the perception and learning of the MCS, it is necessary to model
the task artificially. The state space, action space, and reward function
need to be designed for this task.

4
T. Li et al.
2.2.1. State space
Aortic pressure is an asymmetric periodic curve and it is very difficult
to simulate all the characteristics. Therefore, some simplifications are
made to the simulation of aortic pressure. The aortic pressure curve is
determined horizontally by the patient’s heart rate while vertically by
the highest (aortic systolic pressure (AOPsys)) and lowest (aortic diastolic
pressure (AOPdia)) points, with subtle cases influenced by the physical
system. In the case where the patient’s heart rate is determined, the
personalized simulation of aortic pressure only needs to simulate the
patient’s AOPsys and AOPdia. Thus the control targets of the MCS are
AOPsys, AOPdia and mean aortic flow (Qmean).
To allow the SAC controller to learn the strategy better, the differ­
ence between the system measurement and the patient target value is
chosen as the state. The state st is defined as.
[ ]
st = ΔAOPsys , ΔAOPdia , ΔQmean (3)
whereΔAOPsys = AOPm t t
sys − AOPsys ,ΔAOPdia = AOPdia − AOPdia ,ΔQmean =
m The SAC applies the Bellman equation to update the network used to
Qm t t
mean − Qmean .AOPsys , AOPdia and Qmean are the measured values.AOPsys ,
m m m
AOPtdia and Qtmean are the target values. The units of ΔAOPsys and ΔAOPdia
are mmHg, and the unit of ΔQmean is defined as 10-1L/min in order to
improve the control accuracy.
2.2.2. Action space
The HR is determined by the specific patient. The changes of Pne,
Vacu, C, and Rs are selected as the SAC controller outputs. The action at
is defined by.
at = [ΔPne, ΔVacu, ΔC, ΔRs] (4)
where each variable of at ranging from − 1 to 1 and needs to be scaled
according to a certain ratio. The final real output action is ([7.5 × ΔPne]
kPa, [37.5 × ΔVacu]kPa, [0.5 × ΔC]ml/mmHg, [27 × ΔRs]◦ ). Then the
new parameter values are output to the MCS by adding them to the last
parameter values.
2.2.3. Reward function
In order to improve the learning efficiency of the intelligences, a
reward function with segmented rewards is defined as.
10
r(st , at ) = √̅̅̅̅̅̅̅ + M (5)
3S2
where M is − 2 when |ΔAOPsys|, |ΔAOPdia| and |ΔQmean| are all less than
10, and M is − 10 in other cases. S2 is the variance between the measured
values and the target values, and can be expressed as.
[
1 ( )2 ( )2 ( )2
] The SAC gets values: AOPsys ranges from 80 to 120 (mmHg), the
S2 =
3
AOPmsys − AOPtsys + AOPmdia − AOPtdia + Qmmean − Qtmean
(6)
The key point of multi-objective control is to ensure that multiple
measured values in the regulation are close to multiple objectives at the
same time. Therefore, S2 is chosen as the denominator term of the
reward function. This can avoid the problem of large reward for single-
target approach. And the further the measured values are from the target
√̅̅̅̅̅̅̅̅
values, the fractional term (10/ 3S2 ) approaches 0. Only when all three
measured values are very close to the target value, a large positive
reward can be obtained. The segmented reward(M) is designed to give
positive rewards when the deviation from the target value is small
(within 10) and negative penalties when the deviation is far from the
target value. This reward system accelerates learning and increases SAC
intelligence.
Using the defined state space, action space and reward function, the
proposed SAC controller is shown in Fig. 3(a). A fully connected neural
network structure with 2 hidden layers and 512 neurons are used inside
the SAC controller to analyze the measurement signal and enhance the
5
Biomedical Signal Processing and Control 78 (2022) 103987
agent’s understanding of the MCS.
SAC introduces entropy to increase the algorithm exploration capa­
bility. Entropy can be expressed as Equation (6).
∑
∞
H(π(⋅|st ) ) = − π(⋅|st )logπ(⋅|st ) (7)
t=1
where π is the strategy for choosing the action at state st.
SAC is required to maximize the output entropy in addition to
maximizing cumulative reward when learning a strategy:
[ ]
∑
π* = argmaxπ Eτ∼π r(st , at ) + αH(π(⋅|st ) ) (8)
t
where α is denoted as the entropy regularization factor, which is used to
adjust the importance of the algorithm to entropy. E is the mathematical
expectation.
fit the Q. The algorithm defines the soft Q value as follows:
[ ]
Qsoft (st , at ) = r(st , at ) + γEst+1 ,at+1 Qsoft (st+1 , at+1 ) + αH(π (⋅|st+1 ) ) (9)
Using the Kullback-Leibler (KL) divergence to optimize strategy [24]:
⎛ ( )⎞
exp α1Qπsoft (st , ⋅) ⎟
old
⎜
πnew = argminDKL ⎜
⎝π(⋅|st )|| πold
⎟
⎠ (10)
Z (st )
soft
where Z πold soft(st) is the logarithm partition function, does not
contribute to the policy gradient.
Since the environment is an actual physical system, the st after each
update at requires the MCS to reach steady state. The MCS reaches
steady state after 10 cardiac cycles, so the MCS runs 24 cardiac cycles
after each update at. The SAC controller operates as follows:
(1). The parameters Pne, Vacu, C and Rs are initially selected
randomly and entered into the MCS. The average values of the
AOPsys, AOPdia, Qmean of the last 10 cardiac cycles are selected as
the measured values, and the new state is obtained by making
differences with the patient’s target values.
(2). The SAC controller outputs at to be executed next according to the
current state.
(3). The at is processed and input to MCS to get the new state and
reward. The SAC controller outputs the next action to be per­
formed based on the new state.
range of AOPdia is [AOPsys-40, AOPsys-20] (mmHg), Qmean ranges from 20
to 50 (10-1L/min).
The process of patient-specific personalized simulation is shown in
Fig. 3(b). The simulated regulation process for each group of target
values is divided into initial adjustment and steady-state adjustment.
During the initial adjustment the SAC controller adjusts four parameters:
Pne, Vacu, C, Rs. The MCS runs for 24 cardiac cycles after each parameter
adjustment by the SAC controller. Since the adjustment of C takes a long
time (about 15 s), C is no longer changed when TOT is larger than 5
(according to the experimental statistics). TOT is the times of the
controller adjusting parameters when S2 is less than 4. S2 is the variance
of the measured values from the target values after each controller
adjusting parameters (equation (6)). To reduce the effect of environ­
mental interference, etc., the average of the last 10 cardiac cycles are
taken as the measured values. During steady-state adjustment C is fixed
and the SAC controller adjusts three parameters: Pne, Vacu, Rs. The
adjustment time of Pne, Vacu is around 250 ms. The adjustment time of
Rs is related to the adjustment angle of the electric ball valve (8 s for the
electric ball valve from fully closed to fully open), and is usually around
1 s in the steady-state adjustment phase (the deviations between the
T. Li et al. Biomedical Signal Processing and Control 78 (2022) 103987

Table 1 SAC controller adjusts Pne, Vacu and Rs.

Target values for several types of cardiovascular patients. Fig. 5 shows personalized simulations of aortic pressure and mean
Diseases HR AOPsys AOPdia Qmean flow curves for several typical cardiovascular patients in Table 1. (i) and
(bpm/min) (mmHg) (mmHg) (10-1L/min) (ii) show the aortic pressure curves after a smooth run; (iii) shows the
Healthy 75 120 80 50 mean aortic flow curves. The aortic pressure and mean flow for different
HEN 78 140 100 45 types of disease conditions are very close to the patients’ target values.
HEN + HF 80 135 105 25 Deviations of aortic systolic and diastolic pressures are within 2 mmHg.
HON + HF 90 73 48 22 The deviation of the mean aortic flow is within 0.2 L/min. The aortic
pressure in (ii) oscillates at the beginning of the drop, which is caused by
measured values and the target values in the steady-state adjustment the mechanical check valve.
phase are usually within 5, and the parameter adjustment ranges of the Table 2 shows the aortic pressure and flow ranges obtained after
controller output are small at this time). The controller adjusts the MCS experiments with conditions arbitrarily defined by the algorithm. The
parameters every 20 cardiac cycles if S2 exceeds 1. The measured values aortic systolic pressure ranges from 60 to 150 mmHg, the aortic diastolic
are averaged over 20 cardiac cycles. pressure ranges from 40 to 130 mmHg, and the average aortic flow
Clinical patient-specific data is processed to obtain the control target ranges from 1.5 to 5.5 L/min. In these simulations, the algorithm com­
values: AOPsys, AOPdia, Qmean. The initial parameters of the MCS are pletes parameters adjustment in about 10 min. The average deviations of
randomly generated to obtain the initial state. The SAC controller gen­ aortic systolic and diastolic pressures are within ± 2 mmHg, and the
erates the action according to the initial state, and the action processed average deviation of aortic flow is within ± 0.2 L/min. Due to the
is input to the MCS to obtain the new state. When TOT is larger than 5, continuous learning capability of the DRL, the longer the experimental
the value of C is considered to have reached the ideal state at this time, time is, the smaller the deviation of the measured value from the target
and the C is no longer adjusted. Then the SAC controller performs steady value.
state control of the MCS, checking every 20 cardiac cycles to see if the
specified deviation is exceeded, and if so, the controller tunes Pne, Vacu, 4. Discussion
and Rs.
Previous MCSs [14–17] focused on improving each component of the
3. Results system and aimed to be closer to the real human blood circulation sys­
tem. Because the system requires many parameters to be adjusted and
In order to verify the personalized simulation function of the the control law is complex, researchers have mainly obtained several
developed MCS, the target feature values of several typical cardiovas­ general simulation situations for testing CVDs by simply tuning each
cular patients are designed: Healthy, hypertension (HEN), hypertension parameter. This is time-consuming and inaccurate. Personalized MCS
with heart failure (HEN + HF), hypotension with heart failure (HON + can accurately simulate specific patient physiological conditions,
HF).The specific parameters are shown in Table 1. providing an accurate simulation environment for cardiovascular
The target feature values in Table 1 representing the four specific research and in vitro evaluation of CVDs. Balabani et al., [19] firstly
patients are entered into the system. After comparing the target values to proposed the concept of personalized MCS. Simulation of patient-
the measured values, the SAC controller sends control signals to the MCS specific vascular characteristics (aortic pressure and flow) was ach­
to adjust the Pne, Vacu, C and Rs. The aortic pressure and flow curves ieved by personalized adjustment of parameters such as resistance and
obtained for these cardiovascular patients are shown in Fig. 4 and Fig. 5. compliance. Due to the individual configuration required for system
Fig. 4 shows the simulated regulation process of aortic pressure and resistance and compliance, the time required to generate physical
mean blood flow in a normal person at rest. Initially, the SAC controller manifestations from patient-specific data varies from at least 4 days to
outputs four actions to regulate the MCS, and a large pressure drop in about 2–3 weeks. The maximum pressure deviation is 5 mmHg. Cestari
aortic pressure can be seen at each regulation due to the time required et al., [18] developed an automated algorithm to reproduce a wide range
for each regulation of compliance. C is fixed when TOT is larger than 5. of left atrial pressures and aortic pressures, with 99.1 % accuracy in
MCS enters steady-state adjustment phase. When S2 is greater than 1, the about 1 min. However, aortic flow needs to be determined according to

(a) (b)

Fig. 4. Simulated regulatory process curves of healthy adults in Table 1. Target systolic pressure is 120 mmHg; target diastolic pressure is 80 mmHg; target mean
flow is 50 × 10-1L/min. The blue dotted lines represent the target values and the red solid lines represent the measured values. (a): Aortic pressure. (b): Mean
aortic flow.

6
T. Li et al. Biomedical Signal Processing and Control 78 (2022) 103987

(i) Aortic pressure (ii) Magnified aortic pressure (iii) Mean aortic flow

(a)

(b)

(c)

(d)

Fig. 5. Aortic pressure and mean flow curves for different conditions (Table 1) in 20 cardiac cycles. The blue dotted lines represent the target values and the red solid
lines represent the measured values. (a) Healthy; (b) HEN; (c) HEN + HF; (d) HON + HF; (i) aortic pressure curve for 20 cardiac cycles at steady state; (ii) magnified
aortic pressure curve; (iii) mean aortic flow curve for 20 cardiac cycles at steady state.

the equation and cannot be simulated arbitrarily. This system can only
be used in infants under 1 year of age. Based on previous work, we have
developed a generalized personalized algorithm to solve the MIMO
regulation challenge of MCS. MCS combined with intelligent algorithms
performs well in personalized in vitro simulation. The developed
personalized MCS can reproduce aortic pressure and mean flow, with
parameters adjustment in approximately 10 min. The mean deviations
of aortic systolic and diastolic pressure are ± 2 mmHg and the mean
aortic flow deviation is within ± 0.2 L/min. The personalized simulation
algorithm based on DRL can be applied to other different types of MCSs.
Patient-specific personalized simulations benefit from the fully
automated MCS and the SAC algorithm with powerful characterization
and self-learning capabilities. The systolic-diastolic capacity of the
ventricle, aortic compliance and vascular resistance are the main pa­
rameters that affect MCS. The automatic program adjustment of these
main parameters provides the basis for personalized control algorithms.

7
T. Li et al.
Table 2
Systematic aortic pressure and flow range. The target values are arbitrarily
defined by the algorithm and the diastolic pressure is always lower than the
systolic pressure.
Target Range Deviation
AOPsys (mmHg) 60 ~ 150 ±2
AOPdia (mmHg) 40 ~ 130 ±2
Qmean (L/min) 1.5 ~ 5.5 ±0.2
HR (bpm/min) 50 ~ 110
Fig. 6. Aortic and left ventricular pressure simulation results for hypotensive
heart failure. AOP, aortic pressure; LVP, left ventricular pressure; LAP, left
atrial pressure.
The design of a suitable reward function is crucial to the SAC to learn
control strategy. This paper introduces the variance of the three
measured values and the target values as the denominator of the reward
function to ensure that the SAC intelligence learns the control law for
multiple objectives. The segmented reward term M is also introduced to
enhance the learning of the intelligences. By neural network fitting out
the policy network, the DRL intelligent agent trained with a large
amount of data is able to reproduce the aortic pressure and flow for the
specific patient. The personalized simulation capability of the system is
demonstrated by the simulation results (Fig. 5 and Table 2) for arbitrary
target pressure and flow. The MCS can reproduce specific aortic pressure
and mean flow characteristics over a wide range. In Fig. 4(a), the trained
SAC controller shows strong robustness when the MCS environment
changes during steady-state adjustment resulting in large fluctuations in
aortic pressure.
This paper uses a gas-driven silicone diaphragm to simulate the
function of the left ventricle, which results in a nonlinear process that
makes it difficult to precisely control the degree of each ventricular beat.
This is different from the precisely controllable piston pump used in
[19]. This brings a greater challenge for the precise control of the sys­
tem. Meanwhile, the control of the MIMO nonlinear system is complex.
The intelligent control algorithm (SAC) is introduced to study the con­
trol strategy of the system in different situations. The maximum entropy
mechanism ensures that the algorithm has sufficient exploration capa­
bility, which allows the SAC intelligence to learn how to adjust the
system parameters to approach the multiple objectives in different sit­
uations. The DRL algorithm provides a research idea for the complex
control of MCS.
Although SAC has many advantages, its training process requires a
8
Biomedical Signal Processing and Control 78 (2022) 103987
large amount of data and suitable reward function design, etc. This leads
to a time-consuming development process of the algorithm, which is an
issue that needs attention. In this paper, it takes at least 5 days for one
experimental training to observe whether the algorithm converges or
not. Longer experimental training is often required to obtain better
control accuracy. More efficient algorithms will be researched in the
future to reduce training time.
Fig. 6 shows an example of aortic, left ventricular and atrial pressures
(LAP) curves. The curves of AOP and LAP are comparable to the known
physiological ones. The LVP showed a negative pressure in diastole,
which is inconsistent with the physiological characteristics. This is
because the vacuum generator is always being pumped during diastole
(limited by the solenoid valve opening and closing frequency of 10 Hz).
The simulation of the ventricle will be improved in the future to make
the diastolic pressure of the ventricle compatible with physiological
values. The pulsed input of compressed air and evacuation results in a
steep rise and fall in left ventricular pressure. This is mainly limited by
the opening and closing frequency of the solenoid valve (10 Hz) and the
adjustment frequency of the pneumatic electric proportional valve (4
Hz).
5. Conclusion
In this paper, a personalized MCS based on DRL algorithm is pro­
posed. The designed MCS can be fully controlled automatically. The SAC
controller introduces variance into the reward function to achieve
cooperative control of multi-objectives and regulates multi-parameters
based on the difference between the measured values and the target
values. The personalized simulation capability of the system is demon­
strated by simulation results for arbitrary target pressure and flow. The
proposed MCS based on SAC can rapidly and accurately reproduce the
physiological conditions of a specific patient. DRL control method has
great potential for multi-objective control of MCS.
Personalized MCS offers numerous advantages. It can accurately
reproduce a wide range of physiological and pathological conditions,
providing a more comprehensive assessment for cardiovascular device
development. It can also be used for patient-specific hemodynamic
studies and personalized in vitro evaluation prior to implantation of
clinical CVDs.
In the future, the simulation of the ventricle will be further improved
so that the ventricular indexes are physiologically consistent. Further­
more, a personalized in vitro mock circulation system will be developed
in the systemic and pulmonary circulation.
CRediT authorship contribution statement
Te Li: Conceptualization, Methodology, Software, Validation,
Investigation, Data curation, Writing – original draft. Heng Li: Software,
Validation, Formal analysis, Data curation, Writing – original draft.
Wenbo Cui: Validation, Formal analysis. Nan Xie: Writing – original
draft. Xu Li: Writing – review & editing. Yongqing Wang: Conceptu­
alization, Methodology, Validation, Writing – review & editing, Super­
vision, Funding acquisition.
Declaration of Competing Interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
Acknowledgment
This work is partially supported by Fundamental Research Funds for
the Central Universities (Grant No. DUT22GF301), Changjiang Scholar
Program of Chinese Ministry of Education (Grant. NO. T2017030),
Dalian High-level talent innovation support program (Grant No.
T. Li et al.
2019RQ026).
References
[1] S.S. Virani, A. Alonso, E.J. Benjamin, M.S. Bittencourt, C.W. Callaway, A.P. Carson,
et al., Heart disease and stroke statistics—2020 update: A report from the american
heart association, Circulation 141 (2020).
[2] L.W. Miller, J.G. Rogers, Evolution of left ventricular assist device therapy for
advanced heart failure: A review, Jama Cardiol. 3 (2018) 650–658.
[3] F. Cappon, T. Wu, T. Papaioannou, X. Du, P.-L. Hsu, A.W. Khir, Mock circulatory
loops used for testing cardiac assist devices: A review of computational and
experimental models, The Int. J. Artificial Organs 44 (11) (2021) 793–806.
[4] K. Makinouchi, Y. Ohara, I. Sakuma, G. Damm, K. Mizuguchi, T. Jikuya, et al.,
Internal hydraulic loss in a seal-less centrifugal gyro pump, Artif. Organs 18 (1994)
25–31.
[5] M. Yoshino, M. Uemura, K. Takahashi, N. Watanabe, H. Hoshi, K. Ohuchi, et al.,
Design and evaluation of a single-pivot supported centrifugal blood pump, Artif.
Organs 25 (2001) 683–687.
[6] Y. Yokoyama, O. Kawaguchi, T. Shinshi, U. Steinseifer, S. Takatani, A new pulse
duplicator with a passive fill ventricle for analysis of cardiac dynamics, J. Artif.
Organs 13 (2010) 189–196.
[7] A.M. Leopaldi, R. Vismara, S. van Tuijl, A. Redaelli, F.N. van de Vosse, G.B. Fiore,
et al., A novel passive left heart platform for device testing and research, Med. Eng.
Phys. 37 (2015) 361–366.
[8] D. Timms, M. Hayne, K. McNeil, A. Galbraith, A complete mock circulation loop for
the evaluation of left right, and biventricular assist devices, Artif. Organs 29 (2005)
564–572.
[9] R. Zannoli, I. Corazza, A. Branzi, Mechanical simulator of the cardiovascular
system, Physica Medica 25 (2009) 94–100.
[10] Y. Liu, P. Allaire, H. Wood, D. Olsen, Design and initial testing of a mock human
circulatory loop for left ventricular assist device performance testing, Artif. Organs
29 (2005) 341–345.
[11] M. Vaes, M. Rutten, R. van de Molengraft, F. van de Vosse, Left ventricular assist
device evaluation with a model-controlled mock circulation, ASME Summer
Bioengineering Conference 2007 (2007) 723–724.
[12] J. Fredrick Cornhill, An aortic-left ventricular pulse duplicator used in testing
prosthetic aortic heart valves, The Journal of Thoracic and Cardiovascular Surgery
73 (1977) 550–558.
[13] G. Ochsner, R. Amacher, A. Amstutz, A. Plass, M. Schmid Daners, H. Tevaearai, et
al., A novel interface for hybrid mock circulations to evaluate ventricular assist
devices, IEEE T Bio-Med. Eng. 60 (2013) 507–516.
Biomedical Signal Processing and Control 78 (2022) 103987
[14] A. Packy, G.A. D’Souza, M. Farahmand, L. Herbertson, C.G. Scully, Simulating
radial pressure waveforms with a mock circulatory flow loop to characterize
hemodynamic monitoring systems, Cardiovasc. Eng. Techn. 13 (2) (2022)
279–290.
[15] J. Gehron, J. Zirbes, M. Bongert, S. Schäfer, M. Fiebich, G. Krombach, A. Böning,
P. Grieshaber, Development and validation of a life-sized mock circulatory loop of
the human circulation for fluid-mechanical studies, ASAIO J. 65 (8) (2019)
788–797.
[16] A. Petrou, M. Granegger, M. Meboldt, M. Schmid Daners, A versatile hybrid mock
circulation for hydraulic investigations of active and passive cardiovascular
implants, ASAIO Journal (American Society for Artificial Internal Organs 65 (2019)
(1992) 495–502.
[17] S.D. Gregory, J.P. Pauls, E.L. Wu, A. Stephens, U. Steinseifer, G. Tansley, et al., An
advanced mock circulation loop for in vitro cardiovascular device evaluation, Artif.
Organs 44 (2020) E238–E250.
[18] D.S. Torres, M. Mazzetto, I.A. Cestari, A novel automated simulator of pediatric
systemic circulation: Design and applications, Biomed. Signal Proces 70 (2021)
102926.
[19] G. Franzetti, V. Díaz-Zuccarini, S. Balabani, Design of an in vitro mock circulatory
loop to reproduce patient-specific vascular conditions: toward precision medicine,
J. Eng. Sci. Med. Diagnostics Therapy 2 (2019).
[20] S.S. Mousavi, M. Schukat, E. Howley, Deep reinforcement learning: an overview,
in: Y. Bi, S. Kapoor, R. Bhatia (Eds.), Springer International Publishing, Cham,
2018, pp. 426–440.
[21] N. Liu, Y. Cai, T. Lu, R. Wang, S. Wang, Real–sim–real transfer for real-world robot
control policy learning with deep reinforcement learning, Appl. Sci. (2020).
[22] A. R. Fayjie, S. Hossain, D. Oualid, D. Lee, Driverless Car: Autonomous Driving
Using Deep Reinforcement Learning in Urban Environment, 2018 15th
International Conference on Ubiquitous Robots (UR), 2018, pp. 896-901.
[23] Y. Zheng, X. Xie, T. Su, L. Ma, J. Hao, Z. Meng, et al., Wuji: Automatic Online
Combat Game Testing Using Evolutionary Deep Reinforcement Learning, 2019
34th IEEE/ACM International Conference on Automated. Software Engineering
(ASE), 2019, pp. 772-784.
[24] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, in: Soft Actor-Critic: Off-Policy
Maximum Entropy Deep Reinforcement Learning With a Stochastic Actor, 2018,
pp. 1861–1870.
[25] S. Schampaert, K.A.M.A. Pennings, M.J.G. van de Molengraft, N.H.J. Pijls, F.N. van
de Vosse, M.C.M. Rutten, A mock circulation model for cardiovascular device
evaluation, Physiol. Meas 35 (2014) 687–702.