See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/375375521

LfDT: Learning Dual-Arm Manipulation from Demonstration Translated from a

Human and Robotic Arm

Conference Paper · December 2023

CITATIONS READS

0 21

4 authors, including:

Masato Kobayashi Masashi Hamaya

Osaka University OMRON SINIC X Corporation, Japan, Tokyo
16 PUBLICATIONS 61 CITATIONS 36 PUBLICATIONS 358 CITATIONS

SEE PROFILE SEE PROFILE

Kazutoshi Tanaka
OMRON SINIC X Corporation
55 PUBLICATIONS 251 CITATIONS

SEE PROFILE

All content following this page was uploaded by Kazutoshi Tanaka on 06 November 2023.

The user has requested enhancement of the downloaded file.

© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any
current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating
new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in
other works.

Published article:
M. Kobayashi, J. Yamada, M. Hamaya, K. Tanaka, “LfDT: Learning Dual-Arm Manipulation from Demonstration Translated
from a Human and Robotic Arm,” The 2023 IEEE-RAS International Conference on Humanoid Robots (Humanoids 2023),
2023.
 LfDT: Learning Dual-Arm Manipulation from Demonstration
Translated from a Human and Robotic Arm
Masato Kobayashi∗1 , Jun Yamada∗2 , Masashi Hamaya3 , Kazutoshi Tanaka3

Abstract— Imitation learning (IL) is a promising method Robot1 Human Human Robot2
for programming dual-arm manipulation easily by imitating
demonstrations from human experts. However, IL for dual-
arm manipulation is still challenging because operating two
robotic arms to collect demonstrations requires considerable
effort. Therefore, we present a novel IL framework for dual-arm
manipulation: learning dual-arm manipulation from demon-
stration translated from a human and robotic arm (LfDT). Human-Robot2
Robot1-Human
LfDT collects demonstrations of one human and one robotic demonstration
demonstration
arm. Thus, a human expert can easily and precisely adjust its
arm movements according to the movement of the robotic arm.
Domain translation framework
LfDT collects demonstrations of one human and one robotic
arm, whereas IL methods typically demand demonstrations
of two robotic arms. Therefore, LfDT employs a domain- Robot1-Robot2 Robot1-Robot2
translation network to convert the demonstrations of one demonstration demonstration
human and one robotic arm into demonstrations of two robotic
arms, which are then used to learn dual-arm manipulation via Imitation learning
IL. The experiments demonstrate that LfDT successfully con-
verts the demonstrations and learns the dual-arm manipulation Robot1-Robot2 policy
in both simulation and real-world. Robot1 Robot2
I. I NTRODUCTION
Many real-world tasks such as folding laundry [1], kitchen
support [2], and industrial assembly [3] require dual-arm
manipulation. Dual-arm manipulation often requires two
robotic arms to precisely adjust their movements relative to
each other for cooperation. For example, as demonstrated in
Fig. 1, for a 2D-pushing task, one arm must slide its end- Fig. 1. Overview of LfDT. LfDT collects demonstrations of dual-
effector synchronously with the other arm to cooperatively arm manipulations performed by one human arm of an expert and one
move the object to a target position. If one of the arms robotic arm operated by the other expert. Unlike other methods that collect
demonstrations of two robotic arms operated together by one expert, LfDT
slides its end-effector too quickly or slowly, it is likely that can easily collect the demonstrations because the human expert can easily
the object will rotate in the same position, and the arms and precisely adjust its arm movements according to the movement of the
will fail to complete the task. While dual-arm manipulation robotic arm to cooperate with these arms.
is essential to broaden the scope of robotic applications in
real-world, manually programming dual-arm manipulation is
often time-consuming due to the required precise cooperation requires precise coordination of two robotic arm using
of the arms. passive observation. Kinesthetic teaching also collects the
Imitation learning (IL) is a core research topic for learn- demonstrations by physical guidance of an expert for a
ing robotic manipulation from demonstrations collected by robotic arm [7], [8], [9], [10]. However, IL for dual-arm
human experts [4], [5] instead of manually programming the manipulations is still challenging because operating two
manipulation. Passive observation collects such demonstra- robotic arms to collect demonstrations requires considerable
tions by observing demonstrations of a human expert [6]. effort. Teleoperation allows an expert to operate two robotic
However, it is difficult to learn dual-arm manipulation which arms to collect demonstrations of dual-arm manipulations
∗ Equal contribution. intuitively; however, it requires a specific device [4].
1 Osaka University kobayashi.masato.cmc@osaka-u.ac. Operating multiple robotic arms together by a human ex-
jp, work done during internship at OMRON SINIC X Corporation pert to collect demonstrations requires considerable cognitive
2 University of Oxford jyamada@robots.ox.ac.uk, work done
during internship at OMRON SINIC X Corporation effort. Therefore, Tung et al. presented an IL framework, in
3 OMRON SINIC X Corporation, Hongo 5-24-5, Bunkyo-ku, Tokyo,
which multiple human experts operated multiple robotic arms
Japan {masashi.hamaya, kazutoshi.tanaka}@sinicx to collect demonstrations, and each expert easily operated
.com
† This work was supported by JST, PRESTO Grant Number JP- one robotic arm using a smartphone as a six degrees-of-
MJPR22C6, Japan. freedom (DoF) motion controller [11]. However, it is difficult
to precisely adjust the movements of a robotic arm according
to the movement of the other arms using a smartphone.
For these reasons, we aim to present a novel IL framework,
learning dual-arm manipulation from demonstration trans-
lated from a human and robotic arm (LfDT) to easily collect
demonstrations of dual-arm manipulation. This framework
collects demonstrations of one human and one robotic arm as
illustrated in Fig. 1. A human can intuitively and easily adjust
the movement of its arm to cooperate with the other entity,
even a robotic arm, for manipulation tasks. Owing to the
adjustment of a human arm, the other expert does not have
to operate the robotic arm precisely. Thus, leveraging the
human arm enables these experts to collect demonstrations
of dual-arm manipulation easily.
Although IL methods, such as behavior cloning (BC),
typically demand demonstrations of all robotic arms, LfDT
collects demonstrations of one human and one robotic arm.
Generally, demonstrations by a human arm cannot be repli-
cated directly by a robotic arm due to the physical differences
between the human and robotic arm, such as range of motion,
joint velocity, and joint torque. Therefore, LfDT employs a
domain-translation framework to convert demonstrations of
one human and one robotic arm into demonstrations of two
robotic arms and then learns the dual-arm manipulation by
imitating the converted demonstrations with BC.
The contributions of our work are three-fold: (1) We
propose LfDT, an IL framework for learning dual-arm ma-
nipulation from demonstrations translated from a human
and robotic arm; (2) We introduce a domain-translation
framework that converts the demonstrations of one human
and one robotic arm into demonstrations of two robotic
arms; (3) We empirically demonstrate that LfDT successfully
learns both simulated and real-world dual-arm manipulation.
II. R ELATED WORKS
A. Learning dual-arm manipulation
Dual-arm manipulation requires two robotic arms to co-
operate by precisely adjusting their movements relative to
each other. Therefore, compared to single-arm manipulation,
dual-arm manipulation requires more advanced planning and
control approaches and necessitates substantial engineering
efforts [3]. To mitigate this issue, reinforcement learning
(RL) has been applied to robots for learning dual-arm
manipulation [12], [13], [14]. However, RL for dual-arm
manipulation requires a large number of samples collected
via interaction with an environment, limiting its applications
in the real-world. Consequently, we employ IL for dual-arm
manipulation, which is applicable to robotic tasks in the real-
world.
B. Imitation learning
IL enables a robotic arm to learn manipulation by imitating
the behavior of human experts in demonstrations [15], [16].
One mainstream of IL is BC [4], [17], which uses supervised
learning to learn a function that maps an observation to an
action from demonstrations of experts. Another mainstream
of IL is inverse RL (IRL) [18], [19], [20], which estimates
a reward function from the demonstrations. In contrast to
IRL, BC does not require any additional trials for training
the manipulation. Therefore, in this study, LfDT uses BC,
and it is potentially applicable to all IL methods.
Demonstrations in IL are often collected by teleoperation
using devices such as virtual reality headsets and hand-
tracking systems [4], smartphones [11], 3D motion con-
trollers [21], and keyboards [22]. Prior studies have mostly
tackled single-arm manipulation, and only a few studies
have applied IL to dual-arm manipulation because operating
two robotic arms to collect the demonstrations requires
considerable effort.
Laghi et al. introduced a system that can operate two
robotic arms using one arm of a human expert [23]. Some
studies have introduced bimanual robot teaching systems
using inertial measurement unit (IMU) and electromyogra-
phy (EMG) signals [23] as well as exoskeletons [24]. These
frameworks require a specific device to operate two robotic
arms to collect demonstrations therefrom.
Tung et al. built a system for collecting demonstrations of
multi-arm manipulation by multiple human experts [11]. In
their system, each human expert collected demonstrations by
operating one robotic arm using a smartphone. In the system,
it is difficult for an expert to precisely adjust the movement of
a robotic arm using a smartphone, whereas the other expert
using LfDT can easily and precisely adjust its arm movement
according to the movements of the other robotic arm.
C. Domain transfer
Domain transfer in RL and IL has gained attention for their
generalization across tasks [25]. Prior studies have learned
state-translation mapping by defining a common state space
between two tasks, and have employed unsupervised man-
ifold alignments using hand-crafted features [26]. Several
prior studies have suggested leveraging a set of proxy tasks
to learn the state correspondence or domain-invariant space
for mismatches in morphology [27] and viewpoint [28],
[29]. Moreover, a recent study has proposed learning-state
correspondence from demonstrations of proxy tasks without
access to expert actions [30]. Although domain transfer has
been used for generalization across tasks, we use domain
transfer to convert demonstrations of one human and one
robotic arm into demonstrations of two robotic arms to learn
dual-arm manipulation.
Cycle-consistency is an essential concept in domain-
translation networks [31], [32] to learn bidirectional trans-
lation between different domains. CycleGAN [31] intro-
duced cycle-consistency, which enables bidirectional un-
paired image-to-image translation with generative adversarial
networks [33]. Subsequently, cycle-consistency was extended
for video translation [34] and domain adaptation [35]. Dy-
namic cycles have proposed cycle-consistency for dynam-
ics [32], which have used a forward dynamics model to
convert states and actions from one domain into those in
the other domain. LfDT also uses the cycle-consistency for
dynamics to convert demonstrations of one human and one
robotic arm into demonstrations of two robotic arms.
(1) Robot1-Human Human-Robot2 of the robotic arm xRi (i = 1, 2), the state of an object
demonstration demonstration
{(xR1, xH, xO, aR1)} {(xH, xR2, xO, aR2)} xO , and the action of the robotic arm aRi . The generalized
coordinate of a robot, joint angles, the pose of the end
(2) effector, and visual image can be used for the states, and
Domain translation framework
motor commands and the target pose of the end effector can
Human-Robot (domain X) Robot-Robot (domain Y)
G
be used as the action. Y contains the states of two robotic
xt ŷt yt arms and that of the object, y R1 , y R2 , y O , and actions of
x t⊂
Ri
ŷRit⊂ D

Lid: Identity mapping loss

Ladv: Adversarial loss the arms aR1 , aR2 . We define the states in X and Y as
F-1 x = (xH , xRi , xO ) and y = (y R1 , y R2 , y O ), and the action
ãt
F in Y as a = (aR1 , aR2 ). We define MDPs in X and Y
G : Generator
ỹt+1 as Mx = (Sx , Ax , Px , ρ0x ) and My = (Sy , Ay , Py , ρ0y ),
D : Discriminator
Ldyn: Dynamics cycle loss respectively. Our goal is to learn a policy π : Sy 7−→ Ay for
F: Forward dynamics
xt+1 G ŷt+1 dual-arm manipulation. Actually, LfDT learns π using BC
F-1: Inverse dynamics
minimizing E[||a − π(y)||2 ].

Robot1-Robot2 Robot1-Robot2
demonstration demonstration
{(yR1, yR2, yO, aR1, aR2)} {(yR1, yR2, yO, aR1, aR2)}
(3)
Imitation learning
(yR1, yR2, yO)
Robot1-Robot2 policy π
(aR1, aR2)
Fig. 2. Framework overview. Illustration of the overall LfDT procedure.
(1) Demonstration: LfDT collects demonstrations of one human and one
robotic arm. A human expert first operates Robot1 to manipulate an
object and then operates Robot2 to manipulate this object. (2) Translation:
Subsequently, LfDT trains models of the domain-translation framework to
convert the collected demonstrations of the human and robotic arm into
demonstrations of two robotic arms. (3) Imitation: Finally, LfDT learns a
control policy with BC, using the converted demonstrations.
III. A PPROACH
In this section, we describe our approach, LfDT in detail.
Fig. 2 shows an overview of our approach. While IL methods
generally require demonstrations of all robotic arms for
dual-arm manipulation, LfDT collects demonstrations of one
human and one robotic arm. Then, given such demon-
strations, LfDT employs a domain-translation framework
to translate demonstrations of one human and one robotic
arm into demonstrations of two robotic arms to learn dual-
arm manipulation from the converted demonstrations using
BC. This section consists of the problem formulation, data
collection, and domain-translation framework.
A. Problem Formulation
We define the problem as a Markov decision process
(MDP) formulated as a tuple (S, A, P, ρ0 ) consisting of
states s ∈ S, actions a ∈ A, transition function P (s′ ∈
S|s, a), and initial state distribution ρ0 .
As mentioned above, LfDT translates demonstrations of
one human and one robotic arm into demonstrations of
two robotic arms. We define two domains to contain raw
and translated states: domain X for demonstrations of the
human and robotic arm; and the target domain, domain Y
for deployment of dual-arm manipulation by two robotic
arms. X contains the state of the human arm xH , the state
B. Data collection
LfDT collects demonstrations of cooperation between the
human arm and Robot1 as well as between the human arm
and Robot2, as illustrated in Fig. 1. Collecting both demon-
strations of a human as Robot1 and Robot2 helps to consider
the different roles of these robots in the manipulation. As
mentioned before, a human expert can intuitively manipulate
an object by hand and precisely adjust its movement in
response to the movement of the robotic arm. Thus, we can
collect demonstrations more easily compared to operating
two robotic arms simultaneously, which would require an
operating expert considerable cognitive effort.
LfDT also collects random motion data in the domains of
X and Y to train a generator G, a discriminator D, a forward
dynamics model F , and an inverse dynamics model F −1 as
described in Sec. III-C. Random action commands are sent
to move the robotic arms and the resulting movement data
is recorded.
C. Domain-translation framework
LfDT does not collect any demonstrations of two robotic
arms in the domain Y directly. That is, there is no paired
data of the state of two robotic arms y and the state of a
human and robotic arm x. Thus, we cannot apply supervised
learning to learn a function that maps x to y. Therefore,
LfDT converts demonstrations of the human and robotic arm
in X into demonstrations of two robotic arms in Y using
a domain-translation framework learned from unpaired data
in domains X and Y, as illustrated in Fig. 2. Then, we
employ IL, specifically BC, to learn a policy π for dual-
arm manipulation from the converted demonstrations of the
two robotic arms.
1) Domain-translation: To convert demonstrations of one
human and one robotic arm into demonstrations of two
robotic arms, we train a state translation function that maps
states xt ∈ X to states yt ∈ Y. Through adversarial train-
ing [33], a generator G learns to map x onto the distribution
of y as ŷ to deceive a discriminator D. Meanwhile, D is
trained to distinguish between generated ŷt and real yt input
samples:

min max Ladv (G, D) = Ey∼p(y) [log D(y)]+
G D
Ex∼p(x) [log(1 − D(G(x)))],
where p(∗) is a distribution over data ∗.
In addition to demonstrations in the domain X and random
motion data in Y, random motion data in X involving
states of one human and one robotic arm are used to
avoid overfitting G. Since we collect demonstrations and
random motion data of Robot1-Human and Human-Robot2
respectively, we concatenate the states xt ∈ X with a one-hot
vector representing either Robot1-Human or Human-Robot2
to construct an input to G.
2) Dynamics-consistency: If LfDT employs the domain-
translation without any constraints, the converted state ŷ may
not be consistent with the state transition PY . Therefore, to
make ŷt and ŷt+1 consistent with PY , we apply dynamics-
consistency loss inspired by a previous study [32].
Specifically, if translated states ŷt (= G(xt )) and ŷt+1 (=
G(xt+1 )) are consistent, then, ideally, the state ŷt+1 would
be the same as the ỹt+1 predicted by the forward dynamics
model using the input ŷt and ãt such that ỹt+1 = F (ŷt , ãt ),
where ãt is an action predicted by the inverse dynamics
model given ŷt and ŷt+1 such that ãt = F −1 (ŷt , ŷt+1 ).
Thus, to train G with consistent dynamics, LfDT minimizes
the objective:
Ldyn =E[||ỹt+1 − ŷt+1 ||2 ]
=E[||F (ŷt , ãt ) − ŷt+1 ||2 ]
−1
=E[||F (ŷt , F (ŷt , ŷt+1 )) − ŷt+1 ||2 ].
The models F and F −1 are trained using the following
objectives:
min Lfwd (F ) = E[||yt+1 − F (yt , at )||2 ],
F A. Simulated experiment setup
−1 −1
min
−1
Linv (F ) = E[||at − F (yt , yt+1 )||2 ].
F
3) Partial identity mapping: In addition to the dynamics-
consistency constraint, we also introduce a partial identity
mapping that partially constraints the generator. In particular,
since the state xt in the domain X includes the states of both
the human and robotic arm, the converted state of Robot{i},
ŷtRi (⊂ ŷt ) must have values closed to the raw state of the
same arm xRi t (⊂ xt ) in demonstrations. Thus, we add the
following objective function to constrain G:
min Lid (G) = E[||ŷ Ri − xRi ||2 ].
G
Note that ŷ Ri and xRi have the same state types of Robot{i}.
4) Full objective: In summary, our full objective is as
follows:
Lfull = λadv Ladv (G, D) + λdyn Ldyn (G) + λid Lid (G), (6)
where λadv , λdyn , and λid are coefficients for balancing the
loss terms. LfDT learns F and F −1 from the random motion
data in Y before training G and D.
Demonstration Deployment
(a) 2D dual-arm box push
Robot1 Human Human Robot2 Robot1 Robot2
(1)
Goal
(b) Dual-arm block push
Robot1 Human Human Robot2 Robot1 Robot2
Goal
(c) Dual-arm peg insertion
Robot2 Robot2
Robot1 Human Robot1
Human
Fig. 3. Demonstrations of the human and robotic arm and dual-
arm manipulation. The left and middle columns show demonstrations of
the human and robotic arm. (a) two-dimensional (2D) dual-arm box push:
Robot1 and Robot2 have three links, and the green box needs to be pushed
by the two arms towards a black rectangle goal position. We collected
demonstrations of either Robot1 or Robot2 and a robotic arm with four
links to represent physical differences between the human and robotic arm.
(b) Dual-arm block push: Two robotic arms need to push the elongated
block onto the green line. (c) Dual-arm peg insertion: The peg attached to
Robot2 must be inserted into the hole attached to Robot1.
IV. E XPERIMENTS
(2) In this section, we evaluate LfDT to verify whether LfDT
successfully converts demonstrations of one human and
one robotic arm into demonstrations of two robotic arms
and learns dual-arm manipulation by employing IL on the
converted demonstration in both simulated and real-world
experiments.
(3)
We first evaluated LfDT for “2D dual-arm box push”
(4)
simulated using MuJoCo physics engine [36] as shown in
Fig. 3(a). In this task, Robot1 and Robot2 need to cooper-
atively push a box to a goal while avoiding the rotation of
the box. Robot1 and Robot2 consist of three links in the
domain Y, whereas one of the robots that demonstrate the
task consists of four links to represent physical differences
between the human and robotic arm. The states xRi and
y Ri include the joint angles, joint angular velocity, and end
position of the robots, and xO and y O contain the position
and orientation of the box. The robots are controlled by
(5) applying the joint torque of the robots defined as aRi .
In this experiment, we added random noise to the ini-
tial joint angles of both arms and the initial position and
orientation of the box. The task is considered successful
when the distance between the center of the box and the
goal position and the rotation of the box are smaller than
their thresholds. The demonstrations were collected from
expert policies trained by a soft actor-critic (SAC) [37]. In
the simulated experiment, we collected 0.4M samples of
random motion data for each domain X and Y. In addition,
we also collected 1.6K samples which contain 200 expert
demonstrations of a human and Robot1, and those of the
human and Robot2, respectively.
B. Training details
F and F −1 consist of four fully connected layers of 256
hidden units with leaky ReLU activation functions. G and
D comprise three fully connected layers of 256 hidden units
with leaky ReLU nonlinearity. We used a learning rate of
0.0003, Adam optimizer, and a batch size of 32 for all
experiments.
C. Baselines
To leverage IL using demonstrations of one human and one
robotic arm, the physical differences between the human and
robotic arm require a method to convert the demonstrations
into demonstrations of two robotic arms. However, to the best
of our knowledge, no other method can be used to convert
them except for LfDT. Thus, we compare LfDT against
three ablated baselines, BC without translation, LfDT with-
out dynamics-consistency, and LfDT without partial identity
mapping.
BC without translation learns a policy π R1 and the other
policy π R2 such that aR1 = π R1 (sR1 , sO ) and aR2 =
π R2 (sR2 , sO ) from {(xR1 , xO , aR1 )} and {(xR2 , xO , aR2 )},
respectively. These two policies are separately trained with-
out our domain-translation framework to perform the task.
Since demonstrations of Robot1-Human and Human-Robot2
are independent of each other, we hypothesize that the
policies π R1 and π R2 struggle to learn coordinated behavior.
If BC without translation does not perform well but LfDT
successfully learns the task, it implies that the translation
of the demonstrations in LfDT helps IL learn coordinated
behavior in the dual-arm manipulation.
LfDT without dynamics-consistency and LfDT without
partial identity mapping first convert demonstrations of one
human and one robotic arm into demonstrations of robotic
arms similar to LfDT. However, we remove the dynamics-
consistency or partial identity constraint in the domain-
translation network from LfDT, respectively, to verify the
importance of these constraints.
In the simulated experiment, we evaluate methods based
on the task success rate. Across all methods, we train the
BC policies using 90% of the demonstrations for training
and 10% for validation, and we evaluate the policies when
the validation loss is minimum.
#demonstrations 50 100 200
BC w/o translation 28.3% 6.7% 26.7%
LfDT w/o dynamics-consistency 51.7% 61.7% 51.7%
LfDT w/o partial identity 0% 0% 0%
LfDT 80.0% 81.7% 71.7%
TABLE I
S UCCESS RATES OF 2D DUAL - ARM BOX PUSH .
D. Simulation results
As shown in Tab. I, LfDT outperforms ablated baselines by
a significant margin. This table reports the success rate of our
method and ablated baselines out of 20 trials averaged over
three different seeds. This result implies that our domain-
translation framework successfully translates demonstrations
of one human and one robotic arm into demonstrations of the
two robotic arms in a consistent manner with the dynamics
of the domain of the robotic arms.
On the other hand, BC without translation often fails the
task because the policy cannot learn coordinated behavior by
imitating demonstrations independent of Robot1 and Robot2.
We also ablate our method, LfDT, on dynamics-consistency
and partial identity mapping to verify the importance of these
constraints. The success rate of LfDT without dynamics-
consistency significantly drops, indicating that the dynamics-
consistency constraint contributes to translating dynamically
feasible demonstrations of Robot1 and Robot2. Furthermore,
LfDT without partial identity mapping completely fails the
task, suggesting that the partial identity mapping constraint
for enabling the generator to output states of the robot close
to those in the demonstration is an essential component.
E. Real robot experiment
In this section, we verify whether LfDT can work for a
real-robot setup. We use two robotic arms (myCobot, Ele-
phant Robotics), small-scaled 6DoF robotic arms, for “Dual-
arm block push” and “Dual-arm peg insertion” experiments,
as shown in Fig. 3. The two robotic arms push a block onto
the green line in Dual-arm block push. Robot2 inserts a peg
attached to Robot2 into the hole attached to Robot1 in Dual-
arm peg insertion. The hand of a human expert and the end
effectors of the robotic arms moves in the horizontal plane in
Dual-arm block push and in the vertical plane in Dual-arm
peg insertion, respectively.
We set the position of the end of the robotic arms as xRi
and y Ri , the position of the hand of the human expert as
xH , the position and orientation of the object as xO and
y O , and the target position of the end effector as aRi . We
expected that LfDT translated demonstrations considering
the difference between the pose of the object in the human
hand and that in the robotic gripper. The robotic arms
received the target joint angles, which were calculated from
the target position using inverse kinematics, and moved their
joints toward the target angles. To observe the pose of the
human hand and object, two markers were attached to both
the hand and object, and they were tracked using a camera
(RealSense D435, Intel) and AprilTag [38]. One human ex-
pert collects random motion data and demonstrations. When
collecting demonstrations, the robotic arm replays a motion
programmed in advance instead of real-time teleoperation.
We collect 3300 random motion data samples in X and Y
as well as 390 demonstration data samples which contain ten
demonstrations of Robot1-Human and ten demonstrations of
Human-Robot2 in X in Dual-arm block push and Dual-arm
peg insertion. We evaluate the effectiveness of LfDT by the
task success rate over ten trials. We compare LfDT and BC
!.),/)(#01,$230+.&4

Demonstration
!"#$%&'()'*$%

Deployment
!"+,$-#"%'

(a) Dual-arm block push

!.),/)(#0+"50*%&"('*$%

!"#$%&'()'*$%
Demonstration

Deployment
!"+,$-#"%'

(b) Dual-arm peg insertion

Fig. 4. Snapshots of demonstrations and the deployment of (a) dual-arm block push and (b) dual-arm peg insertion. LfDT successfully learns
and executes the manipulation of real robots from the human-arm and robotic-arm demonstrations.

without translation to evaluate effects of the state translation.
The models are trained using the same parameters of the
simulations as described in Sec. IV-B.
Fig. 4 shows the snapshots of demonstrations of the human
and robotic arm as well as the deployment of the tasks
by the robotic arms. Finally, LfDT shows 100%(= 10/10)
and 80%(= 8/10) success rates for Dual-arm block push
and Dual-arm peg insertion, respectively, while BC without
translation showed 100%(= 10/10) and 20%(= 2/10)
success rates for Dual-arm block push and Dual-arm peg
insertion, respectively. These results indicate that LfDT
works in a real robot setup.
V. D ISCUSSION
In simulations, Robot1 and Robot2, each with three links,
learned 2D dual-arm box task from demonstrations, in which
one of the arms had four links, using LfDT. That is, by
leveraging LfDT, robotic arms can learn dual-arm manipula-
tion from demonstrations of arms with physical differences,
such as human arms. Based on the successful results in the
simulated experiments, we further apply our method for real-
world dual-arm manipulation. As described in Section IV-E,
LfDT successfully learns Dual-arm block push and Dual-
arm peg insertion tasks by converting demonstrations of
one human and one robotic arm into demonstrations of two
robotic arms. The dual-arm using LfDT succeeded the tasks
by coordination of their movements, which the tasks require.
LfDT can translate demonstrations of a human source arm
into ones of a target robotic arm with a different morphology
from the source arm without the action of the arm. Therefore,
LfDT can also translate demonstrations of a source robot
arm, such as a different serial-link arm, a parallel-link
arm, and a continuum robot, into the target serial-link arm.
Furthermore, LfDT can be potentially extended to multi-
arm manipulation by a larger number of robotic arms than
two, in which LfDT would significantly reduce the effort of
demonstration collection.
While LfDT successfully converts demonstrations of the
human and robotic arm into demonstrations of two robotic
arms, it leverages a large size of random motion data
to generate an accurate forward dynamics model and an
accurate inverse dynamics model. These dynamics models
enable LfDT to translate the demonstrations satisfying the
dynamics consistency and to improve its performance as
shown in Tab. I. Also, LfDT learns quasi-static manipulation
in this study. Using a parametric and non-parametric hybrid
model [39], online adaptation [40], transfer learning [41],
meta-learning [42] and continuous learning [43] enable to
train these models using a smaller size of random motion
data. Therefore, in future work, we will combine LfDT with
these methods to learn more complex manipulations, such as
dual-arm object grasping and dynamic manipulation.
VI. C ONCLUSIONS
In this study, we present LfDT, a novel IL framework
for dual-arm manipulation, which collects demonstrations
of one human and one robotic arm and converts these
demonstrations into demonstrations of two robotic arms us-
ing a domain-translation framework. LfDT outperforms other
ablated baselines in the simulated experiments. Furthermore,
we demonstrate that LfDT works in real-robot experiments.
Future work should focus on methods to efficiently train a
dynamics model and an inverse dynamics model for complex
dual-arm manipulation.
ACKNOWLEDGMENT
We would like to thank Motoi Laboratory from Kobe Uni-
versity for allowing us to use their facilities for conducting
the experiments.
R EFERENCES
[1] H. Ha and S. Song, “Flingbot: The unreasonable effectiveness of
dynamic manipulation for cloth unfolding,” in Conference on Robot
Learning, 2022, pp. 24–33.
[2] M. Bollini, S. Tellex, T. Thompson, N. Roy, and D. Rus, “Interpreting
and executing recipes with a cooking robot,” in Experimental Robotics.
Springer, 2013, pp. 481–495.
[3] C. Smith, Y. Karayiannidis, L. Nalpantidis, X. Gratal, P. Qi, D. V.
Dimarogonas, and D. Kragic, “Dual arm manipulation—A survey,”
Robotics and Autonomous Systems, vol. 60, no. 10, pp. 1340–1353,
2012.
[4] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and
P. Abbeel, “Deep imitation learning for complex manipulation tasks
from virtual reality teleoperation,” in IEEE International Conference
on Robotics and Automation, 2018, pp. 5628–5635.
[5] Z. Zhu and H. Hu, “Robot learning from demonstration in robotic
assembly: A survey,” Robotics, vol. 7, no. 2, 2018.
[6] R. Caccavale, M. Saveriano, G. A. Fontanelli, F. Ficuciello, D. Lee,
and A. Finzi, “Imitation learning and attentional supervision of dual-
arm structured tasks,” in Joint IEEE International Conference on
Development and Learning and Epigenetic Robotics. IEEE, 2017,
pp. 66–71.
[7] B. Akgün, M. Cakmak, K. Jiang, and A. Thomaz, “Keyframe-
based learning from demonstration,” International Journal of Social
Robotics, vol. 4, pp. 343–355, 2012.
[8] G. Ye and R. Alterovitz, “Guided motion planning,” Robotics research,
pp. 291–307, 2017.
[9] E. Gribovskaya and A. Billard, “Combining dynamical systems control
and programmingby demonstration for teaching discrete bimanual
coordination tasks to a humanoid robot,” in ACM/IEEE International
Conference on Human Robot Interaction, 2008, pp. 33–40.
[10] G. Franzese, L. de Souza Rosa, T. Verburg, L. Peternel, and J. Kober,
“Interactive imitation learning of bimanual movement primitives,”
IEEE/ASME Transactions on Mechatronics, 2023.
[11] A. Tung, J. Wong, A. Mandlekar, R. Martı́n-Martı́n, Y. Zhu, L. Fei-Fei,
and S. Savarese, “Learning multi-arm manipulation through collabo-
rative teleoperation,” in IEEE International Conference on Robotics
and Automation, 2021.
[12] R. Chitnis, S. Tulsiani, S. Gupta, and A. Gupta, “Intrinsic motivation
for encouraging synergistic behavior,” in International Conference on
Learning Representations, 2020.
[13] R. Chitnis, S. Tulsiani, S. Gupta, and A. K. Gupta, “Efficient bimanual
manipulation using learned task schemas,” IEEE International Con-
ference on Robotics and Automation, pp. 1149–1155, 2019.
[14] F. Amadio, A. Colomé, and C. Torras, “Exploiting symmetries in
reinforcement learning of bimanual robotic tasks,” IEEE Robotics and
Automation Letters, vol. 4, no. 2, pp. 1838–1845, 2019.
[15] S. Schaal, “Is imitation learning the route to humanoid robots?” Trends
in Cognitive Sciences, vol. 3, no. 6, pp. 233–242, 1999.
[16] A. Billard, S. Calinon, R. Dillmann, and S. Schaal, “Survey: Robot
programming by demonstration,” Springer Handbook of Robotics, pp.
1371–1394, 2008.
[17] M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp,
P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang,
J. Zhao, and K. Zieba, “End to end learning for self-driving cars,”
arXiv preprint arXiv:1604.07316, 2016.
[18] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, “Maximum
entropy inverse reinforcement learning,” in National Conference on
Artificial Intelligence, vol. 3, 2008, pp. 1433–1438.
[19] P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforce-
ment learning,” in International Conference on Machine Learning,
2004.
[20] J. Ho and S. Ermon, “Generative adversarial imitation learning,”
in Advances in Neural Information Processing Systems, D. Lee,
M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., vol. 29,
2016.
[21] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunya-
suvunakool, J. Kramár, R. Hadsell, N. de Freitas, and N. Heess,
“Reinforcement and imitation learning for diverse visuomotor skills,”
arXiv preprint arXiv:1802.09564, 2018.
[22] L. Fan, Y. Zhu, J. Zhu, Z. Liu, O. Zeng, A. Gupta, J. Creus-Costa,
S. Savarese, and L. Fei-Fei, “SURREAL: Open-source reinforcement
learning framework and robot manipulation benchmark,” in Confer-
ence on Robot Learning, 2018.
[23] M. Laghi, M. Maimeri, M. Marchand, C. Leparoux, M. Catalano,
A. Ajoudani, and A. Bicchi, “Shared-autonomy control for intuitive
bimanual tele-manipulation,” in IEEE-RAS International Conference
on Humanoid Robots, 2018, pp. 1–9.
[24] H. Lee, J. Kim, and T. Kim, “A robot teaching framework for a
redundant dual arm manipulator with teleoperation from exoskeleton
motion data,” in IEEE-RAS International Conference on Humanoid
Robots, 2014, pp. 1057–1062.
[25] M. E. Taylor and P. Stone, “Transfer learning for reinforcement
learning domains: A survey,” Journal of Machine Learning Research,
vol. 10, no. 56, pp. 1633–1685, 2009.
[26] H. B. Ammar, E. Eaton, P. Ruvolo, and M. E. Taylor, “Unsupervised
cross-domain transfer in policy gradient reinforcement learning via
manifold alignment,” in AAAI Conference on Artificial Intelligence,
2015, pp. 2504–2510.
[27] A. Gupta, C. Devin, P. Abbeel, and S. Levine, “Learning invariant
feature spaces to transfer skills with reinforcement learning,” in
International Conference on Learning Representations, 2017.
[28] Y. Liu, A. Gupta, P. Abbeel, and S. Levine, “Imitation from observa-
tion: Learning to imitate behaviors from raw video via context transla-
tion,” in IEEE International Conference on Robotics and Automation,
2018, pp. 1118–1125.
[29] P. Sharma, D. Pathak, and A. Gupta, “Third-person visual imitation
learning via decoupled hierarchical controller,” Advances in Neural
Information Processing Systems, vol. 32, 2019.
[30] D. S. Raychaudhuri, S. Paul, J. Vanbaar, and A. K. Roy-Chowdhury,
“Cross-domain imitation from observations,” in International Confer-
ence on Machine Learning, 2021, pp. 8902–8912.
[31] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-
to-image translation using cycle-consistent adversarial networks,” in
IEEE International Conference on Computer Vision, 2017.
[32] Q. Zhang, T. Xiao, A. A. Efros, L. Pinto, and X. Wang, “Learn-
ing cross-domain correspondence for control with dynamics cycle-
consistency,” in International Conference on Learning Representa-
tions, 2021.
[33] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,”
in Advances in Neural Information Processing Systems, vol. 27, 2014.
[34] A. Bansal, S. Ma, D. Ramanan, and Y. Sheikh, “Recycle-GAN: Un-
supervised video retargeting,” in European Conference on Computer
Vision, 2018.
[35] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros,
and T. Darrell, “CyCADA: Cycle-consistent adversarial domain adap-
tation,” in International Conference on Machine Learning, 2018, pp.
1989–1998.
[36] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine
for model-based control.” in IEEE/RSJ International Conference on
Intelligent Robots and Systems, 2012, pp. 5026–5033.
 [37] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-
policy maximum entropy deep reinforcement learning with a stochastic
actor,” in International Conference on Machine Learning, 2018, pp.
1861–1870.
[38] E. Olson, “Apriltag: A robust and flexible visual fiducial system,” in
IEEE International Conference on Robotics and Automation, 2011,
pp. 3400–3407.
[39] T.-C. Çallar and S. Böttger, “Hybrid learning of time-series inverse
dynamics models for locally isotropic robot motion,” IEEE Robotics
and Automation Letters, vol. 8, no. 2, pp. 1061–1068, 2022.
[40] J. Fu, S. Levine, and P. Abbeel, “One-shot learning of manipulation
skills with online dynamics adaptation and neural network priors,” in
IEEE/RSJ International Conference on Intelligent Robots and Systems.
IEEE, 2016, pp. 4019–4026.
[41] K. Tanaka, R. Yonetani, M. Hamaya, R. Lee, F. von Drigalski, and
Y. Ijiri, “TRANS-AM: Transfer learning by aggregating dynamics
models for soft robotic assembly,” in IEEE International Conference
on Robotics and Automation. IEEE, 2021, pp. 4627–4633.
[42] K. Morse, N. Das, Y. Lin, A. S. Wang, A. Rai, and F. Meier, “Learning
state-dependent losses for inverse dynamics learning,” in IEEE/RSJ
International Conference on Intelligent Robots and Systems. IEEE,
2020, pp. 5261–5268.
[43] K. Hitzler, F. Meier, S. Schaal, and T. Asfour, “Learning and adaptation
of inverse dynamics models: A comparison,” in IEEE-RAS Interna-
tional Conference on Humanoid Robots. IEEE, 2019, pp. 491–498.

View publication stats