ICACSIS 2013
Development of Augmented Reality (AR) for
Machining Simulation of 3-axis CNC Milling
Gandjar Kiswanto, Dedy Ariansyah
Laboratory of Manufacturing Technology and Automation
Department of Mechanical Engineering - Universitas Indonesia
Kampus Baru UI – Depok 16424
Email : gandjar_kiswanto@eng.ui.ac.id
Abstract - Machining simulation to verify the
correctness of numerical control (NC) program
before physically downloaded and executed to a
real machine tool is a common practice in many
industries. Many simulation system that previously
enhanced and applied based on 3D graphics in
virtual reality (VR) environment, require
measuring of completely dimension of a real
machine tool before modeling of a complete
machine tool. Besides, a large number of
machining rules and theories must be modeled
correctly as well as simulate complex machining
process require prior expertise knowledges. This
paper proposes machining simulation system
which is easier to set up and operate through
integrating with the Augmented Reality (AR). By
integrating AR, machining simulation can be
performed on a real machine tool without the
needs to model the machine itself, machining rules,
and theories thus provides more accurate NC
program. The developed system can assist the user
for positioning collision-free set up which is
commonly occurs especially when conducting
complex motion during machining process. A more
detailed discussion related to AR development for
machining simulation is presented in the paper.
Keywords: Machining simulation, Augmented
Reality, collision-free set up
I. INTRODUCTION
The benefits of Virtual Reality (VR) implemented in
manufacturing system have been rapidly recognized
and researched over the years. Meanwhile the
advances of computer technology and existing of
graphics card accelerator are making the VR
technology increasingly more attractive and realistic
with result that many potential application in
manufacturing system can be integrated to provide a
useful means for products to be manufactured
efficiently. The integration of VR technology to the
one of the manufacturing system that gained interest
in research community is development of Virtual
Machine Tool (VMT) systems. A large amount of
work has been conducted either for the purpose of
designing a complete machine tools or in more
specific by simulating machine tool’s capabilities
/13/$13.00 ©2013 IEEE 143
ISBN: 978-979-1421-19-5
such as Virtual Machining (VMach) in machining
process. Some reviews were carried out and thus find
out that the development of VMT and VMach have
expanded over the years, from simple simulation (e.g.
tool-path generation, NC-code verification, and
material removal simulation) to complex predictions
(e.g. tool deflection, surface errors, and surface
roughness) [1][5].
A system name VR Computer Numerical Control
(CNC) was developed [2] to support virtual training.
This kind of technology mimics the real machining
operation with models and simulations allowing the
interaction with high sense of reality but produces no
output such as materials and physical products instead
it can produce assistive information to the user.
However, this VR CNC system requires the
measuring of completely dimension of a real CNC
before modeling of a complete machine tool in virtual
machining environment which is time consuming. In
addition, a large number of machining rules and
theories must be modeled correctly as well as simulate
complex machining process require prior expertise
knowledge. Unlike the research that making use VR
technology to support the VMT and VMach, this
paper proposes the development of Augmented
Reality (AR) to support the VMach. The proposed
system utilize AR that supplements reality with
virtual information. Ideally, it appears to the user as if
the virtual 3-D objects actually exist in the real
environment [3]. Therefore, the real machining
environment is supplemented with necessary
information (cutting tools, workpiece, fixtures, and
etc.) rather than completely replacing it with synthetic
environment. Through AR, the user need not to
modeling the whole component of a machine tools
and predicting it’s capabilities in virtual system to
conduct the machining process, instead a user can see
the real machining environment as well as conducting
machining simulation on real machine tool efficiently
without having to worry about possibilities of
machine and tool breakages.
Compared to VR, AR offers higher sense of reality
to the user, reducing computation time and efforts as
well as increasingly more accurate to evaluate and
validate the behaviour of the system since only a few
virtual objects are modeled and geomterically
updated.
ICACSIS 2013
Processing Unit
Augmented Reality Module
Workpiece Fixture
database database
Rendering Virtual Object
Database & System
Management
Single
Vision-based
Marker
Tracking and
Multiple Registration
Marker
Camera
Fig. 1. System architecture of AR CNC
The integration of AR technology to CNC
machine tools especially in 3-axis milling machine
tools had been realized many years ago. The authors
[6] developed the machining simuation by combining
reality with virtual information through AR
technology to show the possibilities and feasibilities
of this fusion with objectives to enhance
manufacturing quality and efficiency, and also realise
manufacturing automation. With the improved
bounding boxes method, the Material Removal Rate
(MRR) estimation can be calculated thus near real-
time MRR is visualized. The tracking system in
prepared and unprepared environment which have
been one of the challenges issue encountered in
building the AR system is implemented by using
marker and KLT feature tracker respectively. Many
years later the authors improved the machining
simulation system for traning purpose by making the
interaction of real cutter with virtual workpiece,
cutting force prediction and collision detection,
improving the computational time in tracking system
[7] as well as developing the hybrid feature tracking
system [8].
In this paper, we endeavour to explore the benefits
of AR technology which is used not only for
previously reported purposes, but also for helping the
fixture or workpiece clamping set up which
commonly collide with the real cutter during actual
cutting process according to the NC program
especially include the complex motion of cutting tool
in five-axis CNC milling. Besides, multiple markers
was implemented due to the limitation of marker-
based tracking in which the virtual objects will only
appear when the tracking mark are in view. In
addition, the use of marker-based tracking system was
developed to realise the integration of AR technology
ISBN: 978-979-1421-19-5
Workpiece
Selection
Fixture
Real cutting tool
Selection
Virtual workpiece
and fixture
Workpiece
Selection and
Visualization
Fixture
Selection and
visualization
Operator & Screen
Material
Removal
based spatial display
Visualization
Real Machining
Environment
Vertical
axis
Marker
Horizontal
axis
in five-axis CNC milling. The Structure of the paper
is arranged as follows. Section 2 discusses the
developed system architecture of AR CNC and
integrating AR technology to the real machine tool.
Section 3 presents the implementation of AR
technology in machining process. Finally, discussion
of further researchs and concluding remarks are given
in Section 4.
II. DEVELOPED SYSTEM ARCHITECTURE OF AR CNC
Fig. 1. shows the developed system architecture of
AR CNC. This developed system consists of four
functional modules: a real 3-axis CNC milling,
camera and marker as vision-based tracking,
processing unit, and display device using screen-based
video see-through display. In this proposed system,
ARToolkit [9] library was utilized to build the AR
system by using fiducial marker. As compared with
sensor-based tracking, vision-based tracking is more
convenient since preparation of sensors and related
devices will decrease the flexibility of the system. To
build the integration of AR on a real machine tool,
markers with different pattern have to be attached
properly in the real machining environment to register
the virtual workpiece, fixture, and cutter. However, it
has been realised that the use of single marker is not
sufficient for certain applications which involve the
complex movements of real objects that might
potentially occlude the marker from the view of
camera. Thus, multiple markers was used to resolve
and anticipate the limitation of single marker where
the virtual objects will only appear when the tracking
mark is fully in view, whereas the marker is
potentially occluded by cutting tools and tool holder
during machining process. Screen-based video see-
144
ICACSIS 2013
through display was selected for display device due to
the limited size of screen when using head-worn
display and hand-held display as well as it exhausted
the user especially when the user must work and wear
the display device at the same time during the
simulation. It also represents probably the most cost
efficient AR approach, since only camera video and
standard PC equipment are required.
This AR CNC can simulates machining process by
firstly selecting various shape of virtual workpieces
and fixtures used during machining operation and
augment them according to the pose of marker located
on a real machine tools in a real scene. The cutting
process between the real cutting tool and virtual
workpiece based on the Numerical Control (NC)
program will then be displayed to the users using a
video see-through scene rendering mechanism in a
real-time. The virtual fixture helps to verify the NC
program which is commonly include the collision
with tool holder and cutting tools during machining
process.
A. Marker-based Tracking
One of the key requirement for creating AR
application is a tracking system that capable to
measure accurately the position and the orientation of
the observer’s location in space. The tracking system
used in ARToolkit relies upon computer vision (CV)
algorithm by analyzing the images captured from
camera in live video feed to calculate the pose. By
placing special fiducial markers with unique pattern at
known locations in the environment, the marker can
be easily detected and identified in video live feed and
hence the pose estimation can be calculated. To
achieve the augmentation of virtual objects in a real
scene the position and orientation of marker relative
to the camera have to be recovered which can be
represented by 6 independent variables or six degrees
of freedom (6 DoF) tracking systems (3 translational
coordinates and 3 rotational angles) or (Tcm) through
image processing. Therefore, the user can freely
augment the virtual objects in accordance to the
marker coordinates relative to the camera coordinates
in a real scene.
ܺܿ ܴ11 ܴ12 ܴ13 ܶ‫݉ܺ ݔ‬
ܻܿ ܴ21 ܴ22 ܴ23 ܶ‫ܻ݉ ݕ‬
൦ ൪=൦ ൪൦ ൪
ܼܿ ܴ31 ܴ32 ܴ33 ܶ‫ܼ݉ ݖ‬
1 0 0 0 1 1
ܺ݉ ܺ݉
ܴ3‫ݔ‬3 ܶ3‫ݔ‬1 ܻ݉ ܻ used to substract part of the virtual workpiece.
= ቂ ቃ ൦ ൪ = ܶܿ݉ ൦ ݉൪ (1)
0 0 0 1 ܼ݉ ܼ݉
1 1
Camera calibration which is usually conducted before
the initialization phase to obtain appropriate
parameter intrinsics of camera to compensate lens
distortion is not discussed here. On the other hand, the
user can use multiple markers to realise more complex
applications. Once the relation 6 DoF coordinates
145
ISBN: 978-979-1421-19-5
between one marker and another are known, the pose
of one marker can be reliably estimated by another
visible markers allowing the user displays the virtual
objects anytime as long as one of the markers is fully
in view.
B. Virtual Objects Selection
As preparing the actual machining process, the user
needs to define the workpiece and typical fixture to
clamp the workpiece before performing the machining
process. The various shape of workpieces and fixtures
were modeled virtually and available in the workpiece
and fixture database. The size of virtual objects are
freely modeled according to the defined dimension.
The main goal of virtual objects selection is allowing
the user to match the requirements of machining
simulation that can be performed on any CNC
machines.
C. Material Removal Visualization
The first important thing to build machining
simulation by means of AR technology is to do
synchronization of real machine tool movement with
the updated virtual object during simulation in
recorded or live video feed. This such synchronization
is created through the interaction of real cutter and
virtual workpiece. The idea for creating interaction
between real cutter and virtual workpiece is to define
when or where they collide each other. To define the
collision, the intersection between real cutter and
virtual workpiece has to be calculated by previously
tracking and reconstructing machine coordinate
system (MCS) into tool coordinates system (TCS) and
workpiece coordinates system (WCS) in AR
environment near real-time. To construct TCS and
WCS, we used two predefined marker with different
pattern that was glued to the tool holder and
worktable. This method yields two different
coordinates system which represent TCS and WCS so
that relative position between real cutter and virtual
workpiece can be tracked dynamically in live video
stream near real-time. After that the position of the
cutter tip center with respect to the TCS must be
approximated by doing precalibration. The cutter
position in TCS is then transformed into WCS using
linear transformation matrix. By doing so, the
collision can be detected along the tool path. If the
cutter position lies within the volume of workpiece,
then the collision occurs, and hence intersection
points can be calculated to generate tool swept
volume. The tool swept volume for each NC-path is
The second is modeling scheme method to
represent the virtual workpiece and simulate material
removal process in machining simulation. These
modeling scheme can be categorized into two
approaches: direct solid modeling and spatial
partitioning. Each of these approaches has been
widely adopted to create machining simulation with
varying ranges of applicability and degrees of success.
ICACSIS 2013 ISBN: 978-979-1421-19-5

The major disadvantage of the direct solid modeling

approach to machining simulation is the
computational complexity of regularised Boolean set
operations [4], meanwhile spatial partitioning
approach decompose a solid object into a collection of
basic geometry elements, thus simplifying the Z
processes of regularised Boolean set operation. There
are many kind of basic geometry elements
implemented to discretise the solid object including z-
buffer data structure, dexel data structure, and ray
representation. However, they are not capabe to be
used to address multidimensional milling verification
for workpieces of comparable complexity and
accuracy [4], therefore voxel representation were used Y X
in this paper. The grids size of voxel representation is
dynamically modified according to octree data level
(a)
management. Surfaces that are not affected by cutter
position remain representated in large grids to speed
up the rendering information during simulation.

III. IMPLEMENTATION OF AR CNC

The proposed system was initially developed and

applied in 3-axis CNC milling machine. There are
numerous type of 3-axis CNC milling machine tools
that have been widely available commercially and
used in many industries and institutions. Each of them
has it’s own machine tool characteristics ranging from
type of fixture, tool holder, degree of freedom, size of
machine, power, and etc. The proposed system to
simulate machining process can be conducted on any
kind of machine tools. It provides the various shape of
workpieces and fixtures to be selected and thus match
the requirements to perform machining simulation. (b)
Fig. 2 . Single marker implementation with various shape of
A. Workpiece and Fixtures Set up virtual workpiece: (a) cube, (b) cylinder, and various shape of
clamping fixture: (a) Jigged, (b) T-slot clamping
Before performing machining simulation, the
workpiece and fixture are set up by locating them to Fig. 2. shows the workpiece and fixture set up with
the correct location. Since the virtual objects are different shape of virtual workpieces and virtual
located according to the position and orientation of fixtures with respect to the marker.
the related marker, the movements of the mechanical
unit of CNC machines have to be taken into
consideration when attach the markers on CNC
machines. Due to the ease of implementing marker-
based tracking, this fiducial markers can be attached
to any location according to the movement
configuration of mechanical component in any CNC
machines. In this experiment, the AR CNC system
was applied to the 3-axis CNC milling machine with
2-axis (x and y axis) translation from worktable and 1-
axis (z axis) translation from the tool holder as shown
in Fig. 2a. Since the simulation of machining process
is achieved according to the movement of the real
cutting tool and the virtual workpiece, the marker with
recognized pattern for augmenting virtual workpiece
and fixtures has to be attached on the worktable.
Another marker with different pattern for augmenting
virtual cutting tool was attached to the tool holder.
Fig. 3 Multiple markers occluded by real object ; (a), (b)
composite markers, and (c), (d), (e), separated markers.

146
ICACSIS 2013
However, the movement of cutting tool at certain path
during simulation can potentially occlude the marker
from the view of the camera and making the virtual
objects disappeared. Hence, the multiple markers was
used to anticipate this occlusion problem for making
virtual objects remain appeared by detecting another
visible markers. Fig. 3. shows the implementation of
multiple markers.
Fig. 4. Multiple markers implementation to realise multi-
axis machining
Further, the relation of 6 DoF coordinates marker
between one to another can be developed to realise the
workpiece and fixture set up for multi-axis milling
CNC machine which involve complex motion. Fig. 4.
shows by consider the box as if it is the worktable in
multi-axis milling CNC machine and multiple
markers are attached to it. The virtual objects were
drawn to another marker with respect to itself.
(a)
Fig. 5. Solving occlusion problem via depth buffer: (a) virtual objects occlude the real cutting tool, (b) rendering virtual cutting
tool, (c) rendering real scene pixel within the real cutting tool rendering area.
However, when the marker is occluded or not in
camera view (for example when the worktable were
tilted and the marker is not detected), then the virtual
objects remain appeared by detecting another visible
markers. The rendering information of virtual objects
to the real scene that occlude the real objects has been
147
ISBN: 978-979-1421-19-5
noticed. Initially, virtual objects rendered in real scene
always occlude the real objects thus making the real
cutting tools along certain path occluded by virtual
workpiece when it should be visible and displayed in
front of the workpiece. This occlusion problem was
solved by making use of OpenGL Depth buffer by
rendering the real scene pixels from different marker
attaced to the tool holder within the real cutting tool
rendering area over the virtual workpiece. Fig. 5.
shows the resolved occlusion problem by means of
depth buffer rendering rule.
B. Near Real Time MRR Visualization
In the early stages, the system simulation was
developed on a 3-axis CNC milling. Hence, this
system is limited to 3-axis machining simulation as
well as only consider linear cutter paths. To visualize
material removal on the system proposed, a simple
NC program were created for simplification. Fig. 6.
Shows the simulation of machining process in AR
CNC. During the simulation process, the system
displays a real cutter machining a virtual workpiece in
which the initial workpiece is incrementally converted
into the finished part.
After all, this proposed AR CNC system will
contribute to the following several aspects:
1. NC program which is generated semiautomated by
system or manually by machinists might include
the machining errors and collisions between tool
holder, workpiece and fixture especially when NC
program involve complex motion in multi-axis
machining. therefore, machining simulation is
used to validate the generated information for
machine tools before it is physically downloaded
(b) (c)
and executed. During the simulation, the user can
observe the machining process as it proceed so
that the selection of workpieces and fixtures in
machining simulation can help the user for
positioning collision free set-up during the
machining process.
ICACSIS 2013
2. Since the simulation of process machining are
performed on real machine tool, AR CNC system
provides increasingly more accurate to visualise
the material removal simulation including correct
kinematics without the need to build VMT which
is common in several commercial CAM system
for the purpose of NC program verification.
Fig. 6. AR CNC machining simulation process
IV. FURTHER DISCUSSION AND CONCLUDING
REMARKS
The prosposed system by integrating AR technology
to create machining simulation on any CNC machines
148
ISBN: 978-979-1421-19-5
will be very beneficial to validate NC program and
positioning collision-free set up of workpieces and
fixture. A simple and a few set up required to conduct
the simulation including correct kinematics of the
machine tools movement without the needs to model a
complete machine tools and predicting it’s
capabilities. Since the workpieces and fixture set up
has been realised and can be integrated to multi-axis
machine, the complex motion of multi-axis which is
prone to errors as well as collisions between the
cutting tool and clamping fixture which are
commonly difficult to anticipate can be predicted, as a
result the NC program can be adjusted accordingly.
This system can also be expanded to resolve the
collision situation during the simulated process by
automatically generate and apply the appropriate tilted
and rotation angle to the NC path. The physical aspect
can also be developed and applied in this system to
provide the automatically generated machining
parameters to optimize the NC program.
REFERENCES
[1] Aini Abdul Kadir , Xun Xu , Enrico Hämmerle, Review:
Virtual machine tools and virtual machining-A technological
review, Robotics and Computer-Integrated Manufacturing,
v.27 n.3, p.494-508, June, 2011 .
[2] Kao YC, Cheng HY, Chen YC, “Development of a virtual
controller intergrating virtual and physical CNC”, Materials
Science Forum 505-507, pp. 631-636, 2006.
[3] Azuma R. A survey of augmented reality. Presence:
Teleoperators and Virtual Environments 1997;6(4):355–85.
[4] J. Donggo, K. Kwangso, and J. Jungmin, “Voxel-based virtual
multi-axis machining”, International Journal of Manufacturing
Technology, Vol. 16,cpp. 709-713, 2000.
[5] Tamer M. Wasfy, Ayman M. Wasfy, Hazim El-Mounayri,
Daniel Aw, “Virtual training environment for a 3-axis CNC
milling machine”, Proceedings of IDETC/CIE 2005, ASME
2005 International Design Engineering Technical Conferences
& Computers and Information in Engineering Conference,
Long Beach, California USA.
[6] J. Zhang, S. K. Ong, A. Y. C. Nee. A volumetric Model-Based
CNC Simulation and Monitoring System in Augmented
Environments. Proceedings of the 2006 International
conference on Cyberworlds IEEE.
[7] Zhang J, Ong SK, Nee AYC. AR-assisted in situ machining
simulation: architecture and implementation. In: ACM
SIGGRAPH 7th international conference on virtual-reality
continuum & its applications in industry. SIGGRAPH VRCAI
conference. 2008.
[8] Zhang J, Ong SK, Nee AYC (2010) A Multi-regional
Computation Scheme in an AR-Assisted in situ CNC
Simulation Environment. Computer-Aided Design
42(12):1167–1177.
[9] ARToolKit 2.72, http://www.hitl.washington.edu/
artoolkit/download/ (last accessed 18.07.2012).