Full Field Deformation Measurement by Videogrammetry Using Self Adaptive Window Matching

See discussions, stats, and author profiles for this publication at: https://www.researchgate.
net/publication/331896238
Full‐field deformation measurement by videogrammetry using self‐adaptive

window matching
Article in The Photogrammetric Record · March 2019

DOI: 10.1111/phor.12268
CITATIONS READS
5 197
13 authors, including:
Zhen Ye Cheng Zhao

Tongji University Tongji University
62 PUBLICATIONS 974 CITATIONS 79 PUBLICATIONS 650 CITATIONS
SEE PROFILE SEE PROFILE
Shijie Liu Huan Xie

Tongji University Tongji University
122 PUBLICATIONS 1,689 CITATIONS 157 PUBLICATIONS 2,176 CITATIONS
SEE PROFILE SEE PROFILE
Some of the authors of this publication are also working on these related projects:
NSFC funding View project
National Natural Science Foundation of China under Grant Nos. 41622107 and 41371344 View project
All content following this page was uploaded by Chao Wei on 14 January 2022.
The user has requested enhancement of the downloaded file.

The Photogrammetric Record 34(165): 36–62 (March 2019)
DOI: 10.1111/phor.12268
FULL-FIELD DEFORMATION MEASUREMENT BY

VIDEOGRAMMETRY USING SELF-ADAPTIVE WINDOW
MATCHING
Sa GAO† (1410900@tongji.edu.cn)
Xiaohua TONG* (xhtong@tongji.edu.cn)
Peng CHEN* (chenpeng@tongji.edu.cn)
Zhen YE (yezhen0402@126.com)
Ouling HU (1434924@tongji.edu.cn)
Benkang WANG (1733504@tongji.edu.cn)
Cheng ZHAO (zhaocheng@tongji.edu.cn)
Shijie LIU (liusjtj@tongji.edu.cn)
Huan XIE (huanxie@tongji.edu.cn)
Yanmin JIN (jinyanmin@tongji.edu.cn)
Xiong XU (xvxiong@tongji.edu.cn)
Sicong LIU (sicong.liu@tongji.edu.cn)
Chao WEI (cwei@tongji.edu.cn)
The State Key Laboratory for Disaster Reduction in Civil Engineering and College of
Surveying and Geo-Informatics, Tongji University, Shanghai, China
†Also at Institute of Photogrammetry and Remote Sensing, Technische Universit€

at
Dresden, Germany
*Corresponding authors
Abstract
Full-field deformation measurement of specific materials is an important issue
in civil engineering materials testing. This paper presents a flexible videogrammetric
scheme to measure full-field deformation of an artificial rock-like material under
uniaxial compression. In this scheme, two high-speed cameras were used to measure
the spatial morphological changes of the material surface, which was sprayed with
a speckle pattern. A robust self-adaptive window matching strategy is then proposed
to extract accurate displacements of tracking points. Finally, three-dimensional (3D)
point clouds and full-field deformation can be calculated by photogrammetric and
spatiotemporal analysis. Simulation and empirical tests demonstrated that the self-
adaptive window matching strategy used with high-speed videogrammetric speckle
image sequences can detect more subtle deformation and avoid more mismatches
than a fixed-window matching strategy.
© 2019 The Authors

The Photogrammetric Record © 2019 The Remote Sensing and Photogrammetry Society and John Wiley & Sons Ltd
The Photogrammetric Record
Keywords: deformation measurement, full-field deformation, high-speed camera,

rock-like material, self-adaptive window matching, videogrammetry
Introduction
THE STABILITY AND SAFETY of building materials and structures have become a worldwide
focus of attention in modern urban development because they have an important impact on
human life and activities. Therefore, before the actual use of a specific material, its
structural performance and factor of safety need to be evaluated through tensile/compression
testing, impact testing, thermal testing and other assessments (Due and Robinson, 2013;
Perras and Diederichs, 2014; Safri et al., 2014). However, traditional contact transducers are
not suitable for measuring the full-field deformation of materials, because of their single-
point monitoring, limited measurement range, difficult assembly and other shortcomings
(Liu et al., 2015). With the rapid development of optical engineering, digital image
correlation (DIC) has gradually become a major measurement method in the study of
mechanics, with the great advantage of being a non-contact technique (Chu et al., 1985).
This method can overcome the drawbacks of traditional transducers when used in
combination with videogrammetric techniques. In digital photogrammetry, DIC is usually
known as “image matching”; image-matching algorithms and strategies are the most critical
processing steps because of their influence on measurement accuracy and computational
efficiency. For this reason, image matching has undergone unprecedented development in
photogrammetry (Gruen, 2012; Tong et al., 2014, 2015d).
At present, image-matching algorithms can be roughly divided into global algorithms
and local algorithms (Liu et al., 2015), with semi-global matching (Hirschm€ uller, 2005)
combining characteristics from both of these. A global algorithm typically has smoothness
optimisation, which yields accurate and dense disparity measurements. However, complex
parameter setting, tedious calculation and optimisation of the global energy function are
both time- and memory-consuming (Adhyapak et al., 2007; Shi et al., 2016). In contrast,
local methods often aggregate the support from the neighbouring pixels in a window (Zhang
et al., 2010; Tong et al., 2015a, b). However, a small window may cause poor disparity
estimation due to loss of the intensity variation; conversely, a large window will cause
smooth disparity estimation because of covering a large, varying region (Kanade and
Okutomi, 1994). Therefore, a number of studies have reported on variable-window methods
(Izquierdo, 1999; Tang et al., 2002; Mekuz et al., 2006). Kanade and Okutomi (1994)
employed a Gaussian statistical model to evaluate the local variation of the intensity and the
disparity, allowing an appropriate matching window to be selected in the iterative
stereomatching algorithm. Fusiello et al. (1997) presented a multiple-window scheme to
compute the disparity using left–right consistency, where nine asymmetric correlation
windows were used to calculate the minimum correlation error value. Jeon et al. (2002) used
eight different windows to calculate dense disparity maps and minimise boundary overreach.
Veksler (2003) developed an efficient approach to calculate the window size with a new
window cost function, where the optimal window size is determined by the minimum cost
value. Gupta and Cho (2013) proposed a robust window-based correlation approach for real-
time stereo correspondence. In their algorithm, both a small matching window and a large
matching window are used to compute a dense and sharp disparity map. Li et al. (2016) also
proposed an adaptive phase correlation method to minimise the influence of boundary reach
in precise disparity estimation of unmanned aerial vehicle (UAV) images. However, these
methods are mainly used to solve the disparity problem of stereomatching with variable-
window methods in the application of three-dimensional (3D) reconstruction; they cannot be
© 2019 The Authors

The Photogrammetric Record © 2019 The Remote Sensing and Photogrammetry Society and John Wiley & Sons Ltd 37
GAO et al. Full-field deformation measurement by videogrammetry
directly used to calculate high-precision coordinates of tracking points in speckle image

sequences. The purpose of image sequence processing is to extract subtle deformations
instead of obtaining disparity information. In this paper, the adaptive window matching
method is improved to meet the requirements of micro-deformation measurement.
In the research field of non-contact deformation monitoring, some scholars have carried
out research into the use of DIC in arithmetic improvement, accuracy assessment and
application extension (Pan et al., 2009b; Liu et al., 2014; Zhang et al., 2015). In proposing the
Newton–Raphson (NR) method of partial differential correction to determine displacements
and gradients, Bruck et al. (1989) laid the groundwork for optical measurement in the field of
mechanics. Yao et al. (2005) adopted a digital speckle correlation method to measure the
deformation of composite pressure vessels in real engineering practice. Liu and Gao (2011)
developed a digital speckle correlation method for coefficient of thermal expansion (CTE)
measurement of a film surface, demonstrating an improved accuracy in comparison with
actual thermal measurement. Salazar and Barrientos (2013) presented a speckle correlation
method to measure the surface roughness on an actual aircraft wing with one charge-coupled
device (CCD) camera. Murray et al. (2015) used DIC to measure static displacements at
varying load levels and dynamic displacements at varying vehicle speeds on a reinforced
concrete bridge. However, most of the empirical experiments have always selected fixed
matching windows during the image sequence processing, and the window sizes need to be
large enough to meet the requirements of the deformation measurement. As mentioned above,
a large window will cause smooth displacement estimation because of covering a large,
varied deformation region; the matching results of target points at the deformation edges will
be inaccurate if drastic deformation takes place in their neighbouring pixels.
Therefore, in this paper, according to the characteristics of full-field 3D deformation
monitoring, a robust self-adaptive window matching strategy is presented to extract accurate
displacements in speckle image sequences. Moreover, based on the high-speed video-
grammetric technique, this method can effectively detect the transient and subtle deformations
of the measured material at any epoch. Following the introduction, the camera network
configuration is briefly described. The main analytical algorithms are then introduced, and the
simulation test and uniaxial compression test undertaken in this study are described to
demonstrate the robustness and reliability of the proposed videogrammetric method. In the
final section, a summary of the work and potential further developments are presented.
Methodology
Fig. 1 shows the overall data processing procedure, which consists of three main
elements as follows:
(1) Camera network configuration. The network includes two high-speed cameras, an
artificial light source and the measured speckle object.
(2) Stereoscopic matching and target tracking. The stereoscopic matching of image
points is performed in the initial left–right speckle image pairs through an improved
reliability-guided matching strategy. The target tracking of tracking points is then
performed in the forward–backward speckle image sequences through the self-
adaptive window matching strategy.
(3) Full-field deformation measurement. The interior- and exterior-orientation parameters,
including the lens-distortion parameters, can be acquired by stereo camera calibration.
The structural deformation parameters are then calculated through the generation and
spatiotemporal analysis of the sequential 3D point clouds.
© 2019 The Authors

38 The Photogrammetric Record © 2019 The Remote Sensing and Photogrammetry Society and John Wiley & Sons Ltd
Camera network configuration

High-speed camera arrangement
Digital speckle spraying
Artificial light arrangement
Stereoscopic matching
Coarse matching: Reliably-guided
matching strategy in initial left-right
speckle image pairs
Stereo camera calibration
Interior- and exterior-

Precise matching: Least-squares orientation parameters and
matching (LSM) lens-distortion parameters
Target tracking
Sub-Controller One Sub-Controller Two

Coarse matching: Coarse matching:
Distributed parallel Self-adaptive Self-adaptive
computing architecture window matching window matching
Precise matching: Precise matching:

LSM LSM
Full-field 3D deformation measurement

3D point cloud generation of the
measured surface by forward
intersection in Epoch i
3D full-field displacement estimation

by coordinate subtraction between
Epoch i and Epoch 0
Full-field strain calculation with the

help of the displacement results
Fig. 1. Flowchart of the proposed scheme for acquiring accurate full-field deformation measurements.
Camera Network Configuration

In order to obtain the 3D dynamic deformation information of the material and
structure over a relatively short period of time, videogrammetry, as an optical measurement
technique, has been widely used to monitor the 3D morphological changes of a measured
object in a non-contact way (Tong et al., 2017a). Furthermore, the availability of high-speed
cameras has made it possible to measure rapidly moving objects. Therefore, thanks to the
high frame rate, many scholars have studied deformation monitoring of fast-moving objects
© 2019 The Authors

(Liu et al., 2010; Leifer et al., 2011; Matsumoto et al., 2016). In this study, to achieve full-
field 3D deformation measurement, two high-speed cameras were used to record the same
experimental scene. The overlapping area taken by the cameras should be as large as
possible. Generally speaking, in order to satisfy this condition, the baseline length and
intersection angle between the two cameras must be relatively small. This convergent
photography method leads to poor accuracy of the 3D reconstruction in the direction
perpendicular to the baseline due to propagation of the image-matching error (Alsadik et al.,
2014). To overcome this problem, this research paid more attention to improvement of the
image-matching accuracy. This is because a robust image-matching algorithm can greatly
improve the accuracy of the 3D reconstruction (Abdel-Aziz, 1982; Tong et al., 2015c,
2017b; F€orstner and Wrobel, 2016). For automatic matching of the corresponding images
and accurate image sequence tracking of the target points, robust image-matching strategies
are introduced, which are suitable for high-accuracy full-field deformation measurement.
As shown in Fig. 2, the generation of a speckle pattern on the measured surface can provide
sufficient intensity variations to ensure reliable and accurate matching (Pan et al., 2010; Pan,
2011). Firstly, white matt paint was painted on the surface of the material, and then black matt
paint or black ink was sprayed randomly and evenly on the white surface. The effect of the matt
paint was to overcome light reflection and increase image texture information. The black-and-
white speckle pattern can help to improve image contrast and image-matching accuracy.
Moreover, in the case of high-speed shooting, natural light would be generally the best choice
because of its consistent and homogeneous illumination. However, in the case of insufficient
natural light during the experiment, a high-power halogen lamp as a supplemental light source
was used to improve the image quality of the high-speed camera.
Stereoscopic Matching and Target Tracking

Before stereoscopic matching, the region of interest (ROI) and the target points should
be defined in the initial reference image of one camera. Notably, the selection method of the
target points is similar to the sampling process. In the designated ROI, the locations of the
full-field target points can be determined based on a uniform sampling interval (or grid
(a) (b)
Fig. 2. Speckle pattern on the material surface. (a) The initial speckle image. (b) The deformed speckle image
captured by the high-speed camera.
© 2019 The Authors

step). In mechanics calculations, the regular grid is regarded as a common and convenient
method to determine the target points (Chu et al., 1985; Pan, 2011). This method of
determining targets points is very conducive to the fast and accurate calculation of both
displacement fields and strain fields.
In image matching, the coarse-to-precise matching method performs very well for the
corresponding speckle image sequences. In the precise matching process, stereoscopic
matching and target tracking can adopt the same sub-pixel matching method. As shown in
Fig. 3, the imaging distortion of the target image patch is caused by the optical disparity or
the motion deformation. Therefore, least-squares matching (LSM) is suitable as it can take
into account geometric distortion of the image patch (Ackermann, 1984). Furthermore,
compared with the NR method, LSM can avoid the computational complexity and burden
involved in the optimisation of the NR method’s correlation function because, in LSM, only
the first-order derivatives of the deformed image patch need to be calculated (Pan et al.,
2009a). Moreover, to evaluate the degree of similarity between two image patches, the zero-
mean normalised cross-correlation (ZNCC) measure is chosen because it is insensitive to
both offset and scale changes of the intensity of the image patch (Pan, 2011). In the
proposed scheme, ZNCC is not only the termination criterion of the iterative process in the
LSM, but also the correlation criterion of the coarse matching.
In stereoscopic matching and target tracking, the precise matching of every target point
requires the approximate corresponding location and the support window, which can be
provided by the coarse matching. However, during the coarse matching process,
stereoscopic matching and target tracking adopt different matching strategies because of
their different matching purposes and characteristics.
Stereoscopic Matching of the Initial Left–Right Speckle Image Pairs. Stereoscopic

matching is generally applied in the initial epoch of the corresponding image sequences. In
the proposed scheme, it has two purposes: (i) to calculate the corresponding image points in
the left–right speckle image pairs; and (ii) to provide the initial locations of the tracking
points in the image sequence tracking. In the stereoscopic matching, an epipolar matching
strategy could be used to solve the conventional stereoscopic matching problem. However,
in the generation of epipolar images, the actual distorted coordinates caused by optical lens
Initial image Deformed image
2M+1
Reference image block

Corresponding image block
Fig. 3. Image distortion between the initial image and the deformed (or corresponding) image.
© 2019 The Authors

distortion must be transformed into ideal distortion-free ones (Garcia et al., 2002). The
rectified epipolar images would then be obtained with the help of a greyscale interpolation
algorithm. Thus, it would be time-consuming to rectify all high-speed image sequences in
the subsequent target tracking. Moreover, when using the epipolar constraint, the greyscale
interpolation algorithm would have an impact on the accuracy of the matching results.
Therefore, in order to overcome this problem, an efficient reliability-guided matching
strategy needs to be introduced in the stereoscopic matching, even though the original
intention of this algorithm was to accurately calculate the displacement field in image
sequences (Pan, 2009; Pan et al., 2012).
In order to achieve the purpose of automatic calculation, the traditional reliability-
guided matching strategy is improved to complete two main calculation steps: (1) the target
points are accurately matched one-by-one in the corresponding images using the ZNCC
measure as the reliability parameter; and (2) the search window of every target point is
provided by the known location of its neighbouring point. Fig. 4(a) shows the determination
process of the search window. When one target point (blue dot) has been matched
successfully, the local search windows of its neighbouring points (L1, L2, L3 and L4) can be
provided by its corresponding point (red dot). Therefore, at the beginning of the calculation
process, at least one seed point needs to be selected. Fig. 4(b) shows the matching order of
the target points. In the point list, the neighbouring points of the computed point with the
highest ZNCC value should be matched first. In addition, the deformation parameters of this
computed point are directly provided to its neighbouring points as the initial values of the
parameters of the LSM. Through the reliability-guided matching strategy and LSM, the
dense corresponding image points can be calculated accurately in the initial epoch.
Robust Self-adaptive Window Matching of Forward–Backward Speckle Image

Sequences. In target tracking, the appropriate support window of every target point can be
determined by the self-adaptive window matching strategy. Traditionally, the window size
of the target image patch is often specified through practical experience, and the window
type and window size of all the target points are the same. However, the matching window
has a direct effect on the accuracy of the experimental results because a small target
window will result in matching failure due to the lack of image information, whereas a
large target window will result in subtle deformations not being detected. Therefore, the
optimum target window needs to be calculated through the self-adaptive window matching
strategy. Since the speckle image is randomly sprayed, the black speckles are not evenly
distributed on the white object surface. Addressing this characteristic, the window size of
each target image patch can be calculated separately due to the randomness of the speckle
spraying. Generally speaking, the target image patch in Epoch 0 is regarded as the reference
image patch for the following epochs, because this approach can avoid the error
accumulation which is caused by recursively using the target image patch of the previous
epoch (Tong et al., 2017a).
In this paper, the window size is determined based on spatial correlation and the
information entropy of the target image patch. For the spatial correlation, every target image
patch should be unique within its surrounding region. Due to the randomness of the speckle
spraying, some surrounding image patches that have similar textural information to the
target image patch might result in mismatches in the tracking and matching process.
Therefore, the purpose of the uniqueness of the spatial correlation is to efficiently avoid the
mismatch problem, which is caused by locally similar textures. To a certain extent, it can
also avoid the mismatch problem that is caused by the lack of greyscale (or texture)
© 2019 The Authors

Camera One Camera Two
Point list stored in

L1 Initial
epoch descending order of ZNCC
L4 L2
L1
L3
L4 L2 Seed point, L1, L2, L3, L4
L3
L5
S1 L1 Seed point, L1, L2, L3, L4,
L1 L7 L6
L5, L6, L7
L4 L2 L4 L2
R1
L3 L3
S2 L2 L5
L1 Seed point, L1, L2, L3, L4,
L7 L6 L5, L6, L7, L8, L9
L4 L2 R2
L4 L8
L3
L3 L9
S3 L3 L5
L1 Seed point, L1, L2, L3, L4,
R3 L7 L6
L4 L2 L5, L6, L7, L8, L9, L10,
L4 L8 L11
L3
L11 L9
L10
L1
S4 L4 L5
Seed point, L1, L2, L3, L4,
L4 L2 L7 L6
R4 L5, L6, L7, L8, L9, L10,
L3 L12 L8 L11, L12
L11 L9
L10 ĂĂ
(a) (b)
Fig. 4. Stereoscopic matching strategy. (a) The corresponding search window of the target point is determined
by its neighbouring point. (b) The ZNCC measure values are regarded as the reliability parameter in the
matching order. In this example, it is assumed that the sorting of the ZNCC measure values is
L1 > L2 > L3 > L4 > L5 > L6 > L7 > L8 > L9 > L10 > L11 > L12. Moreover, when one point is matched success-
fully, it will be deleted in the point list (indicated by lines).
information. Unlike traditional stereomatching methods in computer vision, LSM also

requires enough textural information for high-accuracy positioning during the least-squares
iterative convergence. Therefore, as a common texture feature, the information entropy of
the image can be used to evaluate the complexity of the texture. According to equation (1),
the more complex the image texture, the larger the entropy value. Conversely, the more
uniform the image greyscale, the smaller the entropy value. Therefore, in the self-adaptive
window matching strategy, the information entropy value increases with an enlargement of
the window size because more textural information is accommodated in the support window
(Geng, 2016). Moreover, before the target tracking and matching, the threshold value d of
the information entropy needs to be set, as different threshold values will result in different
matching results.
X255
Entropyð f Þ ¼ i¼0
pðiÞ logðpðiÞÞ ð1Þ
© 2019 The Authors


pðiÞ ¼ NðiÞ=Nsum
where
pðiÞ logðpðiÞÞ ¼ 0 if pðiÞ ¼ 0:
Equation (1) is the information entropy equation where N(i) is the number of pixels whose
grey value is i; and Nsum is the total number of pixels in the support window. In this paper,
the base of the logarithmic function is set to 2.
As shown in Figs. 5(a) and (b), the initial positions of the target points have been
provided by stereoscopic matching. Therefore, at the beginning of the target tracking, every
target point is given a small window. Once the matching process finds a unique corresponding
image patch and its information entropy value reaches the predefined threshold value, the
window size will stop growing. Furthermore, the left and right constraints can be further
utilised to ensure the correctness of the image matching when large displacement occurs (Fua,
1993). In this whole process, the ZNCC measure is used to evaluate the degree of spatial
correlation. When computing the ZNCC measure values for various sizes of image patches,
incremental computation can be introduced to improve the computational efficiency (Men
et al., 2009). Finally, LSM is also used to calculate the precise location of every tracking point.
Traditionally, the support region of the target point can be formed using a square
window (Veksler, 2003), and the target point is located exactly at the centre of the image
window. However, in actual experiments, the sudden rupture of a non-rigid body causes the
occurrence of deformation boundaries. The target points around such boundaries will
receive low-precision or even wrong matching results because of the low ZNCC measure
values. Therefore, the multiple-window method is applied at deformation edges. As shown
in Fig. 5(c), according to the four directions around the target point, four different windows
are utilised to calculate the ZNCC measure values, and the window with the largest ZNCC
measure value is retained. The idea is that the window with the largest ZNCC measure
value is most likely to cover the same deformation region. Therefore, during precise
matching, the deformation parameters of the target image patch can be directly provided by
its neighbouring point which is located in the largest ZNCC measure-value window.
Distributed Parallel Computing. In order to achieve the purpose of on-site computing,

parallel computing is necessary to significantly reduce the processing time of massive image
sequences. In the tests undertaken in this study, each camera was equipped with a
minicomputer (subordinate controller), and a principal controller was used to manage and
control the two subordinate controllers. For the real-time capture and storage of the large
volume of image sequences from the high-speed cameras, each subordinate controller was
assembled with a high-speed image capture card and solid-state disk array. A multi-core
central processing unit (CPU) was also installed in each subordinate controller to support
the parallel computing of the image sequence tracking and matching. This is because the
tracking and matching of image sequences takes up a large amount of time. Furthermore,
the ROIs of the stereo cameras need to be partitioned according to the core numbers of the
CPU so that every partition can be calculated in parallel. The detailed calculation
mechanism is shown in Fig. 6. Depending on the number of CPUs and CPU cores,
distributed parallel computing would significantly improve the computational efficiency. For
example, if the distributed network has m CPUs and n processor cores in each CPU, the
distributed parallel computing would only cost about 1/mn of the time of traditional single-
threaded processing.
© 2019 The Authors

(a) Target tracking and matching
Epoch 0 Epoch i
Uniqueness correlation criterion
Target image ZNCC

Local search area
(b) Adaptive-window matching
+1
+1
+1
Window
+1 +1 +1
expansion
7×7 pixels 9×9 pixels 11×11 pixels
13×13 pixels 15×15 pixels 17×17 pixels
(c) Multiple-window matching
Fig. 5. The self-adaptive window matching process. (a) The initial reference image patch is provided by
stereoscopic matching in Epoch 0; the local search area is determined by the location of the target point and the
search radius. (b) The optimal window size is determined by the spatial correlation and the information entropy.
(c) For the target points on deformation boundaries, an appropriate window is chosen from the four different
windows according to the largest ZNCC measure value.
© 2019 The Authors

Full-field 3D Deformation Measurement

Videogrammetry is an important branch of photogrammetry, which mainly focuses on
the morphological measurement of a moving object. Its technical essence is to establish
the relationship between the 3D spatial locations and the two-dimensional (2D) projections
of target points in the corresponding image sequences. Based on the standard collinearity
equation model, the rigorous analytical algorithms are described in this section. After
obtaining the spatial information of the measured object, the main structural parameters
can be further calculated through the spatiotemporal analysis of the sequential 3D point
clouds.
②
Left Stereoscopic matching Right Principal
image image controller
Local network transmission

① Initial image ①
⑤ Sequential corresponding points
③ Initial target points
Left Right
camera camera
Subordinate
controller
④ Image partitioning ④
Parallel computing Parallel computing

by multi-core CPU by multi-core CPU
Fig. 6. Distributed parallel computing architecture. The processing steps are depicted as follows. (1) The initial
images are extracted from the subordinate controller to the principal controller. (2) The initial corresponding
target points are determined through stereoscopic matching. (3) These target points are transferred to the
corresponding sub-controllers. (4) The tracking and matching of the image sequence can be computed in parallel
after image partitioning. (5) The sequential corresponding points are sent to the principal controller to participate
in the 3D reconstruction.
© 2019 The Authors

Full-field 3D Deformation Construction. Stereo calibration and 3D point-cloud

generation are the important elements in this full-field deformation measurement procedure.
The purpose of the stereo camera calibration is to calculate the interior-orientation
parameters, the lens-distortion parameters and the exterior-orientation parameters (Tsai,
1987; Fraser, 1997; Clarke and Fryer, 1998; Pollefeys et al., 1999; Borghese and Cerveri,
2000; Furukawa and Ponce, 2009). In the proposed scheme, a plane-based calibration
algorithm, as the first choice, is utilised to calculate the camera parameters, which has the
advantages of flexibility, robustness and low cost (Zhang, 2000). The Camera Calibration
Toolbox for Matlab can be used to accurately calculate the interior-orientation and
lens-distortion parameters (Bouguet, 2015). During the whole calibration process, the
plane-based calibration algorithm can also determine the spatial relationship between the
camera and the calibration board. Therefore, when the stereo cameras synchronously
capture the images of the same calibration board in a common field of view, the
relative spatial relationship between the cameras can be easily attained through spatial
mathematical transformation, and the exterior-orientation parameters of each camera can be
obtained.
As the stereo cameras remain stable during the whole experimental process, the
exterior-orientation and lens-distortion parameters are constant for any epoch. After the
stereoscopic matching and target tracking, the 2D coordinates of the sequential
corresponding image points can be obtained. Therefore, the 3D object coordinates of the
tracking points can be easily calculated through forward intersection, which is derived from
the classic photogrammetric collinearity equations. At every epoch, the 3D point cloud of
the measured surface can be reconstructed.
Structural Parameter Estimation. Structural dynamic parameters can be derived based

on the spatiotemporal analysis of the sequential spatial information of the measured
structure. After obtaining the sequential 3D point clouds, which are calculated through 3D
reconstruction, the 3D displacement fields of the measured surface can be derived through
coordinate subtraction between the subsequent and initial point clouds. In addition,
according to the specific demands in the material testing, the instantaneous velocity and
acceleration fields of the measured surface can also be calculated through the first- and
second-order differential of the displacement with respect to the time interval (Liu et al.,
2015).
As the main structural parameters in mechanical analysis, the full-field 3D strains of
the structure need to be calculated through differentiating the estimated displacement fields
(Eberly, 1999; Meng et al., 2006; Sutton et al., 2009; Hall et al., 2012). The displacement
equations are constructed in the form of polynomials; thus, they can be changed according
to the different numbers of polynomial basis values (Hall et al., 2012). For example, the
data processing in this research adopts a quadratic basis in equation (2) (Sutton et al., 2009),
with the Lagrangian strain tensors in equation (3) using differentials:
8
< UðX;Y;ZÞ ¼ a0 þ a1 X þ a2 Y þ a3 Z þ a4 XY þ a5 YZ þ a6 XZ þ a7 X 2 þ a8 Y 2 þ a9 Z 2
V ¼ b0 þ b1 X þ b2 Y þ b3 Z þ b4 XY þ b5 YZ þ b6 XZ þ b7 X 2 þ b8 Y 2 þ b9 Z 2 ð2Þ
: ðX;Y ;ZÞ
WðX;Y;ZÞ ¼ c0 þ c1 X þ c2 Y þ c3 Z þ c4 XY þ c5 YZ þ c6 XZ þ c7 X 2 þ c8 Y 2 þ c9 Z 2
© 2019 The Authors

8 @V 2 @W 2
> ¼ @U @U 2
> @X þ 2 þ þ
1
> eXX @X @X @X
>
> @V 2 @W 2
>
> @V @U 2
>
> eYY ¼ þ 1
þ þ
> @Y
@Y @Y@Y
@W 2
2
< @V 2
@W @U 2
eZZ ¼ @Z þ 2
1
@Z þ @Z þ @Z ð3Þ
>
> 1 @U @V @V @W @W
> ¼ 1 @U
þ @V
þ @U
>
>
>
eXY 2 @Y @X 2 @X @Y þ @X @Y þ @X @Y
>
> ¼ 1 @V @W @U @U @V @V @W @W
2 @Z þ @Y þ 2 @Y @Z þ @Y @Z þ @Y @Z
1
>
> eYZ
: 1 @W @U @U @U @V @V @W @W
eZX ¼ 2 @X þ @Z þ 2
1
@Z @X þ @Z @X þ @Z @X :
In equations (2) and (3), (U(X,Y,Z), V(X,Y,Z), W(X,Y,Z)) are the 3D displacement values of
the tracking point; (X, Y, Z) are the 3D ground coordinates of the tracking point; (ai, bi, ci)
are the unknown coefficients for the quadric basis; and (eXX, eYY, eZZ, eXY, eYZ, eZX) are the
Lagrangian strain tensors. During the calculation process of the full-field strain, the local
neighbouring points of every tracking point participate in the fitting of the displacement
function using the least-squares method. After obtaining the coefficients of the polynomial
basis values, the strain tensors can be calculated by the above numerical differential
equations.
Experimental Results and Discussions

Both a simulation test and an empirical test using real speckle images were performed
to demonstrate the robustness and reliability of the proposed scheme. In the simulation test,
synthetic images generated with different densities of speckle points were used to verify the
flexibility and reliability of the self-adaptive window matching strategy. In addition, the
empirical test demonstrated the feasibility of the entire high-speed videogrammetric method.
From comparison of the displacement fields, the self-adaptive window matching strategy
can detect more subtle deformation and avoid more mismatches than the fixed-window
matching strategy. Finally, uniaxial compression deformation of rock-like material was
further calculated and analysed.
Simulation Test: Gaussian Speckle Image with Shift Deformation

In order to verify the image-matching strategy, simulated images can provide robust
speckle features and accurate deformation information. The simulated speckle images can be
generated by the sum of the individual Gaussian speckles (Zhou and Goodson, 2001).
Based on equation (4), the initial speckle image can be obtained through the random point
positions and the radius of the speckle; the deformed speckle images can be generated using
the initial speckle image and different deformation parameters:
8 P
< Iinitial ðu; vÞ ¼ Mi¼1 I i exp ðuui Þ2 þðvvi Þ2
2
R
ð4Þ
: Ideform ðu; vÞ ¼ PM Ii exp ðua0 a1 ua2 vui Þ þðvb
2 2
0 b1 ub2 vvi Þ
i¼1 R2 :
In equation (4), (u, v) are the image coordinates of every pixel in the simulated image; (ui, vi)
are the image coordinates of the ith speckle point; Ii is the peak intensity of the ith speckle
point; M is the number of speckle points; R is the radius of the speckle; (a0, a1, a2, b0, b1, b2)
© 2019 The Authors

are the affine deformation parameters; and (a0, b0) are the shift parameters within these
deformation parameters. Corresponding to the different deformation parameters, different
synthetic simulated speckle images can be obtained. As shown in Fig. 7, the simulated speckle
image with an image size of 800 9 800 is generated with M = 16 718 Ii = rand(100~255)
(“rand” is a random-number generator) and R = 3 pixels. For accurate quantitative analysis of
the self-adaptive window matching strategy, a set of deformation parameters is given.
Therefore, based on this initial image, the deformed speckle images can be generated with
b0 = 075 pixels. Fig. 7(b) intuitively displays three different speckle regions.
In the simulation test, the selection of the grid step would affect the number of the
target points. By trial and error in the experiment, it was found that the selection of 2 pixels
as the grid step could balance the computation efficiency and matching resolution.
According to the pre-defined speckle size and speckle texture distribution, the range of the
window radius was set as [3, 30] to guarantee the successful matching of all the target
points in the experiment. At the same time, through comparison of the matching results
under different scenarios of the threshold values, the threshold value d of information
entropy was set as 6 to obtain the optimal matching result for target point tracking and
matching. By the use of the above-mentioned parameter values, a self-adaptive window
matching strategy used in the experiment could provide the most appropriate matching
window for each target point.
Through the self-adaptive window matching strategy, the window size of every target
point was calculated. As shown in Fig. 8(a), the window sizes are generally small in Region
A. Conversely, the window sizes of most points have reached the maximum window value
in Region C. Through the mean calculation, the average window radii in Regions A, B and
C are 676, 1405 and 2898 pixels, respectively. As shown in Fig. 8(b), the precise
displacement of every point can be calculated by LSM. The average displacement of all the
tracking points is 0739 pixels. Given that the theoretical displacement value is 075 pixels,
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
Pn 2 ffi
the root mean square error (RMSE) DR ¼ ð 1 Di Þ=n of the displacement values of all
the tracking points is 0011 pixels. Moreover, as shown in Fig. 9(a), the deformed speckle
images were generated with different shift parameters (a0 = 001~1 pixels or
b0 = 001~1 pixels). As mentioned above, for each deformed speckle image, the average
displacement of all the tracking points can be calculated. Fig. 9(b) plots the RMSE under
(a) (b) (c)
Fig. 7. The initial speckle image and the deformed speckle image through numerical simulation. (a) The
simulated speckle image has three regions: the high-density region (Region A) has 10 558 speckle points; the
medium-density region (Region B) has 3865 speckle points; and the low-density region (Region C) has 2295
speckle points. (b) The initial speckle image. (c) The deformed speckle image with b0 = 075 pixels.
© 2019 The Authors

Fig. 8. Application of the self-adaptive window matching strategy. (a) Distribution and statistics of the support
window size. (b) The 2D displacement field calculated between Figs. 7(b) and (c).
Fig. 9. Accuracy verification of the proposed scheme. (a) Average displacements of the different deformed
images in the u direction (red) and v direction (blue). (b) The RMSE curves of the displacements. The RMSE
values were obtained through a comparison between the calculated values and the theoretical values.
these different translation parameters. From this figure, the proposed self-adaptive window
matching strategy can achieve a matching accuracy of about 001 pixels. Moreover, the
matching errors at the locations of approximately 0, 05 and 1 pixel are relatively small.
This phenomenon is known as pixel locking in the field of computer vision and particle
image velocimetry (Westerweel, 1997; Shimizu and Okutomi, 2005; Cholemari, 2007). The
pixel-locking effect means that sub-pixel location estimation of the target point is biased
towards approximate multiples of 05 pixel and integral pixel positions (Tong et al., 2015d).
In the image sequence tracking and matching, if inappropriate fixed windows are used, a
large number of mismatched points will be generated. A detailed comparison between the
self-adaptive window matching strategy and the fixed-window approach is provided in the
“Results and Analysis” section.
© 2019 The Authors

Empirical Test: Uniaxial Compression Test with Real Speckle Images
Experiment Design. The binocular-vision measurement system, made of two high-

speed cameras, was used to measure the 3D morphological changes of the rock-like material
under uniaxial compression. Following data processing, which is described in the
“Methodology” section, the 3D morphological changes of the rock-like material surface
could be obtained, namely the displacement field, velocity field and strain field.
In this experiment, the high-speed cameras were the CP80-4-M/C-500 model produced
by Optronis GmbH (Kehl, Germany). The capture frequency of the cameras was set to 400
frames per second (fps). The captured images were greyscale images with an image size of
2304 (H) by 1720 (V) pixels. With the advantages of both high frame rate and high
resolution, this high-speed camera model is particularly suitable for high-precision civil
engineering applications. Through the calibration of the camera-lens system, the distortion
of a 50 mm focal-length lens would cause less than 1 pixel image-point deviation. The
image distortion is thus rather small and can be ignored. Therefore, the image-matching
algorithms can be used directly in the original image sequences. Moreover, in this
experimental environment, the 50 mm prime lens can record a more detailed scene than
would have been achieved with a 20 or 35 mm lens. The network configuration of the high-
speed cameras is illustrated in Fig. 10(a).
The rock-like material comprised medical gypsum and water. A different gypsum type,
water quality and mixing ratio can cause different rock material properties. In this empirical
test, the rock-like specimen was a cube with side lengths of 70 mm.
Fig. 10. Uniaxial compression testing of the rock-like material. (a) The camera network configuration. The
rock-like material (yellow circle) is placed in the test machine and the two high-speed cameras record the image
sequence of the rock-like material using intersection photography. (b) An image captured by the high-speed
camera. During the experiment, the loading plate (red box) moves downward.
© 2019 The Authors

(a) (b)
(c) (d)
(e) (f)
Fig. 11. Tracking and matching results of the different-sized windows. Blues represent low displacement; reds
indicate larger displacement. The black boxes in (a) to (e) represent the mismatched regions in the deformation
boundary. The green box in (f) represents the low-accuracy matching region with the large window.
© 2019 The Authors

(a) δ = 4.8 (b) δ = 5.2
(c) δ = 5.6 (d) δ = 6.0
Fig. 12. Tracking and matching results with different information entropy thresholds d from 48 to 60. Blues
represent low displacement; reds indicate larger displacement. Around the deformation edges (regions with low
ZNCC measurement values), the multiple-window method is applied to avoid mismatching and to improve the
accuracy of the measurement results.
Results and Analysis. During the stereo camera calibration, some control points on the
calibration board were selected as check points which were not involved in the calibration
calculation. After obtaining the orientation parameters and lens-distortion parameters, the
RMSE of these check points in the image space were about 01–02 pixels through back
projection, and the RMSE of these points in the object space were about 002 mm through
forward intersection. In terms of the accuracy of the estimation of the interior-orientation
parameters, the standard errors of the calculated principal distance and principal point
coordinates only reached around 01% of their parameter values. Therefore, the stereo
calibration could meet the accuracy requirement of the measured points in this experiment.
As shown in Fig. 10(b), the three main axes of the object coordinate system were defined
based on a right-handed system, where the X–Y plane of this coordinate system was parallel
© 2019 The Authors

Fig. 13. Three-dimensional reconstruction. The point cloud of the rock-like material surface in: (a) Epoch 0;
(b) Epoch 3000; and (c) Epoch 4000.
to the rock-like surface. Moreover, at the beginning of the data processing, the grid step of
the initial speckle image was set as 5 pixels.
After the tracking and matching, the 2D displacement results could be calculated
through image coordinate subtraction between the initial image and the deformed image. As
shown in Fig. 11, the different sizes of target windows resulted in different matching results.
In Figs. 11(a) to (e), small target windows lead to wrong matching (black boxes) in the
deformation boundary. In Fig. 11(f), the mismatch problem was solved using the large target
window, which contains sufficient image information content. However, some subtle
deformation (green box) cannot be detected. In addition, as can be seen from the matching
results, the use of the large target window achieves the effect of filtering. This is because
the different deformations in the same target window are smoothed due to using the same
deformation parameters during the precise sub-pixel image matching. Therefore, the
traditional fixed-window matching method cannot obtain accurate displacement results.
As described in the section “Robust Self-adaptive Window Matching of Forward–Backward
Speckle Image Sequences”, the proposed scheme can effectively solve the above problems.
Under the self-adaptive window matching strategy, every target point will be assigned the optimal
support window size. In order to avoid unlimited growth of the window size, the range of the
window radius was set as [5, 16] during testing. As shown in Figs. 12(a) to (d), the different
displacement results were calculated according to different threshold values d of the information
entropy. This is because the threshold value essentially influences the window size. When the
threshold value increases, the support window of the target point increases accordingly. If the
threshold value is large enough, the window size of every target point will reach the upper limit.
Therefore, the 2D displacement results in Fig. 12(d) are similar to the results shown in Fig. 11(f).
From the above analysis, the proposed scheme can effectively detect subtle deformation areas and
avoid mismatches. Moreover, in the subsequent data processing, the threshold value of the
information entropy was set as 52 to obtain the optimal deformation monitoring results.
Through the calculation described in the section “Full-field 3D Deformation
Construction”, the 3D point cloud of the rock-like surface at any epoch can be reconstructed.
In Figs. 13(a) to (c), the deformation locations are displayed for Epochs 0, 3000 and 4000.
As the loading plate of the test machine moves downwards, the top of the rock-like material
is first destroyed, and then some cracks caused by the uneven stress appear on the rock-like
surface. Moreover, as shown in Fig. 13(c), the matching errors of the image sequence cause
poor accuracy of the 3D reconstruction due to the destruction of the image texture, so a few
spatial points in the 3D model are missing because of the mismatched points.
© 2019 The Authors

Fig. 14. The 3D displacement fields of the rock-like material surface. In (b), (c), (e) and (f), blues represent
low displacement and reds indicate larger displacement. In (a) and (d), greens represent low displacement. The
full-field displacement changes in (a) to (c) (left) are for Epoch 3000, and in (d) to (f) (right) are for Epoch
4000.
© 2019 The Authors

Fig. 15. The strain fields of the rock-like material surface with the displacement window set to 10.
In the uniaxial compression test, the 3D displacement fields of the rock-like material
surface are shown in Fig. 14. With the movement of the loading plate, more and more
deformation occurs in the upper part of the rock-like material. Moreover, as shown in
Figs. 15 and 16, the full-field strains of the rock-like surface can be attained through
equation (2). In the given displacement window, the strain in the centre of this window is
calculated through the numerical difference of the displacement field. Remarkably, these
figures also show that greater strain occurs in the upper part of the rock-like material.
In addition, the presence of noise may result in the deformation data not being smooth.
Therefore, according to individual specific demands, many studies have adopted filtering
methods to reduce the influence of noise (Savitzky and Golay, 1964; Pan et al., 2007).
However, in this paper all the experimental results are provided without any additional
processing. In short, the measured full-field deformation data can help the civil engineering
technologist to evaluate the structural properties and safety factors of rock material.
Conclusion
This paper has proposed a novel scheme to measure full-field 3D deformation of rock-
like material with non-contact high-speed videogrammetry. In the proposed scheme, an
efficient reliability-guided matching strategy is used for stereomatching, and a robust self-
adaptive window matching strategy is used for target tracking. This approach enables the
entire matching process to be accomplished automatically and efficiently with the help of
distributed parallel computing. Compared with single-threaded processing, distributed
parallel computing can increase the computational efficiency proportionately to the number
of CPUs and CPU cores. From the analysis of the experimental results, the following
conclusions can be drawn:
(1) During the simulation test, the proposed self-adaptive window matching strategy
achieved a matching accuracy of about 001 pixels in the different speckle regions.
During the practical compression test, the proposed matching strategy could detect
more subtle deformation and avoid more mismatches than the fixed-window matching
strategy in the deformation edges.
(2) Two high-speed cameras with a sampling frequency of 400 fps were used to measure the
transient 3D morphological changes of the rock-like material under the actual uniaxial
compression test. The high-speed videogrammetric scheme was able to achieve a spatial
location accuracy of about 002 mm in the 3D localisation of the targets.
© 2019 The Authors

Fig. 16. The strain fields of the rock-like material surface (see also Fig. 15). The normal strain tensor and shear
strain tensor in: (a) to (c) (left) for Epoch 3000; and (d) to (f) (right) for Epoch 4000. Note the colour for zero
varies between the six component figures.
© 2019 The Authors

(3) In the empirical test, the transient 3D point clouds of the rock-like material surface at
every epoch (0025 s interval) were accurately calculated by photogrammetric analysis;
the full-field strain could be further obtained by spatiotemporal analysis of the sequential
point clouds. The experimental results showed that the failure mechanism was consistent
with the experimental design of the uniaxial compression.
The complete high-speed videogrammetric scheme embodies the advantages of being
non-contact, flexible and convenient in civil engineering material testing. However, the
computational complexity of the data processing is still relatively high. Therefore, in future
work, the authors will optimise the proposed scheme and apply more texture features to
assist the self-adaptive window matching strategy (Haralick, 1979). The optimal threshold
value of the texture feature also needs to be determined to attain the best image-matching
results in the deformation analysis. In the distributed parallel computing, GPU-based image
matching will also be considered to accelerate the processing speed (Zhi et al., 2016).
Furthermore, more attention will be paid to the time-series analysis of the sequential point
clouds (Liu et al., 2014).
Acknowledgements
This research was substantially supported by the National Natural Science Foundation
of China (Project Numbers 41631178 and 41401531), the National Key Research and
Development of China (Project Numbers 2018YFB0505400 and 2018YFB0505001) and the
Fundamental Research Funds for the Central Universities.
references
Abdel-Aziz, Y. I., 1982. Accuracy of the normal case of close-range photogrammetry. Photogrammetric
Engineering and Remote Sensing, 48(2): 207–213.
Ackermann, F., 1984. Digital image correlation: performance and potential application in photogrammetry.
Photogrammetric Record, 11(64): 429–439.
Adhyapak, S. A., Kehtarnavaz, N. and Nadin, M., 2007. Stereo matching via selective multiple windows.
Journal of Electronic Imaging, 16(1): article 013012. 14 pages.
Alsadik, B., Gerke, M., Vosselman, G., Daham, A. and Jasim, L., 2014. Minimal camera networks for
3D image based modeling of cultural heritage objects. Sensors, 14(4): 5785–5804.
Borghese, N. A. and Cerveri, P., 2000. Calibrating a video camera pair with a rigid bar. Pattern
Recognition, 33(1): 81–95.
Bouguet, J. Y., 2015. Camera Calibration Toolbox for Matlab. www.vision.caltech.edu/bouguetj/calib_doc
[Accessed: 28th November 2018].
Bruck, H. A., McNeill, S. R., Sutton, M. A. and Peters, W. H., 1989. Digital image correlation using
Newton–Raphson method of partial differential correction. Experimental Mechanics, 29(3): 261–267.
Cholemari, M. R., 2007. Modeling and correction of peak-locking in digital PIV. Experiments in Fluids,
42(6): 913–922.
Chu, T. C., Ranson, W. F. and Sutton, M. A., 1985. Applications of digital-image-correlation techniques to
experimental mechanics. Experimental Mechanics, 25(3): 232–244.
Clarke, T. A. and Fryer, J. G., 1998. The development of camera calibration methods and models.
Due, J. and Robinson, A. J., 2013. Reliability of thermal interface materials: a review. Applied Thermal
Engineering, 50(1): 455–463.
Eberly, D., 1999. Least Squares Fitting of Data. https://www.geometrictools.com/Documentation/LeastSquares-
Fitting.pdf [Accessed: 15th January 2018].
€rstner, W. and Wrobel, B. P., 2016. Photogrammetric Computer Vision: Statistics, Geometry, Orientation
Fo
and Reconstruction. Springer, Berlin, Germany. 816 pages.
Fraser, C. S., 1997. Digital camera self-calibration. ISPRS Journal of Photogrammetry and Remote Sensing,
52(4): 149–159.
© 2019 The Authors

Fua, P., 1993. A parallel stereo algorithm that produces dense depth maps and preserves image features.
Machine Vision and Applications, 6(1): 35–49.
Furukawa, Y. and Ponce, J., 2009. Accurate camera calibration from multi-view stereo and bundle
adjustment. International Journal of Computer Vision, 84(3): 257–268.
Fusiello, A., Roberto, V. and Trucco, E., 1997. Efficient stereo with multiple windowing. IEEE
Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico.
Pages 858–863.
Garcia, D., Orteu, J. J. and Penazzi, L., 2002. A combined temporal tracking and stereo-correlation
technique for accurate measurement of 3D displacements: application to sheet metal forming. Journal of
Materials Processing Technology, 125–126: 736–742.
Geng, Y., 2016. Local stereo matching based on information entropy of image. 3D Research, 7(3): 27.
Gruen, A., 2012. Development and status of image matching in photogrammetry. Photogrammetric Record,
27(137): 36–57.
Gupta, R. K. and Cho, S.-Y., 2013. Window-based approach for fast stereo correspondence. IET Computer
Vision, 7(2): 123–134.
Hall, M. S., Long, R., Hui, C. Y. and Wu, M., 2012. Mapping three-dimensional stress and strain fields
within a soft hydrogel using a fluorescence microscope. Biophysical Journal, 102(10): 2241–2250.
Haralick, R. M., 1979. Statistical and structural approaches to texture. Proceedings of the IEEE, 67(5): 786–
804.
Hirschm€ uller, H., 2005. Accurate and efficient stereo processing by semi-global matching and mutual
information. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2: 807–814.
Izquierdo, E., 1999. Disparity/segmentation analysis: matching with an adaptive window and depth-driven
segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 9(4): 589–607.
Jeon, J., Kim, C. and Ho, Y.-S., 2002. Sharp and dense disparity maps using multiple windows. Lecture Notes
in Computer Science, 2532: 1057–1064.
Kanade, T. and Okutomi, M., 1994. A stereo matching algorithm with an adaptive window: theory and
experiment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(9): 920–932.
Leifer, J., Weems, B. J., Kienle, S. C. and Sims, A. M., 2011. Three-dimensional acceleration measurement
using videogrammetry tracking data. Experimental Mechanics, 51(2): 199–217.
Li, J., Liu, Y., Du, S., Wu, P. and Xu, Z., 2016. Hierarchical and adaptive phase correlation for precise
disparity estimation of UAV images. IEEE Transactions on Geoscience and Remote Sensing, 54(12):
7092–7104.
Liu, Z. and Gao, J., 2011. Deformation-pattern-based digital speckle correlation for coefficient of thermal
expansion evaluation of film. Optics Express, 19(18): 17469–17479.
Liu, J., Jin, L., Liang, X. and Tang, Z., 2010. Videogrammetric system for dynamic deformation
measurement during metal sheet welding processes. Optical Engineering, 49(3): article 033601.
Liu, S., Peng, L. and Yuan, L. L., 2014. Research on deformation analysis of time-varying point cloud.
Applied Mechanics and Materials, 556–562: 3450–3455.
Liu, J., Zhang, Y. and Dong, X., 2015a. Local stereo matching based on the improved matching cost
function and the adaptive window. 8th International Congress on Image and Signal Processing, Shenyang,
China. Pages 287–292.
Liu, X., Tong, X., Yin, X., Gu, X. and Ye, Z., 2015b. Videogrammetric technique for three-dimensional
structural progressive collapse measurement. Measurement, 63: 87–99.
Matsumoto, R., Kubota, M. and Miyazaki, N., 2016. Development of deformation measurement system
consisting of high-speed camera and digital image correlation, and its application to the measurement of
large inhomogeneous deformations around the crack tip. Experimental Techniques, 40(1): 91–100.
Mekuz, N., Derpanis, K. G. and Tsotsos, J. K., 2006. Adaptive step size window matching for detection.
18th IEEE Computer Society International Conference on Pattern Recognition, Kowloon, Hong Kong, 2:
259–262.
Men, G. Z., Chai, J. L. and Zhao, J., 2009. A fast matching algorithm with an adaptive window based on
quasi-dense method. International Conference on Machine Learning and Cybernetics, Baoding, China.
Pages 12–15.
Meng, L. B., Jin, G. C., Yao, X. F. and Yeh, H. Y., 2006. 3D full-field deformation monitoring of fiber
composite pressure vessel using 3D digital speckle correlation method. Polymer Testing, 25(1): 42–48.
Murray, C., Hoag, A., Hoult, N. A. and Take, W. A., 2015. Field monitoring of a bridge using
digital image correlation. Proceedings of the Institution of Civil Engineers – Bridge Engineering,
168(1): 3–12.
Pan, B., 2009. Reliability-guided digital image correlation for image deformation measurement. Applied Optics,
48(8): 1535–1542.
© 2019 The Authors

Pan, B., 2011. Recent progress in digital image correlation. Experimental Mechanics, 51(7): 1223–1235.
Pan, B., Xie, H., Guo, Z. and Hua, T., 2007. Full-field strain measurement using a two-dimensional
Savitzky–Golay digital differentiator in digital image correlation. Optical Engineering, 46(3): article
033601.
Pan, B., Asundi, A., Xie, H. and Gao, J., 2009a. Digital image correlation using iterative least squares and
pointwise least squares for displacement field and strain field measurements. Optics and Lasers in
Engineering, 47(7–8): 865–874.
Pan, B., Qian, K., Xie, H. and Asundi, A., 2009b. Two-dimensional digital image correlation for in-plane
displacement and strain measurement: a review. Measurement Science and Technology, 20(6): article
062001.
Pan, B., Lu, Z. and Xie, H., 2010. Mean intensity gradient: an effective global parameter for quality
assessment of the speckle patterns used in digital image correlation. Optics and Lasers in Engineering,
48(4): 469–477.
Pan, B., Wu, D. and Xia, Y., 2012. Incremental calculation for large deformation measurement using
reliability-guided digital image correlation. Optics and Lasers in Engineering, 50(4): 586–592.
Perras, M. A. and Diederichs, M. S., 2014. A review of the tensile strength of rock: concepts and testing.
Geotechnical and Geological Engineering, 32(2): 525–546.
Pollefeys, M., Koch, R. and van Gool, L., 1999. Self-calibration and metric reconstruction inspite of
varying and unknown intrinsic camera parameters. International Journal of Computer Vision, 32(1): 7–25.
Safri, S. N. A., Sultan, M. T. H., Yidris, N. and Mustapha, F., 2014. Low velocity and high velocity impact
test on composite materials – a review. International Journal of Engineering and Science, 3(9): 50–60.
Salazar, F. and Barrientos, A., 2013. Surface roughness measurement on a wing aircraft by speckle
correlation. Sensors, 13(9): 11772–11781.
Savitzky, A. and Golay, M. J. E., 1964. Smoothing and differentiation of data by simplified least squares
procedures. Analytical Chemistry, 36(8): 1627–1639.
Shi, H., Zhu, H., Wang, J., Yu, S.-Y. and Fu, Z.-F., 2016. Segment-based adaptive window and multi-
feature fusion for stereo matching. Journal of Algorithms and Computational Technology, 10(1): 3–11.
Shimizu, M. and Okutomi, M., 2005. Sub-pixel estimation error cancellation on area-based matching.
International Journal of Computer Vision, 63(3): 207–224.
Sutton, M. A., Orteu, J. J. and Schreier, H. W., 2009. Image Correlation for Shape, Motion and
Deformation Measurements: Basic Concepts, Theory and Applications. Springer, New York, USA. 322
pages.
Tang, L., Wu, C. and Chen, Z., 2002. Image dense matching based on region growth with adaptive window.
Pattern Recognition Letters, 23(10): 1169–1178.
Tong, X., Ye, Z., Xu, Y., Tang, X., Liu, S., Li, L., Xie, H., Wang, F., Li, T. and Hong, Z., 2014.
Framework of jitter detection and compensation for high resolution satellites. Remote Sensing, 6(5): 3944–
3964.
Tong, X., Li, L., Liu, S., Xu, Y., Ye, Z., Jin, Y., Wang, F. and Xie, H., 2015a. Detection and estimation
of ZY-3 three-line array image distortions caused by attitude oscillation. ISPRS Journal of Photogrammetry
and Remote Sensing, 101: 291–309.
Tong, X., Xu, Y., Ye, Z., Liu, S., Li, L., Xie, H., Wang, F., Gao, S. and Stilla, U., 2015b. An improved
phase correlation method based on 2-D plane fitting and the maximum kernel density estimator. IEEE
Geoscience and Remote Sensing Letters, 12(9): 1953–1957.
Tong, X., Xu, Y., Ye, Z., Liu, S., Tang, X., Li, L., Xie, H. and Xie, J., 2015c. Attitude oscillation
detection of the ZY-3 satellite by using multispectral parallax images. IEEE Transactions on Geoscience
and Remote Sensing, 53(6): 3522–3534.
Tong, X., Ye, Z., Xu, Y., Liu, S., Li, L., Xie, H. and Li, T., 2015d. A novel subpixel phase correlation
method using singular value decomposition and unified random sample consensus. IEEE Transactions on
Geoscience and Remote Sensing, 53(8): 4143–4156.
Tong, X., Gao, S., Liu, S., Ye, Z., Chen, P., Yan, S., Zhao, X., Du, L., Liu, X. and Luan, K., 2017a.
Monitoring a progressive collapse test of a spherical lattice shell using high-speed videogrammetry.
Tong, X., Ye, Z., Li, L., Liu, S., Jin, Y., Chen, P., Xie, H. and Zhang, S., 2017b. Detection and
estimation of along-track attitude jitter from ZiYuan-3 three-line-array images based on back-projection
residuals. IEEE Transactions on Geoscience and Remote Sensing, 55(8): 4272–4284.
Tsai, R., 1987. A versatile camera calibration technique for high-accuracy 3D machine vision metrology using
off-the-shelf TV cameras and lenses. IEEE Journal of Robotics and Automation, 3(4): 323–344.
Veksler, O., 2003. Fast variable window for stereo correspondence using integral images. IEEE Computer Society
Conference on Computer Vision and Pattern Recognition, Madison, Wisconsin, USA. Pages 556–661.
© 2019 The Authors

Westerweel, J., 1997. Fundamentals of digital particle image velocimetry. Measurement Science and
Technology, 8(12): 1379–1392.
Yao, X. F., Meng, L. B., Jin, J. C. and Yeh, H. Y., 2005. Full-field deformation measurement of fiber
composite pressure vessel using digital speckle correlation method. Polymer Testing, 24(2): 245–251.
Zhang, Z., 2000. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 22(11): 1330–1334.
Zhang, J., Li, D.-X. and Zhang, M., 2010. Fast stereo matching algorithm based on adaptive window.
International Conference on Audio, Language and Image Processing, Shanghai, China. Pages 138–142.
Zhang, D., Xie, G., Li, Y. and Liu, J., 2015. Strain and mechanical properties of the VCM multilayer sheet
and their composites using the digital speckle correlation method. Applied Optics, 54(25): 7534–7541.
Zhi, X. Y., Yan, J. H., Hang, Y. Q. and Wang, S. F., 2016. Realization of CUDA-based real-time
registration and target localization for high-resolution video images. Journal of Real-Time Image
Processing, 1–12. https://doi.org/10.1007/s11554-016-0594-y
Zhou, P. and Goodson, K. E., 2001. Subpixel displacement and deformation gradient measurement using
digital image/speckle correlation. Optical Engineering, 40(8): 1613–1620.
Resume
Mesurer la deformation d’ensemble d’un objet est un enjeu important pour le test de materiaux en genie
civil. Cet article presente un dispositif videogrammetrique flexible pour mesurer la deformation d’ensemble d’un
materiau artificiel ayant la consistance de la roche et soumis a une compression uniaxiale. Dans ce systeme,
deux cameras a haute vitesse ont ete utilisees pour mesurer les changements de morphologie spatiale de l’objet
apres qu’une texture de speckle a ete pulverisee sur sa surface. Une strategie auto-adaptative robuste
d’appariement de fenêtres est proposee pour extraire des deplacements precis sur les points de contrôle. Ainsi
des nuages de points 3D et une deformation d’ensemble peuvent être calcules par une analyse
photogrammetrique spatiotemporelle. Une simulation et des tests empiriques montrent que la strategie auto-
adaptative d’appariement de fenêtres appliquee a des sequences videogrammetriques a haute vitesse d’images
de speckle permet de detecter des deformations plus tenues et d’eviter plus d’erreurs qu’une strategie
d’appariement de fenêtres fixes.
Zusammenfassung
Eine fl€
achenhafte Deformationsmessung f€ur spezifische Materialien spielt eine wichtige Rolle f€ur die
Materialpr€
ufung im Bauingenieurwesen. Dieser Beitrag stellt einen Entwurf zur videogrammetrischen Erfassung
von fl€achenhaften Deformationen eines k€unstlichen, felsartigen Materials unter einachsiger Kompression vor. In
diesem Entwurf werden zwei Hochgeschwindigkeitskameras eingesetzt, um die r€aumlichen, morphologischen
Ver€anderungen der Materialobfl€ache zu erfassen. Die Materialoberfl€ache wurde mit einem Grautonmuster
uht. Zur Auswertung wird eine robuste, selbstadaptierende fl€achenhafte Zuordnungsstrategie
eingespr€
vorgeschlagen, um die genauen Verschiebungen von Tracking-Punkten festzustellen. Abschließend k€onnen 3D
Punktwolken und die fl€achenhafte Deformation durch photogrammetrische und r€aumlich-zeitliche Analyse
gerechnet werden. Sowohl Simulationen, als auch empirische Versuche haben gezeigt, dass die vorgeschlagene
Zuordnungsstrategie, angewandt auf Bildsequenzen von Hochgeschwindigkeitskameras, feinere Deformationen
erkennen kann, und mehr Fehlzuordnungen im Vergleich zu einer Zuordnungsstrategie mit fest vorgegebenen
Fenstern vermieden werden k€onnen.
Resumen
La medicion de deformacion de materiales especıficos es un tema importante en ingenierıa civil. Este
artıculo presenta un esquema videogrametrico flexible para medir la deformacion de un material artificial
© 2019 The Authors

similar a una roca bajo compresion uniaxial. En este esquema, se utilizaron dos camaras de alta velocidad
para medir los cambios morfologicos espaciales de la superficie del material, que se marcaron con un patron
moteado. Luego se propone una estrategia robusta de correspondencia de ventana autoadaptativa para extraer
desplazamientos precisos de los puntos en seguimiento. Finalmente, las nubes de puntos 3D y la deformacion
se pueden calcular mediante analisis fotogrametrico y espaciotemporal. La simulacion y las pruebas empıricas
demostraron que la estrategia de correspondencia de ventana autoadaptativa utilizada con secuencias de
im
agenes videogrametricas de alta velocidad puede detectar una deformacion mas sutil y evitar mas errores de
correspondencia que una estrategia de correspondencia de ventana fija.
摘要
在土木工程材料测试中, 特殊材料的全场形变测量是一个重要研究问题。本文提出了一种灵活的视频测
量方案以测量岩石材料在单轴压缩下的全场形变。在这个测量方案中, 被测材料的表面需要被喷涂散斑图案,
并且两台高速相机被用来观测材料表面的空间形态变化, 然后本文提出了一种自适应窗口匹配方法以精确提
取全场跟踪点的位移值。最后, 通过摄影测量解析与时空分析计算被测对象的三维点云和全场形变。为了验
证本方法的可靠性, 依次实施模拟实验和实际测试。实验结果表明, 在高速散斑序列影像中, 自适应窗口匹配
策略相对于固定窗口匹配策略能够探测更微小的形变并且避免更多的误匹配。
© 2019 The Authors

View publication stats

Full Field Deformation Measurement by Videogrammetry Using Self Adaptive Window Matching

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Full Field Deformation Measurement by Videogrammetry Using Self Adaptive Window Matching

Uploaded by

Copyright:

Available Formats

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

Full‐ﬁeld deformation measurement by videogrammetry using self‐adaptive

Article in The Photogrammetric Record · March 2019

Zhen Ye Cheng Zhao

SEE PROFILE SEE PROFILE

Shijie Liu Huan Xie

SEE PROFILE SEE PROFILE

NSFC funding View project

The user has requested enhancement of the downloaded file.

FULL-FIELD DEFORMATION MEASUREMENT BY

†Also at Institute of Photogrammetry and Remote Sensing, Technische Universit€

© 2019 The Authors

Keywords: deformation measurement, full-ﬁeld deformation, high-speed camera,

© 2019 The Authors

directly used to calculate high-precision coordinates of tracking points in speckle image

© 2019 The Authors

Camera network configuration

Digital speckle spraying

Artificial light arrangement

Interior- and exterior-

Sub-Controller One Sub-Controller Two

Precise matching: Precise matching:

Full-field 3D deformation measurement

3D full-field displacement estimation

Full-field strain calculation with the

Camera Network Conﬁguration

© 2019 The Authors

Stereoscopic Matching and Target Tracking

© 2019 The Authors

Stereoscopic Matching of the Initial Left–Right Speckle Image Pairs. Stereoscopic

Initial image Deformed image

Reference image block

© 2019 The Authors

Robust Self-adaptive Window Matching of Forward–Backward Speckle Image

© 2019 The Authors

Camera One Camera Two

Point list stored in

information. Unlike traditional stereomatching methods in computer vision, LSM also

© 2019 The Authors

Distributed Parallel Computing. In order to achieve the purpose of on-site computing,

© 2019 The Authors

(a) Target tracking and matching

Target image ZNCC

(b) Adaptive-window matching

7×7 pixels 9×9 pixels 11×11 pixels

13×13 pixels 15×15 pixels 17×17 pixels

(c) Multiple-window matching

© 2019 The Authors

Full-ﬁeld 3D Deformation Measurement

Local network transmission

③ Initial target points

Parallel computing Parallel computing

© 2019 The Authors

Full-ﬁeld 3D Deformation Construction. Stereo calibration and 3D point-cloud

Structural Parameter Estimation. Structural dynamic parameters can be derived based

© 2019 The Authors

Experimental Results and Discussions

Simulation Test: Gaussian Speckle Image with Shift Deformation

© 2019 The Authors

(a) (b) (c)

© 2019 The Authors

© 2019 The Authors

Empirical Test: Uniaxial Compression Test with Real Speckle Images