1 - A Probabilistic Method For Fractured Cultural Relics

A Probabilistic Method for Fractured Cultural Relics
Automatic Reassembly
HAIPING WANG, Wuhan University, Wuhan, Hubei, China

YUFU ZANG, Nanjing University of Information Science & Technology, Jiangsu, China
FUXUN LIANG and ZHEN DONG, Wuhan University, Wuhan, Hubei, China
HONGCHAO FAN, Norwegian University of Science and Technology, Norway
BISHENG YANG, Wuhan University, Wuhan, Hubei, China
Masses of fragile cultural relics are dug out in fragments due to long-standing burying and their fragility, which must be
reassembled to play a role in cultural heritage study. However, it is very challenging to automatically reassemble a large
collection of fragments of unknown geometric shapes. In this article, a novel probabilistic method for fractured cultural
relics automatic reassembly is proposed to solve the problem in terms of good accuracy, efficiency, and robustness. First, a
set of matching units are detected and described by the 2D Link-Chain Descriptors (LCD) and the 3D Spatial-Distribution
Descriptors (SDD). Second, the pairwise reassembly probability is calculated by combining the similarities of LCD and SDD
descriptors, then the collision detection is conducted to eliminate the incorrect overlapping pairs. Finally, a global optimal
reassembly solution is obtained by iterative graph optimization with the constrains of the loop-closures and overlap restric-
tions. Comprehensive experiments with eight challenging datasets demonstrate that the proposed method achieved good
performance in terms of minor reassembly errors, efficiency and robustness to noise, varying point density and completeness.
CCS Concepts: • Applied computing → Arts and humanities; Fine arts; • Computing methodologies → Computer
5
graphics; Shape modeling; Point-based models; Shape analysis;
Additional Key Words and Phrases: Point clouds, probabilistic framework, cultural heritage, feature extraction, registration
ACM Reference format:

Haiping Wang, Yufu Zang, Fuxun Liang, Zhen Dong, Hongchao Fan, and Bisheng Yang. 2021. A Probabilistic Method for
Fractured Cultural Relics Automatic Reassembly. J. Comput. Cult. Herit. 14, 1, Article 5 (January 2021), 25 pages.
https://doi.org/10.1145/3417711
This research is jointly supported by the National Natural Science Foundation of China Projects (Grants No. 41901403 and No. 41725005),
the Fundamental Research Funds for the Central Universities (Grant No. 2042020kf0015), and China Postdoctoral Science Foundation (Grant
No. 2018M642913).
Authors’ addresses: H. Wang, State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, and School of
Geodesy and Geomatics, Wuhan University, 129 Luoyu Road, Wuhan, Hubei, China; email: hpwang@whu.edu.cn; Y. Zang, School of Remote
Sensing & Geomatics Engineering, Nanjing University of Information Science & Technology, 219 Ningliu Road, Nanjing, Jiangsu, China;
email: 3dmapzangyufu@nuist.edu.cn; F. Liang, Z. Dong (corresponding author), and B. Yang (corresponding author), State Key Laboratory
of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, 129 Luoyu Road, Wuhan, Hubei, China; emails:
{liangfuxun, dongzhenwhu, bshyang}@whu.edu.cn; H. Fan, Department of Civil and Environmental Engineering, Norwegian University of
Science and Technology, 7491 Trondheim, Norway; email: hongchao.fan@ntnu.no.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first
page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy
otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions
from permissions@acm.org.
© 2021 Association for Computing Machinery.
1556-4673/2021/01-ART5 $15.00
https://doi.org/10.1145/3417711
ACM Journal on Computing and Cultural Heritage, Vol. 14, No. 1, Article 5. Publication date: January 2021.
5:2 • H. Wang et al.
1 INTRODUCTION
As the witness of human history and civilization, cultural relics have attracted more and more attention in
recent years [60]. A mass of fragile cultural relics (e.g., hollow sculptures, frescos, and porcelain) are dug out at
archaeological sites [27]. These cultural heritages are often found in fragments due to their long-standing burying
and fragility, which must be reassembled and modeled to join their force in cultural heritage study [5]. However,
it is very challenging to reassemble a large collection of fragments due to their poor thickness and the unknown
geometric shapes [66]. A fast and accurate virtual repair method is desperately needed. In the past few years,
automatic reassembling fractured 3D cultural relics has gained increasing importance in the field of archeology
[62], which can be roughly categorized into thin fragments reassembly and thick fragments reassembly according
to the thickness of the fragments [66], or 2D reassembly, 2.5D reassembly, and 3D reassembly depending on
the dimension of fragments [4]. These virtual, computer-assisted repair methods provide plenty of advantages,
including the non-contact access to cultural relics and the ability to manipulate complex and abundant 3D shapes.
Interestingly, domains of forensics and computer-assisted surgery have also shown interest in such techniques
and have carried out related research [4].
In general, the virtual fragments reassembly problem is formulated as a multiview and low-overlap point cloud
registration process [15], which involves the robust and descriptive feature representation of fracture surfaces
[3], the corresponding fracture faces searching and matching, and the global poses re-estimation of multiple
fragments using abundant observations (e.g., loop-closure and overlap constraints) [42]. Although the existing
methods achieved good performance for the thick fragments reassembly, they still have some common limita-
tions for the thin fragments reassembly. First, the robust and descriptive feature representation is challenging
especially for the thin fracture faces [24]. Second, reassembly must be based solely on fragment geometry, as in-
formation such as color and texture have long been destroyed [27]. Third, the overlap between thin fracture faces
is extremely limited, which leads to insufficient common points and poor reassembly accuracy. Additionally, the
archaeological fragments reassembly problem lacks the guidance of a general theoretical framework [40].
To address the above challenges, a novel probabilistic method for fractured cultural relics reassembly is pro-
posed to improve the efficiency and robustness of archaeological fragments reassembly. Compared with the
existing literature, the main contributions of this article are as follows:
• The universality of the probabilistic framework. The proposed probabilistic framework is generally ap-
plicable to many reassembly needs, including restoration of cultural relics, multi-view point cloud regis-
tration, and even medical assistance.
• The proposed method significantly improves the efficiency of fractured cultural relics reassembly by the
fast acquisition of robust matching regions and the time-efficient global graph optimization strategy.
• The proposed method dramatically improves the robustness of fractured cultural relics reassembly to
noise, varying point density and geometric shapes, and makes it possible to reassemble thin fragments
with ultra-low overlap by combining local similarities (i.e., the similarity of 2D link-chain descriptor and
3D spatial-distribution descriptor) and global constraints (i.e., collision detection, overlap restriction, and
loop closure).
The rest of this article is organized as follows. Section 2 reviews the existing related work and analyzes their
strengths and weaknesses. Section 3 proposes a general reassembly framework based on probability theory. In
Section 4, the detailed method is presented based on the framework mentioned. Then the proposed method is
verified by experimental studies in Section 5. Section 6 draws a conclusion.
2 RELATED WORK
In the past few years, automatically reassembling fractured 3D cultural relics has increasingly gained impor-
tance in the field of archeology [4], which can be roughly categorized into thin fragments reassembly and thick
A Probabilistic Method for Fractured Cultural Relics Automatic Reassembly • 5:3
fragments reassembly according to the thickness of the fragments [66], or 2D reassembly, 2.5D reassembly, and
3D reassembly depending on the dimension of fragments [4], or known geometry-based methods and unknown
(i.e., freeform) geometry-based methods based on the assumption of geometric shape [39]. This article will focus
on thin (thick to test, meanwhile), 3D, and unknown geometry-based methods.
2.1 Known Geometry-based Fragments Reassembly

To make simplification and improve robustness, some pioneering studies imposed the assumption of known
geometry to the fractured antique (e.g., relative flatness, axial symmetry, or rotational symmetry). For instance,
Da Gama Leitão and Stolfi [12], Kong and Kimia [31], and Papaodysseus et al. [41] assumed relative flatness to
reduce the problem as two-dimensional (2D) reassembly and utilized the elastic curve matching. These methods
achieved good performance for relatively flat objects, such as murals and steles. However, the assumption of
relative flatness is not fulfilled in other cases. Hong et al. [26], Kampel and Sablatnig [28], Willis and Cooper [59],
and Son et al. [54] addressed the fragments reassembly based on the geometric priors of rotational symmetry,
by estimating of the axis and principle curves extracted from the scanned fragments, which is only applicable to
specialized pottery sherds. In general, the known geometry-based fragments reassembly methods heavily depend
on the assumption of fractured antique shapes, which limit their feasibility to freeform fragments reassembly.
2.2 Freeform Geometry-based Fragments Reassembly

To extend the reassembly methods to the fractured antique with unknown or freeform geometry, some pioneer-
ing studies formulated the fragments reassembly problem as a multi-view and low-overlap point cloud registra-
tion process, which involves three critical processes: the robust and descriptive feature representation of fracture
faces, pairwise registration by the corresponding fracture faces searching and matching, and the global poses
re-estimation of multiple fragments using abundant observations (e.g., loop-closure and overlap constraints).
2.2.1 Matching Units and Descriptors Calculation. For thick cultural relics, scholars believe that even after
abrasion, the overall shape and some local features of the fractured surfaces are still available. Therefore, this
kind of method first extracts the fractured surfaces by region growing strategy [3], then detects the concave
and convex points with the local maximum curvature as the keypoints (i.e., matching units) of the fractured
surfaces [11]. For instance, Huang et al. [27] constructed the local clusters on the fractured surface by using
the growth strategy based on the consistency of volume integral invariants, and the detailed shape description
and boundary description were carried out as the descriptor. Based on the detected matching units, Li et al. [33]
calculated the Point Feature Histograms (PFH) of curvature, normal angle and point-to-plane distance as the
descriptors for correspondences matching. Altantsetseg et al. [2] developed a descriptor that involves not only
the point features but also curves along with the main directions of the point cluster by applying 2D Fourier
series, but this requires an excellent extraction algorithm and an accurate local coordinate system. Weinmann
et al. [58] utilized the distances from the keypoints to the projection surface as the 2D descriptors matrix of the
fracture surface. This kind of method achieved good performance in the scenario of thick cultural relics without
erosion. However, the stable characteristics on the fractured surfaces are not enough to support the construction
of descriptors.
For thin cultural relics, the contours are widely used as matching units. For instance, Zheng et al. [66] built
a local Cartesian coordinate system by calculating normal and tangent directions on sampled contour points
as the matching unit. To accelerate the matching speed, Aiger et al. [1] built 4-points congruent sets (4PCS) as
local descriptors, and exploited the rule that intersection ratios of the diagonals are invariant under the affine
transformation for correspondences matching, afterward, its variants super4PCS [35], GD-4PCS [22], and V4PCS
[61] were proposed to improve the performance. To enhance descriptiveness, Dong et al. [16] put forward a
state-of-the-art descriptor, binary shape context (BSC), by describing density and distance grids with Gaussian
probability model. Raposo and Barreto [47] conditionally combined two contour points to construct descriptors
with connection vector, length and point normal, tangent. To reduce the influence of contours’ multiple overlaps,
Huang et al. [27] describe sub-contours cut by inflexions with volume integral invariants and global tangents.
To improve the robustness, Rasheed and Nordin [48] divided contour more intensively and randomly, and the
slope, curvature and deflection of each segment were calculated. Especially, these methods work well on the data
sets with small overlaps, and require no assumption about the initial positions. However, the local features are
difficult to extract and describe for thin fractured surfaces.
Apart from the geometric information, the color or texture information is also used for thin cultural relics re-
assembly. Oxholm and Nishino [39] developed a continuous linear descriptor using contour’s color information
for the thin cultural relics reassembly. Savelonas et al. [51] employed a heuristic test to evaluate the plausibility
of facet pairs, by comparing the number of feature curves associated with each facet, as well as the geometric
texture of associated intact surfaces. However, the color or texture information may have been changed or
even vanished due to long-time burying. Besides, to improve the reassembly accuracy, curves [4], and textures
[50] derived from original surfaces [27] are also used as constraints for the determination of matching regions
between fracture surfaces.
2.2.2 Fracture Surfaces Matching and Refinement. After matching units detection and descriptors calculation,
the search of corresponding matching units between two fragments has also been widely concerned. Altantset-
seg et al. [2], Zhang et al. [65], and Zheng et al. [66] applied the exhaustive coverage of the search space, tried
to register all of them, and evaluated each time afterwards. To reduce their time complexity, the rough global
matching policy was introduced, Andreadis et al. [4] and Mellado et al. [36] provided coarse alternative pairs
and introduced human intervention for higher accuracy. To ensure the fully automatic process, Parikh et al. [42]
obtained the largest cluster of feature correspondences between fragments by building the adjacency matrix
and applying principal components analysis (PCA) [18] and iterative greedy scheme. Furthermore, alternative
matching pairs were determined by the similarity of descriptors based on spatial features of patches on fracture
surfaces [27], point local features on fracture surface [33], points-plane distances [55], and geometric character-
istics of sub-contours [53]. However, these methods are not as robust as the exhaustive searching-based methods.
For a good compromise between efficiency and robustness of correspondence determinations between pair-wise
fractured facets, the genetic algorithm [56], random sample consistency (RANSAC) [1], norm degradation [34],
and enhanced simulated annealing and sparse ICP algorithm [34] were introduced.
More recently, deep learning has been widely used in image registration and achieved remarkable success
[32] Innovatively, PointNet [44] and PointNet++ [45] were proposed to describe the global and local features of
point clouds. Afterwards, Dang et al. [13], Elbaz et al. [19], Ge [23], and Kang et al. [29] matched the multi-view
point clouds based on robust local features. Additionally, Dang et al. [13] and Lin et al. [21] utilized the neural
network to directly learn spatial transformations between two point clouds. All above can be well extended to
cultural relics splicing. As a pioneer, Rasheed and Nordin [48] applied a three-layer full-connect Neural Network
to determine the corresponding sub-contour generated by intensive cutting between fragments.
To address the potential inconsistent matches and redistribute the reassembly errors of pair-wise fractured
facets matching, some global poses re-estimation methods using abundant observations and constraints (e.g., col-
lision detection [4], cross detection [34], overlap restriction [53], and loop closure constraints) were proposed. For
instance, Huang et al. [27] and Zhang et al. [65] first aligned each pair-wise fractured facet locally and hierarchi-
cally, then refined them with iterative graph optimization strategy to redistribute the accumulated errors. With-
out the correspondences update, these approaches only redistribute the reassembly errors over the fragments and
are incapable of reducing the total registration errors [20]. To overcome this limitation, several joint registration-
based methods have been studied. Zhe et al. [38] proposed a clustering-based method for joint registration of
multiple point clouds extracted from the human femur CT model. Theoretically, this method showed good po-
tential for surface reconstruction in both medical imaging computing (MIC) and computer-assisted interventions
(CAIs) community. However, its outstanding potential for multiple fragments reassembly has not been verified.
Although the existing methods achieved good performance for the thick fragments reassembly, they still have
some common limitations for the thin fragments reassembly. First, a robust and descriptive feature representa-
tion is challenging especially for the thin fracture faces. Second, reassembly must be based solely on fragment
geometry, as information such as color and texture have long been destroyed [27]. Third, the overlap between
thin fracture surfaces is extremely limited, which leads to insufficient common points and poor reassembly accu-
racy. Additionally, the archaeological fragments reassembly problem lacks the guidance of a general theoretical
framework.
3 THE PROPOSED PROBABILISTIC FRAMEWORK

In this article, we formulate the fragments reassembly as a probability optimization problem. Given a collection
of fragments F = { f i } and a set of matching units (e.g., fragments themselves, fracture surfaces, contours, feature
facets) of all fragments S = {s i }, the task of fragments reassembly is to find a globally consistent reconstruction
of the fractured antique.
First of all, let s i and s j be two corresponding matching units from S and ms i and ms j be the overlapping
area of s i and s j , respectively. The task of rigid registration of s i and s j is to estimate the rotation matrix
R ∈ SO (3) and translation vector T ∈ R(3) by using the correspondences between ms i and ms j . Then regard
Z i j = φ(s i , s j , ms i , ms j , R,T ) as the degree of the match (s i , s j ), which can be formulated to maximize the joint
conditional probability [49] as Equation (1):
p(Z i j |s i , s j ) = p(ms i , ms j |s i , s j ) ∗ p(R,T |ms i , ms j , s i , s j ) ∗ p(Z i j |R,T , ms i , ms j , s i , s j ) (1)
For (s i , s j ) is an objective fact independent from Z i j and (R,T ) once (ms i , ms j ) is chosen, Equation (1) can
be simplified as
p(Z i j |s i , s j ) = p(ms i , ms j |s i , s j ) ∗ p(R,T |ms i , ms j ) ∗ p(Z |R,T , ms i , ms j ) (2)
Similarly, after giving (R,T ), (ms i , ms j ) become the prior fact to the posterior Z , then Equation (2) can be
further reduced as
p(Z i j |s i , s j ) = p(ms i , ms j |s i , s j ) ∗ p(R,T |ms i , ms j ) ∗ p(Z i j |R,T ) (3)
where p(ms i , ms j | s i , s j ) is the possibility distribution of (ms i , ms j ) when (s i , s j ) is given; p(R,T | ms i , ms j )
is the possibility distribution of (R,T ) when (ms i , ms j ) is chosen; and p(Z i j | R,T ) is the evaluation and accept-
ability of (ms i , ms j , R,T ).
After all of the (s i , s j )s being analyzed, the globally optimal registration can be donated as the maximization
of Equation (4):

n−1
n
p(Z |F ) = p(S |F ) ∗ (δ (s i , s j ) ∗ p(Z i j |s i , s j )) (4)
i j=i+1
where n is the number of matching units in S. δ (s i , s j ) is an indicator function, which is set as 1, if (s i , s j ) is
chosen as the corresponding pair; otherwise, it equals to 0.
Because F → S is a certain operational process, not a probability distribution, Equation (4) can be updated as

n−1
n
p(Z |F ) = (δ (s i , s j ) ∗ p(Z i j |s i , s j )) (5)
i j=i+1
To avoid the conflicts in the global reassembly solution Z , the probability with restrictions is formulated as
Equation (6):

n−1
n
p(Z |F ) = (δ (s i , s j ) ∗ p(Z i j |s i , s j )) + Rest(Z ) (6)
i j=i+1
Fig. 1. Working flowchart of the proposed method.
Fig. 2. The matching units construction: (b) initial fractured facets, (c) refined fractured facets, and (d) the matching units.
where Rest(Z ) are global constraints (e.g., collision detection, cross detection, overlap restriction, loop closure
constraints) based on the set of overlapping areas.
4 METHODOLOGY
In this section, we propose a fractured cultural relics reassembly algorithm based on the introduced probabilistic
framework (Section 3) with three key components: robust matching units detection and description (Sections 4.1
and 4.2), efficient pairwise registration with the matching units pair (Section 4.3, i.e., p(Z i j | s i , s j )), and the po-
tential inconsistent matches elimination with global constraints (Section 4.4, i.e., p(Z |F )). Figure 1 illustrates an
overview of the proposed fractured cultural relics reassembly algorithm. First, the set of matching units S are
detected by region growing-based strategy, then described by the 2D Link-Chain Descriptors (LCD). Second,
the potential overlapping pair (ms i , ms j ) of each matching unit pair (s i , s j ) is rapidly extracted by applying
the sliding match strategy to the LCD descriptors, then the 3D Spatial-Distribution Descriptors (SDD) are fur-
ther calculated for each potential overlapping area. Third, the similarity between the potential overlapping pair
(ms i , ms j ) is calculated by combining the LCD and SDD descriptors, then the collision detection is conducted
to eliminate the incorrect pairs. Finally, the global optimal reassembly solution is obtained by iterative graph
optimization with the constrains of loop-closure and overlap restriction.
In this article, the fractured surfaces are detected and considered as matching units. First, the initial fractured
facets are extracted by region growing-based strategy with normal-angle constraints (i.e., iteratively expending
the seed point set with the nearest neighbor point whose normal is similar to the seed point’s, where the
normal difference threshold is set as π /6 in all experiments) [46], as shown in Figure 2(b). Second, Hierarchical
Fig. 3. Construction of 2D descriptors: (a) Principal Components Analysis; (b) Projection and refinement; (c) Thinning and
sequencing; (d) Douglas–Peucker algorithm; and (e) 2D Link-Chain Descriptor.
Agglomerative Clustering [3] is employed to merge the facets partitioned improperly based on the adjacency
and normal consistency (π /6 too), and the small or over discrete regions are decomposed to points meanwhile,
as shown in Figure 2(c). Third, labels of scattering points are voted by their neighbor points (points in the radius
of 3–5 times the average point distance in experiments), as shown in Figure 2(d).
4.1 2D Link-Chain Descriptors for Matching Units

For thin fragments, the detailed spatial distribution of fractured surfaces is equivalent to the orientation of
their contours. Therefore, the 2D LCD is proposed to encode the orientation information of fractured surfaces.
First, the Principal Component Analysis (PCA) [18] is used to calculate the eigenvalues (λ 1 ≥ λ 2 ≥ λ 3 ) and their
corresponding eigenvectors (e 1 , e 2 , e 3 ) of the fractured surfaces, shown in Figure 3(a). Second, the points of the
fractured surfaces are projected to the plane perpendicular to e 2 , and further refined by the mean-shift algorithm
(relocating to the center of the neighborhood [10]) to refine the contour preliminarily, shown in Figure 3(b).
Third, as shown in Figure 3(c), a single-track contour sequence is obtained following the procedure: (1) extracting
endpoints based on the unilateral distribution of neighborhood points; (2) initializing the growing direction with
the endpoints coordinate difference; (3) choosing the next point that best fits the growing direction in current
point’s neighborhood; (4) updating the growing direction with the current-next point pair. Fourth, Douglas–
Peucker algorithm [17] is used for the optimum contour segmentation with the point-line threshold ε, shown
in Figure 3(d). Finally, the LCD descriptor d s i of each matching unit s i is acquired by encoding the length of the
line segments and the interior angles between the adjacent line segments as Equation (7), shown in Figure 3(e):
d s i = (l 1 , θ 1 , l 2 , θ 2 · · · ) (7)
Fig. 4. The sliding match strategy: (a) the LCD descriptor d u , d v and inverse d v , (b) sliding match procedures, and (c) the
initial overlapping areas.
4.2 3D Spatial-Distribution Descriptors for Overlapping Areas

Given the matching units s i , s j and their LCD descriptor d s i , d s j , the initial overlapping areas ms i , ms j and
their corresponding LCD descriptors d m s i , d m s j are extracted by applying a sliding match strategy to the LCD
descriptors, shown in Figure 4.
The sliding match procedure is used to consider all possible inflection point alignments of (d u , d v ), where
d v represents the original and inverse d v for endpoints’ unknown inversion (shown in Figure 4(a)), and the
one with the highest similarity is considered as the initial overlapping areas. More specifically, the maximum
extension of each inflection pair is chosen as the corresponding sliding window (SW) as in Figure 4(b), and the
similarity of each sliding window is estimated by Equation (8):
n
SW
d k − d k
max εd − k
SWu SWv
ζ (d SWu , d SWv ) = k
, 0 (8)
k=1 d SWu
+ d SWv
As shown in Figure 4(b), where d SWu is the SW part of d u , ζ is the similarity of d SWu , d SWv , n SW is the
dimension of the SW and d kSWu is the kth value of the SW part of d u . εd is the upper bound of the expected
feature distance on each dimension. The initial overlapping area is determined by
(d m u , d mv ) = arg max (ζ (d SWu , d SWv )) (9)

SW s
where d m u , d mv are considered as the initial matching LCD parts of d u , d v . mu , mv are the corresponding 3D-
overlapping areas between matching units u, v.
After extraction of the overlapping area of every (s i , s j ), the 3D SDD are further calculated and concatenated
with the LCD descriptors to enhance the description of their geometric features. The 4-dimensional features [58]
of overlapping areas derived from eigenvalues (i.e., linearity, planarity, omnivariance, anisotropy) are calculated
by Equations (10)–(14). Finally, the 2D-3D enhanced descriptors Dm s i of each overlapping area ms i is represented
Fig. 5. Rigid transformation.
by Equation (15):
λi
λi = 3 (10)
i =1 λ i
λ1 − λ2
li = (11)
λ1
λ2 − λ3
pl = (12)
λ1

3
om = λ1 ∗ λ2 ∗ λ3 (13)
λ1 − λ3
an = (14)
λ1
Dm s i = (d m s i , lim s i , plm s i , omm s i , anm s i ) (15)
4.3 Pairwise Reassembly Probability Estimation

Given the overlapping area ms i and ms j of matching units s i and s j , the similarity of ms i and ms j is defined as
t
Dm s i − Dm s j
t

nD
ζ (ms i , ms j ) = max εd − t , 0 (16)
t =1
Dm s i + Dm
t
sj

where ζ (ms i , ms j ) is the similarity between ms i , ms j , Dm s i is the enhanced descriptor of ms i , Dm t
si
is the tth
value in Dm s i and n D is the dimension of Dm s i .
Given the matching units s i and s j , the conditional probability of ms i , ms j being the corresponding overlap-
ping areas of s i and s j is defined as

ms i ms j ζ ms i , ms j
p(ms i , ms j |s i , s j ) = tanh max , , 0.1 ∗ e −1 (17)
si s j
m ms
where max( ssi i , s j j , 0.1) is multiplied to prevent local extremum and allow the angle-only match.
After calculation of the overlapping areas ms i , ms j , the initial rotation matrix R 0 ∈ SO (3) and translation
vector T 0 ∈ R(3) are calculated by Equations (18) and (19) as shown in Figure 5:
T = Cms j − Cms i (18)
−1
R 0 = Rm s i ∗ Rm sj
(19)
Fig. 6. Collision detection.
where C m s i , C m s j are the center points of ms i and ms j . Rm s i and Rm s j are the rotation matrix from their local
coordinate systems to the world coordinate system [66]. Then the ICP algorithm [7] is used to refine the initial
rotation R 0 only without applying the T corrections for a much more robust fitting, based on the points belonging
to matching units s i and s j . To restrict R corrections and, p(R,T | ms i , ms j ) can be given as a probability of
modifying R 0 ,T 0 that fits (ms i , ms j ) with R,T that fits (s i , s j ):
AR · AR
p(R,T |ms i , ms j ) = 0
(20)
||A
R ||||AR 0 ||
where AR is the axis-angle vector (α R , β R , γ R ).
There are Cn2 potential overlapping area pairs (ms i , ms j , R,T ) in total, where n is the number of matching
units. The confidence of the calculated transformation R,T is estimated by Equation (21):
p(Z i j |R,T ) = tanh(Cross (Z i j |R,T ) ∗ Fit (Z i j |R,T )) (21)
where Cross (Z i j | R,T ) represent the collision detection calculated by Equation (22):
Cross (Z i j |R,T ) = ReLU(0.5 − IOU f i , f j ) (22)
where IOU f i , f j is the bounding box intersection-over-union of the fragments f i and f j (the matching units
s i , s j are derived from f i and f j ) after transformed by (R,T ), as shown in Figure 6.
ReLU is an activate function shown as

a, a ≥ 0
ReLU(a) = (23)
0, a < 0
And Fit (Z i j | R,T ) is defined as the degree of contact areas under the transformation of R,T :
L s i˙
l =1 δ (C (p l , s j˙ ))ϵd
Fit (Z i j |R,T ) = L s i˙ (24)
l =1 dist (p l , s j˙ )
s s
s i˙ C (p l i˙ , s j˙ ), C (p l i˙ , s j˙ ) ≤ ϵd
dist (p l , s j˙ ) = s (25)
0, C (p l i˙ , s j˙ ) > ϵd
s s s s˙
C (p l i˙ , s j˙ ) = min{d (p l i˙ , p r j )}, ∀p r j ∈ s j˙ (26)
s
s 1, C (p l i˙ , s j˙ ) ≤ ϵd
δ (C (p l i˙ , s j˙ )) = s (27)
0, C (p l i˙ , s j˙ ) > ϵd
s
where p l i˙ is the lth point of s i˙ and L is the point number of s i , ϵd is the upper bound of the correct matching
s s s˙
point pair distance and always set as 2 times average point distance. C (p l i˙ , s j˙ ) and d (p l i˙ , p r j ) are the distance
s s˙
from the point p l i˙ to matching unit s j˙ and point p r j , respectively.
Fig. 7. M for overlap restriction.
Then the pairwise reassembly probability p(Z i j | s i , s j ) can be reached by

p(Z i j |s i , s j ) = p(ms i , ms j |s i , s j ) ∗ p(R,T |ms i , ms j ) ∗ p(Z i j |R,T ) (28)
4.1 Global Reassembly Probability Estimation

The fully connected graph G (V , E) is constructed with node v i ∈ V as a matching unit and each edge e =
(v i , v j ) ∈ E as the vector (R,T , ms i , ms j ). And the weight of edge e = (v i , v j ) is set as p(Z i j | s i , s j ) (i.e., Equa-
tion (28)). To gain an optimal and globally consistent reassembly graph, first initialize with the edge that owns
the maximum weight for every node. As refinements, the iterative graph optimization with the constrains of
overlap restriction and loop-closure is further utilized.
4.4.1 Overlap Restriction for Potentially Incorrect Pairs Detection. The overlap restriction O (M ) of M = {ms i }
is defined as

m a ∩ mb

O (M ) = ReLU − ε O , ∀ m a , mb ∈ M (29)
m a ∪ mb
where ma , mb are two elements of the set of overlapping areas M, and the intersection between them should
be empty set in reality; εO is the upper bound for ignoring the intersection between the overlapping areas ma
and mb . An illustration of overlap restriction is shown in Figure 7. For the initialized global reassembly graph,
assume that there are two potential overlapping area pairs (m1 , m 3 ), (m2 , m 4 ). The intersection and union of
(m1 , m 2 ), (m1 , m4 ) (m3 , m2 ), (m3 , m 4 ) are calculated, the intersection over the union of (m1 , m 4 ) is larger than
the predefined threshold εO . Therefore, there is at least one potentially incorrect pair in (m1 , m 3 ) and (m2 , m 4 ),
which will be further checked and eliminated in the loop-closure constrains.
4.4.2 Loop-closure Constrains for Incorrect Pairs Elimination. The loop-closure constrains LC (C) is defined as

LC (C) = ReLU R i
− I (3) − ε , ∀C q ∈ C (30)
Cq R
i
RC
q C q
∈R

where C is the set of loops in the global reassembly graph, C q is a loop in C; RC q is the set of rotation matrix for
loop C q , RCi q is a rotation matrix in RC q ; I (3) is an identity matrix with a scale of 3 by 3; ε R is the upper bound
for accepting as a correct loop closure. An illustration of loop-closure constrains is shown in Figure 8.
The global optimal strategy under these two constraints can be obtained by maximizing Equation (31):

n−1
n
p(Z |F ) = (δ (s i , s j ) ∗ p(Z i j |s i , s j )) − O (M ) ∗ ∞ − LC (C) ∗ ∞ (31)
i j=i+1
Fig. 8. A loop-closure example.
For instance, there are five fragments of the Flowerpot (Figure 9(a)). First, we detect all of the match-
ing units (pre-selected based on the scale for a clear representation, Figure 9(b)) and calculate their LCDs
as described in Section 4.1. Second, exhaustively estimate the pairwise reassembly probability between
matching units (Figure 9(c)), and initialize the global reassembly graph using the edge owning the max-
imum probability for each matching unit (Figure 9(d)). Then, the overlap restriction is applied to detect
the potentially incorrect edges (e.g., the edges e (3b, 2b), e (4c, 2b), e (5b, 2b) in Figure 9(d). Next, 1 the
loop-closure constrains are utilized to eliminate the incorrect edges. More specifically, the loop-closures
are checked as: R e1→4 (1a,4a) ∗ R e (4b,3a) ∗ R e (3b,2b ) ∗ R e (2a,1b ) ≈ I (3), R e (4c,2b ) ∗ R e (2a,1b ) ∗ R e (1a,4a) I (3), R e (5b,2b ) ∗
4→3 3→2 2→1 4→2 2→1 1→4 5→2
R e (2a,1b ) ∗ R e (1a,4a) ∗ R e (4d,5a) I (3), which means e (3b, 2b) is the correct edge and the edges e (4c, 2b), e (5b, 2b)
2→1 1→4 4→5
are eliminated from the global reassembly graph (the red dashed lines in Figure 9(e), as supplementary, 2 with-
out loop, the one coupled (e.g., e (3b, 2b) and e (2b, 3b)) is accepted, 3 otherwise, the one weighted most is taken
in. Afterwards, the optimal edges connecting node 4c or 5b (i.e., the edge e (5b, 4c) in Figure 9(e) marked in
orange) now are added to the global reassembly graph. Finally, the above steps, including the suboptimal over-
lapping areas detection and description for the eliminated edges, overlapping detections, and loop validations,
are iteratively performed until all the edges in the global reassembly graph meet the conditions of overlap re-
strictions and loop-closure constraints or all of the fragments are reassembled, which can always be achieved
by iteratively carrying out the whole steps with preliminary splicing fragments even in quite difficult situation
(Figure 9(f)).
5 EXPERIMENTS AND ANALYSIS

The method was implemented in C++ under Visual Studio 2013. All experiments were implemented on a com-
puter with 8 GB RAM and an Intel i7-8750H @ 2.20 GHz CPU.
5.1 Datasets Description

The performance evaluation of the proposed fragments reassembly algorithm is presented with eight challenging
datasets of different objects.
First of all, four simulated thin datasets (i.e., blue and white porcelain, Shakyamuni Nirvana sculpture (part),
Left King (part), Left Monk (part)). More specifically, the blue and white porcelain were captured using the Go!
SCAN 50 laser scanner system1 with a measuring range up to 0.5 m, a measuring rate up to 550,000 pts/s, the
maximum resolution of 0.5 mm and accuracy up to 0.05 mm. The Shakyamuni Nirvana sculpture (built between
C.E.783-797 of Tang Dynasty) [25], the Left King and Left Monk (built in C.E.824 of the middle Tang Dynasty)
in the cave158 and cave159 of Mogao Grottoes, Dunhuang, were recorded by the Handyscan 3D laser scanner
system2 with high measureing rate up to 1,300,000 measurements/s, and accuracy up to 0.025 mm.
1 https://3dscannertech.com/creaform-3d-laser-scanners/creaform-goscan-3d-50.
2 https://www.creaform3d.com/en/portable-3d-scanner-handyscan-3d.
Fig. 9. The example of graph optimization strategy.
After acquiring the point clouds of four objects, we utilized the software of PolyWorks3 to reconstruct their
3D models, then used the software of 3DsMax4 and Rayfire5 to simulate the breaking process with varying
parameters (i.e., shell thickness, breaking methods, iterations, chaos, detailzation, and noise strength). Finally,
CloudCompare6 is used to sample points from the mesh of the fragments.
Table 1 lists the details of the four datasets. What makes these datasets tough are (1) varying measuring
principles, accuracies, thickness, chaos, detailzation, and noise strength; (2) a wide range of fractured antiquities
with a significant disparity in scale, era, materials and geometric shapes; (3) the simulated thin fractured surfaces
contains few of stable concave-convex information for corresponding fractured surfaces matching.
3 https://www.innovmetric.com/en/products-solutions/polyworksmodelertm.
4 https://www.autodesk.com/products/3ds-max/overview.
5 http://rayfirestudios.com/products/rayfire-for-unity/.
6 https://github.com/CloudCompare/CloudCompare.
Table 1. Details of the four Datasets
Points Thickness Scale

Dataset Parts Chaos Detail Noise
(millions) (mm) (m)
Blue and white porcelain 6 0.18 5 0.3 70 25 0.62
Shakyamuni sculpture 22 1.53 50 4.5 70 10 0.59
Left King(part) 34 1.43 15 0.46 65 15 0.6
Left Monk(part) 39 0.78 15 0.53 60 12 0.59
Table 2. Key Parameters of the Proposed Reassembly Algorithm
Parameter Description Suggestion Value

ε Point-line threshold in DP FS*(0.003–0.006) Varying
εd Feature distance on each dimension 0.1–0.2 0.15
εo Overlapping threshold 0.1–0.2 0.1
εR Closure-loop detection threshold 0.1–0.2 0.1
where FS is the average fragment scale (1D projection length along the PCA’s main vector) of the
dataset.
Second, Brick, Buddha Head from public dataset 3D Puzzles7 [27] and Tombstone, DoraEmbrasure from the
public project PRECIOUS8 [4] are used as mixed thin-thick datasets and thick datasets to test the performance of
the method proposed on varying situations. Besides the difficulties mentioned above, bad abrasions and thicker
fractured surfaces test the robustness of the proposed strategy designed for thin datasets greatly.
5.2 Parameters Settings

Table 2 shows the parameter details and values adopted in the proposed fragments reassembly method, set by
trial and error.
There are four key parameters in the proposed reassembly algorithm, i.e., ε, εd , εO , and ε R . The parameter
ε is the point-line threshold in Douglas–Peucker algorithm, which is used to ensure the stability of building
LCDs. A small ε results in the excessive segmentation and a large one in the insufficient segmentation, both of
which result in the low robustness of matching. The parameter εd is the upper bound of the expected feature
distance on each dimension. Once exceeded, we believe that this dimension match can be denied. As εd increases,
more matching dimensions will be accepted while more pairwise mismatches too, whereas a too small εd will
lead to truth-value discardings. εO is the upper bound of the intersection between the overlapping areas, which
guarantees that one fracture surface area cannot match two other fracture surface areas at the same time. A
small εO requires high accuracy of R,T , whereas a too large one will lead to pairwise mismatches that cannot be
detected. And ε R is the upper bound of the correct loop closure to remove the incorrect matches in the conflict
of overlap restriction. The smaller the ε R , the more stringent the accuracy requirements for global reassembly.
εd , εO , and ε R are always less influential and can be set as a constant. ε as the key parameter in the LCD
construction is greatly related to the scale and is tolerant in a way, which can be shown in Figure 10. From
Figure 10, we suggest ε ∈ FS*(0.003–0.006). Analyzed as: if too small, then the truth-pair average probability is
very high for over-segmentation of contours. However, with long LCD codes, the error matches can also output
high probabilities according to Equation (17), but the direct maximum rates are low for the low differentiation
of the matches (all around 0.9), which is called the morbid high probability. Whereas, if too large, then the
insufficient segmentations of LCD are ruining the success rate gradually.
7 https://www.geometrie.tuwien.ac.at/ig/3dpuzzles.html.
8 http://www.presious.eu/.
Fig. 10. The choice of ε: (a) the Tombstone; (b) Blue and white porcelain; (c) DoraEmbrasure. Blue lines represent the Truth-
pair average probability, which is the average probability calculated with Equation (17) of the true matching unit pairs.
Orange lines represent the Direct maximum rate, which is the proportion of the fracture surfaces whose most likely match
is the true one. Gray lines represent the Fragment reassembly rate, which is the proportion of the fragments that have
positive contributions for the final graph.
5.3 Experiment Results

Figures 11–18 show the test results of the proposed method on the eight testing datasets, respectively. These
qualitative results demonstrate that the proposed fragments reassembly method performs well on all the four
simulated datasets, 3D Puzzles datasets, and part PRECIOUS datasets, showing the practicability for various of
fractured antiques with significant disparity and freeform geometry and the robustness to varying thickness,
noise, and chaos. Analyzed as: (1) the thinning and sequencing process can improve the method’s robustness
Fig. 11. The registration results of blue and white porcelain with ε = 0.8 mm: (a) the simulated six fragments with random
colors; (b) details of fragments reassembly result from different viewpoints.
Fig. 12. The registration results of Shakyamuni Nirvana sculpture (part) of cave158, Mogao Grottoes, Dunhuang, with ε =
5 mm: (a) the simulated 22 fragments with random colors; (b) details of fragments reassembly result.
of the real erosion effects; (2) the LCD’s sliding match strategy enhanced the method’s part overlapping recall
capability (i.e., “abcde” and “abfd,” we can get “abd” or at least “ab” and “d”); and (3) benefit from the relative
matching strategy and probability thought, the relatively consistent “contour” is enough for matching rather
than an precise contour.
5.4 Reassembly Result Evaluation

For the simulated datasets, as the fragments are generated by simulating the breaking process, we can easily
regard the point clouds of complete objects as the ground truth of the fragments reassembly. Therefore, to quan-
tify the accuracy of the fragments reassembly, the ICP algorithm [7] is utilized to register the reassembly result
(i.e., the point clouds of reassembled fragments) with their corresponding ground truth (i.e., the point clouds of
complete objects). Then the reassembly error is calculated as the distance from each point in the reassembled
fragments to the mesh of the ground truth. Figure 19 shows the graphical representation of the reassembly errors
of the four datasets. It is considered as a negative point-mesh distance if the point is inside the mesh and positive
otherwise.
These experimental results demonstrate that the proposed method performs well in reassembling the frag-
ments on all the four simulated datasets, with average reassembly error less than 0.14 cm and a standard devi-
ation of less than 1.45 cm, which can meet the requirements of cultural heritage virtual restoration. Inevitably,
there are some fitting errors, but it can still reflect the error law of reassembly, as shown in Figure 19: (1) The
Fig. 13. The registration results of Left King sculpture (part) of cave159, Mogao Grottoes, Dunhuang, with ε = 0.6 mm:
(a) the simulated 34 fragments with random colors; (b) details of fragments reassembly result.
Fig. 14. The registration results of Left Monk sculpture (part) of cave159, Mogao Grottoes, Dunhuang, with ε = 0.5 mm:
(a) the simulated 39 fragments with random colors; (b) details of fragments reassembly result.
Fig. 15. The registration results of the brick with ε = 0.01, which has the same unit with model: (a) the six fragments with
random colors; (b) details of fragments reassembly result.
reassembly algorithm is an iterative process, which leads to the outer part is prone to large deviation for the
error accumulations. (2) The reassembly accuracy is related to the object size and the point density: the average
and standard deviation of reassembly errors for Shakyamuni sculpture are 0.16 and 1.34 cm, much larger than
others because of its 4.5 m size; with nearly the same size, Left King (part) with higher point density achieves a
higher accuracy than Left Monk (part).
Fig. 16. The registration results of the Buddha Head with ε = 0.009 ∼ 0.11, which has the same unit with model: (a) the 12
fragments with random colors; (b) details of fragments reassembly result.
Fig. 17. The registration results of Tombstone with ε = 1.4, which has the same unit with model: (a) the five fragments;
(b) details of fragments reassembly result.
Fig. 18. The registration results of DoraEmbrasure with ε = 1, which has the same unit with model: (a) the simulated 12
fragments; (b) details of fragments reassembly result.
For the public datasets, without the truth model, we focus on the partly reassembled dataset DoraEmbrasure:
As shown in Figure 20, for the blue one, on the one hand, it passed through the collision detection with
the matched fragments and the overlap detections with all other fragments, on the other hand, because of the
tortuosity of the projection, the matching probability is higher than the truth value (a straight-line pair). For the
orange one, we found that the minimal fracture surface that can offer the true value was decomposed for its tiny
size in the process I.
5.5 Time Performance Analysis

The runtime of each step is further introduced to verify the high efficiency of the proposed fragments reassembly
method. Table 3 illustrates the number of fragments and the runtime of each stage on the eight testing datasets.
Fig. 19. The graphical representation of the reassembly errors of the four datasets: (a) Blue and white porcelain with the
average error and standard deviation of 0.000 and 0.39 cm; (b) Shakyamuni sculpture with the average error and standard
deviation of 0.14 and 1.45 cm; (c) Left King (part) with the average error and standard deviation of −0.008 and 0.20 cm; and
(d) Left Monk (part) with the average error and standard deviation of −0.014 and 0.24 cm.
Fig. 20. Wrong matches (line boxes) and true locations (dot boxes).
Table 3. Time Consuming
Dataset Fragments P1 P2 P3 P4 Total

BW porcelain 6 23 17 9 2 51
Shakyamuni sculpture 22 55 75 26 4 160
Left King (part) 34 89 122 51 19 281
Left Monk (part) 39 73 119 71 21 284
Brick 6 13 31 12 2 58
Buddha Head 12 35 63 16 10 124
Tombstone 5 11 24 10 3 48
DoraEmbrasure 12 32 75 13 11 131
P1, P2, P3, P4 represent the processes of fracture surfaces segmentation, LCD construc-
tion, pairwise reassembly, and global reassembly, respectively.
It takes only about 284 seconds to reassemble the 39 fragments. It is also worth noting that, compared with Zhao
et al. [63] spending 358s reassembling 8 fragments, Altantsetseg et al. [2] taking 293s reassembling 2 fragments
and Zhang et al. [65]costing 824s reassembling 19 fragments, the proposed method achieves higher calculation
efficiency. What’s more, tests showed that the most time-consuming process is refining the projection line before
the LCD construction and is related greatly to the number of fracture surface points.
Fig. 21. The robustness evaluation: (a) robustness to Noise. X-axis: standard deviation of Gaussian noise. Y-axis: matching
degree calculated with the descriptors of the corresponding overlapping area. (b) Robustness to density. X-axis: point density
decimation. Y-axis: matching degree calculated with the descriptors of the corresponding overlapping area.
5.6 Performance Further Analysis

For a further evaluation, to test the robustness of the Enhanced descriptors to varying noise and point density,
two variants of the proposed method are also designed, which, respectively, only utilize the LCD descriptors and
LCD+SDD7 (seven dimensions, [58]) descriptors for matching units matching instead of the Enhanced descrip-
tors, all other settings and processes are in line with the proposed method. Additionally, to verify the robustness
of the proposed method to the case of fragment loss in reality, some fragments were taken away randomly and
the reassembly test is carried out.
The robustness of each feature descriptor was further tested with simulation disturbances (i.e., Gaussian noise,
varying point density). The matching degree under the set of disturbances is calculated to quantitatively evaluate
the robustness of different feature descriptors.
5.6.1 Robustness to Different Levels of Noise. To assess the stability of the introduced method to varying noise
levels (scanning noise, erosion effects), we independently added Gaussian noise to the x-, y-, and z-axes of each
point on the fractured surfaces. The matching degree of fractured surfaces under Gaussian noise with varying
standard deviations are shown in Figure 21(a). It can be seen that the LCD descriptors are stable due to the
optimization of the mean shift in the 2D description, and have stable fitting results in the cases of small-scale
noise. However, when the noise scale increases further, it is difficult to extract skeleton lines stably with simple
mean-shift, so the stability of LCD descriptors drops sharply. The matching degree of LCD+SDD7 descriptors of
the corresponding areas decreases because of the unacceptable influence of noise in normal direction. And the
proposed enhanced descriptors by combining the LCD and SDD descriptors achieved the best performance in
terms of noise robustness.
5.6.2 Robustness to Varying Point Density. To evaluate the robustness of the proposed method to varying point
density, the point clouds were down-sampled with varying decimations. The matching degree of the descriptors
derived with varying point density is shown in Figure 21(b). It shows that when the down-sampling rate is low,
all the descriptors achieved good performance. Further increasing the down sampling rate, the matching degree
of LCD descriptor sharply decreases at first, followed by the LCD+SDD7 descriptor. The proposed ED descriptor
combining the advantages of LCD and SDD descriptors achieves the best performance in terms of varying point
density.
Fig. 22. The robustness to completeness: (a) 1 in 5 fragments of Tombstone loss; (b) reassembly result of Tombstone; (c) 4
in 22 fragments of Shakyamuni sculpture loss; (d) reassembly result of Shakyamuni sculpture; (e) 9 in 34 fragments of Left
King (part) loss; (f) reassembly result of Left King (part).
5.6.3 Robustness to Completeness. To evaluate the robustness of the proposed method to the completeness of
single cultural relics, we randomly take away several fragments of datasets Tombstone, Shakyamuni sculpture,
and Left King (part) and carry out reassembly processes on the left fragments, shown in Figure 22. Despite
the loss of plenty of fragments (red boxes in Figures 22(a), 22(c), and 22(e)), the reassembly results are still
correct (Figures 22(b), 22(d), and 22(f)). It is analyzed that only one local matching part is needed to determine
the relationship between two fragments (arrows in Figures 22(b), 22(d), and 22(f)), and constraints of overlap
restriction and loop-closure in the global reassembly strategy can eliminate mismatches robustly.
However, once the unique contact fragment of each part is lost, there will be an error in the result. As shown
in Figure 22(f), the green right chest part is the unique fragment connecting the upper and lower parts. Once
lost, local matching errors might occur (Figure 23(b)).
Fig. 23. Inexcusable loss: (a) the unique fragment (green one) loss; (b) reassembly result of Left King (part) with the local
error.
Analyze as follows: with LCD+SDD, the overlapping area of every matching unit pair can be extracted. This
full connectivity can greatly strengthen the method’s completeness robustness with the help of overlap constrain
and loop-closure constrain.
6 CONCLUSIONS
This article proposed a novel probabilistic method for fractured cultural relics reassembly that can adapt to
many methods and guide the whole splicing process, and we verified it with eight challenging fractured cultural
relics datasets. Experiments have shown that the proposed algorithm can solve the problem in terms of good
accuracy, efficiency, and robustness. Although the approach introduced could yield satisfying results on the eight
challenging datasets, it still has some limitations: feature descriptors cannot adapt to large curvature boundary
lines, parameters are not self-adaptive and need to be set through trials, the accuracy cannot meet the modeling
requirements in a way, and some collisions might happen in global reassembly without collision detection, which
is now just removed by more iterations and more overlapping detections. In the future, we will pursue more
automated approaches for the robust descriptor extraction and the global matching strategy with high speed
and accuracy. Additionally, the deep-learning-based research is on the way to improve robustness and efficiency,
which is naively to be an end-to-end differentiable model like or based on the probabilistic framework proposed.
REFERENCES
[1] Dror Aiger, Niloy J. Mitra, and Daniel Cohen-Or. 2008. 4-points congruent sets for robust pairwise surface registration. In Proceedings
of the ACM International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’08). DOI:https://doi.org/10.1145/
1399504.1360684
[2] Enkhbayar Altantsetseg, Katsutsugu Matsuyama, and Kouichi Konno. 2014. Pairwise matching of 3D fragments using fast Fourier
transform. Vis. Comput. 30, 6–8 (2014), 929–938. DOI:https://doi.org/10.1007/s00371-014-0959-9
[3] Andreadis, Anthousis, Mavridis, Pavlos, and Papaioannou, Georgios. 2014. Facet extraction and classification for the reassembly of
fractured 3D objects. Comput. Graph. Forum 33, 1–2 (2014). https://doi.org/10.2312/egp.20141060
[4] Anthousis Andreadis, Georgios Papaioannou, and Pavlos Mavridis. 2015. Generalized digital reassembly using geometric registra-
tion. In Proceedings of the Digital Heritage International Congress (ICDH’15). 549–556. DOI:https://doi.org/10.1109/DigitalHeritage.2015.
7419572
[5] Mafkereseb Kassahun Bekele, Roberto Pierdicca, Emanuele Frontoni, Eva Savina Malinverni, and James Gain. 2018. A survey of aug-
mented, virtual, and mixed reality for cultural heritage. J. Comput. Cult. Herit. 11, 2 (2018). DOI:https://doi.org/10.1145/3145534
[6] C. Sánchez Belenguer. 2015. Surface Registration Techniques Applied to Archaeological Fragment Reconstruction. Universitat Politèc-
nica de València.
[7] Paul J. Besl and Neil D. McKay. 1992. A Method for Registration of 3D Shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14, 2 (1992),
239–256. DOI:https://doi.org/10.1109/34.121791
[8] Åke Björck. 1996. Numerical methods for least squares problems. Numer. Methods Least Squares Probl. (1996). DOI:https://doi.org/10.
1137/1.9781611971484
[9] Benedict J. Brown. 2008. Registration and matching of large geometric datasets for cultural heritage applications. Princeton University.
[10] Yizong Cheng. 1995. Mean shift, mode seeking, and clustering. IEEE Trans. Pattern Anal. Mach. Intell. 17, 8 (1995), 790–799. DOI:https:
//doi.org/10.1109/34.400568
[11] Ananda S. Chowdhury, Suchendra M. Bhandarkar, Robert W. Robinson, and Jack C. Yu. 2009. Virtual multi-fracture craniofacial recon-
struction using computer vision and graph matching. Comput. Med. Imaging Graph. 33, 5 (2009), 333–342. DOI:https://doi.org/10.1016/
j.compmedimag.2009.01.006
[12] Helena Cristina Da Gama Leitão and Jorge Stolfi. 2002. A multiscale method for the reassembly of two-dimensional fragmented objects.
IEEE Trans. Pattern Anal. Mach. Intell. 24, 9 (2002), 1239–1251. DOI:https://doi.org/10.1109/TPAMI.2002.1033215
[13] Haowen Deng, Tolga Birdal, and Slobodan Ilic. 2019. 3D local features for direct pairwise registration. Proc. IEEE Comput. Soc. Conf.
Comput. Vis. Pattern Recogn. 2019 (June 2019), 3239–3248. DOI:https://doi.org/10.1109/CVPR.2019.00336
[14] Luca Di Angelo, Paolo Di Stefano, Luciano Fratocchi, and Antonio Marzola. 2018. An AHP-based method for choosing the best 3D
scanner for cultural heritage applications. J. Cult. Herit. 34, (2018), 109–115. DOI:https://doi.org/10.1016/j.culher.2018.03.026
[15] Zhen Dong, Fuxun Liang, Bisheng Yang, Yusheng Xu, Yufu Zang, Jianping Li, Yuan Wang, Wenxia Dai, Hongchao Fan, Juha Hyyppäb,
and Uwe Stilla. 2020. Registration of large-scale terrestrial laser scanner point clouds: A review and benchmark. ISPRS J. Photogramm.
Remote Sens. 163, (2020), 327–342. DOI:https://doi.org/10.1016/j.isprsjprs.2020.03.013
[16] Zhen Dong, Bisheng Yang, Yuan Liu, Fuxun Liang, Bijun Li, and Yufu Zang. 2017. A novel binary shape context for 3D local surface
description. ISPRS J. Photogramm. Remote Sens. 130, (2017), 431–452. DOI:https://doi.org/10.1016/j.isprsjprs.2017.06.012
[17] David H. Douglas and Thomas K. Peucker. 1973. Algorithms for the reduction of the number of points required to represent a digitized
line or its caricature. Cartogr. Int. J. Geogr. Inf. Geovisualization 10, 2 (1973), 112–122. DOI:https://doi.org/10.3138/fm57-6770-u75u-7727
[18] G. H. Dunteman. 1989. Principal components analysis. (1989).
[19] Gil Elbaz, Tamar Avraham, and Anath Fischer. 2017. 3D point cloud registration for localization using a deep neural network auto-
encoder. Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 2472–2481. DOI:https://doi.org/
10.1109/CVPR.2017.265
[20] Georgios Dimitrios Evangelidis and Radu Horaud. 2018. Joint alignment of multiple point sets with batch and incremental expectation-
maximization. IEEE Trans. Pattern Anal. Mach. Intell. 40, 6 (2018), 1397–1410. DOI:https://doi.org/10.1109/TPAMI.2017.2717829
[21] Lin Gao, Jie Yang, Tong Wu, Yu Jie Yuan, Hongbo Fu, Yu Kun Lai, and Hao Zhang. 2019. SDM-NET: Deep generative network for
structured deformable mesh. ACM Trans. Graph. 38, 6 (2019). DOI:https://doi.org/10.1145/3355089.3356488
[22] Xuming Ge. 2016. Non-rigid registration of 3D point clouds under isometric deformation. ISPRS J. Photogramm. Remote Sens. 121, (2016),
192–202. DOI:https://doi.org/10.1016/j.isprsjprs.2016.09.009
[23] Zan Gojcic, Caifa Zhou, Jan D. Wegner, and Andreas Wieser. 2019. The perfect match: 3D point cloud matching with smoothed densities.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5540–5549. DOI:https://doi.org/10.1109/CVPR.2019.
00569
[24] Ping Guo, Wei Hu, Haibing Ren, and Yimin Zhang. 2018. PCAOT: A manhattan point cloud registration method towards large rotation
and small overlap. In Proceedings of the IEEE International Conference on Intelligent Robot Systems. 7912–7917. DOI:https://doi.org/10.
1109/IROS.2018.8594514
[25] Shizhe He. 1994. The buddhas of past, present and future and their figures in the Mogao Grottoes of Dunhuang. Dunhang Res. 2, (1994).
Retrieved from http://en.cnki.com.cn/Article_en/CJFDTotal-DHYJ402.011.htm.
[26] Je Hyeong Hong, Young Min Kim, Koang Chul Wi, and Jinwook Kim. 2019. PotSAC: A robust axis estimator for axially symmetric pot
fragments. Proceedings of the International Conference on Computer Vision Workshop (ICCVW’19). 1421–1428. DOI:https://doi.org/10.
1109/ICCVW.2019.001799
[27] Qi Xing Huang, Simon Flöry, Natasha Gelfand, Michael Hofer, and Helmut Pottmann. 2006. Reassembling fractured objects by geo-
metric matching. In Proceedings of the ACM International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’06).
569–578. DOI:https://doi.org/10.1145/1179352.1141925
[28] Martin Kampel and Robert Sablatnig. 2003. An automated pottery archival and reconstruction system. J. Vis. Comput. Animat. 14, 3
(2003), 111–120. DOI:https://doi.org/10.1002/vis.310
[29] Rong Kang, Jieqi Shi, Xueming Li, Yang Liu, and Xiao Liu. 2019. DF-SLAM: A deep-learning enhanced visual SLAM system based on
deep local features. Retrieved from http://arxiv.org/abs/1901.07223.
[30] David Koller, Bernard Frischer, and Greg Humphreys. 2009. Research challenges for digital archives of 3D cultural heritage models. J.
Comput. Cult. Herit. 2, 3 (2009). DOI:https://doi.org/10.1145/1658346.1658347
[31] Weixin Kong and Benjamin B. Kimia. 2001. On solving 2D and 3D puzzles using curve matching. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognitition. 2. DOI:https://doi.org/10.1109/cvpr.2001.991015
[32] Canyu Le and Xin Li. 2019. JigsawNet: Shredded image reassembly using convolutional neural network and loop-based composition.
IEEE Trans. Image Process. 28, 8 (2019), 4000–4015. DOI:https://doi.org/10.1109/TIP.2019.2903298
[33] Qunhui Li, Mingquan Zhou, and Guohua Geng. 2012. Pairwise matching of 3D fragments. Proceedings of the International Conference
on Information Management, Innovation Management, and Industrial Engineering (ICIII’12). 479–482. DOI:https://doi.org/10.1109/iciii.
2012.6340022
[34] Pavlos Mavridis, Anthousis Andreadis, and Georgios Papaioannou. 2015. Fractured object reassembly via robust surface registration.
Eurographics. Retrieved from http://www.presious.eu/sites/default/files/EG15_Reassembly_final.pdf
[35] Nicolas Mellado, Dror Aiger, and Niloy J. Mitra. 2014. SUPER 4PCS fast global pointcloud registration via smart indexing. Eurograph.
Symp. Geom. Process. 33, 5 (2014), 205–215. DOI:https://doi.org/10.1111/cgf.12446
[36] Nicolas Mellado, Patrick Reuter, and Christophe Schlick. 2010. Semi-automatic geometry-driven reassembly of fractured archeological
objects. In Proceedings of the 11th International Symposium on Virtual Reality, Archaeology, and Cultural Heritage. 33–38. DOI:https:
//doi.org/10.2312/VAST/VAST10/033-038
[37] Michalis Savelonas, Anthousis Andreadis, and Georgios Papaioannou. 2017. Exploiting unbroken surface congruity for the acceleration
of fragment reassembly. In Eurographics Workshop on Graphics and Cultural Heritage. https://doi.org/http://dx.doi.org/10.2312/gch.
20171305
[38] Zhe Min, Jiaole Wang, and Max Q. H. Meng. 2020. Joint rigid registration of multiple generalized point sets with hybrid mixture models.
IEEE Trans. Autom. Sci. Eng. 17, 1 (2020), 334–347. DOI:https://doi.org/10.1109/TASE.2019.2906391
[39] Geoffrey Oxholm and Ko Nishino. 2013. A flexible approach to reassembling thin artifacts of unknown geometry. J. Cult. Herit. 14, 1
(2013), 51–61. DOI:https://doi.org/10.1016/j.culher.2012.02.017
[40] Georgios Papaioannou, Tobias Schreck, Anthousis Andreadis, Pavlos Mavridis, Robert Gregor, Ivan Sipiran, and Konstantinos Vardis.
2017. From reassembly to object completion: A complete systems pipeline. J. Comput. Cult. Herit. 10, 2 (2017). DOI:https://doi.org/10.
1145/3009905
[41] Constantin Papaodysseus, Thanasis Panagopoulos, Michael Exarhos, Constantin Triantafillou, Dimitrios Fragoulis, and Christos
Doumas. 2002. Contour-shape-based reconstruction of fragmented, 1600 B.C. wall paintings. IEEE Trans. Signal Process. 50, 6 (2002),
1277–1288. DOI:https://doi.org/10.1109/TSP.2002.1003053
[42] Devi Parikh, Rahul Sukthankar, Chen Tsuhan, and Chen Mei. 2007. Feature-based part retrieval for interactive 3D reassembly. In
Proceedings of the IEEE Workshop on Applied Computer Vision (WACV’07). DOI:https://doi.org/10.1109/WACV.2007.25
[43] Marie Morgane Paumard, David Picard, and Hedi Tabia. 2018. Jigsaw puzzle solving using local feature co-occurrences in deep neural
networks. Proceedings of the International Conference on Image Processing (ICIP’18). 1018–1022. DOI:https://doi.org/10.1109/ICIP.2018.
8451094
[44] Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017. PointNet: Deep learning on point sets for 3D classification and
segmentation. Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 77–85. DOI:https://doi.
org/10.1109/CVPR.2017.16
[45] Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. 2017. PointNet++: Deep hierarchical feature learning on point sets in a metric
space. Adv. Neural Inf. Process. Syst. 2017 (Dec. 2017), 5100–5109.
[46] T. Rabbani, F. van den Wildenberg, and G. Vosselman. 2006. Segmentation of point clouds using smoothness constraint. Int. Arch.
Photogramm. Remote Sens. Spat. Inf. Sci. 36, 5 (2006), 248–253
[47] Carolina Raposo and João P. Barreto. 2018. 3D registration of curves and surfaces using local differential information. In Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition. 9300–9308. DOI:https://doi.org/10.1109/CVPR.2018.00969
[48] Nada A. Rasheed and Md Jan Nordin. 2020. Classification and reconstruction algorithms for the archaeological fragments. J. King Saud
Univ. Comput. Info. Sci. 32, 8 (2020), 883–894. https://doi.org/10.1016/j.jksuci.2018.09.019
[49] Sheldon M. Ross. 2014. A first course in probability. Pearson.
[50] M. Ş. Sağıroğlu and A. Erçil. 2005. A texture-based approach to reconstruction of archaeological finds. In Proceedings of the International
Symposium on Virtual Reality, Archaeology, and Cultural Heritage (VAST’05).
[51] M. A. Savelonas, A. Andreadis, G. Papaioannou, and P. Mavridis. 2017. Exploiting unbroken surface congruity for the acceleration of
fragment reassembly. In Proceedings of the Eurographics Workshop on Graphics and Cultural Heritage. 137–144. DOI:https://doi.org/10.
2312/gch.20171305
[52] Robert Sitnik. 2012. Archiving shape and appearance of cultural heritage objects using structured light projection and multispectral
imaging. Opt. Eng. 51, 2 (2012), 021115. DOI:https://doi.org/10.1117/1.oe.51.2.021115
[53] V. Sivapriya, Madhumitha Senthilvel, and Koshy Varghese. 2018. Automatic reassembly of fragments for restoration of heritage site
structures. In Proceedings of the 35th International Symposium on Automation and Robotics in Construction and International AEC/FM
Hackathon: The Future of Building Things (ISARC’18). DOI:https://doi.org/10.22260/isarc2018/0114
[54] Kilho Son, Eduardo B. Almeida, and David B. Cooper. 2013. Axially symmetric 3D pots configuration system using axis of symmetry
and break curve. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 257–264. DOI:https://doi.org/10.
1109/CVPR.2013.40
[55] Eduardo Vendrell-Vidal and Carlos Sánchez-Belenguer. 2014. A discrete approach for pairwise matching of archaeological fragments.
J. Comput. Cult. Herit. 7, 3 (2014). DOI:https://doi.org/10.1145/2597178
[56] Piao Wang, Guohua Geng, Xiaofeng Wang, and Yi Wang. 2018. Method for splicing fragments based on constructing surface texture and
fracture boundary tuple. In ACM International Conference Proceedings Series (ICPS’18). DOI:https://doi.org/10.1145/3207677.3278005
[57] Shuang Wang, Dou Quan, Xuefeng Liang, Mengdan Ning, Yanhe Guo, and Licheng Jiao. 2018. A deep learning framework for remote
sensing image registration. ISPRS J. Photogramm. Remote Sens. 145, (2018), 148–164. DOI:https://doi.org/10.1016/j.isprsjprs.2017.12.012
[58] Martin Weinmann, Boris Jutzi, Stefan Hinz, and Clément Mallet. 2015. Semantic point cloud interpretation based on optimal neighbor-
hoods, relevant features and efficient classifiers. ISPRS J. Photogramm. Remote Sens. 105, (2015), 286–304. DOI:https://doi.org/10.1016/
j.isprsjprs.2015.01.016
[59] Andrew Willis and David B. Cooper. 2006. Estimating a-priori unknown 3D axially symmetric surfaces from noisy measurements
of their fragments. Proceedings of the 3rd International Symposium on 3D Data Processing Visualization and Transmission (3DPVT’06).
334–341. DOI:https://doi.org/10.1109/3DPVT.2006.56
[60] Wen Xiao, Jon Mills, Gabriele Guidi, Pablo Rodríguez-Gonzálvez, Sara Gonizzi Barsanti, and Diego González-Aguilera. 2018. Geoin-
formatics for the conservation and promotion of cultural heritage in support of the UN Sustainable Development Goals. ISPRS J.
Photogramm. Remote Sens. 142, (2018), 389–406. DOI:https://doi.org/10.1016/j.isprsjprs.2018.01.001
[61] Yusheng Xu, Richard Boerner, Wei Yao, Ludwig Hoegner, and Uwe Stilla. 2019. Pairwise coarse registration of point clouds in urban
scenes using voxel-based 4-planes congruent sets. ISPRS J. Photogramm. Remote Sens. 151, (2019), 106–123. DOI:https://doi.org/10.1016/
j.isprsjprs.2019.02.015
[62] Naci Yastikli. 2007. Documentation of cultural heritage using digital photogrammetry and laser scanning. J. Cult. Herit. 8, 4 (2007),
423–427. DOI:https://doi.org/10.1016/j.culher.2007.06.003
[63] Yin Zhao, Wei Li, Li Xin, and Manhein Mary. 2011. An automatic assembly and completion framework for fragmented skulls. In Pro-
ceedings of the IEEE International Conference on Computer Vision (ICCV’11). 2532–2539. DOI:https://doi.org/10.1109/ICCV.2011.6126540
[64] Wei Yu, Maoqing Li, and Xin Li. 2012. Fragmented skull modeling using heat kernels. Graph. Models 74, 4 (2012), 140–151. DOI:https:
//doi.org/10.1016/j.gmod.2012.03.011
[65] Kang Zhang, Wuyi Yu, Mary Manhein, Warren Waggenspack, and Xin Li. 2015. 3D fragment reassembly using integrated template
guidance and fracture-region matching. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15). 2138–2146.
DOI:https://doi.org/10.1109/ICCV.2015.247
[66] Shunyi Zheng, Rongyong Huang, Zheng Wang, and Jian Li. 2015. Reassembling 3D thin fragments of unknown geometry in cultural
heritage. Photogramm. Fernerkundung Geoinfo. 2015, 3 (2015), 215–230. DOI:https://doi.org/10.1127/pfg/2015/0267
Received February 2020; revised June 2020; accepted August 2020

1 - A Probabilistic Method For Fractured Cultural Relics

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

1 - A Probabilistic Method For Fractured Cultural Relics

Uploaded by

Copyright:

Available Formats

A Probabilistic Method for Fractured Cultural Relics

HAIPING WANG, Wuhan University, Wuhan, Hubei, China

ACM Reference format:

2.1 Known Geometry-based Fragments Reassembly

2.2 Freeform Geometry-based Fragments Reassembly

3 THE PROPOSED PROBABILISTIC FRAMEWORK

Fig. 1. Working flowchart of the proposed method.

4.1 2D Link-Chain Descriptors for Matching Units

4.2 3D Spatial-Distribution Descriptors for Overlapping Areas

(d m u , d mv ) = arg max (ζ (d SWu , d SWv )) (9)

Fig. 5. Rigid transformation.

4.3 Pairwise Reassembly Probability Estimation

Fig. 6. Collision detection.

Fig. 7. M for overlap restriction.

Then the pairwise reassembly probability p(Z i j | s i , s j ) can be reached by

4.1 Global Reassembly Probability Estimation

Fig. 8. A loop-closure example.

5 EXPERIMENTS AND ANALYSIS

5.1 Datasets Description

Fig. 9. The example of graph optimization strategy.

Table 1. Details of the four Datasets

Points Thickness Scale

Table 2. Key Parameters of the Proposed Reassembly Algorithm

Parameter Description Suggestion Value

5.2 Parameters Settings

5.3 Experiment Results

5.4 Reassembly Result Evaluation

5.5 Time Performance Analysis

Table 3. Time Consuming

Dataset Fragments P1 P2 P3 P4 Total

5.6 Performance Further Analysis

Received February 2020; revised June 2020; accepted August 2020

You might also like