Int J Adv Manuf Technol (2017) 90:1665–1678

DOI 10.1007/s00170-016-9489-0

ORIGINAL ARTICLE

Vision-based automatic detection of steel surface defects

in the cold rolling process: considering the influence of industrial
liquids and surface textures
Yong Jie Zhao 1 & Yun Hui Yan 1 & Ke Chen Song 1

Received: 9 May 2016 / Accepted: 16 September 2016 / Published online: 29 September 2016
# Springer-Verlag London 2016

Abstract In the cold rolling process, automatic surface defect breal Real image width (mm)
detection could be considered as an important technique to B(x, y) Pixel value at the coordinate (x, y) in the binary
monitor product quality, and so far, substantial kinds of surface map
defects have been already successfully inspected. However, the Ci Center of superpixel i
interference of industrial liquids and produced surface textures d pq Difference between superpixel p and q
has been rarely studied yet. To address this problem, the vision- Dij Difference between superpixel center i and pixel j
based automatic detection method is proposed in this paper D1 ; D2 Difference between pixels and surroundings
containing three main parts: (i) region extraction, (ii) defect (see Fig. 7)
detection, and (iii) industrial liquid quantification. Four sets of f Camera frame rate (fps)
experiments are designed to verify the feasibility, accuracy, and h Industrial liquid region number
stability of the proposed method. The results indicate that the K Preset superpixel number
proposed method can detect most cracks and scratches with the l i ; ai ; bi Color information of superpixel center i in
accuracy of more than 91 %. Also, the proposed method could CIELAB color space
accurately quantify the industrial liquids including their num- la , lg , lb Preset BP neural network fitting inputs
bers (with the average accuracy of more than 90 %) and their L Initial superpixel side length (pixel)
sizes (with the accuracy of more than 91 %); it is therefore Ldet;man Defect length detected by the proposed method
expected to be promising because it could be utilized to track and by manual segmentation (mm)
the online running status of the rolling system and provide Lp Defect circumcircle diameter length expressed
rolling system maintenance suggestions. by pixels (pixel)
L1 Overlapping length along steel sheet width
Keywords Vision-based defect detection . Industrial liquid . direction (see Fig. 14a) (mm)
Cold rolling process . Process monitoring L2 Overlapping length along steel sheet feed
direction (see Fig. 14b) (mm)
m Pixel number within superpixel i
Nomenclature M(x, y) Gray value of the image after using gray value
aimg Image length expressed by pixels (pixel) multiply operation
areal Real image length (mm) n Required camera number to capture the images
bimg Image width expressed by pixels (pixel) along the steel sheet width direction
n_s Industrial liquid number accuracy (%)
numall All the utilized image number
* Ke Chen Song
numcor Number of images where industrial liquid number
unkechen@gmail.com is correctly counted
N Image pixel number
1
School of Mechanical Engineering and Automation, Northeastern N c; N s Normalized weight coefficients
University, Shenyang, People’s Republic of China N L ð pÞ L-neighbor of superpixel p
1666
r_d Defect length accuracy (%)
r_s Industrial liquid size accuracy (%)
R1j First-level label of pixel j
R2p Second-level label of superpixel p
sp Area size of industrial liquid region expressed by
pixels (pixel)
sseg;man Industrial liquid area size by the proposed method
and by manual segmentation (mm2)
S(x, y) Gray value of the image
S ′ (x, y) Transformed gray value of the image
T Adaptive threshold
v Rolling speed (m/s)
W Steel sheet width (mm)
xi ; yi Coordinate of superpixel center i (pixel)
μ Maximum times for a superpixel to be a
L-neighbor candidate
1 Introduction
As a key manufacturing processing technique in the field of
metal making, cold rolling is one of the most widely used
forming processes [1], which could provide high production
rates of high-quality products and account for more than 45 %
of the production of metal products [2]. However, elements
during the rolling process like impurities of raw materials,
overcambered/worn rolls, and inappropriate processing pa-
rameters easily lead to various kinds of defects on metal prod-
uct surfaces (e.g., pits, scratches, cracks, etc. [3]). These de-
fects would further influence produced surface qualities/integ-
rities, mechanical properties, and functional performances of
metal products, leading to a major loss of money and time [4].
With this, the topic on surface defect detection has been wide-
ly studied not only by scholars but also by manufacturers/
engineers.
Most early studies on defect detection have been performed
based on human vision and reached some achievements [5–8].
However, the subjectivity and the speed of raw eyes cannot
meet the requirement of modern productions characterized by
high speeds and tough work environments. Magnetic flux
leakage detecting [9], infrared detecting [10], and eddy current
detecting [11] also have attempted to be employed to inspect
surface defects. The reported problems however have been
related to anti-noise capability, inspection speed, ability to
quantify and categorize defects, and electricity consumption
[9–12].
As a relatively new technique, vision-based automatic de-
tection has been studied recently. This technique usually em-
ploys illumination devices and image sensors to capture sur-
face images and utilizes computer vision techniques or imag-
ing processing methods to inspect the defects in the captured
images. Based on this technique, various kinds of defects have
been successfully detected. Seam defects have been detected
Int J Adv Manuf Technol (2017) 90:1665–1678
by the machine vision and intelligent system based on the
support vector machine (SVM) method [13]. The detection
accuracy (i.e., the parameter to quantify how close the detect-
ed results are to the realistic ones) has been reported to be
90.5 %. Higher accuracy of 99.1 % has been achieved during
the detection of defects including wrinkle, die line, and coat-
ing void by using the multi-scale structural similarity index
quality metric method [14]. Crack defects during the cold
rolling process have been also inspected by employing the
morphology-based algorithm [15]. The crack probability esti-
mation (i.e., the parameter to quantify the similarity between
detected and realistic results based on logistic regression cal-
culations) has achieved 90 %. With the three-level compre-
hensive hierarchical method [16], weld and white rust defects
have been localized and 87.42 % of defects have been suc-
cessfully classified into a specific defect categories (e.g.,
cracks, pits, scratches, voids, etc.).
Although the defects above have been inspected, the de-
fects covered by industrial liquids (e.g., lubricants, coolants,
and rolling oils) have been rarely studied yet. In fact, industrial
liquids would appear on the steel surfaces during the cold
rolling process owing to the cooling requirement, the inappro-
priate adjustment of gaps and pressures between rolls, and the
wear and aging of rolls. These liquids would interfere the
defect detection process and influence the inspection efficien-
cy and accuracy to a large extent [17].
To this end, a vision-based automatic detection technique is
proposed in this paper to detect defects covered by industrial
liquids in the cold rolling process. The proposed method is
automatic and easy-to-handle because only one parameter
needs to be preset before the detection process and, once se-
lected, this parameter is proved to be robust to 100 images
with different kinds of defects covered by liquids. Moreover,
this method could also be utilized to track the online running
status of the rolling system and provide rolling system main-
tenance suggestions.
The remainder of this paper is organized as follows: Sect. 2
briefly introduces the vision-based inspection system setup.
Section 3 presents the architecture and details of the proposed
method. The defect detection experiments by the proposed
method and the validation are presented in Sects. 4 and 5 by
comparing the manually detected results with the ones obtain-
ed by the proposed method. Conclusions are summarized in
Sect. 6.
2 Vision-based detection system
The vision-based detection system mainly consists of two
subsystems: (i) the image acquisition subsystem and (ii) the
processing subsystem (see Fig. 1). The image acquisition sub-
system (including cameras and a light source) is fixed above
the steel surface with a distance of the camera focus range. A
Int J Adv Manuf Technol (2017) 90:1665–1678 1667

Fig. 1 Schematics of the vision-

based detection system

protection cover is employed to protect the system from any
possible damage. The image data captured by the acquisition
system would transmit to the processing subsystem, which
contains the computational server, host computer, and output
device (e.g., printer). The computational server processes the
image data by using image processing methods or computer
vision techniques, and then, the detected defect results are
transmitted to the host computer, which could generate the
online surface quality report if the steel products are qualified
in quality or modify rolling parameters to improve product
quality if substantial defects have been detected.
It could be seen from the steps above that a criteria is
needed to define which pixels or superpixels are “similar”;
thus, the parameter “label” is defined for pixels and
superpixels. The same value of label means sharing similar
visual characteristics. Here, the two-level labeling technique is
proposed as seen in Fig. 3. In this technique, the first-level
label (denoted as “R1” in Fig. 3) is added to each pixel, while
the second-level label (denoted as “R2” in Fig. 3) is attached to
each superpixel. Detailed explanations of how to obtain each
pixel and superpixel label are presented as follows.

3.1.1 Clustering pixels into superpixels

3 The proposed method
The content of the proposed method includes three main parts:
(i) the extraction of the industrial liquid regions (ILRs) and the
defect regions (DRs), (ii) the detection of the surface defects,
and (iii) the monitoring of the rolling system based on the
statistical analysis of the numbers and sizes of the ILRs (see
Fig. 2).
More detailed explanation of the proposed two-level labeling
technique is presented in Fig. 4. The first-level label is gener-
ated by the SLIC algorithm [18] as this algorithm outputs
superpixels that could adhere well to complex target (i.e., sur-
face defect) boundaries. According to the difference between
superpixel centers (i.e., the average pixel values within a

3.1 Extraction of the ILRs and the DRs

The extraction of the ILRs and the DRs involves three steps:
(i) clustering pixels of the captured images into superpixels
which could be considered as a set of pixels having similar
visual characteristics, (ii) clustering superpixels with similar
visual characteristics into the subregions, and (iii) extracting
the ILRs and DRs from the subregions.

Fig. 2 Schematics of the three main parts of the proposed method Fig. 3 Schematics of the proposed two-level labeling technique
1668
Fig. 4 The typical example of the proposed two-level labeling technique
certain superpixels) and neighboring pixels, the superpixel
boundaries are updated iteratively until pixels with similar
visual characteristics are clustered into one same superpixel.
In this algorithm, the initial boundary of superpixels is set
to be square. Assuming that the captured image is consisted by
aimg (in length) × bimg(in width) pixels and the desired number
of superpixels is K, the initial superpixel side length L there-
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
fore could be expressed as aimg  bimg =K . The superpixel
center is determined according to the lowest gradient principle
vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
u 2 
u  2  2  2 !
u l j −l i þ a j−ai þ b j −bi
Dij ¼ u
t þ@
Nc
where the normalized weight coefficients Nc ∈ [1, 20] and
Ns = L [18].
Among all the superpixel centers around the pixel j, the one
having the minimal Dij value (e.g., superpixel center i) is con-
sidered to be the superpixel center that the pixel j should be
clustered into. Consequently, the first-level label R1j of pixel j
is assigned to be the same label value as the one of the center i,
i.e., R1j ¼ R1i .
After all the first-level labels are determined, the
superpixel center Ci = {li, ai, bi, xi, yi} is updated according
to Eq. (2).
X
m therefore, the arrangement of superpixels is not as regular as
Pk
R1k ¼R1i
Ci ¼
m To address this, the “L-neighbor” is defined to search
where the pixel k (presented by Pk = {lk, ak, bk, xk, yk}) is the
pixel that should be clustered into the superpixel center i and
m is the number of pixels that should be clustered into the
superpixel center i.
Int J Adv Manuf Technol (2017) 90:1665–1678
[18] and could be expressed as Ci = {li, ai, bi, xi, yi}, where {l-
i, ai, bi} presents the color information in CIELAB color space
and {xi, yi} is the center coordinate. The first-level label of
each superpixel center (e.g., R1i ) is set to be i (i = 1 , 2 , 3 … N).
Within the 2L (in length) * 2L (in width) region around
each superpixel center, every pixel (e.g., pixel j) is compared
with the center (e.g., superpixel center i) by calculating the
difference Dij (see Eq. (1)) so that the spatial distance and
color information can be considered.
0 ffi
2 1
B x j −xi þ y j −yi C
A ð1Þ
Ns
The step (i) will finish when the difference of every
superpixel center before and after the update is smaller than
0.001.
3.1.2 Clustering superpixels into subregions
Superpixel clustering refers to merging similar superpixels
into subregions by comparing the difference between a certain
superpixel and its neighboring ones. During the comparisons,
searching neighboring superpixels would be hard. This is due
to the fact that superpixels adhere to the target boundaries;
that of pixels, and the definition of pixel neighbor (e.g., four
neighbors or eight neighbors [19]) would not apply.
ð2Þ
the superpixel neighbors (see Fig. 5), including two
main steps: (i) Eight pixels are selected which are sur-
rounding a certain superpixel center (e.g., the orange
point in Fig. 5) with a distance of L pixels at eight
uniform circumferential angles, and (ii) the superpixels
Int J Adv Manuf Technol (2017) 90:1665–1678
Fig. 5 Schematic of the proposed L-neighbor
that the eight pixels belong to could be considered as
the L-neighbor of superpixel p, symbolized by NL(p).
The difference between the superpixel p and the neighbor-
ing superpixel q (q ∈ NL(p)) could be expressed as
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
 2  2  2ffi
d pq ¼ l p −l q þ ap −aq þ bp −bq ð3Þ
If the difference dpq is smaller than the threshold T, the
superpixels p and q would be merged into one subregion.
The second-level label R2q of the neighboring superpixel q is
then assigned to be the same second-level label value as the
one of the superpixel p.
One note here is that a static threshold T would be inappro-
priate because DRs in different images would have inconsis-
tent luminance and/or different color information. Therefore,
the adaptive threshold T is adopted and obtained by the back
propagation (BP) neural network fitting here. During this
fitting, three inputs (denoted as la, lg, and lb) and one output
T could be obtained by experiments as follows:
(i) The input la is defined as the average value of the lumi-
nance l in the CIELAB color space, which could be
gained according to Eq. (4).
maxðl Þ þ minðl Þ
la ¼
2 The aim of the second main part is to (i) extract the edges of
where max(l) and min(l) are the maximal and minimal l values
of pixels within the whole image.
(ii) The input lg is obtained by using the iterative global
threshold algorithm [34], i.e.,
1 X 1 X
X l ðx; yÞ þ X
:1 lðx;yÞ< lgk :1 lðx;yÞ>lgk
l ðx;yÞ< lgk l ðx;yÞ>lgk
lgkþ1 ¼
2
where lgk + 1 is the lg value in the iteration k + 1 and l(x, y) is
the l value of the pixel at the coordinate (x, y).
1669
(iii) The input lb is utilized to represent the average back-
ground value of l and defined as Eq. (6) [20].
X X
l ðx; yÞ
X x< H 2 y< W 2
λ¼4:1
Hλ  Wλ
lb ¼ ð6Þ
4
where λ refers to the four borders of the image (λ = 1, 2, 3, 4,
respectively, refers to the top, bottom, left, and right borders),
Hλ is the width of the top and bottom borders, and Wλ is the
width of the left and right borders.
(iv) The output T is experimentally determined by using 50
sample images randomly chosen from steel surface defect
images.
Based on the above procedures, the step (ii) will be finished
when all the second-level label R2 are assigned to values.
3.1.3 Extracting the ILRs and DRs from subregions
The ILRs and DRs are indirectly obtained by extracting the
background (i.e., the non-ILRs and non-DRs) because the
background is more easily detected than ILRs and DRs, due
to the more consistent visual characteristics. This process is
visualized in Fig. 6.
Figure 6a presents the second-level label assignment result,
while Fig. 6b illustrates the binary map by setting the back-
ground in Fig. 6a to be zeros (black in the figure) and the rest
region in Fig. 6a to be ones (white in the figure). Figure 6c
shows the final results generated by extracting the original
image within the white region in Fig. 6b.
3.2 Detection of the surface
ð4Þ
both ILRs and DRs and then (ii) recognize the DR edges from
them. To achieve this, some potential techniques could be
employed, e.g., Canny and Sobel edge detection algorithms
[21] for the aim (i) and the low-pass and high-pass filters [22]
for the aim (ii). These techniques however cannot cope with
the problems in this study, because interference textures
would be included in the capture images and the variance of
the visual characteristics in the industrial liquid regions would
l ðx; yÞ
be largely varied (see Fig. 7).
In the field of industrial manufacturing, the texture refers to
the machined surface topography due to the kinematics of the
ð5Þ employed manufacturing processes [23]; therefore, the gray
values of the texture topography would be varied along the
image width (see Fig. 7).
As shown in Fig. 7, textures cover the whole ILRs and the
differences between texture pixels and surroundings are very
1670
Fig. 6 Schematic of segmentation process of the proposed method: a the
image after the second-level label assignment, b the binary map by setting
the background to be zeros (black) and the rest region to be ones (white),
close to those between defect pixels and surroundings, as the
point B (the difference of gray value D1 is 72.6) and the point C
(the difference of gray value D2 is 76.4). The texture pixels
would probably be mistaken as defects if the previously men-
tioned edge detection algorithms are directly applied to extract
defect edges. Besides, previously mentioned low-pass or high-
pass filters would also easily filter the defects out together with
the textures because most of the DRs are amid the textures
where a large variance of the gray values exists (the largest
variance in Fig. 7 is around 135.5 between the points A and D).
To fix this, the coupled technique including the gray value
transformation, the edge extraction, and the morphology
method is adopted in this paper, and the detailed flowchart is
presented in Fig. 8 (where all the edges in the figure are thick-
ened for clarity). In this technique, the gray value transforma-
tion and the edge extraction are utilized to generate two kinds
of images. One contains only the ILR edges, while the other
one contains both the ILR and DR edges. The morphology
method is employed to recognize the DR and ILR edges.
The gray value transformation is performed by (i) the linear
transformation, (ii) the gray value multiply, and (iii) the me-
dian filtering. The gray value linear transformation is adopted
to minimize the gray value variation for extracting the ILR
edges. Assuming that S(x, y) represents the gray value of the
images, the transformed image S′(x, y) could be expressed as
8 values could be averaged by the surrounding ones, while the
>
< 0; S ðx; yÞ ¼ 0
d−c
S ðx; yÞ ¼ d−c ð7Þ
>
: ½sðx; yÞ−a þ c; sðx; yÞ≠0
b−a
Fig. 7 The typical captured
image of surface defects covered
by the ILRs and the
corresponding gray values
Int J Adv Manuf Technol (2017) 90:1665–1678
and c the final results generated by extracting the original image within
the white region
where a = minS(x, y) ≠ 0(S(x, y)), b = maxS(x, y) ≠ 0(S(x, y)), and c
and d are the minimum and maximum of the linear transfor-
mation range. c and d are separately set to 0 and 15 to avoid
the overflow in the gray value multiply process.
The transformed image is shown in Fig. 9a which indicates
that the variance is largely reduced to be 1.6 in comparison
with the variance before transformed (135.5 in Fig. 7).
The gray value multiply enhances the contrast between
defects and their surrounding pixels so that the DR edges
could be obtained. The obtained image M(x, y) could be
gained according to
M ðx; yÞ ¼ S ðx; yÞ  S ðx; yÞ ð8Þ
The result is presented in Fig. 9b, where the D1 and D2
values are separately 23.1 and 31.5 in comparison with the
variance before transformed (72.6 and 76.4 in Fig. 7). This
makes the differences between the defects, and their surround-
ings are distinctive from those between the textures and their
surroundings.
The median filtering is the employed operation that could
eliminate the influence of textures. The basic principle of this
filtering is by using the average gray values of the surround-
ings to replace the gray values of the target pixels [24]. Based
on this, the texture pixels with extremely large or small gray
defect pixels would be rarely influenced because other defect
pixels are surrounding them, which makes the average gray
values of the surroundings close to the defect pixels.
Int J Adv Manuf Technol (2017) 90:1665–1678 1671

Fig. 8 The proposed flowchart of the surface defect detection

The filtered image of Fig. 9a is presented as Fig. 9c
where the waviness of the gray values induced by tex-
tures is smaller than that shown in Fig. 9a and the var-
iance between the points A and D is reduced to be below
1.5. The filtered image of Fig. 9b is shown as Fig. 9d
where the difference between the textures and the sur-
rounding pixels is minimized (see D1 in Fig. 9d), while
the difference between the defects and the surrounding
pixels is kept relatively constant before and after the
filtering process (see D2 in Fig. 9d). Therefore, the big
difference between D1 and D2 could be utilized to detect
the DR and ILR edges easily.
To extract the ILR and DR edges, the Canny edge detection
algorithm [21] is adopted, and the results are illustrated in
Fig. 10. The DR edge results extracted by the morphology
subtraction are shown in Fig. 11.
3.3 Monitoring of the rolling system based on the numbers
and sizes of the ILRs
The aim of this part is to count the numbers and sizes of the
ILRs in order to monitor whether the rolling system is running
under a good state.

Fig. 9 The images transformed

by the gray value transformation:
a the images after the linear
transformation, b the image after
the gray value multiply, c the
image after the median filtering of
a, and d the image after the
median filtering of b
1672 Int J Adv Manuf Technol (2017) 90:1665–1678

Fig. 10 The result of the

extracted a ILR edges and b ILR
and DR edges by using the Canny
edge detection algorithm (ILR
industrial liquid region, DR defect
region)

The ILR number here is defined as the number of the un- 4.1 Image acquisition subsystem
connected ILRs. Let the symbol B(x, y) represent the value of
the pixel whose coordinate is (x, y) in the binary map (e.g., In this paper, the image acquisition subsystem (see Fig. 12)
Fig. 6b); then, the ILR number could be determined by the consists of the light-emitting diode (LED) light source
number of the unconnected pixels which could satisfy (HLND-1200-SW2, CCS Co., Inc., Japan) and the area array
B(x, y) = 1, because the liquid region pixels are of white color charge-coupled device (CCD) camera (acA640-90uc, Basler
in the binary map. AG, Germany) with the frame rate (denoted as f) of 90 fps and
Define the first pixel satisfying B(x, y) = 1 as B1(x1, y1), ; the resolution of 640 × 480 (denoted as aimg  bimg ), which
then, the ILR number h could be expressed as corresponds to a realistic region with the size (denoted as areal
8 breal ) of 21:5  16:1 mm. Similar to ref. [25], the light
< 1; if exist B1 ðx1 ; y1 Þ source is located 20±3 cm above the steel surface with the
h ¼ h; if Bμ ðx; yÞΘBμ−1 ðx; yÞμ ¼ 2; 3:: ð9Þ
: inclination angle of 30° in order to provide the uniformly
h þ 1; elsewise
distributed luminance. The CCD camera directly faces the
where AΘB denotes that the pixel A is connected with the steel surface.
pixel B (or there exists a path between the pixels A and B
[19]) and μ = 2 , 3 , … is used to traverse all the pixels that 4.2 Workpiece
could satisfy the condition B(x, y) = 1.
The ILR size could be quantified by using pixels by Given that it is one of the most widely used metal materials in
X the production of products like motors and generators, the
SP ¼ 1 ð10Þ workpiece sample used in this study is the silicon steel sheet
Bðx;yÞ¼1
with the width (denoted as W) of 100 mm containing 3 %
(wt%) silicon content (35W230, Ansteel Co., Inc., China).

4 Experiments
4.3 Industrial liquid
To verify the feasibility of the proposed method, defect detec-
The industrial liquid used in this study is the CRO lubricant
tion experiments are conducted.
(Quakerol Co., Ltd.), which is widely used in industrial
manufacturing like rolling, milling, and finishing [17]. It is
transparent light yellow in color, and therefore, it is believe
to be appropriate to study the interference effects of liquids
during the defect detection process.

4.4 Experiment design

Four sets of images with different numbers of the ILRs and

various kinds of defects are employed to evaluate the pro-
posed methods as seen in Table 1. To eliminate the random
error, 25 images captured by the above setup are employed in
Fig. 11 The DR edge results extracted by the morphology subtraction each set.
Int J Adv Manuf Technol (2017) 90:1665–1678
Fig. 12 a Schematics and b realistic image acquisition subsystem employed in the experiments; c sample image of the captured defects covered by the
ILRs
5 Results and discussion
5.1 Detection of the defects under the ILRs
To evaluate the detection accuracy of the proposed method,
the parameter r_d is defined as Eq. (11), which is based on the
comparison of the defect lengths which are inspected by the
proposed method and which are manually detected.
Ldet
rd¼ ð11Þ
Lman
where Ldet is the defect length detected by the proposed meth-
od and Lman is the defect length that is manually detected.
According to the definition, it could find that the more the
parameter r_d is close to 100 %, the more detection accuracy
the proposed method could perform.
Here, the defect lengths Ldet and Lman are defined as
the diameter of the defect circumcircle (see Fig. 13c),
which could be obtained by first measuring the pixel
number Lp of the circumcircle diameter and then
converting Lp into actual value by
areal  Lp
Ldet;man ¼ ð12Þ
aimg
The result is presented in Table 2. The lowest accuracy
during the experiments is 91.67 %, which could satisfy the
industrial requirements [1] to a large extent, indicating the
accuracy and feasibility of the proposed method.
More detailed image results could be seen in Fig. 13. It can
be seen that the proposed method is applicable to both the
crack (Fig. 13a, b) and scratch (Fig. 13c, d) inspections al-
though there is the interference of the industrial liquids. It
could also be found that the proposed method could be applied
Table 1 The employed experiment design
Experiment no. 1 2 3 4
No. of industrial liquids 1 2 1 2
Defect type Crack Crack Scratch Scratch
1673
with the interference of not only the single ILR (Fig. 13a, c)
but also the multiple ILRs (Fig. 13b, d). Besides, the proposed
method could inspect the defects with the accuracy of more
than 95 %, although there is a large visual characteristic var-
iance of the ILRs (see the difference between the regions A
and B in Fig. 13c). Figure 13 also indicates that the interfer-
ence of the textures induced by the rolling process during the
detection process is minimized.
5.2 Quantification of the industrial liquids
The quantification ability of the ILRs is the basis for monitor-
ing the rolling system, and this ability could be evaluated from
the following two aspects: (i) the ILR number and (2) the ILR
size.
To evaluate from the first aspect, the parameter n_s is de-
fined as
numcor
ns¼ ð13Þ
cumall
where numcor is the number of images where the ILR number
could be accurately counted and numall is the number of all the
images used in the experiment.
To evaluate from the second aspect, the parameter r_s is
defined as
sseg
r s¼ ð14Þ
sman
where sseg is the area size of the ILRs obtained by using the
proposed method and sman is the area size of the ILR results
obtained by manual segmentation. Here, sseg and sman are
obtained by first measuring pixel size sp by using Eq. (10)
and then converting sp into actual sizes by
areal  breal  sp
sseg;man ¼ ð15Þ
aimg  bimg
The quantification ability of the ILRs is presented in
Table 3. It shows that the ILR numbers could be accurately
counted for the single ILR (experiment nos. 1 and 3) with the
mean accuracy of 96 %. This accuracy reduces a little for the
1674
Fig. 13 The defect inspection
results: a the experiment no. 1, b
the experiment no. 2, c the
experiment no. 3, and d the
experiment no. 4
multiple ILRs (experiment nos. 2 and 4), which might be
because for these cases, ILRs join together and the overesti-
mate of the multiple ILRs might happen. But, the accuracy is
still kept more than 90 %.
The requirement of the accurate detection of the ILR area
sizes seems to be a little demanding because industrial liquids
are transparent; therefore, the ILR edges would be hard to
recognize even for human raw eyes due to the small liquid
thicknesses. Also, the size of the ILR is small (smaller than
1.2 cm2), and the ILRs might be interfered by textures.
However, as shown in Table 3, the average accuracy of the
proposed methods for all the experiments is 91.58 % and the
minimal accuracy is almost 80 %, indicating the accuracy of
the proposed method. Besides, the detection stability could
also be evidenced by the small variance between the maximal
and minimal values.
Table 2 Accuracy of defect
inspection Detect accuracy r_d
For crack defects (%)
For scratch defects (%)
Int J Adv Manuf Technol (2017) 90:1665–1678
6 Further discussion
6.1 Stitching captured images to obtain the whole picture
of the produced steel sheet surface
Stitching captured images to obtain the whole picture of
the produced steel sheet surface is an intuitive way to
quantify the quality of the product and monitor the on-
line running status of the production line, which is
discussed in this section.
In industrial rolling process, multiple cameras simulta-
neously take images at different locations along the steel sheet
width direction and the images taken by two adjacent cameras
have overlapping regions (see Fig. 1). The stitching process is
conducted by putting the images captured at the same moment
together with the overlapping length of L1 (see Fig. 14a). In
Experiment no. 1 Experiment no. 2 Experiment no. 3 Experiment no. 4
92.00 91.67 – –
– – 95.83 94.44
Int J Adv Manuf Technol (2017) 90:1665–1678 1675

Table 3 Accuracy of the

qualification of the detected ILRs Accuracy of the qualification of the detected ILRs (%) Experiment nos. 1 and Experiment nos. 2 and 4
3 (single ILR) (multiple ILRs)

n_s r_s n_s r_s

Mean value 96 93.41 90.20 91.58

Maximum – 99.10 – 98.52
Minimum – 83.39 – 79.90

this way, the band image along the sheet width direction could of the stitched whole picture of the steel sheet surface is shown
be gained. in Fig. 14c.
The overlapping length L1 could be calculated by It can be seen from Fig. 14c that more industrial liquids are
presented during the rolling process, and even multiple indus-
ðareal  n−W Þ
L1 ¼ ð16Þ trial liquids emerge at the end of the rolling. It therefore might
ðn−1Þ probably comment that the running status of the production
line is becoming worse, and by using the proposed method,
where n is the required camera number to capture all the im-
the system problems could probably be checked in time so
ages along the steel sheet width direction, which could be
that the loss could be minimized.
expressed as
W  
n> s:t:nεN * ð17Þ
areal 6.2 Inspection speed of the proposed method

Similarly, there is also image overlapping along the sheet
feed direction (i.e., the overlapping between two adjacent
band images). The whole picture of the steel sheet surface
would be obtained by putting together all the band images
with the overlapping length of L2 (see Fig. 14b).
The overlapping length L2 could be expressed as
v era frame rate, which, in this study, is 90 fps, i.e., the inspection
L2 ¼ breal −
f
where v is the rolling speed (in this study v ¼ 1:35 m=s ).
For this study, L1 is 1:875 mm and L2 is 1:1 mm. An example
To synchronize with the production line, the inspection speed
of the proposed method is an important performance parame-
ter, which is discussed in this section.
Because the computational server consists of multiple com-
putation unit systems and each unit system individually process-
es the images taken by one camera, the inspection speed require-
ment of the proposed method is therefore related with the cam-
ð18Þ process for each image should be finished within 0.011 s (equal
to the time interval between two captured images).
Based on experiments, the processing time of the proposed
method to inspect one image is 0.01 s (using a workstation

Fig. 14 Schematic of stitching captured images to obtain the whole images along the feed direction to obtain the whole picture of the steel
picture of the steel sheet surface: a stitching captured images along the sheet surface, and c an exmaple of the stitched whole picture of the steel
steel sheet width direction to obtain the band images, b stitiching band sheet surface
1676
equipped with Intel Xeon 14-Core 3.6 GHz processor and
64 GB RAM), which is smaller than the requirement of
0.011 s. In fact, the processing time of the proposed method
could also be theoretically calculated as follows:
The computation complexity of the proposed method is
shown in Fig. 15, from which it can be seen that the proposed
method includes two circulations, i.e., circulations (i) and (ii).
In circulation (i), each pixel j is utilized to calculate Dij less
than eight times because there are at most eight potential
superpixel centers that pixel j could fall in the 2L  2L region.
The “for-end” loop in circulation (i) therefore would run less
than 8 N times. In the “do-while” loop, the convergence
error could be less than 0.001 (the terminal condition, see
Sect. 3.1.1) within 10 iterations. Hence, the circulation (i)
would run at most 10  8  N times. In the circulation (ii),
each superpixel q could be utilized to calculate d pq less than μ
times (μ is the maximum times that a superpixel could be a L-
neighbor candidate). The circulation (ii) therefore would run a
at most μ  K times. Thus, the running time for the two
circulations could be expressed as ð10  8  N þ μ  K Þ
= 3:6  109 ¼ 0:007 s, where μ ¼ 10 is utilized based on
statistical analysis of superpixel graphs (see an example in
Fig. 4). The theoretical running time of 0.007 s is consistent
with the experimental processing time which is 0.01 s.
Based on above, it seems that the proposed method speed
could satisfy the requirement of production line. However, it
should also note that the proposed defect inspection method is
Fig. 15 Computation complexity of the proposed method
Int J Adv Manuf Technol (2017) 90:1665–1678
usually employed as the one step of the whole inspection
process, and other subsequent operations like feature extrac-
tion and classification will also be performed. Therefore, our
future research direction would be focused on the improve-
ment of the inspection speed of the proposed method.
6.3 Influence of ILR and defect sizes on the inspection
accuracy
Given that realistic ILR and defect sizes varied within a wide
range (ca. area within 8–120 mm2 and length within 0.8–
10 mm in this study) and these sizes might probably influence
the inspection accuracy, the influence of ILR and defect sizes
on inspection accuracy is discussed in this section.
The results are presented in Fig. 16, where more than five
images are utilized for each point and the performance stabil-
ity is illustrated by the standard error. It can be seen from
Fig. 16 that when the ILR area and the defect length, respec-
tively, increase from 8 to 36 mm2 and from 0.8 to 4 mm, the
accuracy is rising from 62 to 89 % and from 54 to 86 %;
however, the rising speed is reduced when the ILR area and
defect length are separately increased to the threshold values
of 36 mm2 and 4 mm. This might be because the threshold
values are close to the preset superpixel sizes. When the ILRs
and defects are smaller than the preset superpixel sizes, the
ILRs and defects would be clustered into a same superpixel
with background so that the inspection accuracy is low. When
the ILRs and defects are larger than the preset superpixel sizes,
the superpixels would probably only contain the ILRs and
defects so that the influence of the ILR area and defect length
on the inspection accuracy is limited. However, if the preset
superpixel sizes are specially designed for small-sized ILRs
and defects, substantial computational efforts and times would
be waste when segmenting large-sized ILRs and defects. In
fact, the selection of the appropriate preset superpixel sizes is
still an open issue for all the superpixel-based methods [26],
which would be our another future research direction.
Moreover, it could also find that the standard errors are
larger when ILR areas and defect lengths are, respectively,
smaller than 36 mm2 and 4 mm, indicating the performance
instability of the proposed method when the actual ILR areas
and defect lengths are small. This might due to the fact that the
proposed method is based on the creation of superpixels with
appropriate segmentation boundaries; therefore, when the
ILRs and defects are small, the differences between ILRs
and background are hard to be detected.
6.4 Failure inspection cases
Although the proposed method is applicable to the steel sur-
face defect detection covered by industrial liquids with the
advantages of accuracy, automation, and easy operation, there
Int J Adv Manuf Technol (2017) 90:1665–1678
Fig. 16 Influence of a ILR areas and b defect lengths on inspection accuracy of the proposed method (more than five images are utilized for each point,
and the performance stability is illustrated by the standard error)
are still failure inspection cases and three typical examples of
them are separately shown in Fig. 17a, b, c.
The ILRs in typical case (i) (see Fig. 17a) are failed to
be detected which might be because the ILRs are in low
contrast with the background so that color information
has less effects on the difference Dij calculation (see
Eq. (1)) than spatial distance during clustering pixels into
superpixels. This means that the pixels within ILRs and
the background pixels would be mistakenly clustered in-
to a same superpixel. Moreover, the failed detection of
ILRs influences the subsequent inspection of defects,
leading to the undesired defect detection results.
Unlike the ILRs in typical case (i), only a part of the ILRs in
typical case (ii) (see Fig. 17b) is in low contrast and the rest
Fig. 17 Three typical examples
of failure inspection cases
induced by a typical case (i), b
typical case (ii), and c typical case
(iii)
1677
part is in high contrast relative to the background (see “A” in
Fig. 17b). The low-contrast part interferes with the ILR detec-
tion and has little effects on defect detection. This might be
because the superpixels are independent; therefore, the
superpixels which are not properly clustered would not influ-
ence the merging of other superpixels into subregions.
In typical case (iii) (see Fig. 17c), some defects are ob-
served between the two adjacent ILRs, but they are failed to
be detected (see “B” in Fig. 17c). The reason might lie in the
high similarity between the defects and the textures, so that the
superpixels containing these defects are mistaken as back-
ground superpixels.
Based on the above analysis, one of the future research
directions of the proposed method might be related with the
1678
defect inspection for the low-contrast ILRs or the defects sim-
ilar to the texture.
7 Conclusions
In this paper, the vision-based automatic detection technique
is proposed for the cold rolling process to detect steel surface
defects covered by industrial liquids and monitor the cold
rolling system based on the qualification of the ILR numbers
and sizes. Based on the experimental results, the following
conclusions could be drawn.
1. The proposed method can detect most cracks and
scratches accurately. All the experiments achieve
the accuracy of more than 91 % although there is
a big interference induced by industrial liquids and
surface textures during the detection process. The
proposed method is believed to be easy-to-handle
because only one parameter (the desired number of
superpixel K) needs to be preset before the detection
process and, once selected, this parameter is proved
to be robust to 100 images with two kinds of de-
fects covered by liquids.
2. Except for defects, the proposed method could also accu-
rately quantify the industrial liquids including their num-
bers (with the average accuracy of more than 90 %) and
their sizes (with the average accuracy of more than 91 %).
The proposed method therefore is expected to be promis-
ing because it could be utilized to track the online running
status of the rolling system and provide rolling system
maintenance suggestions.
Future research might be related with the applicability of
the proposed method for other kinds of surface defects (e.g.,
voids, pits) including low-contrast ILRs and defects, the re-
duction of the proposed method computation to improve its
real-time properties, and the selection of the appropriate preset
superpixel sizes.
Acknowledgments The authors acknowledge the support from the
National Natural Science Foundation of China in undertaking this re-
search work under grant reference number 51374063 and the
Fundamental Research Funds for the Central Universities, the numbers
N140303008 and N141008001.
References
1. Rajput R (2007) A textbook of manufacturing technology:
manufacturing processes. Firewall Media
2. Lambiase F (2013) Optimization of shape rolling sequences by
integrated artificial intelligent techniques. Int J Adv Manuf
Technol 68(1–4):443–452
Int J Adv Manuf Technol (2017) 90:1665–1678
3. Roberts WL (1978) Cold rolling of steel. Dekker
4. Bruzzone AAG, Costa HL, Lonardo PM, Lucca DA (2008)
Advances in engineered surfaces for functional performance.
CIRP Ann Manuf Technol 57(2):750–769
5. Yun JP, Choi SH, Seo B, Park CH, Kim SW (2008) Defects detec-
tion of billet surface using optimized gabor filters. In: Proceedings
of the 17th world congress international federation of automatic
control Seoul, pp 77–82
6. Li X, Tso SK, Guan XP, Huang Q (2006) Improving automatic
detection of defects in castings by applying wavelet technique.
IEEE Trans Ind Electron 53(6):1927–1934
7. Sylla C (2002) Experimental investigation of human and machine-
vision arrangements in inspection tasks. Control Eng Pract 10(3):
347–361
8. Suresh BR, Fundakowski RA, Levitt TS, Overland JE (1983) A
real-time automated visual inspection system for hot steel slabs.
IEEE Trans Pattern Anal Mach Intell 5(6):563–572
9. Maki H, Tsunozaki Y, Matsufuji Y (1993) Magnetic on-line defect
inspection system for strip steel. Iron Steel Eng (USA) 70(1):56–59
10. Strom Y (1992) Automatic surface inspection of continuously cast
billets. Iron Steel Eng (USA) 69(5):29–33
11. Rao BPC, Raj B, Jayakumar T, Kalyanasumdaram P (2002) An
artificial neural network for eddy current testing of austenitic stain-
less steel welds. NDT & E Int 35(6):393–398
12. Gong R, Chu M, Wang A, Yang Y (2015) A fast detection method
for region of defect on strip steel surface. ISIJ Int 55(1):207–212
13. Jia H, Murphey YL, Shi J, Chang TS (2004) An intelligent real-time
vision system for surface defect detection. In: Proceedings of the
17th International Conference on Pattern Recognition, pp 1051–
1055
14. Tolba A, Raafat HM (2015) Multiscale image quality measures for
defect detection in thin films. Int J Adv Manuf Technol 79(1–4):
113–122
15. Landstrom A, Thurley MJ (2012) Morphology-based crack detec-
tion for steel slabs. IEEE J Sel Top Sign Process 6(7):866–875
16. Medina R, Gayubo F, González-Rodrigo LM, Olmedo D, Bermejo
J, Zalama E, Peran JR (2011) Automated visual classification of
frequent defects in flat steel coils. Int J Adv Manuf Technol 57(9–
12):1087–1097
17. Ginzburg VB (2009) Flat-rolled steel processes: advanced technol-
ogies. CRC Press
18. Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Su¨sstrunk S (2012)
SLIC superpixels compared to state-of-the-art superpixel methods.
IEEE Trans Pattern Anal Mach Intell 34(11):2274–2281
19. Gonzalez RC, Woods RE (2002) Digital image processing. Prentice
Hall, Upper Saddle River, NJ
20. Yang C, Zhang L, Lu H, Ruan X, Yang MH (2013) Saliency detec-
tion via graph-based manifold ranking. IEEE conference on com-
puter vision and. Pattern Recogn:3166–3173
21. Oskoei MA, Hu H (2010) A survey on edge detection methods.
University of Essex, UK
22. Mak KL, Peng P, Yiu K (2009) Fabric defect detection using mor-
phological filters. Image Vis Comput 27(10):1585–1592
23. Degarmo EP, Black JT, Kohser RA (2003) Materials and Processes
in Manufacturing (9th ed.). Wiley
24. Arce GR (2005) Nonlinear signal processing: a statistical approach.
John Wiley & Sons
25. Pernkopf F, O’Leary P (2003) Image acquisition techniques for
automatic visual inspection of metallic surfaces. NDT & E Int
36(8):609–617
26. Mehra J, Neeru N (2016) A brief review: super-pixel based image
segmentation methods. International journal of research and.
Engineering 3(7):21–24