See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/330350446

Structured-Illumination Reflectance Imaging for the Detection of Defects in

Fruit: Analysis of Resolution, Contrast and Depth-resolving Features

Article  in  Biosystems Engineering · January 2019

DOI: 10.1016/j.biosystemseng.2019.01.014

CITATIONS READS

2 226

2 authors, including:

Yuzhen Lu
United States Department of Agriculture Agriculture Research Service
38 PUBLICATIONS   249 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Robot-assisted sensing and automation for characterization and monitoring for specialty crops View project

Development and evaluation of an apple harvest-assist and in-field sorting (HAIS) machine View project

All content following this page was uploaded by Yuzhen Lu on 04 February 2019.

The user has requested enhancement of the downloaded file.

 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5

Available online at www.sciencedirect.com

ScienceDirect

journal homepage: www.elsevier.com/locate/issn/15375110

Research Paper

Structured-illumination reflectance imaging for the

detection of defects in fruit: Analysis of resolution,
contrast and depth-resolving features

Yuzhen Lu a, Renfu Lu b,*

a
Department of Biosystems and Agricultural Engineering, Michigan State University, East Lansing, MI 48824, USA
b
United States Department of Agriculture Agricultural Research Service, East Lansing, MI 48824, USA

article info
Structured-illumination reflectance imaging (SIRI), compared to conventional uniform
Article history: illumination based modalities, provides a new means for defect detection of fruit. The
Received 18 June 2018 capability of SIRI for defect detection, however, mainly depends on its image resolution
Received in revised form and contrast and depth-resolving features. This study was therefore aimed at providing
15 October 2018 theoretical and experimental analyses of these features for SIRI, so as to evaluate its po-
Accepted 16 January 2019 tential for detecting surface and subsurface defects of fruit and other food products. The
image formation in SIRI was first modelled in terms of the optical transfer functions, which
led to a general image demodulation methodology and also provided insights into the
Keywords: features of direct component (DC) and amplitude component (AC) images that were
Structured illumination demodulated from the original SIRI pattern images. A set of experiments were performed
Resolution on standard optical targets and apple samples by using illumination patterns over a wide
Contrast range of spatial frequencies of 0.01e1.0 cycles mm
1 to examine the features of SIRI. The
Depth imaging process acted as a low-pass filter, which, coupled with the sample absorption and
Spatial frequency scattering, accounted for the resolution and contrast loss of resulting images. As the
Defect hallmark of SIRI, AC images possessed enhanced resolution and contrast, which could be
explained by the synergistic effect of frequency shifting in the Fourier domain and sup-
pressed light scattering in the sample, and also a depth-resolved ability by varying the
spatial frequency of illumination patterns. However, the depth of tissue interrogation by
the AC images decreased as the spatial frequency of illumination patterns increased. The
effectiveness of AC in defect detection of apples depended on such factors as defect type,
fruit surface morphology (thus variety) and spatial frequency of illumination. Without
resorting to an inverse computational approach, SIRI should be positioned as a subsurface-
surface modality for detecting defects that are on or close to the surface of fruits, especially
thin-skinned fruits like apple. Further research is therefore needed to explore the full
potential of SIRI in defect detection of fruits and other food products.
© 2019 IAgrE. Published by Elsevier Ltd. All rights reserved.

* Corresponding author.
E-mail addresses: luyuzhen@msu.edu (Y. Lu), renfu.lu@ars.usda.gov (R. Lu).
https://doi.org/10.1016/j.biosystemseng.2019.01.014
1537-5110/© 2019 IAgrE. Published by Elsevier Ltd. All rights reserved.
2 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5

An acquired image should be a faithful representation of

Nomenclature the object being imaged, without resolution and contrast loss
(Hecht, 2002). In practise, however, the image resolution and
Acronyms contrast would unavoidably degrade, due to the band-limiting
AC Amplitude component property of the imaging lens, resulting from diffraction ef-
DC Direct component fects, and the discrete sampling process of the camera
PSF Point spread function (Bertero & Boccacci, 1998; Goodman, 2005), and other optical
MFP Mean free path aberrations. Biological samples attenuate and broaden the
MTF Modulation transfer function incident light travelling within the media due to absorption
NEM Noise-equivalent modulation and scattering (Mosk, Lagendijk, Lerosey, & Fink, 2012; Tuchin,
TMFP Transport mean free path 2015), which also contribute to the degradation of image res-
SI Structured illumination olution and contrast. Illumination plays a vital role in an im-
SIRI Structured-illumination reflectance imaging aging system. A well-designed illumination scheme helps
SIM Structured-illumination microscopy produce high-resolution/contrast images to enhance fruit
SNR Signal-to-noise ratio defect detection. Conventional imaging techniques, including
UI Uniform illumination monochromatic, colour, and multispectral and hyperspectral
LCTF Liquid crystal tuneable filter (Lorente et al., 2012; Lu & Lu, 2017b; Qin, Chao, Kim, Lu, &
Burks, 2013), utilise uniformly or quasi-uniformly distributed
Symbols
light (referred to as uniform illumination, or UI), to detect fruit
5 Convolution operator
defects. However, UI is susceptible to the image degradation
CM Michelson contrast
resulting from the aforementioned instrument-sample ef-
f Spatial frequency (cycles mm
1)
fects, and moreover, it has the difficulty controlling light
H Point spread function
penetration for imaging the heterogeneity of the sample,
i Imaginary unit
restricting its ability to detect depth-specific defects. There are
I Acquired pattern images
many types of defects that are externally invisible or even
P Illumination pattern images
deep inside the fruit, which require an imaging technique to
S Sample information
have a sufficient detection depth. However, due to strong light
ma Absorption coefficient (cm
1)
scattering in biological tissues, most optical imaging modal-
meff Effective light attenuation coefficient (cm
1)
ities with UI have difficulty interrogating deep tissues (e.g.,
m0 s Reduced scattering coefficient (cm
1)
more than several millimetres) with adequate resolution and
Q Phase offset (rad)
contrast (Ntziachristos, 2010; Shi & Alfano, 2017).
(u, v) Frequency coordinates in the Fourier space
Structured illumination (SI), in which the light is patterned
(pixel)
or structured in certain manner, provides an alternative to UI
(x, y) Spatial coordinates (pixel)
for sample imaging. SI is used to enhance image resolution
and contrast in microscopy (Dong, Nanda, Shiradkar, Guo, &
Zheng, 2014; Gao, Pedrini, & Osten, 2013; Gustafsson, 2000;
Neil, Juskaitis, & Wilson, 1997; Thomas, Wolstenholme,
1. Introduction Chaudhari, Kipreos, & Kner, 2015), where light diffraction ef-
fects are mainly responsible for image degradation. Depend-
Effective fruit defect detection using an imaging inspection ing on image acquisition and processing procedures, SI
system relies on acquiring high quality images, for which microscopy (SIM) can be further classified into optical
resolution and contrast are two critical image characteristics. sectioning and super-resolution (Dan, Yao, & Lei, 2014); the
Resolution, hereinafter referred to as spatial resolution, dic- former obtains enhanced-contrast optically sectioned images
tates the ability of an imaging system to resolve or distinguish by employing a phase-shifting pattern demodulation algo-
the smallest features that are close to each other (Bigio & rithm (Neil et al., 1997; Thomas, Momany, & Kner, 2013), while
Fantini, 2016; Smith & Webb, 2010). Contrast, on the other the latter, which involves image deconvolution and recon-
hand, is related to the differentiation between different re- struction, can achieve image resolutions beyond the limit
gions of interest within an image, which can be quantified by governed by light diffraction (Gustafsson, 2000, 2005). In
the (relative) difference or variance of the imaged property recent years, SI has been extended from the diffraction regime
(e.g., intensity) in a neighbourhood or across the whole image to the diffusion regime for quantitation and mapping of op-
(Bex & Makous, 2002; Peli, 1990). Compared to resolution, tical scattering and absorption properties (Cuccia, Bevilacqua,
which is largely determined by the imaging sensor (e.g., pixel Durkin, Ayers, & Tromberg, 2009; Cuccia, Bevilacqua, Durkin,
number), contrast is more of a property of instrument-sample & Tromberg, 2005; Mazhar et al., 2010; Weber, Cuccia,
combination, and, in the regime of macroscopic imaging, it is Durkin, & Tromberg, 2009) and depth-resolved tomography
often of more practical significance to defect localization and (D'Andrea, Ducros, Bassi, Arridge, & Valentini, 2010; Konecky
segmentation. The resolution and contrast features of an et al., 2009; Lukic, Markel, & Schotland, 2009) of biological
imaging system can be collectively described in terms of its tissues. Both optical mapping and tomography are based on
modulation transfer function (MTF), which is the modulation the fact that light attenuation depends on scattering and ab-
depth or amplitude response to sinusoidal illumination pat- sorption of tissues as well as spatial frequency of illumination
terns at different spatial frequencies (Boreman, 2001). patterns, and they are realised by projecting a set of
 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5 3

illumination patterns at different spatial frequencies, aided
with light transport model-based inverse algorithms (also
known as inverse problems). Anderson, Cuccia, and Durkin
(2007) first used this approach to quantify and map optical
properties of apples for the discrimination of bruised regions
from sound ones.
Based on these successful applications of SI, our group
developed a new SI reflectance imaging (SIRI) modality for fruit
defect detection (Lu, Li, & Lu, 2016c). Using a forward approach,
the features of images obtained by a demodulation method
were directly analysed. The approach simplifies and expedites
image acquisition and processing, instead of using the more
sophisticated image reconstructions in super-resolution SIM
and the inverse algorithms in optical property mapping and
tomography. SIRI demonstrated superior performance in
detecting subsurface bruising in apples, under both broadband
black/white (Li, Lu, & Lu, 2018) and multispectral imaging (Lu &
Lu, 2017a) modes, due to enhanced image resolutions and
contrasts and depth-resolved features as well. Further effort
was also made on improving the demodulation method for fast
image acquisition (Lu, Li, & Lu, 2016a, 2016b) and on imple-
menting multi-frequency composite sinusoidal patterns (Lu &
Lu, 2017c). Despite the progress being made on SIRI for defect
detection, there still is a lack of basic understanding of the
fundamental features of SIRI, regarding image resolution and
contrast and depth-resolving capability. It is thus desirable and
also necessary to examine these features so as to better assess
the potential of SIRI for fruit defect detection and help optimise
related imaging hardware.
This paper presents a theoretical and experimental analysis
of the fundamental features of SIRI in order to enhance defect
detection of fruit. A theoretical framework was proposed for
modelling image formation in SIRI, which resulted in a general
methodology for image demodulation. Qualitative and quanti-
tative analyses were provided of the SIRI technique on its res-
olution and contrast enhancement and depth-resolving
features. Experiments were performed on standard imaging
targets and defective apple samples to examine the MTF profile
of a SIRI system and validate its fundamental features.
respectively; Sðx; yÞ represents the sample information, more
specifically, the absorption contrast; Hs ðx; yÞ and Hi ðx; yÞ
represent the point spread function (PSF) due to light scat-
tering in the sample and the imaging process, respectively;
and 5 denotes the convolution operation. It should be noted
that noise, which is present in any practical imaging system,
is not included in Eq. (1) and will be considered later.
The convolution operations describe how light scattering
and the imaging process negatively affect the resolution and
contrast of resulting images. Scattering events broaden the
light beam, thus blurring acquired images. The imaging pro-
cess, in essence, acts as a low-pass filter, due to imperfect
focussing of the lens, wave diffraction, the discrete sampling
process of camera and other possible optical aberrations. All
these factors contribute to the resolution and contrast loss of
acquired images. For mathematical convenience, we combine
the PSFs of the light scattering and imaging process into a
single term Hðx; yÞ ¼ Hs ðx; yÞ 5 Hi ðx; yÞ, albeit the two pro-
cesses are independent. Logically, Hðx; yÞ has the effect of low-
pass filtering on the product of the illumination pattern and
the sample information.
In Eq. (1), the illumination pattern Pn ðx; yÞ is generated in a
sinusoidal form as follows:
1
 
Pn ðx; yÞ ¼ 1 þ cos 2pfx x þ qn (2)
2
where fx is the spatial frequency along the x-axis; and qn is the
phase offset. Eq. (2) can be rewritten in a complex exponential
form as follows:
1 1
   
Pn ðx; yÞ ¼ þ expðiqÞexp i2pfx x þ expð
iqÞexp
i2pfx x
2 4
(3)
Substituting Eq. (3) into Eq. (1) yields:
1 1
In ðx; yÞ ¼ IDC ðx; yÞ þ expðiqÞISI ðx; yÞ þ expð
iqÞI*SI ðx; yÞ (4)
2 2
where
1
IDC ðx; yÞ ¼ ½Sðx; yÞ5Hðx; yÞ; (5)
2

2. Theoretical formulation for SIRI 1 

ISI ðx; yÞ ¼ ½expði2pfx xÞSðx; yÞ5Hðx; yÞ ; (6)
2
2.1. Image formation and I*SI ðx; yÞ is the complex conjugate of ISI ðx;yÞ. Further, Eq.
(4) can be expressed in the Fourier domain as follows:
In SIRI, the image formation involves two steps: 1) the incident
 
light interacts with the sample through absorption and scat- ~In ðu; vÞ ¼ 1 ½Sðu; ~ vÞ þ 1 expðiqÞS~ u
fx ; v Hðu;
~ vÞHðu; ~ vÞ
tering, which are responsible for the decay and broadening of 2 4
1  
the light, respectively, and 2) the light reemitted from the þ expð
iqÞS~ u þ fx ; v Hðx;
~ yÞ (7)
4
sample travels through a series of optical devices (e.g., lens,
camera) of the imaging system, and eventually forms a digital where the tilde denotes the Fourier transformed function and
image. Here we treat SIRI as a linear, space-invariant system, ðu; vÞ are frequency coordinates. Eqs. (4) or (7) lays the math-
which leads to the application of transfer function theory to ematical basis for analysing the SIRI system.
an optical imaging system (Goodman, 2005; Saleh, 2011). According to Eqs. (4) and (7), the acquired pattern image is
Then, the image formation can be modelled as follows: composed of three parts. The first part is direct component
(DC), i.e., IDC ðx; yÞ, corresponding to the conventional image
obtained by a UI-based imaging system, with the Fourier
In ðx; yÞ ¼ Pn ðx; yÞSðx; yÞ5Hs ðx; yÞ5Hi ðx; yÞ (1) spectrum centred at the origin. The other two parts are
where ðx; yÞ are the image coordinates; In ðx; yÞ and Pn ðx; yÞ are oscillatory or harmonic components, complex conjugates of
the nth acquired and illumination pattern images, each other, with the Fourier spectra shifted by fx and
fx , and
4 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5

hence they share the same amplitude distribution. We denote As aforementioned, Hðx; yÞ has a low-pass filtering prop-
 
the sum of their amplitude distribution as IAC ðx; yÞ, termed as erty. So the MTF of the imaging system [i.e., MTF ¼ Hðu; ~ vÞ],
amplitude component (AC) image, which is mathematicaly which is due to the light scattering and imaging process,
 
equal to jISI ðx; yÞj or I*SI ðx; yÞ. should be a bell-shaped function, e.g., the one depicted in
Fig. 1(a). The MTF profile describes how the imaging system
2.2. Image demodulation transfers the sample information (i.e., contrast) at a given
spatial frequency (i.e., resolution), and it has the lower and
Image demodulation is aimed to retrieve DC and AC images, upper cut-off frequencies that define the passband of the
i.e., IDC ðx; yÞ and IAC ðx; yÞ, from the acquired pattern images. system, beyond which the sample (high-frequency) informa-
Although Eqs. (5) and (6) provide the makeup of the two images, tion is inaccessible to the imaging system. According to Eq. (7),
they cannot be directly used to solve for the DC and AC images, IAC ðx; yÞ contains the sample information whose Fourier
since the PSFs of the tissue scattering and the imaging system spectrum has been shifted by fx , compared to that in IDC ðx;yÞ.
are unknown. By revisiting Eq. (4), a general method to achieve It is due to the spectral shifting that a certain amount of
this can be developed by establishing and solving a linear sys- higher-frequency sample information, which is originally
tem consisting of different phase-shifted pattern images. inaccessible in IDC ðx; yÞ due to the limited passband of the
Since there are three unknowns in Eq. (4), we need to ac- imaging system, is now down-modulated into the lower-
quire three phase-shifted pattern images from a sample, so as frequency passband and becomes resolvable, as schemati-
to establish a well-posed linear system as follows: cally illustrated in Fig. 1(b). Hence, IAC ðx; yÞ is expected to have
2 3 2 3 higher image resolution and contrast than IDC ðx; yÞ.
I1 ðx; yÞ IDC ðx; yÞ
4 I2 ðx; yÞ 5 ¼ M4 ISI ðx; yÞ 5 It is also important to note that the down-modulating of
(8)
I3 ðx; yÞ I*SI ðx; yÞ higher-frequency information is accompanied by up-
modulating of lower-frequency information, which indicates
where that IAC ðx; yÞ would have lower intensity than IDC ðx; yÞ; higher
2 3 frequency would result in the lower intensity, due to the
2 expðiq1 Þ expð
iq1 Þ
14 decreasing MTF profile with the increasing frequency. It fol-
M ¼ 2 expðiq2 Þ expð
iq2 Þ 5 (9)
2 lows that, assuming an ideal shot-noise limited imaging sys-
2 expðiq3 Þ expð
iq3 Þ
tem, higher frequencies would result in a lower signal-to-
Then, one can solve for IDC ðx; yÞ and ISI ðx; yÞ [further, IAC ðx;yÞ
noise ratio (SNR) for IAC ðx; yÞ. Therefore, there would be a
¼ jISI ðx; yÞj], by simply inverting the matrix M, assuming it is
trade-off between image resolution/contrast and SNR.
not rank-deficient, and pre-multiplying it to the pattern image
It should be pointed out that, an implicit assumption un-
matrix. This method can be naturally extended to multi-
derlying the mathematical understanding of the resolution
frequency composite pattern images, as equivalently imple-
and contrast enhancement is that the sample information
mented in the Fourier domain (Lu & Lu, 2017c).
Sðx; yÞ and the point spread function Hs ðx; yÞ due to light
Consider a special case, in which the phase offsets of illu-
scattering remain unchanged in varying spatial frequencies.
mination patterns are 0, 2p/3 and 4p/3. It can be readily
While this assumption leads to simplification of the analysis,
proved that the general expression given in Eq. (8) is now
one should be aware that different spatial frequencies would
reduced to the classic demodulation approach (Cuccia et al.,
result in different light penetration, and probably a change in
2009; Neil et al., 1997) as follows:
Sðx; yÞ and Hs ðx; yÞ due to the heterogeneity/anisotropy of real
1 samples, which is discussed in Section 2.5.
IDC ¼ ðI1 þ I2 þ I3 Þ (10)
3
2.4. Light penetration
pffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
2
IAC ¼ ðI1
I2 Þ2 þ ðI1
I3 Þ2 þ ðI2
I3 Þ2 (11) In diffuse optics, the light penetration depth in biological tis-
3
sues is usually defined as the reciprocal of the effective light
where the image coordinates ðx; yÞ are omitted for brevity.
attenuation coefficient (meff ), which is equal to the travelling
Simplifications have been made to this method by using two
distance of light corresponding to a decay in power rate by a
phase-shifted images (Lu et al., 2016a; 2016b). In this study, we
factor of 1/e (~37%) (Wilson & Jacques, 1990). According to this,
adhere to the classic method for image demodulation in the
the light penetration depth is estimated to be 2e6 mm for
following experimental demonstrations.
apple tissues with typical optical absorption (ma ) and reduced
scattering (m0 s ) coefficients of 0.1e0.5 cm
1 and 9.0e17.0 cm
1,
2.3. Image resolution and contrast respectively, over the spectral region of 500e1000 nm. This
depth, however, does not indicate the actually detectable re-
Resolution, contrast and light penetration or detection depth gion for a general imaging system under spatially-extended
features are of primary concern for the demodulated images, wide-field or broad-beam illumination. In such cases, the
IDC ðx; yÞ and IAC ðx; yÞ, which are also the most critical for fruit detected signal is mostly due to the photons backscattered
defect detection. The resolution and contrast features can be close to the illumination source [i.e., the decreased source
understood by examining the image formation model as (light entry point)-detector (camera pixel) separations] (Bigio &
described in Eq. (7). A separate discussion on detection depth Fantini, 2016; Dognitz & Wagnieres, 1998), which correspond
and its relationship with image resolution and contrast is to a far more superficial depth of tissue interrogation than that
given in the following subsections. derived from diffuse light attenuation. This is especially
 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5
Fig. 1 e (a) A simulated modulation transfer function (MTF) profile for a structured-illumination reflectance imaging (SIRI)
system, and (b) visualization of the spectral shifting of a SIRI pattern image in the Fourier domain [Note: the notations in the
figure comply with those in Eq. (7)].
important in the reflectance measurement geometry, since
the reemitted light intensity due to the point source decays by
many orders of magnitude within millimetres.
It has been shown that, through Monte Carlo (MC) model-
ling on light transport inside the human skin (Binzoni, Vogel,
Gandjbakhche, & Marchesini, 2008), the optical information in
the image captured under uniform planar-wave illumination,
primarily originates from a small, conic like volume of tissue
situated vertically under each image pixel, with a maximum
detection depth of about 2 mm in the wavelength range of
600e1000 nm. Fruit tissues such as apples are comparable to
human skins in the order of magnitude of light scattering
ability (i.e., m0 s in the wavelength range) (Bashkatov, Genina,
Kochubey, & Tuchin, 2005), suggesting similar, limited detec-
tion depths [See numerical simulations in Fig. 2(b)], albeit the
actual depth depends on specific sample and imaging
hardware.
For the demodulated images in SIRI, IDC ðx; yÞ is equivalent
to that obtained by conventional wide-field UI based imaging
and has a relatively fixed depth of tissue interrogation, while
IAC ðx; yÞ is due to the use of SI and has the ability to control the
detection depth by varying the spatial frequency of illumina-
tion patterns. As shown in Fig. 2(a), due to scattering and ab-
sorption, biological tissues act as a low-pass filter, which
attenuates the reflectance more strongly as the spatial fre-
quency increases (Cuccia et al., 2005). As a result, as shown in
Fig. 2(b), the low-frequency light is more sensitive to a deeper
region of tissue (i.e., larger detection depth), while the high-
frequency light is more sensitive to shallower tissues (i.e.,
smaller detection depth), and IDC ðx; yÞ represents an asymp-
totic case of IAC ðx; yÞ when the spatial frequency is
approaching zero. Hence, IDC ðx; yÞ can image deeper tissues
5
than IAC ðx; yÞ, but IAC ðx; yÞ enables depth-resolved imaging of
tissue, which IDC ðx; yÞ is incapable of.
2.5. Relationship between light penetration and image
resolution/contrast
Examination of the composition of photons backscattered
from a turbid medium will provide a qualitative insight into
the relationship between detection depth and image
contrast/resolution. As schematically illustrated in Fig. 3, the
backscattered photons comprise three types, i.e., ballistic,
weakly scattered and diffusive or multiple-scattered (Wang,
Ho, Liu, Zhang, & Alfano, 1991), corresponding to three
levels of light interrogation in the medium. The ballistic
photons experience one backward and one forward scat-
tering event before being detected, and due to the shortest
travelling path, they suffer from minimal scattering (the
corresponding PSF is a Dirac Delta function) and thus can
deliver images of superior resolution and contrast. However,
the ballistic photons yield information on the very superficial
layer of the medium, of about one mean free path (MFP) of a
photon (Ntziachristos, 2010), around or less than 100 mm for
fruit tissues like apple (assuming a scattering coefficient ms
equal to or larger than 100 cm
1). The weakly scattered
photons provide information on deeper, sub-surface tissues,
and they are still capable of forming well-resolved images
due to limited scattering events. One transport mean free
path (TMFP), usually less than 1 mm for fruit tissues, repre-
sents an upper bound of light penetration, beyond which the
photons become completely diffusive (Ntziachristos, 2010),
significantly degrading the resolution and contrast of result-
ing images. Therefore, the detection depth decreases with the
6 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5

Fig. 2 e (a) Simulated modulation transfer function (MTF) profiles of apple tissues obtained through Monte-Carlo (MC)
simulation and diffusion approximation (Cuccia et al., 2009), assuming a typical pair of optical properties of absorption (ma ¼
0:2 cm
1 ) and reduced scattering (m0 s ¼ 15 cm
1 ) coefficients. In addition to the optical properties, other settings for the MC
simulations include 5 millions of photons, reflective index of 1.35, anisotropy factor of 0.9 and thickness of 10 cm for the
slab-geometry tissue. (b) Simulated median optical detection depth for apple tissues (ma ¼ 0:2 cm
1 and m0 s ¼ 15 cm
1 ),
which is estimated using a scaled lookup table derived from MC simulations to the radiative transport equation in the
spatial frequency domain (Hayakawa, Karrobi, Pera, Roblyer, & Venugopalan, 2018). The median detection depth is the
depth that encloses the photon trajectories responsible for 50% of the detected light, and accordingly, the vertical-capped
lines in the figure correspond to detection depths responsible for 25% (lower) and 75% (upper) of the detected light.

formation for IDC ðx; yÞ and IAC ðx; yÞ, by separately expressing
the associated sample information and scattering PSF as
follows:

1
IDC ðx; yÞ ¼ ½SDC ðx; yÞ5Hs DC ðx; yÞ5Hi ðx; yÞ (12)
2

1  

IAC ðx; yÞ ¼ ½expði2pfx xÞSAC ðx; yÞ5Hs AC ðx; yÞ5Hi ðx; yÞ (13)
2
where SDC ðx; yÞ and Hs DC ðx; yÞ and SAC ðx; yÞ and Hs AC ðx; yÞ
denote the sample information and scattering PSF for DC and
Fig. 3 e Schematic of light scattering within a turbid AC images, respectively. The demodulation approaches for
medium for reflectance measurements, showing three IDC ðx; yÞ and IAC ðx; yÞ presented above remain the same.
types of backscattered photons, i.e., ballistic, weakly Presented in the following sections are several experi-
scattered and diffusive (or multiple scattered), with ments that were conducted to demonstrate the image reso-
corresponding light penetration regimes. lution and contrast enhancement and depth-resolving
imaging features of SIRI.

increasing image resolution and contrast (Gigan, 2017;

Ntziachristos, 2010).
The IDC ðx; yÞ probes a deeper region of tissues than IAC ðx;yÞ;
the former includes a larger contribution from diffusive pho-
tons, thus degrading the resulting resolution and contrast,
while IAC ðx;yÞ, which probes a shallower depth, would contain
more ballistic and weakly-scattered photons, thus producing
better resolution and contrast. Higher spatial frequency is
more likely to enhance image resolution and contrast. This
general relationship is in agreement with the mathematical
interpretation (Section 2.3.) of resolution and contrast
enhancement due to frequency shifting in the Fourier domain.
In case of very high-frequency illumination, only ballistic or
quasi-ballistic photons are captured by the sensor, and the
imaging process becomes the limiting factor for image reso-
lution and contrast.
Since IDC ðx; yÞ and IAC ðx; yÞ have different regimes of tis-
sue interrogation, with different sample information and
light scattering, it is more appropriate to modify the image
3. Materials and methods
3.1. SIRI system
As shown in Fig. 4, the SIRI system mainly consists of a
computer, a digital light projector (Model DLi CEL5500-Fiber,
Digital Light Innovations, Austin, TX, USA) with a resolution of
768  1024 pixels, a 250-W quartz-tungsten-halogen lamp
(Newport Corporation, Irvine, CA, USA) along with a radio-
metric power supply controller and a 12-bit back-illuminated
electron-multiplying charge-coupled device camera (Model
PhotonMax:1024B, air-cooled, Princeton Instruments, Tren-
ton, NJ, USA) with a resolution of 1024  1024 pixels, attached
with a C-mount fixed focal length lens (35 mm, Edmund Optics
Inc., Barrington, NJ). A liquid crystal tuneable filter (LCTF)
(Varispec, Cambridge Research and Instrumentation, Inc.,
Woburn, MA, USA) is placed in front of the lens for multi-
spectral imaging, which allows for selecting any wavelength
 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5
Fig. 4 e Schematic (left) and photo (right) of a structured-illumination reflectance imaging (SIRI) system.
in the spectral range of 650e1000 nm. A fibre optic cable is
used to channel the output of the lamp into the projector, and
a linear polariser is placed in front of the projector, which
works together with the LCTF to suppress specular reflec-
tance. The SIRI system is operated in an enclosed dark
chamber.
Phase-shifted sinusoidal patterns in 8-bit image format were
created in Matlab (The Mathworks, Inc., Natick, MA, USA) and
uploaded to the projector's control software for sample illumi-
nation. Due to the tilting of the projector, the projection area
was approximately an isosceles trapezoid, with two parallel
sides of 110 mm and 130 mm, and the width of 100 mm. The
fringe of illumination patterns was perpendicular to the plane
spanned by the optical axes of the camera and the projector.
Image acquisitions were performed by using a custom-
developed graphical user interface program written in Lab-
VIEW (National Instruments, Austin, TX, USA), in the wave-
length range of 650e830 nm with 20 nm increments, with the
exposure time of 300 ms and on-chip 2  2 binning operations.
3.2. Experiments
3.2.1. Experiment 1: MTF of the SIRI system
The first experiment was to examine the MTF profile of the
SIRI system due to the imaging process. Images were acquired
from a Spectralon white panel (Labsphere, Inc., Sutton, NH,
USA) with a reflectance rate of 98% or higher. By imaging the
white panel, it could be reasonably assumed that the reflec-
tance attenuation was only due to the instrument hardware. A
set of sinusoidal patterns, covering 21 spatial frequencies of
0.01, 0.03, 0.05, 0.08, 0.10, 0.12, 0.15, 0.18, 0.20, 0.22, 0.25, 0.30,
0.35, 0.40, 0.45, 0.50, 0.60, 0.70, 0.80, 0.90 and 1.00 cycles mm
1,
were used for sample illumination. Three phase-shifted
pattern images were acquired at each spatial frequency with
the phase offsets of 0, 2p/3 and 4p/3, respectively, and were
demodulated into DC and AC, according to Eqs. (10) and (11).
The average intensities at different spatial frequencies gave
the frequency response (i.e., MTF) profiles of the imaging
system. Besides, images were also acquired from a dark
background with the light source turned off, representing the
image noise due to the dark current of the image sensor. Other
noise sources, including photon-shot and readout noise, were
not considered due to their small contributions. Knowledge of
the noise magnitude allows to determine the limiting (upper-
7
bound) spatial frequency of the imaging system, which is
defined as the frequency at which the MTF curve falls below
the noise-equivalent modulation (NEM) (Boreman, 2001).
3.2.2. Experiment 2: contrast and resolution
This experiment was to demonstrate the resolution and
contrast enhancement features of SIRI. A negative USAF 1951
target, consisting of different groups of black horizontal and
vertical bars at various spacings on a white background, was
chosen for this experiment. This target was printed on regular
white printing paper (US Letter size) and imaged by SIRI,
illuminated with the same set of patterns as in Experiment 1,
excluding those above 0.5 cycles mm
1. The resulting DC and
AC images were compared in terms of resolution and contrast.
Image resolution was evaluated based on the ability to resolve
the fine details of the USAF 1951 target, while image contrast
was evaluated based on the Michelson contrast metric (CM)
(Bex & Makous, 2002):
CM ¼ ðImax
Imin Þ=ðImax þ Imin Þ (14)
where Imax and Imin denote the maximum and minimum in-
tensities of an image, respectively. For the sake of reliability, it
is common practise to replace the maximum and minimum
with the mean of a number of the highest and lowest in-
tensities (Thomas et al., 2013). In this paper, a third of image
pixels of the highest and lowest intensities were used to
calculate the image contrast.
3.2.3. Experiment 3: depth-resolved imaging
This experiment was to show the depth-resolved imaging
feature of SIRI. Firstly, the detection depth of SIRI was
explored by imaging the USAF 1951 paper target under UI,
which was placed beneath a number of pieces of blank
printing paper. The thickness of each piece of paper was
estimated to be 100 mm. Afterwards, two targets, including the
USAF 1951 and another Snellen eye chart that consists of black
letters of various size printed on regular white printing paper,
were put together and imaged by SIRI with the same set of
illumination patterns as in Experiment 2. Two scenarios were
considered in arranging the two targets for image acquisition:
1) the Snellen target was placed on top of the USAF 1951 target,
and 2) a sheet of blank printing paper was put on top of the
Snellen target that was placed on top of the USAF 1951 target.
In both cases, the two targets were placed on top of a stack of
8 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5
blank paper of the same size and about 1 cm in thickness,
which served as an opaque background.
3.2.4. Experiment 4: defect detection of apples
As shown in Fig. 5, three defective apple samples, two ‘Deli-
cious’ and one ‘Red Rome’ of the equatorial diameter of
75e81 mm, were used to demonstrate the ability of SIRI for
defect detection. One of the two ‘Delicious’ apples, purchased
from a local grocery store, was subjected to subsurface
bruising by impacting its equatorial area with impact energy
of about 0.5 J, which could not be identified visually. The other
two apples were hand-picked from an orchard at the Horti-
cultural Teaching and Research Center of Michigan State
University in Holt, Michigan, USA. While still on the tree, these
apples, as shown in Fig. 5, had been inflicted with physiolog-
ical disorders, i.e., circular dark pits in the fruit skin centred on
the lenticels for the ‘Red Rome’, and skin russeting for the
‘Delicious’, appearing as a brown, corky and rough surface.
SIRI images were acquired from the three apples using the
same set of illumination patterns as in Experiments 2 and 3,
and image acquisitions for the ‘Delicious’ apple with bruise
were done immediately after it had been damaged by impact.
4. Results and discussion
4.1. MTF
Figure 6(a) shows the spectral curves for DC and AC images
acquired from the white panel. These spectra at different
spatial frequencies were due to the joint effect of the LCTF and
camera, which peaked around 730 nm. Figure 6(b) shows the
frequency response curves for DC and AC (i.e., the MTF profile
due to the imaging process of SIRI) images at wavelength
730 nm. As expected, the intensity of DC remained basically
unchanged, while the intensity of AC was lower than that of
DC (even at the lowest spatial frequency), and decreased
steadily with the increase in the spatial frequency, which was
true of other wavelengths as well. The decreasing MTF profile
confirmed the low-pass filtering property of the imaging pro-
cess, which would wash out the high-frequency sample in-
formation during image acquisition.
The MTF profile was well modelled by a Gaussian function,
2
f ðxÞ ¼ a expð
ðx
bÞ
2c2
Þ, where a, b and c are parameters to be
estimated, with the determination coefficient being larger
than 0.999 for all the wavelengths. The limiting frequency was
then determined by setting the function value to the noise
intensity. As indicated in Fig. 6(b), the limiting spatial
Fig. 5 e Photo of a ‘Delicious’ apple (left) with invisible subsurface bruising, a ‘Red Rome’ apple (center) with lenticel dark
spots, and a ‘Delicious’ apple (right) with skin russeting.
frequency was 0.57 cycles mm
1 at the wavelength 730 nm,
where the intensity of AC decreased to less than one tenth of
that of DC. Similarly, the limiting spatial frequency for other
wavelengths was in the range of 0.41e0.59 cycles mm
1. It is
important to stress that these obtained limiting spatial fre-
quencies do not represent the maximum that SIRI is able to
support, since it mainly depends on the imaging hardware
and configurations. For example, one can readily increase the
limiting spatial frequency by setting a longer camera exposure
time without causing signal saturation, especially for those
wavelengths, for which the imaging system has a low spectral
sensitivity [See Fig. 6(a)]. However, based on this experiment,
it is recommended that the spatial frequency used for SIRI
should not exceed 0.6 cycles mm
1.
Since the wavelength of 730 nm gave the highest image
intensity and also relatively large limiting spatial frequency,
only spectral images acquired at this wavelength were used in
further analysis.
4.2. Resolution and contrast
Figure 7 shows an example of resulting DC and AC images for
the USAF 1951 target. All the images could resolve well the
black bars for the groups
1 and
2, but not for the groups
0 and 1, which represented even finer details, as shown in the
close-up sub-image. So comparisons of image resolution were
made based on resolving the bars for the groups of 0 and 1.
Figure 8 shows one DC image and all AC images at different
spatial frequencies for the close-up view of the USAF 1951
target. It should be pointed out that each spatial frequency
produced a DC image but all the DC images remained basically
the same. Hence only the first DC image (i.e., at 0.01 cycles
mm
1) was presented for comparison. Visually, as the spatial
frequency increased, some vertical bars (e.g., those bars in the
lower right corner, with the actual spacing of bars of about
0.5 mm) became more and more resolved, while the resolution
for the horizontal bars was not noticeably improved. This is
because of the fact that the fringes of illumination patterns
were parallel to the vertical bars, so that the resolution
enhancement mainly occurred in the horizontal direction,
that is, the direction in which the frequency components of
illumination patterns are situated in the Fourier domain, thus
making the vertical bars more resolved.
As a further illustration, the pixel profiles for four repre-
sentative groups of vertical and horizontal bars were analysed
for the DC image and the AC images at 0.5 cycles mm
1. As
shown in Fig. 9, with using vertically orientated illumination
patterns, the AC image gave much more resolved pixel profiles
 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5 9

Fig. 6 e (a) Spectral responses of direct component (DC, dashed lines) and amplitude component (AC, solid lines) images at
different spatial frequencies, where the line profiles of different colours, which have been scaled by dividing the maximum
value of DC, correspond to individual spatial frequencies ranging between 0.01 and 1.0 cycles mm¡1, and the downward
arrow indicates the direction of increasing the spatial frequency (fx); (b) frequency responses of the DC and AC images and
the noise level at 730 nm, where each line profile has been normalised to the maximum of the DC image.

Fig. 7 e The direct component (DC) (left) and amplitude component (AC) (middle) images of the USAF 1951 target at the
spatial frequency of 0.15 cycles mm¡1, and the close-up view (right) of the AC image focussing on the groups 0 and 1
elements of the USAF 1951 target.

Fig. 8 e The close-up view of direct component (DC) and amplitude component (AC) images at different spatial frequencies
(cycles mm¡1) as indicted in the name of individual sub-images, for the bars of groups 0 and 1 of the USAF 1951 target.
10 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5

than the DC image for the vertical bars, while the AC and DC
images gave similar pixel profiles for the horizontal bars. In
contrast, with the horizontally orientated illumination pat-
terns, enhanced resolution for the vertical bars was observed
for the AC compared to the DC image. Moreover, the grid illu-
mination patterns, which integrated both vertical and hori-
zontal frequency components, yielded resolution enhancement
for the two types of bars for AC images, but at the cost of
increased image acquisitions (five phase-shifted illumination
patterns are needed for retrieval of DC and AC images).
Although resolution enhancement depends on the orien-
tation of illumination patterns, it may not make a marked
difference in fruit defect detection, since most defects are
non-directional in their spatial distribution. Hence the selec-
tion of illumination direction (i.e., vertical, horizontal or grid
patterns) largely depends on the application and system
setup, so as to ensure minimal pattern distortion and high-
quality demodulated images. In addition, the imaging speed
can be a critical factor given that grid patterns requires more
image acquisitions for demodulation than vertical and hori-
zontal patterns. The following discussion is therefore focused
on the vertical illumination patterns only.
Figure 10 shows the contrast curves for DC and AC images
of the USAF 1951 target at different spatial frequencies. For
both full-size and close-up images, DC had almost a constant
contrast value since it was independent of spatial frequency,
while AC had much higher contrast values than DC and the
contrast increased steadily with the spatial frequency.
These findings demonstrated that AC images enhanced
resolution and contrast compared to DC images, and hence
SIRI is superior over conventional UI based imaging technique
in enhancing image features.
4.3. Depth-resolved imaging
Figure 11 shows the UI images of the USAF 1951 target at
different depths, defined by the number of blank printing paper
sheets stacked on top of the target. The UI image draws an upper
bound of the detection depth that can be reached by SIRI. Clearly,
at the depth of more than three sheets of paper, direct imaging
(without involving any inverse, computational approaches) of
the target was impossible due to substantially deteriorated res-
olution and contrast. Thus the maximum detection depth with
acceptable resolution and contrast was confined to no more than
3 sheets of paper (or less than 400 mm). Although the detection
depth for fruit samples would be different, this experiment
revealed the superficial sampling nature of the reflectance im-
aging under wide-field uniform illumination.
Figure 12 shows the DC and AC images for the two paper
targets, i.e., the Snellen eye target and the USAF 1951 target.
For the Snellen target directly exposed to the camera, the DC
image, which corresponded to the maximum detection
depth, contained features on both targets; and the lower-
frequency AC images are similar to the DC image in
revealing the features of both targets. However, as the
spatial frequency increased, the AC images were

Fig. 9 e The close-up image and pixel profiles for the corresponding lines in the DC and AC images at the spatial frequency
of 0.5 cycles mm¡1: (top pane) vertically orientated illumination patterns, (middle pane) horizontally orientated illumination
patterns, and (bottom pane) two-frequency grid illumination patterns. It should be noted that the grid illumination pattern
has two frequency components of 0.5 cycles mm¡1 in horizontal and vertical directions respectively.
 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5 11

Fig. 10 e The curves of contrast values for the full-size (left) and close-up (right) direct component (DC) and amplitude
component (AC) images of the USAF 1951 target at different spatial frequencies.

Fig. 11 e Uniform illumination images of the USAF 1951 paper target placed beneath 1e6 (from left to right) sheets of blank
printing paper.

increasingly focused on the surface Snellen target, while the
features of the USAF 1951 almost disappeared completely at
the highest spatial frequency of 0.5 cycles mm
1. A similar
pattern was also observed for the second scenario, in which
one sheet of paper placed on top of the Snellen target. These
findings confirmed that the detection depth of AC images
decreased with the spatial frequency and thus depth-
resolved imaging of the sample can be achieved by control-
ling the spatial frequency.
It is important to note that although AC images for
different spatial frequencies would have different detection
depths, they are not truly sectioned or tomographic to provide
information exclusively on certain depth. All the images,
including DC, are still dominated by the shallower target in-
formation. An intuitive approach to obtaining a better depth-
sectioned image is to subtract from the DC or a lower-
frequency AC image a higher-frequency AC image multiplied
by a varying constant. As illustrated in Fig. 13, subtracting 1e2
times of the AC image from the DC image led to enhanced
subsurface features. Such simple subtractions, however,
cannot completely eliminate the information of the shallower
target, due to the nonlinear relationship between the
demodulated images and detection depth. Using inverse
image reconstructions in diffuse optical tomography
(Konecky, 2012; Lukic et al., 2009), it may be possible to obtain
more depth-specific images at greater depths, which is how-
ever beyond the scope of the current study and will be
explored in future work.
4.4. Defect detection
Figure 14 shows the DC and AC images for the three defective
apples. The subsurface bruising in the ‘Delicious’ apple was
invisible in the DC image, which was consistent with visual
inspection, and it still could not be observed clearly in the
lower-frequency (i.e., less than 0.05 cycles mm
1) AC images.
However, as the spatial frequency increased, the bruise region
became well resolved with enhanced contrast between
bruised and normal tissues. In the meantime, due to improved
image resolution and contrast at higher frequencies, the sur-
face lenticels of the fruit became more distinct, which could
interfere with the bruise detection. Above a certain spatial
frequency (e.g., 0.4 cycles mm
1), the bruise started to fade
out, and the image became greatly darkened due to strong
signal attenuation. The spatial frequency for effective bruising
detection would be confined to the range of 0.05e0.40 cycles
mm
1.
The ‘Red Rome’ apple with lenticel spots gave a quite
different picture (see the middle three rows of images in
Fig. 14). The lenticel spots could be readily identified in the DC
and low-frequency (e.g., less than 0.10 cycles mm
1) AC im-
ages, and also resolved in the colour photo (Fig. 5). However, as
the spatial frequency increased, the lenticel spots became less
obvious and even disappeared. This was because the lenticel
spots, although visible from the fruit surface, were not fully
exposed to air, but instead they occurred in the outer cortex
and lower epidermis covered with thin, intact cuticle (Brook,
1968). The high-frequency AC images were mainly focused
on the very surface features, thus overpassing the lenticel
spots. The ‘Red Rome’ did not have so many lenticels as in the
‘Delicious’, because of the difference in the skin morphology
of the two varieties. Likewise, the intensity of the AC images of
‘Red Rome’ decreased with the increasing spatial frequency.
The third apple of ‘Delicious’ variety exhibited skin
russeting, which, relative to the lenticel spots, is more of a
surface defect. Skin russeting results from microscopic cracks,
which probably occurs during early fruit development, fol-
lowed by the formation of a periderm in the hypodermal cell
12 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5

Fig. 12 e The direct component (DC) and amplitude component (AC) images for two paper targets for different spatial
frequencies: (top three rows) the Snellen target is placed on top of the USAF 1951, and (bottom three rows) one sheet of blank
paper is laid on top of the Snellen target that is on top of the USAF 1951 target.

Fig. 13 e The direct component (DC) image (left), amplitude component (AC) image at 0.30 cycles mm¡1 (AC_F0.3) (second
from left), and the reconstructed images (three from right) by subtracting the AC image multiplied by a constant (i.e., 1, 1.5
and 2, respectively) from the DC image. It should be noted that the heuristics for deciding an optimal constant is open to
question.
 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5 13

Fig. 14 e The direct component (DC) and amplitude component (AC) images at different spatial frequencies (cycles mm¡1) for
a ‘Delicious’ apple with subsurface bruising (top three rows), a ‘Red Rome’ apple with lenticel breakdown (middle three
rows) and a ‘Delicious’ apple with surface russeting (bottom three rows).
14 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5
layer, appearing at the fruit surface after the epidermis and
cuticle are shed (Faust & Shear, 1972; Khanal, Shrestha,
Huckstadt, & Knoche, 2013). As shown in Fig. 14 (bottom
three rows), all DC and AC images showed the russeting
defect, and the defect became more distinct as the spatial
frequency increased, except for some intermediate fre-
quencies (i.e., 0.1e0.2 cycles mm
1) which had relatively low
contrast. The spatial frequencies above 0.2 cycles mm
1
seemed to be more appropriate for detecting the defect (on a
side note, attention should be paid to the enhanced non-
defect surface features and decreased image SNR in using
high-frequency patterns). Apart from skin russeting, one also
discerned another circular defect in the AC images at spatial
frequencies of 0.03e0.25 cycles mm
1, which was due to slight
subsurface tissue damage, probably resulting from finger
pressure during apple picking.
The above findings suggest that while AC images are
effective in enhancing certain features at specific spatial fre-
quencies, they are not always superior over the DC image in
fruit defect detection. The actual performance of AC images
would depend on a number of factors, including defect type,
and inherent optical contrast, fruit variety, spatial frequency
of light patterns (Li, 2016; Lu & Lu, 2018), and the wavelength
selected. In essence, it depends on whether adequate image
resolution and contrast can be obtained at the probing depth
where the defect of interest is located. Although AC images
have been proven effective in detecting subsurface bruising
and skin russeting, their effectiveness or potential in detecting
other subsurface and/or surface defects, remains to be fully
explored. Moreover, in high-frequency AC images, surface
lenticels become more apparent, accompanied with a signifi-
cantly diminished image SNR, which may pose more chal-
lenges for accurate defect segmentation and detection.
5. Conclusions
This paper presents a theoretical and experimented analysis
of the fundamental features of SIRI, i.e., image resolution,
contrast and depth-resolving. We have proposed an image
formation model and a general demodulation methodology
for SIRI, and performed a set of experiments to validate its
features. The imaging process of SIRI has a low-pass filtering
property, which, together with the sample absorption and
scattering, account for the contrast and resolution degrada-
tion of resulting images. As the hallmark of SIRI, AC images
exhibit enhanced image resolution and contrast, and also
have a depth-resolving ability, compared to DC images that
correspond to conventional UI based imaging. The resolution
and contrast enhancement can be explained by down-
modulating of high-frequency sample information in the
Fourier domain and suppressed light scattering in the sample,
but they are convolved with a decreased detection depth as
the spatial frequency of illumination patterns increases. The
effectiveness of AC images for defect detection of apples
depend on a number of factors, including defect type, fruit
variety and spatial frequency. Without using inverse compu-
tations, SIRI basically samples a superficial region of fruit
samples. The technique can thus serve as a subsurface-
surface imaging technique for (surface and subsurface)
defect detection of thin-skinned fruits, such as apple and
peach. Further work should be done on exploring its full po-
tential in fruit defect detection.
references
Anderson, E. R., Cuccia, D. J., & Durkin, A. J. (2007). Detection of
bruises on Golden Delicious apples using spatial frequency-
domain imaging. In Paper presented at the advanced biomedical
and clinical diagnostic systems V, San Jose, CA, USA.
Bashkatov, A. N., Genina, E. A., Kochubey, V. I., & Tuchin, V. V.
(2005). Optical properties of human skin, subcutaneous and
mucous tissues in the wavelength range from 400 to 2000 nm.
Journal of Physics D Applied Physics, 38(12), 2543e2555.
Bertero, M., & Boccacci, P. (1998). Introduction to inverse problems in
imaging. Bristol, UK: Institute of Physics Publishing.
Bex, P. J., & Makous, W. (2002). Spatial frequency, phase, and the
contrast of natural images. Journal of the Optical Society of
America A, 19(6), 1096e1106.
Bigio, I. J., & Fantini, S. (2016). Quantitative biomedical optics: Theory,
methods, and applications. Cambridge, UK: Cambridge
University Press.
Binzoni, T., Vogel, A., Gandjbakhche, A. H., & Marchesini, R.
(2008). Detection limits of multi-spectral optical imaging
under skin surface. Physics in Medicine and Biology, 53(3),
617e636.
Boreman, G. D. (2001). Modulation transfer function in optical and
electro-optical systems. Bellingham, WA: SPIE.
Brook, P. J. (1968). Pre-havest lenticel spot of apples. New Zealand
Journal of Agricultural Research, 11(2), 237e244.
Cuccia, D. J., Bevilacqua, F., Durkin, A. J., Ayers, F. R., &
Tromberg, B. J. (2009). Quantitation and mapping of tissue
optical properties using modulated imaging. Journal of
Biomedical Optics, 14(2), 024012.
Cuccia, D. J., Bevilacqua, F., Durkin, A. J., & Tromberg, B. J. (2005).
Modulated imaging: Quantitative analysis and tomography of
turbid media in the spatial-frequency domain. Optics Letters,
30(11), 1354e1356.
D'Andrea, C., Ducros, N., Bassi, A., Arridge, S., & Valentini, G.
(2010). Fast 3D optical reconstruction in turbid media using
spatially modulated light. Biomedical Optics Express, 1(2),
471e481.
Dan, D., Yao, B., & Lei, M. (2014). Structured illumination
microscopy for super-resolution and optical sectioning.
Chinese Science Bulletin, 59(12), 1291e1307.
Dognitz, N., & Wagnieres, G. (1998). Determination of tissue
optical properties by steady-state spatial frequency-domain
refelectometry. Lasers in Medical Science, 13(1), 55e65.
Dong, S., Nanda, P., Shiradkar, R., Guo, K., & Zheng, G. (2014).
High-resolution fluorescence imaging via pattern-illuminated
Fourier ptychography. Optics Express, 22(17), 20856e20870.
Faust, M., & Shear, C. B. (1972). Russeting of apples, an
interpretive review. HortScience, 7, 233e235.
Gao, P., Pedrini, G., & Osten, W. (2013). Structured illumination for
resolution enhancement and autofocusing in digital
holographic microscopy. Optics Letters, 38(8), 1328e1330.
Gigan, S. (2017). Optical microscopy aims deep. Nature Photonics,
11, 14e16.
Goodman, J. W. (2005). Introduction to fourier optics. Greenwood
Village, CO: Roberts & Company.
Gustafsson, M. (2000). Surpassing the lateral resolution limit by a
factor of two using structured illumination microscopy.
Journal of microscopy, 198(2), 82e87.
Gustafsson, M. (2005). Nonlinear structured-illumination
microscopy: Wide-field fluorescence imaging with
 b i o s y s t e m s e n g i n e e r i n g 1 8 0 ( 2 0 1 9 ) 1 e1 5
theoretically unlimited resolution. Proceedings of the National
Academy of Sciences, 102(37), 13081e13086.
Hayakawa, C. K., Karrobi, K., Pera, V., Roblyer, D., &
Venugopalan, V. (2018). Optical sampling depth in the spatial
frequency domain. Journal of Biomedical Optics, 24(7), 071603.
Hecht, E. (2002). Optics. Reading, MA: Addison-Wesley.
Khanal, B. P., Shrestha, R., Huckstadt, L., & Knoche, M. (2013).
Russeting in apple seems unrelated to the mechanical
properties of the cuticle at maturity. HortScience, 48(9),
1135e1138.
Konecky, S. D. (2012). Diffuse optical tomography: Imaging
physics and image formation principles. In M. A. Anastasio, &
P. La Riviere (Eds.), Emgerging imaging technologies in medicine
(pp. 123e138). Boca Raton, FL: CRC Press.
Konecky, S. D., Mazhar, A., Cuccia, D., Durkin, A. J.,
Schotland, J. C., & Tromberg, B. J. (2009). Quantitative optical
tomography of sub-surface heterogeneities using spatially
modulated structured light. Optics Express, 17(17),
14780e14790.
Li, R. (2016). Development of a strutured illumination reflectance
imaging system for enhanced defection of subsurface and surface
defects in apple fruit. USA: Master, Michigan State University.
Li, R., Lu, Y., & Lu, R. (2018). Structured illumination reflectance
imaging for enhanced detection of subsurface tissue bruising
in apples. Transactions of the ASABE, 61(3), 809e819.
Lorente, D., Aleixos, N., Go  mez-Sanchis, J., Cubero, S., Garcı́a-
Navarrete, & Blasco, J. (2012). Recent advances and
applications of hyperspectral imaging for fruit and vegetable
quality assessment. Food and Bioprocess Technology, 5(4),
1121e1142.
Lukic, V., Markel, V. A., & Schotland, J. C. (2009). Optical
tomography with structured illumination. Optics Letters, 34(7),
983e985.
Lu, Y., Li, R., & Lu, R. (2016a). Fast demodulation of pattern images
by spiral phase transform in structured-illumination
reflectance imaging for detection of bruises in apples.
Computers and Electronics in Agriculture, 127, 652e658.
Lu, Y., Li, R., & Lu, R. (2016b). Gram-Schmidt orthonormalization
for retrival of amplitude images under sinisoidal patterns of
illumination. Applied Optics, 55(25), 6866e6873.
Lu, Y., Li, R., & Lu, R. (2016c). Structured-illumination reflectance
imaging (SIRI) for enhanced detection of fresh bruises in
apples. Postharvest Biology and Technology, 117, 89e93.
Lu, Y., & Lu, R. (2017a). Development of a multispectral
structured-illumination reflectance imaging (SIRI) system and
its application to bruise detection of apples. Transactions of the
ASABE, 60(4), 1379e1389.
Lu, Y., & Lu, R. (2017b). Non-destructuctive defect detection of
apples by spectroscopic and imaging technologies: A review.
Transactions of the ASABE, 60(5), 1765e1790.
Lu, Y., & Lu, R. (2017c). Using composite sinusoidal patterns in
structured-illumination reflectance imaging (SIRI) for
View publication stats
15
enhanced detection of apple bruise. Journal of Food Engineering,
199, 54e64.
Lu, Y., & Lu, R. (2018). Detection of surface and subsurface defects
of applels using structured-illumination reflectance imaging
with machine learning algorithms. Transactions of the ASABE,
61(6), 1831e1842.
Mazhar, A., Dell, S., Cuccia, D., Gioux, S., Durkin, A. J.,
Frangioni, J. V., et al. (2010). Wavelength optimization for rapid
chromophore mapping using spatial frequency domain
imaging. Journal of Biomedical Optics, 15(6), 061716.
Mosk, A. P., Lagendijk, A., Lerosey, G., & Fink, M. (2012).
Controlling waves in space and time for imaging and focusing
in complex media. Nature Photonics, 6, 283e292.
Neil, M. A. A., Juskaitis, R., & Wilson, T. (1997). Method of
obtaining optical sectioning by using structured light in a
conventional microscope. Optics Letters, 22(24), 1905e1907.
Ntziachristos, V. (2010). Going deeper than microscopy: The optical
imaging frontier in biology. Nature Methods, 7(8), 603e614.
Peli, E. (1990). Contrast in complex images. Journal of the Optical
Society of America A, 7(10), 2032e2040.
Qin, J., Chao, K., Kim, M. S., Lu, R., & Burks, T. F. (2013).
Hyperspectral and multispectral imaging for evaluating food
safety and quality. Journal of Food Engineering, 118(2), 157e171.
Saleh, B. (2011). Introduction to subsurface imaging. New York,
NY,USA: Cambridge Uiversity Press.
Shi, L., & Alfano, R. (2017). Deep imaging in tissues and biomedical
materials using linear and nonlinear optical methods. Singapore:
Pan Stanford Publishing Pte. Ltd.
Smith, N. B., & Webb, A. (2010). Introduction to medical imaging:
Physics, engineering and clinical applications. New York, NY, USA:
Cambridge University Press.
Thomas, B., Momany, M., & Kner, P. (2013). Optical sectioning
structured illumination microscopy with enhanced
sensitivity. Journal of Optics, 15(9), 094004.
Thomas, B., Wolstenholme, A., Chaudhari, S. N., Kipreos, E. T., &
Kner, P. (2015). Enhanced resolution through thick tissue with
structured illumination and adaptive optics. Journal of
Biomedical Optics, 20(20), 026006.
Tuchin, V. (2015). Tissue optics: Light scattering methods and
instruments for medical diagnostics. Bellingham, WA, USA: SPIE
Press.
Wang, L., Ho, P. P., Liu, C., Zhang, G., & Alfano, R. R. (1991).
Ballistic 2-D imaging through scattering walls using an
ultrafast optical Kerr gate. Science, 253(5021), 769e771.
Weber, J. R., Cuccia, D. J., Durkin, A. J., & Tromberg, B. J. (2009).
Noncontact imaging of absorption and scattering in layered
tissue using spatially modulated structured light. Journal of
Applied Physics, 105(10), 102028.
Wilson, B. C., & Jacques, S. L. (1990). Optical reflectance and
transmittance of tissues: Principles and applications. IEEE
Journal of Quantum Electronics, 26(12), 2186e2199.