This article has been accepted for inclusion in a future issue of this journal.

Content is final as presented, with the exception of pagination.

IEEEJOURNALOFSELECTEDTOPICSINAPPLIEDEARTHOBSERVATIONSANDREMOTESENSING

ShadowDetectionandRemovalforOccluded
ObjectInformationRecoveryinUrbanHigh
ResolutionPanchromaticSatelliteImages
NanSu,StudentMember,IEEE,YeZhang,ShuTian,StudentMember,IEEE,YimingYan,andXinyuanMiao
objectsbythelargeshadow.In

AbstractThe existence of shadows in very highresolutionthat case, the objects in the

panchromaticsatelliteimagescanoccludesomeobjectstocausethe shadowregionsaredifficultto
reductionorlossoftheirinformation,particularlyinurbanscenes.To
beextractedforfurtherapplica
recover the occluded information of objects, shadow removal is a
tions. Therefore, in order to
significant processing procedure for the image interpretation and
application.Inthispaper,weproposeanovelframeworkofshadowrestore obscured objects,
detectionandremovalforpanchromaticsatelliteimagestorestorethe shadow detection and shadow
obscured object information. In shadow detection, we present anremoval is an essential
automaticsoftshadowdetectionmethodbythecombinedapplication preprocessing step of urban
ofabimodalhistogramsplittingmethodandimagemattingtechnique. highresolution remote sensing
Soft detection results can exhibit both umbra areas and penumbra images. Many effective
areas to describe the shadow distribution precisely. In shadow algorithms of shadow removal
removal, we propose a spatial adaptive nonlocal sparse shadow
havebeenproposedfornatural
removalmethodtooperateattwolevels.Fortheinitialstep,weapply
images or remote sensing
thelinecorrectionmethodtoenhanceshadowareasroughlyinglobal.
In the refined process, we study the characteristics of objects and multispectralimages.However,
shadows,andanalyzespatialrelationshipbetweenthem.Thesecond thereisagreatlackofshadow
linearradiometriccorrectionandnonlocalsparsemodelareusedto removal method for
simultaneouslycontrolthebrightnessandsmoothnessoftherecovered panchromatic imagery, while
shadowareastobethesameasthecorrespondingnonshadowareas the panchromatic images
basedongroupmatrixwithsimilarpatches.Ourmethodcanrestore usuallycanprovidemorehigh
the uniform objects in the shadow areas. Highresolution resolution to be useful for the
panchromaticshadowimagesofdifferentcasesanddifferentsatellites
application of objects in the
areprocessedbytheproposedmethod.Theexperimentalresultsverify
satellite sensors. For the
the effectiveness and superiority of the proposed method by
purpose of the information
comparingthreeothermethods.

recovery of obscured objects,

IndexTermsAutomaticsoftshadowdetection,obscuredinforwe intend to analyze the

mationofobjects,panchromaticsatelliteimages,spatialadaptive characteristics of shadows and
nonlocalsparse(SANS)shadowremoval.

objects in the panchromatic

images of urban areas and
I.INTRODUCTION
remove the shadows to obtain
ATELLITEimagingfortheobservationofEarthhasabil shadowfreeimages.
Many current researches
itytoobtainveryhighresolutiondata,rangedfrom0.5to 2indicate that shadow detection
m in the panchromatic band. Indeed, highresolution images istheindispensablestepinthe
exhibit more detail information to increase the objectoriented complete processing chain of
application potential, e.g., a building accurate location, detailed
featureextraction,and3Dreconstruction[1],[2].Unfortunately, shadow removal. Popular
most highresolution satellite images contain shadows as un methods of shadow detection
desired information in urban areas, which strongly affect the canbefocusedontheshadow
interpretationofsatelliteimages.Theshadowscausethepartialor
property, such as shadow
totallossradianceinformation,particularlythatofoccluded
regionshavinglowerintensity,
ManuscriptreceivedSeptember30,2015;revisedApril6,2016;accepted higherhue,andsaturation.For
May7,2016.ThisworkwassupportedbytheNationalNaturalScienceFoun instance, shadow areas are
dationofChinaunderGrant61471148.(Correspondingauthor:NanSu.)
TheauthorsarewiththeDepartmentofInformationEngineering,HarbinIndetermined by the threshold
stituteofTechnology,Harbin150001,China(email:wlw01@163.com;zhye@ method with bimodal distribu
hit.edu.cn;tianshu_0202@126.com;yanyiming@hrbeu.edu.cn;1176868698@tion of image histogram [3].
qq.com).
Colorversionsofoneormoreofthefiguresinthispaperareavailableonline Moreover, a lot of literature
athttp://ieeexplore.ieee.org.
works show that the threshold
DigitalObjectIdentifier10.1109/JSTARS.2016.2570234
detection results are more

accuracy using nearinfrared

band for the multispectral
images [4], [5]. Based on the
bimodal thresholding method,
several shadow detection
algorithms are proposed.
Mohamed and Khan
Iftekharuddin [6] employs
morphological filtering to rule
out small wrong shadow
regions,and[7]carriesoutre
segmentation with multilevel
threshold to refine detected
results. For the panchromatic
imagery, a few researches are
done, which also employ the
threshold method to segment
shadowsasthemainstep[8].In
all the mentioned methods
above, shadows are detected
absolutely by a binary mask
that cannot describe shadows
accurately, because of the
complex scenes and the
existence of penumbras in the
highresolutionsatelliteimages
of urban areas. In a recent
study, soft shadow detection
has been proposed in some
literature works for solving
nonuniform shadow problems.
Lietal.[9]andGuoetal.[10]
bothemploytheimagematting
techniqueofLevin[12],which
calculates the shadow
coefficients to measure the
shadow probability of each
pixel. But they carry out the
original matting method to
mark shadow and nonshadow
bymanualscribblesandignore
the characteristics of the
shadows. In this paper, we
propose a novel method to
combinethethresholdmethod
andtheimagemattingmethod
and to obtain the soft shadow
detectionresultsautomatically.

Theshadowremovalisthe
coreofthispaper.Inorderto

recovertheinformationofshadowareas,therearethreemajor
methodsbyimageenhancementtechnique:1)gamma
193914042016IEEE.Personaluseispermitted,but
republication/redistributionrequiresIEEEpermission.See

http://www.ieee.org/publicati
ons

standards/publications/rights/i information.
ndex.htmlformore

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

IEEEJOURNALOFSELECTEDTOPICSINAPPLIEDEARTHOBSERVATIONSANDREMOTESENSING

technique. The detection

mainly contains three steps.
correction, 2) histogram matching, and 3) linear correlation [5],First, the shadow image is
[14][17].Inarelatedstudy,themethodsareanalyzedtoobtaina classified to shadow and
smoothshadowfreeimage.In[5],shadowareasarerestoredbya nonshadowareas
piecewise linear function correlation with different ratio pa
rametersaccordingtodifferentdarknesslevels.Tobettersolvethe
problem of nonuniform shadow, Li et al. [9] presents to use a
spatiallyadaptivenonlocalregularizedoperatorstoconstrainthe
shadowscaleandupdateshadowfreeimage.Inthesimilarway,
using linear radiometric correction and polynomial fitting to
restoretheshadowarea,Zhang et al. [8] analyze theinnerand
outer of shadow boundary to calculate the parameters for the
boundarypostprocessing.Guoetal.[10]alsoproposetheideaof
pair wise regions thatare the same material with differentillu
mination.Asidefromthesemethods,forthemultispectralsatellite
images,Songetal.[11]presentanexamplebasedlearningmethod
for shadow removal via examples training of shadow and
nonshadow, which models the relationship of them based on
Markovrandomfield.However,afewworksareinvolvedinthe
shadow removal processing in highresolution panchromatic
imagery, particularly for compound shadows in urban scenes,
whichisstillachallengingproblem.Andalmostnoonecaresthe
restorationoftheobjectsinshadowareas.

Inthispaper,therearethreemaincontributions.First,we
buildanovelautomaticshadowdetectionandremovalframe
work for the highresolution satellite panchromatic images.
Second, a bimodal histogram splitting method and image
matting technique are both used to automatically obtain soft
shadowdetectionresults.Finallyandmostimportantly,dueto
theshadowremovalbylinearcorrectionleadingtobenoisierin
the shadow areas, we propose the spatial adaptive nonlocal
sparse(SANS)methodtocombinethenonlocalsparsemodel
andthesecondlinearradiometriccorrectiontosimultaneously
controlthebrightnessandsmoothnessoftherecoveredshadow
areas.Thenonlocalsparsemodelmainlysolvesnoisyproblem
andthesecondlinearcorrectioncanreachthebrightnessunifor
mity.Inparticular,forthetwocasesoftheshadowareasinthis
paper,weimprovetheiterativesingularvaluethresholdofthe
nonlocalsparsemodelforcase1andusetheoriginalthreshold
forcase2.Theproposedmethodcanrestoretheobjectsinthe
shadowareastobethesameimagequalityasthecorresponding
nonshadowareas.
Theremainderofthispaperisorganizedasfollows.Section
IIshowsthewholeflowoftheproposedmethod.InSectionIII,
theautomaticsoftshadowdetectionalgorithmisdescribed.We
introduce the proposed SANS shadow removal algorithm in
detail in Section IV. The experimental results are shown by
comparing three other methods in Section V, and the
conclusioninSectionVI.
II.PROPOSEDMETHOD
Theproposedframeworkinvolvesthetwomainprocedures
asshowninFig.1:shadowdetectionandshadowremoval.In
thedetectionstep,wepresentanautomaticsoftshadowdetec
tionmethodbybimodalhistogramsplittingandimagematting

roughly by hard threshold.

Thehardmapofbinarymask
cannot provide the precise
edgesbetweenthetwoareas,
due to the presence of
penumbra. Second, the
shadowareasareerodedand
dilated by morphological
operators and the difference
middle areas are filled with
theoriginalimage.Third,we
employ the image matting
method to calculate the
shadow coefficient for each
pixel based on the mask
image.0isshadow,1isnon
shadow, and the penumbra
areaisfrom0to1toindicate
theshadowprobability.
In the removal step, we
propose a SANS shadow
removal method. Our removal
method is divided into two
levels:theinitialremovaland
the refined removal. In the
initial removal level, the line
correctionmethodisemployed
to enhance the shadow area.
The initial result is aimed to
make the same object across
shadow and nonshadow areas
belong roughly to the same
category. In the refined
process,weanalyzethespatial
relationship between objects
andshadows,anddigintotheir
characteristics. The obscured
information of objects in the
shadow areas is recovered
preciselybytheunitofpatches
based on the group matrix.
Eachgroupconsistsofnonlocal
patches with similar structure
from shadow areas and
nonshadow areas. The second
linear radiometric correction
and nonlocal sparse model are
used to simultaneously control
the brightness and smoothness
of the recovered areas by the
correspondingnonshadowareas
with similar structure in the
same group matrix. Therefore,
ourSANSremovalmethodcan
obtain the recovered objects
across shadow and nonshadow

areastobethesameimagequality.Inthefollowingsections,the only partly blocked. The

detailofthealgorithmisprovided.
penumbra is obvious in very
highresolution images. In
additiontoshadowareas,other
III.SHADOWDETECTION
areasunderthedirectlightare
Shadowsareformedbecause ofsomethingblockingthelight
called nonshadow areas or
source, as shown in Fig. 2 [8], [18], [19]. The phenomenon
sunlitareasinthispaper.Inour
frequently occurs in dense urban areas. Solving the shadow
work, we mainly research the
problemisveryimportantforurbanobjectapplicationsinthehigh
cast shadow areas, and
resolutionsatelliteimages.Shadowscanbeclassifiedsimplyas
distinguish the umbra and the
theselfshadowandthecastshadow.Theselfshadowisformed
penumbra in the shadow
whenafaceoftheobjectisnotilluminateddirectlybythelight
detection.
source.Thecastshadowisformedwhensomeobjectsblockthe
lightsourcetofallonotherobjects.Andthecastshadowisoftwo
A.HardShadowDetection
types: the umbra and the penumbra. The umbra is not directly
illuminatedbythelightsourcecompletely,whilethepenumbrais For satellite multispectral

images, the accurate shadow

detection is obtained by
spectral features. For high
resolution

satellite
panchromatic images, the
principal characteristic of the
shadow is low intensity to
occupy the lower end of the
histogram.In[3]and[5],both
the histograms of PAN band
imagesfromQBandIKONOS
satellites are shown, and the
histograms show an obvious
bimodal distribution. The
bimodalhistogramsplitting

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
SUetal.:SHADOWDETECTIONANDREMOVALFOROCCLUDEDOBJECTINFORMATIONRECOVERYINURBANHIGHRESOLUTION

Fig.1.Flowchartoftheproposedmethod.

Fig.2.Formationoftheshadow.

other bands in the visual

spectrum, has the ability to
method,whichchoosesthemeanofthetwopeaksasthethreshold distinguishtheshadowareasby
value,isemployedtoobtaintheinitialshadowdetectionresultsin properthreshold,
bothpapers.Songetal.[11]comparetheratiosofnonshadowand
shadow pixel values in each band of QB and WV2 images to
concludethattheNIRbandimageshavethestrongestabilityto
distinguish shadow areas in the experiments of single band.
Panchromaticband,whichisapproximatedbytheaverageofall

although the shadow detection

results in only panchromatic
band are slightly worse than
thoseinonlyNIRband.Inthis
paper,thethresholdmethodof
bimodal histogram splitting is
selectedfortworeasons.First,
harddetectionisonlyaninitial

detectiontolocatetothecorrespondingshadowareasinthisstep. soft shadow detection step.

The detailed detection of boundary areas between shadow and Second, our main aim is to
nonshadowareas,namelythepenumbraareas,isconductedbythe detect large shadow areas,

whichcanobscureothersmall

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

IEEEJOURNALOFSELECTEDTOPICSINAPPLIEDEARTHOBSERVATIONSANDREMOTESENSING

C.SoftShadowDetection

buildings.Large shadowareas including alargenumberoflow

For the soft detection, we
grayvaluepixelsaresimplydistinguishedbyasuitablethreshold.
apply an image matting
Inourwork,lowerthanthemeanofthetwopeaksisadoptedas
technique[9],[10]toderivethe
thethresholdfromthehistogramoftheoriginalshadowimage,as
probabilityvalueofeachpixel
shownin(1),promisingthatourpreliminaryharddetectionresults
belongingtotheshadow.That
belongtoshadowcompletely
means, 0 and 1 represent the
Bm = hard
I , 2 (Pl + Pr )
pixelbelongingtoshadowand
1
nonshadow, respectively, and
the value of between 0 and 1
whichdenotestheoperatorofhardthresholding;Bmisthebinarymeans the pixel belonging to
mask result of 1hard shadow detection; I stands for the original
shadowimage; 2 (Pl + Pr ) isourselectedthresholdvalue;the penumbra in the soft
andPlandPrrepresentthepeaksofthehistogram. isempiricallyshadow map. The original
setto10inourexperiments.
matting method need that the
original image is marked by
B.MorphologicalOperationandMask
scribbles, where white and
Afterthehardshadowdetection,abinarymaskimageisob
black scribbles represent
tained,whichdistinguishesshadowandnonshadowabsolutely.In
absolute shadow and
actual, illumination often changes gradually resulting in the
nonshadow

regions,
existence of penumbra between shadow and nonshadow. The
respectively. In our study, the
penumbraisthetransitionregionfromshadowtononshadow,that
twostepsabovementionedtake
isnotexactlyapartofshadowornonshadow.Forrecoveringthe
theplaceofscribblestorealize
shadowfreeimageprecisely later,the penumbrais notignored.
the automatic detection
Especially in the high resolution satellite images, the width of
algorithm.
penumbramightreach2pixelsormore,whichcancausestrong
Given an original shadow
boundaryeffectsinrecoveryusingabinarymask.Inthissection,
image, image matting can
the morphological operation is employed to get the suspected
separate the foreground
penumbraforfollowingwork.Thebinarymaskimageisdilated
image from background
anderoded,andthedifferenceregionsbetweenthetworesultsare
image, representing shadow,
filledwiththecorrespondingoriginalimage.Thefinalmaskimage and nonshadow, respectively.
islikeFig.1(3)shownatthebottomofthepage.
Based on [12], the corre

sponding foreground and

background RGB values are
Dif f [x, y] = d(Bm[x, y]) e(Bm[x, y])
assumed to be a linear
(2)
combination. For the
whereDiffstandsforthedifferenceimagebetweendilationandpanchromatic images, this
erosion; d and e representdilateanderodeoperation,respec methodcanalsoworkwellfor
tively;Mrepresentsthefinalmaskimage;DNisthegrayvalueonlythegrayvalue.Itcanbe
oftheoriginalimage.
expressedas

Inthissection,weemploythemorphologicaloperationswitha
large structuring element for additional useful to remove some
I [x, y] = [x,
smallshadowareasbasedontheharddetectionresults.Forthis
y]S[x, y] + (1
procession,wehavetwomainconsiderations.First,theexistence
[x, y])N [x, y] (4)
ofmanysmalldiscontinuousshadowareascausedbynoiseisnot
trueshadowinhighresolutionimages.Second,ourmainaimisto whereI[x, y]istheintensity
restore the occluded buildings by shadow removal. Only large oftheoriginalimageat[x, y];
SandNrepresenttheshadow
shadowscanoccludebuildings,whilesmallshadowscannotaffect and the nonshadow (the
the
the interpretation and extraction of buildings in our work. foreground and
background),

respectively.
Therefore,smallshadowsaredeletedbymorphologicalfiltering.

Levin et al. [12] carries out

the matting algorithm to

calculate the shadow
probabilityforeachpixelby
minimizing the energy
function based on the
followingformulation:
T

E() =
whereisthepredicted
shadowprobabilitybythe
matting

method;isourmarklabelaccordingtothe
finalmaskimage,

thatmeans= 0isshadowareaand= 1is

nonshadowarea;

LismattingLaplacianmatrix;
and D is a diagonal matrix
withD(i,i)=1.
IV.
SHADOW
REMOVAL
Shadow removal to restore
theoccludedinformationofob
jects is the core in this paper.
The proposed removal method
including two levels is the
initialcorrectionandtherefined
restoration. The initial
correctionstepcompensatesthe
intensity for shadow area by
global parameter, aiming to
make the same object across
shadow area and nonshadow
areabelongroughlytothesame
category. The refined
restoration step utilizes non
local similarity and local
sparsityoftheinitialcorrected
image to restore the shadow
areas,makingtheobscuredpart
ofobjectsintheshadowareaof
the same brightness and
smoothness level as the other
partinthenonshadowarea.

0,

M [x, y] =

1,

DN,

if d(Bm[x, y]) = 0 and e(Bm[x, y]) = 0

if

d(Bm[x, y]) = 1

if

D[x, y] = 1

and

e(Bm[x, y]) = 1

(3)

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
SUetal.:SHADOWDETECTIONANDREMOVALFOROCCLUDEDOBJECTINFORMATIONRECOVERYINURBANHIGHRESOLUTION

used to recover the central

shadow patch. In this section,
After the soft shadow detection, initial shadow correction is we propose SANS method to
performed. In fact, the relationship between the intensity ofrecover the smoothness and
shadow and nonshadow areas is not exactly of the linear type. brightness in detail
However,weneedtobuilditforthesakeofgettingasimplebut simultaneously, which can
satisfactoryremovalmodeltoobtaintheinitialshadowremoval reach uniform image quality
results.Asdoneintherelatedliteratureofshadowremoval[4], betweentherecoveredshadow
[5],[8],theyallchooselinearcorrelationcorrectionandprovethe areasandthenonshadowareas.
methodtobeeffectiveforrestoringthebrightnessofshadow.And 1) Build Nonlocal Sparse
inalloftheabovementionedliteraturetheyempiricallyobserve Model in Shadow Removal:
theshadowandnonshadowareasfollowingGaussiandistribution. After the first correction, the
Becausethedistributionofnonshadowandshadowareasarethe initial predicted shadowfree
leftandrightofthethresholdvalueinthehistogram,respectively. image may be still obviously
We can regard as two Gaussian distributions according to the nonuniform. In particular for
bimodal curve. Therefore, in our method we also assume two the obscured part of object
conditions:1)therelationshipbetweentheintensityoftheshadowinformation,thebrightnessand
andthatofcorrespondingnonshadowisofthelineartype,and2) the smoothness are very
the intensity of shadow and nonshadow follow a Gaussian different from those of the
distribution. Based on the two hypotheses, we employ linear nonshadow area. In order to
correlation method in our initial shadow correction step, which ensure the whole recovered
usesthemeanvalueandstandarddeviationofallthesunlitareas objectswiththesameproperties
tocompensatetheintensityofshadowareasinthewholeimage. in greater detail, the nonlocal
Further,integratingtheshadowprobabilityofoursoftdetection sparse model is applied to
results, the corrected gray value of the preliminary predicted restore the shadow image
precisely [21], [22]. In our
shadowfreeimagecanbeexpressedas
SANS method, each unit of
shadowareaisfurtherrestored
In [x, y] = (1 [x, y])Is [x, y] + [x, y]
by using corresponding
(6) k s
su n lit
nonshadow area with

(Is [x, y] sh a d ow ) + su n lit

similarstructure.
sh a d ow
where In istheintensityofshadowaftertheinitialcorrection; Is isthe Weattempttorepresent the
intensityofshadowintheoriginalshadowimage;and istheshadow
probabilityofoursoftdetection.
shadow image by the sparse
sh a d ow , su n lit , sh a d ow , and su n lit denotethemeanvalue
andstandarddeviationofshadowareasandnonshadowareas. codinginthetraditionalpatch
space. But in the traditional
sparse representation, the
B.RefinedRestoration
patchesareindependent,which
Shadow areas in singleband panchromatic images show the cannot build the relationship
maincharacteristicsthattheradiationinformationislostandthe between the shadow area and
brightnessislow,namelyradiometricdegradation.Afterthelinear the corresponding nonshadow
radiation enhancement in the initial removal step, the low gray area of the same objects [23].
valueintervalofshadowareasisdrawntomoregrayscalerange For the shadow patches being
leadingtomorenoise.Forthisreason,theinitialrestoredshadowrestored by using the similar
areas are both spatial and radiometric degradation, which are nonshadow patches, the
noisyandnonuniform.Intherefinedrestorationstep,weaimto structure similarity among the
recover the shadow area in detail with the information of thepatches needs to be better
correspondingnonshadowareabytheunitofpatch.Therefore,we exploited. In our SANS
need build the relationship between the shadow areas and the method,thebasicunitofsparse
corresponding nonshadow areas. However, the neighbors of the representation for the shadow
shadowpatchesinthelocalareaarelikelystilltobetheshadow imageisgroup[24].Eachpatch
patches, which cannot provide useful information to restore the extractedfromshadowregions
shadow area. Nonlocal model can take advantage of patches ismatchedwithasetofnonlo
arbitrarilyfarawayfromthecentralpatchinthehighresolution cal patches having similar
panchromatic satellite images. The similar structural nonshadow structure. Stack all the similar
patchescanbebetter
patchesintheformofamatrix
toconstructagroup.Thisgroup
matrix simultaneouslycontains

A.InitialShadowRemoval

the shadow and nonshadow

patches to establish their
connectionspatialadaptively.
AsisshowninFig.3,for
eachcentralpatchinshadow
area
(seeredpatchinFig.3),with

size ps ps,wesearch

its t bestmatchedpatchesinthe
nonlocal space, which may
conclude shadow patches and
nonshadow patches (see green
patches and yellow patches in
Fig. 3). iG k is named as group
withsizepstcontainingallthe
matched patches with similar
structures.Fig.3showsthateach
patch ik s is represented as a
vector, and the group iG k is
integratedtoamatrix.Analogous
tothesparsecodinginthepatch
space,thegroupiG kisexpressed
as iG k = DG k G k by a
dictionary DG k and sparse
vectors G k .Accordingtothis
concept of group, the whole
shadow image can be expressed
bystructuredsparselycodeas(7)

I = DG G =

n
R

k=1

where DG and G denote the

concatenationofallDGkandG k ,
respectively; RG k is actually an
operatorthatextractsthe groupiG k
from shadow image I, that means
iGk=RGk(I);anditstransposeis
T
denotedbyRG k;nisthenumber
of groups, and it is obvious that
eachpatchcorrespondstoagroup.
Like traditional sparse algorithm,
our nonlocal sparse model can be
formulated into the following
minimizationproblem[25],[26]:

(D

) = arg
min

Gk

D
G

1,2

i,
G

1,2

i= 1

+ i,

2
2

+ i,m

(8)

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

IEEEJOURNALOFSELECTEDTOPICSINAPPLIEDEARTHOBSERVATIONSANDREMOTESENSING

Fig.3.Constructionofthegroupmatrix.

(regularizationparameter),whichisappliedtothesingular
whichconsidersthegroupsparsitydefinedbyapseudoma
norm
,
G p ,q
setp=1, valuethresholdingtorestoretheshadowareagroupmatrixby
trix
andin ourmethodology
q=2.ThegroupsparsityregularizerG 1,2
isthesum SVD.
ofstandarddeviationsassociatedwithsparsecoefficientvector.
Givenshadowimage,theempiricaldistributionofsingular

= [1 , 2 , . . . , n ] isthesparsevector,thesparsecoeffi valuescanbemodeledbyaLaplacianwithchangingvariances.
k

= [

,...,

] arethe ith rowofmatrix

, Withtheprior,followingthumbrulethesingularvaluesthresh

i,1 , i,2

cientsi

i,m

andm=t+ 1denotetmatchedpatchesofashadowpatchik s.

oldparametercanbesettoi= 2 2/i.[23],wherei

Inthispaper,weapplySVDtoobtaintheselfadaptivelearning representsthelocallyshadowareaestimatedvarianceandis
dictionaryforeachgrouptoformastandardlowrankapproxi agiveninitialvaluetodenotetheglobalvariance.
mationproblemas[27],ratherthanagivendictionaryDforthe Incase1,firsttomakethesameobjectacrosstheshadowand
entireimageinthetraditionalsparsealgorithm
nonshadowareasreachthesamebrightnesslevel,thebrightness
oftheshadowpatchescanbecorrectedonceagainbytheother
K
nonshadowpatcheswithsimilarstructureinthegroupmatrix
G
(D

,V

)=
D

Gk

Gk

V
k

Gk

arg min

2
2

+
k =1

nonshad
ow

(9)

= (1 )iG

k s

(i

)
g s h a d o w

g s h a d o
(D

,V

) = SV D(i

= S (

Gk

w
(10)

+ g

n o ns ha d ow

(11)

whereGk= diag{1,2, . . . ,K}(K= min(m, n))isadiagonal shadowpatchesincase2,while

matrix;andSrepresentsthesoftthresholdingoper
the matrix contains both
ation.ThepredictedshadowfreeimageisobtainedbyIG=

shadowpatchesandnonshadow
DG G VG .
patchesincase1.
2)SmoothnessandBrightnessRestoredSimultaneously:
In[23],theiterativesingular
Therefinedrestorationconsistsoftwoparts:thebrightnessandthe
value regularization from the
smoothness reach uniform image quality between the restored
spatiallyadaptiveperspectiveis
shadowareasandthenonshadowareas.Inthissection,wesolve
employedtoattainthethreshold
theoptimizationproblembasedonnonlocalsparsemodelforthe
smoothness,thatcanmainlyimprovespatialdegradation.Andthe
brightnessdifferenceinthesameobjectsbetweentheshadowand
nonshadow areas is restored precisely for radiometric
enhancementbythesecondlinearcorrection.Thebasicunitsofthe
two refined restoration process are both group matrix including
similarpatchesfromshadowandnonshadowareas.
Accordingtothespatialrelationshipbetweentheobjectsandthe
shadows,twopossiblecasesareputforwardasshowninFig.4.In
case 1,the object staysacrosstheshadow areaand nonshadow
area, which leads to the matched patches located both in the
shadowandnonshadowareas.Incase2,thewholeobjectstaysin
theshadowarea,whichleadstoallthematchedpatchesstillinthe
shadowarea.Or,morespecifically,thegroupmatrixonlycontains

where iGk is the intensity of shadow

patches after the second correction in a
group; iG k s is the intensity of shadow
patches in a group; is the shadow
probabilityofoursoftde

tection.
g sh a d ow g n o n sh a d ow g

denote
n o n sh a d ow

g sh a d ow

the mean value and standard

deviation of shadow patches
and nonshadow patches in a
group.

Then,thesin

/i ,andtheiterativereg

ployed to attain the shadow

area as smooth as the
corresponding nonshadow
area. The core concept is to
add filtered noisy shadow
image back to the predicted
shadowfreeimage[28]
(k + 1 )
i

(k
=i

Further, then iterative regularization is extended to update the

estimationofthelocalshadowareavarianceiandtheglobalvariance
alternatively
i(k + 1 )

(k ) 2

= max((si

) /s ((k ) )2 , 0)

wherekistheiterationnumber.Thegroupmatrixincase1canbe
viewedastwonewmatrices,and si isthesingularvalueofthe
shadowmatrix,andsisthenumberofthecolumnsintheshadow
matrix.Thatmeansiisupdatedonlybyshadowpatches

(k ) =

2 i

G k n

G k s l2

(14)

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
SUetal.:SHADOWDETECTIONANDREMOVALFOROCCLUDEDOBJECTINFORMATIONRECOVERYINURBANHIGHRESOLUTION

Fig.4.Twocasesofthespatialrelationshipbetweenshadowsandobjects.

wheredenotesascalingfactor.iGkn, iGksarethe
shadow
respectively.TheyareacquiredonOctober9,2011.Thethird
matrixandthenonshadowmatrixinthesamegroupmatrix.That imagehavingthesameareaasthefirstisimagedbyQuickBird
meansisupdatedbythedifferencebetweenshadowpatches
satellitewith0.6mresolution.ItisacquiredonNovember11,
andnonshadowpatches.Butthedifferencevalueismuchlarger 2007.ThethreeimagesshowtheurbanareaofSanFranciscoin
thantheinitialvaluethatisnotupdated.Becausescenesmay
theUSA.ThelastimageisalsoaWorldView2satelliteimage
changeandtheinitialvalueissetagain.
with0.5mresolution.ItistakenonJanuary19,2010inthe
Incase2,duetoallthepatchesofthegroupmatrixbeinginthe RiodeJaneiroofBrazil.Alltheexperimentaldatahavingcom
shadow,nononshadowinformationwithsimilarstructurecanbe monpointsthattheyallhavelargeshadowsoccludingseveral
usedtorestoretheshadowarea.Andthefirstlinearcorrectionfor buildingobjectsandmanykindsoflandcoversintheurban
alltheshadowareasisperformedbythestatisticalinformation scenes.Theshadowimagesareprocessedtoremoveshadow
oftheglobalnonshadowarea,whilethesecondlinearcorrection fortheobscuredbuildingobjectsrecoverytodemonstratethe
fortheshadowpatchisperformedbythestatisticalinformation effectivenessandrobustnessoftheproposedalgorithm.
ofthenonshadowpatcheswithsimilarstructureinthesame
groupmatrix.However,therearenononshadowpatchesinthe B.ExperimentalStepsandResultsContrast
groupmatrix.Itmeansthegroupmatrixisrestoredonlyby
Fig.5showsaWorldView2satelliteimagemainlycontaining
thestatisticalinformationoftheglobalnonshadowareainthe
twobuildingspartlyoccludedbyabigshadowlikecase1.The
originalshadowimage.Andthebrightnessofthegroupneeds
shadowremovaltorestorethecompletebuildingsisthepurpose
notbecorrectedonceagainincase2.
oftheexperiments.Forthesoftshadowdetection,thebimodal
Similartothecase1,thesmoothnessofthegroupcanbe
thresholdandmorphologicaloperationarefirstemployedto
alsorestoredbytheiterativeregularization.Theupdateofthe
shadowimageisthesameasthecase1.However,thei
and marktherelevantareas.Fig.5(b)(e)showstheprocessofthe
automaticsoftshadowdetectioninourwork.Afterthethreshold
canbeupdatedbytheglobalinformation,ratherthanthe

method,theshadowdetectionresultincludesalotofnoiseand

nonshadowinformationinthegroup[29],[30]

smallshadowareasbeingnothingtodowithoccludedbuildings
inFig.5(b).Therefore,thenoiseandsmallshadowareasare
i
= max((i
) /m ( ) , 0)
(15) removedbythemorphologicaloperationwithalargestructure
denotesthesingularvalueofthegroupmatrix,andm element(theunitissettobe100inthispaper),asshownin
wherei
Fig.5(c).Thesamplesofshadowandnonshadowaremarked
isthenumberofthecolumnsinthegroupmatrix
intheoriginalshadowimage,wherewhitemarksrepresentab
(k ) = 2
(16) soluteshadowareas,blackmarksrepresentnonshadowareas,
I I (k + 1 ) l2
andothersrepresentthependingarea,asshowninFig.5(d).
isthepre Shadowareasandnonshadowareasareerodedbythemorpho
(k + 1 )
(k + 1 )

(k )

(k )

whereIisthewholeoriginalshadowimage,I

dictedshadowfreeimageafterthek+1iteration.

A.DatasetDescription

logicaloperationrespectively(theerodeunitstake5andthe
dilateunits15forshadowandnonshadowinthispaper)toform
thependingareafilledtheoriginalimage.FromFig.5(e),it
isshownthatthesoftshadowdetectionresultbyimagemat
tingrepresentstheshadowprobabilityforeachpixel.Thesoft

Toevaluatetheperformanceoftheproposedmethod,four
differentareasinthethreehighresolutionpanchromaticim
agesofurbanareused.Theexperimentalpanchromaticimages
aredifferentsatellitesanddifferentshadowcases.Thefirstand

shadowcandescribethegradualintensitychangefromshadow
tononshadowarea,whichishelpfultotheconsistenceofthe
recoveredshadowfreeimage.Inordertoprovethesuperiority
oftheproposedmethodforshadowremoval,wecomparethe

V.EXPERIMENTALRESULTS

secondshadowimagesarefromaWorldView2imagewith
0.5mresolutionandbelongtotheshadowcase1andcase2,

shadowremovalresultswiththeexperimentalresultsofthree
othermethods,i.e.,Liusmethod[5],Zhangsmethod[8],and

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

IEEEJOURNALOFSELECTEDTOPICSINAPPLIEDEARTHOBSERVATIONSANDREMOTESENSING

Fig.5.Shadowremovalresultsandexperimentcontrast.(a)Originalshadowimage.(b)Shadowdetectionresultsafterthreshold.(c)Morphologicaloperationbasedon
(b).(d)Hardshadowdetectionandmorphologicalmask.(e)Softshadowdetectionresults.(f)Initialshadowremovalresult.(g)ProposedSANSmethod.

(h)ShadowremovalresultsbyLiusmethod.(i)Zhangsmethod.(j)Lismethod

Lis method [9]. Three obvious shortcomings in the shadow

removal results of Lius method are shown in Fig. 5(h):1) ap
parent boundary exists between the original shadow and non
shadowregion;2)therecoveredshadowareaismorenoisythan
nonshadowarea,causingtherecoveredimagenonuniform;and3)
someshadowpixelsareoverexposedintherecoveredshadowarea.
To Zhangs method (see Fig. 5(i)), the penumbras are handled
well,wherenoobviousboundariesexist.Oneprimarydefectis

the same as Lius method,

whichisthefailureto

control the smoothness in the

recoveredshadowarea.There
stored shadow areas are also
noisy.Lietal.[9]alsousesoft
shadowdetectiontoreducethe
influencesofthepenumbraand
employ spatially adaptive
nonlocal operators to suppress

noiseintheshadowremoval.FromFig.5(j),theshadowremoval and the uniform of the

resultis moresmooththanLiusmethodand Zhangsmethod. brightnessinthesamebuildings
However,partoftexturedetailsofrecoveredbuildingsisnotclear, betweentherecoveredshadow

areasandnonshadowareashas

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
SUetal.:SHADOWDETECTIONANDREMOVALFOROCCLUDEDOBJECTINFORMATIONRECOVERYINURBANHIGHRESOLUTION

Fig.6.Analysisoftheobjectsrecovery.(a)Selecttwoobjectsacrosstheshadowandnonshadowareaandmarkthelinetoanalyzethegrayvalueprofilerecovery.

(b)Grayvalueprofileofrecoveredresultstotheobject2.(c)Grayvalueprofileofrecoveredresultstotheobject.

of the original shadow image,

the shadowfree result of our
stillagaptocomparewithourmethod.Incontrast,inthepro SANS method,andthe results
posedmethod,thesoft shadowdetectioninsteadofthehard oftheothertwomethods,

detectionavoidstheboundarylinesbetweenoriginal shadow
and nonshadow areas. Even more important, the smoothness
andthebrightnessoftherecoveredshadowareaareofthesame
levelasthoseofthenonshadowarea,whichmakethewhole
shadowfreeresultimageofconsistentquality.
Summarizingthecomparativeexperimentalresults,ourSANS
algorithm is more superior than the other three methods for
shadow removal based on objects of the panchromatic satellite
images. First, the soft shadow detection is employed to be
successful for the penumbra region, that makes no obvious
boundarylinesappear.Moreover,thenonlocalstructuresimilarity
andlocalsparsityareusedtosolvetheproblemofthecompound
shadow concluding multiple object types. Our SANS method
controls both the brightness and the smoothness to avoid the
recoveredshadowareanoisyandoverexposed,whichmakesthe
same object type to reach consistent image quality between
recoveredshadowareaandnonshadowarea.Finally,ourmethod
showsgoodperformancetobemoreusefulforfurtherapplication,
i.e.,objectdetectionandobjectextraction.

C.AnalysisoftheObjectsRecovery
Tofurtherinvestigatetheperformanceoftheshadowremoval,
particularly the restoration of the object, we select two objects
acrosstheshadowandnonshadowareasanddisplaytheirprofiles

as shown in Fig. 6. From the

intensity profiles of the same
objects, three key points are
compared:1)forthetransition
of the profiles, the red lines
(SANS method) have smooth
transitionfromtheleftofdash
lines to the right (from
nonshadow area to recovered
shadow area), because of the
penumbra being handled well;
2)forthepeaksofcurvesinthe
right of the dash lines, the
peaks of the red curves are
nearly the same as the left of
dash lines, which indicate the
brightness of the recovered
shadow area being controlled
wellinourmethod;and3)for
the tendency of curves in the
rightofthedashlines,thered
curves and the nonshadow
curves almost have the same
smoothness, while the yellow
and greencurves vibrate more
frequently. It means that the
recovered objects by our

method have the same quality between the shadow area and comparisonsfortheintensityof
nonshadow area, and the two other methods are noisier. In the recovered objects between
particular,theintensityprofilesfromtwobuildingsoftheredlines therecoveredshadowareaand
(ourmethod)andblacklines(Lismethod)havesimilarlevelon nonshadow area, the proposed
smoothness.However,ourmethodcontrolstheaverageandpeakmethod shows a good
ofthebrightnessbetterthanLismethod.Itprovesthatthesecond performance.
linearcorrectionwithsimilarnonshadowpatchesisbeneficialfor
the brightness consistence of the recovered buildings. From the D.AnalysisoftheRecovered

ObjectsEdgeExtracting
To further verify the
efficiency of the application
of the object based on the
shadow removal method, the
modifiedLine

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

10

IEEEJOURNALOFSELECTEDTOPICSINAPPLIEDEARTHOBSERVATIONSANDREMOTESENSING

Fig.7.Analysisoftherecoveredobjectsedgeextracting.(a)Edgeextractingoftheoriginalimage.(b)EdgeextractingoftheshadowremovalimagebyLius
method.(c)Zhangsmethod.(d)Lismethod.(e)OurSANSmethod.

objects.

SegmentDetectoralgorithmisemployedtoextractthelinefeature
ontheedgeoftherecoveredbuildingobjects,asshowninFig.7.
Obviously,theedgefeatureisextractedpreciselyandcompletely,
onlythattheimagequalityoftheobjectsintheshadowareais
recoveredasgoodasthatinthenonshadowarea.InFig.7,the
extractedresultsoftherecoveredobjectsbyfourshadowremoval
methods are compared. Each group results from left to right
displaytheenlargedviewoftherecoveredobjects,theextracted
edgefeature,andoverlapoftheedgesandtheobjects.Fromthe
perspectiveofsubjective,thetworecoveredobjectsareuniform
andclearbytheproposedmethodintheenlargedview.Whatis
more,itisobviousthatbuildingedgefeatureisextractedthemost
completelyinthelastrowbytheproposedmethod(seeFig.7(d)).
Therestoredshadowfreeimagebytheproposedmethodshowsthe
advantageoftheapplicationonthelinearfeatureextractionofthe

E.AnalysisoftheRobustness
toProcessDifferentImages
To further verify the
robustnessandeffectivenessof
the proposed SANS shadow
removal method, different
satellite images and different
shadowcasesareperformedto
compare the three other
methods. As shown in Fig. 8,
the results of shadow removal
aretheoriginalshadowimage,
Liusmethod,Zhangsmethod,
Lis method and our SANS
methodinturn.Inaddition,the
enlargedviewoftherecovered

buildingsisshowninanotherrow.FromtheresultsoftheLius to solve the problem of the

method and Zhangs method, three obvious shortcomings are penumbra. 2) The recovered
shown: 1) fake edges appear between shadow and nonshadow areas are noisier than
areas, due to the presence of penumbra. Zhangs method nonshadow areas. The three
consideredtheboundarypostprocessing;however,itisnotrobust experiments are all poor

performance to control the

spatialsmoothnessofrecovered
areasbythetwomethods.3)

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
SUetal.:SHADOWDETECTIONANDREMOVALFOROCCLUDEDOBJECTINFORMATIONRECOVERYINURBANHIGHRESOLUTION

largeview.Theresultsinturn:theoriginalimage,Liusmethod,Zhangs
method,LismethodandtheproposedSANSmethod.

Fig.8.Resultsofthreeshadowremovalmethodsandtherecoveredobjectsin

11

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

12

IEEEJOURNALOFSELECTEDTOPICSINAPPLIEDEARTHOBSERVATIONSANDREMOTESENSING

objectsinlargeview.Theresultsinturn:theoriginalimage,Liusmethod,
Zhangsmethod,LismethodandtheproposedSANSmethod.

Fig.8.(Continued)Resultsofthreeshadowremovalmethodsandtherecovered

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
SUetal.:SHADOWDETECTIONANDREMOVALFOROCCLUDEDOBJECTINFORMATIONRECOVERYINURBANHIGHRESOLUTION

TABLEI
CAMPARETHEMEANVALUEANDSTANDARDDEVIATIONOFTHERECOVERED
SHADOWAREASAMPLESBYTHREEMETHODSTOTHATOF
CORRESPONDINGNONSHADOWAREA
(a)WorldView2
images
Mean
value
Corresponding
nonshadow
area
Recovered
shadowarea
byLius
method
Recovered
shadowarea
byZhangs
method
Recovered
shadowarea
byLismethod
Recovered
shadowarea

Standard
deviation

(b)QuickBirdimages
Mean
value

Standard
deviation

(c)WorldView2
images
Mean
value

Standard
deviation

55

2.5

54

6.4

82

3.9

73

14.5

61

21.2

97

19.7

63

10.8

57

12.4

100

17.3

63

6.4

58

9.0

98

5.6

62

4.9

56

7.6

93

2.4

byourSANS
method

Thebrightnessisnonuniforminpartofresults.Thesecondex
perimentalresultbyLiusmethod(seeFig.8(b))isthemost
obvious.Theintensityoftherecoveredshadowareasisobvi
ously lower than the corresponding nonshadow areas. From
Lisresults,themajorprobleminLiusandZhangsmethods
isimproved,whichisnoisyintherestoredshadowareas.For
the accuracy of the brightness and the restoration of texture
features, our method shows better performance than Lis
method. Obviously in experiment (c), for the texture of the
highestbuilding,thesurroundingbrightnessofitsroofandthe
penumbra, the proposed SANS method outperforms all the
threeothermethods.Inconclusion,softshadowdetectionis
successfultoavoidtheboundaryeffectofpenumbra.Andthe
SANS method is effective in controlling the brightness and
smoothnessoftherecoveredareastobesimilartotherelated
nonshadowareasinarobustway.Fromthelargeviewofthe
objects,eithercase1orcase2,itisevidentthattherestored
objectsareaccurateanduniformbytheproposedmethod.
Tothethreeexperiments,alltheresultsbyfourmethodsin
experiment (a) have best performance and the results of
QuickBirdimageinexperiment(b)areallworsethantheseof
WorldView2.Theremaybe three mainreasons that: 1)the
image of experiment (a) globally has lower gray value and
smallergrayrange,leadingtolessnoiseafterlinearscalingin
shadowareas.Itreducesthedifficultyforshadowremoval;2)
due to the angle of satellite imaging, this QuickBird image
concludesmanyselfshadowstomakeitdifficultforshadow
removal;and3)theradiationinformationislostcompletelyin
the original QuickBird image, because we cannot see some
detailsoftheobjectsinshadowthrougheyes.Theinformation
cannotberecovered,ifitisnotvisibletothehumaneye.The

discussion above is a
subjectiveevaluation.

13

For the quantitative

evaluation, we selected the
samples of the recovered
shadow areas from the results
by three methods and the
samples in the corresponding
nonshadowareas.Thesamples
are selected on the objects in
images (b) and (c), while the
sample of image (a) is on the
ground.Because(a)belongsto
theshadowcase2,theobjects
stayintheshadowcompletely,
whichisdifficulttobeassessed
without similar nonshadow
areas.ItisshowninFig.8,blue
squareissamplesofnonshadow
areaandredsquareissamples
ofrecoveredshadowarea.

Intheexperiment,themean
value and standard deviation
oftherecoveredareasamples
are closer to those of
nonshadow area samples,
indicating that the method is
better. As can be seen from
Table I, our method
performed better than the
three other methods in all
three images. The results
indicate that the proposed
SANS method can promise
the brightness (mean value)
andthesmoothness(standard
deviation) of recovered area
tobethesameasthoseofthe
similar nonshadow area. The
obscured information of
objectsshowsbetterrecovery.
VI.
CONCLU
SION

Inthispaper,wehaveput
forward an automatic soft
shadowdetectionmethodand
a SANS shadow removal
method for the very high
resolution panchromatic
satelliteimages.Ourworkhas
dealt with the challenging
problem of the shadow
removal in the panchromatic
images,particularlytorecover
the obscured information of
objectsprecisely.

For shadow detection, we combine the bimodal histogram method.Fromtheexperimental

splittingmethodandimagemattingtechniquetoobtainthesoft results,ourmethodtosolvethe
detection results automatically. The soft shadow detection main problems is summarized
resultscanshowtheprobabilityvalueofeachpixelbelonging asfollows:
totheshadow,tosolvethepresenceofpenumbras.
1) The soft shadow
For shadow removal to restore the occluded information of
objects, we propose two levels removal method concluding the
initialcorrectionandtherefinedrestoration.Theinitialcorrection
step compensates the intensity for shadow areas by global
parameter.Intherefinedrestorationstep,weutilizegroupsparse
representationtobuildtherelationshipbetweentheshadowarea
and the corresponding nonshadow area of the same objects.
Shadow areas have been restored precisely using nonshadow
patcheswith similar structurein the samegroup,whatismore,
makingtheobscuredpartofobjectsbethesamebrightnessand
smoothness level as the other part in the nonshadow area. The
uniformshadowfreeimageshavebeenobtainedbytheproposed

detection is performed
automatically. The
shadow probability is
calculated accurately to
show the distribution of
umbra and penumbra.
Theproposedautomatic
method is effective to
avoid the boundary
effects after recovery,
duetothepresenceofthe

penumbra.

2)

The proposed SANS

shadow removal method
can restore the obscured
informationofobjectsin
shadow areas ef
fectively. We analyze
two cases of the spatial
relationship between the
objectsandshadows.The
shadowareasarerestored
by similar structure
patches based on the
group

sparse
representation, and the
smooth results are
obtained.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

14

IEEEJOURNALOFSELECTEDTOPICSINAPPLIEDEARTHOBSERVATIONSANDREMOTESENSING

[14]

P. Sarabandi, F.
Yamazaki,M.Matsuoka,andA.
Kiremidjian,Shadowdetection
and radiometric restoration in
satellite high resolution
images,inProc.IEEEGeosci.
RemoteSens.Symp.,vol.6,Sep.
2004,pp.37443747.

3)

BasedongroupmatrixofourSANSmethod,wepresenta
twicelinecorrectionmethodtocontrolthebrightnessofthe
recovered areas. First correction uses the whole image
parameters to enhance shadow areas roughly. Second
correctionmakesuseoftheparametersofsimilarpatchesin [15]
A.Buades,B.Coll,andJ.
M. Morel, A nonlocal
thegroupmatrixtocorrecteachpatchinshadowareas,and
algorithm for image
uniformshadowfreeimagesareobtained.
denoising,inProc.IEEEConf.
Comput. Vision Pattern.
Inconclusion,theproposedSANSmethodcombinesnonlocal
Recognit.,vol.2,2005,pp.60
sparse model and the second linear radiometric correction to
65.
simultaneously control the brightness and smoothness of the
[16] Y.Lou,X.Zhang,S.Osher,
recoveredshadowareas.Theuniformobjectsintheshadowareas
andA.Bertozzi,Imagerecovery
canberestoredautomaticallybyourmethod.
via nonlocal operators, J. Sci.
Comput.,vol.42,no.2,pp.185
197,Feb.2010.

[17]
REFERENCES

[1]

A. Makarau, R. Richter, R. Muller, and P. Reinartz, Adaptive

shadow detection using a blackbody radiator model, IEEE Trans.
Geosci.RemoteSens.,vol.49,no.6,pp.20492059,Jun.2011.

[2]

V.J.D.Tsai,Acomparativestudyonshadowcompensationof
color aerial images in invariant color models, IEEE Trans. Geosci.
RemoteSens.,vol.44,no.6,pp.16611671,Jun.2006.

[3]

M.PaulDare,Shadowanalysisinhighresolutionsatelliteimagery
ofurbanareas,PhotogrammetricEng.RemoteSens.vol.71,no.2,

42. 169177,2005.
[4]

L.Luca,F.Melgani,andG.Mercier,Acompleteprocessingchain
forshadowdetectionandreconstructioninVHRimages, IEEETrans.
Geosci.RemoteSens.,vol.50,no.9,pp.34403452,2012.

[5]

W. Liu and F. Yamazaki, Objectbased shadow extraction and

correction of highresolution optical satellite images, IEEE J. Sel. Topics
Appl.EarthObserv.RemoteSens.,vol.5,no.4,pp.12961302,Aug.2012.

[6]

E. Mohamed and M. Khan Iftekharuddin, Shadow detection of

manmade buildings in highresolution panchromatic satellite images,
IEEETrans.Geosci.RemoteSens.,vol.52,no.9,pp.53745386,2014.

[7]

L.FrancoisPierreSarel,F.VandenBergh,andB.TikanathJugpershad
Maharaj,Theeffectsofsegmentationbasedshadowremovalonacrossdate
settlementtypeclassificationofpanchromaticQuickBirdimages,IEEEJ.Sel.
Top.Appl.EarthObserv.RemoteSens.,vol.6,no.3,

42. 12741285,Jun.2013.
[8]

H.Zhang,K.Sun,andW.Li,Objectorientedshadowdetectionand
removal from urban highresolution remote sensing images, IEEE Trans.
Geosci.RemoteSens.,vol.52,no.11,pp.69726982,Nov.2014.

[9]

H.Li,L.Zhang,andH.Shen,Anadaptivenonlocalregularized
shadowremovalmethodforaerialremotesensingimages,IEEETrans.
Geosci.RemoteSens.,vol.52,no.1,pp.106120,Jan.2014.

[10]

G. Ruiqi, Q. Dai, and D. Hoiem, Paired regions for shadow

detectionandremoval,IEEETrans.PatternAnal.Mach.Intell.,vol.35,
no.12,

42. 29562967,Dec.2013.
[11] H. Song, B. Huang, and K. Zhang, Shadow detection and

reconstruction in highresolution satellite images via morphological

filteringandexamplebasedlearning,IEEETrans.Geosci.RemoteSens.,
vol.52,no.5,pp.25452554,2014.

[12]

A. Levin, D. Lischinski, and Y. Weiss, A closedform solution to

naturalimagematting,IEEETrans.PatternAnal.Mach.Intell.,vol.30,no.2,

42. 228242,Feb.2008.
[13] F.P.S.Luus,F.VandenBergh,andB.T.J.Maharaj,Adaptive

thresholdbasedshadowmaskingforacrossdatesettlementclassification
ofpanchromatic QuickBirdimages, IEEEGeosci.RemoteSens.Lett.,
vol.11,no.6,pp.11531157,2014.

G. Gilboa and S. Osher,

Nonlocal operators with
applicationstoimageprocessing,
Multiscale Model. Simul., vol. 7,
no.3,pp.10051028,2008.

[18]

M.Bertalmio,V.Caselles,
and E. Provenzi, Issues about
retinex theory and contrast
enhancement, Int. J. Comput.
Vis., vol. 83, no. 1, pp. 101
119,Jun.2009.

[19]

H. Li, L. Zhang, and H.

Shen, A perceptually inspired
variationalmethodfortheuneven
intensity correction of remote
sensing images, IEEE Trans.
Geosci.RemoteSens.,vol.50,no.
8,pp.30533065,Aug.2012.

[20]

F.Yamazaki,W.Liu,and
M.Takasaki,Characteristicsof
shadow and removal of its
effects for remote sensing
imagery, in Proc. IEEE
Geosci. Remote Sens. Symp.,
vol. 4, Jul. 2009, pp. IV426
IV429.

[21]

H. Shen et al., Missing

information reconstruction of
remote sensing data: A technical
review, IEEE Geosci. Remote
Sens.Mag.,vol.3,no.3,

42. 6185,Sept.2015.
[22] X.Li,H.Shen,L.Zhang,

and H. Li, Sparsebased

reconstruction of missing
information in remote sensing
images from spectral/temporal
complementary information,
ISPRS J. Photogrammetry
RemoteSens.,vol.106,pp.1
15,2015.

[23]

W.Dong,G.Shi,andX.
Li,Nonlocalimagerestoration
with bilateral variance
estimation: A lowrank
approach, IEEE Trans. Image
Process., vol. 22, no. 2, pp.
700711,2013.

[24]

J.Zhang,D.Zhao,andW.
Gao, Groupbased sparse
representation for image
restoration,IEEETrans.Image
Process., vol. 23, no. 8, pp.
33363351,Aug.2014.

[25]

E. Reinhard, M.
Adhikhmin, B. Gooch, and P.
Shirley,Colortransferbetween
images,IEEEComput.Graph.
Appl.,vol.21,no.5,pp.3441,
Sep./Oct.2001.

[26]

F.L.Gadallah,F.Csillag,
andE.J.M.Smith,Destriping
multisensory imagery with
moment matching, Int. J.
RemoteSens.,vol.21,no.12,

42. 25052511,2013.
[27] J.Cai,E.Candes,andZ.

Shen, A singular value

thresholding algorithm for
matrix completion, SIAM J.
Optim.,vol.20,no.4,pp.1956
1982,2010.

[28]

S. Osher, M. Burger, D.
Goldfarb, J. Xu, and W. Yin,
An iterative regularization
methodfortotalvariationbased

imagerestoration,MultiscaleModel.Simul.,vol.4,no.2,pp.460489,
2005.

[29]

J.XuandS.Osher,Iterativeregularizationandnonlinearinverse
scale space applied to waveletbased denoising, IEEE Trans. Image
Process.,vol.16,no.2,pp.534544,Feb.2007.

[30]

M. Charest, M. Elad, and P. Milanfar, A general iterative

regularizationframeworkforimagedenoising,inProc.40thAnnu.Conf.
Inf.Sci.Syst.,2006,pp.452457.

NanSu(S13)receivedtheB.S.degreeinelectronic
andinformationengineering,andtheM.S.degreein
signal and information processing from the Harbin
InstituteofTechnology,Harbin,China,in2010and
2012,respectively,whereheiscurrentlyworkingto
wardthePh.D.degreeininformationandcommuni
cationengineering.
Hisresearchinterestsincludehighresolutionre
mote sensing image processing and 3D reconstruc
tionofurbanarea,andapplication.

YeZhang receivedtheB.S.degreeincommunica
tion engineering, and the M.S. and Ph.D. degrees

both in
communica
tion and
electronic
system,
from the
Harbin
Institute of
Technology
(HIT),
Harbin,
China, in
1982,1985,
and 1996,
respectivel
y.
Since
1985,hehas
been a
workingasa
Teacher at
HIT.
Between
1998 and
1999, he
was a
Visiting
Scholar at
the
University
of Texas at

San
Antonio.
Hisresearch
interests
include
remote
sensing
hyper
spectral
image
analysis and
processing,
image/video
compression
and
transmission
, as well as
multisource

information collaboration processing

and applications. He is currently a
Professor and the Doctoral
Supervisor in information and
communicationengineering.Heisthe
Director of Institute of Image and
InformationTechnologyattheSchool
of Electronic and Information
Engineering,HIT.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
SUetal.:SHADOWDETECTIONANDREMOVALFOROCCLUDEDOBJECTINFORMATIONRECOVERYINURBANHIGHRESOLUTION

YimingYan wasbornonJuly4,1982inWeihe ForestryBureauWeihe,Heilongjiang,Xinyuan Miao was

China.HereceivedtheB.Eng.degreefromtheHarbinInstituteofTechnology,Harbin, born in Shandong,
China,in2006andtheM.Eng.degreein2008.HeiscurrentlyworkingtowardthePh.D. China, in 1992. He
degree at the Institute of Image and Information Technology, Harbin Institute of received the B.Sc.
degree from the
TechnologyinthefieldofstereoRSinformationprocessing.

Harbin Institute of
Technology, Harbin,
China, in 2015,
whereheiscurrently
ShuTian(S14)wasborninBaicheng,China,in1986.HereceivedtheB.Eng.degree working toward the
fromtheChangChunUniversityofScienceandTechnology,Changchun,China,in M.Sc.degreeinthe
2012.HeiscurrentlyworkingtowardthePh.D.degreeattheInstituteofImageand field of spectral
Information Technology, Harbin Institute of Technology in the field of 3Dunmixing.
His research
reconstructionand3Dvisualization.
interests include the

15

physical processes
of hyperspectral
remote sensing,
including spectral
unmixing and the
retrieval of surface
emissivity from
thermal infrared
remotesenseddata.