This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 1

BOEW: A Content-based Image Retrieval

Scheme using Bag-of-Encrypted-Words in
Cloud Computing
Zhihua Xia, Member, IEEE, Leqi Jiang, Dandan Liu, Lihua Lu, and Byeungwoo Jeon, Senior Mem-
ber, IEEE,

Abstract—Content-based Image Retrieval (CBIR) techniques have been extensively studied with the rapid growth of digital images.
Generally, CBIR service is quite expensive in computational and storage resources. Thus, it is a good choice to outsource CBIR
service to the cloud server that is equipped with enormous resources. However, the privacy protection becomes a big problem, as the
cloud server cannot be fully trusted. In this paper, we propose an outsourced CBIR scheme based on a novel bag-of-encrypted-words
(BOEW) model. The image is encrypted by color value substitution, block permutation, and intra-block pixel permutation. Then, the
local histograms are calculated from the encrypted image blocks by the cloud server. All the local histograms are clustered together,
and the cluster centers are used as the encrypted visual words. In this way, the bag-of-encrypted-words (BOEW) model is built to
represent each image by a feature vector, i.e., a normalized histogram of the encrypted visual words. The similarity between images
can be directly measured by the Manhattan distance between feature vectors on the cloud server side. Experimental results and
security analysis on the proposed scheme demonstrate its search accuracy and security.

Index Terms—Searchable encryption; Computation outsourcing; Content-based image retrieval; Bag-of-Words.

F
1 I NTRODUCTION

T HE world has witnessed a rapid development of imag-

ing devices, such as digital cameras, medical imaging
equipments, smart phones, and so on. Accordingly, the
number of digital images increases dramatically. In order
to retrieve similar images quickly from large amount of im-
ages, many practical Content-based Image Retrieval (CBIR)
techniques have been developed. However, typical image
database is simply too large, including millions of images
and each may be larger than 40 megabytes [1]. Thus, CBIR
service generally requires heavy storage and computation.
Such demands make it attractive to outsource CBIR services
to the cloud server. In this way, the image owner needs
not to store the image database locally, and can efficiently
retrieve the desired images from the cloud server [2].
Apart from the enormous benefits of CBIR outsourcing,
the privacy of images becomes the biggest concern to the
image owner. Both the image database and the query image
should be protected properly.
Contribution. In this paper, we propose an outsourced
CBIR scheme where the image content is properly protected.
• Zhihua Xia, Leqi Jiang, Dandan Liu, and Lihua Liu are with Jiangsu
Engineering Center of Network Monitoring, Jiangsu Collaborative Inno-
vation Center on Atmospheric Environment and Equipment Technology,
and School of Computer and Software, Nanjing University of Information
Science & Technology, Nanjing, China. Zhihua Xia is also with the
College of Information & Communication Engineering, Sungkyunkwan
University, Korea.
E-mails: xia zhihua@163.com, jdz jlq@hotmail.com, jsrgldd@qq.com,
millie Lu@163.com
• Byeungwoo Jeon is with the College of Information & Communication
Engineering, Sungkyunkwan University, Korea.
E-mail: bjeon@skku.edu
Manuscript received November 9, 20**; revised August 26, **.
The main contributions are summarized as follows:
1) A BOEW model is proposed for CBIR outsourcing. We
propose to encrypt images by blocks and make sure that
the secure and useful local features can be directly ex-
tracted from the encrypted blocks. k -means clustering
algorithm is deployed to generate the encrypted visual
words. The final feature vectors, also the encrypted
ones, are then constructed with the visual words. The
similarity between the feature vectors can be directly
measured by Euclidean or Manhattan distance. The
proposed BOEW could be a valuable model in encrypt-
ed image processing.
2) As a case study, we propose to encrypt image by color
value substitution, block permutation, and intra-block
pixel permutation. With the specially-designed encryp-
tion method, secure local histograms can be directly
extracted from the encrypted images on cloud server
side. The index construction can also be finished by
cloud server. Compared with the scheme using secure
global histogram [3], [4], our method achieves a much
better retrieval accuracy.
The rest of this paper is organized as follows. Section
2 introduces the related works. Section 3 presents the tech-
nical overview. Section 4 addresses the proposed scheme
design. Section 5 gives the security analysis. Experiments
and results are presented in Section 6. Finally, conclusion is
made in Section 7.
2 R ELATED WORKS
CBIR techniques have been studied for more than twenty
years and shown its maturity in many real-world appli-

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018
cations [5], [6], [7], [8]. However, it cannot be outsourced
directly to the cloud due to the privacy concern. It is
noteworthy that, in addition to the original image data, the
image features would also leak the information about image
contents if they are not well protected.
Searchable encryption (SE) enables the clients to store
the encrypted data at the cloud, meanwhile supports data
search over cipher-text domain [9]. However, many of the
existing SE schemes are designed for text documents [10],
[11], [12]. Lu et al. [13] proposed the first privacy-preserving
CBIR scheme over the encrypted image database. The
scheme utilized the set of visual words to represent images.
The similarity between images was measured by Jaccard
distance between the sets of visual words. The min-hash
algorithm and order-preserving encryption were employed
to protect the visual words. In another work [14], Lu et
al. investigated three image feature protection techniques
including bitplane randomization, random projection, and
randomized unary encoding. The bitplane randomization
and random unary encoding support the calculation of
Hamming distance in the encryption domain. The random
projection supports the approximate calculation of L1 dis-
tance in the encryption domain. In [15], Lu et al. compared
the three mentioned methods with the homomorphic en-
cryption and indicated that the homomorphic encryption
consumed much more computation and communication
resources. Yuan et al. [16] protected the image features using
local sensitive hashing and Cuckoo Hashing to support
secure similarity search. This method was used to discovery
the social connections between image owners. Xia et al.
[17] proposed a privacy-preserving CBIR scheme based on
Scale-Invariant Feature Transform (SIFT) features and Earth
Mover’s Distance (EMD). The calculation of the EMD is in
fact a linear program problem. The linear transformation
was utilized to protect the privacy information during the
solution process of EMD problem. Yuan et al. [18] designed
an encrypted image search scheme based on the secure k NN
(k -nearest neighbors) algorithm and constructed a tree index
to improve the search efficiency. In [19], Chen et al. proposed
a Markov process-based retrieval scheme over encrypted
images. The image content was protected by encrypting
the Huffman table in JPEG files. The Markov features were
directly extracted from the DCT coefficients which were
decoded with the encrypted Huffman table. In [20], [21],
Weng et al. proposed a framework for privacy-preserving
multimedia retrieval. The media features were protected by
the robust hashing and partial encryption by image owner.
Then encrypted part of hash introduced search ambiguity
to server. The similar images were retrieved using the
unencrypted part of hash on the server side and refined
with the whole plaintext hash on the query user side. In
[22], Xia et al. proposed a privacy-preserving CBIR scheme.
Four MPEG descriptors were used to represent the images.
Secure k NN algorithm was employed to protect the image
features. Locality-sensitive hash was used to increase the
search efficiency. In addition, the authors incorporated an
encryption-domain watermarking method to the scheme so
as to deter the image users’ illegal distribution. In [23],
Zhang et al. proposed a secure outsourced CBIR scheme
with fine-grained access control. A key-agent was intro-
duced to identify which images can be accessed by a user.
1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
2
Fig. 1: System model. In the initialization phase, the image
owner encrypts and uploads the images to the cloud server.
Then, the cloud server extracts features from the encrypted
images to generate the index. In the query phase, the image
owner uploads an encrypted query image to the cloud serv-
er. Next, the cloud server extracts features from the query
image and search the index. Finally, the images similar to
the query image are returned to the image owner.
The works mentioned above are good solutions to the
CBIR service outsourcing. Nevertheless, there exists a com-
mon disadvantage. As the big volume of storage and large
computation complexity, both the image feature extraction
and index construction are resource-consuming operations.
In the previous outsourced CBIR scheme, it is data own-
er to undertake the tasks of feature extraction and index
generation [13], [14], [15], [16], [17], [18], [19], [21], [22].
Bellafqira et al. [24], [25] proposed two secure CBIR schemes
in homomorphic encryption domain. The SIFT [24] and
discrete wavelet transform features [25] can be extracted
directly from the encrypted images. However, these features
can only support the exact search because the tiny differ-
ences between the plaintext images can make the encrypted
images completely different. In [26], Xu et al. proposed a
privacy-preserving CBIR scheme based on partial encryp-
tion. The images were orthogonally decomposed and then
divided into two different parts. One part was encrypted to
protect the image content and the other was used to extract
features for image retrieval. However, the unencrypted part
can lead serious information leakage. Ferreira et al. [3], [4]
proposed an Image Encryption Scheme which is tailor-made
for constructing a CBIR scheme (IES-CBIR). In this scheme,
the color values are protected by random permutation and
the pixels are shuffled by rows and columns. After the
encrypted image are uploaded to the cloud server, the
HSV (Hue-Saturation-Value) color histograms are extracted
from the encrypted images at the cloud server side. The
similarities between images can be directly measured by
Hamming distances between the corresponding histograms.
In this way, the image owner undertakes the image en-
cryption only. Other tasks, such as feature extraction, index
generation, and search operation, are outsourced to the
cloud server. However, the global histogram is too rigid for
image retrieval problems. Liu et al. [27] tried to improve
the retrieval accuracy with difference histogram. But the
improvement is not significant. In this paper, we propose
an outsourced CBIR scheme based on secure local features.
The proposed scheme also outsources the tasks of feature
extraction, index construction, and search operation to the
cloud server, meanwhile achieving much higher retrieval
precision by using the proposed BOEW model.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018
3 S YSTEM OVERVIEW AND PRELIMINARIES
3.1 System model
Similar to [28], the proposed scheme involves two types of
entities, i.e., the image owner and the cloud server, as shown
in Fig. 1.
Image owner has a large-scale image database, I =
{Ii }ni=1 with the corresponding identity set ID = {IDi }ni=1 ,
to be outsourced for cost saving and convenient utilization.
To preserve privacy, the image database needs to be encrypt-
ed before being uploaded, generating an encrypted image
set C = {Ci }n i=1 . Besides the image encryption, the image
owner would like to outsource the computation and storage
tasks to the cloud server as much as possible.
After storing the images on the cloud server, the image
owner may want to retrieve the desired images which are
similar to a query image. In the query phase, the image
owner in our scheme needs only to encrypt the query
image and then uploads it. The feature extraction and search
operation are finished by the cloud server.
Cloud server stores the encrypted images for the image
owner and provides CBIR service for the image owner. In
addition to the search operation, the cloud server in our
scheme also takes charge of the index generation.
3.2 Security model
Similar to the previous SE schemes [12], [16], [28], [29], [30],
[31], an honest-but-curious cloud server is considered in
our scheme. That is to say, the cloud server follows the
designated protocol specification correctly, but may pre-
serve and analyze the history of communication to acquire
sensitive information. In this paper, we do not consider the
information leakage due to the access pattern. In addition,
if images Ii and Ij are clustered together or return as the
search results to the same query, it is not difficult to deduce
that images Ii and Ij are similar to each other. This kind of
information leakage is also not considered in this paper as
many previous works [3], [4], [13], [14], [15], [17], [18], [19],
[21], [22], [24], [25].
3.3 Preliminaries
3.3.1 Bag-of-Word model
CBIR methods extract visual features to represent the im-
ages. Then, the similarity between images will be measured
by the distance between feature vectors. The features can
be extracted either globally from the entire image or locally
from the small regions. Generally, the local features are more
robust and likely to retrieve more accurate results [32]. One
popular CBIR approach using local features is the bag-of-
words (BOW) model [33].
BOW model firstly appeared in the field of natural
language processing and information retrieval. The model
regards the document as a collection of words, ignoring the
word order and grammar of the documents. Afterward the
BOW model is widely applied in image retrieval [34], [35].
In this model, local features are extracted from images in the
database, and then are jointly clustered. The cluster centers
are defined as “visual words” to form the vocabulary. Then,
all the local features are represented by its nearest visual
word and an image can be finally denoted by a histogram
1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
3
of the visual words. Specifically, the BOW model has three
steps:
(i) Local feature extraction. The first step is to extract
local features from the images in database. Some typical
local features include SIFT [36] and SURF [37], which are
invariant to illumination, deformation, and rotation. As the
development of cloud computing, some researchers have
designed several secure SIFT feature extraction methods
which can calculate the SIFT features from the particularly
encrypted images. However, the extracted features are also
the encrypted ones and cannot be directly employed for
the image retrieval [38], [39]. For efficiency and security, we
calculate the color histograms from the image blocks as local
features.
(ii) Vocabulary construction. The second step is to con-
struct the vocabulary. Typically, k -means method can be
employed to cluster the local features into k classes. The
cluster centers are defined as visual words. The full set of
the visual words constitute the vocabulary.
(iii) Histogram calculation. The last step is to calculate
the histogram of visual words. Please note that, all the
local features are represented by their nearest cluster centers
(visual words). Finally, each image is represented by a k -bins
histogram of visual words.
3.3.2 Color space
Color images can be displayed in many color spaces. In
this paper, we describe our system with HSV color model
as [3], [4]. HSV is a cylindrical-coordinate representation
of the points in RGB color model. HSV stands for hue,
saturation, and value. In matlab, the components of HSV
are measured by the decimals in [0, 1]. Here we quantize
them into integers in [0, 100] as [3], [4] for the convenience
of local histogram calculation. In our experiments, we also
test our scheme in RGB and YUV color spaces.
4 T HE P ROPOSED SCHEME
4.1 Overview of the proposed scheme
The proposed scheme includes six algorithms which are
respectively executed by two entities, i.e., KeyGen, ImgEnc,
TrapGen, and ImgDec executed by the image owner, and
IndexGen and Search executed by the cloud server.
First of all, given an image database I = {Ii }n i=1 , the
image owner runs KeyGen to generate a set of secret keys K.
Then, the owner runs ImgEnc to encrypt the image database,
generating an encrypted image database C = {Ci }n i=1 .
The encrypted image database C is uploaded to the cloud
server. Upon the receipt of encrypted image database C , the
cloud server runs IndexGen to build a secure index Idx. To
retrieve similar images, the image owner runs TrapGen to
generate a trapdoor and then submits it to the cloud server.
Once receiving the query request from the image owner, the
cloud server runs Search to find θ similar images which
are returned to the image owner as the search results. After
receiving the search results, the image owner runs ImgDec
to decrypt the retrieved images. A summarization of the
algorithms is presented in Fig. 2.
4.2 Privacy-preserving CBIR protocol
Detailed algorithm processes are as following.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018
Image owner side:
• K ← KeyGen(1κ ) is the key secret generation algo-
rithm which takes as input the security parameter κ,
and returns the secret key K.
• C ← ImgEnc(K, I, ID, blksize) is the image encryp-
tion algorithm which takes as inputs the secret key
K, the image database I , identity set of images ID,
and the predefined block size blksize, and returns
the encrypted image database C .
• Iq ← ImgDec(K, Rq , blksize) is the decryption al-
gorithm which takes as inputs the secret key K, the
set of result images Rq and block size blksize, and
returns the decrypted image set Iq .
• TD ← TrapGen(K, Iq , blksize) is the trapdoor gener-
ation algorithm which takes as inputs the secret key
K, the query image Iq and the block size blksize, and
returns the trapdoor TD.
Cloud server side:
• Idx ← IndexGen(C, blksize) is the index generation
algorithm which takes as inputs the image database
C and the block size blksize, and returns the index
Idx.
• Rq ← Search(C, Idx, TD, blksize) is the search algo-
rithm which takes as inputs the encrypted database
C , the index Idx, the trapdoor TD and the block size
blksize, and returns the encrypted similar images
Rq .
Fig. 2: Overview of the algorithms in the proposed scheme
4.2.1 Key Generation
K ← KeyGen(1κ ). Images are made up of two kinds
of information, i.e., color and texture information, both
of which should be protected properly [3], [4]. In our
scheme, color information is protected by substituting the
color values, while the texture information is protected by
shuffling the pixel positions. This scheme processes the
images in HSV color space. To protect the image content,
we firstly substitute the color values in three components
with three different secret keys. Next, the image is divid-
ed into non-overlapping blocks which are then shuffled.
Finally, the pixels in each block are shuffled to further
protect the texture information. Accordingly, we generate
a pseudo-random permutation generator and several secret
keys for the whole encryption process, i.e., K = {RandPerm,
{keyv ∗ }∗∈{H,S,V } , keyb , {keyp ∗ }∗∈{H,S,V } }.
Here, the secret keys {keyv ∗ }∗∈{H,S,V } are utilized to
generate random permutations to substitute color values
in H, S, V components, respectively. The random permu-
tations are generated from the range [0...100] as follows:
{pmtv ∗,# } ← RandPerm(keyv ∗ , [0...100]),
where ∗ ∈ {H, S, V }, # ∈ {1, ..., Npmt }, Npmt is amount
of the random permutations for each color component.
The secret key keyb is used to generate pseudo-random
permutations from the range [1...blknum], where blknum
is the total number of non-overlapping blocks in an image.
Here, the random permutation generated by keyb is used to
shuffle image blocks and is generated as follow:
1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
4
pmtb ← RandPerm(keyb , [1...blknum], ID). (2)
We apply the same block permutation secret key to the
three components in an image, but different keys to different
images for better security without any influence on retrieval
accuracy. In the process of calculation, the ID and the secret
key keyb are merged to be the seed of the pseudo-random
permutation generator.
The secret keys {keyp ∗ }∗∈{H,S,V } are used to generate
random permutations to shuffle pixels in blocks. The ran-
dom permutations of three components are generated as
follows:
pmtp∗j ← RandPerm(keyp ∗ , [1...blksize], ID, j), (3)
where ∗ ∈ {H, S, V }. Here we generate different keys for
different image blocks. This increases the security but has
no influence on the local histogram calculation, and thus
will not influence the image retrieval.
Among the secret keys above, the RandPerm, blksize
and keyv ∗ are needed to request an valid image query.
After receiving the encrypted images, all the secret keys are
needed for image decryption.
4.2.2 Image encryption
C ← ImgEnc(I, ID, K, blksize). As presented above, there
are three steps in the image encryption including color
value substitution, block permutation, and intra-block pixel
permutation. For each step, we present a sub-algorithm to
specify its process (see Algorithm 1, 2, and 3).
As shown in Algorithm 1, we utilize a polyalphabetic
cipher to encrypt the color values. Polyalphabetic cipher
encrypts the values by substituting values with multiple
tables which in this paper are the permutations of the
elements in the set {0,..., 255}. In this way, the same pixel
value at different positions can be substituted with different
values, which helps to resist the statistic attack [40], [41]. As
presented by Algorithm 2, and 3, for block permutation and
intra-block permutation, we generate random permutations
to shuffle the block position and intra-block pixel position. It
is quite straightforward. Finally, with these sub-algorithms,
the whole process of the image encryption is defined as in
Algorithm 4.
4.2.3 Index generation
Idx ← IndexGen(C, blksize). In our scheme, the construc-
tion of index is also outsourced to cloud server so as to
further reduce the burden of the image owner. Inspired by
the famous BOW model, we propose a novel BOEW model
(1) to support the image retrieval while preserving the image
privacy. Similar to the BOW model, our method consists of
three steps.
(i) Local histogram extraction. Firstly, the encrypted images
are divided into non-overlapping blocks as that in the image
encryption process. Next, we calculate a local histogram
from each block, and denote the local histogram of j -th
block in i-th image as hij = (bijτ )303 τ =1 , where i ∈ {1, ..., n},
j ∈ {1, ..., blknumi }. It is worth noting that the local
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 5

Algorithm 1 ValueSubstitution Algorithm 3 IntrablockPermut

Input: Image I and secret keys {keyv ∗ }∗∈{H,S,V } Input: I ′′ , ID, {keyp ∗ }∗∈{H,S,V } , blksize
Output: I ′ Output: C
1: Generate the secret permutations pmtv H,# , pmtv S,# , 1: Denote the final encrypted image as C whose compo-
and pmtv V,# , where # ∈ {1, ..., Npmt } ; nents are denoted as CH , CS , and CV . Divide the com-
2: Generate three random sequences sqntH , sqntV , sqntS . ponents CH , CS , and CV into non-overlapping blocks
c
The length of sequences are equal to the pixel amount denoted by blkH , blkSc , and blkVc ;
of the image I and the elements of these sequences are 2: Divide the components of image I ′′ into non-
′′
within the set [1, ..., Npmt ]; overlapping blocks denoted by blkH , blkS′′ , and blkV′′ ;
3: Denote pi as ith pixel in image I , and pi H , pi S , pi V as 3: for ∀∗ ∈ H, S, V do
the three components of the pixel; 4: for ∀ blk ′′ ∗j ∈ blk∗′′ do
4: Denote p′i as the corresponding pixel in the encrypted 5: Generate the secret permutation for j -th block
image I ′ , and p′iH , p′iS , p′iV as the three components of blk ′′ ∗j as pmtp ∗j ;
′′
the pixel; 6: for ∀ blk∗j [i] ∈ blk ′′ ∗j do
5: for each p′i ∈ I ′ do 7: blk ∗j [i] ← blk ′′ ∗j [pmtp∗j [i]];
c
6: p′iH ← pmtv H,sqntH [i] [pi H ], p′iS ← 8: end for
′
pmtv S,sqntS [i] [pi S ], piV ← pmtv V,sqntV [i] [pi V ] 9: end for
7: end for 10: end for

Algorithm 2 BlockPermut Algorithm 4 ImgEnc

′
Input: I , ID, keyb , blksize Input: I, ID, K, and blksize,
Output: I ′′ 1: for ∀ Ii ∈ I do
1: /* blksize denotes the size of image blocks, which is an open 2: Ii′ = ValuePermut(Ii , {keyv ∗ }∗∈{H,S,V } );
parameter in our scheme. */ 3: Ii′′ = BlockPermut(Ii′ , IDi , keyb , blksize);
2: Calculate the total number of non-overlapping blocks in 4: Ci = IntrablockPermut(Ii′′ , IDi , {keyp ∗ }∗∈{H,S,V } ,
image I ′ as blknum = imgsize/blksize; blksize);
3: Generate the secret permutation pmtb; 5: end for
4: Divide the image I ′ into non-overlapping blocks denot-
ed as blk ′ ;
5: Divide the resulting image I ′′ into the blocks denoted as the feature vector make up a linear index as shown in Table
blk ′′ ; 1.
6: for ∀blk ′′ [i] ∈ I ′′ do
7: blk ′′ [i] ← blk ′ [pmtb[i]]; 4.2.4 Trapdoor generation
8: end for
TD ← TrapGen(K, Iq , IDIq , blksize). In our scheme, the
trapdoor is generated by encrypting the query image with
the color value substitution, block permutation, and intra-
histograms in our scheme are the encrypted ones since the block pixel permutation. The encrypted query image is then
color values have been encrypted by the substitution. uploaded to the cloud server as the trapdoor.
Compared to the classic local features, such as SIFT
which is robust to scaling, rotation, affine distortion, and 4.2.5 Search operation
illumination changes, the local histogram has its limitation. Rq ← Search(C, Idx, TD). Upon receiving the trapdoor,
Some researchers proposed to outsource the calculation of i.e., the encrypted query image, the cloud server divides
SIFT features in a privacy-preserving manner [38], [39], [42]. it into blocks and calculates the set of local histograms
blknum
However, these methods need several rounds of commu- {hqj }j=1 q , where blknumq is the total number of blocks
nications between non-collusive cloud servers. In addition, in the query image Iq . Next, a feature vector fq = (fqj )kj=1 is
the applications of the outsourced SIFT in secure CBIR need calculated using the vocabulary and {hqj }. The Manhattan
the additional communication between the server and query distance between fq and the feature vector fi in the index
users, which is a undesirable burden to the users.
(ii) Vocabulary generation. Cluster all the local histograms
{hij } into k classes with the k -means clustering algorithm TABLE 1: The linear index
[43]. The k cluster centers are defined to be the encrypted
Image identity Feature vector
visual words which make up the vocabulary.
(iii) Global feature extraction. After generating the vocabu- ID(C1 ) f1 = {f11 , f12 , ...f1j , ..., f1k }
lary, all the local histograms in an image are represented by ... ...
their nearest visual words. Then, the occurrence histogram ID(Ci ) fi = {fi1 , fi2 , ...fij , ..., fik }
of the visual words are calculated and normalized to repre-
... ...
sent the image. As a result, each image is represented by a
k ID(Cn ) fn = {fn1 , fn2 , ...fnj , ..., fnk }
feature vector f = (fi )i=1 . Finally, the image identities and

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018
can be directly calculated. Finally, the θ most similar images
are returned to the image owner as the search results.
4.2.6 Image decryption
Iq ← ImgDec(Rq , K, blksize). The decryption process is an
exact reverse to the encryption process. Firstly, the intra-
block pixel permutation is reversed. Secondly, the block per-
mutation is reversed. Finally, the color value permutation is
recovered. Here, we do not describe the decryption process
in detail because it is quite straightforward.
4.2.7 Image update
Idx ← ImgUpdate(Idx, ID, TD, updatetype). When the
owner needs to add an image, he encrypts the image and
submits the encrypted image and the ID to the cloud server.
The server generates the corresponding feature vector as the
trapdoor generation process and adds the pair of image ID
and feature vector to the index. When an image needs to be
deleted, the image owner just notices the server to delete the
encrypted image and the corresponding item in the index.
5 S ECURITY ANALYSIS
As stated in subsection 3.2, we consider an honest-but-
curious cloud server which follows the protocol correctly,
but may analyze the owner’s data to acquire privacy in-
formation such as the content of encrypted images and the
image features. The security of the scheme is analyzed under
the ciphertext-only attack (COA), known-background attack
(KBA), and chosen-plaintext attack (CPA) models.
5.1 Security under COA model
In the COA model, the adversary is only able to get the
ciphertext. For formal statement, the functionality F and the
corresponding information leakages of our scheme under
the COA model are summarized in Fig. 3. Our security
proofs follow the paradigm in secure multi-party com-
putations [44]. The execution of our scheme involves the
interaction between cloud server and image owner, which is
defined as the real experiment. In the proposed scheme, the
honest-but-curious cloud server is the potential adversary
A. In an ideal experiment, a simulator S is defined as the
one that can simulate the view of adversary A by using the
functionality F only, constructing the ideal experiment. The
proposed scheme is proved secure once the two experiments
are indistinguishable.
Theorem 1. Our scheme is secure against an honest-but-
curious probabilistic polynomial time (PPT) adversary under
the COA model. The security strength depends on the image
size, block size, and the number of permutations in color value
encryption.
Proof.
• Security of image content. The simulator S simulates
a set of images I S and the corresponding identity
set IDS according to the storage leakage as shown
in Fig. 3. The simulator S knows the total number
of images and the size of each image. However, S
can only fill the images with the randomly generated
pixels. As stated in subsection 4.2, images consist
1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
6
of the color information and texture information,
both of which are contained in all H, S, V compo-
nents. In our scheme, this information is protect-
ed respectively by the substitutions and random
permutations. Specifically, the color information in
H, S, V components are protected by color value
substitution with different secret keys. The color
information between the real images and simulated
ones are indistinguishable according to the property
of random permutation. For a random permutation
with length of 101, the computational complexity of
a distinguisher D, executed by S , in distinguishing
the color values is 101! because D needs to figure
out the correct one from 101! permutations, which
means a log2 (101!) ≈ 531 bits security strength. The
texture information is protected by block permuta-
tion and intra-block pixel permutations. The secu-
rity strengths of block permutation and intra-block
pixel permutations are equal to log2 (blknum!) and
log2 (blksize!) bits, respectively. The image content
is made up of all of these information and hence
the security strength of image encryption (SecImg )
in our scheme can be calculated as:
SecImg =3 × Npmt × log2 (101!)
+log2 (blknum!) (4)
+3 × blknum × log2 (blksize!)(bits)
• Security of features. In our scheme, image features are
calculated from the local histograms of encrypted
color values. In addition, the adversary can obtain
the encrypted global histogram of an image if he
wants. With a simulated image I S , S can calculate
the local and global histograms of the simulated im-
age. The computational complexity of a distinguisher
D in distinguishing the histogram is 3 × Npmt × 101!
which means a 3 × Npmt × 531 bits security strength.
• Security of query image. In our scheme, query image
is encrypted in the same way as images on cloud
server. Thus the security strength equals to SecImg .
In this scheme, the images are encrypted by the combi-
nation of value substitution, block permutation, and intra-
block pixel permutation. The image can be well protected,
although the security is related to the image size. Despite
the proposed image encryption method cannot be the best
one in terms of security, we would like to point that our en-
cryption method is designed to support the direct extraction
of features from encrypted images for image retrieval.
5.2 Security under KBA model
In the KBA model, besides the information leakage de-
fined in Fig. 3, the adversary also knows certain statistical
properties of natural images, which degrades the security
strength of the proposed scheme. For example, as illustrated
in the first subfigure of Fig. 4, color values do not occur
uniformly, and the adjacent color values have the similar
occurrence probabilities. After the encryption with a single
permutation, the histogram bins have been shuffled, which
protects the image statistical features to some extent. How-
ever, the distribution statistics are still reserved as shown in
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 7

The ideal functionality F of our scheme as well as information leakages.

(i) F .StoreImage(I, ID, K, blksize):
• Functionality. Image owner encrypts all the images in I , and generates a set of encrypted images C . Next,
Image owner uploads C, ID, blksize to the cloud server.
• Storage leakage. The information leaked here includes C, ID, blksize, the size of each images, and the total
number of images.
(ii) F .IndexGen(C, blksize):
• Functionality. Cloud server extracts local histograms from images blocks, and constructs the vocabulary by
cluster algorithm, and calculates a feature vector for each image in C .
• Feature leakage. The information leaked here includes the encrypted local histograms, the similarities between
the local histograms, and the distributions of local histograms. What’s more, the cloud server can calculate the
encrypted global histogram of image if he wants.
(iii) F .Query(Iq ):
• Functionality. Image owner encrypts the query image, and submits the encrypted image to cloud server as
query trapdoor. After searching on the index, the cloud server returns the most similar images to the image
owner.
• Query leakage. The information leaked here includes the encrypted query image and the similarity between
the images in the database.

Fig. 3: The functionality F and information leakage in our framework

the second subfigure of Fig. 4, which decreases the compu-
tational complexity to figure out the secret permutation. In
our scheme, the polyalphabetic cipher is utilized to encrypt
the pixel values, which will flatten the color value histogram
of the encrypted image and thus offering stronger security
[40], [41]. In this paper, we calculate the histogram entropies
of an original image and its encrypted versions generated
using different number of permutations Npmt as shown in
Fig. 5. The higher entropy is received with larger Npmt ,
which indicates higher uniformity, i.e., better security [40],
[41]. However, there is a trade-off between the security and
retrieval accuracy, which will be shown in next section.
5.3 Security under CPA model
In the CPA model, besides the information leakage defined
in Fig. 3, the adversary can obtain the ciphertexts of ar-
bitrary plaintexts. In fact, all the permutation-only image
ciphers can be completely broken by the chosen-plaintext
attack if images are encrypted by the same secret key
[45]. But in our scheme, the images are encrypted by the
combination of position permutation and pixel substitution.
The secret keys of position permutation are distinct for each
image. In addition, the pixel position is not shuffled by
rows and columns which significantly degrades the security
[45]. Thus, the images encrypted in our scheme cannot be
recovered even under the CPA model. However, in order to
support the image retrieval, the secret key of the color value
substitution is the same for all images, and this secret key
is unsafe under the CPA model [46]. This will cause infor-
mation leakage about image content. With the secret key of
color value substitution, the cloud server can generate the
valid trapdoor for the query image chosen by itself. Then,
the cloud server can search the encrypted image database
with the trapdoor, and all the retrieved images may contain
the similar content with the chosen query image. In this way,
the cloud server can figure out the image content to some
extent.
5.4 Access pattern
Access pattern is a high-lever privacy compared with the
data confidentiality. The consequences caused by access pat-
tern leakage depend on the application scenario. In an image
retrieval scheme, the cloud server knows which images
are retrieved by the image owner by analyzing the access
pattern. Generally, the researchers do not consider the access
pattern leakage in the design of an SSE scheme [3], [4], [9],
[12], [13], [14], [15], [16], [17], [18], [19], [21], [22], [24], [25],
[29], [30], [31].
Oblivious RAM techniques have been researched for
many years to deal with access pattern leakage, but usually
suffers to huge computation and communication burdens
[47], [48]. It is possible to incorporate Oblivious RAM tech-
niques into the SE scheme to mitigate the access pattern
leakage. But the computation and communication burdens
are big problems for the image retrieval.
6 E XPERIMENTAL RESULTS
This section evaluates the performance of the proposed
scheme in terms of encryption effectiveness, retrieval ac-
curacy, and efficiency. We implement the proposed scheme
with MatLab 2017a on a Win-10 operation system with Intel
Core (TM) i7-6900K CPU 3.20 GHz and 64 GB memory.
Inria Holidays database [49] is used as the experiment
database which contains 1491 color images (2.65 GB) in
500 categories. Inria Holidays database provides a Python
evaluation package to calculate mAP and was used in
privacy-preserving image retrieval schemes [4], [26], which
facilitates performance comparison. For clarity, we symbol-
ize several important parameters as shown in Table 2.
6.1 Effectiveness of image encryption
In our scheme, the images are encrypted by color value sub-
stitution, block permutation, and intra-block pixel permuta-
tion. Fig. 6 illustrates the separate and joint effect of the three

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 8

8.2

Original image 8
0.1
Ratio

7.8
0.05

Entropy
0 7.6
0 50 100 150 200 250 300
7.4
Npmt=1
0.1 7.2
Ratio

0.05 7
0
0 50 100 150 200 250 300 6.8
1 2 3 4 5 10 20 50 100 200 500
Npmt=5 N
pmt
0.1
Ratio

0.05
0 Fig. 5: Entropies of global histograms of encrypted images
0 50 100 150 200 250 300 using different Npmt .
Npmt=10
0.1 TABLE 2: Symbol of parameters
Ratio

0.05
0 Parameters Symbol
0 50 100 150 200 250 300
Size of block blksize
Npmt=20
0.1 Number of cluster centers k
Ratio

0.05 Total number of local histograms blknumT

0
0 50 100 150 200 250 300 Number of used local histograms in clustering blknumU
Npmt=50 Number of permutations in color value substitution Npmt
0.1
Ratio

0.05
0
0 50 100 150 200 250 300 6.2 Retrieval accuracy
Npmt=100 In our experiment, mean average precision (mAP) is used to
0.1 measure the retrieval accuracy. The Python evaluation pack-
Ratio

0.05 age of Inria Holidays Dataset is directly used to calculate the

0
0 50 100 150 200 250 300 mAP of the proposed method for fair comparison. Firstly,
Npmt=500 we use only one permutation in color value encryption,
0.1 i.e., Npmt = 1, to determine the proper parameters such as
Ratio

0.05 blksize and k . Next, we test our scheme with different Npmt
0 as the larger Npmt can protect the image content better but
0 50 100 150 200 250 300
may decrease the retrieval accuracy.
Values in HSV components

6.2.1 Accuracy with one permutation in color value encryp-

Fig. 4: Occurrence ratios of color values of an original image
and its encrypted versions with different Npmt . It can be
observed that the histogram gets more flattened if the image
is encrypted with more permutations.
protecting steps. As shown in Fig. 6 (b), with a small blksize
in a big image, the intra-block pixel permutation does not
protect the image content a lot. The image is blurred, but the
content can be almost fully learned. Conversely, the block
permutation makes a good randomization to the image with
a small blksize (see Fig. 6 (c)). The color value substitution
with Npmt = 1 has fully permuted the color information
but leaves much texture information revealed. The image
content can be further blurred by increasing the number
of permutations as show in Fig. 6 (d-f). Finally, the image
content can be well protected by the combination of the
three steps (see Fig. 6 (g-h)).
tion
The retrieval accuracy of the proposed BOEW-CBIR scheme
mainly depends on two parameters, i.e., the size of the
block (blksize) and the number of cluster centres (k ). We
test our scheme under a large range of parameter settings.
As shown in Table 3, our scheme achieves the highest mAP
with blksize set to be 20 × 20 and k set to be 3000. With
setting of blksize = 20 × 20 and k = 3000, we compare
our scheme with the scheme in [4] that uses the secure
global histogram and the scheme in [26] based on partial
encryption. In addition, we implement our scheme in RGB
and YUV color space with this setting. As shown in Table 4,
the proposed scheme also performs well in YUV and RGB
color spaces. Compared to the scheme using secure global
histogram [4], our scheme increases the mAP from 0.54564
to 0.64244 in HSV color model, which is a considerable
improvement.
6.2.2 Accuracy with multiple permutations in color value
encryption
In this paper, we use multiple permutations to protect the
color values to resist the statistic attack. However, this may

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 9

TABLE 3: The mAP of our scheme with different parameter pairs

blksize
k
15 × 15 20 × 20 25 × 25 50 × 50 100 × 100 200 × 200 Average
100 0.57095 0.57713 0.57907 0.57099 0.55027 0.50554 0.55899
500 0.62097 0.62816 0.63544 0.62718 0.60810 0.56265 0.61375
1000 0.63398 0.63161 0.62967 0.63117 0.60523 0.58453 0.61937
2000 0.63473 0.62966 0.63059 0.63280 0.61689 0.57699 0.62028
3000 0.63080 0.64244 0.63613 0.63039 0.60846 0.57851 0.62112
4000 0.62960 0.63583 0.63647 0.64002 0.62398 0.58542 0.62522
5000 0.63293 0.63715 0.63510 0.63089 0.62129 0.57146 0.62147
8000 0.63170 0.63626 0.64046 0.63771 0.62205 0.59026 0.62641
Average 0.62321 0.62728 0.62787 0.62514 0.60703 0.56942 0.61333

TABLE 4: The mAPs of two previous schemes and our scheme in different color spaces

Schemes BOEW-HSV BOEW-YUV BOEW-RGB IES [4] SSE [13] reported in [4] Partial-encryption based scheme [26]
mAP 0.64244 0.62641 0.61107 0.54564 0.49075 0.5604

degrade the retrieval accuracy. Here, we set blksize = block. It takes blknumT × O(blksize) time for the
20 × 20 and k = 3000 and test the retrieval accuracy with local histogram calculation. Table 7 shows more time
different Npmt . As shown in Fig. 7, the accuracy does not is consumed with a smaller blksize to extract local
decrease a lot when we increase Npmt up to 50. histogram from the whole Inria holiday database.
• Time consumption of local histogram clustering. In our
6.3 Efficiency implementation, the built-in k -means function in
In this section, we present the time consumptions of image MatLab is used to cluster the local histograms. The
encryption, index generation, and search operation. time cost of the k -means algorithm depends on the
predefined number of class k , the number of used
6.3.1 Time consumption of image encryption local features blknumU , and the number of itera-
In the proposed scheme, it takes O(imgsize) time for color tions t. The time complexity of k -means algorithm
value substitution, O(blknum) for block permutation, and is O(k × blknumU × t). In our experiment, we set
blknum × O(blksize) for intra-block pixel permutation to the maximal iteration times to be 300. When we set
encrypt an image. In this paper, we separately test the blksize to be 20 × 20, it generates 17,099,758 local
time consumptions of color value substitution with different histograms from Inria Holiday database. Our scheme
Npmt and time consumptions of pixel position permutation runs out of memory (64GB) when we use all of these
(including block and intra-block permutations) with differ- local histograms. Thus, in this experiment, we only
ent blksize, which are listed in Table 5 and 6. As listed in use a part of local histograms to generate the cluster
Table 5, it does not increase time consumption a lot to use centers.
multiple color value permutations, but as listed in Table 6, At first, we set a similar blknumU for different
more time is consumed with smaller blksize in the position blksize so as to figure out proper parameters blksize
permutations. The image encryption is very time consum- and k that obtain good retrieval accuracies. In Ta-
ing. However, the proposed image encryption method is ble 8, we list the total number of local histograms
much more efficient than some standard encryption meth- blknumT when different blksize are set. We also list
ods. For example, it takes 548.29s to encrypt the first image the number of used local histograms blknumU in
(3942KB) in Inria Holidays database using DES. Note that clustering. Finally, the time consumption of cluster-
the previous outsourced CBIR schemes [13], [14], [15], [17], ing with different blksize and k are given in Table
[18], [19], [21], [22], [24], [25] need to encrypt images and 8. Next, the influence of blknumU on our scheme
upload encrypted ones to the cloud server, which is also is tested when blksize is set to 20 × 20 as shown
quite time-consuming. in Table 9. Table 8 and 9 shows that more time
is consumed with larger blknumU and k . Table 9
6.3.2 Time consumption of index generation shows that the retrieval accuracy does not decrease
In the proposed scheme, feature generation includes three a lot even though a very small part (1/5000) of local
steps, local histogram calculation, clustering of local his- histograms are used for clustering.
tograms, and generation of feature vectors. The time con- • Time consumption of feature generation. After histogram
sumptions of these steps are separately tested. clustering, a normalized histogram of the encrypted
words is calculated from each image as the feature
• Time consumption of local histogram calculation. In the vector. The time complexity of feature vector calcula-
proposed scheme, the cloud server divides the im- tion is O(blknumT ×k) for the entire image database.
ages into blocks and extracts a histogram from each

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 10

TABLE 5: Time consumptions in color value substitution with different Npmt

Npmt 1 2 3 4 5 10 20 50 100 200 500

Time consumption (s) 5390.60 5478.60 5412.40 5387.60 5369.20 5370.70 5375.90 5424.60 5463.50 5643.00 5889.80

TABLE 6: Time consumptions in block and intra-block permutations with different blksize

blksize 15 × 15 20 × 20 25 × 25 50 × 50 100 × 100 200 × 200

Time consumption (s) 5437.90 3869.40 3149.80 2185.80 1949.90 1796.60

TABLE 7: Time consumptions in local histogram calculation

blksize 15 × 15 20 × 20 25 × 25 50 × 50 100 × 100 200 × 200

Time consumption (s) 618.40 551.41 467.43 383.58 350.84 332.51

TABLE 8: Time consumptions in the clustering with different blksize and k (s)

blksize
15 × 15 20 × 20 25 × 25 50 × 50 100 × 100 200 × 200
blknumT 30406710 17099758 10887924 2707247 671176 158828
blknumU 76844 171252 170763 169546 167916 158828
blknumU/blknumT 1/400 1/100 1/64 1/16 1/4 1
100 88.11 226.64 445.03 310.68 447.44 415.88
500 219.19 659.61 880.04 707.60 597.67 559.11
1000 231.34 872.96 862.16 836.25 859.65 791.93
2000 306.92 1158.16 1095.38 1119.35 1087.68 988.78
k
3000 329.95 1431.94 1421.01 1359.65 1348.08 1252.13
4000 382.25 1684.34 1573.37 1599.92 1570.11 1500.51
5000 431.61 1915.47 1904.63 1843.33 1806.84 1694.72
8000 576.66 2625.00 2591.51 2538.60 2523.75 2362.38

The time consumption of global feature generation
for Inria Holiday dataset with different parameters
are listed in Table 10.
6.3.3 Time consumption of searching
With the linear index, it needs to search on the whole index
to find the θ most similar images. The time complexity of
search is O(k × n). In our scheme, the length of feature
vector equals to the number of cluster center k . Table 11
lists the searching time of our schemes with different k . The
results are averaged from 500 queries.
6.4 Discussion
The proposed scheme is robust to parameters. It achieves
a good retrieval accuracy as long as we choose blksize in
the range of 15 × 15 to 100 × 100 and k in range [500,
8000]. In addition, our scheme can get a satisfying retrieval
accuracy even though a very small part of local histograms
are used for clustering. Compared to the previous privacy-
preserving CBIR schemes [13], [14], [15], [17], [18], [19], [21],
[22], [24], [25], our scheme outsources the index construction
to the cloud server, which further relieves the image owner’s
burden. Compared to the scheme using the global histogram
[3], [4], our scheme improve the mAP from 0.56544 to
0.64244. It is a considerable improvement.
The proposed scheme can be improved from several
aspects. Firstly, it is possible to extract better local features
under the proposed BOEW model. It depends on how the
intra-block pixels are processed and encrypted. For exam-
ple, one may extract and encrypt the gradient information
from image blocks as local features. The utilization of texture
information may achieve better retrieval accuracy. However,
the information leakage is a problem to solve. Secondly,
our scheme is constructed on the basis of uncompressed
images. Storage consumption could be a practical problem.
According to [3], [4], the encrypted image can also be
compressed to reduce the storage cost despite the correla-
tion among the pixels is destroyed. We do not discuss the
image compression a lot in this paper. Finally, it could be a
meaningful work to apply the BOEW model to the retrieval
of encrypted JPEG images.
7 C ONCLUSIONS
In this paper, a novel privacy-preserving CBIR scheme is
proposed. A novel bag-of-encrypted-words (BOEW) model
is designed to achieve a good retrieval accuracy. As a case
study, we protect the image content by color value substitu-
tion, block permutation, and intra-block pixel permutation.
Local histograms are calculated as local features. k -means
algorithm is utilized to generate encrypted visual words.
The histogram of the visual words is calculated to represent
the image. The similarity between images can be directly
measured by the Manhattan distance between feature vec-
tors on the cloud server side. Besides the search operation,

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 11

TABLE 9: Performance of the proposed scheme with different number of used local histogram, blksize = 20 × 20,
blknumT = 17, 099, 758

blknumU/blknumT 1/100 1/200 1/500 1/1000 1/2000 1/5000

Time consumption in clustering (s) 1381.84 407.86 136.49 68.44 39.82 20.71
mAP 0.64244 0.63902 0.6282 0.63519 0.63254 0.62967

TABLE 10: Time consumptions in the global feature generation (s)

blksize
15 × 15 20 × 20 25 × 25 50 × 50 100 × 100 200 × 200
100 301.24 109.66 67.97 23.05 14.79 11.13
500 735.27 343.93 214.13 61.94 23.32 15.15
1000 1210.27 636.99 401.51 108.95 36.44 17.85
2000 2196.60 1170.73 768.25 202.83 61.68 25.40
k
3000 3209.22 1733.15 1110.68 294.43 87.06 33.98
4000 4207.75 2299.12 1466.80 388.36 112.16 42.75
5000 5204.52 2858.70 1826.36 483.31 135.73 50.20
8000 8284.66 4546.23 2906.85 751.16 211.83 74.05

TABLE 11: Time consumptions of searching at the server side with linear index (s)

k 100 500 1000 2000 3000 4000 5000 8000

Time 0.0044 0.0087 0.0119 0.0211 0.0300 0.0381 0.0464 0.0704

the index construction in our scheme can be also outsourced
to the cloud server.
The proposed scheme can be further improved. Firstly,
it could be a meaningful future work to design better
local descriptors under our BOEW model. Secondly, how to
protect the image content under CPA model needs further
studies. Finally, it could be interesting to apply the BOEW
model to JPEG images.
ACKNOWLEDGMENTS
This work is supported in part by the National Nat-
ural Science Foundation of China under grant num-
bers 61672294, 61502242, 61702276, U1536206, U1405254,
61772283, 61602253, 61601236, and 61572258, in part by
Six peak talent project of Jiangsu Province (R2016L13), in
part by National Key R&D Program of China under grant
2018YFB1003205, in part by NRF-2016R1D1A1B03933294,
in part by the Jiangsu Basic Research Programs-Natural
Science Foundation under grant numbers BK20150925 and
BK20151530, in part by the Priority Academic Program
Development of Jiangsu Higher Education Institutions (PA-
PD) fund, in part by the Collaborative Innovation Center
of Atmospheric Environment and Equipment Technology
(CICAEET) fund, China. Zhihua Xia is supported by BK21+
program from the Ministry of Education of Korea.
R EFERENCES
[1] J. M. Lewin, R. E. Hendrick, C. J. D’Orsi, P. K. Isaacs, L. J.
Moss, A. Karellas, G. A. Sisney, C. C. Kuni, and G. R. Cutter,
“Comparison of full-field digital mammography with screen-film
mammography for cancer detection: results of 4,945 paired exam-
inations.” Radiology, vol. 218, no. 3, pp. 873–80, 2001.
[2] C. S. Lu, “Homomorphic encryption-based secure sift for privacy-
preserving feature extraction,” Proceedings of SPIE The International
Society for Optical Engineering, vol. 7880, no. 2, pp. 788 005–17, 2011.
[3] B. Ferreira, J. Rodrigues, J. Leitão, and H. Domingos, “Privacy-
preserving content-based image retrieval in the cloud,” in IEEE
34th Symposium on Reliable Distributed Systems. IEEE, 2015, pp.
11–20.
[4] B. Ferreira, J. Rodrigues, J. Leitao, and H. Domingos, “Practical
privacy-preserving content-based retrieval in cloud image reposi-
tories,” IEEE Transactions on Cloud Computing, vol. PP, no. 99, pp.
1–1, 2017.
[5] Y. Rui, T. S. Huang, M. Ortega, and S. Mehrotra, “Relevance feed-
back: a power tool for interactive content-based image retrieval,”
IEEE Transactions on Circuits and Systems for Video Technology, vol. 8,
no. 5, pp. 644–655, 1998.
[6] Y. Liu, D. Zhang, G. Lu, and W.-Y. Ma, “A survey of content-based
image retrieval with high-level semantics,” Pattern Recognition,
vol. 40, no. 1, pp. 262–282, 2007.
[7] C. B. Akgül, D. L. Rubin, S. Napel, C. F. Beaulieu, H. Greenspan,
and B. Acar, “Content-based image retrieval in radiology: current
status and future directions,” Journal of Digital Imaging, vol. 24,
no. 2, pp. 208–222, 2011.
[8] X. Zhang, W. Liu, M. Dundar, S. Badve, and S. Zhang, “Towards
large-scale histopathological image analysis: Hashing-based im-
age retrieval,” IEEE Transactions on Medical Imaging, vol. 34, no. 2,
pp. 496–506, 2015.
[9] R. Curtmola, J. Garay, S. Kamara, and R. Ostrovsky, “Searchable
symmetric encryption: Improved definitions and efficient con-
structions,” Journal of Computer Security, vol. 19, no. 5, pp. 79–88,
2011.
[10] J. Li, Q. Wang, C. Wang, N. Cao, K. Ren, and W. Lou, “Fuzzy
keyword search over encrypted data in cloud computing,” in 2010
Proceedings IEEE INFOCOM. IEEE, 2010, pp. 1–5.
[11] N. Cao, C. Wang, M. Li, K. Ren, and W. Lou, “Privacy-preserving
multi-keyword ranked search over encrypted cloud data,” IEEE
Transactions on parallel and distributed systems, vol. 25, no. 1, pp.
222–233, 2013.
[12] Z. Xia, X. Wang, X. Sun, and Q. Wang, “A secure and dynamic
multi-keyword ranked search scheme over encrypted cloud data,”
IEEE Transactions on Parallel and Distributed Systems, vol. 27, no. 2,
pp. 340–352, 2016.
[13] W. Lu, A. Swaminathan, A. L. Varna, and M. Wu, “Enabling search

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018
(a) (b)
(c) (d)
(e) (f)
(g) (h)
Fig. 6: The visual effect of encryption with blksize =
20 × 20, (a) the original image (100301.jpg in Inria Holidays
database), the size of which is 3264 × 2448, (b) with intra-
block pixel permutation only, (c) with block permutation
only, (d) with color value substitution only under Npmt = 1,
(e) with color value substitution under Npmt = 5, (f) with
color value substitution under Npmt = 20, (g) with the
combination of the three steps under Npmt = 1, and (h)
with the combination of the three steps under Npmt = 20.
over encrypted multimedia databases,” Proceedings of SPIE The
International Society for Optical Engineering, vol. 7254, pp. 725 418–
11, 2009.
[14] W. Lu, A. L. Varna, A. Swaminathan, and M. Wu, “Secure image
retrieval through feature protection,” in IEEE International Confer-
ence on Acoustics, 2009, pp. 1533–1536.
[15] W. Lu, A. L. Varna, and M. Wu, “Confidentiality-preserving image
search: A comparative study between homomorphic encryption
and distance-preserving randomization,” IEEE Access, vol. 2, pp.
125–141, 2014.
[16] X. Yuan, X. Wang, C. Wang, A. Squicciarini, and K. Ren, “Enabling
1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
12
0.65
0.6
mAP
0.55
0.5
0.45
1 2 3 4 5 10 20
Npmt
Fig. 7: The mAPs with different Npmt
privacy-preserving image-centric social discovery,” in Distributed
Computing Systems (ICDCS), 2014 IEEE 34th International Conference
on. IEEE, 2014, pp. 198–207.
[17] Z. Xia, Y. Zhu, X. Sun, Z. Qin, and K. Ren, “Towards privacy-
preserving content-based image retrieval in cloud computing,”
IEEE Transactions on Cloud Computing, vol. 6, no. 1, pp. 276–286,
2018.
[18] J. Yuan, S. Yu, and L. Guo, “Seisa: Secure and efficient encrypted
image search with access control,” in 2015 IEEE Conference on
Computer Communications (INFOCOM). IEEE, 2015, pp. 2083–2091.
[19] H. Cheng, X. Zhang, J. Yu, and F. Li, “Markov process-based re-
trieval for encrypted jpeg images,” EURASIP Journal on Information
Security, vol. 2016, no. 1, pp. 1–9, 2016.
[20] L. Weng, L. Amsaleg, A. Morton, and S. Marchand-Maillet, “A
privacy-preserving framework for large-scale content-based infor-
mation retrieval,” IEEE Transactions on Information Forensics and
Security, vol. 10, no. 1, pp. 152–167, Jan 2015.
[21] L. Weng, L. Amsaleg, and T. Furon, “Privacy-preserving out-
sourced media search,” IEEE Transactions on Knowledge and Data
Engineering, vol. 28, no. 10, pp. 2738–2751, 2016.
[22] Z. Xia, X. Wang, L. Zhang, Z. Qin, X. Sun, and K. Ren, “A privacy-
preserving and copy-deterrence content-based image retrieval
scheme in cloud computing,” IEEE Transactions on Information
Forensics and Security, vol. 11, no. 11, pp. 2594–2608, 2016.
[23] L. Zhang, T. Jung, K. Liu, X. Y. Li, X. Ding, J. Gu, and Y. Liu, “Pic:
Enable large-scale privacy preserving content-based image search
on cloud,” IEEE Transactions on Parallel and Distributed Systems,
vol. 28, no. 11, pp. 3258–3271, Nov 2017.
[24] R. Bellafqira, G. Coatrieux, D. Bouslimi, and G. Quellec, “Content-
based image retrieval in homomorphic encryption domain,” in
2015 37th Annual International Conference of the IEEE Engineering in
Medicine and Biology Society (EMBC). IEEE, 2015, pp. 2944–2947.
[25] ——, “An end to end secure cbir over encrypted medical
database,” in Engineering in Medicine and Biology Society (EMBC),
2016 IEEE 38th Annual International Conference of the. IEEE, 2016,
pp. 2537–2540.
[26] Y. Xu, J. Gong, L. Xiong, Z. Xu, J. Wang, and
Y. qing Shi, “A privacy-preserving content-based image
retrieval method in cloud environment,” Journal of Visual
Communication and Image Representation, vol. 43, pp. 164 –
172, 2017. [Online]. Available: http://www.sciencedirect.com/
science/article/pii/S104732031730007X
[27] D. Liu, J. Shen, Z. Xia, and X. Sun, “A content-based image
retrieval scheme using an encrypted difference histogram in cloud
computing,” Information, vol. 8, no. 3, 2017. [Online]. Available:
http://www.mdpi.com/2078-2489/8/3/96
[28] T. Hoang, A. A. Yavuz, and J. G. Merchan, “A secure searchable
encryption framework for privacy-critical cloud storage services,”
IEEE Transactions on Services Computing, 2019.
[29] M. Kuzu, M. S. Islam, and M. Kantarcioglu, “Efficient similarity
search over encrypted data,” in IEEE International Conference on
Data Engineering, 2012, pp. 1156–1167.
[30] F. Hahn and F. Kerschbaum, “Searchable encryption with secure
and efficient updates,” in ACM Sigsac Conference on Computer and
Communications Security, 2014, pp. 310–320.
[31] L. Weng, L. Amsaleg, A. Morton, and S. Marchand-Maillet, “A
privacy-preserving framework for large-scale content-based in-
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 13

formation retrieval,” IEEE Transactions on Information Forensics & Leqi Jiang is a Ph.D. student in School of
Security, vol. 10, no. 1, pp. 152–167, 2015. Computer and Software, Nanjing University of
[32] J. Zhang, M. Marszałek, S. Lazebnik, and C. Schmid, “Local fea- Information Science & Technology. He received
tures and kernels for classification of texture and object categories: his M.S. degree in Software Engineering from
A comprehensive study,” International Journal of Computer Vision, Nanchang Hangkong University in 2016. His re-
vol. 73, no. 2, pp. 213–238, 2007. search interests include encrypted image pro-
[33] D. Nister and H. Stewenius, “Scalable recognition with a vocab- cessing, data security in cloud.
ulary tree,” in Computer vision and pattern recognition, 2006 IEEE
computer society conference on, vol. 2. Ieee, 2006, pp. 2161–2168.
[34] Z. Wu, Q. Ke, J. Sun, and H.-Y. Shum, “A multi-sample, multi-
tree approach to bag-of-words image representation for image
retrieval,” in IEEE 12th International Conference on Computer Vision.
IEEE, 2009, pp. 1992–1999.
[35] J. Wang, Y. Li, Y. Zhang, C. Wang, H. Xie, G. Chen, and X. Gao,
“Bag-of-features based medical image retrieval via multiple as-
signment and visual words weighting.” IEEE Transactions on Med-
ical Imaging, vol. 30, no. 11, pp. 1996–2011, 2011.
[36] D. G. Lowe, “Distinctive image features from scale-invariant key-
points,” International Journal of Computer Vision, vol. 60, no. 60, pp.
Dandan Liu received her BS in network engi-
91–110, 2004.
neering from Nanjing University of Information
[37] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust
Science & Technology in 2015, China. She is
features (surf),” Computer vision and image understanding, vol. 110,
currently pursuing her MS in computer science
no. 3, pp. 346–359, 2008.
and technology at the College of Computer and
[38] Z. Qin, J. Yan, K. Ren, C. W. Chen, and C. Wang, “Towards efficient
Software, in Nanjing University of Information
privacy-preserving image feature extraction in cloud computing,”
Science & Technology, China. Her research in-
in Proceedings of the 22nd ACM International Conference on Multime-
terest is encrypted image retrieval.
dia. ACM, 2014, pp. 497–506.
[39] ——, “Secsift: Secure image sift feature extraction in cloud com-
puting,” ACM Transactions on Multimedia Computing, Communica-
tions, and Applications (TOMM), vol. 12, no. 4s, p. 65, 2016.
[40] G. Chen, Y. Mao, and C. K. Chui, “A symmetric image encryption
scheme based on 3d chaotic cat maps,” Chaos, Solitons & Fractals,
vol. 21, no. 3, pp. 749–761, 2004.
[41] X. Chai, Y. Chen, and L. Broyde, “A novel chaos-based image
encryption algorithm using dna sequence operations,” Optics and
Lasers in engineering, vol. 88, pp. 197–213, 2017.
[42] Z. Qin, J. Yan, and K. Ren, “Private image computation: The case of
cloud based privacy-preserving sift,” in Computer Communications Lihua Lu received her BS in network engineer-
Workshops (INFOCOM WKSHPS), 2014 IEEE Conference on. IEEE, ing from Nanjing University of Information Sci-
2014, pp. 179–180. ence & Technology in 2016, China. She is cur-
[43] K. Krishna and M. N. Murty, “Genetic k-means algorithm,” IEEE rently pursuing her MS in computer science and
Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), technology at the College of Computer and Soft-
vol. 29, no. 3, pp. 433–439, 1999. ware, in Nanjing University of Information Sci-
[44] R. Canetti, “Universally composable security: A new paradigm for ence & Technology, China. Her research interest
cryptographic protocols,” in 42nd IEEE Symposium on Foundations is encrypted image retrieval.
of Computer Science. IEEE, 2001, pp. 136–145.
[45] A. Jolfaei, X.-W. Wu, and V. Muthukkumarasamy, “On the security
of permutation-only image encryption schemes,” IEEE Transaction-
s on Information Forensics and Security, vol. 11, no. 2, pp. 235–246,
2016.
[46] A. Clark and E. Dawson, “A parallel genetic algorithm for crypt-
analysis of the polyalphabetic substitution cipher,” Cryptologia,
vol. 21, no. 2, pp. 129–138, 1997.
[47] B. Pinkas and T. Reinman, “Oblivious ram revisited,” in Annual
Cryptology Conference. Springer, 2010, pp. 502–519.
[48] E. Shi, T.-H. H. Chan, E. Stefanov, and M. Li, “Oblivious ram
Byeungwoo Jeon received the B.S. degree
with o ((logn) 3) worst-case cost,” in International Conference on
(Magna Cum Laude) in 1985, the M.S. degree
The Theory and Application of Cryptology and Information Security.
in 1987 in electronics engineering from Seoul
Springer, 2011, pp. 197–214.
National University, Seoul, Korea, and the Ph.D.
[49] H. Jegou, M. Douze, and C. Schmid, “Hamming embedding
degree in electrical engineering from Purdue U-
and weak geometric consistency for large scale image search,”
niversity, West Lafayette, in 1992. Since Septem-
Computer Vision–ECCV 2008, pp. 304–317, 2008.
ber 1997, he has been with the faculty of the
School of Electronic and Electrical Engineer-
ing, Sungkyunkwan University, Korea, where he
is currently a full Professor. He has authored
many papers in the areas of video compression,
prepost processing, and pattern recognition. His research interests in-
Zhihua Xia received the BS degree in Hunan
clude multimedia signal processing, video compression, statistical pat-
City University, China and PhD degree in com-
tern recognition, and remote sensing.
puter science and technology from Hunan Uni-
versity, China, in 2006 and 2011, respectively.
He works as an associate professor in School
of Computer & Software, Nanjing University of
Information Science & Technology. His research
interests include digital forensic and encrypted
image processing. He is a member of the IEEE.

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.