18th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing
Censoring Sensitive Data From Images
Ştefan Postăvaru
University of Bucharest
Bitdefender
Bucharest, Romania
spostavaru@bitdefender.com
Abstract—In the recent years, the vast volume of digital
images available enabled a large range of learning methods to
be applicable, while making human input obsolete for many
tasks. In this paper, we are addressing the problem of removing
private information from images. When confronted with a
relatively big number of pictures to be made public, one may
find the task of manual editing out sensitive regions to be
unfeasible. Ideally, we would like to use a machine learning
approach to automate this task. We implement and compare
different architectures based on convolutional neural networks,
with generative and discriminative models competing in an
adversarial fashion.
Keywords-image censoring; convolutional neural networks;
generative adversarial network; autoencoder;
I. I NTRODUCTION
In this paper we propose several architectures with dif-
ferent training approaches, in order to solve the task of
removing private information from images. Our goal is to
create reproductions that do not contain data considered
”sensitive” (such as the person’s name, location identifiers,
private belongings), but preserve as much information as
possible from the original images. We will refer the process
of removing data from pictures in order to hide a certain
aspect, as image anonymization.
In the past few years, there have been multiple methods
proposed for image inpainting [1],[8] using a generative and
a discriminative model in an adversarial architecture. We
adapt this approach and create models for greyscale image
anonymisation, improving upon the ideas presented in [3].
In addition, we also create new models for color images
label removal.
A blind inpainting setup is characterized by the lack of
spatial information of the data that should be removed.
We first try to model the relation between annotated and
clean images with a convolutional denoising autoencoder
(DAE), trained on aligned examples. Its prediction is further
enhanced by training a different model to separate patches
with text from clean ones, effectively transiting to a non-
blind inpainting problem. Regions without labels are copied
from the source image, while the rest are taken from
the DAE prediction. We then proceed with an adversarial
architecture for color images anonymization, where a dis-
criminator learns to distinguish between generated images
2470-881X/16 $31.00 © 2016 IEEE
DOI 10.1109/SYNASC.2016.67
Ionuţ-Mihăiţă Pleşea
mihai.plesea@gmail.com
coming from annotated and clean sources, while a generator
tries to increase the error of the discriminator. Finally, for
the non-blind inpainting case with adversarial networks, we
test the same architecture presented in [12] on a common
dataset, for comparison purposes.
II. R ELATED W ORK
The task of image anonymisation can be formalized as
having the original data X, a dependent variable S, find a
representation R such that S cannot be deduced from R [3].
In this situation, the objective function to be minimized is:
L = αC(X, R) + βD(R, S), (1)
where C(·) represents the cost of reconstructing the orig-
inal data from its representation, and D(·) measures the
dependence between the hidden variable and the learned
representation. In this case, the representation coincide with
the input X, the task summarizing to finding the distribution
of data p(X), therefore a generative model is used.
To quantify the dependence D(·), a separate model (dis-
criminator) is trained in parallel with the generative one,
in an adversarial fashion. The discriminator learns relations
between the representation and the hidden variable S, while
the generator tries to minimize this dependence. Formally, if
the generative model parameters are θ and the discriminator
has parameters ϕ, the dependence is defined as:
Dθ,ϕ (R, S) = EX,R [S log(ADV (R)) + (1 − S) log(1 − ADV (R))] (2)
Finding the optimal parameters for both models is a min-
max problem:
min max Dθ,ϕ (R, S) (3)
θ ϕ
In addition to alternating training, in practice it is mandatory
to preserve balance between the generator and discriminator
performances. A too good critique would provide weak
gradients for the generator, while a discriminator with poor
performance would produce uninformative gradients [4]. We
incorporate the idea presented in [3] when designing our
deep adversarial architecture and extend it for color images
as well.
An alternative method is to consider the examples as
samples coming from two different distributions, p(XD1 ),
443
p(XD2 ), and make aligned pairs for training (xD1 , xD2 ).
In this case, we are interested in modeling the conditional
P (XD2 |XD1 ). We can consider images with sensitive in-
formation to come from a different distribution than ”clean”
examples, obtained by applying a stochastic corruption
process. Denoising autoencoders are generative models that
learn representations robust to partial corruptions of the input
[10], being a natural choice to model this relation. We are
re-purposing this idea to fit our method for anonymizing
images with good results.
In [1] they proposed an adversarial architecture to solve
the image inpainting problem. A denoising autoencoder is
trained to reproduce images from incomplete input, along
a discriminator that separates real from generated images,
forming a generative adversarial network (GAN [4]). This
approach requires aligned examples for training. By presum-
ing the location of empty region known, one can use the
non-blind inpainting approach as presented in [12]. Given a
trained generator G and discriminator D, the following loss
functions are defined:
Lcontextual (z) = kM ⊙ G(z) − M ⊙ Xk (4)
Lperceptual (z) = log(1 − D(G(z))) (5)
Ltotal = αLcontextual + βLperceptual (6)
Where z is a random noise vector and M a mask matrix
corresponding to the hidden region. By minimizing Ltotal
with respect to z, the reconstruction G(z) will converge to
a complete representation of the initial, partial data. Private
information removal can be viewed as a special case of
image inpainting, where the sensitive data is discarded and
the previous method is used to generate new, plausible
information.
In the next section we will motivate the chosen learning
models and present the implemented architectures.
III. A PPROACH
A. Choosing the loss function
Considering normal images as being samples from a ob-
served random variable X, annotated examples are samples
from another random variable X̃, created by a stochastic
corruption process C(X̃|X), adding random names uni-
formly over pictures. We then try to predict the conditional
P (X|X̃), i.e. reconstruct original images maximizing their
likelihood, using a DAE. Let Y be the model prediction
and θ the parameters of the model, the loss function to be
minimized is:
L(θ) = − log(P (X|X̃)) (7)
Considering each pixel color from every channel (pi )
to be independently and normally distributed, with mean
pi (Y (X̃)) and standard deviation σ, the loss function be-
comes:
X X1 is constructed from X by replacing a random pixel
L(θ) = − log N (pi (X)|pi (Yθ (X̃), σ 2 )) (8)
444
(a) Clean image (b) Labelled image
Figure 1: Randomly annotated example
1X
L(θ) ∝ (pi (X) − pi (Yθ (X̃)))2 (9)
n
So, under these considerations, choosing the mean-squared
error (MSE) as the criterion function seems reasonable.
B. Denoising Autoencoder
The training set is composed of pairs (x, x̃) where x
is a clean image and x̃ is its annotated counterpart, ob-
tained by randomly applying a name over the picture (Fig.
1). As presented in [11], data in high dimension tend to
concentrate near a lower-dimensional non-linear manifold.
Corrupted examples x̃ are naturally further away form this
manifold. Training a denoising autoencoder is equivalent
to learning to project the corrupted examples back into
the original manifold. The DAE learns on mini-batches,
with labeled pictures as input and their clean counterpart as
examples, minimizing the MSE between them with Adam
[6] optimized gradient descent (Fig. 2). Different tested
architectures, extended from the autoencoder presented by
[5], are detailed in subsection III-E.
Figure 2: General architecture of a denoising autoencoder.
The annotated inputs pass through a series of N convolu-
tional layers with 3x3 kernels, stride 1 and separated by
max-pooling layers and rectified linear unit (ReLU) transfer
layers. After 2 fully connected layers, the signal passes
through a reversed architecture of the first N convolutinal
layers. The gradient to be propagated back is computed from
the mean-squared error between the network output and the
clean counterpart of the input.
C. Patch Network
We make the observation that pixels without text (un-
labeled) are identical in both images from a pair, so
p(pi (X) = pi (X̃)|pi (X̃), i ∈ U ) = 1, where U denotes the
set of unlabeled pixels indexes. Let X be a fixed image,
with its correspondent from X̃ and Xk+1 is constructed
recursively from Xk . By making the assumption that pixels
are independently distributed, we have:
Y
k+1 Y
n ones, that are indistinguishable by a good discriminator.
p(Xk+1 |X̃) = p(pi (X)|pi (X̃), i ∈ U ) p(pi (X)|pi (X̃))
i=1 i=k+2
Y
k Y
n
p(Xk+1 |X̃) ≥ p(pi (X)|pi (X̃), i ∈ U ) p(pi (X)|pi (X̃))
i=1 i=k+1
p(Xk+1 |X̃) ≥ p(Xk |X̃)
So, an image with with more unlabeled pixels copied from
the input is as good if not better than the original prediction.
For this purpose, we trained a patch classifier on regions of
size 8x8 cropped from the training set images. The patch size
was chosen such that the text characteristics are preserved,
clean regions are filtered and the picture can be split in
equal-sized regions. In [3] they proposed the same approach,
with a filter size of 5x5 pixels. Patches classified as clean
are copied directly from the annotated image, while labeled
regions are copied from the prediction. The patch classifier
is a neural network with 4 convolutional layers and 2 fully
connected layers with sigmoidal output (Fig. 3), trained with
binary cross-entropy criterion and Adam optimization.
Figure 3: Patch Network with 3x3 convolutions and ReLU
after the second and last convolution followed by two fully
connected layers with 1024 units and a Dropout layer in
between. A final Sigmoid layer assigns probabilities for one
of the two classes: “clean” and “with text/unclean”.
Green: ReLU transfer after layer
D. Autoencoder with critique
Although in our dataset obtaining aligned images is trivial
that may not be so for real world examples. This makes
reproducing the corruption process much more difficult, thus
increasing the computational expense for generating such
examples. Ideally, we would like to train a model on a
mixed dataset, without the need to make pairs (x, x̃). As
proposed by [3], we test an approach with two models:
an autoencoder and a critique. The discriminator learns to
distinguish between clean and annotated images generated
by the autoencoder. The generator is trained to replicate
its input and also to increase the discriminator error for
annotated examples. With this approach, we aim to create
a model able to generate clean images from annotated
(10) Compared to the DAE approach, the adversarial autoencoder
can be trained on a dataset without aligned examples. The
architecture of the autoencoder is the same as the one used
(11)
for the denoising approach (Fig. 2). The discriminator is
a neural network with 3 convolutional layers and 2 fully
(12)
connected layers (Fig. 4), trained to minimize the binary
cross-entropy criterion:
1X
n
BCE(o, t) = − (ti log(oi )+(1−ti ) log(1−oi )) (13)
n i=1
In parallel, the autoencoder is trained to minimise the MSE
with respect to its input, and maximize the discriminator
error (1).
Figure 4: Adversarial architecture. The discriminator takes
the generator output and tries to predict whether the input
is an annotated picture or not, minimizing the binary cross-
entropy criterion. The autoencoder parameters are updated
based on the gradient from the MSE criterion with its input
and the negative gradient of its output with respect to the
discriminator BCE.
Red: ReLU + MaxPooling after layer
E. Models for greyscale and color images
Different architectures are used for greyscale and color
images. In the first case, the encoder is represented by a
neural network with 6 convolutional layers grouped in 3
parts separated by max-pooling layers and ReLU transfer
layers, ending with one fully connected layer. All convolu-
tions have a kernel size of 3x3 with stride 1 and padding 1
to preserve dimensions, and all max-polling have size 2x2
with stride 1. The decoder has the mirrored architecture of
the encoder (Fig. 5). The examples are reshaped to a tensor
of size (64, 64, 1).
For color images, the previous architecture was not com-
plex enough to reproduce tensors of size (64, 64, 3). We
adjusted the denoising autoencoder (Fig. 7) to increase the
layers depth faster in order to learn more regularities [7].
The training process and the results obtained are presented
in the next section.
445

Figure 5: DAE architecture for greyscale images, En-
coder+Decoder. Having an annotated image is input, it tries
to reproduce its clean output, minimizing the MSE criterion.
Green: ReLU transfer after layer
Red: ReLU + MaxPooling (Encoder)/MaxUnpooling (De-
coder) after layer
Figure 7: DAE Color architecture with Encoder + Decoder.
Every convolution is followed by a ReLU transfer layer. Af-
ter the second convolution layer we add 2x2 Pooling layers
after convolutional layers. Last layer is fully connected with
8*8*256 parameters.
Green: ReLU transfer after layer
Red: ReLU + MaxPooling (Encoder)/MaxUnpooling (De-
coder) after layer

F. Non-blind Inpainting Table I: Greyscale examples

Labeled input DAE output Patched DAE output

(a) Generator (as presented in [9])

(b) Discriminator (as presented in [9])

IV. E XPERIMENTS
For comparison purposes, we implemented the architec-
ture presented in [12] and trained a generator (6a) that pro-
duces images with faces from random noise and a discrim- We used images from 10k US Adult Faces Database [2] to
inator (6b) that distinguish between samples coming from train and test our models. After the pictures were resized to
the distribution of real images and the ones coming from the 64x64 pixels, we generated labeled examples for each image
distribution of the generator. The rectangular regions corre- by randomly adding a name over it (Fig. 1). We created a
sponding to annotations were cropped and new information separate set by translating images to greyscale using ITU-R
299 587 114
was generated based on the remaining contextual data. 601-2 luma transform R 1000 + G 1000 + B 1000 .

446
A. Greyscale Images Table II: Color examples
We aligned annotated images with their normal counter- Input DAE Patched Adversarial
parts and trained a denoising autoencoder on these pairs. In
this case, the examples are tensors with shape (64, 64, 1),
and the implemented architecture is presented in Fig. 5. We
used 7000 examples for training, 2000 for validation and
1000 for the test set. The hyper-parameters were chosen
empirically and are set to: batch size 64, learning rate 0.1
and default Adam. Processed images from the test set have
the annotation removed but do present some noise (Table I,
Column 2).
A patch classifier was also trained to distinguish regions
with text (Fig. 3). For this model, the training set consists
of 10k cropped regions from annotated images, with a 1:1
ratio of patches with and without text. The empirically
validated hyper-parameters are: batch size 256 and learning
rate 0.001. To generate the final output, regions ”corrupted”
with annotations are copied from the denoising autoencoder error, in order to adjust the learning accordingly. By setting
output, while clean patches are copied from the input image a constant threshold (φ = 0.3 empirically chosen) we can
(Table I, Column 3). The produced results are similar to the stop the generator training if the discriminator accuracy on
ones obtained by [3] (Fig. 8). labeled examples is bellow this margin, i.e. the discrim-
inator can’t detect reproduced images with text. Also, if
the discriminator accuracy is higher than 1 − φ, we stop
its training and let the generator improve its predictions.
However, we couldn’t find a satisfactory balance between
good input reproduction and text-removal (Table II, Column
4). It seems that the gradient coming from the adversarial
loss (D(R, S)), into the generator, tends to attenuate all
particularities of a picture, not just the annotations. On the
other side, assigning a bigger weight to the input replication
Figure 8: Results obtained by [3], with an adversarial cost (C(X, R)) would make the generator reproduce the
autoencoder and patch filtering sensitive information. The results are oscillating between a
constant output (Fig. 9a) and images with clear captions
B. Color Images (Fig. 9b).
1) Denoising Autoencoder: For color images, we first
implemented denoising autoencoder architecture presented
in Fig. 7. Again, we used 7000 aligned examples for training,
2000 for validation and 1000 for test, minimizing the mean-
squared error with gradient descent and Adam optimization.
The empirically chosen hyper-parameters are: batch size (a) Constant output (b) Annotations remain
64 and learning rate 0.0001. As in the greyscale case, the Figure 9: Different adversarial hyper-parameters results.
reproduced images do not contain the annotations, but are
noisy (Table II, Column 2).
We adjusted the previous patch classifier (Fig. 3) to accept C. Non-blind Inpainting
color images and trained it on the new set. Constructing The generator and discriminator were trained on 9000
patched images by filtering clean regions does provide better examples. On the remaining images, we cropped the rect-
results (Table II, Column 3). angles with annotations and obtained a reconstruction of
2) Autoencoder with critique: We conducted next exper- the original data by minimizing equation 4 with stochastic
iments with an adversarial strategy. Captioned and clean gradient descent and momentum. The final output is com-
images are mixed in the training set and labeled accordingly posed from clean regions of the input and generated regions
in order to train the discriminator (Fig. 4). During training, corresponding to the missing information. The results (Table
we watched the discriminator classification performance for III) present plausible representations, but given the higher
the two types of images and the autoencoder reproduction loss of information during the masking process, in most

447
 Table III: Non-blind Inpainting
Input Masked Image Output
cases the contextual information does not suffice to uniquely
determine the original input.
V. C ONCLUSION AND F UTURE W ORK
This work presented different approaches to censor
sensitive information from greyscale and color images.
Using a DAE architecture with aligned examples, the
private data is removed, but the reconstructed images
contain artifacts. We showed that if the sensitive regions
can be detected with a high probability, for a specific
problem, copying ”clean” regions from the original image
does always improve the generated prediction. A step
forward would be detecting and replacing irregular shapes,
instead of fixed rectangular patches. This approach has
a clear limitation, being unable to generalize beyond the
type of corruption learned from the training set. Another
addition can be separating the sensitive parts into a new
image, the model producing two outputs in this case.
In the adversarial setup where the discriminator input
is given only by samples coming from the generator, the
later one tends to collapse the original bi-modal distribution
(label/unlabeled examples) to a singularity (constant
output), or to preserve both modes. We concluded that
better results can be produced if the discriminator does not
only learn from the generator distribution, but also from
the distribution of real examples, forcing the generator to
fit the later one.
R EFERENCES
[1] A. Allen and W. Li. Generative adversarial denoising
autoencoder for face completion, 2016. http://www.cc.
gatech.edu/∼hays/7476/projects/Avery Wenchen/.
[2] W. Bainbridge. 10k us adult faces database. http://
wilmabainbridge.com/facememorability2.html.
[3] H. Edwards and A. Storkey. Censoring representations
with an adversary. Technical report, University of
448
Edinburgh, Department of Informatics, Edinburgh, UK,
EH8 9AB, 2016.
[4] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Ben-
gio. Generative adversarial nets. Technical report,
Universite de Montreal, Departement d’informatique et
de recherche operationnelle, Montreal, QC H3C 3J7,
2014.
[5] S. Khallaghi. Training autoencoders on imagenet using
torch 7, 2016. http://siavashk.github.io/2016/02/22/
autoencoder-imagenet/.
[6] D. Kingma and J. Ba. Adam: A method for stochastic
optimization, 2015.
[7] F.-F. Li, A. Karpathy, and J. Johnson. Convolutional
neural networks for visual recognition, 2016. http://
cs231n.stanford.edu/.
[8] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and
A. A. Efros. Context encoders: Feature learning by
inpainting, 2016. https://arxiv.org/abs/1604.07379.
[9] A. Radford, L. Metz, and S. Chintala. Unsupervised
representation learning with deep convolutional gener-
ative adversarial networks, 2016.
[10] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Man-
zagol. Extracting and composing robust features with
denoising autoencoders. Technical report, Universite
de Montreal, 2008.
[11] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-
A. Manzagol. Stacked denoising autoencoders: Learn-
ing useful representations in a deep network with a
local denoising criterion. Journal of Machine Learning
Research, 2010.
[12] R. Yeh, C. Chen, T. Y. Lim, M. Hasegawa-Johnson, and
M. N. Do. Semantic image inpainting with perceptual
and contextual losses. Technical report, Dept. of
Electrical and Computer Engineering, University of
Illinois, 2016.