A Multiplicative Model for Conditional Image Manipulation Using DeepSIM

Rupesh Kumar Dwivedi Vaibhav Nath V.Arul

School of Computer Science and
Engineering
Galgotias University, Uttar Pradesh
rupesh_dwivedi.scsebtech@galgotiasuniver
sity.edu.in
School of Computer Science and
Engineering
Galgotias University, Uttar Pradesh
vaibhav_nath.scsebtech@galgotiasuniversit
y.edu.in
School of Computer Science and
Engineering
Galgotias University, Uttar Pradesh
v.arul@galgotiasuniversity.edu.in

Abstract paper is very exciting model for image manipulation using

This paper is mainly focused for generating conditional
image with the help of single augmented training sample.
The conditional image manipulation is accomplished with
the help of one model named as DeepSIM. DeepSIM is
stands for Deep Similarity which is responsible for creating
super resolution images but we will make some
manipulation in the previous model and make it to
manipulate images also we have seen that a single image
can be manipulated by using extension augmentation and
we can use the thin-spline-plate (TSP) for augmentation.
This model learns mapping between primitive image and
image itself. At the time of manipulation our generator
allows to make complex changes in primitive images. Our
model is suitable for image manipulation tasks.
1. Introduction
Deep neural network is the best for image manipulation tasks.
It works like human brain so whatever we will make with
help of neural network it will have a boosting performance. If
we are using Deep neural network for image manipulation
task then we need large data sets upon which we can train our
model. We have seen many times in machine learning models
especially in image manipulation the large data sets might be
give unpredictable output. This problem has been seen in
many cases where the output was not predicted and even we
were getting same output. It happens when we train out
model on large datasets. Training generative image for single
image and then making some changes this is the hot research
which is going on this time. People are writing so many
research paper on image manipulation and making new
model. But some model works and some comes and goes.
The Model of DeepSIM which we have proposed in this
Training Pair
single augmented training sample. DeepSIM easy to
implement but highly effective model for performing many
kinds of image manipulation tasks.
This model inference that first the primitive image of the
given real image has been generated by the model. And then
some changes has been made by the user and at the end the
final result has been generated with the help of given
model.To achieve this kind of image manipulation task we
have used a mapping network which learns how to map
between a primitive representation of an image and a real
image. After the training of primitive representation of an
image user can explicitly perform the changes according to
their requirements. Which makes this model very good model
for image manipulation task. Because it takes the modified
image and mapped into the network to generate the real
image.we will make with help of neural network it will have
a boosting performance. If we are using Deep neural network
for image manipulation task then we need large data sets
upon which we can train our model. We have seen many
times in machine learning models especially in image
manipulation the large data sets might be give unpredictable
output. This problem has been seen in many cases where the
output was not predicted and even we were getting same
output. It happens when we train out model on large datasets.
Training generative image for single image and then making
some changes this is the hot research which is going on this
time. People are writing so many research paper on image
manipulation and making new model. But some model works
and some comes and goes. The Model of DeepSIM which we
have proposed in this paper is very exciting model for image
manipulation using single augmented training sample.
DeepSIM easy to implement but highly effective model for
single image manipulation tasks. The purpose to make this
model is because existing model is working upon
unsupervised machine learning algorithm. But this model
help us to train single image using supervised method.
Input Output

Fig 1: Our model has fabricated the desired result for given images.

DeepSIM model has been trained for single image “face” has been trained and some manipulation has been
which we can see from left to right in image first we can see performed on the face noise. In second example one “dog”
image has been manipulated using our model and in third
example a racing car image where one extra wheel has been
added. Now we can see that our model has provided us a
desired output.
To achieve this kind of image manipulation task we have
used a mapping network which learns how to map between
a primitive representation of an image and a real image.
After the training of primitive representation of an image
user can explicitly perform the changes according to their
requirements. Which makes this model very good model for
image manipulation task. Because it takes the modified
image and mapped into the network to generate the real
image.
The image manipulation model has been explored by
several papers in this field. Some of them did very
extraordinary works in this field after proposing their
Training Image Pair
Figure-3: This is the example of Design and Fashion. On the left side we have training Image pair, in the middle we have primitive
manipulated image passed as input to the mapping network which generated output on the right side left to right.
1. Related Work
Deep Conditional generative model: The main work of this
model is to generate image-to-image translation of
primitive representation of image and real image keeping
all the aspect of image constant. Our model uses some of
the features of this model that’s why we thought we should
have to mention this model here it would very good for one
who is reading this paper will get to know what kind of
technique we have used to make this model so efficient.
This model does not use augmentation is the key for
conditional image manipulation it uses unsupervised
technique to perform conditional image generation using
this model. This model also used Generative Adversarial
Networks (GANs). To make it possible this model by now
papers. I want to mention two of them SinGAN [28] and
InGAN [29] which has the seminal works. SinGAN uses an
unsupervised learning way to perform unconditional image
manipulation upon a single image. On the other hand
InGAN uses unsupervised learning way to perform
conditional image manipulation upon a single image. This
paper used supervised learning method to work upon
conditional image manipulation that’s why it is easy to
apply and understandable. And in this process of making
this model we have seen that augmentation is the key to
make DeepSIM model effective for conditional image
manipulation tasks. In this model we have breaked the
ongoing practices of image manipulation and proposed the
conditional image manipulation for manipulating images
from primitive to real images. And we have also seen that
using thin spline model (TSP) for augmentation image
manipulation is the key to perform conditional image
manipulation.
Input Output
presented model.
Learning by a single-image: As we have seen to work on a
machine learning algorithm we have use a large datasets. But
sometimes large datasets creates problem.
This model learns how to map using a deep learning mapping
network.
Fig-4: Natch looking manipulation: On left hand we can see
the primitive image and in middle the training image and at
the right the manipulated image.
 Classical Image manipulation: There are many fields where image manipulation has attracted researcher from decades like image
processing, computational photography, graphics communities and many more that’s why this model very efficient. Everything
related to this model we cannot share here that’s why we are providing reference [31]. There are lots of image manipulations model
but not good enough to make it possible using supervised way of image manipulation. There is very good and notable image
manipulation technique: Poisson Image Editing [26], Seam Carving [3], and PatchMatch[4].

Figure-5: How this model works on challenging manipulation: In this model left side we have Training Image Pair and at right side
we can see the output in which we can see that this model mapped this image and manipulated according to our desire manipulation.

3. Learning Multiplicative Generator from a Generative Adversarial Networks (GANs). To make it possible
Single Image using DeepSIM this model by now presented model.

A conditional generative adversarial network (CGANs) ℓrec (x,y;G)=ℓperc (G(x),y) (1)

is learned by this model by using a exclusive image
Conditional GAN Loss: The main work of this model is to
which consists of a principle image and its primordial
generate image-to-image translation of primitive
representation. To make our model constraint for
representation of image and real image keeping all the aspect
datasets we have augmented the data with the help of
of image constant. Our model uses some of the
thin-spline-plate (TSP). This approach has several
featuresofthismodelthat’swhywethoughtweshould have to
advantages (i) single image training. (ii) The output
mention this model here it would very good for one who is
should reflect the primitive representation. Now we will
reading this paper will get to know what kind of technique we
explore each topic one by one related to this model.
have used to make this model so efficient.
3.1. Model
ℓadv(x, y; D, G)=log(D(x, y) + log(1 −D(x,G(x))) (2)
This model uses standard way to make (CGAN)
ℓtotal (x,y;D,G)=ℓrec (x,y;G) +α · ℓ adv(x, y;D,G) (3)
working effectively (particularly Pix2PixHD [35]). We
×d ×3 are
will denote our training pair (x, y) where y ∈ Rdx y

the input images (where dx and dy are rows and columns).

x ∈ Rdx×dy ×dp is the corresponding primitive images

dimensions. The main work of this model is to generate
image-to-image translation of primitive representation of
image and real image keeping all the aspect of image
constant. Our model uses some of the features of this
model that’s why we thought we should have to mention Figure 6: Output of three different primitive images. There has been
this model here it would very good for one who is a training performed on the source image which we can see left
reading this paper will get to know what kind of side at the starting of (Fig-6). Then we have passes this source
image to our mapping network which generates the result as
technique we have used to make this model so efficient.
combined images.
This model does not use augmentation is the key for
conditional image manipulation it uses unsupervised
technique to perform conditional image generation using
this model. This model also used.
 3.2. Augmentation
In this model we have breaked the ongoing practices of
image manipulation and proposed the conditional image
manipulation for manipulating images from primitive to real
images. And we have also seen that using thin spline model
(TSP) for augmentation image manipulation is the key to
perform conditional image manipulation. As we have seen
many other research papers where these model were not
included those model are not working well enough. That’s
why we thought we have to use this (TSP) model for
conditional image manipulation. This model also able to
learn the internal statistics of target image.
3.3. Optimization
During training we have a tendency to change our primitive
image representation according to our choice. And we can
also perform manipulation with high optimization which
make this model very efficient . The main work of this model
is to generate image-to-image translation of primitive
representation of image and real image keeping all the
aspect of image constant. Our model uses some of the
features of this model that’s why we thought we should have
to mention this model here it would very good for one who
is reading this paper will get to know what kind of technique
we have used to make this model so efficient. This model
does not use augmentation is the key for conditional image
manipulation it uses unsupervised technique to perform
conditional image generation using this model.
3.4 Primitive Images
We have used a mapping network which learns how to map
between a primitive representation of an image and a real
image. After the training of primitive representation of an
image user can explicitly perform the changes according to
their requirements. Which makes this model very good model
for image manipulation task. Because it takes the modified
image and mapped into the network to generate the real image.
Our model uses some of the features of this model that’s why
we thought we should have to mention this model here it
would very good for one who is reading this paper will get to
know what kind of technique we have used to make this model
so efficient. This model does not use augmentation is the key
for conditional image manipulation it uses unsupervised
technique to perform conditional image generation using this
model. This model also used Generative Adversarial Networks
(GANs). To make it possible this model by now presented
model.
4. Experiments
A conditional generative adversarial network (CGANs) is
learned by this model by using a exclusive image which
consists of a real image and its demonstrative representation.
To make our model constraint for datasets we have
augmented the data with the help thin-spline-plate (TSP). This
approach has many advantages (i) exclusive image training.
(ii) The result should reflect the demonstrative representation.
Now we will explore each topic one by one related to this
model.

Method S1 S2 S3 S4 S5
L S L S L S L S L S
.
Pix2PixHD-SIA 0.44 0.51 0.47 0.49 0.41 0.5 0.53 0.26 0.46 0.44
Ours - no VGG 0.14 0.05 0.26 0.11 0.11 0.07 0.28 0.14 0.19 0.08
Ours 0.12 0.07 0.21 0.12 0.1 0.04 0.22 0.12 0.14 0.06
Figure-7: In this experiment we have seen visual comparison between the images and tried to make manipulation and we found that
our model worked successfully to accomplish this experiment.
3.5 Analysis
We have seen all the primitive representation and real image
where this model worked very efficiently now we will see
what is run time of this model it will run according to our
desired time limit or it will exceed the limit. The run time
analysis is the most important aspect for every kind of
algorithm. We thought we have to also do this analysis for
this algorithm too. To make analysis of this algorithm we
compared this algorithm with different-different other image
processing algorithm and we found that this algorithm beats
almost image processing algorithm. Deep neural network is
the best for image manipulation tasks. It works like human
brain so whatever we will make with help of neural network
it will have a boosting performance. If we are using Deep
neural network for image manipulation task then we need
large data sets upon which we can train our model. We have
seen many times in machine learning models especially in
image manipulation the large data sets might be give
unpredictable output.
This model inference that first the primitive image of the
given real image has been generated by the model. And
then some changes has been made by the user and at the
end the final result has been generated with the help of
given model.
3.6 Conclusion
This proposed model based upon augmentation and TSP is
the image manipulation model. This model performs image
manipulation upon complex images which makes this
model more powerful than other image manipulation
model. This model can perform complex image
manipulation with high accuracy. This model takes input
what a user wants to manipulate in image because it
already train the primitive representation of image in which
manipulation has been performed. And also we have seen
the method which works upon single image have
significantly different level of performance and accuracy
than other methods. There is one limitation of single image
based method is that we have to pass every image to our
network separately which is very critical otherwise these
method made extraordinary accuracy and efficiency to
perform image manipulation.
References
[1] Nur Arad, Nira Dyn, Daniel Reisfeld, and Yehezkel Yeshu-run.
Image warping by radial basis functions: Applications to facial
expressions. CVGIP: Graph. Models Image Pro- cess.,
56(2):161–172, Mar. 1994.3
[2] Yuki M Asano, Christian Rupprecht, and Andrea Vedaldi. A
critical analysis of self-supervision, or what we can learn from
a single image. arXiv preprint arXiv:1904.13132, 2019.3
[3] Shai Avidan and Ariel Shamir. Seam carving for content-
aware image resizing. In SIGGRAPH, 2007.2
[4] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B
Goldman. Patchmatch: A randomized correspon- dence algorithm for
structural image editing.2
[5] Thaddeus Beier and Shawn Neely. Feature-based image
metamorphosis. SIGGRAPH Comput. Graph.,26(2):35–42,
July 1992. 3
[6] Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. Learn-ing texture
manifolds with the periodic spatial GAN. CoRR, abs/1705.06566,
2017.2
[7] Wengling Chen and James Hays. Sketchygan: Towards di- verse and
realistic sketch to image synthesis, 2018.3
[8] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim,
and Jaegul Choo. Stargan: Unified genera- tive adversarial networks for
multi-domain image-to-image translation. In CVPR, 2018.3
[9] EkinDCubuk,BarretZoph,DandelionMane,VijayVasude-van, and Quoc V
Le. Autoaugment: Learning augmentationpolicies from data. arXiv
preprint arXiv:1805.09501, 2018.3
[10] Tali Dekel, Chuang Gan, Dilip Krishnan, Ce Liu, and William T Freeman.
Smart, sparse contours to represent and edit images. arXiv preprint
arXiv:1712.08232, 2017.3
[11] Gianluca Donato and Serge Belongie. Approximate thin plate spline
mappings. In European conference on computer vision, pages 21–31.
Springer, 2002. 2,5
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu,
DavidWarde-Farley,SherjilOzair,AaronCourville,andYoshua Bengio.
Generative adversarial nets. In NIPS, pages 2672– 2680, 2014.3
[13] Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton:
An image-based virtual try-on network. In CVPR, 2018.3
[14] Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and
David H. Salesin. Image analogies.SIG-
GRAPH, 2001. 2
[15] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-
image translation with conditional adver- sarial networks. In CVPR,
2017.3
[16] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jungkwon Lee, and Jiwon
Kim. Learning to discover cross-domain relations with generative
adversarial networks. In ICML, 2017.3
[17] Junsoo Lee, Eungyeup Kim, Yunsung Lee, Dongjun Kim, Jaehyuk
Chang, and Jaegul Choo. Reference-based sketch image colorization
using augmented-self reference and dense semantic
correspondence. In CVPR, 2020.3
[18] Jianxin Lin, Yingxue Pang, Yingce Xia, Zhibo Chen, and Jiebo Luo.
Tuigan: Learning versatile image-to-image trans- lation with two
unpaired images. pages 18–35, 2020.3