Deep Learning for Virtual Makeup: Transfer, Recommendation

and Removal
Dr. Taleb Alashkar (Ph.D.)a , Amjad Hussien
a
Algomus Inc. 950 Stephenson Hwy, USA

Abstract

Keywords:

1. Introduction
Virtual makeup is becoming a very hot topic in computer vision, machine learning and
augmented reality fields. The ability to integrate different methods and techniques from
these different fields to make the user experience with cosmetics purchases easier, more
interesting and attractive can play a vital role in increasing client satisfaction by allowing
consumers to try the product virtually before ordering it online [1]. Client experience in
beauty salons can be enriched by makeup transfer from models’ faces to customer’s faces
[2] along with an automatic facial feature analyzer. Recommendations of makeup styles
based on facial feature analysis can increase engagement and sales in salons, stores or online
[3]; Also, virtual makeup removal can vastly improve the on-site virtual makeup try-on
experience in cosmetic stores [4].

2. Aim
To build an advanced e-beauty system for virtual facial makeup that includes: makeup
transfer, makeup removal, facial analyzer and makeup recommendation.

3. Material and methods

The starting point in our research toward a fully automated e-beauty system for virtual
makeup is to collect a big before-after facial makeup database with very detailed manual
annotation for all facial features and makeup styles. We apply a deep convolution neural
network in a supervised manner to read all facial traits and classify them, then providing the
best makeup advice automatically according to the face analyzer results. From the dataset
available, a generative adversarial network (GAN) [5] model is trained for both makeup
transfer from one face to another and makeup removal at the same time. Also, starting
from the annotated dataset, a progressive neural network [6] model is used for makeup
recommendation.

March 12, 2018

4. Results
The accuracy of the makeup-related facial features classification outperforms current
state of the art methods and it succeeds in giving meaningful makeup style recommenda-
tions for corrective makeup; The makeup transfer and removal using our proposed GAN
model gives high resolution and accurate facial images with a new makeup style for makeup
transfer and without makeup for makeup removal. Also, the recommender system using
progressive neural network model gives realistic suggestions and high user satisfaction with
the recommended styles.

5. Conclusions
The proposed complete and fully automatic e-beauty system that includes facial traits
classification, makeup related advice generation, makeup removal and transfer; and makeup
recommendation can be deployed in beauty salons, cosmetics stores, and online shopping
platforms to increase client satisfaction and enrich the user experience.

6. Keywords
Virtual Makeup, Generative Adversarial Network, eBeauty, Makeup transfer, Makeup
Removal.

7. References
1. A. Rahman ; T. Tran ; S. Hossain ; A. Saddik, “Augmented Rendering of Makeup
Features in a Smart Interactive Mirror System for Decision Support in Cosmetic Products
Selection”, IEEE/ACM 14th International Symposium on Distributed Simulation and Real
Time Applications (DS-RT), 2010.
2. G. Dong and S. Terence; “Digital face makeup by example”, IEEE Computer Vision
and Pattern Recognition Conference, 2009.
3. T. Alashkar, S. Jiang, W. Shuyang, Y. Fu “Examples-Rules Guided Deep Neural
Network for Makeup Recommendation”, Thirty-First AAAI Conference on Artificial Intelli-
gence, 2017.
4. W. Shuyang, Y. Fu “Faces behind makeup”, Thirty AAAI Conference on Artificial
Intelligence, 2016.
5. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A.
Courville, Y. Bengio, “Generative Adversarial Nets”, Conference on Neural Information Pro-
cessing System, 2014.
6. A.Rusu, N.Rabinowitz, G.Desjardins, H.Soyer, J.Kirkpatrick, K.Kavukcuoglu,
R.Pascanu, R.Hadsell, “Progessive Neural Network”, Arxive, 2016.

March 12, 2018

Author biography
Dr. Taleb Alashkar (Ph.D.) completed his Master and Ph.D degree in France from
the University of Dijon 2012, and University of Lille, 2015 respectively. He has one year of
research experience from Northeastern University in Boston. He is currently working as a
Machine Learning Scientist at Algomus Inc. in Troy, MI. Taleb has published more than
10 papers in reputed journals and international conferences/workshops and he has been
serving as reviewer and principal committee member for more than 15 journals, interna-
tional conferences and workshops in computer vision, multimedia and artificial intelligence
domains.
Amjad Hussien is an entrepreneur at heart. He co-founded Algomus where he is the
CEO during the day and AI algorithms designer at night. He has built a passionate team
of designers, machine learning scientists and software engineers with tremendous industry
knowledge of manufacturing, distribution and retail .
Amjad and his team have built a powerful Opportunity Finder and Anomaly Detector AI
powered engine trained for manufacturing, distribution and retail verticals and architected
to run on very large and diverse data sets.
Amjad is an Honors Electrical Engineering & Computer Science graduate and a senior
member of IEEE. He earned his Master’s degree in Electrical Engineering & Computer
Science from UMIST, in the United Kingdom with a focus on artificial intelligence and
an MBA from MIT. Amjad is an active researcher and practitioner in the field of applied
natural language processing and deep learning. He has completed Advanced Management
Programs at Harvard and Stanford. Amjad was recognized in the Crain’s ’s ’40 under 40’
Executive recognition award program for his pioneering work in advanced analytics.

March 12, 2018