You are on page 1of 7

WVC 2019

Facial Makeup Detection using the CMYK


Color Model and Convolutional Neural Networks
Marcello G. Bertacchi Ismar F. Silveira
Mackenzie Presbyterian University Mackenzie Presbyterian University
São Paulo, Brasil São Paulo, Brasil
bertacchi@acm.org ismar@mackenzie.br

Abstract—This work presents a facial makeup detection tech- methods and experiments, section 6 describes the obtained
nique using CMYK and Neural Networks. The main goal is to results, ending with final remarks in section 7.
detect facial makeup using the CMYK color model, and analyzing
its results by comparing it to the HSV color model, which is A. Goals
widely used in the literature. In the detection process, each
image was separated into regions of interest (the eyes and the The general goal of this work is to achieve success in the
whole face). Five image databases were chosen, all varying in detection of facial makeup on an image containing frontal face
lighting and environment conditions. In HSV, 91% of accuracy and eyes regions, by means of Convolutional Neural Networks
was achieved on the eye region and 92% on the face. In CMYK, along with the CMYK color model.
the results obtained had 97% of accuracy on the eye region and
Among the specific objectives are:
95% on the face. Therefore, based on the results achieved, the
CMYK color model, even though it is mainly used in Printing, • to analyze the viability of the CMYK color model,
deserves attention in the area of Computer Vision, involving which does not have Computer Vision as its main area
Makeup Detection. of activity, by means of tests with combinations and
Index Terms—makeup detection, computer vision, CMYK,
neural networks, HSV
separation of Cyan, Magenta, Yellow and Key channels.
This model, which is widely used in printing, was chosen
I. I NTRODUCTION because of its satisfactory performance in skin detection,
At first, the recognition of characteristics was made in- as showed by Sawicki and Miziolek (2015) [7], [8].
• to compare the results obtained by the CMYK color
tuitively, that is, an individual memorized facial features of
another individual so that, in future, a recognition process model against those of the HSV model, which is widely
could take place. A method of recognition was born in ancient used in Image Processing and Computer Vision.
• to use neural networks in order to achieve more assertive
Egypt, where a procedure for measuring certain parts of
the body was studied and applied for comparison purposes. results in images with unfavorable conditions. For this
This technique evolved, later, in the nineteenth century, in reason, Convolutional Neural Networks were chosen and
a professional environment, being used to fight crime. Body implemented.
metrics were collected by means of measurement, and stored B. Related Work
in records for future reference towards people identification
Technological advancement allowed several systems, among
[1].
them the recognition of facial features, to be introduced
Nowadays, the recognition of facial features scenario also
into digital environments. However, as a counterpoint, this
included the application of facial makeup. According to Ueda
opened doors for certain vulnerabilities, including alterations
and Koyama (2010), excessive application of makeup may
on the facial features through cosmetics. Based on this idea,
interfere in the effectiveness of face-to-face recognition. Sub-
A. Dantcheva, C. Chen, and A. Ross (2012) implemented
sequently, as presented by A. Dantcheva, C. Chen, and A. Ross
an analysis that determined how much the application of
(2012), the interference caused by makeup in the effectiveness
makeup interferes on the effectiveness of facial recognition
of facial recognition algorithms was duly proven, allowing the
systems. The result obtained showed that there is interference,
digital makeup detection to be considered as a relevant subject
especially when applied around the ocular region, thus likely
[2]–[6].
The structure of this article is organized as follows: section 1 to compromise directly the security proposed by these systems
is the introduction, sections 2 and 3 contain a brief introduction [3].
on the CMYK (Cyan, Magenta, Yellow and Key) and HSV In that same year, Varshovi (2012) developed a work of
(Hue, Saturation and Value) color models, section 4 show the makeup recognition. This research was one of the first to
image databases used in the implementation, section 5 presents implement a database with images of women before and
after the application of cosmetics. The HSV color model was
chosen due to its capability to separate the saturation, hue, and
978-1-7281-5337-7/19/$31.00 ©2019 IEEE intensity channels individually. The results achieved 90.62%

54
WVC 2019

effectiveness in detecting eye shadow and 93.33% in detecting • Value: is when a color is exposed to a particular measure
lipstick [9]. of light intensity. When this measurement has the value
Using algorithms that extracts characteristics from a face close to the minimum, the color becomes darker and
shape, texture, and color, C. Chen, A. Dantcheva, and A. Ross darker until it reaches black (value = 0).
(2013) conducted a survey that determined whether or not it
was possible to find makeup in images. The HSV color model III. CMYK
Saturation channel was chosen because it clearly showed the Unlike the RGB model, CMYK is not based on the color
makeup on the eyes and lips. The final result was 93.5% addition system, in which three primary color lights (Red,
of accuracy in the detection of facial makeup. The Fig. 1 Green, and Blue) are combined in such a way that it is possible
demonstrates the separation of an image with makeup in each to achieve a color set of the light spectrum. The CMYK model
of the three RGB color model channels (Red, Green and Blue), is derived from the subtractive system, which uses pigments
as well as in the HSV color model [10]. instead of lights. CMYK, used mainly for printing, is an
abbreviation of four colors, which are, Cyan, Magenta, Yellow
and Key. The first three colors are secondary colors in the
additive system, but, in the subtractive, involving pigments,
they are primary colors [7], [12]–[15]:
• Cyan: is a lighten and pale color, which resembles blue,
and is also known as cold blue or grayish sky.
• Magenta: is a color that, in the additive system, is the
mixture between blue and red. The colors that most re-
semble magenta are scarlet violet, amaranth and crimson.
• Yellow: is a slightly more pale yellow.
• Key: is the amount of black color that is used, being
achieved in the subtractive model by combining all the
Fig. 1: A facial image separated in each of the three RGB
primary colors, which are Cyan, Magenta and Yellow.
and HSV channels [10, p. 2].
However, in printing cases, mixing three colors to achieve
one, generates an excessive amount of ink. For this
More recently, N. Kose, L. Apvrille, and J.-L. Dugelay
reason, the black color was added, thus allowing an
(2015), based on the last work cited above, developed a new
appropriate density to be achieved.
method that, through the Local Gabor Binary Pattern and
Histogram of Gradients algorithms, extracts shape and texture CMYK primary colors, combined with each other, generate
characteristics of facial images. In the final results, it achieved primary RGB colors (Red, Green, and Blue). While the
an accuracy of 97.7% [11]. additive system results in a set of one or more colored light
Finally, Sawicki and Miziolek (2015) proposed the appli- beams on a surface, the subtractive mode absorbs light through
cation of the CMYK color model for skin detection. The pigments, or inks. The colors that are not absorbed, combined
result obtained in detection by means of the Cyan channel generate the color that gives the name to the pigment. For
reached the accuracy rates of 97.6%, 91.3% and 95.5% in the example, an ink that absorbs all colors of light except red and
ECU1 , SFA2 and Kawulok3 databases. A relevant point in this blue, which are reflected, is called magenta [15].
research was that the RGB and HSV color models did not IV. I MAGES DATABASE
exceed the mark of 80% [7]. For the tests stage, five Image Databases were chosen,
II. HSV namely: YouTube Makeup (YMU), Virtual Makeup Database
(VMU), Makeup In the Wild (MIW), Facial Cosmetics
HSV is a color model with a principle similar to that of Database (FCD) and Makeup Induced Face Spoofing (MIFS).
RGB. This model is widely used in researches related to Image YMU is composed of 99 images of women’s front faces,
Processing and Computer Vision, and on Image Segmentation collected in makeup tutorials from YouTube.
[8]. VMU is a synthetic Image Database containing images of
HSV abbreviated means Hue, Saturation and Value [12], 51 women, taken from the Face Recognition Grand Challenge
[13]: database (FRGC).
• Hue: is the quality or attribute of color, which is repre- MIW is an Image Database with photographs captured at
sented by the wavelengths of the spectrum. everyday situations, which led to a variety of poses, lack of
• Saturation: is a variation between white color and pure balanced lighting and different levels of cosmetics application.
color, or, in other words, is the intensity of color. Its a total of 154 images, where 77 contain no makeup and 77
1 Face and Skin Detection Image Database created by Edith Cowan Univer-
contain cosmetic application.
sity.
FCD consists of a set of facial images with a variety
2 http://www.sel.eesc.usp.br/sfa/ of makeup application levels. Each individual has multiple
3 http://sun.aei.polsl.pl/∼mkawulok/gestures/ images ranging from very light to heavy levels of application,

55
WVC 2019

(a) YMU (b) VMU (c) FCD (d) MIW (e) MIFS

Fig. 2: Sample images from each of the five chosen databases.

and at least one image without any makeup. These images The tests environment was arranged in such a way that 25%
were also captured from makeup tutorials on YouTube. FCD of the images from all five databases were used as samples,
has a total of 389 images of 50 individuals. while 75% were used for the neural networks training.
MIFS is an Image Database created to test the effectiveness The statistical classification method used as metrics for this
of the spoofing4 technique. The images were captured from work is known as Confusion Matrix. The loss and accuracy
spoofing tutorials on YouTube. curves obtained from each experiment are also presented. The
abbreviations TP, TN, FP and FN mean, respectively, True
V. M ETHODS AND E XPERIMENTS Positive, True Negative, False Positive, and False Negative.
All images from the five databases were converted from The evaluation metrics used are as follows [17]–[19], in which
RGB to CMYK using the ICC Profile UncoatedFOGRA29. ACC (Accuracy) is the model accuracy,
After this step, the images were converted to each of the
(TP + TN )
four channels of the CMYK color model, and then to a ACC = , (1)
pairwise combination of channels. The combination of the (TP + TN + FP + FN )
M+K channels was chosen thanks to a satisfactory returned
result, where makeup can be better discerned. True Positive Rate (TPR), known as Recall, is the model’s
It was noted that, some of the converted images showed a ability to predict correctly the initial condition (for instance,
good amount of noise due to the bad conditions of illumination an image that contains makeup),
and capture. For this reason, there was a screening, where
these images were removed, keeping only the ones that were TP
TPR = , (2)
not influenced by the factors mentioned previously. A total of (TP + FN )
1343 faces images were approved.
Finally, the combination of the CMYK channels (M+K),
True Negative Rate (TNR), known as Specificity, which
as it was previously set, highlighted in grayscale the makeup
determines whether the system predicts correctly the absence
application on the lips and the eyes. For this reason, ROIs (Re-
of the initial condition (for example, an image that does not
gions of Interest) were cut out from the original images using
contain makeup),
the OpenCV library. The final processing, performed through
Convolutional Neural Networks, returned results pointing the TN
presence or absence of makeup in the eyes as well as in the TNR = , (3)
(TN + FP )
whole face.
The configuration of the neural networks for the HSV model
in each of the ROIs were as follows: False Positive Rate (FPR), also known as the False Alarm
Rate, is the inverse of the TNR,
• Face: five Convolution layers, five Max Pooling layers,
three Dropouts, and three hidden layers. FP
• Eyes: five Convolution layers, five Max Pooling layers,
FPR = = (1 − TNR), (4)
(FP + TP )
six Dropouts, and five hidden layers.
The configuration of the neural networks for the CMYK
F -measure, is the harmonic mean between the Precision
model in each of the ROIs were as follows:
(also known as Positive Predictive Value) and the Recall,
• Face: five Convolution layers, five Max Pooling layers,
three Dropouts, and three hidden layers. 2 · TP
F -measure = (5)
• Eyes: five Convolution layers, five Max Pooling layers, (2 · TP + FP + FN )
five Dropouts, and five hidden layers.
4 spoofing is defined as the act of trying to make, as a joke, someone believe Finally, ROC (Receiver Operator Characteristics) curves
in something that is not true [16]. were used for a better analysis and understanding of the

56
WVC 2019

evaluation metrics. In these ROC Charts the True Positive Rate VI. R ESULTS AND D ISCUSSION
(TPR) and False Positive Rate (FPR) are displayed on the In this section the results of HSV and CMYK makeup
Y and X axes [18]. The AUC (Area Under the Curve), which detection are presented and discussed.
is a method to calculate the area below the ROC curve, was
also used, always returning a value between 0 and 1, where A. HSV Makeup Detection
1 equals the ideal prediction. The value equal to or greater In this stage, two neural networks were trained towards
than 0,5 represents the correct predictions, while below this detection of makeup in HSV images. The results obtained for
number implies opposite predictions [18], [20]. the face and eye regions are as follows.
For a better performance in the processing and experiments, 1) Face Region: In this ROI, a total of 1343 images of
pairs of CMYK channels were combined, generating the result frontal faces were chosen from the database. The neural
shown in Fig. 3. The combination of (C+M) returned a widely network training happened during 300 epochs. The result was
prominent makeup on the eyes and lips. In (Y+K) the makeup 92% of accuracy. The Confusion Matrix returned 145 True
becomes a little more faded, but still visible on the eyes and Positives, 12 False Positives, 16 False Negatives and 163 True
lips. In (M+K) the result was satisfactory, where the makeup Negatives.
on the eyes and lips obtains more balance compared to the The Fig. 5 and Fig. 6 show the model loss and accuracy,
other combinations. In (C+K) the eyes makeup are highlighted, respectively.
while on the lips the makeup is mostly hidden. In (C+Y) the
makeup is highlighted but with noise. Finally, in (M+Y), the
makeup is highlighted on both ROIs, but with a higher noise
frequency on the face, making it difficult to present a clear
and detailed makeup form in the image.

Fig. 5: Loss curve from the HSV neural network on facial


images.

Fig. 3: Example of pair combinations for each CMYK


channel.

In Fig. 4, it is presented a comparison between the original


image, the HSV Saturation channel, and the CMYK combi-
nation (M+K). This chosen combination, compared with the
other channels combinations, has returned a more highlighted
makeup, with more details and less noise on each ROI.
Through this combination (M+K), it was possible to remove
shadow and noise on parts of the face besides the eyes and
lips.
Fig. 6: Accuracy curve from the HSV neural network on facial
images.

2) Eyes Region: In this ROI, a total of 2429 eyes images


were chosen from the database. The neural network training
happened during 300 epochs. The result was 91% of accuracy.
The Confusion Matrix returned 160 True Positives, 3 False
Fig. 4: Comparison between the S (Saturation) channel and Positives, 38 False Negatives and 284 True Negatives.
the CMYK channels (M+K) combination. The Fig. 7 and Fig. 8 show the model loss and accuracy,
respectively.

57
WVC 2019

Fig. 7: Loss curve from the HSV neural network on eyes Fig. 10: Accuracy curve from the CMYK neural network on
images. facial images.

The result was 95% of accuracy. The Confusion Matrix re-


turned 153 True Positives, 8 False Positives, 8 False Negatives
and 167 True Negatives.
2) Eyes Region: In this ROI, the same amount as in
the HSV process was chosen. The neural network training
happened during 300 epochs. The Fig. 11 and 12 show the
model loss and accuracy, respectively.

Fig. 8: Accuracy curve from the HSV neural network on eyes


images.

B. CMYK Makeup Detection


At this stage, two neural networks were implemented to
detect makeup in CMYK images. The results obtained for the
face and eye regions are as follows.
1) Face Region: In this ROI, the same amount as in Fig. 11: Loss curve from the CMYK neural network on eyes
the HSV process was chosen. The neural network training images.
happened during 300 epochs. The Fig. 9 and Fig. 10 show the
model loss and accuracy, respectively.

Fig. 12: Accuracy curve from the CMYK neural network on


Fig. 9: Loss curve from the CMYK neural network on facial eyes images.
images.
The result was 97% of accuracy. The Confusion Matrix

58
WVC 2019

returned 187 True Positives, 5 False Positives, 11 False Neg- TABLE I: Comparison between the evaluation metrics for both
atives and 282 True Negatives. model’s performances.
In the Fig. 13a, it is possible to see that the result was Color Model ROI ACC TPR TNR F-measure
satisfactory in both probabilistic and discrete curves for the
CMYK model for faces, in which TPR reached a value close Face 0,92 0,90 0,93 0,91
HSV
Eyes 0,91 0,80 0,99 0,89
to 1 and FPR was close to 0. The AUC of the probabilistic
curve resulted in a value of 0,99. In Fig. 13b, for the HSV CMYK
Face 0,95 0,95 0,95 0,95
Eyes 0,97 0,94 0,98 0,96
model, TPR and FPR were balanced between themselves,
both 0,1 away of the ideal performance value. The AUC of
the probabilistic curve reached a value of 0,97.
In the Fig. 14a, it is possible to see that the result was
satisfactory in both curves for the CMYK model for eyes, in On the detection of makeup, the CMYK model obtained
which TPR reached a value close to 1 and FPR to 0. The a satisfactory result, especially when it comes to predicting
AUC of the probabilistic curve resulted in a value of 0,99. In the absence of cosmetics application. The results achieved for
Fig. 14b, for the HSV model, FPR reached a value close to absence and presence of makeup in the face region model were
0, however, reaching a value of only 0,8 in TPR. The AUC similar in both TPR and TNR. In the eyes region, TPR was
of the probabilistic curve reached a value of 0,97. lower than TNR, implying a less accurate prediction of the
As presented in Table I, for all models (face and eyes), in presence of makeup.
accuracy (ACC ) and harmonic mean (F -measure), CMYK
performed better than HSV. For all cases, the AUC of the HSV was less accurate in detecting the presence of face and
probabilistic curve in CMYK was 0,02 above the result eye makeup, since TPR was lower than TNR. The model to
obtained in HSV. The harmonic mean of the HSV model to achieve greater prominence was the one for the eye region,
achieve the best result was the one for the face ROI, reaching reaching the value of 99% in the detection of absence of
a value of 91%. In CMYK, the harmonic mean to achieve the makeup.
best result was that of the eyes ROI, reaching a value of 96%.

(a) CMYK (a) CMYK

(b) HSV (b) HSV

Fig. 13: Comparison between ROC Curves for both CMYK Fig. 14: Comparison between ROC Curves for both CMYK
and HSV face makeup detection. and HSV eyes makeup detection.

59
WVC 2019

VII. S UMMARY AND F UTURE W ORK A special thanks to Everton Lages da Silva, who motivated
This work made a performance analysis between the CMYK and inspired the authors to pursuit this work subject, related
and HSV color models in the facial makeup detection process. to the CMYK color model.
This process was initiated by means of four neural networks, R EFERENCES
which, trained, returned a final result that proves the superi-
[1] A. K. Jain, A. Ross, and S. Prabhakar, “An introduction to biometric
ority of the CMYK model compared to HSV in such process. recognition,” IEEE Transactions on Circuits and Systems for Video
With CMYK, on the face region, it was achieved an Technology, vol. 14, no. 1, pp. 4–20, 2004.
accuracy of 95%. The probability of predicting the absence [2] S. Ueda and T. Koyama, “Influence of make-up on facial recognition,”
Perception, vol. 39, no. 2, pp. 260–264, 2010.
of makeup (TNR) was 95%, the same value obtained in the [3] A. Dantcheva, C. Chen, and A. Ross, “Can Facial Cosmetics Affect
probability of predicting individuals with makeup (TPR). On the Matching Accuracy of Face Recognition Systems?” Proc. of 5th
the eye region, an accuracy of 97% was returned. In this, the IEEE International Conference on Biometrics: Theory, Applications and
Systems (BTAS), 2012.
TNR returned a value of 98%, while the TPR returned a [4] B. Miller, “Vital signs of identity [biometrics],” IEEE Spectrum, vol. 31,
value of 94%. This shows that the ability of the model to no. 2, pp. 22–30, feb 1994.
detect the absence of makeup is superior to that of detecting [5] T. Ahonen, A. Hadid, and M. Pietikãinen, “Face Description with Local
Binary Patterns: Application to Face Recognition,” IEEE TRANSAC-
the presence of cosmetics. TIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,
With HSV, on the face region, it was achieved an accuracy vol. 28, no. 12, pp. 2037–2041, 12 2006.
of 92%, and there was similarity in the prediction probabilities [6] T. Ojala, M. Pietikäinen, and T. Mäenpää, “Multiresolution Gray Scale
and Rotation Invariant Texture Classification,” Pattern Analysis and
of absence and presence of makeup. The TNR reached 93%, Machine Intelligence, IEEE Transactions on, vol. 24, pp. 971–987, 7
and the TPR reached 90%. On the eyes region, an accuracy of 2002.
91% was achieved, showing a high probability of detecting the [7] D. J. Sawicki and W. Miziolek, “Human colour skin detection in CMYK
colour space,” IET Image Processing, vol. 9, no. 9, pp. 751–757, 2015.
absence of makeup (99%) and a low probability of detecting [8] N. A. Ibraheem, M. M. Hasan;, R. Z. Khan;, and P. K. Mishra,
the presence of makeup (80%). In general, the probability of “Understanding Color Models: A Review,” Journal of Science and
detecting absence of makeup was notable in all ROIs in the Technology, vol. 2, no. 3, pp. 265–275, 2012.
[9] S. Varshovi, “Facial Makeup Detection Using HSV Color Space and
HSV model. Texture Analysis,” Master’s thesis, Concordia University, 2012.
In conclusion, it is perceived that CMYK obtained better [10] C. Chen, A. Dantcheva, and A. Ross, “Automatic facial makeup
performance than HSV in the process of detection of facial detection with application in face recognition,” Proc. of 6th IARP
International Conference on Biometrics (ICB), 2013.
makeup. However, HSV achieved a satisfactory result in the [11] N. Kose, L. Apvrille, and J.-L. Dugelay, “Facial makeup detection
detection of absence of makeup. Neural network models have technique based on texture and shape analysis,” 2015 11th IEEE In-
also proved themselves capable of handling the chosen set ternational Conference and Workshops on Automatic Face and Gesture
Recognition (FG), pp. 1–7, 2015.
of images from five different databases, effectively detecting [12] A. V. Hurkman, Color Correction Handbook, 2014.
facial and eyes makeup. In both ROIs the results obtained [13] M. Tkalčič and J. F. Tasič, “Colour spaces - Perceptual, historical and
in the detection of presence and absence of cosmetics were applicational background,” IEEE Region 8 EUROCON 2003: Computer
as a Tool - Proceedings, vol. A, pp. 304–308, 2003.
satisfactory, which demonstrates that the CMYK is a viable [14] A. Carlsson, A. Eriksson, and M. Isik, “Automatic Detection of Images
alternative to applications related to makeup, involving Com- Containing Nudity,” pp. 1–53, 2008.
puter Vision. This shows how much the CMYK model can [15] B. A. Wandell and L. D. Silverstein, “Digital color Reproduction,” in
The Science of Color. Elsevier, 2003, ch. 8, pp. 281–316.
contribute to the technology environment, opening doors to [16] C. U. Press and K. Woodford, Cambridge Advanced Learner’s Dictio-
more research on this subject, and cooperating towards the nary. Cambridge University Press, 2003.
constant evolution of computer technology. [17] F. Clésio, “Métricas de avaliação de modelos de classificação/predição,”
2014. [Online]. Available: https://mineracaodedados.wordpress.com/tag/
As future work, it is recommended that the process of com- matriz-de-confusao/
bining the CMYK channels be further explored, based on the [18] T. Fawcett, “An Introduction to ROC Analysis,” Pattern Recognition
success obtained by combining the Magenta and Key (M+K) Letters, vol. 27, no. 8, pp. 861–874, 2006.
[19] J. Brownlee, “Classification accuracy is not enough: More
channels. Likewise, it is suggested an implementation of a performance measures you can use,” 2014. [Online]. Available:
contrast normalization procedure in the images preprocessing https://mineracaodedados.wordpress.com/tag/matriz-de-confusao/
step, thus allowing the use of images that were previously [20] P. Sonego, A. Kocsor, and S. Pongor, “ROC analysis: Applications to
the classification of biological sequences and 3D structures,” Briefings
removed because of poor lighting and environment conditions. in Bioinformatics, vol. 9, no. 3, pp. 198–209, 2008.
Finally, unlike the binary classification used in this work (with [21] C. Chen, A. Dantcheva, and A. Ross, “An Ensemble of Patch-based
or without makeup), it is advised to have a level classification Subspaces for Makeup-Robust Face Recognition,” Information Fusion
Journal, vol. 32, pp. 80–92, 2015.
of makeup, such as “none”, “lightweight” and “heavy”. [22] C. Chen, A. Dantcheva, T. Swearingen, and A. Ross, “Spoofing Faces
Using Makeup: An Investigative Study,” Proc. of 3rd IEEE International
ACKNOWLEDGMENT Conference on Identity, Security and Behavior Analysis (ISBA), 2017.
The authors would like to thank N. Kose, L. Apvrille, and
J.-L. Dugelay for sending a free CD-WR media containing the
Facial Cosmetics Database images database [11].
Also, thanks to A. Dantcheva, C. Chen, A. Ross, and T.
Swearingen for making available the YMU, VMU, MIW and
MIFS makeup image databases [3], [10], [21], [22].

60

You might also like