Introduction Methodology Experimental Evaluation Conclusions

Face Recognition based on a

Collection of Binary Classifiers

Author: Rafael Henrique Vareto

Advisor: William Robson Schwartz
Co-Advisor: Filipe de Oliveira Costa

Smart Surveillance Interest Group

Department of Computer Science
Federal University of Minas Gerais

October 16th, 2017

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 1

Introduction Methodology Experimental Evaluation Conclusions
Face Recognition

Overview

1 Introduction
Face Recognition

2 Methodology
Background Methods
Open-set Face Identification
Face Verification

3 Experimental Evaluation
Common Attributes
Open-set Face Identification
Face Verification

4 Conclusions

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 2

Introduction Methodology Experimental Evaluation Conclusions
Face Recognition

General Applications

Image source : Google Images

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 3
Introduction Methodology Experimental Evaluation Conclusions
Face Recognition

Tasks

Chellappa, R., Sinha, P., and Phillips, P. J. (2010). Face recognition by computers and humans. Computer.
Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 4
Introduction Methodology Experimental Evaluation Conclusions
Face Recognition

Motivation

Great demand
Partial awareness
Open-set identification
Law enforcement scenario
Irrelevant subjects
Potential suspects
Verification
Access control scenario
Claim identity
Imposter / Legitimate

Image source : Google Images

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 5
Introduction Methodology Experimental Evaluation Conclusions
Face Recognition

Objectives

General

Determine whether a face image is known

Determine whether two face images have same identity

Specific

Accuracy and simplicity trade-off

Associate both methods with Partial Least Squares
Combine several classifiers

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 6

Introduction Methodology Experimental Evaluation Conclusions
Face Recognition

Hypotheses

Hypothesis One
Response histograms proceed differently whether we present probe
face images whose identity are enrolled in the gallery or whether we
examine unseen individuals.

Hypothesis Two
Face images of the same subject hold small differences, but this
difference increases when we consider a pair of images from different
subjects.

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 7

Introduction Methodology Experimental Evaluation Conclusions
Face Recognition

Contributions

Major

Good results with few learning data

Secondary

New approach for open-set face identification

New approach for face verification
Few trade-off parameter algorithms
Broad evaluation and discussion

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 8

Introduction Methodology Experimental Evaluation Conclusions

Overview

1 Introduction
Face Recognition

2 Methodology
Background Methods
Open-set Face Identification
Face Verification

3 Experimental Evaluation
Common Attributes
Open-set Face Identification
Face Verification

4 Conclusions

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 9

Introduction Methodology Experimental Evaluation Conclusions
Background Methods

Partial Least Squares (PLS)

Regression and dimensionality reduction

Supervised setting
NIPALS algorithms

Rosipal, R. and Krämer, N. (2006). Overview and recent advances in partial least squares. In Subspace, latent structure and feature
selection.
Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 10
Introduction Methodology Experimental Evaluation Conclusions
Background Methods

Partial Least Squares for Face Hashing (PLSH)

Closed-set face identification task

Galleries consisting of numerous subjects
Generate list of candidates

Santos Junior, C., Kijak, E., Gravier, G., and Schwartz, W. R. (2016). Partial least squares for face hashing. In Neurocomputing.
Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 11
Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Proposed Methods

Tasks

Open-set Face Identification

Face Verification

Vareto, R., Silva, S., Costa, F., and Schwartz, W. R. (2017). Towards Open-Set Face Recognition using Hashing Functions. In Inter-
national Joint Conference on Biometrics (IJCB).
Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 12
Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Open-set Face Identification

Training Stage

Subject partitioning
Binary classifiers

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 13

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Open-set Face Identification

Training Stage

Subject partitioning
Binary classifiers

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 14

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Open-set Face Identification

Training Stage

Subject partitioning
Binary classifiers

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 15

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Open-set Face Identification

Training Stage

Subject partitioning
Binary classifiers

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 16

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Open-set Face Identification

Testing Stage

Present to each classifier

Increment response score r

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 17

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Open-set Face Identification

Testing Stage

Present to each classifier

Increment response score r

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 18

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Open-set Face Identification

Testing Stage

Present to each classifier

Increment response score r

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 19

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Open-set Face Identification

Testing Stage

Present to each classifier

Increment response score r

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 20

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Open-set Face Identification

Testing Stage

Present to each classifier

Increment response score r

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 21

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Open-set Face Identification

Thresholding Response Histograms

HTS HTS
τ1 = 1 HTS τ3 = 1 , p = d0.15 × |H|e
AVG(HTS + HTS ) τ2 = 1 AVG(HTS + ... + HTS )
2 3 HTS 2 p
2

Known subject Unknown subject

30 10
8
20 6
4
10
2
0
0
2
4
10
6
0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 22

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Open-set Face Identification

Binary Classifiers

Partial Least Squares

Artificial Neural Network

ReLU
64 neurons

SOFTMAX
2 neurons Yes

Is this face
Feature Vector in the
gallery?

No
…

Lin, M.-I. B., Groves, W. A., Freivalds, A., Lee, E. G., and Harper, M. (2012). Comparison of Artificial Neural Network and Partial Least
Squares regression models for predicting respiratory ventilation : an exploratory study. In European journal of applied physiology.
Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 23
Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Proposed Methods

Tasks

Open-set Face Identification

Face Verification

Vareto, R., Silva, S., Costa, F., and Schwartz, W. R. (2017). Face Verification based on Relational Disparity Features and Partial Least
Squares Models. In Conference on Graphics, Patterns and Images (SIBGRAPI).
Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 24
Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Face Verification

Relational Disparity Features

Absolute difference
New feature vector

fi

fj
f(i, j) = |f i - f j|
VGGNet

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 25

Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Face Verification
Training Stage

Relational disparity features

Binary Classifiers

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 26

Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Face Verification
Training Stage

Relational disparity features

Binary Classifiers

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 27

Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Face Verification
Training Stage

Relational disparity features

Binary Classifiers

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 28

Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Face Verification
Training Stage

Relational disparity features

Binary Classifiers

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 29

Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Face Verification
Testing Stage

Present to each classifier

Majority voting

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 30

Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Face Verification
Testing Stage

Present to each classifier

Majority voting

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 31

Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Face Verification
Testing Stage

Present to each classifier

Majority voting

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 32

Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Face Verification
Testing Stage

Present to each classifier

Majority voting

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 33

Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Face Verification
Testing Stage

Present to each classifier

Majority voting

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 34

Introduction Methodology Experimental Evaluation Conclusions
Common Attributes

Overview

1 Introduction
Face Recognition

2 Methodology
Background Methods
Open-set Face Identification
Face Verification

3 Experimental Evaluation
Common Attributes
Open-set Face Identification
Face Verification

4 Conclusions

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 35

Introduction Methodology Experimental Evaluation Conclusions
Common Attributes

Datasets

Face Verification

LFW-A PubFig

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 36

Introduction Methodology Experimental Evaluation Conclusions
Common Attributes

Datasets

Open-set Face Identification

FRGC v1.0 PubFig83 VGGFace

VGGFace’s subset with 1000 subjects and 15 samples each

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 37
Introduction Methodology Experimental Evaluation Conclusions
Common Attributes

Feature Descriptors

Histogram of Oriented Gradients (HOG)

Shape information
Re-scaled to 128 × 144
Overlapping 16 × 16 blocks, 8-pixel stride

VGGFace CNN Feature Descriptor (VGGFace)

Derived from VGG16 architecture

Deep features obtained from FC6 layer
4096-size feature vector

Dalal, N. and Triggs, B. (2005). Histograms of oriented gradients for human detection. In Conference on Computer Vision and Pattern
Recognition.
Parkhi, O. M., Vedaldi, A., and Zisserman, A. (2015). Deep face recognition. In British Machine Vision Conference.
Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 38
Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Overview

1 Introduction
Face Recognition

2 Methodology
Background Methods
Open-set Face Identification
Face Verification

3 Experimental Evaluation
Common Attributes
Open-set Face Identification
Face Verification

4 Conclusions

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 39

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Evaluation Protocol

Literature protocol for PubFig83

Random 75 individuals compose known set

Eight remaining individuals as unknown
90% known subjects’ samples for training

New protocol for FRGCv1 and VGGFace

Varying known individuals : 10, 50 and 90%

Remaining individuals as unknown
50% known subjects’ samples for training

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 40

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Threshold Influence
Thresholds
HTS HTS
τ1 = 1 HTS τ3 = 1 , p = d0.15 × |H|e
AVG(HTS + HTS ) τ2 = 1 AVG(HTS + ... + HTS )
2 3 HTS 2 p
2

Receiver Operating Characteristic

1.0

0.8
True Positive Rate

0.6

0.4

0.2 Threshold 1 (AUC = 0.96)

Threshold 2 (AUC = 0.96)
Threshold 3 (AUC = 0.92)
0.0
0.0 0.2 0.4 0.6 0.8 1.0
False Positive Rate

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 41

Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Literature Comparison
H PLS-HOG outperforms H PLS-VGG

FRGCv1 dataset exp4

Approach AUC STD Execs

Least Squares 0.869 - -
SVM-Single 0.853 - -
Chebyshev 0.838 - -
W SVM-VGG 0.862 0.014 10
W SVM-HOG 0.515 0.027 10
H PLS-VGG 0.850 0.008 10
H PLS-HOG 0.910 0.022 10
H FCN-VGG 0.877 0.021 10
H FCN-HOG 0.613 0.105 10

Average AUC for 10 executions

Santos Junior, C. and Schwartz, W. R. (2014). Extending face identification to open-set face recognition. In SIBGRAPI.
Scheirer, W. J., Jain, L. P., and Boult, T. E. (2014). Probability models for open set recognition. In TPAMI
Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 42
Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Identification Results
Complete identification pipeline

Open-set Receiver Operating Characteristic

1.0

0.8
Detection and Identification Rate

0.6

0.4
PubFig83 - 100 models (AUC = 0.95)
PubFig83 - 300 models (AUC = 0.96)
0.2 PubFig83 - 500 models (AUC = 0.96)
FRGCv1-exp4 - 100 models (AUC = 0.76)
FRGCv1-exp4 - 300 models (AUC = 0.78)
0.0 FRGCv1-exp4 - 500 models (AUC = 0.78)
0.0 0.2 0.4 0.6 0.8 1.0
False Alarm Rate

Schwartz, W. R., Guo, H., Choi, J., and Davis, L. S. (2012). Face identification using large feature sets. In TIP
Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 43
Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Descriptor Selection

Picking best descriptor

Proper contrast on PubFig83

VGGFace outperforms HOG

Approach AUC STD Execs

H PLS-HOG 0.658 0.014 5
H PLS-VGG 0.954 0.008 5
H FCN-HOG 0.640 0.021 5
H FCN-VGG 0.974 0.005 5

Average AUC for 05 executions

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 44
Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Single Classifier Evaluation

Classifier behavior

FRGCv1 dataset exp4

PLS-VGG (%) FCN-VGG (%)

AVG 73.559 77.026
STD 01.943 01.648
MIN 70.614 73.245
MAX 77.631 80.592

Average hit rate for 100 executions

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 45
Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Additional Evaluation

Variable number of binary classifiers

PubFig83 dataset

Number of classifiers 100 300 500

AUC 0.946 0.954 0.972
H PLS-VGG
STD 0.009 0.015 0.004
AUC 0.970 0.972 0.980
H FCN-VGG
STD 0.000 0.004 0.007

Average AUC for 10 executions

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 46
Introduction Methodology Experimental Evaluation Conclusions
Open-set Face Identification

Additional Evaluation
Variable number of known individuals

Known individuals 10% 50% 90%

AUC 0.872 0.877 0.856
H FCN-VGG
FRGCv1 4

STD 0.015 0.022 0.014

AUC 0.794 0.850 0.856
H PLS-VGG
STD 0.078 0.009 0.022
AUC 0.866 0.862 0.848
W SVM-VGG
STD 0.035 0.015 0.019
AUC 0.996 0.974 0.964
H FCN-VGG
STD 0.011 0.008 0.005
VGGFace

AUC 0.972 0.962 0.924

H PLS-VGG
STD 0.004 0.004 0.005
AUC 0.841 0.839 0.835
W SVM-VGG
STD 0.013 0.007 0.007
50-50% known/unknown and train/test split
100 binary classifiers for H FCN and H PLS
Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 47
Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Overview

1 Introduction
Face Recognition

2 Methodology
Background Methods
Open-set Face Identification
Face Verification

3 Experimental Evaluation
Common Attributes
Open-set Face Identification
Face Verification

4 Conclusions

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 48

Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Evaluation Protocol

LFW’s Unrestricted, labeled outside data protocol

Less restrictive
Subjects’ identity exploitation
Additional training pairs
Add other datasets
Employed on PubFig

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 49

Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Method Evaluation

Evaluation Mode

Same-dataset evaluation
Cross-dataset evaluation

Algorithm variations

Cross setting (H PLS V-Cross)

Development-evaluation setting (H PLS V-Dev-Eval)
One fold per model setting (H PLS V-One-F-M)
Random setting (H PLS V-Random)

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 50

Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Literature Comparison

Receiver Operating Characteristic

1.0
Less data on LFW-A

0.8 0.997
DeepFace =
4.4M
True Positive Rate

0.6
0.996
PSE =
2.4M
0.4 DeepFace (area = 0.997)
MMDFR (area = 0.999) 0.999
pose+shape+exp (area = 0.996) MMDFR =
Skytop-Gaia (area = 0.987) 400K
0.2 HPLSV-Random (area = 0.966) 0.966
HPLSV-One-F-M (area = 0.958) HPLSV =
HPLSV-Cross (area = 0.906) 13K
0.0
0.0 0.2 0.4 0.6 0.8 1.0
False Positive Rate

DeepFace : Learn feature representation and train a siamese network

PSE : Synthesizing CASIA WebFace from 495K to 2.4M
MMDFR : Learn feature representation and train a joint bayesian model
Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 51
Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Literature Comparison

Evaluation Mode

PubFig Dataset
Average EER for 10 executions

Approaches EER (%) STD

DRM-WV 2.78 0.57
RNP 10.79 0.83
H PLS V-Dev-Eval 13.65 2.11
H PLS V-Random 14.73 2.02
H PLS V-One-F-M 16.63 3.05
CHISD 19.15 0.71
GEDA 23.90 1.29

EER is the point where FAR = FRR

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 52
Introduction Methodology Experimental Evaluation Conclusions
Face Verification

Parameter Evaluation

Variable number of hashing models

Need of multiple PLS

No significant improvement with more than 100 models

Number of Models
Approaches 1 100 300 500
H PLS V-Cross-LFW-A 0.620 0.886 0.897 0.899
H PLS V-Random-LFW-A 0.880 0.959 0.960 0.960
H PLS V-Dev-Eval-PubFig 0.801 0.941 0.942 0.942
H PLS V-Random-PubFig 0.810 0.936 0.937 0.938

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 53

Introduction Methodology Experimental Evaluation Conclusions

Overview

1 Introduction
Face Recognition

2 Methodology
Background Methods
Open-set Face Identification
Face Verification

3 Experimental Evaluation
Common Attributes
Open-set Face Identification
Face Verification

4 Conclusions

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 54

Introduction Methodology Experimental Evaluation Conclusions

Conclusions
Advantages

Simple and straightforward

Good results with few data

Limitations

Massive datasets
Incremental enrollment

Future Works

List of candidates for face identification

Adding other datasets for face verification
Employ fine-tuning techniques

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 55

Introduction Methodology Experimental Evaluation Conclusions

Contribution

Publications

Vareto, R., Silva, S., Costa, F., and Schwartz, W. R. (2017).

Towards Open-Set Face Recognition using Hashing
Functions. In International Joint Conference on Biometrics
(IJCB).
Vareto, R., Silva, S., Costa, F., and Schwartz, W. R. (2017). Face
Verification based on Relational Disparity Features and
Partial Least Squares Models. In Conference on Graphics,
Patterns and Images (SIBGRAPI).

Best Paper Runner-up Award at the International Joint Conference on Biometrics, 2017.
Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 56
Introduction Methodology Experimental Evaluation Conclusions

Acknowledgments

William R. Schwartz and Filipe O. Costa

Guillermo C. Chávez and Jefersson A. Santos
SSIG’s friends and colleagues

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 57

Introduction Methodology Experimental Evaluation Conclusions

LFW Protocol example

Rafael Henrique Vareto SSIG - DCC - UFMG October 16th, 2017 59