Chap 7: Generative vs.

Discriminative
learning

Instructor
Dr. Sanjay Chatterji
 Several Pair of Concepts

Generative v.s. Discriminative

Parametric v.s. Non-parametric

Supervised v.s. Unsupervised

The gap between them is becoming increasingly small.

 An analogy

● The task is to determine the language that

someone is speaking
● Generative approach:
− is to learn each language and determine as to
which language the speech belongs to
● Discriminative approach:
− is determine the linguistic differences without
learning any language – a much easier task
 Some Examples
Example 1 Example 2
P(X|Y = 0) = 0.01 X Y P(X,Y) Y=2 Y=3
P(X|Y = 1) = 0.03
P(Y=0) = P(Y=1) = 0.5 2 3 X=1
P(Y=1/X) = ? 1 2 X=2
P(Y=0/X) = ?
P(Y=0, X) = ? 2 3 P(Y|X) Y=2 Y=3
P(Y=1, X) = ? 1 3 X=1
Example 3 X=2
Problem: Identify whether a sentence is in French or in Polish
Generative Model Discriminative model
Learn French Learn the difference of these two languages
Learn Polish Classify
Classify
 Generative Methods
● Model class-conditional pdfs and prior probabilities
● “Generative” since sampling can generate
synthetic data points
● Popular models
− Gaussians, Naïve Bayes, Mixtures of multinomials
− Mixtures of Gaussians, Mixtures of experts, HMM
− Sigmoidal belief networks, Bayesian networks, Markov
random fields
 Discriminative Methods

● Directly estimate posterior probabilities.

● No attempt to model underlying probability
distributions.
● Focus computational resources on given task.
● Better performance.
● Popular models
− Logistic regression, SVM, Traditional neural networks
− K-Nearest neighbour, Conditional Random Fields (CRF)
 Generative vs. Discriminative

● Generative learning: learns a joint model over

all variables P(x,y)

● Discriminative learning: learns a conditional

model of the output variables given the input
variables P(y|x)
 Discriminative v.s. Generative Models

Generative and discriminative learning are key problems

in machine learning and computer vision.
If you are asking, “Are there any
faces in this image?”, then you
would probably want to use
discriminative methods.

If you are asking, “Find a 3-d

model that describes the runner”,
then you would use generative
methods.

ICCV W. Freeman and A. Blake

Discriminative v.s. Generative Models

Discriminative models, either

explicitly or implicitly, study
the posterior distribution
directly.

Generative models approaches

to model the likelihood and
prior separately.
Graphical Model Relationship
 Generative Classifier
● Given variables x =(x1 ,..,xM ) and class
variable y
● Joint pdf is p(x,y)
− Called generative model since we can generate
more samples artificially
● Given a full joint pdf we can
− Marginalize
− Condition
− By conditioning the joint pdf we form a classifier
 Pros and Cons of Discriminative Models
Some general views, but might be outdated
Pros:
Focused on discrimination and marginal distributions.
Easier to learn/compute than generative models (arguable).
Good performance with large training volume.
Often fast.
Cons:

Limited modeling capability.

Can not generate new data.
Require both positive and negative training data (mostly).
Performance largely degrades on small size training data.
 Intuition about Margin

Elderly
Infant ?

? Woman
Man
Problem with All Margin-based
Discriminative Classifier

It might be very miss-leading to return

a high confidence.
 Wikipedia Definition of Generative and Discriminative Model

• Given an observable variable X and a target variable Y, a

generative model is a statistical model of the joint probability
distribution on X × Y, P ( X , Y )
• A discriminative model is a model of the conditional

probability of the target Y, given an observation x,

symbolically, P (Y|X = x)
• Classifiers computed without using a probability model are

also referred to loosely as "discriminative".

Generative classifiers: naive Bayes classifier and linear
discriminant analysis;
Discriminative model: logistic regression;
non-model classifier: perceptron and support vector machine.
Thank You