Backprop, 25 years later…

Garrison W. Cottrell
Gary's Unbelievable Research Unit (GURU) Computer Science and Engineering Department Institute for Neural Computation UCSD

1

But first…
Hal White passed away March 31st, 2012

• Hal was “our theoretician of neural
nets,” and one of the nicest guys I knew. • His paper on “A heteroskedasticityconsistent covariance matrix estimator and a direct test for heteroskedasticity” has been cited 15,805 times, and led to him being shortlisted for the Nobel Prize. • But his paper with Max Stinchcombe: “Multilayer feedforward networks are universal approximators” is his second most-cited paper, at 8,114 cites.
Back propagation, 25 years later 2

But first…

In yet another paper (in Neural Computation, 1989), he wrote

“The premise of this article is that

learning procedures used to train artificial neural networks are inherently statistical techniques. It follows that statistical theory can provide considerable insight into the properties, advantages, and disadvantages of different network learning methods…” This was one of the first papers to make the connection between neural networks and statistical models - and thereby put them on a sound statistical foundation.
Back propagation, 25 years later 3

few could appreciate as deeply as Dave did what they had when they discovered it. 2011 • Many had invented back propagation. Rumelhart passed away on March 13.We should also remember… Dave E. 25 years later 4 . Back propagation.

Back propagation. 25 years later 5 .too many for us to explicitly program them in our models • Rather we need some way to indirectly set them via a procedure that will achieve some goal by changing the synaptic strengths (which we call weights). which we think encode the “knowledge” in the network . • These neurons are connected by 1014 .1015 synapses.What is backpropagation. and why is/was it important? • We have billions and billions of neurons that somehow work together to create the mind. • This is called learning in these systems.

25 years later 6 .Learning: A bit of history • Frank Rosenblatt studied a simple version of a neural net called a perceptron: • A single layer of processing • Binary output • Can compute simple things like (some) boolean functions (OR.) Back propagation. etc. AND.

Learning: A bit of history output net input Back propagation. 25 years later 7 .

Learning: A bit of history Back propagation. 25 years later 8 .

1969) Back propagation.Learning: A bit of history • Rosenblatt (1962) discovered a learning rule for perceptrons called the perceptron convergence procedure. 25 years later 9 . • Guaranteed to learn anything computable (by a two-layer perceptron) • Unfortunately. not everything was computable (Minsky & Papert.

25 years later 10 . compute the output as: If net   then output = 1 else output = 0 output net input Back propagation.Perceptron Learning Demonstration • Output activation rule: • First. compute the net input to the output unit: wixi = net • Then.

compute the net input to the output unit: wixi = net If net   then output = 1 else output = 0 • Learning rule: If output is 1 and should be 0. not 0) Back propagation. then lower weights to active inputs and raise the threshold () If output is 0 and should be 1. then raise weights to active inputs and lower the threshold () (“active input” means xi = 1.Perceptron Learning Demonstration • Output activation rule: • First. 25 years later 11 .

25 years later 12 . • Slow! Learning on some patterns ruins learning on others. Back propagation.Characteristics of perceptron learning • Supervised learning: Gave it a set of input-output examples for it to model the function (a teaching signal) • Error correction learning: only correct it when it is wrong. • Random presentation of patterns.

Perceptron Learning Made Simple • Output activation rule: • First. 25 years later 13 . compute the net input to the output unit: wixi = net If net   then output = 1 else output = 0 • Learning rule: If output is 1 and should be 0. then raise weights to active inputs and lower the threshold () Back propagation. then lower weights to active inputs and raise the threshold () If output is 0 and should be 1.

then raise weights to active inputs and lower the threshold () • Learning rule: wi(t+1) = wi(t) + *(teacher .output)*xi ( is the learning rate) • This is known as the delta rule because learning is based on the delta (difference) between what you did and what you should have done:  = (teacher . then lower weights to active inputs and raise the threshold () If output is 0 and should be 1.Perceptron Learning Made Simple • Learning rule: If output is 1 and should be 0.output) Back propagation. 25 years later 14 .

it can learn to compute. Back propagation. e. 1969) • Minsky & Papert said: • if you had hidden units.Problems with perceptrons • The learning rule comes with a great guarantee: anything a perceptron can compute. and we don’t think one will ever be discovered. you could compute any boolean function. 25 years later 15 . • Problem: Lots of things were not computable.g. • But no learning rule exists for such multilayer networks. XOR (Minsky & Papert..

Problems with perceptrons Back propagation. 25 years later 16 .

Aside about perceptrons • They didn’t have hidden units . • Support Vector Machines essentially do this in a principled way. followed by a (highly sophisticated) perceptron learning algorithm. Back propagation. 25 years later 17 .but Rosenblatt assumed nonlinear preprocessing! • Hidden units compute features of the input • The nonlinear preprocessing is a way to choose features by hand.

• Works a lot like the perceptron algorithm: • • • • Randomly choose an input-output pattern present the input. & Williams (1985) • Discovered a learning rule for networks with hidden units.Enter Rumelhart. 25 years later 18 . Hinton. let activation propagate through the network give the teaching signal propagate the error back through the network (hence the name back propagation) • change the connection strengths according to the error Back propagation.

25 years later 19 . INPUTS • The actual algorithm uses the chain rule of calculus to go downhill in an error measure with respect to the weights • The hidden units must learn features that solve the problem Back propagation. Activation Hidden Units Error ... Hinton.Enter Rumelhart.. & Williams (1985) OUTPUTS ..

XOR Back Propagation Learning OR AND Random Network XOR Network • Here. can solve the problem Back propagation. 25 years later 20 .two features that when combined appropriately. the hidden units learned AND and OR .

25 years later 21 . depending on initial conditions. Back propagation.XOR Back Propagation Learning Random Network OR AND XOR Network But. there are an infinite number of ways to do XOR .backprop can surprise you with innovative solutions.

25 years later 22 .Why is/was this wonderful? • • • • • Efficiency Learns internal representations Learns internal representations Learns internal representations Generalizes to recurrent networks Back propagation.

25 years later 23 .Hinton’s Family Trees example • Idea: Learn to represent relationships between people that are encoded in a family tree: Back propagation.

25 years later 24 .Hinton’s Family Trees example • Idea 2: Learn distributed representations of concepts: localist outputs Learn: features of these entities useful for solving the task Input: localist people localist relations Localist: one unit “ON” to represent each item Back propagation.

25 years later 25 .People hidden units: Hinton diagram • The corresponding people in the two trees are above/below one another. italian below Back propagation. english above.

People hidden units: Hinton diagram • What does the unit 1 encode? What is unit 1 encoding? Back propagation. 25 years later 26 .

People hidden units: Hinton diagram • What does unit 2 encode? What is unit 2 encoding? Back propagation. 25 years later 27 .

People hidden units: Hinton diagram • Unit 6? What is unit 6 encoding? Back propagation. 25 years later 28 .

25 years later 29 .Relation units • What does the upper right one code? Back propagation.

• Can explain some human phenomena Back propagation. 25 years later 30 .e..Lessons • The network learns features in the service of the task .i. it learns features on its own. • This is useful if we don’t know what the features ought to be.

I make two points: • The perceptron algorithm is still useful! • Representations learned in the service of the task can explain the “Visual Expertise Mystery” Back propagation. 25 years later 31 .Another example • In the next example(s).

. . PCA .A Face Processing System Happy Sad Afraid Angry Surprised Disgusted Gabor Filtering . . Neural Net Object (IT) Level Pixel (Retina) Level Perceptual (V1) Level Category Level Back propagation. 25 years later 32 . .

PCA . .The Face Processing System Bob Carol Ted Alice Gabor Filtering . 25 years later 33 . . . . Neural Net Object (IT) Level Pixel (Retina) Level Perceptual (V1) Level Category Level Back propagation.

. Neural Net Object (IT) Level Pixel (Retina) Level Perceptual (V1) Level Category Level Back propagation. . PCA . .The Face Processing System Bob Carol Ted Alice Gabor Filtering . . 25 years later 34 .

25 years later 35 . Neural Net Pixel (Retina) Level Perceptual (V1) Level Object Feature (IT) level Level Category Level Back propagation. .The Face Processing System Bob Carol Ted Cup Can Book Gabor Filtering . . PCA . . .

25 years later 36 .The Gabor Filter Layer • Basic feature: the 2-D Gabor wavelet filter (Daugman. 85): • These model the processing in early visual areas * Convolution Magnitudes Subsample in a 29x36 grid Back propagation.

25 years later 37 .600 numbers • We use PCA to reduce this to 50 numbers • PCA is like Factor Analysis: It finds the underlying directions of Maximum Variance • PCA can be computed in a neural network through a competitive Hebbian learning mechanism • Hence this is also a biologically plausible processing step • We suggest this leads to representations similar to those in Inferior Temporal cortex Back propagation.Principal Components Analysis • The Gabor filters give us 40.

Holons. 1987. O’Toole et al..How to do PCA with a neural network (Cottrell. Cottrell & Fleming 1990.. Back propagation. 25 years later 38 . Cottrell & Metcalfe 1990. Principal Components. 1991)  A self-organizing network that learns whole-object representations (features. eigenfaces) Holons (Gestalt layer) Input from Perceptual Layer . Munro & Zipser.

eigenfaces) Holons (Gestalt layer) Input from Perceptual Layer .. Holons. Cottrell & Fleming 1990. Munro & Zipser. Cottrell & Metcalfe 1990. Back propagation. 1991)  A self-organizing network that learns whole-object representations (features. O’Toole et al. 25 years later 39 . Principal Components. 1987.How to do PCA with a neural network (Cottrell..

Cottrell & Metcalfe 1990. Munro & Zipser.. Back propagation.How to do PCA with a neural network (Cottrell. O’Toole et al. 1987. Holons. 25 years later 40 . Principal Components. 1991)  A self-organizing network that learns whole-object representations (features. eigenfaces) Holons (Gestalt layer) Input from Perceptual Layer . Cottrell & Fleming 1990..

Holons. Back propagation. 1987. O’Toole et al. Cottrell & Metcalfe 1990. Munro & Zipser. Principal Components.. 1991)  A self-organizing network that learns whole-object representations (features.How to do PCA with a neural network (Cottrell. eigenfaces) Holons (Gestalt layer) Input from Perceptual Layer . 25 years later 41 .. Cottrell & Fleming 1990.

O’Toole et al. Cottrell & Fleming 1990. Munro & Zipser. Holons. eigenfaces) Holons (Gestalt layer) Input from Perceptual Layer .. 1991)  A self-organizing network that learns whole-object representations (features. Back propagation. 1987.How to do PCA with a neural network (Cottrell. 25 years later 42 . Principal Components. Cottrell & Metcalfe 1990..

Back propagation.. eigenfaces) Holons (Gestalt layer) Input from Perceptual Layer . O’Toole et al. Principal Components. Munro & Zipser. 1991)  A self-organizing network that learns whole-object representations (features.. 25 years later 43 . 1987. Holons. Cottrell & Metcalfe 1990. Cottrell & Fleming 1990.How to do PCA with a neural network (Cottrell.

25 years later 44 . O’Toole et al. Holons. 1991)  A self-organizing network that learns whole-object representations (features. Principal Components.. Back propagation. 1987.. eigenfaces) Holons (Gestalt layer) Input from Perceptual Layer . Cottrell & Fleming 1990. Munro & Zipser. Cottrell & Metcalfe 1990.How to do PCA with a neural network (Cottrell.

Cottrell & Metcalfe 1990. Face. etc. Carol. Can. 25 years later 46 . Padgett & Cottrell 1996. Book. Categories Holons Input from Perceptual Layer . • Excellent generalization performance demonstrates the sufficiency of the holistic representation for recognition Back propagation. Bob. Afraid. Output: Cup. Dailey et al. 1999. Sad.The Final Layer: Classification (Cottrell & Fleming 1990... Happy. Dailey & Cottrell. 2002) The holistic representation is then used as input to a categorization network trained by supervised learning. Greeble. Ted.

The Final Layer: Classification
• Categories can be at different levels: basic, subordinate. • Simple learning rule (~delta rule). It says (mild lie here): • add inputs to your weights (synaptic strengths) when
you are supposed to be on, • subtract them when you are supposed to be off. • This makes your weights “look like” your favorite patterns – the ones that turn you on. • When no hidden units => No back propagation of error. • When hidden units: we get task-specific features (most interesting when we use the basic/subordinate distinction)

Back propagation, 25 years later

47

Facial Expression Database
• Ekman and Friesen quantified muscle movements (Facial Actions)
involved in prototypical portrayals of happiness, sadness, fear, anger, surprise, and disgust. • Result: the Pictures of Facial Affect Database (1976). • 70% agreement on emotional content by naive human subjects. • 110 images, 14 subjects, 7 expressions.

Anger, Disgust, Neutral, Surprise, Happiness (twice), Fear, and Sadness This is actor “JJ”: The easiest for humans (and our model) to classify
Back propagation, 25 years later 48

Results (Generalization)
Expression Happiness Surprise Disgust Anger Sadness Fear Average Network % Correct 100.0% 100.0% 100.0% 89.2% 82.9% 66.7% 89.9% Human % Agreement 98.7% 92.4% 92.3% 88.9% 89.2% 87.7% 91.6%

• Kendall’s tau (rank order correlation): .667,
Back propagation, 25 years later

p=.0441 • Note: This is an emergent property of the model!
49

3. 25 years later 50 .0011] Again.Correlation of Net/Human Errors • Like all good Cognitive Scientists. we like our • • • • models to make the same mistakes people do! Networks and humans have a 6x6 confusion matrix for the stimulus set. Back propagation. [F (1.567. This suggests looking at the off-diagonal terms: The errors Correlation of off-diagonal terms: r = 0. p = 0. this correlation is an emergent property of the model: It was not told which expressions were confusing.28) = 13.

But the Gabor magnitude representation is noninvertible. We can learn an approximate inverse mapping. 25 years later 51 .” Back propagation.Examining the Net’s Representations • • • • We want to visualize “receptive fields” in the network. however. We used linear regression to find the best linear combination of Gabor magnitude principal components for each image pixel. • Then projecting each unit’s weight vector into image space with the same mapping visualizes its “receptive field.

so subtracting the resulting “average face” shows more precisely what the units attend to: • Apparently local features appear in the global templates. 25 years later 52 .Examining the Net’s Representations • The “y-intercept” coefficient for each pixel is simply the average pixel value at that location over all faces. Back propagation.

iness Modeling Six-Way Forced Choice Surprise Fear Sadness Disgust Anger Happ- • Overall correlation r=. 25 years later 58 .9416. with NO FIT PARAMETERS! Back propagation.

61 • The model fits the data best at a precategorical layer: The layer we call the “object” layer. NOT at the category level Back propagation. 25 years later 59 .36 PERCENT CORRECT DISCRIMINATION MODEL OBJECT LAYER R=0.Model Discrimination Scores iness Surprise Fear Sadness Disgust Anger Happ- HUMAN MODEL OUTPUT LAYER R=0.

25 years later 62 . p<.001 Back propagation.Reaction Time: Human/Model iness Surprise Fear Sadness Disgust Anger Happ- MODEL REACTION TIME (1 - MAX_OUTPUT) HUMAN SUBJECTS REACTION TIME Correlation between model & data: .6771.

1 -0.9 0.Mix Detection in the Model Mixed-In Expression Detection 1.3 -0.1 -0.3 Average Rank Score 1. 25 years later 63 .3 0. Back propagation.7 0.5 0.5 1.5 10 30 50 Mixed-in expression (humans) Unrelated expressions (humans) Mixed-in expression (networks) Unrelated expressions (networks) Percent Mixing Can the network account for the continuous data as well as the categorical data? YES.1 0.

Back propagation.Human/Model Circumplexes These are derived from similarities between images using non-metric Multi-dimensional scaling. For networks: similarity is correlation between 6-category output vectors. 25 years later 64 . For humans: similarity is correlation between 6-way forcedchoice button push.

it nevertheless: • Organizes the faces in the same order around the circumplex • Correlates very highly with human responses. • Has about the same rank order difficulty in classifying the emotions Back propagation.Discussion • Our model of facial expression recognition: • Performs the same task people do • On the same stimuli • At about the same accuracy • Without actually “feeling” anything. without any access to the surrounding culture. 25 years later 66 .

The Visual Expertise Mystery • Why would a face area process BMW’s?: • Behavioral & brain data • Model of expertise • results Back propagation. 25 years later 69 .

25 years later 70 .” **Courtesy of Jim Tanaka. University of Victoria Back propagation.Are you a perceptual expert? Take the expertise test!!!** “Identify this object with the first name that comes to mind.

Not an expert “2002 BMW Series 7” .“Car” .Expert! Back propagation. 25 years later 71 .

Expert! Back propagation.“Bird” or “Blue Bird” .Not an expert “Indigo Bunting” . 25 years later 72 .

25 years later .Democrat! “Jerk” or “Megalomaniac” 73 .Expert! Back propagation.Not an expert “George Dubya”.“Face” or “Man” .

. 1971) Indigo Bunting Subordinate Species Level Back propagation. 25 years later 74 .How is an object to be named? Animal Bird Superordinate Level Basic Level (Rosch et al.

Entry Point Recognition Animal Semantic analysis Entry Point Bird Visual analysis Downward Shift Hypothesis Indigo Bunting Fine grain visual analysis Back propagation. 25 years later 75 .

Tanaka & Taylor. • Participants provided their own controls.Dog and Bird Expert Study • Each expert had a least 10 years experience in their respective domain of expertise. 1991 Back propagation. 25 years later 76 . • None of the participants were experts in both dogs and birds.

25 years later 77 .Object Verification Task Superordinate Animal Plant Basic Subordinate Bird Robin Dog Sparrow YES NO YES NO Back propagation.

900 Novice Domain Expert Domain 800 700 Downward Shift 600 Superordinate Animal Back propagation. 25 years later Basic Bird/Dog Subordinate Robin/Beagle 78 .Dog and bird experts recognize objects in their domain of expertise at subordinate levels.

25 years later 79 . Bush Indigo Bunting 2002 Series 7 BMW Back propagation.Is face recognition a general form of perceptual expertise? George W.

2001 80 Back propagation. 25 years later .Face experts recognize faces at the individual level of unique identity 1200 Objects Faces 1000 800 Downward Shift 600 Superordinate Basic Subordinate Tanaka.

see also Gauthier. Nature Neuro. 2001. Bentin. Curby & Collins. Perez & McCarthy. 2003. 1996 Novice Back propagation. Allison. 25 years later Domain Expert Domain81 . Curran. Puce.Event-related Potentials and Expertise Face Experts Object Experts N170 Tanaka & Curran.

Neuroimaging of face, bird and car experts
Cars-Objects
Fusiform Gyrus

Birds-Objects

Car Experts
Fusiform Gyrus

“Face Experts”

Bird Experts
Gauthier et al., 2000
Back propagation, 25 years later

Fusiform Gyrus
82

How to identify an expert?
Behavioral benchmarks of expertise
• Downward shift in entry point recognition • Improved discrimination of novel exemplars from learned and related categories

Neurological benchmarks of expertise
• Enhancement of N170 ERP brain component • Increased activation of fusiform gyrus

Back propagation, 25 years later

83

End of Tanaka Slides

84

• Kanwisher showed the FFA is specialized for faces for what??? •But she forgot to control Back propagation. 25 years later 85 .

1999) • Subjects trained over many hours to recognize individual Greebles. 25 years later 86 . Back propagation. • Activation of the FFA increased for Greebles as the training proceeded.Greeble Experts (Gauthier et al.

birds. dogs. Proceedings of the Cognitive Science Society Back propagation. do they? Sugimoto & Cottrell (2001). 25 years later 87 . they don’t share features. • But the question remains: why would an area that presumably starts as a face area get recruited for these other visual tasks? Surely.The “visual expertise mystery” • If the so-called “Fusiform Face Area” (FFA) is specialized for face processing. then why would it also be used for cars. or Greebles? • Our view: the FFA is an area associated with a process: fine level discrimination of homogeneous categories.

Sugimoto & Cottrell (2001).“basic” level areas and the “expert” (FFA) area. • We will create • • • • Basic level models . Proceedings of the Cognitive Science Society Back propagation. Then we can deconstruct the winner to see why they won.Solving the mystery with models • Main idea: • There are multiple visual areas that could compete to be the Greeble expert . Then we will put them in a race to become Greeble experts.that’s what makes it an expert • Perhaps these features will be useful for other fine-level discrimination tasks. • The expert area must use features that distinguish similar looking inputs -.trained to identify individual objects.trained to identify an object’s class Expert level models . 25 years later 88 .

” Back propagation.Model Database • A network that can differentiate faces. books. cans OR books) is an “expert.” •A network that can also differentiate individuals within ONE class (faces. cups and cans is a “basic level network. cups. 25 years later 89 .

(Experts) can cup book Bob Ted Carol Greeble1 Greeble2 Greeble3 can cup book face (Non-experts) Hidden layer Back propagation.Model • Pretrain two groups of neural networks on different tasks. 25 years later Greeble1 Greeble2 Greeble3 90 . • Compare the abilities to learn a new individual Greeble classification task.

Back propagation. leads to faster learning of Greebles (can’t try this with kids!!!). 25 years later 91 . books. The more expertise. the FCA (Fusiform Can Area) would win.Expertise begets expertise Amount Of Training Required To be a Greeble Expert Training Time on first task • • • • Learning to individuate cups. If our parents were cans. FFA would win. cans. or faces first. the faster the learning of the new task! Hence in a competition with the object area.

0 -max(output)) Human data --.Entry Level Shift: Subordinate RT decreases with training (rt = uncertainty of response = 1.Subordinate Basic Network data RT Back propagation. 25 years later # Training Sessions 92 .

How do experts learn the task? • Expert level networks must be sensitive to within-class variation: • Representations must amplify small differences • Basic level networks must ignore within-class variation. • Representations should reduce differences Back propagation. 25 years later 93 .

25 years later 94 . Back propagation. • We can then observe how tightly clustered stimuli are in a low-dimensional subspace • We expect basic level networks to separate classes. but not individuals. • We expect expert networks to separate classes and individuals.Observing hidden layer representations • Principal Components Analysis on hidden unit activation: • PCA of hidden unit activations allows us to reduce the dimensionality (to 2) and plot representations.

25 years later 95 .Subordinate level training magnifies small differences within object representations 1 epoch 80 epochs 1280 epochs Face greeble Basic Back propagation.

Greeble representations are spread out prior to Greeble Training Basic greeble Face Back propagation. 25 years later 96 .

We used linear regression to find the best linear combination of Gabor magnitude principal components for each image pixel.Examining the Net’s Representations We want to visualize “receptive fields” in the network. But the Gabor magnitude representation is noninvertible. however. 25 years later 98 . We can learn an approximate inverse mapping.” • • • • Back propagation. • Then projecting each hidden unit’s weight vector into image space with the same mapping visualizes its “receptive field.

Two hidden unit receptive fields AFTER TRAINING AS A FACE EXPERT AFTER FURTHER TRAINING ON GREEBLES HU 16 HU 36 NOTE: These are not face-specific! Back propagation. 25 years later 99 .

The Face and Object Processing System FFA Book Can Cup Face Bob Sue Jane Expert Level Classifier Gabor Filtering PCA Book Can Cup Face Basic Level Classifier LOC Pixel (Retina) Level Perceptual (V1) Level Gestalt Level Hidden Layer Category Level 105 Back propagation. 25 years later .

25 years later 106 . 2004.Effects accounted for by this model • • • • • • • • • • Categorical effects in facial expression recognition (Dailey et al. 2001. in preparation) • • • Back propagation. Joyce & Cottrell.. 2005) Priming effects (face or character) on discrimination of ambiguous chinese characters (McCleery et al. 1999) How the FFA could be recruited for other areas of expertise (Sugimoto & Cottrell. 2000) How a specialized face processor may develop under realistic developmental constraints (Dailey & Cottrell. Tran et al. and the lack of interaction between the two (Cottrell et al. L&C. featural. 1999) Cultural effects in facial expression recognition (Dailey et al. Tong et al. under review) Memory for faces (Dailey et al.. 2006) Holistic processing of identity and expression.. 2004. 2005) The other race advantage in visual search (Haque & Cottrell. 2005) Generalization gradients in expertise (Nguyen & Cottrell.. and inverted face discrimination (Zhang & Cottrell. 2002) Similarity effects in facial expression recognition (ibid) Why fear is “hard” (ibid) Rank of difficulty in configural. under review) Age of Acquisition Effects in face recognition (Lake & Cottrell. 2005. 1998.

25 years later • Backprop is important because it was the first relatively efficient method for learning internal representations • Recent advances have made deeper networks possible • This is important because we don’t know how the brain uses transformations to recognize objects across a wide array of variations (e.g. 25 years later 107 .. the Halle Berry neuron) Back propagation.Backprop.

25 years later 108 .. the “Halle Berry” neuron… Back propagation.• E.g.

109 .

Sign up to vote on this title
UsefulNot useful