Executive

Functioning: BRIEF &

ROCF
Carolina Lancastre, Elisabeth
Harvey, Tooba Fatima, Yang Yu
Overview

● Executive Functioning

● Tests:

○ Rey-Osterrieth Complex Figure (ROCF)

○ Behavior Rating Inventory of Executive Functioning (BRIEF)

● Validity

● Reliability

● Cultural-Fairness

● Clinical Applications
What is Executive Functioning?

● Executive Functioning
○ “top-down mental processes needed when you have to concentrate and pay attention, when
going on automatic or relying on instinct would be ill-advised, insufficient or impossible”
(Diamond, 2013)
● “It is not that the individual does not know what to do. It is that somehow it
does not get done.” (Barkley, 2012)
Executive Functioning

3 Core Executive Functions:

1) Working memory - the capacity to store and transform information.

2) Inhibition - the ability to prevent or block a cognitive or behavioral response, as well as selective
attention.

3) Cognitive flexibility - the ability to shift between sets of tasks or rules

Why is executive function important?

1) It predicts how one will achieve in Reading and Mathematics (Blair & Razza, 2007; Bull, Espy &
Wiebe, 2008).

2) Poor executive function is connected to ADHD (Clark, Prior, & Kinsella, 2002).

3) It gives us the ability to ignore irrelevant or distracting stimuli and focus our attention.

4) Gives us the capacity to control automatic or intuitive responses.

5) It is key for successful everyday functioning and problem solving.

 ROCF: Rey-Osterrieth
Complex Figure Copying
Test
ROCF - an overview

Objective: Evaluation of different cognitive functions, such as visuo-spacial abilities, memory attention,
planning and working memory (executive functions).

Instrument type: Test

Publication Date: 1941/1944

Duration: Not timed, but length of time needed recorded

ROCF

● Format:
○ Examinees are asked to reproduce a complicated line drawing.
○ They are not told initially that they have to remember it
● 3 Conditions
○ Copy: examinees copy image to the best of their ability with image in front of them
○ Immediate Recall: examinees draw image to the best of their ability from memory
○ Delayed Recall: after a period of time (~20-30 min), examinees draw the image again to the
best of their ability from memory
Developmental Scoring System (DSS)

Publication Date: 1985

Purpose: A qualitative approach that takes developmental progression into consideration.

Scores are obtained from three aspects: organization, accuracy and errors.

Reliability: Interrater reliability = 0.95

Boston Qualitative Scoring System

Publication date: 1994-1999

Purpose: Designed as a Quantifiable approach to rating the qualitative features of the ROCF

ROCF is divided into three elements, configural elements, clusters and details

Scores are obtained from 17 dimensions of qualitative scores and 6 summary scores.

Reliability:In the manual, interrater reliability and test-retest reliability were reported.

Kappa coefficient for interrater reliability ranged from 0.53-1.00 for copy condition

Test-retest reliability in copy condition = 0.50, immediate recall = 0.66, and delayed recall = 0.68
BQSS (continued)

● n = 108, 46 with right hemisphere stroke, 20 with left hemisphere stroke, 21

with dementia and 20 in normal control
● Interrater reliability coefficient: 0.63-0.99, except the asymmetry scale = 0.20
● Internal consistency: 0.78-0.91
ROCF with BQSS - Validity

Does the ROCF scored with BQSS measure Executive Functioning?

Content validity: The extent to which it matches the executive function domain. First developed to
measure visual constructional ability and visual memory, in brain-impaired patients, so there is some
construct underrepresentation and irrelevance.

Construct Validity:

a) Convergent Validity: With Original 36-point scoring system - 0.95 (but only copy)

With the Wisconsin Card Sorting Test - ranged from -.29 to -.39. “Statistically significant, far from
robust.” 16% shared variance.

With Controlled Oral Word Association Test - 0.33 to 0.49

ROCF - Validity (cont.)

b) Discriminant validity: - several studies done to show sensitivity and to differentiate between groups
(HIV+, Alzheimer, dementia, traumatic brain injury). Scores between 72% and 82%, which demonstrate
sensitivity but the groups where small (between 16 and 33)

In conclusion, some aspects of executive functioning, like organizing and planning can be evaluated
using the ROCF with the BQSS.
ROCF-Cultural Fairness

● Can be used on wide age range of adult
participants
● Conditions for test administration are
universal for all examinees
● Mostly white participants from the U.S.
Northeast and Canada of middle/upper
middle class
● Clear gender differences with female
participants scoring significantly lower
● Has mostly only been tested on those with
confirmed or suspected
neuropsychological disorders (not for in-
group differences)

Sources: Rey, & Osterrieth, 1941; Stern, Javorsky, Singer, Harris, Somerville,
Duke, & Kaplan, 1999
BRIEF: Behavior Rating
Inventory of Executive
Functioning
BRIEF - an overview
Objective: Designed to assess impairment of executive function, behavior in the home and school
environment.

Intended Population: ages 5-18

Instrument type: Scale

Duration: 15/20 minutes, can be administered as group or individual

Publication dates: 1996/2000

BRIEF: Test Materials

● Includes Teacher Form and Parent Form, which ask about the child’s behaviors as observed.
○ The BRIEF 86 items on both the Parent and Teacher Forms.

● For each behavior, parents or teachers choose if the behaviors is:

○ N: Never a Problem
○ S: Sometimes a Problem
○ O: Often a Problem

● Scoring:
○ Raw scores for scale and all sub-scales are converted to t-scores and percentiles.
○ Negativity Scores are calculated for some items.
○ An Inconsistency Scale is built into the instrument.
BRIEF: Clinical Scales

● It includes 8 clinical scales:

○ Inhibit: ability to inhibit, resist, or not act on an impulse.

○ Shift: ability to shift attention from one situation, activity, or aspect of a problem to another.
○ Emotional Control: emotional regulation.
○ Initiate: beginning a task/activity, independently generating ideas.
○ Working Memory: capacity to hold information in the mind in order to complete tasks.
○ Plan/Organize: ability to manage current and future-oriented task demands.
○ Organization of Materials: orderliness of work, play, and storage spaces.
○ Monitor: work-checking habits during or after child completes tasks.
BRIEF: Sub-Scale and Composite Score

● Behavioral Regulation Index (sub-scale):

○ Inhibit + Shift + Emotional Control scales.
○ Represents a child’s ability shift cognitive set and modulate emotions/behavior via
appropriate inhibitory control.

● Metacognition Index (sub-scale):

○ Initiate + Working Memory + Plan/Organize + Organization of Materials + Monitor scales.
○ Represents child’s ability to initiate, plan, and organize.

● Global Executive Composite:

○ Summary of all clinical scales.
BRIEF - Validity

● Content Validity: degree to which content measures what it intends to measure.

○ Agreement was sought among a panel of 12 pediatric neuropsychologists.
○ Each expert assigned items to a primary or secondary scale.
○ Items with poor agreement were flagged, and checked against inter-rater agreement.

Item (Organization of Materials Sub- Item-Total Author Ratings Expert Ratings

Scale) Correlations (%) (%)

Backpack is disorganized. .76 100 88

Has a messy desk. .79 100 44

Cannot find things in room/school desk. .82 100 88

BRIEF - Validity

● Construct Validity: examines correlations with other, existing scales measuring similar constructs.

○ ADHD Rating Scale-IV

○ Child Behavior Checklist
○ Teacher’s Report Form
○ Behavior Assessment System for Children
○ Conner’s Rating Scale
 BRIEF Scale/Index Inattention Hyperactivity-Impulsivity

Inhibit .42 .73

Shift .39 .59

Emotional Control .39 .56

Initiate .55 .36

Working Memory .60 .44

Plan/Organize .63 .33

Organization of Materials .49 .15

Monitor .54 .45

Behavioral Regulation .44 .70

Metacognition .67 .38

Global Executive Composite .63 .60

 ● Parents of 200 clinically referred children completed BRIEF and CBC.

BRIEF Scale/Index CBC Scale Correlation

Withdrawn .50

Initiate Anxious/Depressed .52

Attention Problems .50

Working Memory Attention Problems .64

Attention Problems .58

Inhibit
Aggressive Behavior .73

Shift Emotional Control .67

Child Behavior Checklist

 ● Teachers of clinically referred children completed the Teacher Report Form and BRIEF

BRIEF Scale/Index Teacher’s Report Form Correlation

Inhibit .83

Shift .70
Aggressive Behavior
Emotional Control .81

Monitor .71

Initiate .69
Attention Problems
Working Memory .74

BRIEF Social Problems .35-.63

Teacher’s Report Form

 ● Completed by parents and teachers of 80 clinically referred children.

BRIEF Scale/Index Teacher’s Report Form Correlation

Aggression .76

Hyperactivity .63
Behavioral Regulation
Attention Problems .48

Conduct Problems .14

Aggression .49-.84

BRIEF Learning Problems .41-.69

Anxiety/Somatization -

Behavior Assessment System for Children

 ● Completed by parents of 25 clinically referred children.

BRIEF Scale/Index Teacher’s Report Form Correlation

Restless-Disorganized .71

Behavioral Regulation Conduct Disorder .77

Hyperactive-Immature .57

Obsessive-Compulsive -
BRIEF
Antisocial -

● Collectively, these correlations provide evidence for convergent/divergent validity.

● Executive functions as measured on the BRIEF correlated less strongly with measures
of emotional functioning.

Conner’s Rating Scale

BRIEF - Predictive Validity & Clinical Utility

● The Working Memory and Inhibit scales are especially useful in predicting ADHD-I and ADHD-C.
○ Successful predictions ranging from 70-85%
BRIEF - Reliability

● Internal Consistency: The degree to which items in a single scale are measuring the underlying
construct. α = 0.8-0.98 for both Parent and Teachers form, indicating the internal consistency is
high.

● Interrater Reliability: The degree to which two independent observers rate a child in a similar
manner. Parent/Teacher r = 0.3-0.5. Two of the scales are notably lower: Initiate r =0.18,
Organization of Materials r= 0.15

● Test-Retest Reliability: The stability of a measure over time for behaviors that are presumed to
remain relatively constant.
 Scale/Index Parent clinical a Parent normative b Teacher normative c

Inhibit 0.76 0.84 0.91

Shift 0.72 0.78 0.83

Emotional Control 0.79 0.79 0.92

Initiate 0.77 0.80 0.87

Working Memory 0.82 0.85 0.86

Test Retest
Correlation Plan/ Organize 0.80 0.85 0.88

Organization of 0.84 0.79 0.83

Materials

Monitors 0.80 0.76 0.87

Behavioral Regulation 0.80 0.84 0.92

Metacognition 0.83 0.88 0.90

Global Executive 0.81 0.86 0.91

Composite
BRIEF - Cultural Fairness
● Considers needs of those who have difficulties ● Doesn’t consider the linguistic functioning of

with “traditional” testing with alternative forms those who are bi/multilingual.

available ● Defines what tasks are considered “complex” and

● Observes executive functioning contextually “automatic” for children.

(both in school and in-home). ● Assumes that given environment is suitable for

● Allows for people without extensive assessment observation/evaluation.

training to gather data ● More than half of participants came from

● Considers wide range of variables that may affect suburban backgrounds (~59%).

executive functioning ● Majority of parents observing were mothers

○ EX: ADHD, Traumatic Brain Injury, etc. (~83%)

● Diverse sample used ● Potential bias from depending on rater-child

relationship

Source: Guy, Isquith, & Gioia, 2004

Conclusion

● BRIEF
○ Better standardization, validity, and reliability.
○ Provides a good overview of executive functioning
○ Allows clinicians to narrow down or further explore further particular executive functioning
domains that require attention.

● ROCF
○ Helpful as a supplemental test or to explore certain functions, such as working memory
capacity, of executive functioning.

Both should be used when trying to evaluate executive functions, but none would be sufficient, on their
own or together, for a diagnosis.
Discussion Question

Most executive functioning tests have relatively low ecological validity, since the
conditions in which they are tested are usually artificial. In such conditions, it is
difficult to test real life executive functioning.

Can you think of a way of increasing ecological validity for these tests?
References

Barkley, R.A. (2012). Executive functions: What they are, how they work, and why they evolved. New York:
Guilford Press.

Blair, C. & Razza, R.P. (2007). Relating effortful control, executive function, and false belief
understanding to emerging math and literacy ability in kindergarten. Child Development, 78(2), 647-663.
doi: 10. 1111/j.1467-8624.2007.01019.x

Bull, R., Espy, K.A., & Wiebe, S.A. (2008). Short-term memory, working memory, and executive
functioning in preschoolers: Longitudinal predictors of mathematical achievement at age 7 years.
Developmental Neuropsychology, 33(3), 205-228. http://doi.org/10.1080/87565640801982312

Canham, R., Smith, S., & Tyrrell, A. (n.d.). Automated scoring of a neuropsychological test: the Rey
Osterrieth complex figure. Proceedings of the 26th Euromicro Conference. EUROMICRO 2000.
Informatics: Inventing the Future. doi:10.1109/eurmic.2000.874519

Clark, C., Prior, M., & Kinsella, G.J. (2002). The relationship between executive functioning abilities,
adaptive behaviour, and academic achievement in children with externalising behaviour problems.
Journal of Child Psychology and Psychiatry 43(6), 785-796. doi: 10.1111/1469-7610.00084.
Diamond, A. Executive Functions. Annual Review of Psychology, 64, 135-168. doi: 10.1146/annurev-
psych--113011-143750.

Folbrecht, J.R., Charter, R.A., Walden, D.K., & Dobbs, S.M. (1999). Psychometric properties of the Boston
Qualitative Scoring System for the Rey-Osterrieth Complex Figure. The Clinical Neuropsychologist, 13(4),
442-449. doi: 10.1076/1385-4046(199911)13:04;1-y;ft442.

Gioia, G. A. (2000). Behavior Rating Inventory of Executive Function: BRIEF. Odessa, FL: Psychological
Assessment Resources.

Guy, S.C., Isquith, P.K, & Gioia, G.A. (2004). Behavioral Rating inventory of Executive Functioning.

Ogino, T., Wantabe, K, Nakado, K., Kado, Y., Morooka, T., Takeuchi, A., & Ohtsuka, Y. (2009). Predicting
executive function task scores with the Rey-Osterrieth Complex Figure. Brain and Development, 31(1),
52-57. doi: 10.1016/j.braindev.2008.07.003

Rey, A., & Osterrieth, P.A.A. (1941). Rey-Osterrieth Complex Figure Copying Test.
http://dx.doi.org/proxy.bc.edu/10.1037/t07717-000

Shin, M.S., Park. S.Y., Park, S.R., Seol, S.H., & Kwon, J.S. (2006). Clinical and empirical applications of the
Rey-Osterrieth Complex Figure Test. Nature Protocols, 1(2), 892-899. doi: 10.1038/nprot.2006.115.
Somerville, J., Tremont, G., & Stern, R.A.(2000). The Boston Qualitative Scoring System as a measure of
executive functioning in Rey-Osterrieth Complex Figure performance. Journal of Clinical and
Experimental Neuropsychology, 22(5), 613-621. doi: 10.1076/1380-3395(200010)22:5;1-9;ft613

Stern, R.A., Javorsky, D.J., Singer, E.A., Harris, N.S., Somerville, J.A., Duke, L.M., & Kaplan, E. (1999). The
Boston Qualitative Figure for the Rey-Osterrieth Complex Figure.

Stern, R.A., Singer, E.A., Duke, L.M., Singer, N.G., Morey, C.E., Daughtry, E.W., & Kaplan, E. (1994). The
Boston Qualitative Scoring System for the Rey-Osterrieth complex figure: Description and interrater
reliability. The Clinical Neuropsy- chologist, 8, 309–322.

Waber, D.P. & Holmes, J.M. Assessing children’s copy productions of the Rey–Osterrieth Complex
Figure. J. Clin. Exp. Nueropsychol. 7, 264–280 (1985).

Wantabe, Ogino, T., Nakado, K., Kado, Y., Morooka, T., Takeuchi, A., & Ohtsuka, Y. (2005). The Rey-
Osterrieth Figure as a measure of executive function in childhood. Brain and Development, 8, 564-569.
doi: 10. 1016/j.braindev.2005.02.007