Assessment: Meaning and Interpretation

Dr Barry Johnson Dyslexia Action

The aim of this session is to update diagnostic assessors on developments within the area of ability-attainment discrepancy analysis and, in so doing, help them to be more confident with their day-to-day assessment practices.

Disclaimer This presentation reflects the personal thoughts and opinions of Barry Johnson, HPC registered, educational psychologist and not the policies of Dyslexia Action.

Is an ability attainment-discrepancy analysis for sp.l.d. diagnosis ...

Irrelevant

Necessary?

Sufficient?

Useful?

IQ

Piano playing skills

5

IQ

Piano playing skills

6

IQ

Piano playing skills

7

High IQ High Level Piano Skills Clevercloggies

High IQ Low Level Piano Skills

Low IQ High Level Piano Skills

Low IQ Low Level Piano Skills

High IQ High Level Piano Skills Clevercloggies

High IQ Low Level Piano Skills Dyspianolexics

Low IQ High Level Piano Skills

Low IQ Low Level Piano Skills

High IQ High Level Piano Skills Clevercloggies

High IQ Low Level Piano Skills Dyspianolexics

Low IQ High Level Piano Skills Idiotsavant-piano syndrome

Low IQ Low Level Piano Skills

10

High IQ High Level Piano Skills Clevercloggies

High IQ Low Level Piano Skills Dyspianolexics

Low IQ High Level Piano Skills Idiotsavant-piano syndrome

Low IQ Low Level Piano Skills Dimbutnoisies

11

IQ

Memory for sounds

Finger dexterity

Musical Aptitude

Piano playing skills

Music sight reading
12

IQ

Memory for sounds

Finger dexterity

Musical Aptitude

Piano playing skills

Music sight reading
13

IQ

Memory for sounds

Finger dexterity

Musical Aptitude

Piano playing skills

Music sight reading
14

Aptitude-Achievement Consistency Analysis

Aptitude-Achievement Consistency Analysis Coo! What is this? A contemporary alternative to the traditional ability-attainment discrepancy model used to help in the diagnosis of dyslexia/sp.l.d.
15

Aptitude-Achievement Consistency Analysis

What are the main criticisms of ability-attainment discrepancy analysis? (1) no close & clear link between assessment and teaching. (2) use of arbitrary statistical cut-offs to judge degree of discrepancy/clinical worth. (3)over-reliance on old models of intelligence - g or VIQ/Vis. IQ (4)Pragmatics time, statistical knowledge, etc. (5) Need for statistical knowledge (CCET plug!)
16

Aptitude-Achievement Consistency Analysis

But, there is a claim that ability-attainment discrepancy analysis is still useful/needed in that it: is clearly evident in daily practice - assessments and reports. mentioned in DfES DSA Working Party Report. has an oblique reference in SEN Code of Practice. is offered in test manuals have ability-attainment look-up tables in Appendices. is listed in Dyslexia Action s Consulting Psychologists quality assurance requirements for report writing. Is used as an optional facility for Dyslexia Index - Martin Turner s Dyslexia Portfolio (GL-Assessment). Is fed by consumer/cultural demand underachievement.
17

Aptitude-Achievement Consistency Analysis

SEN Code of Practice
7:39 However, academic attainment is not in itself sufficient for LEAs to conclude that a statutory assessment is or is not necessary. An individual child s attainment must always be understood in the context of the attainments of the child s peers, the child s rate of progress over time and, where appropriate, expectations of the child s performance. A child s apparently weak performance may, on examination of the evidence, be attributable to wider factors associated with the school s organisation. Careful consideration of evidence of low attainment may reveal good progress from a low base.
18

Aptitude-Achievement Consistency Analysis

SpLD Working Group 2005/DfES Guidelines (DSA)

Underlying Ability / Achievement differentials

Although a discrepancy between underlying ability and attainment in literacy skills is not a diagnostic criterion (Frederickson & Reason 1995, Howe 1997, Miles 1996, Stanovich & Stanovich 1997, Siegel 1999), where such discrepancies do exist, they provide further supporting evidence.
19

SpLD Working Group 2005/DfES Guidelines (cont.) Gathering information about underlying ability is an important component of assessment. The assessment of verbal and non-verbal ability throws light on the extent to which students are likely to be able to develop compensatory strategies, and informs specialist teaching intervention. The effect of SpLD on a student s learning can be evaluated more effectively when underlying ability is taken into account.
20

SFE revised guidance for key delivery partners: It is recommended that a full diagnostic assessment of a higher education student for dyslexia should contain the following

21

Evidence of any significant difficulties in reading, writing, spelling and numeracy relevant to the standards in these skills required in higher education. Evidence of a significant discrepancy between the abilities in reading, writing, spelling and numeracy and the level of those abilities expected of the student in terms of their general intellectual ability. Evidence of cognitive disabilities or neurological anomalies (affecting memory, visual perception, phonological processing, including speed of processing or motor coordination) which are likely to have adverse effects on learning. Evidence that the learning difficulties identified are likely to affect learning in higher education.
22

Aptitude-Achievement Consistency Analysis

Some difficulties experienced by assessors: Lack of a range of up-to-date cognitive ability and attainment tests. Lack of co-normed ability-attainment look-up tables. Lack of statistical knowledge in order to compare scores if tests used are not co-normed standard error of measurement, simple regression, standard error of estimate, confidence ranges, etc.
23

24

Aptitude-Achievement Consistency Analysis

If one uses an IQ test and an attainment test that have not been co-normed, then regression to the mean needs to be taken account of when comparing discrepancy scores. Regression to the mean is affected by the correlation coefficients between the IQ and attainment tests. The lower the correlation, the bigger the regression effect.
25

Simple Regression formula

yd rxy X  100  100 !

Regressed score

IQ
Correlation coefficient for IQ & Attainment Test
26

Expected Regressed Scores

Test A
rIQ TEST 0.4

Test B
0.6

Test C
0.8

Test D
0.9

IQ 130 100 70 112 100 88 118 100 82 124 100 76 127 100 73
27

28

Note correlation coefficients

29

No regression at mean

30

Significant regression at tails of distributions

31

Aptitude-Achievement Consistency Analysis

There are five dimensions of importance when applying A-ACA to a diagnosis of sp.l.d.: (1) The examination of the degree to which a weakness in academic achievement is consistent/positively correlated with related cognitive abilities/aptitudes. (2) A-ACA is concerned in part with evaluation of consistency among ability/aptitude & academic achievement of below normal limits - below the normal range is 85-115 standardised scores.

32

Aptitude-Achievement Consistency Analysis

(3) The evidence that the child has intact ability and academic functioning in areas largely unrelated to the presenting problem. (4) The use of CHC model of cognitive abilities. (5) The use of aptitudes (narrow areas of cognitive ability) as a conceptual bridge between broad cognitive abilities and academic skills. Interpretation is focused at the cluster level, yielding better reliability. (6) focus on exclusionary factors
33

Aptitude-Achievement Consistency Analysis

Broad abilities Narrow aptitudes Attainments

34

Aptitude-Achievement Consistency Analysis

Rather than conceiving of cognitive abilities and academic achievements as mutually exclusive, they may be better thought of as lying on an ability continuum that has the most general types of abilities at one end and the most specialized types of knowledge at the other end (Carroll, 1993).
35

Aptitude-Achievement Consistency Analysis

You may have noticed that USA terminology is confusing this is the reason attainments/skills are commonly referred to as abilities. CHC theory has influenced the use of such conceptual labels.

36

Aptitude-Achievement Consistency Analysis

Brief summary of CHC Theory The Cattell-Horn-Carroll Theory of cognitive development is a synthesis of the models by Raymond Cattell, John Horn, and John Carroll. Cattell postulated that there were two overall abilities people have: Crystallized intelligence and fluid intelligence. Crystallized intelligence reflected abilities that were relatively static (such as learned information) while fluid intelligence was more related to novel problem solving. (WRIT type model)
37

Aptitude-Achievement Consistency Analysis

John Horn expanded this model by adding seven to nine (depending on your theoretical orientation) broad abilities. They include:-

38

Crystallized Intelligence (Gc) Processing Speed (Gs)

Fluid Reasoning (Gf)

Auditory Processing (Ga) Long-Term Retrieval (Glr) Quantitative Reasoning (Gq)

39

Short-Term Memory (Gsm) Reading/ Writing ability (Grw)

Visual-Spatial Processing (Gv)

Aptitude-Achievement Consistency Analysis

After analyzing past years of data, John Carroll came up with his own set of broad abilities. He also offered a three-stratum theory of cognitive development. Stratum III represents g or overall intelligence. Stratum II represents the broad abilities Stratum I represents the narrow abilities grouped under the broad abilities (for example, the broad construct of Gsm includes the narrow abilities of working memory and memory span.)
40

Aptitude-Achievement Consistency Analysis

John Horn and John Carroll agreed to synthesize their theories (Carroll s three stratum theory with Horn s broad abilities). The result is the CHC theory of cognitive development.

41

F lu id In te l l i g e n c e ( G f)

Q u a n tita tive K n o w le d g e (G q )

C r y st a l l i ze d In t e l l i g e n c e (G c)

R e a d in g a n d W r itin g (G rw )

S h o r t-T e r m M em ory (G s m )

V i su a l P r o c e ssi n g (G v )

A u d i to r y P r o c e ssi n g (G a )

L o ng -T e r m Sto r a g e & R e tr ie va l (G l r )

P r o c e ssin g Speed ( G s)

D e c i si o n / R e a c ti o n Tim e/S p eed (G t)

G e n e ra l Se q u e n tia l R e a so n in g In d u c tio n
Q u a n tita tiv e R e a so n in g P ia ge tia n R e a so n in g

M a th . K now . M a th . A ch.

L a n gu a ge D e v e lo p .
L e x ic a l K n o w l. L is te n in g A b ility
G e n e ra l In fo .

R e a d in g D e c o d in g R e a d in g C om p. V erbal L anguage C om p. C lo ze A b ility S p e llin g A b ility W r itin g A b ility

M e m o ry Sp a n
L e a rn in g A b ilitie s

Visu a liza tio n

Sp a tia l R e la tio n s V isu a l M em ory

P h o n . C d g .: A n a l y si s P h o n . C d g .: S y n t h e si s

A sso c . M em ory M n g fu l . M em ory F ree R e c a ll M em ory

P e rc e p tu a l Sp e e d
R a te -o fT e st T a k in g N um ber F a c ility

Sim p le R e a c tio n T im e C h o ic e R e a c tio n T im e Se m a n tic P r o c e ssin g Sp e e d M e n ta l C o m p a r iso n Sp e e d

Sp e e c h Sn d . D isc rim . R e s. to A u d . Stim . D isto rtio n M e m ory fo r So u n d P a tte rn s G e ne ra l Sn d . D isc rim . T e m pora l T r a c k in g

C lo su re Sp e e d
F le x ib ility o f C lo su re Sp a tia l Sc a n n in g Se r ia l P e r c e p tu a l In te gra tio n L e n gth E stim a tio n

Id e a tio n a l F lu e n c y A sso c . F lu e n c y
E x p r e s s io n a l F lu e n c y N a m in g F a c ility

Sp e e d o f R e a so n in g

In fo . about C u ltu re
G e n e ra l S c ie n c e In fo . G e o g ra p h y A ch.

E n g lish U sa g e K n o w le d g e
R e a d in g Speed

C omm. A b ility O ra l P ro d u c tio n & F lu e n c y G ra m . Se n sitiv ity

F o r e ign L a n gu a ge P ro fic ie n c y F o r e ign L a n gu a ge A p titu d e

P e rc e p tu a l I llu sio n s P e rc e p tu a l A lte r n a tio n s Im a ge ry

M u sic a l D isc r im . & Ju d ge m e n t

M a in ta in in g & Ju d gin g R h y th m Sn d - I n te n sity D u r a tio n D isc rim . Sn d - F r e q . D isc r im . H e a r in g & Sp e e c h T h re sh o ld A b so lu te P itc h

W o rd F lu e n c y F igu ra l F lu e n c y F igu ra l F le x ib ility

S e n s itiv ity to P ro b le m s O r igin a lity / C r e a tiv ity

L e a rn in g A b ilitie s

So u n d L o c a liza tio n

42

Aptitude-Achievement Consistency Analysis

Cognitive ability and attainment test design in the USA is aligning with the CHC model. Up-to-date cognitive ability tests such as the WISC IV and WAIS IV have better alignment with CHC than older tests. XBA (Cross Battery Assessment) - commonly referred to in USA. Lot of research collation linking aptitudes with attainments, feeding back into dynamic changes of CHC models.
43

Aptitude-Achievement Consistency Analysis

Seems as though this process of taxonomy linking aptitudes with attainments - is scientific, but is also helpful for moderation purposes it promotes agreement in field. Is dynamic and changing according to ongoing research findings.

44

Aptitude-Achievement Consistency Analysis

Following the evolution of CHC theory, cross battery assessment emerged as a way to assess students. The principle is to select tests from varied batteries which best match the referral concern. The examiner then puts the scores into a cross-battery template, where the stratum II and I levels are aggregated (the subtests were matched with their stratum I and II counterparts by an expert consensus study.) Those scores are then averaged. Although combining subtests and averaging may not seem like best practice, it has been noted to be an empirically defensible practice (See the FAQ section in http://www.crossbattery.com)
45

Aptitude-Achievement Consistency Analysis

In essence, diagnostic assessors need: (1) a causal model/framework in their heads that link underlying aptitudes with the apparent learning difficulties (2) tests to sample these aptitudes (3) statistical means (confidence ranges will do) by which they can observe consistencies/inconsistencies across these aptitudes and attainments.
46

Aptitude-Achievement Consistency Analysis

(con.) (4) confidence to conclude the evidence (5) reduced reliance on a model that either drives the perceived need to have a measure of untainted/pure g or where an ability-attainment discrepancy is perceived to be the main factor to drive the diagnosis of dyslexia/sp.l.d. (5) assertiveness to apply and defend a normative weakness model of disability/learning difficulties.
47

Aptitude-Achievement Consistency Analysis

In USA there is a huge test industry: very powerful and influential. The big hitters of tests in this area are:

48

Aptitude-Achievement Consistency Analysis

WJ III (2001) Abilities/WJ III Woodcock Johnson III Cognitive abilities/Cognitive Achievement SB5 (2003) Stanford Binet Intelligence Scales KABC-II (2004)/KTEA II Kaufman Assessment Battery for Children, Second Edition (KABC-II) /Kaufman Test of Educational Achievement Second Edition DAS-II (2007) Differential Ability Scales Second Edition (DAS-II)

49

Given the historical predominance of the discrepancy model, evaluation of consistency may appear rather strange at first. An aptitude score is comprised specifically of tests that are most directly relevant to the development and acquisition of specific academic skills and thus is the best predictor of the corresponding achievement area. For example, an individual with low reading ability and isolated cognitive deficits in one or more aptitude areas related to reading achievement (e.g., phonological awareness, processing speed, short-term memory) will most likely demonstrate consistency between scores of reading aptitude and reading achievement. Likewise, a high reading aptitude score would predict high reading achievement the two scores are more likely to be consistent with each other rather than discrepant. Flanagan & Ortiz (2006)

50

Aptitude-Achievement Consistency Analysis

Let us return to the A-ACA position on the importance of the evaluation of consistency among ability/aptitude & academic achievement of below normal limits - below the normal range is 85-115 standardised scores. Note: this de-emphasises the importance of abilityattainment discrepancy when attainment scores are average or above-average. It also introduces a quantifiable criterion to facilitate decision making in borderline cases and reduce overall numbers of sp.l. being diagnosed.
51

Aptitude-Achievement Consistency Analysis

(cont.) It is relevant to formal/legal definitions and interpretations of disability in that a strict qualification/cut-off is given. Evidence of average/above-average levels within the academic domain of concern would lead to sp.l.d. not to be diagnosed. The null hypothesis stops at the first stage. Hunting for low scores or large discrepancies between scores (mass profiling/Bonferroni problem) is discouraged.
52

Aptitude-Achievement Consistency Analysis

Note that the lower cut off point of <85 s.s. is already acknowledged as valid for JCQ access regulations in UK. Disability legislation does drive strict/severe interpretations of disability that use psychometric principles.

53

54

Aptitude-Achievement Consistency Analysis

How would adoption of A-ACA change diagnostic practice? (1) Better co-normed tests that span the abilityaptitude-attainment areas and in-line with CHC models. (2) Stricter definitions of sp.l.d. and transparency of diagnosis (3) less use of FSIQ/VIQ/VisIQ attainment comparison
55

Aptitude-Achievement Consistency Analysis

What would all this look like? Would there be much change to our assessment paradigms and practices? Not as great as you think, particularly with better availability of co-normed up-to-date tests that merge and span the ability-aptitude-attainment range. Even without these, cross-assessment battery selection would help, provided assessors used slightly more sophisticated statistics to help them. We do have many research findings that have highlighted aptitude-attainment links to literacy development neatly summarised in, for example, :
56

Aptitude-Achievement Consistency Analysis

Snowling, M.J. & Hume, C. (2010). Evidencebased interventions for reading and language difficulties: creating a virtuous circle. British J. of Ed. Psychology, 81, 1-23

57

Aptitude-Achievement Consistency Analysis

What would reports look like? Not that much different, but advantages are that the diagnostic process would be transparent, shared and therefore more consistent, and would facilitate moderation processes to be applied in cases of borderline cases particularly where external criteria were not present (such as DSA) .
58

Aptitude-Achievement Consistency Analysis

Megan reports having difficulties with single word reading recognition. I found that she has weakness with her phonetic coding (narrow Ga), naming facility (narrow Glr) and speed of processing ( Gs) which are probably causally liked to her confirmed below average single word reading skills. In that she also has average abilities within the broad ability areas of general knowledge, fluid reasoning and attainment area of numerical operations, I would conclude that she has a specific learning difficulty Below-average aptitude-achievement consistency Average/above average ability and/or attainment
59

Aptitude-Achievement Consistency Analysis

Blodwyn, a H.E. student, is concerned that she has a specific learning difficulty in that she believes she has weak spelling skills. However, my assessment demonstrates that her spelling skills lie within the average range for her age and that her other core literacy skills lie either within the average of above average ranges. In that her core literacy skills are not at below average levels, the presence of a sp.l.d. in the area of concern can be discounted Below-average aptitude-achievement consistency x Average/above average ability and/or attainment
60

Aptitude-Achievement Consistency Analysis

Ieuan has below-average levels for both reading aptitude and core reading skill levels for his age. He does not have average or above-average aptitude/ability levels in other areas such as quantitative reasoning, memory and vocabulary knowledge. I therefore conclude that he does not have sp.l.d. but has general learning difficulties Below-average aptitude-achievement consistency Average/above average ability and/or attainment x
61

Dr Barry Johnson Dyslexia Action Knowle House 4 Norfolk Park Road Sheffield S2 3QE

T 01142 815909 M 07970 662268

bjohnson@dyslexiaaction.org.uk

62