Session #65

Draft Results of Quality Assurance Program Data from Award Year 2006-07

David Rhodes and Anne Tuccillo

Goals
• Share draft results of programwide analysis of 2006-07 data • Illustrate additional ways to analyze ISIR data

2

Background
• Quality Assurance (QA) Program • Participants granted regulatory flexibility to decide which ISIR data they verify • ISIR Analysis Tool
3

The 2006-07 Analysis
• 146 Quality Assurance Program institutions • Each school drew a random sample of at least 350 applicants • Each school verified the ISIR information for the entire random sample of applicants • We analyzed 68,077 records
4

Previous data collections
2005-06
ISIR Records subject to school verification 140

2004-05
Random sample of all applicants 133

Population Number of schools

5

See a pattern?
• Alternating focus –Random sample of all applicants –Institutionally verified applicants • This year (2007-08) –All QA Program institutions –ISIRS subject to institutional verification

6

Is analysis over time appropriate? • For individual schools, yes with care • For program-wide analyses the answer “should” be no, but…..

7

When Looking Across Years Keep in Mind:
• The two different populations • Changes to institutional verification criteria between years • Other differences between award years • Percents and averages not counts
8

Why we “shouldn’t”
• QA schools supplying data differ slightly from year to year (146 ≠140 ≠133) • Unmeasured differences in institutional verification across years

9

Key areas of analysis
• Description of population • Critical ISIR fields • How changes affect aid eligibility • Improper payments in the Pell Grant program

10

Characteristics of Aid Applicants at QA Program Schools 2006-07
Eligible for Pell Selected for CPS Verification Selected for School Verification Dependent 0% 25% 50%
40% 44% 62% 58%

75% 100%

11

Dependent and independent record counts over time
150,000 125,000 100,000 75,000 50,000 25,000 0 2006-07 2005-06 Dependent 25,622 42,455 94,866 14,785 24,310 2004-05 42,297

Independent

12

Dependent and independent percentages over time
100% 75% 50% 25% 0% 2006-07 2005-06 Dependent 2004-05 Independent 62% 69% 62% 38% 31% 38%

13

Percentage of dependent applicants with changes to the most commonly changed ISIR Fields: 2006-07
Parent AGI Parent Total from Worksheet B Mother Income from Work Father Income from Work Parent Tax Filing Status 0% 29% 29% 26% 23% 22% 25% 50% 75% 100%

14

Percentage of independent applicants with changes to the most commonly changed ISIR Fields: 2006-07
Student Income from Work Student AGI Student Total from Worksheet B Student Tax Filing Status Student Total from Worksheet C

19% 18% 14% 12% 11%

0%

25%

50%

75%

100%

15

Changes to critical ISIR fields among dependent students over time
100% 75% 50% 25% 0% Parent AGI 2006-07 Parent Total from Worksheet B 2005-06 2004-05 29% 33% 30% 29% 27% 32%

16

Changes to critical fields among independent students over time
100% 75% 50% 25% 0% Student AGI 2006-07 Student Total from Worksheet B 2005-06 2004-05 18% 25% 19% 14% 15% 14%

17

Dependent records: percent of ISIR fields experiencing an EFC change - 2006-07
Parent AGI Parent Total from Worksheet B Mother Income from Work Father Income from Work Parent Tax Filing Status

89% 90% 88% 89% 87% 25% 50% 75% 100%

0%

18

Independent records: Percent of ISIR fields experiencing an EFC change - 2006-07
Student Income from Work Student AGI Student Total from Worksheet B Student Tax Filing Status Student Total from Worksheet C 0% 25% 50% 66% 68% 68% 62% 71% 75% 100%

19

Comparison of changes to EFC among dependent students with a change to the indicated ISIR field
100% 75% 50% 25% 0% Parent AGI 2006-07 Parent Total from Worksheet B 2005-06 2004-05 89% 88% 91% 90% 90% 93%

20

Comparison of changes to EFC among independent students with a change to the indicated ISIR field
100% 75% 50% 25% 0% Student AGI 2006-07 Student Total from Worksheet B 2005-06 2004-05
21

66% 67% 71%

68% 66% 72%

Percentage of dependent records with a change to the indicated ISIR field and a change to a Pell Grant: 2006-07
Parent AGI Parent Total from Worksheet B Mother Income from Work Father Income from Work Parent Tax Filing Status

49% 38% 46% 42% 43% 25% 50% 75% 100%

0%

22

Percentage of independent records with a change to the indicated ISIR field and a change to a Pell Grant: 2006-07
Student Income from Work Student AGI Student Total from Worksheet B Student Tax Filing Status Student Total from Worksheet C 0% 25% 36% 38% 34% 31% 35% 50% 75% 100%

23

Comparison of change to a Pell Grant among dependent students with a change to the indicated field
100% 75% 50% 25% 0% Parent AGI 2006-07 Parent Total from Worksheet B 2005-06 2004-05 49% 58% 42% 38% 54% 35%

24

Comparison of change to Pell Grants among independent students with a change to the indicated field
100% 75% 50% 25% 0% Student AGI 2006-07 Student Total from Worksheet B 2005-06 2004-05
25

38%

48%

37%

34%

45%

34%

Average EFC change among dependent records with a change to the indicated ISIR field: 2006-07
Parent AGI Parent Total from Worksheet B Mother Income from Work Father Income from Work Parent Tax Filing Status

648 819 645 913 1,039 0 400 800 1,200

26

Average EFC change among independent records with a change to the indicated ISIR field: 2006-07
Student Income from Work Student AGI Student Total from Worksheet B Student Tax Filing Status Student Total from Worksheet C

100 186 387 490 470 0 500 1,000 1,500

27

Average EFC change among dependent students with a change to the indicated field over time
1,500 1,000 648 500 0 Parent AGI 2006-07 Parent Total from Worksheet B 2005-06 980 819 1,214

28

Average EFC change among independent students with a change to the indicated field over time
1,500 1,000 591 500 103 0 Student AGI 2006-07 Student Total from Worksheet B 2005-06 371 387

29

Average Pell change for dependent records with a change to the indicated ISIR field: 2006-07
-$270 -$314 -$257 -$305 -$344 -$600 -$400 -$200
Parent AGI Parent Total from Worksheet B Mother Income from Work Father Income from Work Parent Tax Filing Status

$0

30

Average Pell change for independent records with a change to the indicated ISIR field: 2006-07
-$102 -$122 -$257 -$347 -$368
Student Income from Work Student AGI Student Total from Worksheet B Student Tax Filing Status Student Total from Worksheet C

-$600

-$400

-$200

$0

31

Average Pell Grant change for dependent records with a change to the indicated field over time
Parent AGI $0 -$200 -$270 -$400 -$469 -$600 2006-07 2005-06 -$314 -$504 Parent Total from Worksheet B

32

Average Pell Grant change for independent records with a change to the indicated field over time
Student AGI $0 -$200 -$400 -$600 2006-07 2005-06 -$122 -$257 -$344 -$459 Student Total from Worksheet B

33

Potential improper payments in Pell Grants prior to verification at QA Schools: 2006-07
9.1% potential overpayments 6.7% potential under-payments

84.2% Pell Grant dollars not at risk prior to verification

34

Improper Payments in Pell Grants before and after school verification
10.0% Percent of Pell Dollars 6.7% 5.0% 0.0% -5.0% -10.0% -3.6% Potential under-awards -0.7% -2.4% Potenial overawards 2.9% Net potenial over-awards

-9.1% Before verification After school verification

35

Improper Payments in Pell Grants before and after CPS verification
10.0% Percent of Pell Dollars 6.7% 5.0% 0.0% -1.7% -5.0% -10.0% Potential under-awards -2.4% 4.1% Potenial overawards Net potenial under-awards 2.4%

-9.1% Before verification After CPS verification

36

Lessons Learned
• While verification is crucial in some cases, many ISIR records do not experience a meaningful change from their initial transaction value • QA school verification procedures target larger EFC and Pell changes • QA school verification reduces improper payment risks in Pell

37

What Do the Findings Mean for Schools and FSA? Given that relatively few ISIR records experience a change that affects eligibility for need-based aid, verification efforts should strive to focus on the records that matter.

38

Two Ways to Focus
• Look for the records that matter • Look for the records that don’t

39

Suggested Additional Analysis Look for ways to reduce the number of school verified records that experience no or only trivial changes to aid eligibility after verification

40

Additional Analysis
• ISIR Analysis Tool reports • Exporting data from the Tool

41

ISIR Analysis Tool
• Key Filters –EFC change < 400 & EFC change > -400 –EFC change > 400 –EFC change < -400 –Institutional verification criteria

42

ISIR Analysis Tool
• Key Reports –Sample summary –Field Increment (EFC or AGI) • Drill down to refine understanding

43

New Analytic “Recipe”
See handout

44

Exporting Data
See handout

45

Institutional Profiles

Comparison of Programwide and school specific data

2006-07 Institutional Profile
Example of a 2006-2007 Institutional Profile

47

Profile Differs from Year to Year • 2005-06 provided data of improper payments PREVENTED by institutional verification • 2006-07 provides data on estimated levels of POTENTIAL improper payments in a schools applicant population

48

Contact Information
We appreciate your feedback and comments. We can be reached at:
Email: david.rhodes@ed.gov anne.tuccillo@ed.gov

49