1

Back-to-Basics Discretionary Grants
Data Collection, Processing & Reporting

Discretionary Grants: Data Collection, Processing & Reporting

Employment and Training Administration U.S. Department of Labor Frances Perkins Building 200 Constitution Avenue, NW Washington, DC 20210 www.doleta.gov

Sessi on Obj ectives
2

1

Data Collection

2 Data Processing 3 Reporting Requirements Note: The discretionary grantees that are the focus of this session are those subject to USDOL’s common measures (e.g., High-Growth and Community-Based Job Training Grantees)
Discretionary Grants: Data Collection, Processing & Reporting

Perf or mance M anagement
3

Data Processing Data Collection

“What matters in the end is Completion. Performance. Results. Not just making promises, Reports and Information but making good on promises.
- President’s Management Agenda See Handouts 1 and 2 Discretionary Grants: Data Collection, Processing & Reporting

Federal

Requ irements

4

Data Collection & Reporting Requirements

Grant Agreement

29 CFR Parts 95 & 97

Public Law 107-288

29 CFR Part 37

Discretionary Grants: Data Collection, Processing & Reporting

Data Collection—Introduction
for the management of the project

5

 Process used to gather and collect information  To develop a tracking mechanism, must address
four fundamental questions:  What information needs to be collected and in what format?  Where does the grantee obtain the information?  When does the information need to be collected?  How does the grantee know the information is
accurate/valid?

Discretionary Grants: Data Collection, Processing & Reporting

Data Collection—What Information To Collect?
What information is needed?  Start with grant proposal and grant award documents

6

 Review goals, objectives, & expected outcomes for your grant  Review Work Plan, Service Flow Chart, Performance Measures,
Project Deliverables, etc.

 Involve internal staff and external partners in the process of
identifying data collection elements

 They know what format it comes in and when the information is
collected

Discretionary Grants: Data Collection, Processing & Reporting

Data Collection— Service Flow
Assessment Service Planning Service Delivery
Application/ Intake/Eligibility

7

Case Management Cycle
Exit & Follow-Up

Job Development

See Handout 3 Discretionary Grants: Data Collection, Processing & Reporting

Data Collection— Participant Characteristics
 Participant contact and emergency contact information (e.g.,
name, address, telephone)

8

Types of individual characteristic information to collect includes:

 Information to assess a person’s eligibility for the project (tailored  EEO information (e.g., age, sex, ethnicity, race, and disability
status)

to each grantee based on definition/information from the proposal)

 Veteran’s characteristics and status
Continued

Discretionary Grants: Data Collection, Processing & Reporting

Data Collection— Participant Characteristics
Continued from previous slide

9

 Education information (e.g., highest school grade completed)

 Employment status at enrollment and past/current
employment information

 Information about supportive service needs and additional
reportable characteristics (e.g., offender, runaway, low income, single parent, etc.)

Discretionary Grants: Data Collection, Processing & Reporting

Data Collection— Participant Services
Collect information about the services provided to individual participants  Types of services in which the participant is enrolled
 Some examples include career guidance services, job
search assistance activities, supportive services, customized training, on-the-job training, apprenticeship training, occupational skills training, Adult Basic Education/ English Second Language training

10

 Participation or service enrollment dates  Service completion dates  Training provider information

Discretionary Grants: Data Collection, Processing & Reporting

Data Collection— Participant Outcomes
 Common Measures Initiative
 Why use common measures?
 Ability to describe in a similar manner the core purposes
of the workforce system—how many people found jobs? Did they stay employed? What did they earn?

11

 Enhance ability for different programs to collaborate by
reducing multiple reporting requirements

 Ability to analyze program outcomes and develop effective
program strategies

 Ability to compare results of similar government programs

Discretionary Grants: Data Collection, Processing & Reporting

DOL Common Measures Policy
funded grants and some discretionary grants

12

 Common measures is required for ETA formula High growth and community-based grantees are subject to
the common measures policy

 Training and Employment Guidance Letter
(TEGL) No. 17-05, dated February 17, 2006 details DOL policy for common measures

See TEGL 1705 Discretionary Grants: Data Collection, Processing & Reporting

What Common Measures Apply to Your Grant?
 Adult Measures
 Entered Employment  Employment Retention  Average Earnings

13

 Youth Measures
 Placement in Employment or Education  Attainment of a Degree or Certificate  Literacy or Numeracy Gain

Discretionary Grants: Data Collection, Processing & Reporting

Required Definitions— Participant
 Participant = An individual determined eligible
to participate in the program who receives a service funded by the program in either a physical location (e.g., One-Stop Center) or remotely through electronic technologies  You need to determine what services trigger
participation in your grant

14

 You may need to capture information about services
provided remotely

Discretionary Grants: Data Collection, Processing & Reporting

Required Definitions—Other Terms Related to Participation
 Participation Date: Date of first program-funded service
 It’s possible to record an earlier date if services were being
provided by another partner at the time of participation

15

 Discretionary grantees, however, cannot report a participation
date that begins before the actual period of performance in the grant award document

 Participation Quarter: Calendar quarter containing the
participation date (e.g., If 1st service is 4/22/07 then participation quarter is 2nd CY quarter)

 Participant Cohort: Group of individuals who share the same
participation quarter

Discretionary Grants: Data Collection, Processing & Reporting

Required Definitions—Exit
Exit: A participant who hasn’t received a program or partnerfunded service for 90 consecutive days and no future services are scheduled has exited the program

16

1. Hasn’t received a service 2. No service received for 90 consecutive days 3. No future services scheduled (does not include any follow-up
services or circumstances where the participant voluntarily withdraws or drops out of the grant)

For example, if a participant completes training on 2/13/2007 and planned to take one additional specialty course but then failed to do so, the exit date would be retroactive to 2/13/2007.

Discretionary grantees cannot report an exit date that extends beyond the actual period of performance in the grant award document

Discretionary Grants: Data Collection, Processing & Reporting

Required Definitions— Other Terms Related to Exit
date  For example, the exit quarter for individuals exiting on
2/13/2007 would be Q1 of 2007

17

 Exit Quarter: Calendar quarter containing the exit

 Exit Cohort: Group of individuals who exit during
the same calendar quarter.  For example, all participants that exit between 1/01/07 and
3/31/07 are in the same exit cohort

17

Discretionary Grants: Data Collection, Processing & Reporting

Service Delivery
Eligible & Receives Grant Services Last Service Participant Received & No Receiving Grant Future Services Services Scheduled Gap in Services Allowed
(see TEGL 17-05 Page 22)

18

Participant Receives Follow-up Services Follow-up Discussed in TEGL 17-05
(see TEGL 17-05 Pages 20-21)

Date of 1st Service Triggers Participation Date

No Services Received for 90 Consecutive Days

Last Service Date Triggers Exit Date
Discretionary Grants: Data Collection, Processing & Reporting

Basic Common Measurement — Rules and Concepts 19
 Only participants are included in performance calculations
 Specific conditions exclude participants from calculations, such as being
employed on the date of participation means excluded from the entered employment measure or missing SSN, (3) terminates due to health/medical or family care, (4) dies, (5) is a reservist called to active duty, and (6) relocates to a residential or non-residential program (applies to youth).

 Global exclusions if participant becomes (1) institutionalized, (2) invalid

 Calculations are based on exiters
 Exception is Youth Literacy/Numeracy because based on one year from
participation date

 Each measure is based on a specific cohort
 For example, a person who was unemployed on their participation date
and exits on 4/28/07 would be in the entered employment rate cohort of Quarter 3 of 2007 (July 1-September 30, 2007)

 Specific data sources must demonstrate the outcomes
Discretionary Grants: Data Collection, Processing & Reporting

Exit Cohorts
Exit Cohorts Oct 1 Dec 31 Jan 1 – Mar 31 Jan 1 – Mar 31 Apr 1 – Jun 30 Apr 1 – Jun 30 Apr 1 – Jun 30 Jul 1 – Sep 30 Jul 1 Sep 30 Jul 1 – Sep 30 Jul 1 – Sep 30 Exit Q1 Q2 Q3 Participant Exit Quarter 1st Quarter After Exit Quarter 2nd Quarter After Exit Quarter 3rd Quarter After Exit Quarter Oct 1 Dec 31 Oct 1 – Dec 31 Oct 1 Dec 31 Jan 1 – Mar 31 Jan 1 Mar 31

20

Apr 1 – June 30

Discretionary Grants: Data Collection, Processing & Reporting

Data Coll ecti on— Where to Get the Inf or mat ion?
Source Documentation

21

 Social Security Card  Driver’s License/ID Card  Hospital Records  Intake/Eligibility Forms  Attendance Sheets  Sign-In Sheets  School Records  Activity Forms  Assessment Results

 Pay stubs  Progress Reports  Surveys  Self-attestation forms  Copy of Diploma  Training Certificates  Interviews  Public Agency Records  Student ID

Discretionary Grants: Data Collection, Processing & Reporting

Wage Records—Data Source for Outcomes Measures

22

 Discretionary grantees that are State and local workforce
investment boards or one-stop providers will most likely have access to wage record data

 Employers are required to send confidential quarterly reports of

wage earners and their wages for the respective calendar quarter

 The majority of discretionary grantees, however, will not
have access to Unemployment Insurance (UI) wage records

 The following websites provide fee-based access to employment
verification and wage information, but not all employers provide information to these sources: (1) http://www.theworknumber.com/; (2) https://www.verifyjob.com/; and (3) http://www.usverify.com/incomeverify.html

See Handout 6 Discretionary Grants: Data Collection, Processing & Reporting

Wage Records—Data Source for Outcomes Measures (cont’d)
 Wage records are the preferred source of
information for calculating common measures

23

 The proposed reporting package for HG/CB job
training grantees would require grantees to submit quarterly participant records for each exiter to ETA containing 4 data elements

 ETA would then use these records to obtain
wage record data and calculate the common measures for grantees

 This is proposed!
See Handout 6 Discretionary Grants: Data Collection, Processing & Reporting

Supplemental Data Sources
 Most discretionary grantees will need to use

24

supplemental data sources to verify employment and wages

 Copy of a participant’s pay stub  Employer contact—telephone or written verification  Participant contact—telephone, participant surveys or written
self-attestation form

 Case management notes

 All supplemental data sources must be documented in
participant case files and are subject to audit

See Handout 6 Discretionary Grants: Data Collection, Processing & Reporting

Data Sources for Measures Not Related to Employment
Pertains to all three youth measures  Youth Placement in Employment or Education  Administrative records utilized for placement in
education (wage records or supplemental data for employment portion)

25

 Attainment of Degree/Certificate
 Administrative records utilized

 Literacy and Numeracy Gains
 Assessment instrument

25

Discretionary Grants: Data Collection, Processing & Reporting

Entered Employment
Of those who are not employed at the date of participation:
Number of adult participants who are employed in the 1st quarter after the exit quarter Number of adult participants who exit during the quarter

26

Discretionary Grants: Data Collection, Processing & Reporting

Entered Employment
Employed at Participation Date?
YES

27

Excluded from Measure

NO

Numerator Employed in 1st quarter after exit?
YES

Denominator
NO
Discretionary Grants: Data Collection, Processing & Reporting

Employment Retention
Of those who are employed in the first quarter after the exit quarter:
Number of adult participants who are employed in both the second and third quarters after the exit quarter Number of adult participants who exit during the quarter

28

Discretionary Grants: Data Collection, Processing & Reporting

Employment Retention
Employed in 1st quarter after exit quarter?
YES NO

29

Excluded from Measure

Numerator Employed in 2nd and 3rd quarters after exit quarter?
NO
Discretionary Grants: Data Collection, Processing & Reporting

YES

Denominator

Earnings Measure
Of those adult participants who are employed in the first, second and third quarters after the exit quarter:
Total earnings in the second quarter plus total earnings in the third quarter after the exit quarter Number of adult participants who exit during the quarter

30

Discretionary Grants: Data Collection, Processing & Reporting

Average Earnings
Employed in 1st quarter after exit quarter?
No

31

Excluded from Measure

Yes

No

Employed in 2nd and 3rd quarters after exit quarter?

Numerator = Wages from quarters 2 & 3, post-exit included Denominator = Number of adult participants who exit during the quarter

Yes

Discretionary Grants: Data Collection, Processing & Reporting

Youth Placement in Employment or Education

32

Of those who are not in post-secondary education or employment (including the military) at the date of participation:
Number of youth participants who are employed (including military) or enrolled in post-secondary education and/or advanced training/occupational skills training in the first quarter after the exit quarter Number of youth participants who exit during the quarter

Discretionary Grants: Data Collection, Processing & Reporting

Youth Attainment of a Degree or Certificate
Of those who are enrolled in education--at the date of participation or at any point during the program:
Number of youth participants who attain a diploma, GED, or certificate by the end of the third quarter after the exit quarter Number of youth participants who exit during the quarter

33

Discretionary Grants: Data Collection, Processing & Reporting

Youth Literacy and Numeracy Gains

34

Of those out-of-school youth who are basic skills deficient:
Number of youth participants who increase one or more educational functioning levels Number of youth participants who have completed one year of participation in the youth program (one year from date of first program service) plus the number of youth participants who exit before completing a year in the youth program

Discretionary Grants: Data Collection, Processing & Reporting

Data Collection— Is That It?
 While you are required to track information to
report on the common measures, other management information continues to be necessary  Collect information relevant to the day-to-day
operations of your project (e.g., information about enrollments and counts on various service dimensions)

35

 May collect information to help case managers, job
developers, managers, etc. make programmatic decisions

Discretionary Grants: Data Collection, Processing & Reporting

Data Collection—When to Collect the Information?
Important to develop an information management plan addressing:

36

 How will data be collected?  Information collected verbally, with paper documents or
electronically

 Who is responsible for the data collection?  Assign responsibility for data collection to specific staff to
ensure that expectations are clearly understood

 In what format will data be collected?  MS Excel spreadsheet, MS Access, Proprietary system  When will data be collected?  Incorporate deadlines & establish timeframes for all
collection tasks
Discretionary Grants: Data Collection, Processing & Reporting

Data Collection— Is the Information Reliable?
 Ensure data elements are well defined and

37

understood by project staff—this includes contractors  Train all staff on definitions and processing requirements  Note that original source of information more reliable
than secondary information

 Ensure duplication of effort does not occur in
the collection of data  Costly  Lack of Consistency

Discretionary Grants: Data Collection, Processing & Reporting

Data Processing—Introduction
38

 Process used to input data and information into
a MIS, find and correct errors in the data, and compile and aggregate the information into a user-friendly format questions:  Who has responsibility for data entry, compilation and
processing? integrity?

 Grantees must address these fundamental

 How should they validate the data and ensure its  What management information system will be used to
maintain the data?

Discretionary Grants: Data Collection, Processing & Reporting

Data Processing— Who Has Responsibility?
 The grantee is responsible for ensuring that a
system is in place to track participant characteristics, services and outcomes

39

 It is highly recommended that grantees maintain access
to data processing and reporting at all times

 You need to know what is going on with your grant!

 The grantee may contract out for services, but
should provide input on how the data is maintained and gathered

Discretionary Grants: Data Collection, Processing & Reporting

Data Processing— Who Has Responsibility?

40

 All staff, including sub-recipients staff, need to
clearly understand their role in creating, collecting, maintaining, and reporting program data  Understand interrelationship of staff member duties  Understand consequences of missing deadlines, not
sharing information, etc.

 Reduce errors by training staff and making sure they
meet specified qualifications

Discretionary Grants: Data Collection, Processing & Reporting

Data Processing— Is the Data Reliable?
 Data needs to be reliable…accurate, complete
and consistent

41

 Factors affecting data reliability:  Lack of data collection and data processing
policies and procedures

 Inaccurate and incomplete data  Insufficient staff training  Differences in definitions  Insufficient system controls
Discretionary Grants: Data Collection, Processing & Reporting

Data Processing—Is the Data Reliable?
 Data errors can occur at any point in record keeping.
 Assign ranges of possible values to each data element
 A person’s age could range from 16-100, anything outside the
range would be flagged by the system

42

To help maintain data quality, consider the following:

 Check consistency across data elements
 Same identification number but different race or sex

 Look for missing data  Conduct periodic reviews of information keyed into MIS

compared with original sources (i.e., sample of case files and MIS cases) back to original source

 Remember data must be auditable—must be able to track

Discretionary Grants: Data Collection, Processing & Reporting

Data Processing—What System Should We Use?
 How sophisticated or elaborate does the grantee’s MIS
need to be?

43

 It varies, but high-performing organizations have an MIS that

produces information/reports to assist staff in addressing issues and improving performance

 Examples include MS Access, MS Excel, or a proprietary system
such as Client Tracking System  Contract with local workforce investment areas to process and aggregate data

 What must grantee MIS be able to do?
 At a minimum, capture all required data elements, perform any 
necessary calculations and report information to the grantee and its partners In some cases, ETA may provide the required data collection instruments and a system for collecting and reporting the information

See Handout 5 Discretionary Grants: Data Collection, Processing & Reporting

Data Processing—What System Should We Use?

44

 Key principles of an effective MIS:
 Enter information only once  Enter information in a timely manner  Enter information at the most cost-effective level of the
organization (e.g., line staff versus technical staff)

 Information available on demand  Access available to all levels in the organization, as
needed

Discretionary Grants: Data Collection, Processing & Reporting

Reports and Information
 Ability of grantee to use data processing and
processing techniques to develop reports

45

 Need for an information system that can give
you reports that tell a story about your grant  What other data help tell your story?  What about customer satisfaction data?  Don’t collect data simply to ‘feed the Federal data
machine,’ but to inform program design and management

Discretionary Grants: Data Collection, Processing & Reporting

Reports and Information— Federal Requirements
 Reporting requirements based on Uniform
Administrative Requirements in 29 CFR Parts 95.51 and 97.40  Comparison of actual accomplishments with goals and
objectives for the project

46

 Reasons why established goals were not met and
corrective action being taken

 Other pertinent information, including technical assistance
needs, best practices or any promising approaches

 In some cases, ETA has a required report format
for your use with established due dates

Discretionary Grants: Data Collection, Processing & Reporting

Reports and Information— Common Measures
 Common measures are based on specific
cohorts  Cohorts based on calendar quarters  January 1 to March 31, 2007 = CY 2007 Q1  April 1 to June 30, 2007 = CY 2007 Q2  July 1 to September 30, 2007 = CY 2007 Q3  October 1 to December 31, 2007 = CY 2007 Q4  If using wages to calculate performance on employment, earnings, and retention there will be a lag in when the wage information is available

47

Discretionary Grants: Data Collection, Processing & Reporting

Reports and Information— Table of Reporting Periods
Reporting Time Periods Oct 1 Dec 31 Jan 1 – Mar 31 Apr 1 – Jun 30 Jul 1 – Sep 30 Exit Q1 Q2 Q3 Participant Exit Quarter 1st Quarter After Exit Quarter 2nd Quarter After Exit Quarter 3rd Quarter After Exit Quarter Jul 1 Sep 30 Oct 1 – Dec 31 Jan 1 Mar 31

48

Discretionary Grants: Data Collection, Processing & Reporting

Reports and Information— Is the Information Reliable?
 Employ mechanism(s) to ensure reports
generated and outcomes reported are calculated correctly

49

 Useful to have a data entry/MIS handbook
 Reference tool for system users that details data
collection protocols, definitions of terms, coding assignments, acceptable data sources, operational rules for data entry, available edit checks and report operations

Discretionary Grants: Data Collection, Processing & Reporting

Concl usi on
Measuring performance can be a powerful mechanism for managing performance!
Data Collection
 Intake/Eligibility forms  Attendance sheets  Activity forms  Progress reports  Records of completion  Exit & follow-up forms  Satisfaction surveys

50

Data Processing
 Data Entry  Organizing data  Compiling data  Aggregating data  MIS  Excel spreadsheets  Access databases  Analyzing data

Information & Reports
 Internal management  Quarterly ETA reports  Stakeholder reports  Customer satisfaction  Evaluation reports  Contractor reports

Discretionary Grants: Data Collection, Processing & Reporting

51

THANK YOU
Are there any questions?

Discretionary Grants: Data Collection, Processing & Reporting