AIR Contract

Contract No.
: 14-652
Attachment A
Detailed Description of
Performance Duties
Contract No.: 14-652

Attachment C
Standard Terms and Conditions

Attachment D
Single Audit Act Requirements
Not Applicable
Contract #14-652 Attachment F

ITN and All Attachments and Exhibits
State of Florida
Department of Education
INVITATION TO NEGOTIATE
FLORIDA STANDARDS ASSESSMENTS
BID NUMBER: ITN 2014-47
DEADLINE FOR TECHNICAL QUESTIONS: November 8, 2013 @ 2:00 PM

(There is no deadline for administrative questions)
REPLIES ARE DUE BY: DECEMBER 12, 2013 @ 2:00 PM, EASTERN TIME (ET)
MAIL OR DELIVER REPLIES TO:

Florida Department of Education
Bureau of Contracts, Grants and Procurement Management Services
325 West Gaines Street
332 Turlington Building
Tallahassee, Florida 32399-0400
Attention: Fran Shewan
Phone: (850) 245-9884
Florida Standards Assessments, ITN 2014-47
Table of Contents
INVITATION TO NEGOTIATE REGISTRATION
1.0
GENERAL INSTRUCTIONS TO RESPONDENT
SECTION 2 CONTRACT CONDITIONS
2.0
GENERAL CONTRACT CONDITIONS
6
7
7
SECTION 1 - OVERVIEW
SECTION 1 INSTRUCTIONS
SECTION 3 - INTRODUCTION
6
6
3.0
3.1
3.2
3.3
3.4
INTENT
PURPOSE
BACKGROUND
DEFINITIONS
CRITICAL EVENT DATES
7
7
8
8
10
4.0
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
4.9.
4.10
4.11
4.12
PRE-SOLICITATION CONFERENCE:
SITE INSPECTION:
VISITORS PASS TO THE TURLINGTON BUILDING
REPLY QUESTIONS & ANSWERS
PROCUREMENT PROTESTS / NOTICE OF RIGHTS
ORAL INSTRUCTIONS / CHANGES TO THE INVITATION TO NEGOTIATE (ADDENDA)
MODIFICATIONS, RESUBMITTAL AND WITHDRAWAL
RESTRICTIONS ON COMMUNICATIONS WITH DEPARTMENT STAFF
CONFIDENTIAL, PROPRIETARY, OR TRADE SECRET MATERIAL
POOR PERFORMANCE NOTICE
WITHDRAWAL OF A REPLY
DISCLOSURE OF REPLY CONTENTS
AWARD
11
11
11
11
11
12
12
12
13
13
13
14
14
5.0
5.1
5.2
5.3
5.4
5.5
5.6
5.7
5.8
5.9
5.10
5.11
5.12
5.13
5.14
5.15
5.16
5.17
5.18
5.19
AUTHORIZED TO DO BUSINESS IN THE STATE OF FLORIDA

LICENSED TO CONDUCT SERVICES IN THE STATE OF FLORIDA
OTHER CONDITIONS
IDENTICAL EVALUATION OF REPLIES
DISCLOSURE STATEMENT
SUB-CONTRACTING
CONTRACTUAL OBLIGATIONS
METHOD OF PAYMENT
CONVICTED VENDOR LIST
DIVERSITY IN CONTRACTING
COSTS INCURRED IN RESPONDING
SUBMISSION OF REPLIES BY SUBSIDIARIES OR AFFILIATES
PROHIBITION OF GRATUITIES
INDEPENDENT PRICE DETERMINATION
FINANCIAL CONSEQUENCES
PERFORMANCE BOND
PARTICIPATION IN FUTURE STAGES OF THIS PROJECT
ACCESSIBLE ELECTRONIC INFORMATION TECHNOLOGY
SCRUTINIZED COMPANIES LISTS (AS APPLICABLE)
LIQUIDATED DAMAGES
14
14
14
14
15
15
15
15
15
15
16
16
16
16
16
16
17
17
17
17
6.0
REPLY SUBMISSION
19
SECTION 4 SPECIAL INSTRUCTIONS
SECTION 5 SPECIAL CONDITIONS
SECTION 6 SPECIAL INSTRUCTIONS REPLY FORMAT & CONTENT
11
14
19
6.1
6.2
6.2.1
6.2.2
6.2.3
6.2.4
MAIL OR DELIVER REPLIES TO: (DO NOT FAX OR E-MAIL)

REPLY FORMAT INSTRUCTIONS
PRELIMINARY ADMINISTRATIVE REVIEW
TECHNICAL REPLY (PART I) (20 HARD COPIES)
PRICE REPLY (PART II) (20 HARD COPIES)
PRESENTING THE REPLY
SECTION 7 SCOPE OF SERVICES
7.0
7.1.0
7.1.1
7.1.2
7.2.0
7.2.1
7.2.2
7.3.0
7.3.1
7.3.2
7.3.3
7.3.4
7.3.5
7.3.6
7.3.7
7.3.8
7.3.9
7.3.10
SCOPE OF SERVICES
INTRODUCTION
NOTICE TO RESPONDENTS
PROJECT SCHEDULE
PROGRAM DESCRIPTIONS
EXPECTED NUMBER OF DISTRICTS, SCHOOLS, AND STUDENTS
POTENTIAL TEST ADMINISTRATION DATES
TEST DESIGN AND DEVELOPMENT
TEST CONTENT
TEST CONFIGURATION
DEVELOP ITEMS
REVIEW MEETINGS
DEVELOP FIELD-TEST ITEM SETS AND FORMS
DEVELOP STAND-ALONE ELA/L FIELD-TEST ITEM SETS AND FORMS
CONSTRUCTING TEST FORMS
PRODUCTION OF PRINTED TEST MATERIALS
PRODUCTION OF COMPUTER-DELIVERED PRODUCTS
PRODUCTION OF PAPER-BASED AND COMPUTER-BASED TEST, STYLE GUIDE, AND BRAILLE
PRODUCTION PROCEDURES
7.3.11 PRODUCE SPECIAL FORMATS AND ACCOMMODATIONS FOR STUDENTS WITH DISABILITIES
7.4.0 TEST ADMINISTRATION
7.4.1 PACK AND DISTRIBUTE MATERIALS
7.4.2 MISSING MATERIALS REPORT AND INVENTORY
7.4.3 DISPOSITION AND STORAGE OF MATERIALS
7.4.4 RETRIEVAL OF ANSWER SHEETS AND TEST BOOKS
7.4.5 INNOVATIVE PRODUCT DESIGN
7.4.6 PRINTED PRODUCTS ADMINISTRATION
7.4.7 TEST ADMINISTRATION ANCILLARY MATERIALS
7.4.8 ONLINE MANAGEMENT SYSTEM
7.4.9 COMPUTER-BASED TEST ADMINISTRATION OVERVIEW
7.4.10 CUSTOMER SERVICE AND TECHNICAL SUPPORT
7.5.0 SCANNING, SCORING, AND REPORTING
7.5.1 SCORING AND REPORTING SPECIFICATIONS
7.5.2 PREPARING SCHOOL AND STUDENT INFORMATION (MSID AND PREID)
7.5.3 SCANNING/EDITING PBTS AND EXTRACTING CBT RECORDS
7.5.4 SCORING
7.5.5 REPORTING
7.5.6 VERIFYING DATA
7.6.0 SCALING, EQUATING, SCORING, AND SPECIAL PSYCHOMETRIC STUDIES
7.6.1 ASSESSMENTS
7.6.2 SCALING, EQUATING, AND SCORING (PSYCHOMETRICS)
7.6.3 TECHNICAL OVERSIGHT AND SPECIAL STUDIES
7.7.0 PROGRAM MANAGEMENT
7.7.1 PROVISIONS GOVERNING THE WORK OF THE CONTRACTOR
19
19
19
20
23
23
23
23
23
24
25
25
26
26
27
28
29
30
30
32
32
32
34
35
36
36
39
39
43
43
44
44
44
46
49
52
58
59
59
60
62
63
68
73
77
78
79
81
87
87
7.7.2
7.7.3
7.7.4
7.7.5
7.7.6
7.7.7
7.7.8
7.7.9
7.8
CONTRACTOR STAFF AND RESPONSIBILITIES

SUBCONTRACTORS
PROGRAM MANAGEMENT COMMUNICATION
PROGRAM MANAGEMENT MEETINGS
PROGRAM MANAGEMENT REPORTS
MEETINGS INVOLVING FLORIDA EDUCATORS AND OTHER CITIZENS
COMMUNICATION BETWEEN THE CONTRACTOR AND DISTRICTS
REIMBURSABLE FUNDING CATEGORIES
DELIVERABLES (SEE ATTACHED SECTION 7.8 DELIVERABLES PROVIDED IN AN EXCEL
SPREADSHEET.)
SECTION 8 OPENING, EVALUATION AND AWARD
87
89
89
90
90
91
93
94
96
96
8.0
REPLY OPENING
96
8.1
REPLY EVALUATION AND NEGOTIATION PROCESS
96
8.2
EVALUATIONS
97
8.3
POSTING OF INTENDED AWARD
98
8.3.1 PROTEST OF INTENDED AWARD
98
8.3.2 INABILITY TO POST
98
8.4
AWARD OF THE CONTRACT
98
APPENDIX A: TRANSITION TO COMPUTER-BASED TESTING AND ADDITIONAL TEST ADMINISTRATION
DATES
99
APPENDIX B: CRITICAL AND EXTREMELY CRITICAL ACTIVITIES
101
APPENDIX C: FLORIDA PUBLIC SCHOOL STUDENT PARTICIPATION ESTIMATES, DISTRICT NUMBERS, AND
SPECIAL DISTRICT/SCHOOL NUMBERS
104
APPENDIX D: SAMPLE TEST DESIGN
107
APPENDIX E: PRODUCT SPECIFICATIONS AND DISTRIBUTION REQUIREMENTS
108
APPENDIX F: TEST SECURITY REQUIREMENTS, STATUTE, AND RULE
114
APPENDIX G: TEST ADMINISTRATION MANUAL WORKFLOW
118
APPENDIX H: TEST ACCOMMODATIONS
124
APPENDIX I: DATA PRIVACY & SECURITY POLICY
131
APPENDIX J: REQUIREMENTS FOR SCORING PERFORMANCE ITEMS
137
ATTACHMENT 1: RESPONDENTS PRICE REPLY
145
ATTACHMENT 2: DRUG-FREE WORKPLACE
146
ATTACHMENT 3: DISCLOSURE STATEMENT
147
ATTACHMENT 4: REFERENCES
148
ATTACHMENT 5: MINORITY SUB CONTRACTORS UTILIZATION SUMMARY
149
ATTACHMENT 6: VENDOR CERTIFICATION REGARDING SCRUTINIZED COMPANIES LISTS
150
ATTACHMENT 7: CONTRACT STANDARD TERMS AND CONDITIONS
151
State of Florida
INVITATION TO NEGOTIATE REGISTRATION

**************************************************************************************************************************
COMPLETE AND RETURN THIS FORM
**************************************************************************************************************************
Reply Number: ITN 2014-47
Title: Florida Standards Assessments
Date & Time Reply Due: December 12, 2013 @ 2:00 PM, EASTERN TIME (ET)
Potential Respondents should notify the Florida Department of Education, Bureau of Contracts, Grants and
Procurement Management Services by returning this Intent to Submit Reply Form as soon as possible after
downloading. Complete the information below and send this sheet only to below e-mail address
(preferred). Form can also be faxed to fax number (850) 245-0719, or mailed to 325 West Gaines Street,
332 Turlington Building, Tallahassee, Florida 32399-0400.
Company Name:
Contact Person:
Address:
City, State, Zip:
Telephone: ( )
Fax Number: ( )
Internet E-Mail Address:

Signed:
Date:
For further information on this process, you may contact Fran Shewan fran.shewan@fldoe.org at (850)
245-9884.
(Revised 10/06/2012)
STATE OF FLORIDA
DEPARTMENT OF EDUCATION
INVITATION TO NEGOTIATE
BID NUMBER: ITN 2014-47
SECTION 1 - OVERVIEW
SECTION 1 INSTRUCTIONS
1.0
GENERAL INSTRUCTIONS TO RESPONDENT

This section contains instructions explaining the solicitation process and the actions necessary to respond.
General Instructions to Respondent (Form PUR 1001 incorporated herein by reference) is a downloadable
document which must be downloaded for review. This document need not be returned with the Respondents
Reply. Form PUR 1001 may be accessed at http://dms.myflorida.com/business_operations/state_purchasing
under Documents, Forms, References and Resources.
In the event of any conflict between Form PUR 1001 and other instructions provided in this document, the
additional instructions in this document shall take precedence over the Form PUR 1001 unless the conflicting
term is required by any section of the Florida Statutes (F.S.), in which case the statutory requirements shall
take precedence.
It is entirely the respondents responsibility to examine this ITN, to confirm that the Departments requirements
are clearly stated, and to submit its reply in a timely, complete, and procedurally correct manner. The
products and services described in this ITN will be procured in accordance with Chapter 287, Florida Statutes
(F.S.), and Chapter 60A-1, Florida Administrative Code (FAC).
THE DEPARTMENT HAS CHOSEN TO USE THE ITN FORMAT FOR THIS PROCUREMENT BECAUSE IT
WANTS VENDORS TO PROPOSE THE BEST METHOD FOR ACHIEVING THE GOAL OF THIS ITN AND
SOLVING THE PROBLEM STATED HEREIN. THEREFORE, ALTHOUGH THE ITN MAY USE
MANDATORY WORDS LIKE SHALL, WILL, OR MUST, AND MAY DEFINE CERTAIN ITEMS AS
REQUIREMENTS, THE DEPARTMENT RESERVES THE RIGHT, IN ITS DISCRETION, TO WAIVE ANY
DEVIATIONS FROM THESE PROVISIONS AND RESOLVE ANY ISSUES IN THE NEGOTIATION PHASE.
THEREFORE, THE DEPARTMENT RESERVES THE RIGHT TO REVIEW THE ENTIRE REPLY TO
DETERMINE IF IT ACHIEVES A LEVEL OF COMPETENCY WORTHY OF FURTHER NEGOTIATIONS;
REGARDLESS OF WHETHER INDIVIDUAL REQUIREMENTS HAVE BEEN ADDRESSED OR NOT.
HOWEVER, VENDORS THAT FAIL TO PROVIDE SIGNIFICANT PORTIONS OF THE SOLUTION OR
ADDRESS SIGNIFICANT PORTIONS OF THE PROCUREMENT MAY STILL BE DEEMED
NONRESPONSIVE. IN ADDITION, THERE IS NO GUARANTEE THAT SUCH DEVIATIONS WILL BE
DEEMED IN THE STATES BEST INTEREST OR ANY REPLY CONTAINING THOSE DEVIATIONS
PARTICIPATE IN THE NEGOTIATIONS. USE OF THE TERMS SHALL, WILL, AND MUST INDICATE
THE DEPARTMENTS INITIAL VIEW OF THE VALUE OF SUCH ITEMS. VENDORS HAVE THE
OPPORTUNITY TO SUGGEST ALTERNATIVES IN THE ITN PROCESS, BUT THERE IS NO GUARANTEE
THAT THE DEPARTMENT WILL AGREE THE DEVIATIONS ARE IN ITS BEST INTEREST OR CREATE
THE BEST VALUE FOR THE STATE.
SECTION 2 CONTRACT CONDITIONS

2.0
GENERAL CONTRACT CONDITIONS

Standard terms and conditions that will apply to the contract which results from the solicitation event are
provided in this section. General Contract Conditions (Form PUR 1000 incorporated herein by reference) is
a downloadable document which must be downloaded for review. This document need not be returned with
the
Respondents
Reply.
Form
PUR
1000
may
be
accessed
at
http://dms.myflorida.com/business_operations/state_purchasing under Documents, Forms, References and
Resources.
In the event of any conflict between the PUR 1000 form and any other Special Conditions, the Special
Conditions shall take precedence over the PUR 1000 form unless the conflicting term in the PUR form is
required by any section of the F.S., in which case the statutory requirements shall take precedence.
SECTION 3 - INTRODUCTION
3.0
INTENT
The State of Florida Department of Education (hereinafter referred to as the "Department") is soliciting written
replies from qualified vendors to establish a term contract of which the term is anticipated to begin upon
execution of the contract and be effective for 36) months thereafter. Award(s) will be made to the responsible
and responsive vendor that the Department determines will provide the best value to the state.
The resulting contract(s) may be renewed for up to three additional 12 month terms, see Form PUR 1000 for
renewal requirements.
Although this ITN may result in the award of multiple contracts, the term Contractor is used throughout this
ITN and refers to each contractor providing the required products and services for each grade and subject for
which the respective contract is awarded.
3.1
PURPOSE
The purpose of this Invitation to Negotiate (ITN) is to solicit replies for activities related to the implementation
of computer-based and paper-based statewide summative assessments aligned to the Florida Standards for
administrations beginning in spring 2015 and continuing through 2016-17 for the base contract with three oneyear optional renewals for the administrations in 2017-18, 2018-19, and 2019-20. Henceforth, in this
document, this test shall be called the FSA (Florida Standards Assessment)
In accordance with Governor Rick Scotts Executive Order, dated September 23, 2013, this ITN is seeking
Replies that:
A. Provides timely and informative reports of results;
B. Includes costs that are in line with current assessment costs;
C. Ensures that testing time for students is not significantly different from current assessments;
D. Provides for summative assessments to occur as close as possible to the end of the school year;
E. Measures student mastery of the standards taught, including comparability to other states;
F. Includes test quality metrics that are as rigorous as current assessments;
G. Provides results that can be used in conjunction with Floridas school accountability system;
H. Requires technology parameters that are defined and can be supported, including appropriate
accommodations for exceptional students.
This ITN addresses four broad assessment components as described below. Respondents may submit bids
on any individual assessment component (A, B, C or D), any group of assessment components, or to provide
all assessment components (A, B, C and D):
A. Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 3-8
B. Computer-based and paper-based assessments in Mathematics in Grades 3-8
C. Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 9-11
D. Computer-based and paper-based Endof-Course (EOC) assessments in Algebra 1, Geometry, and
Algebra 2
A preferred schedule and plan for transition to computer-based testing is included in Appendix A. However,
depending on the final award for the individual assessments, the Department may vary when tests will be
administered as computer-based or paper-based. The final determination of which tests will be computerbased or paper-based is at the Departments discretion.
In addition, all Respondents agree, by submitting a reply, that any other state or school district in the United
States may purchase the services provided in the resulting contract, upon the same terms and conditions, and
price, as contained in the contract, or as permitted under the laws of that jurisdiction.
3.2
BACKGROUND
The Florida Comprehensive Assessment Test 2.0 (FCAT 2.0) and Florida End-of-Course (EOC)
Assessments implement the requirements of Section 1008.22, Florida Statutes (F.S.), requiring assessments
of the academic achievement of Floridas public school students. The FCAT 2.0/EOC Assessment program is
composed of state-developed criterion-referenced assessments. FCAT 2.0 Reading and Mathematics have
been administered each spring since 2011 to assess students in selected grade levels. FCAT 2.0 Science
was introduced in spring 2012, FCAT 2.0 Writing was introduced in spring 2013, and the Florida EOC
Assessments have been phased in beginning in 2011. Prior to FCAT 2.0, FCAT assessments were
administered each spring since 1998, and included Reading, Mathematics, Writing, and Science. The final
administration of FCAT 2.0 Writing, FCAT 2.0 Reading, and FCAT 2.0 Mathematics will be in spring 2014.
The final administration of the FCAT 2.0 Reading Retake for students with this graduation requirement will be
in spring 2016, and the Florida Algebra 1 and Geometry EOC Assessments will be eliminated after the
summer 2017 administration. (Administration of the FCAT 2.0 Reading Retake and the Florida EOC
Assessments will be handled in another procurement.)
The Sunshine State Standards, which specify challenging expectations for the educational achievement of
Florida students, were revised and approved by the State Board of Education as the Next Generation
Sunshine State Standards (NGSSS). The Reading/Language Arts and Mathematics revised standards were
approved by the State Board in 2007. Science and Social Studies Standards were approved in February
2008. The NGSSS and associated course descriptions for the Florida End-of-Course Assessments are
available at www.flstandards.org. In 2010, Florida adopted the Common Core State Standards for English
Language Arts/Literacy and for Mathematics. As of the release of this ITN, these standards were scheduled
to be reviewed further by Floridas stakeholders. The degree to which the standards, and by extension the
assessments, will be adjusted is to be determined. Florida plans for the Florida Standards Assessments to be
operational beginning in 2014-15.
3.3
DEFINITIONS
After the award, said Respondent will be referred to as the "Contractor". For the purpose of this document,
the term "Respondent" means a potential Contractor acting on its own behalf and on behalf of those
individuals, partnerships, firms, or corporations comprising the Respondents team. The term "Reply" means
the complete response of the Respondent to the ITN, including properly completed forms and supporting
documentation. The term contract refers to the agreement between the Department and the Contractor
resulting from this ITN. Best value means the highest overall value to the state based on objective factors
that include, but are not limited to, price, quality, design, and workmanship. A responsive bid is a Reply
submitted by a responsive and responsible vendor which conforms in all material respects to the solicitation.
However, for purposes of determining what is material, please refer to Section 1 of this ITN. Use of the terms
shall, will, and must, or the designation of items as requirements, does not mandate that such items are
material. The Department reserves the right to waive deviations, in its discretion, and resolve any issues in
the negotiation phase. Deliverable means a tangible, specific, quantifiable and measurable event or item
that must be produced to complete a project or part of a project directly related to the scope of services.
Additional definitions/terms as follows:
a.
b.
c.
d.
e.
f.
g.
h.
i.
j.
k.
l.
m.
n.
o.
p.
q.
r.
s.
t.
u.
v.
w.
AL Achievement Level
Braille Paper-based accommodation provided for eligible students with disabilities
CBT Computer-based test; computer-based testing
Cohort For the purposes of this ITN, refers to a group of students that enters grade 9 and is enrolled at
the high school for a total of four years
District Assessment Coordinator Primary contact(s) in each school district with whom FDOE
communicates regarding the administration of K-12 statewide assessments and who oversees the
administration of statewide assessments at all participating schools in that district
DAR District Aggregated Results files used for reporting purposes
DIF Differential Item Functioning
DSR District Student Results file used for reporting purposes
DSS Developmental Scale Score
EOC- End-of-Course Refers to an assessment that is associated with a specific course
FDOE (DOE) Florida Department of Education
FPO Placeholder for a graphic used in a publication
FSA Florida Standards Assessment, the current name of the assessments to be administered as part of
this ITN
IEP Individual Educational Plan for students with disabilities (ESE)
IRT Item Response Theory
Large Print Paper-based or computer-based accommodation provided for eligible students with
disabilities; large print usually refers to at least 18 point font
MS (Manuscript) The initial production round when a document is composed and first reviewed
MSID (Master School Identification File) A file maintained by the Department and available online used
to identify all active schools that will participate in statewide assessment administrations
One-Item-Per-Page Paper-based accommodation provided for eligible students with disabilities; only
one item appears on each printed page
P&D Packaging and distribution; often used in reference to the specifications that describe a
contractors processes of packaging and distributing all statewide assessment materials prior to each
assessment administration
PBT Paper-based test; paper-based testing
PreID (Preidentification) Process for identifying students prior to an administration to print labels with all
of the students information (PreID labels) or to create computer-based student records that will be used
for testing
PT (Program team) Contractor staff members with program-wide expertise who are expected to assist
in developing accurate instructions and providing specific information related to the contractors
processes and procedures regarding Floridas statewide assessments
x. Rangefinding Process required for scoring purposes by which student responses to performance tasks
are used to illustrate score points on a rubric
y. RD Results Delivery (RD1, RD2, etc.)
z. SAR State Aggregated Results files used for reporting purposes
aa. School Assessment Coordinator A designated school-level assessment contact who oversees
administration of statewide assessments at that school, training of test administrators and proctors, and
preparing and returning test materials
bb. Seals Method of maintaining security of test content until a test session begins; paper-based tests may
have one or more paper or plastic seals that students can easily break without tearing the test book, and
computer-based tests may have an electronic means of sealing a test session (e.g., content is locked until
a specific code is entered that allows access)
cc. Section 504 Plan Plan detailing accommodations for students with temporary disabilities (may not be
ESE)
dd. Section 508 Compliant (ADA Compliant) Section 508 requires that electronic and information
technologies which are procured or developed, be maintained by federal agencies and be accessible to
people with disabilities, unless it poses an undue burden to do so. Accessibility standards needed to
comply with this law were developed by the United States Architectural and Transportation Barriers
Compliance Board. Section 508 of the Rehabilitation Act of 1973 is in the federal code of regulations (29
U.S.C. 794(d).
ee. SES (Scaling, Equating, and Scoring) The set of processes used to ensure that assessment results are
valid, reliable, and accurate
ff. SS Scale score
gg. SSR State Student Results files used to check data for reporting purposes
hh. STMs Sample Test Materials
ii. Test Administrator A certified educator who administers statewide assessments to students in a testing
room; responsible for reading administration scripts to students and maintaining a secure testing
environment
jj. TAC (Technical Advisory Committee) A panel of experts who review and respond to proposed
modifications and processes for the assessment program
kk. TAM Test administration manual
ll. TC (Technology Coordinator) A designated coordinator at the school or district level responsible for
preparing for computer-based testing and addressing technical issues during testing
mm.TDC Test Development Center, which is overseen by the FDOE and staffed with editors and Florida
educators; TDC provides quality assurance for all test content and related products, whether customized
or derived from existing products
nn. TM Test Materials Shipment (TM1, TM2, etc)
3.4

These are tentative dates for critical events, and are subject to change.
October 25, 2013 ITN Released
November 8, 2013 @ 2:00 PM, ET - Deadline for Technical Questions
November 15, 2013 - Department Response to Technical Questions received by deadline
December 12, 2013 @ 2:00 PM, ET - Technical & Price Replies are due
10
December 12, 2013 @ 2:30 PM - Technical Replies will be opened in Room 332 Turlington Building
(Step 1)
January 10, 2014 - The Department will evaluate the Technical Replies in Room 1706 Turlington Building
@ 9:00 AM (Step 2)
Price Replies will be evaluated by the Bureau of Contracts, Grants and Procurement Management Services
staff during the evaluation stage.
January 28, 2014 - The Negotiation Committee will start negotiations on or about (Step 4)
February 13, 2014 BAFOs due on or about
February 25, 2014 ITA Meeting on or about
The Intent to Award will be posted for 72 hours upon completion of the negotiation meeting and a final
decision by the Department.
SECTION 4 SPECIAL INSTRUCTIONS

4.0
PRE-SOLICITATION CONFERENCE:
A Pre-Solicitation Conference will not be held.
4.1
SITE INSPECTION:
4.2
VISITORS PASS TO THE TURLINGTON BUILDING

Each visitor to the Turlington Building is required to sign in and obtain a Visitor's Pass at the security desk in
the main lobby. Please allow at least 15 minutes prior to Reply due time if hand-delivering the Reply to the
Bureau of Contracts, Grants and Procurement Management Services.
4.3
REPLY QUESTIONS & ANSWERS

Any technical questions arising from this ITN must be forwarded, in writing, to the purchasing agent identified
below. The Department's written response to written inquiries submitted timely by Respondents will be posted
on the Florida Vendor Bid System (VBS) at www.myflorida.com (click on Business, then click on Doing
Business with the state, under Everything for Vendors and Customers, click on the Vendor Bid System, then
Search Advertisement; select the Department of Education in the Agency drop down window and initiate
search), under this Reply number. It is the responsibility of all potential Respondents to monitor this site for
any changing information prior to submitting a Reply.
A Site Inspection will not be held.
Only timely received written inquiries will be officially addressed by the Department.
WRITTEN QUESTIONS should be submitted to:
Bureau of Contracts, Grants and Procurement Management Services, Fran Shewan, 325 West Gaines
Street, 332 Turlington Building, Tallahassee, Florida 32399-0400, fshewan@fldoe.org (email preferred),
or via Fax Number: (850) 245-0719. See SECTION 3.4 Critical Event Dates for question due dates.
4.4
PROCUREMENT PROTESTS / NOTICE OF RIGHTS

Pursuant to F.S., Section 120.57(3) (b):
Any person who is adversely affected by the agency decision or intended decision shall file with the agency a
notice of protest in writing within 72 hours after the posting of the notice of decision or intended decision. With
respect to a protest of the terms, conditions, and specifications contained in a solicitation, including any
11
provisions governing the methods for ranking bids, proposals, or replies, awarding contracts, reserving rights
of further negotiation, or modifying or amending any contract, the notice of protest shall be filed in writing
within 72 hours after the posting of the solicitation. The formal written protest shall be filed within 10 days after
the date the notice of protest is filed. Failure to file a notice of protest or failure to file a formal written protest
shall constitute a waiver of proceedings under this chapter. The formal written protest shall state with
particularity the facts and law upon which the protest is based. Saturdays, Sundays, and state holidays shall
be excluded in the computation of the 72-hour time periods provided by this paragraph.
Section 120.57(3) (a) provides:
Failure to file a protest within the time prescribed in Section 120.57(3), F.S. or failure to post the bond or other
security required by law within the time allowed for filing a bond shall constitute a waiver of proceedings under
Chapter 120, F.S."
Florida Administrative Code (F.A.C.) Rule 28-110.002(2) defines the term decision or intended decision, and
includes the solicitation terms (and any addenda), the award of the contract, and a rejection of all bids.
At the time of filing the Formal Written Protest the protestor must also file a Protest Bond payable to the
Department in an amount equal to one percent (1%) of the estimated contract amount. F.S., Section
287.042(2) (c) and F.A.C. Rule 28-110.005 contain further terms relating to the Protest Bond, including how to
determine the estimated contract amount. In lieu of a Protest Bond, the Department will accept cashiers
checks, official bank checks or money orders. The bond shall be conditioned upon the payment of all costs
and charges that are adjudged against the protestor in the administrative hearing in which the action is
brought and in any subsequent appellate court proceeding.
The Notice of Protest, Formal Written Protest, and Protest Bond shall be filed with the issuing office as defined
in SECTION 4.3 above.
4.5
ORAL INSTRUCTIONS / CHANGES TO THE INVITATION TO NEGOTIATE (ADDENDA)

No negotiations, decisions, or actions will be initiated or executed by a Respondent as a result of any oral
discussions with a state employee. Only those communications which are in writing from the Bureau of
Contracts, Grants and Procurement Management Services, will be considered as a duly authorized
expression on behalf of the Department until negotiation is awarded.
Notices of changes (addenda) will be posted on the VBS, under this Reply number. It is the responsibility of all
potential Respondents to monitor this site for any changing information prior to submitting a Reply. All
addenda should be acknowledged by the Respondents signature and subsequent submission of addenda
with Reply when so stated in the addenda.
4.6
MODIFICATIONS, RESUBMITTAL AND WITHDRAWAL

Respondents may modify submitted replies at any time prior to the Reply due date. Requests for modification
of a submitted Reply shall be in writing and should be signed by an authorized representative of the
Respondent. Upon receipt and acceptance of such a request, the entire Reply will be returned to the
Respondent and not considered unless resubmitted by the due date and time. Respondents may also send a
change in a sealed envelope to be opened at the same time as the Reply. The ITN number, opening date and
time should appear on the envelope of the modified Reply.
Unless specifically requested by the Department, any amendments, revisions, or alterations to Replies will not
be accepted after the closing for the receipt of Replies.
4.7
RESTRICTIONS ON COMMUNICATIONS WITH DEPARTMENT STAFF

Respondents shall not communicate with any Department staff concerning this ITN except for the Department
contact person identified in SECTION 4.3 REPLY QUESTIONS & ANSWERS of this ITN. Only those
communications which are in writing from the Bureau of Contracts, Grants, and Procurement Management
12
Services shall be considered as a duly authorized response on behalf of the Department. For violation of this
provision, the Department reserves the right to reject a Respondents Reply.
Respondents to this solicitation or persons acting on their behalf may not contact, between the release of the
solicitation and the end of the 72-hour period following the agency posting the notice of intended award,
excluding Saturdays, Sundays, and state holidays, any employee or officer of the executive or legislative
branch concerning any aspect of this solicitation, except in writing to the procurement officer or as provided in
the solicitation documents. Violation of this provision may be grounds for rejecting a Reply.
4.8
CONFIDENTIAL, PROPRIETARY, OR TRADE SECRET MATERIAL

The Department takes its public records responsibilities as provided under Chapter 119, F.S., and Article I,
Section 24 of the Florida Constitution, very seriously. If Respondent considers any portion of the
documents, data or records submitted in response to this solicitation to be confidential, trade secret
or otherwise not subject to disclosure pursuant to Chapter 119, F.S., the Florida Constitution or other
authority, Respondent must clearly mark and identify in its Reply those portions which are
confidential, trade secret or otherwise exempt. Respondent must also simultaneously provide the
Department with a separate redacted copy of its Reply. This redacted copy shall contain the Departments
solicitation name, number, and the name of the Respondent on the cover, and shall be clearly titled Redacted
Copy. The Redacted Copy shall be provided to the Department at the same time Respondent submits
its Reply to the solicitation and must only exclude or obliterate those exact portions which are claimed
confidential, proprietary, or trade secret. The Respondent should also provide two (2) electronic
copies (compact disc (CD), flash drive, etc.) of their Redacted Copy.
Respondent shall be responsible for defending its determination that the redacted portions of its Reply are
confidential, trade secret or otherwise not subject to disclosure. Further, Respondent shall protect, defend,
and indemnify the Department for any and all claims arising from or relating to Respondents determination
that the redacted portions of its Reply are confidential, proprietary, trade secret or otherwise not subject to
disclosure.
If Respondent fails to submit a Redacted Copy with its Reply, the Department is authorized to produce the
entire documents, data or records submitted by Respondent in answer to a public records request for these
records.
4.9.
POOR PERFORMANCE NOTICE

The Respondent should provide for both the Respondent and its employees, subcontractors, and
subcontractor employees, copies of any and all documents regarding complaints filed, investigations made,
warning letters or inspection reports issued, any notice of breach, notice of default, termination notice,
suspension notice, or any disciplinary action initiated or taken under any contract or job performance within
the past seven (7) years. For each instance listed, provide a narrative summary of the contracts purpose and
scope of work, the Respondents performance, including the concerns of the project owner, and any major
adverse findings. In addition, provide the contract or job number, the name of the owner, the term of the
contract, the name, address, and telephone number of the owners contract manager. Please also include
any relevant documentation evidencing the performance issues.
The Department reserves the right to seek further information on this matter from the Respondent or to make
inquiries with the project owner. The information obtained from this review may be reflected in the
Respondents score or used to declare the Respondent not a responsible vendor.
4.10 WITHDRAWAL OF A REPLY

A Respondent may withdraw a Reply by written notice to the Department on or before the deadline specified
for the receipt of Replies in SECTION 3.4 CRITICAL EVENT DATES of this ITN. Such written notice is to be
13
submitted to the Issuing Office at the address specified in SECTION 4.3 REPLY QUESTIONS AND
ANSWERS of this ITN.
4.11 DISCLOSURE OF REPLY CONTENTS
All documentation produced as part of this solicitation shall become the exclusive property of the state and
may not be removed by the Respondent or its agents. All Replies shall become the property of the state and
shall not be returned to Respondent. The Department shall have the right to use any or all ideas or
adaptations of the ideas presented in any Reply. Selection or rejection of a Reply shall not affect this right.
4.12 AWARD
As in the best interest of the State, the right is reserved to award based on all or none, groups of services,
or any combination thereof, to a responsive, responsible Respondent. As in the best interest of the state,
the right is reserved to reject any and/or all Replies or to waive any minor irregularity in replies received. In
addition, the Department reserves the right, in its discretion, to correct deviations during the negotiation phase.
Conditions which may cause rejection of Replies include, without limitation, evidence of collusion among
Respondents, obvious lack of experience or expertise to perform the required work, failure to perform, or meet
financial obligations on previous contracts.
SECTION 5 SPECIAL CONDITIONS

5.0
AUTHORIZED TO DO BUSINESS IN THE STATE OF FLORIDA

Foreign corporations and foreign limited partnerships should be authorized to do business in the state of
Florida. Domestic corporations should be active and in good standing in the State of Florida. Such
authorization and status should be obtained by the Reply due date and time, but in any case, must be
obtained prior to contract execution. For authorization, contact:
Florida Department of State
Tallahassee, Florida 32399
(850) 245-6053
5.1
LICENSED TO CONDUCT SERVICES IN THE STATE OF FLORIDA

If the services being provided require that individuals be licensed by the Florida Department of Business and
Professional Regulation or any other state or federal agency, such licenses should be obtained by the Reply
due date and time, but in any case, must be obtained prior contract execution. For state licensing, contact:
Florida Department of Business and Professional Regulation
(850) 487-9501
5.2
OTHER CONDITIONS
Other conditions which may cause rejection of replies include, without limitation, evidence of collusion among
Respondents, obvious lack of experience or expertise to perform the required work, failure to perform or meet
financial obligations on previous contracts.
5.3
IDENTICAL EVALUATION OF REPLIES

Whenever two (2) or more replies which are equal with respect to price, quality, and service are received, the
Department will determine the order of award using the criteria established in 60A-1.011, F.A.C. The "DrugFree Workplace Program Certification" can be found as Attachment 2.
14
5.4
DISCLOSURE STATEMENT
The Disclosure Statement Form (Attachment 4) should be signed and submitted with the Reply.
5.5
SUB-CONTRACTING
This contract or any portion thereof shall not be sub-contracted except as permitted herein, with the prior
written approval of the Department. No sub-contract shall, under any circumstances, relieve the Contractor of
its liability and obligation under this contract; and despite any such sub-contracting the Department shall deal
through the Contractor, which shall retain the legal responsibility for performing the Contractor obligations.
The Prime Contractor shall report all Minority Subcontractors, identifying the Name, Address, Type of
Certification and Dollar Amount on the Utilization Summary form, attached as Attachment 5. A list of
subcontractors shall be provided to the Departments contract manager upon execution of the Contract. The
Prime Contractor shall submit the Utilization Summary form with each invoice submitted for payment. The
form must be submitted with all invoices, regardless if funds have not been spent with a Minority
Subcontractor for the period covered by the invoice. The Office of Supplier Diversity, Florida Department of
Management Services, will assist in furnishing names of qualified minorities. The Office of Supplier Diversity
can be reached at (850) 487-0915; the Internet Web address is
http://dms.myflorida.com/other_programs/office_of_supplier_diversity_osd/.
5.6
CONTRACTUAL OBLIGATIONS
The Departments Contract Standard Terms and Conditions are incorporated in this ITN as Attachment 6 and
will govern the relationship between the Department and the Contractor. A Reply submitted by the successful
Respondent(s) shall be incorporated into the final contract(s). The terms and conditions of this ITN shall
control notwithstanding any statement to the contrary by the Respondent, unless such terms and
conditions are modified by the Department during the negotiation phase.
5.7
METHOD OF PAYMENT
Compensation and payment will be made in accordance with the terms and conditions of the contract.
5.8
CONVICTED VENDOR LIST

A company placed on the Convicted Vendor List may not submit a Reply or be awarded a contract to provide
any goods or services pursuant to Rule 60A-1.006 F.A.C. The Convicted Vendor List is published at:
http://www.dms.myflorida.com/business_operations/state_purchasing/vendor_information/convicted_suspen
ded_discriminatory_complaints_vendor_lists/convicted_vendor_list.
5.9
DIVERSITY IN CONTRACTING
The State of Florida is committed to supporting its diverse business industry and population through ensuring
participation by minority-, women-, and service-disabled veteran business enterprises in the economic life of
the state. The state of Florida Mentor Protg Program connects minority-, women-, and service-disabled
veteran business enterprises with private corporations for business development mentoring. We strongly
encourage firms doing business with the State of Florida to consider this initiative. For more information on
the Mentor Protg Program, please contact the Office of Supplier Diversity at (850) 487-0915.
The State is dedicated to fostering the continued development and economic growth of small, minority-,
women-, and service-disabled veteran business enterprises. Participation by a diverse group of Vendors
doing business with the state is central to this effort. To this end, it is vital that small, minority-, women-, and
service-disabled veteran business enterprises participate in the States procurement process as both
Contractors and subcontractors in this solicitation. Small, minority-, women-, and service-disabled veteran
business enterprises are strongly encouraged to contribute to this solicitation.
15
Information on Certified Minority Business Enterprises (CMBE) and Certified Service-Disabled Veteran
Business Enterprises (CSDVBE) is available from the Office of Supplier Diversity at:
http://dms.myflorida.com/other_programs/office_of_supplier_diversity_osd/.
5.10 COSTS INCURRED IN RESPONDING
This ITN does not commit the Department or any other public agency to pay any costs incurred by the
Respondent in the submission of a Reply or to make necessary studies or designs for the preparation thereof,
nor to procure or contract for any articles or services.
5.11 SUBMISSION OF REPLIES BY SUBSIDIARIES OR AFFILIATES
A Respondent, its subsidiaries, affiliates, or related entities is limited to one (1) Reply. Submission of more
than one (1) Reply per activity by a Respondent may cause the rejection of all Replies submitted by the
Respondent. In the alternative, the Department my decide, in its sole discretion, which Reply to
evaluate and consider. A subsidiary or affiliate of a prime Respondent may also be included as a
subcontractor in another Respondents Reply.
5.12 PROHIBITION OF GRATUITIES
By submission of a Reply, the Respondent certifies that no elected or appointed official or employee of the
state of Florida has or will benefit financially or materially from this procurement. Any contract arising from this
procurement may be terminated by the Department if it is determined that gratuities of any kind were either
offered to or received by any of the aforementioned officials or employees from the Respondent or its agents
or employees.
5.13 INDEPENDENT PRICE DETERMINATION
A Respondent shall not collude, consult, communicate, or agree with any other Respondent regarding this
procurement as to any matter relating to the Respondents Reply.
5.14 FINANCIAL CONSEQUENCES
The Contractor shall submit to the Department for payment the above deliverables and those proposed and
accepted by the Department that are satisfactorily completed in accordance with the terms and conditions set
forth in SECTION 7.0 SCOPE OF SERVICES and SECTION XXXXX DELIVERABLES. Payment shall be
delayed for any deliverables that are not satisfactorily completed until all deficiencies are corrected and the
services have been accepted by the Department. In addition, delays in providing completed deliverables will
subject the Contractor to damages.
5.15 PERFORMANCE BOND
The Contractor shall supply to the Department a Performance Bond in the amount of ten percent (10%) of the
amount of the annual award. The surety shall be in a form acceptable to the Florida Department of Education,
such as a bond, cashiers check, certified check or money order. A Surety must be authorized to do business
in the state of Florida. The Performance Bond shall be executed and furnished to the Department within ten
(10) calendar days prior to the Contractor beginning work under the contract. The performance bond must
state that it includes coverage of liquidated damages assessed against the contractor. A performance bond is
not required from Florida State Universities.
Cap: For each calendar, contract or fiscal year which corresponds to the period of the performance bond
specified, the cumulative total of liquidated damages for which the Contractor(s) may be liable shall not
exceed, annually, ten percent (10%) of the total contract amount at the time of assessment of liquidated
damages.
16
5.16 PARTICIPATION IN FUTURE STAGES OF THIS PROJECT

As stated in Chapter 287.057 F.S.
17(a)c A person who receives a contract that has not been procured pursuant to subsections (1) through (3)
to perform a feasibility study of the potential implementation of a subsequent contract, who
participates in the drafting of a solicitation or who develops a program for future implementation, is
not eligible to contract with the agency for any other contracts dealing with that specific subject
matter, and any firm in which such person has any interest is not eligible to receive such contract.
However, this prohibition does not prevent a vendor who responds to a request for information from
being eligible to contract with an agency.
5.17 ACCESSIBLE ELECTRONIC INFORMATION TECHNOLOGY
Respondents submitting Replies to this solicitation must provide electronic and information technology resources
in complete compliance with the accessibility standards required by Section 282.601-282.606, F.S., and Rule 608.002, F.A.C. These standards establish a minimum level of accessibility.
5.18 SCRUTINIZED COMPANIES LISTS (as applicable)
Section 287.135, Florida Statutes, requires that at the time a company submits a bid or proposal for a contract
for goods or services of $1 million or more, the company must certify that the company is not on either the
Scrutinized Companies with Activities in Sudan List or the Scrutinized Companies with Activities in the Iran
Petroleum Energy Sector List. Both lists are created pursuant to section 215.473, Florida Statutes.
Replies of $1 million or more should include the attached Scrutinized Companies Lists Form (Attachment 6)
to certify the respondent is not on either of those lists. The Form should be submitted with the Technical
Reply.
5.19 LIQUIDATED DAMAGES
Accurate and timely delivery is imperative and, as a result, the contract will include the following provisions for
the completion of the Work Tasks/Activities and Interim Deliverables and Final Deliverables identified in this
ITN.
The Contractor(s)s failure to complete Work Tasks/Activities and Interim Deliverables and Final Deliverables
both correctly and on time will result in substantial injury to the Department and the State of Florida, but the
amount of damages resulting from such injury cannot be calculated with certainty. Each such failure to
complete Work Tasks/Activities and Interim Deliverables and Final Deliverables both correctly and on time is
hereinafter referred to as a default. Defaults shall be deemed corrected on the date that the Work
Tasks/Activities and Interim Deliverables and Final Deliverables have been correctly completed. For each
default, the Contractor(s) shall be liable to the Department for liquidated damages and not as a penalty, as set
forth below:
a. Tasks/Activities and Interim Deliverables: For each default on performance and completion of Work
Tasks/Activities and Interim Deliverables (see SECTION 7.8), it is agreed that for each and every day
work remains uncompleted beyond the time set for its completion the Contractor(s) shall be liable to the
Department in a specified amount of three thousand dollars ($5,000.00) per business day as liquidated
damages for such delay, and not as a penalty, from the date of notice by the Department until the default
has been corrected.
THE REMAINDER OF PAGE INTENTIONALLY LEFT BLANK
17
b. Final Deliverables: For each default on completion of Final Deliverables (see SECTION 7.8), the
Contractor(s) shall be liable to the Department in a specified amount per calendar day from the date of
notice by the Department until the default has been corrected. The specified amount of damages is
defined below:
Schedule of Liquidated Damages
For Final Deliverables
Number of business days
Liquidated Damages
elapsed after the default
First
$5,000.00
Second
$15,000.00
Third
$30,000.00
Fourth
$45,000.00
Fifth
$60,000.00
Sixth
$75,000.00
Seventh/ subsequent days
$125,000.00 per day
c. Cap: For each calendar, contract, or fiscal year which corresponds to the period of the performance bond
specified in ITN Section 5.15, the cumulative total of liquidated damages for which the contractor may be
liable shall not exceed, annually, ten percent (10%) of the total contract amount at the time of assessment
of liquidated damages.
d. Collection: At its option, the Department shall collect liquidated damages by making claims against the
performance bond, from time to time, until the bond has been exhausted or by deducting the liquidated
damages from contract payments to the contractor. In the alternative, the Contractor may make payment
to the Department in the amount of such liquidated damages.
e. Deadline Adjustments:
(1) In the event the contractor wishes to propose a change of deadline for a Critical Work Task, the
contractor may propose the change to the Department, in writing. The Department has no obligation
to agree to the change, but shall notify the contractor, in writing, as to whether or not it agrees to the
proposed change. If the Department agrees, the modified deadline shall become the effective
deadline for purposes of assessing liquidated damages.
(2) In the event the Department wishes to change a deadline for an Extremely Critical Work Task to a
date which shortens the contractors time for meeting the deadline, it shall notify the contractor of the
new deadline in writing in the form of a contract amendment. If the contractor determines that by
exercise of every reasonable effort, the contractor will be unable to meet the new deadline, the
contractor shall offer a proposed shortened deadline which the contractor can meet by the exercise of
every reasonable effort.
(3) In the alternative, the contractor may propose a contract amendment in which the additional
personnel hours needed to achieve the shortened deadline are compensated by a price adjustment.
(4) If the parties are unable to reach agreement, the Department shall determine the deadline. However,
the issue of whether liquidated damages apply to the contractors failure to meet the shortened
deadline shall be reached as follows. The parties shall submit to binding arbitration the issue of what
date (earlier than the original deadline) the contractor could have completed the Extremely Critical
Work Task on time and correctly by exercising every reasonable effort. The date determined through
arbitration shall be the deadline for purposes of assessing liquidated damages. Arbitration shall be
conducted pursuant to the procedures of the American Arbitration Association, subject to the Florida
Arbitration Code, Chapter 682, F.S.
18
f.
Waiver: The Department, in its sole discretion, may waive the imposition of liquidated damages or a
portion thereof in a given instance. Such waiver, in any instance, shall not constitute a waiver in any
future instance, nor establish any right on behalf of the contractor to a waiver.
g. Department Approval: In situations where in the contractor must obtain the Departments approval of an
activity or product before the contractor can complete a Critical Work Task, the contractor shall be
responsible for providing a reasonable time for the Department to complete its review and for the
contractor to correct any deficiencies. In the event the Departments review is not conducted within a
reasonable time, as described in Section 7.7.4, that shall be grounds for the contractor to propose a
deadline adjustment pursuant to Subsection 5.19(e)(1), above. The parties contract managers may
agree, in writing, in advance, as to the reasonable time for the Departments review of a specific activity or
product.
SECTION 6 SPECIAL INSTRUCTIONS REPLY FORMAT & CONTENT

6.0
REPLY SUBMISSION
Both Technical and Price Replies will be received as specified in SECTION 3.4 CRITICAL EVENT DATES.
All Replies and associated forms should be signed and dated in ink by a duly authorized representative of the
Respondent. The overall Reply should be written in a concise manner, which is conducive to effective
evaluation and selection.
All Replies and related documents submitted in response to this ITN shall become the property of the state.
6.1
MAIL OR DELIVER REPLIES TO: (Do Not Fax or E-Mail)

Florida Department of Education
Attn: Fran Shewan
6.2
REPLY FORMAT INSTRUCTIONS

This section contains instructions that describe the expected format for the Reply. All Replies submitted
should contain two parts and be marked as follows:
PART I TECHNICAL REPLY NUMBER ITN 2014-47
(One Separately Sealed Package for Technical)
PART II PRICE REPLY NUMBER ITN 2014-47
(One Separately Sealed Package for Prices)
THE SEPARATELY SEALED PACKAGES MAY BE MAILED TOGETHER IN ONE ENVELOPE OR BOX.
6.2.1 Preliminary Administrative Review

The absence of any of these documents may result in a determination that the Reply is nonresponsive and the Reply shall not be evaluated. The Reply forms furnished must be used when
submitting the Reply. Forms are to be filled out in ink or typewritten. Administrative Review information
shall be included in the Technical Reply portion (Part I) of the Reply and consist of the following:
All Addenda
DUNS Number
19
Disclosure Statement Form (Attachment 3)

Reference Form (Attachment 4) Provide at least three (3) references, which demonstrate efforts
comparable to the one described in this ITN. The Department reserves the right to contact the references
regarding the services provided. Any information provided will be subject to the requirements of the
Florida Public Records Law.
Transmittal Letter Provide a Transmittal Letter (on Company Letterhead) that contains the following:
a statement confirming that the person signing the Reply is authorized to represent the Respondent
and bind the Respondent relative to all matters contained in the Respondent's Reply
the companys federal tax identification number
the companys DUNS number
a statement confirming that the Respondent has read, understands, comply and agrees to all
provisions of this ITN, unless changed during the negotiation process. By submitting a Reply,
the Respondent agrees to the terms and conditions of the ITN notwithstanding any statement
in the Reply to the contrary. However, the Department will attempt to resolve any
disagreements in the negotiation sessions.
a statement confirming that the Respondent is authorized to conduct business in Florida in

accordance with the provisions of Chapter 607, F.S. In lieu of such statement, the Respondent
alternatively should confirm that authorization to do business in Florida will be secured prior to the
award of the contract.
a statement confirming that the Respondent is registered on the MyFloridaMarketPlace website in
accordance with the provisions by the State of Florida. In lieu of such statement, the Respondent
should alternatively confirm that registration authorization will be completed prior to the award of the
contract.
a statement confirming that the Respondent has electronically registered a valid W-9 with the
Department of Financial Services (DFS). DFS is ready to assist any vendors with questions, and
vendors must submit their W-9 forms electronically at https://flvendor.myfloridacfo.com. Contact the
DFS Customer Service Desk at (850) 413-5519 or FLW9@myfloridacfo.com with any questions. In
lieu of such statement, the Respondent should alternatively confirm that registration will be completed
prior to the award of the contract.
6.2.2 Technical Reply (Part I) (20 hard copies)

(Do not include price information in Part I)
The Respondent should submit one (1) original in hard copy, nineteen (19) hard copies and twenty (20) in
electronic format (compact disc (CD), flash drive, etc.) copies in Microsoft Word 5.0 or higher, or
Adobe Acrobat of the Reply which is to be divided into the sections described below. The Department
expects all Replies to be in this format. Failure of the Respondent to follow this outline may result in the
rejection of the Reply, or result in a lower evaluation score. Any rejection of a Reply, or other adverse affect
on a Respondents Reply, due to its failure to properly organize its Reply shall be at the sole risk of the
Respondent. The Technical Reply should be submitted in a separate sealed package marked "TECHNICAL
REPLY FOR ITN 2014-47".
1. EXECUTIVE SUMMARY
The Executive Summary should be written in non-technical language to summarize the Respondents
overall capabilities and approaches for accomplishing the services specified herein. Evidence should
include, but not be limited to, respondents mission, date of founding, size, and experience. This section
should also demonstrate the Respondents clear understanding of the Departments goal to securely and
reliably develop, administer, score, and report Floridas assessments in an efficient and cost-effective
manner. The Executive Summary should be limited to no more than six (6) pages.
20
2. QUALIFICATIONS AND EXPERIENCE

The respondent should provide a description of qualifications and prior experience performing tasks
similar to those required in this ITN. The discussion should include a description of the respondents
background and relevant experience that qualifies it to provide the products and services required by this
ITN.
The respondent should document the contracted services for previous assessment projects similar to the
one described in this ITN and any other projects the respondent believes will document its corporate
capability. For each, the documentation should include a description of the services and products
delivered; the contract period; and the name, address, and telephone number of a contact person for each
of the contracting agencies.
If subcontractors are to be used for any portion of the work activities, the respondent should name the
subcontractor(s) and document the experience and qualifications of the subcontractor(s) in performing
tasks identical or similar to those they will be asked to do. A separate chart in the proposal may be used to
identify all of the subcontractors proposed to be involved in the project and the services they are expected
to provide. The respondent also should provide examples of materials that demonstrate the quality of the
work done by the respondent on similar projects.
The proposal should include information sufficiently descriptive to permit the proposal evaluation
committee to consider the experience and capability of the prime contractor as well as the subcontractors.
As part of the proposal evaluation, the proposal evaluation committee will contact one or more of the
agencies for whom the respondent previously had been engaged to obtain information and
recommendations. Committee members may also depend upon and use their own personal knowledge of
the respondent, the subcontractors, and the proposed personnel as they evaluate each proposal.
3. TECHNICAL PLAN
The technical plan should provide a description of the respondents proposal to meet the requirements of
this ITN. This discussion should encompass all of the requirements of Section 7.1.0 through Section
7.6.3.6.3 and associated deliverables of this ITN and should be organized and numbered in accordance
with the organization and numbering system used in Sections Section 7.1.0 through Section 7.6.3.6.3.
The technical plan should describe the methodologies, quality management standards and best practices
that will be employed in managing and completing the requirements. The descriptions should be
thorough, clear, yet as concise as possible. The Respondent should also identify where significant
difficulties may be anticipated and describe any specific techniques proposed to be used to resolve the
difficulties.
TECHNICAL APPROACH
The Respondent should explain the approach, capabilities, and means to be used in accomplishing the tasks
in the Scope of Services, and where significant development difficulties may be anticipated and resolved. Any
specific techniques to be used should also be addressed.
TECHNICAL APPROACH ORGANIZATION OF THE INVITATION TO NEGOTIATE
Section 7.1 of the ITN states the purpose of the ITN; describes the current Florida statewide assessment
program and modifications to the program that will be implemented under the ITN; and provides information
for preparing replies.
Sections 7.2 through 7.7 describe the scope of work for the years covered by this ITN. References are
made in Sections 7.2 through 7.7 to the appendices, which provide detailed information about printed
products, reports, shipments, and other elements of Floridas assessment program. Each section includes a
description of products and services to be provided.
21
Section 7.2.0: Program Descriptions. An overview of the standards-based assessments requiring services
described in this ITN.
Section 7.3.0: Test Development. A description of the test development tasks from item writing through the
production of final test forms.
Section 7.4.0: Test Administration. A description of the test administration tasks, including the delivery and
return of test materials, the production of test administration ancillary materials, providing customer service
and technical support, needs for the online management system, and administering computer-based
assessments.
Section 7.5.0: Scanning, Scoring, and Reporting. A description of the scanning, scoring, and reporting
tasks from preparing school and student information through the reporting of results. The psychometric
aspects of scoring are covered in Section 7.6.0.
Section 7.6.0: Scaling, Equating, Scoring, and Special Psychometric Studies. A description of the
scaling, equating, scoring, and special psychometric studies and tasks.
Section 7.7.0: Program Management. A description of how the work of the contract should be managed,
including meetings, staff, and reimbursable funding categories.
SYSTEM DEMONSTRATION (ELECTRONIC DEMONSTRATION)
The Respondent should submit two (2) copies with the Technical Reply, in electronic format (compact
disc (CD), DVD, flash drive, etc.) a demonstration (demo) of the proposed system. The system demo
provided should not exceed 25 slides or 30 minutes of presentation.
Do not include pricing information with the Technical Reply or the demo. Any inclusion of prices/discounts in
the Technical Reply or demo may be grounds for deeming the Respondents Reply nonresponsive.
After the Evaluation Phase is completed, the Department will notify the Respondents invited to participate in
Negotiations. Only those Respondents entering into Negotiations will be allowed and required to present an
on-site oral presentation of the proposed operational system. The Respondents on-site presentation will be
conducted during the Negotiation meeting. The actual date, time and duration of Negotiation
meetings/presentations are to be determined.
4. RESPONDENT'S MANAGEMENT PLAN
The management plan should describe the organization and document its capacity to provide the
products and services required by this ITN. This discussion should specifically encompass the
requirements of Section 7.7.0 through Section 7.7.10 and associated deliverables of this ITN and should
be organized and numbered in accordance with the organization and numbering system used in Section
7.7.0 through Section 7.7.10. The descriptions should be complete, clear, and concise.
The management plan should identify by name the managers and professional personnel responsible for
this project. Charts showing the organizational structure and the incorporation of proposed project staff
into the structure should be included, as well as the amount of time in FTE person-days that each person
will devote to the major tasks and activities.
The Respondent should include a description of the methodology to be used to manage the project work,
including control of costs, assuring quality, identifying and minimizing risk, and maintaining schedules;
Vitae should be included as an appendix to the proposal and should include relevant educational
background and professional experience. Personnel assigned by the contractor should demonstrate
qualifications and competencies of the position to which assigned.
It is assumed that the contractor may use outside printers for some materials, such as preprinted forms.
Printers will be documented as subcontractors, and the management plan should identify the proportion of
22
materials to be printed by the contractor and by outside vendors. Procedures for quality control and
security during printing should be described.
6.2.3 Price Reply (Part II) (20 hard copies)
The Respondent should submit one (1) original hard copy, nineteen (19) hard copies and twenty (20)
electronic format (compact disc (CD), flash drive, etc.) copies in Microsoft Excel 5.0 or higher. The
Respondents price information should be submitted on the form provided in this ITN. Prices submitted in the
Price Reply form, Attachment 1, will be used to evaluate the price component of the evaluation. Payment for
services provided under the contract will be made based on the deliverable schedule submitted, Section 7.8,
The Price Reply and deliverable schedule should be submitted together in a separate sealed package marked
"PRICE REPLY FOR ITN 2014-47".
6.2.4 Presenting the Reply
The Reply should be limited to a page size of eight and one-half by eleven inches (8" x 11"). Type size
should not be less than a 12 point font. The Reply shall contain a table of contents, be typed single-spaced
and have separate parts, each clearly labeled including page numbers. The information to be addressed in
each part is described in the above sections. The absence of information or the organization of information in
a manner inconsistent with the terms of this ITN may result in the rejection of the Reply, or a reduced
evaluation score. Bindings and covers will be at the Respondent's discretion; however, elaborate
notebooks/hard back binders are discouraged.
Unnecessarily elaborate special brochures, art work, expensive paper and expensive visual and other
presentation aids are neither necessary nor desired.
All Respondent materials should be packaged so that each box of materials shipped to the Department does
not exceed 25 pounds.
SECTION 7 SCOPE OF SERVICES

7.0
SCOPE OF SERVICES
7.1.0 Introduction
This ITN addresses four broad assessment components as described below. Respondents may submit bids
on any individual assessment component (A, B, C or D), any group of assessment components, or to provide
all assessment components (A, B, C and D):
A. Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 3-8
B. Computer-based and paper-based assessments in Mathematics in Grades 3-8
C. Computer-based and paper-based assessments in English Language Arts/ Literacy in Grades 9-11
D. Computer-based and paper-based Endof-Course (EOC) assessments in Algebra 1, Geometry, and
Algebra 2
The Department will continue to phase in computer-based assessments in Grades 3-8, transitioning
completely to computer-based by 2017-2018 according to the preferred schedule in Appendix A. However,
depending on the final award for the individual assessments, the Department may vary when tests will be
administered as computer-based or paper-based. The final determination of which tests will be computerbased or paper-based is at the Departments discretion.
This ITN will specify a complete package of products and services so that the FSA will be developed,
administered, scored, and reported separately from other assessments aligned to Floridas Next Generation
Sunshine State Standards under a contract currently held by NCS Pearson. This ITN does not include the
23
Science, Social Studies, FCAT 2.0 Reading Retake, and Florida EOC Assessments aligned to the Next
Generation Sunshine State Standards that will be covered in a separate procurement.
Although this ITN may result in the award of multiple contracts, the term Contractor is used throughout and
refers to each contractor providing the required products and services for each grade and subject for which the
contract is awarded.
7.1.1 Notice to Respondents
1. This ITN provides for implementation of Floridas standards-based assessment program. The contract will
include development, psychometric services, administration, scoring, and reporting of Florida Standards
Assessments in English Language Arts/Literacy and Mathematics for the test administration cycles for
2014-2015 through 2016-17 with possible extension of services in three one-year renewal periods, 201718 through 2019-2020. Replies will be evaluated on the basis of the respondents qualifications and
experience and the technical and cost replies for the tasks described herein for the entire span of the
ensuing contract.
2. This ITN defines the requirements for implementing Floridas standards-based assessment program. The
ITN and the selected contractors reply, together with clarifying documents, define the work to be
conducted under contract. These documents will be incorporated into the contract resulting from the state
standards-based assessment program project award. Because Floridas standards-based assessment
program is technical and complex, it is possible that a responsive reply may not totally or clearly reflect
ITN requirements in all details. If the reply of a respondent selected as a result of the bidding process is
inconsistent with the ITN, the requirements of the ITN will prevail; the selected contractor will be expected
to perform all ITN requirements without an increase in cost above the proposed cost. Furthermore, by
submitting a bid the respondent is agreeing to be bound by the terms of the ITN notwithstanding any
language in the reply to the contrary. Any discrepancies between the terms of the ITN and the reply will
be resolved in the negotiation process at the Departments discretion, and when it is in the states best
interest.
3. The Department reserves the right to substitute or modify tasks on a generally equal basis. Substitutions
or modifications will be made prior to the selected contractor's beginning significant efforts to complete the
task. All design changes, substitutions, or modifications will be given to the contractor in writing. It is not
the intent of this provision to increase the scope of work of the contract but to protect the Department as
design changes to improve the assessment become known during the implementation of development
and administration activities. The Department also reserves the right to amend the contract resulting from
this bid competition to add, modify, or delete designated tasks, products, and/or services related to the
core activities of this project.
4. In developing the solution pursuant to this procurement, a respondent may propose to provide a custom built
solution, a solution comprised of pre-existing, proprietary materials or products that are the intellectual
property of the respondent or third parties, or a combination thereof. A respondent may propose that all
intellectual property developed be owned solely by the Department, owned jointly by the Department and the
respondent, or, at a minimum, that the Department be granted a perpetual and non-exclusive license for the
use, reproduction and modification of the solution for use in Florida. The Department reserves the right to
request alternative intellectual property arrangements or combinations of such arrangements during
negotiations with respondents, in order to obtain a reply that provides the best value to the state. All
permissions or licenses required for the use of materials or products that are the intellectual property of a third
party must be obtained by the respondent. All materials, products, and licenses must be transferred to the
Department at the end of the contract period such that the Department can maintain and sustain the solution
beyond the end of the contract period.
5. The selected contractor will work closely with the Department and with groups of Florida educators to
complete all tasks. All procedures, processes, and products used by the contractor to complete contract
work must be approved by the Department. Throughout the contract period, the contractor will confer with
24
the Department on a continuing and consistent basis and will be involved in frequent face-to-face
meetings with the Department, as necessary.
6. Several reimbursable funding categories identified in Section 7.7.9 are set aside for specific functions
during the course of the project. Expenditures in these categories will be reconciled on a quarterly basis.
7. The dates for reporting assessment results necessitate a restricted time period to complete the work of
scoring activities. Respondents are expected to propose efficient and innovative procedures to meet
these required reporting dates and are expected to demonstrate in their replies that they have the
capability and are willing to commit the human and technological resources needed to complete the
scanning, student response file processing, scoring, and reporting tasks in the required time period.
8. The fundamental purpose of Floridas standards-based assessment program is to provide accurate
information on student achievement. The contractor will utilize every means required to ensure that
information created by the project is accurate. The selected contractor will be responsible for correcting
at the contractors expense any errors arising from activities that are the responsibility of the contractor.
This may involve activities such as conducting analyses to identify the cause and extent of errors;
reprogramming or reproducing products or other materials; replacing data files; reproducing reports; and
shipping replacement products or reports to the Department or districts using expedited shipping services.
9. News releases or public announcements pertaining to this ITN, a contract award resulting from
the ITN, or work completed under a contract will not be made without prior written approval from
the Department.
7.1.2 Project Schedule
The deliverables schedule presents a list of major project activities and the dates for completion where these
can be determined. Many of the deliverable dates are contingent upon finalized administration dates and will
be determined during the negotiation process. Appendix B indicates critical and extremely critical tasks and
provides general guidelines for the timeframes for completion, but these are also contingent on finalized
administration dates. The dates provided in 7.2.2 are subject to change during the negotiation phase.
Respondents should provide a detailed schedule that incorporates the activities included in this ITN for the
entire life of the contract. The schedule will indicate essential steps for the completion of project tasks,
initiation and completion dates, and task responsibility. Tasks and responsibilities will include an explanation
of work effort required and full-time equivalent staff (FTE) committed to this work. The reply should also
provide total annual FTE and identify individuals and/or staff roles that will work full-time on this
project. Respondents should be aware that the schedules for each project year overlap. To accurately
estimate required annual resources, schedules for preceding and succeeding years need to be considered.
Following the contract award, the schedule will be further developed into a working document that will be
updated weekly, indicate all task completion dates, and be revised as needed with Department approval (See
Section 7.7.6). A draft work schedule, based on a sample provided by the Department, will be prepared by the
selected contractor for submission to the Department one week prior to the initial planning meeting, which will
be held no later than one month after the execution of a contract for the project. The schedule will be
submitted in hard copy and electronic format using Department-approved spreadsheet software that permits
sorting of the file by start or end date and major task and subtask groupings.
7.2.0 Program Descriptions
The selected contractor will provide the Department with a criterion-referenced achievement test, aligned to
and measuring mastery of the Florida Standards in English Language Arts/Literacy (ELA/L) and Mathematics
proficiency for administration to Florida public school students. For purposes of responses to this ITN, the
Florida Standards are to be considered those adopted by the Florida State Board of Education in July, 2010.
Any changes to the standards following additional Florida public review scheduled for fall 2013, and resulting
changes to the assessment will be addressed through the amendment of the contract that may be awarded as
a result of this ITN.
25
The primary purpose of the assessment is to provide student academic achievement and learning gains data
to students, parents, teachers, school administrators, and school district staff. This data is to be used by
districts to improve instruction; by students, parents, and teachers to guide learning objectives; by education
researchers to assess data; and by the public to assess the cost benefit of the expenditure of taxpayer dollars.
The assessment must be designed to:
1. Assess the achievement level and annual learning gains of each student in ELA/L and Mathematics.
2. Provide data for making decisions regarding school accountability, recognition, and improvement of
operations and management, including schools operating for the purpose of providing educational
services to youth in Department of Juvenile Justice programs.
3. Identify the educational strengths and needs of students and the readiness of students to be promoted to
the next grade level, to graduate from high school, and to succeed in college and in careers.
4. Assess how well educational goals and curricular standards are met at the school, district, and state level,
and at the national level, to the extent possible.
7.2.1 Expected number of districts, schools, and students
The expected number of students, shipping destinations, and schools are provided in Appendix C as a basis
for preparing replies. Respondents should calculate the quantities of materials needed to implement the test
administrations on the basis of these estimates. If the total number of tests administered across all grades
and test administrations exceeds the number of tests described in this appendix by more than ten percent
(10%), the Department will execute a formal change to the contractual agreement.
Appendix C also lists the 67 school districts and approximately 13 special districts. For the school districts and
the special districts combined, the contractor will ship testing materials to approximately 80 separate
destinations. In the 67 school districts and the special districts combined, approximately 4,000 separate
administration sites (including private schools, community colleges, etc.) will be involved in the test
administrations, and the contractor will expect that number to increase by as much as five percent (5%)
annually, calculated based on the previous year. Some of this increase is due to legislative changes that
increase the number of students participating in Floridas statewide assessment program. If the increase of
administration sites is beyond the anticipated five percent (5%), the incremental cost increase will be handled
in an amendment to the contract.
7.2.2 Potential Test Administration Dates
Potential projected starting dates for future assessment administrations relevant to this ITN are shown in Table
7.2.2 below for illustrative purposes only. The respondents reply should propose starting dates for spring
administrations that occur as late in the school year as possible. Replies should take into account the scoring
of constructed- and extended-response items, data verification activities, and reports of results by the required
dates (Appendix B).
26
The FSA EOC Assessment administrations will be at the same time as the Florida EOC Assessment
administrations. These dates are also provided, for reference, in Appendix A.
Table 7.2.2 EXAMPLE: Potential Test Administration Dates for Floridas Standard-Based Assessments
Table 7.2.2 EXAMPLE: Potential Test Administration Dates for Floridas Standard-Based Assessments
Year
Grades 3-11 ELA/L

Grades 3-8 Mathematics
Grade 10 ELA/L Retake
2014-15
March 2-13, 2015
N/A
2015-16
February 29-March 11, 2016
October 12-23, 2015

February 29-March 11, 2016
2016-17
March 6-17, 2017
October 10-21, 2016

March 6-17, 2017
2017-181
March 5-16, 2018
October 9-20, 2017

March 5-16, 2018
2018-191
March 4-15, 2019
October 8-19, 2018

March 4-15, 2019
2019-201
March 2-13, 2020
October 7-18, 2019

March 2-13, 2020
1 Optional
Algebra 1 EOC
Geometry EOC
Algebra 2 EOC
April 7-25, 2015
July 13-24, 2015
September 14-25, 2015
November 30-December 18, 2015
April 4-22, 2016
July 11-22, 2016
November 30-December 21, 2016
April 3-21, 2017
July 10-21, 2017
December 4-22, 2017
April 2-20, 2018
July 16-27, 2018
December 3-21, 2018
April 8-26, 2019
July 15-26, 2019
December 2-20, 2019
April 6-24, 2020
July 13-24, 2020
Renewal Period
7.3.0 Test Design and Development

All services related to test design, item development and review, item banking, and test production for the FSA
are the responsibility of the contractor selected under this ITN. Respondents may propose a customized
assessment, or products and services derived from existing products developed and published by the
contractor, or developed as part of other initiatives. It is understood that by the time the contract is executed
for the services requested in this ITN, test items may have already been developed for assessments that will
be operational in 2014-2015. However, the Department expects to be closely involved in the development and
review of new test items to be field tested and developed in subsequent years. To the extent possible, the
Department also expects to be involved in any test construction activities for 2015 assessments still to be
completed after contract execution. The Departments primary goal with respect to test design is ensuring that
the assessment measures student mastery of the Florida Standards, and accomplishes this in the most costeffective way without significantly increasing testing time beyond that of current Florida assessments. A
sample test design is provided in Appendix D.
The Department has established a Test Development Center (TDC), staffed with editors and Florida
educators, to provide quality assurance for all test content and related products, whether customized or
derived from existing products. The selected contractor will work closely with the Department and TDC staff to
ensure that the test philosophy, design, item specifications, and product quality are aligned with Floridas
expectations. The Department and TDC will also coordinate closely with the selected contractor to conduct
27
rangefinding meetings with educators in order to generate training materials for the scoring of performance
tasks. While the selected contractor will be required to coordinate all meeting activities and materials, the
Department and TDC will identify invitees and assist in facilitating all meetings. Required meetings are
referenced throughout the ITN. See Section 7.7.7 for a summary of details about educator/citizen meetings.
Test development procedures should include review and revision of proposed test and item specifications,
review of test blueprints, review and evaluation of items available for use on the test (item development plan),
field testing for needed items to complete the desired test design, and final operational test creation. All
phases of test development shall use accepted validity, reliability and other testing principles including
Universal Test Design. A plan for complete data review of field-tested items should be delineated. The
Department must approve the items and the test forms before they become operational. Respondents who
propose solutions derived from existing products that are used by other states must take into consideration the
requirement for including state educators in the quality control aspect of item development and scoring of
performance tasks. The extent of educator involvement will be discussed during negotiations and agreed
upon prior to contract execution.
To the greatest extent possible, the Department is seeking a solution that provides comparability of results
with other states. Such comparability could be obtained by a number of acceptable means. The assessment
could be one that is also administered in other states, or the design could include embedding of a small
number of norm-referenced items within each criterion-referenced test form. A design that uses embedded
norm-referenced items would not be expected to yield student-level comparative data, rather in this case, the
Department would expect reporting at an aggregate level. The final design will be discussed during
negotiations and agreed upon prior to contract execution.
7.3.1 Test Content
1. The FSA will consist solely of test items as described in the previous section. If an existing shelf product
test includes additional components beyond those described above, it will be necessary to remove these
items.
2. If the test being proposed is to be derived from an existing product, respondents should:
a. Describe the general content being measured by the proposed test.
b. Provide full-length inspection copies of the test and test administration manual for each grade level.
c. Provide a test blueprint that governs construction of the test, including the number of items per
objective and reporting category.
d. Describe the objectives being measured by the test.
e. Describe the taxonomic classification system used to classify each test item and the proportional
assignment of items within the taxonomic classification system.
f.
Describe the range of content difficulty included on each level of the test including the extent to which
the items at each level assess content that may be difficult for some students.
g. Describe the method by which text complexity and grade-level appropriateness was determined for
reading passages
h. Describe how grade-level appropriateness of item content for each level of the test was determined
i.
Describe the extent to which the items included at each level assess higher order thinking skills. If
the taxonomic classification system (See e. above) includes a thinking skills dimension, please
include this information in that description.
j.
Describe the manner in which the test is configured into subparts for both test administration and
score reporting purposes, including how reporting categories are related to the taxonomic
classification system.
28
k. Provide an indication of the minutes of testing time required for the separate content sections.
3. The respondent should describe the special materials or manipulatives required or permitted for correct
administration of the proposed test (e.g., online rulers or Department-approved calculators). Detailed
descriptions of or access to each required item should be provided with the technical reply.
4. Reading passages should expose students to grade-level text of appropriate complexity. Texts should
reflect a balance of authentic literature (stories, drama, and poetry) and informational text/literary
nonfiction, which would include passages that address the literacy standards requirement for history/
social studies, science, and technical subjects that students are expected to read. Selections should
include a combination of single texts and paired passages of sufficient complexity for close, analytical
reading which would allow students to discern and locate evidence from the text to support assertions in
multiple-choice items or writing responses. Respondents may propose passages from three sources,
including public domain, permissioned, and commissioned sources. Solely for cost comparison purposes,
respondents may assume a test design that includes 1/3 of each passage type, but respondents may
propose solutions that best meet the primary requirements of the ITN. The final test design will determine
the proportion and genre percentages for passages.
5. English Language Arts/Literacy items aligning to each passage may include multiple-choice items as well
as short, constructed responses to text-dependent questions. One extended response item should be
drawn from a paired or multiple-text passage set. Extended writing prompts should allow students to
produce opinions/arguments, informative/explanatory responses, and narratives.
6. The first tests to be administered by the selected contractor under this ITN will be the spring 2015 ELA/L
and Mathematics tests in Grades 3 high school.
a. The selected contractor will be required to provide stand-alone computer-based calculator
applications for 4-function, scientific, or graphing calculator use on computer-based tests, depending
on final test design. These computer-based calculator applications will be made available to districts
and schools for downloading to local computers and for online access and practice throughout the
year.
b. For any Florida test produced for students, the selected contractor will provide braille, large-print,
regular-print, and one-item-per-page versions at each grade level for students with disabilities. The
contractor must also provide computer-based accommodations for students with various disabilities,
as described in Section 7.3.11.2. A demonstration of these accommodations may be included in the
respondents presentation.
c. The selected contractor will also print and ship Braille and large print sample test materials upon
district request for students who will require these paper-based accommodations. These sample test
materials are considered generic, meaning once they are created by the contractor, there is no
expectation that they will be updated during the life of this contract.
7.3.2 Test Configuration
This section provides information on the configuration for all test documents and related materials for both
print and computer test delivery.
7.3.2.1 Test and Answer Document Design
Respondents should describe the proposed document design and layout for each grade and
subject test, for computer-based and paper-based tests, as described for each year in Appendix A.
The selected contractor designs and produces camera-ready art for the test forms that are used to
print test books and answer books or prepare computer-based test items in delivery-ready format.
Contractors are expected to provide the necessary human and other resources necessary to meet
extremely critical deadlines for delivery of documents and the computer-based test system and
files.
29
Respondents have been requested to submit bids for a paper-based and computer-based format
for each assessment. However, even if a computer-based assessment is the accepted solution,
any tests offered on the computer must also be prepared for use in a paper-based format (Braille,
large print, regular print, one-item-per-page) as noted in these tables. In this instance, the paperbased format will only be offered by special request for up to five percent (5%) of the students and
can be used at the Departments discretion.
To the extent that a respondents design solution allows for some customization, the Department
will consider replies that include:
1. Security-controlled booklets containing only FSA Reading passages and no test items for the
computer-based tests to be available by special request for up to five percent (5%) of the
students. See Section 7.3.11.1.
2. Blank work folders for all students taking the FSA Mathematics computer-based tests. The
work folder consists of four pages for students to do scratch work, the fourth page containing a
full-page grid.
3. Blank worksheets for all students taking the FSA ELA/L computer-based tests. The worksheet
consists of two pages for students to make notes.
4. A copy of the appropriate reference sheet, if proposed by the respondent and accepted by the
Department, may also be provided for students taking the assessment for which it is proposed.
5. Work folders, worksheets, and reference sheets/periodic tables are not numbered or
inventoried by the contractor; they are destroyed by the school after results arrive.
For respondents who do not offer such customization, replies should address how test design and
presentation eliminate the need for these considerations.
The Department may choose to modify the design of any state-specific test documents/files prior to
any test administration, and reserves the right to change this configuration beyond the
specifications but within the scope of the contract, if necessary, through change orders or contract
amendments. Appendix E provides additional details concerning these materials.
7.3.3 Develop Items
For items developed after contract execution, it is the responsibility of the selected contractor to ensure that
the quantity of items developed and submitted for review annually is sufficient to create operational test forms
for each administration. Respondents should propose item and passage counts by item type, standard or
cluster, as appropriate, needed to be developed annually in order to meet these needs. The contractor must
identify and recruit a sufficient number of highly-qualified writers for passages and items to ensure variety and
quality of development. Resumes for these writers should be provided and approved by the Department prior
to contract execution. Note that no active Florida educators will be approved as item writers.
The contractor must conduct in-depth training of passage/item writers at least annually. Department content
experts will attend the passage/item writer training to provide input and oversight for the process.
The selected contractor will be responsible for internal reviews to ensure item clarity, content accuracy,
adherence to Test Item Specifications, psychometric properties, difficulty, readability requirements, and
potential bias or insensitivity. It is unacceptable to submit to the Department proposed test items at any time
that contain blatant errors, such as violation of Test Item Specifications, spelling errors, or grammar mistakes.
The Department will not review any submissions with these types of obvious errors even if the impact creates
schedule challenges for the contractor, for which the contractor will be responsible.
7.3.4 Review Meetings
For test items developed for use on assessments following the spring 2015 administration, the selected
contractor will be responsible for organizing and convening meetings for the purpose of reviewing assessment
30
items for content and bias and community sensitivity issues by Department-appointed review committees
composed of Florida educators and other citizens. A detailed list of meetings to be conducted under this
contract and the funding mechanism for these meetings are provided in Sections 7.7.7 and 7.7.9 of the ITN.
For each meeting, the selected contractor will be required to maintain original forms of all committee member
test security and non-disclosure agreements. These will be made available to the Department on an asneeded basis.
The meetings will be held in a Florida location that is easily accessible (hub cities) to the majority of
participants and capable of handling the logistical requirements of a large-scale, multi-committee meeting.
The selected contractor will prepare assessment items and all other materials required to conduct the
meetings. Separate content review meetings will be conducted for each subject/grade combination.
If more than one contract is awarded as a result of this ITN, the Department will coordinate with the
contractors to determine the most efficient method of planning and conducting the review meetings. The
respondent may propose recommended solutions in the event of the award of multiple contracts.
7.3.4.1 Bias and Sensitivity Committee Reviews
The selected contractor is responsible for coordinating all logistics and for preparing materials for the
bias and community sensitivity review committees. The committees are normally composed of
representatives of Florida citizens and educators from various backgrounds who review each newly
developed test item. The primary purpose of each committee is to consider whether the subject
matter and language used is free of potential bias and acceptable to Florida students, parents, and
other community members.
Respondents should propose methods for conducting these reviews.
7.3.4.2 Item Content Review Committees
For the subject/grade combination content review meetings, each item should be projected on a
screen and changes should be made to items onscreen throughout the review process. The selected
contractor will be responsible for supplying the computers, projection devices, and other hardware
and software needed to project and modify items onscreen. The contractor will also provide
calculators, reference materials such as Test Item Specifications, textbooks relevant to the grade and
subjects being reviewed, dictionaries, and thesauri for each room as is appropriate to the nature of
the meeting. Print copies of the items being reviewed will be provided to each participant in securitycontrolled notebooks.
At the conclusion of each meeting, Department and contractor staff will review and evaluate changes
and ideas recommended by each committee. The Department will have final approval of all changes
to be made.
After these meetings, the selected contractor will conduct an editorial and content review of the items
to ensure that changes have been applied correctly and that all editorial and content issues are
corrected. This process is referred to as the Cleans review process. The Department will review
and have final approval of all items.
The selected contractors development project manager, the development team, and necessary
assistants will participate in the assessment item content review meetings to manage the meetings
and to be responsible for organizing, distributing, secure material tracking, and data entry functions
required for meetings. Arrangements will be required for collecting and keeping secure materials
locked in secure storage when not in use and for copying materials during the meeting.
7.3.4.3 Data Review Committees
Once items have been field tested, the contractor is responsible for convening a meeting to review
the statistical data on each item from field testing to identify any issues from a psychometric
perspective. This Data Review Committee may include state content experts, grade-level teachers,
31
accessibility and accommodations experts, and psychometricians, as appropriate. The primary

purpose is to ensure that all dichotomously-scored items and tasks behave as expected, include all
correct and appropriate responses, and are fair for all students. This committee may review
differential item functioning (DIF) statistics to ensure that an item did not unintentionally favor or
disfavor a particular subgroup of students. Additional considerations include determining whether an
item can be used as is or needs to be revised and, if so, whether it will be returned to the field test
pool for further testing. The contractor will be responsible for applying any recommended edits to
items and will submit the revised items for final Department approval.
7.3.4.4 Content Advisory Committees
Of particular importance for the design and development of Florida assessments are the Content
Advisory committees. The members of the ELA/L and Mathematics Content Advisory committees
advise the Department about the scope of the assessments for these areas. Their recommendations
may include: standards to be assessed; the number of benchmarks, passages, and items to be
addressed per grade level; the item types recommended for each assessed standard; and the
content of the Test Item Specifications documents.
7.3.5 Develop Field-Test Item Sets and Forms
For designs using embedded field testing, it is the responsibility of the selected contractor to group the
developed items into item sets that will be included in the operational forms of the test for each content area.
In making the decision as to which items will be field-tested, the selected contractors field-test proposals must
consider the areas of greatest need in the item bank, the range of content and rigor included in the items, and
the number of items available to be field tested.
7.3.6 Develop Stand-Alone ELA/L Field-Test Item Sets and Forms
Depending on the measurement model for the proposed ELA/L assessment, the Department will consider a
stand-alone field test for ELA/L extended constructed-response tasks. Respondents who propose a standalone field test should prepare field test sampling specifications detailing the sampling methodology. The
Department prefers that such a field test, if proposed, take place in the December time frame of each year of
administration. A field test administration is not listed in other sections of this ITN (schedules, test
administration manuals, etc.); however, if it is determined during negotiations that a stand-alone field test will
take place, all test administration and scoring schedules and deliverables will be agreed upon prior to contract
execution.
7.3.7 Constructing Test Forms
This section describes the requirements for constructing test forms, including the creation and maintenance of
test construction specifications. The test construction process must be supported by sophisticated computer
software, preferably a component of the item banking system that will generate test statistics and related
curves. The construction of operational forms must be completed in time to meet the requirements of this ITN
for delivery of test materials to districts.
7.3.7.1 Test Construction Specifications
The specific processes, guidelines, requirements, and schedule to be followed in constructing all
forms of the assessments are to be provided to the Department as detailed, expert-developed, and
referenced specifications. The test construction specifications must also include information about
the criteria for item selection, content representativeness, design of the assessment, evaluation
criteria for reviewing statistical characteristics of items, and desirable assessment characteristics. A
section of the specifications must address considerations and requirements for those assessments
that are pre-equated, if applicable, as well as a section for unique requirements of constructing
computer-based assessments. Careful and extensive development of the creation of linking or
anchor item sets must be included with details about statistical and content requirements for selecting
sets of items and criteria for annual refreshing/replacement of portions of the anchor item set.
Additionally, the specifications must include necessary information regarding item context/position
32
based upon previous use. The selected contractor must also retain assessment construction records
during the lifetime of the contract. These records must include assessment summaries (item IDs and
item/assessment statistics) of the various versions of proposed assessments and reasons for
decisions made regarding each version, and Department approval documentation for the final version
of each assessment.
7.3.7.2 Test Construction System
The contractor will provide a computer-based test construction system that interfaces with the item
banking system to select and sequence items, generate test curves, and provide statistical summary
tables and test curves upon demand. The system proposed by the respondent should be fully
described in the reply including technical specifications, screen shots and access to a demonstration
version of the system.
7.3.7.3 Construct the English Language Arts/Literacy, and Algebra 1, Algebra 2, and Geometry
Retake Forms
Grade 10 English Language Arts/Literacy and Algebra 1, Algebra 2, and Geometry retake tests will
be administered under this contract beginning in fall 2015. Respondents should propose pre-equated
retake forms that can be scored for reporting on accelerated timelines. Solutions for retakes that do
not include performance tasks will be considered, even if performance tasks are included in the
operational forms. Overall test characteristics must mirror those of the operational assessments;
however, the Department will discuss the design of the retakes during negotiation and a decision will
be finalized prior to contract execution.
7.3.7.4 Prepare Camera-Ready or Screen-Ready Test Forms
For each grade or subject test, the selected contractor is responsible for preparing camera-ready
documents or screen-ready files after the test items have been selected, composed in forms per
grade/subject, and reviewed. Preparing test documents for publication includes developing test book
configurations, test maps, specifying page layouts, proofing composed PDFs, and completing digital
proofs in conjunction with the Department and external reviewers, as appropriate to the selected
contractors proposed solution.
7.3.7.5 Interpretive Products
The selected contractor will develop interpretive products (IP) to interpret or describe test items for
specific subject areas or to explain test results. The contractor will be expected to propose sample
items, review publication PDF and digital rounds, provide alternate text for ADA-compliant files,
create or refine annotations for student responses, and ensure coherence and consistency with the
operational test. Interpretive products follow the production and review guidelines outlined in Section
7.3.10. Required products include the following:
Understanding Floridas Assessment Reports
This publication describes the various reports of test results distributed to students, schools, and
districts. An explanation and an image of each type of report are provided. The booklet also includes
information about the test content areas and a glossary of technical terms. The contractor will deliver
the electronic ADA-compliant PDF for posting to the DOE website so files can be posted to coincide
with the first results delivery (RD1).
Sample Test Materials (STMs)
STMs will be delivered for all assessments, including assessments that are primarily computerbased, and include 10-20 developed sample items in a sample test booklet for students with a
separate answer key booklet for teachers. Both documents are produced in ADA-compliant PDF files
for unsecure web posting and in large print and Braille accommodated versions. The printed
accommodations are delivered to schools each fall so students can practice answering various item
types. The ADA-compliant version is delivered at the same time so all students have the same
opportunity for practice. Respondents should propose designs for sample test books and answer
33
keys for the ADA-compliant PDF version and for printed large print and Braille accommodated
versions.
English Language Arts/Literacy and Mathematics Scoring Guides
The contractor will be expected to post electronic publications that provide information to help
educators understand the scoring of constructed-response items included on student reports. These
publications provide information about the test design, standards and benchmarks, item types, and
how items are scored. They also provide scoring rubrics and the annotated papers used to train
scorers for the constructed-response items. Each sample item from the scoring papers is displayed
as it appeared on the test and responses for each score point, images of student responses, and
annotations explaining why the response received a particular score are provided.
Computer-Based Test (CBT) Practice Tests
Respondents should specify a plan for delivering computer-based practice tests for all computerbased assessments, including accommodated computer-based assessments to be reviewed and
revised as needed in conjunction with Department and external reviewers. The purpose for these
practice tests is to familiarize students with the computer-based testing system and to provide
students with an opportunity to practice answering various item types.
All practice tests will include items similar to test format and content. The practice session may
combine subject items or use separate practice sessions for each subject area as the Department will
determine. Practice sessions will include approximately six (6) to twenty (20) items per subject.
Review and development of the student tutorials/practice sessions must follow the Departments
procedures for forms review, outlined in Section 7.3. In addition, the Department reserves the right to
request updates to the practice sessions to reflect any changes in CBT platform functionality or
necessary changes to the practice questions throughout the life of the contract. The contractor must
anticipate that these updates may occur and must be prepared to provide these updates at no
additional cost. (These would not qualify as authors alterations defined in Section 7.7.9.3.)
Additional information about the computer-based format and practice tests is provided in Table
7.4.9.6.
Test Item Specifications
Respondents should propose a detailed plan for producing and reviewing test item specifications for
ELA/L and Mathematics grades and subjects in consultation with the Department and Florida
educators. The specifications govern the writing and review of passages, prompts, stimuli, and test
items and generally identify the item/task requirements and constraints for measuring each standard.
The publications may include introductory information and appendices as necessary to fully explain
how the tests and items are developed. The test item specifications may include clarification of the
standard being assessed, specific content and item limitations, and item formats by which each
standard will be assessed. Sample items may be included for each item type. Item writers are the
primary audience for these publications, but Florida educators also use these materials, so ADAcompliant PDF files will be posted to the Department website. These publications will not be
distributed in print. Respondents may propose approximate document lengths and document design.
In conjunction with the review of the test item specifications, an external review panel of Florida
educators will review the test design and philosophy contained in each manuscript and will make
recommendations for improvements to the item specifications for each assessment.
7.3.8 Production of Printed Test Materials
1. The selected contractor is responsible for producing all versions of the materials and products developed for
this project, including test forms.
2. The selected contractor is responsible for proofreading documents during all stages of production. The
contractor should propose well-defined procedures to ensure drafts are checked by computer for spelling and
34
proofed for complete accuracy by qualified proofreaders from the respondent's staff before they are sent to
the Department for review. Qualified persons within the respondent's organization and independent editors,
who will be responsible for proofreading, should be named in the reply and their resumes attached. The
respondent should include a provision for the cost of changes made during the final proof stage of production
as authors alterations to be authorized by the Department.
3. The selected contractor will design and implement a procedure for assessing the quality of printing. The
procedure must include on-site quality control plans to ensure 100% correct printing, sealing, and collating test
documents. In their replies, respondents should explain what arrangements they have reached with printers
to establish technology or procedures to prevent miscollating test and answer books. Printing subcontractors
that produce miscollated test and answer books may not be approved for subsequent test administrations.
Procedures for ensuring printing quality must include plans for proofing all materials before and after printing.
The selected contractor will be responsible for replacing misprinted or otherwise defective materials at the
contractors expense within a time period acceptable to the Department.
4. In addition to the quality of printing, the Department requires stringent test security procedures to be followed
during all stages of document production and printing. In their replies, respondents should explain the security
plan to be implemented while test documents are in production and at the printer. See Appendices F and I for
these restrictions.
5. If errors occur during production, the selected contractor will be responsible for providing to the Department
within 30 days of the event a root-cause analysis, providing details on the cause and steps for improvement.
7.3.9 Production of Computer-Delivered Products
1. The selected contractor is responsible for producing all versions of computer-delivered products developed for
this project, including test forms. These products include all information, text, and graphics that will be
presented to computer-based test users.
2. The Department recognizes that the development, review, and approval processes for computer presentation
of information and materials parallels similar work for printed tests and products.
3. The respondent should specify its suggested plan for computer-based-test production with the objective of
ensuring that industry-leading quality is attained. The plan for the test content review should include steps
required to develop computer-based tests and practice tests to be reviewed and revised as needed prior to
approval in conjunction with Department and external reviewers, as appropriate to the respondents proposed
solution.
4. The functionality of CBT forms (e.g., tools, reference sheets, navigation) must be checked carefully to ensure
that errors have not been introduced, and that the functionality remains consistent across forms and
administrations. The Department will provide a checklist that can be used as a starting point for the
functionality review, but the selected contractor must be prepared to devote significant time to thoroughly
check each form.
5. A final review round will include an end-to-end check of the product, including checks to ensure user access
to the product and accurate navigation and functionality. The respondent should include a provision for the
cost of changes made following approval of a computer-based-test form as authors alterations to be
authorized by the Department (see Section 7.7.9.3).
6. Qualified persons within the respondent's organization and independent editors, who will be responsible for
proofreading, should be named in the reply and their resumes attached.
7. The Department requires stringent test security procedures to be followed during all stages of document
production and printing. In their replies, respondents should explain the security plan to be implemented while
test documents are in production. See Appendix F for these restrictions.
35
8. If errors occur during production, the selected contractor will be responsible for providing a root-cause
analysis to the Department within 30 days following the event, providing details on the cause and steps
for improvement.
7.3.10 Production of Paper-Based and Computer-Based Test, Style Guide, and Braille Production Procedures
1. The selected contractor will be responsible for all aspects of production for the CBT system, secure webbased reporting site, and printed materials, including formatting, graphics, and key entry. For each
publication, the contractor will submit product specifications that identify printing plans, type size and
style, ink and paper color, paper quality, and layout. Printing examples that show type size and style will
be included. The contractor will provide copies of the final electronic files for products.
2. The selected contractor will also print any additional materials needed to implement the project, such as
transmittal memoranda, labels for packing and packing lists.
3. The selected contractor will be responsible for all aspects of production for publishing printed products,
including formatting, graphics, and key entry. For each publication, the contractor will submit printing
plans that identify type size and style, ink and paper color, paper quality, and layout. Printing examples
that show type size and style will be included. The contractor will present these plans annually. The
Department expects attractive, quality printed materials at reasonable cost. Guidelines for the quantities
of each product to be printed are indicated in Appendix E.
4. Editorial specifications must include editorial and stylistic considerations for the development of test
questions and the layout of test pages. The selected contractor is responsible for updating and revising
the specifications annually as style and layout requirements may change. The contractor will work with
the Department to identify the components of the specifications that need revision and will revise and print
copies for all users.
5. The selected contractor will submit plans that identify screen layouts and navigation elements, font style
and size requirements, background and foreground color palettes, and requirements for graphics. The
contractor will identify and propose a branding plan to identify and provide consistency for Florida
assessment components delivered via computer. The Department expects an attractive, high quality,
engaging, and functional presentation. The contractor will present this information, as well as the
proposed scheduling and process for review rounds, in a specifications document that addresses
computer-based presentations.
7.3.11 Produce Special Formats and Accommodations for Students with Disabilities
7.3.11.1 Paper-Based Accommodations
For all assessments, the selected contractor will provide Braille and large print versions for visually
impaired students. For all assessments, the selected contractor will also provide one-item-perpage/one-sided accommodated versions as unique accommodations. For test documents, the
selected contractor will create the electronic files in the correct format needed to produce these
special formats. Although multiple forms of the tests may be constructed for each administration,
only one form for each test will be converted to Braille, large print, and one-item-per-page versions.
Large print documents will be printed in a minimum of 18-point type on approved paper and bound
using agreed upon methods. Reformatting of documents may be necessary to meet these
specifications. The selected contractor is responsible for having Braille materials proofed by an
independent party that includes a certified Braille reader. The Department may also employ the
services of a Braille proofreader. See Appendices E and G for more details and specifications on
special formats of test documents.
1. Respondents should include proposed subcontract publishers of Braille and large print materials
in their response. This publisher will produce the accommodated versions of the test and
answer books and other documents at the contractors expense. Test administrator notes and
36
scripts to accompany each administrations Braille test versions will also be developed by the
contractor.
2. While most blind students in Florida require contracted Braille, there will be some students who
require uncontracted Braille. The selected contractor will be responsible for providing either
format based on district requests. Approximately 50 copies of the Braille versions and up to 400
copies of the large print version may be required for each grade level and subject area per
administration. Test administrator notes and scripts to accompany each administrations Braille
test versions will be provided by the contractor as a supplement to the manual, and will undergo
the same iterative review process as other printed materials.
3. If requested by districts, the selected contractor will also provide large print and/or Braille
versions of sections of the test administration manuals. These publications will be produced so
that they will be delivered to districts along with the shipment of the regular print versions of
these products as identified in Appendix E. A maximum of five (5) each per format may be
produced for each administration.
4. The selected contractor will maintain and update an internal document that identifies which
person, by position and name that will perform each task in the Braille materials process. Each
task must be described in detail and the document must be accessible online with other
downloadable materials.
5. On request, the selected contractor will develop and deliver one-item-per-page documents to
districts. Approximately 25 copies of the one-item-per-page version may be required for each
grade level per administration. The need for these specialized documents will be determined by
the Department through its unique accommodations requests process and the Department will
provide an approved list identifying students needing this accommodation to the contractor.
6. The one-item-per-page documents will be packed and shipped by the selected contractor for
identified students. These documents will be printed on 8 x 11 paper, three-hole punched,
and placed in a binder so that individual pages may be removed as necessary. Because these
are also used for students who require true black and white print, the selected contractor will
provide for the Departments approval a sample of the materials to be used that demonstrates
the paper quality and font/style. The document package will include the appropriate scannable
answer document for student responses and the contractor will include directions for
administration and management of this test version. The contractor will be responsible for
transcription of one-item-per-page student responses.
7. The respondents reply should describe the processing of large print, Braille, and one-item-perpage versions of answer documents, including handling and transcription during the scoring
process. All Braille, large print, and one-item-per-page materials require separate packaging but
must be shipped so that they arrive in districts at the same time as the materials for regular
students.
8. The contractor will also deliver paper-based regular print or large print Reading Passage
Booklets for computer-based Reading tests. The booklets, which contain reading passages but
do not contain test items, will be for use by eligible students who need to utilize handwritten
strategies in order to respond to test items on the computer. See Section 7.3.2.1.
7.3.11.2 Computer-Based Accommodations
Floridas computer-based tests must also be available in paper-based format as an accommodation
at all grades and subjects. The use of this format will be dictated by a students IEP or Section 504
plan. The Department anticipates that the number of students taking a paper-based test (PBT)
based upon allowable accommodations will not exceed five percent (5%) of the total population for
each grade and subject tested on computer, and the selected contractor will produce this quantity five
37
percent (5%) of paper-based tests. For these students, paper-based accommodations described in
Section 7.3.11.1 must be available.
The reply should address the capability of the computer-based system to provide the following
special accommodations for students and to handle any additional requirements related to the use of
these accommodations.
1. Screen Reader
The respondents reply should include a description of a screen reader for computer-based
tests. The respondent should propose a method for reading text within the computer-based
system that does not require additional, proprietary software installed on the users computer.
The screen reader should be able to read all text and/or text descriptions of graphics or pictures,
including graphics or pictures used as answer options. The system should also have the
capability to use modified text if necessary (e.g., reading an algebraic equation with grouping
symbols), and allow user control of the reading speed and volume. The screen reader should be
designed for use by both sighted and visually impaired students. The Department recognizes
that the contractor may have different types of screen readers for these two purposes, but this
difference should be transparent to the user. The Department must approve the usability and
voice of the screen reader. In addition, the contractor must ensure that the pronunciation of the
screen reader is accurate, consistent across forms and subject areas, and would be
recognizable to students. If the screen reader does not have accurate, recognizable
pronunciation, it must be modifiable to meet the Departments standards. In addition, the
contractor must check the pronunciation on every form to ensure it is correct, is tracking
properly, and is understandable prior to it being provided to the Department for review. The
Department prefers that the contractor adhere to the Departments screen reader specifications
to ensure consistent performance of the screen reader. However, the Department will review
screen readers that meet the intent of this section and the needs of the Department.
2. Increased Font Size (Large Print and Zoom)
The respondents reply should describe available options on the CBT system to enlarge font
size, graphics, and CBT interface (i.e., buttons, tools, cursor, scrollbar) for students with
disabilities and describe how these options are set for a student. See Section 7.4.9.6 for large
print and zoom size requirements.
3. Font and Background Color (Color Contrast)
The respondents reply should describe available options on the CBT system for changes to the
font color and background color in order to optimize the ability of students with visual
impairments to read the screen and describe how these options are set for a student. See Table
7.4.9.6 for the font/background color combinations.
4. Assistive /Adaptive Devices
The respondents reply should describe the ability of the CBT system to interface with
assistive/adaptive devices. The reply should include a description of specific devices that may
be used and known devices that are not supported, as well as any special requirements
necessary for use, and should describe how options for use are set for a student.
5. English-to-Heritage Language Translation Dictionary
The respondents reply should describe the provision on the CBT system of an on-screen
English-to-heritage language translation dictionary for English Language Learners, if this can be
made available. The accommodated student should be able to select a word and be presented
with a translation sensitive to the context of the text. The language translation dictionary should
be available in Spanish and Haitian Creole. The respondents reply should describe the highlevel tasks involved, note the general timelines required, including the major/essential
deliverables and services required.
38
6. English-to-Sign Language Translation Dictionary

The respondents reply may describe the provision on the CBT system of an on-screen Englishto-sign language translation dictionary, if this can be made available. The accommodated
student should be able to select a word and be presented with a translation sensitive to the
context of the text. The respondents reply should describe the high-level tasks involved, note
the general timelines required, including the major/essential deliverables and services required.
7.4.0 Test Administration
The contractor is responsible for all aspects of test administration described in this section.
7.4.1 Pack and Distribute Materials
1. The contractor is responsible for all arrangements and costs associated with packing, distributing, and
retrieving physical materials. There must be 100% accounting for all secure materials (e.g., test books,
test and answer books) distributed to and returned by districts using barcode labeling systems, as well as
for any physical media (e.g., CDs) used to transmit electronic files. The contractor must guarantee that
distribution procedures are accurate and make corrections to the system in the event of errors.
Respondents replies should include descriptions of the procedures they will use to complete these tasks.
Respondents should identify carriers they propose to use and the procedures for delivery and retrieval of
test materials. The Department must approve all carriers.
2. The contractor will prepare packaging and distribution (P&D) specifications for each test administration.
The contractor will use the information in the specifications to program packaging systems and train
personnel involved in the processes. The packaging specifications will include the contractors
procedures for packing and distributing materials to districts and retrieving materials from districts. The
specifications will include a description of how the materials are packaged, how quantities of materials will
be calculated, examples of packing and inventory lists for boxes sent to districts and schools, methods
used for distributing and retrieving materials, a description of the procedures used to inventory materials
as they are returned, as well as any special instructions or administration-specific information. The
contractor will also include samples of all memoranda required in the delivery/pickup process. For
reference, the Department will provide the contractor with the most recent packaging and distribution
specification document at the initial contract management meeting. Proof of Department approval for
each specification document will be evidenced through email confirmation of approval.
3. The number of districts and special schools that serve as districts may change slightly during the life of
the contract. Appendix C provides information about the number of districts, schools, and students that
will receive test materials. Respondents should anticipate shipping materials to district assessment
coordinators at approximately eighty (80) separate sites. Materials will be shipped to a central district
location and distributed to the schools by the district.
4. Several school districts have contracts for services with school sites that are not geographically located
within the school district itself. School district personnel supervise the sites and the district assessment
coordinator provides training for site-based test administrators related to the administration, including
security, packing, and shipping. To enable prompt receipt and return of test materials, the contractor may
be required to ship materials directly to and retrieve materials directly from these remote sites. The
Department will work with the district assessment coordinators and the contractor to identify and approve
the remote sites to which the contractor will be permitted to ship and retrieve materials without going
through the district office. For the purpose of preparing replies, respondents should expect that up to
twenty-five (25) such locations might be identified.
5. At the beginning of the contract, the Department will provide the contractor with a data file containing a list
of the districts and schools (names and identification numbers). The contractor will be responsible for
maintaining this list, updating it, and providing a copy of the data file to the Department at the end of the
contract or at other times within the contract period on request.
39
6. The contractor will collect preidentification information from districts for each administration via the online
management system and assume that preidentification will be used for 100% of the school population.
These files will be used to load student data into computer-based testing systems, as well as produce
barcode PreID labels and rosters for paper-based tests. The contractor will establish a system to ensure
that the labels/documents delivered to districts contain all requested data, reflect the options selected by
districts, are accurate, and are printed at a level of quality that permits accurate scanning and precludes
the possibility of smudging, smearing, and/or flaking. Contractors will provide quality control throughout
the printing process to ensure the quality of label/document printing. The contractor will provide a
separate roster of students using the same sort order used to prepare the student labels/documents. This
roster will be used by districts and schools to verify the accuracy of certain information that must be
correct before the label/document can be used.
7. A list of the primary materials to be shipped for each administration, the quantities to be packaged for
schools, districts, and the Department, and other packaging specifications is provided in Appendix E.
8. The contractors specifications for packing, distributing, and receiving materials will include the following
provisions:
a. The contractor will shrink-wrap materials in quantities specified by the Department. Some materials
will not be shrink-wrapped (e.g., Test Administration Manuals). All secure materials will be shrinkwrapped.
b. Schools within a district will be ordered by school number on all lists and for shipping purposes,
unless the district has requested that materials be sorted in route code order, in which case the
materials will be sorted by route code, by school number.
c. The contractor will package and ship materials by district size as determined by the Department,
beginning with the largest district. The largest district will receive materials at the beginning of the
delivery window; the smallest district will receive materials last. District rank by size information will
be updated annually by the Department.
d. The contractor will label the boxes of test materials with the message TO BE OPENED BY
ASSESSMENT COORDINATOR ONLY. District boxes will be differentiated from school boxes by
color (e.g., white district boxes and brown school boxes). Boxes containing the district/school
packing lists, materials return kits, and ancillary materials must appear different (e.g., different color,
colored stripe) from all other district and school boxes for easy identification. School boxes will be
labeled with the name/number of the district and name/number of the school in at least a 16 point
font. Label design must be approved by the Department. Only boxes directed to the district-level
staff, such as boxes containing district overage, will be labeled with the name of the district
assessment coordinator. The contractor will label district boxes (that contain overage and other
materials for the district) on all four sides and the top, and number boxes as Box 1 of X, where X is
the total number of boxes sent to that district. The contractor will label school boxes on all four sides
and the top, and number boxes as Box 1 of X, where X is the total number of boxes sent to that
school. If materials for more than one administration or type of material are shipped during the same
window, the contractor must have a way to distinguish these shipments from each other (e.g., colorcoded label on boxes). Boxes containing secure test materials must never indicate secure content
on the outside of the boxes.
e. For districts receiving more than one pallet in a shipment, a map listing the identity and location of
boxes on each pallet will be provided electronically and will be provided at the time of delivery as a
hard copy to the district assessment coordinator to assist with distribution of boxes in the district.
f.
The contractor must have a method to collect route code information from districts and package
materials accordingly. If district assessment coordinators provide internal district delivery route code
40
information, the contractor will use this information to build pallets for districts to match district
delivery route codes.
g. The contractor will pay charges on all materials shipped to and from each district. The contractor will
make arrangements for and pay for shipment if, due to a delivery error, a district is asked to ship
materials to another district. The contractor must use an overnight delivery service for such
shipments. The contractor is not responsible for any costs schools may incur in shipping test
materials from their schools to the district office, unless the need to ship is the result of a packaging
error by the contractor. The contractor must also have a method for districts to order additional test
and ancillary materials, as needed, in the online management system. The contractor must be aware
that requests for additional materials are often critical and such shipments must be shipped overnight
for early-morning delivery, on request.
h. The contractor must receive Department approval before shipping materials to districts. Approval will
be provided after the Department has received and proofed examples of printed products and
reviewed sample shipments, as described in the packaging and distribution specifications. Approval
is sent via an Approval to Ship form.
i.
Secure materials will be packaged sequentially in ascending barcode order (for example, if a school
receives 200 secure documents and 100 documents fit in each box, Box 1 will contain the documents
with barcode numbers 1-100, and the document with barcode #1 will be at the top of the documents
contained in Box 1 and the document with barcode #100 will be at the bottom of the documents
contained in Box 1. Box 2 will contain the documents with barcode numbers 101-200, and the
document with barcode #101 will be at the top of the documents contained in Box 2 and the
document with barcode #200 will be at the bottom of the documents contained in Box 2.). Packing
lists must list all box contents in order as they are packaged with the first item on the list being the
first item (on top) in the first box.
j.
The contractor will package materials for each shipment by school in returnable boxes (doublewalled) suitable for multiple uses. The contractor will provide additional boxes statewide for the
return of materials that are available as an additional order for districts.
k. No box will be packed to weigh more than 25 pounds, and boxes will be of the dimensions, or similar
to, those indicated in Table 7.4.1. If test materials require different box dimensions than those listed
below, the specifications for boxes will be approved by the Department during the negotiation phase.
Table 7.4.1 Box Dimensions*
Table 7.4.1 Box Dimensions*
BOX SIZE
MATERIALS
9.5 x 12 x 13, Double-walled
Regular Print
9.5 x 12 x 6.5, Double-walled
Regular Print
19 x 15 x 4.5
Large Print/Braille
19 x 15 x 6
Large Print/Braille
19 x 15 x 9
Large Print/Braille
* Box height is dependent upon the quantity of materials in each shipment.
l.
Upon completion of the delivery of each shipment, the contractor will provide proof of delivery to the
Department indicating, but not limited to, a description of the materials, date/time of delivery, and the
name of the staff member accepting the delivery. Proof of 100% delivery will be used for invoicing
purposes.
41
m. The contractor will provide fifteen percent (15%) materials overage to each district for paper-based
test materials and computer-based ancillary materials (work folders, etc.). The Department and
districts will decide how materials overage will be distributed to districts and schools for each
shipment. Districts may select from four (4) distinct algorithms for overage distribution:
All fifteen percent (15%) overage to the district
All fifteen percent (15%) overage to the school
Ten percent (10%) overage to the district and five percent (5%) overage to the school
Five percent (5%) overage to the district and ten percent (10%) overage to the school
n. The contractor must anticipate that some districts will require an additional pickup after the specified
pickup date. This may occur in up to 15 districts for as many as 500 boxes for each spring
administration, and in up to 10 districts for as many as 200 boxes for each retake and End-of-Course
administration.
o. Large-print, Braille, and one-item-per-page materials are to be packaged and labeled separately and
delivered to districts during the same delivery window as the shipment of other test materials in Test
Materials Shipment 2 (TM2).
p. Preidentification labels and rosters will be packaged with test materials (included in TM2). If
applicable, Wave 2 labels will be shipped separately.
q. It may be necessary for the contractor to ship test materials to districts/schools and the Department in
multiple shipments per testing window, not including requests for additional materials. The number of
shipments may vary depending on the administration. The respondent should refer to the
approximate shipment dates provided in Appendix B. The contractor must be prepared to deliver
some materials on alternate schedules per district request (e.g., to avoid conflicts with spring break or
closures) and must have a method to effectively track these requests and ensure that shipping
companies meet the revised dates. Shipping companies must never contact districts directly to
arrange alternate delivery/pickup dates unless agreed upon by the Department.
r.
The contractor will prepay charges on return shipments from districts. Return labels, prepaid postage
labels, instructions, and other materials needed for return will be provided in a Materials Return Kit
shipped in TM2.
s. Districts will return answer documents using overnight delivery, if deemed necessary to meet
reporting deadlines. Materials will be returned according to a pickup schedule developed by the
Department and the contractor that allows schools enough time to package materials and also meets
scoring and calibration deadlines.
t.
The contractor will be responsible for mailing or shipping any miscellaneous materials to the
Department and districts by overnight delivery service or other means, as requested. The contractor
is required to secure the services of shippers who will provide inside delivery and unload large
shipments onto loading docks. All carrier vehicles must display the carrier name, and districts must
be made aware of what carrier name to expect prior to the pickup or delivery date. Districts may or
may not have a loading dock or the appropriate equipment to support the delivery and pickup of
materials (e.g., shrink-wrap; hand-trucks). In this event, the contractors shipper must be prepared to
provide such services/materials. District staff is not expected to assist in loading or unloading
materials, but will be available to provide instructions to the carrier. Carriers must communicate with
district staff face-to-face when delivering or picking up materials, never leave materials unattended,
and be prepared to protect materials in the event of inclement weather.
u. The contractor must develop procedures to monitor the receipt of all materials and develop error logs.
The date materials are received and any errors made by districts in packaging and completing forms
42
must be documented. When problems arise, the contractor must contact districts and the Department
to resolve them. Error logs will be identified by school and district, and will document all failures to
follow the established procedures and, if appropriate, how the errors were resolved. The error logs
will be delivered to the Department within three weeks after the scheduled receipt of materials from
all districts.
v. The contractor will use a barcode numbering system to verify all secure materials returned after
testing. The contractor will provide a written report to the Department documenting the check-in of all
secure materials. f the contractors system for barcode verification is demonstrated to be unreliable
(e.g., materials not scanned properly), each document will be required to be scanned twice, creating
independent data files that can be matched for scanning errors. Beyond these measures, the
contractor will be required to manually check answer documents to ensure 100% accuracy of secure
document check-in.
w. The contractor will include in its check-in procedures a method of checking for and retrieving used
answer documents erroneously packaged with used and unused materials, invalidated materials, or
test books. These orphan answer documents will be processed and scored under extended
timelines and individual reports will be generated and shipped to the districts.
x. The contractor will produce statistical reports summarizing responses to the online test administrator,
school coordinator, technology coordinator, and district coordinator comment forms. A summary,
analysis, and list of recommended actions derived from the comment form responses, as well as a
data file containing all responses, will be delivered to the Department no later than two (2) months
after completion each administration. Contractor staff will be responsible for summarizing the openended responses received from district and school staff. This summary must be a categorized report,
not simply a list of open-ended responses.
7.4.2 Missing Materials Report and Inventory
The contractor will prepare a missing materials report for all secure materials, based on the scanning that is
completed during materials check-in. The report must be produced after all materials have been scanned,
including as many orphan and late-returned documents as possible, and must not reflect materials that are not
missing (e.g., contractor has verified against the soiled/destroyed documents list). The contractor must
implement quality control measures to ensure the accuracy of the reports. The contractor will produce a
preliminary missing materials report for Department review. The Department will review and approve the
preliminary missing materials reports for distribution to districts. The Department requires ten working days to
review preliminary reports prior to approval. Reports will be prepared for each affected school listing the title
of the missing products and the corresponding barcode numbers. School reports will be delivered to district
assessment coordinators with a district-level summary and memorandum, and a state-level summary report
will be delivered to the Department. Any missing materials returned by districts will be recorded in the missing
materials inventory maintained by the contractor. The contractor will deliver a final summary report of missing
materials to the Department by the date indicated in the Deliverables Schedule.
7.4.3 Disposition and Storage of Materials
After testing, the contractor will inventory and store unused test and answer books, manuals, and other
materials for a period of at least six (6) months. During this time, the contractor will be required to ship
quantities of these materials, as necessary, to the Department and districts.
Replies should include a plan for secure destruction of excess test materials that requires Department
approval. However, the contractor will store 100 copies of each test and answer book for each administration
throughout the life of the project. In addition, the contractor will store some digital record of the CBT
assessments as they appeared to the students for that administration. Any materials that may be used in
subsequent assessments will be stored by the contractor throughout the life of the project. Additionally, six (6)
months after the contract ends, the subcontractors and print vendors will be required to destroy all electronic
files and used print copies according to the test security requirements approved by the Department.
43
On verification of the individual barcode numbers of all secure materials returned by the districts and
acceptance by the Department of accurate results files, the contractor will store used documents containing
student responses, at contractor expense, for a period of two (2) years. Likewise, all student CBT response
files must be electronically archived for a two-year period. At the end of the two-year period, the contractor
will ship or destroy the materials according to instructions from the Department. The contractor must submit a
letter to the Department requesting permission to destroy specific materials. This process is already welldefined by the Department and will be shared at the initial contract management meeting. Test security
requirements will be maintained throughout the destruction process. It may be necessary to retain a small
number of used documents for a longer time period due to Department security investigations. At the end of
the contract, the contractor will be expected to ship these materials to the new contractor.
Destruction of secure documents and CBT student response files must be requested in writing and authorized
by the Department. The contractor and all subcontractors must submit certificates of destruction that describe
in writing the specific items/files destroyed.
7.4.4 Retrieval of Answer Sheets and Test Books
In addition to the document retrieval specified in Section 7.4.1, the contractor will periodically be required to
retrieve up to 1200 used test and answer books per grade, per administration from storage. Likewise,
individual student response files from the computer-based testing system for up to 1200 students per grade,
per subject, per administration may need to be retrieved. The contractor must maintain information related to
warehouse location for each schools used materials to allow for retrieval as may be required for Department
investigations or audits. In some cases, retrieval of answer documents will require the contractor to conduct a
manual verification of machine scoring. The contractor will be responsible for costs associated with retrieval,
review, and possible delivery of these materials to the Department. In addition, if errors are found, the
contractor may be required to re-score and re-report these documents.
7.4.5 Innovative Product Design
New, innovative designs are required for all Florida-specific products developed by the contractor. Designs for
publications will include the organization, format, and page or screen layouts for administration manuals,
reports of assessment results, information publications, and other materials. Designs will also be produced for
other products, as appropriate. The contractor will produce all graphics, charts, and illustrations for the
products for which it is responsible and share designs and graphics with other contractors, as required by the
Department. The design process will include furnishing conceptual art, design development, and final cover
designs. The contractor will produce all graphics, charts, and illustrations and will secure the full range of
copyrights and/or permissions that allow for production in print, secure web, and unsecure web format.
The contractor must arrange for personnel with professional expertise and experience in the areas of
visual/graphic arts and document/software/web design to work on the design and development of documents
and computer/online interfaces. If the contractor does not have qualified, professional personnel devoted
primarily to visual/graphic arts and document/web design on staff, a subcontractor or subcontractors must be
identified to provide these design services. Respondents should indicate in their replies the qualified
personnel/subcontractors who will provide these professional services and include their vitae or other
documentation of expertise and experience in the reply.
7.4.6 Printed Products Administration
This section provides guidelines, including descriptions of and duration of review rounds, for all printed
products required for test administration services in this ITN. Contractors must provide high-quality, carefully
edited materials for each round of review by the Department.
7.4.6.1
Stages of Development of Printed Products Administration

The contractor is responsible for producing the first written draft and all subsequent versions of all
materials and products developed for this project, and for submitting documents for Department
review. The review period for the Department is seven (7) work days, but longer review times will be
required if multiple documents are under review simultaneously. For each review round, the contractor
44
will post an electronic version (PDF) by 9:00 a.m. ET on the first scheduled day of Department review.
If materials are delivered or posted after 9:00 a.m., the review schedule will be revised accordingly.
Documents for CBT-related materials will be posted to a secure website for review in their presentation
format.
The delivery of administration materials is a critical task and workflow must be planned carefully. The
overlap of reviews could create a burden on both the Department and contractor staff. Excessive
burdens can be avoided with sufficient staffing, advanced planning, and an early start. The contractor
must propose a detailed materials development schedule for managing the workflow and volume of
documents to be reviewed in each seven (7) day period in a format approved by the Department. The
contractor must develop a schedule for the production of test administration manuals that follows the
review schedule of test books/answer documents or computer-based test forms by one round. The
schedule must consider what is reasonable given the Departments staffing constraints. All proposed
schedules must be approved by the Department. The number of documents to be reviewed in any
seven (7) day period must be negotiated with and approved by the Department. The Department is
committed to reviewing products submitted by the contractor as efficiently as possible.
The contractor is responsible for proofreading all documents during the production stages. The
contractor will propose well-defined procedures to ensure that all drafts are checked by computer for
spelling and proofed for complete accuracy by qualified proofreaders before they are sent to the
Department for review. The contractor will submit a written report by the proofreaders indicating
proposed results and actions. Before moving to digital proof stage, the Department will review and
approve the proofreaders report. Respondents should indicate in their replies the qualified
personnel/subcontractors/independent editors who will be responsible for proofreading and include
their vitae or other documentation of expertise and experience in the reply.
Test Administration Manuals
For respondents proposing solutions derived from other products, the Department understands that
certain test administrations must be standardized, however, there are certain Florida requirements
that must be met. Replies from such respondents should indicate which aspects of test
administration are considered customizable, and a final determination will be agreed upon during
the negotiation phase. For instance, the Department may require the production of test
administration manuals specifically for Florida that include the standard information, or the
production of Florida-specific supplementary publications that complement the contractors more
generic publications. The following information details the Departments expectations for Floridaspecific stand-alone products.
For each year of the contract (Fall Summer administrations), the following test administration
manuals will be produced: One (1) ELA/L and Mathematics manual for paper-based tests; One (1)
ELA/L and Mathematics manual for computer-based tests; One (1) ELA/L Retake manual (beginning
with the first retake administration in 2015); (1) End-of-Course manual. Administration requirements
and scripts for all assessed grade levels/subjects will be included in the manuals. Instructions for
Special Format Tests will include scripts necessary for accommodated testing (e.g., Braille, one-itemper-page, and others) and will be provided as separate addenda to the manuals. Computer-based
manuals will also include scripts for the practice tests necessary to familiarize students with the
computer-based testing system. All manuals are published in print and on the contractors website.
Minimally, manuals will contain instructions and scripts for administering each portion of the
assessments for the appropriate grade levels for that administration. They also include information
about security of materials, packing and returning materials, receipt and distribution of materials, and
the Departments test administration and security policies and procedures, which are subject to change
with each administration. Revisions will be made as necessary to reflect program changes.
45
The contractor must plan to schedule production of test administration manual covers as separate
mini-deliverables that will eventually be married to the final products. Each cover requires multiple
review rounds by graphics designers, editors, and content specialists, and careful planning is required
to meet production deadlines.
Test Administration Manuals are to be packaged (no shrink wrap) and delivered to districts for each
administration as a part of TM1 according to the specifications in Appendix E, Part 3. All manuals
must also be available in a Section-508 compliant (see Appendix E, Section 2) electronic format on
both the Departments and contractors websites within seven (7) days of Departmental approval of
the web-tagged files. The suggested workflow for the Department and the contractor is provided in
Appendix G to help the respondent anticipate the level of work required on the review rounds.
7.4.7 Test Administration Ancillary Materials
For the purpose of this ITN, test administration ancillary materials refers to printed or online products other
than manuals produced by the contractor. Ancillary materials are developed with review rounds and email
approval. Some ancillary materials will be printed and shipped with test materials (e.g., document count
forms, return labels, paper bands). Some will be included as perforated pages in the manuals (e.g., security
agreements, chain of custody form, blank administration record/security checklist, security log). Some will be
developed and provided in an online format only (e.g., comment forms).
This section contains details regarding the ancillary materials required for administration. These materials are
designed to ensure overall security, efficient and secure handling of materials, and standardized
administration activities. In addition to the materials listed here, the contractor will provide, based on
Department request and approval, a number of memoranda to assist in communications with districts
regarding test administration processes and requirements.
The contractor will provide the required publication production and software development resources to
produce the ancillary materials using current staff that have the necessary expertise and serve in such
capacities on a full-time basis, through temporary staffing, or by subcontracting. Respondents should identify
in their replies the staff who will be assigned to develop the ancillary materials and/or subcontractors proposed
to complete these project tasks, as well as their qualifications (education and experience) to do so.
Most of the products identified in this section will be provided in formats that permit them to be accessed
electronically via the internet on both the contractors and Departments websites. Posted products must be
available for viewing and downloading and must be provided in an ADA-compliant format, that is, a format
compliant with Section 508 [29 U.S.C. 794(d) & 36 CFR Part 1194] of the Rehabilitation Act, as amended,
and s. 282.601, et seq., F.S. The Department will run standard Section 508-compliance checks on all
documents to be posted on the Department website to ensure their accessibility. The contractor will be
required to modify any noncompliant elements and provide compliant documents. The contractor is
responsible for securing all copyright requirements, if applicable, for materials submitted for nonsecure web
posting. (See Appendix E, Part 2 for additional information about Department guidelines for Section 508 pdf
publications.)
The contractor will work with the Department to identify the specific requirements for files to be made
accessible on the online management system and will provide the required files in a timely manner.
The contractor is responsible for developing the following materials for distribution to districts for each
administration. These materials will be produced following the quantities and specifications identified in
Appendix E. The contractor is also required to provide various other materials required to implement
administration activities, including forms, labels, and documents.
7.4.7.1
Training Resources for School and District Technology Coordinators (CBT)

The contractor must provide training materials to support CBT test administrations. These
materials may include screen-shots and will be delivered via narrated PowerPoint web
presentations, WebEx, training module, or other similar mode through the CBT test platform.
46
These training materials will consist of an outline of key issues to be covered during school and
district technology coordinator training with explanations accompanied by full-color graphics,
including screen-shots, representing critical elements or live demonstrations of processes in the
online management system.
All training materials are to be made available to districts in a Section-508 compliant electronic
format on both the Departments and contractors website within seven (7) days of Departmental
final approval of the web-tagged files.
7.4.7.2
Instructions and Materials for Braille Tests

The contractor must develop Braille scripts and Braille notes for the administration of Braille tests
for each subject and grade. The braille notes are secure materials and must be handled
accordingly. The Braille notes and Braille scripts are distributed with the secure test materials, and
they must be approved by the Department prior to production. In addition, some items may require
manipulatives to allow access for visually impaired students. These manipulatives are provided,
prepared, and shipped to the districts by the contractor.
7.4.7.3
Student Preidentification Labels

The contractor will produce student preidentification labels for each student to be tested on a
paper-based assessment for whom preidentification data were provided by the districts. (See
Section 7.5.2 for more information about the preidentification process.) Labels will be provided for
each student answer document as needed. Local school staff will be responsible for placing labels
on student answer documents.
One label for each answer document not already preprinted
Sorted as specified by each district (may be different for each grade/subject and different for
schools within a district)
Sorted with page breaks as specified by the district/Department
3 x 3 adhesive labels or approved equivalent
A supplementary supply of labels will be produced for spring administrations (Wave 2) based
on PreID information gathered in January
7.4.7.4
Student PreID Rosters

The contractor will provide a preidentification list containing all information included on the student
PreID labels for each school in a format approved by the Department. The list is produced from
the same file used to print student PreID labels and contains the same information. (See Section
7.5.2 for more information about the preidentification process.) One row of information per student
will be used to verify the accuracy of student demographic information. Rosters will be provided in
hard copies to schools and electronically via the contractors secure website. 8 x 11 paper is
preferred, but the Department may approve 8 x 14 if this is required for readability.
7.4.7.5
Document Count Form (DCF)

The contractor will provide Document Count Forms (DCF). The DCF is a pregridded, scannable, 8
X 11 document used to inventory answer documents returned by each school. The form
specifies the subject and grade level test placed underneath it and ensures that the correct number
of documents is scanned. Forms must be preidentified with district/school names and numbers,
and blank forms will also be sent as this information may also be hand-gridded. The district/school
number should match the numbers on the answer documents underneath them, and the number of
documents gridded on the DCF should match the number of documents beneath it. If
discrepancies are discovered during scanning, the contractor will work to resolve them, as
described in the Scoring and Reporting Specifications.
47
If the respondent employs a different acceptable method/form for the process described above, the
Department and the contractor will agree on the final process during the negotiation phase.
7.4.7.6
Administration Record/Security Checklist

Security barcode numbers of all secure materials listed in ascending order by product, will be
provided to a school/district with space to indicate assignment and return of materials. The
Department expects that the checklist will also include box numbers and section breaks between
packages and will be provided electronically in an editable format to districts.
If the respondent employs a different acceptable method/form for the process described above, the
Department and the contractor will agree on the final process during the negotiation phase.
7.4.7.7
Comment Forms
The contractor will work with the Department to create the following ADA-compliant online forms
that will be posted to the contractors website for each administration:
Test Administrator Comment Form (One for paper-based and one for computer-based)
School Coordinator Comment Form (One for paper-based and one for computer-based)
District Coordinator Comment Form (One for paper-based and one for computer-based)
Technology Coordinator Form (CBT only)
CBT Student Comment Form
The final output of the comment form responses is described in #26 under Section 7.4.1.
7.4.7.8
Return Shipping Labels

The contractor will provide preprinted, adhesive, color-coded (as required by the Department)
labels for district use in returning materials.
7.4.7.9
Materials Return Kit District

The contractor will provide envelopes containing return shipping labels, bills of lading, return
instructions memorandum, and Braille notes and scripts, if applicable.
7.4.7.10
Materials Return Kit School

The contractor will provide envelopes containing document count forms, paper bands, and colorcoded return shipping labels.
7.4.7.11
Miscellaneous Return Materials

The contractor will provide envelopes and boxes necessary for packaging large-print, Braille, and
other special program and special format materials for district use in returning materials.
7.4.7.12
Paper Bands
Printed paper bands, preferably 4 x 24 adhesive, will be provided by the contractor for bundling
scorable materials and provided to schools and districts. The adhesive must not cover the entire
width of the band to avoid adhering to the documents contained in the band. The bands must be
printed on one side to identify the type of answer document in bundle. Bands must be 8pt.
Carolina paper or approved equivalent. In addition, paper band quality must be suitable for filling in
information lines on band with multiple pen/pencil types (e.g., markers, ball point pens, pencils)
without smearing/smudging.
If the respondent employs a different acceptable method for the process described above, the
Department and the selected contractor will agree on the final process during the negotiation
phase.
48
7.4.7.13
Rulers / Manipulatives / Calculators

If any test items require the use of a ruler or manipulative, the contractor will produce/provide these
materials for testing in a reliable, cost-efficient manner. Respondents should describe proposed
functionality and features of calculators required for Mathematics tests and the Department will
provide an approved list of calculators for student use. (Contractors will not be required to provide
hand-held calculators to districts.) Any testing accessories required for computer-based tests must
be built into the platform and must also be available for paper-based tests. If materials not listed in
this ITN are needed for testing, they should be clearly identified in the respondents reply and the
Department will specify its requirements during the negotiation phase.
7.4.7.14
Miscellaneous Memos, Forms, Labels, and Other Products

The contractor will provide the following products:
Miscellaneous memoranda to the district coordinators may be necessary for each

administration. Letters and memoranda will be of a length to be determined by the
Department, up to five (5) pages each, and length of documents may change at each
production round.
Cover memoranda for all shipments of materials indicating what is being shipped and how it is
packed.
Packing lists must be provided with all shipments (district coordinator receives a hard copy of
the district and all school packing lists and receives all lists electronically as soon as they are
available).
Pallet lists (maps) must be provided to any district that has materials on pallets.
Other miscellaneous memoranda, labels, forms, etc., will be required to implement the
program.
7.4.8 Online Management System

The contractors online management system described in this section will be the primary resource for all
student data management (paper-based and computer-based), as well as test administration and reporting
tasks. The expectations of this ITN are written with the full understanding that capabilities for an online
management system are expanding quickly. The Department is interested in reviewing innovative procedures
and techniques for managing student assessment online, which take advantage of the maturation of
technology in educational assessment through the life of the contract arising from this ITN. Therefore, the
respondent may submit a reply that varies from the expectations herein as long as the solution meets the
needs of the Department and the intent of the ITN. Respondents should agree to work with the Department to
amend the specifications, expectations, and requirements to ensure that the Florida testing program keeps
pace with the expanding capabilities of test delivery and management via computer. Any such variations will
be resolved and agreed to during the negotiation phase.
The online management system should be a secure site that requires usernames and passwords and
accounts should be customizable based on the access a user will have (state-level, district-level, school-level,
etc.). Depending on the access level, a user should be able to view other organizations in the system (i.e., a
state-level user may select and view any district or school in the state) and have the ability to make changes
or alter data in those organizations.
The Department will provide the contractor with a list of the current names, addresses, email addresses, and
phone and fax numbers of the district assessment coordinators. The contractor will populate the passwordprotected online system with this information to create initial district-level user accounts. New accounts should
be able to be created individually or via file upload to create multiple new accounts or make updates to
existing accounts.
49
The system should allow districts to upload enrollment counts, preidentification files, and to place additional
orders for materials. Using the Departments Master School Identification (MSID) file, the contractor will
populate the system with all active districts and schools. The system must have the ability to add additional
schools that are not on the MSID file (e.g., private schools), and these schools may require special school
types as determined by the department.
Components of the contractors online systems (e.g., websites, databases) must be submitted to the
Department for review of content, layout, aesthetic quality, and functionality. The contractor will make any
requested changes to such systems. The online systems may not be used for contractor advertisements or
other promotional purposes. The branding of these systems should be primarily for Floridas assessment
system, but may include contractor logos and other information as approved by the Department.
The system should also accommodate CBT delivery. All systems that identify, prepare for, and monitor
student test participation must have the capacity to distinguish between paper-based and computer-based
tests, including all applications of preidentification and enrollment files. These systems must also be able to
accurately track student participation by delivery mode (paper-based, computer-based, or both) and reflect the
test delivery mode(s) in each student file.
The system should be designed to provide user access to various non-secure and secure links including:
general system introduction and descriptive information;

updates to current operational status and special situations;
self-running software to test the suitability of computers for test delivery;
self-running software to download a secure browser to ensure test and system security;
all user guides, manuals, training, and ancillary materials in downloadable format;
procedures to add, modify, or remove students or student information in preparation for testing or
reporting;
record test group codes and accommodations for computer-based tests;
ability to invalidate computer-based tests;
procedures for school and district staff to update contact information;
access to an online help-desk chat system;
system status website to display operability status of testing components;
access to instances of all computer-delivered tests serving Florida K-12 education programs;
access to order additional test materials and track shipments;
access to packing lists, pallet maps, security checklists (pre-populated with security numbers of secure
materials), and material summaries;
results files and reports;
web-based student practice tests;
self-paced training resources for test setup and administration tasks;
Training center to enable practice for test setup tasks and to run an infrastructure test to test capacity at
testing sites; and an
infrastructure test.
The system must collect and maintain records for students as they move through Floridas public school
systems. An individual student's test results must be maintained in electronic archive for the life of the
contract and must be returned to the Department at contract termination.
50
7.4.8.1
Test Staff Contact Information

It is necessary for the contractor to maintain accurate lists of district and school information and
staff associated with Florida public education testing programs as specified by the Department.
This information must be easy to update by school staff as information changes. This information
must include each users security authorization level and provide an accessible log, in viewable
and printable format, of each users participation in the system, including type of participation, date
and time of participation, etc. (see also Section 7.4.1).
7.4.8.2
Test Staff Passwords

The system must securely deliver and maintain system access passwords to users as necessary.
The passwords must access a hierarchical permissions structure available for school, district, and
state users. The contractor will work with the Department to determine expiration timelines for
passwords and user accounts.
7.4.8.3
Test Session Requirements

The system must have available during all steps in initiating and delivering tests:
a school look-up sub-system;
a searchable, sortable student look-up sub-system;
rosters of students in each test session.
During the test session, automated backup, recovery, and restart procedures for the system must
be in place. The system will recover data from any unforeseen test interruption and return the testtaker to the point of interruption. The system must also show student status and progress through
the items in the test. The Department should be able to view uploaded student responses realtime while students are completing the assessments in the event of an unforeseen test interruption.
Contractor and department users must have the ability to restart a submitted test. The tests should
only be available during specified administration windows. The features and interface of the
system may be customizable, and the contractor will consider requests for customization from the
Department.
7.4.8.4
Security Requirements
Security of tests and individual test results must be maintained. Access to student test results
must be restricted to authorized Department staff and school district personnel based on a secure,
encrypted, password protected system.
The system must assure proper identification of each student and the accurate matching of the
student to his/her test results. Preferably, the system must ensure that a student can take each
computer-based test just once per login unless authorized by the department. If an error is found
(e.g., student tests with the wrong login information) the system must have the ability to link the test
to the correct student profile.
The ability to make changes to data or processes in any part of the system must be based on a
password-protected, hierarchical permissions structure.
Respondents should respond to each of the following items:
Describe the security controls over all system aspects.
Describe the levels of security provided in the system.
Describe how the ability to limit access to specific system functions or modules is provided.
Describe the log-in security method and assurances to maintain individual confidentiality of
test results.
Describe the authentication process to ensure that an individual is the person logging into the
system (e.g., the student taking the test is who s/he says s/he is).
51
The contractor and Department staff will create a report that includes a checklist of all operable
aspects of the system to be used each invoice period. The contractor will prepare the report listing
any malfunctions or unavailability of the system for Department signoff each period.
7.4.9 Computer-Based Test Administration Overview
The expectations in this ITN are written with the full understanding that computer-based testing (CBT)
capabilities are expanding quickly. The Department is interested in reviewing innovative procedures and
techniques for item development, test delivery, scoring (see Section 7.5.4), and reporting (see Section 7.5.5)
which take advantage of the maturation of technology in educational assessment through the life of the
contract arising from this ITN. Therefore, the respondent may submit a reply that varies from the expectations
herein as long as the solution meets the needs of the Department and the intent of the ITN. Respondents
should agree to work with the Department to amend the specifications, expectations, and requirements to
ensure that the Florida testing program keeps pace with the expanding capabilities of test delivery and
management via computer. Any such variations will be resolved and agreed to during the negotiation phase.
The computer-based test system should be capable of being customized to meet the needs of Florida
students, school and district personnel, and the Department. The reply should indicate procedures used to
ensure that the computer-based tests accurately assess students possessing a diverse range of physical,
sensory, and cognitive abilities; the system should ensure adherence to universal design concepts. The
system should be easy to use for all participants, intuitive, and deliver a consistent look and feel for all
computer-delivered test programs provided by the Department. This includes item layouts, navigation tools,
and administrative controls such as managing participant demographic and performance information, as well
as CBT setup activities. The contractor will modify any proposed system to achieve these objectives.
The contractor and Department staff will create a report that includes a checklist of all operable aspects of the
system to be used each invoice period. The contractor will prepare the report listing any malfunctions or
unavailability of the system for Department signoff each period.
If the Department determines that the computer-based system is not performing adequately, exhibits
malfunctions that may significantly affect the student testing experience, or is unable to meet the requirements
listed in this ITN, the contractor will propose a solution to the Department, including providing paper-based
tests at no additional cost, that allows for students to participate in the assessments and for the Department to
gather the necessary data.
7.4.9.1
Hardware and Software Requirements

The system should:
Support both wired and wireless connections to the system and all eligible devices that meet
the specifications in the table in Section 7.4.9.2. The respondent should also commit to
continued support for future updates similar to those made available on the latest version of
Partnership for Assessment of Readiness for College and Careers' (PARCC) Technology
Guidelines that can be found at (http://www.parcconline.org/technology). Other assistive
technologies may be needed for students requiring accommodations.
Support eligible devices with capabilities to lock down the device to temporarily disable
features, functionalities, and applications that could present a security risk during test
administration. Features that will need to be controlled during test administration include, but
are not limited to, unrestricted Internet access, cameras (still and video), screen capture (live
and recorded), email, instant messaging, Bluetooth connections, application switching, and
printing. The contractor will develop, deliver, and continuously improve capabilities to lock
down the device for the applicable operating systems and browsers.
52
Support the current version within six months of release and two prior versions of the web
browsers listed below (as of June 2013).
Web Browser Support List:
Apple Safari
Google Chrome
Microsoft Internet Explorer
Mozilla Firefox
7.4.9.2
Eligible Device Specifications

At the time of the development of this ITN, the device specifications that are recommended for
Floridas public schools are provided below. The respondents platform for computer-based testing
should be able to perform, at a minimum, on devices that meet the specifications in Tables
7.4.9.2.A and 7.4.9.2.B.
Table 7.4.9.2.A. Device Specifications for Desktop, Laptop, Netbook, and Thin Client/VDI Computers
Table 7.4.9.2.A Device Specifications for Desktop, Laptop, Netbook, and Thin Client1/VDI Computers
Operating System
Windows
Specifications
Windows 7 or newer
Mac OS
Mac OS X 10.7 or newer
Linux
Linux: Ubuntu 11.10, Fedora 16 or newer
Chrome OS
Chrome OS 19 or newer
Memory
1 GB RAM or greater
Connectivity
Screen Size
Screen Resolution
Input Device Requirements
Computers must be able to connect to the Internet via wired or wireless networks.
9.5 inch screen size or larger
1024 x 768 resolution2 or higher
Keyboard
Mouse or Touchpad or Touchscreen
The input device must allow students to select/deselect, drag, and highlight text, objects,
and areas. The input device must allow students to enter letters, numbers, and symbols
and shift, tab, return, delete, and backspace. To meet security guidelines, each
Bluetooth/wireless keyboard must be configured to pair with only a single computer during
assessment administration.
Headphone/Earphone and
Microphone Requirements
(Some student
accommodations may require
headphones/ earphones (e.g.,
text to speech).
Additional Guidance
Other assistive technologies may be needed for students requiring accommodations.

Headphones/Earphones
Microphone
1 Each
computer operating in a thin client environment must meet or exceed minimum

hardware specifications, as well as bandwidth and security requirements.
2Computers
must accommodate the 1024 x 768 screen resolution minimum without

panning. Some netbook computers may have screen resolutions slightly less than the
1024 x 768 minimum, yet may meet all other minimum requirements. Depending on
netbook model specifics, school technology administrators may be able to reset screen
resolution to meet guidelines.
53
Table 7.4.9.2.B. Device Specifications for Tablets

Table 7.4.9.2.B Device Specifications for Tablets
Operating System
Android
Specifications
Android 4.0 or newer
(with 1 GB RAM or greater)
Apple iOS
iPad 2 or newer running iOS 6 or newer

(with 512 MB RAM or greater)
Windows
Memory
Connectivity
Windows 8 or newer
(with 1 GB RAM or greater)
By operating system
Tablets must be able to connect to the Internet via wired or wireless networks.
Screen Size
9.5 inch screen size or larger
Screen Resolution
1024 x 768 resolution2 or higher
Input Device
Requirements
Keyboard
Touchscreen or Mouse
Due to the onscreen space occupied by a tablets virtual keyboard, assessments
will require external keyboards for test takers using tablets so as not to limit or
obscure the view of test item content and related functionalities when text input is
required.
External keyboards must allow students to enter letters, numbers, and symbols and
shift, tab, return, delete, and backspace. Tablet touchscreen interfaces can be used
for student interactions with the assessments other than text input, including to
select/deselect, drag, and highlight text, objects, and areas. To meet security
guidelines, each Bluetooth/wireless keyboard must be configured to pair with only a
single computer during assessment administration.
Headphone/Earphone
and Microphone
Requirements (Some
student accommodations
may require headphones/
earphones (e.g., text to
speech).
Other assistive technologies may be needed for students requiring

accommodations.
Headphones/Earphones
Microphone
54
7.4.9.3
System Performance Requirements

The system must demonstrate and report that it can meet the following performance requirements
in a controlled Internet access environment, requiring a bandwidth connection of no more than five
(5) kilobits per second (kbps) per simultaneous test taker using a caching solution or fifty (50) kbps
per simultaneous test taker without caching:
deliver less than one second mean screen refresh rate for 90% of all users; and,
ensure that no users have a refresh rate of greater than five (5) seconds.
The respondent should describe the ability to support a substantial number of simultaneous online
users of the system (up to fifty percent (50%) of the current Membership in Florida Public Schools
as determined through the most recent Survey 2 data under the Education Information &
Accountability Services (EIAS) found at http://www.fldoe.org/eias/eiaspubs/pubstudent.asp). The
respondent should consider any services provided to other states when presenting this
information. The contractor must continually monitor and report to the Department the number of
simultaneous online users of the system during student testing periods.
7.4.9.4
Connectivity Requirements
The respondent should consider existing and planned communications infrastructures when
proposing its system. Preferably, school, district, and Department technology architecture and
computing hardware should not have to be replaced.
7.4.9.5
Test Site Set-Up/Certification Requirements

The contractor will develop specifications for, and deliver a system to operate/deliver online tests
and that serves as:
a secure browser, which both [a] locks access to all other computer and web programs,
applications, and files, and [b] masks and directly accesses the testing host web site; and,
a system check, monitoring required computer software and connectivity readiness for testing
and which can be run from individual machines.
The contractor will be responsible for assisting schools in certifying that computers are properly
prepared and can make the necessary Internet connections to successfully deliver tests. The
contractor will develop and make available an infrastructure test, consisting of mock items, that
replicates the size of the largest CBT and is run on the actual testing platform (i.e., users log in to
the testing platform as if they were logging into a live test). The contractor will assist the
Department in developing a guide to walk users through running the trial on machines that will be
used for testing to help them identify technical issues and make adjustments prior to live testing.
The respondent should describe the proposed test site set-up, certification, and system check
measures in the reply.
55
7.4.9.6
Computer-Based Test Presentation Requirements

Table 7.4.9.6 CBT Tools and Features
The following table lists required and preferred tools and features in the CBT platform.
Accommodated features are required. More information about the test accommodations that must
be provided for students are provided in Appendix H.
Tool
Description
Required Features
Calculator
Computer-based calculators will be available to students for the mathematics tests.

Numbers and functions on the on-screen calculator will be activated by mouse-click on
the displayed button, or by keystroke of the appropriate keyboard key. User-acceptance
testing will be required to assure that calculations are performed in the same sequence
and with the same rounding rules applied as the hand-held calculators. Respondents
should describe proposed functionality and features of calculators to be included in
computer-based test forms.
Reference Sheet
The system must be capable of displaying a pop-up reference sheet for mathematics
tests. This will be capable of being enlarged and the Department prefers that, when
open, the reference sheet not obscure portions of the test item.
Ruler
If it is determined that some mathematics test items require the use of a ruler, a pop-up
ruler will be available to students. The ruler will be easily moved and rotated on-screen
by the student.
Preferred Features
Straight Edge
A pop-up straight edge, without measurement increments. The straight edge will be
easily moved and rotated on-screen by the student.
Option Eliminator
A feature for eliminating options. However, the feature cannot eliminate the option by
placing any mark on or in close proximity to the bubble that would be used for marking a
response. It must allow the student to easily read the option after it has been
eliminated. Students must be able to undo the elimination.
Bookmark
Mark test items for review and navigate easily to return to previous items, including
those bookmarked for later review. Bookmarked items should be visible on an item
summary page at the end of the assessment.
Highlighter
Highlight text on all tests and to erase the highlighting. If a student highlights in a
reading passage, the highlighting should remain through all items related to that
passage until the student erases it.
Notes
Allow students to create electronic notes for reading passages if desired. These notes
must not hinder reading of the passage. Notes should travel for each passage and item
associated with that passage, then refresh with each new passage.
Footnote Pop-up
Reading items require a pop-up footnote. That is, if the student clicks on a footnote
number occurring in the passage, the text for the footnote appears. Footnotes must
also appear at the end of the passage.
56

Tool
Description
Student
Tutorials/Practice
Sessions
Each student that takes the computer-based test must have the opportunity to learn how
to use the tools, select/enter responses, and navigate the system in a short practice
session with instructions that may be either teacher-directed or independently
conducted, as described in Section 7.3.7.5. The contractor will produce and make
available a web-based practice test for each assessment. The contractor may also
provide access to a selectable practice test at the beginning of any test session
within the platform.
Accommodated Features (Required)
Variable Font
The system must be able to display items and text using variable font sizes. For
example, footnotes for words in passages must appear smaller at the end of the
passage than the text used in the passage.
AdministratorSelectable Zoom
The system must be able to Zoom in increments from approximately 12 point to

approximately 72 point available for the student to zoom or for the administrator to
preset. Graphics must be able to be enlarged also. Any buttons, tools, the
pointer/cursor, and the scrollbar must also enlarge based on the zoom level selected.
AdministratorSelectable
Large-Print Font
The system must allow for selection of a large print font of at least 14 point font and up
to 72 point font. The student must then be able to zoom as indicated above.
AdministratorSelectable Variable
Font and Background
Colors
The system must allow for selection from a pallet of colors for font and background
colors. The Department prefers fonts of black, dark blue, light blue, pink, and yellow
with backgrounds of white, black, brown, or dark blue. Other colors may be
recommended. Any buttons, tools, the pointer/cursor, and the scrollbar must also
change based on the colors/fonts selected.
AdministratorSelectable Screen
Reader
Provide an integrated (preferably password-enabled) selectable screen reader capable

of reading via headset to the student, all text, mathematical symbols, and text within
graphics. The contractor must ensure that the screen reader has one consistent voice
and is not operating-system specific. The contractor must run quality control checks on
all products, test forms, graphics, and items to ensure the screen reader pronounces
words correctly.
AdministratorSelectable Assistive
Devices Integration
Support assistive/adaptive devices commonly available to support computer input and

interaction to persons with disabilities.
7.4.9.7
Computer-Based Test Screen Layout Requirements

The contractor is required to recommend a set of screen layouts for the presentation of computerdelivered test items. (See also Sections 7.3.11.2 and 7.4.8) These layouts must include:
prominently displayed student identifying information [name, student ID, etc., as specified by
the Department];
icons to access the item tools/references noted above; and
icons presented with consistent location and look and feel across all screen layout options.
57
Additionally, the system must code and capture in the item bank the basic screen layout used
to deliver the item on computer to students.
7.4.9.8
Training Delivery Requirements

The contractor will develop specifications for and produce a training plan to provide initial training
and supply materials to support on-going training for audiences including, but not limited to:
the Department K-12 Student Assessment program team;
other Department staff;
district assessment coordinators;
school assessment coordinators;
school and district hardware/network technical staff; and,
school and district test administrators.
The Department prefers the contractor to deliver in-person training to approximately 500 key
Department and district personnel in as many as ten (10) sessions of not more than one-half ()
day each, held throughout Florida prior to the first implementation of the new computer-delivery
platform, which will occur in the spring of 2015. Additional training should be provided via WebEx
or similar web-delivered, interactive sessions scheduled at least six (6) weeks in advance of the
opening of each online assessment window. These training requirements are in addition to the
training materials described in Section 7.4.7.1.
7.4.9.9
Technology-Supported Item Types

The Department is interested in leveraging empirically-validated item formats which take
advantage of the capabilities of computer technology. Examples of these item types may include:
Drag-and-Drop in which students select from multiple possible responses to select one, via
mouse-click or key-stroke combination, and then drag it to insert (drop) it in an indicated
blank to correctly complete a mathematics formula, text-editing task, etc. This process may be
repeated for the same test item so that multiple answers are placed into multiple blanks.
Rotate and other image transformation items.
Other formats as the contractor may recommend or the Department may request.
The respondent may include the capacity to support these types of items in the reply.
7.4.10 Customer Service and Technical Support

The contractor will provide customer service via toll-free phone and email between 7:00 a.m. and 8:30 p.m.,
Eastern Time. The service will be staffed appropriately, as needed, with personnel designated to respond to
assessment program calls and perform other Florida assessment program tasks specific to the assessments
detailed in this ITN. Customer service staff must document all communications in a log and furnish it daily to
the Department during each administration or upon request as needed. The contractor will develop a call topic
matrix, with the Departments guidance, to ensure that calls are responded to in a consistent and appropriate
manner.
Beginning not less than four weeks before and extending one (1) full week after each test administration, the
contractor must use additional customer service personnel to handle computer-based test administration calls,
and personnel who are able to provide advanced technical support must be immediately available for any
issues that cannot be resolved by call center staff. The Department is interested in systems which provide
help desk functionality via telephone and Internet chat systems.
The system should employ a Department auditable job-ticket method of tracking issue types, status, and
individual client trouble calls/emails including current aggregate status information for resolution.
58
7.5.0 Scanning, Scoring, and Reporting

This section of the ITN provides a description of the scanning, scoring, and reporting requirements for this
project. Section 7.5.2 provides details for preparing school and student information, and Sections 7.5.3, 7.5.4,
and 7.5.5 provide the requirements for the scanning, scoring, and reporting, respectively. The steps
necessary to verify the accuracy of these procedures are described in Section 7.5.6.
The contractor must assign experienced staff to complete the scanning, scoring, and reporting tasks and to
coordinate with the test development, the test administration, and the psychometric staff during various
phases of the scanning, scoring, and reporting. As described in Section 7.7.2, it is anticipated that one of the
four project managers will focus on scoring and reporting and one will focus on computer-based testing. The
one-page vitae for each of the professional personnel must indicate their scanning, scoring, and reporting
experience within the organization to ensure the critical and extra-critical dates may be met.
The contractors scoring and reporting staff must coordinate with the development team to ensure the file
layouts accurately reflect the final tests; with the test administration team to obtain information about testing
procedures, schools, and students; and with the psychometric staff to ensure timely file verification and
transfer during the scoring and reporting process.
The contractor must verify the accuracy of all files and reports prior to submission to the Department for
review. The contractor will develop staffing plans to complete its review of files and reports and then provide
the Department time to complete its review.
Because of the secure nature of the personally identifiable student information that is used for this assessment
program, stringent security measures must be in place for the Department and the selected contractor. See
Appendix I for more information on these measures.
7.5.1 Scoring and Reporting Specifications
In April of each year, the contractor will submit the first draft of the Scoring and Reporting Specifications
for all tests within a school year. These specifications provide details to ensure accurate and timely scanning,
scoring, and reporting for the following school year. The specifications are initially drafted by the contractor
using the specifications from the prior school year, updating procedures or products as instructed by the
Department. This initial draft includes anticipated schedules for planning the next school year, details on file
naming conventions, as well as the scanning, editing, scoring, and reporting procedures. These specifications
do not contain confidential information or exact replicas of products; they provide the general requirements
and processing steps using information from the prior year to assist in planning. If the initial draft needs
extensive editing, a second draft must be submitted by the contractor.
These specifications are used to communicate between the contractor and the Department throughout the
school year with details added as necessary during the year to ensure that procedures and products are
sufficiently described and any changes in procedures are documented. The contractor will anticipate having
additional documentation for its internal communications.
At the initial planning meeting, the Department will provide the contractor with the 2013-2014 Scoring and
Reporting Specifications with a list of updates for the contractor to make for the initial draft of the 2014-15
specifications. The reply should clearly indicate how the scoring and reporting staff will coordinate with the
other staff members to enable the contractor to develop the Scoring and Reporting Specifications each April
for the following school year while at the same time conducting the current years spring reporting activities.
During the school year, the contractor will maintain an addendum to the specifications noting updates, issues,
or changes to the specifications. This specifications addendum will be used to document changes made in the
specifications during the school year and to update the specifications for the next school year. If necessary,
the contractor may be requested to provide the Department with updated versions of the specifications
throughout the year. The contractor will submit the final specifications addendum at the same time as
the last closeout file of student results for that school year.
59
After the initial draft of the specifications has been reviewed by the Department and any edits or comments
have been incorporated by the contractor, development of file layouts and other procedures may begin for the
first administration in that school year.
See Appendix B for the critical and extra critical dates associated with the reports described in this section.
7.5.2 Preparing School and Student Information (MSID and PreID)
1. To aid in the accuracy of reporting results, before the initial wave of reporting the contractor must perform
checks to be sure that school names and numbers are accurate and that each students scores are
reported to the correct school. Due to the numerous types of special schools (e.g., home schools and
virtual schools) and numerous students testing in schools that are different than the schools where their
scores are reported, school numbers must often be updated after scanning. The contractor must provide
staff necessary to make updates to student information and school information between scanning and
reporting when conflicting information has been provided. The contractor will assist the Department and
districts to ensure that each students scores are reported to the correct school during the initial wave of
reporting.
2. The Department maintains a Master School Identification (MSID) website with active school information.
The Department will determine dates that the contractor will download information from the MSID website
to be used for each administration. Due to multiple waves of reporting for each administration, as well as
reporting annual EOC results, it is important to use consistent school information and to report scores
using the information from the MSID website. The Department will provide the contractor with any
changes or updates to any information on this site, such as changing a school name for the annual
reporting. The contractor will not make changes to school information or add/delete schools unless
directed by the Department.
3. To reduce the amount of information a student must provide during testing, for each administration the
contractor will obtain student information from each district prior to testing. This preidentification (PreID)
information includes student demographic information, requests for special print materials, and school
numbers of the school where the student will test and the school where their results will be reported.
In the past, the Department has obtained PreID information in a text file from each district during one time
period each fall and summer, and two time periods each spring. The PreID information was then loaded
into the CBT system or used to create adhesive PreID labels for answer documents.
The reply should include a description of how the contractor would obtain and update the PreID
information to provide the most accurate results. Respondents may propose other options to adhesive
labels, such as pregridding (preslugging) answer documents. The Department wants to obtain the most
cost efficient methods that will ensure accurate results.
4. The Department will determine the dates the districts will be permitted to submit PreID information, the
number of PreID windows, how and which information may be updated, the rules for accepting or
rejecting records or files, and the final method of preidentifying the answer documents.
5. The contractor will draft the PreID file layouts and the PreID instructions for each administration and
incorporate changes as requested by the Department. The Department will send the final PreID
instructions and file layout to the districts. The contractor will also post the layout on the online
management system as instructed by the Department.
6. The contractor must complete numerous steps to ensure the accuracy of the PreID information submitted
by each district, and the contractor must work with district staff when necessary to ensure that the PreID
files are properly uploaded in a timely manner with just one unique record for each student in the district.
a. First, the contractor must be sure that only schools that are active in the MSID file or approved by the
Department (e.g., a participating private school or community college) are used in each districts
PreID file. Then, the contractor must ensure that each field in the PreID file submitted by the district
60
is completed with valid characters as described in the layout and that the values are reasonable. For
example, if all records submitted for Algebra 1 are coded as first-time takers in a district or all are
coded as female, the contractor must alert the district and permit updates as necessary. Also, the
contractor must compare the numbers of students in each district to the number of students tested in
the previous comparable administration to identify any issues with the number of records in the
districts file and take corrective action as necessary. The reply should include a description of the
system it will use to ensure the accuracy and reasonableness of these files.
b. Two days prior to the close of each PreID window, the contractor will contact the districts that have
not successfully submitted a PreID file and aid the district to ensure each district is able to submit its
data.
c. During the PreID window, to assist district staff in reviewing the files as read by the contractor, the
contractor creates PreID Frequency Reports with n-counts for various demographic groups for each
district to review in the online management system. The contractor will also generate a PreID
Frequency Report with state totals for the Department to review.
d. The district may submit additional files to make corrections during the PreID window. The contractor
will draft the format of these reports and make updates as provided by the Department. The reply
should include examples of the types of reports that might be used to aid districts in reviewing the
PreID data and methods to make updates and corrections while ensuring the final files include one
record per student with accurate information.
e. After the PreID window closes, the contractor provides the Department with a state PreID file
containing all the accepted records that were submitted by the districts. This file will be in the same
format as the files provided by the districts, one record per student, except with a unique PreID
Number, generated by the contractor, for each student. The PreID Number is used for each test the
student is assigned within that administration and the PreID Number is unique across administrations
for each school year. For example, if a student in grade 8 is taking ELA/L and Algebra 1 in the
spring, that student will have one PreID number for both spring assessments, and that number will
not be used again during that school year.
f.
If a students school is incorrect in the districts PreID file in (e.g., student moved), school or district
staff will grid the student information on the cover of the answer document or add student information
using the online management system prior to testing.
g. If a student will be tested in a school other than the school used for reporting, PreID labels are sent to
the school where the student will test or the CBT record is set up where the student will test. This is
called the Alternate District/School Number.
h. The contractor will print PreID Student Rosters and include them with the shipments of PreID labels
(or preidentified answer documents) to districts. These rosters list the information on the PreID labels
and an explanation of the codes used on the PreID labels. The contractor will provide hard-copy
PreID Student Rosters with the PreID labels and electronic copies on the contractor's secure website
as specified in the Scoring and Reporting Specifications. Each spring, via a second PreID window to
provide updates closer to testing, the contractor will provide a supplemental shipment of PreID labels
and rosters to districts that update their PreID files.
i.
As permitted by the Department, districts may subsequently edit some of the information using the
online management system or add new students for the CBT system. Students who were not
included in the PreID files that are taking paper tests will not be loaded into the PreID system, and
PreID labels are not sent for new students taking paper tests.
61
7.5.3 Scanning/Editing PBTs and Extracting CBT Records

Scanning and editing requirements in this section refer to the PBTs. The contractors CBT system must be
capable of creating a file with each students information and the exact responses entered by each student,
including the fill-in response and performance tasks, character by character before editing, similar to the scan
files described in this section for the PBTs. The reply should include a description of how the scanning and
editing procedures in this section will be completed, as well as a description of the types of CBT extract files
that will be available for the Department prior to the creation of the scored files described in the next section of
this ITN.
1. For each administration, the test administration manuals describe in detail the test materials provided to
districts and schools and the procedures for returning the materials and completing information in the CBT
system. The contractor will establish procedures to verify that the district and school staff completed
these procedures correctly. The contractor will work with the Department and districts if any anomalies
are found and make corrections as necessary. This includes mistakes made by districts such as returning
to-be-scored answer documents to the wrong site or miscoding a school number on a document.
2. The reply should include a description of where the answer documents will be scanned, the types of
scanners to be used and their capacity during the time this project will be scored, general editing and
proofing procedures of documents scanned, the types of documents and procedures the contractor
proposes to ensure that the correct number of answer documents have been received and scanned to be
scored. In the past districts packaged to-be-scored answer documents with a Document Count Form
(DCF header) on top of documents of the same type. Alternate methods to ensure that the correct number
of answer documents is scanned from each school may be proposed. See Section 7.4.1 for more
information on the return of materials.
3. The contractor will provide the Department with scan files for the PBTs and the extract files from the CBT
system prior to creating the scored files. The contractor will draft layouts for these scan and extract files
and incorporate changes as requested by the Department. The Department will specify the fields in the
file, valid values for the fields, and the contents of each file (e.g., by grade or by subject).
4. The contractor will provide two versions of each scan file. The original scan file contains the exact
information as scanned and the second, the edited scan file, contains the final information from the
scanning system incorporating any edits made as described in the Scoring and Reporting Specifications.
For example, if the student wrote her name on the top of the answer document, but did not grid the
bubbles on the document, the original scan file would have a blank in the name field and the edited scan
file would have the name as entered by the contractor based on the hand-written name. The edit rules
are designed to ensure a student will receive the correct report during the initial wave of reporting. The
reply should include a description of how the edits would be made to original scan file data to ensure
accurate and timely reporting of results. If necessary, the contractor will update the edited scan files to
ensure the correct information is transferred to the first version of the State Student Results file (SSR).
5. If any pages with responses cannot be scanned, all responses key entered into the system must be
checked by a qualified person other than the person who initially key entered the responses.
6. The contractor will maintain edit logs that indicate why records on scan files were edited and by whom.
The logs will indicate the names of anyone who key entered or verified responses.
7. The contractor will provide the original scan files and the edited scan files before scored files are created.
See Section 7.5.6 for more information on the steps for verifying data and file delivery. If any edits were
not made correctly, the contractor will update the edit files.
See Section 7.6.2.2 for information on the records that must be scanned or extracted prior to the
beginning of the psychometric tasks. The contractors psychometric and scoring/reporting staff must
coordinate schedules and activities to ensure that all files are transferred as expected (times, names,
layouts, etc.).
62
7.5.4 Scoring
The basic types of scored files created by the contractor and history files used and maintained by the
contractor are described in this section. The contractor drafts detailed layouts for each file and then
incorporates changes requested by the Department.
7.5.4.1
Types of Scored Files

From the PBT edited scan files and the CBT extract files, scored files are created. Scored files that
contain data for students throughout the state are called State Student Results (SSR) files. These
files contain information from the scan and extract files, information from the students preidentification labels when used, students scores, and additional information regarding item (test
question) data and scoring. The districts are also provided a truncated version of the SSR with just
their students data, but no item data. This file is called the District Student Results (DSR) file.
1. Before the full-state SSRs are prepared, the psychometric staff (Department and contractor),
begin their calibration and equating activities with a subset of the state records. These files
are called calibration SSRs. Initially they only contain the raw score data, then after the
calibration and equating, they are completed with the other derived scores as specified by the
layout. See Section 7.6.0 for more information on the calibration, equating, and scaling
procedures. Coordination of the details for the transfer of these files from scoring/reporting
staff and psychometric staff is crucial to the timely release of reports and files.
2. Aggregated files contain school, district, and state data based on the scores reported on the
SSRs. These files contain information that is reported to the schools and districts as well as
additional information such as standard deviations of the scores. The files with all the data are
called State Aggregated Results (SAR) files, and the districts receive a District Aggregated
Results (DAR) with the data for the state and their schools and district.
3. After the initial release of files and reports, there are up to three waves of late reporting for
students whose scores were not included in the original release of data, whose records had to
be updated, or whose scores are released after suppression. The contractor will generate late
reporting SSRs, DSRs, and provide student reports in a manner similar to the initial wave of
reporting. SARs and DARs are not created for late reporting.
4. After the last wave of late reporting, the contractor creates and provides close-out SSRs and
DSRs that contain the final information and scores for that administration.
5. After each summer EOC administration, the contractor will create annual SSRs, DSRs,
SARs, and DARs by concatenating the SSRs and aggregating data combined for the fall,
winter, spring, and summer administrations for the school year for subject tests that require
multiple test administrations within a school year. The Department will specify how to handle
any anomalies in reporting across administrations (e.g., a school name changed).
6. For each spring close-out SSR and each annual SSR, the Department inserts demographic
information from its Survey 3 membership file of student data and aggregates the data for
approximately 50 demographic groups. The Department will provide these Demographic
SSRs and SARs to the contractor. From these files, the contractor will create the
Demographic DSRs and DARs and post these files to the online management system for
district-level access.
7. The contractor posts most files to the online management system, but may also be required to
provide some files to the districts on CD/DVDs. See Table 7.5.5 for additional information.
7.5.4.2
Types of History Files

Three types of history files may be used for this project. The Department will provide the
contractor with the layouts and necessary history files to begin the project, and then the contractor
will maintain the files and update the files and layouts as instructed by the Department.
63
1. One type of history file may be used for the ELA/L Retake. Depending on policy decisions
regarding scoring, the ELA/L Retake history file may be called the Alternate Passing Score
(APS) file and would provide information on students who were grandfathered into a lower
passing scale score requirement. Students who take the first administration of the Grade 10
ELA/L before standards are set may be grandfathered into a lower passing cut score than set
for other students. If this occurs, for each administration, the contractor will use this file to
determine if a student is eligible for the APS and will update the records for students who test.
The Scoring and Reporting Specifications will detail the matching criteria.
2. The second type of history file will be used for the Algebra 1, Geometry, and Algebra 2 EOC
Assessments for FDOE verification of the testing status (Retaker vs. First-time Tester) of each
participating student. This kind of history file provides information for students beginning with
the first time the subject test is administered statewide. Each administration, the contractor
will use these files to determine if a student is a first-time tester or a retaker. The Scoring and
Reporting Specifications will detail the matching criteria and process for using these files.
Some aggregated data are based on first-time testers only, so the accuracy of the first-time
status for each student is very important.
3. A third type of history file will be used to track scores across grade levels for the Grades 3-11
ELA/L and Grades 3-8 Mathematics students. Starting with the first ELA/L and Mathematics
administration, the contractor will begin maintaining a file with student data for Grades 3 10
ELA/L and Grades 3 8 Mathematics, called the ELA/L and Mathematics History File.
Scores for up to 7 administrations will be added to this file using the closeout files for each
administration. The contractor will update the file after each administration and will provide it
to FDOE at the end of the contract. The Scoring and Reporting Specifications will detail the
matching criteria and process for using these files.
The contractor will update these history files each administration and provide a copy to the
Department.
7.5.4.3
Types of Scores and Codes on the SSR/DSRs

1. The scoring process begins with the Test Maps that are created by the contractor when a test
form is created. The Test Maps are used to identify item placement in the test book and other
information about the item. The Test Maps include information regarding which items are core
items and field test items, which responses are correct, and the category used for subscores
within each content area.
2. If the Test Map or a core item is determined to have a flaw, the Department may choose not to
include an item in the students scores. The contractor must have procedures to provide the
flexibility to edit the Test Map information during the scoring process, if necessary.
3. Excluding field test item data, subject raw scores and raw subscores are then generated by
the contractor.
4. See Section 7.6.1 for information regarding the calculation of scale scores (SS) by the
contractor.
5. Once the SS are determined, the score is classified in one of the Achievement Levels (AL)
by the contractor.
6. Florida statute requires that passing scores be set for all grades and subjects. The contractor
will report the scores as specified by the Department.
7. The ELA/L Retake may have two (2) cut scores for passing, depending on policy decisions
that will be made prior to spring 2014. If so, the contractor will first determine if the student is
64
on the APS file described in this section and then determine the passing status for each
student.
8. For each special school, there is a defined School Type number that is used to determine
which reports are prepared for a school and whether the results are included in district and
state aggregated data. For example, home schools are school type 10, and they are not
included in aggregated results within a district.
9. Scores are not reported for some answer documents (e.g., students who did not answer
enough questions to have a score reported). The student Score Status Flag determines
whether the student scores are reported and aggregated. The Scoring and Reporting
Specifications provide details on these codes.
10. Many codes are related to each other and all values must be internally consistent. For
example, the School Type Codes and the ESE Primary Exceptionality codes are used to
determine the Curriculum Group and the Standard Curriculum Codes. The Ethnicity
Code and the Race Codes are used to generate the Report Race Code. Students who use
large print and Braille documents must have specific Primary Exceptionality Codes and
accommodation codes that must be checked and updated by the contractor if necessary.
11. Numerous other codes (approximately 15-20 for each subject) are provided on the SSRs.
These types of codes may change occasionally based on changes in the assessment program
and needs of the schools and districts. The contractor must be flexible in modifying the codes
and including additional codes as needed to aid in the accurate and timely reporting of useful
results.
12. After the scores and codes are verified by the contractor, the first version of the full-state State
Student Results (SSR) file is provided to the Department. The contractor will verify that the
data were properly transferred from the corresponding edited scan file, that all scores, flags,
and codes were set correctly, and that all fields contain valid values as defined by the layout.
See Section 7.5.6 for additional information on the steps for verifying mock data, early district
return data, and state data.
7.5.4.4
Handling Special Print Documents

1. This project includes Large Print, Braille, and One-Item-Per-Page (LP, BRL, and OIPP)
special print test materials. The contractor is responsible for numerous tasks to prepare the
special print documents for scoring and reporting. The scores for students using these
materials are included in the initial release of reports.
2. The Scoring and Reporting Specifications will include the steps the contractor follows from the
time these materials are unpacked until the scores are reported. If any to-be-scored answer
documents are not packaged correctly, the contractor is responsible for properly preparing the
materials or contacting the district if necessary.
3. The contractor is responsible for the transcription of the responses from the LP book to the
regular print book according to guidelines provided by the Department. The contractor may
suggest ways to complete this process using the CBT system to expedite the scoring process.
4. The contractor is responsible for transcribing Braille answer documents to regular print or into
the CBT system. If a student uses an OIPP document, the contractor is responsible for the
transcription of the responses from the OIPP book to the regular print book according to
guidelines provided by the Department. Transcription services may be subcontracted, but all
transcriptions must be completed in time to report the students results in the initial reporting.
This process must be carefully monitored and transcription must be double checked by a
person other than the person who initially transcribed the responses.
65
5. The contractor will create a Special Document Tracking Log to inventory and track the
processing of all special print documents. The contractor and the Department will determine
the format of the log to track the answer document information and times for the log to be
provided to the Department.
7.5.4.5
Handscoring
Handscoring refers to the processes necessary for determining the rating of a students response
on the writing prompts or on the performance-based tasks in ELA/L and Mathematics that
respondents may propose. Both paper-based and computer-based tests will contain performance
tasks, and so the Contractor must have the capability to score both hand-written and key-entered
student responses. The Department requires handscoring processes that are reliable and valid, as
well as efficient in terms of time and expenditures. This section describes procedural, design, and
implementation requirements for scoring student writing and performance tasks. Respondents
may propose other processes and procedures that accomplish the tasks in an equally reliable and
valid manner, and are at least as efficient as those outlined below. The final processes will be
agreed upon during the negotiation phase. See Appendix J for more details regarding
handscoring.
1. Students constructed responses should be scored by trained readers using imaging
technology. Respondents should provide a detailed explanation of the system that will be
used.
2. Respondents should also provide a detailed description of how the security of the test items
and student responses will be maintained throughout the handscoring process.
3. Trained professional readers will score written answers using criteria established by the
Department in coordination with the contractor. Readers must have a college degree and
must qualify to score Florida student responses by accurately scoring a series of training and
qualifying papers. The readers must be monitored carefully and continuously throughout
scoring for their accuracy (validity) and reliability (inter-rater agreement). In addition, team
leaders must read behind each reader to determine the accuracy of their scoring. Readers
must also be periodically given papers that have been previously scored to determine the
accuracy of their scoring (validity responses). Quality control metrics will be established by the
Department after contract execution. Student responses for constructed-response items in
ELA/L tests in grades 3 and 10 and for Algebra 1 will be scored by at least two scorers (100%
second read). For all other ELA/L and Mathematics performance tasks, responses will be
scored by one reader, with twenty percent (20%) of these papers having a second reader. For
these papers that are read a second time, if the two (2) readers scores do not match, a formal
resolution process will be established in order to obtain exact or adjacent scores for quality
control purposes only.
4. Handscoring may be done at the contractors established scoring site(s). Respondents may
also propose solutions that include distributed scoring. The scoring sites must be secure
facilities that employ procedures and technology to control access and maintain the security of
materials. Access to the sites must be limited to full and part-time employees and authorized
visitors. Secure materials must be accounted for, and computers containing secure materials
must be secured from unauthorized access or copying of files.
5. The contractor should incorporate the following requirements into written handscoring
specifications prior to scoring. The handscoring specifications will be a detailed guide to
conducting handscoring that will be used by the contractors handscoring managers and the
Department. Draft handscoring specifications must be delivered by the contractor for review
and approval at least three (3) months before the first test administration. The handscoring
specifications should include the following:
66
a. Handscoring schedules.
b. Site requirements (includes security and access during scoring).
c. Scoring design (includes expected number of reads; groupings of items assigned to
readers; assignment of first and second reads).
d. Personnel (includes minimum requirements, such as education and experience, for
scoring directors, team leaders, readers; number of personnel; security agreements).
e. Rangefinding meetings (includes staffing, meeting procedures).
f.
Rangefinder review meetings (includes staffing; meeting procedures).
g. Training and qualifying materials (includes listing/description of required materials;

annotations for training and qualifying rounds; review and approval process).
h. Validity sets (includes number and type of validity responses; transcription).
i.
Scorer degree verification procedures.
j.
Training procedures.
k. Qualifying standards required of scorers before live scoring of student work.

l.
Handscoring reports (includes listing/description of handscoring reports; distribution).
m. Scoring process (includes administration of validity sets; rules for determining and
assigning resolution reads).
n. Monitoring for scorer drift and other sources of scorer inconsistency and retraining
(includes read-behind guidelines; use of daily and cumulative reader reports).
o. Processing requirements (includes programming and design requirements for
implementing online handscoring of images).
p. Providing electronic copies of annotations used for training materials, calibration, and
recalibration of readers,
6. Department staff must be provided on-line access to all handscoring systems and reports and
will communicate frequently with the contractor throughout the scoring process. During live
scoring daily handscoring status calls must be held between the Department and the
contractor.
7. At the end of handscoring, the contractor must produce a report that captures at least the
following results for all prompts and performance-based items and tasks that required
handscoring:
a. Item or task ID, the form number, and sequence number
b. Total number of reads for each item
c. Inter-Rater Reliability statistics including number and percentage of perfect agreement,
number and percentage of adjacent agreement, and number and percentage of nonadjacent agreement
d. Validity Statistics including the number of validity reads, number and percentage of
perfect agreement, number and percentage of adjacent agreement, and number and
percentage of non-adjacent agreement
e. Distribution for scores and nonscorable codes from handscoring
67
f.
The mean, median, modal score, and score variance and standard deviations for scorable
responses
g. Final quality-assurance monitoring reports from the handscoring system that include
rescoring and retraining rates
h. Scoring notes handscoring.
The handscoring statistical data will be provided in an electronic data file in a format
agreed upon by the contractor and the Department.
Provide copies of all materials used in training and recalibration.
8. It is expected that Department staff and Florida educators will be involved in the process to
select student responses that will be used in training and qualifying scorers.
7.5.5 Reporting
This section includes requirements for reporting results to students, schools, and districts. This includes the
posting of the scored files (DSRs, SARs, and DARs) described in Section 7.5.4, the release of hard copy
student reports and PDFs of other reports described in this section, and the creation of files produced for the
Departments website.
Although the reports used in the past may be used as a model for these reports, the Department would
consider changes to the report design to be more attractive or to present the data more clearly. The reply
should include a description of how the respondent will design the reports, sample reports, and any
suggestions for improvements to the current models. The report design is reviewed by the Department after
each administration. If requested, the selected contractor will make modifications to the design after an
administration and prior to the next administration. In addition, if high-level changes (change in Commissioner
of Education, significant policy change) occur after designs have been approved; the contractor will make the
appropriate modifications to the report design.
If the respondent does not have qualified, professional staff devoted primarily to visual graphic arts and report
design, a subcontractor(s) should be identified to provide these design services. The reply should include the
names and documentation for the qualified staff or the subcontractors who will provide these design services.
The final report design will take numerous iterations to ensure that all types of scores have been clearly
presented. The Department will determine the final colors, design, and content of all reports.
A general description of each type of report is provided below. Most of the reports are designed by test, that
is, one (1) report form for English Language Arts/Literacy, one (1) for Mathematics, one (1) for each EOC
subject test, and one (1) for Retake.
Student Reports
For each student that tested by test.
Includes the students scores (e.g., scale scores, achievement levels, raw subscores, passing status),
information about the scores (e.g., passing scores, points possible), and comparison data (e.g., state
means).
Two (2) hard copies (one (1) for parent and one for school) shipped to districts, packaged by school.
Printed front and back with up to eight (8) colors plus black.
Some text translated to Spanish and Haitian Creole. The Spanish and Haitian Creole translations on the
student reports must be created by someone familiar with the terminology used in Floridas schools and
proofed by an independent party, that is, someone not involved in the original translation.
Delivered with cover memoranda and packing lists following specifications similar to those for distributing
test materials.
Hard copies produced for initial reporting and late reporting waves.
68
For ELA/L and Mathematics test, the student report will include historical scores that are up to seven (7)
test administrations with a bar graph.
At all grades/subjects containing performance tasks, each student will receive a report showing his/her
response to one performance task in each grade/subject. The report will include a copy of the answer
given, the number of points earned on the item, descriptive information about the question, and scores
(points earned) earned on all of the performance items.
School Report of Students
For each school by subject test.
Includes the scores for all students in the school that tested.
One (1) PDF by school posted on the online management system for school access for initial reporting
only.
One (1) PDF of all schools in a district posted in the online management system for district access for
initial reporting only.
One (1) color plus black.
PDFs produced for initial reporting and late reporting wave.
Reports created for the late reporting wave include only the students with reports in that wave. The
reports will denote that they are part of late reporting and the reports will be delivered to districts on
CD/DVD.
District Report of Schools
For each district by test.
Includes aggregated data for the schools in the district (mean scores, percent passing, etc.).
One (1) PDF by test by district posted in the online management system for school and district access.
One (1) Excel spreadsheet for all districts and schools by test with the same data as the PDFs for the
Department to use for its web postings. This may be a slightly modified version of the PDF without color.
PDFs produced for initial reporting only.
State Report of Districts
For each test.
Includes aggregated data for the districts in the state.
One (1) PDF by test, posted in the online management system for school and district access.
One (1) Excel spreadsheet by test with the same data as the PDF for the Department to use for its web
postings. This may be a slightly modified version of the PDF without color.
District Summary
For each subject.
Includes the aggregated data for each grade level tested in the district and the state.
One (1) PDF by subject by district posted in the online management system for school and district access.
State Summary
For each subject.
Includes the aggregated data for each grade level tested in the state.
69
One (1) PDF by subject posted in the online management system for school and district access.
CD/DVD/Jump Drive with Student Report Shipments
The initial shipment of Student Reports includes a CD, DVD, or jump drive with the DSRs and DARs and
their layouts.
The late reporting shipments include a CD, DVD, or jump drive with the DSRs and layout and the PDFs of
the School Reports of Students for the late reporting students.
The Scoring and Reporting Specifications provide specific details on handling special student score flags and
school types and the specific information and scores provided on each report.
The packaging and distribution requirements for test materials (see Section 7.4.1) also pertain to the packaging
and distribution of the reports.
Table 7.5.5 below shows these reports grouped as they are released to districts and schools.
Table 7.5.5 Results Released to Districts and Schools
Table 7.5.5 Results Released to Districts and Schools
Results
Release
Number
Released To
Method of Release
Contents of Release
To Districts
Online Management
System
District Student Results File (DSR .txt)

School Report of Students (PDF of all students in district)
To Districts
Online Management
System
State Summary (PDF)

District Summary (PDF)
State Report of Districts (PDF)
District Report of Schools (PDF)
State Aggregated Results File (SAR .txt)
District Aggregated Results File (DAR .txt)
To Schools
Online Management
System
State Summary (PDF)

District Summary (PDF)
State Report of Districts (PDF)
District Report of Schools (PDF)
School Reports of Students (PDF of own school only)
To Districts and
Schools
Shipments to
Districts
To Districts and
Schools
Waves of Shipments
to Districts
Late Reporting Student Reports (hard copies)

Late Reporting School Reports of Students (PDFs on
CD/DVD)
District Student Results Files (DSR .txt)
To Districts and
Schools
Shipment to Districts
Writing Images (PDFs on CD/DVD) (spring only)
Online Management
System
Demographic SARs, DARs, and DSRs
To Districts
Initial Release Student Reports (hard copies)

CD/DVD with DSR, SAR, DAR and Layouts
70
7.5.5.1
Initial Reporting Guidelines

The Department coordinates the schedule for the release of data to the public. The contractor must be
flexible with the reporting release schedule. The general guidelines are provided in this section.
Detailed guidelines are specified in the Scoring and Reporting Specifications.
The basic steps for the initial reporting of full-state data for Department review are listed below. The
Department may request slight modifications to these steps.
a. The contractor posts the full-state scan and edit files version A for the Department on the
SFTP site.
b. If necessary, the Department provides an issue log for the state scan and edit files.
c. The contractor posts the full-state SSR version A for the Department and any independent
checkers specified by the Department (see Section 7.5.6).
d. Contractor provides data and recommendations to the Department based on the review for
anomalous scores; the Department will review and identify any data to hold for investigation.
e. If necessary, the Department will provide an issue log for the SSR files.
f.
The contractor posts the full-state SSR version B and the SAR version B (no version A)
g. If necessary, the Department will provide an issue log.

h. If necessary, the contractor corrects and reposts SSRs and SARs until approved.
i.
The contractors psychometric and scoring/reporting staff verify the accuracy of the final files.
j.
As requested for initial checking, the contractor creates DSRs and DARs and posts on the
SFTP site for the Department review and updates as necessary until approved.
k. During this process, the Department will identify a version of the files to be used to check the
shipment of student reports and CD/DVDs. The contractor ships a sample of this shipment to
the Department to check.
l.
If requested, the contractor posts additional samples of district and school PDFs of the reports
on the SFTP site.
m. After the PDFs are approved by the Department, the contractor will create all DSRs and DARs
and reports on the SFTP site and, for state access only, in the online management system.
n. After Department approval and guidance, the files and reports will be available for districts and
schools in the online management system.
o. The student reports and the CD/DVDs will be shipped for delivery by the critical date in the
contract.
7.5.5.2
Late Reporting Guidelines

For each administration, various reasons make late reporting necessary. For example, to-bescored answer documents were sent to the wrong location, updates were made based on the
review of documents for graduating students (see Section 7.5.6.5), or the original student report
was sent to the wrong school or student.
Immediately after the initial wave of reporting, the contractor begins checking scores of graduating
students as specified in Section 7.5.6.5. Any updates will be provided in the first wave of late
reporting.
The contractor will prepare for up to three waves of late reporting. The contractor and the
Department check the late reporting files in a manner similar to the initial reporting wave.
71
Depending on the administration and number of inquiries from districts, the Department will
determine the late reporting schedule. The contractor must be prepared to adhere to a rigid late
reporting schedule to resolve any issues with reporting for one administration before the next
administration begins.
The late reporting shipments include the following reports and files.
Student Reports two (2) hard copies as in initial reporting with a note or footnote denoting it
is a late report.
CD/DVD with DSR and layout (no DAR)
CD/DVD of School Report of Students (not in the online management system) with a note or
footnote denoting it is a late report.
After the initial release of data, the Department opens a website called the Score Inquiry System
for district staff to inquire about missing reports or to ask for a students score to be reviewed. This
system is designed so the contractor and the Department can communicate requests to search for
a students answer document or report, or to review a students score. Maintenance of this system
is the responsibility of the Department.
The contractor must search for missing answer documents in the not-to-be scored materials if
requested by the district to find any documents that were mislabeled or returned incorrectly.
7.5.5.3
Close-Out and Annual Reporting Guidelines

After the last wave of late reporting, the contactor creates close-out SSRs for each administration.
These SSRs contain the final records for all student with erroneous records removed. For
example, if the district/school number was incorrect in the initial reporting and corrected in late
reporting, the close-out SSR contains the corrected record. The incorrect record is removed.
The close-out files are provided to the Department for review and approval, but they are not posted
for districts.
After the summer EOC close-out files have been approved, the contractor creates annual SSRs
following guidelines provided by the Department for tests that require multiple administrations
within a school year. These annual SSRs contain the records for students from each
administration that school year. A few fields may be slightly altered from the close-out files for
each administration, for example, if a school name changes during the school year, the name will
be updated so the SAR shows annual aggregated data for that school using the correct name.
7.5.5.4
Demographic Reporting Guidelines

After the spring close-out SSRs are approved by the Department, the Department inserts
demographic information from its Survey 3 membership file of student data and aggregates these
data for approximately 50 demographic groups. The Department then returns the Demographic
SSR and SAR to the contractor to create Demographic DSRs and DARs.
After the Demographic SARs are created and approved, the contractor also creates Excel
spreadsheets with the data that can be used for the Departments interactive demographic
websites, which can be viewed here for reference:
https://app1.fldoe.org/fcatdemographics/ and
http://app1.fldoe.org/FEocDemographics/MainMenu.aspx.
The Department will specify the format for the spreadsheets.
After the annual SSRs and SARs are created for the subjects that require the multiple test
administrations within a school year, this same process is completed for those files.
72
7.5.6 Verifying Data

Due to the tight timelines and quantity of data, all initial scoring and reporting processes are checked with
mock data prior to the administration. Then, once actual data are available, a small sample of answer
documents are used to begin checking, called early district return data. After all answer documents have
been scanned, the files and reports are checked for completeness, accuracy, and possible irregularities
(cheating) as described in this section. The verification process may require numerous iterations of the files to
ensure that each students scores are accurately reported.
The reply should include a description of the procedures to be used to verify the accuracy of data as outlined
in this section. This includes the staffing necessary to complete the mock data process before the
administration begins and the verification of each version of the files given the tight time constraints.
The Department may have independent contractors assist in a review of the procedures used in this project
and verification of the files and reports. Currently the Department has contracts with two independent parties
for this purpose. One (1) contractor is primarily concerned with reviewing the procedures and the
specifications for the scanning, scoring, and reporting with an emphasis on the psychometric aspects and
primarily coordinating with the psychometric staff. The other contractor is primarily concerned with ensuring
the procedures are followed, verifying codes and scores on the scan files, the SSRs/DSRs, the SAR/DARs,
the Student Reports, and the PDFs of the other reports coordinating with the scoring and reporting staff.
The contractor will be responsible for coordinating with the Departments independent contractors for the
review. This may include touring the scanning site and discussions with the contractors scanning, scoring,
psychometric, and reporting staff. During the psychometric activities, the independent contractors may
participate in the meetings and discussions, and during the scoring activities, the contractor may be asked to
provide a portion of the reports and files to the independent contractors when reports and files are provided to
the Department.
7.5.6.1
Checking Mock Data

1. All mock data files, reports, and procedures are generated and delivered using methods as
close as possible to the same way final files will be generated and delivered. This process
begins with mock PreID files, mock PreID labels, mock PreID Frequency Reports, mock
history files, and mock answer documents, and then all headers, files, and reports for the initial
wave of reporting are generated using the mock data. The mock student reports are packed
and shipped as specified by the Department with the media containing the DSR and DAR, and
the mock files and PDFs of reports are also posted to the SFTP site for the Department
review.
2. Actual school information from the MSID file will be used in the mock data process to check
the processing of the special school types and special districts.
3. Mock data cases may be generated using different methods. To check scanning of PBTs,
some mock cases must be manually gridded, but others may be auto-gridded (preslugged)
documents. Some CBT data must be generated by the contractor staff simulating the
processes that a CBT student would use for each form of each test. To have enough cases
to check all scoring and aggregating rules without using too many paper answer documents
and time entering CBT responses, some additional cases may be computer generated,
meaning for PBTs physical documents are never gridded or scanned for those cases and, for
CBTs no one sat at the computer and key entered the responses in the same manner as a
student enters the responses.
4. For PBTs, only the cases to check erasures must be manually gridded and other references in
this section to cases with gridding may be auto-gridded.
73
5. For each CBT, the PBT accommodated form must also be checked. Large Print and Braille
paper documents are not used for mock processing; however the regular print answer
documents and the headers are created with the appropriate Large Print and Braille codes.
6. The Scoring and Reporting Specifications indicate the various cases that must be included in
the mock data. To easily check that all values scanned on PBTs and key entered on CBTs
are properly recorded on the files, the mock data will include cases to check the values using
specific patterns (e.g., alternating A,B,C,D,A,B,C,D responses). Cases must be included to
be sure that each value on the SSR, SARs, and reports are accurate. This includes
processing erasures, multigrids, demographic values, score flag codes, school type values,
scores, etc.
7. The contractor must design the mock data files to check all procedures through the release of
results. To reduce the amount of time to review these data and iterations required, the
contractor will complete their review of all mock cases, possible values of codes and scores,
and all reports before providing the data to the Department. The reply should include a
description of how the respondent will complete these mock data, mock files, and mock
reports prior to the first day of testing for each administration.
7.5.6.2
Checking Early District Return Data

1. To begin the verification of actual (live) data, the Department will identify some districts to be
processed first, called Early Districts. The contractor will arrange for the pickup of answer
documents from these districts so they will be scanned first for Department review. The CBT
records for students in these districts will be extracted and provided to the Department for
review.
2. The selected contractor must anticipate that the spring scan check will be completed at the
contractors scanning site. The scan check for other administrations may be conducted at the
contractors scanning site or at the Department. If the scan checking is performed at the
Department, the contractor will post PDFs on the SFTP site of the headers and answer
documents or ship the required documents to the Department to complete the checking.
3. For the on-site review of answer documents, the contractor will provide the Department the
original scan and the edited scan files at least three (3) business days prior to the on-site
review.
4. The contractor will pay travel expenses for up to five (5) Department staff from Tallahassee to
the contractors scanning site to complete this checking. These expenses must be included in
the reimbursable funds described in Section 7.7.9.
5. During the on-site review, the contractor will provide computers to Department staff with the
following:
Access to the CBTs, access to email, access to a high-speed printer, and, if requested,
SAS.
Easy access to the answer documents. Batches and stacks will be logically sequenced
with the headers used.
LP, Braille, and OIPP documents and updated Special Document Tracking logs.
Key Entry verification logs (e.g., double entry of responses if document is unscannable)
6. The Departments independent contractors may participate in this review or they may visit the
site prior to this review.
7. Depending on the Department staff availability, the on-site review may be completed during
one week or in two consecutive weeks.
74
8. The reply may include additional and alternate methods to aid in meeting the data verification
and reporting requirements in the time allowed.
7.5.6.3.
Checking Data for Anomalous Scores

Before data are released to the public, the contractor must review the student and school scores
for anomalous data, that is, data that may not be a valid measure of student achievement. The
contractor must review the data for testing irregularities, such as a student copying another
students answers or an administrator changing answers.
If the respondent does not have staff to perform these analyses, a subcontractor should be
proposed. The Department will approve the subcontractor and final procedures to be followed for
this checking.
Anomalous scores would include the following.
Students in the same testing group have similar response patterns
A student has an excessive number of erasures from wrong answers to right answers
Students in a testing group have an excessive number of erasures from wrong to right
answers
There was an unusual increase in school performance from a comparable administration
The guidelines and procedures used to perform the anomalous score checking will be refined by
the Department with the contractors assistance.
If an irregularity is found in the data, the Department may request for the contractor to hold (i.e., not
release) some data. This is handled using the Score Status Flags between the first and second
versions of the full-state SSRs. Schedules must be set to allow time for this review and
updating before the initial reporting for each administration.
7.5.6.4.
Checking Full-State Data Initial Reporting

1. The contractor will verify that all files and all PDFs of reports have been produced according to
the Scoring and Reporting Specifications before providing them to the Department. In addition
to the checks described in this section, the contractors psychometric staff must also approve
the final SSRs and SARs before final reports are generated and data are released to the
public.
2. The verification begins by checking that the DCFs have been completed properly so student
reports will be sent to the correct school. If the school identified on the header is different from
the school on the students answer document, and both are active schools, the contractor
must identify the correct school for the student by contacting the district, if necessary.
3. The contractor will compare the edited scan files to the SSR and then the SSR to the SAR,
and compare reports to the SSR and SAR to be sure all information (scores, codes, school
numbers, etc.) is accurate and flowing through the systems correctly before submitting to the
Department for review. The contractor will verify that all scores, flags, and codes were set
correctly, and all fields contain valid values as defined by the layout.
4. The contractor will compare the final number of students tested in each school to the number
of students on the PreID files to identify any anomalies, such as a missing box. To ensure all
answer documents have been received for scoring prior to the delivery of the first scan file to
the Department, the contractor will review the numbers of records and contact the Department
and districts, using guidelines defined in the Scoring and Reporting Specifications, of any
concerns. The contractor will assist districts in returning any materials that were not handled
properly.
75
5. As specified by the Department, the contractor will also provide sample shipments including
the student reports and the CD/DVDs with files simulating the shipments to districts. The
Department will determine which districts shipments will be included in this verification to
ensure that a variety of reports are checked. They may be different from the districts selected
for early scan if those records are not diverse. As described earlier in this section, the
contractor may also be requested to send some reports to the Departments independent
contractor.
6. Before the files and reports are available in the online management system, the contractor and
the Department will verify that the correct reports and files are available for district and school
users prior to the release of the data. The contractor will provide documentation to show that
the correct files and reports are being released as specified in the reporting schedule.
See Section 7.5.5 for more information on the reports.
7.5.6.5
Checking Graduating Student Scores (Manual Rescore)

Due to the high stakes of the tests for graduating students, the contractor must perform additional
checks for graduating students who test on paper for Grade 10 ELA/L and for Algebra 1. The
checks must be completed so any updates may be included in the first wave of late reporting.
1. For the tests required for graduation, the contractor must identify student in each grade who
failed by 5 or fewer Scale Score points or 15 Developmental Scale Score points for
Manual Rescoring. If there is an APS cut score established for the assessment, the APS cut
score must be used for selecting the students.
2. For each of these students, the contractor will review each answer for the subject failed and
verify that the items have been scored as the student intended. For example, the contractor
will look for unintentional stray marks in the answer bubbles that do not appear to be an
intentional grid, will confirm the hand written name on the document is the same as on the
PreID label if one (1) is used, and will also confirm that no incomplete erasures in the griddedresponse columns were recorded as a mark (e.g., double gridding because of incomplete
erasure). The item responses will be compared to the values on the SSR for any irregularities
in editing or processing.
3. If any editing, scanning, or processing irregularities are detected, or if an incomplete erasure
would change an item response, the students paper must be rescored with the updated
response(s), and a new SS must be calculated to determine the passing status of the student.
If the student passes, the record must be updated for the first wave of late reporting. If not,
no change is made in the record.
4. A new student report will be generated and shipped to the district and the student will be
included on the School Report of Students sent with the other late reporting wave 1 student
data, if any.
5. The contractor will provide the Department with the number of answer documents reviewed by
subject and the number that will have a Passing Status changed to Y when the new files are
submitted.
6. This process will be completed within sixty (60) days of the original release of scores for the
administration.
7.5.6.6
Checking Late Reports

The contractor and the Department check the late files, reports, and CD/DVDs in a manner similar
to the initial files and reports, although the files and reports produced are slightly different. The
Department checks the files and reports after the contractor.
See Section 7.5.5.2 for information on the procedures for the late reporting waves.
76
7.5.6.7
Checking Close-Out and Annual EOC Files

The contractor checks the close-out files by comparing each to the files previously generated for
the administration to ensure that records reported to the incorrect schools have been removed and
any scores or information for students have been updated or added as necessary.
The contractor will ensure that there are not erroneous duplicate records caused by the multiple
waves of reporting on the close-out files. The contractor must review the Late Reporting issue logs
to ensure erroneous records have been removed from these files. The Department checks the
files and reports after the contractor.
The contractor checks the annual EOC files similar to the initial files, ensuring that the firsttime/retaker status codes are properly set. The Department checks the files and reports after the
contractor.
7.5.6.8
Checking Demographic Files

Each spring the contractor will check the Demographic SSRs created by the Department to verify
that no records or scores were changed; only Survey 3 data will have been inserted. The
contractor will verify that only valid values are in the Survey 3 fields before creating the
Demographic SARs.
The contractor will check the Demographic DSRs, SARs, and DARs prior to release in a manner
similar to the initial release of the files. The Demographic Excel spreadsheets must also be
checked by the contractor to be sure they contain the same information as on the SAR. The
Department checks the files and reports after the contractor.
See Section 7.5.5.4 for information on the procedures for the demographic files.
7.6.0 Scaling, Equating, Scoring, and Special Psychometric Studies

During the timeframe of this ITN, the contractor will be responsible for drawing samples from the populations
of students, for calibrating field-test items, for constructing the new test forms for the future administrations,
and for Scaling, Equating, and Scoring (SES) of the assessment system. The system is complex and requires
the contractor to provide knowledgeable and experienced staff, including psychometricians and programmers.
Other than the annual regular psychometric operations, such as sampling, test construction, and SES for the
assessment system, the contractor will conduct a set of special psychometric studies for these assessments,
described in Section 7.6.3.6. The contractor will also cooperate to conduct another set of psychometric
studies as Florida transitions from FCAT 2.0 and EOC Assessments to the Florida Standards Assessments
(FSA) systems. As the Department transitions to the FSA, the contractor will assist the Department in the
establishment of FCAT 2.0 and EOC equivalent score scales, appropriately calibrate items in the new
assessments, properly combine performance indicators, construct meaningful score scales, possibly establish
vertical scales for ELA/L and Mathematics assessments, assist in defining achievement level descriptors, build
cut scores with criterion-referenced meaning on the new scales, link the performance of Floridas students to
the scales of other assessments to make comparisons, and conduct other special analyses that may be
required by the Department or required for a peer review under the federal Elementary Secondary Education
Act (ESEA) program. This section of the ITN provides descriptions of related services and products and the
expected quality assurance steps that must be employed to ensure that all tested students receive
assessment results that are valid, reliable, and accurate.
Florida uses a Technical Advisory Committee (TAC) composed of selected Florida District Coordinators of
Assessment and selected experts from around the nation representing the areas of measurement and
statistics, students with disabilities, English language learners, and related alternate assessments. The TAC
members listen to issues and offer guidance on implementation of Floridas assessment program. Currently
Floridas TAC meets twice a year in March and in November. The contractor is responsible for convening
and conducting these meetings.
77
The contractor may utilize proprietary software for any of the annual regular psychometric activities or the
special psychometric studies for the assessment system during the transition to the FSA, but is required to
provide this software for use by all relevant staff, including the Department and any subcontractors. Whatever
software is selected must be able to handle the large number of test cases, including large calibration
samples, and also accommodate full-state calibration with numbers of records in excess of 200,000 at a time.
The contractor will be responsible for assembling a psychometric team to support work performed under this
contract consisting of the contractors psychometricians and programmers, and at least one (1) outside
entity subcontracted to participate in all SES activities to independently perform these particular
operations and verify all results. The Departments psychometric team consists of two psychometricians plus
three graduate interns pursuing their doctorate in educational measurement to be part of the team replicating
all SES activities and overseeing the entire process.
A separate procurement is being issued for the FCAT 2.0 and Florida EOC Assessments for subjects not
covered as part of this ITN (Science, Social Studies, etc.). The contractor selected as part of this ITN process
should anticipate coordinating with the contractor from the FCAT 2.0/Florida EOC Assessments procurement
for some of the tasks described in this scope of work.
7.6.1 Assessments
7.6.1.1
Florida Standards Summative Assessments (Scaling, Equating, and Scoring)

1. The contractor will develop and maintain test construction specifications describing the details
of test construction, including content and psychometric guidelines for building equivalent
forms. Item selection will be based on matching target test characteristics, test information
functions, and standard error curves in addition to meeting other content and psychometric
requirements, plus further constraints.
2. For the scaling of items, the Department prefers to use a three-parameter logistic Item
Response Theory (IRT) model for multiple choice items, two-parameter logistic IRT model for
gridded-response items, and two-parameter generalized partial credit model for constructedresponse items. For the equating of the tests to their base year scales, the Department
prefers to employ test characteristic curve equating using the Stocking & Lord procedure and
triangulating the equating with IRT based other equating procedures, such as Haebera,
Mean/Mean, and Mean/Sigma. For scaling student performance, the Department prefers to
use Maximum Likelihood Estimation to score examinee response patterns.
For
dimensionality, the Department supports using a variety of item types measuring various
student skills on the Florida Standards, but maintains validity for a single dimension. However,
these are just the Departments preferences for scaling, equating, scoring, and dimensionality.
The contractor must conduct thorough research studies and collect empirical evidence to
determine whether these options are psychometrically defensible, whether items measuring
various constructs can be combined into a single assessment instrument and scored, and
whether the preferred options can be implemented operationally to obtain assessment results
that are valid, reliable, and accurate. If research and empirical evidence(s) suggest the use of
a different set of techniques in scaling, equating, and scoring of the assessments, the
respondent should provide a detailed description of the measurement models proposed for
use in calibration, equating method for creating interchangeable scale scores, and scoring
algorithm for placing examinees onto a score scale. The contractor will be responsible for
designing sampling plans, drawing the samples from populations of examinees, and supplying
the data files to the scaling and equating team described in Section 7.6.2.1 so that the
required analyses can be performed in a timely manner. The contractor will be responsible for
calibrating the tests, placing item parameter estimates onto the base years scale, and
providing the final scaled item parameters for the purpose of scoring. After the Department
approves the final scaled item parameters, the contractor will score student responses, the
78
entity subcontracted to participate in all SES activities will independently verify the scale
scores, and the Departments psychometric team will replicate and verify the scale scores.
3. For the Assessments in Algebra 1, Algebra 2, and Geometry, the contractor must construct
multiple forms of these assessments by following content and psychometric requirements as
indicated in the test construction specifications. As for the scaling, equating, and scoring
process for these assessments, the Department desires to follow the same process as
indicated in the previous paragraph, but since there are multiple forms of the assessments at
this time, the Department prefers to concurrently calibrate these forms so that all item
parameter estimates can first be placed onto a common scale, and then, equating these item
parameter estimates to the base year scale via anchor items. In other words, equivalency of
the multiple forms of new tests is verified through a concurrent calibration, plus post-equating
via common items between the item bank and the new test forms. However, as indicated
earlier, the contractor must show evidence that the Departments preferences are
psychometrically defensible and operationally feasible. The respondent may include in the
reply a different proposal for scaling, equating, and scoring of these assessments to obtain
assessment results that are valid, reliable, and accurate. Also, when multiple forms of these
assessments are being created, the contractor must construct the new test forms that are
sufficiently different in an effort to discourage cheating and to ensure a valid measure of
student learning. In addition, the department will consider the use of pre-equated forms for
administrations other than the spring administration.
4. Because every effort is made to develop items and construct tests according to universal
design principles, we anticipate all Braille versions of the assessments will contain the same
items as the regular version of the tests. If they do not contain the same items, the contractor
will develop a separate scoring process to be approved by the Department and used by the
contractor and subcontractor.
7.6.1.2
Retake Assessments (Scaling, Equating, and Scoring)

The contractors content and psychometric teams will work together with the Departments content
and psychometric teams to select retake test forms, which may consist of a previously-used intact
form, or another proposed design that yields a test of similar difficulty to the operational form. The
inclusion of performance tasks in the retake form may be proposed, however, consideration will
also be given to forms that do not include performance tasks.
The contractor will administer the selected test form and will score student responses. The entity
subcontracted to participate in all SES activities must independently verify the scale scores
generated by the contractor. The same procedure will be repeated to select different forms for
subsequent years.
7.6.2 Scaling, Equating, and Scoring (Psychometrics)

1. The contractor will construct Scaling, Equating, and Scoring (SES) specifications for the assessments.
The specifications will describe the specific steps for implementing the scaling, equating, and scoring
requirements and will explain how, by when, and by whom all of the necessary steps will be completed
and verified. The specifications will also address the steps, procedures, and software required for
replication of the scaling, equating, and scoring procedures by the Department. The specifications will be
reviewed by the Department and TAC members, if necessary, and updated by the contractor for each
subsequent administration. The Department will provide the contractor with copies of existing
specification documents.
2. Before operational SES activities start, the contractor and other members of scaling and equating team
must conduct dry-run SES activities on mock data files to make sure all operations laid out in the SES
specifications can be successfully executed during the operational SES process. For this purpose, the
contractor will prepare mock data for the assessments using the State Student Results file (SSR), as
79
indicated in Section 7.5.6.1. The purpose of this mock data will be to provide a vehicle for the SES dry
run; i.e., a practice check of the operation of computer programs used in scaling, equating, creating
adjusted item parameters, and scoring. The psychometric staff for the identified calibration team will be
available for daily discussions and consultation throughout the parallel dry-run calibration activities.
3. It is the responsibility of the contractor to keep all the documents/electronic files from the scaling,
equating, and scoring activities, such as the classical and IRT item statistics, alternative equating
solutions summarizing the reasons why anchor items are removed or kept in the anchor set, the impact of
decisions on the transformation coefficients and scale scores, and the other pertinent decisions made by
the Department. The format of these documents must be proposed by the contractor and mutually
agreed upon by the contractor and the Department.
4. The SES specifications must include how the contractors subcontractor will independently verify the
scaling, equating, and scoring operations. For the entire SES process, the contractor will provide all of the
resources, including software, staff support, and data files to permit parallel calibration, equating, and scoring
of data during the same time period that contractor are producing the scaled item parameter estimates.
7.6.2.1
Scaling and Equating Team

The contractor will assist the Department in coordinating the work of the scaling and equating team
during dry-run activities and regular administrations each spring. The contractor is required to
subcontract with at least one outside entity to work on the scaling and equating team during all
SES activities. The contractors, the subcontractors, and the Departments psychometric staff will
each independently perform SES tasks to verify results. The specifications will clearly describe the
scaling and equating team consisting of the Departments, the contractors, and third partys
psychometricians and programmers, as well as the roles assigned to each entity. The
Departments psychometric team will oversee the entire SES process. Respondents should
indicate in their replies the qualified personnel/subcontractors who will provide these professional
services and include their vitae or other documentation of expertise and experience.
7.6.2.2
Selection of Calibration Samples

Anchor, core, and field-test items are calibrated on representative samples. Thus, the contractor is
responsible for designing sampling plans and drawing the representative samples from the
respective populations, which will include at least 65% of the respective population, including
approximately 90% of Miami-Dade. The contractor will also plan in a way so that a representative
set of approximately 5,000 students take anchor and field-tests forms. The contractors sampling
specifications must include analyzing the most recent SSR files and establishing a road map for
gathering the required data proven to represent the demographic characteristics of the state and
student performance. The specifications will also address the steps, procedures, and software
required for replication of the sampling procedures by the Department. The specifications will be
reviewed by the Department and TAC members, if necessary, and updated by the contractor for
each subsequent administration.
Before SES activities start, the contractor must also check and verify the representativeness of the
most recent data, and then provide the files to scaling and equating team members so that
necessary analyses can be performed in a timely fashion.
The contractor must repeat this process for each assessment administered each spring.
80
7.6.3 Technical Oversight and Special Studies

7.6.3.1
Contractors Role
The contractor must provide psychometric direction and oversight for all aspects of the assessment
programs described in this ITN. The psychometricians assigned to this project must have
extensive experience in state assessment programs. The psychometricians will plan, conduct,
analyze, and report on test construction, scaling, equating, scoring, and the special studies
described in this section.
7.6.3.2
Technical Report
Each year, the contractor will produce a technical report based on an overall analysis of that years
administrations. The technical report will also include a comparison of the characteristics of the
current test administration to previous administrations. The technical report will be reviewed by the
Department and by the Technical Advisory Committee (TAC) prior to completion of the final copy of
the report. The technical report will include a section on field-test administrations. The report will
include tabular and graphic displays of data to illustrate the characteristics and quality of test
scores. At a minimum, the technical report will include the topics listed below in the draft table of
contents.
A. DRAFT TECHNICAL REPORT TABLE OF CONTENTS
B. BACKGROUND OF ASSESSMENTS
C. ITEM DEVELOPMENT & TEST CONSTRUCTION
Item Development
Item Review
Field Testing
Item Statistics
o Classical Statistics
o Item Response Theory Statistics: Difficulty, Discrimination, and Guessing
o Differential Item Functioning (DIF) Statistics
o Model Fit
o Local Independence
o Dimensionality (factor analyses, etc.)
Test Construction
D. ADMINISTRATION
Eligibility
Administration Procedures
Accommodations
Test Security
Data Forensics Program
E. REPORTS
Appropriate Uses for Scores and Reports
Reports Provided
Description of Scores
Appropriate Score Uses
Cautions for Score Use
F. PERFORMANCE STANDARDS
Setting Performance Standards
Interim Performance Standards
81
G. SAMPLING
Rationale
Procedure
Representativeness
H. SCALING
Rationale
Measurement Models
Scale Scores
Scale Drift
Vertical Scaling
Checking IRT Assumptions (Model Fit, Local Dependence, Dimensionality, etc.)
I.
EQUATING
Rationale
Pre-Equating
Post-Equating
Development Procedure for Future Forms
Sources of Error Variances to Score Instability
Linking Assessments During Transitioning Years
Linking Assessments to Other Scales for Performance Comparision
J. RELIABILITY
Estimating Reliability
Standard Error of Measurement
Student Classification Accuracy and Consistency
Reliability for sub groups in the population
Reliability of year-to-year changes in school means
K. VALIDITY
Perspectives on Test Validity
Criterion Validity
Content and Curricular Validity
Construct Validity
Consequential Validity
L. CONSTRUCTED-RESPONSE SCORING
Scoring Process
Quality Control
Handscoring Reliability and Validity
Rater Effects
M. QUALITY CONTROL PROCEDURES
Quality Control for Test Construction
Quality Control for Non-Scannable Documents
Quality Control in Data Preparation
Quality Control in Production Control
Quality Control in Scanning
Quality Control in Editing and Data Input
Quality Control for Test Form Equating
Quality Control in Scoring and Reporting
82
N. SUMMARY STATISTICS
Item level statistics
Test level statistics
Review and comparision of raw scores, sub-scores, scale scores, distributions for total
and sub-populations across years to monitor the trends across years and sub-populations
Review and comparision of percent proficiency rates across years on different populations
Frequency distributions of student achievement (all students and by sub-group) across
years by grades and subject
Scale drift indicators
Correlations of student performance across the various subject areas tested
Following each field test or operational administration, differential item functioning (DIF) analyses,
to detect possible item bias, will be conducted, preferably using Mantzel-Haenszel method. DIF
analyses are conducted for Caucasian, African-American, and Hispanic racial/ethnic groups and by
gender. Values for items resulting from these analyses will be included in the item-banking
system. Changes in DIF values across administrations will be analyzed and presented in the
Technical Report.
The contractor will provide a detailed item analysis for each test administration and provide the
results in the annual Technical Report. There are several item level statistics, such as pointbiserial, corrected point-biserial, fit, etc. available in the current item bank. Thus, the contractor is
responsible for calculating these statistics following each administration and populating the findings
into their respective positions in the bank.
The Department will work with the contractor to determine the contents of the written report. The
contractor will provide several printed copies of the report and also an electronic version in the
format determined by the Department.
7.6.3.3
Transitioning from FCAT 2.0 and EOC Assessments to Florida Standards Assessments
(FSA) Linking the Scales
As the assessment system transitions to the FSA in spring 2015, the scales of the new FSA will not
yet have any meaning attached. Therefore, the Department requires technical assistance from the
contractor so that the scales of these assessment systems can be linked through a special and
defensible psychometric operation. The contractor will provide a detailed scale linking plan for
transitioning from FCAT 2.0 Reading & Mathematics and EOC Algebra 1 & Geometry to FSA.
Linking through a special psychometric operation may go beyond employing a simple linking
procedure, since each score unit on the two separate scales may have to be linked to each other
so that achievement levels can be determined for student and school accountability purposes. The
Department may directly engage additional contractors, or the contractors subcontractor (if
proposed) to triangulate the results.
The plan will include alternative options for linking, how the alternatives will be operationalized,
scoring plans, and procedures. The plan should include a comprehensive and detailed timeline for
all tasks and events. The plan will be reviewed and vetted by Florida TAC members after
submission, and may require revision as a result of this review. This review will address the
appropriateness and validity of the psychometric procedure selected for linking, as well as the
validity, reliability and defensibility of the linked scores.
The respondents reply should describe available psychometric methods for establishing interim
scores for high-stakes student-level decisions, including an explanation of the defensibility of the
proposed methods. During the transition years, as student performance on other assessments will
be used to determine the achievement of testing standards, the linking/concordant score studies
will be conducted by the contractor and concordant scores will be recommended to the
83
Department. The contractor selected through this ITN process will be expected to secure external
legal opinion regarding the defensibility of the linking method to be employed.
7.6.3.4
Implement Achievement Level Standard Setting Meetings

The contractor is responsible for facilitating the Departments process to establish achievement
levels and associated cut scores for the Florida Standards ELA/L, Mathematics, Algebra 1, Algebra
2, and Geometry Assessments; achievement levels, passing scores, retention cut for grade 3
ELA/L, graduation standards for ELA/L, and achievement levels and passing scores for the Algebra
1, Algebra 2, and Geometry Assessments in consultation with Florida educators and citizens.
The contractor will be responsible for developing a Standard Setting Plan for each standard-setting
process utilizing IRT item values and appropriate, reliable, valid, and defensible procedures. The
Standard Setting Plan will address the nature of the proficiency level standards, methods for
determining the standards, and procedures for validating and analyzing the quality of information
reported using the achievement levels. In addition to describing the general standard-setting
strategy, the Standard Setting Plan will describe in detail how participants will be identified and how
the standard-setting procedure will be implemented, such as the materials to be prepared by the
contractor, how test forms will be organized, and how the contractor will analyze the data and
present impact results.
The contractor will be responsible for organizing and implementing the standard-setting processes,
based on the Standard Setting Plans, and for assisting the Department in conducting the standardsetting meetings. Each standard-setting process will involve meetings of standard-setting
committees, which will involve Florida teachers and instructional leaders, followed by a reactor
meeting involving other Florida education stakeholders. The educator meetings in summer 2015
will result in content-based achievement level descriptions for each grade and recommended cut
scores for each test. The Department desires to present proposed cut scores to the State Board of
Education in January following each meeting; therefore the final report of activities of each meeting
should be available for the Department no later than November following each meeting. The final
report of standard setting activities should include in detail process and activities followed during
the standard setting, including a description of materials used, statistical background, participants,
changing of recommendations through rounds, and participant evaluation of meeting processes.
7.6.3.5
Vertical Scaling
The contractor will be responsible for conducting the vertical scaling for the Florida Standards
ELA/L and Mathematics Assessments. This scale provides for reporting growth continuously from
grades 3 to 10 in ELA/L and from grades 3 to 8 in Mathematics. In other words, each students
growth between adjacent grade levels will be calculated and reported on this scale as a type of
value-added score.
The contractors plan for vertical scaling, provided prior to the selection of items and construction of
forms for the appropriate administration, should recommend and describe the process of
determining hierarchical common content areas across grades, a recommended method for
vertical scaling that takes into account potential uses of the scale for NCLB and Floridas
accountability system, the data collection procedures, and how and by whom and by when all of
the necessary steps will be completed. The vertical scaling plan must be approved by the
Department and presented to the Technical Advisory Committee. After vertical scaling studies are
conducted, the contractor will prepare a technical report including a description of the process,
statistical background, methodology, the recommended scale, and characteristics of the scale.
The final report will be presented to the Technical Advisory Committee.
The contractor should anticipate replicating the vertical scaling study three years after the
completion of the initial scaling study. This work may occur in the optional renewal period and is
contingent on contract renewal.
84
7.6.3.6
Scaling and Equating Studies

As explained in Sections 7.6.1.1 and 7.6.1.2, the Florida Standards Summative Assessments will
be pre-equated during test construction and post-equated after each spring test administration.
The retake assessments, such as Grade 10 ELA/L or Algebra 1 Retake will be pre-equated only,
consisting of a previously-used intact form, or another proposed design that yields a test of similar
difficulty to the operational form. In other words, during the time frame of this ITN, there will be
several equating activities to put the new test forms on the same scale through a series of
intermediate equating procedures (known as equating chains). In this case, equating error from
each of these intermediate equating procedures may accumulate to a point where the
comparability of scale scores across time is questioned. For this reason, it is essential for the
Department to employ practices that ensure comparable scores across time in the presence of
equating chains. The contractor is required to submit a research plan to be conducted in spring
2018 to investigate the extent of scale drift on the assessments. The research plan(s) must be
approved by the Technical Advisory Committee and the Department, and the resulting report(s)
must be submitted to the Department and presented to the Technical Advisory Committee.
7.6.3.6.1 Investigate the Comparability between Devices Delivering Assessments
EOC Assessments will be administered online and delivered by several different
devices throughout the state since school districts are using many different machines
when administering the assessments, such as laptops and desktops. The Department
prefers to make sure these devices used in delivering the assessments are not
providing any advantage or disadvantage to the performance of students. As a result,
there is a need to investigate whether there is an impact of certain devices on the
assessment results. The contractor must determine the groups of testing devices,
based on the types of devices that work with the contractors platform, whose impact
will be investigated, create a plan for the study, present the plan to the Department staff
and TAC members, and upon approval from the Department, start investigating any
impact on item-level, test form-level, and reported score-level. The results of the study
and contractors recommendations must also be presented to both the Department staff
and TAC members.
7.6.3.6.2 Linking Florida Standards Assessments to Other Assessments Measuring the
Same Standards
The Department intends to compare Florida student performance on FSA to other
students performance on the same standards from other states, also representing the
nation. Thus, contractor must link the scales of Florida Standards Assessments to the
scales from other assessments measuring the same standards.
By August of 2015, the contractor will complete a comprehensive curriculum mapping
study to determine the degree of overlap between Floridas standards and those in
other states in order to determine the degree of alignment. The mapping study results
will be analyzed by the Department and contractors staff to determine if Floridas
standards are sufficiently like those of other states to ensure a valid assessment for
purposes of reporting comparative information or linked data on Florida Standards
Assessments.
Assuming there is adequate alignment, the contractor will work with the Department to
design, plan, and conduct a linking study. The contractors plan for linking should
recommend and describe the process of determining the linking methodologies, the
data collection procedures, and how and by whom and by when all of the necessary
steps will be completed. The linking plan must be approved by the Department and
presented to the Technical Advisory Committee. After the linking studies are
conducted, the contractor will prepare a technical report including a description of the
85
process, statistical background and methodology, and characteristics of the scale. The
final report will be presented to the Technical Advisory Committee. The contractor will
also generate data files with linked scores and prepare reports.
If a linking study requires collecting data from either a common set of students taking
both assessments or from different sets of students taking common set of items, the
contractor selected through this ITN will be responsible for scoring the test forms
administered to students for linking, including scanning/imaging, processing,
handscoring, calibration (generation of item statistics), scoring, and production of
student results files including linked scaled scores. The scoring will be conducted after
the completion of all scoring and reporting for the operational tests given in Florida
during spring of 2016. It is anticipated that the results files from the linking study will be
provided to the Department by July 2016.
As an outcome of the linking study, the contractor will provide scoring tables displaying
how Floridas scale scores are aligned to the ones of other assessments measuring the
same standards. Upon first use of these linking tables in reporting the results of the
spring 2017 Florida Standards Assessments, the contractor will provide psychometric
services to verify the reporting as well as the reliability and validity of the linked scores
for spring 2017.
The contractor should anticipate replicating the linking study three years after the
completion of the initial linking study. This work may occur in the optional renewal
period and is contingent on contract renewal.
7.6.3.6.3 Linking Florida Standards Assessments to International Assessments
The Department intends to compare Florida student performance on Florida Standards
Assessments to other students performance on International Assessments, such as
Program for International Student Assessment (PISA) and Trends in International
Mathematics and Science Study (TIMSS). Thus, contractor must link the scales of
Florida Standards Assessments to the scales from international assessments.
By August 2015, the contractor will develop a study plan for the linking Florida
Standards Assessments to International Assessments. The contractors plan will
include, review of the most promising and most feasible methods likely to be used in
international linking, discuss the technical feasibility of each method given the nature
and design of the Florida Standards Assessments and those of international
assessments, such as PISA and TIMSS, propose method(s) for linking the scales of
assessments, propose alternative ways for collecting data, explain the process of
establishing the linking relationship(s) in detail, and provide a comprehensive and
detailed timeline for all tasks and events. The contractor will also share the report with
the Department and its TAC members.
Upon approval from the Department, the contractor will finalize the plan for the linking
and implement all of the activities. The contractor selected through this ITN will be
responsible for scoring the test forms administered to students for linking, including
scanning/imaging, processing, handscoring, calibration (generation of item statistics),
scoring, and production of student results files including linked scaled scores. The
scoring will be conducted after the completion of all scoring and reporting for the
operational tests given in Florida during spring of 2016. It is anticipated that the results
files from the linking study will be provided to the Department by August 2016.
As an outcome of the linking study, the contractor will provide scoring tables displaying
how Floridas scale scores are aligned to the ones of international assessments. Upon
first use of these linking tables in reporting the results of the spring 2017 Florida
86
Standards Assessments, the contractor will provide psychometric services to verify the
reporting as well as the reliability and validity of the linked scores for spring 2017.
7.7.0 Program Management
Respondents should provide their overall description of the approach to manage, implement, and support
Floridas assessment program that specifically addresses the entire scope of work in this ITN. Respondents
should identify and describe all resources available to support all program activities. A management plan for
the contract should be incorporated into the respondents reply. In the event that multiple contracts are
awarded, additional program management capacity will be required to coordinate among the other contractors
and the Department.
Respondents should designate a Senior Program Manager who will have overall, daily responsibility for all
processes and deliverables under the full scope of this program.
The Chief of the Departments Bureau of K-12 Assessment will be assigned as the program manager of the
contract resulting from this ITN under the overall supervision of the Director of Assessment, Division of
Accountability, Research and Measurement.
7.7.1 Provisions Governing the Work of the Contractor
The procedural requirements identified in this section govern the work of the contractor. This section
establishes specific requirements and general principles by which the program will be managed.
The work described in Sections 7.2.0 through 7.6.0 reflects design, development, validation, and
implementation requirements of Floridas standards-based testing program. In preparing replies, respondents
should consider the following provisions governing the activities required by these sections.
a. All of the processes used by the contractor to complete the work identified in Sections 7.2.0 through 7.7.0
must be approved by the Department. All products must be approved by the Department at milestone
stages of development as described in the Production Schedule developed at the beginning of the contract.
The Department reserves the right to specify details of tasks and products.
b. Design control of all aspects of this program remains the responsibility of the Department. Changes in the
design made after the award of the contract will be at the discretion of the Department.
c.
The Department reserves the right to require changes in materials, produced in any medium, at any stage
of the program prior to final printing, production, or deployment.
d. Products and processes are modified to some extent for each assessment. For example, modifications
may be made to the design, format, or wording of printed and online products from one administration to the
next. All such modifications are to be expected as part of the scope of this program and will be at
contractor expense. Modifications may also be required for other products and procedures, such as data
analyses, file formats, and report designs. Consequently, all printed products developed under this contract
are to be only produced in quantities for the particular assessment, unless otherwise specified in writing by
the Department.
7.7.2 Contractor Staff and Responsibilities
The work described in the ITN constitutes a large, complex, challenging program that requires year-round
activity and close attention to overlapping tasks for different years and the impact on project resources. The
contractor is expected to provide a sufficient number of highly-qualified personnel to work closely with the
Department to manage the contract. Respondents should carefully consider whether they have sufficient and
knowledgeable human resources to complete the work activities of the ITN and should demonstrate in their
replies that they have sufficient and knowledgeable human resources to complete program work. Companies
should consider bidding only if they are committed to assigning the number and quality of staff and staff time
required to complete a program of this magnitude.
The contractor must assign professional staff members who are qualified, experienced, and capable of
providing technical assistance to the Department. Please note the following staff requirements:
87
a. The contractor will assign at least one (1) full-time senior program manager, four (4) full-time
project managers, and two (2) full-time administrative assistants. The full-time senior program
manager will concentrate on ensuring quality and timeliness of the full range of services and products
required by this ITN. The contractors management team is expected to lead project and support staff
and to manage implementation activities throughout the contract period. Although this section outlines
the minimum staffing requirements for this program, the contractor must provide adequate staff to
maintain the highest quality products and must deliver these products according to the Departments
timelines.
b. It is anticipated that the four (4) full-time project managers will focus on (a) test development, (b) test
administration, (c) scoring and reporting, and (d) computer-based testing. Respondents may propose a
modification to this project assignment for the managers.
c. Implementing the program requires considerable administrative activities involving the maintenance of
schedules, the development of specifications, production of management reports and documents,
arrangement of meeting and travel for educators and Department staff, and communication with district
and other personnel. Administrative assistants are necessary to serve in a dedicated role as meeting
organizers and as direct contacts for the many Florida citizens that may attend assessment meetings
throughout the year. The Department requires a single point of contact to discuss each meeting's
requirements and venue options and to communicate with participants throughout the travel process, and
to address issues that may require late-notice changes in travel plans.
d. The managers should be members of the regular organizational staff with experience within the company
or institution in coordinating the types of activities identified by this ITN. In their replies, respondents are
expected to demonstrate that the individuals identified as managers have sufficient authority across
departments within the organization to ensure that the work of the contract has the necessary priority to
be completed with the highest quality and on time.
e. Respondents should include pertinent staffing charts in their replies that show the regular institutional
organization staffing and the proposed organization of program staff for this work. In these charts, the
full-time equivalency devoted to work under the contract for each staff member with significant
responsibilities will be identified. Persons proposed to fill all key positions will be identified by name.
f.
Respondents should indicate, by name, the professional personnel to be responsible for major contract
activities, with an estimation of the amount of time, as a percent of full-time, each person will devote to
each task. This information will be included in a separate chart in the management reply. An appendix to
the technical reply should contain one (1) page vitae for each of the professional personnel, including
subcontractors, to be assigned to the program, indicating relevant educational background and
professional experience.
g. In a separate chart(s), respondents should identify the full extent of staff resources to be allocated to the
program to conduct activities related to the item and test development; computer-based test system and
content support; ancillary materials development; scanning, imaging, data processing, and reporting
activities; and scoring, equating, and technical analysis/special studies activities for each year of the
program. The chart should identify the major activities by month and the number of staff by category
and the amount of time in full-time equivalencies (equivalent person-days) assigned to complete the
activities.
h. In addition, the contractor must provide for five (5) contractor staff positions funded through the
reimbursable category and described in Section 7.7.9.6.
88
7.7.3 Subcontractors
In addition to any subcontractors required in this ITN, the prime contractor may choose to employ one (1) or
more additional subcontractors for the completion of these tasks. If the respondent proposes to employ a
subcontractor(s), the qualifications and experience of the subcontractor(s) should be documented in the reply
at the same level of detail as those of the respondent.
The prime contractor will assume responsibility for all services offered in the reply whether or not they are
performed or produced by the contractor or by subcontractors. The Department will consider the prime
contractor to be the sole point of contact for contractual matters, including payment of any and all charges
resulting from the contract.
The respondent should describe how the work of all subcontractors will be monitored and guided to ensure
adherence to requirements in this ITN, including resulting specifications and schedules. The contractor will
provide for quality assurance activities in oversight of the subcontractors performance.
A separate chart in the reply should identify all of the subcontractors proposed to be involved in the program,
the assessment components they will be involved with, and the specific services they will provide.
Throughout the life of the contract, any proposed changes in subcontractors must be approved by the
Department. The contractor will provide the Department with a qualifications and experience summary for
review and consideration for acceptance in advance of a proposed subcontractors start of work on program
processes or deliverables. As part of the Annual Report, all subcontractors, their services, and Departmental
approval status will be listed.
A detailed description of the subcontracted services to be provided under the ensuing contract should be
included in the reply. All subcontracted services should be documented. Each service (e.g., printing, computer
operations, test development, psychometric services, legal counsel, quality control, reporting) should be fully
described, including resources, level of service, scheduling, and quality commitments. The respondent should
identify the proportion of materials, etc., to be generated by any and all involved subcontractors.
No contractor- or subcontractor-provided work related to customer service or to any secure document or data
will be provided by a persons or agencies physically located outside of the United States.
7.7.4 Program Management Communication
Weekly program management conference calls will be conducted at an agreed upon day and time in order to
ensure that all requirements and timelines are maintained and to address needs for problem solving and timesensitive adjustments. The Department may determine periodically that weekly, focused conference calls are
required as activities dictate. The contractor will provide a toll-free conference line for all calls between contractor
and Department staff required for the effective coordination of the assessment program.
To meet timelines for completing tasks, both the Department and the contractor will need to use secure
electronic communications, including, but not limited to, a secure FTP site, a website, a teleconferencing site,
etc., provided by the contractor in addition to overnight express delivery of materials as necessary. In
addition, the Department and the contractor will need access to a secure site that can be used with districts
and with subcontractors to transfer secure materials.
The contractor and the Department may determine that, for selected meetings, video teleconference meetings
will be more time and cost efficient than travel to a common meeting site. In this case, the contractor will
coordinate all logistics and provisions for the teleconference.
To assist Department staff with communication during on-site visits to the contractors offices or other work
sites, the contractor will provide access to lockable office space, including a telephone with a speaker, a fax
machine, and a computer with Internet access and a local printer.
89
7.7.5 Program Management Meetings

The successful operation of the program will require regular management meetings between the contractor and
Department staff. At the Departments request, the management meetings will take place in Tallahassee, the
contractors headquarters or other worksite, or in a location convenient and agreed upon by the contractor and the
Department. Contractors should anticipate that the first comprehensive program management meeting will take
place at the contractors headquarters within thirty (30) days of contract execution.
The contractor must expect that up to fifteen (15) of their staff will travel to attend approximately two (2)
comprehensive program management meetings per year. The contractor will also send all relevant staff to up to
four (4) additional annual management meetings that are project specific, such as the scoring and reporting
debrief. All relevant subcontractors must also attend these meetings. The contractor will be responsible for the
logistics, facilities, and travel costs of their staff and required subcontractors staff for all management meetings.
These meetings will also involve relevant Department staff. Provisions will be made for electronic participation of
any staff member unable to travel to these meetings. In the event that multiple contracts are awarded, the
contractor should expect part of the meetings will include all other contractors and Department staff, and that part
of the meetings will be reserved for one-on-one meetings between the Department and each contractor to focus
on topics and issues specific to each contractor.
The cost reply should include funding specifically for Department staff travel for program management meetings
and other oversight activities. Program oversight activities include work related to overall program management,
management of key processes, including development, test administration, scanning, handscoring, scoring,
programming, printing, and reporting. The contractor will be responsible for paying the cost of travel, lodging, and
per diem for up to ten (10) Department staff members to attend up to fifteen (15) trips annually. The cost of the
trips has been based on travel between Tallahassee and the contractors headquarters or between Tallahassee
and the contractors or subcontractors headquarters, scanning/processing sites, or scoring/handscoring sites.
Funds set aside for this purpose are described in Section 7.7.9.5.
7.7.6 Program Management Reports
Five types of reports are to be prepared by the contractor to facilitate and document program management
communications with the Department: (1) the Program Schedule, (2) the Weekly Management and Weekly Action
Item Reports, (3) a Technical Report, (4) Annual Program Review, including the Operational Addendum, and (5)
the Accounting Report. In addition to written reports, the Department may require the contractors participation
in presentations to Department senior management or other entities at places and times determined by the
Department.
1. The Program Schedule will be developed for each year of the program and will include both high-level
management summary information as well as specific dates associated with critical processes, milestones,
and deliverables for individual projects listed in the program management plan. The Program Schedule will
be derived from the program management plan at the beginning of each test administration year, and will be
updated quarterly. The Department expects submission of the Program Schedule in Microsoft Excel. If
significant changes occur between quarterly submissions of the program schedule, the Department may
request an updated schedule reflecting the changes at any time.
2. The contractor will provide a Weekly Management and Action Item Report to the Department. A template for
this report will be provided to the contractor, and the Department reserves the right to modify the design,
format, and content of this report at any time. The Weekly Management Report and Action Item Report will
report the status of recent and upcoming tasks, highlighting those leading up to deliverables and critical tasks,
report progress on program issues, identify unresolved issues, and maintain a list of requests requiring
change orders or amendments. The weekly report will be provided to the Department by no later than noon
one day prior to the weekly program management conference calls and will reflect the previous weeks work.
3. The contractor will produce a Technical Report and provide the initial draft document to the Department by
September 30 of each year of the contract arising from this ITN. The Department will work with the
90
contractor to identify the specific information to be provided in the Technical Report each year. A
proposed report structure is provided in Section 7.6.3.2.
4. The contractor will produce two copies of the Annual Program Review each year to be provided only on
CDs/DVDs/jump drives inside plastic sleeves in loose-leaf binders. Along with other materials described
below, the Program Review will include a record of the contractors services and deliverables in response to
contractual/ITN requirements. This review must identify major program accomplishments and successes, as
well as areas of the program that require changes and improvement in the upcoming year. The review is
intended to be a history of program events and to describe recommended improvements or changes for
subsequent years. The annual review will be based on information gathered by the contractors program and
project management staff throughout the year. The review will address all aspects of program service and
products including management, technology, interpretive products, communications, development, test
production, test administration, materials shipping/pickup, scoring, and reporting processes. It will also follow
the annual administrative cycle, beginning with fall and ending with the following summer.
The Annual Program Review will include:
details of meetings held with educators (dates, locations, purpose)
milestones attained/deliverables completed for each project area
problems encountered, root cause analyses, and corrective action(s) taken
performance metrics reports as required by the Department noting changes over time
The Annual Program Review will include electronic files of deliverables and documentation on services
required by this ITN, beginning with the fall test administration and ending with the summer test
administration. It will include:
PDFs of the test and answer books,
PDFs of all screenshots for all computer-based test approved forms,
PDFs of all passage booklets, reference sheets, large print test books, one-item-per-page test books,
large print reference sheets, large print response grid instruction sheets, and CBT work folders,
PDFs (untagged and tagged, if applicable) of all interpretive and ancillary products,
Native application files of all interpretive and ancillary products,
PDFs, or other appropriate file formats such as video, of all other products, including memoranda and
report templates,
Missing materials and resolution reports,
Final Program Schedule reflecting planned vs. actual dates of task completion,
Weekly Management and Action Item reports,
File Layouts, and
Specifications documents.
5. The collected Accounting Reports will provide a summary of the Quarterly Accounting Reports, including
expenditures by program task and level of effort as well as expenditures and balances of Reimbursable
Funding Categories identified in Section 7.7.9, including balances resulting from executed change orders and
contract amendments.
7.7.7 Meetings Involving Florida Educators and Other Citizens
The contractor will be responsible for arranging, organizing, and paying the cost of bringing participants to
specific meetings as identified in this ITN. This section identifies points at which meetings with groups of
Florida educators and other citizens will be conducted to propose development and/or management
guidelines, to draft products, or to review work and products produced by the contractor. The Department will
designate Florida educators, and in some instances Florida citizens or consultants from outside the state, to
participate in these meetings.
91
All meetings are facilitated by Department staff unless determined otherwise by the Department. Contractor
staff will attend the meetings with Florida educators, as required. The contractor is responsible for on-site meeting
organization, support, and management. Generally, the contractors manager or team leaders for the program
area and an administrative assistant responsible for meeting planning will be present at meetings for overall
management and support purposes. The contractor will also be expected to send staff appropriate to assist and
participate in each meeting. Generally, for all meetings involving separate subject/grade groups, the contractor will
provide the appropriate professional staff to take part in each of the separate subject/grade groups. The contractor
will be responsible for all expenses associated with contractor staff travel and participation. All expenses must be
in accordance with the Department of Financial Services Guidelines for State Expenditures which prohibits the
purchase of food and beverages and other unallowable costs. The Guideline may be accessed through the
following link: http://www.myfloridacfo.com/aadir/reference_guide/.
The contractor will be responsible for arranging, organizing, attending, and paying travel, lodging and per diem
costs required to bring these participants together for the meetings. For meetings to be held in Florida, they
will typically be in a central Florida location, a hub city (e.g., Jacksonville), or in Tallahassee.
Table 7.7.1 provides information about the number, expected length, approximate number of participants, and
estimated cost of such meetings.
Table 7.7.7 Meetings of Educators and Other Citizens
Meeting
Area
Item/ Passage
Bias
English
Language
Arts/ Literacy
(ELA/L),
Mathematics
7.3.4.1
Oct 2015,
2016, 2017
Oct
2018,
2019,
2020
Approx #
of
attendees/
ind mtg
30
Item/ Passage
Community
Sensitivity
ELA/L,
Mathematics
7.3.4.1
Apr 2015,
2016; May
2017
30
Jacksonville
or Central
Florida
Item Content
Review
ELA/L,
Mathematics
Test Items
7.3.4.2
1/
subject &
grade
Oct 2015,
2016, 2017
Jacksonville
or Central
Florida
Passage
Content
Review
ELA/L
7.3.4.2
Aug 2015,
2016, 2017
15
Jacksonville
or Central
Florida
Data Review
Mathematics
7.3.4.3
May or Jun
2015,
2016, 2017
Tallahassee
Standard
Setting
ELA/L and
Mathematics
7.6.3.4
1/
subject
Aug 2015
May
2018,
2019,
2020
Oct
2018,
2019,
2020
Aug
2018,
2019,
2020
May or
Jun
2018,
2019
N/A
300
Central FL
Standards
Reactor
ELA/L and
Mathematics
7.6.3.4
Aug 2015
N/A
30
Tallahassee
ITN
Ref.
Mtgs/
Year
Base
Contract
Renewal
Days/
Mtg
Likely Site
Jacksonville
or Central
Florida
92

Approx #
of
attendees/
ind mtg
25/subject/
meeting
ITN
Ref.
Mtgs/
Year
Base
Contract
Renewal
Days/
Mtg
3 per
subject/fi
rst year;
2 per
subject
after first
year
2
TBD
TBD
Spring and
Nov 2015,
2016, 2017
Spring
and Nov
2018,
2019
20
Tallahassee,
Jacksonville,
Central FL, or
site of FERA
Sep
2017;
May and
Sep
2018,
2019,
May
2020
Sep
2017,
2018,
2020
10/subject
per grade
(3
grades/cou
rses per
subject)
Tallahassee
Tallahassee
Meeting
Area
Content
Advisory
ELA/L,
Mathematics
7.3.4.4
Technical
Advisory
Committee
Technical
7.6.0
10
Field Test
Rangefinding
ELA/L,
Mathematics
7.5.4.5
2/
subject
May and
Sep 2015,
2016, May
2017
11
Operational
Rangefinder
Selection
ELA/L,
Mathematics
7.5.4.5
1/
subject
Sep 2015,
2016,
Likely Site
Tallahassee,
Jacksonville,
or Central FL
7.7.8 Communication between the Contractor and Districts

Florida has 67 school districts conforming to the boundaries of Floridas 67 counties. Within each school
district is a district coordinator of assessment assigned to work directly with the Department to implement
statewide assessment programs. All aspects of test administration and reporting are administered within the
districts through the district coordinator of assessment. Other district personnel involved in statewide test
administration include one testing coordinator per school and the teachers or guidance counselors who serve
as test administrators. While there are some testing sites that are not part of the public school system (e.g.,
private schools), most communication will still be directed through the district coordinator of assessment.
All communication between the contractor and school districts must be directed through or receive prior approval
by the Department. Memos required to implement various aspects of the assessment will be prepared by the
contractor and approved by the Department prior to release. The contractor will be required to email, mail, or fax
memoranda or sets of instructions to district coordinators of assessment as instructed by the Department.
Districts will be instructed to communicate directly with the contractor only for the purpose of clarifying meeting
travel logistics, ordering additional testing materials, correcting computer-based system problems, or to report
problems with the delivery/pickup of materials. For this purpose, the contractor will dedicate for the use of Floridas
district and school personnel a toll-free help desk telephone, operated Monday through Friday 6:00 AM through
8:00 PM, Eastern Time, a toll-free fax number, and a web-based help site available during the same hours cited for
the help-desk telephone. This customer service is distinct from that described in Section 7.4.10.
93
7.7.9 Reimbursable Funding Categories

Six funding categories are designated for specified program functions and may be used only for those
functions. Use of these funds requires authorization by the Department contract or program manager or
program area leads. Authorization may be provided by specific email or other written communication, or by
Department approval of a specific plan for a meeting(s) or activities requiring use of these funds. Amounts are
designated for administration tasks. Respondents should include these amounts in their cost replies.
Monies not expended from these funds will be deducted from final contract payment at the conclusion of the
contract during contract resolution. The reimbursable fund balances will be reported by the contractor
quarterly with each invoice submission in the Accounting Report. The amounts specified in this section may
be increased or decreased by an appropriately negotiated amendment at any time during the lifetime of the
contract. The Accounting Report will indicate expenditures from, and the balances remaining, in these funds.
The final contract invoice will be accompanied by the final balance sheet for these funds. The amounts
allocated for these funds should be identified separately in the respondents cost reply.
7.7.9.1
Consultant Services
The contractor will provide to the Department technical advice and consultant service, as required,
from outside the contractors staff. The contractor will recommend and the Department will select
the consultant(s) to be used, who may or may not have been recommended by the contractor. A
contract between the consultant and the contractor will be required and will include the amount
allocated for honorarium, travel, lodging, and per diem as authorized by the Department.
Reimbursement for travel will be at rates authorized by Section 112.061, F. S., for travel and per
diem. The respondent should include in the reply the amount of $84,375.00 as a fixed amount for
the base contract period and $28,125.00 as a fixed amount for each year of the optional renewal
period.
7.7.9.2
Contingency Services
It is possible that unforeseen circumstances will make it either necessary or desirable to perform
tasks not specifically anticipated at this time, for example, data analyses beyond those described
herein, unexpected data processing, acquisition of additional materials, or programming for test
functionality not yet envisioned. These circumstances cannot be defined in advance, and the
Department must be able to request such extra services in a timely fashion so as not to impede
implementation of the assessments. In the reply, the respondent should include $84,375.00 as a
fixed amount for the full term of the base contract period and will include $28,125.00 as a fixed
amount for each year of the optional renewal period.
7.7.9.3
Authors Alterations and Design Control

The Department reserves the right to make changes in materials at any stage of the program prior
to final production or delivery. Any changes to the Department approved, production-ready product
(e.g., digital proofs or final computer-based test files) that are required due to contractor error will
be paid for by the contractor. Otherwise, changes made to the Department approved, productionready product will be paid for as authors alterations. If any alterations are necessary, the
Department will identify in writing to the contractor the alterations to be made. In the reply, the
respondent should include $84,375.00 as a fixed amount for the full term of the base contract
period and will include $28,125.00 as a fixed amount for each year of the optional renewal period.
7.7.9.4
Provision for Meetings with Florida Educators

The respondent will include a fixed amount to cover the costs of travel and per diem for Florida
educators, Florida citizens, and others participating in the meetings identified in Section 7.7.7. The
contractor will be responsible for the travel reimbursement for participants at rates authorized by
Section 112.061, F. S., for travel and per diem. In the reply, the respondent should include
$5,491,496.00 as a fixed amount for these meetings with Florida educators during the base contract
period and $1,830,498.00 for each year of the optional renewal period.
94
7.7.9.5
Provision for Meetings with Department Staff

A fixed amount will be included in the reply to cover the costs of travel and per diem for the
Department of Education staff or others participating in meetings with the contractor on behalf of
the Department. Reimbursement for staff will be at rates authorized by Section 112.061, F.S.
These funds will not be used to fund travel and per diem of the contractors staff. These funds will
be expended only with the approval of the Department. In the reply, the respondent should include
$438,750.00 as a fixed amount for ten (10) staff members to attend as many as fifteen (15)
meetings per year requiring four (4) travel days each during the contract during the base contract
period and $146,250.00 for each year of the optional renewal period.
7.7.9.6
Contractor Staff Positions

To improve work flow, work quality, and communications, the contractor will fund salary and
benefits for five (5) professional positions to work under the direction of the Bureau of K-12
Assessment within the Florida Department of Education. These positions will be three editors and
two Systems Project Consultants. The Department will work closely with the contractor to interview
and select individuals who are acceptable to both parties. The contractor will employ its standard
recruiting, screening, and hiring procedures to secure individuals to fill these positions. The
employees will be full-time employees of the contractor. The employees will receive benefits
typically provided to the contractors employee classification system. The employees will be paid
at a rate comparable to similarly skilled and experienced Department employees using the
contractors employee classification system.

The basic duties of the Computer-Based Test Editor (1) and the Materials Editors (2) include: (a)
write and edit materials for content, such as items, tests, reports, training materials, brochures,
scripts, or on-screen presentations; (b) develop, analyze, and select appropriate information and
present it in a form and at a level suitable for the intended audience; (c) review written products
and materials for conformity with standards of objectivity, style, and presentation; and, (d) perform
a variety of editorial duties related to editorial production. Candidates must possess one (1) year of
specialized experience directly related to these duties. Candidates may substitute two (2) full
academic years of related graduate level education in lieu of the work experience requirement.
The basic duties of the Systems Project Consultants (2) include: (a) participating in weekly
meetings and communicating issues between the FDOE and the contractors program managers,
(b) making suggestions for improvement to the contractors procedures to aid FDOE staff in data
verification activities, (c) assisting FDOE staff debugging SAS computer programs and analyzing
data files, (d) traveling, at the contractors expense, with FDOE staff to check scanning at the
contractors site, (e) assisting in the verification of reports of results using SAS computer programs,
(f) assisting in the verification of posting of files and reports on the portal, and (g) communicating
issues with District Coordinators of Assessment related to scoring and reporting activities and
issues. Candidates must have a thorough understanding of large scale standardized testing
programs and data verification techniques. The candidate must also have experience managing
data activities such as those required for this project, verifying data files, and using SAS, Excel,
and other Microsoft Office products.
The respondent should include in the response $1,800,000.00 as a total amount during the base
contract period and $600,000.00 for each year of the optional renewal period.
7.7.10
Final Delivery of Materials

The contractor agrees to deliver to the Department, upon request, all materials and products in all
forms that are developed for and used in conjunction with this program, including test books,
answer documents (paper and PDF), final electronic source files of interpretive products, software
source code, and ancillary materials, PDFs, and electronic files within 30 days following
acceptance by the Department of the final report for the program. Payment of the final program
95
invoice will not be made until all materials and certification of destruction, as appropriate, are
received and approved by the Department and final payment resolution is agreed to by both
parties.
At the completion of the contract, all data files, including those containing items or graphics,
remaining in the possession of the contractor will be destroyed once all handoffs are made to the
next contractor. Written verification of destruction will be provided to the Department as part of the
final contract report. Destruction will not take place until authorized by the Department.
7.8
DELIVERABLES (See Attached Section 7.8 Deliverables provided in an Excel Spreadsheet.)

SECTION 8 OPENING, EVALUATION AND AWARD
8.0
REPLY OPENING
TECHNICAL & PRICE REPLIES WILL BE OPENED BY THE DEPARTMENT'S BUREAU OF CONTRACTS,
GRANTS AND PROCUREMENT MANAGEMENT SERVICES PERSONNEL AT 325 WEST GAINES
STREET, 332 TURLINGTON BUILDING , TALLAHASSEE, FLORIDA, AS SPECIFIED IN SECTION 3.4
8.1
REPLY EVALUATION AND NEGOTIATION PROCESS

Using the evaluation criteria specified below, in accordance with Section 287.057, F.S., the Department shall
evaluate and rank responsive Replies and, at the Departments sole discretion, proceed to negotiate with one
or more Respondent(s) selected, as follows:
A. Selected Respondent(s) will be invited to commence contract negotiations. If necessary, the Department
will request revisions to the approach submitted by the selected Respondent(s) until it is satisfied that the
contract will serve the Departments needs. The process will continue until a contract is negotiated and
executed. The Department may in its sole discretion, award and enter into contracts with more than one
Contractor, if in the best interest of the state.
B. The Department reserves the right to negotiate with all responsive and responsible Respondents, serially
or concurrently, to determine the best-suited solution. The ranking of Replies indicates the perceived
overall benefits of the proposed solution, but the Department retains the discretion to negotiate with other
qualified Respondents as deemed appropriate.
C. Before award, the Department reserves the right to seek clarifications, to request Reply revisions, and to
request any information deemed necessary for proper evaluation of Replies. Respondents may be
requested to make a presentation, provide additional references, provide the opportunity for site visits,
etc. The Department reserves the right to require attendance by particular representatives of the
Respondent. Any written summary of presentations or demonstrations shall include a list of attendees, a
copy of the agenda, and copies of any visuals or handouts, and shall become part of the Respondents
Reply. Failure to provide requested information may result in rejection of the Reply.
D. The focus of the negotiations will be on achieving the solution that provides the best value to the state.
E. In submitting a Reply Respondent agrees to be bound to the terms of this ITN, however, the Department
reserves the right to negotiate different terms and related price adjustments if the Department determines
that it is in the states best interest to do so.
F. The Department reserves the right to reject any and all Replies, if the Department determines such action
is in the best interest of the state or the Department. The Department reserves the right to negotiate
concurrently or separately with competing Respondents. The Department reserves the right to accept
portions of a competing Respondents Reply and merge such portions into one project, including
96
contracting with the entities offering such portions. The Department reserves the right to waive minor
irregularities in Replies.
Criteria for Evaluation
The Department shall evaluate and rank responsive Technical Replies and score them on a scale of 1 to 100
using the following criteria. The Department anticipates awarding the Contract (if any award(s) is made) after
negotiations to the responsible and responsive vendor determined to provide the best value to the state.
Each Respondent will receive one score for all criteria, other than price, regardless of the number of
assessment components it proposes to deliver. This score will constitute the Total Technical Score for the
Respondent. Respondents will also receive a separate price evaluation for each assessment component it
submits a price for (A-E in Attachment 1). The Department will add the Total Technical Score for each
Respondent to the price evaluation score for an assessment component to determine the total score for the
Respondent for that assessment component. Respondents will be ranked for each assessment component in
this manner.
The following criteria will be used to evaluate and rank Replies.
A)
B)
C)
D)
Executive Summary
Qualifications and Experience
Technical Plan
Management Plan
Total
N/A
10 points
50 points
15 points
75 points
E)
Price
25 points
25 points
Total
Price analysis is conducted through the comparison of Price Replies submitted. The Department shall score
responsive Price Replies up to 25 points using the below.
Price analysis is conducted through the comparison of Price Replies submitted.
The method of awarding points for the Price Reply will be done in accordance with the provisions of Section
287.0572, F.S., which requires use of the present value methodology. The present value discount rate will be
used in the computations and evaluation. To determine that discount rate, use the rates identified in Release
H.15, Select Interest Rates (Weekly), available online at http://www.federalreserve.gov/releases/h15/. Use the
most recent copy of the Federal Reserve Bulletin published at the time of issuance of the ITN. See the
Critical Event Dates for the release date.
The maximum points will be awarded to the lowest acceptable Price Reply. Replies with higher costs will
receive the fraction of the maximum points proportional to the ratio of the lowest Price Reply to the higher
Price Reply. The fractional value of points to be assigned will be rounded to one decimal place. The criteria
for price evaluation shall be based upon the following formula:
(Low Price/Respondent's Price) x Price Points = Respondents Awarded Points
The total maximum number of points that can be earned in the evaluation process is 100 points.
8.2
EVALUATIONS
The ITN is designed to assess the most points to the Respondent presenting the best solution for the required
services. The Evaluation Committee will consider only those Replies provided after the preliminary
administrative review (See SECTION 6.2.1) first completed by the Departments Bureau of Contracts, Grants
and Procurement Management Services.
Each member of the Evaluation Committee will be provided a copy of each Technical Reply. Replies will be
evaluated on the criteria established in the section above entitled "Criteria for Evaluation" in order to assure
97
that Replies are uniformly rated. The Evaluation Committee will assign points, utilizing the technical
evaluation criteria identified herein and the Procurement Office will complete a technical summary.
The Department reserves the right to request oral presentations/seek clarification on any Reply submitted.
Information requested and received will be evaluated by the Committee based on the criteria established in
SECTION 6.2.2 above. During this stage Respondents will be asked to provide any clarifications needed by
the Evaluation Committee to assist in evaluating their Reply. Information received in this stage will be added
to the Respondents Reply and evaluated as a part of the appropriate section above.
8.3
POSTING OF INTENDED AWARD

The intended award will be posted in the Bureau of Contracts, Grants and Procurement Management
Services, 325 West Gaines Street, 332 Turlington Building, Tallahassee, Florida and on the VBS and will
remain posted for a period of seventy-two (72) hours. (See SECTION 3.4 CRITICAL EVENT DATES.)
8.3.1 Protest of Intended Award

Any Respondent who is adversely affected by the Department's recommended award or intended decision
must file a written Intent to Protest with the Department at the address of posting. See SECTION 4.4
PROCUREMENT PROTESTS/NOTICE OF RIGHTS for protest information.
8.3.2 Inability to Post
If the Department is unable to post as defined above, the Department will post a public notice on the VBS.
The Department will provide written notification of any future posting in a timely manner.
8.4
AWARD OF THE CONTRACT

Services will be authorized to begin when the Contractor receives a fully executed contract from the
Departments Contract Administrator.
****
98
APPENDIX A:
TRANSITION TO COMPUTER-BASED TESTING AND ADDITIONAL TEST ADMINISTRATION DATES
All high school and End-of-Course assessments are currently computer-based assessments and are anticipated to
remain as such. Grade 11 English Language Arts/Literacy and Algebra 2 End-of-Course Assessments will be
implemented as computer-based assessments.
Table A.1: Transition to FSA Computer-Based Tests in Florida
Grades 3-8: ELA/L and Mathematics
Grade
Subject
2013-2014
2014-2015
2015-2016
2016-2017
2017-2018
ELA/L
PBT
PBT
PBT
PBT
Transition to
CBT
Mathematics
PBT
PBT
PBT
Transition to
CBT
CBT
ELA/L
PBT
PBT
Transition to
CBT
CBT
CBT
Mathematics
PBT
PBT
PBT
Transition to
CBT
CBT
ELA/L
PBT
Transition to
CBT
CBT
CBT
CBT
Mathematics
CBT
CBT
CBT
CBT
CBT
ELA/L
CBT
CBT
CBT
CBT
CBT
Mathematics
Transition to
CBT
CBT
CBT
CBT
CBT
ELA/L
CBT
CBT
CBT
CBT
CBT
Mathematics
PBT
Transition to
CBT
CBT
CBT
CBT
ELA/L
Transition to
CBT
CBT
CBT
CBT
CBT
Mathematics
PBT
PBT
Transition to
CBT
CBT
CBT
Key
CBT = Was computer-based test in prior year
Transition to CBT = First year of computer-based test in this grade/ content
PBT = Administered as a paper-based test
99
The following dates are for reference only and refer to assessments that are not included as part of this ITN.
Dates for the assessments that are included in this ITN are provided in Table 7.2.2.
Table A.2: Additional Administration Dates for FCAT 2.0 and Florida EOC Assessments
(Assessments not included in this ITN)
Florida End-of-Course Assessments (U.S. History,
FCAT 2.0
Civics, Biology 1, Algebra 1 Retake, Geometry Retake)
Fall
Spring
Year
Reading
Reading
Spring Science
Fall
Winter
Spring
Summer
Retake
Retake
Nov. 30 April 25May July 1122,
Oct. 516,
April 11
April 1115,
Sept. 1425,
2015-16
Dec. 18,
27, 2016
2016
2015
15, 2016
2016
2015
2015
April 1721,
Sept. 1223,
2016-17
NA
NA
Dec. 16,
26, 2017
2017
2017
2016
2016
April 1620,
Sept. 1122,
2017-18
NA
NA
Dec. 15,
25, 2018
2018
2018
2017
2017
April 1519,
Sept. 1021,
2018-19
NA
NA
Dec. 19,
24, 2019
2019
2019
2018
2018
April 1317,
Sept. 920,
Dec. 220, April 27May July 1324,
2019-201
NA
NA
2020
2019
2019
22, 2020
2020
April 1216,
Sept. 1425,
2020-211
NA
NA
Dec. 18,
21, 2021
2021
2021
2020
2020
1 Optional Renewal Period
100
APPENDIX B
CRITICAL AND EXTREMELY CRITICAL ACTIVITIES
Final dates for these activities and a final activities list will be determined in the negotiation process of this ITN and included in the final contract. The table below
describes the critical and extra critical tasks, the frequency for distribution, and any notes about the approximate delivery dates. Failure to provide these materials
by the agreed-upon dates will be cause for the assessment of liquidated damages.
Task Activity
Critical (C) or
Extra Critical
(XC)
Delivery of practice tests and sample test

materials to districts.
Frequency
Varies by deliverable
Delivery Notes (Approximations)

By November/December for assessments that are
introduced in the spring, regular sample tests (PDF or
computer-based) are delivered the initial year of the
assessment or the first year of a format change (i.e.,
transition from paper-based to computer-based).
ITN Ref.
7.3.7.5,
7.4.9.6
Special format print materials (e.g., Braille, large print)

are delivered upon request each year.
Delivery of the printed End-of-Course
test administration manuals and ancillary
materials to districts for the paper-based
and computer-based administrations.
Prior to each fall

administration (see note in
7.4.1 regarding the
possibility of 2 manuals
for 2014-15 school year)
Usually 70 calendar days before each administration.
7.4.1
Delivery of the printed ELA/L and

Mathematics test administration manuals
and ancillary materials to districts for the
paper-based and computer-based
administrations.
Prior to each spring

administration
7.4.1
Delivery of the printed ELA/L Retake test

administration manuals and ancillary
Prior to each fall

administration (beginning
7.4.1
101
Task Activity
Critical (C) or
Extra Critical
(XC)
materials to districts for the paper-based

and computer-based administrations.
Frequency
ITN Ref.
in fall 2015)
Delivery of the printed test materials to

districts.
XC
Prior to each
administration.
Usually 3 weeks before each administration.
7.4.1
Delivery of the preidentification labels to

districts.
XC
Prior to each
administration
Usually 3 weeks before each administration.
7.4.1
Delivery of the computer-based tests to

districts.
XC
Prior to each
administration
The first day of the test administration window, unless

the system requires pre-loading the assessments. (At
that point, they should be delivered in sufficient time
for the schools to load the test prior to the
administration.)
7.4.9
After each administration
Varies by results release deliverable. Initial results

delivered as early as two weeks after the end of the
administration.
For EOC administrations:
RD1-RD3 (state, district, and school reports)
must be delivered within two weeks of the
last day of the administration window, per
statute.
RD4 (hard copies of student results) must be
delivered within 1 week of RD1-3.
RD5 (late reporting) should be completed
within 2 months of RD4.
RD7 (demographic reports) will be delivered
in August/September for the previous school
year.
For ELA/L and Mathematics (Spring):
7.5.5
Delivery of test results to districts.
RD1 XC
Results Delivery (RD) details for 1-7 are

provided in Table 7.5.5. This includes
print and electronic deliveries.
RD2 XC
RD3 XC
RD4 XC
RD5 C
RD6 C
RD7 C
102
Task Activity
Critical (C) or
Extra Critical
(XC)
Frequency
RD1-RD3 (state, district, and school reports)

delivered between the end of April and the
middle of May.
RD4 (hard copies of student results) must be
delivered before the week of June 8, per
statute.
RD5 (late reporting) should be completed
within 3 months of RD4.
RD6 (writing image media) is usually
delivered mid-July.
RD7 (demographic reports) will be delivered
in August/September for the previous school
year.
103
ITN Ref.
APPENDIX C
FLORIDA PUBLIC SCHOOL STUDENT PARTICIPATION ESTIMATES, DISTRICT NUMBERS, AND
SPECIAL DISTRICT/SCHOOL NUMBERS
STUDENT PARTICIPATION ESTIMATES
Table C.1 indicates the anticipated number of students that would be tested in each grade-level test administration.
The number of students expected to participate in ELA/L Retake administrations (Table C.2) will be calculated as a
percentage of the grade 10 population.
Table C.3 indicates the expected test takers for each EOC Assessment. This calculation assumes 200,000 new
students for each spring cohort. (Cohort, for the purposes of this ITN, refers to a group of students that enters grade
9 and is enrolled at the high school for a total of four years.)
Table C.1 Anticipated Number of Students Participating in Assessment Administrations
Table C.1 Anticipated Number of Students Participating in Assessment Administrations
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Grade 9
Grade
10
Grade
11
2013-14
212,774
201,371
198,642
203,053
210,018
211,085
222,533
206,675
198,131
2014-15
2015-16
2016-17
217,735
225,524
223,753
202,712
207,553
215,105
203,183
204,609
209,564
202,436
207,154
208,854
206,962
206,010
210,865
213,634
210,031
209,364
228,413
229,570
226,066
216,413
218,707
219,248
202,282
205,215
206,493
2017-181
224,786
213,553
217,105
213,364
212,354
214,365
225,364
216,066
207,248
2018-191 225,132 214,786

2019-201 224,893 215,132
215,553
216,734
221,105
219,654
216,864
224,634
215,854
220,268
230,365
231,467
215,364
220,540
206,707
203,211
Table C.2 Anticipated Number of Students Participating in ELA/L Retake Administrations

Table C.2 Anticipated Number of Students Participating in ELA/L
Retake Administrations
Fall Retake
Spring Retake
1
ELA/L
65%
45%1
1 As a percent of anticipated Grade 10 enrollment for applicable years
from Table C.1.
104
Table C.3 Anticipated Number of Students Participating in EOC Administrations

Table C.3 Anticipated Number of Students Participating in EOC Administrations
School
Year
Spring
Summer
Fall
Winter
2015-16
Spring
Summer
Fall
Winter
2016-17
Spring
Summer
Fall
Winter
2017-181
Spring
Summer
Fall
Winter
2018-191
Spring
Summer
Fall
Winter
2019-201
Spring
Summer
2014-15
Algebra 1
200,000
100,000
75,000
56,250
242,188
131,641
98,730
74,048
255,536
141,652
106,239
79,679
259,759
144,820
106,239
79,679
259,759
144,820
106,239
79,679
259,759
144,820
Algebra 2
200,000
50,000
12,500
3,125
200,781
50,195
12,549
3,137
200,784
50,196
12,549
3,137
200,784
50,196
12,549
3,137
200,784
50,196
12,549
3,137
200,784
50,196
Geometry
200,000
50,000
12,500
3,125
200,781
50,195
12,549
3,137
200,784
50,196
12,549
3,137
200,784
50,196
12,549
3,137
200,784
50,196
12,549
3,137
200,784
50,196
TOTAL
600,000
200,000
100,000
62,500
643,750
232,031
123,828
80,322
657,104
242,044
131,337
85,954
661,328
245,212
131,337
85,954
661,328
245,212
131,337
85,954
661,328
245,212
105
DISTRICT NUMBERS AND SPECIAL DISTRICT/SCHOOL NUMBERS

NOTE: The districts/schools participating in each test administration may vary. For example, some administer only
the tests required for graduation.
01-Alachua through 67-Washington
68-
Florida School for the Deaf and the Blind (FSDB)

68-0011 Deaf Elementary School
68-0014 Blind Elementary School
68-0012 Deaf Middle School
68-0015 Blind Middle School
68-0013 Deaf High School
68-0016 Blind High School
68-0019 Career Education and Transition High School
69-0011 Okeechobee Youth Development Center

71- Florida Virtual Schools
71-0300 FLVS Full-Time K-8
71-0400 FLVS Full-Time 9-12
71-0500 FLVS Part-Time 6-8
71-0600 FLVS Part-Time 9-12
71-0700 FLVS Part-Time K-5
72-75 University Laboratory (Lab) Schools
72-0011 A.D. Henderson University School & FAU High School
72-0020 FAU/SLCSD Palm Pointe Educational Research School @ Tradition
73-0341 Florida State University School (FSUS)
73-0351 The Pembroke Pines Florida
74-0351 Florida A&M University (FAMU) Developmental Research School
75-0391 University of Florida (UF) Lab School
P.K. Yonge Developmental Research School
80- State Colleges with Adult High School Programs
80-0005 Daytona State College (Volusia)
80-0007 Florida State College at Jacksonville (Duval)
80-0011 Indian River State College (St. Lucie)
80-0020 Pensacola State College (Escambia)
80-0025 Seminole State College (Seminole)
80-0026 South Florida State College (Highlands)
97 Office of Independent Education (Florida Tax Credit Scholarship students)
98 Ahfachkee School
Special Programs (The following special programs are designations within each district. The XX is replaced by the
two-digit district number.)
XX-3518 McKay Scholarship Students
XX-9998 Home Education Program Students
106
APPENDIX D
SAMPLE TEST DESIGN
Table D.1 below is provided as an example of one possible test design that may meet the criteria described in
Section 7.3.0. Respondents may propose other designs, and a final design will be discussed in negotiation and
agreed upon between the Department and the selected contractor.
Table D.1 Example Testing Sessions Grades 3-8 and High School
Table D.1 Example Testing Sessions Grades 3-8 and High School
Content Area
English Language
Arts/Literacy
Mathematics
Specification
Session 1
Session 2
Session 3
Time
80 minutes
80 minutes
80 minutes
Item Types
Multiple Choice and

Short Constructed
Response
Extended Writing in
Response to Text
Time
90 minutes
Multiple Choice
and Short
Constructed
Response
90 minutes
Item Types
Multiple Choice,
Gridded/Fill-in
Response, Short
Constructed
Response, and
1-2 Extended
Performance Tasks
Multiple Choice,
Gridded/Fill-in
Response, Short
Constructed
Response, and
1-2 Extended
Performance
Tasks
107
APPENDIX E
PRODUCT SPECIFICATIONS AND DISTRIBUTION REQUIREMENTS
Appendix E is organized as follows:
Part 1 Operational Test Materials Product Specifications & Distribution Requirements
Part 2 Florida Department of Education PDF Guidelines for Section 508/ADA Documents
Part 3 Ancillary Materials Shipment Schedule & Print Quantities
APPENDIX E, Part 1
OPERATIONAL TEST MATERIALS
PRODUCT SPECIFICATIONS AND DISTRIBUTION REQUIREMENTS
Test Documents General Considerations
The contractor is responsible for producing camera-ready forms of these tests, printing the test documents,
and distributing them to school districts. The following requirements apply to all test documents:
A method to collect demographic information and other special information (e.g., invalidation)
o Preferred: Scannable document covers include the student demographic grids and other special
coding sections.
A method to track and log secure materials
o Preferred: Unique security barcode numbers will be printed on each book containing test questions.
Large Print and Braille (contracted and uncontracted) version of the test. Copies of the regular book (the same
form) must accompany the shipment of the large print and Braille materials. Braille notes also accompany the
Braille versions.
o Preferred: One form of each test (usually form 1) is produced in large print (minimum of 18 point
type on 14 x 17 approved paper) and in Braille.
Forms will be spiraled for random distribution.
Documents will be shrink-wrapped in packs of 5 and 25. Quantities smaller than 25 may be desirable and will
be approved by the Department when the document size makes lifting and packing difficult or hazardous.
Test documents must be high-quality and durable
o Preferred: Test documents are 8 x 11, saddle stitched or bound; 60# white opaque cougar or
approved equivalent paper will be used for answer documents and interior pages of test books.
Test documents should have high-quality, attractive printing and design
o Preferred: Covers are printed in one color of ink plus black; Interior pages are printed in one color,
some with both scannable and non-scannable ink; Color coding or other identification marks are
included on the spine of the documents to identify them when stacked.
Test Documents Specific Considerations:
1. Grades 3-11 ELA/L Test Documents
Up to 40 forms of scannable, sealed test and answer and/or answer documents
Number of document types may vary if different portions of a test are in separate documents (e.g.,
writing folders)
1 Reading Passage Booklet as a CBT test accommodation.
1 standalone CBT Worksheet for each CBT tester.
2. Grades 3-8 Mathematics Documents
Up to 40 forms of scannable, sealed test and answer and/or answer documents
performance tasks)
1 standalone CBT Work Folder for each CBT tester.
One standalone reference sheet per student, as applicable
108
3. Grade 10 ELA/L Retake

Preferably, one form of a scannable, sealed test and answer and/or answer document
performance tasks)
1 Reading Passage Booklet as a CBT test accommodation.
1 standalone CBT Worksheet for each CBT tester.
4. End-of-Course Tests
1 paper accommodation form of the Algebra 1 End-of-Course test and answer book (scannable,
with an outside seal).
1 paper accommodation form of the Geometry End-of-Course test and answer book (scannable,
1 paper accommodation form of the Algebra 2 End-of-Course test and answer book (scannable,
1 standalone CBT Work Folder for each CBT tester.
1 standalone reference sheet for each Algebra 1, Geometry, and Algebra 2 tester.
5. Other Accommodations
1 large print paper accommodation form for each test
1 Braille form divided into two volumes for each test
1 one-item-per-page accommodation form for each test
6. Accommodation Standalones
1 large print reference standalone sheet to accompany each applicable assessment form
1 large print response grid instructions standalone sheet to accompany each applicable assessment
form
6 pages of blank Braille paper to accompany each Braille assessment form
1 large print form of the CBT work folder for each CBT Mathematics assessment
1 large print form of the CBT worksheet for each CBT ELA/L assessment
Shipment Schedule and Print Quantities for ELA/L and Mathematics Assessments
All test materials are shipped in TM2
All test documents may contain up to 64 pages
All grades 3-11 tests will have up to 40 test forms
Grade 10 ELA/L Retake will have one test form
Schools will receive one test form (with standalones, as applicable) per preidentified student
Districts will receive 15% overage of all test materials
For paper-based administration subjects, 10% extra test material is produced
Shipment Schedule and Print Quantities for EOC Assessments
All test materials are shipped in TM2
All test documents may contain up to 64 pages
Spring EOC tests will have up to 4 unique test forms
Fall, Winter, and Summer EOC tests will have one test form each
Schools will receive one test form (with standalones, as applicable) per preidentified student
Districts will receive 15% overage of all test materials
For paper-based administration subjects, 10% extra test material is produced
109
APPENDIX E, Part 2
FLORIDA DEPARTMENT OF EDUCATION
PDF GUIDELINES FOR SECTION 508/ADA DOCUMENTS
The following guidelines may be revised throughout the contract to reflect changes in technology or requirements.
The contractor will be notified of any changes, as appropriate.
In order for PDF documents to be posted on the DOE website or any site that is affiliated with the DOE, they must
pass the Full Accessibility check without error in Adobe Acrobat 9.0 or X.
DOE uses Adobe Acrobat 9.0 and X to check accessibility of PDF documents. Currently, Adobe Acrobat 8.0 has
known issues with the accessibility checker. Although a document may pass the full check, this does not necessarily
mean the document is compliant according to DOE Web standards. The following additional checks are required:
1. PDF documents must have the correct reading order. The file must be able to be read by assistive technology in
a logical manner. This can be checked in several ways, which include the following items.
DOE uses at least two of the following to verify reading order:
a. Saving the file as an accessible .txt file and reading it to be sure it is correct
b. Using the reading order tool in Adobe Acrobat
c. If proficient using a screen reader (not the Adobe Acrobat read out loud), listen to the document
d. Using the Reflow view in Adobe Acrobat
e. Using the Order navigation panel
2. The document must have correct Tab order.
Tab order refers to how a user can tab through the content. The user should be able to tab through the
document in the order it is intended to be read. If the tab order is incorrect, the assistive technology user may
jump from one page to another and back and not realize it. Another example of incorrect tab order is the
practice of structuring tables or columned lists using tabs.
3. The document must have appropriate alt text on all images that have meaning. (Alt text means alternative text
that appropriately describes the image for an assistive technology user.)
a. Graphs, charts, flowcharts, etc., cannot simply have the word graph in alt text. It must contain enough
descriptive information for an unsighted user.
b. Graphics that do not have meaning need to be tagged as artifacts or backgrounds. If the primary purpose
of an image is decoration or if describing an image will take the listener out of the testing experience and
serve more as a distraction to the task at hand, then it would fall under this rule. During regular production
rounds, TDC content leads will determine which elements will be tagged as background.
c. Putting a blank space for the alt text area of an image in an attempt to get the screen reader to skip it is
not acceptable in most cases, as a screen reader will still say the word graphic. Tagging graphics that
have no meaning as artifacts or background will cause the screen reader to ignore them.
d. Graphs and charts need to be described completely to convey the same information to a listener as the
sighted user would get. If the graph is explained in detail in the content text of the document, you can refer
to that content in the alternate text. For example, if right below a pie chart is a paragraph explaining the
chart, then the alt text on the chart could simply state Pie Graph which is explained in detail below. This
is very helpful if the person converting the file to PDF is not the one who created it.
e. Alternate text should not be provided on anything except for image tags.
f. Links are not clickable in alternative text; therefore, including links in alternative text should be avoided.
Permission must be obtained from the DOE for special circumstances, such as the approach for tagging the
Periodic Table of the Elements in Interpretive Products.
4. If the State Seal, or any other image, is in the header and footer of a document, the first occurrence of the
header and footer needs to be tagged and readable by a screen reader. All other occurrences should be tagged
110
as artifacts or backgrounds. This will let the listener know that there is at least one state seal (or other image)
present, but does not repeat on every page that it appears. This rule also applies to headers and footers that
are repeated on each page of a document. Only new information in the header and footer selections, such as
page numbers, should be tagged.
5. Page numbers need to be tagged in such a way that the screen reader reads them. This is helpful in navigating
the document.
6. Tables must be tagged as tables, so they are read correctly. A common problem with tables is that they are
being tagged as paragraphs.
Tables and table components will be tagged as such: Table headers will have the <TH> tags, table data will
have <TD> tags , and table rows will have <TR> tags. The proper order for tagging a table is as follows:
<Table>, <TR>, and <TH>; or <Table>, <TR>, <TD>. The header section of a complex table (i.e, a table with
more than one header row) should begin with <THead>, and the body section should begin with <TBody>. To
check a table, compare the layout in the normal view against the layout in the View menu and select Reflow. If
the layout stays the same and does not shift, the table is properly tagged.
7. Documents greater than 10 pages in length should contain a clickable table of contents, except when it is not
feasible. The table of contents should contain the correct tag structure using <TOC> and <TOCl> tags. For
documents that contain a table of contents that is not clickable, the table of contents needs to be made into
Bookmarks in the PDF and each table of contents listing needs to be made clickable within the table of contents.
The Bookmarks Panel should appear on the left side of the screen for documents that do not contain a table of
contents but are greater than 10 pages in length. This Bookmarks Panel will act as the documents table of
contents. During regular production rounds, TDC content leads will determine which elements in TDC
documents will be bookmarked.
This is needed not only for easy navigation for those using assistive technologies, but it is a convenience to
sighted visitors as well, since it allows them to click on an item in the table of contents and be directed
immediately to the content of the document to which it refers.
8. Content headings must be tagged as headers and not tagged as paragraphs.
Headings need to be tagged with <H> tags (or Heading 1 <H1>, Heading 2 <H2>, etc.). This helps the listener
know what subject matter is contained beneath the header and allows for easier navigation.
9. Files must be saved (in a reduced file size) to Adobe Acrobat version 7.0 with "Fast Web View" enabled for
documents tagged in Adobe Acrobat 9.0 or X. (The DOE has discovered that converting files created in Adobe
Acrobat 9.0 to Adobe Acrobat 5.0 increases rather than decreases the file size, and converting files created in
Adobe Acrobat X to Adobe Acrobat 6.0 or earlier versions distorts the font in interpretive products, so this should
be avoided.) Reducing the file size ensures that users who have older versions of Acrobat are able to access
the document. This also decreases the file size for quicker download.
10. Links must be functional.
a. Hyperlinks must not be broken and must link to the correct page. This includes e-mail addresses that are
clickable.
b. If a link is split between two lines, both lines need to be linked correctly.
c. Hyperlinks need to be standard, which means in blue font and underlined. This allows for sighted users to
quickly locate links within the document.
11. Color should not inhibit the readability of text, and color alone is not to be used to convey meaning. Some
assistive technologies do not reproduce colored text, and users who are colorblind often cannot differentiate
every color.
a. If color is used to convey meaning for sighted users, add a symbol in front of the colored items as well. ag
this symbol to indicate the meaning of the colored item.
111
b. Under certain DOE Web Services approved circumstances, the color-coded information may be described
using white text that is not viewable to the sighted user. This invisible information must be tagged so that it
is read prior to the colored text.
12. The following information needs to be noted in the Document Properties:
TitleUsers can set up a screen reader to read the titles instead of the long file names.
SubjectSpecific to the document, and may or may not be the same as the document title.
KeywordsAs with other Web documents, these words are used for search engines and need to be specific to
the documents contents.
Language(Advanced tab) Specified to indicate the primary language of the document. The individual tag
properties need to indicate text within a document that may be of a different language (e.g., paragraph in
Spanish needs to specify Spanish and paragraph in Haitian Creole needs to specify French).
AuthorAt this time, the DOE is not requiring the inclusion of Author; however, if the name of the author is
pertinent to the overall document, it is suggested that a name be included.
13. Lists should be tagged correctly and not tagged as text. The correct order options for list tags are as follows:
<L>, <LI>, <LBody>; <L>, <LI>, <Lbl>, <LBody>; or <L>, <LI>, <LI_Label><LI_Title>. If lists have lists within list
items, they need to be tagged correctly. Always tag multiple-choice answer choices/options as lists. Sample
multiple-choice items appear in Sample Test Materials and in Item Specification books.
14. Forms that need to be filled out should contain the appropriate form fields and check boxes so users may fill
them out on the Web. Examples of forms should be indicated as such and do not need to contain form fields
and check boxes, because they are samples. The answer grids for sample testing materials are not considered
forms at this time for feasibility reasons. Form field properties need to be filled out so that the information that
needs to be entered into the form fields is clear. (The "Tooltip" in the form's properties is the portion of the tag
that is read by the screen reader.) If two form fields have the same "Name" in a document, then both fields will
populate when one of them is filled out; therefore, each form field must have a unique name in the form field's
properties. When tagging form fields, it is important to check the accuracy of the Tab order (see list item 2).
15. Pages cannot be blank. They must contain some text, such as Notes in the header or the disclaimer, This
page is intentionally blank.
16. Pages with content in landscape format should be presented in landscape format, so users may read the content
without having to print the page. In other words, the Pearson contractor shall rotate landscaped pages inserted
sideways so the content appears right side up to the viewer.
17. Words set in bold, underline, or italics may not be used alone to convey meaning. For example, a word set in
italics cannot have a coded meaning such as italics= story title.
a. Add a symbol in front of the italicized, bolded, or underlined information, and tag the symbol to indicate the
meaning italics, bolding, or underlining should convey.
b. Under certain DOE Web Services approved circumstances, the italicized, bolded, or underlined information
may be described using white text that is not viewable to the sighted user. This invisible information must
be tagged so that it is read prior to the colored text.
18. No text should be entered in the Actual Text field in the Touchup Properties box. If that is done, some readers
will only read the actual text and then stop instead of reading the entire document.
Note: DOE PDF accessibility guidelines were initially effective September 11, 2007. They were revised by DOE Web
Services to include Adobe Acrobat 9.0 on January 5, 2010, and by DOE Editorial staff on February 17, 2010, for
added clarifications and for the purposes of this document. Some DOE staff are now using Adobe Acrobat X, and the
Department is currently in the process of officially updating PDF accessibility guidelines to include Adobe Acrobat X
(3/27/12).
112
APPENDIX E, PART 3 Ancillary Materials Shipment Schedule & Print Quantities

Product
Shipment Pages
Number of Copies
State
District
Schools
Electronic
Delivery
(508/ADA
compliant)
Test Administration Manual
TM1
Up to 400
25
10% overage
1/20 students
Student PreID Labels (Wave 1)
TM2
N/A
N/A
N/A
1/student
Student PreID Rosters (Wave 1)

Document Count Forms (Preidentified)
TM2
N/A
N/A
N/A
TM2
N/A
N/A
N/A
Document Count Forms (NonPreidentified)
TM2
N/A
N/A
N/A
1 per subject
2 per
subject/grade
5 per
subject/grade
Online Comment Forms
N/A
N/A
N/A
N/A
N/A
Yes
Security Checklist
N/A
variable
Yes
District Packing List
TM1, TM2,
TM3
variable
Yes
School Packing List
TM1, TM2,
TM3
variable
Yes
District Materials Return Kit
TM2
N/A
School Materials Return Kit

Return Materials (e.g., envelopes,
boxes, special formats)
TM2
N/A
TM2
N/A
As necessary
As necessary
N/A
As necessary
As necessary
Bills of Lading
Paper Bands
TM2
N/A
10
15% over
1/20 student
documents
Student PreID Labels (Wave 2)
TM3
N/A
1/student (As
necessary)
TM3
N/A
1/subject (As
necessary)
As necessary
As necessary
(Spring ELA/L and Mathematics only)

Student PreID Rosters (Wave 2)
(Spring ELA/L and Mathematics only)
Misc. Memoranda
variable
Yes
113
APPENDIX F
TEST SECURITY REQUIREMENTS, STATUTE, AND RULE
Chapter 1008.24 of Florida Statutes and Florida State Board of Education Rule 6A-10.042 establish the requirement
that Florida Department of Education tests are to be maintained in a secure manner during development,
administration, and scoring in order to preserve the integrity of the tests. When not in use, all test materials are to be
kept in secure, locked storage. Individuals who have access to secure test materials are not to copy or otherwise
reproduce test questions or reveal test questions verbally or in writing. Persons who are involved in administering or
proctoring the tests or preparing examinees for the tests are not to participate in, direct, aid, counsel, assist in, or
encourage any activity which could result in the inaccurate measurement or reporting of the examinees'
achievement. Examinees' answers to questions are not to be interfered with in any way by persons administering or
scoring the tests. Persons violating test security requirements are guilty of a first degree misdemeanor, punishable
by a fine of not more than $1,000.00 or imprisonment for not more than 90 days, or both.
The security requirements and penalties established by the rule and statute must be provided by the contractor to
each person who has access to tests or test questions during the development, printing, administration, or
scoring of the tests.
A copy of the Statute and Rule is part of this appendix.
Restrictions on Printing for Security Purposes
The prospective contractors should indicate their ability to comply with the following conditions relative to printing of
the required tests. These conditions are necessary as a means of maintaining test security. Lack of compliance with
these conditions may subject the reply to rejection if the Department determines it is in its best interest to do so.
1. All test negatives and plates must be maintained under lock and key by the printing supervisor.
2. Unauthorized personnel must not be permitted access to the test negatives, plates, or copies.
3. All plates and negatives must be destroyed by the contractor upon completion of this contract.
4. The Department reserves the right to conduct on-site spot checks of the printing processes.
5. All press pull-ins, trim, and waste material must be shredded at the end of each day's press run by a person
authorized to do so by the contractor.
6. Each production run must be made under close supervision of the printing supervisor.
114
FLORIDA TEST SECURITY STATUTE

1008.24
Test administration and security.
(1)
A person may not knowingly and willfully violate test security rules adopted by the State Board of
Education for mandatory tests administered by or through the State Board of Education or the Commissioner of
Education to students, educators, or applicants for certification or administered by school districts pursuant to s.
1008.22, or, with respect to any such test, knowingly and willfully to:
(a)
Give examinees access to test questions prior to testing;
(b)
Copy, reproduce, or use in any manner inconsistent with test security rules all or any portion of any
secure test booklet;
(c)
Coach examinees during testing or alter or interfere with examinees responses in any way;
(d)
Make answer keys available to examinees;
(e)
Fail to follow security rules for distribution and return of secure test as directed, or fail to account for all
secure test materials before, during, and after testing;
(f)
Fail to follow test administration directions specified in the test administration manuals; or
(g)
Participate in, direct, aid, counsel, assist in, or encourage any of the acts prohibited in this section.
(2)
A person who violates this section commits a misdemeanor of the first degree, punishable as provided
in s. 775.082 or s. 775.083.
(3)
A school district may contract with qualified contractors to administer and proctor statewide
standardized assessments required under s. 1008.22 or assessments associated with Florida approved courses
under s. 1003.499, as approved by the Department of Education in accordance with rules of the State Board of
Education. Assessments may be administered or proctored by qualified contractors at sites that meet criteria
established by rules of the State Board of Education and adopted pursuant to ss. 120.536(1) and 120.54 to
implement the contracting requirements of this subsection.
(4) (a) A district school superintendent, a president of a public postsecondary educational institution, or a
president of a nonpublic postsecondary educational institution shall cooperate with the Commissioner of Education in
any investigation concerning the administration of a test administered pursuant to state statute or rule.
(b) The identity of a school or postsecondary educational institution, the personally identifiable information
of any personnel of any school district or postsecondary educational institution, or any specific allegations of
misconduct obtained or reported pursuant to an investigation conducted by the Department of Education of a testing
impropriety are confidential and exempt from s. 119.07(1) and s. 24(a), Art. I of the State Constitution until the
conclusion of the investigation or until such time as the investigation ceases to be active. For the purpose of this
paragraph, an investigation shall be deemed concluded upon a finding that no impropriety has occurred, upon the
conclusion of any resulting preliminary investigation pursuant to s. 1012.796, upon the completion of any resulting
investigation by a law enforcement agency, or upon the referral of the matter to an employer who has the authority to
take disciplinary action against an individual who is suspected of a testing impropriety. For the purpose of this
paragraph, an investigation shall be considered active so long as it is ongoing and there is a reasonable, good faith
anticipation that an administrative finding will be made in the foreseeable future. This paragraph is subject to the
Open Government Sunset Review Act in accordance with s. 119.15 and shall stand repealed on October 2, 2014,
unless reviewed and saved from repeal through reenactment by the Legislature.
(5)
Exceptional students with disabilities, as defined in s. 1003.01(3), shall have access to testing sites.
The Department of Education and each school district shall adopt policies that are necessary to ensure such access.
History.s. 370, ch. 2002-387; s. 1, ch. 2009-143; s. 7, ch. 2013-225.
115
Florida Test Security State Board of Education Rule

6A-10.042 Maintenance of Test Security.
(1) Tests implemented in accordance with the requirements of Sections 1004.93, 1008.22, 1008.29, 1008.30,
1012.55, and 1012.56, Florida Statutes, shall be maintained and administered in a secure manner such that the
integrity of the tests shall be preserved.
(a) Test questions shall be preserved in a secure manner by individuals who are developing and validating the
tests. Such individuals shall not reveal in any manner, verbally or in writing, the test questions under development.
(b) Tests or individual test questions shall not be revealed, copied, or otherwise reproduced by persons who are
involved in the administration, proctoring, or scoring of any test.
(c) Examinees shall not be assisted in answering test questions by any means by persons administering or
proctoring the administration of any test.
(d) Examinees answers to questions shall not be interfered with in any way by persons administering,
proctoring, or scoring the examinations.
(e) Examinees shall not be given answer keys by any person.
(f) Persons who are involved in administering or proctoring the tests or persons who teach or otherwise prepare
examinees for the tests shall not participate in, direct, aid, counsel, assist in, or encourage any activity which could
result in the inaccurate measurement or reporting of the examinees achievement.
(g) Each person who has access to tests or test questions during the development, printing, administration, or
scoring of the tests shall be informed of specifications for maintaining test security, the provisions in statute and rule
governing test security, and a description of the penalties for breaches of test security.
(h) During each test administration, school district and institutional test administration coordinators and
contractors employing test administrators and proctors shall ensure that required testing procedures are being
followed at all test administration sites. Officials from the Department are authorized to conduct unannounced
observations of test administration procedures at any test administration site to ensure that testing procedures are
being correctly followed.
(2) Test materials, including all test booklets and other materials containing secure test questions, answer keys,
and student responses, shall be kept secure and precisely accounted for in accordance with the procedures specified
in the examination program administration manuals and other communications provided by the Department. Such
procedures shall include but are not limited to the following:
(a) All test materials shall be kept in secure, locked storage prior to and after administration of any test.
(b) All test materials shall be precisely accounted for and written documentation kept by test administrators and
proctors for each point at which test materials are distributed and returned.
(c) Any discrepancies noted in the number or serial numbers of testing materials received from contractors shall
be reported to the Department by designated institutional or school district personnel prior to the administration of the
test.
(d) In the event that test materials are determined to be missing while in the possession of an institution or
school district, designated institutional or school district personnel shall investigate the cause of the discrepancy and
provide the Department with a report of the investigation within thirty (30) calendar days of the initiation of the
investigation. At a minimum, the report shall include the nature of the situation, the time and place of occurrence,
and the names of the persons involved in or witness to the occurrence. Officials from the Department are authorized
to conduct additional investigations.
(e) In those cases where the responsibility for secure destruction of certain test materials is assigned by the
Department to designated institutional or school district personnel, the responsible institutional or school district
116
representative shall certify in writing that such destruction was accomplished in a secure manner.
(f) In those cases where test materials are permitted by the Department to be maintained in an institution or
school district, the test materials shall be maintained in a secure manner as specified in the instructions provided by
the Department. Access to the materials shall be limited to the individuals and purposes specified by the
Department.
(3) In those situations where an employee of the educational institution, school district, or contractor, or an
employee of the Department suspects a student of cheating on a test or suspects other violations of the provisions of
this rule, a report shall be made to the department or test support contractor, as specified in the test administration
procedures, within ten (10) calendar days. The report shall include a description of the incident, the names of the
persons involved in or witness to the incident, and other information as appropriate. Officials from the Department
are authorized to conduct additional investigations.
(4) Violations of test security provisions shall be subject to penalties provided in statute and State Board Rules.
Specific Authority 1001.02(1), 1008(24)(1) FS. Law Implemented 1001.02, 1008.24 FS. HistoryNew 7-5-87, Amended 10-26-94.
117
APPENDIX G
TEST ADMINISTRATION MANUAL WORKFLOW
The production stages for test administration manuals are defined as follows and are first defined for the Department, and
then defined for the contractor. This is provided to help the respondent anticipate the level of work required on the review
rounds:
Table G.1 Test Administration Manual Workflow for FDOE and Contractor for Developing Florida-specific
Standalone Products
Table G.1 Test Administration Manual Workflow for FDOE and Contractor
Round
Task
oo. Identify which prior administrations TAMs to use as a starting point for each section of the
current manual.
FDOE
Manuscriptpp. Identify high-level changes, additions, and omissions to the scripts and other sections to be
(MS) Round included.
qq. Introduce known or potential policy updates.
rr. Verify accuracy of all FDOE policies outlined in TAM; apply content; policy updates, where
necessary.
Review all TAM text by comparing current TAM to most recent and most relevant prior
TAM(s); verify that relevant text from each section of the prior TAM is reflected in current
TAM.
Identify any new naming conventions that need to be incorporated (e.g., changing
calculator to approved four-function calculator); note any global edits.
FDOE
Verify accuracy of links, email addresses, phone numbers, statutes, dates, and schedules.
Round
Review instructions in Test Administrator (TA), School Coordinator (SC), and Technology
Coordinator (TC) sections to verify consistency (e.g., items packed in DAC box).
Confirm TAM accurately indicates which ancillaries will be posted; verify file format(s).
Add queries for contractor program team in green text boxes.
Include placeholders (text in purple text box) in sections to be addressed at Round 2
(e.g., outstanding policy decisions).
File clean-up before returning TAM to contractor:
Change internal queries from orange to purple.
Remove FDOE internal queries where possible.
118
Round
Task
Confirm that all Round 1 edits have been applied.
Confirm that queries to contractor from Round 1 have been addressed.
Replace all placeholders with content.
For assessments with CBT & PBT TAMs:
Review instructions in PBT TAM against instructions in CBT TAM for consistency.
FDOE
Round
2
Compare CBT and/or PBT test form to TAM to

o Verify accuracy of scripts (incl. electronic practice test scripts),
o Verify instructions,
o Verify seal configurations,
o Verify page references,
o Verify titles, and
o Verify tool descriptions.
Verify session schedule references in scripts (e.g., Session 2, Day 2); verify session length
references in script.
Ensure screen shots and instructions are up-to-date.
Respond to contractor queries (light blue text boxes); apply checkmarks to approve
suggested edits, as needed.
Ensure that contractor program team checks have been made; include reminders/queries
regarding checks that must be performed; schedule conference call/WebEx, if necessary, to
discuss questions/concerns regarding Program Team (PT) verification.
Conduct roundtable read-through, comparing to multiple prior TAMs for consistency.
Ensure FDOE specialists (e.g., individuals from other offices) verify information (e.g., any
updates to students to be tested, any updates to ESE accommodation information)
File clean-up before returning TAM to contractor:
Change internal queries from orange to purple.
Remove FDOE internal queries where possible.
119
Round
Task
Verify contractor program team checks have been performed by appropriate program team
members.
FDOE
Round
3
Check TOC, headers, footers, pagination, and internal page references for accuracy.
Perform a cold read of the entire TAM to identify any final text edits to be made.
Verify all FPOs (placeholders) have been replaced with accurate, up-to-date graphics.
Verify checklists and ancillaries in appendices for accuracy and consistency (i.e., that forms
are approved versions).
Review third-party proofreaders queries/suggested edits; apply checkmarks to approve
suggested edits or indicate revised text using verbatim text edits in yellow text boxes.
File clean-up before returning TAM to contractor.
Remove FDOE internal queries.
Round
Task
FDOE
Final
Round

Perform a final visual check of the document.
120
Round
Task
Draft new TAM using TAM Information Sheet and MS round FDOE markup in previous
TAM.
Communicate with FDOE about edits to be applied based on MS round markup. Identify
version(s) of previous TAM from which language has been pulled if not the approved final
version (e.g., language in x, y, and z sections of CBT TAM pulled form EOC TAM round 3).
Include queries to FDOE for sections that require confirmation, revision, or additional
information.
Contractor
Round
1
Insert FPO (placeholder) graphics; insert forms/images from contractor P&D program team
when available and note the following:
o Graphics should depict information for the applicable admin, such as correct start
date and logical start time.
o For CBT: Screenshots should be taken using the correct user profile (e.g., CBT
Coordinator).
o Sample students should have age appropriate date-of-birth fields and grade levels
should match the administration.
Perform a clean read of TAM and insert queries where text may need to be updated based
on comments at debrief, comment form feedback analysis, call center data, etc.; suggest
changes; highlight any tentative revisions for FDOE to accept/reject/revise.
File clean-up before posting TAM to FDOE:
Remove internal queries where possible.
121
Round
Task
Apply FDOE edits. If an FDOE edit seems incorrect or inappropriate, contact FDOE to inquire.
All edits should be applied unless, during contractors review, FDOE is consulted and agrees
to leave out edit.
Consult appropriate contractor program staff (per Florida Program Team org chart) and
acquire approval/sign-off on relevant script, technical, and P&D sections of TAM, including a
review of naming conventions for P&D materials.
Insert FPO (placeholder) graphics; insert forms/images from contractor P&D program team
when available and note the following:
Coordinator).
Perform a role play to check operational scripts against currently available test forms/books
and verify accuracy of instructions, page references, screen shots, tool descriptions, etc.
Contractor
Round
2
Verify session schedule references in scripts (e.g., Session 2, Day 2); verify session length
references in script.
Verify the online management system instructions by intended user role.
Verify the online management system and computer-based platform screen shots.
Verify emails, phone numbers, URLs, dates, and schedules.
Verify which ancillaries will be posted on the online management system, and in what file
format(s) (check against approved ancillaries matrix).
Perform a clean read and provide queries/suggested additional edits. Suggested edits should
be indicated using blue comment boxes for FDOE to approve or reject in the next review
round (i.e., should not be incorporated into the body of the TAM prior to FDOE approval and
should not be made using yellow text boxes, which could be mistaken for FDOE verbatim
edits).
Indicate content that has and has not yet been verified (e.g., add a text box indicating that
school box contents have not yet been verified).
Remove contractor internal queries.
Replace FDOE queries, as necessary, and ensure contractor responses are included.
122
Round
Task
Apply FDOE edits. If an FDOE edit seems incorrect or inappropriate, contact FDOE to
inquire. All edits should be applied unless, during contractors review, FDOE is consulted and
agrees to leave out edit.
Confirm all FPOs have been replaced with updated artwork.
Coordinator).
Pearson
Round
3
Verify online management system instructions are accurate and up to date.

Ensure all program team reviews are complete and that verification is indicated in TAM; where
suggested edits arise during PT verification, these edits should be placed in blue comment
boxes for FDOE approval prior to incorporation into text.
Perform final full review of entire TAM for content, formatting, consistency of information and
instructions across sections (e.g., TA, SC, and TC have same instructions and understanding
of responsibilities).
Consult appropriate contractor program staff (per Florida Program Team org chart) and
acquire approval/sign-off on relevant script, technical, and P&D sections of TAM.
Verify TOC, headers/footers, pagination, internal page references.
Round
Task
Confirm all edits have been applied.
Confirm that approved versions of all checklists and ancillaries are being used.
Perform a final, full clean read.
Contractor
Final
Round

Confirm that when the final version is posted to FDOE, no marks or comments are included except
where the copyeditor has made an unrequested edit or where FDOE had a query and contractor is
providing a response.
123
APPENDIX H
TEST ACCOMMODATIONS
Test Accommodations for Students with Disabilities (General Information)
As specified in this appendix, accommodations may be provided to students with disabilities enrolled in public
schools with current IEPs or Section 504 plans. Care must be taken to provide a student with only the
accommodations permitted for that student. In addition, students with disabilities must be provided the opportunity to
participate in practice activities for the tests with appropriate allowable accommodations.
Determination of appropriate accommodations in assessment situations for students with disabilities shall be based
on the individual needs of each student. Decisions on accommodations shall be made by the IEP or Section 504
team and recorded on the IEP or Section 504 plan. Students with disabilities should be oriented to any test situation
through test-taking instruction designed to familiarize them with testing format and procedures. This orientation
should take place near the time of testing. Guidelines recommended for making accommodation decisions include:
1. Accommodations should facilitate an accurate demonstration of what the student knows or can do.
2. Accommodations should not provide the student with an unfair advantage or interfere with the validity of a test;
accommodations must not change the underlying skills that are being measured by the test.
3. Accommodations must be the same or nearly the same as those needed and used by the student in completing
classroom instruction and assessment activities.
4. Accommodations must be necessary for enabling the student to demonstrate knowledge, ability, skill, or
mastery.
Students with disabilities who are not currently enrolled in public schools or receiving services through public school
programs and require accommodations in order to participate in a test administration may have access to
accommodations if the following information is provided:
Evidence that the student has been found eligible as a student with a disability as defined by s. 1003.01(3) (a),
F.S., or Rule 6A-19.001(6), FAC, and
Documentation that the requested accommodations are regularly used for instruction.
For further information, contact the Bureau of Exceptional Education and Student Services at 850-245-0475 or visit
the following website: www.fldoe.org/ese/fcatasd.asp.
Computer-Based Accommodations for Students with Disabilities
Students with disabilities may participate in CBT administrations using computer-based accommodations as
indicated on their IEPs or Section 504 plans. CBT accommodations may be included in the category of flexible
presentation accommodations for CBT administrations. They may be provided to students along with other
allowable test administration accommodations.
CBT accommodations must be accessible through the computer-based testing platform that will be used for nonaccommodated students. Accommodations should be administrator-selectable.
A. Large Print Font
A fixed large print font size (minimum 18-point) that is an enlargement of the regular type is available for
students who require it. Computer interface (scrollbar, buttons, cursor/pointer, etc.) is also enlarged accordingly.
B. Color Contrast
Options for variable font color and background color display preselected font and background color combinations
on the computer screen. Specific highlighter colors are also provided for students who require them.
The color combinations may include, but are not limited to, the following:
White Text/Black Background (Blue Tracking; Green Highlighter)
Black Text/Beige Background (Green Tracking; Yellow Highlighter)
124
Black Text/Light Green Background (Yellow Tracking; Orange Highlighter)
Without the use of color contrast, the screen displays the following:
Black Text/White Background (Blue Tracking; Blue Highlighter)
C. Zoom
Feature allows students to self-select print size and magnification. Zoom levels include: 1x, 1.25x, 1.5x, 1.75x,
2x, 2.25x, 2.5x, 2.75x, 3x, 3.5x, 4x, 5x, 6x, 7x, 8x, 10x. Computer interface (scrollbar, buttons, cursor/pointer,
etc.) also zooms accordingly.
D. Screen Reader
The screen reader provides text-to-speech audio presentation of directions, test items, and answer choices. The
Department will provide the contractor with screen reader specifications at the beginning of the contract detailing
the various requirements and procedures for read-aloud text and technology.
E. Assistive Devices
The assistive devices form allows the use of computer-based assistive devices with no other accommodated
features enabled for students who require assistive devices but do not require other accommodated CBT forms.
Paper-Based Accommodations for Students with Disabilities
Students with disabilities may participate in CBT administrations using paper-based accommodations as indicated on
their IEPs or Section 504 plans. Paper-based accommodations may be included in the category of flexible
presentation accommodations for CBT administrations and may be provided to students along with other allowable
test administration accommodations.
In order to receive paper-based test materials for any CBT administration, the requirement for a paper-based test OR
one or more of the following conditions should be documented on a students IEP or Section 504 plan:
The student cannot access a computer.
The student requires an accommodation that cannot be implemented in conjunction with a computer-based
administration (e.g., the student requires a specialized setting or uses adaptive furniture that cannot
accommodate a computer workstation).
The student requires an accommodated CBT form that is not available.
The student has a flexible scheduling/extended time accommodation that requires testing one session over more
than one day.
The student is hospital/homebound.
Other conditions, in addition to those listed above, may be considered by the students IEP or Section 504 plan team.
A. Regular Print
A regular print paper version of the test may be requested for a student who requires it.
B. Braille
A contracted or uncontracted Braille version may be requested for a student who uses Braille materials. Some
test items may be altered in format for Braille versions of the test as authorized by the Department. Test items
that have no application for the Braille reader will be deleted as authorized by the Department. Student
performance standards that cannot be assessed in the Braille format will be deleted according to the
requirements of s. 1008.22, F.S.
C. Large Print
A paper-based large print version of the test may be requested as a unique accommodation for eligible students
who are unable to access the computer-based large print or zoom accommodation via the computer-based
testing platform.
125
D. One-Item-Per-Page
One-item-per-page materials may be requested as a unique accommodation for paper-based tests for use by
students who require one test item on a page, fewer test items on a page, increased space between test items,
or true black-and-white print.
E. Reading Passage Booklet
Eligible students participating in computer-based Reading tests may be provided regular print or large print
booklets that contain the reading passages but do not contain the test items. Although Reading Passage
Booklets are included in the category of paper-based accommodations, students who require them will test on
the computer.
Administration Accommodations for Students with Disabilities
Administration accommodations must be provided as indicated on student IEPs or Section 504 plans. Listed below
are allowable administration accommodations for Floridas statewide assessments. Not all allowable
accommodations are listed.
A. Flexible Presentation
Paper-based test materials (regular print, braille, large print) and accommodated computer-based test forms
(large print font, color contrast, zoom, screen reader) may be requested for students who require them.
Oral presentation may be provided for all directions and for items/answer choices other than passages and
items/answer choices in Reading tests. Reading passages, items, and answer choices may not be read
aloud or presented orally, but must be read by the student through visual or tactile means.
Mathematics, science, and history items/answer choices may be read aloud by a test administrator
or presented orally via screen reader.
Signed presentation may be provided for all directions and for items/answer choices other than passages
and items/answer choices in Reading tests. Reading passages, items, and answer choices may not be
signed, but must be read by the student through visual or tactile means. The test administrator may
translate using the same method of sign language that the student regularly uses in the classroom, but must
be careful not to use signs that might lead the student to the correct response. In such cases, fingerspelling
may be used as an alternative.
For oral or signed presentation, the test administrator may sign or read aloud directions, items, and answer
choices, other than passages and items/answer choices in Reading tests, to the student in the manner that
is regularly used in the classroom. Examples of allowable oral/signed presentation include, but are not
limited to, reading/signing items to a group of students, reading/signing every item to a student individually,
reading/signing items only when a student requests.
Test directions may be repeated, clarified, or summarized.
A student may be provided with a copy of directions from the administration script that is read by the test
administrator.
A student may be allowed to demonstrate that he or she understands the directions (e.g., repeating or
paraphrasing).
A student may read aloud test items and answer choices to him/herself. This would require use of a device
such as a WhisperPhone, or would require that the student be tested in a separate setting so that other
students are not disturbed.
Verbal encouragement (e.g., keep working, make sure to answer every question) may be used; however,
it may not be used to cue a student regarding correct/incorrect responses.
Students may use magnification devices (e.g., CCTV/video magnifiers, reading loupes, handheld
magnifiers). Devices must be used without accessing image-upload features (e.g., devices with a memory
card must have the memory card removed during testing).
A student may use a straightedge to maintain or enhance visual attention to test items.
126
Portions of the test may be masked to direct attention to uncovered item(s).

Colored transparencies/overlays may be used.
Test books may be secured to a work area if no adhesives are used on scannable documents. If
necessary, scannable documents may also be secured with adhesives if responses are transcribed into
replacement documents.
Positioning tools, such as a reading stand, may be used.
A student may use a highlighter to highlight key words or phrases in directions, passages, test items, and
answer choices.
B. Flexible Responding
The student may use varied methods to respond to the test, including written, signed, and verbal responses.
(Written responses may include the use of devices such as the Graphic Aid for Mathematics or the
geoboard for students using braille.) A test administrator or proctor may transcribe student responses to the
format required by the test. Recorded responses must accurately reflect the response of the student,
without addition or enhancement by the test administrator or proctor.
A student may dictate responses to a proctor.
A student may use speech-to-text technology to record answers.
A student may indicate answers to test items using the mouse, keyboard, or assistive technology that uses
either device.
A student may use a computer switch to indicate answers.
A student may use a computer/alternative keyboard to indicate answers.
A student may use a pointing device to indicate answers.
A student may use other communication devices to indicate answers.
A student may use mathematics grids/guides to organize mathematical computation.
A student may use special paper such as raised line, shaded line, or color-coded (would require that
responses are then transcribed by school/district personnel).
Test administrators may check periodically to be sure that students are entering answer choices correctly on
the computer or marking in the correct spaces in the test book.
C. Flexible Scheduling
A student may be administered a test during several brief sessions, allowing frequent breaks during the test
sessions. Between sessions and during breaks (e.g., restroom breaks, lunch breaks), students must be
closely monitored to ensure that they do not share responses or change responses to items that were
already completed.
A student may use a specific time of day for specific sessions.
A student may be provided extended time to complete a test. Extended time must be provided in
accordance with the students IEP or Section 504 plan. Extended time is not unlimited time; it should
align with the accommodation used regularly in the students classroom instruction and assessments. The
student is not required to use all of the extended time that is allowed and may end the testing session prior
to the expiration of the extended time.
Other considerations for Scheduling accommodations when administering Floridas statewide assessments
to students with disabilities include:
Each test or test session must be completed within one school day. Students who have a flexible
scheduling accommodation that allows them to test over more than one day will be required to participate in
testing using paper-based test materials.
127
For students using paper-based test materials to test over more than one day, the following conditions must
be maintained to ensure the validity of the test administration across days:
The student may not be permitted to change his or her responses to items that were completed on a
previous day.
The test administrator must closely supervise the administration of the test on an individual basis to
ensure that answers from the previous day are not changed and that the student does not preview parts
of the test to be answered the following day.
The test administrator must use a paper clip or binder clip to secure the answer documents and prevent
the student from reviewing his or her answers from the previous day. ALL CLIPS MUST BE
REMOVED AFTER TESTING. Staples or tape should NOT be used on books containing student
answers.
D. Flexible Setting
A student may be administered a test in a familiar place with a test proctor present, and/or a familiar person
who has been appropriately trained may administer the test. Procedures for test security must also be
followed and precautions taken in order to ensure that the test remains secure.
A student may be administered a test individually or in a small group setting. A small group should be of a
size comparable to the normal instruction group size indicated on the students IEP or Section 504 plan.
If a student requires use of an accommodation that may disturb other students, the student should be tested
in a separate setting.
A student may use special lighting.
A student may use adaptive or special furniture.
Special acoustics, such as FM systems, may be used to enhance sound or special rooms may be used to
decrease auditory distractions.
The opportunity for movement may be increased or decreased.
Stimuli may be reduced (e.g., by limiting the number of items around the students desk or computer
station).
White noise/sound machines or approved music may be used to reduce auditory distractions.
E. Assistive Devices and Tools
Any student who requires software programs to ensure access to the computer and to approved assistive
technology (e.g., switch scanning software, specialized on-screen keyboards, mouse emulators) must be
able to use these devices with the computer-based testing platform.
A student may use varied assistive devices to access the computer (e.g., alternative keyboards, trackballs,
joysticks, switch scanning systems, touch screens, mouth sticks, head wands, eye-gaze or head control
systems).
Visual magnification and auditory amplification devices may be used.
Real coins may be used to represent coins depicted in test items. Coins may not be used as counting
devices.
For students with visual impairments, an abacus and products such as the Graphic Aid for Mathematics,
Math Window, and the geoboard may be used. These devices are approved substitutes for paper and
pencil computation.
Other assistive technology typically used by the student in classroom instruction may be used provided the
purpose of the testing is not violated. Implementation of assistive devices must ensure that test responses
are the independent work of the student.
Students who use sign language as their primary means of communication may use an English/sign or
sign/English translation dictionary. The dictionary must be similar to one used in the instructional setting
128
and may not contain definitions of words. The dictionary may contain the sign picture, the word, synonyms,
and an index.
Manipulative materials, including, but not limited to, counters, base-10 blocks, clock faces, or geometric
shapes, are not allowed to be used by any student during the test administration.
FOR BRAILLE MATERIALS ONLY: While the use of manipulatives is not an allowable accommodation,
occasionally an item in a Braille test will require the use of manipulatives to give the student a tactile
reference. If/when such items occur, test administrators will be provided with the instructions allowing
students to use the manipulatives for those specific test items only. This modification is approved by
the Department and applies only to students using Braille test materials.
Unique Accommodations
In accordance with Rule 6A-1.0943, FAC, school districts may request unique accommodations for individual
students with disabilities. Unique accommodations usually involve alterations of existing test materials. In addition,
they must be regularly used by the student for classroom instruction and must not alter the underlying content of the
test. Each unique accommodation must be approved by the Commissioner of Education or a designee prior to its
use. Written requests for unique accommodations must be submitted by the district assessment coordinator to the
Bureau of K-12 Student Assessment at the FDOE. Accommodation requests must be reviewed by district level staff
before being sent to the FDOE. In addition, the signatures of both the district exceptional student education (ESE)
coordinator and the district assessment coordinator are required.
Administration Accommodations for English Language Learners (ELLs)
Districts are required to offer accommodations to students identified as ELLs. Allowable accommodations for ELLs
are listed below. The test may be administered with any one of these modifications or a combination of
accommodations that are determined to be appropriate for the particular needs of ELLs. However, all testing, with or
without accommodations, must be completed during the prescribed testing dates.
A. Flexible Setting
ELLs may be offered the opportunity to be tested in a separate room with the English for Speakers of
Other Languages (ESOL) or heritage language teacher acting as test administrator. Parents must be
informed of this option for students not of legal age and shall be given the opportunity to select the
preferred method of test administration.
B. Flexible Scheduling
ELLs may take a test session during several brief periods within one school day; however, each test or
test session must be completed within one school day.
ELLs may be provided additional time; however, each test or test session must be completed within one
school day.
C. Assistance in Heritage Language
ELLs may be provided limited assistance by an ESOL or heritage language teacher using the students
heritage language for all directions and for items/answer choices other than passages and items/answer
choices in Reading tests. This should not be interpreted as permission to provide oral presentation of
items/answer choices in English or in the students heritage language.
The ESOL or heritage language teacher may answer student questions about the general test
directions in their heritage language. If the test is administered to a group of students, the teacher may
answer questions about directions for the benefit of the group. Questions of clarification from individual
students must be answered on an individual basis without disturbing other students.
Mathematics, Science, and History items: The teacher may answer specific questions about a word
or phrase that is confusing the student because of limited English proficiency, but is prohibited from
giving assistance that will help the student solve problems and answer test items. A students
129
questions must not be answered in a way that would lead the student to infer the correct answer to an
item.
D. Approved Dictionary
ELLs must have access to an English-to-heritage language translation dictionary and/or heritage
language-to-English translation dictionary, such as those made available to ELLs in an instructional
setting. However, a dictionary providing definitions written exclusively in the heritage language or in
English may not be provided.
130
APPENDIX I
DATA PRIVACY & SECURITY POLICY
TABLE OF CONTENTS
I.
INTRODUCTION
II. DEFINITIONS
III. PRIVACY OF PERSONAL INFORMATION
A. Basic Privacy Protections
B. Access to the Department PII
IV. INFORMATION SECURITY PROGRAM
A. Information Security Risk Assessment
B. Security Controls Implementation
C. Security Monitoring
D. Security Process Improvement
E. Breach Remediation
F. Organization, Responsibilities and Administration
G. Personnel Security Policy Overview
V. ENFORCEMENT
I.
INTRODUCTION
Personally Identifiable Information ("PII") on students, including information on their performance on the
assessments and limited demographic information, will be collected by the Contractor. Disclosure of PII to the
Department and its contractors is authorized by the Family Educational Rights and Privacy Act ("FERPA") for the
purposes of
(1) Conducting studies, for or on behalf of the Department and Floridas school districts, to develop, validate,
and administer predictive tests or to improve instruction (34 CFR 99.31(a)(6)(i)(A) & (C)); and
(2) Assisting in the evaluation of federal- and state-supported education programs and ensuring compliance
with federal legal requirements related to such programs in particular related to state-level assessments
and accountability systems (34 CFR 99.31(a)(3) & 99.35).
More specifically, the data will be used by the Department or by one or more of the Departments Contractors
to validate, pilot and field test, and improve the assessments;
to report assessment results back to the Department and Florida school districts in a form that is useful to
them;
to prepare reports on student performance for the Department and Florida school districts, and the public
(PII may not be included in public reports or in reports to states or school districts that were not the source
of the PII);
to analyze test results for purposes of accountability, including promotion and graduation decisions for
individual students; teacher and school leader evaluations; school accountability determinations;
determinations of principal and teacher professional development and support needs; and teaching,
learning, and program improvement; and
to carry out studies designed to improve instruction on behalf of the Department and Florida school districts.
Access to student data remains the legal responsibility of the Department and Florida school districts, in
accordance with FERPA.
131
This Policy describes, in general, (i) what steps the Department takes to protect PII that is accessed by or
provided to the Department or the Contractor; (ii) how that information is used; (iii) with whom the Department
shares that information, and (iv) the steps the Department takes to protect the security of that information.
Monitoring of the implementation of this Policy is the shared responsibility of the Department and the Contractor.
Each of the privacy and security provisions in this Policy is effective upon contract execution.
II. DEFINITIONS
Capitalized terms referenced herein but not otherwise defined shall have the meanings as set forth below:
Authorized User means an individual Department or Contractor employee authorized by the Department to
access PII maintained by the Department or the Department Contractor.
"Destroy" or "Destruction" means the act of ensuring the PII cannot be reused or reconstituted in a format which
could be used as originally intended and that the PII is virtually impossible to recover or is prohibitively expensive
to reconstitute in its original format.
FERPA means the Family Educational Rights and Privacy Act of 1974 (codified at 20 U.S.C. 1232g) and its
implementing regulations, as they may be amended from time to time. The regulations are issued by the U.S.
Department of Education, and are available at http://www2.ed.gov/policy/gen/reg/ferpa/index.html.
"Contractor" means each Department contractor, including any subcontractor(s), that may be required to
maintain or handle PII in the course of providing services in support of the Department.
"Personally Identifiable Information" (or "PII") means any information defined as personally identifiable
information under FERPA.
III. PRIVACY OF PERSONAL INFORMATION
A. Basic Privacy Protections
1. Compliance with Law and Policy. All PII uploaded to or made accessible to the Department or
Contractor will be handled, processed, stored, transmitted and protected in accordance with all
applicable federal data privacy and security laws (including FERPA) and with this Policy.
2. Training. Employees of the Department (including temporary and contract employees) and the
employees (including temporary and contract employees) of the Contractor are educated and trained
on the proper uses and disclosures of PII and the importance of information privacy and security. Such
training will include training for new employees and refresher training for current employees.
3. Personnel Guidelines. All the Department and Contractor employees are required to be aware of and
work to protect the confidentiality, privacy, and security of PII. The Department, the Contractor, and
their respective personnel shall not access PII except to comply with a legal obligation under federal or
state law, regulation, subpoena or action by the Department that requires such access, or if there is
legitimate need for the information to maintain data systems or to perform required services. The
following list provides a general description of the internal policies with which the Department, the
Contractor, and their respective personnel are required to comply:
Limit internal access to PII to the Department and Contractor personnel with proper authorization
and allow use and/or disclosure internally, when necessary, solely to personnel with a legitimate
need for the PII to carry out the educational purposes of the Department.
Allow access to PII by external parties only as authorized by the Department.
Require that materials containing PII in electronic form are stored solely within encrypted data
repositories and PII are not available on unencrypted shared drives that are used by other users or
on a local drive. PII must be encrypted in transit as well.
132
When PII is no longer needed, delete access to PII, in accordance with secure destruction
procedures that the Department will provide.
Permit the Department and Contractor employees to store or download information onto a local or
encrypted portable devices or storage only when necessary, and to create a written record for
retention verifying that the information is encrypted and stored in password-protected files, and that
devices containing the information have appropriate security settings in place (such as encryption,
firewall protection, anti-virus software and malware protection).
Require that any downloaded materials consisting of PII remain in the United States.
Prohibit the unencrypted transmission of information from the Department, or any other source of
PII, wirelessly or across a public network to any third party.
B. Access to PII
1. The Department makes the final determination regarding access to PII.
2. Parent Complaints. The Department shall be responsible for handling any complaints from parents (or
students 18 and over) regarding the disclosure or use or accuracy of PII.
3. Privacy Administrators. The Department will designate a privacy administrator who is responsible for
managing access to PII maintained by the Department or the Contractor by designating authorized
users, and for determining the scope of PII to which they have access. The Privacy Administrator shall
be responsible for making all administrative decisions regarding access to and use of PII.
IV. INFORMATION SECURITY PROGRAM
The security of the PII is of critical importance to the Department. The Department's IT Security Program
consists of technical, physical and administrative safeguards to protect the PII. The Department's IT Security
Program is designed to identify, manage and control the risks to system and data availability, integrity, and
confidentiality, and to ensure accountability for system actions. The Department's Security Program includes,
and the security program of each Contractor is required to include, the following key general processes which
may be more fully described in other materials as necessary:
A. Information Security Risk Assessment
The Department periodically conducts, and the Contractor is required to periodically conduct, an accurate
and thorough assessment of the potential risks and vulnerabilities to the confidentiality, integrity, and
availability of electronic, paper, or other records containing PII maintained by the Department and the
Contractor; to report such risks as promptly as possible to the Department or other official within the
Department designated to be responsible for data privacy and security compliance; and to implement
security measures sufficient to reduce identified risks and vulnerabilities. Such measures shall be
implemented based on the level of risks, capabilities, and operating requirements. These measures must
include as appropriate and reasonable the following safeguards:
1. Administrative Safeguards
a. Sanctions: Appropriate sanctions against the Department or contractor personnel who fail to
comply with the Department security policies and procedures, with the potential for criminal referral
if warranted.
b. System Monitoring: Procedures to regularly review records of information systems activity,
including maintaining access logs, access reports, security incident tracking reports, and periodic
access audits.
c. Security Oversight: Assignment of one or more appropriate senior officials within the Department
and the Contractor, as applicable, to be responsible for developing, implementing, and monitoring
of safeguards and security issues.
d. Appropriate Access: Procedures to determine that the access of the Department and the
Contractor personnel to PII is appropriate and meets a legitimate need to support their roles in
133
business or educational operations. Procedures for establishing appropriate authorization and

authentication mechanisms for the Department and the Contractor personnel who have access to
PII.
e. Employee and Contractor Supervision: Procedures for regularly monitoring and supervising the
Department and Contractor personnel who have access to PII.
f.
Access Termination: Procedures for terminating access to PII when employment ends, or when an
individual no long has a legitimate need for access.
g. Recording Requests and Disclosures: Disclosures of PII to, and requests for disclosures of PII
from, third parties -- other than employees of the Department or of the Contractor that provided the
PII to the Department -- are recorded.
2. Physical Safeguards
a. Access to PII: Procedures that grant access to PII by establishing, documenting, reviewing, and
modifying a users right of access to a workstation, software application/transaction, or process.
b. Awareness Training: On-going security awareness through training or other means that provide
the Department and the Contractor personnel (including management) with updates to security
procedures and policies (including guarding against, detecting, and reporting malicious software).
Awareness training should also address procedures for monitoring log-in attempts and reporting
discrepancies, as well as procedures for safeguarding passwords.
c. Incident Response Plan: Procedures for responding to, documenting, and mitigating where
practicable suspected or known incidents involving a possible breach of security and their
outcomes.
d. Physical Access: Procedures to limit physical access to PII and the facility or facilities in which
they are housed while ensuring that properly authorized access is allowed, including physical
barriers that require electronic control validation (e.g., card access systems) or validation by human
security personnel.
e. Physical Identification Validation: Access is physically safeguarded to prevent tampering and theft,
including procedures to address control and validation of a persons access to facilities based on
his or her need for access to the PII.
f.
Operational Environment: Procedures that specify the proper functions to be performed, the
manner in which they are to be performed, and the physical attributes of the surroundings of
facilities where PII is stored.
g. Media Movement: Procedures that govern the receipt and removal of hardware and electronic
media that contain PII into and out of a facility.
h. Encryption and Final Disposition of Information: Procedures addressing encryption of all data at
rest and in transit and the final disposition of PII. Procedures must include processes for the
continued encryption of PII through the time when its secure deletion/destruction has been
requested by the Department and the PII is deleted/destroyed.
3. Technical Safeguards
a. Data Transmissions: Technical safeguards, including encryption, to ensure PII transmitted over an
electronic communications network is not accessed by unauthorized persons or groups.
b. Data Integrity: Procedures that protect PII maintained by the Department or the Contractor from
improper alteration or destruction. These procedures will include mechanisms to authenticate
records and corroborate that they have not been altered or destroyed in an unauthorized manner.
134
c. Logging off Inactive Users: Inactive electronic sessions shall be designed to terminate
automatically after a specified period of time.
B. Security Controls Implementation
The Department and the Contractor will develop procedures addressing the acquisition and operation of
technology, the specific assignment of duties and responsibilities to managers and staff, the deployment of
risk-appropriate controls, and the need for management and staff to understand their responsibilities and
have the knowledge, skills and motivation necessary to fulfill their duties.
C. Security Monitoring
In combination with periodic security risk assessments, the Department and the Contractor will use a variety
of approaches and technologies to make sure that risks and incidents are appropriately detected, assessed
and mitigated on an ongoing basis. The Department and the Contractor will also assess on an ongoing
basis whether controls are effective and performing as intended, including intrusion monitoring and data loss
prevention.
D. Security Process Improvement
Based on the Department's security risk assessments and ongoing security monitoring, the Department and
the Contractor will gather and analyze information regarding new threats and vulnerabilities, actual data
attacks on the Department and the Contractor, and new opportunities for managing security risks and
incidents. The Department and the Contractor will use this information to update and improve their risk
assessment strategy and control processes.
E. Breach Remediation
The Department and the Contractor strive to keep PII maintained by the Department and the Contractor
secure and to use reasonable administrative, technical, and physical safeguards to do so. The Department
and the Contractor will maintain and update incident response plans that establish procedures to follow in
case a breach occurs. The Department and the Contractor will also identify individuals within their
respective organizations responsible for implementing incident response plans if a breach should occur.
If the Department or the Contractor determine that a breach has occurred, when there is a reasonable risk
of identity theft or other harm, or where otherwise required by law, the Department or the Contractor, as
applicable, will notify affected parties as promptly as possible, and will cooperate as needed to enable
compliance with all breach of confidentiality laws.
The Department and Contractor employee are required to report as promptly as possible to the Department
leadership (or his or her designee) and persons responsible for managing their respective organizations
incident response plan any incident or threatened incident involving unauthorized access to or acquisition of
PII of which they become aware. Such incidents include any breach or hacking of the Departments
electronic data system or any loss or theft of data, other electronic storage, or paper. As used herein,
"Electronic Data System" means all information processing and communications hardware and software
employed in the Department's or the the Contractor's business, whether owned by the Department or the
Contractor, or operated by its employees or agents in performing work for the Department or the Contractor.
F. Personnel Security Policy Overview
The Department and the Contractor shall mitigate the risks posed by users of the Department- or the
Contractor-maintained PII by:
1. Performing appropriate background checks and screening of new Department and Contractor
personnel, in particular those who will have access to the Department-maintained PII;
2. Obtaining agreements from the Department and the Contractor internal users covering confidentiality,
nondisclosure and authorized use of PII, including PII contained in or accessed through the
Department; and
135
3. Providing training to support awareness and policy compliance for new hires and annually for all
Department and Contractor personnel.
V. ENFORCEMENT
The Department and the Contractor will consistently enforce this Policy with appropriate discipline for its own
employees. The Department and the Contractor, as applicable, will determine whether violations of this Policy
have occurred and, if so, will determine the disciplinary measures to be taken against any director, officer,
employee, agent or representative who violates this Policy.
The disciplinary measures may include counseling, oral or written reprimands, warnings, probation or
suspension without pay, demotions, reductions in salary, or termination of service or employment, as well as
criminal referral to law enforcement, if appropriate.
Persons subject to disciplinary measures may include, in addition to the violator, others involved in the
wrongdoing such as (a) persons who fail to use reasonable care to detect a violation, (b) persons who withhold
material information regarding a violation, and (c) supervisors who approve or condone the violations or attempt
to retaliate against employees or agents or representatives of the Department or the Contractor for reporting in
good faith violations or violators.
The Department also may take appropriate actions authorized under contract or by law regarding the
Contractors failure to comply with the terms of this Policy. It is noted that if the U.S. Department of Education
finds that the Contractor has violated FERPA requirements related to disclosure, the Contractor may be
debarred from access to PII from the Department for at least five (5) years.
136
APPENDIX J
REQUIREMENTS FOR SCORING PERFORMANCE ITEMS
This appendix supplements the ITN description of the procedures required for scoring constructed response items on
Florida Standards Assessments in English Language Arts/Literacy (ELA/L) and Mathematics. The ITN requires that
student responses for constructed response items ELA/L tests in grades 3 and 10 and Algebra 1 be scored by at
least two readers (100% second scored) using imaging technology or other technological applications that are
equally responsive to the test administration and reporting schedule. As a quality control measure, 20% of all other
constructed response items for ELA/L and Mathematics tests will require at least two readers scores. The
Department desires to implement processes that are reliable and valid as well as efficient in terms of time and
expenditures. Therefore, respondents should, at a minimum, follow the processes described in this appendix.
Enhancements to these processes are acceptable when approved by the Department. For example, respondents
may propose an alternate method or formula for second reads to ensure quality control of scoring constructed
response items.
J.1
Overview of Handscoring of English Language Arts/Literacy and Mathematics Constructed

Response Items
Respondents should submit scoring rubrics to be used in conjunction with other training materials for
scoring constructed-response items for all grades and assessments. Field testing of items occurs in the
spring of each year with items embedded within the operational tests. Near the end of operational scoring,
contractor staff will begin selecting responses from the field test sample that represent a full range of
student work. These responses will be presented to educator committees in mid-spring at field test
rangefinding meetings. From these meetings, complete training and qualifying sets are created to train
scorers who will score a sufficient number of responses to yield reliable statistical data for each constructedresponse item. Every effort must be made to expedite scoring of the field test responses in order to have
the item statistics available to construct and produce the following years operational tests. Based on the
number of constructed response items field tested each spring, the Department will consider an additional
fall field test rangefinder meeting for the purpose of preparing training materials for constructed response
items that were not scored in the summer. The statistics for these items will be banked for possible use on
future operational tests. After all rangefinder meetings are completed, a final scoring material review
meeting between the Department and contractor staff will be conducted to finalize training, qualifying, and
initial validity materials for scorer training.
Depending on the measurement model for the ELA/L assessment, the Department will consider a standalone field test for ELA/L extended-response tasks. Respondents who propose a stand-alone field test
should prepare field test sampling specifications detailing the sampling methodology. The Department
prefers that such a field test, if proposed, take place in the December time frame of each year of
administration.
J.2
Conduct Field-Test and Operational Rangefinding Meetings for ELA/L and Mathematics
Prior to the scoring of constructed-response items field tested during the spring administrations, the
contractor will be responsible for organizing and implementing rangefinding meetings to select training
papers for the tasks. The meetings take place in Tallahassee and will involve Florida educators,
Department staff, and the contractors scoring directors and assistant directors over a period of up to three
weeks each spring and each fall for field-test rangefinder meetings and each fall for operational rangefinder
meetings. For all educator meetings, the Department and contractor staff will hold concurrent meetings for
subjects and grades whenever possible.
The contractors subject area lead scoring directors, subject/grade scoring directors, and assistant scoring
directors will participate in meetings with up to 10 Florida educators and a Department staff member who
will serve as chair.
137
The contractors staff will maintain a record of all scores and will keep notes during the rangefinder selection
process to document the rationale for assigning scores. Following the meeting, the contractors staff will
meet with Department staff to finalize the student responses to be used as rangefinders and to identify
others that may be used as training, qualifying, and validity responses. The contractors staff is also
responsible for providing written annotations in a format approved by the Department for the training
materials. Requirements for selecting validity sets are identified in the following section. Contractor staff is
also responsible for developing as part of the scoring guides any additional scoring notes or criteria required
to conduct accurate scoring of the constructed-response tasks. The Department will approve all selections,
annotations, or other materials to be used in training. See Section 7.5.4.5 for associated tasks.
J.3
Conduct Scoring Material Review Meetings

Meetings to review proposed scoring materials for operational tests will be conducted in Tallahassee at least
three months prior to the beginning of handscoring training for the spring test administrations. The
contractors subject area lead scoring director and the subject/grade scoring directors will participate in
meetings with Department staff to review scoring criteria and rangefinder papers that were used to score the
tasks during field test scoring, or during the previous operational use.
The purpose of each review meeting is to review the criteria, supplement the initial set of rangefinder papers
with additional papers, if necessary, and to ensure that the contractors scoring directors and Department
staff share the same, detailed understanding of the scoring criteria for the operational constructed-response
tasks. The scoring standards established by the initial field-test rangefinder selection will be maintained
during the subsequent review. See Section 7.5.4.5 for associated tasks.
J.4
Produce Scorer Training Materials

Department staff will work closely with the contractor to prepare scoring materials. Frequent communication
between Department and contractor staff can be expected following rangefinder selection and scoring
material review meetings to initiate and complete the development of scoring materials. All scoring material
will be submitted to the Department for review and approval according to an agreed-upon schedule.
Scoring materials must be approved at least four weeks prior to the beginning of training and scoring. The
contractor will be responsible for developing a detailed schedule to be included in the handscoring
specifications identifying steps and schedules in the development of scoring materials. The contractor is
responsible for producing the following scoring materials for each operational and field-test constructedresponse task.
Scoring guides
Training sets
Qualifying sets
Validity sets
Daily calibration sets
Targeted calibration
The contractor will develop a system for cataloging and storing all scoring materials developed during the
course of the project.
Scoring Guides - Scoring guides will be produced for each reader to use during training and scoring. The
scoring guides will contain the rubrics, scoring criteria, constructed-response tasks, passages (if applicable),
glossary of key terms, and other scoring guidelines designated by the Department and the contractor. The
contractor will be responsible for writing annotations for the scoring guides, and the Department will give
final approval to all annotations.
Training Sets Contractor must develop a sufficient number of training (practice) sets for ELA/L and
Mathematics constructed-response items for scorer training to ensure valid and reliable scores for the
duration of the project. The Department will work with contractor staff to determine the number and types of
138
training sets to be used. The contractor will be responsible for writing the annotations for the training papers
for all content areas, and the Department will give final approval to the annotations. The respondent should
propose the process for administering the training sets.
Validity Papers - Initial validity responses are selected from student responses to field-test administrations
and do not appear in any other scoring materials. These responses allow the Department and contractor to
monitor scoring accuracy and to maintain a consistent focus on the established rubrics and scoring
guidelines. A sufficient number of initial validity responses must be approved in time to be loaded into the
electronic scoring system before the scoring session begins. Validity responses are administered to readers
online as part of the electronic scoring system and must be indistinguishable from live responses. The
validity insertion rate will be agreed upon during the negotiation phase; however, for cost purposes,
respondents should assume a validity insertion rate of 1 in 10 for extended ELA/L items and 1 in 25 for all
other constructed response items. Once handscoring begins, scoring directors must refresh the validity pool
during live scoring by escalating proposed validity responses to the Department for approval. All proposed
validity papers for constructed-response items must have two agreement scores assigned. These scores
may be generated by two scorers or by one scorer and a back-reading score. The frequency distribution of
approved validity papers in the pool throughout the project should mirror the operational frequency
distribution to the greatest extent possible. Responses that are approved by the Department will be moved
into the electronic scoring system for random distribution to scorers. To avoid validity responses being
repeatedly seen by the same scorers, adding and deleting validity papers within the electronic system is a
continuous process throughout the project.
The contractor will describe the estimated number of readers to be employed and plans for the
administration of validity responses.
Daily Calibration Sets In addition to review of selected papers from the Scoring Guide at the start of each
shift, daily calibration sets will be sent via the electronic scoring system at least once a day usually after
lunch or break. These 2-3 paper sets will be proposed by scoring directors and approved by the
Department. The purpose of daily calibration is to highlight scoring issues or remind scorers of what to look
for at a certain score point. Following the scoring of the calibration sets, the scoring directors will conduct a
short discussion to further explain the scoring standards therein.
Targeted Calibration Sets Targeted calibration sets consist of 2-3 responses that target an individuals
scoring trend and may be used as individual training/remediation for scorers whose quality metrics are
below the standard for any reason.
At the completion of scoring, the contractor will securely ship to the Department an electronic file of all
scoring materials prepared for and utilized during scoring.
J.5
Scoring Directors for Handscoring

The contractor will assign its most qualified scoring staff to be scoring directors for Floridas Standards
Assessments. All scoring directors must have worked in scoring director roles on a regular, continuing
basis. The project and site scoring directors must be highly qualified to serve in these positions. They must
have an appropriate educational background and extensive experience in directing state-level performance
scoring projects as members of the contractors regular scoring staff. The contractor will submit resumes of
all handscoring staff to the Department for approval before assigning them to the project.
The contractor will appoint a full-time project director to serve as overall director for Florida Standards
Assessment handscoring activities. The project director must be available on a daily basis to discuss issues
with the site scoring directors and the Department either in person, by e- mail, by phone or fax throughout
the training and scoring sessions.
The contractor must provide at least one full-time lead scoring director for each content area with expertise
in their respective subject. This contractor staff will be available as liaison for Department content specialists
for all handscoring activities.
139
The contractor will provide a site scoring director at each site who is dedicated solely to scoring of Floridas
tests during the scoring period. Site scoring directors will be on site throughout the training and scoring
sessions and will personally assist the scoring directors during the training of team leaders and readers and
throughout the scoring sessions.
The project director must be a senior scoring manager for the contractor. Each site scoring director must
have served as a site scoring director for at least two projects in the past three years involving projects
similar in size and scope to Floridas.
The scoring directors who will direct the scoring for each subject/grade combination must be highly qualified
to serve in this position. They must have an appropriate educational background and must have had
experience serving as scoring directors at this level for previous scoring projects conducted by the
contractor. The scoring directors must participate in the field test and operational rangefinder and scoring
material review meetings. At least two scoring directors will be required for each grade and subject in order
to be available in the event of the scheduling of two shifts or two scoring rooms. For the ELA/L extended
constructed response items, it may be necessary to assign three scoring directors for this project.
At least one assistant scoring director will be appointed to assist each scoring director in all aspects of
directing scoring. The assistant scoring directors must also have an appropriate educational background
and must have had experience serving as scoring directors or extensive experience serving as team leaders
for previous scoring projects conducted by the respondent. The assistant scoring directors must also
participate in all rangefinder meetings.
Lead scoring directors, scoring directors, and assistant scoring directors must have their re sumes approved
by the Department prior to assignment to the project.
As stated above, Floridas handscoring involves selection of rangefinders for field-test tasks at the same
time that responses to the operational test are being scored. The scoring director and the assistant scoring
director for each subject/grade will work together as a team to ensure that field test rangefinder selection
and ongoing direction of operational scoring are both conducted at the highest levels of quality. It may be
required that contractor staff assigned to the project be present at field test rangefinding meeting while other
contractor staff remains at the scoring site to oversee completion of operational scoring.
For each assessment, the project and site scoring directors will ensure that all additional training required
for scoring is conducted prior to team leader training. Respondents should describe how they intend for this
training to occur.
J.6
Team Scoring Leaders

The contractor will hire one team leader for every 10 to 12 readers. The team leaders may be especially
recruited by the contractor or identified through the reader recruitment process. Team leaders must go
through the same screening process as readers. The Department requires that team leaders have previous
experience as readers and as team leaders if at all possible. At a minimum, team leaders must be
experienced readers.
Extensive training sessions approximately four days in length will be conducted for team leaders before
readers are trained. The scoring directors and assistant scoring directors for each set of constructedresponse tasks will conduct the training. The team leader training sessions will include the successful
completion of a minimum of three training sets and two qualifying rounds. For ELA/L extended constructedresponse writing tasks, Florida currently requires five training sets and three qualifying rounds.
The initial training will include the training and qualifying procedures that will be used for the readers. In
addition, there will be training on managing interaction with scorers, interpreting scoring reports, completing
forms, and following all procedures required by the contractor and the Department.
140
J.7
Recruit and Hire Readers

Respondents should include an analysis of the number of people that must be recruited, hired, and
subsequently qualified as readers to complete the scoring within the time required to return reports to
districts by the dates designated in the project schedule. The analysis must include the number and percent
of people who will be expected to qualify, expected reader-attrition rate during each scoring session, and
production required by reader and, if necessary, by shift at each site to complete the work. The respondent
should describe the number of team leaders needed for each subject/grade. This detailed analysis should
be completed in the reply for the Florida Standards Assessments in English Language Arts/Literacy and
Mathematics. Respondents should also indicate expected changes in reader numbers across successive
administrations.
All readers are required to have earned a bachelor's degree in mathematics or the area of English
Language Arts, or some related field. Readers of mathematics constructed-response tasks must have
either been certified to teach the level of mathematics being assessed, have a bachelors degree in
mathematics education or mathematics, or must have completed a degree program with a sufficiently strong
emphasis in mathematics to have the mathematics knowledge needed to effectively score responses to
mathematics items.
Potential readers in Florida may include retired Florida teachers or substitute teachers in the state of Florida
provided that they will not teach in Floridas public schools (or work as a substitute teacher therein) for the
remainder of the current school year. The Respondent is responsible for conveying this information and
obtaining signed verification that potential readers agree to adhere to this stipulation.
The contractor must work with the Department to implement a screening process for hiring readers to satisfy
the Departments requirements. Potential readers must document that they meet the minimal education
requirements during the screening process. Respondents should describe the screening process to select
scorers.
Potential readers with related degrees are those with degrees that involve course work and experience
related to the content areas being scored. Potential candidates for training and qualifying to score
mathematics shall have degrees in fields such as engineering, computer science, or liberal arts with an
emphasis in mathematics. Likewise, for scoring English Language Arts/Literacy, candidates shall have
degrees in fields such as English, English education, elementary education, reading education, history,
psychology, or journalism. Potential readers will provide information about degree, certification, and course
work taken. Minimum college coursework that represents a sufficiently strong emphasis in mathematics
needed to score in this area is college-level credit in at least three courses to score at the high school level,
two courses for the middle-school level, and one course to score at the elementary level. The contractor will
provide for an independent, third-party verification that the potential reader has earned the identified degree.
It has been the Departments experience that obtaining third-party verifications is more expensive and timeconsuming for potential readers holding degrees conferred by foreign universities and degrees conferred by
U.S. colleges and universities fifteen or more years ago. The contractor must take this into consideration
when hiring potential readers. Potential readers whose degrees cannot be verified by a third party before
the start of live scoring must be dismissed. No scores must be entered or maintained in the scoring system
from readers with unverified qualifications.
Potential readers also include those with a bachelors degree or higher who have successfully completed a
scoring project in the content area and at the elementary, middle, or high-school level being scored,
qualifying and performing at levels required by Floridas program. Potential readers will ultimately
demonstrate their qualification to score through successful completion of training and meeting the qualifying
standard on scoring qualification rounds, as indicated in the Handscoring Specifications document.
The contractor will document qualifications of readers assigned to be trained in an electronic file (the
Qualification File) specifying degrees earned; relevant teaching, educational, or work experience; and
previous scoring experience relevant to scoring Floridas assessments. The Qualification Files will not
141
include personally identifiable information such as name, address, and telephone number. Documentation
will be provided to Department staff, as needed, as it is accumulated throughout the hiring and training
period. After scoring has begun, the contractor will only maintain Qualification Files for readers who
successfully qualify and score student responses to Floridas assessments.
The Department will monitor adherence to the screening process, review reader files as needed, approve
acceptable related degrees, and provide general guidance on the screening process. Typically, each
scoring site will be monitored by at least one representative from the Department for two to three weeks
beginning with training. Prior to and during this on-site monitoring, the Department will provide oversight
and monitoring of both potential Team Leader and reader qualifications. This monitoring will include a
review for acceptability of potential readers for assignment to the handscoring program,
In the month prior to handscoring training, weekly conference calls between the contractor and the
Department will provide discussions surrounding site readiness, hiring/staffing, system readiness, and other
pertinent topics related to handscoring. After recruitment of potential readers is complete, the Department
staff will review a sample of potential readers qualifications. Staff review of qualifications will be conducted
utilizing a random sample of the Qualification Files. Staff will select additional potential scorer IDs for review
as deemed necessary, e.g., for candidates who are having little success in scoring accurately during
training. All qualifications for team leaders and a minimum of one in five candidates documentation will be
reviewed (e.g., for an initial hiring of 150 candidates for training, approximately 30 sets of documentation will
be requested). Once scoring training begins, daily conference calls between the contractor and the
Department will be conducted for project management purposes.
As part of the screening process, all potential readers must sign a statement indicating that they understand
the following conditions:
Applicants who do not successfully complete the training and qualifying requirements, they will not be
hired as readers.
Hired readers may be dismissed if, after being trained to score, their scoring performance does not
meet the requirements of the Department or the respondent acting on behalf of the Department.
Readers who are absent from scoring for two or more consecutive scoring days must repeat the
training and qualifying procedures described above.
Applicants must agree that they will not teach or work as a substitute teacher in Floridas public
schools for the remainder of the current school year.
Applicants hired for training must also sign an agreement with the Department that they will maintain the
security of Floridas test materials in addition to security agreements required by the contractor.
J.8
Training and Qualifying of Readers

The contractor will conduct separate training sessions for each constructed-response task or task set.
Training is provided separately for each grade level. The scoring director will conduct training with the
assistance of team leaders, under the direction of site scoring directors. The purpose of the training is to
ensure that each person who scores Floridas student responses has met the Department's standards for
scoring. The training process is essential for ensuring that scores assigned to performance task responses
provide valid and reliable information. During training, the trainees will be provided with a scoring guide,
highlighters and pens or pencils to make notations on training materials. At the conclusion of training,
qualified readers will be taught how to use the alert system to identify students whose responses indicate
the need for intervention. The contractor is responsible for developing training procedures in consultation
with the Department. The Department will have final approval on all training techniques.
To qualify as a reader, candidates must meet the established standards for the qualification sets approved
by the Department.
Assignment of scorers to items will be done in consultation with Department staff.
142
J.9
Handscoring Reports
The contractor will collect reader performance statistics and provide electronic access to daily and
cumulative reader handscoring reports to team leaders and above, including Department staff. The reports
must include both real-time and partial- to full-day reports. Real-time reports must include reader inter-rater
reliability and validity reports. The same reports are required for both operational and field-test scoring. The
Department will determine, in consultation with the contractor, which of these reports will be available to the
Department in real-time. The reports listed below are described generically. The contractor and the
Department will work together initially to determine the format of this information that should be used for
reporting to the Department. An annual review of the format and function of the reports will be conducted
and some modifications may be required prior to the next test administration requiring handscoring based
upon that review. The Department will have final approval on the design and user access level of each
report.
At a minimum, reports containing the following information will be produced and will include both numbers
and percentages, as appropriate.
1. Cumulative Training Report by Reader. This report identifies, by performance task or performance
task sets, by reader, by team, and by site (subject/grade combination), performance on training sets.
2. Cumulative Training Report by Document. This report identifies and summarizes, by performance task
or performance task set, by team, and by site, performance on documents.
3. Cumulative Qualifying Report by Reader. This report identifies, by performance task or performance
task set, by reader, by team, and by site, performance on qualifying sets.
4. Cumulative Qualifying Report by Document. This report summarizes, by performance task or
performance task set, by team, and by site, performance on documents.
5. Cumulative Reader Performance Report. This report includes the readers scores on each training set,
qualifying set, and validity set; daily inter-rater reliability statistics; and daily score point distribution.
This report will be updated daily. The report will be accessed on request of the scoring director or the
team leader.
6. Daily and Cumulative Score Point Distribution. This report summarizes, by performance task or
performance task set, by team, by site, and by rubric (0-2, 0-4, and 0-6), daily and cumulative
distributions of score-point assignments.
7. Daily and Cumulative Validity. This report summarizes, by performance task or performance task set,
by team, and by site, daily and cumulative performance on validity sets.
8. Daily and Cumulative Inter-rater Reliability. This report summarizes, by performance task or
performance task set, by site, by team, and by rubric, daily and cumulative levels of inter-rater
agreement statistics.
9. Score Point Distribution for Standard Curriculum Students. This report summarizes, by team and by
site, the cumulative distribution and mean of score-point assignments for standard curriculum students,
for writing performance tasks only.
10. Daily and Cumulative Score Point Distribution and Inter-rater Reliability by Reader. This report
identifies and summarizes, by performance task or performance task set, by site, by team, by rubric, by
reader, daily and cumulative score-point assignment distributions and levels of inter- rater agreement
statistics (including perfect, adjacent, and disparate agreement; high/low scoring summaries; and
required resolutions).
11. Daily and Cumulative Validity by Reader. This report summarizes, by performance task or
performance task set, by site, by team, by rubric, by reader, statistics indicating daily and cumulative
143
levels of agreement with validity set scores (including perfect, adjacent, and disparate agreement and
high/low scoring summaries).
12. Daily and Cumulative Validity by Validity Set. This report summarizes, by site, by team, by validity set,
by reader, statistics indicating daily and cumulative levels of agreement with validity set scores
(including a listing of validity set scores with readers assigned scores; perfect, adjacent, and disparate
agreement; and high/low scoring summaries).
13. Daily and Cumulative Read-Behind Reports. This report summarizes, for image scoring applications,
by performance task or performance task set, by team, and by site, the result of online read-behinds by
team leaders or scoring directors.
14. Daily and Cumulative Timing Report. This report summarizes, by site, by team, by performance task
or performance task set, the daily and cumulative number of readers engaged in scoring and statistics
on actual time spent scoring performance tasks or performance task sets (including mean and range).
15. Daily and Cumulative Completion Report. This report, by site and by team, summarizes the daily and
cumulative number of student responses read and the proportion of student responses remaining to be
read. This report also compares on a daily and cumulative basis the amount of readings completed in
comparison to projected completion targets.
By the time of the first handscoring planning meeting with the Department, the contractor will provide an
annotated list and samples of all reports, including reports used internally by the contractor to monitor the
quality and pace of the scoring session. The Department may request that some internal reports be used
for Florida and available to Department staff.
At the completion of operational and field-test scoring, the contractor will provide the Department with final
summaries of all cumulative handscoring reports. The summary reports are to be made available as
electronic files and will be posted to a secure transfer file system maintained by the contractor and shared
with the Department to access, manage, and transfer files as needed throughout the project. The contractor
will produce a technical report that summarizes the score reports and provides details related to the
reliability and validity of the field-test and operational handscoring. These materials will be reviewed at an
annual handscoring debrief.
J.10
Scoring Student Responses

The rubrics used for all constructed-response tasks must be approved by the Department and will be used
in conjunction with approved scoring guides and additional training materials throughout the handscoring of
student responses. Constructed-response items for ELA/L tests in grades 3 and 10 and Algebra 1 will be
scored by at least two readers (100% second scored) using imaging technology or other technological
applications that are equally responsive to the test administration and reporting schedule. As a quality
control measure, 20% of all other constructed response items for ELA/L and Mathematics tests will require
at least two readers scores. Responses receiving first and second reads of an individual students response
to an item will be randomly distributed across all readers designated to score that item. Assignment of
responses across schools must be randomized to the extent that an individual reader scores responses
from several schools within the same time period and does not score responses from the same school in
succession.
The Department will provide the contractor with a specific set of scoring rules for resolving disagreement
between first and second readings with third and, infrequently, fourth readings and for calculating final
scores. The following should be noted:
Third, independent readings are conducted by team leaders.

Complex resolutions and resolutions involving fourth readings and unscorable decisions are resolved
by scoring directors.
All discrepancies for short-response tasks shall be resolved by a third reader
144
ATTACHMENT 1
RESPONDENTS PRICE REPLY
** PLACEHOLDER FOR EXCEL PRICE SHEET **
145
ATTACHMENT 2
DRUG-FREE WORKPLACE
(will be considered in case of identical tie replies)
Preference shall be given to businesses with drug-free workplace programs. Whenever two or more bids which are
equal with respect to price, quality, and service are received by the state or by any political subdivision for the
procurement of commodities or contractual services, a bid received from a business that certifies that it has
implemented a drug-free workplace program shall be given preference in the award process. Established
procedures for processing tie bids will be followed if none of the tied vendors have a drug-free workplace program. In
order to have a drug-free workplace program, a business shall:
1) Publish a statement notifying employees that the unlawful manufacture, distribution, dispensing, possession, or
use of a controlled substance is prohibited in the workplace and specifying the actions that will be taken against
employees for violations of such prohibition.
2) Inform employees about the dangers of drug abuse in the workplace, the business's policy of maintaining a drugfree workplace, any available drug counseling, rehabilitation, and employee assistance programs, and the
penalties that may be imposed upon employees for drug abuse violations.
3) Give each employee engaged in providing the commodities or contractual services that are under bid a copy of
the statement specified in subsection (1).
4) In the statement specified in subsection (1), notify the employees that, as a condition of working on the
commodities or contractual services that are under bid, the employee will abide by the terms of the statement
and will notify the employer of any conviction of, or plea of guilty or nolo contendere to, any violation of Chapter
893 or of any controlled substance law of the United States or any state, for a violation occurring in the
workplace no later than five (5) days after such conviction.
5) Impose a sanction on, or require the satisfactory participation in a drug abuse assistance or
rehabilitation program if such is available in the employee's community, by any employee who is so
convicted.
6) Make a good faith effort to continue to maintain a drug-free workplace through implementation of this
section.
As the person authorized to sign the statement, I certify that this firm complies fully with the above
requirements.
Vendor's Signature
146
ATTACHMENT 3
DISCLOSURE
STATEMENT
PARTNERSHIP OR INDIVIDUAL
I hereby certify that I, if an individual, or each of us, if a partnership, doing business as of
,
(Name of Individual or Partnership)
(am)(is) not now involved in nor have I ever engaged in any private business venture or enterprise, directly or indirectly, with
the Commissioner of Education, the Deputy Commissioner of Education, any Associate Commissioner of Education,
Division Director, or Bureau Chief within the Florida Department of Education.
I further certify that neither I, nor any partner, if a partnership, nor anyone acting in my or our behalf has requested that any
of the above designated persons or any other employee of the Department of Education exert any influence to secure the
appointment of
under this proposed agreement.
(Name of Individual or Partnership)
(1)
Signature
Signature
(1) If partnership, each partner must sign and execute.
Signature
COMPANY OR CORPORATION
I hereby certify that neither I nor any owner, officer, director, or shareholder of
,a
(Name of Corporation/Company)
(1) corporation, licensed to do business in Florida, is presently involved in or has

(Name of State of Inc.)
been engaged in any private business venture or enterprise, directly, or indirectly, with the Commissioner of Education, the
Deputy Commissioner of Education, any Associate Commissioner of Education, Division Director, or Bureau Chief within the
Florida Department of Education.
I further certify that neither I nor any owner, officer, director, or shareholder of this corporation or anyone acting on behalf of
this corporation or any of its owners, officers, directors, or shareholders has requested that any of the above designated
persons or any other employee of the Department of Education exert any influence to secure the appointment of
under this proposed agreement.
(Company / Corporation)
(2)
Signature
Title
(1) If company is not incorporated, insert "not incorporated" in this space.
(2) If incorporated, this statement is to be executed by same person who will execute Contract, if awarded.
147
ATTACHMENT 4
REFERENCES
RESPONDENT NAME: _____________________________________
PROVIDE THE FOLLOWING REFERENCE INFORMATION FOR A MINIMUM OF THREE (3) BUSINESSES WHERE
SERVICES OF SIMILAR SIZE AND SCOPE HAVE BEEN COMPLETED.
BUSINESS NAME:
ADDRESS:
CONTACT PERSON:
PHONE NUMBER:
FAX NUMBER:
E-MAIL ADDRESS:
DATE AND DESCRIPTION
OF SERVICES:
BUSINESS NAME:
ADDRESS:
CONTACT PERSON:
PHONE NUMBER:
FAX NUMBER:
E-MAIL ADDRESS:
OF SERVICES:
BUSINESS NAME:
ADDRESS:
CONTACT PERSON:
PHONE NUMBER:
FAX NUMBER:
E-MAIL ADDRESS:
OF SERVICES:
148
ATTACHMENT 5
MINORITY SUB CONTRACTORS UTILIZATION SUMMARY
STATE OF FLORIDA DEPARTMENT OF EDUCATION
The Departments Supplier Diversity initiative strives to ensure the promise of Floridas future is shared by all of its
residents, regardless of race, ethnicity, disability, neighborhood or background. To that end, the Department is
dedicated to support, track and increase its small, minority-, women-, and service-disabled veteran business
enterprise spending with Prime Contractors and subcontractors. This form was developed to assist in these efforts.
The Prime Contractor shall report all small, minority-, women-, and service-disabled veteran business enterprise
Subcontractors, identifying the Name, Address, Type of Certification and Dollar Amount on the form below. The
Prime Contractor shall submit this form with each invoice submitted for payment, whether or not funds have been
spent with a small, minority-, women-, and service-disabled veteran business enterprise subcontractor for the period
covered by the invoice. The Office of Supplier Diversity, Florida Department of Management Services will assist in
furnishing names of qualified minorities. The Office of Supplier Diversity can be reached at (850) 487-0915; the
Internet Web address is http://dms.myflorida.com/other_programs/office_of_supplier_diversity_osd .
PRIME CONTRACTOR:
CONTRACT NO.:
CONTRACT TITLE:
MBE CONTRACTORS
Full Name, Address, Telephone Number
State
Certified
NonCertified
NonProfit
Dollar
Amount
Total Amount $
Certified True and Correct by:
Submit Report to:
_________________________________________
Prime Contractor
_________________________________________
Title
_________________________________________
Date
Mrs. Janice Brown

Bureau of Contracts, Grants & Procurement
Management Services
For additional information, you may call Mrs. Brown at (850) 245-0819, or e-mail janice.brown@fldoe.org
149
ATTACHMENT 6
VENDOR CERTIFICATION REGARDING
SCRUTINIZED COMPANIES LISTS
Respondent Vendor Name: _____________________________________________________

Vendor FEIN: ___________________
Vendors Authorized Representative Name and Title: ___________________________________
Address: _____________________________________________________________________
City: _____________________ State: _____________________________ Zip: ______________
Phone Number: ____________________________________
Email Address:
Section 287.135, Florida Statutes, prohibits agencies from contracting with companies, for goods or services over
$1,000,000, that are on either the Scrutinized Companies with Activities in Sudan List or the Scrutinized Companies
with Activities in the Iran Petroleum Energy Sector List. Both lists are created pursuant to section 215.473, Florida
Statutes.
As the person authorized to sign on behalf of Respondent, I hereby certify that the company identified above in the
section entitled Respondent Vendor Name is not listed on either the Scrutinized Companies with Activities in Sudan
List or the Scrutinized Companies with Activities in the Iran Petroleum Energy Sector List. I understand that pursuant
to section 287.135, Florida Statutes, the submission of a false certification may subject company to civil penalties,
attorneys fees, and/or costs.
CERTIFIED BY: __________________________________________________________,

Who is authorized to sign on behalf of the above referenced company.
Authorized Signature Print Name and Title: _________________________________________
150
ATTACHMENT 7
CONTRACT STANDARD TERMS AND CONDITIONS
STATE OF FLORIDA DEPARTMENT OF EDUCATION
I.
II.
III.
IV.
Pursuant to S. 287.058(1), Florida Statutes (F.S.):

A. Bills for fees or other compensation for services or expenses shall be submitted in detail sufficient for a proper preaudit and
postaudit thereof.
B. Travel expenses will be reimbursed only if expressly authorized by the terms of the Contract. Bills for any travel expenses
shall be submitted in accordance with s. 112.061, F.S.
C. The Department may unilaterally cancel this Contract if the Contractor refuses to allow access by members of the public to all
documents, papers, letters and materials made or received in conjunction with the Contract that are subject to Chapter 119,
F.S., and are not exempt from public inspection by s 119.071, F.S., or by other provisions of general or special law.
D. The Deliverables specified in the Contract must be received and accepted in writing by the Departments Contract Manager
before Contractor is entitled to payment.
E. To complete this Contract, all services must be performed and/or goods received on or before the date(s) specified in the
Contract.
F. If this Contract is expressly renewable, it may be renewed for a period that may not exceed three years or the term of the
original contract, whichever is longer. The renewal price for the contracted service is set forth in the bid, proposal, reply. Cost
for renewal shall not be changed. Renewals shall be contingent on satisfactory performance evaluations by the Department
and subject to the availability of funds. Exceptional purchase contracts pursuant to s. 287.057(3)(a) and (c), F.S., may not be
renewed.
The Contractor shall prepare an invoice for the amount due and mail it to the Department of Education Comptroller after having
delivered the products and services required under this Contract to the Contract Manager. The invoice shall set forth details
sufficient for a proper pre-audit and post-audit including, where applicable, the products and services delivered and completion
dates. Upon receipt of the invoice, the Department of Education Comptroller will request confirmation from the Contract Manager
that the delivered products and services are satisfactory and payment is due. If for any reason they are not satisfactory, payment
will be withheld until the unsatisfactory condition or conditions are corrected. Upon receipt of the Contract Managers approval, the
Department of Education Comptroller shall process each invoice in accordance with the provisions of s. 215.422, F.S.
A. Contractor agrees to submit invoice within thirty (30) days of the Departments acceptance of deliverables. It is understood
that should Contractor fail to submit invoice within thirty (30) days following the Departments acceptance of the deliverables,
the Department shall not be responsible for payment thereof under this contract or quantum meruit.
Section 215.422, F.S., provides that agencies have five (5) working days to inspect and approve goods and services, unless bid
specifications or the Contract specifies otherwise. With the exception of payments to health care providers for hospital, medical, or
other health care services, if payment is not available within forty (40) days, measured from the latter of the date the invoice is
received or the goods or services are received, inspected and approved, a separate interest penalty set by the Comptroller pursuant
to s. 55.03, F.S., will be due and payable in addition to the invoice amount. To obtain the applicable interest rate, please contact the
Departments Fiscal s. at 850/245-0401 or Purchasing Office at 850/245-0735. Payments to health care providers for hospitals,
medical, or other health care services, shall be made not more than thirty-five (35) days from the date of eligibility for payment is
determined, and the daily interest rate is .02740 percent. Invoices returned to a vendor due to preparation errors will result in a
payment delay. Invoice payment requirements do not start until a properly completed invoice is provided to the agency. A Vendor
Ombudsman, whose duties include acting as an advocate for vendors who may be experiencing problems in obtaining timely
payment(s) from a State Agency, may be contacted at 866/352-3776 or by calling the Chief Financial Officers Hotline, 800/3422762.
As used in this Contract, the term Deliverable refers to tangible commodities, as defined in s. 287.012(5), F.S., which the
Contractor provides pursuant to the Contract and to reports or other tangible or documentary evidence which demonstrate that the
Contractor has performed the services required by the Contract. The following provisions govern Deliverables, as applicable:
A. Each Deliverable must be physically delivered to the Departments Contract Manager, or to a person designated by the
Contact Manager. If delivery is made to a designee, the Contractor shall give written notice to the Contract Manager of the
delivery. A Deliverable is not received until the Contract Manager has physical control of deliveries or has written notice that
the designee has physical control.
B. In each case in which the approval of a Deliverable is dependent upon tests being conducted by the Department or Contractor,
independently or jointly, the Departments inspection and approval of the Deliverable shall not be subject to the five (5) day
provision in s. 215.422, F.S., but shall be governed by the terms and conditions of the acceptance testing plan as stated in
Attachment A, until approved in accordance with the plan.
C. In each case of a Deliverable of information technology, as defined at s. 287.012(14), F.S., unless specified otherwise in
Attachment A, the acceptance testing plan is deemed to include as a minimum the reliable performance of the information
technology in accordance with its design specifications in:
1. a test environment that simulates the production environment as much as is reasonably possible; and
2. the production environment for which it is intended for a period of time sufficient for the information technology to have
experienced the major foreseeable exigencies of the production functions.
D. The Departments inspection, including testing when applicable, shall determine whether or not the Deliverables appear to be
in compliance with the Contract. The Contractor shall be notified in writing of any apparent deficiency. The written notice shall
151
V.
VI.
VII.
VIII.
IX.
X.
XI.
detail the specific action required by the Contractor to correct the deficiency. The Contractor shall timely correct such
deficiency and resubmit the deliverable for acceptance.
The Contractor represents and agrees that information submitted in support of its requests for payment is the basis of payment and
is true and accurate to the best of knowledge of the responsible signatory. A violation of this provision shall subject the violator to
the provisions of s. 68.082, F.S., pertaining to false claims against the State, and/or s. 837.06, F.S., pertaining to false official
statements.
This paragraph applies if this Contract expires in a fiscal year subsequent to the fiscal year in which the Contract is entered. The
State of Floridas fiscal year comprises July 1 through June 30. The Departments and State of Floridas performance and obligation
to pay under this Contract is contingent upon an annual appropriation by the Legislature. If the Legislature fails to make the
necessary appropriation, the Department will determine if there are other unencumbered funds which are available and which can
be lawfully expended to pay for the Departments obligations hereunder. If the Department determines that there are no such funds,
the Department shall promptly notify the Contractor. The giving of notice shall be deemed to have cancelled this Contract by mutual
consent, with the date of notice being the date of cancellation.
Notwithstanding anything to the contrary contained in a State Term Contract, Contractor warrants that all commodities, as defined in
s. 287.012, F.S., shall meet the specifications of the Contract and shall be merchantable and fit for the particular purposes intended
by the Contract.
The Contractor further warrants that as to each Deliverable produced pursuant to this Contract, Contractors production of the
Deliverable, and the Departments use of the Deliverable, will not infringe on the copyrights of any third party. This provision applies
to each work of authorship in which copyrights subsist pursuant to 17 U.S.C. Sections 102-105 and to each exclusive right
established in 17 U.S.C. Section 106. In furtherance of this provision the Contractor additionally warrants that:
A. As to each work of software or other information technology, as defined in s. 287.012(15), F.S., in which copyrights subsist,
the Contractor has acquired the rights by conveyance or license to any third party software or other information technology,
which was used to produce the Deliverable;
B. As to each image and sound recording incorporated into a Deliverable, the Contractor has acquired the necessary rights,
releases, and waivers from the person whose image or sound is included, or from the holder of the copyrights subsisting in the
literary, musical, dramatic, pantomime, choreographic, pictorial, graphic, sculptural, motion pictures, audiovisual work or sound
recording from which the included image or sound recording was taken.
The Contractor further warrants that the Contractor shall not disclose to any third party, without the express, prior, written approval
of the Department, any personally identifiable information about any student. This applies to information which came from any
record or report of a Florida public education institution or from any education record which is subject to the Family Educational
Rights and Privacy Act, 20 U.S.C. Section 1232g. The terms record a report and student shall have the meanings prescribed in
s. 1002.22(2)(c) and (d), F.S. The term educational record shall have the meaning prescribed in 20 U.S.C. Section 1232g(a)(4).
In the event that the Governor and Cabinet are required to impose a mandatory reserve on appropriations, the Department shall
amend this Contract to place in reserve the amount determined by the Department of Education to be necessary because of the
mandatory reserve. Such amendments may provide for adjustments in the Deliverable products and services as may be necessary.
Intellectual property is subject to following additional provisions:
A. Anything by whatsoever designation it may be known, that is produced by, or developed in connection with, this Contract shall
become the exclusive property of the of the State of Florida and may be copyrighted, patented, or otherwise restricted as
provided by Florida or federal law. Neither the Contractor nor any individual employed under this Contract shall have any
proprietary interest in the product.
B. With respect to each Deliverable that constitutes a work of authorship within the subject matter and scope of U.S. Copyright
Law, 17 U.S.C. Sections 102-105, such work shall be a "work for hire" as defined in 17 U.S.C. Section 101 and all copyrights
subsisting in such work for hire shall be owned exclusively by the Department pursuant to s. 1006.39, F.S., on behalf the State
of Florida.
C. In the event it is determined as a matter of law that any such work is not a "work for hire", Contractor shall immediately assign
to the Department all copyrights subsisting therein for the consideration set forth in the Contract and with no additional
compensation.
D. The foregoing shall not apply to any preexisting software, or other work of authorship used by Contractor, to create a
Deliverable but which exists as a work independently of the Deliverable, unless the preexisting software or work was
developed by Contractor pursuant to a previous Contract with the Department or a purchase by the Department under a State
Term Contract.
E. The Department shall have full and complete ownership of all software developed pursuant to the Contract including without
limitation:
1. The written source code;
2. The source code files;
3. The executable code;
4. The executable code files;
5. The data dictionary;
6. The data flow diagram;
7. The work flow diagram;
8. The entity relationship diagram; and
152
XII.
XIII.
XIV.
XV.
XVI.
XVII.
XVIII.
XIX.
XX.
XXI.
XXII.
XXIII.
XXIV.
XXV.
9. All other documentation needed to enable the Department to support, recreate, revise, repair, or otherwise make use of
the software.
The Department reserves the right, at its option, to issue a change order to delete work tasks reducing the total Contract amount by
up to 10%. An addition of work tasks within the scope of the Contract, an increase in the total Contract amount, or a decrease of
more than 10% of the total Contract amount, shall be implemented only by a Contract amendment signed by both the Department
and the Contractor.
Pursuant to s. 216.347, F.S., no funds awarded under this Contract may be used for the purpose of lobbying the Legislature, the
judicial branch, or a State agency.
The Contractor shall grant access to all records pertaining to the Contract to the Departments Inspector General, General Counsel
and other agency representatives, the State Auditor General, the Office of Program Policy and Government Accountability, and the
Chief Financial Officer.
The Contractor agrees to permit onsite visits by designated Department employees or agents to conduct audits to ensure
compliance with Section 20.055, Florida Statutes. These audits may require Department access to records and data, computers and
communications devices, and other materials whether owned or operated by the Contractor. Access may include, but is not limited
to, user level and/or system level access to any computing or communications device; access to information (electronic, hardcopy,
etc) that may be produced, transmitted or stored on the Contractor's equipment or premises; access to work areas; and access to
interactively monitor and log traffic on the Contractor's networks.
The Contractor must carry general liability insurance, which shall include errors and omissions coverage. The amount of coverage
shall be a minimum of $1,000,000 or the aggregate total of all contractual agreements between the Contractor and the agencies and
political subdivisions of the State of Florida, whichever is greater. The Contractor shall add the Department as an additional insured
on the general liability coverage. The insurance shall cover all of the Contractors operations under this Contract and shall be
effective throughout the Term of this Contract, as well as any renewals or extensions thereto. It is not the intent of this Contract to
limit the types of insurance otherwise required by this Contract or that the Contractor may desire to obtain or be required to obtain
by law. The Contractor must submit a Certificate of Insurance indicating coverage for general liability purposes and additional
insured coverage, and shall maintain and pay for same throughout the Term of this Contract. A Certificate of Insurance indicating
adequate coverage shall be submitted to the Department prior to the time the Contract is entered. Any and all insurance policies
shall be through insurers qualified to do business in Florida.
The Contractor agrees to provide the Department upon execution of this Contract with a performance bond or other security
deposited with the Department in the total amount of the Contract or another amount if specified in the procurement specifications or
Attachment A, guaranteeing that the Contractor will perform all work according to this Contract, within the time and price specified in
the Contract. A performance bond shall be issued from a surety company, qualified to do business in Florida.
The Contractor may not assign or subcontract all or any portion of this Contract without the advance written consent of the
Department.
In all cases in which the Contractor, with the advance written consent of the Department, assigns or subcontracts, all or any portion
of the Contract:
A. The Contractor shall monitor the subcontractor or assignee and establish controls to avoid or mitigate risks identified by the
Department or the Contractor; and
B. The Contractor shall allow the Department to monitor subcontractor or assignee activity and compliance, and the Contractor
shall require the subcontractor or assignee to promptly submit to the Department, at the Departments request, complete and
accurate documentation pertaining to the subcontract or the Contract.
The Contractor shall coordinate with and assist the Departments Contract Manager in the performance of the latters
responsibilities, which include without limitation:
A. Monitoring the activities of the Contractor;
B. Receiving and reviewing the reports of the Contractor to determine whether the objectives of the Contract are being
accomplished;
C. Receiving and reviewing the invoices for payment of funds to assure that the requirements of the Contract have been met and
that payment is appropriate;
D. Evaluating the process used by the Contractor to monitor the activities of any subcontractor or assignee; and
E. Accessing, directly, the subcontractors and assignees, as the Contract Manager deems necessary.
This Contract may not be modified unless in writing signed by the Department and the Contractor.
The Department and the Contractor waive application of the principle of contract construction that ambiguities are to be construed
against a contracts drafter, and agree that this Contract is their joint product.
The Department and the Contractor acknowledge that they have had their respective attorneys review and approve this Contract or
that they have had the opportunity to do so.
This Contract shall be governed by the laws of the State of Florida, and venue for purposes of any action brought to enforce or
construe the Contract shall lie in Leon County, Florida.
Failure of the Department to declare any default immediately upon the occurrence or knowledge thereof, or delay in taking any
action in connection therewith, does not waive such default. The Department shall have the right to declare any such default at any
time and take such action as might be lawful or authorized under the Contract, at law, or in equity. No Department waiver of any
term, provision, condition or covenant of the Contract shall be deemed to imply or constitute a further Department waiver of any
other term, provision, condition or covenant of the Contract, and no payment by the Department shall be deemed a waiver of any
default under the Contract.
153
XXVI.
XXVII.
XXVIII.
XXIX.
XXX.
XXXI.
XXXII.
XXXIII.
XXXIV
Time is of the essence with regard to each and every obligation of the Contractor contained in the Contract. Each such obligation is
deemed material, and a breach of any such obligation (including a breach resulting from the untimely performance thereof) shall
constitute a material breach.
The Contractor shall indemnify and hold harmless the Department, its attorneys, agents and employees, from and against any and
all third party claims, suits, debts, damages, and causes of action, whatsoever, whether arising in law or in equity, arising out of or
relating to Contractor performance or failure to perform under this Contract. The indemnification shall include reasonable attorney
fees and costs incurred by the Department, its attorneys, agents and employees, in the defense of any such claim, suits or causes
of action, as aforesaid.
This Contract may be cancelled by written agreement of the Department and the Contractor specifically referencing this Contract.
Such agreement shall specify the remaining measures necessary to be taken by each party.
The Department reserves the right to cancel this contract without cause by giving the Contractor thirty (30) days written notice.
Should Contractor fail to perform to Contract terms and conditions, Contractor shall be notified in writing, stating the nature of the
failure to perform and providing a time certain (which shall be not less than ten (10) days following receipt of such notice) for
correcting the failure. Such failure to perform shall otherwise be dealt within accordance with Rule 60A-1.006, F.A.C.
A person or affiliate who has been placed on the convicted vendor list following a conviction for a public entity crime may not submit
a bid on a contract to provide any goods or services to a public entity, may not submit a bid on a contract with a public entity for the
construction or repair of a public building or public work, may not submit bids on leases of real property to a public entity, may not be
awarded or perform work as a contractor, supplier, subcontractor, or consultant under a contract with any public entity, and may not
transact business with any public entity in excess of the threshold amount provided in s. 287.017, F.S., for CATEGORY TWO for a
period of thirty-six (36) months from the date of being placed on the convicted vendor list.
The employment of unauthorized aliens by any contractor is considered a violation of Section 274A(e) of the Immigration and
Nationality Act. If the contractor knowingly employs unauthorized aliens, such violation shall be cause for unilateral cancellation of
the contract. In addition, pursuant to State of Florida Executive Order No. 11-116, Contractor shall utilize the U.S. Department of
Homeland Securitys E-Verify system to verify the employment of all new employees hired by the Contractor during the contract
term. Also, Contractor shall include in related subcontracts a requirement that subcontractors performing work or providing services
pursuant to the Contract utilize the E-Verify system to verify employment of all new employees hired by the subcontractor during the
contract term.
Certification Regarding Debarment, Suspension, Ineligibility and Voluntary Exclusion Lower Tier Covered Transactions
A. The prospective lower tier participant certifies, by submission of this proposal, that neither it nor its principals is presently
debarred, suspended, proposed for debarment, declared ineligible, or voluntarily excluded from participation in this transaction
by any Federal department or agency.
B. Where the prospective lower tier participant is unable to certify to any of the statements in this certification, such prospective
participant shall attach an explanation to this proposal. The Department may cancel this contract if an attached explanation is
not acceptable to the Department or the Federal government.
MyFloridaMarketPlace
A. MyFloridaMarketplace Vendor Registration
Each Vendor doing business with the State of Florida for the sale of commodities or contractual services as defined in section
287.012, Florida Statutes, shall register in MyFloridaMarketPlace, in compliance with Rule 60A-1.030, Florida Administrative
Code, unless exempt under Rule 60A-1.030(3) Florida Administrative Code.
B. MyFloridaMarketplace Transaction Fee
The State of Florida, through the Department of Management Services, has instituted MyFloridaMarketPlace, a statewide
eProcurement system. Pursuant to section 287.057(23), Florida Statutes (2010), all payments for commodities and/or
contractual services as defined in Section 287.012, Florida Statutes, shall be assessed a Transaction Fee of one percent
(1.0%), which the Vendor shall pay to the State, unless exempt under Rule 60A-1.032, Florida Administrative Code.
Notwithstanding the provisions of Rule 60A-1.030, et seq., the assessment of a transaction fee shall be contingent upon
Federal approval of the transaction fee assessment program and continued payment of applicable federal matching funds.
For payments within the State accounting system (FLAIR or its successor), the Transaction Fee shall, when possible, be
automatically deducted from payments to the Vendor. If automatic deduction is not possible, the Vendor shall pay the
Transaction Fee pursuant to Rule 60A-1.031(2), Florida Administrative Code. By submission of these reports and
corresponding payments, Vendor certifies their correctness. All such reports and payments shall be subject to audit by the
State or its designee.
The Vendor shall receive a credit for any Transaction Fee paid by the Vendor for the purchase of any item(s) if such item(s)
are returned to the Vendor through no fault, act, or omission of the Vendor. Notwithstanding the foregoing, a Transaction Fee
is non-refundable when an item is rejected or returned, or declined, due to the Vendors failure to perform or comply with
specifications or requirements of the agreement.
Failure to comply with these requirements shall constitute grounds for declaring the Vendor in default and recovering
reprocurement costs from the Vendor in addition to all outstanding fees. VENDORS DELINQUENT IN PAYING
TRANSACTION FEES MAY BE EXCLUDED FROM CONDUCTING FUTURE BUSINESS WITH THE STATE.
154
Attachment 1
ITN 2014-47 Florida Standards Assessments
RESPONDENT:_________________________________________________
We propose to provide the services being solicited for ITN 2014-47 Florida Standards Assessments. All work shall be performed in accordance with the specifications of this ITN, which has been reviewed and
understood. The below prices are all inclusive. Payment for services will be made based on the deliverables and dates specified. There shall be no additional costs charged for work performed
under this ITN, unless changes are made during the negotiation phase.
Respondents may submit bids on any individual assessment components (A, B, C or D), any group of assessment components, or to provide all assessment components (A, B, C and D). The prices on this
sheet will be used for evaluation purposes. All accommodations and deliverable costs should be included in each component price. In addition, Respondents should complete the pricing per deliverable in
SECTION 7.8, for each assessment component, which will be used for negotiation purposes and any resulting contract.
PRICING
ASSESSMENT COMPONENTS
YEAR 1
YEAR 2
YEAR 3
TOTAL ORIGINAL
CONTRACT TERM
(YEARS 1 - 3)
GRAND TOTAL
RENEWAL YEAR RENEWAL YEAR RENEWAL YEAR
(ORIGINAL TERM + ALL
1
2
3
RENEWALS)*
A. Computer-based and paper-based assessments in English

Language Arts/ Literacy in Grades 3-8 per schedule in
Appendix A.
B. Computer-based and paper-based assessments in
Mathematics in Grades 3-8 per schedule in Appendix A.
C. Computer-based and paper-based assessments in English
Language Arts/Literacy in Grades 9-11 per schedule in
Appendix A.
D. Computer-based and paper-based End-of-Course (EOC)
assessments in Algebra 1, Geometry, and Algebra 2 per
schedule in Appendix A.
E. All assessments (A-D) per schedule in Appendix A.
* Evaluation points will be calculated on these costs.
Vendor Name:
Street Address:
City, State, Zip Code:
Telephone #:
Fax:
Authorized Agent (typed):
Authorized Agent (Signature):
145
Attachment 1b - DELIVERABLES PRICING

Specify Component A, B, C, D or E (A-D): __________________________
A
1
3
4
5
6
7
8
9
10
11
12
13
Deliverable Category
Test Development
Contractor Deliverable
Periodic: Production Specifications
Document
Test Development
Fall ELA/L Retake: Handscoring

Specifications
Test Development
Spring ELA/L Retake: Periodic:

Handscoring Specifications
Deliverable Description
This secure document includes the specific
processes, procedures, and design guidelines for all
print, computer-based, and accommodated tests,
ancillaries, and interpretive products. ITN Reference
7.3.10
Evidence of Completion
(1) Provide Department email reflecting approval of
the final production specifications document.
(2) The final production specifications file provided
via SFTP.
The contractor should incorporate the handscoring

requirements into written handscoring
specifications prior to scoring. The handscoring
specifications will be a detailed guide to conducting
handscoring that will be used by the contractors
handscoring managers and the Department. The
applicable administrations/assessments will be
determined in the final negotiations. ITN Reference
7.5.4.5
(1) Provide Department email reflecting approval of Dates to be negotiated

the final handscoring specifications.
(2) The final handscoring specifications provided via 2016
2017
SFTP.
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Due Date(s)
June 1, 2014
June 1, 2015
June 1, 2016
(Renewal 1 - June 1, 2017)
The contractor should incorporate the handscoring (1) Provide Department email reflecting approval of Dates to be negotiated
2016
specifications will be a detailed guide to conducting SFTP.
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5
14
15
16
17
18
19
20
Deliverables Pricing Sheet
F
Deliverable Price

A
1
21
22
23
24
25
Test Development
Spring ELA/L and Mathematics:
7.5.4.5
(2) The final handscoring specifications provided via
SFTP.
Due Date(s)
Dates to be negotiated
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Test Development
Fall EOC: Handscoring Specifications

7.5.4.5

2016
SFTP.
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
Test Development
Winter EOC: Handscoring Specifications The contractor should incorporate the handscoring
7.5.4.5

2016
SFTP.
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
26
27
28
29
30
31
32
33
34
35
36
37
38
F
Deliverable Price

A
1
39
40
41
42
43
Test Development
Spring EOC: Handscoring Specifications
7.5.4.5
SFTP.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Test Development
Summer EOC: Handscoring

Specifications

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Periodic: Style guide for test items
This document is designed to provide a consistent (1) Provide Department email reflecting approval of
grammar and style standard for the proofing and
the final style guide document.
editing of ELA/L and Mathematics test materials.
(2) The final style guide file provided via SFTP.
The document may be included as part of the
production specifications document. ITN Reference
7.3.10
44
45
46
47
48
49
50
51 Test Development
52
53
54
55
56
June 1, 2014
June 1, 2015
June 1, 2016
F
Deliverable Price

A
1
D
each test item specification document.
(2) Final PDF files provided via SFTP.
Due Date(s)
Deliverable Price
Test Development
Periodic: Interpretive Products: Test
Item Specifications Documents
Test item specifications will be produced for all
grades/subjects and may be grouped together in
one booklet by grade band or content area. They
are based on the standards and define both the
content and the format of the test items. Their
primary purpose is to establish guidelines for test
item writers and reviewers. ITN Reference 7.3.7.5
Test Development
Periodic: ELA/L and Mathematics

Sample Scoring Guides
The contractor will be expected to post electronic (1) Provide Department email reflecting approval of
each scoring guide.
publications that provide information to help
educators understand the scoring of constructedresponse items included on student reports. These
publications provide information about the test
design, standards and benchmarks, item types, and
how items are scored. They also provide scoring
rubrics and the annotated papers used to train
scorers for the constructed-response items. Each
sample item from the scoring papers is displayed as
it appeared on the test and responses for each
score point, images of student responses, and
annotations explaining why the response received a
particular score are provided. ITN Reference 7.3.10
(Dates TBD - These should coincide with the

approved report release dates, which are part of
contract negotiations.) 2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Test Development
Periodic: Understanding Florida's

Assessment Reports
This publication describes the various reports of test (1) Provide Department email reflecting approval of
Understanding Florida's Assessment Reports.
results distributed to students, schools, and
districts. An explanation and an image of each type (2) Final PDF files provided via SFTP.
of report are provided. The booklet also includes
information about the test content areas and a
glossary of technical terms. The contractor will
deliver the electronic ADA-compliant PDF for
posting to the DOE website so files can be posted to
coincide with the first results delivery (RD1). ITN
Reference 7.3.7.5

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
November 1, 2015
57
58
59
60
61
62
63
64
65
66
67
68
69

A
1
Test Development
70
71 Test Development
72
73
74
75
76 Test Development
77
78
79
80
B
Periodic: Test Design and Blueprint
C
The Test Design and Blueprint will clearly delineate
the number of items, types and placement of items,
time allotted for each test session, and location of
field-test items for all grades and subjects. ITN
Reference 7.3.1 and 7.3.2.1
D
Test Design and Blueprint
Due Date(s)
Deliverable Price
June 1, 2015
(2) Appearance of completed schedule in weekly

management report
Periodic: Content Item Review "Cleans" After Content Item Review, the contractor, in
conjunction with the Department, will conduct an
editorial and content review of the items to ensure
that changes have been applied correctly and that
all editorial and content issues are corrected. ITN
Reference 7.3.4.2
(1) Provide Department email reflecting completion February 1, 2016

of the Cleans process
February 1, 2017
(Renewal 1 - February 1, 2018)
management report
Periodic: Field Test Item Sets and Forms Whether an embedded or stand-alone model is
Review
used, the contractor, in conjunction with the
Department, will construct field-test forms. ITN
Reference 7.3.5 and 7.3.6
(1) Provide Department email reflecting completion

of the field test forms process
management report
May 1, 2015
May 1, 2016
(Renewal 1 - May 1, 2017)

A
1
Test
Development
81
82
83
84
85
86
87 Test Development
88
89
90
91 Test Development
92
93
94
Periodic: Onsite Test Construction
The contractor will host content experts and
leadership from the Department for the express
purpose of constructing assessments for all
grades/courses and subjects. To the extent possible,
the Department expects to be involved in any test
construction activities for 2015 assessments still to
be completed after contract execution (in July
2014). ITN Reference 7.3.7
Due Date(s)
1) Provide Department email reflecting completion of July 15, 2014
the test construction process
July 15, 2015
July 15, 2016
(Renewal 1 - July 15, 2017)
management report
Periodic: Item Development Plan
To ensure the quality and number of

items/passages developed is sufficient to create
future operational test forms, the contractor will
submit an item development plan delineating the
number and types of items to be developed. ITN
Reference 7.3.0
1) Provide Department email reflecting completion of

the Item Development Plan.
management report.
December 1, 2015
December 1, 2016
(Renewal 1 - December 1, 2017)
Periodic: Item Writer Training
The contractor must identify and recruit a sufficient

number of highly-qualified writers for passages and
items to ensure variety and quality of development.
The contractor, in conjunction with the Department,
will conduct an annual training session for all Item
Writers. ITN Reference 7.3.3

Item Writer Training.
management report.
December 1, 2015
December 1, 2016
F
Deliverable Price

A
1
Test
Development
95
96
97
98
99
100
101 Test Development
102
103
104
105
106
108
109
110
111
112
Test Administration
113
114
115
116
117
Periodic: Meetings with Educators
The contractor is responsible for organizing and
convening meetings for the purpose of reviewing
assessment items and otherwise supporting the
assessment program. These meetings are
referenced in the ITN in 7.3.4 and 7.7.7. This
deliverable includes the cost of preparing for and
administering the meeting, but it does not include
the cost of travel for meeting participants, which is
covered as a reimbursable category in 7.7.9.
(1) Include support for all meetings since the
previous invoice.
(2) Provide the meeting costs, a final agenda, and a
final list of attendees for each meeting.
Periodic: English-to-Language
Translation Dictionary
The respondents reply should describe the

(1) Provide evidence of Department acceptance and
provision on the CBT system of an on-screen English- approval (email is acceptable) of the English-toto-heritage language translation dictionary for
Language Translation Dictionary.
English Language Learners. ITN Ref 7.3.11.2
Periodic: English-to-Sign Language

The respondents reply may describe the provision

on the CBT system of an on-screen English-to-sign
language translation dictionary. ITN Ref 7.3.11.2
Fall ELA/L Retake: Inventory missing

materials and provide a report.
Once all materials are scanned in and inventoried, 1) Final missing materials report.
2) Provide Department email reflecting approval of
districts are contacted and asked to locate and
the report.
return any secure materials identified as missing.
After a thorough search has been conducted by
districts, materials have been accounted for, and
any additional searches have been exhausted at the
contractors facilities, the contractor provides a final
report listing any outstanding missing materials. ITN
Reference 7.4.2

approval (email is acceptable) of the English-to-Sign
E
Due Date(s)
(Dates to be negotiated.) 2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
(TBD - Part of negotiation) 2015

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
After each test administration, contingent on final
administration dates. Finalize during negotiations.
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
F
Deliverable Price

A
1
Test Administration
B
Spring ELA/L Retake: Inventory missing
the report.
report listing any outstanding missing materials.
ITN Reference 7.4.2
Due Date(s)
2016
Inventory missing materials and provide districts are contacted and asked to locate and
the report.
a report.
ITN Reference 7.4.2

2015
the report.
ITN Reference 7.4.2

2015
118
119
120
121
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
122
Test Administration
123
124
125
126
127
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
128
Test Administration
129
130
131
132
133
Fall EOC: Inventory missing materials

and provide a report.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
F
Deliverable Price

A
1
Test Administration
134
135
136
137
Winter EOC: Inventory missing materials Once all materials are scanned in and inventoried, 1) Final missing materials report.
the report.
ITN Reference 7.4.2
Due Date(s)
2015
Spring EOC: Inventory missing materials Once all materials are scanned in and inventoried, 1) Final missing materials report.
the report.
ITN Reference 7.4.2

2015
the report.
ITN Reference 7.4.2

2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
138
Test Administration
139
140
141
142
143
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
144
Test Administration
145
146
147
148
149
150
Summer EOC: Inventory missing

2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
F
Deliverable Price

A
1
Test
Administration
151
152
153
154
155
156
157 Test Administration
158
159
160
161
162
164
165
166
167
168
169
B
Fall ELA/L Retake: Develop packaging
and distribution specifications for each
test administration.
A document detailing all processes, procedures, and (1) Digital copy of the approved specifications.
requirements for the packaging and distribution of (2) Provide Department email reflecting approval of
test materials, ancillary materials, and student
the Specifications.
reports. ITN Reference 7.4.1
Spring ELA/L Retake: Develop packaging A document detailing all processes, procedures, and (1) Digital copy of the approved specifications.
and distribution specifications for each requirements for the packaging and distribution of (2) Provide Department email reflecting approval of
the Specifications.
Spring ELA/L and Mathematics: Develop A document detailing all processes, procedures, and (1) Digital copy of the approved specifications.
packaging and distribution specifications requirements for the packaging and distribution of (2) Provide Department email reflecting approval of
for each test administration.
the Specifications.
E
Due Date(s)
Prior to each test administration, TBD based on
packaging requirements of the selected contractor
(must be finalized in sufficient time for the
contractor to complete the work before the critical
delivery dates). Finalize during negotiations.
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
F
Deliverable Price

A
1
Test
Administration
170
171
172
173
174
175
177
178
179
180
181
183
184
185
186
187
188
B
Fall EOC: Develop packaging and
distribution specifications for each test
administration.
Winter EOC: Develop packaging and

administration.
Spring EOC: Develop packaging and

administration.
requirements for the packaging and distribution of (2)Provide Department email reflecting approval of
the Specifications.
the Specifications.
the Specifications.
E
Due Date(s)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2015.
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
F
Deliverable Price

A
1
Test
Administration
189
190
191
192
193
194
195
B
Summer EOC: Develop packaging and
administration.
the Specifications.
PERIODIC: ELA/L Retake: Deliver test

administration manuals, including
packing lists, work folders, applicable
memos, etc.
For each administration, all test administration

manuals and ancillary materials must be
successfully developed and delivered to districts by
the critical or extremely critical date listed in
Appendix B. ITN Reference 7.4.1
(1) Provide proof of delivery spreadsheet listing the

following for each shipment: date/time delivered,
who received materials, destination address,
quantity and type of material delivered. Must show
proof of successful delivery to all participating
districts.
Test Administration
PERIODIC: ELA/L and Mathematics

Computer-Based Tests: Deliver test
memos.


districts.
Test Administration

Paper-Based Test: Deliver test
memos.

Appendix B. ITN Reference 7.4.1, 7.4.6.1, and 7.4.7

districts.
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
E
Due Date(s)
2015.
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
See Appendix B (Contingent upon final, approved
administration dates. Will be determined during
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
F
Deliverable Price

A
1
Test Administration
PERIODIC: EOC: Deliver test
memos.
Fall ELA/L Retake: Deliver paper-based

test materials, including ancillary
materials (i.e., memos, packing lists,
materials needed for return materials)
and accommodations.
For each administration, all test materials and

ancillary materials must be successfully developed following for each shipment: date/time delivered,
and delivered to districts by the critical or extremely who received materials, destination address,
critical date listed in Appendix B. This includes paper- quantity and type of material delivered. Must show
based accommodations for computer-based
assessments. ITN Reference 7.3.8, 7.4.1, and 7.4.7 districts.
Test Administration
Spring ELA/L Retake: Deliver paperbased test materials, including ancillary

and accommodations.

Test Administration
Spring ELA/L and Mathematics: Deliver

paper-based test materials, including
ancillary materials (i.e., memos, packing
lists, materials needed for return
materials) and accommodations.

216
217
218
219
220
221
222
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
districts.
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
F
Deliverable Price

A
1
Test Administration
Fall EOC: Deliver paper-based test
materials, including ancillary materials
(i.e., memos, packing lists, materials
needed for return materials) and
accommodations.
Test Administration
Winter EOC: Deliver paper-based test

accommodations.

Test Administration
Spring EOC: Deliver paper-based test

accommodations.

Test Administration
Summer EOC: Deliver paper-based test

accommodations.

242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
E
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
F
Deliverable Price

A
1
Test Administration
Fall ELA/L Retake: Deliver
preidentification labels and rosters.
For each administration, all preidentification labels
and rosters must be successfully delivered to
districts by the critical or extremely critical date
listed in Appendix B. ITN References 7.5.2, 7.4.7.3,
and 7.4.7.4
districts.
Test Administration
Spring ELA/L Retake: Deliver


and 7.4.7.4

districts.
Test Administration


and 7.4.7.4

districts.
Test Administration
Fall EOC: Deliver preidentification labels For each administration, all preidentification labels
and rosters.
and 7.4.7.4

districts.
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
E
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
F
Deliverable Price

A
1
Test Administration
Winter EOC: Deliver preidentification
labels and rosters.
and 7.4.7.4
districts.
Test Administration
Spring EOC: Deliver preidentification

labels and rosters.

and 7.4.7.4

districts.
Test Administration
Summer EOC: Deliver preidentification

labels and rosters.

and 7.4.7.4

districts.
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
314
315
316
317
318
Fall ELA/L Retake: Deliver and maintain Provide evidence that the online management
the online management system for the system met the requirements of the contract and
life of the contract.
was operable for the invoice period. ITN Reference
7.4.8
(1) Provide a report listing met requirements and any

(or no) malfunctions or unavailability of the site that
occurred during the period, signed by the
Department team. Delivered at the initiation of the
contract; report should cover from 15 days before
the previous invoice period through 15 days prior to
the current invoice billing date.
E
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
November 2015
November 2016
(Renewal 1 - November 2017)
F
Deliverable Price

A
1
Test
Administration
319
320
321
322
323
324
326
327
328
329
330
331
333
334
335
336
337
339
340
341
342
343
Due Date(s)
Deliverable Price
Spring ELA/L Retake: Deliver and
maintain the online management
system for each assessment.
Provide evidence that the online management
system met the requirements of the contract and
7.4.8

and maintain the online management

7.4.8

Fall EOC: Deliver and maintain the

online management system for each
assessment.

7.4.8

November 2015
November 2016
Winter EOC: Deliver and maintain the

assessment.

7.4.8

February 2016
February 2017
(Renewal 1 - February 2018)
May 2016
May 2017
(Renewal 1 - May 2018)
May 2015
May 2016
May 2017

A
1
Test
Administration
344
345
346
347
348
349
Due Date(s)
Deliverable Price
Spring EOC: Deliver and maintain the
assessment.
7.4.8
Summer EOC: Deliver and maintain the

assessment.

7.4.8

357
359
360
361
362
Fall ELA/L Retake: Deliver a computerbased assessment platform that meets

the specifications in the contract for
each computer-based administration.
Provide evidence that the computer-based

assessment platform met the requirements of the
contract and was operable for the invoice period.
This includes the availability of the system check
and an infrastructure test. ITN Reference 7.4.9

November 2015
November 2016
363
365
366
367
368
Spring ELA/L Retake: Deliver a computer- Provide evidence that the computer-based
based assessment platform that meets assessment platform met the requirements of the

May 2016
May 2017
350
352
353
354
355
356
369
August 2015
August 2016
August 2017
(Renewal 1 - August 2018)
August 2015
August 2016
August 2017

A
1
Test
Administration
370
371
372
373
374
375
Due Date(s)
Deliverable Price
Spring ELA/L and Mathematics: Deliver a
computer-based assessment platform
that meets the specifications in the
contract for each computer-based
administration.
376
378
379
380
381
Fall EOC: Deliver a computer-based

assessment platform that meets the
specifications in the contract for each
computer-based administration.


November 2015
November 2016
382
384
385
386
387
Winter EOC: Deliver a computer-based



February 2016
February 2017
Spring EOC: Deliver a computer-based



388
390
391
392
393
394
395
May 2015
May 2016
May 2017
May 2015
May 2016
May 2017

A
1
Test
Administration
396
397
398
399
400
401
402
404
405
406
407
408
Due Date(s)
Deliverable Price
Summer EOC: Deliver a computer-based
PERIODIC: Develop and deliver approved

practice tests and tutorials for the
computer-based and paper-based
assessments (sample test materials and
computer-based practice tests).
According to production schedules, all practice

materials must be developed and approved on a
the practice tests and tutorials that reflect the
timeline to be established that offers students
requirements for each year of the assessment. For
enough time to practice prior to each
year 1 of the contract, this will be PDFs for all grades
administration. All practice tests (computer-based and subjects, and computer-based practice tests for
and paper-based) must contain 10-20 sample items. all grades and subjects that will be offered as
For the paper-based practice materials, there should computer-based subjects the first year. As grades
be a separate test booklet and a separate answer
and subjects are added each year (per the schedule
key booklet for teachers. These paper-based
in Appendix A), the appropriate computer-based
practice materials are produced in ADA-compliant practice tests must also be released. This delivery
PDF files for posting. Practice materials must also be includes the annual delivery of requested paperproduced for all accommodated versions (computer- based accommodated versions, which are the only
based accommodations and paper-based
versions that are shipped to districts. (All others are
accommodations, such as braille and large print).
published or linked for distribution.)
ITN Reference 7.3.7.5 and 7.4.9.6
PERIODIC: Develop and deliver a training

plan for the initial training and supply
materials to support ongoing training for
the audiences identified in the contract.
The contractor must train Department, district, and

school staff on the setup, features, management,
and use of the computer-based platform and online
management system at the beginning of the
contract. ITN Reference 7.4.7.1 and 7.4.9.8
August 2015
August 2016
August 2017
November 2014
November 2015
November 2016
409
Test Administration
410
(1) Provide Department email confirming and

approving completion of the initial training and
completion of the materials to support ongoing
training.
(2) Provide Department email approving the final,
posted training materials for each administration.
September 2014

A
1
Test
Administration
411
412
413
414
415
417
418
419
420
421
423
424
425
426
PERIODIC: Develop and deliver materials The contractor must train Department, district, and (1) Provide Department email approving the final,
to support ongoing training for the
audiences identified in the contract.
management system. ITN Reference 7.4.7.1 and
7.4.9.8
The contractor will provide customer service and
PERIODIC: Provide customer service
throughout the year, with additional
provide call/email logs from each week to the
customer service provided at peak times Department for review. ITN Reference 7.4.10
leading up to and following each test
administration.
PERIODIC: Inventory, store, and destroy The contractor will inventory and store unused test
used and unused test materials as
and answer books, manuals, and other materials for
described in the contract.
a period of at least six months, and used documents
containing student responses for a period of two
years. Destruction of secure documents must be
requested in writing and authorized by the
Department. ITN Reference 7.4.3
Due Date(s)
August 2015
August 2016
(1) Provide a report listing met requirements and any November 2015
(or no) errors or unavailability of the customer
November 2016
service system that occurred during the period,
November 2017
signed by the Department team. Invoice at the end of (Renewal 1 - November 2018)
each assessment year (end of Summer EOC
administration) for the assessments from the
previous year.
(1) Provide Department approval to destroy the
materials and evidence of destruction. Invoice at the
end of each assessment year (end of Summer EOC
administration) for the assessments from that year.
November 2015
November 2016
November 2017
427
Test Administration
Fall ELA/L Retake: Produce reports and

recommendations from all online
comment forms.
Includes data file of all responses, analysis, and

recommended actions, with summary of selected
the comment form summaries and analyses.
response questions from all online comment forms.
ITN Reference 7.4.1 and 7.4.7.7
Test Administration
Spring ELA/L Retake: Produce reports

and recommendations from all online
comment forms.

428
429
430
431
432
433
434
435
436
437
438
439

negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Test Administration
Spring ELA/L and Mathematics: Produce Includes data file of all responses, analysis, and
reports and recommendations from all recommended actions, with summary of selected
online comment forms.
Test Administration
Fall EOC: Produce reports and

comment forms.

Test Administration
Winter EOC: Produce reports and

comment forms.

Test Administration
Spring EOC: Produce reports and

comment forms.

440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Test Administration
Summer EOC: Produce reports and
comment forms.
Test Administration
PERIODIC: Deliver the Infrastructure

Trial Test
Develop and make available an infrastructure test,

consisting of mock items, that replicates the size of
the largest CBT and is run on the actual testing
platform (i.e., users log in to the testing platform as
if they were logging into a live test). ITN References
7.4.8 and 7.4.9.5

the final product and its availability. Product must be
available at least 6 months prior to the first CBT
administration. Updates to the test may be
requested during the life of the contract, depending
on its performance when used by school districts.
Test Administration
Fall ELA/L Retake: Deliver ComputerBased Test forms (including

accommodated versions)
Test forms are developed and available for use on

the first day of each CBT administration ITN
Reference 7.3.11.2

the final product and its availability.
Test Administration
Spring ELA/L Retake: Computer-Based

Test form delivery (including

Reference 7.3.11.2

466
467
468
469
470
471
472
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
August 1, 2014
473
474
475
476
477
478
479
480
481
482
483
484
485

negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Test Administration
Computer-Based Test form delivery
(including accommodated versions)
Reference 7.3.11.2
Test Administration
Fall EOC: Computer-Based Test form

delivery (including accommodated
versions)

Reference 7.3.11.2

Test Administration
Winter EOC: Computer-Based Test form Test forms are developed and available for use on
versions)
Reference 7.3.11.2
(1) Provide Department email approval of the final

product and its availability.
Test Administration
Spring EOC: Computer-Based Test form Test forms are developed and available for use on
versions)
Reference 7.3.11.2

486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Test Administration
Summer EOC: Computer-Based Test
form delivery (including accommodated the first day of each CBT administration ITN
versions)
Reference 7.3.11.2
Test Administration
Fall ELA/L Retake: Retrieve used and

unused test materials after testing
Test Administration
Spring ELA/L Retake: Retrieve used and After each test administration, the contractor will
retrieve test materials from the sites identified in
the ITN for scanning, scoring, and storing, as
described in the ITN. This includes accommodated
materials and all used and unused materials. ITN
Reference 7.4.1.
Test Administration
Spring ELA/L and Mathematics: Retrieve After each test administration, the contractor will
used and unused test materials after
testing
Reference 7.4.1.
512
513
514
515
516
517
518
519
520
521
522
523
After each test administration, the contractor will

Reference 7.4.1.
524
525
526
527
528
529
530
531
532
533
534
535
536
537
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
(1) Provide a proof of retrieval spreadsheet listing
the following for each shipment: date/time received, administration dates. Will be determined during
quantity and type of material received, and the
negotiations.)
2015
associated district. Must show proof of successful
2016
delivery to all participating districts.
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
the following for each shipment: date/time received,

negotiations.)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Test Administration
Fall EOC: Retrieve used and unused test After each test administration, the contractor will
materials after testing
Reference 7.4.1.
Test Administration
Winter EOC: Retrieve used and unused

test materials after testing

Reference 7.4.1.
Test Administration
Spring EOC: Retrieve used and unused


Reference 7.4.1.
Test Administration
Summer EOC: Retrieve used and unused After each test administration, the contractor will
Reference 7.4.1.
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
D
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Scoring
and Reporting
564
565
566
B
PERIODIC: Develop Annual Scoring and
Reporting Specifications document
567
568
569
570 Scoring and Reporting
571
572
573
574
576
577
578
579
581
582
583
584
585
587
588
589
590
C
The specific processes, guidelines, requirements,
and schedule to be followed in scanning, scoring
and reporting all of the assessments are to be
provided annually to the Department as detailed,
expert-developed, and referenced specifications.
ITN Reference 7.5.1
D
1) Completed specifications posted to the online
management system.
(2) Provide Department email approval of the Scoring
and Reporting Specifications
E
Due Date(s)
April 2014
April 2015
April 2016
(Renewal 1 - April 2017)
Fall ELA/L Retake: Development of PreID The Contractor will collect and monitor district
freq reports for Department and
PreID data submission and will submit PreID
districts to review.
Frequency Reports and the reports of issues
encountered during the PreID upload for all
assessments. ITN Reference 7.5.2
(1) PreID frequency reports posted to the online

management system.
(2) Provide Department email reflecting receipt of
the PreID frequency reports.
Spring ELA/L Retake: Development of

PreID freq reports for Department and
The Contractor will collect and monitor district


management system.
(Renewal 2 - September 2018)

January 2016
January 2017
(Renewal 1 - January 2018)

Development of PreID freq reports for
Department and districts to review.


management system.

January 2015
January 2016
January 2017
Fall EOC: Development of PreID freq

reports for Department and districts to
review.


management system.
September 2015
September 2016

September 2015
September 2016
F
Deliverable Price

A
1
Scoring
and Reporting
591
592
593
594
595
597
598
599
600
601
603
604
605
606
607
609
610
611
612
Winter EOC: Development of PreID freq The Contractor will collect and monitor district
reports for Department and districts to PreID data submission and will submit PreID
review.
management system.
Due Date(s)
October 2015
October 2016
(Renewal 1 - October 2017)
Spring EOC: Development of PreID freq

review.


management system.

January 2015
January 2016
January 2017
Summer EOC: Development of PreID



management system.
Fall ELA/L Retake: Development of the

file layout and Full-State PreID data file
in .txt format
The Contractor will work with the Department to

create a PreID layout for each administration for the
district to use to upload student demographic and
testing information. The contractor will post the
layout to online management system. After each
PreID window closes, the contractor will create and
send to the Department a full state PreID file that
includes a record for each student and test
assigned. ITN Reference 7.5.2
(1) Final PreID file layout posted to the online

management system and Secure File Transfer
Protocol (SFTP).
(2) PreID data file posted to the SFTP.
(3) Provide Department email reflecting the receipt
and approval of the file layout and data file.

June 2015
June 2016
June 2017
(Renewal 1 - June 2018)
September 2015
September 2016
F
Deliverable Price

A
1
Scoring
and Reporting
613
614
615
616
617
619
620
621
622
623
625
626
627
628
the file layout and Full-State PreID data create a PreID layout for each administration for the
file in .txt format
Protocol (SFTP).
Due Date(s)
January 2016
January 2017

Development of the file layout and Full- create a PreID layout for each administration for the
State PreID data file in .txt format

Protocol (SFTP).
January 2015
January 2016
January 2017
Fall EOC: Development of the file layout The Contractor will work with the Department to
and Full-State PreID data file in .txt
format

Protocol (SFTP).
September 2015
September 2016
F
Deliverable Price

A
1
Scoring
and Reporting
629
630
631
632
633
635
636
637
638
639
641
642
648
649
650
Winter EOC: Development of the file
layout and Full-State PreID data file in
.txt format
Protocol (SFTP).
Due Date(s)
October 2015
October 2016
Spring EOC: Development of the file

.txt format


Protocol (SFTP).
January 2015
January 2016
January 2017
Summer EOC: Development of the file

.txt format


Protocol (SFTP).
June 2015
June 2016
June 2017
643
644
645
647

special documents transcription logs
The Contractor will use a Special Document Tracking (1) Final transcription log posted to SFTP.
log to inventory and track the processing of all
special documents. The contractor will provide the the special documents transcription log.
final transcription log to Department. ITN Reference
7.5.4.4

October 2015
October 2016
May 2016
F
Deliverable Price

A
1
652
653
654
655
657
658
659
660
661
663
664
665
666
668
669
670
671
673
674
675
676
677
679
680
681
682
683
the special documents transcription logs log to inventory and track the processing of all
7.5.4.4
Development of the special documents
transcription logs
Fall EOC: Development of the special

documents transcription logs
7.5.4.4
7.5.4.4
Winter EOC: Development of the special The Contractor will use a Special Document Tracking (1) Final transcription log posted to SFTP.
7.5.4.4
Spring EOC: Development of the special The Contractor will use a Special Document Tracking (1) Final transcription log posted to SFTP.
7.5.4.4
Summer EOC: Development of the

7.5.4.4
E
Due Date(s)
May 2017
May 2015
May 2016
May 2017
October 2015
October 2016
December 2015
December 2016
(Renewal 1 - December 2017)
May 2015
May 2016
May 2017
July 2015
July 2016
July 2017
(Renewal 1 - July 2018)
F
Deliverable Price

A
1
Scoring
and Reporting
684
685
686
687
Fall ELA/L Retake: Early district and fullstate scan/edit and CBT extract data
files: Development of the file layouts,
scan/edit and CBT extract data files
The contractor will work with the Department to
develop the layouts for scan/edit and CBT extract
data files. For each administration, the contractor
unpacks and scans the documents to be scored and
provides the scan files by grades for ELA/L and
Mathematics and by subject for EOC. The Edited
scanned data are loaded to online management
system with the extracts for any CBT data.
Additional processing is performed to resolve PreID
data with gridded data, invoke any defaults
necessary, and make some transformations for final
scoring and creation of the State Student Results file
during the editing process. ITN References 7.5.3,
7.5.6.2, and 7.5.6.4
(1) Final approved file layouts, early/full state
scan/edit, and CBT extract data files, posted to online
management system and SFTP.
(2) Provide Department email reflecting receipt and
approval of the file layouts and data files.
Due Date(s)
October 2015
October 2016
688
690
691
692
Spring ELA/L Retake: Early district and

full-state scan/edit and CBT extract data

7.5.6.2, and 7.5.6.4

March 2016
March 2017
(Renewal 1 - March 2018)
693
F
Deliverable Price

A
1
Scoring
and Reporting
694
695
696
697
698
Spring ELA/L and Mathematics: Early
district and full-state scan/edit and CBT
extract data files: Development of the
file layouts, scan/edit and CBT extract
data files
7.5.6.2, and 7.5.6.4
Due Date(s)
March 2015
March 2016
March 2017
699
701
702
703
Fall EOC: Early district and full-state

scan/edit and CBT extract data files:
Development of the file layouts,

7.5.6.2, and 7.5.6.4

October 2015
October 2016
704
F
Deliverable Price

A
1
Scoring
and Reporting
705
706
707
708
Winter EOC: Early district and full-state
7.5.6.2, and 7.5.6.4
Due Date(s)
December 2015
December 2016
709
711
712
713
714
Spring EOC: Early district and full-state


7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017
715
F
Deliverable Price

A
1
Scoring
and Reporting
716
717
718
Summer EOC: Early district and full-state
7.5.6.2, and 7.5.6.4
719
720
721
723
724
725
726
728
729
730
731
733
734
735
736
737
Fall ELA/L Retake: Anomalous score data The contractor will analyze the full state SSR and
(1) Anomalous score data files posted to SFTP.
files: Development of the anomalous
return a Suppression file with student invalidation (2) Provide Department email reflecting receipt of
score data files
to the Department. ITN Reference 7.5.6.3
the data files.
Spring ELA/L Retake: Anomalous score

data files: Development of the
anomalous score data files
The contractor will analyze the full state SSR and

the data files.

Anomalous score data files:
Development of the anomalous score
data files

the data files.
E
Due Date(s)
July 2015
July 2016
July 2017

October 2015
October 2016
April 2016
April 2017
April 2015
April 2016
April 2017
F
Deliverable Price

A
1
Scoring
and Reporting
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
B
Fall EOC: Anomalous score data files:
data files
the data files.
Scoring and Reporting
Winter EOC: Anomalous score data files: The contractor will analyze the full state SSR and
data files
the data files.
Spring EOC: Anomalous score data files: The contractor will analyze the full state SSR and
data files
the data files.
Summer EOC: Anomalous score data

score data files

the data files.
Fall ELA/L Retake: Initial reporting SSRs

and DSRs data files: Development of
SSR/DSR file layouts and multiple
versions of the SSR and DSR data files.

develop the State Student Results (SSR) and District
Student Results (DSR) file layouts and use the layout
to develop the SSR/DSR data files. The SSR files
contain all students tested in the State. A DSR file
contains data for the students in that district. ITN
Reference 7.5.5
762
763
764
(1) Final initial reporting SSR and DSR data files

posted to SFTP and online management system.
E
Due Date(s)
October 2015
October 2016
December 2015
December 2016
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017
October 2015
October 2016

F
Deliverable Price

A
1
Scoring
and Reporting
765
766
767
768
769
771
772
773
774
775
777
778
779
780
782
783
784
785
Spring ELA/L Retake: Initial reporting
SSRs and DSRs data files: Development
of SSR/DSR file layouts and multiple
Reference 7.5.5
Due Date(s)
April 2016
April 2017
Spring ELA/L and Mathematics: Initial

reporting SSRs and DSRs data files:
Development of SSR/DSR file layouts
and multiple versions of the SSR and
DSR data files.

Reference 7.5.5

April 2015
April 2016
April 2017

Reference 7.5.5


Reference 7.5.5

Fall EOC: Initial reporting SSRs and DSRs

data files: Development of SSR/DSR file
layouts and multiple versions of the SSR
and DSR data files.
Winter EOC: Initial reporting SSRs and

DSRs data files: Development of

October 2015
October 2016
December 2015
December 2016
F
Deliverable Price

A
1
Scoring
and Reporting
786
787
788
789
B
Spring EOC: Initial reporting SSRs and
790
791
793
794
795
796
797
799
800
801
802
804
805
806
807
Summer EOC: Initial reporting SSRs and

Fall ELA/L Retake: Initial reporting SARs

and DARs data files: Development of
SAR/DAR file layouts and multiple
versions of the SAR and DAR data files.

SARs and DARs data files: Development
of SAR/DAR file layouts and multiple
Reference 7.5.5

Reference 7.5.5

The contractor will work with Department Scoring

and Reporting Staff to develop the State Aggregated
Results (SAR) and District Aggregated Results (DAR)
file layouts and use the layout to develop the
SAR/DAR data files. The SSR files contain all of the
scores and codes described in the SSR layout for all
students tested in the State. A DSR file contains data
for the students in that district. ITN Reference 7.5.5
(1) Final initial reporting SAR and DAR data files

posted to online management system and SFTP.
approval of the data files.


E
Due Date(s)
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017
October 2015
October 2016
April 2016
April 2017
F
Deliverable Price

A
1
Scoring
and Reporting
808
809
810
811
812
813
815
816
817
818
820
821
822
823
reporting SARs and DARs data files:
Development of SAR/DAR file layouts
and multiple versions of the SAR and
DAR data files.
Due Date(s)
April 2015
April 2016
April 2017
Fall EOC: Initial reporting SARs and DARs

data files: Development of SAR/DAR file
layouts and multiple versions of the SAR
and DAR data files.


October 2015
October 2016
Winter EOC: Initial reporting SARs and

DARs data files: Development of


December 2015
December 2016


F
Deliverable Price

A
1
Scoring
and Reporting
824
825
826
827
B
Spring EOC: Initial reporting SARs and
828
829
831
832
Summer EOC: Initial reporting SARs and

833
834
835
837
838
839
840
Fall ELA/L Retake: Late reporting SSRs

SSR/DSR and multiple versions of DAR
data files.
Due Date(s)
April 2015
April 2016
April 2017



for the students in that district. ITN Reference
7.5.5.2 and 7.5.6.6
(1) Final late reporting SSR and DSR data files posted November 2015
to online management system and SFTP.
November 2016
(2) Provide Department email reflecting receipt and (Renewal 1 - November 2017)
July 2015
July 2016
July 2017
F
Deliverable Price

A
1
Scoring
and Reporting
841
842
843
Spring ELA/L Retake: Late reporting SSRs
data files.
7.5.5.2 and 7.5.6.6
(1) Final late reporting SSR and DSR data files posted

7.5.5.2 and 7.5.6.6
844
845
847
848
849
850
851
853
854
855
856
Spring ELA/L and Mathematics: Late

Development of SSR/DSR and multiple
versions of DAR data files.
Fall EOC: Late reporting SSRs and DSRs The contractor will work with Department Scoring
data files: Development of SSR/DSR and and Reporting Staff to develop the State Aggregated
multiple versions of DAR data files.
7.5.5.2 and 7.5.6.6
E
Due Date(s)
June 2016
June 2017
June 2015
June 2016
June 2017

November 2016
F
Deliverable Price

A
1
Scoring
and Reporting
857
858
859
B
Winter EOC: Late reporting SSRs and
data files.
860
861
863
864
865
Spring EOC: Late reporting SSRs and

data files.
866
867
869
870
871
872
873
Summer EOC: Late reporting SSRs and

data files.
7.5.5.2 and 7.5.6.6

7.5.5.2 and 7.5.6.6

7.5.5.2 and 7.5.6.6
E
Due Date(s)
February 2015
February 2016
June 2015
June 2016
June 2017
October 2015
October 2016
October 2017
F
Deliverable Price

A
1
Scoring
and Reporting
874
875
876
877
878
880
881
882
883
885
886
887
888
889
891
892
893
B
Fall ELA/L Retake: Close out SSR data
files: Development of the closeout SSR
data files for all assessments.
The contractor will generate close out files after all (1) Final closeout SSR files posted to SFTP.
waves of late reporting are complete for each
administration of all assessments. For the EOC, the approval of the data files.
closeout SSR contains data for the first-time testers
for each administration. The Department will review
and approve these data files. ITN Reference 7.5.5.3
and 7.5.6.7
E
Due Date(s)
December 2015
December 2016
Spring ELA/L Retake: Close out SSR data The contractor will generate close out files after all (1) Final closeout SSR files posted to SFTP.
files: Development of the closeout SSR waves of late reporting are complete for each
and 7.5.6.7
October 2016
October 2017
Spring ELA/L and Mathematics: Close

out SSR data files: Development of the
closeout SSR data files for all
assessments.
and 7.5.6.7
October 2015
October 2016
October 2017
Fall EOC: Close out SSR data files:

Development of the closeout SSR data
files for all assessments.
and 7.5.6.7
December 2015
December 2016
F
Deliverable Price

A
1
Scoring
and Reporting
895
896
897
898
899
901
902
903
904
905
907
908
909
910
911
913
914
915
916
917
Winter EOC: Close out SSR data files:
and 7.5.6.7
Due Date(s)
March 2015
March 2016
Spring EOC: Close out SSR data files:

and 7.5.6.7
October 2015
October 2016
October 2017
Summer EOC: Close out SSR data files:

and 7.5.6.7
October 2015
October 2016
October 2017
Periodic: Development of the

Demographic DSR/DAR data files
The Department will generate and provide the

approved Demographic SSR and SAR to the
contractor. The contractor will create Demographic
DSR and DAR files for the districts and post the
Demographic DSR/DAR files to the online
management system for districts to download. ITN
Reference 7.5.5.4 and 7.5.6.8
(1) Final Demographic DSRs and DARs data files

September 2015
September 2016
September 2017
F
Deliverable Price

A
1
918
919
920
921
922
923
924
925
926
927
B
Fall ELA/L Retake: Initial reporting files
and PDFs of reports for districts and
schools: Development of the data files
and PDFs of reports for the initial
reporting
The contractor will post the data files and reports to (1) Final data files and reports posted to online
the online management system for districts and
schools to download. ITN Reference 7.5.5
approval of files and reports.
Due Date(s)
See Appendix B. Dates to be determined as part of
contract negotiation.
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
files and PDFs of reports for districts and the online management system for districts and
schools: Development of the data files schools to download. ITN Reference 7.5.5
(2) Provide Department email reflecting receipt and 2016
2017
reporting
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

The contractor will post the data files and reports to
reporting files and PDFs of reports for
districts and schools: Development of
the data files and PDFs of reports for the
initial reporting
(1) Final data files and reports posted to online

Fall EOC: Initial reporting files and PDFs The contractor will post the data files and reports to
of reports for districts and schools:
Development of the data files and PDFs schools to download. ITN Reference 7.5.5
of reports for the initial reporting

928
929
930
931
932
933
934
935
936
937
938

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
F
Deliverable Price

A
1
Winter EOC: Initial reporting files and
PDFs of reports for districts and schools: the online management system for districts and
Spring EOC: Initial reporting files and

PDFs of reports for districts and schools:
Development of the data files and PDFs
Summer EOC: Initial reporting files and

Fall ELA/L Retake: Initial reporting

Student Reports and CD/DVDs or jump
drives. Submit proof of delivery:
Development of the student reports and
media for the initial reporting
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
D
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
The contractor will print and deliver initial reporting (1) Provide Department email reflecting approval of See Appendix B. Dates to be determined as part of
Student Reports and CD/DVDs or jump drives. The the samples of the student reports and CD/DVD or
contractor will submit proof of delivery to the
jump drives.
2015
Department after all reports and media are
(2) Provide proof of delivery spreadsheet listing the 2016
(Renewal 1 - 2017)
delivered to the districts. ITN Reference 7.5.5
(Renewal 2 - 2018)
(Renewal 3 - 2019)
districts.
F
Deliverable Price

A
1
961
962
963
The contractor will print and deliver initial reporting
Student Reports and CD/DVDs or jump drives. The
964
965
966
967
968

reporting Student Reports and CD/DVDs
or jump drives. Submit proof of delivery:

969
970
D
the samples of the student reports and CD/DVD or
jump drives.
districts.
Due Date(s)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)

jump drives.
districts.

2015
2016
2017
(Renewal 1 - 2018)

jump drives.
districts.

2015
2016
(Renewal 1 - 2017)
971
972
973
974
975
976
Fall EOC: Initial reporting Student

Reports and CD/DVDs or jump drives.
Submit proof of delivery: Development
of the student reports and media for the
initial reporting

(Renewal 3 - 2020)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
F
Deliverable Price

A
1
977
978
979
Winter EOC: Initial reporting Student
initial reporting
980
981
jump drives.
districts.
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
Spring EOC: Initial reporting Student

initial reporting


jump drives.
districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Summer EOC: Initial reporting Student

initial reporting


jump drives.
districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
982
983
984
985
986
987
988
989
990
991
992
993
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Fall ELA/L Retake: Performance Task
Reports and proof of delivery:
Development of the hard copy of the
Performance Tasks reports
For each test administration, the student report will
include a students written response to selected
performance tasks in each subject that contains the
performance task. The responses will be reproduced
on a separate page that includes an explanation of
the tasks and the scoring criteria. The contractor
will print and deliver performance task reports.
Submit proof of delivery to the Department after all
reports and media are delivered to the districts. ITN
Reference 7.5.5
approval of the samples of the performance task
reports.
districts.
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
Spring ELA/L Retake: Performance Task


Reference 7.5.5

reports.
districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

Performance Task Reports and proof of
delivery: Development of the hard copy
of the Performance Tasks reports

Reference 7.5.5

reports.
districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Fall EOC: Performance Task Reports and
proof of delivery: Development of the
hard copy of the Performance Tasks
reports
Reference 7.5.5
reports.
districts.
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
Winter EOC: Performance Task Reports

and proof of delivery: Development of
the hard copy of the Performance Tasks
reports

Reference 7.5.5

reports.
districts.

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
Spring EOC: Performance Task Reports

reports

Reference 7.5.5

reports.
districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
F
Deliverable Price

A
1
1026
1027
1028
Summer EOC: Performance Task Reports
reports
Reference 7.5.5
reports.
districts.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
The contractor will print and deliver late Student

Reports and CD/DVDs or jump drives. The

approval of the samples of the late reporting student
reports and CD/DVD or jump drives.
districts.

2015
2016
(Renewal 1 - 2017)

districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
1029
1030
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1031
1032
1033
1034
1035
Fall ELA/L Retake: Late reporting

Student Reports and CD/DVDs. Submit
student reports and media for the late
reporting
(Renewal 2 - 2018)
(Renewal 3 - 2019)
1036
1037
1038
1039
1040
1041
Spring ELA/L Retake: Late reporting

reporting

(Renewal 3 - 2020)
F
Deliverable Price

A
1
1042
1043
1044
B
reporting Student Reports and CD/DVDs.
late reporting
C
1045
1046
1047
Fall EOC: Late reporting Student Reports

and CD/DVDs. Submit proof of delivery:
media for the late reporting

Winter EOC: Late reporting Student

Reports and CD/DVDs. Submit proof of
delivery: Development of the student
reports and media for the late reporting

1048
1049
1050
1051
districts.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)

districts.

2015
(Renewal 2 - 2019)
(Renewal 3 - 2020)
(1) Provide Department email reflecting receipt and See Appendix B. Dates to be determined as part of
approval of the samples of the late reporting student contract negotiation.
2015
(Renewal 1 - 2017)
quantity and type of material delivered. Must show (Renewal 2 - 2018)
(Renewal 3 - 2019)
districts.
1052
1053
1054
1055
1056
1057
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
F
Deliverable Price

A
1
1058
1059
1060
1061
B
Spring EOC: Late reporting Student
C
1062
districts.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1063
1064
1065
1066
1067
1068
1069
1071
1072
1073
1074
1076
1077
1078
1079
Summer EOC: Late reporting Student



districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Fall ELA/L Retake: Final Mock PreID

label/Roster, data files, Media &
Reports: Development of the mock
PreID label/Roster, Mock data files and
reports/medias for mock data.
The contractor will submit approved final mock

PreID label, data files and reports to the
Department via the SFTP or deliver the samples to
the Department. All the mock processes need to
follow the procedures specified in the SR
specifications. ITN Reference 7.5.6.1
(1) Mock PreID Label/Roster, data files, media and

reports delivered to the Department or posted to the
secure file transfer protocol (SFTP).
approval of the mock data files and reports/media.
September 2015
September 2016
Spring ELA/L Retake: Final Mock PreID



March 2016
March 2017
F
Deliverable Price

A
1
Scoring
and Reporting
1080
1081
1082
1083
1084
Spring ELA/L and Mathematics: Final
Mock PreID label/Roster, data files,
Media & Reports: Development of the
mock PreID label/Roster, Mock data files
and reports/medias for mock data.
Due Date(s)
March 2015
March 2016
March 2017
Fall EOC: Final Mock PreID label/Roster,

data files, Media & Reports:
Development of the mock PreID
label/Roster, Mock data files and


September 2015
September 2016
Winter EOC: Final Mock PreID



November 2015
November 2016
1095
1097
1098
1099
1100
Spring EOC: Final Mock PreID



March 2015
March 2016
March 2017
1101
1103
1104
1105
1106
1107
Summer EOC: Final Mock PreID



June 2015
June 2016
June 2017
1085
1087
1088
1089
1090
1092
1093
1094
F
Deliverable Price

A
1
Fall ELA/L Retake: Handscoring of
Performance Tasks
Handscoring refers to the processes necessary for
determining the rating of a students response on
the writing prompts or on the performance-based
tasks in ELA/L and Mathematics that respondents
may propose. Both paper-based and computerbased tests will contain performance tasks, and so
the Contractor must have the capability to score
both hand-written and key-entered student
responses. The Department requires handscoring
processes that are reliable and valid, as well as
efficient in terms of time and expenditures. The
handscoring process must follow the handscoring
specifications. The final handscoring report, as
described in the ITN, is required as evidence for
payment. ITN Reference 7.5.4.5
Due Date(s)
(1) Provide Department email reflecting approval of See Appendix B. Dates to be determined as part of
the final handscoring report.
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
Spring ELA/L Retake: Handscoring of

Performance Tasks

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
F
Deliverable Price

A
1
Handscoring of Performance Tasks
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Fall EOC: Handscoring of Performance

Tasks

contract negotiation.2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
F
Deliverable Price

A
1
1130
1131
1132
1133
Winter EOC: Handscoring of
Performance Tasks
Due Date(s)
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
Spring EOC: Handscoring of

Performance Tasks

contract negotiation. 2015
1134
1135
1136
1137
1138
1139
1140
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
1141
1142
1143
1144
1145
1146
1148
1149
1150
1151
1153
1154
1155
1156
1158
1159
1160
1161
1162
Summer EOC: Handscoring of
Performance Tasks
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Fall ELA/L Retake: Final Report

Mockups: Development of the report
mockups for all assessments
The contractor will post the final approved report

mockups for all assessments to the SFTP. ITN
Reference 7.5.5
(1) Final set of approved report mockups posted to

the SFTP.
approval of the report mockups.
Spring ELA/L Retake: Final Report


Reference 7.5.5

the SFTP.

Report Mockups: Development of the
report mockups for all assessments

Reference 7.5.5

the SFTP.
June 2015
June 2016
January 2016
January 2017
January 2015
January 2016
January 2017
F
Deliverable Price

A
1
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
Fall EOC: Final Report Mockups:
Development of the report mockups for mockups for all assessments to the SFTP. ITN
all assessments
Reference 7.5.5
the SFTP.
Winter EOC: Final Report Mockups:

all assessments
Reference 7.5.5

the SFTP.
Spring EOC: Final Report Mockups:

all assessments
Reference 7.5.5

the SFTP.
Summer EOC: Final Report Mockups:

all assessments
Reference 7.5.5

the SFTP.

history files for EOC and
ELA/Mathematics
The contractor will create and update history files (1) History files posted to the SFTP.
after each test administration using the
requirements specified in the SR specifications. ITN approval of the history data files.
Reference 7.5.4.2

the history files for EOC and
ELA/Mathematics
Reference 7.5.4.2
E
Due Date(s)
June 2015
June 2016
October 2015
October 2016
January 2015
January 2016
January 2017
April 2015
April 2016
April 2017
December 2015
December 2016
October 2016
October 2017
F
Deliverable Price

A
1
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
Development of the history files for EOC after each test administration using the
and ELA/Mathematics
Reference 7.5.4.2
Fall EOC: Development of the history

files for EOC and ELA/Mathematics
Winter EOC: Development of the history The contractor will create and update history files (1) History files posted to the SFTP.
Reference 7.5.4.2
Spring EOC: Development of the history The contractor will create and update history files (1) History files posted to the SFTP.
Reference 7.5.4.2

ELA/Mathematics
Reference 7.5.4.2

APS (Alternate Passing Score) file for
Retake
The contractor will create and update an APS file for (1) APS file posted to the SFTP.
Retake test after each test administration using the (2) Provide Department email reflecting receipt and
rule specified in the SR specifications. NOTE: This is approval of the APS file.
only if policy decisions mean that an APS file is
needed. ITN Reference 7.5.4.2
1225
1226
1227
Reference 7.5.4.2
E
Due Date(s)
October 2015
October 2016
October 2017
December 2015
December 2016
March 2016
March 2017
October 2015
October 2016
October 2017
October 2015
October 2016
October 2017
November 2015
November 2016

F
Deliverable Price

A
1
Scoring
and Reporting
1228
1229
1230
1231
1232
1234
1235
1236
1237
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
the APS (Alternate Passing Score) file for Retake test after each test administration using the (2) Provide Department email reflecting receipt and
Retake
Due Date(s)
May 2016
May 2017
Fall ELA/L Retake: Manual Rescore Log:

Development of the log to record the
PBT test with passing scores for manual
rescoring.
The contractor will create a log to identify students

who qualified for manual rescoring and provide this
log to the Department. The log will follow the
requirements specified in the SR Specification. ITN
Reference 7.5.6.5
(1) Manual rescore log posted to the online

management system.
log.

October 2015
October 2016
Spring ELA/L Retake: Manual Rescore

Log: Development of the log to record
the PBT test with passing scores for
manual rescoring.

Reference 7.5.6.5

management system.
log.
Spring ELA/L and Mathematics: Manual

Rescore Log: Development of the log to
record the PBT test with passing scores
for manual rescoring.

Reference 7.5.6.5

management system.
log.
Fall EOC: Manual Rescore Log:

rescoring.

Reference 7.5.6.5

management system.
log.
April 2016
April 2017
April 2015
April 2016
April 2017
October 2015
October 2016
F
Deliverable Price

A
1
Scoring
and Reporting
1254
1255
1256
1257
1258
1260
1261
1262
1263
1264
1266
1267
1268
Winter EOC: Manual Rescore Log:
rescoring.
Reference 7.5.6.5
management system.
log.
Spring EOC: Manual Rescore Log:

rescoring.

Reference 7.5.6.5

management system.
log.
Summer EOC: Manual Rescore Log:

rescoring.

Reference 7.5.6.5

management system.
log.
Periodic: Writing Image CD/DVD or jump

drives and proof to delivery:
Development of the Writing Image
CD/DVD or jump drives.
The contractor will create and ship Writing images

for all students who submitted a folder to be scored
and ship these CDs to the districts with one CD/DVD
or jump drives for each grade tested. The contractor
will submit proof of delivery to the Department
after all of the Writing Image CD/DVD or jump
drives are delivered to districts. ITN Reference 7.5.5
1) Provide Department email reflecting receipt and

approval of the sample CD/DVD or jump drives.
districts.
1269
1270
1271
1272
1273
1274
1275
1276
Due Date(s)
Deliverable Price
December 2015
December 2016
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017
July 2015
July 2016
July 2017

A
1
Scaling,
Equating, Scoring,
1277
and Special Psychometric
1278
Studies
1279
The Contractor will deliver Test Construction
Specifications for ELA/Literacy and Mathematics,
Algebra 1, Algebra 2, Geometry, and Retake
Assessments. The final specifications shall be
evaluated and approved by the Department. To the
extent possible, the Department expects to be
involved in any test construction activities for 2015
assessments still to be completed after contract
execution. ITN Reference 7.6.0, 7.6.1, and 7.6.2
(1) Test Construction Specifications are secure, and
therefore the Contractor shall provide a report which
indicates the completed status of Specifications
posted to Secure File Transfer Protocol (SFTP).
approval of the Test Construction Specifications.
Periodic: Development of Test Forms for All paper-based and computer-based forms for the
Field Test and/or Operational
tests, as agreed upon by the Department and
Administrations
memorialized in the approved Test Construction
Specifications, must be submitted and approved.
ITN Reference 7.6.0 and 7.6.1
(1) Test forms are secure, and therefore the

contractor shall post them to SFTP. The Department
shall maintain a copy of the materials on file as
needed for evidence of completion of the
deliverable.
approval of the final paper-based and computerbased forms.
July 2014
July 2015
Detailed plan for sampling of students from the

(1) Final versions of sampling plans posted to SFTP.
State. Sampling plan must describe the initial
analyses on the SSR files, introducing plans for how approval of the sampling plans.
to collect data, and the list of post-hoc analyses for
representativeness checks when data from test
administrations are in for operational processing.
ITN Reference 7.6.2.2
July 2014
July 2015
Periodic: Development of the Test
Construction Specifications
1280
1281
1282
1283 Scaling, Equating, Scoring,
1284 Studies
1285
1286
1287
1288
Studies
1290
1291
1292
1293
1294
Studies
1296
1297
1298
1299
1300
Periodic: Development of the Sampling

Plans
Periodic: Drawing Samples for

Operational and/or Field Test
Administrations
Delivering the data files including sampled units

from the populations and the respective reports.
Sampling report must describe the initial analyses
on the SSR files, procedures for how to collect data,
evaluate the representativeness of resulting sample
and a list of post-hoc analyses for
ITN Reference 7.6.0, 7.6.1, and 7.6.2.2
(1) Data files, including sampled units from the

populations, posted to SFTP.
approval of the samples.
Due Date(s)
Deliverable Price
May 2014
May 2015
May 2016
July 2016
July 2016
August 2014
August 2015
August 2016

A
1
Scaling,
Equating, Scoring,
1301
and
Special
Psychometric
1302
Studies
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
Scaling, Equating, Scoring,

Studies

1314
Studies
1315
1316
1317
1318
Periodic: Development of the SES
Specifications
Preparing a detailed plan for conducting all scaling (1) SES Specifications (secure) posted to SFTP.
and equating plans and procedures, scoring plans
approval of the SES Specifications.
and procedures. The plan should include a
comprehensive and detailed timeline for all tasks
and events. The plan should also provide a detailed
description of the outcomes produced from the
operational and/or field test SES activities, including
data files and reports.
The plan must be vetted by TAC members.
Due Date(s)
December 2014
December 2015
December 2016
Periodic: Performing SES Activities
Conducting all scaling and equating, and scoring

activities as laid out in the specifications.
ITN Reference 7.6.0, 7.6.1, and 7.6.2
May 2015
May 2016
May 2017
(Renewal 3 -May 2020)
(Dates to be negotiated) 2015
2016
Spring ELA/L and Mathematics: Develop The contractors sampling specifications must
Calibration Sampling Specifications
include analyzing the most recent SSR files and
establishing a road map for gathering the required
data proven to represent the demographic
characteristics of the state and student
performance. The specifications will also address
the steps, procedures, and software required for
replication of the sampling procedures by the
Department. The specifications will be reviewed by
the Department and TAC members, if necessary,
and updated by the contractor for each subsequent
administration. ITN Reference 7.6.2.2
(1) Submit Final Scaled Item Parameter Estimates

and Scored Files (secure files) to the Department,
posted to SFTP.
approval of the files.
(1) Submit final Calibration Sampling Specifications
via SFTP.
approval of the specifications.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Scaling,
Equating, Scoring,
1319
and
Special
Psychometric
1320
1321
1322
1323
Spring EOC: Develop Calibration
Sampling Specifications
E
Due Date(s)
2016
2017
Preparing Technical Reports, as described in section

7.6.3.2. Note that Table of contents listed in 7.6.3.2
includes the minimum requirements for the report.
Collecting Feedback from the Department and
Technical Advisory Committee (TAC), finalizing the
reports after incorporating the feedback, and
submitting the final versions of the reports to the
Department. ITN Reference 7.6.3.2
(1) Submit initial versions and the final versions of

Technical Reports (secure) to the Department,
posted to SFTP.
approval of the technical reports.
December 2015
December 2016
December 2017
Periodic: Development of Scale Linking

Plan: Scale Linking Plan for transitioning
from FCAT 2.0 Reading & Mathematics
and EOC Algebra 1 & Geometry to the
Florida Standards Assessments
Scale Linking Plan for transitioning from FCAT 2.0

Reading & Mathematics and EOC Algebra 1 &
Geometry to the Florida Standards Assessments ITN
Reference 7.6.3.3

Scale Linking Plans (secure) to the Department,
posted to SFTP.
approval of the linking plans.
November 2014

Plan: Collecting Legal Opinion
Collecting legal opinion, subject to the Departments (1) Submit final version of the legal opinion
document (secure) to the Department, posted to
approval, about the defensibility of linking
SFTP.
methodology. ITN Reference 7.6.3.3
Periodic: Creating and Finalizing

Technical Reports
1328
1329

Studies
D
via SFTP.
Studies
1330
C
The contractors sampling specifications must
Studies
1324
1326 and Special Psychometric
1327
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

1332
1333
Studies
1334
October 2014
F
Deliverable Price

A
1
Scaling,
Equating, Scoring,
1335
Periodic: Linking Scales during
Conducting all linking operations and scoring
transitioning years to find out FCAT 2.0 activities as laid out in the approved linking
and EOC equivalents based on the
performance of students on the Florida
Standards Assessments
(1) Submit final look-up tables and scored files
(secure files) to the Department, posted to SFTP.

Studies
Periodic: Development of Standard

Setting Work Plan
(1) Submit the final version of Standard Setting Work January 2015
Plan and relevant materials, such as slides (secure
materials), posted to SFTP.
approval of the documents.

Studies
Periodic: Implement Achievement Level Hold the required meetings and collect
Standard Setting Meetings
recommended cut scores from all relevant groups,
as described in ITN Reference 7.6.3.4
(1) Submit the initial version of the Standard Setting October 2015
Technical Report (secure file), including
recommended cuts and impact data, to the
Department via SFTP.
approval of the technical report and excel tables with
interactive impact data and graphs.

Studies
Periodic: Finalizing Standard Setting

Technical Report
(1) Submit the final version of the Standard Setting

November 2015
Technical Report (secure file) to the Department via
SFTP.

Studies
May 2015
1336
Prepare a Standard Setting Work plan for setting
achievement level standards for ELA/Literacy,
Mathematics, Algebra 1 &2, and Geometry
assessments. Plan must include every detail of the
methodology, internal and external data files, and
the timeline for all of the activities during educator
and reactor panel meetings. The Plan must be
vetted by TAC members and the methodology must
be approved by the department. ITN Reference
7.6.3.4
1337
1338
1339

7.6.3.4.
Department.
ITN Reference 7 6 3 4
Due Date(s)
Deliverable Price

A
1
Scaling,
Equating, Scoring,
1340
Studies
1341
Studies
Periodic: Development of Vertical
Scaling Specifications
Preparing a detailed plan for conducting all vertical
scaling procedures. The plan should include a
and events, as described in section 7.6.3.5
ITN Reference 7.6.3.5 (Repeated in contract
renewal)
(1) Post the plan for Vertical Scaling (secure file) via
SFTP.
approval of the Vertical Specifications.
Due Date(s)
November 2014
Periodic: Performing Vertical Scaling

Activities and Preparing Technical
Report

activities as laid out in the specifications. ITN
Reference 7.6.3.5 (Repeated in contract renewal)
(1) Submit Final Technical Report and Scored SSR

Files (secure files), including vertically aligned scale
scores, to the Department via SFTP.
approval of the documents and files.
August 2015
Periodic: Development of Scale Drift

Study Specifications
Preparing a detailed plan for conducting scale drift

study. The plan should include a comprehensive and
detailed timeline for all tasks and events, as
described in section 7.6.3.6
(1) Submit the plan for conducting the scale drift

study (secure) to the Department via SFTP.
approval of the scale drift study specifications.
Periodic: Performing Scale Drift

Research and Preparing Technical
Report

Reference 7.6.3.6
(1) Submit Final Technical Report and rescaled item (Renewal 2 - August 2018)
parameters (secure files) to the Department via SFTP.
1343
1344

Studies
1345
Studies
1347
F
Deliverable Price

A
1
1348 Studies
1349
Studies
1351
Periodic: Development of Specifications Preparing a detailed plan for conducting
for Comparability Between Devices
comparability study. The plan should include a
Delivering Assessments
The plan should include the degree of
comparability of itemlevel and testlevel statistics
across devices.
The plan should include the degree of comparability
of itemlevel and testlevel statistics across various
input types.
For the questions above, the plan should address to
what degree of comparability can be obtained
through scaling items onto a single metric, linking or
concordance.
The plan should also include what interactions exist
among device, input type and item/task
characteristics, grade, subject, and relevant student
characteristics.
ITN Reference 7.6.3.6.1
(1) Submit the plan for conducting the comparability November 2014
study (secure file) to the Department via SFTP.
approval of the study specifications.
Periodic: Performing Study for

Comparability Between Devices
Delivering Assessments, Preparing its
Final Report, and Recommendations
with TAC Committee
(1) Submit Final Report and Recommendations to the August 2015

Department. These materials are secure, and
therefore contractor shall post them to SFTP.
Conducting all of the planned analyses, preparing its

technical report, and sharing the findings and
recommendations with TAC members.
Due Date(s)
Deliverable Price

A
1
Studies

Studies
1355
Studies
1357
Periodic: Development of a study plan
for linking Florida Standards
Assessments to other assessments
measuring the same standards
Preparing a detailed plan for conducting the linking
The plan should include a review of the most
promising and most feasible methods in linking,
discuss the technical feasibility of such methods,
and explain the process of establishing the linking
relationship(s) in detail. ITN Reference 7.6.3.6.2
(1) The plan for conducting comparability study is
November 2015
secure, and therefore contractor shall post them to
SFTP.
Periodic: Linking Florida Standards

Conducting all of the planned analyses, preparing a

final technical report, and sharing the results of
linked relationships with TAC committee to discuss
the validity of interpretations. ITN Reference
7.6.3.6.2
August 2016
(1) Submit Final Report and Recommendations
(secure files) to the Department via SFTP.

for the linking Florida Standards
Assessments to International
Assessments
The Contractors plan will include, review of the

most promising and most feasible methods likely to
be used in international linking, discuss the
technical feasibility of each method given the nature
and design of the Florida Standards Assessments
and those of international assessments such as PISA
and TIMSS, propose method(s) for linking the scales
of assessments, propose alternative ways for
collecting data, explain the process of establishing
the linking relationship(s) in detail, and provide a
and events, as described in section 7.6.3.9.
The contractor will share the report with the
Department and its TAC members. Upon approval
from the Department, the contractor will finalize
the plan for the linking.
(1) The plan for conducting the linking study is

SFTP.
1352
1353
November 2015
Due Date(s)
Deliverable Price

A
1
Studies
1377
Assessments
7.6.3.6.3
Periodic: Production Schedule
The production schedule shows the Departments

beginning and ending review dates by test
document for each review round. ITN Reference
7.7.2
(1) Appearance of schedule in weekly management

report
proposed or revised production schedule.
Periodic: Program Management

Meetings
The contractor will coordinate regular management

meetings between the contractor and Department
staff. The number and type of meetings are
provided in 7.7.5 of the ITN.

previous invoice.
Periodic: Program Management Staff
The work described in the ITN constitutes a large,

complex, challenging program that requires yearround activity and close attention to overlapping
tasks for different years and the impact on project
resources. The contractor is expected to provide a
sufficient number of highly-qualified personnel to
work closely with the Department to manage the
contract. Specific requirements are in 7.7.2 of the
ITN.
(1) Include support for program management staff

operational for the invoice period.
(2) Provide an organization chart with the
appropriate staff indicated for each invoice period.
(Quarterly invoices to occur in February, May,
August, and November)
1358
1359
1360 Program Management
1361
1362
1363
1364
1365
1367
1368
1369
1370
1371
1373
1374
1375
1376
Due Date(s)
Deliverable Price
April 15, 2014

April 15, 2015
April 15, 2016
(Renewal 1 - April 15, 2017)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

A
1
Program
Management
1378
1379
1380
1381
1382
1383
1385
1386
1387
1388
1389
Reports: Weekly Management and
Action Item Report
The Weekly Management Report and Action Item
Report will report the status of recent and
upcoming tasks, highlighting those leading up to
deliverables and critical tasks, report progress on
program issues, identify unresolved issues, and
maintain a list of requests requiring change orders
or amendments. The weekly report will be provided
to the Department by no later than noon one day
prior to the weekly program management
conference calls and will reflect the previous weeks
work. ITN Ref 7.7.6
(1) For each invoice period, provide one
management report from each month since the
previous invoice period. (Quarterly invoices to occur
in February, May, August, and November)
Periodic: Annual Program Review

Review will include a record of the contractors
approval of the annual program review for the
services and deliverables in response to
previous administration year.
contractual/ITN requirements. This review must
identify major program accomplishments and
successes, as well as areas of the program that
require changes and improvement in the upcoming
year. The review is intended to be a history of
program events and to describe recommended
improvements or changes for subsequent years. The
annual review will be based on information
gathered by the contractors program and project
management staff throughout the year. The review
will address all aspects of program service and
products including management, technology,
interpretive products, communications,
development, test production, test administration,
materials shipping/pickup, scoring, and reporting
processes. It will also follow the annual
administrative cycle, beginning with fall and ending
with the following summer. ITN Reference 7.7.6
Due Date(s)
Deliverable Price
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
November 2015
November 2016
November 2017

A
1
Program
Management
1390
1391
1392
1393
1394
Reports: Collect Accounting Reports
A summary of the Quarterly Accounting Reports,
(1) For each invoice period, provide the quarterly
including expenditures by program task and level of accounting reports. (Quarterly invoices to occur in
effort as well as expenditures and balances of
February, May, August, and November)
Reimbursable Funding Categories identified in
Section 7.7.9, including balances resulting from
executed change orders and contract amendments.
ITN Ref 7.7.6
Periodic: Final Delivery of Materials
The contractor agrees to deliver to the Department, (1) Provide evidence of Department acceptance and
upon request, all materials and products in all forms approval (email is acceptable) of the final program
that are developed for and used in conjunction with report and acceptance of all materials.
this program, including test books, answer
documents (paper and PDF), final electronic source
files of interpretive products, software source code,
and ancillary materials, PDFs, and electronic files
within 30 days following acceptance by the
Department of the final report for the program. ITN
Ref. 7.7.10
Due Date(s)
Deliverable Price
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
1395
Program Management
1396
[TBD - either November 2017, November 2018,

November 2019, or November 2020, depending on
whether the base is extended into one, two, or three
one-year renewals]
Contract #14-652
Attachment G
American Institutes for Research
Addendums: 1, 2 and 3

BUREAU OF CONTRACTS, GRANTS AND PROCUREMENT
MANAGEMENT SERVICES
Addendum # 1
REPLY NO.: ITN 2014-47
OCTOBER 31, 2013
This addendum is issued to provide applicable changes to the above referenced ITN.
Section 7.8 DELIVERABLES 8 x 14 document hereby replaces the current 7.8 DELIVERABLES
8 x 11 document due to formatting issues. To obtain a copy of 7.8 DELIVERABLES in Microsoft
Excel format, send an email request to the Purchasing Analyst as instructed in Section 4.3.
Potential proposers have 72 hours from posting of addenda to protest the requirements of
each addendum. Failure to file a protest within the time prescribed in Section 120.57(3), or
failure to post the bond or other security required by law within the time allowed for filing a
bond shall constitute a waiver of proceedings under Chapter 120, Florida Statutes.
YOUR REPLY WILL NOT BE COMPLETE WITHOUT THIS DOCUMENT SIGNED AND INCLUDED!
Vendor Name written:
Authorized Signature:
Mailing Address:
City, State & Zip Code:
Telephone:
ITN # 2014-47 Addendum #1
Facsimile:
E-Mail Address:
2014-47 ITN for Florida Standards Assessments DELIVERABLES-DRAFT (V2)

A
1
3
4
5
6
7
8
9
10
11
12
13
Test Development
Document
Test Development

Specifications
Test Development

7.3.10
via SFTP.

7.5.4.5

2017
SFTP.
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Due Date(s)
June 1, 2014
June 1, 2015
June 1, 2016
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5
14
15
16
17
18
19
20
DRAFT
1 of 72

A
1
21
22
23
24
25
Test Development
7.5.4.5
SFTP.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Test Development

7.5.4.5

2016
SFTP.
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
Test Development
7.5.4.5

2016
SFTP.
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
26
27
28
29
30
31
32
33
34
35
36
37
38
DRAFT
2 of 72

A
1
39
40
41
42
43
Test Development
7.5.4.5
SFTP.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Test Development

Specifications

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
7.3.10
44
45
46
47
48
49
50
51 Test Development
52
53
54
55
56
DRAFT
3 of 72
June 1, 2014
June 1, 2015
June 1, 2016

A
1
D
Test Development
Due Date(s)
Test Development

each scoring guide.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Test Development

Assessment Reports
Reference 7.3.7.5

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
November 1, 2015
57
58
59
60
61
62
63
64
65
66
67
68
69
DRAFT
4 of 72

A
1
Test Development
70
71 Test Development
72
73
74
75
76 Test Development
77
78
79
80
DRAFT
B
E
Due Date(s)
June 1, 2015

management report
Reference 7.3.4.2

February 1, 2017
management report
Review

5 of 72

management report
May 1, 2015
May 1, 2016

A
1
Test
Development
81
82
83
84
85
86
87 Test Development
88
89
90
91 Test Development
92
93
Due Date(s)
July 15, 2015
July 15, 2016
management report

Reference 7.3.0

management report.
December 1, 2015
December 1, 2016


management report.
December 1, 2015
December 1, 2016
94
DRAFT
6 of 72

A
1
Test
Development
95
96
97
98
99
100
102
103
104
105
106
108
109
110
111
112
Test Administration
113
114
115
116




the report.
Reference 7.4.2
117
DRAFT
7 of 72
previous invoice.

E
Due Date(s)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)

A
1
Test Administration
B
the report.
ITN Reference 7.4.2
Due Date(s)
2016

the report.
a report.
ITN Reference 7.4.2

2015
the report.
ITN Reference 7.4.2

2015
118
119
120
121
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
122
Test Administration
123
124
125
126
127
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
128
Test Administration
129
130
131
132

133
DRAFT
8 of 72
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)

A
1
Test Administration
134
135
136
137
the report.
ITN Reference 7.4.2
Due Date(s)
2015
the report.
ITN Reference 7.4.2

2015
the report.
ITN Reference 7.4.2

2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
138
Test Administration
139
140
141
142
143
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
144
Test Administration
145
146
147
148
149

150
DRAFT
9 of 72
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)

A
1
Test
Administration
151
152
153
154
155
156
158
159
160
161
162
B
the Specifications.
the Specifications.
the Specifications.
164
165
166
167
168
169
DRAFT
E
Due Date(s)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
10 of 72

A
1
Test
Administration
170
171
172
173
174
175
177
178
179
180
181
B
administration.

administration.

administration.
the Specifications.
the Specifications.
the Specifications.
183
184
185
186
187
188
DRAFT
E
Due Date(s)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2015.
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
11 of 72

A
1
Test
Administration
189
190
191
192
193
194
195
the Specifications.


districts.
Test Administration

memos.


districts.
Test Administration

memos.


districts.
202
203
204
205
206
207
208
DRAFT
administration.

memos, etc.
197
198
199
200
201
209
210
211
212
213
214
215
12 of 72
E
Due Date(s)
2015.
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)

A
1
Test Administration

and accommodations.

Test Administration

and accommodations.

Test Administration


224
225
226
227
228
229
230
231
232
233
234
DRAFT
memos.
216
217
218
219
220
221
222
235
236
237
238
239
240
241
13 of 72
districts.
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)

A
1
accommodations.
Test Administration

accommodations.

Test Administration

accommodations.

Test Administration

accommodations.

248
249
250
251
252
253
254
255
256
257
258
259
260
DRAFT
Test Administration
242
243
244
245
246
247
261
262
263
264
265
266
267
14 of 72
E
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)

A
1
and 7.4.7.4
districts.
Test Administration


and 7.4.7.4

districts.
Test Administration


and 7.4.7.4

districts.
Test Administration
and rosters.
and 7.4.7.4

districts.
274
275
276
277
278
279
280
281
282
283
284
285
286
DRAFT
Test Administration
268
269
270
271
272
273
287
288
289
290
291
292
15 of 72
E
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)

A
1
Test Administration
labels and rosters.
and 7.4.7.4
districts.
Test Administration

labels and rosters.

and 7.4.7.4

districts.
Test Administration

labels and rosters.

and 7.4.7.4

districts.
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
314
315
316
317
7.4.8
318
DRAFT
16 of 72

E
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
November 2015
November 2016

A
1
Test
Administration
319
320
321
322
323
324
326
327
328
329
330
331
333
334
335
336
337
339
340
341
342
E
Due Date(s)
7.4.8


7.4.8


assessment.

7.4.8

November 2015
November 2016

assessment.

7.4.8

February 2016
February 2017
343
DRAFT
17 of 72
May 2016
May 2017
May 2015
May 2016
May 2017

A
1
Test
Administration
344
345
346
347
348
349
E
Due Date(s)
assessment.
7.4.8

assessment.

7.4.8

357
359
360
361
362



November 2015
November 2016
363
365
366
367
368

May 2016
May 2017
350
352
353
354
355
356
369
DRAFT
18 of 72
August 2015
August 2016
August 2017
August 2015
August 2016
August 2017

A
1
Test
Administration
370
371
372
373
374
375
E
Due Date(s)
administration.
376
378
379
380
381



November 2015
November 2016
382
384
385
386
387



February 2016
February 2017



388
390
391
392
393
394
395
DRAFT
19 of 72
May 2015
May 2016
May 2017
May 2015
May 2016
May 2017

A
1
Test
Administration
396
397
398
399
400
401
402
404
405
406
407
408
Due Date(s)




August 2015
August 2016
August 2017
November 2014
November 2015
November 2016
409
Test Administration
410
DRAFT
20 of 72

training.
September 2014

A
1
Test
Administration
411
412
413
414
415
417
418
419
420
421
423
424
425
426
7.4.9.8
administration.
Due Date(s)
August 2015
August 2016
November 2016
November 2017
previous year.
November 2015
November 2016
November 2017
427
Test Administration

comment forms.

Test Administration

comment forms.

428
429
430
431
432
433
434
435
436
437
438
439
DRAFT
21 of 72

negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

A
1
Test Administration

comment forms.

Test Administration

comment forms.

Test Administration

comment forms.

447
448
449
450
451
452
453
454
455
456
457
458
DRAFT
Test Administration
440
441
442
443
444
445
446
459
460
461
462
463
464
465
22 of 72
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

A
1
Test Administration
comment forms.
Test Administration

Trial Test

7.4.8 and 7.4.9.5

Test Administration


Reference 7.3.11.2

Test Administration


Reference 7.3.11.2

466
467
468
469
470
471
472
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
August 1, 2014
473
474
475
476
477
478
479
480
481
482
483
484
485
DRAFT
23 of 72

negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

A
1
Reference 7.3.11.2
Test Administration

versions)

Reference 7.3.11.2

Test Administration
versions)
Reference 7.3.11.2

Test Administration
versions)
Reference 7.3.11.2

493
494
495
496
497
498
499
500
501
502
503
504
DRAFT
Test Administration
486
487
488
489
490
491
492
505
506
507
508
509
510
511
24 of 72
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

A
1
Test Administration
Reference 7.3.11.2
versions)
Test Administration

Test Administration
Reference 7.4.1.
Test Administration
testing
Reference 7.4.1.
512
513
514
515
516
517
518
519
520
521
522
523

Reference 7.4.1.
524
525
526
527
528
529
530
531
532
533
534
535
536
537
DRAFT
25 of 72
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)

negotiations.)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

A
1
Reference 7.4.1.
Test Administration


Reference 7.4.1.
Test Administration


Reference 7.4.1.
Test Administration
Reference 7.4.1.
544
545
546
547
548
549
550
551
552
553
554
555
556
DRAFT
Test Administration
538
539
540
541
542
543
557
558
559
560
561
562
563
26 of 72
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

A
1
Scoring
and Reporting
564
565
566
B
567
568
569
571
572
573
574
576
577
578
579
581
582
583
584
585
587
588
589
590
DRAFT
ITN Reference 7.5.1
management system.
E
Due Date(s)
April 2014
April 2015
April 2016

management system.



management system.

January 2016
January 2017



management system.

January 2015
January 2016
January 2017

review.

27 of 72

management system.
September 2015
September 2016

September 2015
September 2016

A
1
Scoring
and Reporting
591
592
593
594
595
597
598
599
600
601
603
604
605
606
607
609
610
611
review.
management system.
Due Date(s)
October 2015
October 2016

review.


management system.

January 2015
January 2016
January 2017



management system.

in .txt format


Protocol (SFTP).
612
DRAFT
28 of 72

June 2015
June 2016
June 2017
September 2015
September 2016

A
1
Scoring
and Reporting
613
614
615
616
617
619
620
621
622
623
625
626
627
file in .txt format
Protocol (SFTP).
Due Date(s)
January 2016
January 2017


Protocol (SFTP).
January 2015
January 2016
January 2017
format

Protocol (SFTP).
September 2015
September 2016
628
DRAFT
29 of 72

A
1
Scoring
and Reporting
629
630
631
632
633
635
636
637
638
639
641
642
648
649
650
DRAFT
.txt format
Protocol (SFTP).
Due Date(s)
October 2015
October 2016

.txt format


Protocol (SFTP).
January 2015
January 2016
January 2017

.txt format


Protocol (SFTP).
June 2015
June 2016
June 2017
643
644
645
647

7.5.4.4
30 of 72

October 2015
October 2016
May 2016

A
1
652
653
654
655
657
658
659
660
661
663
664
665
666
668
669
670
671
673
674
675
676
677
679
680
681
682
683
DRAFT
7.5.4.4
transcription logs

7.5.4.4
7.5.4.4
7.5.4.4
7.5.4.4

7.5.4.4
31 of 72
E
Due Date(s)
May 2017
May 2015
May 2016
May 2017
October 2015
October 2016
December 2015
December 2016
May 2015
May 2016
May 2017
July 2015
July 2016
July 2017

A
1
Scoring
and Reporting
684
685
686
687
7.5.6.2, and 7.5.6.4
Due Date(s)
October 2015
October 2016
688
690
691
692


7.5.6.2, and 7.5.6.4

March 2016
March 2017
693
DRAFT
32 of 72

A
1
Scoring
and Reporting
694
695
696
697
698
data files
7.5.6.2, and 7.5.6.4
Due Date(s)
March 2015
March 2016
March 2017
699
701
702
703


7.5.6.2, and 7.5.6.4

October 2015
October 2016
704
DRAFT
33 of 72

A
1
Scoring
and Reporting
705
706
707
708
7.5.6.2, and 7.5.6.4
Due Date(s)
December 2015
December 2016
709
711
712
713
714


7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017
715
DRAFT
34 of 72

A
1
Scoring
and Reporting
716
717
718
7.5.6.2, and 7.5.6.4
719
720
721
723
724
725
726
728
729
730
731
733
734
735
736
737
DRAFT
the data files.
score data files


the data files.

data files

the data files.
E
Due Date(s)
July 2015
July 2016
July 2017

October 2015
October 2016
April 2016
April 2017
April 2015
April 2016
April 2017
35 of 72

A
1
Scoring
and Reporting
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
DRAFT
B
data files
the data files.
data files
the data files.
data files
the data files.

score data files

the data files.


Reference 7.5.5

E
Due Date(s)
October 2015
October 2016
December 2015
December 2016
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017
October 2015
October 2016

36 of 72

A
1
Scoring
and Reporting
765
766
767
768
769
771
772
773
774
775
777
778
779
780
782
783
784
Reference 7.5.5
Due Date(s)
April 2016
April 2017

DSR data files.

Reference 7.5.5

April 2015
April 2016
April 2017

Reference 7.5.5


Reference 7.5.5


and DSR data files.

785
DRAFT
37 of 72

October 2015
October 2016
December 2015
December 2016

A
1
Scoring
and Reporting
786
787
788
789
B
790
791
793
794
795
796
797
799
800
801
802
804
805
806



Reference 7.5.5

Reference 7.5.5





807
DRAFT
38 of 72
E
Due Date(s)
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017
October 2015
October 2016
April 2016
April 2017

A
1
Scoring
and Reporting
808
809
810
811
812
813
815
816
817
818
820
821
822
DAR data files.

and DAR data files.


October 2015
October 2016



December 2015
December 2016
823
DRAFT
39 of 72
Due Date(s)
April 2015
April 2016
April 2017


A
1
Scoring
and Reporting
824
825
826
827
B
828
829
831
832

833
834
835
837
838
839
840
DRAFT

data files.



7.5.5.2 and 7.5.6.6
November 2016
40 of 72
Due Date(s)
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017

A
1
Scoring
and Reporting
841
842
843
data files.
7.5.5.2 and 7.5.6.6

7.5.5.2 and 7.5.6.6
844
845
847
848
849
850
851
853
854
855

7.5.5.2 and 7.5.6.6
856
DRAFT
41 of 72
E
Due Date(s)
June 2016
June 2017
June 2015
June 2016
June 2017

November 2016

A
1
Scoring
and Reporting
857
858
859
B
data files.
860
861
863
864
865

data files.
866
867
869
870
871
872

data files.
7.5.5.2 and 7.5.6.6

7.5.5.2 and 7.5.6.6

7.5.5.2 and 7.5.6.6
873
DRAFT
42 of 72
E
Due Date(s)
February 2015
February 2016
June 2015
June 2016
June 2017
October 2015
October 2016
October 2017

A
1
Scoring
and Reporting
874
875
876
877
878
880
881
882
883
885
886
887
888
889
891
892
B
and 7.5.6.7
Due Date(s)
December 2015
December 2016
and 7.5.6.7
October 2016
October 2017

assessments.
and 7.5.6.7
October 2015
October 2016
October 2017

and 7.5.6.7
December 2015
December 2016
893
DRAFT
43 of 72

A
1
Scoring
and Reporting
895
896
897
898
899
901
902
903
904
905
907
908
909
910
911
913
914
915
916
917
DRAFT
and 7.5.6.7
Due Date(s)
March 2015
March 2016

and 7.5.6.7
October 2015
October 2016
October 2017

and 7.5.6.7
October 2015
October 2016
October 2017


44 of 72

September 2015
September 2016
September 2017

A
1
918
919
920
921
922
923
924
925
926
927
B
reporting
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2017
reporting
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

initial reporting


928
929
930
931
932
933
934
935
936
937
938
DRAFT
45 of 72

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

A
1



950
951
952
953
954
955
960
DRAFT
944
945
946
947
948
949
958
959
939
940
941
942
943
956
957
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
jump drives.
2015
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
districts.
46 of 72

A
1
961
962
963
964
965
966
967
968


969
970
jump drives.
districts.
Due Date(s)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)

jump drives.
districts.

2015
2016
2017
(Renewal 1 - 2018)

jump drives.
districts.

2015
2016
(Renewal 1 - 2017)
971
972
973
974
975

initial reporting

976
DRAFT
47 of 72
(Renewal 3 - 2020)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

A
1
977
978
979
initial reporting
980
981
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

initial reporting


jump drives.
districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

initial reporting


jump drives.
districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
987
992
993
DRAFT
jump drives.
districts.
982
983
984
985
986
988
989
990
991
48 of 72
(Renewal 3 - 2020)

A
1
Reference 7.5.5
reports.
districts.
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)


Reference 7.5.5

reports.
districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)


Reference 7.5.5

reports.
districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
DRAFT
49 of 72
(Renewal 3 - 2020)

A
1
reports
Reference 7.5.5
reports.
districts.
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

reports

Reference 7.5.5

reports.
districts.

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

reports

Reference 7.5.5

reports.
districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
DRAFT
50 of 72

A
1
1026
1027
1028
reports
Reference 7.5.5
reports.
districts.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)


districts.

2015
2016
(Renewal 1 - 2017)

districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
1029
1030
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1031
1032
1033
1034
1035

reporting
(Renewal 2 - 2018)
(Renewal 3 - 2019)
1036
1037
1038
1039

reporting

1040
1041
DRAFT
51 of 72
(Renewal 3 - 2020)

A
1
1042
1043
1044
B
late reporting
1045
1046
1047




1048
1049
1050
1051
districts.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)

districts.

2015
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
(Renewal 1 - 2017)
(Renewal 3 - 2019)
districts.
1052
1053
1054
1055
1056
1057
DRAFT
52 of 72
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

A
1
1058
1059
1060
1061
B
1062
districts.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1063
1064
1065
1066
1067
1068
1069
1071
1072
1073
1074
1076
1077
1078



districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)



September 2015
September 2016



March 2016
March 2017
1079
DRAFT
53 of 72

A
1
Scoring
and Reporting
1080
1081
1082
1083
1084
Due Date(s)
March 2015
March 2016
March 2017



September 2015
September 2016



November 2015
November 2016
1095
1097
1098
1099
1100



March 2015
March 2016
March 2017
1101
1103
1104
1105
1106
1107



June 2015
June 2016
June 2017
1085
1087
1088
1089
1090
1092
1093
1094
DRAFT
54 of 72

A
1
Performance Tasks
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

Performance Tasks

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
DRAFT
55 of 72

A
1
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

Tasks

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
DRAFT
56 of 72

A
1
1130
1131
1132
1133
Performance Tasks
Due Date(s)
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

Performance Tasks

1134
1135
1136
1137
1138
1139
1140
DRAFT
57 of 72
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

A
1
1141
1142
1143
1144
1145
1146
1148
1149
1150
1151
1153
1154
1155
1156
1158
1159
1160
1161
1162
DRAFT
Performance Tasks
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.
58 of 72
June 2015
June 2016
January 2016
January 2017
January 2015
January 2016
January 2017

A
1
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
DRAFT
all assessments
Reference 7.5.5
the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

ELA/Mathematics
Reference 7.5.4.2

ELA/Mathematics
Reference 7.5.4.2
59 of 72
E
Due Date(s)
June 2015
June 2016
October 2015
October 2016
January 2015
January 2016
January 2017
April 2015
April 2016
April 2017
December 2015
December 2016
October 2016
October 2017

A
1
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
DRAFT
and ELA/Mathematics
Reference 7.5.4.2

Reference 7.5.4.2
Reference 7.5.4.2

ELA/Mathematics
Reference 7.5.4.2

Retake
Reference 7.5.4.2
E
Due Date(s)
October 2015
October 2016
October 2017
December 2015
December 2016
March 2016
March 2017
October 2015
October 2016
October 2017
October 2015
October 2016
October 2017
November 2015
November 2016

60 of 72

A
1
Scoring
and Reporting
1228
1229
1230
1231
1232
1234
1235
1236
1237
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
DRAFT
Retake

rescoring.

Reference 7.5.6.5

management system.
log.

manual rescoring.

Reference 7.5.6.5

management system.
log.


Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.
61 of 72
E
Due Date(s)
May 2016
May 2017

October 2015
October 2016
April 2016
April 2017
April 2015
April 2016
April 2017
October 2015
October 2016

A
1
Scoring
and Reporting
1254
1255
1256
1257
1258
1260
1261
1262
1263
1264
1266
1267
1268
rescoring.
Reference 7.5.6.5
management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.



districts.
1269
1270
1271
1272
1273
1274
1275
1276
DRAFT
62 of 72
E
Due Date(s)
December 2015
December 2016
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017
July 2015
July 2016
July 2017

A
1
Scaling,
Equating, Scoring,
1277
1278
Studies
1279
Administrations

deliverable.
July 2014
July 2015

July 2014
July 2015
1280
1281
1282
1284 Studies
1285
1286
1287
1288
Studies
1290
1291
1292
1293
1294
Studies
1296
1297
1298
1299
1300
DRAFT

Plans

Administrations

63 of 72

E
Due Date(s)
May 2014
May 2015
May 2016
July 2016
July 2016
August 2014
August 2015
August 2016

A
1
Scaling,
Equating, Scoring,
1301
and
Special
Psychometric
1302
Studies
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313

Studies

1314
Studies
1315
1316
1317
Specifications
Due Date(s)
December 2014
December 2015
December 2016

May 2015
May 2016
May 2017
2016
1318
DRAFT
64 of 72

posted to SFTP.
via SFTP.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

A
1
Scaling,
Equating, Scoring,
1319
and
Special
Psychometric
1320
1321
1322
1323
E
Due Date(s)
2016
2017


posted to SFTP.
December 2015
December 2016
December 2017


Reference 7.6.3.3

posted to SFTP.
November 2014

SFTP.

Technical Reports
1328
1329

Studies
D
via SFTP.
Studies
1330
C
Studies
1324
1327
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

1332
1333
Studies
1334
DRAFT
65 of 72
October 2014

A
1
Scaling,
Equating, Scoring,
1335

Studies

Setting Work Plan

Studies

Studies

Technical Report

November 2015
SFTP.

Studies
Due Date(s)
May 2015
1336
7.6.3.4
1337
1338
1339
DRAFT

7.6.3.4.
Department.
66 of 72

A
1
Scaling,
Equating, Scoring,
1340
Studies
1341
Studies
renewal)
SFTP.
Due Date(s)
November 2014

Report


August 2015




Report

Reference 7.6.3.6
1343
1344

Studies
1345
Studies
1347
DRAFT
67 of 72

A
1
1348 Studies
1349
Studies
across devices.
input types.
concordance.
characteristics.

with TAC Committee


1351
DRAFT
68 of 72
E
Due Date(s)

A
1
Studies

Studies
1355
Studies
November 2015
SFTP.


7.6.3.6.2
August 2016

Assessments


SFTP.
1357
DRAFT
1352
1353
69 of 72
November 2015
Due Date(s)

A
1
Studies
7.6.3.6.3

7.7.2

report

Meetings


previous invoice.

ITN.

Due Date(s)
April 15, 2014

April 15, 2015
April 15, 2016
1377
DRAFT
Assessments
1358
1359
1361
1362
1363
1364
1365
1367
1368
1369
1370
1371
1373
1374
1375
1376
70 of 72
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

A
1
Program
Management
1378
1379
1380
1381
1382
1383
1385
1386
1387
1388
Action Item Report
work. ITN Ref 7.7.6

1389
DRAFT
71 of 72
E
Due Date(s)
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
November 2015
November 2016
November 2017

A
1
Program
Management
1390
1391
1392
1393
1394
ITN Ref 7.7.6
Ref. 7.7.10
E
Due Date(s)
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
1395
Program Management
1396
DRAFT
72 of 72

one-year renewals]

MANAGEMENT SERVICES
Addendum # 2
November 12, 2013
This addendum is issued to provide applicable changes to the above referenced ITN. Changes made to the ITN are coded as:
words stricken are deletions; words underlined are additions.
1)
Section 3.4 CRITICAL EVENT DATES is hereby revised to read:
3.4

These are tentative dates for critical events, and are subject to change.
October 25, 2013 ITN Released
November 8, 2013 November 14, 2013 @ 2:00 PM, ET - Deadline for Technical Questions
November 15, 2013 November 18, 2013 - Department Response to Technical Questions received by deadline
December 12, 2013 December 16, 2013 @ 2:00 PM, ET - Technical & Price Replies are due
December 12, 2013 December 16, 2013 @ 2:30 PM - Technical Replies will be opened in Room 332
Turlington Building (Step 1)
January 10, 2014 - The Department will evaluate the Technical Replies in Room 1706 Turlington Building @ 9:00
AM (Step 2)
Price Replies will be evaluated by the Bureau of Contracts, Grants and Procurement Management Services staff during
the evaluation stage.
January 28, 2014 - The Negotiation Committee will start negotiations on or about (Step 4)
February 13, 2014 BAFOs due on or about
February 25, 2014 ITA Meeting on or about
The Intent to Award will be posted for 72 hours upon completion of the negotiation meeting and a final decision by the
Department.
2)
Section 4.3 REPLY QUESTIONS & ANSWERS, third paragraph is hereby revised to read:
WRITTEN QUESTIONS should be submitted to:
Bureau of Contracts, Grants and Procurement Management Services, Fran Shewan, 325 West Gaines Street, 332
Turlington Building, Tallahassee, Florida 32399-0400, fshewan@fldoe.org fran.shewan@fldoe.org (email
preferred), or via Fax Number: (850) 245-0719. See SECTION 3.4 Critical Event Dates for question due dates.
Potential proposers have 72 hours from posting of addenda to protest the requirements of each addendum. Failure to
file a protest within the time prescribed in Section 120.57(3), or failure to post the bond or other security required by
law within the time allowed for filing a bond shall constitute a waiver of proceedings under Chapter 120, Florida
Statutes.
Mailing Address:
Telephone:
Facsimile:
E-Mail Address:

MANAGEMENT SERVICES
Addendum # 3
NOVEMBER 18, 2013
This addendum is issued to provide applicable changes to the above referenced ITN and to provide the Answers to
Questions submitted timely by vendors during the Questions and Answer period. Changes made to the ITN are coded as:
words stricken are deletions; words underlined are additions.
1. Section 5.19 Liquidated Damages, a. Tasks/Activities and Interim Deliverables is revised to read:
a. Tasks/Activities and Interim Deliverables: For each default on performance and completion of Work
Tasks/Activities and Interim Deliverables (see SECTION 7.8), it is agreed that for each and every day work remains
uncompleted beyond the time set for its completion the Contractor(s) shall be liable to the Department in a
specified amount of three five thousand dollars ($5,000.00) per business day as liquidated damages for such delay,
and not as a penalty, from the date of notice by the Department until the default has been corrected.
2. Section 7.3.2.1, page 30, first paragraph is removed and replaced with the following:
Any tests offered on the computer must also be prepared for use in a paper-based format (Braille, large print,
regular print, one-item-per-page) as noted in these tables. See Appendix A. In this instance, the paper-based
format will only be offered by special request for up to five percent (5%) of the students and can be used at the
Departments discretion.
3. Clarification: Section 7.4.9.3 System Performance Requirements
The reference to "simultaneous online users" refers to the number of users who are logged in to the system.
4. Section 7.6.3.4 Implement Achievement Level Standard Setting Meetings, third paragraph, fourth sentence
is revised to read:
The Department desires to present proposed cut scores to the State Board of Education in January November
following each meeting; therefore the final report of activities of each meeting should be available for the
Department no later than November September following each meeting.
5. Table 7.7.7 has been revised. Line number 2 Item/Passage Community Sensitivity Base Contract dates and
the Renewal month has changed. See new revised Table 7.7.7.
ITN 2014-47 Addendum #3
Table 7.7.7.1 provides information about the number, expected length, approximate number of participants,
and estimated cost of such meetings.
Revised Table 7.7.7 Meetings of Educators and Other Citizens
Meeting
Area
Item/ Passage
Bias
7.3.4.1
Oct 2015,
2016, 2017
Oct 2018,
2019, 2020
Item/ Passage
Community
Sensitivity
English
Language
Arts/ Literacy
(ELA/L),
Mathematics
ELA/L,
Mathematics
Approx #
of
attendees/
ind mtg
30
7.3.4.1
Oct May
2018, 2019,
2020
30
Jacksonville
or Central
Florida
Item Content
Review
ELA/L,
Mathematics
Test Items
7.3.4.2
1/ subject &
grade
Oct Apr
2015,
2016,; May
2017
Oct 2015,
2016, 2017
Oct 2018,
2019, 2020
Jacksonville
or Central
Florida
Passage
Content Review
ELA/L
7.3.4.2
Aug 2015,
2016, 2017
Aug 2018,
2019, 2020
15
Data Review
Mathematics
7.3.4.3
May or Jun
2018, 2019
Standard
Setting
ELA/L and
Mathematics
7.6.3.4
1/ subject
May or Jun
2015, 2016,
2017
Aug 2015
Jacksonville
or Central
Florida
Tallahassee
N/A
300
Central FL
Standards
Reactor
ELA/L and
Mathematics
7.6.3.4
Aug 2015
N/A
30
Tallahassee
Content
Advisory
ELA/L,
Mathematics
7.3.4.4
TBD
TBD
25/subject/
meeting
Tallahassee,
Jacksonville,
or Central FL
Technical
Advisory
Committee
Technical
7.6.0
3 per
subject/ first
year; 2 per
subject after
first year
2
Spring and
Nov 2015,
2016, 2017
Spring and
Nov 2018,
2019
20
10
Field Test
Rangefinding
ELA/L,
Mathematics
7.5.4.5
2/ subject
May and
Sep 2015,
2016, May
2017
Sep 2017;
May and
Sep 2018,
2019, May
2020
11
Operational
Rangefinder
Selection
ELA/L,
Mathematics
7.5.4.5
1/ subject
Sep 2015,
2016
Sep 2017,
2018, 2020
10/subject
per grade
(3
grades/cour
ses per
subject)
8
Tallahassee,
Jacksonville,
Central FL,
or site of
FERA
Tallahassee
ITN
Ref.
Mtgs/ Year
Base
Contract
Renewal
Days
/ Mtg
Likely Site
Jacksonville
or Central
Florida
Tallahassee
6. Appendix K is hereby added to indicate the budget for reimbursables that will be added during the
final contract negotiations.
7. Section 7.7.9 Reimbursables has been removed and replaced to read:
7.7.9
Reimbursable Funding Categories
Six funding categories are designated for specified program functions and may be used only for those functions.
Use of these funds requires authorization by the Department contract or program manager or program area
leads. Authorization may be provided by specific email or other written communication, or by Department
approval of a specific plan for a meeting(s) or activities requiring use of these funds. Amounts are designated
for administration tasks. Respondents should not include these amounts in their cost replies.
Monies not expended from these funds will be deducted from final contract payment at the conclusion of the
contract during contract resolution. The reimbursable fund balances will be reported by the contractor quarterly
with each invoice submission in the Accounting Report. The amounts specified in this section may be increased
or decreased by an appropriately negotiated amendment at any time during the lifetime of the contract. The
Accounting Report will indicate expenditures from, and the balances remaining, in these funds. The final contract
invoice will be accompanied by the final balance sheet for these funds. The amounts allocated for these funds
should not be identified separately in the Respondents cost reply.
7.7.9.1
Consultant Services
The contractor will provide to the Department technical advice and consultant service, as required,
from outside the contractors staff. The contractor will recommend and the Department will select the
consultant(s) to be used, who may or may not have been recommended by the contractor. A contract
between the consultant and the contractor will be required and will include the amount allocated for
honorarium, travel, lodging, and per diem as authorized by the Department. Reimbursement for travel
will be at rates authorized by Section 112.061, F. S., for travel and per diem. The Respondent should
be aware that include in the reply the amount of $84,375.00 as a fixed amount for the base contract
period and $28,125.00 as a fixed amount for each year of the optional renewal period will be available
for reimbursable expenses. These expenses, however, should not be included in the Respondents
price reply.
7.7.9.2
Contingency Services
It is possible that unforeseen circumstances will make it either necessary or desirable to perform
tasks not specifically anticipated at this time, for example, data analyses beyond those described
herein, unexpected data processing, acquisition of additional materials, or programming for test
functionality not yet envisioned. These circumstances cannot be defined in advance, and the
Department must be able to request such extra services in a timely fashion so as not to impede
implementation of the assessments. In the reply, the The Respondent should be aware that include
$84,375.00 as a fixed amount for the full term of the base contract period and will include $28,125.00
as a fixed amount for each year of the optional renewal period will be available for reimbursable
expenses. These expenses, however, should not be included in the Respondents price reply.
7.7.9.3
Authors Alterations and Design Control

The Department reserves the right to make changes in materials at any stage of the program prior to
final production or delivery. Any changes to the Department approved, production-ready product
(e.g., digital proofs or final computer-based test files) that are required due to contractor error will be
paid for by the contractor. Otherwise, changes made to the Department approved, production-ready
product will be paid for as authors alterations. If any alterations are necessary, the Department will
identify in writing to the contractor the alterations to be made. In the reply, the The Respondent
should be aware that include $84,375.00 as a fixed amount for the full term of the base contract
period and will include $28,125.00 as a fixed amount for each year of the optional renewal period will
be available for reimbursable expenses. These expenses, however, should not be included in the
Respondents price reply.
7.7.9.4
Provision for Meetings with Florida Educators

The Respondent will include a fixed amount to cover the costs of travel and per diem for Florida
educators, Florida citizens, and others participating in the meetings identified in Section 7.7.7. The
contractor will be responsible for the travel reimbursement for participants at rates authorized by
Section 112.061, F. S., for travel and per diem. In the reply, the The Respondent should be aware that
include $5,491,496.00 as a fixed amount for these meetings with Florida educators during the base
contract period and $1,830,498.00 for each year of the optional renewal period will be available for
reimbursable expenses. These expenses, however, should not be included in the Respondents price
reply.
7.7.9.5
Provision for Meetings with Department Staff

A fixed amount will be included in the reply to cover the costs of travel and per diem for the
Department of Education staff or others participating in meetings with the contractor on behalf of the
Department. Reimbursement for staff will be at rates authorized by Section 112.061, F.S. These
funds will not be used to fund travel and per diem of the contractors staff. These funds will be
expended only with the approval of the Department. In the reply, the The Respondent should be
aware that include $438,750.00 as a fixed amount for ten (10) staff members to attend as many as
fifteen (15) meetings per year requiring four (4) travel days each during the contract during the base
contract period and $146,250.00 for each year of the optional renewal period will be available for
reimbursable expenses. These expenses, however, should not be included in the Respondents price
reply.
7.7.9.6
Contractor Staff Positions

To improve work flow, work quality, and communications, the contractor will fund salary and benefits
for five (5) professional positions to work under the direction of the Bureau of K-12 Assessment within
the Florida Department of Education. These positions will be three editors and two Systems Project
Consultants. The Department will work closely with the contractor to interview and select individuals
who are acceptable to both parties. The contractor will employ its standard recruiting, screening, and
hiring procedures to secure individuals to fill these positions. The employees will be full-time
employees of the contractor. The employees will receive benefits typically provided to the
contractors employee classification system. The employees will be paid at a rate comparable to
similarly skilled and experienced Department employees using the contractors employee
classification system.
The basic duties of the Computer-Based Test Editor (1) and the Materials Editors (2) include: (a) write
and edit materials for content, such as items, tests, reports, training materials, brochures, scripts, or
on-screen presentations; (b) develop, analyze, and select appropriate information and present it in a
form and at a level suitable for the intended audience; (c) review written products and materials for
conformity with standards of objectivity, style, and presentation; and, (d) perform a variety of editorial
duties related to editorial production. Candidates must possess one (1) year of specialized
experience directly related to these duties. Candidates may substitute two (2) full academic years of
related graduate level education in lieu of the work experience requirement.
The basic duties of the Systems Project Consultants (2) include: (a) participating in weekly meetings
and communicating issues between the FDOE and the contractors program managers, (b) making
suggestions for improvement to the contractors procedures to aid FDOE staff in data verification
activities, (c) assisting FDOE staff debugging SAS computer programs and analyzing data files, (d)
traveling, at the contractors expense, with FDOE staff to check scanning at the contractors site, (e)
assisting in the verification of reports of results using SAS computer programs, (f) assisting in the
verification of posting of files and reports on the portal, and (g) communicating issues with District
Coordinators of Assessment related to scoring and reporting activities and issues. Candidates must
have a thorough understanding of large scale standardized testing programs and data verification
techniques. The candidate must also have experience managing data activities such as those
required for this project, verifying data files, and using SAS, Excel, and other Microsoft Office
products.
The Respondent should be aware that include in the response $1,800,000.00 as a total amount during
the base contract period and $600,000.00 for each year of the optional renewal period will be available
for reimbursable expenses. These expenses, however, should not be included in the Respondents
price reply.
8. Attachment 1a Deliverables Pricing Sheet Instructions and Attachment 1b Deliverables Pricing Sheet
are hereby added.
To obtain a copy of Attachment 1b Deliverables Pricing Sheet in Microsoft Excel format, send an email
request to: Fran Shewan, 325 West Gaines Street, 332 Turlington Building, Tallahassee, Florida 323990400, fran.shewan@fldoe.org (email preferred), or via Fax Number: (850) 245-0719.
Questions and Answers
Please be advised all questions are keyed as submitted.
ITN 2014-47 Questions and Answers

Question #
Vendor Name
Data
Recognition
Corporation
Vendor
Question #
8
8
Vendor Question/Comments
Answer
Can the Department provide more information

on the requirements for performance tasks?
Are examples of Performance Tasks available
to review?
The Department has no strict expectations nor requirements regarding

performance tasks beyond what is described in Sections 7.3.0 and 7.3.1. The
overall assessment should meet the key criteria outlined in Sections 3.1 and
7.2.0.
Are there any requirements to release

operational items or operationallyrepresentative items during the contract (once
or an annual basis)?
There is an expectation to release operational or operationally-representative

items for certain interpretive products (see 7.3.7.5) and for the individual student
reports for grades/subjects containing performance tasks (see 7.5.5). These
releases may be limited to one item per grade and subject.
Is there a bank of items that need to be

reviewed and realigned to the Florida
Standards? If so, approximately how many
items would need to be reviewed for possible
use?
At this time, there are no plans to review and realign items in Florida's Item Bank
to the Florida Standards.
It is understood that an English-to-Heritage

No.
Language Translation Dictionary is requested
for online assessments. Is there an expectation
to translate any of the paper/pencil deliveries of
the forms to Spanish and Haitian Creole?
How has district route code information been Districts submit route codes electronically via an Excel spreadsheet.
previously provided to contractors? Is this
information provided by districts in electronic or
paper format?
Please define the types of materials that would
be considered "miscellaneous materials." Are
these additional or supplemental testing
materials? Can the Department provide an
estimate of the number of such shipments and
the quantities of districts involved?
Miscellaneous materials in this case would apply to non-scheduled materials

deliveries. For instance, additional orders require supplemental shipments, as
needed, and it is difficult to anticipate how many orders will be placed for each
administration. In addition, the department may request samples of materials or
used secure materials that are not included in scheduled reviews or shipments.
Please clarify the difference between the

Item/Passage Bias Meeting and the
Item/Passage Community Sensitivity
Meeting? Will a meeting be conducted with
Florida educators and the latter conducted with
other citizens? Are both review committees
reviewing potential concerns for bias and
sensitivity issues?
The Community Sensitivity Committee is composed of Florida citizens and

reviews all passages, prompts, and items for issues of potential concern to the
members of the community at large. Reviewers are asked to consider whether the
subject matter and language of each passage, prompt, and item is acceptable to
Florida students, their parents, and other members of Florida communities. The
Bias Review Committee is composed of Florida educators representative of
Florida's demographic make-up. Reviewers look for gender, racial/ethnic,
linguistic, religious, geographic, and socioeconomic bias in passages, prompts,
and items. Comments made by each of these committees are collected, reviewed,
and acted upon by Content Item Review Committees. Item Review Committees
are composed of Florida educators who are content experts.
Please confirm that Attachment 5 is a form that

is used in management of the awarded contract
and does not need to be submitted with the
proposal.
The Respondent may submit Attachment 5 with their reply; however, it is a

requirement for the Contractor to submit this form with each invoice submitted for
payment, whether or not funds have been spent with a small, minority-, women-,
and service-disabled veteran business enterprise subcontractor for the period
covered by the invoice.
1 of 8
CTB/ McGraw
Hill
It is unlikely that the number of copies

requested (20 with original and copies), for
both the technical proposal and the price
proposal could be packaged in one box each
weighing less in 25 pounds. Are multiple,
clearly labeled, boxes acceptable?
How should bidders reply to requirement 7.7.9? The price reply should not include Reimbursable Funding Categories (7.7.9). See
If bidders include requirement 7.7.9 and our
Addendum 3 and Appendix K.
response in the technical proposal, will
proposals be deemed non responsive?
10
11
12
13
14
15
Does Florida have items that are aligned

Common Core that are available in for 3 8
English 9, 10, 11 and the End of Course
Mathematics?
No.
For grades 3 8, ELA and Math, how many

operational forms are required?
The Department will consider any number of operational forms, however, currently
for Grades 3-8 in ELA/L and Mathematics, the Department uses one unique
operational form for each grade and subject spiraled in such a way as to yield up
to forty field test and anchor forms. Only one of these forms is used for all
accommodated forms, however.
In order to meet the two week turnaround for

EOC reporting, it appears that pre-equating will
be required. Thus, a field test will need to be
conducted to establish reliable item
parameters. The EOC items should be field
tested on FL students who recently completed
the associated courses to obtain valid and
reliable item parameters. To have the
assessments pre-equated for a spring 2015
administration, these items will have to be field
tested either in spring 2014 (preferred) or fall
2014. Will students be enrolled in courses in
spring or fall 2014 that are similar enough to
those expected to be offered in spring 2015
that we can expect fall or spring 2014 field test
parameters to be validly applied to the
operational assessments? If not, will DOE
consider post-equating and possibly extending
the time between assessment and reporting?
As indicated in the scope of work, equivalency of the multiple forms of each EOC
assessment administered in spring is verified through concurrent calibration, plus
post-equating via common items between the item bank and the new test forms.
Thus, EOC assessments administered in the spring must be post-equated, and
the selected contractor must come up with a plan that will allow implementation of
these psychometric operations in a short timeframe.
Yes.
Please confirm that the cost of this independent This is to confirm that the Department's independent contractor in 7.5.6 is handled
contractor should NOT be included in this bid. under a separate procurement and is not the responsibility of the selected
contractor.
In Section 7.5.6.5, you indicate that the
For the tests required for graduation, the contractor must identify students in each
contractor must identify student in each grade grade who failed by 0.1 or fewer points in standard deviation units for Manual
who failed by 5 or fewer Scale Score points or Rescoring.
15 Developmental Scale Score points for
Manual Rescoring. Can you clarify the
difference in the criteria 5 or fewer Scale Score
points and 15 Developmental Scale Score
points?
2 of 8
16
17
18
19
20
Typically, legal defensibility is established by The external technical advisory committee does not meet the intent of this
documenting the following of industry standard requirement. A third-party consultant, who has expertise in both measurement and
procedures, including the presentation of the law, must review the linking procedure.
linking method to an appropriately credentialed
technical advisory committee for review, and
approval (sometimes with modifications based
on TAC recommendations). Please confirm that
an external technical advisory committee
review meets the intent of this requirement.
It seems that the RFP indicates standard

setting will occur in educator meetings summer
2015 with proposed cut scores presented to the
State Board of Education in January 2016. Is it
correct to assume that the first years score
reports will not include performance level
information?
As a correction, the Department desires to present proposed cut scores to the

State Board of Education in November following the standard-setting meetings;
therefore the final report of activities of each meeting should be available for the
Department no later than September following each meeting. See Addendum 3. It
is correct to assume that 2014-15 score reports will not include performance level
information. Scores may be reported in thirds and/or on some type of scale to be
determined after contract execution. Proposals may address these or other
possible approaches.
10
What percentage of students will be Online

during each Transition to CBT
grade/administration?
Each administration that a grade/subject test is scheduled to be computer-based,

the expectation is that all participating students are testing online with the
exception of a small number of students who require paper-based
accommodations (e.g., regular print, large print, braille, one-item-per-page).
11
By full-time do you mean these Scoring

Directors are full-time time employees of the
vendor, not full-time to the FSA program?
Scoring directors are full-time employees of the contractor and, during operational
handscoring, should be 100% dedicated to the FSA program.
12
Can the Scoring Director for 3-8 Math also

A single Mathematics scoring director is sufficient.
serve as the Scoring Director for EOC
Geometry and Algebra I & II, or do you require
one for each section?
13
If Bidders are only responding to one or more See #10.

but not all four components (A-D) what should
be the amount of reimbursable funds they
should include in the Price Reply? It seems the
prices provided in the ITN are to cover all four
components.
22
14
Please confirm that in the Price Reply line/row The Respondent will submit a separate price for each component (A-D) they
E is simply the total of the prices in rows A-D. choose to bid on and additional pricing (E) showing combined cost with the
Or, if vendors are submitting bids for the entire anticipated cost savings if the Respondent is awarded all components.
scope (A-D), do they simply complete line E?
23
15
Should bidders add rows to the Price Reply

document to provide for the Required
Reimbursable Funds?
21
24
16
See #10 and #22.
Please confirm that in the Price Reply line/row See #22.

E is simply the total of the prices in rows A-D.
Or, if vendors are submitting bids for the entire
scope (A-D), do they simply complete line E?
3 of 8
25
26
27
28
29
30
31
17
Pearson
What dates should bidders use for the first

year of the contract? Will the first contract
year be he award date through June 30,
2014? Or, would it cover more than 12 months
and run from the award date through June 30,
2015? Or, would no contract be in place prior
to June 30, 2014?
The contract dates will be negotiated and determined prior to contract execution.
The Department anticipates contract execution will be prior to June 2014. The
original term of the contract will commence upon signing from both parties and be
effective for a period of thirty-six (36) months. The contract will overlap the State
of Floridas fiscal year which comprises July 1 through June 30.
The ITN states that a performance bond in the Section 5.15 PERFORMANCE BOND supercedes the Standard Contract Term
amount of 10% of the total annual award is
and Condition regarding bond amount.
required. On page 153, Attachment 7, Contract
Standard Terms and Conditions, Section XVII,
it states that a performance bond in the total
amount of the contract is required. Please
confirm that Section 5.15 governs this ITN and
that a performance bond in an amount equal to
10% of the total contract value will be required.
Section 7.3.4.3 makes no mention of a specific

subject matter and yet it is cross-referenced in
the Meetings table in Section 7.7.7 to a
meeting that mentions only Mathematics and
only has nine participants over two days.
Please clarify the subjects and grades/courses
exams for which the data will be reviewed in
the Data Review meeting and whether the total
number of participants (9) is accurate.
The Data Review Committees meet to adjudicate Mathematics fill-in response

items (such as inserting a numeric answer) to ensure items behave as expected,
include all correct and appropriate answers, and are fair for all students. Nine
participants over a two-day period should be sufficient.
For costing purposes, does the Department

Yes.
expect the base bid to include scoring of
performance tasks and constructed responses
in the calibration samples used for equating?
The RFP states, For these papers that are
read a second time, if the two (2) readers
scores do not match, a formal resolution
process will be established in order to obtain
exact or adjacent scores for quality control
purposes only.
Please clarify this requirement does the
Department require a resolution process only
when the two (2) readers scores are nonadjacent?
Non-adjacent scores, discrepant condition codes, or a paper that was scored by

one reader and given a condition code by another reader would require a
resolution process.
As a cost savings measure, may the contractor

propose a lower n-count for handscoring of
field test constructed responses and
performance tasks?
If the contractor provides evidence to the Department indicating the sample drawn
from the respective population is sufficiently large and representative and will
allow valid and reliable parameter estimates, and the contractor receives approval
from the Department, the contractor may propose a lower n-count for handscoring
of field test constructed responses and performance tasks.
If the contractor is applying for all four

The proposal should meet the minimum requirements for staffing as described in
components, is it acceptable to propose a total the ITN. Staffing beyond that is at the respondent's discretion, but sufficient staff
of four full-time project managers?
should be proposed in order to manage the project.
4 of 8
32
33
34
35
36
37
Item/Passage bias and sensitivity reviews are

listed in the table as meetings occurring at
separate times during the year. Would the
department consider holding bias and
sensitivity at the same time?
The ITN has been corrected to reflect that both meetings will be held at the same
time. See Addendum 3.
Does the department have any preference in

how the pricing by deliverable is formatted?
See Addendum 3. Detailed instructions are provided in Attachment 1a.
Could the department provide some

clarification around the evaluation process in
regards to comparing respondents who bid on
different components? For example, if a
respondent wants to bid on all components can
they just provide a price for Component E or do
they also have to provide prices for A-D
separately? Also, if the highest scoring
respondent bid only on Component A, and all
other respondents only bid on Component E,
what would happen?
Scoring will be conducted in accordance with Section 8.1 Criteria for Evaluation.
Each price component will be evaluated and points awarded (A - E respectively).
Respondents submitting pricing on component E only will not be evaluated and
scored against other components (A - D).
So that all bidders may respond appropriately,

will the Florida Department of Education
identify its requirement with regard to unique
sequence numbers on multi-page response
documents (e.g., litho codes)?
The Department requires the use of lithocodes on multi-page response

documents. Each answer document has a unique lithocode. The lithocode is a
machine-readable code and eye-readable number printed on each page in the
document. The front half of the answer document has the lithocode on the even
numbered pages and the back half of the document has it on the odd numbered
pages. If a stack of documents is dropped after the spine has been spliced off, the
lithocode number may be used to restack the document ensuring that each
students pages are placed under the correct cover page.
During the scanning process, the lithocode on the first page is compared to the
other lithocodes on the pages of that students document. If any lithocode differs
from the lithocode on the first page, or is missing, the value is flagged for the
editor to review and correct.
A. Please explain how a not for profit entity can

be a respondent to this if they subcontract with
a University College of Education to develop
the test(s), and also subcontract with a data
infrastructure vendor such as IBM for data
security & data transfer functions?
All replies should be from independent contractors and not individuals or entities
that would anticipate being a subcontractor of an independent contractor. Not for
profit entities may submit responses that include other entities as subcontractors
by clearly identifying who the subcontractors are and what work they will be
responsible for. The successful respondent is ultimately responsible for all work
on the contract.
10
Osceola County
School Board
B. Are the tests developed under BID ITN 2014- The tests developed are for 2014-15 and beyond, for up to six total years.
47 to be used for the current 2013-2014 school
year, or for the 2014-2015 school years going
forward for the next five years?
5 of 8
38
39
40
41
42
43
American
Institutes for
Research
a. Q: How will the Department evaluate bids

for individual Components versus bids for all
Components? b. Q: Is the evaluation of any
Component linked to any other Component?
For example, is the evaluation of a bid for
Component A completely separate from the
evaluation of other Components? c. Q: If the
evaluation of any Component is linked to any
other Component, how will the evaluation
points be assigned?
See #22.
What is covered by the perpetual licenses

While proposals including perpetual licenses will be considered, the terms of
covering periods after the term of the contract? those licenses will be governed by the license itself as detailed by the provider.
Test items?
The terms will be subject to negotiations.
Due to efficiencies, we will be able to offer
lower prices if awarded multiple components.
How should Vendors present prices in
Attachment 1 (and by deliverable in 7.8) for
these efficiencies if awarded multiple
Components?
The current pricing structure is designed to reflect a discount when all

components are bid together (see Component E pricing). If discounts are
available for multiples other than a bid on all components, those contingencies
may be explored during negotiations and in Best and Final Offers.
a. Q: Are reports 2 or 4 pages? Or perhaps 2

pages per subject? b. Q: Does the Department
require full-color printing, or is black ink on preprinted color forms acceptable?
While the number of pages may vary from our current model, the student report
should include the same information currently provided in our reports. The
Department will accept responses that include black ink on pre-printed color
forms, and the pre-printed color forms may include up to 8 colors plus black as
indicated in the ITN.
Section 3.0 indicates the initial term (Contracts See #25.

Years 1, 2, and 3) shall only be 36 months.
Given the award is anticipated February 2014
and the Original Contract Term includes the
2016-17 administrations, this would be longer
than 36 months. In completing the schedule
and Pricing forms, shall contractors use the
following periods for the Original Contract
Term? a. Year 1 February 2014 Award
June 30, 2015 b. Year 2 July 1, 2015 - June
30, 2016 c. Year 3 July 1, 2016 June 30,
2017
This section is very clear with regard to notices

of breach, termination, and suspension.
However, vendors may interpret this sections
reporting requirement for complaints,
investigations, warning letters, or disciplinary
actions differently. a. Q: For these items,
would the Department accept a common
definition of instances that required a Vendor
to pay penalties, fines, or liquidated damages?
b. Q: Are Vendors required to report the
amount of any penalties, fines, and liquidated
damages? c. Q: Where in the Technical Reply
should the Vendor include the required
documentation for Poor Performance?
6 of 8
a. Respondent should not limit reporting as suggested here but include any
information that falls within the parameters of the listed actions or similar actions.
b. Yes, as this information would be considered relevant. c. The Department
recommends this information be included in the Qualifications and Experience
section of the Technical Reply.
44
Computerized
Assessments
and Learning
45
46
Strategic
Planning Online
In lieu of a bid covering all elements of the ITN, The Department will only consider replies that include all aspects of components
please indicate whether a bidder may submit a bid. Respondents may, however, include subcontractors for portions of the work.
partial bid to provide Online Management and
Computer-Based Test Administration Systems,
in the form of a perpetual and non-exclusive
Open license for the use, reproduction and
modification of the solution for use in Florida, to
support all four assessment components (as
summarized in Section 7.1.0), including the
requirements of Sections 7.4.8 and 7.4.9, and
the full integration of the assessment content to
support the requirements of Section 7.4.5,
without bidding to create the underlying
assessment content, if the bidder agrees to
work in conjunction with the selected
assessment content provider.
If the answer to Question 1 above is

NA.
affirmative, how should cost be provided in the
Price Reply (Attachment 1) for the Online
Management and Computer-Based Test
Administration Systems, including initial
onboarding and services to operate and
maintain the systems for the duration of the
contract?
How do I go about bidding for this job?
Detailed instructions are included in the ITN available through the Florida Vendor
Bid System.
http://www.myflorida.com/apps/vbs/vbs_www.ad.view_ad?advertisement_key_nu
m=109751
Can you provide more detail about what your

department is looking for?
No.
How many users will be using this software?
The System Performance Requirements, including the anticipated number of

simultaneous users, are provided in 7.4.9.3 of the ITN.
47
48
49
What softwares are you using today to

facilitate this process?
The current summative assessment contractor uses TestNav 6.9 to implement

computer-based testing.
Appendix J, Page 130, Section J4 Will

Department approval be required for the
handscoring targeted calibration sets?
Yes.
Appendix J, Page 142, Section J9 Can the

State provide established reader Qualification
thresholds?
Section 7.5.4.5.5 describes the development of handscoring specifications prior to

the beginning of handscoring activities. The Department will conduct discussions
with the successful bidder regarding reader qualification metrics, and these
metrics will be included in the handscoring specifications.
Appendix J, Page 143, Section J9 RFP, Page

66, Section 7.5.4.5.3 Handscoring Can the
State provide desired reader performance
thresholds for IRR, validity items, and
calibration sets?
Section 7.5.4.5.5 describes the development of handscoring specifications prior to

the beginning of handscoring activities. The Department will conduct discussions
with the successful bidder regarding expected reader performance metrics (IRR,
Validity, etc.), and these metrics will be included in the handscoring specifications.
Appendix B, Page 103, Critical and Extremely

Critical Activities Can you please confirm that
EOC administrations will not require any
handscoring activities?
All grade/course assessments with constructed response items will require

handscoring activities.
50
51
52
53
Measurement
Incorporated
7 of 8
54
Rally! Education
55
American
Institutes for
Research
Am I reading correctly that the only document

due at this time, prior to Dec 12, is the one
page form in section 1 of the BID NUMBER:
ITN 2014-47?
The Invitation to Negotiate Registration, Intent to Submit Form located on page 5

of the ITN may be submitted anytime. The Form PUR 1001 referenced in Section
1 is an instructional document and need not be returned with the Respondent's
Reply.
1. Is it possible for districts to update the FL

DOE data system at any point in time?
This question is not related to ITN 2014-47.
2. Is it possible for the state to extract either all This question is not related to ITN 2014-47.
of the data in the system or data updates to the
system nightly?
56
57
1. What is the date the rostering system will be This question is not related to ITN 2014-47.
online?
58
2. Will it be possible for districts to update the This question is not related to ITN 2014-47.
FL DOE rostering system at any point in time?
59
3. Will it be possible for the state to extract

either all of the data in the rostering system or
data updates to the rostering system nightly?
60
1. Is FCAT explorer a state-owned and

operated system, or do you expect similar
capabilities from the assessment contractor?
2. Will the new contractor be required to

provide any content for FCAT Explorer? If yes,
what content will the new contractor be
required to provide?
61
62
63
ACT
1. Would Florida please share any results,

summaries, or reports available regarding
Florida collected with or reported from the
PARCC/Smarter Balanced Technology
Readiness Tool (www.techreadiness.net)?
Florida did not participate in the Readiness Tool activities.
1. With regard to Section 5.19 Liquidated

The correct amount is $5,000. See Addendum 3.
Damages, item a. there is a discrepancy
regarding the statement, Contractor(s) shall
be liable to the Department in a specified
amount of three thousand dollars ($5,000) per
business day as liquidated damages. Please
clarify if the amount is $3,000 or $5,000?
8 of 8
Potential proposers have 72 hours from posting of addenda to protest the requirements of each addendum. Failure
to file a protest within the time prescribed in Section 120.57(3), or failure to post the bond or other security required
by law within the time allowed for filing a bond shall constitute a waiver of proceedings under Chapter 120, Florida
Statutes.
Mailing Address:
Telephone:
Facsimile:
E-Mail Address:
ITN 2014-47 Appendix K: Florida Standards Assessments Reimbursables Budget

CONTRACT NUMBER:
ITN Reference
Project Budget Cost Categories
Year 1
$ Project
Budget
Year 2
$ Project
Budget
Year 3
$ Project
Budget
Years 1-3
$ Project
Total
Renewal
Year 1
$ Project
Budget
Renewal
Year 2
$ Project
Budget
Renewal
Year 3
$ Project
Budget
Renewal
Years
$ Project
Total
Section 7.7.9.1 Consultant Services
$28,125.00
$28,125.00
$28,125.00
$84,375.00
$28,125.00
$28,125.00
$28,125.00
$84,375.00
Section 7.7.9.2 Contingency Services
$28,125.00
$28,125.00
$28,125.00
$84,375.00
$28,125.00
$28,125.00
$28,125.00
$84,375.00
$28,125.00
$28,125.00
$28,125.00
$84,375.00
$28,125.00
$28,125.00
$28,125.00
$84,375.00
$1,830,498.67
$1,830,498.67
$1,830,498.67
$5,491,496.00
$1,830,498.00
$1,830,498.00
$1,830,498.00
$5,491,494.00
Section 7.7.9.5 Provision for Meetings with Department Staff
$146,250.00
$146,250.00
$146,250.00
$438,750.00
$146,250.00
$146,250.00
$146,250.00
$438,750.00
Section 7.7.9.6 Contractor Staff Positions
$600,000.00
$600,000.00
$600,000.00
$1,800,000.00
$600,000.00
$600,000.00
$600,000.00
$1,800,000.00
$2,661,123.67
$2,661,123.67
$2,661,123.67
$7,983,371.00
$2,661,123.00
$2,661,123.00
$2,661,123.00
$7,983,369.00
Section 7.7.9.3 Author's Alterations/Design Control

Section 7.7.9.4 Provision for Meetings with Florida Educators
Original Term and Renewal Total: $15,966,740.00
ITN 2014-47
Page 1 of 1
APPENDIX K
Attachment 1a
ITN 2014-47
DELIVERABLES PRICING SHEET INSTRUCTIONS
The Respondent should complete the Deliverables Pricing Sheet (column F of Attachment 1b) for each component they are
responding to and submit all pricing sheets as part of the Price Reply. See 3.1 and Attachment 1 for the components,
including that Component E includes Components A, B, C, and D and may show anticipated cost savings if the Respondent
is awarded all components. The Deliverables Pricing Sheet will be used for negotiation purposes and any resulting contract.
1. In Attachment 1b, do not modify the content in columns A E.
2. Complete the Specify Component field in the heading of the Deliverables Pricing Sheet. Remember that this
should be completed for each assessment component included in the reply.
3. Complete column F for each applicable deliverable. Column B indicates whether the deliverable is Periodic (e.g.,
Program Management Reports, Conduct Onsite Test Construction) or Administration-Specific (e.g., Fall EOC,
Spring ELA/L and Mathematics).
a. Prices for Periodic deliverables should be provided for all components, except where the content of the
component is not applicable (e.g., delivering Writing Image CDs would not be applicable for Component B:
Mathematics in Grades 3-8).
b. Prices for Administration-Specific deliverables should be completed for only the deliverables related to that
component (e.g., Summer EOC deliverables would not be applicable for Component A: ELA/L in Grades
3-8).
4. The total of the Deliverable Prices in column F of the Deliverables Pricing Sheet should match the Grand Total
(Original Term + All Renewals) in Attachment 1 for that component.
ITN 2014-47 Deliverables Pricing Sheet Instructions
Page 1 of 1

A
1
3
4
5
6
7
8
9
10
11
12
13
Test Development
Document
Test Development

Specifications
Test Development

7.3.10
via SFTP.

7.5.4.5

2017
SFTP.
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Due Date(s)
June 1, 2014
June 1, 2015
June 1, 2016
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5
14
15
16
17
18
19
20
F
Deliverable Price

A
1
21
22
23
24
25
Test Development
7.5.4.5
SFTP.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Test Development

7.5.4.5

2016
SFTP.
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
Test Development
7.5.4.5

2016
SFTP.
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
26
27
28
29
30
31
32
33
34
35
36
37
38
F
Deliverable Price

A
1
39
40
41
42
43
Test Development
7.5.4.5
SFTP.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Test Development

Specifications

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
7.3.10
44
45
46
47
48
49
50
51 Test Development
52
53
54
55
56
June 1, 2014
June 1, 2015
June 1, 2016
F
Deliverable Price

A
1
D
Due Date(s)
Deliverable Price
Test Development
Test Development

each scoring guide.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Test Development

Assessment Reports
Reference 7.3.7.5

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
November 1, 2015
57
58
59
60
61
62
63
64
65
66
67
68
69

A
1
Test Development
70
71 Test Development
72
73
74
75
76 Test Development
77
78
79
80
B
C
D
Due Date(s)
Deliverable Price
June 1, 2015

management report
Reference 7.3.4.2

February 1, 2017
management report
Review

management report
May 1, 2015
May 1, 2016

A
1
Test
Development
81
82
83
84
85
86
87 Test Development
88
89
90
91 Test Development
92
93
94
Due Date(s)
July 15, 2015
July 15, 2016
management report

Reference 7.3.0

management report.
December 1, 2015
December 1, 2016


management report.
December 1, 2015
December 1, 2016
F
Deliverable Price

A
1
Test
Development
95
96
97
98
99
100
102
103
104
105
106
108
109
110
111
112
Test Administration
113
114
115
116
117
previous invoice.




the report.
Reference 7.4.2

E
Due Date(s)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
F
Deliverable Price

A
1
Test Administration
B
the report.
ITN Reference 7.4.2
Due Date(s)
2016
the report.
a report.
ITN Reference 7.4.2

2015
the report.
ITN Reference 7.4.2

2015
118
119
120
121
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
122
Test Administration
123
124
125
126
127
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
128
Test Administration
129
130
131
132
133

2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
F
Deliverable Price

A
1
Test Administration
134
135
136
137
the report.
ITN Reference 7.4.2
Due Date(s)
2015
the report.
ITN Reference 7.4.2

2015
the report.
ITN Reference 7.4.2

2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
138
Test Administration
139
140
141
142
143
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
144
Test Administration
145
146
147
148
149
150

2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
F
Deliverable Price

A
1
Test
Administration
151
152
153
154
155
156
158
159
160
161
162
164
165
166
167
168
169
B
the Specifications.
the Specifications.
the Specifications.
E
Due Date(s)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
F
Deliverable Price

A
1
Test
Administration
170
171
172
173
174
175
177
178
179
180
181
183
184
185
186
187
188
B
administration.

administration.

administration.
the Specifications.
the Specifications.
the Specifications.
E
Due Date(s)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2015.
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
F
Deliverable Price

A
1
Test
Administration
189
190
191
192
193
194
195
B
administration.
the Specifications.

memos, etc.


districts.
Test Administration

memos.


districts.
Test Administration

memos.


districts.
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
E
Due Date(s)
2015.
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
F
Deliverable Price

A
1
Test Administration
memos.

and accommodations.

Test Administration

and accommodations.

Test Administration


216
217
218
219
220
221
222
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
districts.
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
F
Deliverable Price

A
1
Test Administration
accommodations.
Test Administration

accommodations.

Test Administration

accommodations.

Test Administration

accommodations.

242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
E
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
F
Deliverable Price

A
1
Test Administration
and 7.4.7.4
districts.
Test Administration


and 7.4.7.4

districts.
Test Administration


and 7.4.7.4

districts.
Test Administration
and rosters.
and 7.4.7.4

districts.
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
E
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
F
Deliverable Price

A
1
Test Administration
labels and rosters.
and 7.4.7.4
districts.
Test Administration

labels and rosters.

and 7.4.7.4

districts.
Test Administration

labels and rosters.

and 7.4.7.4

districts.
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
314
315
316
317
318
7.4.8

E
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
November 2015
November 2016
F
Deliverable Price

A
1
Test
Administration
319
320
321
322
323
324
326
327
328
329
330
331
333
334
335
336
337
339
340
341
342
343
Due Date(s)
Deliverable Price
7.4.8


7.4.8


assessment.

7.4.8

November 2015
November 2016

assessment.

7.4.8

February 2016
February 2017
May 2016
May 2017
May 2015
May 2016
May 2017

A
1
Test
Administration
344
345
346
347
348
349
Due Date(s)
Deliverable Price
assessment.
7.4.8

assessment.

7.4.8

357
359
360
361
362



November 2015
November 2016
363
365
366
367
368

May 2016
May 2017
350
352
353
354
355
356
369
August 2015
August 2016
August 2017
August 2015
August 2016
August 2017

A
1
Test
Administration
370
371
372
373
374
375
Due Date(s)
Deliverable Price
administration.
376
378
379
380
381



November 2015
November 2016
382
384
385
386
387



February 2016
February 2017



388
390
391
392
393
394
395
May 2015
May 2016
May 2017
May 2015
May 2016
May 2017

A
1
Test
Administration
396
397
398
399
400
401
402
404
405
406
407
408
Due Date(s)
Deliverable Price




August 2015
August 2016
August 2017
November 2014
November 2015
November 2016
409
Test Administration
410

training.
September 2014

A
1
Test
Administration
411
412
413
414
415
417
418
419
420
421
423
424
425
426
7.4.9.8
administration.
Due Date(s)
August 2015
August 2016
November 2016
November 2017
previous year.
November 2015
November 2016
November 2017
427
Test Administration

comment forms.

Test Administration

comment forms.

428
429
430
431
432
433
434
435
436
437
438
439

negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Test Administration
Test Administration

comment forms.

Test Administration

comment forms.

Test Administration

comment forms.

440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Test Administration
comment forms.
Test Administration

Trial Test

7.4.8 and 7.4.9.5

Test Administration


Reference 7.3.11.2

Test Administration


Reference 7.3.11.2

466
467
468
469
470
471
472
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
August 1, 2014
473
474
475
476
477
478
479
480
481
482
483
484
485

negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Test Administration
Reference 7.3.11.2
Test Administration

versions)

Reference 7.3.11.2

Test Administration
versions)
Reference 7.3.11.2

Test Administration
versions)
Reference 7.3.11.2

486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Test Administration
versions)
Reference 7.3.11.2
Test Administration

Test Administration
Reference 7.4.1.
Test Administration
testing
Reference 7.4.1.
512
513
514
515
516
517
518
519
520
521
522
523

Reference 7.4.1.
524
525
526
527
528
529
530
531
532
533
534
535
536
537
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)

negotiations.)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Test Administration
Reference 7.4.1.
Test Administration


Reference 7.4.1.
Test Administration


Reference 7.4.1.
Test Administration
Reference 7.4.1.
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
D
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Scoring
and Reporting
564
565
566
B
567
568
569
571
572
573
574
576
577
578
579
581
582
583
584
585
587
588
589
590
C
ITN Reference 7.5.1
D
management system.
E
Due Date(s)
April 2014
April 2015
April 2016

management system.



management system.

January 2016
January 2017



management system.

January 2015
January 2016
January 2017

review.


management system.
September 2015
September 2016

September 2015
September 2016
F
Deliverable Price

A
1
Scoring
and Reporting
591
592
593
594
595
597
598
599
600
601
603
604
605
606
607
609
610
611
612
review.
management system.
Due Date(s)
October 2015
October 2016

review.


management system.

January 2015
January 2016
January 2017



management system.

in .txt format


Protocol (SFTP).

June 2015
June 2016
June 2017
September 2015
September 2016
F
Deliverable Price

A
1
Scoring
and Reporting
613
614
615
616
617
619
620
621
622
623
625
626
627
628
file in .txt format
Protocol (SFTP).
Due Date(s)
January 2016
January 2017


Protocol (SFTP).
January 2015
January 2016
January 2017
format

Protocol (SFTP).
September 2015
September 2016
F
Deliverable Price

A
1
Scoring
and Reporting
629
630
631
632
633
635
636
637
638
639
641
642
648
649
650
.txt format
Protocol (SFTP).
Due Date(s)
October 2015
October 2016

.txt format


Protocol (SFTP).
January 2015
January 2016
January 2017

.txt format


Protocol (SFTP).
June 2015
June 2016
June 2017
643
644
645
647

7.5.4.4

October 2015
October 2016
May 2016
F
Deliverable Price

A
1
652
653
654
655
657
658
659
660
661
663
664
665
666
668
669
670
671
673
674
675
676
677
679
680
681
682
683
7.5.4.4
transcription logs

7.5.4.4
7.5.4.4
7.5.4.4
7.5.4.4

7.5.4.4
E
Due Date(s)
May 2017
May 2015
May 2016
May 2017
October 2015
October 2016
December 2015
December 2016
May 2015
May 2016
May 2017
July 2015
July 2016
July 2017
F
Deliverable Price

A
1
Scoring
and Reporting
684
685
686
687
7.5.6.2, and 7.5.6.4
Due Date(s)
October 2015
October 2016
688
690
691
692


7.5.6.2, and 7.5.6.4

March 2016
March 2017
693
F
Deliverable Price

A
1
Scoring
and Reporting
694
695
696
697
698
data files
7.5.6.2, and 7.5.6.4
Due Date(s)
March 2015
March 2016
March 2017
699
701
702
703


7.5.6.2, and 7.5.6.4

October 2015
October 2016
704
F
Deliverable Price

A
1
Scoring
and Reporting
705
706
707
708
7.5.6.2, and 7.5.6.4
Due Date(s)
December 2015
December 2016
709
711
712
713
714


7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017
715
F
Deliverable Price

A
1
Scoring
and Reporting
716
717
718
7.5.6.2, and 7.5.6.4
719
720
721
723
724
725
726
728
729
730
731
733
734
735
736
737
score data files
the data files.


the data files.

data files

the data files.
E
Due Date(s)
July 2015
July 2016
July 2017

October 2015
October 2016
April 2016
April 2017
April 2015
April 2016
April 2017
F
Deliverable Price

A
1
Scoring
and Reporting
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
B
data files
the data files.
data files
the data files.
data files
the data files.

score data files

the data files.


Reference 7.5.5
762
763
764

E
Due Date(s)
October 2015
October 2016
December 2015
December 2016
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017
October 2015
October 2016

F
Deliverable Price

A
1
Scoring
and Reporting
765
766
767
768
769
771
772
773
774
775
777
778
779
780
782
783
784
785
Reference 7.5.5
Due Date(s)
April 2016
April 2017

DSR data files.

Reference 7.5.5

April 2015
April 2016
April 2017

Reference 7.5.5


Reference 7.5.5


and DSR data files.


October 2015
October 2016
December 2015
December 2016
F
Deliverable Price

A
1
Scoring
and Reporting
786
787
788
789
B
790
791
793
794
795
796
797
799
800
801
802
804
805
806
807



Reference 7.5.5

Reference 7.5.5





E
Due Date(s)
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017
October 2015
October 2016
April 2016
April 2017
F
Deliverable Price

A
1
Scoring
and Reporting
808
809
810
811
812
813
815
816
817
818
820
821
822
823
DAR data files.
Due Date(s)
April 2015
April 2016
April 2017

and DAR data files.


October 2015
October 2016



December 2015
December 2016


F
Deliverable Price

A
1
Scoring
and Reporting
824
825
826
827
B
828
829
831
832

833
834
835
837
838
839
840

data files.
Due Date(s)
April 2015
April 2016
April 2017



7.5.5.2 and 7.5.6.6
November 2016
July 2015
July 2016
July 2017
F
Deliverable Price

A
1
Scoring
and Reporting
841
842
843
data files.
7.5.5.2 and 7.5.6.6

7.5.5.2 and 7.5.6.6
844
845
847
848
849
850
851
853
854
855
856

7.5.5.2 and 7.5.6.6
E
Due Date(s)
June 2016
June 2017
June 2015
June 2016
June 2017

November 2016
F
Deliverable Price

A
1
Scoring
and Reporting
857
858
859
B
data files.
860
861
863
864
865

data files.
866
867
869
870
871
872
873

data files.
7.5.5.2 and 7.5.6.6

7.5.5.2 and 7.5.6.6

7.5.5.2 and 7.5.6.6
E
Due Date(s)
February 2015
February 2016
June 2015
June 2016
June 2017
October 2015
October 2016
October 2017
F
Deliverable Price

A
1
Scoring
and Reporting
874
875
876
877
878
880
881
882
883
885
886
887
888
889
891
892
893
B
and 7.5.6.7
E
Due Date(s)
December 2015
December 2016
and 7.5.6.7
October 2016
October 2017

assessments.
and 7.5.6.7
October 2015
October 2016
October 2017

and 7.5.6.7
December 2015
December 2016
F
Deliverable Price

A
1
Scoring
and Reporting
895
896
897
898
899
901
902
903
904
905
907
908
909
910
911
913
914
915
916
917
and 7.5.6.7
Due Date(s)
March 2015
March 2016

and 7.5.6.7
October 2015
October 2016
October 2017

and 7.5.6.7
October 2015
October 2016
October 2017



September 2015
September 2016
September 2017
F
Deliverable Price

A
1
918
919
920
921
922
923
924
925
926
927
B
reporting
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2017
reporting
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

initial reporting


928
929
930
931
932
933
934
935
936
937
938

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
F
Deliverable Price

A
1



939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
D
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
jump drives.
2015
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
districts.
F
Deliverable Price

A
1
961
962
963
964
965
966
967
968


969
970
D
jump drives.
districts.
Due Date(s)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)

jump drives.
districts.

2015
2016
2017
(Renewal 1 - 2018)

jump drives.
districts.

2015
2016
(Renewal 1 - 2017)
971
972
973
974
975
976

initial reporting

(Renewal 3 - 2020)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
F
Deliverable Price

A
1
977
978
979
initial reporting
980
981
jump drives.
districts.
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

initial reporting


jump drives.
districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

initial reporting


jump drives.
districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
982
983
984
985
986
987
988
989
990
991
992
993
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Reference 7.5.5
reports.
districts.
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)


Reference 7.5.5

reports.
districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)


Reference 7.5.5

reports.
districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
(Renewal 3 - 2020)
F
Deliverable Price

A
1
reports
Reference 7.5.5
reports.
districts.
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

reports

Reference 7.5.5

reports.
districts.

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

reports

Reference 7.5.5

reports.
districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
F
Deliverable Price

A
1
1026
1027
1028
reports
Reference 7.5.5
reports.
districts.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)


districts.

2015
2016
(Renewal 1 - 2017)

districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
1029
1030
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1031
1032
1033
1034
1035

reporting
(Renewal 2 - 2018)
(Renewal 3 - 2019)
1036
1037
1038
1039
1040
1041

reporting

(Renewal 3 - 2020)
F
Deliverable Price

A
1
1042
1043
1044
B
late reporting
C
1045
1046
1047




1048
1049
1050
1051
districts.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)

districts.

2015
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
(Renewal 1 - 2017)
(Renewal 3 - 2019)
districts.
1052
1053
1054
1055
1056
1057
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
F
Deliverable Price

A
1
1058
1059
1060
1061
B
C
1062
districts.
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1063
1064
1065
1066
1067
1068
1069
1071
1072
1073
1074
1076
1077
1078
1079



districts.

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)



September 2015
September 2016



March 2016
March 2017
F
Deliverable Price

A
1
Scoring
and Reporting
1080
1081
1082
1083
1084
Due Date(s)
March 2015
March 2016
March 2017



September 2015
September 2016



November 2015
November 2016
1095
1097
1098
1099
1100



March 2015
March 2016
March 2017
1101
1103
1104
1105
1106
1107



June 2015
June 2016
June 2017
1085
1087
1088
1089
1090
1092
1093
1094
F
Deliverable Price

A
1
Performance Tasks
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

Performance Tasks

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
F
Deliverable Price

A
1
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

Tasks

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
F
Deliverable Price

A
1
1130
1131
1132
1133
Performance Tasks
Due Date(s)
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

Performance Tasks

1134
1135
1136
1137
1138
1139
1140
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
1141
1142
1143
1144
1145
1146
1148
1149
1150
1151
1153
1154
1155
1156
1158
1159
1160
1161
1162
Performance Tasks
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.
June 2015
June 2016
January 2016
January 2017
January 2015
January 2016
January 2017
F
Deliverable Price

A
1
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
all assessments
Reference 7.5.5
the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

ELA/Mathematics
Reference 7.5.4.2

ELA/Mathematics
Reference 7.5.4.2
E
Due Date(s)
June 2015
June 2016
October 2015
October 2016
January 2015
January 2016
January 2017
April 2015
April 2016
April 2017
December 2015
December 2016
October 2016
October 2017
F
Deliverable Price

A
1
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
and ELA/Mathematics
Reference 7.5.4.2

Reference 7.5.4.2
Reference 7.5.4.2

ELA/Mathematics
Reference 7.5.4.2

Retake
1225
1226
1227
Reference 7.5.4.2
E
Due Date(s)
October 2015
October 2016
October 2017
December 2015
December 2016
March 2016
March 2017
October 2015
October 2016
October 2017
October 2015
October 2016
October 2017
November 2015
November 2016

F
Deliverable Price

A
1
Scoring
and Reporting
1228
1229
1230
1231
1232
1234
1235
1236
1237
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
Retake
Due Date(s)
May 2016
May 2017

rescoring.

Reference 7.5.6.5

management system.
log.

October 2015
October 2016

manual rescoring.

Reference 7.5.6.5

management system.
log.


Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.
April 2016
April 2017
April 2015
April 2016
April 2017
October 2015
October 2016
F
Deliverable Price

A
1
Scoring
and Reporting
1254
1255
1256
1257
1258
1260
1261
1262
1263
1264
1266
1267
1268
rescoring.
Reference 7.5.6.5
management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.



districts.
1269
1270
1271
1272
1273
1274
1275
1276
Due Date(s)
Deliverable Price
December 2015
December 2016
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017
July 2015
July 2016
July 2017

A
1
Scaling,
Equating, Scoring,
1277
1278
Studies
1279
Administrations

deliverable.
July 2014
July 2015

July 2014
July 2015
1280
1281
1282
1284 Studies
1285
1286
1287
1288
Studies
1290
1291
1292
1293
1294
Studies
1296
1297
1298
1299
1300

Plans

Administrations


Due Date(s)
Deliverable Price
May 2014
May 2015
May 2016
July 2016
July 2016
August 2014
August 2015
August 2016

A
1
Scaling,
Equating, Scoring,
1301
and
Special
Psychometric
1302
Studies
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313

Studies

1314
Studies
1315
1316
1317
1318
Specifications
Due Date(s)
December 2014
December 2015
December 2016

May 2015
May 2016
May 2017
2016

posted to SFTP.
via SFTP.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Deliverable Price

A
1
Scaling,
Equating, Scoring,
1319
and
Special
Psychometric
1320
1321
1322
1323
E
Due Date(s)
2016
2017


posted to SFTP.
December 2015
December 2016
December 2017


Reference 7.6.3.3

posted to SFTP.
November 2014

SFTP.

Technical Reports
1328
1329

Studies
D
via SFTP.
Studies
1330
C
Studies
1324
1327
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

1332
1333
Studies
1334
October 2014
F
Deliverable Price

A
1
Scaling,
Equating, Scoring,
1335

Studies

Setting Work Plan

Studies

Studies

Technical Report

November 2015
SFTP.

Studies
May 2015
1336
7.6.3.4
1337
1338
1339

7.6.3.4.
Department.
Due Date(s)
Deliverable Price

A
1
Scaling,
Equating, Scoring,
1340
Studies
1341
Studies
renewal)
SFTP.
Due Date(s)
November 2014

Report


August 2015




Report

Reference 7.6.3.6
1343
1344

Studies
1345
Studies
1347
F
Deliverable Price

A
1
1348 Studies
1349
Studies
1351
across devices.
input types.
concordance.
characteristics.

with TAC Committee


Due Date(s)
Deliverable Price

A
1
Studies

Studies
1355
Studies
1357
November 2015
SFTP.


7.6.3.6.2
August 2016

Assessments


SFTP.
1352
1353
November 2015
Due Date(s)
Deliverable Price

A
1
Studies
1377
Assessments
7.6.3.6.3

7.7.2

report

Meetings


previous invoice.

ITN.

1358
1359
1361
1362
1363
1364
1365
1367
1368
1369
1370
1371
1373
1374
1375
1376
Due Date(s)
Deliverable Price
April 15, 2014

April 15, 2015
April 15, 2016
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

A
1
Program
Management
1378
1379
1380
1381
1382
1383
1385
1386
1387
1388
1389
Action Item Report
work. ITN Ref 7.7.6

Due Date(s)
Deliverable Price
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
November 2015
November 2016
November 2017

A
1
Program
Management
1390
1391
1392
1393
1394
ITN Ref 7.7.6
Ref. 7.7.10
Due Date(s)
Deliverable Price
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
1395
Program Management
1396

one-year renewals]
Table 7.7.7.1 provides information about the number, expected length, approximate number of participants,
and estimated cost of such meetings.
Meeting
Area
Item/ Passage
Bias
7.3.4.1
Oct 2015,
2016, 2017
Oct 2018,
2019, 2020
Item/ Passage
Community
Sensitivity
English
Language
Arts/ Literacy
(ELA/L),
Mathematics
ELA/L,
Mathematics
Approx #
of
attendees/
ind mtg
30
7.3.4.1
Oct May
2018, 2019,
2020
30
Jacksonville
or Central
Florida
Item Content
Review
ELA/L,
Mathematics
Test Items
7.3.4.2
1/ subject &
grade
Oct Apr
2015,
2016,; May
2017
Oct 2015,
2016, 2017
Oct 2018,
2019, 2020
Jacksonville
or Central
Florida
Passage
Content Review
ELA/L
7.3.4.2
Aug 2015,
2016, 2017
Aug 2018,
2019, 2020
15
Data Review
Mathematics
7.3.4.3
May or Jun
2018, 2019
Standard
Setting
ELA/L and
Mathematics
7.6.3.4
1/ subject
May or Jun
2015, 2016,
2017
Aug 2015
Jacksonville
or Central
Florida
Tallahassee
N/A
300
Central FL
Standards
Reactor
ELA/L and
Mathematics
7.6.3.4
Aug 2015
N/A
30
Tallahassee
Content
Advisory
ELA/L,
Mathematics
7.3.4.4
TBD
TBD
25/subject/
meeting
Tallahassee,
Jacksonville,
or Central FL
Technical
Advisory
Committee
Technical
7.6.0
3 per
subject/ first
year; 2 per
subject after
first year
2
Spring and
Nov 2015,
2016, 2017
Spring and
Nov 2018,
2019
20
10
Field Test
Rangefinding
ELA/L,
Mathematics
7.5.4.5
2/ subject
May and
Sep 2015,
2016, May
2017
Sep 2017;
May and
Sep 2018,
2019, May
2020
11
Operational
Rangefinder
Selection
ELA/L,
Mathematics
7.5.4.5
1/ subject
Sep 2015,
2016
Sep 2017,
2018, 2020
10/subject
per grade
(3
grades/cour
ses per
subject)
8
Tallahassee,
Jacksonville,
Central FL,
or site of
FERA
Tallahassee
ITN
Ref.
Mtgs/ Year
Base
Contract
Renewal
Days
/ Mtg
Likely Site
Jacksonville
or Central
Florida
Tallahassee
ITN 2014-47 Appendix K: Florida Standards Assessments Reimbursables Budget

CONTRACT NUMBER:
ITN Reference
Project Budget Cost Categories
Year 1
$ Project
Budget
Year 2
$ Project
Budget
Year 3
$ Project
Budget
Years 1-3
$ Project
Total
Renewal
Year 1
$ Project
Budget
Renewal
Year 2
$ Project
Budget
Renewal
Year 3
$ Project
Budget
Renewal
Years
$ Project
Total
Section 7.7.9.1 Consultant Services
$28,125.00
$28,125.00
$28,125.00
$84,375.00
$28,125.00
$28,125.00
$28,125.00
$84,375.00
Section 7.7.9.2 Contingency Services
$28,125.00
$28,125.00
$28,125.00
$84,375.00
$28,125.00
$28,125.00
$28,125.00
$84,375.00
$28,125.00
$28,125.00
$28,125.00
$84,375.00
$28,125.00
$28,125.00
$28,125.00
$84,375.00
$1,830,498.67
$1,830,498.67
$1,830,498.67
$5,491,496.00
$1,830,498.00
$1,830,498.00
$1,830,498.00
$5,491,494.00
Section 7.7.9.5 Provision for Meetings with Department Staff
$146,250.00
$146,250.00
$146,250.00
$438,750.00
$146,250.00
$146,250.00
$146,250.00
$438,750.00
Section 7.7.9.6 Contractor Staff Positions
$600,000.00
$600,000.00
$600,000.00
$1,800,000.00
$600,000.00
$600,000.00
$600,000.00
$1,800,000.00
$2,661,123.67
$2,661,123.67
$2,661,123.67
$7,983,371.00
$2,661,123.00
$2,661,123.00
$2,661,123.00
$7,983,369.00
Section 7.7.9.3 Author's Alterations/Design Control

Section 7.7.9.4 Provision for Meetings with Florida Educators
Original Term and Renewal Total: $15,966,740.00
ITN 2014-47
Page 1 of 1
APPENDIX K
Contract #14-652, Attachment H

Technical Response
Table of Contents
6.2.1 Preliminary Administrative Review
Signed Addenda
DUNS Number
Disclosure Statement Forms
Reference Forms
Transmittal Letter
Letters of Commitment
6.2.2 Technical Reply

1. Executive Summary
Overview of Solution .......................................................................................................1-1
Who We Are.....................................................................................................................1-3
Project Leadership Team.................................................................................................1-4
2. Qualifications and Experience
Background......................................................................................................................2-1
AIR and DRC, A Proven Team ........................................................................................2-2
Relevant Experience on Similar Projects........................................................................2-4
Ability to Absorb Additional Workload Resulting From this Project ..............................2-8
Financial Capability ........................................................................................................2-9
Detailed Descriptions of Similar Projects to Florida Assessment Standards................2-12
4.9 Poor Performance Notice ........................................................................................2-45
Presented by the American Institutes for Research
Page i
3. Technical Plan
7.1.0 Introduction .........................................................................................................7.1-1
Summary of Proposed Solution....................................................................................7.1-1
Policy Drives FSA Requirements .................................................................................7.1-3
7.1.1 Notice To Respondents.......................................................................................7.1.-9
7.1.2 Project Schedule ...............................................................................................7.1-10
7.2.0 Program Descriptions...........................................................................................7.2-1
7.2.1 Expected Number of Districts, Schools, and Students .......................................7.2-1
7.2.2 Potential Test Administration Dates ...................................................................7.2-1
7.3.0 Test Design and Development...............................................................................7.3-1
7.3.1 Test Content ........................................................................................................7.3-2
ELA/L Test Content .................................................................................................7.3-3
Mathematics Test Content .....................................................................................7.3-19
ELA/Literacy Sample Items ...................................................................................7.3-32
Math Sample Items ................................................................................................7.3-44
7.3.2 Test Configuration.............................................................................................7.3-56
7.3.2.1 Test and Answer Document Design ...........................................................7.3-56
7.3.3 Develop Items....................................................................................................7.3-60
7.3.4 Review Meetings ................................................................................................7.3-72
7.3.4.1 Bias and Sensitivity Committee Reviews ..................................................7.3-73
7.3.4.2 Passage and Item Content Review Committees ........................................7.3-77
7.3.4.3 Data Review Committees...........................................................................7.3-78
7.3.4.4 Content Advisory Committees ..................................................................7.3-79
Page ii
7.3.5 Develop Field-Test Item Sets and Forms ..........................................................7.3-79

Gap Analysis to Determine Needs of the Item Bank .............................................7.3-80
Proposed Number of Field-Test Items and Forms .................................................7.3-80
EFT for Online Forms ............................................................................................7.3-82
EFT for Paper Forms..............................................................................................7.3-83
7.3.6 Develop Stand-Alone ELA/L Field-Test Sets and Forms ...................................7.3-84
7.3.7 Constructing Test Forms...................................................................................7.3-85
7.3.7.1 Test Construction Specifications ...............................................................7.3-86
7.3.7.2 Test Construction System..........................................................................7.3-88
FormBuilder Output Files..................................................................................7.3-89
7.3.7.3 Construct the English Language Arts/Literacy, and Algebra 1,
Algebra 2, and Geometry Retake Forms................................................................7.3-90
7.3.7.4 Prepare Camera-Ready or Screen-Ready Test Forms ...............................7.3-92
7.3.7.5 Interpretive Products .................................................................................7.3-95
Interpretive Guide for Floridas Paper Assessment Reports..............................7.3-95
Proposed Content for the Online Reporting System User Guide......................7.3-96
Production of Interpretive Guide and Online Reporting
System User Guide ............................................................................................7.3-98
Scoring Guides.................................................................................................7.3-101
7.3.8 Production of Printed Test Materials...............................................................7.3-106
7.3.9 Production of Computer-Delivered Products ..................................................7.3-111
7.3.10 Production of Paper-Based and Computer-Based Test, Style Guide,
and Braille Production Procedures ..........................................................................7.3-115
7.3.11 Produce Special Formats and Accommodations for Students
with Disabilities........................................................................................................7.3-116
7.3.11.1 Paper-Based Accommodations ..........................................................7.3-116
7.3.11.2 Computer-Based Accommodations ...................................................7.3-119
Page iii
7.4.0 Test Administration ..............................................................................................7.4-1

7.4.1 Pack and Distribute Materials .............................................................................7.4-1
7.4.2 Missing Materials Report and Inventory ...........................................................7.4-10
7.4.3 Disposition and Storage of Materials.................................................................7.4-10
7.4.4 Retrieval of Answer Sheets and Test Books ......................................................7.4-10
7.4.5 Innovative Product Design ................................................................................7.4-11
7.4.6. Printed ProductsAdministration...................................................................7.4-12
7.4.6.1 Stages of Development of Printed ProductsAdministration...................7.4-12
7.4.7 Test Administration Ancillary Materials ...........................................................7.4-17
7.4.7.1 Training Resources for School and District Technology
Coordinators (CBT) ...............................................................................................7.4-18
7.4.7.2 Instructions and Materials for Braille Tests...............................................7.4-20
7.4.7.3 Student PreID Labels .................................................................................7.4-20
7.4.7.4 Student PreID Rosters ...............................................................................7.4-20
7.4.7.5 Document Count Form (DCF) ...................................................................7.4-21
7.4.7.6 Administration Record/Security Checklist ................................................7.4-21
7.4.7.7 Comment Forms ........................................................................................7.4-22
Gathering Feedback from All User Groups .......................................................7.4-22
Building Fully Customizable and Dynamic Surveys .........................................7.4-22
7.4.7.8 Return Shipping Labels..............................................................................7.4-23
7.4.7.9 Materials Return KitDistrict ...................................................................7.4-24
7.4.7.10 Materials Return KitSchool ..................................................................7.4-24
7.4.7.11 Miscellaneous Return Materials ...............................................................7.4-24
7.4.7.12 Paper Bands .............................................................................................7.4-24
7.4.7.13 Rulers/Manipulatives/Calculators.............................................................7.4-24
7.4.7.14 Miscellaneous Memos, Forms, Labels, and Other Products ....................7.4-25
Page iv
7.4.8 Online Management System .............................................................................7.4-25

7.4.8.1 Test Staff Contact Information ..................................................................7.4-27
7.4.8.2 Test Staff Passwords...................................................................................7.4-27
7.4.8.3 Test Session Requirements ........................................................................7.4-27
Creating and Managing Test Sessions................................................................7.4-27
7.4.8.4 Security Requirements ..............................................................................7.4-34
Physical Security ...............................................................................................7.4-34
Network Security ...............................................................................................7.4-34
Software Security...............................................................................................7.4-35
7.4.9 Computer-Based Test Administration Overview...............................................7.4-35
7.4.9.1 Hardware and Software Requirements ......................................................7.4-37
7.4.9.2 Eligible Device Specifications ....................................................................7.4-37
7.4.9.3 System Performance Requirements ..........................................................7.4-37
7.4.9.4 Connectivity Requirements .......................................................................7.4-39
7.4.9.5 Test Site Set-Up/Certification Requirements.............................................7.4-39
7.4.9.6 Computer-Based Test Presentation Requirements ....................................7.4-39
Tools and Features.............................................................................................7.4-39
Accommodations and Embedded Supports.......................................................7.4-40
7.4.9.7 Computer-Based Test Screen Layout Requirements .................................7.4-45
Student Interface Layout...................................................................................7.4-45
7.4.9.8 Training Delivery Requirements ................................................................7.4-47
Proposed Plan for Early Communication ..........................................................7.4-49
Proposed Training Plan......................................................................................7.4-49
7.4.9.9 Technology-Supported Item Types ............................................................7.4-53
7.4.10 Customer Service and Technical Support ..................................................7.4-65
Page v
7.5.0 Scanning, Scoring, and Reporting ........................................................................7.5-1

7.5.1 Scoring and Reporting Specifications .................................................................7.5-1
7.5.2 Preparing School and Student Information ........................................................7.5-7
Overview of Proposed Plan ..................................................................................7.5-8
Specific Requirements .........................................................................................7.5-9
7.5.3 Scanning/Editing PBTs and Extracting CBT Records .......................................7.5-11
7.5.4 Scoring...............................................................................................................7.5-19
Flow of Data .......................................................................................................7.5-20
7.5.4.1 Types of Scored Files .................................................................................7.5-21
7.5.4.2 Types of History FIles ................................................................................7.5-23
7.5.4.3 Types of Scores and Codes on the SSR/DSRs............................................7.5-23
7.5.4.4 Handling Special Print Documents ............................................................7.5-24
Processing of Accommodated Materials ............................................................7.5-24
Braille Transcription ..........................................................................................7.5-25
Large Print and One-Item-Per-Page Transcription............................................7.5-25
Special Document Tracking Log........................................................................7.5-26
7.5.4.5 Handscoring ...............................................................................................7.5-26
Staffing ...............................................................................................................7.5-27
Scoring Facilities ...............................................................................................7.5-29
Image Handscoring ............................................................................................7.5-29
Quality Control of the Handscoring Imaging System........................................7.5-30
Selecting and Training Scorers..........................................................................7.5-33
Scoring Procedures ............................................................................................7.5-35
Page vi
7.5.5. Reporting ..........................................................................................................7.5-41

Online Reports...................................................................................................7.5-43
AIRs Paper Reports ...........................................................................................7.5-56
7.5.5.1 Initial Reporting Guidelines.......................................................................7.5-63
7.5.5.2 Late Reporting Guidelines .........................................................................7.5-68
7.5.5.3 Close-Out and Annual Reporting Guidelines ............................................7.5-70
7.5.5.4 Demographic Reporting Guidelines ...........................................................7.5-70
7.5.6 Verifying Data....................................................................................................7.5-71
7.5.6.1 Checking Mock Data ..................................................................................7.5-71
7.5.6.2 Checking Early District Return Data.........................................................7.5-74
7.5.6.3 Checking Data for Anomalous Scores .......................................................7.5-75
7.5.6.4 Checking Full-State DataInitial Reporting .............................................7.5-81
Online Reporting System Verification...............................................................7.5-81
Scoring and Reporting Specifications................................................................7.5-81
Student Reponse and Score Data are Correct...................................................7.5-81
Reporting System Software Accurately Reports and Aggregates
Student Scores...................................................................................................7.5-82
7.5.6.5 Checking Graduating Student Scores (Manual Rescore) ..........................7.5-82
7.5.6.6 Checking Late Reports...............................................................................7.5-84
7.5.6.7 Checking Close-Out and Annual EOC Files ..............................................7.5-84
7.5.6.8 Checking Demographic FIles .....................................................................7.5-84
7.6.0 Scaling, Equating, Scoring, and Special Psychometric Studies..............................7.6-1
Calibrating, Scaling, and Equating ...............................................................................7.6-2
Florida Technical Advisory Committee (TAC).............................................................7.6-6
7.6.1 Assessments ........................................................................................................7.6-6
Page vii
7.6.1.1 Florida Standards Summative Assessments (Scaling, Equating,

and Scoring) .............................................................................................................7.6-6
7.6.1.2 Retake Assessments (Scaling, Equating, and Scoring) ................................7.6-8
7.6.2 Scaling, Equating, and Scoring (Psychometrics) ................................................7.6-9
7.6.2.1 Scaling and Equating Team........................................................................7.6-11
7.6.2.2 Selection of Calibration Samples...............................................................7.6-11
7.6.3 Technical Oversight and Special Studies ..........................................................7.6-13
7.6.3.1 Contractors Role .......................................................................................7.6-13
7.6.3.2 Technical Report........................................................................................7.6-13
Item Analysis .....................................................................................................7.6-16
7.6.3.3. Transitioning from FCAT 2.0 and EOC Assessments to
Florida Standards Assessments (FSA)Linking the Scales.....................7.6-18
Additional Considerations .................................................................................7.6-20
7.6.3.4 Implement Achievement Level Standard Setting Meetings.......................7.6-20
Plan for Setting Performance Standards ...........................................................7.6-20
7.6.3.5 Vertical Scaling ..........................................................................................7.6-28
Vertical Linking Background .............................................................................7.6-28
Using Growth Scales for Accountability............................................................7.6-29
Technical Approach to Developing the Vertical Scale.......................................7.6-30
7.6.3.6 Scaling and Equating Studies ....................................................................7.6-31
7.6.3.6.1 Investigate the Comparability between Devices
Delivering Assessments .....................................................................................7.6-36
7.6.3.6.2 Linking Florida Standards Assessments to Other
Assessments Measuring the Same Standards ....................................................7.6-39
7.6.3.6.3 Linking Florida Standards Assessments to
International Assessments.................................................................................7.6-40
Page viii
4. Management Plan
7.7.0 Program Management..............................................................................................4-1
7.7.1 Provisions Governing the Work of the Contractor ................................................4-2
7.7.2 Contractor Staff and Responsibilites .....................................................................4-3
AIRs Assessment Program Structure .........................................................................4-3
Project Organization Chart ....................................................................................4-9
Staff Loading Chart...............................................................................................4-10
Staff and Total Person Days by Month by Category Chart ..................................4-11
AIRs Proposed Project Staff......................................................................................4-12
DRC Proposed Project Staff ......................................................................................4-25
7.7.3 Subcontractors .....................................................................................................4-30
7.7.4 Program Management Communication................................................................4-33
7.7.5 Program Management Meetings ...........................................................................4-34
Communications Plan...............................................................................................4-35
7.7.6 Program Management Reports .............................................................................4-36
Management Documents...........................................................................................4-37
Change Management .................................................................................................4-39
Issue Management.....................................................................................................4-45
Risk Management ......................................................................................................4-46
Annual Program Review and Accounting Reports....................................................4-46
7.7.7 Meetings Involving Florida Educators and Other Citizens ..................................4-47
Meeting Logistics.......................................................................................................4-48
Committee Travel, Communication, and Reimbursements .....................................4-49
Contractor Staff Travel and Expenses ......................................................................4-50
Page ix
7.7.8 Communication between the Contractor and Districts.......................................4-50

Proposed Methods for Communicating with Districts ..............................................4-50
Description of Process to Draft, Review, and Approve All
District Communications..........................................................................................4-53
Dedicated Customer Support Line for Districts .......................................................4-53
7.7.9 Reimbursable Funding Categories .......................................................................4-53
7.7.10 Final Delivery of Materials .................................................................................4-54
Transition Experience...............................................................................................4-55
Conclusion ................................................................................................................4-60
Appendix A: Resumes
Appendix B: Materials List
Appendix C: Sample Print Materials
Appendix D: Sample User Role Chart
Appendix E: Sample Panelist Evaluation Form
Appendix F: Sample Interpretive Guides and User Guides
Appendix G: Sample Family Reports
Appendix H: Sample Communications Plan
Appendix I: Sample Issues Log
Appendix J: Sample Risk Register Format
Appendix K: Sample Change Control Log
Appendix L: Sample Handscoring Reports
Appendix M: Required Documentation
Appendix N: Smarter Balanced Draft Policy
Page x
ATTACHMENT 4
REFERENCES
RESPONDENT NAME: American Institutes for Research
PROVIDE THE FOLLOWING REFERENCE INFORMATION FOR A MINIMUM OF THREE (3) BUSINESSES
WHERE SERVICES OF SIMILAR SIZE AND SCOPE HAVE BEEN COMPLETED.
BUSINESS NAME:
Ohio Department of Education
ADDRESS:
CONTACT PERSON:
25 South Front Street

Mail Stop 507
Columbus, OH 43215
Mr. Jim Wright, Director of Assessment
PHONE NUMBER:
(614) 387-2218
FAX NUMBER:
(614) 995-5568
E-MAIL ADDRESS:
james.wright@ode.state.oh.us

OF SERVICES:
Ohio Achievement Assessments (OAA)

Ohio Test of English Language Acquisition (OTELA)
Ohio Alternate Assessment for Students with Significant Cognitive
Disabilities (AASCD)
3/20026/2014 (all projects)
Ohio Computer-Based Assessments (OCBA)
11/20136/2018
AIR successfully completed more than a decade of continuous support to Ohio,
including item development and test delivery for more than 950,000 students.
We support testing for the achievement tests in reading, mathematics, and
science (as well as writing and social studies until 2009) in grades 38;
graduation tests (OGT) in five subjects, including science and social studies;
English language proficiency (OTELA); and alternate assessments for students
with disabilities.
Page 1 of 3
BUSINESS NAME:
Delaware Department of Education
ADDRESS:
CONTACT PERSON:
401 Federal Street

Dover, DE 19901-3693
Brian M. Touchette, Director, Accountability Resources
PHONE NUMBER:
(302) 735-4090
FAX NUMBER:
(302) 739-4654
E-MAIL ADDRESS:
btouchette@doe.k12.de.us

OF SERVICES:
Delaware Comprehensive Assessment System

DCAS Alternate Assessment
12/200906/2014 (All projects)
AIR developed and delivers the online Delaware Comprehensive Assessment
System (DCAS). DCAS includes assessments for 100,000 students in grades 2
10 in reading, mathematics, science, and social studies and end-of-course
exams in U.S. History, Biology, Algebra II, and Integrated Math III. It includes
fully integrated benchmark and formative assessments and expansive learning
supports.
AIR also developed and delivers the adaptive, task-based DCAS Alternate
Assessment for Students with Disabilities.
Page 2 of 3
BUSINESS NAME:
Hawaii Department of Education
ADDRESS:
1390 Miller Street

Honolulu, HI 96813
CONTACT PERSON:
Cara Tanimura, Director, Systems Accountability Office
PHONE NUMBER:
(808) 586-3283
FAX NUMBER:
(808) 586-3234
E-MAIL ADDRESS:
cara_tanimura@notes.k12.hi.us

OF SERVICES:
Hawaii State Assessments 7/20066/2014

Hawaii End-of-Course Prime 8/20126/2014
Hawaii Alternate Assessment 6/20126/2015
AIR delivered the paper-based Hawaii State Assessments (HSA) from 2006
through 2010, before the HSA moved completely online for the 20102011
school year. We currently test 120,000 students in grades 38, 10, and 11 in
reading, mathematics, and science each year, including ~1,000 technologyenhanced, machine-scored constructed-response items.
We also administer the assessment for students taught in the Hawaiian
Language Immersion Program at grades 3 and 4; online high school end-of
course assessments (American History, American Government, Algebra I and II,
and Expository Writing); and the adaptive, task-based Alternate Assessment
for Students with Disabilities.
Page 3 of 3
Overview of Solution
For more than a decade Florida has been
nationally recognized as a pioneer in innovative education reform. Floridas reforms have
yielded unprecedented improvements in student achievement, demonstrated by improved
National Assessment of Educational Progress
(NAEP) scores, a closing of the achievement
gap, and increased graduation rates. Florida
has experienced success in implementing
major accountability reforms for schools and
districts, with the A to F grading system; for
students, with statewide third-grade progression requirements and high school graduation
standards; and for teachers and administrators, with educator evaluation based on student outcomes. Even the accreditation standards for teacher preparation institutions now
include components to account for the effectiveness of the newly minted educators based
on their effectiveness in improving student
outcomes. Each of these education reforms
depends on fair, accurate, and transparent
measures of student achievement. Continuity
of the reforms depends on the integrity and
success of the underlying assessments and
data.
The Florida Standards Assessments (FSA) will
yield the data on which Floridas reforms rely.
These policy uses are high stakes, and require
timely data. Florida seeks common core tests
that have been field-tested and are ready for
operational administration by spring 2015.
Florida cannot afford to risk not having an
assessment system in place that will serve
these purposes. The assessment must measure
the full breadth and depth of the Common
Core State Standards (CCSS), ensuring full
and accurate measurement of the intended
curriculum for each student.
Governor Scott has indicated that the assessment system must

not significantly increase the testing time
required;
not significantly increase the cost of testing
to states, districts, or schools;
allow students to test as late in the school
year as possible;
meet the same high psychometric standards as the current assessments, including
reliability and evidence of validity;
provide a basis for comparing performance
to other states; and
require technology that can be supported
without significant additional investment.
In addition, the ITN anticipates continued policy innovation. Therefore, the state requires
the ability to integrate custom, Floridaowned items in the test starting after the
initial administration without breaking
trend lines (we propose to integrate fieldtest items even the first year);
the ability to review items starting the first
year to ensure that they meet Florida community expectations and cohere with community standards; and
innovative, thoughtful use of technology to
measure the important skills and knowledge embodied in the standards in deeper
ways without imposing additional barriers
to accessibility.
Our proposed solution offers Florida two different options for the baseline, first-year test,
with the two options differentiated by cost,
testing time, and the types of measurement
used. Both options offer flexibility for the test
Page 1-1
to evolve at Floridas discretion over time. The

two options for the baseline, first-year test
include
the Student Assessment of Growth and
Excellence (SAGE); and
the Smarter Balanced Assessment (Smarter
Balanced).
SAGE was developed in partnership with the
state of Utah to measure the full breadth and
depth of the CCSS. Smarter Balanced was
developed by the Smarter Balanced
Assessment Consortium. Both item banks were
designed to support adaptive assessments, and
therefore offer broad and robust pools from
which to select fixed forms. The two options
differ in cost, testing, and turnaround time and
both will be fully field-tested and calibrated in
spring 2014. The terms of use of each test
allow Florida to enhance them with Florida
items, to modify the blueprints even the first
year, and to select fixed-form tests.
Within, we propose to deliver items selected
from either bank, assembled onto fixed forms.
These items will be vetted by Florida committees before administration. Each year, beginning in 20142015, we will embed Floridaowned items in the tests, building a Floridaowned item bank.
Our proposal begins with a brief survey of the
policy demands placed on the FSA, and identifies technical and timeline requirements. For
example, Floridas Grade 3 Reading guarantee
requires that the FSA offer a reliable Reading
score (not only the combined English
Language Arts score), and that the cut score
separating level 1 from level 2 be established
with this use in mind. We offer a plan to meet
those needs and deliver results by mid-May,
when they are required. Throughout, we offer
Page 1-2
a plan that will ensure that the FSA is appropriate for the uses to which it must be put, and
that it will do so meeting the highest professional standards.
Our plan achieves very fast turnaround,
enabling Florida to push testing windows into
April, if so desired, allowing more instruction
before assessment. We shorten the timeline
through innovative, but well-proven, uses of
technology. Two key innovations include the
use of powerful machine-scored constructedresponse items to maintain measurement rigor
while allowing immediate reporting, and the
introduction of real-time-ID labels.
Real-time-ID speeds paper processing. In the
past, cleanup of data from online tests proceeded much faster than paper tests. For
online testing all students must be positively
identified prior to testing. Paper tests often
rely on hand-gridding of identifiers for mobile
students. This imparts a double blow on timelinesthe data is error prone because it is
hand gridded, and cleanup cannot start until
after the books are scanned. Real-time-ID
labels allow schools or districts to enroll students and print labels in real time. The error
rate is nearly eliminated because the data can
be validated on entry. Because the student
associated with the label is known immediately, cleanup can begin as soon as the label is
printed.
Cumulatively, our approach provides Florida a
robust, calibrated common core item bank
from which to build first-year forms. It provides national and international benchmarks,
and standard setting, psychometric, and validity studies that will support the high-stakes
uses for which the FSA is destined. We offer
well-proven systems that use technology to
improve measurement, enhance quality, and
speed results.
Who We Are
Based in Washington, DC, the American
Institutes for Research (AIR) is a not-for-profit
organization providing research, assessment
services, and technical assistance. We are over
1,500 people working to improve lives in communities at home and around the world. Our
mission calls for us to turn the best research
into practice, which we do in every arena in
which we work. In student assessment, our
commitment is evidenced by our success
delivering:
a testing system with a very small footprint
in the schools. We have designed the system to work on the newest mobile devices
as well as minimally equipped, older computers, with minimal bandwidth, and with
little or no technical support in the schools.
Within, readers will find that three times
more computers in use in Florida schools
meet our minimum requirements than
meet the minimum requirements specified
in the ITN.
tests to all students, regardless of their
needs. Our system offers the most robust
set of accessibility tools in operational use
in the industry, as well as compatibility
with most assistive devices. In some states
where we work, all students eligible for the
general education accountability tests test
online, including students who are blind,
have severely restricted movement, are
deaf, or (through our secure print-ondemand feature) risk seizure if they look at
computer screens.
a broad array of machine-scored item
types, including a variety of machinescored constructed-response item types
that can be scored with near-perfect reliability. For example, in Mathematics, we
were the first to introduce machine-scored
items in which students respond with an
equation or expression, and exploit sophis-
ticated symbolic algebra algorithms to

score the responses. Our equation-scoring
engine can even award partial credit, for
example, awarding credit for an equation
with the correct structure, even if the
answer is not perfectly correct.
These items facilitate measurement of deep
knowledge and skills while simultaneously
making instant reporting possible. In many
states in which we work, students and educators receive electronic reports within seconds
of completing tests.
We are pleased to be able to offer a mature
solution. AIR has been delivering adaptive and
fixed-form online tests for 7 years. In addition
to millions of operational tests in Oregon,
Delaware, Hawaii, Minnesota, and Ohio, our
online testing system recently delivered the
Smarter Balanced Assessment Consortiums
pilot test to over half a million students across
21 states. This spring, we will serve approximately 5 million Smarter Balanced students.
Our system is versatile, works well within
schools, and is easy to useall characteristics
that enable us to extend service to so many
states, schools, and students in such short
order.
For this project we have teamed with Data
Recognition Corporation (DRC). Our solid
partnership with DRC extends across many
states, including the Smarter Balanced
Assessment Consortium. Among the very few
organizations that have the test processing
capacity to serve Florida on the required timelines, only DRC has achieved ISO 9001:2008
certification, ensuring that their processes are
replicable and reliably followed. DRC brings 30
years of experience, and has successfully
processed and handscored millions of tests for
large-scale statewide assessments, including
Alabama, Alaska, Louisiana, Nebraska, Ohio,
Oklahoma, Pennsylvania, South Carolina, and
Washington. One of the most experienced test-
Page 1-3
ing contractors in the nation, DRC scores and

reports assessment results for more than
4.5 million students across the country.
Should Florida select our team, DRC plans to
establish handscoring facilities in Florida,
keeping significant employment in the state
and contributing to the states economic development.
Project Leadership Team

Capable, experienced organizations with the
capacity to do the job are a necessary condition for success. Focusing these capabilities
effectively requires experienced and capable
leadership. We offer Florida our A-team lineup.
Key staff include
Kevin Murphy, a director of state programs
at AIR, typically oversees the project directors on multiple projects. Mr. Murphy will
be committed full time to ensuring the success of Floridas program as our proposed
full-time senior program manager. A Project
Management Institute (PMI) certified manager, Mr. Murphy has more than
12 years of experience directing multi-million dollar, statewide assessment projects.
For 8 years at AIR, he has successfully led
the paper-and-pencil testing program in
Ohio, managed projects to transition
Hawaii and Delaware from paper-and-pencil
testing programs to 100% online, and managed Minnesotas dual-mode program. He is
experienced in all aspects of assessment
projects, including test development, paperand-pencil and computer administration,
psychometrics, scoring, reporting, and professional development. Mr. Murphy also
brings 6 years of classroom teaching experience. Mr. Murphy received his B.A. in
business from Lehigh University and his
M.A. in secondary education from
Hunter College.
Page 1-4
Dr. Harold Doran, our proposed lead psychometrician, has served Florida on its
value-added project, the data for which will
eventually come from the FSA. Florida has
seen firsthand Dr. Dorans ability to render
sophisticated technical information comprehensible to the stakeholders whose lives
it affects. Florida has also watched as Dr.
Doran interacted with some of the nations
leading technical experts on its Valueadded Technical Advisory Committee. In
addition to supporting the psychometrics of
large-scale assessment programs, Dr. Doran
is an expert in state assessment policy and
has helped states complete studies needed
for a successful assessment peer review
with the federal government, including
work for the District of Columbia,
Delaware, Hawaii, Indiana, Michigan,
Oregon, and Wisconsin. Dr. Doran was
appointed by then-Secretary of Education
Margaret Spellings to serve on the U.S.
Department of Educations National
Technical Advisory Committee.
Sandra Durden, our proposed project manager for test development, has more than
20 years of experience in the field of education, including 10 years in the assessment
industry and 10 years as a K8 educator. At
AIR, she is working on performance tasks
for the Smarter Balanced Assessment
Consortium and item development for state
programs and K2 formative assessments.
Prior to joining AIR, Ms. Durden worked at
Pearson and Harcourt, where she served
most recently as director of research and
development and was responsible for all
phases of new product research, design,
and development for learning assessments.
Stacy Tsakeris will serve as project manager, computer-based testing. Ms. Tsakeris is
managing the quality control for the
Smarter Balanced field testing in 24 states
and managing the test content documentation of the Open Source software that will
be available to the Smarter Balanced states.

Ms. Tsakeris has more than 10 years of
experience in software project management, and has earned her PMP, her M.B.A.,
and her B.S. in computer science and engineering. She reports directly to Selina
Tolosa, who directs our software group. For
Florida, Ms. Tsakeris will be supported by
our most experienced team of software
project managers and engineers. These
include
Jeremy Fisher, software project
manager, test delivery, M.S. (systems
development management);
Aaron Newman, senior application
developer, M.S. (electrical engineering
and computer science);
Shailesh Gattewar, software project
manager, student registration, M.B.A.
and M.A. (computer science), PMP;
Anil Chevuri, senior application
developer, B.E. (electrical engineering);
Zorina Alliata, software project
manager, item banking, M.B.A., PMP;
Dipesh Mamtora, senior application
developer, M.B.A. and M.S. (electrical
engineering);
Brian Kane, software project manager,
Ph.D. (astronomy), PMP; and
Paul Van Wamelen, senior application
developer, Ph.D. (mathematics).
DRCs Peter Tressel (M.B.C., PMP) is the
proposed project manager, paper-and-pencil
operations. Mr. Tressel will serve as DRCs
project manager, overseeing and facilitating
DRCs work on the Florida Standards
Assessments and leading manufacturing
and processing of paper tests. Mr. Tressel
has nearly 15 years of project management
experience directly related to large-scale,
statewide assessment programs. In his current role at DRC, Mr. Tressel provides over-
all management and direction for DRC as a

subcontractor to AIR in Minnesota and
Ohio. In addition to his work in the testing
industry, Mr. Tressel has more than 20
years of related experience with a variety of
writing and editing projects and activities.
Ms. Sherri Wolf will lead our handscoring
efforts on behalf of Florida and set up and
manage the new scoring centers in Florida.
Ms. Woolf is vice president of performance
assessment services. Ms. Woolf will oversee
all handscoring activities for the project.
Ms. Woolf has more than 20 years of educational assessment experience, and has
managed performance assessment activities
for Alabama, Arkansas, Kentucky,
Louisiana, Minnesota, Nebraska, North
Carolina, Ohio, Oklahoma, Pennsylvania,
and South Carolina.
In addition to this leadership team, AIR and
DRC have assigned full-time project managers
for handscoring (Wendy Marik), test administration (Meredith Durgin), and scoring and
reporting (Cathy Kugler).
Summary
We offer Florida the choice between two large,
common core item banks, each of which is
field-tested and delivered in our system. That
system is designed to work within the constraints of real school environments. We find
nearly a million computers in Florida that
meet our systems requirementstriple the
number that meet the requirements outlined
in the ITN.
Our proposed plan is guided by clear and
detailed understanding of the intended uses of
the FSA, and is designed to support those uses
with the highest professional quality. We propose to do so while shortening the timeline
between testing and reporting, allowing Florida
to move testing windows closer to the end of
the school year.
Page 1-5

Backround
In many ways, AIR has been working on this
proposal for more than a decade. During that
time, AIRs team has proven our ability to support states in the development of innovative
and customized standard-based assessments,
groundbreaking psychometric advances, and
the transition to online testing.
In addition to developing customized statewide
assessments beginning with the first California
State High School Exit Exam in the 1990s, AIR
is a leader in developing assessments to measure the full breadth and depth of the Common
Core State Standards (CCSS) that underlie the
Florida Standards Assessment (FSA). These
efforts include the Student Assessment of
Growth and Excellence (SAGE) developed in
partnership with the state of Utah and the
Smarter Balanced Assessment Consortium
(Smarter Balanced). For both of these efforts,
AIRs test development staff and technologies
led the development of the vast majority of
technology enhanced items. For Smarter
Balanced, AIR also developed a full set of
open-source scoring engines. We bring experience with traditional and technology-based
items for standards-based assessment using
evidence-centered design, permissioned and
commissioned stimuli, and a unique understanding of the nature of building an item bank
for testing. AIR has consistently experienced
item acceptance rates above 90% in our state
assessment contracts, at volumes of 15,000
items per year. On the National Assessment of
Educational Progress (NAEP) contract, we
completed item development with National
Assessment Governing Board (NAGB) acceptance rates above 95%.
AIR is the undisputed leader in online testing
for statewide assessment. We have transitioned
states to online in a single year and have suc-
cessfully delivered the identical technologyand media-intensive assessments where previous contractors have failed. We currently
deliver all online statewide assessments in
Oregon, Hawaii, Delaware, Minnesota, and
Ohio, and have recently won and started contracts to deliver the online testing in Utah,
Missouri, and Michigan beginning in spring
2014. Just this month, AIR also won new contracts to continue delivering online testing in
Ohio and Oregon. For the Smarter Balanced
Assessment Consortium, we successfully delivered the testing system for the 2013 pilot test
for more than 800,000 students in 23 states
and will deliver more than 15 million tests
during a single Field Test window in spring
2014, including 95% of the students in
California.
Our testing delivery system is proven and
leads to faster and more comprehensive adoption of online testing. We have led multiple
states to 99+% online testing in the first year
of testing programs, while other vendors have
not achieved 90% after 10 years. Our system
has the smallest technology footprint possible
in the school and does not require proctor
caching. AIR has proven our ability to deliver
tests with 1/5 the bandwidth required by the
planned PARCC system, with full reliability,
expanding the number of schools that can successfully test online. We have proven the ability to deliver on the widest range of devices,
from the oldest computers to the newest
tablets.
We are committed to 100% accessibility in
online testing for all students and are proud
that virtually all students are receiving fully
accessible adaptive tests in Hawaii and
Oregon. AIR is the only system with an operational track record delivering the full set of
PARCC accommodations. No existing system is
more accessible than ours, and only our sys-
Page 2-1
tem has been proven in real statewide

accountability tests.
Beyond online testing, AIR has demonstrated
the ability to be a vendor and partner that
reaches beyond state-of-the-art practices in
educational assessment and delivers innovative
models, materials, and strategies that advance
the field of measurement and improve the
validity of student scores. In student assessment, our commitment is evidenced by our
success in delivering innovations, including
the following:
The first and only ESEA-approved, fullyear, multiple-opportunity, online adaptive
assessments are now operational in three
states, and we are starting projects to deliver the same in Utah, Missouri, and
Michigan.
Innovative, machine-scored constructedresponse items are operational.
Item sharing of summative multiple-choice
and technology-enhanced items are available across states, years before Smarter
Balanced or PARCC.
The first statewide online adaptive formative assessments are delivered in the same
system as the summative assessments and
measure the same constructs on the same
scale.
The first ESEA-approved adaptive alternate
assessments that measure growth for students with significant disabilities are as
psychometrically sound as the assessments
for the general population.
The largest and most complex, full-color,
variable data score-reporting system in the
industry is available, including powerful
online reports and learning supports that
empower educators and individual families
to use information from assessments effectively.
Page 2-2
Founded in 1946 as a not-for-profit organization, AIR conducts our work with strict independence, objectivity, and non-partisanship.
We are 1,600+ people working to improve lives
in communities at home and around the
world. Our mission calls for us to turn the best
research into practice, which we do in every
arena in which we work. AIR brings 60+ years
of experience in aptitude and proficiency testing, including 37 years in K12 educational
assessments. Over the years, AIR staff members have worked on 40+ state and national
assessments (e.g., NAEP, the National
Assessment of Adult Literacy, the Voluntary
National Tests), on international assessments
(e.g., TIMSS, PISA), and on assessments in
Brazil, Namibia, Honduras, Tajikistan,
Macedonia, and other countries.
AIR offers Florida a record of extraordinary
support, particularly in the transition of new
testing programs. Delivering fast transitions
with quality results is not unusual in AIRs
experience. During the past seven years, AIR
has successfully transitioned more than a
dozen separate testing programs in Minnesota,
Delaware, Ohio, Oregon, Hawaii, South
Carolina, and New Mexico from other contractors, and AIR met all key deliverables. As the
following project descriptions demonstrate,
AIR meets our timelines and deliverable
requirements, including situations when
aggressive timelines for project start-up/transition involve large item development efforts,
creating new item banks from other sources,
transitioning to entirely new standards or new
assessment designs, meeting field-test windows, and delivering online testing systems.
We offer this track record as evidence to support our commitment and our plan to meet
Floridas timelines and requirements.
AIR and DRC, A Proven Team

For this project, we have teamed with Data
Recognition Corporation, who has provided
identical services with AIR on projects in Ohio

and Minnesota since 2007:
Human scoring of identified constructed
response items, including associated
rangefinding
Production, distribution, collection, and
processing of all paper/pencil testing
In addition, we have proposed to expand AIRs
capacity by involving DRC in two research
studies (7.6.3.3 Transitioning from FCAT 2.0
and EOC Assessments to Florida Standards
Assessments (FSA) Linking the Scales and
7.6.3.6.1 Investigate the Comparability
between Devices Delivering Assessments) and
engaging DRC item development specialists to
work alongside AIR test development leads.
DRC is one of the most experienced fullservice testing contractors in the nation, with
responsibility for projects involving the testing
of more than 4.5 million students. DRCs current education contract base includes assessment projects for 12 statesAlabama, Alaska,
Idaho, Louisiana, Michigan, Minnesota,
Nebraska, Ohio, Pennsylvania, South Carolina,
Washington, and Wisconsinand the Smarter
Balanced Assessment Consortium. DRC also
provides printing services for large-scale
assessment programs in Connecticut,
Michigan, North Carolina, and Tennessee.
Over the past 33 years, DRC has evolved into
a full-service educational assessment company
with a staff of more than 640 full-time employees and over 4,500 seasonal and temporary
employees in 15 locations around the country.
With its relentless focus on quality and customer service, reviewers will note that DRC
does not have any issues to report under the
Poor Performance Notice section of this RFP.
testing, and software development, AIR and

DRC currently support integrated online and
paper/pencil programs in Ohio and Minnesota.
As a team, AIR and DRC have established
inter-organizational communications and procedures, ensuring that consistently high-quality deliverables are produced on time by multiple teams. Few in the industry can match
DRCs record of error-free service. The reason
for this excellent record is simple: DRCs quality processes focus on doing things right the
first time, identifying and resolving issues
before they affect students and educators in
client states. DRC is also currently ISO
9001:2008 certified in three major areas of the
company: Document Services, Educational
Operations, and Performance Assessment
Scoring. Attaining ISO 9001 certification has
enhanced DRCs already strong foundation of
business quality controls.
AIRs successful subcontracting with DRC is
based on three critical components:
1. Clear delineation of responsibility and
expectations.
2. Accountability mechanisms.
3. Trust and integrity on the part of each
player.
AIRs formal subcontract agreements outline
its commitments, which include an acknowledgment of the required documents, areas of
responsibility, and the products for which DRC
is responsible. AIR staff review subcontractor
work and hold it to the same high standards as
AIR holds itself. This formal, clear communication of expectations details the means by
which developed products will be deemed
acceptable.
AIR and DRC are a proven team. In addition to

partnering on three separate Smarter Balanced
contracts covering a wide range of responsibilities, including item development, pilot/fieldPresented by the American Institutes for Research
Page 2-3
Relevant Experience on Similar

Projects
Exhibit 2-1 (at the end of this section) presents a summary of relevant AIR assessment
projects. Below we highlight several projects as
evidence of AIRs relevant significant expertise
in large-scale test development and record of
accomplishment in the development and
refinement of high-quality assessment instruments.
20122014. Through two prime contracts
and one subcontract, AIR is supporting the
Smarter Balanced Assessment Consortium in
seven major activities, covering the research
of new item types, the development of several
thousand items, the delivery of the pilot and
field tests to up to 6 million students in 20+
states, and the development of the core, opensource test delivery system and automated
scoring technology. The following are some
highlights:
Working with teacher focus groups from
across the Consortium, AIR developed
approximately 1,700 math and ELA items
for the spring 2013 pilot administration.
AIR and its partners have conducted
approximately 700 cognitive labs in the
Consortium states of California, Hawaii,
Oregon, Pennsylvania, Michigan,
Connecticut, South Carolina, Iowa, and
Idaho.
AIR delivered the testing system for the pilot
test for more than 800,000 students.
AIR, working with DRC, developed the
manuals and communications materials
and provided the recruiting and Help Desk
support for the pilot test.
Oregon Assessment of Knowledge and Skills
(OAKS) 20072014. AIR administers Oregons
Assessment of Knowledge and Skills Online, or
OAKS Online, the first NCLB-approved online
Page 2-4
adaptive state assessment system. AIR delivers

approximately 2 million adaptive tests annually to more than 335,000 Oregon students in
grades 312 in mathematics, reading, science,
social sciences, and writing in a 7-month testing window each year. AIR also helped develop
and delivers Oregons Online English Language
Proficiency Assessments (ELPA). The following
are highlights beyond excellent performance:
AIR developed and implemented the first
adaptive Braille testing.
AIRs psychometrics and statistics team
worked to help Oregon develop technical
reports and design and implement studies
that were submitted for the states ultimately successful NCLB peer review.
From 2007 to 2013, AIR field-tested
approximately 15,000 items with a rapid
turnaround time; some field-test items were
analyzed and inserted into the operational
item pool within a month of being fieldtested.
AIRs item banking system is used by
Oregon teachers to develop items.
Delaware Comprehensive Assessment System
(DCAS) 20092014. AIR developed and delivers the online Delaware Comprehensive
Assessment System (DCAS). DCAS includes
assessments for 100,000 students in grades
210 in reading, mathematics, science, and
social studies and end-of-course exams in U.S.
History, Biology, Algebra II, and Integrated
Math III. It includes fully integrated benchmark and formative assessments and expansive learning supports. AIR also developed and
delivers the adaptive, task-based DCASAlternate Assessment for Students with
Disabilities. The following are some highlights:
More than 99% of Delaware students have
tested online since the initial statewide
field test.
AIR imported nearly 12,000 existing

English and Spanish items and aligned
them to new Delaware standards.
AIR developed 4,000+ new items aligned to
Delaware standards, including hundreds of
technology-enhanced, machine-scorable,
and natural language items.
There has been exponential growth in the
use of AIRs Learning Point Navigator formative system for instructional resources and
diagnostic assessments.
Hawaii State Assessments (HSA) 20062014.
AIR delivered the paper-based Hawaii State
Assessments (HSA) from 2006 through 2010,
before the HSA moved completely online for
the 20102011 school year. We currently test
120,000 students each year in reading, mathematics, and science at grades 38, 10, and 11,
including ~1,000 technology-enhanced,
machine-scored constructed-response items.
We also administer the assessment for students
taught in the Hawaiian Language Immersion
Program at grades 3 and 4. In 2012, AIR won
contracts to develop online high school end-ofcourse assessments (American History,
American Government, Algebra I and II, and
Expository Writing) and our adaptive, taskbased Alternate Assessment for Students with
Disabilities. Both of these new assessments
were developed on a very fast track and were
operational in spring 2013. The following are
highlights beyond excellent performance:
More than 99% of Hawaii students have
tested online since the initial statewide
field test.
Fully-adaptive Braille testing has been

implemented.
AIR developed 2,400+ new items, including
hundreds of technology-enhanced,
machine-scorable, and natural language
items.
Ohio Assessment System 20022014. AIR
successfully completed more than a decade of
continuous support to Ohio, including item
development and test delivery for more than
950,000 students. We support testing for the
achievement tests in reading, mathematics,
and science (as well as writing and social studies until 2009) in grades 38, graduation tests
(OGT) in five subjects including science and
social studies, English language proficiency
testing (OTELA), and alternate assessments
for students with disabilities. The following
are highlights beyond excellent performance:
In 2013, AIR oversaw the scoring of more
than 3 million tests for more than 950,000
students.
AIR delivered Ohios first online pilot tests
in social studies.
AIR implemented our enhanced, full-color,
variable paper score reports for the first
time.
AIR implemented the transition from a
portfolio alternate assessment to our adaptive alternate assessment measuring
growth.
Minnesota Assessment System 20112015.
AIR is beginning our third year of supporting
the online and paper-pencil Minnesota
Assessment System for approximately 500,000
students. We support testing for the Minnesota
Comprehensive Assessments (MCAs) in reading, mathematics, and science in grades 38,
10, and 11; monthly Graduation-Required
Assessments for Diploma (GRAD) in writing,
reading, and mathematics; and alternate
assessments (MCA-Modified and the Minnesota
Page 2-5
Test of Academic Skills [MTAS]). All of these

have online components, including multipleattempt adaptive testing in long windows and
fixed-form testing in short windows, based on
subject area. Highlights include the following:
AIR delivered Minnesotas highly technology-enhanced science assessments with 90%
less bandwidth requirements than the previous contractor, significantly enhancing
school satisfaction with the assessment.
AIR developed highly interactive science
simulations for science assessments.
AIR enhanced procedures and resources
required for success.
As shown in the project descriptions above,
AIR has demonstrated the ability to deliver
custom state assessment programs for Florida
and at scale across states. In K12 assessments, AIR is successfully supporting state
assessment projects across the country, delivering approximately 8 million tests for more
than 2 million students each year.
~3 million assessments for more than
950,000 students in Ohio, including grades
38 annually since 2003, and the highstakes Ohio Graduation Tests
~2 million online interim and summative
assessments for 335,000 students in
Oregon annually since 2007
~1 million online interim and summative
assessments in Hawaii (120,000 students)
and Delaware (100,000 students) annually
since 2009
Dual paper/pencil and online testing program delivering ~1.4 million assessments
for 500,000 students in Minnesota since
2011
Page 2-6
Alternate Assessments of Students with

Disabilities in Ohio, Minnesota, Hawaii,
Oregon, South Carolina, and New Mexico
Tests of English language acquisition on
paper in Ohio and via computer in Oregon
As some additional metrics of our proven
capacity, AIR annually
develops approximately 15,000 test items;
deploys approximately 150 separate assessments in paper/pencil and online, including
approximately 1,000 fixed forms or adaptive item pools;
answers more than 40,000 help desk calls;
and
delivers more than 25 technical and/or
standard setting reports.
AIRs ability to deliver at these quantities with
high-quality is based on our personnel, internal procedures, and industry-leading tools and
procedures.
Other Resources
In addition to the personnel, organization, and
internal procedures described above, AIR has
in place all of the required facilities and other
resources required to successfully administer
statewide assessment programs. Below we
highlight some of the other important
resources we will use on this project.
Hardware and Software Resources to Support
Online Assessment Programs. Three essential
types of computing capacity are required to
support this project:
A robust, high-availability managed hosting
environment for the item banking, test
delivery, and reporting systems, along with
the related supporting systems
A day-to-day work environment for AIRs

staff of content, psychometrics, score
reporting, and other staff to support everything from e-mail and file storage to complex SAS analysis programs and Variable
Data Intelligent PostScript Printware (VIPP)
programming
A full-scale development environment for
all systems, including a vast array of software tools to manage software development
AIR hosts our ITS, TDS, reporting, and related
critical assessment systems and websites in a
managed hosting facility, because the scale and
investment available from a dedicated hosting
company far exceed what any one company
can provide. We use several of Rackspaces
managed hosting facilities to host the online
testing for Ohio, Oregon, Delaware, Hawaii,
Smarter Balanced, and other clients, as well as
other core AIR assessment systems. Rackspace
is rated one of the Magic Quadrant leaders
by Gartner Research. Rackspace secured facilities are designed with redundancy at every
potential point of failure. Multiple redundant
Tier 1 Internet connections, router and switch
configuration, real-time and backup power
supplies are just examples of the components
of Rackspaces Business Continuity Plan
(BCP). Rackspace considers much of this
information proprietary, precisely to maintain
security, but Rackspace security officials are
happy to meet with ODE to provide specific
information in support of AIRs proposal.
Rackspace delivers the best network performance in the industry, using only quality bandwidth from Tier 1 backbone providers. Its network guarantees 100% network uptime. The
company has multi-gigabytes of high quality
provisioned data capacity with virtually unlimited scalability. Rackspace builds in 100%
availability through multiple sites, multiple
redundant network connections, a redundant
router and switch configuration, and multiple
redundant real-time and backup power sources.
It has complete network security and provides

24/7/365 support to us.
AIR manages the day-to-day work computing
environment for our staff. We host all servers
and applications in our main facility in
Washington, DC, and support asynchronous
real-time replication between key servers
located at our main Washington data center
and dedicated backup servers located at an IT
recovery location in our California offices,
using TCP/IP protocols real-time backups. Staff
are equipped with any needed desktop/laptop
computers and peripherals, software, wireless
broadband, iPhones, Blackberries, and other
communication, and are fully supported by
60 staff and our own 24-hour Help Desk.
Finally, AIR maintains a full, on-site development environment for each system. This environment is housed within our professional
grade web services hosting center, supported
by a team of engineers monitoring the continuous operations of the network, including
servers, security, and hosted applications. This
infrastructure includes 100+ servers carefully
monitored, administered, and supported by
our Web Services operations engineers. AIR
has a comprehensive BCP that incorporates
plans for specific recovery time objectives
(RTO) for all core business system; alternate
headquarters and remote access for virtually
all of AIRs staff; and extensive insurance to get
us back up and running quickly in the event of
natural disasters (including floods), sprinkler
leakage, broken water pipes, loss of utilities,
and many other insurable events. AIR employs
a full-time chief security officer, Robert
McMahon, who holds a Certified Functional
Continuity Professional (CFCP) from Disaster
Recovery Institute (DRI) International.
AIR Processing and Scoring Center. AIRs
Processing and Scoring Center, located in
Columbus, Ohio, is a state-of-the-art facilities
Page 2-7
for all packaging, shipping, receiving, processing, scanning, scoring, and storing of test materials as well as hosting the AIR Assessment Help
Desk. This facility, staffed by 13 full-time and
up to 300 part-time scoring, processing, Help
Desk, and other temporary staff, supports
assessments in Ohio, New Mexico, Delaware,
Hawaii, Oregon, Minnesota, and Utah. The
centers high-speed scanners, streamfeeders,
200+ workstations, and other equipment are
supported by software systems that interface
directly with our pre-ID, student tracking,
print tracking, and other systems.
Copyright Acquisition Team. AIR has a team
of staff experienced in copyright acquisition
for assessment projects, including both paid
and open-source materials for all content
areas, including science and social studies.
Their expertise includes written fiction and
non-fiction, images, audio, and video for both
paper/pencil and online assessments. In total,
AIR acquires more than 1,000 permissions
each year.
Physical and Virtual Meeting Resources. AIR
has conference facilities in our Washington,
DC headquarters and our other domestic
offices for small and large meetings. AIR also
has made significant investments in the hardware, software, and connectivity required to
support virtual meetings. We use well-known,
industry-standard solutions that are easily
accessible by other vendors, state clients,
districts, and schools. AIR has a dedicated
10-person conference services team skilled at
planning and supporting virtual conferences,
including for example, WebEx, Tandberg Video
Conferencing, Skype, Microsoft Lync, Join.me,
and Global Crossing traditional audio conference lines.
Section 508 Capabilities and Testing Best
Practices. Section 508 accessibility requirements were enacted in 1998 as an amendment
to the Rehabilitation Act of 1973. The regula-
Page 2-8
tions seek to ensure that people who have disabilities, such as the visually impaired, have
equal access to all information. AIR is fully
compliant with the provisions of Section 508
of the Rehabilitation Act and is proud of its
reputation for delivering high-quality, complete, and accurate Section 508-compliant
products to clients. AIR has developed and
refined processes for (a) creating Section 508compliant Web sites; applications; e-mails;
videos; multimedia; PDF files; and Microsoft
Word, Excel, and PowerPoint files; (b) testing
and validating our work; and (c) performing
quality assurance (QA) checks. As a result,
AIR is the go to contractor for creating
Section 508-compliant materials and Web sites
for Federal Agencies and state clients, including the U.S. Department of Education, Centers
for Disease Control and Prevention (CDC),
multiple National Institutes of Health agencies,
the National Library of Medicine). In several
instances, we have remediated the accessibility work of other contractors, and we have
trained Federal clients how to create Section
508 accessible products.
Ability to Absorb Additional Workload

Resulting from This Project
Florida is rightly concerned with the operational capacity of offerors. The Florida
Standards Assessment is one of the largest
assessment programs in the industry.
AIR can deliver the Florida Standards
Assessment with our existing operational
capacity described above.
All of the personnel we have proposed are current AIR employees and are available to start
working on this project in early 2014. Using
the knowledge we have acquired supporting
statewide assessments in California, Ohio,
Oregon, Delaware, Hawaii, Minnesota, South
Carolina, New Mexico, Utah, and elsewhere
and our experience helping states transition
effectively to online testing, we have identified

staff and committed them at levels that we
believe are fully adequate to complete all deliverables in a timely and effective manner.
Furthermore, we have contingency plans to
add staff as necessary. As a leading applied
behavioral and social science research organization with more than 1,500 employees, AIR
retains a research staff possessing sufficient
breadth and depth of capability across all areas
of this Florida contract. This provides ample
reserve capacity from which to draw staff as
necessary. In addition to the staff identified in
this proposal, AIR and its partners employ
scores of other individuals who fully meet the
staffing requirements in this ITN. Specifically,
because of the other work we do, AIR has deep
resources of content specialists, psychometricians, programmers, and other staff who can
be brought to bear on this contract if necessary. For example, we have calculated that AIR
has the psychometric capacity for more than
twice our projected workload. We maintain
this capacity because AIR employs psychometricians for much more than the administration of operational assessment projects. AIR
psychometricians spend all or portions of their
time performing research for NAEP and other
clients, designing technical innovations, performing other contracted research, and otherwise advancing the field. This other work can
be scheduled around the very specific timeframe demands of state assessment projects.
The same is true for reading, mathematics,
and other content specialists at AIR who normally work on research and technical assistance projects. We have identified and trained
these staff to be ready to augment our staff on
short notice.
additional human scorers within existing

equipment and facilities.
We will host the online assessments in AIRs
private cloud at Rackspace, which runs on
hardware dedicated to AIR and is essentially
infinitely scalable by the addition of more
hardware. Our private cloud insulates Floridas
performance from exogenous loads, while geographic distribution provides redundancy and
eliminates server-side bandwidth restrictions.
Financial Capability
AIR has the financial resources to support this
contract. As a result of the quality of our
research, assessment, technical assistance, and
dissemination efforts, as well as the dedication
and hard work of our employees, AIR has
experienced tremendous growth in the past
decade. In fiscal year (FY) 2012, our revenues
exceeded $300 million, and as a not-for-profit
corporation, all of our net assets are retained
by the organization to support future work.
Unrestricted net assets currently exceed $120
million. Among other purposes, these funds
are available to ensure long-term financial stability of the organization and to support
investments in AIRs Assessment Program
research agenda and intellectual property to
support our contracts with Florida and our
other clients.
In addition to personnel, AIR and DRC currently have all of the facilities required to support Florida. We will process and hand score
assessments in existing DRC (Minneapolis)
facilities. DRC has an unused capacity to
process 10 million tests and support 1,000
Page 2-9
Exhibit 2-1: Summary of AIR Assessment Projects Similar to Florida
Program
Summary
Smarter Balanced Assessment

Consortium (multiple projects)
Administering online pilot and field tests for Delivering large-scale online testing system that
works on all PARCC-mandated platforms and with
~6m students in 21 states; development
all PARCC-required accommodations
and delivery of the open-source Test Delivery
Cognitive lab research and item development on
System; Tier 2/3 Help Desk
innovative online item types
Developing pilot and field-test items for
Development of open-source machine scoring
Smarter Balanced; responsible for
engines
development of open-source machine
scoring engines; small-scale tryouts
Providing communication and recruiting
support, test administration manuals,
training modules, webinars, and Tier 1 Help
Desk services for ~1m students
Oregon Assessment of
Knowledge and Skills Online
Full-year, multiple-opportunity, online

summative and interim assessments for all
students
Proven online testing system, including delivering

ELA and Mathematics assessments at
elementary, middle, and high school
Successful delivery of online English Language
Proficiency Assessment with all accommodations
Delaware Comprehensive
Assessment System
Full-year, multiple-opportunity, online

summative, interim, and formative
assessments for all students

ELA and Mathematics assessments at
elementary, middle, and high school
Successful development and delivery of hundreds
of technology-enhanced, machine-scorable, and
natural language items
Similarity to Florida Standards Assessments

Hawaii State Assessment System Full-year, multiple-opportunity, online
Science assessments at elementary, middle, and
summative and interim assessments for all
high school
students (20102014); paper/pencil testing
Successful development and delivery of hundreds
20062010
of technology-enhanced, machine-scorable, and
natural language items
Hawaii End-of-Course
Assessments
Online summative assessments for all

students
Separate contract to extend the Hawaii State

Assessments to include EOC assessments
Utah Student Assessment of

Growth and Excellence
Online interim and summative assessments

for all students
Developing thousands of items and preparing for

Utah's first statewide online assessments
Page 2-10
Exhibit 2-1: Summary of AIR Assessment Projects Similar to Florida (continued)

Program
Summary
Ohio Achievement Assessments
Full paper/pencil testing program, with

online adaptive formative assessments;
online pilot tests of science and social
studies assessments; K12 assessments
Successful development and delivery of ELA

and Mathematics assessments at elementary,
middle, and high schools
Successful online pilots to support transition to
online testing, with statewide field testing in 2014
Ohio Graduation Tests
Full paper/pencil testing program, with

online adaptive formative assessments
Successful development and delivery of highstakes graduation tests
Minnesota Assessment System
Dual-mode (paper/pencil and online)

summative assessments for all students
Similar to Florida in that contract covers general

assessments and high-stakes graduation testing
and combines paper/pencil and online testing
National Assessment of
Educational Progress (NAEP)
Test development, psychometric analysis,

validity studies
Expert psychometric expertise
Similarity to Florida Standards Assessments
Page 2-11
Detailed Descriptions of Similar Projects

to Florida Standards Assessments
1. Pilot Item/Task/Stimulus Research and
Development
2. Pilot/Field Test Administration and Open
Source Delivery System
3. Pilot Test Administration Support
4. Field Test Item Development and Scoring
Customer:
Contract Numbers and Value:
1. 00005043.0
$5,579,243
2. 20130180
$14,739,288
3. 20130205
$2,711,500
4. Pending
Pending
Customer Address:
c/o Old Capitol Building, P.O. Box 47200
Olympia, Washington 98504-7200
Contact:
Tony Alpert, Chief Operating Officer
Phone: 503-622-9070
E-mail: Tony.Alpert@smarterbalanced.org
Start and End Date of Projects:
1. 03/2012
2. 08/2012
3. 10/2012
4. Pending
07/2013
09/2014
06/2013
Pending
Similarity in Size and Complexity to Florida

Successfully delivered the Smarter
Balanced small-scale trial fall 2012 test
administration and the spring 2013 pilot
test to more than 577,000 students across
21 states using AIRs online Test Delivery
System.
Assisted with the development of more
than 5,000 items for the Smarter Balanced
pilot test.
Page 2-12
Conducted the rubric validation of all

machined-scored items on the pilot test.
Delivered more than 5,000 mathematics
and English language arts items, including
Smarter Balanceds technology-enhanced
items, for the small-scale trials and the
spring pilot test.
Provided Help Desk services and support
in 21 states participating in both
assessments.
Developed and delivered open-source
scoring engines for machine-scored
constructed-response items.
Project Description
Through two prime contracts and two subcontracts, AIR is supporting the Smarter Balanced
Assessment Consortium. The first subcontract,
requiring pilot item, task, and stimulus
research and development resulted in AIRs
successful delivery of the grade-level pilot
assessments for more than 550,000 students.
AIR conducted four teacher focus groups to
obtain feedback and insights about the different item types and innovations that would be
incorporated into future Consortium assessments. In addition to supporting research
focus groups, AIR developed approximately
150 mathematics items, 350 ELA items, and
20 research questions that were administered
to students across the Consortium states in
cognitive laboratories. AIR trained partner
vendors to also administer laboratories.
Additionally, AIR supported small-scale trials,
the first administration of items to students
across the 21 Consortium states. More than
11,500 students participated in the 2 1/2 week
test administration. The research conducted
from both the cognitive labs and small-scale
trials helped to detect general effects, while
also providing an opportunity for test developers to gather in-depth evidence regarding new
approaches to measurement. Using the evi-
dence taken from the small-scale trials, AIR

developed over 1,600 mathematics and ELA
items for the spring 2013 pilot test.
For the spring 2014 field test, AIR is developing the following new tools:
Masking
AIR also supported scoring engine research by

collecting released items and responses from
several Consortium states to conduct training
and provide an initial analysis of the AIROSE
Essay Scoring Engine. AIR built scoring models
and provided a data analysis of scoring information collected throughout the administration of the fall 2012 small-scale trials. In addition, AIR developed, trained, and validated two
open-source, automated scoring engine prototypes, an essay scoring engine, and an equation scoring engine. The coding is available in
the open-source community for feedback and
potential improvement.
The subsequent contracts supported and will
support the spring 2013 pilot test and the
spring 2014 field test administration in addition to the pilot and field test open source test
delivery system. Through both contracts, AIR
oversaw the recruitment of more than one million students to participate in the pilot test.
Also, AIR developed the test administration
training materials for the pilot test including
user manuals, online training modules, supporting system documentation, and a series of
live webinars for the test administrators and
school coordinators participating in the pilot
test. For the spring 2013 pilot test, AIR developed the following tools for the Test Delivery
System:
Writing tools (spell check, bold, indent,
bullets, etc.)
Expandable passages
Global notes
Embedded ruler
Closed captioning
Line reader
AIR administered the spring 2013 pilot test
assessments with the following online accommodations:
Text-to-Speech
Braille
Spanish translation
Spanish and English glossary
AIR also implemented a fully supported Tier 1
and Tier 2 Help Desk for the pilot test.
Developing the Consortiums open-source Test
Delivery System, AIR integrated all of the
Consortiums systems including the item bank,
reporting system, and the Test Delivery
System.
The last subcontract supports field test item
development and scoring. Under this contract,
AIR will develop a percentage of the 22,000
items for the spring 2014 field test.
Additionally, AIR will be responsible for the
successful import and rendering of multiple
item types into AIRs Test Delivery System
prior to the field test administration.
Already, AIR performed machine scoring, used
on the pilot test, through a process referred to
as rubric validation. Machine scoring, or rubric
validation, was implemented on approximately
904 ELA items (natural language, grid, and hot
text item types) and 1,760 mathematics items
(natural language, equation, and graphic
response item types).
Embedded protractor
Highlight
Strikethrough
Page 2-13
Oregon Assessment of Knowledge and

Skills Online (OAKS)
Customer:
Oregon Department of Education
Contract Number and Value:
ODE1007-07
$30,860,249
Customer Address:
255 Capitol Street NE
Salem, Oregon 97310
Developed the full set of technical reports

to support Oregons successful NCLB peer
review approval for the first statewide,
online adaptive testing system.
Delivered, hand-scored, and reported
online English Language Proficiency
Assessments (ELPA). As with OTELA,
Oregons online ELPA was based on the
CCSSO-developed English Language
Development Assessment (ELDA) project,
for which AIR served as the development
contractor.
Contact:
Kathleen Vanderwall, Ph.D., Manager
Phone: 503-947-5721
E-mail: kathleen.vanderwall@state.or.us
Start and End Date of Project:
03/2007
09/2014

Successfully delivered more than 2 million
tests per year to more than 400,000 students for Oregons online statewide assessments in ELA (grades K12), mathematics
(grades 38 and high school), reading/literature (grades 38 and high school), science
and social sciences (grades 5, 8, and high
school), and writing (grades 11 and 12).
Provided AIRs Item Banking System for
Oregons teachers to develop and store
items.
Provided AIRs Online Scoring System
for Oregons teachers to score writing
assessments.
Delivered assessments online for 100% of
Oregons students through an extensive
suite of accommodations and tools (e.g.,
Braille, large-print, print-on-demand,
text-to-speech).
Page 2-14
Project Description
AIR administers Oregons Assessment of
Knowledge and Skills Online, or OAKS Online,
the first NCLB-approved online adaptive state
assessment. AIRs Psychometrics and Statistics
team worked to help Oregon develop technical
reports by designing and implementing studies
that were submitted for the states ultimately
successful NCLB peer review. Within a
9-month testing window that takes place each
year, AIR delivers approximately 2 million
adaptive tests to more than 335,000 Oregon
students in grades 312 in mathematics, reading, science, and social studies.
Oregon uses AIRs Item Tracking System (ITS)
independently to support its item development
process for OAKS online. AIRs ITS is completely integrated without the Test Delivery
System and can be used by state officials and
educators with minimal support. Since Oregon
educators write all OAKS items, OAKS Online
is the only AIR assessment project that does
not rely on an item development contractor
for summative assessments.
Field-testing of items is fully integrated with
the online testing program, allowing AIR to
dramatically increase the item pool in a short
period of time. AIR has supported the following in ITS:
Since 2007 and through spring 2013, AIR

field-tested approximately 15,000 items
with a rapid turnaround time; some fieldtest items were analyzed and inserted into
the operational item pool within a month of
being field-tested.
tionality to improve the system. AIR implemented several related upgrades.
AIR actively supported the DelawareHawaii-Oregon-Utah item-sharing

consortium.
Provided an online training system for

districts and schools to use with test
administrators.
AIR has also assisted the Oregon

Department of Education (ODE) with optimizing items and item graphics for web
display.
Provided online practice tests to prepare

students and to familiarize stakeholders
with the test.
Immediate availability of student data from the

online system boosts the usefulness of online
testing compared to paper-and-pencil testing,
in which teachers must wait several weeks to
see how students performed. For example,
Once students finish testing, teachers have
almost immediate access to data at student,
classroom, school, district, and state levels.
OAKS Online gives educators immediate
access to assessment results describing students specific strengths and weaknesses,
which allows educators to easily identify
student and curricular needs and target
instruction to address those needs.
To relay on-the-spot results to ODE, the
OAKS system imports updates to student
data daily and exports student test results
nightly to ODE.
To support Oregons educators and students,
AIR provided several services for OAKS
Online:
Trained regional support staff and provided

user support via our Help Desk during
administration throughout the year (Help
Desk hours: 7 a.m. to 6 p.m. PST).
Currently implementing new operational

item types (which were field-tested in
20092012) to increase the range of cognitive complexity captured by the test,
adding new features to increase accessibility, and adding a writing test.
Since 2009, AIR has been providing support
for Oregons Online Writing and Online
English Language Proficiency Assessments
(ELPA). For writing tests, AIR imported items
and deployed tests in the OAKS system in
addition to providing AIRs online distributed
performance scoring system. AIR supported
multiple in-state scoring sessions with groups
of Oregon educators who trained, qualified,
and scored the writing responses. For ELPA,
AIR imported 2,000 items and developed more
than 500 items for field-testing. Also, AIR configured and deployed ELPA in the OAKS system along with AIRs performance scoring system. These constructed-response items, or
writing responses, are scored in AIRs
Processing and Scoring Center.
Deployed a system for local administrators

to manage system permissions and access
for nearly 20,000 test administrators.
Interacted with the field regularly to collect
feedback on ease of use and identified func-
Page 2-15
Assessment System
Delivered interim assessments in all subjects.

DOE2010-02
$25,413,531
Prepared reports for Delawares Technical

Advisory Committee and convened all
required Content, Bias and Sensitivity,
Rangefinding, Rubric Validation, Data
Review, PLD, and Standard Setting
meetings.
Customer Address:
401 Federal Street
Dover, Delaware 19901-3693
Delivered comprehensive Technical Reports

annually including all required validity
studies to support Peer Review approval.
Customer:
Delaware Department of Education
Contact:
Project Description
Brian M. Touchette, Director

Phone: 302-735-4090
E-mail: btouchette@doe.k12.de.us
12/2009
06/2014

Developed and successfully delivered since
2009 Delawares first online statewide
assessments, totaling approximately
500,000 tests to 100,000 students in ELA
and mathematics (grades 210, plus high
school end-of-course assessments), science
(grades 5, 8, and 10, plus end-of-course
assessment in Biology), and social studies
(grades 5 and 7, plus end-of-course
assessment in American History).
Realigned Delawares item banks to new
standards and developed more than 4,000
items including more than 1,000
simulations and machine-scored
The Delaware Comprehensive Assessment

System (DCAS) project includes summative
assessments linked to fully integrated benchmark and formative assessments and expansive learning supports. In addition to developing more than 3,000 items for grade-level
assessments for 100,000 students in grades
210 in reading, mathematics, science, and
social studies, AIR developed more than 2,000
items in one year to help Delaware field-test
and create seven end-of-course (EOC) assessments. The assessments in Algebra I,
Integrated Math I, English II, U.S. History, and
Biology were field-tested during the 20102011
school year and became operational in spring
2012. Two additional assessments (Algebra II
and Integrated Math III) completed field-testing and became operational in January 2013.
This project, awarded in December 2009,
required an incredibly fast start and entailed a
complete transition from paper-and-pencil
testing to online testing. The Delaware
Department of Education (DDOE) required a
full statewide field test of the entire item bank
Delivered assessments online for nearly

100% of Delaware students through an
extensive suite of accommodations and
tools (e.g., large-print, print-on-demand,
text-to-speech).
Page 2-16
with all students in spring 2010. During the

first three months, AIR
developed test specifications and blueprints;
imported nearly 12,000 English and
Spanish existing items and aligned them to
new Delaware standards;
developed 2,000+ new items aligned to
Delaware standards including hundreds of
technology-enhanced, machine-scorable,
and natural language items;
processed all items through appropriate
DDOE and committee reviews;
incorporated PISA items to allow the development of international benchmarks;
developed written and web-based communications materials for all stakeholders;
implemented roster data transfers with
Delawares student information systems;
configured and launched our secure browser and Test Delivery System (TDS) to
DDOE requirements;
provided training to 8,644 Test
Administrators; and
delivered the spring 2010 field test beginning on April 1, 2010.
The field test was a success. All schools were
able to test using AIRs TDS. When field-testing
was complete, AIR conducted data review and
convened standard settings in 20 grades/subjects to establish and report score scales in all
subjects. The standard settings were informed
by vertical scales to help articulate performance standards during a bookmark standard
setting activity, resulting in standards that
were reasonable and orderly across all grades.
We also linked DCAS performance to international benchmarks. All Delaware schools are
now testing online. DDOE has chosen to have

multiple, shorter testing windows in which
specific assessments (e.g., summative, blended,
benchmark) are available. AIR accommodated
this revised assessment schedule with a simple
configuration of our TDS. With the operational
launch, we deployed AIRs reporting module,
which provides a single place for all reports for
the DCAS, the end-of-course exams, and
benchmark growth assessments. Similar diagnostic information is available whether a student takes a summative or a benchmark
growth assessment. Educators can clearly see
which tests a student has taken and when they
were completed and can access instructional
diagnostics at a level appropriate to support
instructional decision-making. AIR completed
the development of multiple paper and Braille
forms in each grade/subject as accommodations.
AIR is actively supporting the DelawareHawaii-Oregon-Utah item-sharing consortium.
AIR selected approximately 4080 items per
grade/subject for embedded field-testing; the
focus is on technology-enhanced items. AIR
delivers blended summative/benchmark assessments in Delaware. Extensive learning supports and resources complete the system, providing educators with targeted instructional
material for student use, giving students interactive practice, and finishing with scorable
activities to help teachers gauge the success of
their lessons.
Learning Point Navigator (LPN) integrates with
AIRs Online Reporting System to provide
resources targeted to identify areas of weakness or to extend standards-aligned topics for
students who are excelling. The system provides immediate access to standards-aligned
professional development, lesson plans, and
student activities. Both educators and students
Page 2-17
(and their parents) can access the system to

obtain or use learning resources. Teachers can
access this system independently, or they can
link directly to appropriate resources from the
online testing system. One of the most powerful uses of Navigator is to extend the diagnostic
reach of the assessments. Rather than take
instructional time for more testing, teachers
can use Navigator to distill information from
instructional activities and assignments. LPNs
content libraries support multiple, diverse
libraries of resources. Each resource can be
aligned to each set of content standards represented in the database of content standard
maps. The content standard maps represent
potentially diverse systems of content standards. They represent the content hierarchy
for each set of standards incorporated. In the
20112012 school year, over 3,500 teachers
accessed LPN in Delaware. Over 23,000 stu-
Page 2-18
dents worked on teacher-created assignments.

In total, over 116,000 assignments were completed by Delaware students. Linked directly
to a group of students strengths and weaknesses, easy to access and use, LPN has made
an immediate impact in Delaware as shown by
its usage by teachers and students in only its
second year of implementation. AIR continues
to add instructional resources to the LPN
bank. It has doubled in size since it opened in
the 20102011 school year.
1. Hawaii State Assessments (HSA)

2. Hawaii End-of-Course Prime (EOC)
Customer:
Hawaii Department of Education
1. CO-69111
$73,778,252
2. CO-10361
$4,413,289
Customer Address:
1390 Miller Street
Honolulu, Hawaii 96813
Contact:
Cara Tanimura, Director
Phone: 808-586-3283
E-mail: cara_tanimura@notes.k12.hi.us
1. 07/2006
06/2014
2. 08/2012
06/2014

Developed and delivered since 2009 more
than 500,000 online statewide assessments
for 120,000 students annually in ELA and
mathematics (grades 38 and high school),
science (grades 4 and 8), and end-ofcourse (EOC) assessments in Biology and
Physical Sciences.
Supported Hawaiis paper-and-pencil
testing program from 20062009 and the
successful transition to online testing.
Realigned Hawaiis item banks to new
standards and developed more than 4,000
new items including more than 1,000
simulations and machine-scored
Delivered assessments online for 100% of
Hawaii students, through an extensive
suite of accommodations and tools (e.g.,
Braille, large-print, print-on-demand,
text-to-speech).
Convened since 2006 Hawaiis Technical

Advisory Committee and all required
Content, Bias and Sensitivity, Rangefinding,
Rubric Validation, Data Review, PLD, and
Standard Setting meetings.
Delivered comprehensive Technical Reports
annually including all required validity
studies to support Peer Review approval.
Project Description
AIR developed and delivered the paper-andpencil Hawaii State Assessment (HSA) from
2006 to 2010. Currently AIR tests 120,000 students online each year in reading, mathematics, and science (grades 38, 10, and 11). In
addition, AIR supports Hawaiis end-of-course
(EOC) assessments in Algebra, Biology and
Writing.
The Hawaii State Assessment offers up to three
testing opportunities on criterion-referenced
tests, measuring the states standards, and
includes selected-response and machinescored constructed-response items. As the
incumbent paper-and-pencil contractor, AIR
prepared the HSA item bank for online testing,
following which AIR
developed online test specifications and
blueprints;
developed 1,400+ new items including hundreds of technology-enhanced, machinescorable, and natural language items;
processed new items through appropriate
HIDOE and committee reviews;
developed written and web-based communications materials for all stakeholders;
implemented roster data transfers with
Hawaiis student information systems;
configured and launched our secure browser and TDS to HIDOE requirements;
Page 2-19
provided training to over 5,500 test administrators; and

delivered the spring 2010 field test.
The HSA field test was a huge success. When
field-testing was complete, AIR conducted data
review and convened standard settings to
establish and report score scales in all subjects. Currently, all Hawaii schools test online.
Hawaii limits field-testing to the spring, and
AIR developed more than 1,000 items, many
of which are technology-enhanced, for embedded field-testing in 2011, 2012, and 2013. AIR
is also supporting the Delaware-HawaiiOregon-Utah item-sharing consortium and
assisting Hawaii in selecting items from the
combined bank for future field-testing. In total,
AIR developed 4,000+ items in reading, writing
and mathematics through review and acceptance by content and fairness committees.
With its operational launch, AIR successfully
deployed a reporting module, which provides a
single place for all reports. For paper-and-pencil accommodations, AIR deployed an adaptive-print function and adaptive Braille function, providing students with the same adaptive testing model and multiple opportunities
as students taking Hawaiis general forms.
AIR provided extensive support for accurate

benchmarks to assess and evaluate student
progress in three major areas:
Improvement of curriculum and instruction
and ensuring consistency throughout the
state. The exams help classroom teachers
focus on the most important concepts and
skills and identify areas where the curriculum needs to be strengthened.
Development and delivery of content that
adequately measures skills students need to
enter and succeed in first-year, credit-bearing courses. Post-secondary institutions will
be able to use the results of the exams to
tell high school students whether they are
ready for college-level work, or if they have
content and skill gaps that need to be filled
before they enroll in college.
Development and delivery of reports that
assist state education leaders, educators,
and the public assess progress to identify
areas for improvement and evaluate the
impact of state strategies for improving
overall achievement.
Similar to the HSA, AIR developed and delivered Hawaiis EOC assessments with machinescorable items that display student scores on
the computer monitor immediately after students complete the exam. AIR developed more
than 4,000 multiple-choice and constructedresponse items that measure student proficiency of content standards and course benchmarks. Students take each exam during the
last three weeks of the course.
Page 2-20
Utah Student Assessment of Growth

and Excellence (SAGE)
Customer:
Utah State Office of Education
136199
$39,303,646
Customer Address:
250 East 500 South, P.O. Box 144200
Salt Lake City, Utah 84114-4200
Contact:
John Jesse, Director
Phone: 801-538-7811
E-mail: John.Jesse@schools.utah.gov
02/2013
01/2018

Developing Utahs first online statewide
assessments for 500,000 students,
incorporating interim, formative, and
summative assessments in ELA and
mathematics (grades 312) and science
(grades 48).
Developing end-of-course assessments in
Biology, Chemistry, Physics, and Earth
Science.
Delivered more than 10,000 items
including simulations and machine-scored
Supporting extensive suite of
accommodations (e.g. Braille,
text-to-speech, large-print).
Deploying online reporting system for
machine-scorable items.
Growth and Excellence (SAGE). Currently,

Utah administers the states paper-and-pencil
criterion-referenced tests (CRT), but will be
transitioning to the states SAGE, an online
adaptive assessment. Including an interim,
formative, and summative assessment in addition to expansive learning supports, AIR will
provide support for approximately 45,000 students per grade in ELA (grades 311), mathematics and secondary math I, II, and III
(grades 38), and science assessments in
Biology, Chemistry, Earth Science, and Physics
(grades 48). AIR has delivered many of the
5,000 new items, including 3,600 ELA items
that will replace the entire ELA item bank.
AIR has conducted content and bias committee reviews, and is planning rangefinding,
rubric validation, data review, and standard
setting meetings immediately after the Spring
2014 Operational Field Test. AIRs formative
assessment tool will include items aligned to
the Utah Core Standards in mathematics, science and language arts from existing Utah item
pools and from item sharing with other states.
The formative item pool will include the same
item types present in the summative item pool
(e.g., technology-enhanced, machine-scored
constructed-response). Student result data
from the summative system will flow through
to the formative system and direct educators
to content resources.
Beginning in early 2014, the new formative
assessment system will add functionality to
support the administration of teacher and district-created items and tests. Beginning in
summer 2013 and continuing throughout the
contract, AIR and the USOE are providing inperson and webinar trainings for staff. Training
materials, system tutorials, and other needed
materials will be provided. A fully staffed Help
Desk will be available to support teacher,
school, and district use.
Project Description
AIR is developing and delivering Utahs first
online assessments, the Student Assessment of
Page 2-21
1. Ohio Achievement Assessments (OAA)

2. Ohio Test of English Language
Acquisition (OTELA)
3. Ohio Alternate Assessment for
Students with Significant Cognitive
Disabilities (AASCD)
Customer:
Contract Numbers and Value:
1. CSPOA02021
2. CSP0A03031
3. CSP906007
$179,705,163
$76,105,403
$312,062,163
Customer Address:
Columbus, Ohio 43215-4183
Contact:
Jim Wright, Director
Phone: 614-466-0223
E-mail: James.Wright@education.ohio.gov
03/2002
06/2014

Successfully deliver 14 achievement tests
in reading, mathematics, and science for
use in Ohios 38 classrooms that require
the scoring of 1.75 million tests for more
than 800,000 students.
Developed and successfully delivered
Ohios grades 38 assessments from initial
design and specifications through standard
setting and all operational administrations.
The administration of the OAA currently
includes grades 38 reading and
mathematics and grades 5 and 8 science;
administrations from 20062009 included
grades 5 and 8 social studies and grades 4
and 7 writing.
Page 2-22
Conducted two successful online pilots

(2012 and 2013) to support Ohios
transition to online testing using social
studies content, exceeding Ohios
participation and performance
expectations.
Developed more than 8,000 items and
conducted both independent field tests and
embedded field tests across more than
1,100 field-test forms.
Realigned all of the existing items from the
OAA bank to Ohios New Learning
Standards.
Collaborated with the Ohio Department of
Education (ODE) on the development of
test specifications, item specifications, and
a style guide for the paper-and-pencil
assessments. Developed test specifications,
item specifications and a style guide for
more than 1,000 new elementary- and middle-school level items, including simulations and other machined-scored constructed-response items.
Built and produced operational fixed-form
tests for annual spring and fall administrations of the OAA, OTELA, and Alternate
Assessment, including up to 150 OAA
forms each spring with embedded field-test
sets. Also, AIR produced all accompanying
ancillaries and special versions for each
administration.
Worked with the English Language
Development Assessment (ELDA)
Consortium to develop its assessment of
English language learners (ELLs).
Subsequent to that development, AIR
worked with ODE to develop OTELA forms
from the ELDA bank and write and field
test OTELA items for the state of Ohio to
supplement the test bank.
Have successfully delivered the annual
OTELA administration since 2007.
Convened each of the following Ohio committees: Technical Advisory Committee,

Test Steering Committee, English Language
Learners Advisor Committee, and the
Alternate Assessment Advisory Committee.
Convened all required Content Advisory,
Fairness and Sensitivity, Rangefinding, Data
Review, Performance-Level Descriptor
(PLD), and Standard Setting committees, as
the program dictates. Since 2002, AIR has
convened more than 800 meetings for the
state of Ohio.
Developed and delivered to Ohio families
and educators more than five million pages
of customized, color score reports every
spring that incorporate variable information based on research and assessment
results.
Developed and delivered innovative professional development efforts (e.g., item writing workshops, train-the-trainer initiatives)
and extensive suite of online tools (Success
Suite).
Delivered comprehensive technical reports
annually, including all required validity
studies to support peer review approval.
Project Description
AIR has managed statewide assessments for
Ohio since work began in 2002 for the spring
2003 administration of the grade 3 reading
field test. AIR develops and administers the
Ohio K8 Assessment System, a comprehensive battery that currently consists of 14
achievement tests in reading, mathematics,
and science for use in Ohios K8 classrooms.
In spring 2013, AIR oversaw the scoring of
1.75 million tests for more than 800,000 students. For students with disabilities, Ohios
alternate assessments measured the same content areas. English language learners are
assessed under the OTELA, which is also managed by AIR. Since its inception, the program
has been supported by extensive professional
development, including workshops, online

tools, and electronic reporting.
During the past decade, AIR developed items
and conducted both independent field tests
and embedded field tests for Ohio with more
than 8,000 items across more than 1,100
embedded and independent field-test forms.
We managed every aspect of test delivery,
scoring and reporting, including application of
millions of hand-scores to constructedresponse items. AIR also worked with Ohio to
develop a series of accommodated and special
version forms to support students. These
include bilingual test booklets, English audio
and foreign-language CDs, a network of support for translations in many languages,
Braille, and large-print.
Each year, AIR develops personalized assessment score reports for more than 850,000
Ohio families and educators. These enhanced,
full-color paper score reports offer actionable
data for families, teachers, principals, and
superintendents. Web-based tools empower
educational leaders to explore their assessment data and conduct their own analyses.
AIR has improved awareness about the Ohio
assessment program by making the test procedures and score results easier for educators to
use and understand.
AIRs Test Information Distribution Engine
(TIDE) helps keep district contact information current, allows districts to upload student pre-ID files, assists administrators in
ordering test materials, and tracks shipments of test materials as efficiently as
possible.
Online and paper practice tests enable
teachers to familiarize students with the
test format, item types, and testing process
before they take the operational test, allowing administrators and students to be prepared before testing day.
Page 2-23
Interpretive guides, delivered with the

score reports, provide information about
interpreting the data on score reports and
make the score reports more accessible to
parents and educators.
Statewide professional development initiatives on both test administration and
score interpretation arm educators with
the information they need to take full
advantage of Ohios strengthened testing
program.
AIR developed and supports OTELA. AIR was
the lead contractor in the development of the
original ELDA in partnership with the Council
of Chief State School Officers (CCSSO), Ohio,
and other participating states in 20052007.
Under subsequent contracts, AIR worked with
Ohio to create the OTELA from the ELDA and
has delivered OTELA flawlessly since 2007.
This work involves approximately 42,000 students in grades K12 across five grade bands.
AIR is responsible for all aspects of this paperand-pencil program, including developing item
and forms, producing special versions (Braille
and large-print), applying more than 50,000
hand-scores, and coordinating all associated
print, pack, ship, receive, process, scan, and
reporting activities.
During the past two years, AIR supported
Ohios transition from the traditional portfolio
assessment for students with disabilities to
implement the Alternate Assessment for
Students with Significant Cognitive Disabilities
(AASCD). The model allows the state to implement an alternate assessment that is as psychometrically sound as the assessments for the
general population and also measures growth
in individual students. The AASCD was also
the first of Ohios operational tests to include
an online component. Test administrators
were required to enter their students scores
into an online Data Entry Interface. AIR
worked with ODE to employ a train-the-trainer
model that reached nearly 10,000 test adminPage 2-24
istrators, and the test was successfully administered to more than 16,000 students. Ohios
transition was completed this past spring
(2013) following a standard setting workshop
and the distribution of score report data and
nearly 100,000 pages of paper reports based
on the new alternate assessment model.
AIR also developed Ohios original Diagnostics
Assessments in 2003 and the revamped the
entire series in 2013.
This recent initiative required the development of new items, committee reviews, field
testing, and operational form building in a little over 12 months time to ensure Ohios
Diagnostic Assessments were aligned to the
Common Core State Standards.
In spring 2012 and spring 2013, AIR worked
with ODE to offer an online pilot opportunity
across the state, assessing grade 8 and grade 4
social studies content, respectively. These
pilots were in support of Ohios commitment
to next-generation testing and provided the
state and its districts the chance to assess
their preparedness for online assessments,
interact with new and innovative item types,
and practice implementing online testing in a
low-stakes environment.
In addition to supporting Ohios statewide
assessment for grades 38, online testing
pilots, English language acquisition assessment, diagnostic assessments, and alternate
assessment, AIR and Ohio have partnered on
two research studies on modified assessments
and the two percent student population.
These studies were completed as a partnership
between AIR, Ohio, Minnesota, and Oregon
under a General Supervision Enhancement
and an Enhanced Assessment Grant, as well as
a partnership between AIR and Ohio under an
Institute of Education Sciences Grant.
Ohio Graduation Tests (OGT)

Customer:
CSP901307
$258,000,935
Customer Address:
Columbus, Ohio 43215-4183
Contact:
Jim Wright, Director
Phone: 614-466-0223
E-mail: James.Wright@education.ohio.gov
12/2006
06/2014

Developed and successfully delivered the
high-stakes graduation assessments to
more than 250,000 Ohio high school
students per year, with 19 administrations
since 2007.
Realigned all of the existing ELA and
mathematics items from the OGT bank to
Ohios New Learning Standards.
Developed test specifications, item
specifications and a style guide for 17
different assessments, including
simulations and other machined-scored
Convened each of the following Ohio
committees: Technical Advisory
Committee, Test Steering Committee,
English Language Learners Advisor
Committee, and the Alternate Assessment
Advisory Committee. Convened all
required Content Advisory, Fairness and
Sensitivity, Rangefinding, Data Review,
Performance-Level Descriptor (PLD), and

Standard Setting committees, as the program dictates. Since 2002, AIR has convened more than 800 meetings for the state
of Ohio.
Developed and annually deliver to Ohio
families and educators more than one million pages of customized, color score
reports incorporating variable information
based on research and assessment results.
Developed and delivered innovative professional development efforts (e.g., item writing workshops, train-the-trainer initiatives)
and extensive suite of online tools (Success
Suite).
Delivered comprehensive technical reports
annually, including all required validity
studies to support peer review approval.
Project Description
AIR has managed the Ohio Graduation Test
(OGT) program since 2006 when work began
for the summer 2007 administration. At the
end of grade 10, the tests measure student
competencies in five subjects: reading, mathematics, writing, science, and social studies.
Students must pass all five parts to receive a
high school diploma. Students who do not pass
one or more parts of the test at the end of
grade 10 have three opportunities a year to
retake the tests. More than 180,000 students
take the test each spring. In summer and fall,
the testing populations are approximately
15,000 and 60,000, respectively.
At the start of the contract, AIR worked with
ODE to phase out Ohios prior graduation test,
the NPT, with four administrations from
20072008. This test was reserved for students
whose graduation requirements had been
established prior to the implementation of the
OGT for sophomores.
Page 2-25
AIR successfully transitioned the item bank

from the previous contractor and front-loaded
a significant amount of item development to
augment the existing item bank. AIR developed approximately 8,000 items across five
subjects during the first two years. In
20082012, we field tested these new items on
more than 400 embedded field-test forms
(reading, mathematics, science, and social
studies) and 14 independent field-test forms
(writing). Given the high stakes associated
with the OGT, AIR works with Ohio to provide
extensive support for educators and students.
The most frequently used feature is AIRs
online formative testing system, which has
been used by more than 60,000 Ohio high
school students. Using a large bank of released
items, teachers can create customized formative assessments for students. The items
become available for immediate use on these
tests when they are deployed in our item
release site. The following are examples of
capabilities offered by this customization feature:
Students and teachers can access complete
formative tests on their own.
Teachers can design short tests, drawing on
one or more content standards, or they can
select items by other characteristics such
as item type or benchmark.
Teachers can assemble these tests in their
own account and assign them to individual
students or to rosters of students working
on similar material. When a student logs on
to his or her secure MySuccess account,
the assignment appears and the student is
able to take the online test.
Page 2-26
Teachers can access the data that accumulate over time. The results are reported in
the context of grade-level expectations
because the available items come from
operational tests.
The student can immediately see a score
based on multiple-choice items. The score
is updated when the teacher scores constructed-response items.
AIR has supported 18 administrations of the
OGT. To date, AIR has
implemented its Test Information
Distribution Engine (TIDE) to keep district
contact information current, allowed districts to upload student pre-ID files, assisted administrators in ordering test materials, and tracked shipments of test materials
as efficiently as possible;
developed, administered, and overseen the
scoring of and reporting of more than
1.2 million subject tests for more than
250,000 students annually;
developed special versionsEnglish audio
CDs, foreign language CDs, Braille and
large-print test booklets, and oral translation scripts;
applied more than seven million constructed-response scores per year;
conducted and analyzed the results of
embedded and independent field tests;
developed operational and breach test
forms for all subjects annually;
created annual technical reports and provided support and documentation to all
Technical Advisory, Test Steering, English
as a Second Language (ESL), Students with
Disabilities (SWD), Content, and Fairness
committees; and
implemented customized assessment score
reports for families and educators including
enhanced, full-color paper score reports
and full electronic reporting, offering
actionable data for families, teachers, principals, and superintendents.
During the past two years, AIR supported
Ohios transition from the traditional portfolio
assessment for students with disabilities to
implement the Alternate Assessment for
Students with Significant Cognitive Disabilities
(AASCD). The model allows the state to implement an alternate assessment that is as psychometrically sound as the assessments for the
general population and also measures growth

in individual students. The AASCD was also
the first of Ohios operational tests to include
an online component. Test administrators
were required to enter their students scores
into an online Data Entry Interface. AIR
worked with ODE to employ a train-the-trainer
model that reached nearly 10,000 test administrators, and the test was successfully administered to more than 16,000 students. Ohios
transition was completed this past spring
(2013) following a standard setting workshop
and the distribution of score report data and
nearly 100,000 pages of paper reports based
on the new alternate assessment model.
Page 2-27
Minnesota Assessment System

Customer:
Minnesota Department of Education
B55607
$61,941,034
Customer Address:
1500 Highway 36 West
Roseville, Minnesota 55113-4266
Contact:
Linda Sams, Manager, Statewide Testing
Phone: 651-582-8431
E-mail: linda.sams@state.mn.us
04/2011
10/2014

Developed and successfully delivered
grade-level and high-stakes high school
graduation tests to more than 400,000
students in a dual-mode paper/pencil and
online testing program.
Developed more than 3,000 items in ELA,
mathematics, and science (grades 5, 8, and
high school). Science items include
extensive use of simulation and
machine-scored constructed response
items.
Prepared reports for Minnesotas Technical
Advisory Committee and convened all
required Content, Bias and Sensitivity,
Rangefinding, Rubric Validation, Data
Review, PLD, and Standard Setting
meetings.
Delivered comprehensive Technical
Reports annually.
Page 2-28
Project Description
AIR completed the second year of supporting
the online and paper-and-pencil Minnesota
Assessment System for approximately 500,000
students. AIR supported testing for the
Minnesota Comprehensive Assessments
(MCAs) in reading, mathematics, and science
(grades 38, 10, and 11); Graduation-Required
Assessments for Diploma (GRAD) in writing,
reading, and mathematics; and alternate
assessments (MCA-Modified and Minnesota
Test of Academic Skills [MTAS]).
Upon award of this contract, AIR imported
approximately 20,000 items (in various formats) from the previous contractor. For MCA
Reading, AIR delivered both the paper-andpencil census reading administration and the
online independent field test. For the last year
of these paper-and-pencil assessments, AIR
created and administered multiple linked and
equated forms to approximately 60,000 students at each grade level. For the online independent field-test, AIR deployed its field-test
item-selection algorithm.
The algorithm employed by AIRs field-test
engine ensures efficient samples for all items
by treating districts and schools as strata and
randomly selecting from among the leastadministered passages/items within each stratum; averaging out position effects by randomly administering test items across test positions; and enabling more robust linkages
among items, since each item is linked with
every other item across hundreds of unique
test forms to a degree not manageable through
a set of fixed forms. AIR also administered the
online MCA Math (offering up to three
attempts for each student) and MCA Science.
Both of these online assessments incorporate
AIRs embedded field-testing engine for new
items.
AIR created and administered multiple forms

of the online MCA-Modified in reading and
math and the paper-and-pencil MTAS. In addition to the initial administrations of the GRAD
assessment for 60,000 students in grade 10,
AIR offers monthly online retests for the
Reading GRAD and the Mathematics GRAD.
AIR also administers the Written Composition
GRAD, which requires students to write an
essay on paper in response to a prompt. The
Written Composition GRAD is first given to

students in grade 9; students who have not yet
taken or passed the Written Composition
GRAD may take the retest during the
November, April (seniors only), and July
administrations. AIR and MDE support this
testing program with extensive online training
for test administrators, incorporating animations and other interactive features.
Page 2-29
DRCs Relevant Project Summaries

ALABAMA
Alabama State Department of Education
Division of Student Instructional Services
Office of Student Assessment
50 North Ripley Street, Room 3306
Montgomery, AL 36104
Dr. Gloria Turner
Director, Student Assessment
334-242-8038 (phone)
334-242-7341 (fax)
gturner@alsde.edu
Ms. Rebecca Mims (AHSGE)
Education Specialist
334-242-8038 (phone)
334-242-7341 (fax)
rmims@alsde.edu
skills in Reading, Mathematics, English

Language Arts, Social Studies, and Biology.
There are four administrations per year; up to
200,000 students are assessed annually. The
Alabama State Department of Education has
contracted with DRC to provide testing services for this assessment, including project management, item development (since 2004),
materials development, and printing, packaging and shipping, scoring, psychometric services, and reporting.
Alabama Reading and Mathematics Test+
(ARMT+)
Contract Years: 20092014
Ms. Kanetra Germany (ARMT+)

Education Specialist
334-242-8038 (phone)
334-242-7341 (fax)
kgermany@alsde.edu
Subject Areas and Grade Levels: Criterion-referenced assessments comprising multiplechoice and constructed-response items in
grades 38 for Reading and Mathematics (multiple-choice and constructed-response) and
grades 5 and 7 for Science (multiple-choice
only).
Project Description
Alabama High School Graduation Exam

(AHSGE)
20012004
20042007
20072010
20102015
Subject Areas and Grade Levels: Criterion-referenced, multiple-choice exam in Reading,
Mathematics, Language, Social Studies, and
Biology for high school students for grades
912.
Project Description
The Alabama High School Graduation
Examination was designed to ensure that students receiving an Alabama high school diploma have a minimum comprehension of basic
Page 2-30
The Alabama State Department of Education

has contracted with DRC to develop, administer, and report on the Alabama Reading and
Mathematics Test (ARMT+). These criterionreferenced assessments are administered to
approximately 60,000 students per grade each
year. Beginning in 2012, the previously developed Alabama Science assessment was administered under this contract along with the
Reading and Mathematics tests. DRC is
responsible for project management; item and
test form development; publication and printing; packaging, distribution, and collection;
processing and scanning; scoring, including
handscoring; psychometric services; reporting;
and customer service.
Project Description
ALASKA
Alaska Department of Education and
Early Development
Division of Teaching and Learning Support
801 West Tenth Street, Suite 200
Juneau, AK 99801
Mr. Erik McCormick
Director of Assessment, Accountability and
Information Management
907-465-8686 (phone)
907-465-8400 (fax)
erik.mccormick@alaska.gov
Alaska Comprehensive System of Student

Assessment: Standards-Based Assessments
(SBAs) and High School Graduation
Qualifying Examination (HSGQE)
(Science Assessments:
20052009)
20092016
Subject Areas and Grade Levels: Criterion-referenced tests based on Alaska standards in
Reading, Writing, and Mathematics for students in grades 310 and high school; and in
Science for students in grades 4, 8, and 10.
Assessments include a balance of constructedresponse and multiple-choice items.
The Alaska Standards-Based Assessments

(SBAs) measure the extent to which Alaskan
students in grades 310 are attaining statewide
student performance standards. The Alaska
Science Assessments were implemented to
measure student performance in Science and
to meet the requirements of the No Child Left
Behind (NCLB) legislation. The High School
Graduation Qualifying Exam (HSGQE) is a
high-stakes assessment that students must
pass to receive high school diplomas. DRC has
been contracted by the Department to develop
and administer these tests, which are given to
approximately 10,000 students in the spring
and 6,000 students in the fall, per grade,
across 54 districts and 514 schools. DRCs
responsibilities include project management;
item development and test construction; pilot
and field testing; conducting review committee
meetings; materials development and printing;
packaging, distributing, and collecting test
materials; scanning, editing, and scoring
answer documents, including handscoring;
standard setting; psychometric services;
reporting; and data warehousing.
Page 2-31
Project Description
IDAHO
Idaho State Department of Education,
Assessment
650 West State Street
Boise, ID 83720
Idaho Standards Achievement Tests
Dr. Angela Rishell
ISAT Coordinator
208-332-6976 (phone)
208-334-2228 (fax)
arishell@sde.idaho.gov

Subject Areas and Grade Levels: Criterion-referenced, multiple-choice tests in Reading,
Language usage, and Mathematics for students
in grades 310, and in Science for students in
grades 5, 7, and 10.
Page 2-32
The Idaho State Department of Education has

contracted with DRC to develop, administer,
and report on the Idaho Standards
Achievement Tests (ISAT). The assessments
include only multiple-choice items and are primarily administered via computer two times
per year (November/December and April/May).
Approximately 150,000 students (600,000
actual tests completed) take the tests via computer annually, with a small number of tests
administered via paper-and-pencil. DRCs
duties include project management; item
development and test construction; field testing; conducting review committee meetings;
materials development and production; electronic administration and scoring of tests (performed by a subcontractor); printing and distributing a small number of testing materials;
preparation of training materials; customer
support; scanning, editing and scoring; psychometric services; and production and distribution of reports and parent brochures.
LOUISIANA
Louisiana Department of Education
Division of Standards, Assessments,
and Accountability
1201 North Third Street G224
Baton Rouge, LA 70804
Dr. Bill Morrison
Assistant Superintendent, Office of Assessment
225-342-3398
bill.morrison2@la.gov
Louisiana Educational Assessment Program

(LEAP), Graduation Exit Examination
(GEE), Louisiana Alternate Assessments
(LAA 1 and LAA 2), and English Language
Contract Years: 19982001 (LEAP and GEE)
20012005 (LEAP, GEE, and
LAA 1)
20052011 (LEAP, GEE,
LAA 1, LAA 2, and ELDA)
20112015 (LEAP, GEE,
LAA 1, LAA 2, and ELDA)
Subject Areas and Grade Levels: Grades 4 and
8, along with high school. These criterion-referenced tests include multiple-choice and constructed-response items in English Language
Arts (ELA), Mathematics, Science, and Social
Studies.
Project Description
This program consists of standards-based tests
for grades 4 and 8, along with a graduation
exam, with approximately 60,000 students per
grade taking the tests. The program also
includes the Louisiana Alternate Assessments
(LAA) for grades 311 and the English
Language Development Assessment (ELDA) for
K12 students. DRCs responsibilities include
project management; test construction; printing of all materials; writing and editing manu-
als, interpretive guides, and released items

documents; conducting workshops for test
coordinators; packing, distributing, and collecting materials; scanning, editing, and scoring answer documents, including handscoring;
psychometric services; and reporting.
In addition, DRC recently developed items for
use in transitional assessments and transitional assessment guides for Louisiana. These
assessments will help the state transition to
Common Core State Standards (CCSS) assessments and new assessments aligned to the
Next Generation Science Standards, and new
Social Studies Standards and Grade Level
Expectations (GLEs) over the next three years.
Integrated Louisiana Educational
Assessment Program (iLEAP): Augmented
Norm-Referenced Tests and CriterionReferenced Tests
Subject Areas and Grade Levels: Augmented
norm-referenced tests and criterion-referenced
tests for grades 3, 5, 6, and 7 in ELA,
Mathematics, Science, and Social Studies, and
grade 9 in ELA and Mathematics. The assessments include both multiple-choice and constructed-response items.
Project Description
The iLEAP program consists of both augmented norm-referenced tests and criterion-referenced tests, which are composed of multiplechoice items, writing prompts, and
Mathematics constructed-response items.
Approximately 55,000 students per grade take
the tests. DRC and our subcontractor,
Riverside Publishing, are working together to
develop and implement the iLEAP program.
DRCs project activities include project management; item development (of Science and
Social Studies items) and forms construction;
conducting review committee meetings;
Page 2-33
designing and printing scannable documents;

packing, distributing, and collecting test materials; scanning, editing, and scoring answer
documents, including handscoring; psychometric services; and reporting.
Louisiana Item Development for Science
and Social Studies Assessments
Subject Areas and Grade Levels: Grades 38
Science and Social Studies, and Biology
Page 2-34
Project Description
DRC will produce Social Studies, Science, and
Biology assessment guides and test items and
related materials aligned to the new Louisiana
Social Studies Standards and Grade Level
Expectations and to the Next Generation
Science Standards. DRC will develop 2,008
multiple-choice items and 58 extendedresponse item sets, along with item-specific
scoring rubrics. DRCs responsibilities include
project management, including schedule management; development of assessment guides;
item development of multiple-choice and
extended-response test questions with scoring
rubrics; and item content/bias review meeting
facilitation and logistics.
MICHIGAN
Michigan Department of Education
Office of Standards & Assessment
Bureau of Assessment & Accountability
P.O. Box 30008
Lansing, MI 48909
Mr. Andrew J. Middlestead
Test Development Manager
Michigan Department of Education
517-335-0568 (phone)
517-335-1186 (fax)
middlesteada@michigan.gov
Michigan K12 Item Development

Subject Areas and Grade Levels: Grades K12
in Mathematics, Reading, Writing, Science,
and Social Studies; and English language proficiency in Reading, Writing, Listening, and
Speaking.
Project Description
The Michigan Department of Education has
contracted with DRC for the development,
quality assurance, revision, and stakeholder
review of assessment items and contexts for all
subjects and tests included in the State of
Michigans K12 statewide assessments. This
includes criterion-referenced assessments in
grades 39 and 11; alternate assessments;
English language proficiency assessments;
End-of-Course (EOC) assessments; and interim benchmark assessments. Item types
include multiple-choice, short answer, extended-response, selected-response, activity based,
performance task, technology-enhanced, and
computer-based simulation tasks. DRCs
developing training methods and materials;
recruiting, training, and performance evalua-
tion of item/context writers and reviewers;

planning, coordinating, and conducting all
item/context development and review meetings; translation services; and reporting of procedures and results.
As of 2013, DRCs additional scope responsibilities include: development and review of Social
Studies items for Michigans alternate assessment, conducting an online pilot test and subsequent cognitive labs (spring 2014) for
Michigans alternate assessment in Social
Studies (administered to students in grades 5,
8, and 11), and item development training for
Michigan teachers to develop Mathematics and
ELA items for the Smarter Balanced
Assessment Consortium.
Michigan Interim Assessment
Subject Areas and Grade Levels: Grades K2
in Reading and Mathematics; grades 3high
school in Science and Social Studies; high
school Reading and Mathematics.
Project Description
As a part of the item development contract,
the Michigan Department of Education has
contracted with DRC for the development and
pilot testing of assessment items and contexts
for the Michigan Interim Assessment program.
This is a new online testing program; item
types include multiple-choice and technologyenhanced items. DRCs responsibilities include
project management; item/context development and forms construction; review committee meeting planning and facilitation; cognitive
labs; administration of field tests; research/psychometric services; customer service; and
technical support.
Page 2-35
Michigan Administration, Scoring and

Reporting of Statewide Assessments
Subject Area and Grade Levels: Grades 311
ELA, Reading, Writing, Math, Science, and
Social Studies
Page 2-36
Project Description
DRC, AIR, Measurement Incorporated, and
Measured Progress were selected for the
administration, scoring, and reporting of
Michigans statewide assessments (given to 1.5
million students, across 3,500 schools). DRCs
responsibilities include printing, packaging,
and shipping of all test materials for Michigans
fall and spring assessments. Materials include:
test booklets, answer documents, manuals, and
accommodated materials (Braille, enlarged
print, audio, video, and translations).
Project Description
NEBRASKA
Nebraska Department of Education
301 Centennial Mall South
Lincoln, NE 68509
Dr. Valorie Foy
Director of Statewide Assessment
402-471-2818 (phone)
402-471-4311 (fax)
valorie.foy@nebraska.gov
Mr. John Moon
Project Manager
402-471-1685 (phone)
402-471-4311 (fax)
john.moon@nebraska.gov
Nebraska State Accountability (NeSA) and

Check 4 Learning (C4L)
Contract Years: 20092013 (NeSAWriting,
20102013)
20142018
Subject Area and Grade Levels: Grades 38
and high school in Reading, Mathematics,
Science, and Writing.
The Nebraska Department of Education has

contracted with DRC for the development and
administration of the NeSA. Tests consist of
multiple-choice items for Reading,
Mathematics, and Science, as well as Writing
prompts. Items are based on Nebraska academic content standards, with approximately
22,000 students tested per grade in the spring
of each year. Operational tests with embedded
field testing are primarily administered online
via DRCs INSIGHTTM Online Learning
System, with a small number of tests being
administered in a paper-and-pencil format.
Practice tests are delivered in online format
only. In November 2013, DRC began administering the C4L formative assessments. The
C4L system is used by teachers and administrators to deliver on-demand, classroom-based
assessments at the point of instruction in
order to monitor student learning throughout
the year. DRCs responsibilities include project
management; test development services; materials production, printing, distribution, and
collection; computer-based test administration
via our online testing engine, DRC INSIGHT;
training; customer service; processing and
scanning; scoring, including handscoring for
NeSA-Writing; psychometric services; and
reporting.
Page 2-37
Project Description
OKLAHOMA
Oklahoma State Department of Education
Office of Accountability and Assessments
2500 North Lincoln Boulevard
Oklahoma City, OK 73105-4599
Ms. Joyce DeFehr
Executive Director, Office of Accountability and
Assessments
405-521-3341 (phone)
joyce_defehr@sde.state.ok.us
Oklahoma School Testing Program:

Oklahoma Core Curriculum Tests (OCCT)
Subject Area and Grade Levels: Criterion-referenced assessments in Reading, Writing,
Mathematics, Science, Social Studies,
Geography, and U.S. History/Constitution/
Government for students in grades 38.
Page 2-38
This program is an annual, statewide assessment administered to approximately 45,000

students per grade. DRC is under contract
with the Oklahoma State Department of
Education to provide ongoing development,
implementation, and management of the program. DRCs project responsibilities include
overall project management; item and test
development; psychometric services; standard
setting; printing of all test materials; packaging, distribution, and collection; scanning,
scoring, and handscoring; reporting; in-state
training; and customer service. With our computerized testing subcontractor, Computerized
Assessments and Learning (CAL), we also provide online offerings, including operational
assessments, practice tests, administrative
setup, and online reports.
PENNSYLVANIA
Pennsylvania Department of Education
Bureau of Assessment and Accountability
333 Market Street
Harrisburg, PA 17126-0333
Mr. John Weiss
Director, Bureau of Assessment and
Accountability
717-214-4394 (phone)
717783-6642 (fax)
jweiss@state.pa.us
Pennsylvania System of School Assessment

(PSSA)
19992003
20032008
20082014
Subject Areas and Grade Levels: Criterion-referenced tests in Reading, Writing,
Mathematics, and Science administered to
students in grades 38.
Project Description
The Pennsylvania Department of Education
has contracted with DRC to provide development, implementation, and management of the
PSSA. These are standards-based, criterion-referenced assessments, consisting of openended, multiple-choice, and technologyenhanced items. Each year, approximately
150,000 students are tested per grade, and
more than 1.6 million tests are administered.
DRCs responsibilities include project management; customer service; item development and
test construction; conducting review meetings;
designing and printing materials; packaging,
distribution, and collection; scanning and scoring all answer booklets, including handscoring;
psychometric services; standard setting;
rangefinding; and reporting.
Prior to January 2012, the contract included

grade 11 (with retests for students in grade 12)
and also included the PSSAModified test,
which had modified achievement standards in
Reading, Mathematics, and Science.
In spring 2013, DRC began delivering online
versions of the PSSA through our online testing engine, DRC INSIGHT.
Pennsylvania Voluntary Model Curriculum,
Classroom Diagnostic Tool, and Keystone
Exams
Project Description
The Pennsylvania Department of Education
contracted with DRC for the development of
the Voluntary Model Curriculum, Classroom
Diagnostic Tool, and Keystone Exams. All
three components are aligned to the
Pennsylvania curriculum framework, the
Pennsylvania Common Core Standards
(PACC), and assessment anchors in the areas
of Reading, Writing, Mathematics, Science, and
Social Studies. Diagnostic tests are administered multiple times throughout the year. EOC
tests are administered three times per. DRC
provides online test administration for the
Classroom Diagnostic Tool and Keystone
Exams with our proprietary online testing system, DRC INSIGHTTM.
The Voluntary Model Curriculum consists of
K12 curricular resources and materials developed in collaboration with Pennsylvania teachers, administrators, and higher education faculty. Curricular resources include units and
lesson plans as well as a variety of support
materials. DRCs responsibilities include project management, curriculum development
(learning progressions/units/lesson plans)
aligned to PACC, development of enriched performance tasks, committee facilitation, meet-
Page 2-39
ing planning and coordination, designing an

online committee feedback system, and curriculum field testing.
The Classroom Diagnostic Tool is an online
test system that supports the Keystone Exams
by providing instructional feedback to students
and teachers through criterion-referenced,
multiple-choice, online computer-adaptive
assessments of the prerequisite skills of students entering grades 312. Like the Keystone
Exams, the assessments are modular in format, based on the same academic content, and
consist of multiple-choice items (with realtime scoring). The system is available for use
in schools and classrooms throughout the
school year, and teachers can easily access
online curricular resources directly from student and class reports. DRC is responsible for
providing the following: project management;
item development and item bank aligned to
PACC; online administration with computeradaptive testing algorithm; educator feedback
system; field testing; materials production
(print and online); customer service; scoring;
psychometric services; data management;
reporting; and professional development
resources.
Page 2-40
The Keystone Exams are a series of EOC summative assessments required for graduation
(multiple-choice, constructed-response, and
technology-enhanced) in Mathematics, Algebra
I, Algebra II, Geometry, Literacy, Composition,
Science, Biology, and Chemistry. Testing began
for three of the courses in 2011 with rollout of
the remaining six courses by 2015. The exams
are offered in both online and paper-and-pencil format. Each test comprises three modules containing specific course-related content topics, permitting students who dont
achieve proficiency to retake only the modules
in which they were not successful. A student
takes a Keystone Exam after completion of the
appropriate course; the exam accounts for 30%
of the students final grade in the course.
DRCs responsibilities include project management; item development aligned to PACC;
forms construction; coordinating committee
reviews; field testing; materials production and
printing; packaging, distribution, collection,
and processing; customer service; online test
administration; scanning and scoring, including handscoring and automated scoring; psychometric services; standard setting; comparability study; data management; conducting
report focus groups; and reporting.
SOUTH CAROLINA
South Carolina Department of Education
Office of Assessment
1429 Senate Street, Room 607
Columbia, SC 29201
Ms. Susan Creighton
Education Associate
803-734-8535 (phone)
803-734-8886 (fax)
screight@ed.sc.gov
assessment is administered in March, and the

ELA, Mathematics, Science, and Social Studies
assessments are administered each May. The
writing test consists of an extended-response
item and multiple-choice items. The remaining
tests consist of multiple-choice items only. The
SCPASS program replaced the PACT program.
High School Assessment Program (HSAP)
South Carolina Palmetto Assessment of

State Standards (SCPASS)
20082013
19982008
19971998
(SCPASS)
(SCPASS)
(PACT)
(PACT Field Test)
Subject Areas and Grade Levels: Statewide,

criterion-referenced assessments for a grades
38 assessment in Writing and grades 38
assessments in ELA, Mathematics, Science,
and Social Studies (50,000 students per
grade/subject annually).
Project Description
The South Carolina Department of Education
has contracted with DRC to provide overall
management and administration of the
Palmetto Assessment of State Standards
(SCPASS) for grades 38. The SCPASS
includes both online and paper-and-pencil
administration. DRCs responsibilities include
project management; item development; materials production; packaging, distribution, and
collection of test materials; customer service;
coordinating and presenting workshops (instate and via WebEx); online testing via the
DRC INSIGHT test engine; scanning and scoring, including handscoring; psychometric services; and reporting. The SCPASS is administered in two parts to more than 50,000 students per grade/subject annually. The Writing

20082013
20032005
19982003
19891998
(HSAP)
(HSAP)
(BSAP)
(BSAP)
(BSAP)
Subject Areas and Grade Levels: Statewide,

criterion-referenced assessments (multiplechoice and constructed-response) for high
school students in ELA and Mathematics.
Approximately 90,000 students are tested per
subject annually and administered three times
per year.
Project Description
has selected DRC as the contractor for the
overall administration and management of the
High School Assessment Program (HSAP).
DRCs responsibilities include project management; materials production; packaging, distribution, and collection; customer service; coordinating and presenting workshops (in-state
and via WebEx); scanning and scoring, including handscoring; psychometric services; and
reporting. The HSAP is administered each fall,
spring, and summer and includes tests in the
content areas of ELA and Mathematics. The
HSAP is the only state exit examination in
South Carolina. Students first take the tests in
the spring of their second year of high school.
If a student does not pass, retesting continues
each spring and fall. The ELA test consists of
one extended-response writing item, two constructed-response reading items, and multiple-
Page 2-41
choice items. The Mathematics test consists of

three constructed-response items as well as
multiple-choice items. The HSAP replaced the
Basic Skills Assessment Program (BSAP),
which was phased out in 2005 and tested
approximately 23,000 students three times per
year in Reading, Mathematics, and Writing.
DRC was responsible for administering the
BSAP from 19892005 and responsibilities
included materials production; packaging, distribution, and collection; scanning and scoring,
including handscoring; psychometric services;
and reporting.
South Carolina English Language
Contract Years: 20122014 (with three
one-year renewal contracts)
20082012
20072008
20062007
Subject Areas and Grade Levels: The ELDA
consists of three tests designed to measure
academic and social language proficiency in
the domains of speaking, listening and reading,
and writing. Approximately 45,000 students in
grades K12 are tested annually in the spring.
Project Description
management and administration of the ELDA,
which is administered annually to all students
in grades K12. The tests consist of multiplechoice and constructed-response items. DRCs
producing test materials; customer service;
coordinating and presenting workshops (instate and via WebEx); printing, packaging, distributing, and collecting materials; image scanning and scoring, including handscoring; psychometric services; and reporting.
Page 2-42
In 2012, DRC conducted an online pilot test

via the DRC INSIGHT online testing system for
grades 312 in grades and subjects.
South Carolina End-of-Course Examination
Program (EOCEP)
Contract Years: 20122014 (with three
one-year renewal contracts)
20082012
Subject Areas and Grade Levels: The EOCEP
includes tests in English 1, Algebra
1/Mathematics for the Technologies 2, Biology,
and U.S. History and Constitution.
Project Description
management and administration of the
EOCEP, which is a high-stakes assessment
administered to high school students in
English 1, Algebra 1/Mathematics for the
Technologies 2, Physical Science, Biology, and
U.S. History and Constitution. More than
55,000 students per course are tested annually. The tests consist of multiple-choice items
only and are administered three times per year
using online or paper-and-pencil modes.
Online test administration is done using our
online testing system, DRC INSIGHT. DRCs
producing test materials; coordinating and presenting workshops (in-state and via WebEx);
printing, packaging, distributing, and collecting
materials; online test delivery; customer service; scanning and scoring; psychometric services; and reporting.
Project Description
WASHINGTON
Office of Superintendent of Public Instruction
State of Washington
P.O. Box 47200
Old Capital Building
600 South Washington
Olympia, WA 98504-7200
Ms. Robin Munson
Assistant Superintendent, Division of
Assessment and Student Information
360-725-6356 (phone)
360-725-6509 (fax)
robin.munson@k12.wa.us
Mr. Mike Middleton
Director, Assessment Business & Special
Populations
(360) 725-6434 (phone)
(360) 725-0424 (fax)
michael.middleton@k12.wa.us
Washington Comprehensive Assessment

Program (WCAP): Measurements of Student
Progress (MSP), High School Proficiency
Exam (HSPE), and End-of-Course (EOC)
Tests
The MSP consists of standards-based tests for

grades 38. The HSPE measures the proficiency of high school students and serves as the
states exit exam. Approximately 80,000 students per grade take the MSP and HSPE tests.
DRCs contract also includes mandated EOC
tests in Mathematics and Biology. DRCs
modifying items to include technologyenhanced features (MSP); typesetting and
printing of all materials; writing and editing
manuals; conducting workshops for test coordinators; packing, distributing, and collecting
materials; scanning, editing, and scoring
answer documents, including automated scoring and hand scoring with teacher scoring; and
reporting results. DRC also provides online
versions of the Reading and Mathematics tests
for grades 38 and Science tests for grades 5
and 8 of the MSP via the DRC INSIGHT online
testing system.

Subject Areas and Grade Levels: The MSP
(grades 38 Reading, Mathematics, Science,
and Writing) and the HSPE (high school
Reading and Writing) are criterion-referenced
tests consisting of multiple-choice and constructed-response items. The EOC tests are
currently based on the specific Mathematics
courses of Algebra I, Integrated Mathematics 1,
Geometry, and Integrated Mathematics 2; EOC
tests in Biology were added for the first time in
2012.
Page 2-43
Project Description
WISCONSIN
WIDA Consortium
Wisconsin Center for Education Research
University of WisconsinMadison
1025 West Johnson Street, MD #23
Madison, WI 53706
Ms. Meredith Alt
ASSETS Project Manager
608-262-8759 (phone)
mlalt@wisc.edu
Dr. Carsten Wilmes
Assistant Director, Assessment
608-263-5547 (phone)
608-263-3733 (fax)
312-718-2444 (cell)
wilmes@wisc.edu
Assessment Services Supporting English

Learners (ELs) Through Technology
Systems (ASSETS)
The Wisconsin Department of Public

Instruction and World-Class Instructional
Design and Assessment (WIDA) were awarded
a four-year Enhanced Assessment Grant by the
U.S. Department of Education to develop a
next generation assessment system for ELs.
The project funded by the grant, known as
Assessment Services Supporting ELs through
Technology Systems (ASSETS), will develop an
online assessment system that will measure
student progress in attaining the English language skills they need to be successful in
school, postsecondary studies, and work.
DRC was selected as WIDAs technology partner to provide online versions of the test via
the DRC INSIGHT online testing system,
beginning with field testing in January 2014.
DRC will provide overall management of the
test delivery system and customer support for
schools and districts administering these
assessments. The DRC eDIRECT portal will be
used for student registration, and thousands of
students in the WIDA Consortiums member
states will be taking the field tests.
WIDA supports academic language development and academic achievement for linguistically diverse students through high quality
standards, assessments, research, and professional development for educators. The WIDA
Consortium includes 33 member states.
Page 2-44
4.9 POOR PERFORMANCE NOTICE

In this section we present a list of issues AIRs
clients have asked our assessment programs to
rectify during the past seven years. In all
cases, AIR worked closely with the client to
remedy the situation. Across all projects, AIR
has paid a total of $25,000 in liquidated damages during the past seven (7) years.
Contract
Hawaii State
Assessment
New Mexico
Alternate
Performance
Assessment
The contact information for these contracts is

presented in the project descriptions earlier in
this section of the proposal.
Documentation/Resolution
Purpose and Scope of Work
Issue/Concern/Date
Statewide assessment project covering

K8 and high school assessments in ELA,
Math, and Science
In fall 2008, AIR provided an incomplete

missing materials list.
In spring 2007, a scanning error occurred on
student test booklets in Hawaii, caused by the
malfunction of scoring subcontractors
electronic scanners. The error was uncovered
by AIR in response to a query from a Hawaii
test coordinator.
No formal notice of any kind was issued.

AIR corrected the problem.
See Appendix M (Item 1) for
documentation.
In spring 2007, 400 grade 6 booklets shipped

to one middle school were damaged. The
booklets were all printed on the same
machine at one of AIRs contracted print sites.
The errors were reported by schools when
they were getting ready to test or during
testing.

Replacements for the damaged testing
materials were express delivered to the
affected school.
In 2009, student test data files contained

miscodes for test completion status.
See Appendix M (Item 2) for the letter

from the New Mexico State Department
of Education (NMPED) related to this
issue. AIR provided corrected data files
and paid liquidated damages of $25,000.
No printed reports were affected, nor
were Adequate Yearly Progress (AYP)
results.
Alternate assessment project covering ELA

and Math in four grade bands and Science
in three grade bands
Pre-ID data were not merged into bio-update.
AIR terminated the subcontract and

reprocessed all of the spring 2007
Reading and Math test booklets on AIR
equipment to ensure the scanning errors
were corrected. We also produced a new
student-level data file with corrected
scores and proficiency levels.
Page 2-45
Contract
Purpose and Scope of Work
Issue/Concern/Date
Documentation/Resolution
Ohio Achievement
Assessments

K8 assessments in ELA, Math, and
Science, English language proficiency, and
alternate assessments for students with
disabilities
2009 test included a released item.

The released item was scored, and any
student responses that were similar to
the exemplars posted online were further
analyzed.
Oregon
Assessment of
Knowledge and
Skills
Statewide online assessment project

delivering more than 2 million tests to
more than 335,000 students
The spring 2008 Subcontractors Online Test

Delivery System failed to operate as specified.

AIR terminated the subcontract for not
meeting contractual requirements and
installed AIRs Test Delivery System.
In 2008, there was a delay in inserting field

test items and a delay in the secure browser
release for certain operating systems. In 2009,
there was incorrect placement of one
field-test item. In
2009, the nightly process of delivering
student results data files from AIRs server to
Oregons server was incomplete. In 2010,
there was an incomplete resolution of two
help desk calls.
No formal notice of any kind was issued

for any of these issues. All were resolved
with normal project escalation
procedures.
In 2011, there was a one-day delay in the

opening of the spring testing window and a
two-hour delay in the opening of the
End-of-Course (EOC) field tests.
In 2013, there was online testing disruption.
On April 16 and April 23, schools reported that
students were experiencing long load times,
and some students were logged out of tests
unexpectedly during brief periods on the
16th. On both days, students were testing at
all times during the day, and no student data
were lost as proctors and students were able
to use the functions to pause and resume
tests as designed.
See the letter from the Delaware

Department of Education (DDOE) in
Appendix M (Item 3), along with AIRs
response (Item 4).
Report issued by the Human Resources
Research Organization (HumRRO) showed
no widespread or consistently negative
effects on student scores; it did not find
evidence that the disruption impacted
school scores. Report cited AIR system
capability and the Minnesota Department of Education (MDE) policy allowing
students to review items or pause or
retake the test. Report can be found at
http://www.education.state.mn.us/mde/
schsup/testadmin/index.html.
In 2013, there were management and

communication issues.
See the documentation in Appendix M

(Item 5).
Delaware
Comprehensive
Assessment
System
Minnesota
Assessments
Page 2-46
Statewide summative and interim

assessments project in grades 210 in
Reading, Mathematics, Science, and Social
Studies.
online and paper-and-pencil testing for
approximately 500,000 students in
Reading, Mathematics, Science, and
alternate assessments
7.1.0 Introduction
7.1.0 Introduction
Summary of Proposed Solution
Florida seeks Common Core tests that have
been field-tested and are ready for operational
administration by Spring 2015. Given the centrality of Floridas assessment program to the
states accountability system and approach to
continuous improvement, the Department
must have an assessment system in place that
will serve these purposes. The assessment
must measure the full breadth and depth of
the Common Core State Standards, ensuring
full and accurate measurement of the intended
curriculum for each student.
Governor Scott has indicated that the assessment system must
not significantly increase the testing time
required;
not significantly increase the cost of testing
to states, districts, or schools;
allow students to test as late in the school
year as possible;
meet the same high psychometric standards as the current assessments, including
reliability and evidence of validity;
provide a basis for comparing performance
to other states; and
require technology that can be supported
without significant additional investment.
In addition, the ITN recognizes Floridas leading role in education across the nation, and
anticipates continued policy innovation.
Therefore, the state requires
the ability to integrate custom, Floridaowned items in the test starting after the
initial administration (we propose to integrate field-test items even in the first year)
without breaking trend lines;
the ability to review items even in the first

year to ensure that they meet Florida community expectations and cohere with community standards; and
innovative, thoughtful use of technology to
measure the important skills and knowledge embodied in the standards in deeper
ways without imposing additional barriers
to accessibility.
Here we outline a solution, described in detail
throughout the rest of this proposal, that
meets all of these requirements. Our proposed
solution offers Florida two different options for
the baseline, first-year test, with the two
options differentiated by cost, testing time,
and the types of measurement used. Both
options offer flexibility for the test to evolve at
Floridas discretion over time.
Both of the options are being field-tested this
spring, and each will field test an extensive
item bank, from which Florida can select
forms. Each of the options is being field-tested
using AIRs online test delivery system, which
can scale to serve as many simultaneous
users as necessary;
works with the technology currently found
within schools, supporting much lower
minimum requirements than identified in
the ITN; and
offers the broadest range of operational
item types and accommodations in the
industry.
The two options for the baseline, first-year test
include:
1. The Student Assessment of Growth and
Excellence (SAGE)
2. The Smarter Balanced Assessment
(Smarter Balanced)
Page 7.1-1
SAGE was developed in partnership with the

state of Utah to measure the full breadth and
depth of the Common Core State Standards.
Smarter Balanced was developed by the
Smarter Balanced Assessment Consortium.
Both item banks were designed to support
adaptive assessments, and therefore offer
broad and robust pools from which to select
fixed forms. The two options differ in cost,
testing, and turnaround time and both will be
fully field-tested and calibrated this spring.
The terms of use of these tests allow Florida to
enhance them with Florida items, to modify
the blueprints even in the first year, and to
select fixed-form tests.
The SAGE item bank is made up of approximately 300400 items per grade and subject.
In its adaptive form, SAGE mathematics takes
approximately 90 minutes to administer and
the ELA assessment takes approximately 150
minutes (elementary) to 180 minutes (secondary) to administer, including the writing
assessment. SAGE ELA measures Common
Core reading, writing, language, and listening.
The SAGE assessment is predominantly
machine scored, with a proportion of the automated scores on the writing assessment
reviewed by human scorers. Should Florida
adopt this model, AIR will recommend that
any high-stakes use of the writing assessment
be based on human scores rather than
machine scores.
Smarter Balanced is targeting an item bank of
approximately 1,000 items per grade/subject.
The Smarter Balanced assessment consists of
an adaptive assessment and a performance
task in each subject. The performance task is
preceded by a non-secure classroom activity in
advance of the performance task administration. Excluding the classroom activity, the
Smarter Balanced math test is designed to take
approximately three hours to administer, and
the Smarter Balanced ELA test will take
approximately three and a half hours to
Page 7.1-2
administer. Like SAGE, the Smarter Balanced

ELA test will measure reading, writing, language, and listening. The Smarter Balanced
Assessment Consortium includes 26 states,
23 of which will participate in the field test in
spring 2014. The final calibrations will result
from the field test, establishing a scale on
which comparisons can be made to the
participating states. The Smarter Balanced
Assessment performance tasks are mostly
human scored, and the remainder of the test
is mostly machine scored, although the nonperformance task portion of the test includes
up to three human-scored items per subject.
Both options offer Florida the required flexibility. Either assessment can be administered asis (as an adaptive assessment), or exploited as
a fully field-tested and calibrated Common
Core item pool. If the assessment is treated as
an item pool, Florida can design its own blueprints and select items from the pools to support creating FSA fixed forms.
Based on cost, testing time, and turnaround
time, Florida may want to take SAGE as the
base from which to build. During the first year
items from Smarter Balanced can be integrated with SAGE to establish the linkage that will
support cross-state comparisons. Similarly, if
PARCC items exist, the terms of the PARCC
grant require that they be available to states
like Florida, and a similar linkage can be
established. Alternatively, Florida may choose
to license the Smarter Balanced item bank
from which to build FSA forms. We note that
the SAGE test has been accepted by the state
of Ohio as a backup plan in case PARCC is not
ready in time, so SAGE may provide direct
comparisons to Ohio and Utah, with other
comparisons accomplished through linking
studies.
Exhibit 7.1.0-1 summarizes these two options,
and how they meet the key objectives articulated by Governor Scott and elaborated in the
ITN.
Exhibit 7.1.0-1: Summary of Options and How They Meet Floridas Objectives
Objective
Smarter Balanced
SAGE
Timely and informative results
Performance tasks may be administered earlier, with

primarily computer-scored portion administered later.
The item bank covers full depth and breadth of content.
Primarily machine scored allowing rapid turnaround of

online grades. Robust item bank covers full depth and
breadth of content.
Costs in line with current assessments
Slightly more.
Less than or equal.
Similar testing time to current

assessments
Approximately 78 hours including both subjects.
Approximately 4 hours including both subjects.
Summative assessments as close

as possible to the end of the year
Rapid turnaround for online grades allows later testing

while meeting timelines.
Rapid turnaround for online grades allows later testing

while meeting timelines.
Student mastery of the standards

taught, including inter-state
comparisons
Direct comparisons to Smarter Balanced states.
Comparisons to Smarter Balanced and (potentially)

PARCC states through linking study. Direct comparisons
to Utah and potentially Ohio.
Test quality as rigorous as current

assessment
Test design supports validity and reliability comparable to

current FCAT 2.0.
Test design supports validity and reliability comparable

to current FCAT 2.0.
Results that can be used for Floridas

accountability system
Assessment provides a criterion-based assessment of the

Common Core State Standards and oers a vertical
(developmental) scale as well as an on-grade scale.
Assessment provides a criterion-based assessment of the

Common Core State Standards and oers a vertical
(developmental) scale as well as an on-grade scale.
Requires technology within required

parameters, including
accommodations
Delivered with AIRs systems, imposes far less demanding

technology requirements than set forth in the ITN.
Delivered with AIRs systems, imposes far less demanding

technology requirements than set forth in the ITN.
Allows incorporation of Floridacreated content
Florida can use the items along with items created for and
eld-tested in Florida.
Florida can use the items along with items created for
and eld-tested in Florida.
Responsive evolution in Floridas

accountability system
License to use items allows FSA to diverge from the

Consortium assessment over time.
License to use items allows FSA to diverge from the SAGE

assessment over time.
Responsive to Florida community

sensitivities
Large item bank oers opportunity for Florida community

review to eliminate inappropriate items without sacricing
coverage of the pool.
Large item bank oers opportunity for Florida community

review to eliminate inappropriate items without sacricing
coverage of the pool.
Innovative, thoughtful use of

technology
Item specications designed to use technology to elicit

deeper measurement while maintaining accessibility.
Item specications designed to use technology to elicit

deeper measurement while maintaining accessibility.
Policy Drives FSA Requirements

For the past decade, Florida has been nationally recognized as a pioneer in innovative education reform that has led to unprecedented
improvements in student achievement. This is
demonstrated by improved National
Assessment of Educational Progress (NAEP)
scores, a closing of the achievement gap, and

increased graduation rates. Florida has experienced success in implementing major accountability reforms for schools and districts, with
the A to F grading system; for students, with
statewide third-grade progression requirements
and high school graduation standards; and for
Page 7.1-3
teachers and administrators with educator

evaluation based on student outcomes. Even
the accreditation standards for teacher preparation institutions now include components to
account for the effectiveness of new educators
based on their effectiveness in improving student outcomes. Each of these education
reforms depends on fair, accurate, and transparent measures of student achievement.
Continuity of the reforms depends entirely on
the integrity and success of the underlying
assessments and data.
Below, we summarize some key policies that
the FSA will be called on to support. These
policies drive both the timelines and characteristics of the assessment. In preparing this
proposal, this policy review provided the
touchstone for our technical recommendations
and timelines.
School Grades and Federal

Accountability
A to F school grades provide transparent,
objective, and easily understood data to parents, educators, and the public to spur
improvement among all schools. Being able to
measure student proficiency and learning
growth measures aimed at meeting a standard
are critical components of school grading.
Being able to measure growth during the first
year of the assessment transition is important
for half of the school grading components that
focus on growth in reading and math. Florida
currently has a federal waiver from the
requirements of Adequate Yearly Progress
(AYP) as described under the Elementary and
Secondary Education Act for which school
grades are substituted for AYP designations.
Ensuring that there is baseline data to measure growth in 20142015 is critical to A to F
school grading. AIR recognizes it is the desire
of the Department to have grades for the
20142015 school year released in the late fall
Page 7.1-4
2015. As a result, standard setting will occur

over the summer for proficiency standards and
determining achievement levels. Growth computations will need to be made during the
same timeframe to ensure a years worth of
progress in a years time is measurable from
the FCAT to the new assessment for inclusion
in school grading.
Grade 3 Reading Guarantee

Florida has placed a command focus on reading in the primary grades (K3), and research
has repeatedly confirmed the critical importance of developing literacy skills in those
years (Musen, 2010; National Institute for
Literacy, 2009; Strickland & Riley-Ayers,
2006). Reading is the foundation of learning.
When you can read, you can learn. In 2001,
29% of Floridas grade 3 students were performing at the lowest achievement level on
FCAT reading. In 2002, Florida began requiring retention for students who performed in
the lowest achievement level at the end of
grade 3. This policy, along with the other
investments that Florida made in early reading, paid dividends: by 2013, only 18% of grade
3 students fell into that lowest category.
The grade 3 reading guarantee imposes both
timeline and technical demands on the assessment. Parents and educators need time to
respond to the grade 3 reading results. The
reporting deadline has historically been midto-late May. First, the reading guarantee
imposes at least two technical requirements.
First, the ELA assessment must provide a reliable reading score. Second, the cut-score
between the lowest level and the next level
must accurately and fairly classify students
whose reading puts them at sufficiently
increased risk to warrant retention.
The use of the reading test for matriculation
into grade 4 requires an achievement standard
that has been explicitly set for this purpose,
and requires evidence of the validity of the

judgment. This poses a particular challenge for
the 20142015 school year. We have proposed
a standard setting plan and validation plan
that will support this use even in the first year
of implementation.
High School Graduation

State policy has required a high school exit
exam since the 1970s. Over time, Florida has
made the exam more rigorous, which made
the diploma more meaningful. Before 2002,
students had to pass a grade 8 minimum skills
test to earn a diploma.
In 2002, Florida raised the rigor of the exam
from a grade 8 to a grade 10 competencybased exam. Rising to meet expectations, more
students are actually passing the tougher test.
When expectations rise, students and teachers
rise to meet and often exceed them.
As Florida transitions to the FSA, the assessment must prove a sufficiently rigorous measure of the Common Core State Standards
(CCSS) that a high school diploma is meaningful to graduates, employers, and postsecondary
institutions. Potential graduates will need
assessment results before the end of the school
year and the results must be reflective of 21st
century skills.
While the standards established by others for
Smarter Balanced or SAGE will be designed to
represent career and college readiness, Florida
may want to evaluate those standards for
themselves and potentially establish independent standards.
Teacher Evaluation
Beginning in school year 20112012 for reading/language arts and math teachers (and
school year 20142015 for all teachers), at
least 50 percent of a teachers evaluation must
be based on student learning growth assessed

annually and measured by statewide assessments or, for subjects not measured by
statewide assessments, by district assessments
(per section 1008.22[8], Florida Statutes).
The Department contracted with the American
Institutes for Research (AIR) to assist in the
development, evaluation, and implementation
of a value-added model (VAM) to be used for
teacher evaluation. The goal of the project is
to provide a fair, accurate, and transparent
measure of teacher effectiveness that districts
can incorporate into their teacher evaluation
systems to bring about educational improvement and to provide information about student
learning to individual teachers, principals, and
other stakeholders.
AIR remains committed to working in partnership with the Department and the Student
Growth Implementation Committee (SGIC),
using a collaborative and iterative process during the assessment transition to develop, analyze, and implement value-added measures of
student academic performance in Florida public schools at grade levels K12 and ensure the
design is not adversely impacted by the transition to new statewide student assessments.
The NGSS adopted in 2007 and tested on the
FCAT 2.0 predate the CCSS but reflect an
attempt to increase the rigor of the standards
that they replaced. It appears that these standards were a step in the same direction as the
CCSS. This is important because a true growth
measure must reflect growth along a common
dimension. When the prior-year score measures a different construct than the current
year score in a value-added model, biases likely arise. As the Department saw when trying to
implement value-added models for End-ofCourse (EOC) tests, a pre-test measuring a different construct resulted in attenuated relationships between pre-tests and post-tests.
This attenuation leads to the value-added
measure representing a sort of compromise
Page 7.1-5
between a growth measure and a status measure, and is likely to favor teachers of higherperforming students, which does not serve the
goal of improving education for all of Floridas
students.
From the perspective of comparability over
time, the transition to the FSA faces the same
challenges faced when Florida transitioned
from the FCAT 1.0 to the FCAT 2.0. In that
transition the skills measured appeared sufficiently similar to support the value-added
analysis.
Charter School Reporting and

Agreements
Charter schools are public schools of choice.
They are very popular and among the fastest
growing school choice options in Florida.
Charter schools are largely free to innovate,
and often provide more effective programs and
choice to diverse groups of students. Since
1996, the number of charter schools in Florida
has grown to more than 578 schools in 46 districts in 20122013. Charter school student
enrollment now exceeds 200,000 students.
Section 1002.33(23), Florida Statutes, requires
the Florida Department of Education to prepare an annual statewide analysis of student
achievement in charter schools versus the
achievement of comparable students in traditional public schools. The analysis examines
the average performance of charter school students and traditional public school students
using state assessment data from the FCAT 2.0
Reading, Math, and Science, and the Algebra
EOC exams and is due to the legislature upon
completion.
The governing body of the charter school is
required to report its progress annually to
its sponsor, and subsequently to the
Commissioner of Education per Section
1002.33(9)(k), Florida Statutes. The
Page 7.1-6
Department has developed a uniform, online

annual accountability report to be completed
by charter schools that is pre-populated with
student assessment data. Charter schools are
subject to the same accountability requirements as other public schools, including
reports of student achievement information
that links baseline student data to the schools
performance projections identified in the
charter. The deadline for schools to submit
the 20122013 Charter School Annual
Accountability Report online was November 1,
2013. The deadline for final submission by districts is January 31, 2014.
In addition, charter schools sponsors may
have additional reporting and accountability
requirements as designed by the district.
Teacher Preparation Program Approval

The teacher preparation program approval
process requires that each teacher preparation
program go through an initial approval process
first based on the institution's documented
ability to deliver a competency-based program.
Once initial approval is received, the institution must submit an annual report based on
the candidate's performance within their program and during the candidate's first year of
teaching.
Programs undergo a continued approval
process through annual reports submitted to
the Department and site visit reviews completed at the institution every seven years.
Currently, 36 colleges and universities in
Florida have state-approved initial teacher
preparation programs.
The Department continues to review and revise
recommendations for improvements to teacher
preparation programs. Recommendations
made by the Race to the Top Teacher Leader
Preparation Implementation Committee
(TLPIC) are the foundation and a framework
for improvement of programs, as well as our

teacher workforce and ultimately our Pre-K12
students. The TLPIC incorporates value-added
model data and student achievement data in
reading and math by subgroup into the program review and approval requirements.
As we see from Exhibit 7.1.0-2, the policy

demands to be placed on the FSA impose the
following technical requirements:
Summary of Policy-Driven Technical

and Timeline Requirements
The preceding discussion highlighted some of
the key demands that Floridas educational
policy will make on the FSA. Exhibit 7.1.0-2
summarizes the technical requirements
imposed by the uses to which the data will be
put, and the implied timelines.
A reliable developmental (vertical) scale

Linkage to the FCAT 2.0
Evidence of the strong correlation between
the trait measured by the FCAT 2.0 and the
FSA
A reliable separate score for Grade 3 reading
Performance standards for Grade 3 reading
(at least) before May 2015
Evidence of the validity of the inference
that grade 3 students in the lowest per-
Exhibit 7.1.0-2: Dates and Requirements Driven by Policies Relying on FCAT

Program/policy
Requirements to serve policy
Approximate annual
reporting date
Implied date for

final results
June 15
School grades
1. A vertical scale to support the growth component of the

school grades measure.
2. A strong linkage to the FCAT 2.0 to support the growth
measure in 20142015.
Grade 3 promotion
Before May 23
1. The Grade 3 ELA test must provide for a separate reading
score.
2. Timely performance standards established to support the
retention decision.
3. Evidence of the validity of the inference that performance in
the lowest FSA performance level indicates increased
likelihood of lower longer-term performance across subjects.
High school graduation
1. Evidence of the validity of the inference that students

achieving the procient standard are college/career ready
and do not need remediation in postsecondary institutions
or are able to obtain livable wage employment.
Before May 14
May 1
Teacher evaluation
1. Strong alignment between the Common Core traits

measured by the FCAT 2.0 and the FSA.
2. Evidence of the FCAT 2.0FSA alignment.
July 31
Summer 2015
Charter school reporting and

agreements
Same as other public schools, so all of the above.
Depends on charter renewal date,

no later than November
Teacher preparation
accreditation
1. Strong correlation between the traits measured by the

FCAT 2.0 and the FSA.
2. Evidence of the FCAT 2.0FSA correlation
Fall/winter
May 15
Page 7.1-7
formance level are at increased risk of

longer-term failure
that examinees meeting the high school
proficient standard are career and college
ready
Our proposed plan meets all of these requirements, and does so on a more favorable timeline than required in the ITN. Below, we provide a brief summary of the technical plan.
Preview of Proposed Plan

AIR has crafted a plan that meets these deadlines and requirements and at the same time,
we are able to push most of the testing windows as late as April 1 in the first year. As
described in the previous section, the foundation of the plan begins with the item bank
available from Smarter Balanced or SAGE. AIR
will work with Florida to select fixed forms
from these item banks or, for online grades
and at the Departments discretion, select item
pools for use in an adaptive test (with a paper
fixed form offered as an accommodation).
Throughout this proposal, readers will see
where these policy uses guide our proposed
technical approach. In particular, we propose a
plan to meet the following technical requirements:
A reliable developmental (vertical) scale.
Both Smarter Balanced and SAGE are
developing a vertical scale during their field
tests this spring. Using AIR technology,
these vertical scales will be based on much
larger and more robust linking sets than is
typically achievable in fixed-form assessments, providing for a more reliable, robust
vertical scale.
Linkage to the FCAT 2.0. In Section 7.6.3.6
we propose a plan to link the FSA to the
FCAT 2.0.
Page 7.1-8
Evidence of the strong correlation between

the trait measured by the FCAT 2.0 and
the FSA. In Section 7.6.3.6 we propose a
validity study that yields accurate estimates of the correlation between the latent
traits.
A reliable separate score for grade 3 reading. Both the Smarter Balanced and SAGE
banks will yield reliable scores for reading.
Performance standards for grade 3 reading (at least) before May 2015. Although
data from Florida will not necessarily be
available, we propose a standard setting
plan that will enable Florida to establish
standards based on data representative of
other states and national and international
benchmarks in fall 2014.
that grade 3 students in the lowest performance level are at increased risk of
longer-term failure. By definition, this evidence must arise over time because it is
about longer-term outcomes. If Florida
adopts the SAGE assessment as a foundation for the FSA, Utah is interested in a
study comparing the longer-term outcome
of low-performing third graders in Utah
(where the retention policy is not in place)
with those of low-performing Florida grade
3 students (where the retention policy is in
place). This study, described in Section
7.6.3.6, will provide empirical evidence
supporting the validity of this use of the
assessment.
that examinees meeting the high school
proficient standard are career and college
ready. Utah is planning a study to be conducted in spring 2014 to compare the performance on high school SAGE of college
freshmen in remediation with those of college freshmen in standard courses.
Discrimination between the groups will
support inferences about career and college
readiness, as well as provide input to the

standard setting in the form of the score
that best discriminates between the groups.
Our plan meets the timeline requirements as
well, and offers Florida the option of pushing
most testing dates back to allow almost a
month of additional instruction. We propose
online assessments that are entirely machinescored. As you will see, these assessments
include many constructed-response items (for
example, math tests where students respond
with sets of equations or drawings) that can be
scored with near-perfect reliability and validity. When administered on paper, these items
are hand-scored; however, the limited number
of grade/subjects that are administered on
paper enable us to complete the hand-scoring
quickly, with the exception of the ELA performance tasks, which will require more handscoring.
To meet these deadlines, we propose that
the ELA performance tasks be administered in early March, consistent with the
testing window in the ITN;
the remainder of the tests be administered
during a 12-day window starting approximately April 1, allowing nearly an additional month of instruction before assessment;
and
the grade 3 reading assessment (which
must be processed much more quickly) be
entirely machine-scorable.
In addition, we have proposed a logistical
process that eliminates the need for extensive
cleanup at the end of the testing window. We
propose a real-time-ID process in which newly
arriving students can be entered into our registration system, which will print ID labels identical to the pre-ID labels in real time. This
process augments, rather than replaces, the
pre-ID process. It avoids the errors associated
with hand-gridding information on answer doc-
uments and enforces a wide variety of data

integrity checks. By introducing the information into the system before testing, rather than
after scanning, the process will allow Florida to
clean up the demographic data while testing is
still taking place.
We are confident that our proposed plan will
enable Florida to continue its impressive educational reform and improvement policies
uninterrupted.
7.1.1 Notice to Respondents

AIR acknowledges each of the nine paragraphs
in the notice to respondents.
This proposal refines requirements mentioned
in Paragraph 4. AIR proposes to use pre-existing proprietary software to deliver this program. We are willing to license the software to
Florida after the duration of the contract for a
price to be negotiated. However, our system is
compatible with the open-source Smarter
Balanced online system that will be released
into open source next year. While the capabilities of our proprietary system exceed the
open-source system and the architecture is different, the items can be delivered using the
open-source system. Our base proposal is that
Florida adopt the open-source system for
future reference, although it will need some
architectural changes to scale to the required
capacity.
We note that Smarter Balanced is in the
process of publishing interoperability standards, which we are developing in partnership
with them. We can deliver Florida items in an
interoperable format that conforms to those
standards.
The base tests proposed here, SAGE or
Smarter Balanced, remain the property of
their owners. Use of those items beyond the
contract period will require Florida to negoti-
Page 7.1-9
ate with the item owners. However, the base

plan will deliver enough Florida-owned items
to replace the foundation item bank before the
end of the contract period.
tion milestones for scoring and reporting student results.
7.1.2 Project Schedule
Musen, L. (2010, May). Early reading proficiency. Annenberg Institute for School
Reform. Retrieved from http://annenberginstitute.org/pdf/leadingindicator_reading.pdf
As required by the ITN, we present AIRs

staffing plan in Section 7.7.2. This plan includes
detailed descriptions of the roles and qualifications of our proposed staff along with estimated
FTE for lead staff and overall activities.
The attached sample project schedule, at the
end of this section, lists the major deliverables
and tasks as required by the Department. In
addition, many tasks and processes, internal to
AIR, have been detailed. These tasks and
processes support the major deliverables,
ensuring that all stakeholders are kept
apprised of the overall project status. The AIR
program team will be responsible for maintaining the project schedule; milestones and deliverables will be discussed during the weekly
status meetings.
References
National Institute for Literacy. (2009). Early

beginnings: Early literacy knowledge and
instruction. Retrieved from
http://lincs.ed.gov/publications/pdf/NELPEar
lyBeginnings09.pdf
Strickland, D.S., & Riley-Ayers, S. (2006,
April). Early literacy: Policy and practice in
the preschool years. National Institute for
Early Education Research, 10. Retrieved
from http://nieer.org/resources/policybriefs/10.pdf
Some highlights of the schedule include the

following:
Meeting all of Floridas reporting deadlines,
with Florida standards, in 20142015
Delivering 20142015 test forms built in
collaboration with Florida
Scheduling of meetings with Florida educators and stakeholders
Listing of proposed testing windows for
20142015
Delivering results in accordance with
Floridas requirements
Listing of supporting tasks and processes
for meeting major schedule deliverables
The table on the next page provides an easy
one-page reference to the key post-administraPage 7.1-10
Test Window (Other)

Schools return material to DRC
Materials arrive at DRC
Scanning
Online Validation Handscoring
Late Batch Processing
Receive prelim Student Results File from DRC
Merge MC and handscoring data
Prepare data file for Department review
Department approval
Release data to ORS
Generate paper report files
Paper report QC
Print/distribute reports
RD-1 Posted in ORS for Schools (DSR, School Report
of Students)
RD-2 Posted in ORS for Schools
(State, District Summary; State Report of Districts;
District Report of Schools; SAR; DAR)
RD-3 Posted in ORS for Schools
(State, District Summary; State Report of Districts;
District Report of Schools; School Reports of
Students)
RD-4 Delivery to Districts for Schools and Districts
(Initial Release Student Reports (hard copies);
CD/DVD with DSR, SAR, DAR and Layouts)
(Late Reporting Student Reports (hard copies); Late
Reporting School Reports of Students (PDFs on
CD/DVD))
(Writing Images (PDFs on CD/DVD) (spring only))
(Demographic SARs, DARs, and DSRs)
Test Window (Writing)

Schools return material to DRC
Materials arrive at DRC
Scanning
Paper Handscoring
Key Post-Administration Milestones
April 1-18, 2015

4/19/2015
4/20/2015
4/24/2015
4/26/2015
4/27/2015
4/28/2015
4/28/2015
5/1/2013
5/13/2015
5/14/2015
5/15/2015
5/20/2015
5/21/2015
5/14/2015
5/14/2015
5/14/2015
6/5/2015
8/8/2015
7/15/2015
8/31/2015
4/21/2015
4/24/2015
5/1/2015
5/5/2015
5/7/2015
5/8/2015
5/10/2015
5/15/2013
5/22/2015
5/23/2015
5/24/2015
5/30/2015
5/31/2015
5/23/2015
5/23/2015
5/23/2015
6/5/2015
8/8/2015
7/15/2015
8/31/2015
Grade 10 ELA/L
Reading
April 1-14, 2015
Grade 3 ELA/L
Reading
Key Post-Administration Milestones
3/20/2015
3/22/2015
3/28/2015
4/22/2015
4/19/2015
4/20/2015
4/24/2015
4/26/2015

8/31/2015
7/15/2015
8/8/2015
6/5/2015
5/14/2015
5/14/2015
5/14/2015
4/27/2015
4/28/2015
4/28/2015
5/1/2013
4/30/2015
5/14/2015
5/3/2015
5/6/2015
5/7/2015
Mar 5-19, 2015
April 1-18, 2015
8/31/2015
7/15/2015
8/8/2015
6/5/2015
5/30/2015
5/30/2015
5/30/2015
4/17/2015
4/21/2015
4/24/2015
5/15/2015
5/16/2015
5/20/2015
5/22/2015
5/28/2015
5/29/2015
5/30/2015
5/31/2015
5/31/2015
6/4/2015
April 1-14, 2015
Grade 3-11 ELA/L
Grade 10 ELA/L
Writing
Spring 2015
8/31/2015
7/15/2015
8/8/2015
6/5/2015
5/30/2015
5/30/2015
5/30/2015
4/17/2015
4/21/2015
4/24/2015
5/15/2015
5/16/2015
5/20/2015
5/22/2015
5/28/2015
5/29/2015
5/30/2015
5/31/2015
5/31/2015
6/4/2015
Apr 1-14, 2015
Grades 3-8
Mathematics
8/31/2015
N/A
7/1/2015
5/5/2015
4/28/2015
4/28/2015
4/28/2015
4/16/2015
4/17/2015
4/20/2015
4/22/2015
4/22/2015
4/24/2015
4/24/2015
4/27/2015
4/28/2015
4/28/2015
5/1/2015
5/4/2015
5/5/2015
Apr 1-14, 2015
End of Course
Algebra I
8/31/2015
N/A
7/1/2015
5/15/2015
5/8/2015
5/8/2015
5/8/2015
4/27/2015
4/28/2015
4/29/2015
5/1/2015
5/1/2015
5/4/2015
5/4/2015
5/5/2015
5/7/2015
5/8/2015
5/12/2015
5/14/2015
5/15/2015
Apr 7-25, 2015
End of Course
(Algebra 2, Geometry)
11/30/2015
N/A
10/14/2015
8/14/2015
8/7/2015
8/7/2015
8/7/2015
7/24/2015
7/25/2015
7/27/2015
7/29/2015
7/29/2015
7/30/2015
7/30/2015
7/31/2015
8/6/2015
8/7/2015
8/10/2015
8/12/2015
8/13/2015
Jul 13-24, 2015
End of Course
Algebra I
11/30/2015
N/A
10/14/2015
8/14/2015
8/7/2015
8/7/2015
8/7/2015
7/24/2015
7/25/2015
7/27/2015
7/29/2015
7/29/2015
7/30/2015
7/30/2015
7/31/2015
8/6/2015
8/7/2015
8/10/2015
8/12/2015
8/13/2015
Jul 13-24. 2015
End of Course
(Algebra 2,
Geometry)
Summer 2015
3/30/2016
12/20/2015
1/27/2016
11/13/2015
11/6/2015
11/6/2015
11/6/2015
10/23/2015
10/24/2015
10/27/2015
10/29/2015
10/29/2015
10/30/2015
10/30/2015
11/2/2015
11/5/2015
11/6/2015
11/9/2015
11/11/2015
11/12/2015
Oct 12-23, 2015
Grade 10 ELA/L
Retake
2/29/2016
N/A
12/16/2015
10/16/2015
10/9/2015
10/9/2015
10/9/2015
9/26/2015
9/28/2015
9/30/2015
10/1/2015
10/1/2015
10/2/2015
10/2/2015
10/3/2015
10/8/2015
10/9/2015
10/12/2015
10/14/2015
10/15/2016
Sept 14-25, 2015
End of Course
(Algebra, Algebra
2, Geometry)
Fall 2015
Winter 2015
5/11/2016
N/A
3/11/2016
1/11/2016
1/4/2016
1/4/2016
1/4/2016
12/21/2015
12/23/2015
12/26/2015
12/27/2015
12/27/2015
12/28/2015
12/28/2015
12/29/2015
1/3/2016
1/4/2016
1/7/2016
1/9/2016
1/10/2016
Dec 2-20, 2015
End of Course
(Algebra, Algebra 2,
Geometry)
Page 7.1-11
ID
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
FloridaStandardsAssessment(FSA)ProgramSchedule
TestAdministrationWindows
20142015
Spring2015
Grade311ELA/L(WritingGrades4,8and10)
Writing
Reading
Grades38Mathematics
EOCAlgebra1
EOCAlgebra2,Geometry
Summer2015
EOCAlgebra1
Fall2015
Grade10ELA/LRetake
EOCAlgebra1,Geometry,Algebra2
Winter2015
20152016
Spring2016
Writing
Reading
Grades38Mathematics
EOCAlgebra1
Summer2016
EOCAlgebra1
Fall2016
Grade10ELA/LRetake
Winter2016
20162017
Spring2017
Writing
Reading
Grades38Mathematics
EOCAlgebra1
TaskName
Page 7.1-12
Thu3/5/15
Thu3/5/15
Thu3/5/15
Thu3/5/15
Wed4/1/15
Wed4/1/15
Wed4/1/15
Tue4/7/15
Mon7/13/15
Mon7/13/15
Mon7/13/15
Mon9/14/15
Mon10/12/15
Mon9/14/15
Wed12/2/15
Wed12/2/15
Mon3/7/16
Mon3/7/16
Mon3/7/16
Mon3/7/16
Fri4/1/16
Fri4/1/16
Fri4/1/16
Thu4/7/16
Wed7/13/16
Wed7/13/16
Wed7/13/16
Wed9/14/16
Wed10/12/16
Wed9/14/16
Fri12/2/16
Fri12/2/16
Tue3/7/17
Tue3/7/17
Tue3/7/17
Tue3/7/17
Mon4/3/17
Mon4/3/17
Mon4/3/17
Fri4/7/17
207days
37days
32days
32days
13days
10days
10days
14days
10days
10days
10days
30days
10days
10days
13days
13days
208days
37days
29days
11days
10days
10days
10days
14days
10days
10days
10days
30days
10days
10days
14days
14days
206days
37days
37days
11days
10days
10days
10days
14days
Fri12/18/15
Fri4/24/15
Fri4/17/15
Fri4/17/15
Fri4/17/15
Tue4/14/15
Tue4/14/15
Fri4/24/15
Fri7/24/15
Fri7/24/15
Fri7/24/15
Fri10/23/15
Fri10/23/15
Fri9/25/15
Fri12/18/15
Fri12/18/15
Wed12/21/16
Tue4/26/16
Thu4/14/16
Mon3/21/16
Thu4/14/16
Thu4/14/16
Thu4/14/16
Tue4/26/16
Tue7/26/16
Tue7/26/16
Tue7/26/16
Tue10/25/16
Tue10/25/16
Tue9/27/16
Wed12/21/16
Wed12/21/16
Tue12/19/17
Wed4/26/17
Wed4/26/17
Tue3/21/17
Fri4/14/17
Fri4/14/17
Fri4/14/17
Wed4/26/17
Mon3/3/14
Finish
Page1
Mon3/3/14
Start
1day
Duration
Notes
ID
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
Summer2017
EOCAlgebra1
Fall2017
Grade10ELA/LRetake
Winter2017
TestDevelopment
ProductionSpecificationsDocument
Draft/reviewproductionsspecificationsdocument
Departmentreview
Applyedits
Finalize/approveproductionspecs
HandscoringSpecifications
FallELA/LRetake
Drafthandscoringspecs
Reviewhandscoringspecs
Applyedits
Finalize/approvehandscoringspecs
SpringELA/LRetake
Applyedits
SpringELA/LandMathematics
Applyedits
FallEOC
Applyedits
WinterEOC
Applyedits
SpringEOC
TaskName
Mon3/17/14
Wed3/19/14
Mon3/17/14
Wed3/26/14
Fri4/4/14
Fri5/1/15
Fri5/1/15
Mon5/4/15
Wed5/13/15
Fri5/22/15
Tue1/20/15
Tue1/20/15
Thu1/22/15
Mon2/2/15
Wed2/11/15
Tue1/20/15
Tue1/20/15
Wed1/21/15
Fri1/30/15
Tue2/10/15
Fri7/3/15
Fri7/3/15
Mon7/6/15
Wed7/15/15
Fri7/24/15
Fri9/25/15
Fri9/25/15
Mon9/28/15
Wed10/7/15
Fri10/16/15
Tue1/20/15
Tue1/20/15
15days
1day
7days
7days
1day
17days
1day
7days
7days
2days
18days
1day
7days
7days
2days
17days
1day
7days
7days
2days
17days
1day
7days
7days
2days
17days
1day
7days
7days
2days
17days
1day
Mon5/25/15
Fri5/1/15
Tue5/12/15
Thu5/21/15
Mon5/25/15
Thu2/12/15
Tue1/20/15
Fri1/30/15
Tue2/10/15
Thu2/12/15
Wed2/11/15
Tue1/20/15
Thu1/29/15
Mon2/9/15
Wed2/11/15
Mon7/27/15
Fri7/3/15
Tue7/14/15
Thu7/23/15
Mon7/27/15
Mon10/19/15
Fri9/25/15
Tue10/6/15
Thu10/15/15
Mon10/19/15
Wed2/11/15
Tue1/20/15
Fri4/4/14
Wed3/19/14
Tue3/25/14
Thu4/3/14
Fri4/4/14
Wed7/26/17
Wed7/26/17
Wed7/26/17
Wed10/25/17
Wed10/25/17
Wed9/27/17
Tue12/19/17
Tue12/19/17
Finish
Page2
Thu7/13/17
Thu7/13/17
Thu7/13/17
Thu9/14/17
Thu10/12/17
Thu9/14/17
Mon12/4/17
Mon12/4/17
Start
10days
10days
10days
30days
10days
10days
12days
12days
Duration
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Notes
Page 7.1-13
Page 7.1-14
124
121
122
123
Wed10/7/15
Fri10/16/15
Mon10/19/15
Mon9/28/15
Fri9/25/15
Tue10/6/15
Thu10/15/15
Fri10/16/15
Page3
Tue9/29/15
Thu10/8/15
Mon10/19/15
Mon9/28/15
1day
7days
7days
1day
Fri9/25/15
Mon9/28/15
Wed10/7/15
Fri10/16/15
1day
7days
7days
1day
Mon10/12/15
Wed10/21/15
Fri10/30/15
Mon11/2/15
Mon3/3/14
Wed3/12/14
Fri3/21/14
Mon6/2/14
Mon11/2/15
Wed4/9/14
Wed3/19/14
Fri3/28/14
Tue4/8/14
Wed4/9/14
Fri4/11/14
Tue4/15/14
Thu4/17/14
Mon4/21/14
Tue4/22/14
Mon3/3/14
Tue3/4/14
Thu3/13/14
Mon6/2/14
Wed3/19/14
Wed3/19/14
Wed3/19/14
Thu3/20/14
Mon3/31/14
Wed4/9/14
Thu4/10/14
Mon4/14/14
Wed4/16/14
Fri4/18/14
Tue4/22/14
Mon10/12/15
Tue10/13/15
Thu10/22/15
Mon11/2/15
Thu1/29/15
Mon2/9/15
Wed2/11/15
Mon5/25/15
Fri5/1/15
Tue5/12/15
Thu5/21/15
Mon5/25/15
Finish
Wed1/21/15
Fri1/30/15
Tue2/10/15
Fri5/1/15
Fri5/1/15
Mon5/4/15
Wed5/13/15
Fri5/22/15
Start
1day
7days
7days
1day
DraftTestItemSpecs
Departmentreview
Applyedits
Finalize/approveTestItemspecs
ELA/LandMathematicsSampleScoringGuides
Draftscoringguides
Departmentreview
Applyedits
Finalize/approvescoringguides
UnderstandingFlorida'sAssessmentReports
DraftUnderstandingFlorida'sAssessmentReports
document
Departmentreview
Applyedits
Finalize/approveUnderstandingFlorida'sAssessment
Reportsdocument
TestDesignandBlueprint
110
111
112
113
114
115
116
117
118
119
120
Duration
7days
Applyedits
7days
2days
SummerEOC
17days
1day
7days
Applyedits
7days
2days
Styleguidefortestitems
Reviewcurrent/draftupdatedstyleguide
1day
Departmentreview
7days
Applyedits
7days
Finalize/approvestyleguide
1day
ItemTrackingSystem(ITS)
424days?
AttributesandSpecifications
16days
DraftITSspecs
1day
Departmentreview
7days
Applyedits
7days
Finalize/approvestyleguide
1day
ConfigureITS
2days
ITSTestPlan
2days
DeployITS
2days
AIRTesting
2days
ITSgolive
1day
InterpretiveProducts:TestItemSpecificationsDocuments
TaskName
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
ID
Datestobenegotiated
Shouldcoincidewithscorereportduedates
FinishdatefromITN
Datestobenegotiated
Shouldcoincidewithscorereportduedates
FinishdatefromITN
Datestobenegotiated
Notes
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
ID
Blueprintcreated
BlueprintQC/testing
Finalize/approvedblueprint
ItemDevelopment
Year1Developmentplan
Selectitemsfor3fixedformsfromeachpool
CominedBias/DataReviewCommitteemeeting
ContentReviewCommitteemeeting
CommunitySensitivityReviewonline
Buildfixedforms
CreateOrderedItemBooklets
Developnewitems
ItemBankanalysis
Developmentplandrafted
Departmentreview/approvalofdevelopmentplan
ItemWriterTraining
Itemwriting
Itemreviewbatches(AIR)
Itemreviewbatches(Department)
Resolution
BiasandSensitivityCommittees
Securelocation
InvitestoBSCs
Meetingprep
BSCmeetingdays
Resolution
ItemContentReviewCommittees
Securelocation
InvitestoICRs
Meetingprep
ICRmeetingdays
Resolution
ContentAdvisoryCommittees
Securelocation
InvitestoCACs
Meetingprep
CACmeetingdays
Resolution
WebPreview
AIRsendsitemsforwebapprovaltotheDepartment
Departmentreviewsandapprovesitems
ContentItemReviewCleans
TaskName

Wed6/4/14
Wed6/4/14
Thu10/15/15
Thu10/22/15
Mon10/26/15
Thu5/29/14
Thu5/29/14
Wed11/11/15
Wed11/18/15
Fri11/20/15
Mon11/23/15
Tue11/24/15
Fri1/29/16
5days
1day
5days
2days
2days
5days
1day
5days
2days
2days
1day
5days
2days
Mon11/23/15
Mon11/30/15
Mon2/1/16
Wed6/4/14
Thu5/29/14
Tue11/17/15
Thu11/19/15
Mon11/23/15
Tue6/10/14
Wed6/4/14
Wed10/21/15
Fri10/23/15
Tue10/27/15
Wed6/4/14
Wed5/28/14
Wed10/28/15
Fri10/30/15
Tue11/3/15
Tue9/22/15
Wed9/23/15
Wed9/30/15
Wed10/7/15
Wed10/14/15
Fri10/16/15
Tue10/20/15
Wed10/21/15
Wed9/3/14
Wed9/10/14
Wed9/17/14
Wed9/17/14
Mon9/22/14
Mon9/22/14
Fri5/22/15
Fri5/29/15
Tue9/22/15
Finish
Page4
Thu5/29/14
Wed5/28/14
Thu10/22/15
Thu10/29/15
Mon11/2/15
5days
1day
5days
2days
2days
Mon9/1/14
Mon9/8/14
Thu9/11/14
Thu9/4/14
Wed9/17/14
3days
3days
5days
10days
4days
Wed9/16/15
Wed9/23/15
Thu9/24/15
Thu10/1/15
Thu10/8/15
Thu10/15/15
Mon10/19/15
Wed10/21/15
Mon9/1/14
16days?
5days
1day
5days
5days
5days
2days
2days
1day
Mon5/11/15
Mon5/25/15
Tue9/22/15
Start
10days
5days
1day
Duration
FinishdatefromITN
FinishdatefromITN
Notes
Page 7.1-15
TaskName
167
FieldTesting
168
SamplingforEFTs
169
DeliversamplingplantoDepartment
170
DataReviewCommittees
171
Securelocation
172
InvitestoDRCs
173
Meetingprep
174
DRmeetingdays
175
Resolution
176
FieldTestItemSetsandFormsReview
177
OnsiteTestConstruction
178
MeetingswithEducators
179
EnglishtoLanguageTranslationDictionary
180
EnglishtoSignLanguageTranslationDictionary
181 TestAdministration
182
Inventorymissingmaterialsandprovidereport
183
FallELA/LRetake
184
Missingmaterialsfollowup
185
Report
186
SpringELA/LRetake
187
188
Report
189
190
191
Report
192
FallEOC
193
194
Report
195
WinterEOC
196
197
Report
198
SpringEOC
199
200
Report
201
SummerEOC
202
203
Report
204
Deliverpreidentificationlabelsandrosters
205
FallELA/LRetake
206
SpringELA/LRetake
207
208
FallEOC
ID
Page 7.1-16

Mon10/26/15
Fri10/23/15
Mon10/26/15
Wed4/15/15
Tue4/14/15
Wed4/15/15
Wed4/15/15
Tue4/14/15
Wed4/15/15
Mon4/27/15
Fri4/24/15
Mon4/27/15
Mon12/21/15
Fri12/18/15
Mon12/21/15
Mon4/27/15
Fri4/24/15
Mon4/27/15
Mon7/27/15
Fri7/24/15
Mon7/27/15
Mon11/30/15
Mon10/5/15
Wed3/25/15
Wed3/25/15
Mon4/6/15
Thu1/22/15
Thu1/15/15
Thu3/26/15
Mon3/30/15
Wed4/1/15
Thu3/19/15
Tue7/15/14
Mon3/21/16
Finish
Page5
Mon9/14/15
Mon9/14/15
Mon10/26/15
Wed3/4/15
Wed3/4/15
Wed4/15/15
Wed3/4/15
Wed3/4/15
Wed4/15/15
Mon3/16/15
Mon3/16/15
Mon4/27/15
Mon11/9/15
Mon11/9/15
Mon12/21/15
Mon3/16/15
Mon3/16/15
Mon4/27/15
Mon6/15/15
Mon6/15/15
Mon7/27/15
Wed3/25/15
Mon10/5/15
Wed3/25/15
Wed3/25/15
Mon4/6/15
Fri1/16/15
Thu1/15/15
Fri3/20/15
Fri3/27/15
Tue3/31/15
Thu2/19/15
5days
1day
5days
2days
2days
21days
31days
30days
1day
31days
30days
1day
31days
30days
1day
31days
30days
1day
31days
30days
1day
31days
30days
1day
31days
30days
1day
179days
1day
1day
1day
1day
Mon3/7/16
Start
11days
Duration
AftereachtestadminDatestobenegotiated
FinishdatefromITN
FinishdatefromITN
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Notes
249
250
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
ID
WinterEOC
SpringEOC
SummerEOC
PackagingandDistributionSpecifications
FallELA/LRetake
Draftpackaginganddistributionspecs
Reviewspecs
Applyedits
Finalize/approvespecs
SpringELA/LRetake
Reviewspecs
Applyedits
Reviewspecs
Applyedits
FallEOC
Reviewspecs
Applyedits
WinterEOC
Reviewspecs
Applyedits
SpringEOC
Reviewspecs
Applyedits
SummerEOC
Reviewspecs
Applyedits
TestAdministrationManuals(includingpackinglists,work
folders,applicablememos,etc.)
ELA/LRetake
DraftTAM
TaskName
Fri7/17/15
Mon5/11/15
Tue8/11/15
Mon7/20/15
Wed7/29/15
Fri8/7/15
Tue8/11/15
Mon1/5/15
Fri12/12/14
Tue12/23/14
Thu1/1/15
Mon1/5/15
Thu1/29/15
Wed1/7/15
Fri1/16/15
Tue1/27/15
Thu1/29/15
Tue7/14/15
Mon6/22/15
Wed7/1/15
Fri7/10/15
Tue7/14/15
Tue10/6/15
Mon9/14/15
Wed9/23/15
Fri10/2/15
Tue10/6/15
Thu1/29/15
Wed1/7/15
Fri1/16/15
Tue1/27/15
Thu1/29/15
Tue5/12/15
Mon4/20/15
Wed4/29/15
Fri5/8/15
Tue5/12/15
Mon11/30/15
Mon4/6/15
Mon7/6/15
Finish
Page6
Mon5/11/15
Mon5/11/15
Mon7/20/15
Mon7/20/15
Tue7/21/15
Thu7/30/15
Mon8/10/15
Fri12/12/14
Fri12/12/14
Mon12/15/14
Wed12/24/14
Fri1/2/15
Wed1/7/15
Wed1/7/15
Thu1/8/15
Mon1/19/15
Wed1/28/15
Mon6/22/15
Mon6/22/15
Tue6/23/15
Thu7/2/15
Mon7/13/15
Mon9/14/15
Mon9/14/15
Tue9/15/15
Thu9/24/15
Mon10/5/15
Wed1/7/15
Wed1/7/15
Thu1/8/15
Mon1/19/15
Wed1/28/15
Mon4/20/15
Mon4/20/15
Tue4/21/15
Thu4/30/15
Mon5/11/15
17days
1day
7days
7days
2days
17days
1day
7days
7days
2days
17days
1day
7days
7days
2days
17days
1day
7days
7days
2days
17days
1day
7days
7days
2days
17days
1day
7days
7days
2days
17days
1day
7days
7days
2days
50days
1day
Mon11/30/15
Mon4/6/15
Mon7/6/15
Start
1day
1day
1day
Duration
Contigentuponfinal,approvedadmindatesDatestobenegotiated
PriortoeachtestadminDatestobenegotiated
Notes
Page 7.1-17
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
ID
Departmentreview#1
UpdateTAM#1
Departmentreview#2
UpdateTAM#2
Departmentreview#3
UpdateTAM#3
Departmentreview#4
UpdateTAM#4
Finalize/approveTAM
ELA/LandMathematicsComputerBasedTests
DraftTAM
Departmentreview#1
UpdateTAM#1
Departmentreview#2
UpdateTAM#2
Departmentreview#3
UpdateTAM#3
Departmentreview#4
UpdateTAM#4
Finalize/approveTAM
ELA/LandMathematicsPaperBasedTest
DraftTAM
Departmentreview#1
UpdateTAM#1
Departmentreview#2
UpdateTAM#2
Departmentreview#3
UpdateTAM#3
Departmentreview#4
UpdateTAM#4
Finalize/approveTAM
EOC
DraftTAM
Departmentreview#1
UpdateTAM#1
Departmentreview#2
UpdateTAM#2
Departmentreview#3
UpdateTAM#3
Departmentreview#4
UpdateTAM#4
Finalize/approveTAM
TaskName
7days
5days
7days
5days
7days
5days
7days
5days
1day
50days
1day
7days
5days
7days
5days
7days
5days
7days
5days
1day
50days
1day
7days
5days
7days
5days
7days
5days
7days
5days
1day
50days
1day
7days
5days
7days
5days
7days
5days
7days
5days
1day
Duration
Page 7.1-18
Wed5/20/15
Wed5/27/15
Fri6/5/15
Fri6/12/15
Tue6/23/15
Tue6/30/15
Thu7/9/15
Thu7/16/15
Fri7/17/15
Tue1/6/15
Wed10/29/14
Fri11/7/14
Fri11/14/14
Tue11/25/14
Tue12/2/14
Thu12/11/14
Thu12/18/14
Mon12/29/14
Mon1/5/15
Tue1/6/15
Tue1/6/15
Wed10/29/14
Fri11/7/14
Fri11/14/14
Tue11/25/14
Tue12/2/14
Thu12/11/14
Thu12/18/14
Mon12/29/14
Mon1/5/15
Tue1/6/15
Tue1/6/15
Wed10/29/14
Fri11/7/14
Fri11/14/14
Tue11/25/14
Tue12/2/14
Thu12/11/14
Thu12/18/14
Mon12/29/14
Mon1/5/15
Tue1/6/15
Finish
Page7
Tue5/12/15
Thu5/21/15
Thu5/28/15
Mon6/8/15
Mon6/15/15
Wed6/24/15
Wed7/1/15
Fri7/10/15
Fri7/17/15
Wed10/29/14
Wed10/29/14
Thu10/30/14
Mon11/10/14
Mon11/17/14
Wed11/26/14
Wed12/3/14
Fri12/12/14
Fri12/19/14
Tue12/30/14
Tue1/6/15
Wed10/29/14
Wed10/29/14
Thu10/30/14
Mon11/10/14
Mon11/17/14
Wed11/26/14
Wed12/3/14
Fri12/12/14
Fri12/19/14
Tue12/30/14
Tue1/6/15
Wed10/29/14
Wed10/29/14
Thu10/30/14
Mon11/10/14
Mon11/17/14
Wed11/26/14
Wed12/3/14
Fri12/12/14
Fri12/19/14
Tue12/30/14
Tue1/6/15
Start
Notes
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
293
ID
PaperBasedTestManuals(includingancillarymaterials
andaccommodations)
FallELA/LRetake
Draftpaperbasedtestmanual
Departmentreview#1
Update#1
Departmentreview#2
Update#2
Departmentreview#3
Update#3
Departmentreview#4
Update#4
Finalize/approvepaperbasedtestmanual
SpringELA/LRetake
Departmentreview#1
Update#1
Departmentreview#2
Update#2
Departmentreview#3
Update#3
Departmentreview#4
Update#4
Departmentreview#1
Update#1
Departmentreview#2
Update#2
Departmentreview#3
Update#3
Departmentreview#4
Update#4
FallEOC
Departmentreview#1
Update#1
Departmentreview#2
Update#2
Departmentreview#3
Update#3
TaskName
50days
1day
7days
5days
7days
5days
7days
5days
7days
5days
1day
50days
1day
7days
5days
7days
5days
7days
5days
7days
5days
1day
50days
1day
7days
5days
7days
5days
7days
5days
7days
5days
1day
50days
1day
7days
5days
7days
5days
7days
5days
Duration
Fri7/17/15
Mon5/11/15
Wed5/20/15
Wed5/27/15
Fri6/5/15
Fri6/12/15
Tue6/23/15
Tue6/30/15
Thu7/9/15
Thu7/16/15
Fri7/17/15
Thu12/11/14
Fri10/3/14
Tue10/14/14
Tue10/21/14
Thu10/30/14
Thu11/6/14
Mon11/17/14
Mon11/24/14
Wed12/3/14
Wed12/10/14
Thu12/11/14
Tue1/6/15
Wed10/29/14
Fri11/7/14
Fri11/14/14
Tue11/25/14
Tue12/2/14
Thu12/11/14
Thu12/18/14
Mon12/29/14
Mon1/5/15
Tue1/6/15
Fri6/19/15
Mon4/13/15
Wed4/22/15
Wed4/29/15
Fri5/8/15
Fri5/15/15
Tue5/26/15
Tue6/2/15
Finish
Page8
Mon5/11/15
Mon5/11/15
Tue5/12/15
Thu5/21/15
Thu5/28/15
Mon6/8/15
Mon6/15/15
Wed6/24/15
Wed7/1/15
Fri7/10/15
Fri7/17/15
Fri10/3/14
Fri10/3/14
Mon10/6/14
Wed10/15/14
Wed10/22/14
Fri10/31/14
Fri11/7/14
Tue11/18/14
Tue11/25/14
Thu12/4/14
Thu12/11/14
Wed10/29/14
Wed10/29/14
Thu10/30/14
Mon11/10/14
Mon11/17/14
Wed11/26/14
Wed12/3/14
Fri12/12/14
Fri12/19/14
Tue12/30/14
Tue1/6/15
Mon4/13/15
Mon4/13/15
Tue4/14/15
Thu4/23/15
Thu4/30/15
Mon5/11/15
Mon5/18/15
Wed5/27/15
Start
Notes
Page 7.1-19
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
ID
Departmentreview#4
Update#4
WinterEOC
Departmentreview#1
Update#1
Departmentreview#2
Update#2
Departmentreview#3
Update#3
Departmentreview#4
Update#4
SpringEOC
Departmentreview#1
Update#1
Departmentreview#2
Update#2
Departmentreview#3
Update#3
Departmentreview#4
Update#4
SummerEOC
Departmentreview#1
Update#1
Departmentreview#2
Update#2
Departmentreview#3
Update#3
Departmentreview#4
Update#4
AdditionalManuals/UserGuides
TIDEUserGuide
DraftUserGuide
InternalReview
DepartmentReview
UpdateUserGuide
TaskName
Page 7.1-20

Fri10/3/14
Fri10/3/14
Fri10/10/14
Fri10/17/14
Fri10/24/14
20days
5days
5days
5days
3days
Thu10/30/14
Thu10/9/14
Thu10/16/14
Thu10/23/14
Tue10/28/14
Thu6/11/15
Thu6/18/15
Fri6/19/15
Fri9/11/15
Mon7/6/15
Wed7/15/15
Wed7/22/15
Fri7/31/15
Fri8/7/15
Tue8/18/15
Tue8/25/15
Thu9/3/15
Thu9/10/15
Fri9/11/15
Tue1/6/15
Wed10/29/14
Fri11/7/14
Fri11/14/14
Tue11/25/14
Tue12/2/14
Thu12/11/14
Thu12/18/14
Mon12/29/14
Mon1/5/15
Tue1/6/15
Fri4/17/15
Mon2/9/15
Wed2/18/15
Wed2/25/15
Fri3/6/15
Fri3/13/15
Tue3/24/15
Tue3/31/15
Thu4/9/15
Thu4/16/15
Fri4/17/15
Finish
Page9
Wed6/3/15
Fri6/12/15
Fri6/19/15
Mon7/6/15
Mon7/6/15
Tue7/7/15
Thu7/16/15
Thu7/23/15
Mon8/3/15
Mon8/10/15
Wed8/19/15
Wed8/26/15
Fri9/4/15
Fri9/11/15
Wed10/29/14
Wed10/29/14
Thu10/30/14
Mon11/10/14
Mon11/17/14
Wed11/26/14
Wed12/3/14
Fri12/12/14
Fri12/19/14
Tue12/30/14
Tue1/6/15
Mon2/9/15
Mon2/9/15
Tue2/10/15
Thu2/19/15
Thu2/26/15
Mon3/9/15
Mon3/16/15
Wed3/25/15
Wed4/1/15
Fri4/10/15
Fri4/17/15
Start
7days
5days
1day
50days
1day
7days
5days
7days
5days
7days
5days
7days
5days
1day
50days
1day
7days
5days
7days
5days
7days
5days
7days
5days
1day
50days
1day
7days
5days
7days
5days
7days
5days
7days
5days
1day
Duration
Notes
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
ID
Finalize/approveUserGuide
PostUserGuide
TDSUserGuide
DraftUserGuide
InternalReview
DepartmentReview
UpdateUserGuide
PostUserGuide
ORSUserGuide
DraftUserGuide
InternalReview
DepartmentReview
UpdateUserGuide
PostUserGuide
DeliverandMaintainOnlineManagementSystemReport
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
TDS
TDSConfigurationSpecifications
DraftTDSconfigspecs
Departmentreview
Finalize/approveTDSconfigspecs
ConfigurationofTDSandrelatedsystems
TDStestplan
DeployTDS
AIRTestingandResolution
DeliverComputerBasedAssessmentPlatformReport
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
SecureBrowser
TaskName

Tue7/28/15
Tue7/28/15
Tue7/28/15
Tue7/28/15
Tue7/28/15
Tue7/28/15
Tue7/28/15
Mon11/3/14
Mon11/17/14
Mon12/1/14
Mon12/15/14
Mon1/12/15
Mon2/9/15
Mon2/16/15
5days
5days
5days
5days
5days
5days
5days
10days
10days
10days
20days
20days
5days
5days
Mon8/3/15
Mon8/3/15
Mon8/3/15
Mon8/3/15
Mon8/3/15
Mon8/3/15
Mon8/3/15
Fri11/14/14
Fri11/28/14
Fri12/12/14
Fri1/9/15
Fri2/6/15
Fri2/13/15
Fri2/20/15
Mon8/3/15
Mon8/3/15
Mon8/3/15
Mon8/3/15
Mon8/3/15
Mon8/3/15
Mon8/3/15
Wed10/29/14
Thu10/30/14
Thu10/30/14
Thu10/9/14
Thu10/16/14
Thu10/23/14
Tue10/28/14
Wed10/29/14
Thu10/30/14
Thu10/30/14
Thu10/9/14
Thu10/16/14
Thu10/23/14
Tue10/28/14
Wed10/29/14
Thu10/30/14
Finish
Page10
Wed10/29/14
Thu10/30/14
Fri10/3/14
Fri10/3/14
Fri10/10/14
Fri10/17/14
Fri10/24/14
Wed10/29/14
Thu10/30/14
Fri10/3/14
Fri10/3/14
Fri10/10/14
Fri10/17/14
Fri10/24/14
Wed10/29/14
Thu10/30/14
Start
1day
1day
20days
5days
5days
5days
3days
1day
1day
20days
5days
5days
5days
3days
1day
1day
Duration
FinishdatefromITN
FinishdatefromITN
FinishdatefromITN
FinishdatefromITN
FinishdatefromITN
FinishdatefromITN
FinishdatefromITN
Notes
Page 7.1-21
Page 7.1-22
PERIODIC:Inventory,store,anddestroyusedandunusedtest
materialsasdescribedinthecontract.
ReportsandRecommendationsAllOnlineCommentForms
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
PERIODIC:DelivertheInfrastructureTrialTest
Forms:DeliverComputerBasedTestForms(including
accommodatedversions)
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
436
437
438
439
440
441
442
443
444
445
446
447
448
449
435
433
432
431
434
Duration
Configuresecurebrowser
5days
Buildandtestsecurebrowser
10days
SecureBrowserUAT
5days
SecureBrowserGoLiveDate
1day
Portal
AIRcreatesnewportalperDepartmentspecs
25days
PortalUAT
10days
AIRimplementschanges
10days
Finalize/approveportal
5days
Portalgolive
1day
TrainingandPracticeTests
PERIODIC:Developanddeliverapprovedpracticetestsand
tutorialsforthecomputerbasedandpaperbased
assessments(sampletestmaterialsandcomputerbased
practicetests).
PERIODIC:Developanddeliveratrainingplanfortheinitial
trainingandsupplymaterialstosupportongoingtraining
fortheaudiencesidentifiedinthecontract.
PERIODIC:Developanddelivermaterialstosupport
ongoingtrainingfortheaudiencesidentifiedinthe
contract.
PERIODIC:Providecustomerservicethroughouttheyear,
withadditionalcustomerserviceprovidedatpeaktimes
leadinguptoandfollowingeachtestadministration.
TaskName
419
420
421
422
423
424
425
426
427
428
429
430
ID
Thu9/25/14
Thu10/9/14
Thu10/23/14
Thu10/30/14
Fri10/31/14
Fri8/22/14
Fri9/26/14
Fri10/10/14
Fri10/24/14
Fri10/31/14
Page11
Fri8/1/14
Mon11/2/15
Mon11/2/15
Mon8/3/15
Mon9/1/14
Mon11/3/14
Fri10/3/14
Fri10/17/14
Fri10/24/14
Mon10/27/14
Finish
Mon9/29/14
Mon10/6/14
Mon10/20/14
Mon10/27/14
Start
Notes
TaskName
482
483
484
485
486
487
488
489
490
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
Page12
Wed2/25/15
Wed1/28/15
Wed2/25/15
Mon1/12/15
13days
Wed10/29/14
Wed11/5/14
Thu11/6/14
Wed11/12/14
Thu1/22/15
Mon2/2/15
Wed2/4/15
Thu2/5/15
Tue2/3/15
Fri2/20/15
Mon2/23/15
Tue2/24/15
Wed10/29/14
Wed11/5/14
Thu11/6/14
Wed11/12/14
Fri1/9/15
1day
1day
1day
1day
10days
Thu11/20/14
Thu11/27/14
Fri11/28/14
Fri12/12/14
Tue12/16/14
Wed12/17/14
Wed12/24/14
Mon12/29/14
Tue12/30/14
Tue4/1/14
Finish
Mon2/2/15
Wed2/4/15
Thu2/5/15
Tue2/3/15
Fri2/20/15
Mon2/23/15
Tue2/24/15
Fri11/14/14
Fri11/21/14
Fri11/28/14
Mon12/1/14
Mon12/15/14
Wed12/17/14
Thu12/18/14
Thu12/25/14
Tue12/30/14
Start
5days
5days
1day
10days
2days
1day
5days
3days
1day
Duration
Order/PreIDfiletoDRC
1day
OrderconfirmationfilefromDRCtoAIR
1day
AIRsignoffonDRCorderconfirmationfile
1day
NewschoolstoDRCforcalculators
1day
AIRdeliversdistrictschoolassociationfiletoDRC
1day
MaterialsshipmttrackingfilesfromDRCtoloadintoTIDE1day
MaterialsshipmttrackingfilesloadedintoTIDE
1day
TIDElatePreID
BeginacceptinglatePreIDrecords
1day
Specificationsdocument
TIDE
ConfigurationSpecifications
DraftTIDEconfigspecs
Departmentreview
Finalize/approveTIDEspecs
ConfigureTIDE
TIDEtestplan
DeployTIDE
AIRTestingofTIDEcomponents
UserAcceptanceTestingandResolution
TIDEGoesLive
TIDEOrder/PreIDSystemConfiguration
Materialslist/calculationsfromDRC
SpringfailcountsfromSAS
PreloadinformationduetoTIDEteam
TIDEspecsduetoTIDEteam
UATfortheentireadministration
TIDEontimeorderandPreID
OntimeTIDEorderandPreIDwindow
TIDEactivitiesfollowingthecloseoftheontimewindow
450
SpringEOC
451
SummerEOC
452
ReturnofUsedandUnusedMaterials
453
FallELA/LRetake
454
SpringELA/LRetake
455
456
FallEOC
457
WinterEOC
458
SpringEOC
459
SummerEOC
460 ScoringandReporting
461
PERIODIC:DevelopAnnualScoringandReporting
ID
Notes
Page 7.1-23
Page 7.1-24
527
528
529
530
531
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
ID
FinaldayforacceptinglatePreIDrecords
LatePreIDrecordstoDRC
LatePreIDnumbersfromDRCforTIDE
PreIdentification:Deliverlablesandrosters
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
PreIDFreqReportsDevelopment
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
FileLayoutandFullStatePreIDDataFileDevelopment
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
Developmentofthespecialdocumentstranscriptionlogs
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
Earlydistrictandfullstatescan/editandCBTextractdata
files:Developmentofthefilelayouts,scan/editandCBT
extractdatafiles
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
TaskName
Mon10/5/15
Fri2/27/15
Wed3/25/15
Mon9/7/15
Mon11/30/15
Wed3/25/15
Mon7/6/15
2days
2days
2days
2days
2days
2days
2days
Thu10/1/15
Tue3/1/16
Sun3/1/15
Thu10/1/15
Tue12/1/15
Thu10/1/15
Sun5/1/16
Fri5/1/15
Thu10/1/15
Tue12/1/15
Fri5/1/15
Wed7/1/15
Tue9/1/15
Fri1/1/16
Thu1/1/15
Tue9/1/15
Thu10/1/15
Thu1/1/15
Mon6/1/15
Tue9/1/15
Fri1/1/16
Thu1/1/15
Tue9/1/15
Thu10/1/15
Thu1/1/15
Mon6/1/15
Tue10/6/15
Mon3/2/15
Thu3/26/15
Tue9/8/15
Tue12/1/15
Thu3/26/15
Tue7/7/15
Thu2/26/15
Fri2/27/15
Mon3/2/15
Finish
Page13
Thu2/26/15
Fri2/27/15
Mon3/2/15
Start
1day
1day
1day
Duration
Notes
562
563
564
565
566
567
568
554
555
556
557
558
559
560
561
546
547
548
549
550
551
552
553
532
533
534
535
536
537
538
539
540
541
542
543
544
545
ID
SpringEOC
SummerEOC
ORS
ConfigurationSpecifications
DraftORSconfigspecs
Departmentreview
Finalize/approveORSspecs
ConfigureORS
ORStestplan
DeployORS
AIRTestingofORSandResolution
UserAcceptanceTestingandResolution
ORSGoesLive
Anomalousscoredatafiles:Developmentoftheanomalous
scoredatafiles
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
InitialreportingSSRsandDSRsdatafiles:Developmentof
SSR/DSRfilelayoutsandmultipleversionsoftheSSRand
DSRdatafiles.
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
InitialreportingSARsandDARsdatafiles:Developmentof
SAR/DARfilelayoutsandmultipleversionsoftheSARand
DARdatafiles.
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
TaskName
5days
5days
5days
4days
4days
5days
5days
5days
1day
Duration
Thu10/1/15
Fri4/1/16
Wed4/1/15
Thu10/1/15
Tue12/1/15
Wed4/1/15
Wed7/1/15
Thu10/1/15
Fri4/1/16
Wed4/1/15
Thu10/1/15
Tue12/1/15
Wed4/1/15
Wed7/1/15
Thu10/1/15
Fri4/1/16
Wed4/1/15
Thu10/1/15
Tue12/1/15
Wed4/1/15
Wed7/1/15
Fri12/19/14
Fri12/26/14
Fri1/2/15
Thu1/8/15
Wed1/14/15
Wed1/21/15
Wed1/28/15
Wed2/4/15
Thu2/5/15
Sun3/1/15
Wed7/1/15
Finish
Page14
Mon12/15/14
Mon12/22/14
Mon12/29/14
Mon1/5/15
Fri1/9/15
Thu1/15/15
Thu1/22/15
Thu1/29/15
Thu2/5/15
Start
Notes
Page 7.1-25
Page 7.1-26
595
596
597
598
599
600
601
602
587
588
589
590
591
592
593
594
586
578
579
580
581
582
583
584
585
570
571
572
573
574
575
576
577
569
ID
LatereportingSSRsandDSRsdatafiles:Developmentof
SSR/DSRandmultipleversionsofDARdatafiles.
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
CloseoutSSRdatafiles:DevelopmentofthecloseoutSSR
datafilesforallassessments.
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
Periodic:DevelopmentoftheDemographicDSR/DARdata
files
InitialreportingfilesandPDFsofreportsfordistrictsand
schools:DevelopmentofthedatafilesandPDFsofreports
fortheinitialreporting(RD13)
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
InitialreportingStudentReportsandCD/DVDsorjump
drives.Submitproofofdelivery:Developmentofthe
studentreportsandmediafortheinitialreporting(RD4)
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
PerformanceTaskReportsandproofofdelivery:
DevelopmentofthehardcopyofthePerformanceTasks
reports
TaskName
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
Duration
Mon11/23/15
Mon5/18/15
Wed5/13/15
Mon10/26/15
Mon1/18/16
Wed5/13/15
Mon8/24/15
Thu11/12/15
Thu5/7/15
Mon5/4/15
Thu10/15/15
Thu1/7/16
Mon5/4/15
Thu8/13/15
Tue12/1/15
Sat10/1/16
Thu10/1/15
Tue12/1/15
Sun3/1/15
Thu10/1/15
Thu10/1/15
Tue9/1/15
Sun11/1/15
Wed6/1/16
Mon6/1/15
Sun11/1/15
Sun2/1/15
Mon6/1/15
Thu10/1/15
Finish
Page15
Mon11/23/15
Mon5/18/15
Wed5/13/15
Mon10/26/15
Mon1/18/16
Wed5/13/15
Mon8/24/15
Thu11/12/15
Thu5/7/15
Mon5/4/15
Thu10/15/15
Thu1/7/16
Mon5/4/15
Thu8/13/15
Start
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Notes
635
636
637
638
639
640
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
611
612
613
614
615
616
617
618
603
604
605
606
607
608
609
610
ID
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
LatereportingStudentReportsandCD/DVDs.Submitproof
ofdelivery:Developmentofthestudentreportsandmedia
forthelatereporting(RD5)
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
FinalMockPreIDlabel/Roster,datafiles,Media&Reports:
DevelopmentofthemockPreIDlabel/Roster,Mockdata
filesandreports/mediasformockdata
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
HandscoringofPerformanceTasks
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
FinalReportMockups:Developmentofthereportmockups
forallassessments
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
TaskName
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
Duration
Mon6/1/15
Fri1/1/16
Thu1/1/15
Mon6/1/15
Thu10/1/15
Thu1/1/15
Tue9/1/15
Tue3/1/16
Sun3/1/15
Tue9/1/15
Sun11/1/15
Sun3/1/15
Mon6/1/15
Mon3/28/16
Mon3/3/14
Wed9/16/15
Mon2/29/16
Mon5/23/16
Wed9/16/15
Mon12/28/15
Mon11/23/15
Mon5/18/15
Wed5/13/15
Mon10/26/15
Mon1/18/16
Wed5/13/15
Mon8/24/15
Finish
Page16
Mon3/28/16
Mon3/3/14
Wed9/16/15
Mon2/29/16
Mon5/23/16
Wed9/16/15
Mon12/28/15
Mon11/23/15
Mon5/18/15
Wed5/13/15
Mon10/26/15
Mon1/18/16
Wed5/13/15
Mon8/24/15
Start
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Datestobenegotiated
Notes
Page 7.1-27
Page 7.1-28
Periodic:DevelopmentofTestFormsforFieldTestand/or
OperationalAdministrations
Periodic:DevelopmentoftheSamplingPlans
Periodic:DrawingSamplesforOperationaland/orFieldTest
Administrations
Periodic:DevelopmentoftheSESSpecifications
Periodic:PerformingSESActivities
SpringELA/LandMathematics:DevelopCalibrationSampling
Specifications
SpringEOC:DevelopCalibrationSamplingSpecifications
Periodic:CreatingandFinalizingTechnicalReports
Periodic:DevelopmentofScaleLinkingPlan:ScaleLinking
PlanfortransitioningfromFCAT2.0Reading&Mathematics
andEOCAlgebra1&GeometrytotheFloridaStandards
Assessments
664
670
671
672
667
668
669
665
666
662
663
654
655
656
657
658
659
660
661
651
652
653
643
644
645
646
647
648
649
650
SummerEOC
DevelopmentofthehistoryfilesforEOCand
ELA/Mathematics
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
DevelopmentoftheAPS(AlternatePassingScore)filefor
Retake
FallELA/LRetake
SpringELA/LRetake
ManualRescoreLog:Developmentofthelogtorecordthe
PBTtestwithpassingscoresformanualrescoring
FallELA/LRetake
SpringELA/LRetake
FallEOC
WinterEOC
SpringEOC
SummerEOC
Periodic:WritingImageCD/DVDorjumpdrivesandproofto
delivery:DevelopmentoftheWritingImageCD/DVDorjump
drives.
Scaling,Equating,Scoring,andSpecialPsychometricStudies
Periodic:DevelopmentoftheTestConstructionSpecifications
TaskName
641
642
ID
Duration
Start
Page17
Tue12/1/15
Sat11/1/14
Mon12/1/14
Fri5/1/15
Tue7/1/14
Fri8/1/14
Tue7/1/14
Thu5/1/14
Thu10/1/15
Fri4/1/16
Wed4/1/15
Thu10/1/15
Tue12/1/15
Wed4/1/15
Wed7/1/15
Wed7/1/15
Sun11/1/15
Sun5/1/16
Tue12/1/15
Sat10/1/16
Thu10/1/15
Tue12/1/15
Tue3/1/16
Thu10/1/15
Thu10/1/15
Wed4/1/15
Finish
Datestobenegotiated
Datestobenegotiated
Notes
688
689
690
691
692
693
694
695
696
697
698
699
700
687
686
685
684
683
682
680
681
677
678
679
675
676
674
673
ID
Periodic:LinkingFloridaStandardsAssessmentsto
InternationalAssessments
ProgramManagement
Periodic:ProductionScheduleInitialSchedule
Draft
FDOEreview
Scheduleupdated
Scheduleapproved
Periodic:ProgramManagementMeetings
ComprehensiveProgramManagementmeeting
ComprehensiveProgramManagementmeeting1
Periodic:DevelopmentofStandardSettingWorkPlan
Periodic:ImplementAchievementLevelStandardSetting
Meetings
Periodic:FinalizingStandardSettingTechnicalReport
Periodic:DevelopmentofVerticalScalingSpecifications
Periodic:PerformingVerticalScalingActivitiesandPreparing
TechnicalReport
Periodic:DevelopmentofScaleDriftStudySpecifications
Periodic:PerformingScaleDriftResearchandPreparing
TechnicalReport
Periodic:DevelopmentofSpecificationsforComparability
BetweenDevicesDeliveringAssessments
Periodic:PerformingStudyforComparabilityBetween
DevicesDeliveringAssessments,PreparingitsFinalReport,
andRecommendationswithTACCommittee
Periodic:DevelopmentofastudyplanforlinkingFlorida
StandardsAssessmentstootherassessmentsmeasuringthe
samestandards
Periodic:LinkingFloridaStandardsAssessmentstoother
assessments
Periodic:DevelopmentofastudyplanforthelinkingFlorida
StandardsAssessmentstoInternationalAssessments
Periodic:DevelopmentofScaleLinkingPlan:CollectingLegal
Opinion
Periodic:LinkingScalesduringtransitioningyearstofindout
FCAT2.0andEOCequivalentsbasedontheperformanceof
studentsontheFloridaStandardsAssessments
TaskName

Wed3/12/14
Wed3/12/14
Fri9/12/14
Thu3/12/15
Sat9/12/15
Sat3/12/16
915days
1day
1day
1day
1day
1day
Tue9/12/17
Wed3/12/14
Fri9/12/14
Thu3/12/15
Sat9/12/15
Sat3/12/16
Mon3/31/14
Mon4/7/14
Mon4/14/14
Tue4/15/14
Mon8/1/16
Sun11/1/15
Mon8/1/16
Sun11/1/15
Sat8/1/15
Sat11/1/14
Sat7/1/17
Wed8/1/18
Sun11/1/15
Sat11/1/14
Sat8/1/15
Thu1/1/15
Thu10/1/15
Fri5/1/15
Wed10/1/14
Finish
Page18
Mon3/31/14
Tue4/1/14
Tue4/8/14
Tue4/15/14
Start
1day
5days
5days
1day
Duration
Notes
Page 7.1-29
Page 7.1-30
739
939
940
941
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
ID
AnnualManagementMeeting
AnnualManagementMeeting1
Periodic:ProgramManagementStaff
QuarterlyStaffingreport
QuarterlyStaffingreport1
Periodic:ProgramManagementReports:Weekly
ManagementandActionItemReport
WeelyProgramManagementreport
Periodic:AnnualProgramReview
Periodic:AnnualProgramReview1
TaskName
991days
522days
1day
1day
1day
1day
1day
980days
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
915days
915days
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
1day
991days
Duration
Tue12/26/17
Sun11/12/17
Thu11/12/15
Sat11/12/16
Mon9/12/16
Sun3/12/17
Tue9/12/17
Tue12/12/17
Wed3/12/14
Thu6/12/14
Fri9/12/14
Fri12/12/14
Thu3/12/15
Fri6/12/15
Sat9/12/15
Sat12/12/15
Sat3/12/16
Sun6/12/16
Mon9/12/16
Mon12/12/16
Sun3/12/17
Mon6/12/17
Tue9/12/17
Tue12/12/17
Sun11/26/17
Sun11/26/17
Mon5/26/14
Tue8/26/14
Wed11/26/14
Thu2/26/15
Tue5/26/15
Wed8/26/15
Thu11/26/15
Fri2/26/16
Thu5/26/16
Fri8/26/16
Sat11/26/16
Sun2/26/17
Fri5/26/17
Sat8/26/17
Sun11/26/17
Tue12/26/17
Finish
Page19
Tue3/11/14
Thu11/12/15
Thu11/12/15
Sat11/12/16
Mon9/12/16
Sun3/12/17
Tue9/12/17
Wed3/12/14
Wed3/12/14
Thu6/12/14
Fri9/12/14
Fri12/12/14
Thu3/12/15
Fri6/12/15
Sat9/12/15
Sat12/12/15
Sat3/12/16
Sun6/12/16
Mon9/12/16
Mon12/12/16
Sun3/12/17
Mon6/12/17
Tue9/12/17
Tue12/12/17
Mon5/26/14
Mon5/26/14
Mon5/26/14
Tue8/26/14
Wed11/26/14
Thu2/26/15
Tue5/26/15
Wed8/26/15
Thu11/26/15
Fri2/26/16
Thu5/26/16
Fri8/26/16
Sat11/26/16
Sun2/26/17
Fri5/26/17
Sat8/26/17
Sun11/26/17
Tue3/11/14
Start
Notes
Periodic:FinalDeliveryofMaterials
Destructionofallelectronicfilesandusedprintcopies
944
TaskName
942
943
ID
1day?
1day
1day
Duration
Wed4/18/18
Sun11/12/17
Wed11/1/17
Finish
Page20
Wed4/18/18
Sun11/12/17
Wed11/1/17
Start
11/1/17,11/1/18,11/1/19,or11/1/20dependingonwhetherbaseis
extended
Notes
Page 7.1-31

In the following sections, AIR presents our
proposal to provide Florida with a criterionreferenced achievement test aligned to and
measuring mastery of Florida Standards in
English Language Arts/Literacy (ELA/L) and
Mathematics proficiency. In our response, we
detail our approach to each requirement of the
ITN and offer innovative solutions for assessment, scoring, reporting, and psychometric
studies.
We recognize the following purposes and
objectives of the assessment:
1. Assess the achievement level and annual
learning gains of each student in ELA/L and
Mathematics
2. Provide data for making decisions regarding
school accountability, recognition, and
improvement of operations and management, including schools operating for the
purpose of providing educational services
to youth in Department of Juvenile Justice
programs
3. Identify the educational strengths and
needs of students and the readiness of students to be promoted to the next grade
level, to graduate from high school, and to
succeed in college and in careers
4. Assess how well educational goals and curricular standards are met at the school, district, and state level, and at the national
level, to the extent possible
The preceding section describes additional
uses of the FSA based on existing Florida policy, and how our proposed plans will meet
those requirements.
7.2.1 Expected Number of Districts,

Schools, and Students
We have prepared this proposal to include the
numbers of districts, schools, and students
identified in Appendix C of the ITN. This
translates into specific materials lists and
hand-scoring quantities, which we document
in Appendix B.
We recognize that the number of administration sites will increase by a projected 5 percent per year.
7.2.2 Potential Test Administration

Dates
Our project schedule presented in Section
7.1.2 presents proposed test windows for
20142015. As described earlier, we propose a
modified logistical plan that enables us to
move the demographic cleanup period to coincide with the testing window, shortening timelines for the main data delivery by weeks.
Exhibit 7.1.0-2, presented earlier, presents
approximate delivery dates required by
Floridas existing policies, and we are able to
meet those dates with the following administration schedule:
The ELA performance tasks will be administered in early March, consistent with the
testing window in the RFP.
The remainder of the tests will be administered during a 12-day window starting
approximately April 1, allowing nearly an
additional month of instruction before
assessment.
The grade 3 reading assessment, which
must be processed much more quickly, will
be entirely machine-scorable.
Page 7.2-1
We note that for reading and mathematics, this

allows nearly an additional month of instruction before testing. As more grades move
online, these dates can be pushed further back
toward the end of the year.
Page 7.2-2

In 2010, Florida adopted the Common Core
State Standards (CCSS) in English Language
Arts/Literacy (ELA/L) and Mathematics.
Florida seeks an assessment that will provide
an accurate and meaningful measurement of
student mastery in those standards without
significantly increasing the testing time beyond
that of the current state assessments.
AIR proposes to provide a statewide assessment system that will not only measure student achievement of Floridas standards but do
so with a delivery system that works within
the existing infrastructure in Florida and
which offers accessibility features that will
make the test accessible to virtually every student in the state. AIR has led the industry in
the development of online assessments that
provide efficient, accurate, and meaningful
results to stakeholders. We are proud of the
partnerships we have forged with our state
clients. These collaborative relationships,
along with our own research and development,
have afforded us opportunities to continue
improving the design of our assessments.
As described in the introduction, we offer the
Department an assessment based on one of
two complete and robust item banksone
drawn from the Student Assessment of Growth
and Excellence (SAGE) and the other developed by the Smarter Balanced Assessment
Consortium. Each of these item banks was
developed to support an adaptive assessment,
which means that each bank is designed to
cover the range of expected achievement in
each grade and contains many items. Florida
has the option of delivering either of these
assessments in its adaptive form or drawing
upon the robust banks to construct fixed forms.
In either case, we understand that it is
Floridas intent to develop items throughout
this contract, embedding them for field testing.
In all likelihood, these items will cause the

FSA to diverge from the base test over time.
Hence, in either case, the comparison to other
states would likely be through a linkage to
other states.
Whether the Department chooses to begin
with the SAGE item bank or the Smarter
Balanced item bank, we recommend that the
FSA be linked to the Smarter Balanced scale to
facilitate cross-state comparisons. If the FSA
begins with the Smarter Balanced assessment,
the linkage will occur naturally. If the FSA
begins with SAGE, the Department can license
the Smarter Balanced bank for a small number
of students (approximately 3,0004,000 per
grade) and intersperse a small number
(approximately 68) of items per students in
embedded field-test slots to link to the Smarter
Balanced scale.
Substantial public information is available on
the Smarter Balanced assessment. That assessment is somewhat more expensive to license
and includes more hand scoring (hence, requiring longer timelines and imposing higher costs)
than the SAGE. Therefore, this section focuses
more on the SAGE assessment, though we
encourage the department to consider administering the Smarter Balanced assessment.
SAGE item banks comprise approximately
300500 items per grade in both ELA/L and
Mathematics, respectively. These item banks
represent the full suite of online item types
AIR offers, including our innovative machinescored constructed-response (MSCR) items.
The SAGE ELA/L assessment is designed to
take approximately 90 minutes, which is significantly shorter than the 160-minute testing
time proposed in Appendix D of the ITN. The
test comprises Reading, Listening, and
Language items, which assess the full range of
content standards in the Common Core. The
Reading portion contains both single and
paired passages, with a range of informational
Page 7.3-1
and literary genres that reflect the ratio established by the Common Core (i.e., progressively
more informational passages in the upper
grades). Listening items are based on short
auditory stimuli. Editing items give students
the opportunity to edit short draft passages by
correcting errors in context.
Additionally, SAGE offers a writing component, based on two extended writing prompts.
The writing portion of the test is designed to
take approximately 60 minutes at the lower
grades and 90 minutes at the upper grades.
Consistent with Common Cores evidencebased approach to writing, the SAGE writing
test provides students with a series of short
stimuli around a common theme or topic.
These are used as a springboard for an authentic writing performance task. The testing times
ensure that students have enough time to read
the stimuli and craft a response that incorporates their research into their reasoning.
The SAGE Mathematics assessment is estimated to require less than 90 minutes. Like ELA/L
tests, Mathematics tests are not administered
in sessions, but they can be segmented to
allow students to access calculators only in
designated portions of the test.
SAGE Mathematics tests assess the breadth
and depth of the CCSS. Just as the CCSS align
across grades to form a coherent progression
of skills, so do the blueprints of the tests at
each grade, which represent an evolving hierarchy of mathematical skills. Each
Mathematics assessment also presents students
with a range of cognitively demanding items
(i.e., the depth of knowledge taxonomy) as
well as a variety of response mechanisms. In
fact, at least 20% of the items on each online
Mathematics assessment are MSCR items.
The SAGE requires 90 minutes or less for each
grade and subject (ELA/L comprises two subjectsReading and Writing) to administer the
Page 7.3-2
SAGE in its adaptive form. Hence, across

ELA/L and Mathematics, the entire battery of
tests takes approximately 270 minutes, or
approximately 4.5 hours. This compares favorably to either consortium assessment, as well
as the current Florida Comprehensive
Assessment Test 2.0 (FCAT 2.0). A fixedform version would require somewhat longer
assessments to cover the expected range of
performance. In no case would the ELA/L and
Mathematics assessment require more testing
time than Florida currently requires.
We note that it is possible for Florida to participate in the SAGE field test this year (spring
2014) and embed Florida-specific items in
embedded field-test slots next year (spring
2015).
Whatever solution Florida chooses for a firstyear item bank, AIR will work closely with the
Department to ensure that items are acceptable to both the Department as well as Floridas
stakeholders. We will hold item review meetings in advance of the field test so the
Department, educators, and community members can review and approve each test item
before it is presented to Floridas students.
In the next section we provide a detailed
description of the SAGE assessments and how
they meet Floridas requirements. Section
7.3.3 describes AIRs structured item-development process and introduces some of the tools
available to our test developers to create openended, machine-scored items that measure
deep knowledge while reducing costs and
speeding reporting.
7.3.1 Test Content

Our proposal offers Florida the choice between
two exceptional Common Core tests, with a
plan to link either to the Smarter Balanced
scale. The Smarter Balanced test specifications
are public and well publicized, so here we
focus on describing the SAGE test content. We

do so first for ELA/L, then for Mathematics.
ELA/L Test Content

Overview of the Next Generation
Sunshine State Standards in English
Language Arts/Literacy
In July of 2010, the Florida State Board of
Education adopted new standards in the
English language arts to place emphasis on
coherence, focus, and rigor for all grades.
These standards, the Next Generation
Sunshine State Standards (Common Core),
represent significant shifts in Floridas
approach to language arts content, which now
emphasizes
building knowledge through content-rich
nonfiction;
reading, writing, and speaking grounded in
evidence from text, both literary and informational; and
regular practice with complex text and its
academic language.
The Common Cores College and Career
Readiness Standards are divided into four
strands, or disciplines: Reading, Writing,
Speaking/Listening, and Language.
In Reading, the strands are organized into substrands, which include Reading Literature,
Reading Informational Text, and Reading
Foundations. Within each sub-strand is a skill
domain, which defines the focus of content within each strand. In Reading Literature and
Reading Informational text, the skill domains are
Key Ideas and Details;
Craft and Structure; and
Integration of Knowledge and Ideas.
Within each skill domain are standard statements, which define specifically what students
should know and be able to do at each
grade level.
The Common Core Writing standards define
the types of texts students should be able to
produce. These standards require that students
write arguments or opinion pieces and provide support;
write informative/explanatory texts to
examine a topic; and
write narratives to develop real or imagined
experiences or events.
At each grade band, these standards become
more complex. In kindergarten through grade 5,
there is an expectation that students will
demonstrate an increasingly sophisticated use
of language and syntax as well as a more coherent organization of ideas. Additionally, writing
should be based on more demanding content
sources. In grades 6 through 12, students
should show an increased sophistication in all
aspects of language use, including vocabulary
structure. Students should be able interact with,
and respond to, increasingly complex texts as
they formulate their ideas in the upper grades.
The Common Core Listening standards require
that students comprehend the main idea and
supporting details of a text that is read aloud
or presented in diverse media formats. As students progress through the grades, their listening skills should become more refined.
Students should become proficient at understanding an oral text as well as delineating a
speakers argument and evaluating the claims
that are supported by reasons and evidence.
Language standards are designed to gauge a
students ability to demonstrate a command of
the conventions of standard English grammar
and usage. The Common Core provides a clear
Page 7.3-3
progression of proficiency in language standards from grade to grade.

The item banks that AIR has developed for
SAGE are comprised of items that align to
these Common Core standards, assess higher
order thinking skills, and follow the item and
passage specifications that are described
below. Our development of SAGE has provided
opportunities to continue refining new item
types that are specifically designed to meet the
rigorous cognitive demands of the Common
Core. AIRs machine-scored constructedresponse items allow us to assess a students
ability to answer questions about a passage
and support those responses with text-based
evidence. (We refer readers to the English
Language Arts/Literacy sample items at the
end of this section.) Such innovations provide
a more thorough measurement of evidencebased literacy skills, which is the cornerstone
of the Next Generation Sunshine State
Standards in English Language Arts/Literacy.
based on the passage specificationsto perform an in-depth analysis of each passage. The
passage specifications call for a close examination of both quantitative measures, such as
word counts and Lexile readabilities, as well as
qualitative measures, such as passage structure
and levels of meaning, all of which are defined
as important measures of text complexity in
Appendix A of the Common Core: Research
Supporting Key Elements of the Standards.
Exhibit 7.3.1-1 describes the metrics we use
when we examine these qualitative measures.
The grade-level details in this exhibit provide a
description of the passage characteristics in literary texts that are designed for grades 9 and
10. Research supports this complexity rating
system, indicating that variations in both general and specific text characteristics impact a
readers ease of comprehension. By incorporating these research findings into the development of the passage specifications, we believe
we have developed a more accurate and thorough measure of text complexity.
Passage Selection and Specifications

The passages that AIR developed for SAGE
represent a variety of genres and topics. AIRs
content experts developed informational texts
from multiple content areas, such as history,
science, and technical subjects. SAGE literary
texts represent authentic pieces from multiple
genres, including stories, poetry, and drama.
The ratio of informational to literary texts
increases at each grade band with a greater
percentage of informational texts in the upper
grades. Finally, we developed both single passages as well as paired passages in which students are asked to synthesize information
across texts.
To ensure that all passages align to the correct
grade level and provide sufficient complexity
for close analytical reading, AIR developed
detailed passage specifications. Content
experts used passage complexity worksheets
Page 7.3-4
Exhibit 7.3.1-1: Passage Metrics

Difficulty Factor Passage Metric Description
Levels of meaning
in literature
1. Single, concrete interpretation

with few generalizations
necessary
2. Some themes not explicitly
stated
3. Multiple, successively abstract or
general, levels of meaning (key
theme or themes implied)
Grade-Level Details
(sample for grades 910)
1. a. Characters are static and characteristics are
explicitly stated.
b. Setting is used as an aesthetic enhancement, not as a way to convey meaning. Mood and
tone are used to enhance the setting of the story,
but are not critical in conveying the meaning or
theme
c. Actions have straightforward meanings and clear,
immediate effects.
d. Symbols are straightforward, common, and
closely linked to their meanings, both in terms of
proximity and explanatory language.
Example
2. a. Characters are dynamic and a single character
may have multiple motives.
b. Characteristics are implied through clear action or
dialogue.
c. Setting serves to underscore the theme and
conveys mood or tone, which supports
understanding of the explicit theme.
d. Actions have straightforward, explicit meanings,
but the effects are not fully realized until later in
the passage.
e. Symbols are straightforward and common, but
may not be supported by explanation or
elaboration (e.g., childrens bare feet symbolize
poverty, which is not explained but can be
deduced through context).
f. There may be some simple analogies or allusions
to other works.
3. a. Characters are complex with multiple motives
and/or inner conflicts.
b. Characterization is implied through subtle actions,
others reactions, and oblique dialogue.
c. The setting is used to reveal the theme.
d. Setting conveys mood or tone, which is crucial to
understanding the implicit theme.
e. Reader may need to understand historical
context to fully comprehend text.
f. Actions have subtle and/or complex meanings,
the effects of which may not be immediately
realized.
g. Symbols are complex, uncommon, and/or make
assumptions about students historical, scientific,
or literary knowledge.
h. There may be complex analogies or allusions to
other works.
Research-Based Evidence
Research shows that concrete passages are
more comprehensible and easier to recall
than abstract passages (Sadoski, Getz, &
Fritz, 1993).
Comprehension for concrete passages also
increases in relation to how easily the reader
can imagine the contents of the text (Riding
& Taylor, 1976).
Characterization, in particular, plays a role in
a texts difficulty. When a characters actions
are clearly linked to the characters
emotional state, the text is much more
readily comprehensible (Gillioz, Gygax, &
Tapiero, 2012).
Similarly, readers draw inferences from
descriptions of a characters actions and
stated preferences (i.e., descriptions of
specific traits as being either positive or
negative) (Mensink & Rapp, 2011).
However, when a character exhibits behavior
that is inconsistent with a perceived trait, the
characterization takes longer for readers to
process and comprehend (Sparks & Rapp,
2011).
An increase in dialogue between characters
has a similar effect, as tested readers
response times to items about dialogue
scenes were slower than for nondialogue
scenes (Long & De Ley, 2000).
Beyond-text inferences involving aspects of
stories such as morals, authors messages,
and relations to the readers lives proved the
most difficult for students (McConaughy,
1982).
The use of figurative language and meanings
also increases the difficulty of a text.
(Rommers, Dijkstra, & Bastiaansen, 2013).
It is easier to understand texts when their
words stand for their literal meanings.
Figurative language such as satire, irony, and
allusions are more difficult to interpret than
figurative language such as imagery or
metaphors (Fisher, Frey, & Lapp, 2012).
Page 7.3-5
Exhibit 7.3.1-1: Passage Metrics (continued)

Difficulty Factor Passage Metric Description
Structure
1. Clear, consistent narrative

structure; single point of view;
events in chronological order
2. One factor varies (structure, point
of view, chronology)
3. Two or more factors vary
Avoid requiring graphics for
comprehension for accessibility
reasons
Grade-Level Details
(sample for grades 910)
1. Story is presented in a straightforward fashion without
any shifts in time or narrator. At this grade level, this
includes significant digression into details and setting,
as long as the chronology is consistent.
2. a. Narrator shifts with a clear signal that it is doing
so.
b. Includes simple chronology shifts, such as clearly
introduced flashbacks or memories.
c. Structure varies with a mixture of prose and verse
OR progresses in a nonlinear fashion.
Example
3. a. Narrator
shifts but may not give a clear signal that
it is doing so.
b. Includes complex chronology shifts, such as
flashbacks or memories.
c. Structure varies with a mixture of prose and verse
OR progresses in a nonlinear fashion.
Language
1. Simple, common word choice;

explicit and literal use
2. May include unfamiliar
vocabulary; abstract meaning;
figurative, ironic, or sarcastic use
3. Generally dense, using figurative
or purposefully ambiguous, and
often unfamiliar, language
Research-Based Evidence
Research shows that texts structured in a
linear and/or hierarchical manner are easier
to comprehend (Calisir & Gurel, 2003).
There are a number of aspects of text
structure that affect the ease of comprehension, including shifts in perspective (Fisher,
Frey, & Lapp, 2012) and character shifts (Rich
& Taylor, 2000).
Flashbacks and narrator changes in a story
significantly impact readers abilities to recall
or retell stories, with more flashbacks and
more narrator changes throughout a story
compounding this effect (Kucer, 2010).
1. Uses high-frequency, grade-appropriate vocabulary

that relies on denotative meaning. Minimal use of
literary devices. Syntax is clear and consistent.
2. a. Uses unfamiliar, above-grade-level words
b. Uses at-grade-level words with intended multiple
connotations in order to convey multiple
meanings.
c. Uses common colloquialisms and/or simple
dialect.
d. Uses simple literary devices and figurative
language.
Texts that use common, high-frequency

words are easier to understand than texts
that use archaic or unfamiliar words. As the
amount of familiar vocabulary increases, so
does the level of text comprehension
(Schmitt, Jiang, & Grabe, 2011).
3. a. Words are unfamiliar, archaic, or academic.

b. Some words cannot be fully comprehended with
context clues.
c. Uses authentic, complex dialect colloquialisms,
and/or vernacular, which may make assumptions
about students prior experience.
d. Uses complex or abstract figurative language or
literary devices.
Archaic, formal, and domain-specific

vocabulary is more difficult than casual or
familiar vocabulary (Fisher, Frey, & Lapp,
2012).
Texts that use unfamiliar language (e.g., Old

English) and/or unfamiliar cultural
references are more difficult to understand
(Fisher, Frey, & Lapp, 2012).
Both the commonness of words and a

readers prior experience impact comprehension. That is, those who read texts with easy
vocabulary and are familiar with the topic
are able to more easily recall and summarize
a text (Freebody & Anderson, 1983).
Total Score
Page 7.3-6
Item Writing and Specifications

In addition to passage specifications, our content experts developed item specifications to
guide the SAGE item development process.
Writers used these specifications while developing SAGE items to ensure a solid alignment
with the Common Core standards and to make
the best use of our item types.
The item specifications were developed using a
vertical alignment for each standard, wherein
the suggested task demands and cognitive
complexity of items build upon those of the
previous grade level, just as the standards
themselves do.
Additionally, the item specifications provide
models for item writers. The models include
item samples that target different Depths of
Knowledge (DoKs) and difficulty levels. These
item models also annotate the information in
order to communicate the intent of the standard and DoK and to clarify for the writer how
to manipulate the item difficulty while keeping
the cognitive demands the same.
The item specifications include the following
information:
Content Standard: This identifies the standard being assessed.
Content Limits: This section delineates the
specific content that the standard measures
and the parameters within which items
must be developed to assess the standard
accurately, including the lower and upper
complexity limits of items.
Acceptable Response Mechanisms: This
section identifies the various ways in which
students may respond to an item or
prompt. Here we note whether evidencebased selected response (two-part items),
extended response, hot text, multiple
choice, multi-select, and/or short answer
(to be scored automatically with our proposition scorer) may be used, and if so, how.
DoK Demands: This section is broken into
three sectionsDoK, task demand, and
response mechanism. The task demands
explain the skills the students may be
required to demonstrate and connect these
skills to each applicable DoK. The task
demands break down the cognitive complexity to show how each DoK level
requires differences in higher-order thinking. Finally, the DoK and task demand are
connected to appropriate response mechanisms used to assess these skills.
Sample Items: In this section, sample items
present a range of response mechanisms
and their corresponding expected difficulties (easy, medium, and hard). Notes delineating the cognitive demands of the item
and an explanation of its difficulty level are
detailed for each sample item.
Exhibit 7.3.1-2 shows a sample of the item
specifications our content experts developed
for grade 7 literacy standard RL.7.1. It outlines
the limits of the items content in order to
fully address the standard. This includes specifying the type of evidence required and the
amount. Furthermore, as the standard requires
citing several pieces of textual evidence, we
limit the acceptable response mechanisms to
hot text, wherein the student selects the evidence within the text itself, and multi-select,
which allows students to choose two or more
disparate pieces of evidence. The DoK sections
explain the demands for each DoK level and
provide the acceptable response mechanisms.
The cognitive demands increase from supporting an explicit inference with explicit evidence
(DoK 1) to providing implicit evidence for an
inference that the student makes (DoK 3).
Note that at DoK 1 an item may be a one-part
hot text response, while at DoK 3 a hot text
item must be two parts to ensure that the student first makes the inference then supports it
with evidence. Similarly, the item models pro-
Page 7.3-7
Exhibit 7.3.1-2: Sample Item Specification Sheet

Content Standard
Literacy RL.7.1: Cite several pieces of textual evidence to support analysis of what the text says explicitly as well as inferences drawn from
the text.
Content Limits
The item may ask for evidence that is directly stated in the text or implied.
The item may ask for specific and exact quotes, or a summary/description of evidence.
The item may require the student to draw inferences from the text.
The item should ask for several pieces of evidence.
Acceptable Response
Mechanisms
Hot Text
Requires the student to select words or phrases from the text to support an analysis or inference.
Requires the student to select an inference from four choices AND then to select words or phrases from the text to support the inference
[two-part Hot Text].
Multi-select
Requires the student to select multiple direct quotes or descriptions of textual evidence to support an explicit or implicit statement from
the text.
1, 2, 3
DoK
DoK Demands
DoK
Task Demand
Response Mechanism
DoK 1
Select several pieces of textual evidence to support a statement from the text where
both the statement and support are explicit.
1. Hot Text Response

2. Multi-select Response
DoK 2
Select several pieces of textual evidence to support an inference from the text. The
inference is provided by the item writer.
1. Hot Text Response [one- and two-part]

DoK 3
Select several pieces of textual evidence to support an inference from the text. The
student must draw the inference.
1. Hot Text Response [two-part]

Item Models
DoK1
Sample Item
Difficulty
Select two phrases that prove Vicky has lost track Easy
of time.
Which sentences from the text show that Jimmy

is fascinated by the immigrant family?
The Moon by Night
Easy
Choices will be direct quotes from the text. A

multi-select response helps meet the standard,
since several pieces of evidence should be
synthesized. In this item, the student must draw
an inference, making the item a DoK 2. However,
the difficulty is easy since the inference is not
deep and the evidence is explicit and proximate
to the site of the inference itself.
My Antonia
Medium
The student must interpret Jimmys descriptions

of the immigrant family. The answer options
present inferences the student must analyze,
making this a DoK 2 item. The descriptions,
however, synthesize explicit and implicit
information for the student, which makes this
item more difficult than the last. The item is not
hard because the options enumerate the ideas
for the student rather requiring him/her to
provide the information himself/herself.
My Antonia
[Multi-select]
DoK2
How can readers tell that Jimmy is fascinated by

the immigrant family?
[Multi-select]
Page 7.3-8
Passage
The student should select two (to meet the

standard) descriptions of the landscape that
indicate time passing, such as the tide, the sun,
etc. This evidence is explicitly stated in the
passage, making this a DoK 1 item. The
proximate location of the detail itself makes the
item easy.
[Hot Text]
DoK2
Notes, Comments
Exhibit 7.3.1-2: Sample Item Specification Sheet (continued)

Item Models
DoK2
Sample Item
Part A:
Select two phrases that show Jimmy has a vivid
imagination.
Difficulty
Notes, Comments
Passage
Hard
The student must highlight two parts of Jimmys

description of Otto. The student must interpret
the implicit descriptions of and reasons for
Jimmys vivid imagination. While the author
explicitly states that Jimmy reads books about
outlaws, the student must understand that the
reading fuels his imagination. This, in turn,
colors his description of Otto. Requiring students
to draw this connection makes the item a DoK 2.
The complexity of the text in combination with
the fact that the student must synthesize
multiple pieces of the text increases the items
difficulty.
My Antonia
Medium
The student must understand from the quoted

text that Vicky is apprehensive. The student
must not only draw an inference to discover
what the quote reveals about Vicky, but he/she
must also synthesize the text as a whole. These
cognitive demands make the item a DoK 3.
The complexity of the passagethe in-depth
nature of the inferencesmake the item
moderately hard. However, the item is not
classified as hard because it provides options to
the student and does not require the student to
sift through the entire text.
The Moon by Night
Hard
To answer this question, the student must first

recognize that Vickys mood has changed from
contemplative and stubborn to embarrassed.
The student then must select evidence to
support this mood (e.g., I stared down at my
bare feet,keeping me from looking at my
rightfully enraged older brother,he was right
and I was wrong). These two steps require the
student to draw and support inferences, which
makes the item a DoK 3. The difficulty of this
item is increased because here the student must
sift through the entire text for support.
The Moon by Night
Part B:
Select the best reason for Jimmys vivid
imagination.
[two-part Hot Text]
DoK3
[quote about holding onto childhood]

Which sentences from the text highlight the
same emotion conveyed in the quote?
[Multi-select]
DoK3
Select a sentence that shows Vickys mood has

changed since the beginning of the passage.
[Hot Text]
vide explicit examples of items at each DoK

level and at various difficulty levels. This level
of detail provides the item writer with guidance when developing items, ensuring that the
items address the standard and are correctly
aligned to the DoK and difficulty levels.
SAGE ELA items were developed by item writers who consistently followed theses item specifications. During each level of review, items
were compared to the item specifications to
ensure alignment to the standard, grade-level

appropriateness and adherence to the content
limits set forth in the item specifications.
Taxonomic Classification Systems
Within each grade, items are aligned to a
depth of knowledge classification system.
Depth of knowledge refers to the cognitive
complexity of an item. SAGE items are categorized based on four levels of DoK:
Page 7.3-9
1. DoK 1Recall. Students recall, recognize,

locate, or describe basic facts, details, events,
or ideas that are explicitly stated in texts.
2. DoK 2Skill/Concept. Students use their
basic knowledge (DoK 1) and extend their
thinking to specify, explain, or show relationships; explain cause and effect; summarize results, concepts, or ideas; make basic
inferences or logical predictions from the
text; identify main ideas or themes; make
accurate generalizations about texts; locate
evidence to support central ideas or inferences; distinguish relevant or irrelevant
information; analyze structure; and compare literary elements.
3. DoK 3Strategic Thinking. Students go
beyond basic inferences to extend their
thinking into non-routine cognitive tasks,
such as explaining, generalizing, or connecting ideas using supporting evidence;
making inferences about explicit and/or
implicit themes; describing how an authors
choices or techniques affect the readers
understanding of a text; analyzing information within texts; analyzing relationships
between concepts and ideas; and analyzing
authors craft.
4. DoK 4Extended Thinking. At this highest
level, students engage in extended thinking
and complex cognitive tasks, including
explaining how concepts or ideas in a text
specifically relate to other content domains;
drawing conclusions and applying those
conclusions to new situations; analyzing
multiple sources for evidence or multiple
works by the same author; analyzing works
across genres, time periods, or themes;
gathering and analyzing multiple sources of
information; and synthesizing information
across multiple texts. SAGEs extended writing tasks represent the DoK 4 items.
Depending upon the grade band, the actual
percentage of items aligned to DoK 1, DoK 2,
DoK 3, and DoK 4 may vary. At each grade of
Page 7.3-10
the SAGE English Language Arts/Literacy

assessment, we maintain the following distribution ranges for DoK:
DoK 1:
1119%
DoK 2:
1730%
DoK 3:
1423%
DoK 4 (writing): 29%

While the exact number of items on each form
may vary, these ranges ensure that all students, regardless of grade level, receive a substantial proportion of items that assess higherorder thinking skills.
Reporting Categories
SAGE English Language Arts/Literacy reporting categories are consistent with the Common
Cores classification system of strands and substrands. Reports reflect a students mastery in
each of the categories illustrated in Exhibit
7.3.1-3:
Exhibit 7.3.1-3: English Language Arts/Literacy
Reporting Categories for SAGE
Reporting Category/Domains
Reading Standards for Literature
Reading Standards for Informational Text
Listening Comprehension (Informational)
Writing
Language
The relative weight placed on each of the standards within the separate grade bands determines the percentage of items in the reporting
categories at each grade level. Consistent with
the Common Core, the balance shifts from an
even distribution of Reading Literature and
Reading Informational Text items in the early
grades to a higher percentage of informational
items in the upper grades. (Please see Exhibit
7.3.1-4.) shows the relative percentage of each form that will be covered by each reporting category for grades 311.
Exhibit 7.3.1-4: Summary of Reporting Categories for Grades 311

ELA Grade 6
ELA Grade 3
Reporting Category /
Domain Percentages
Domain Percentages
Min.
Max.
17%
20%
20%
21%
24%
11%
11%
11%
14%
Writing
29%
29%
Writing
29%
29%
Language
17%
17%
Language
17%
17%
Min.
Max.
20%
20%
20%
ELA Grade 7
ELA Grade 4
Domain Percentages
Domain Percentages
Min.
Max.
17%
20%
20%
21%
24%
11%
11%
11%
14%
Writing
29%
29%
Writing
29%
29%
Language
17%
17%
Language
17%
17%
Min.
Max.
20%
20%
20%
ELA Grade 8
ELA Grade 5
Domain Percentages
Domain Percentages
Min.
Max.
17%
20%
20%
21%
24%
11%
11%
11%
14%
Writing
29%
29%
Writing
29%
29%
Language
17%
17%
Language
17%
17%
Min.
Max.
20%
20%
20%
Page 7.3-11
Exhibit 7.3.1-4: Summary of Reporting Categories for

Grades 311 (continued)
ELA Grade 9
Domain Percentages
Min.
Max.
16%
19%
23%
26%
11%
14%
Writing
29%
29%
Language
17%
17%
in student writing, AIR has developed a robust

bank of extended writing tasks for SAGE that
provide students with meaningful contexts in
which to construct their responses. Each writing prompt presents students with a variety of
stimuli (usually at least two to three per task)
that serve as a springboard for an informed
piece of writing. Students are given research
articles, charts and graphs, narratives, etc., to
serve as the basis for their written response.
Students can then use this information, along
with their own reasoning, to formulate an
essay that is not only a clear and coherent
expression of their own thinking, but one that
is also grounded in research and evidence.
ELA Grade 10
Domain Percentages
Min.
Max.
13%
16%
26%
29%
11%
14%
Writing
29%
29%
Language
17%
17%
ELA Grade 11
Domain Percentages
Min.
Max.
13%
16%
26%
29%
11%
14%
Writing
29%
29%
Language
17%
17%
Writing Tasks
Because the Common Core standards require
a greater awareness of audience and purpose
Page 7.3-12
In the SAGE operational field test, each student will complete two writing pieces, a short
expository piece and a long opinion/argumentative piece. The following sections describe
each writing domain in greater detail.
Informative/Explanatory Essay
Informative/explanatory writing is focused on
conveying information accurately. Informative
writing seeks to enlighten the reader about
processes or procedures, phenomena, states of
affairs, terminology, etc. To produce this kind
of writing, students draw from what they
already know as well as from primary and secondary sources. Students develop a controlling
idea and a primary focus as they relate facts,
details, and examples.
SAGE writing prompts present students with
two or three passage stimuli on a single topic
from science, technical subjects, or social
studies. The reading level of the stimulus does
not exceed the easy Lexile range for the grade
level to enable the students to attend to the
content of the passages and not struggle over
unfamiliar language and non-content-related
vocabulary. Moreover, they help ensure students are assessed on their writing skills and
not their reading abilities.
The stimulus is followed by a prompt that asks

students to write a short essay about the topic.
The students will be required to synthesize
information across the passages to write the
essay and must cite specific information from
the passages to support their main ideas. For
example, the prompt may require students to
describe the steps in a process, describe problems that need to be solved, etc.
Opinion/Argumentative Essay
Appendix A of the Common Core asserts that
persuasive writing is a reasoned, logical way
of demonstrating that the writers position,
belief, or conclusion is valid. SAGE
opinion/argumentative prompts ask students to
analyze primary and secondary sources, make
sound judgments, and present their opinions
and arguments in a coherent way that weaves
personal opinion with evidence from the texts.
The stimuli that we developed for SAGE present opposing points of view about a topic so
that students have enough information to take
a stand. The stimuli are followed by a prompt
that asks students to write an opinion/argumentative essay. The students will be required
to synthesize information across the passages
to write the essay and must cite specific information from the passages to support the ideas
they present.
Sample Test Blueprint

A sample test blueprint for SAGE grade 7 ELA
is shown in Exhibit 7.3.1-5. Each of the proposed reporting categories (domains) is noted
by green highlighting. The domains (pink) and
standards (white) which support each of the
reporting categories are further delineated in
the blueprint.
Note that minima and maxima are specified at
each level of the content hierarchy and within
each standard. This ensures that the content
and demands of each sub-strand are fully

addressed.
These blueprints are defined for an adaptive
administration. The total numbers may need
to increase if Florida decides to use the bank
to create a fixed-form test. The longer test is
often necessary to cover the range of performances seen in the population.
The distribution of item ranges is designed to
accomplish several important goals. First, the
Florida Standards Assessments must reflect a
representative sample of the knowledge and
skills that students are expected to achieve by
the end of each school year. In addition, each
domain must be assessed with sufficient items
to support reliable subscale performance-level
reporting. The blueprints are designed to
accomplish both goals, with items targeting the
full breadth of the academic content standards
but constrained enough to ensure that there
are sufficient items in each domain, thereby
achieving reliable subscale reporting.
Moreover, item ranges for DoK levels and item
types ensure that students, regardless of overall ability, are administered test items representing the full range of cognitive complexity
and are provided the opportunity to demonstrate their knowledge and skills through the
range of item types.
The distribution of item ranges is designed to
accomplish several important goals. First, the
Florida Standards Assessments must reflect a
representative sample of the knowledge and
skills that students are expected to achieve by
the end of each school year. In addition, each
domain must be assessed with sufficient items
to support reliable subscale performance-level
reporting. The blueprints are designed to
full breadth of the academic content standards
but constrained enough to ensure that there
are sufficient items in each domain, thereby
achieving reliable subscale reporting.
Page 7.3-13
Exhibit 7.3.1-5: Sample SAGE Grade 7 ELA Blueprint

Description
MIN
MAX
Literature
12
14
Key Ideas and Details
RL.7.1
Cite several pieces of textual evidence to support analysis of what the text says explicitly as well as inferences
drawn from the text.
RL.7.2
Determine a theme or central idea of a text and analyze its development over the course of the text; provide
an objective summary of the text.
RL.7.3
Analyze how particular elements of a story or drama interact (e.g., how setting shapes the characters or plot).
Craft and Structure
RL.7.4
Determine the meaning of words and phrases as they are used in a text, including figurative and connotative
meanings; analyze the impact of rhymes and other repetitions of sounds (e.g., alliteration) on a specific verse
or stanza of a poem or section of a story or drama.
RL.7.5
Analyze how a dramas or poems form or structure (e.g., soliloquy, sonnet) contributes to its meaning.
RL.7.6
Analyze how an author develops and contrasts the points of view of different characters or narrators in a text.
Integration of Knowledge and Ideas
RL.7.7
Compare and contrast a written story, drama, or poem to its audio, filmed, staged, or multimedia version,
analyzing the effects of techniques unique to each medium (e.g., lighting, sound, color, or camera focus and
angles in a film).
RL.7.9
Compare and contrast a fictional portrayal of a time, place, or character and a historical account of the same
period as a means of understanding how authors of fiction use or alter history.
Informational
15
Key Ideas and Details
RL.7.1
Cite several pieces of textual evidence to support analysis of what the text says explicitly as well as inferences
drawn from the text.
RL.7.2
Determine two or more central ideas in a text and analyze their development over the course of the text;
provide an objective summary of the text.
RL.7.3
Analyze the interactions between individuals, events, and ideas in a text (e.g., how ideas influence
individuals or events, or how individuals influence ideas or events).
Craft and Structure
RL.7.4
Determine the meaning of words and phrases as they are used in a text, including figurative and connotative
meanings; analyze the impact of rhymes and other repetitions of sounds (e.g., alliteration) on a specific verse
or stanza of a poem or section of a story or drama.
RL.7.5
Analyze the structure an author uses to organize a text, including how the major sections contribute to the
whole and to the development of the ideas.
RL.7.6
Determine an authors point of view or purpose in a text and analyze how the author distinguishes his or her
position from that of others.
Integration of Knowledge of Ideas
RL.7.7
Compare and contrast a text to an audio, video, or multimedia version of the text, analyzing each mediums
portrayal of the subject (e.g., how the delivery of a speech affects the impact of the words).
RL.7.8
Trace and evaluate the argument and specific claims in a text, assessing whether the reasoning is sound and
the evidence is relevant and sufficient to support the claims.
RL.7.9
Analyze how two or more authors writing about the same topic shape their presentations of key information
by emphasizing different evidence or advancing different interpretations of facts.
Standard
Page 7.3-14
Exhibit 7.3.1-5: Sample SAGE Grade 7 ELA Blueprint (continued)

Standard
Description
MIN
MAX
Language/Vocabulary Acquisition and Use
L.7.4
Determine or clarify the meaning of unknown and multiple-meaning words and phrases based on grade 7
reading and content, choosing flexibly from a range of strategies.
L.7.5
Demonstrate understanding of figurative language, word relationships, and nuances in word meanings.
Language/Editing Task
Literary
L.7.1
Demonstrate command of the conventions of standard English grammar and usage when writing or speaking.
L.7.2
Demonstrate command of the conventions of standard English capitalization, punctuation, and spelling
when writing.
Informational
L.7.1
L.7.2
when writing.
Listening
10
Comprehension and Collaboration
10
SL.7.1c
Pose questions that elicit elaboration and respond to others questions and comments with relevant
observations and ideas that bring the discussion back on topic as needed.
SL.7.1d
Acknowledge new information expressed by others and, when warranted, modify their own views.
SL.7.2
Analyze the main ideas and supporting details presented in diverse media and formats (e.g., visually,
quantitatively, orally) and explain how the ideas clarify a topic, text, or issue under study.
SL.7.3
Delineate a speakers argument and specific claims, evaluating the soundness of the reasoning and the
relevance and sufficiency of the evidence.
Writing
Pts.
Task 1: Short Essay
10
Production and Distribution
W.7.4
Produce clear and coherent writing in which the development, organization, and style are appropriate to
task, purpose, and audience.
W.7.5
With some guidance and support from peers and adults, develop and strengthen writing as needed by
planning, revising, editing, rewriting, or trying a new approach, focusing on how well purpose and audience
have been addressed.
W.7.6
Use technology, including the Internet, to produce and publish writing and link to and cite sources as well as
to interact and collaborate with others, including linking to and citing sources.
Use of Information
W.7.7
Conduct short research projects to answer a question, drawing on several sources and generating additional
related, focused questions for further research and investigation.
W.7.8
Gather relevant information from multiple print and digital sources, using search terms effectively; assess the
credibility and accuracy of each source; and quote or paraphrase the data and conclusions of others while
avoiding plagiarism and following a standard format for citation.
W.7.9
Draw evidence from literary or informational texts to support analysis, reflection, and research.
Page 7.3-15
Exhibit 7.3.1-5: Sample SAGE Grade 7 ELA Blueprint (continued)

Standard
Description
MAX
Writing
Pts.
Conventions
L.7.3
Use knowledge of language and its conventions when writing, speaking, reading, or listening.
L.7.4
L.7.5
L.7.6
Acquire and use accurately grade-appropriate general academic and domain-specific words and phrases;
gather vocabulary knowledge when considering a word or phrase important to comprehension or expression.
Task 2: Long Essay
10
Production and Distribution
W.7.4
Produce clear and coherent writing in which the development, organization, and style are appropriate to task,
purpose, and audience.
W.7.5
With some guidance and support from peers and adults, develop and strengthen writing as needed by
planning, revising, editing, rewriting, or trying a new approach, focusing on how well purpose and audience
have been addressed.
W.7.6
Use technology, including the Internet, to produce and publish writing and link to and cite sources as well as
to interact and collaborate with others, including linking to and citing sources.
Use of Information
W.7.8
Gather relevant information from multiple print and digital sources, using search terms effectively; assess the
credibility and accuracy of each source; and quote or paraphrase the data and conclusions of others while
avoiding plagiarism and following a standard format for citation.
W.7.9
Draw evidence from literary or informational texts to support analysis, reflection, and research.
Conventions
L.7.1
L.7.2
when writing.
L.7.3
Use knowledge of language and its conventions when writing, speaking, reading, or listening.
L.7.4
L.7.5
L.7.6
Acquire and use accurately grade-appropriate general academic and domain-specific words and phrases;
gather vocabulary knowledge when considering a word or phrase important to comprehension or expression.
Page 7.3-16
MIN
Moreover, item ranges for DoK levels and item types ensure that students, regardless of overall
ability, are administered test items representing the full range of cognitive complexity and are provided the opportunity to demonstrate their knowledge and skills through the range of item types.
Tools Available in the Testing Environment
AIRs test delivery platform supports a number of tools that make the ELA assessment more
accessible, including a zoom function, a passage-expansion tool, and a highlighting tool. Exhibit
7.3.1-6 shows the test environment before the use of any tool.
The passage-expansion tool (Exhibit 7.3.1-7) allows students to click an arrow in the passage
itself in order to expand the stimulus and better view the text.
Exhibit 7.3.1-6: Test Environment before Use of Any Tool
Exhibit 7.3.1-7: Test Environment after Use of the Passage-Expansion Tool
Page 7.3-17
Similarly, the zoom function (Exhibit 7.3.1-8) increases accessibility to the text by allowing the
student to zoom in or out as necessary, in both regular view and when the passage is expanded.
Exhibit 7.3.1-8: Test Environment after Use of the Zoom Function
Finally, students may highlight sections of the text as they deem necessary by using their mouse
to select text and then clicking their mouse to highlight text (Exhibit 7.3.1-9).
Exhibit 7.3.1-9: Test Environment after Use of the Zoom Function
Page 7.3-18
Mathematics Test Content

Overview of the Florida Next
Generation Sunshine State Standards in
Mathematics
In 2010, Florida adopted the CCSS in
Mathematics as part of Floridas suite of Next
Generation Sunshine State Standards. Three
characteristics distinguish these new standards
in Mathematics:
1. The standards focus on fewer topics in a
more in-depth way for each grade.
2. The topics align across grades to form
coherent learning progressions.
3. The standards identify and emphasize
mathematical practices that cut across topics
and are integral to mathematical thinking.
By the 20132014 school year, these new standards are to be fully implemented.
The 2010 Next Generation Sunshine State
Standards in Mathematics focus more narrowly
than the 2007 Next Generation Sunshine State
Standards in Mathematics, as well as prior
Mathematics standards. For example, in grade
6, teachers are expected to focus on these four
critical areas:
1. Connecting ratio and rate to whole number
multiplication and using concepts of ratio
and rate to solve problems
2. Completing understanding of division of
fractions and extending the notion of number to the system of rational numbers,
which includes negative numbers
3. Writing, interpreting, and using expressions
and equations
4. Developing understanding of statistical
thinking
The high school standards are grouped in conceptual categories: Number and Quantity,
Algebra, Functions, Modeling, Geometry, and
Statistics and Probability. Depending on the
specific high school Mathematics course, certain conceptual categories are more heavily
emphasized than others.
The Florida Mathematics End-of-Course (EOC)
Assessments measure student achievement of
the Next Generation Sunshine State Standards
for three specific courses: Algebra I, Geometry,
and Algebra II.
1. In Algebra I, students begin to explore the
real number system, reason about units,
rewrite algebraic expressions, master linear
and quadratic functions, apply knowledge of
the Cartesian plane to linear equations and
systems, and apply basic statistics and simple probability within a modeling context.
2. In Geometry, students further develop their
understanding of geometric transformations
within a modeling context, apply measurement formulas (particularly surface area
and volume of complex shapes), solve
applied problems involving the Pythagorean
Theorem, use the distance formula in coordinate systems to connect equations and
graphs of circles, and apply knowledge of
algebra to prove basic geometric theorems
in the Cartesian plane.
3. In Algebra II, students perform more complex arithmetic with polynomials, solve
equations with extraneous solutions, solve
quadratic equations with complex number
roots, use their knowledge of algebra and
geometry to connect equations and graphs
of conic sections, apply knowledge of right
triangle trigonometry to the entire unit circle, and fit and model exponential functions to a set of data.
The SAGE assessments for high school,
although they follow a more integrated path-
Page 7.3-19
way, include only items aligned to the CCSS.

Therefore, items designated for a particular
course under the SAGE assessment will be
designated for the appropriate course in
Florida.
These emphases align across grades to form
coherent learning progressions in which the
content from one grade supports the topics to
be learned in subsequent grades. Elementary
school standards, which have a heavy emphasis on numbers in base 10, fractions, and operations and algebraic thinking, are designed to
prepare students for success in middle school
Mathematics. Middle school standards, which
include ratio and proportional reasoning, equations, and functions, emphasize the knowledge
needed for high school Mathematics. The high
school standards are designed to ensure college and career readiness. In particular, the
high school standards emphasize mathematical
modeling and allow for more advanced studies
in Mathematics. Throughout the grades, the
standards emphasize the eight Standards for
Mathematical Practice identified in the CCSS:
1. Make sense of problems and persevere in
solving them
2. Reason abstractly and quantitatively
3. Construct viable arguments and critique
the reasoning of others
4. Model with mathematics
5. Use appropriate tools strategically
6. Attend to precision
7. Look for and make use of structure
8. Look for and express regularity in repeated
reasoning
These eight practices do not stand apart from
the Standards for Mathematical Content (i.e.,
the actual Mathematics content covered by the
standards). Connections need to be made
between the Standards for Mathematical Practice
Page 7.3-20
and the Standards for Mathematical Content,

particularly within assessments. For example,
when students are given tasks that demand
understanding of a concept rather than pure
recall, it is easier to see the connection between
the Standards for Mathematical Practice and the
Standards for Mathematical Content. Similarly,
higher Depth of Knowledge (DoK) tasks require
particular attention to the intersection between
these two types of standards.
The SAGE item bank comprises only items
that are aligned to the Common Core, make
use of the Mathematics Practices, and follow
the item specifications described in the following section.
Item Specifications for Mathematics
Item specifications, designed by content
experts at AIR, strategically guide the item
development process. They are detailed documents which outline content limits and model
tasks for a particular standard. Detailed item
specifications include the following:
Content Limits: This section delineates the
specific content measured by the standard
and the extent to which the content is different across grade levels. For example, in
grade 3, fraction denominators are limited
to 2, 3, 4, 6, and 8.
Acceptable Response Mechanisms: This section identifies the various ways in which students may respond to a prompte.g., multiple choice, graphic response, proposition
response, equation response, multi-select.
Math Practice Cluster: The item specifications outline to which practice cluster(s) a
particular standard could be alignedPC1,
PC2, PC3, or none. See section titled
Practice Clusters (under Taxonomic
Classification Systems) for more detailed
information on how the Standards for
Mathematical Practice were grouped

together to define these Practice Clusters.
Depth of Knowledge: The task demands of
each standard can be classified as DoK1,
DoK2, and/or DoK3.
Task Demands: In this section, the standards are broken down into specific task
demands aligned to the standard. In addition, each task demand is assigned appropriate response mechanism, DoK, and
Practice Clusters specifically relevant to
that particular task demand.
Examples and Sample Items: In this section, sample items are delineated along
with their corresponding expected difficulties (easy, medium, and hard.) Notes for
modifying the difficulty of each task
demand are detailed with suggestions for
the item writer. The suggestions for adapting the difficulty based on the task demands
are research-based and have been reviewed
by both content experts and a cognitive
psychologist.
The SAGE Mathematics items were developed
by item writers who consistently followed the
item specifications. During each level of
review, items were compared to the item specifications to ensure their alignment to the
standard, grade-level appropriateness, and
adherence to the content limits set forth in the
item specifications.
Exhibits 7.3.1-10 and 7.3.1-11 demonstrate
item specifications for different standards.
Exhibit 7.3.1-10 shows how two standards,
5.G.1 and 5.G.2, which are related topics and
taught simultaneously, can be grouped together for assessment purposes. Exhibit 7.3.1-11
gives sample secondary Mathematics item
specifications for a single standard.
Taxonomic Classification Systems

Within each grade or course, all items are
aligned according to two taxonomic classification systems, DoK and PC.
Depth of Knowledge (DoK)
Depth of Knowledge refers to the cognitive
complexity of the item and the cognitive
demands upon the student. Based upon work
done by Webb, there are four levels of DoK:
1. DoK1Recall. Students recall basic mathematical ideas, perform basic arithmetic
operations using established algorithms,
and identify examples of general mathematics principles.
2. DoK2Skill/Concept. Students apply their
basic knowledge (DoK1) and extend their
thinking to problem solve, identify relationships, and draw conclusions.
3. DoK3Strategic Thinking. Students go
beyond basic problem solving (i.e., word
problems) to extend their thinking to nonroutine problem solving, hypothesize, and
critique arguments or problem solving
strategies.
4. DoK4Extended Thinking. At this highest
level, students engage in extended problemsolving activities, which require integration
of multiple standards. For example, students may engage in a performance task
which includes a common stimulus and
four to six associated items related to the
stimulus.
Depending upon the grade band, the percentage of items aligned to DoK1, DoK2, and DoK3
may vary. For example, in the elementary
grades (3-5), the following distribution ranges
are recommended, based upon the task
demands and content of the standards:
DoK1: 2030%
Page 7.3-21
Exhibit 7.3.1-10: Sample Elementary Item Specifications (5.G.1 and 5.G.2)

Content Standard
Math.Content.5.G.A.1 Use a pair of perpendicular number lines, called axes, to define a coordinate system, with the intersection of the lines
(the origin) arranged to coincide with the 0 on each line and a given point in the plane located by using an ordered pair of numbers, called
its coordinates. Understand that the first number indicates how far to travel from the origin in the direction of one axis, and the second
number indicates how far to travel in the direction of the second axis, with the convention that the names of the two axes and the
coordinates correspond (e.g., x-axis and x-coordinate, y-axis and y-coordinate).
Math.Content.5.G.A.2 Represent real-world and mathematical problems by graphing points in the first quadrant of the coordinate plane
and interpret coordinate values of points in the context of the situation.
Content Limits
Points in first quadrant

Includes origin and points on positive x- and y-axes
Whole numbers
Real-world contexts must have axes scaled to whole numbers (not letters)
Acceptable Response
Mechanisms
Graphic responseMay require plotting a point on the coordinate plane

Equation responseMay require identifying the x- or y-value of an ordered pair given a point on the coordinate plane
Proposition responseMay require explaining the meaning of a points coordinates
Math Practice Cluster(s)
PC1 and PC2
DoK
1, 2
Model Task
Any situation that includes the use (or understanding) of the coordinate plane.
Context
DoK Demands
DOK
Task Demand
Response Mechanism
PC1
DoK 1
Recognize and identify the location of an ordered pair (e.g.,

plot ordered pairs; state coordinates given a point on the
coordinate plane.)
1. Equation response
2. Graphic response
DoK 2
Interpret the location of a point on the

coordinate plane.
1. Equation response
2. Graphic response
DoK 3
N/A
PC2
PC3
None
X
Example
Context
Consider the point (2, 5).
Context Easier
Graphic is shown or coordinates are given.

No context.
Context More Difficult
Parameters are given.

Context may be included.
Interpretation required.
Item Models
Sample Item
Difficulty
Notes, Comments
DoK1
Plot the point (2, 5) on the coordinate plane.
Easy
Recognizing and identifying a point represented

by an ordered pair.
DoK2
Write a statement that describes the location of the point (2, 5).
Medium
Explaining and interpreting the meaning of an

ordered pair.
DoK2
Point A is located on the x-axis.

Point B is located exactly 5 units above point A. Which could be the
coordinates of points A and B? Explain your reasoning.
Hard
Reasoning about and comparing the location of

points on the coordinate plane given
parameters.
Page 7.3-22
Exhibit 7.3.1-11: Sample Secondary Mathematics Item Specifications (G-C.5)

Content Standard
Math.Content.HSG-C.B.5 Derive using similarity the fact that the length of the arc intercepted by an angle is proportional to the radius, and
define the radian measure of the angle as the constant of proportionality; derive the formula for the area of a sector.
Content Limits
Radian introduced only as a unit of measure

Note: Emphasize the similarity of all circles. Note that by similarity of sectors with the same central angle, arc lengths are proportional to the
radius. Use this as a basis for introducing radian as a unit of measure. It is not intended that it be applied to the development of circular
trigonometry in this course.
Acceptable Response
Mechanisms
Equation responseMay require creating equations related to circles

Graphic responseMay require creating circles (Note: This is currently being developed)
Simulation responseMay require generating tables and creating circles
Math Practice Cluster(s)
PC1
DoK
Model Task
Any situation that involves comparing circles.
Context
DoK Demands
DoK
Task Demand
DoK 1
N/A
DoK 2
Understand that sectors with different arcs

have arc lengths that are proportional.
Response Mechanism
PC1
PC2
PC3
None
1. Graphic response
2. Simulation response
Understand that sectors with the same arc of

two different circles are proportional.
DoK 3
N/A
Example
Context
A circle has points A, B, and C with center O. The arc lengths of AB = 110, and BC = 85. What is the arc length of CA and the measure of angle
COA?
Context Easier
Reduce the number of sectors in the circle.
Context More Difficult
Provide the radius of a circle.
Item Models
Sample Item
Difficulty
Notes, Comments
DoK2
A circle has points A and B with center O. The arc lengths of

AB = 110. What is the angle of AOB?
Easy
Recalling definitions and facts to show the

relationship between two entities.
DoK2
A circle has points A, B, and C with center O. The arc lengths of

AB = 110, and BC = 85. What is the arc length of CA and angle
of COA?
Medium
Interpreting given information to solve a

problem.
DoK2
A circle has a radius of 3 units and a central angel equal to 45

degrees. What is the area of the circle not covered by the sector?
Hard
Analyzing the given information to solve

non-routine problems.
Page 7.3-23
DoK2: 4060%
DoK3: 1020%
As the grades progress, there is less of an
emphasis on DoK1 and a greater emphasis on
DoK2 and DoK3 task demands. By high school,
the following distribution of DoK task demands
is recommended:
DoK1: 1025%
DoK2: 4060%
DoK3: 2030%
These ranges ensure that even in lower grades,
a student will receive a substantial proportion
of items that assess higher order thinking skills
and that these students are not limited to
lower level DoK items. DoK ranges hold true
for students of all ability levels, not just the
higher level students.
Practice Clusters
The Practice Clusters were created by AIR by
extracting the three biggest ideas from the
eight Standards for Mathematical Practice.
These three ideas are problem solving, mathematical reasoning, and the ability to generalize. These practice clusters are presented in
greater detail below.
1. Practice Cluster 1: Use Mathematics to
Solve Problems. Problem solving and modeling go hand in hand with mathematics
education. Many times application of mathematical concepts is the truest way to grasp
a students understanding. This practice
cluster requires the student to use mathematics to model real-world situations and
solve problems. The student is able to
abstract from a concrete situation to a
mathematical model. In order for a student
to plan a solution path or find a solution to
a problem, he or she may apply characteristics of a similar problem or try out special
Page 7.3-24
cases. These strategies allow for further

exploration of the problems characteristics
or evaluation of an interim or final solution. Successful problem solving requires
the ability to connect variables to their
real-world values and use them to evaluate
the reasonableness of the solution. When
solutions are not reasonable, the student
can identify why. Math practices 1, 4, and 5
are clearly present in this cluster; however,
this does not mean the other practices
should be excluded.
2. Practice Cluster 2: Use Mathematical
Reasoning. Reasoning is fundamental for
true mathematical understanding. Being
able to present mathematical concepts in a
sensible, logical manner to prove or disprove arguments is essential. Students can
reason by using stated assumptions and
previously established results. They can
make conjectures and develop a logical
path to test their conjecture. Students will
identify cases covered by a situation and
recognize supporting examples and counter
examples. Reasoning inductively with arguments that make sense given the context of
the problem is a necessary skill. Students
will be able to evaluate chains of logic and
determine the domain in which a solution
applies. The communication will be precise
in terms of language, labeling, equations,
and specific units of measure. This practice
cluster represents practices 2, 3, and 6 but
is not meant to exclude the other practices.
Reasoning, and communicating ones reasoning, cannot be done in isolation of the
other practices.
3. Practice Cluster 3: Use Characteristics of
Problems to Generalize. Mathematical proficiency means being able to identify structure or patterns within a problem and use
this to make generalizations. The student
will be able to apply generalizations to infer
characteristics about a solution, especially
when the information may not be available
for a complete solution. Many times these

generalizations will lead to mathematical
shortcuts. The student can identify structures and/or patterns to analyze the characteristics of mathematical properties. Math
practices 7 and 8 are clearly a part of this
practice cluster. However, it is important to
realize that many times this practice cluster will also be used in combination with
the other two.
The blueprint will specify which content standards match each practice cluster or clusters.
All items will be aligned to the content standards. Each item may also align to one or
more practice clusters. Our expectation is that
out of the 4550 items on a computer-adaptive
test, approximately 30 of the items will also
align to one or more practice clusters. All
SAGE assessment items are tagged with the
corresponding practice cluster, expected difficulty, and DoK.
For any particular grade, the reporting categories are determined by the relative weight
placed on each of the standards within that
reporting category. For certain grades, domains
are grouped together (e.g., geometry is grouped
with measurement and data in the elementary
grades.) Exhibit 7.3.1-12 shows the relative
percentage of each form that will be covered by
each reporting category for grades 38.
Further, the grade 6 assessment will be segmented into two sections, calculator and nocalculator. The content of the items will dictate which section they are to be assessed.
SAGE assessment items for grade 6 were
specifically written with a calculator or noncalculator segment in mind. However, in general, the following reporting categories are
deemed appropriate for the calculator segment: Ratios and Proportional Reasoning,
Expressions and Equations, Geometry, and

Grades 38.
Mathematics Grade 3
Reporting Category/
Domain (Percentage)
Min.
Max.
Operations and Algebraic Thinking
29
38
Number and Operations in Base 10
18
22
Number and OperationsFractions
27
31
Measurement and Data and Geometry
18
22
Mathematics Grade 4
Reporting Category/
Domain (Percentage)
Min.
Max.
18
22
28
32
28
32
14
22
Mathematics Grade 5
Reporting Category/
Domain (Percentage)
Min.
Max.
16
20
30
36
28
34
18
22
Page 7.3-25

Grades 38 (continued)
Mathematics Grade 6
Reporting Category/
Domain (Percentage)
Min.
Max.
Ratio and Proportions
28
32
Expressions and Equations
28
34
The Number System
18
22
Geometry and Statistics and Probability
16
20
Mathematics Grade 7
Reporting Category/
Domain (Percentage)
Min.
Max.
Ratio and Proportions
22
26
16
20
The Number System
18
22
Geometry
18
22
Statistics and Probability
18
22
Mathematics Grade 8
Reporting Category/
Domain (Percentage)
Min.
Max.
Functions
20
24
20
24
Geometry and the Number System
34
40
Statistics and Probability
16
20
Statistics and Probability. Our test delivery

system allows students to complete only one
segment before they continue to the other segment. For example, if students are allowed to
use the calculator in the first segment only,
Page 7.3-26
the test administrator would allow students to

continue to the second section after collecting
handheld calculators (if allowed by local policy), and the second segment would not have
an online calculator available.
Sample Test Blueprint
A sample test blueprint for EOC: Geometry is
shown in Exhibit 7.3.1-13. Each of the proposed reporting categories (domains) is noted
by green highlighting. The clusters (pink) and
standards (white), which support each of the
reporting categories, are further delineated in
the blueprint. An asterisk indicates a standard
related to making Mathematical models.
Note that minima and maxima are specified at
each level of the content hierarchy. So, in the
example in Exhibit 7.3.1-13, the cluster
G.CO.1 requires a minimum of 4 items. It does
not specify which of the standards within
G.CO.1 must be included. Instead, it ensures
that most of the standard be covered by allowing a maximum of only one item from each of
the standards below it. Hence, at least 4 of the
5 standards will have to be covered.
These blueprints are defined for an adaptive
administration. The total numbers may need
to increase if Florida decides to use the bank
to create a fixed-form test. The longer test is
often necessary to cover the range of performance seen in the population.
Tools Available in the Testing Environment
AIRs test delivery platform supports a wide
variety of tools, including a ruler, compass,
and straightedge. The SAGE exam does not
make use of the ruler. Readers are referred to
the detailed description in Section 7.4.9.
Calculators
AIRs online calculators, provided through the
test delivery system, allow for calculators that
Exhibit 7.3.1-13: Sample Geometry End-of-Course Blueprint

Standard
Description
MIN
MAX
G.CO
Congruence
20
22
G.CO.1
Experiment with transformations in the plane.
G.CO.1.1
1. Know precise definitions of angle, circle, perpendicular line, parallel line, and line segment,
based on the undefined notions of point, line, distance along a line, and distance around a circular arc.
G.CO.1.2
2. Represent transformations in the plane using, e.g., transparencies and geometry software;
describe transformations as functions that take points in the plane as inputs and give other
points as outputs. Compare transformations that preserve distance and angle to those that
do not (e.g., translation versus horizontal stretch).
G.CO.1.3
3. Given a rectangle, parallelogram, trapezoid, or regular polygon, describe the rotations and reflections
that carry it onto itself.
G.CO.1.4
4. Develop definitions of rotations, reflections, and translations in terms of angles, circles, perpendicular
lines, parallel lines, and line segments.
G.CO.1.5
5. Given a geometric figure and a rotation, reflection, or translation, draw the transformed figure using,
e.g., graph paper, tracing paper, or geometry software. Specify a sequence of transformations that will
carry a given figure onto another.
G.CO.2
Understand congruence in terms of rigid motions.
G.CO.2.6
6. Use geometric descriptions of rigid motions to transform figures and to predict the effect of a given
rigid motion on a given figure; given two figures, use the definition of congruence in terms of rigid
motions to decide if they are congruent.
G.CO.2.7
7. Use the definition of congruence in terms of rigid motions to show that two triangles are congruent if
and only if corresponding pairs of sides and corresponding pairs of angles are congruent.
G.CO.2.8
8. Explain how the criteria for triangle congruence (ASA, SAS, and SSS) follow from the definition of
congruence in terms of rigid motions.
G.CO.3
Prove geometric theorems.
G.CO.3.9
9. Prove theorems about lines and angles. Theorems include: vertical angles are congruent; when a
transversal crosses parallel lines, alternate interior angles are congruent and corresponding angles are
congruent; points on a perpendicular bisector of a line segment are exactly those equidistant from the
segment's endpoints.
G.CO.3.10
10. Prove theorems about triangles. Theorems include: measures of interior angles of a triangle sum to
180; base angles of isosceles triangles are congruent; the segment joining midpoints of two sides of a
triangle is parallel to the third side and half the length; the medians of a triangle meet at a point.
G.CO.3.11
11. Prove theorems about parallelograms. Theorems include: opposite sides are congruent, opposite angles
are congruent, the diagonals of a parallelogram bisect each other, and conversely, rectangles are
parallelograms with congruent diagonals.
G.CO.4
Make geometric constructions.
G.CO.4.12
12. Make formal geometric constructions with a variety of tools and methods (compass and straight edge,
string, reflective devices, paper folding, dynamic geometric software, etc.). Copying a segment; copying
an angle; bisecting a segment; bisecting an angle; constructing perpendicular lines, including the
perpendicular bisector of a line segment; and constructing a line parallel to a given line through a
point not on the line.
G.CO.4.13
13. Construct an equilateral triangle, a square, and a regular hexagon inscribed in a circle.
Page 7.3-27
Exhibit 7.3.1-13: Sample Geometry End-of-Course Blueprint (continued)

Description
Standard
MIN
MAX
G.SRT
Similarity, Right Triangles, and Trigonometry
11
C.SRT.1
Understand simiarity in terms of similarity transformations.
1. Verify experimentally the properties of dilations given by a center and a scale factor.
G.SRT.1.1a
a. A dilation takes a line not passing through the center of the dilation to a parallel line, and leaves a line
passing through the center unchanged.
G.SRT.1.1b
b. The dilation of a line segment is longer or shorter in the ratio given by the scale factor.
G.SRT.1.2
2. Given two figures, use the definition of similarity in terms of similarity transformations to decide if they
are similar; explain using similarity transformations the meaning of similarity for triangles as the
equality of all corresponding pairs of angles and the proportionality of all corresponding pairs of sides.
G.SRT.1.3
3. Use the properties of similarity transformations to establish the AA criterion for two triangles to be similar.
G.CO.1.1
G.SRT.2
Prove theorems involving similarity.
G.SRT.2.4
4. Prove theorems about triangles. Theorems include: a line parallel to one side of a triangle divides the
other two proportionally, and conversely; the Pythagorean Theorem proved using triangle similarity.
G.SRT.2.5
5. Use congruence and similarity criteria for triangles to solve problems and to prove relationships in
geometric figures.
G.SRT.3
Define trigonometric ratios and solve problems involving right triangles.
G.SRT.3.6
6. Understand that by similarity, side ratios in right triangles are properties of the angles in the triangle,
leading to definitions of trigonometric ratios for acute angles.
G.SRT.3.7
7. Explain and use the relationship between the sine and cosine of complementary angles.
G.SRT.3.8
8. Use trigonometric ratios and the Pythagorean Theorem to solve right triangles in applied problems.
G.C
Circles
11
G.C.1
Understand and apply theorems about circles.
G.C.1.1
1. Prove that all circles are similar.
G.C.1.2
2. Identify and describe relationships among inscribed angles, radii, and chords. Include the relationship
between central, inscribed, and circumscribed angles; inscribed angles on a diameter are right angles;
the radius of a circle is perpendicular to the tangent where the radius intersects the circle
G.C.1.3
3. Construct the inscribed and circumscribed circles of a triangle and prove properties of angles for a
quadrilateral inscribed in a circle.
G.C.2
G.C.2.5
Find arc lengths and areas of sectors of circles.
5. Derive using similarity the fact that the length of the arc intercepted by an angle is proportional to the
radius, and define the radian measure of the angle as the constant of proportionality; derive the formula
for the area of a sector.
G.GPE
Expressing Geometric Properties With Equations
11
G.GPE.1
Translate between the geometric description and the equation for a conic section.
G.GPE.1.1
1. Derive the equation of a circle of given center and radius using the Pythagorean Theorem; complete the
square to find the center and radius of a circle given by an equation.
G.GPE.1.2
2. Derive the equation of a parabola given a focus and directrix.
Page 7.3-28
Exhibit 7.3.1-13: Sample Geometry End-of-Course Blueprint (continued)

Standard
Description
MIN
MAX
G.GPE
Expressing Geometric Properties With Equations
11
G.GPE.2
Use coordinates to prove simple geometric theorems algebraically.
G.GPE.2.4
4. Use coordinates to prove simple geometric theorems algebraically. For example, prove or disprove that a
figure defined by four given points in the coordinate plane is a rectangle; prove or disprove that the point
(1, 3) lies on the circle centered at the origin and containing the point (0, 2).
G.GPE.2.5
5. Prove the slope criteria for parallel and perpendicular lines and use them to solve geometric problems
(e.g., find the equation of a line parallel or perpendicular to a given line that passes through a given
point).
G.GPE.2.6
6. Find the point on a directed line segment between two given points that partitions the segment in a
given ratio.
G.GPE.2.7
7. Use coordinates to compute perimeters of polygons and areas of triangles and rectanglese.g., using
the distance formula.*
G.GMD
Geometric Measurement and Dimension and Modeling with Geometry
11
G.GMD.1
Explain volume formulas and use them to solve problems.
G.GMD.1.1
1. Give an informal argument for the formulas for the circumference of a circle, area of a circle, volume of a
cylinder, pyramid, and cone. Use dissection arguments, Cavalieri's principle, and informal limit arguments.
G.GMD.1.3
3. Use volume formulas for cylinders, pyramids, cones, and spheres to solve problems.*
G.GMD.2
Visualize relationships between two-dimensional and three-dimensional objects.
G.GMD.2.4
4. Identify the shapes of two-dimensional cross-sections of three-dimensional objects, and identify

three-dimensional objects generated by rotations of two-dimensional objects.
G.MG.1
Apply geometric concepts in modeling situations.
G.MG.1.1
1. Use geometric shapes, their measures, and their properties to describe objects (e.g., modeling a tree
trunk or a human torso as a cylinder).*
G.MG.1.2
2. Apply concepts of density based on area and volume in modeling situations (e.g., persons per square
mile, BTUs per cubic foot).*
G.MG.1.3
3. Apply geometric methods to solve design problems (e.g., designing an object or structure to satisfy
physical constraints or minimize cost; working with typographic grid systems based on ratios).*
56
66
TOTALS
are grade-level appropriate and completely

configurable. For example, in a grade 6 assessment, the calculator would include basic fourfunction capabilities. In an Algebra II assessment, scientific, graphing, and regression calculators would also be available. Here, we
describe the SAGE configurations. See
Exhibits 7.3.1-14 and 7.3.1-15 for examples of
these calculators.
AIR also allows students, teachers, and parents
to download home versions of the online cal-
culators (which are identical to the ones

offered in the testing environment) to practice
using the grade-level specific calculators. Once
these are downloaded, an Internet connection
is not required to use them at home, in the
classroom, or in a lab.
The SAGE assessment items were written with
these functionalities in mind. For example, in
grade 6, a general four-function calculator is
only allowed in one segment, as previously
described. In grades 7 and 8, the scientific cal-
Page 7.3-29
Exhibit 7.3.1-14: Basic Calculator
culator is allowed. And in high school courses,

the scientific, graphing, and regression calculators are used by students to answer questions
where the technology supports but does not
replace their own thinking.
Special Versions
AIR will provide Braille and large print versions of these tests, as described in Section
7.3.11. Our online accommodations, which are
the most complete in the industry, are
described in Section 7.4.9.
We note that the SAGE and Smarter Balanced
exams are ready for Braille delivery online,
using technology invented at AIR. The interactive Braille option may be preferable if the
Department chooses an adaptive approach.
Exhibit 7.3.1-15: Scientific, Graphing, and Regression Calculator.
Page 7.3-30
AIR will deliver Braille

and large print sample
materials upon request,
as described in Section
7.3.7.5.
Sample Items
English Language Arts / Literacy Items

Item Type
Hot Text
Alignment to Common
Core Standards for ELA
CCSS.ELA-Literacy.RI.11-12.5 Analyze and evaluate the effectiveness of the structure an author uses in his or
her exposition or argument, including whether the structure makes points clear, convincing, and engaging.
DoK
Rationale/Description
This item requires students to evaluate an authors argument and critique his rhetorical approach. Students
must understand the authors overall argument as well as identify fallacies in the logic.
Page 7.3-32
Sample Item
Page 7.3-33
Item Type
Proposition Scorer
Alignment to Common
CCSS.ELA-Literacy.RI.11-12.6 Determine an authors point of view or purpose in a text in which the rhetoric
is particularly effective, analyzing how style and content contribute to the power, persuasiveness, or beauty
of the text.
DoK
This item requires students to evaluate an authors argument. Students must understand the authors
overall point of view in order to explain its relationship to the quotation.
Page 7.3-34
Sample Item
Page 7.3-35
Item Type
Multiple Choice
Alignment to Common
CCSS.ELA-Literacy.RI.11-12.7 Integrate and evaluate multiple sources of information presented in different

media or formats (e.g., visually, quantitatively) as well as in words in order to address a question or solve
a problem.
DoK
This item requires students to make an inference based on a text. In this item, students analyze data
presented in a text and use the information to draw a conclusion.
Page 7.3-36
Sample Item
Page 7.3-37
Item Type
Writing Prompt
Alignment to Common
CCSS.ELA-Literacy.W.6.2 Write informative/explanatory texts to examine a topic and convey ideas, concepts,
and information through the selection, organization, and analysis of relevant content.
DoK
This item requires students to develop an essay based on the ideas presented in passages. Students must
explore the relationship between concepts, draw conclusions about those relationships, and support their
conclusions with evidence and examples.
Page 7.3-38
Sample Item
Page 7.3-39
Item Type
Multiple Choice
Alignment to Common
CCSS.ELA-Literacy.RL.9-10.4 Determine the meaning of words and phrases as they are used in the text,
including figurative and connotative meanings; analyze the cumulative impact of specific word choices on
meaning and tone (e.g., how the language evokes a sense of time and place; how it sets a formal or
informal tone).
DoK
This item requires students to analyze the cumulative effect of a characters dialogue and its overall impact
on the text.
Page 7.3-40
Sample Item
Page 7.3-41
Item Type
Proposition Scorer
Alignment to Common
CCSS.ELA-Literacy.RL.9-10.4 Determine the meaning of words and phrases as they are used in the text,
including figurative and connotative meanings; analyze the cumulative impact of specific word choices on
meaning and tone (e.g., how the language evokes a sense of time and place; how it sets a formal or
informal tone).
DoK
This item requires students to examine figurative meaning in a text by analyzing the authors use of a literary
device (e.g., symbolism).
Page 7.3-42
Sample Item
Page 7.3-43
Math Items
Item Type
Hot Spot
Alignment (i.e., Standards)
3.MD.3. Draw a scaled picture graph and a scaled bar graph to represent a data set with several categories.
Solve one- and two-step how many more and how many less problems using information presented in
scaled bar graphs. For example, draw a bar graph in which each square in the bar graph might represent 5 pets.
DoK
A typical DoK1 item for this standard might ask students to simply identify the bar graph which matches a
table of given data. A DoK2 item like this requires more than basic recall and reproduction. In this case,
students must interpret the context, establish relationships among given information, and construct their
own bar graph.
Further, there is no single solution. This open-ended item allows students to demonstrate their levels of
understanding in ways that go beyond one particular solution.
Page 7.3-44
Sample Item
Page 7.3-45
Item Type
Multiple Choice
5.NBT.4. Use place value understanding to round decimals to any place.
DoK
Students apply their understanding of place value to interpret two given conditions and identify a number
which satisfies both conditions.
This is a DoK2 item because it is not a straightforward round this number. Students must apply their
knowledge of rounding and number sense to successfully solve this problem.
Page 7.3-46
Sample Item
Page 7.3-47
Item Type
Equation Response
4.OA.1. Interpret a multiplication equation as a comparison, e.g., interpret 35 = 5 7 as a statement that

35 is 5 times as many as 7 and 7 times as many as 5. Represent verbal statements of multiplicative
comparisons as multiplication equations.
4.OA.2. Multiply or divide to solve word problems involving multiplicative comparison, e.g., by using
drawings and equations with a symbol for the unknown number to represent the problem, distinguishing
multiplicative comparison from additive comparison.
DoK
Students interpret situations involving multiplicative comparison, an oft-difficult concept for fourth graders.
Students create an equation to represent the multiplicative relationship and offer one possible set of
solutions. As a constructed-response item, students go beyond identifying the solutionthey create
their own. Further, this open-ended item allows for multiple equations and an infinite number of solutions.
Page 7.3-48
Sample Item
Page 7.3-49
Item Type
Graphic Response
G-CO.5 Given a geometric figure and a rotation, reflection, or translation, draw the transformed figure using,
e.g., graph paper, tracing paper, or geometry software. Specify a sequence of transformations that will carry
a given figure onto another.
DoK
This item requires students to apply their knowledge of reflections and translations to construct the
original triangle based on the final triangle and given information. Students construct their own responses
by using the technology to draw shapes in the coordinate plane.
Page 7.3-50
Sample Item
Page 7.3-51
Item Type
Equation Response
F-BF.1.1 Write a function that describes a relationship between two quantities.

F-TF.2.5 Choose trigonometric functions to model periodic phenomena with specified amplitude, frequency,
and midline.
DoK
This item requires students to interpret data presented in the form of a graph. From this graph, students
must recognize the function as a cosine function, as well as the various components of the cosine function
modeled by the graph. Students go beyond basic recall of mathematical ideas. They construct their own
function by interpreting and analyzing the key pieces of the graph, namely, the height of the axle, the
diameter of the Ferris wheel, and the number of rotations per minute. These three key pieces of information
are not explicitly stated by the graph; rather, students must determine them from the graph and then
construct the function modeled by the graph. Thus, this is an example of a DoK3 item.
Page 7.3-52
Sample Item
Page 7.3-53
Item Type
Equation Response
F-BF.2.3
Identify the effect on the graph of replacing f (x) by f (x) + k, k f (x), f (kx), and f (x + k) for specific values of k
(both positive and negative); find the value of k given the graphs. Experiment with cases and illustrate an
explanation of the effects on the graph using technology.
DoK
After students have constructed a function which models the given animation and resulting graph, they
construct a new function based on a different number of rotations per minute. In doing so, they
investigate possible scenarios within the context of a safe Ferris wheel ride. Thus, this is an example of a
DoK3 item.
Page 7.3-54
Sample Item
Page 7.3-55
7.3.2 Test Configuration

Our online tests are designed to be highly configurable. Readers are referred to Section
7.4.9.7 for some sample layouts and skins. We
will work with the department to design the
specific styles and select the layouts to be used
on the Florida Standards Assessment (FSA).
For paper tests, we propose to use consumable
test booklets for the younger students and a
separate answer document for the older students.
Section 7.3.2.1 provides a summary of the test
and answer document design for paper-based
tests (PBTs). Please see Appendix B for the
Materials List.
7.3.2.1 Test and Answer

Document Design
AIR, along with team member DRC, will work
closely with the Department to develop accurate and engaging materials and create aesthetically appealing test materials that appropriately address the rigorous quality requirements of the Department. Non-scannable test
booklets will be designed and developed by
AIR, with print-ready files provided to DRC for
printing and distribution. Please see Section
7.3.7.4 for a detailed discussion of AIRs materials design and production procedures and
Section 7.3.8 for more information on DRC
materials production and printing. For
scannable test booklets and scannable answer
documents, AIR and DRC will partner to produce materials that are compatible with DRCs
image scanning systems (please see Section
7.5.3 for more information on DRCs image
scanning technology and processes).
AIR and DRC will ensure that the answer documents, test booklets, and other test materials
are provided in appropriate formats to schools
and districts. The formats will maximize security and functionality. AIR and DRC underPage 7.3-56
stand that we will also provide supplemental

assessment materials, including secure FSA
Reading passages booklets, available by special
request for up to five percent (5%) of the students taking computer-based tests (CBTs);
blank four-page work folders for all students
taking the FSA Mathematics CBTs, with the
fourth page containing a full-page grid; blank
two-page worksheets for all students taking the
FSA English Language Arts/Literacy (ELA/L)
CBTs; and reference/periodic table sheets, as
appropriate. AIR and DRC understand that
work folders, worksheets, and reference/periodic table sheets will not be numbered or
inventoried, rather they will be destroyed by
the districts/schools after test results have
been delivered. AIR and DRC will ensure that
the Department receives error-free products.
Please also see Section 7.4.7 for more information on other ancillary test administration
materials that will be provided by AIR and
DRC. Appendix B provides AIRs and DRCs
proposed Materials List for the FSA.
Test Booklets
For PBTs and for the regular print paper versions of the CBTs, AIR proposes using consumable (scannable) test booklets for grades 3 and
4 and non-scannable test booklets for all other
grades and subjects. Corresponding scannable
answer documents will be developed to correspond to the non-scannable test booklets.
Scannable test booklets for grades 3 and 4 will
be printed 8 x 11 on 60# reflective stock.
AIR will work with the Department to produce
the test booklets. Once test booklets have
received final approval by the Department,
AIR will provide the final print-ready files to
DRC for printing. The Department will approve
the answer documents prior to printing. DRC
will also develop Braille (contracted and
uncontracted), large print, and one-item-perpage (OIPP) versions of test booklets; please
see Section 7.3.11 for a discussion on special

format materials production.
The test booklets will be color-coded by grade
and subject area. The color scheme for test
booklets will be duplicated across corresponding answer documents and test administration manuals. The test booklets will be customized to show the name of the assessment
program, the Department logo, the form name,
grade, subject, and a place for adhering student Preidentification (PreID) labels. As appropriate, test booklets will be divided into sections that will be administered either on the
same day or on consecutive days. AIR is open
to discussing alternatives for test booklet and
answer booklet design with the Department
upon contract award.
DRC will ensure that all printing of test booklets adhere to strict quality control procedures.
Materials will be packaged and shrink-wrapped
in set and subset quantities specified by the
Department and secured so that each student
is the first to open a booklet. For a full description of DRCs materials packaging and distribution processes, please see Section 7.4.1.
Image Scannable Answer Documents

After the non-scannable test booklets have
been finalized, separate customized, scannable
answer documents will be created, developed,
and printed. Answer documents will be colorcoded to match test booklet and manual colors, as appropriate. AIR and DRC will work
with the Department to design the scannable
answer documents. DRCs custom-designed
answer documents will feature a light background grid and light background lines on the
response area for constructed-response and
essay items, as well as directions to help students easily navigate between the test booklet
and the answer document. In particular, AIR
and DRC recommend that each constructedresponse and essay item be clearly marked in
the test booklet as to the exact location where
students should respond in the answer document, along with instructions after the student
response area directing students to return to
the appropriate page in the test booklet.
Incorporated into each answer document will
be a specially created section for adhering student PreID labels, which contain student identification and demographic information. Please
see Section 7.4.8 for more information on
AIRs PreID process for the FSA.
The answer documents will be designed, produced, and printed to fully support test administration and DRC processing and scanning
technologies and systems. The Department
will approve the answer documents prior to
printing. The scannable answer documents will
be printed 8 x 11 on 60# reflective stock
for all grades (except grades 3 and 4, which
will use scannable test booklets) and content
areas.
Exhibit 7.3.2.1-1 summarizes the assumed
document design for the PBTs. We recognize
that even for CBT paper versions will be necessary for up to 5 percent of the population.
The ITN specifies four additional materials to
accompany paper tests:
Security-controlled booklets containing
passages
Blank work folders for Mathematics
Blank work folders for ELA
Copies of appropriate reference sheets.
AIR can provide all of these in hard copy, and
we have included this price as an option in our
bid. Our base bid, however, includes only the
blank work folders for Mathematics, which
provide scratch paper for students working on
math problems. Our online system provides
well-proven, secure alternatives for the other
documents.
Page 7.3-57
Exhibit 7.3.2.1-1: Summary of the Test and Answer Document Design for Paper Tests and Paper Versions of Online Tests
Assessment
Default Font Size
Test Booklet Pages
Answer Document Pages
Comments
Grade 3 ELAWriting
14 pt.
56
N/A
Scannable test booklet
Grade 3 ELAReading, Listening, Language
14 pt.
64
N/A
Grade 3Mathematics
14 pt.
64
N/A
Grade 4 ELAWriting
14 pt.
64
N/A
14 pt.
64
N/A
Grade 4Mathematics
14 pt.
64
N/A
Grade 5 ELAWriting
12 pt.
40
N/A
Non-scannable test booklet
Grade 5 ELAWriting
12 pt.
N/A
16
Scannable answer document
12 pt.
48
N/A
12 pt.
N/A
16
Grade 5Mathematics
12 pt.
48
N/A
Grade 5Mathematics
12 pt.
N/A
16
Grade 6 ELAWriting
12 pt.
40
N/A
Grade 6 ELAWriting
12 pt.
N/A
16
Scannable answer booklet
12 pt.
48
N/A
12 pt.
N/A
16
Grade 6Mathematics
12 pt.
48
N/A
Grade 6Mathematics
14 pt.
N/A
16
Grade 7 ELAWriting
14 pt.
40
N/A
Grade 7 ELAWriting
14 pt.
N/A
16
14 pt.
48
N/A
12 pt.
N/A
16
Grade 7Mathematics
12 pt.
48
N/A
Grade 7Mathematics
12 pt.
N/A
16
Grade 8 ELAWriting
12 pt.
40
N/A
Grade 8 ELAWriting
12 pt.
N/A
16
12 pt.
48
N/A
12 pt.
N/A
16
Grade 8Mathematics
12 pt.
48
N/A
Grade 8Mathematics
12 pt.
N/A
16
Grade 9 ELAWriting
12 pt.
40
N/A
Grade 9 ELAWriting
12 pt.
N/A
16
12 pt.
48
N/A
12 pt.
N/A
16
Page 7.3-58
Exhibit 7.3.2.1-1: Summary of the Test and Answer Document Design for Paper Tests and Paper Versions of Online Tests
Assessment
Default Font Size
Test Booklet Pages
Answer Document Pages
Comments
Grade 10 ELAFall Retest
14 pt.
48
N/A
Grade 10 ELAFall Retest
14 pt.
N/A
16
Grade 10 ELASpring Retest
14 pt.
48
N/A
Grade 10 ELASpring Retest
14 pt.
N/A
16
Grade 10 ELAWriting
14 pt.
40
N/A
14 pt.
N/A
16
Grade 10 ELAReading
12 pt.
48
N/A
12 pt.
N/A
16
Grade 11 ELAWriting
12 pt.
40
N/A
Grade 11 ELAWriting
12 pt.
N/A
16
12 pt.
48
N/A
12 pt.
N/A
16
EOC Algebra IFall
12 pt.
48
N/A
EOC Algebra IFall
12 pt.
N/A
16
Scannable answer booklet
EOC Algebra IIFall
12 pt.
48
N/A
EOC Algebra IIFall
12 pt.
N/A
16
EOC GeometryFall
12 pt.
48
N/A
EOC GeometryFall
14 pt.
N/A
16
EOC Algebra IWinter
14 pt.
48
N/A
EOC Algebra IWinter
14 pt.
N/A
16
EOC Algebra IIWinter
14 pt.
48
N/A
12 pt.
N/A
16
EOC GeometryWinter
12 pt.
48
N/A
EOC GeometryWinter
12 pt.
N/A
16
EOC Algebra ISpring
12 pt.
48
N/A
EOC Algebra ISpring
12 pt.
N/A
16
12 pt.
48
N/A
12 pt.
N/A
16
EOC GeometrySpring
12 pt.
48
N/A
EOC GeometrySpring
12 pt.
N/A
16
Page 7.3-59
Our system includes secure print on demand,

which enables the test administrator to print
the passage at which a student is looking if (a)
the student has an accommodation set to allow
this (or it can be offered to all students); and
(b) the student presses the request button.
Secure print on demand prints a unique identifier on the document, so that the documents
can be collected and inventoried. The system
prints a cover page on the top and the bottom
of the requested passage to ensure that it is
not viewed by the test administrator (TA) or
others. The cover page identifies the student
who should receive the printout. This approach
is particularly powerful if the Department
chooses an adaptive option, in which students
might receive a wide variety of passages.
We offer several notepad options that students
can use while taking the ELA tests. The system can be configured to offer a single notepad
that persists across items for the duration of
the testing session or a separate notepad for
each item, again persisting for the duration of
the testing session. These note-taking options
have proven sufficient, and we have not
received requests for hard-copy work folders
for ELA tests.
The online system can supply online reference
sheets, customized to the Departments specifications. At the Departments request, we can
make these printable from the TA workstation.
We reiterate that we have included the option
to have these hard-copy documents provided
in addition to the online version, if the
Department so chooses.
We are happy to work with the Department to
modify any configurations of our online testing
system, within the highly flexible boundaries
of our systems configurability. These configurations may be modified at Department direction each administration year within the scope
of this contract with sufficient notice to be
negotiated with the schedule.
Page 7.3-60
7.3.3 Develop Items

Introduction
AIRs structured process follows specific and
proven protocols designed to produce highquality assessment items from the development of the item specifications documents to
the items becoming operational. In section
7.4.9 we describe the many types of machinescored constructed-response (MSCR) items
that we currently support, in addition to
selected-response and human-scored constructed-response items. Our test developers
have the tools they need to provide a wide
variety of cutting-edge, technologically
advanced items that are cognitively demanding, psychometrically sound, and scored
instantaneously using rubrics created by our
test development staff.
As we will describe in this section, AIR has a
comprehensive, structured development
process to ensure the highest quality of developed items. We have the industrys richest
repertoire of item types and machine-scoring
engines. Our test developers are empowered
with tools to create these sophisticated items
independently. They are supported by our
Item Tracking System (ITS), which structures
and manages the test development process,
implementing quality assurance (QA) procedures at every step. Our proposed team brings
years of assessment and instructional experience, including our work with Utahs Student
Assessment of Growth and Excellence (SAGE)
assessment as well as with the Smarter
Balanced Assessment Consortium (SBAC).
Item development quantities will depend on
several policy decisions to be made by the
Department. Our base bid includes the development and field-testing of 75 items per
grade/subject, with separate development of
writing performance tasks. Our bid also
includes a priced option for additional development at the Departments discretion.
Item Development Process

AIRs content development process is managed
by our Item Tracking System (ITS), which fills
the following three critical roles:
Here, we present the process in more detail by

describing our ITS and explaining how it will
be used to manage the item development
process effectively. Our item development
process is predicated on a high level of interaction with our clients, and below we describe
how ITS manages the workflow and maintains
records about item development that meet the
Departments requirements.
1. Content development and management tool

2. Item bank
3. Publication system supporting both paper
and online publication
ITS is a full-featured, customizable item content
management and banking system that enforces
agreed-upon item review levels throughout the
development process. This item development
workflow leads items from inception, through a
series of content, fairness, graphic, and other
reviews to final publication. The system captures the outcomes and rationales at each
review and maintains previous drafts of each
item. The workflow management ensures that
each item receives each review in the designated sequence, and that the review is conducted
(or recorded in the case of committee review)
by an authorized person. Every version of every
item is archived, along with each comment
received in any review. Reviewers have immediate access to all older versions providing version
control throughout development.
ITS allows remote Internet access by item writers and reviewers and by our clients while ensuring complete security with individualized passwords for all users, limited access for external
users, and strong encryption of all information.
Upon publication, the ITS tracks the items use
on a form or in an adaptive item pool. After
items are used, ITS stores the resulting statistics, including exposure statistics, classical
item statistics, and statistics based on Item
Response Theory (IRT).
AIRs ITS is a full-featured, customizable item

content management and banking system,
which enforces agreed-upon item review levels.
Preparing online tests allows a workflow that
moves an enormous amount of work off the
critical path, allowing analogs of reviews to
take place even before the forms are constructed. For example, cover art, layouts, and
other styling are determined in advance and
can be viewed in ITS and the test delivery system well in advance of form construction.
ITS ensures that every item follows through
the entire sequence of development and provides clients and AIR management on-demand
reports of the content and status of the inventory of items. Each item is shepherded through
a sequence of reviews (described in this section) and signoffs before it is locked for fieldtest or operational administration.
As we will discuss in this section, ITS is integrated with the item display engine used by our
test delivery system. This feature, combined
with our unique web approval process allows
the display of online items to be locked well
before forms are built, taking the blackline
process off the critical path for online tests.
Reviewers can look at the items exactly as they
will appear to students and lock them for
publication in exactly that format.
Exhibit 7.3.3-1 summarizes our item development sequence, which can be customized to
support unique requirements in Florida.
Page 7.3-61
Exhibit 7.3.3-1: Item Development and Review Sequence
Item writing
AIRs experienced and multitalented staff has
worked with many projects including SAGE,
SBAC, and many other state assessment contracts. AIRs item writers have at least a bachelors degree in the content area for which they
will be writing items, and many of our item
writers have advanced degrees. Item writers
produce their best items when they understand the cognitive capabilities of the students
in the grade level for which they are writing.
Therefore, when giving assignments to item
writers, we are careful to match the expertise
of the writer, as defined by either previous
item writing experience or teaching experience, with the grade level(s) with which he or
she has the most familiarity.
Page 7.3-62
Additionally, AIR ensures that item writers

have a full understanding of the response
mechanisms described above. In order for item
writers to use our tools to the fullest potential,
those writers must be well grounded in our latest technologies. We take time to thoroughly
train item writers, ensuring that they have the
background knowledge needed to develop
items and rubrics that our test development
staff can easily build upon and refine throughout the review process.
AIR will recruit a sufficient number of item
writers to ensure variety and quality in the
development process. We will provide the
Department with the resumes of our item writers prior to development. It is critical that
item and passage writers have a full under-
standing of the assessment for which they are

writing. We therefore encourage the
Departments participation in item/passage
writer trainings. We will work with the
Department to develop training materials that
address not only the item/passage/test specifications but that also cover any additional
Florida-specific guidelines and incorporate any
specific Department preferences. For example,
we understand that the acceptable topics for
passages might vary from region to region
based on cultural, religious, and political
views. We will work closely with the
Department to ensure that our writers have all
the information they need to develop the quality content that Floridas educators expect.
Writers can access ITS from any computer
with an Internet connection. This is ideal for
two reasons: 1) it gives writers the flexibility to
work from any place of their choosing (e.g.,
home, office, library), and 2) it allows AIR
access to the best writers in the country.
When drafting items in ITS, writers are provided the complete texts of the target standards
and complexity level on the left side of the
screen. On the right side of the screen, the
writers are presented with a type-specific item
template that has sections for each item component (e.g., stem, answer choices). These
templates offer all of the common features of
word processors (e.g., spell check, formatting)
and provide structure to the writers.
ITS offers item developers the ability to independently create imaginative, technologically
advanced items to measure deep knowledge.
Introduction to AIRs Machine-Scored

Constructed-Response (MSCR) Item
Development Tools
AIRs MSCR item development tools put the
power of both item and rubric creation into
the hands of item writers. Below, we provide
some screenshots and brief descriptions of a

few of these tools to help reviewers understand
the power and flexibility available to our
test developers.
Exhibit 7.3.3-2 presents a shot of the first
screen that our test developers see when they
design a graphic-response item. Students can
respond by drawing, moving, arranging, or
selecting graphic regions. The scoring rubric
allows for each answer to be scored using scoring logic created by the item writer.
Test developers have utmost flexibility to identify features of student responses to score.
These features are not simple (e.g., whether the
correct object is put in the correct place) but
can involve abstraction. For example, if a student is asked to design an experiment, the
rubric can discern whether the objects representing the experimental variable actually vary
across conditions or cover the range of inquiry,
among other capabilities. These concepts are
abstracted and many different responses may
reflect those abstract features. This ability
enables our machine rubrics to justify the
partial credit assigned in terms of the skills that
particular response features exemplify.
Exhibit 7.3.3-3 shows the fourth screen the
test developer sees and shows how our system
allows for many different responses.
As shown in Exhibits 7.3.3-17.3.3-3, the tools
provide the ability to test responses a student
might offer in order to evaluate the rubric.
This capability is available to test developers
throughout the review process. The test developer can test the scoring rubric and make corrections to the scoring logic instantaneously.
At any point in the review process, item writers can make changes to the rubrics scoring
logic instantaneously. Those changes can be
tested immediately by scoring the item in the
Review section of the tool. Exhibit 7.3.3-4
shows this scoring capability.
Page 7.3-63
Exhibit 7.3.3-2: Graphic-Response Item Development Tool
The item writer constructs the scoring rubric

to allow for variations in student response.
The item writer creates every aspect of

the display, including features such as
buttons, images, and snap points.
When creating equation items, our test developers have access to the Equation Editor tool.
Student responses can be simple numeric
responses or complex equations or even sets of
equations. This tool allows for multiple answers
and the development of multistep items. As
shown in Exhibit 7.3.3-5, test developers can
customize the equation palette to show the
appropriate functions. Just as the key pad is
customizable, the answer spaces are as well.
Additional answer spaces can be added as needed by the item writer. The scoring rubric allows
for each answer to be scored using scoring logic
created by the item writer. This shot shows how
the test developer can customize the keypad
and answer spaces for different items.
The Simulation Tool (see Exhibit 7.3.3-6)
allows our test developers to independently
Page 7.3-64
Upload and place background

and palette images.
create sophisticated simulations and to define

rubrics that evaluate the examinees sequence
of interactions with the simulator. Their
sequence of choices and interactions can be
scored by the application of scoring logic within the scoring rubric. The item writer creates
inputs, animations, and outputs that give the
student an opportunity to create their answer.
As with the other item types, if the interactions are to be scored, the test developer can
point and click his or her way to a scoring
rubric that is abstracted from the specific
details of a given response to the underlying
features that reflect the knowledge or skills.
Exhibit 7.3.3-6 shows how the test developer
creates the inputs and animations used in simulation items.
Variables relate to the

objects (points, lines,
curves, icons, locations)
in the answer space.
Scoring Assertions relate features of

responses to the skills or knowledge,
which they putatively reflect.
Rationales describe the scoring
assertions for human consumptions
and can explain the score in terms
of the skills displayed.
The item writer will review the scoring logic by

testing possible student answers. He or she can
manipulate the answer space and apply changes
to the rubric instantaneously.
Page 7.3-65
Exhibit 7.3.3-5: Editor Equation Tool

Additional
answer spaces
can be added for
multistep items.
The item writer can

customize the key pad
for each item.
Exhibit 7.3.3-6: Simulation Tool

Create animations
and select the
animation behavior.
The item writer

will create and
customize the inputs.
Page 7.3-66
These tools, along with many more, are integrated into ITS, providing our test developers
the power and flexibility to use technology to
create sophisticated items. We encourage our
test developers to use these tools to create true
MSCR items.
Item and Passage Review

AIR reviews items through a structured
sequence of reviews. Although the exact
sequence of reviews varies across programs, all
our review processes include content reviews;
language accessibility, bias, and sensitivity
reviews; editorial reviews; and lead developer
approval. ITS ensures that each review is complete before an item can make the transition
to the next level of review. ITS captures every
comment and revision to an item by requiring
reviewers to enter annotations and preserving
every version.
During the content review process, reviewers
ensure that the attributes associated with an
item accurately reflect the item content.
Reviewers focus on each items match to its
item specifications as well as on each items
content and structure. Reviewers also check
every item against its alignment to the academic standards to evaluate the correctness of
the alignment. They make sure that the item
and related materials comply with AIRs guidelines for clarity, style, accuracy, and appropriateness for the population being assessed. For
a constructed-response or technologyenhanced item, the specialists study the items
presentation to ensure that it yields a clear
response and check the scoring guide to make
sure that it adequately accounts for all possible
responses. For a multiple-choice item, the
reviewers study the clarity and importance of
each question, check that the correct response
answers the question posed, and ensure that
the incorrect options are indeed incorrect. In
all cases, the reviewers pay special attention to
the quality, equity, and accuracy of the item,
as well as to any manipulatives that form an

integral part of items in some content areas
and to tasks and items intended for students
who take alternate assessments.
The reviewers then accept the item and classification as written, make revisions, or recommend that the item be discarded. Whatever their
recommendation, ITS stores the rationale and, if
an item is revised, archives the older version(s).
Before the Department content staff and committees review an item, AIRs lead item developer for the content area must indicate
approval. The lead item developer will consider all aspects of the item and related materials
for content, language accessibility, bias, sensitivity, and editorial issues, referring to the
items history through ITS as appropriate.
AIR has one of the industrys most thorough
and rigorous processes for item and passage
review. The sections that follow describe our
typical review process at each stage. This is
the review process AIR implemented in the
development of items for both the SAGE and
SBAC assessments. We understand that it is
the Departments intent to develop new items
in subsequent years to embed in future field
tests. The following sections describe the
review process we propose for all new Florida
Standards Assessments items as well.
Preliminary Review Process

During the preliminary review process, AIRs
test developers review items either individually
or as a group. AIR finds the group process beneficial because the exchange of ideas and open
discussion generally yield items that are clearer, more precise, and better aligned to the academic standards.
AIR has developed an internal review process
that aligns with the Departments expectations
for quality item development. At every stage of
Page 7.3-67
the item review process, beginning with preliminary review, test developers analyze items
to ensure that
the item aligns well with the Florida Next
Generation Sunshine State Standards;
the item matches the item specification for
the target being assessed;
the item is based on a quality ideai.e., it
assesses something worthwhile in a reasonable way;
the item is properly aligned to a DOK level;
the vocabulary used in the item is appropriate for the grade/age; subject matter; and
considerations of language accessibility,
bias, and sensitivity;
the content is accurate and straightforward;
the graphic and stimulus materials are
actually necessary to answer the question;
the stem is clear, concise and succinct,
meaning it
has enough information to know what is
being asked;
is stated positively (and does not rely on
negativessuch as no, not, none,
neverunless absolutely necessary); and
ends with a question.
for selected response items, the set of
response options are as succinct and short
as possible (without repeating text), meaning they are
parallel in structure, grammar, length,
and content;
sufficiently distinct from one another;
all plausible (but with only correct
option); and
ordered by length.
there is no obvious or subtle cluing;
the score points for constructed-response
items are clearly defined; and
Page 7.3-68
for MSCR items, the items score as intended at each score point in the rubric.
When reviewing MSCR items and performance-based assessments, the test developers
review these items in ITS web preview, which
allows them to see the item as it will be rendered for students in the online testing environment. Test developers interact with each
item to verify that it directs the student to a
clear understanding of the task, allows the student to provide a coherent response, and contains scoring guidelines that adequately
account for all logical responses.
Based on this meticulous review of each item,
the preliminary reviewers accept the item and
classification as written or revise the item,
attributes, or classification or all three.
Another alternative is to reject the item
because it is too problematic in content or does
not align to any standard, or both. Whatever
the recommendation, the review comments are
noted in ITS, and if an item is revised, its previous version is automatically archived.
Content Review One

Content Review One is generally conducted by
an individual test development specialist,
rather than as a group review. The Content
One reviewer carefully examines each item
based on all the criteria above. In most cases,
he or she plays a more senior role on the content team and brings more years of knowledge
and experience to the review. This reviewer
approaches the item both from the perspective
of the Departments guidelines as well as his or
her own experience in test development.
Content Review One is where items are
checked to ensure that the revisions made at
the preliminary review step did not introduce
errors or content inaccuracies. The Content
One reviewer looks at all aspects of the item
and reviews the comments saved in ITS from
the preliminary review to verify that any issues

noted there have been adequately addressed.
Edit Review
AIR editors will review every item for clarity,
correctness, and appropriateness of language
for the grade level assessed and conformity
with acceptable item-writing practices. Editors
ensure clearly worded, understandable, and
fair presentation of items, instructions, and
administrative documents. The editorial
process is related to, but separate from, the
item development process so that editors can
look at the items objectively.
Editors have numerous tasks. First, editors perform basic line editing for correct spelling, punctuation, grammar, and mathematical and scientific notation, ensuring consistency of style
across the items. AIR editors have adopted standard reference resources, using Merriam
Websters Collegiate Dictionary (11th edition)
and Websters Third International Dictionary,
Unabridged for spelling and capitalization;
Associated Press Stylebook and Briefing on
Media Law for usage; Words Into Type for grammar; and the Publication Manual of the
American Psychological Association for reference citations. Subject-specific reference
sources include Websters New Biographical
Dictionary and Websters American
Biographies; The Harvard Dictionary of Music;
Jansons History of Art; Websters New
Geographical Dictionary; Dorlands Illustrated
Medical Dictionary; Mathematics Dictionary
(James and James); Scientific Style and Format:
The CBE Manual for Authors, Editors, and
Publishers (Council of Biology Editors); and The
Macmillan Dictionary of Measurement.
AIR will use the Departments style guides that
synthesize frequently used and assessment-relevant information that addresses such test-specific issues as numerals, abbreviations, symbols, and terminology. AIR editors have
become very familiar with the Departments

preferred format and style.
Second, editors ensure that all items are accurate in content and will query the item developer when questions arise. The editors compare reading passages against the original publications and make sure that all information is
internally consistent across stimulus materials
and items, including names, facts, or cited
lines of text that appear in the item. The editors ensure that the keys are correct and that
all information in the item is acceptable and
correct. For example, editors verify common
facts (e.g., the diameter of Earth, the scientific
name of the fruit fly, the correct spelling of
Far from the Madding Crowd, the formula for
calculating the volume of a cube, the proper
use of the semicolon, and the dates of the Civil
War). Mathematics assessments present a specific challenge because of the potential for
transposing numerals and the difficulty of
identifying computation errors by sight.
Therefore, our editors perform all calculations
to ensure accuracy.
Third, AIRs editors review all material for fairness and language accessibility issues.
Although external committees and the lead
item developers look at all material, editors
raise questions before the material reaches the
committees and then recheck all material that
undergoes any changes.
Finally, our editors confirm that items reflect
the accepted guidelines for good item construction. For example, in Mathematics items,
they ensure that options given with calculatoractive items reflect errors in thinking, not
errors in calculator use. In all items, they look
for language that is simple, direct, and free of
ambiguity with minimal verbal difficulty.
Editors confirm that a problem or task and its
stem are clearly defined and concisely worded
with no unnecessary information. For multiple-choice items, editors check that options
Page 7.3-69
are parallel in structure and fit logically and

grammatically with the stem and that the key
accurately and correctly answers the question
posed, is not inappropriately obvious, and is
the only correct answer to an item among the
distractors. For constructed-response items,
editors review the rubrics for appropriate style
and grammar.
Senior Review
We seek to deliver innovative, complete, and
high-quality items to the Department. A subject area content lead must sign off on the
overall quality, including content, fairness, language accessibility, and style before it is
deemed ready for Department review. Senior
reviewers are generally AIR staff who work
closely with their Department counterparts
and are keenly aware of the specific expectations for item content and quality.
By the time an item arrives at the senior review
level in ITS, it has been thoroughly vetted by
both content reviewers and editors. The senior
reviewer looks back at the items entire review
history, making sure that all the issues identified in that item have been adequately
addressed. For MSCR items, the senior reviewer
carefully checks the rubric and scoring logic by
responding to the task in ITS web preview just
as the student would in the testing environment. He or she checks full-credit, partial-credit
and no-credit responses to verify that the scoring is working as intended. The senior reviewer
verifies the overall content of each item, confirming its accuracy, alignment to the standard,
and consistency with the Departments expectations for the highest quality.
Client Review and Resolution

The department review is a critical step in the
overall quality and adequacy of the assessment. AIR will deliver newly developed items
in batches to allow our test development spePage 7.3-70
cialists to calibrate these items to the expectations of the Department reviewers. The initial
batches will contain items that target a representative sample of the standards, item types,
and DOK. After the initial batch delivery, the
item distribution across the content and DOK
in subsequent batches will vary according to
the item development plan.
After we receive feedback for each item batch,
our test development specialists and
Department staff will hold a conference call to
discuss the Departments comments and finalize revisions.
Committee Review
Each item and passage will be reviewed by a
committee of Floridas stakeholders. AIR will
document those reviews in ITS as items move
through the review sequence. Section 7.3.4
provides greater detail about the committee
review process.
Annotation of Items for Accessibility

One feature of Floridas Next Generation testing systems is their focus on accessibility for
all students. One of the elements critical to
our success has been the ITS workflow that
supports the embedding of accessibility
markup into each item as it is being developed, rather than as a post-hoc process
applied to completed test forms. The development of transcription notes (a process that we
call tagging for web-delivered items) proves
most tractable when it is done upon approval
of the base item. That way, tagging is accomplished (and reviewed by the Department) on
a flow basis, rather than requiring the review
of thousands of items at the same time.
Currently our tagging capabilities include
translation, text-to-speech, Braille, sign language, and word lists or glossaries (which may
be multilingual).
Tagging Process
Once items have been vetted by Floridas committees and approved by the Department, they
are eligible to move into the annotation phase
of development. Here, items can have text-tospeech (TTS) and text-to-Braille features
added, among others. AIRs team of TTS specialists put the items through a series of
review steps to ensure the annotations are
clear and precise. Once the annotations have
been created and reviewed internally by AIR,
the Department will have the opportunity to
review the annotations and confirm them.
Signed Translation (ASL)Text-based content is presented in American Sign

Language and/or Signed English.
Language TranslationPresentation of
items and passages, including figures,
graphs, and maps, in select languages displayed next to the English version.
Glossaries and Word ListsFeature that
allows students to view glossary terms for a
specific word or phrase. A glossary may
present information in English or in an
alternate language.
Final Client Signoff

In addition to text-to-speech and text-toBraille, items can be associated with a variety
of accessibility tools and accommodations that
can be delivered to students automatically
based on their profile. Online delivery of AIRs
assessment system ensures that students can
take a test individualized to meet their needs.
Some accessibility features are completely
handled by the test delivery system, without
any special changes to content. For example,
items can be zoomed or magnified without
special preparation of the content. Other features require annotations, which must be
added as part of the content development
process. Accessibility features requiring annotations within the content include:
Spoken Presentation of Items and
PassagesReading text-based content and
providing descriptions of visual material such
as figures, graphs and maps, through TTS.
BrailleTactile presentation of text-based
content and visual materials such as figures, graphs, and maps through a refreshable Braille display, screen reader (JAWS),
and embosser. Both contracted and uncontracted Braille, as well as Nemeth Braille,
are supported.
Once the full complement of annotations is

complete, we have a final web approval phase
in which our clients see the final product displayed as it will render to the students. At this
phase, clients can approve an item, request
revisions, or reject the item. This final signoff
is analogous to a blueline review but includes
the review of general and special versions.
AIR implements a host of quality procedures
after this final signoff. Readers are referred to
Section 7.3.9 for a discussion of these processes, which include review of each item on each
platform.
Intended Development
For pricing purposes, AIR has assumed that we
will develop 75 items per grade/subject each
year. Of course, this figure will depend heavily
on Floridas preferred plan for administration,
how tightly coupled Florida would like to
remain with the base test, and other factors.
We propose to establish a target in negotiations, should the Department choose to negotiate with us.
Page 7.3-71
7.3.4 Review Meetings

Introduction
Florida engages its educators and stakeholders
in the test development process through a
series of committee reviews designed to ensure
that the test, blueprint, items, and passages all
accurately measure the intended construct,
and does so in a way that is fair to all students
and respects community sensitivities. In a typical development year, these committees are
engaged as the items are developed.
The test to be delivered in 2015 presents a
slightly different situation. Timelines dictate
that the items to be used in the 2015 assessment will come from a third-party source.
Here, we have proposed to use items from
Smarter Balanced Assessment Consortium or
the Student Assessment of Growth and
Excellence (SAGE).
The fact that these items already exist and
were independently developed to measure the
Common Core State Standards (CCSS) does
not eliminate the need to ensure that they
conform to Floridas expectations. Each of the
proposed item banks is large because it is
designed to support an adaptive test. This
means that the bank is large enough to select
items from it for use on fixed forms, or to
eliminate unacceptable items even if the test is
adaptively administered.
In preparation for the 2015 test, we propose to
engage the Florida committees in the development of the test blueprint (if the Department
chooses not to adopt the SAGE or Smarter
Balanced blueprint), and the selection of
items. In subsequent years, we propose to
engage the committees in their usual roles in
the development process.
As detailed in Section 7.3.1, the bank developed for SAGE contains approximately
Page 7.3-72
300500 items per subject per grade (grades

311 and End-of-Course Mathematics). The
items in the bank consist of items developed to
measure Common Core State Standards
(CCSS)reading, writing, language, listening,
and math. AIRs proposal will enable Florida to
select fixed forms or review and approve the
pool for an adaptive pool. The pool will be
robust and provide ample opportunity for
Floridas Content and Fairness committees to
review the selection.
As stated in the ITN, the Department will
select participants for committee meetings.
Once appointed, AIR will coordinate with the
Department and secure contact information
and a description of each committee members
background. AIR will maintain all committee
forms (e.g., reimbursements, evaluations, nondisclosure agreements) and will make these
forms available to the Department upon
request.
AIR will work with the Department to select a
city in which to hold the committee meetings.
The cities selected, probably Jacksonville,
Tallahassee, or a location in central Florida,
will be ones that are easily accessible by a
majority of participants. AIR will then invite
the committee members to the meetings and
make all necessary travel and hotel arrangements.
AIR will be responsible for all materials needed
for the meetings. AIR will prepare and ship
hard copy booklets and also will be prepared
to provide calculators, dictionaries, and any
other reference materials the committee may
need. Throughout the meetings, AIR will maintain the security of test items. AIR proposes
that all passages and items will be printed on
light green paper, making them instantly recognizable as secure materials. All materials will
be monitored throughout the meetings and
stored in locked areas when they are not in
use. Additionally, since the Item Tracking
System (ITS) can only be accessed through a

secure login, passages and items being
reviewed online also will be secure. AIR will
work with the Department to provide general
test security guidelines to the committees
prior to passage and item review meetings.
7.3.4.1 Bias and Sensitivity

Committee Reviews
AIR standards require that tests exclude items
that disadvantage examinees of different races,
genders, ethnicities, regions, handicapping
conditions, or language background. Further,
the standards require documentation that such
disadvantage does not occur. (Note that this
does not imply that all groups perform equally
well on all items, but rather that any systematic differences are due to differences in the proficiency levels of the groups on the targeted
trait.)
AIR pairs language accessibility with the bias
review because language accessibility is an
aspect of fairness. This review accentuates
AIRs commitment to ensuring that items and
any associated stimulus materials are without
bias, sensitive issues, controversial language,
stereotyping and statements that reflect negatively on race, ethnicity, gender, culture,
region, disability, or other social and economic
conditions and characteristics.
Passage and Item Bias Meetings

20142015 School Year
An integral practice to ensure these standards
are met is the Passage and Item Bias
Committee meetings. For the first year of the
contract, AIR proposes combining the Bias and
Data Committee meetings to review the SAGE
or Smarter Balanced passages and items.
Participants will review the bank to determine
if passages and items meet Floridas bias and
fairness standards. Since the items will already

have been field-tested, the committee will not
recommend any changes to content. Rather,
the committee will recommend passages and
items to be selected into the pool.
Our baseline bid is based on the assumption
that AIR and the Department will select
enough items for three fixed forms from each
pool (approximately 150 items per grade/subject) to take to the committee. We propose to
organize the committees by grade band, with
610 members per grade band for three grade
bands. We anticipate that each grade level will
take approximately 3 days to complete the
review. AIR proposes to hold the meetings in
late summer or fall of 2014.
The workings of the reviews and the details
will follow the plans outlined below.

Other Years
AIR proposes that our typical roles for committee meetings be used for both embedded
field test (EFT) items developed for the 2015
test and for subsequent years of the contract.
This annual review will take place after passages and items have passed through preliminary stages of item development and bias
review by staff at the Department and AIR. As
part of the review process, all items and itemrelated materials (e.g., stimuli, manipulatives)
are reviewed by an outside panel of Florida
educational stakeholders. This Bias Review
Committee will comprise Florida state educators from across the state, so that there will be
a diverse representation of the Florida population.
AIR will work with the Department to create
materials to train committee members. AIR
proposes the following topics to be covered in
the training to establish guidelines for the Bias
Committee meetings:
Page 7.3-73
Sensitivity
Diversity
Stereotyping
Inflammatory or controversial subjects
Ethnocentrism
Differentially familiar subjects
Appropriate language and grammatical constructions
Appropriately accessible graphics
AIR will distribute training materials that
encompass the pertinent bias guidelines to
each committee member. The materials also
will include documentation to cover the following:
Purpose of bias review
For Bias meetings, we propose two committees

organized by grade band. We anticipate the
Bias meetings lasting 2 days with approximately 30 committee members organized into
working groups. AIR will work with the
Department to establish the target number of
passages and items to review each day. AIR
proposes that the meetings to review the EFT
items developed for the 2015 test be held in
late summer of 2014. As described in the ITN,
AIR proposes to hold the meetings in October
in subsequent years.
Passage and Item Community

Sensitivity Committee
Guidelines
Examples for guidelines
Exceptions for guidelines
AIR content staff will support the facilitator
from the Department to ensure that the bias
guidelines are used to discuss all passages and
items. The possible outcomes for items and
stimuli that are reviewed at the Bias
Committee meeting are: accept as appears
(AA), accept as revised (AR), and reject (RJ).
AIR will take notes and record committees
decisions in ITS.
AIR will prepare the following materials well in
advance of the meeting:
Meeting agenda
Bias Committee member contract
Ancillary background information the committee member will need to participate
fully in the discussions at the meeting
AIR will also print hard copies of passage and
item books and will be prepared to project the
Page 7.3-74
passages and items from ITS web preview.

After the Bias meetings, AIR and the
Department will meet to reconcile decisions
made by the committee. AIR will then implement changes approved by the Department.
Before moving forward to final review, passages

and items will be reviewed by the Community
Sensitivity Committee. This panel of community members will be composed of Florida citizens who represent various organizations and
groups across the state. The committee members will review passages and items, and consider the different groups represented by the
people of Florida. The committee will work to
ensure that the passages and items do not
inadvertently disadvantage any groups of students.
The community review is particularly important given the political focus on the Common
Core State Standards. The SAGE exam has
withstood examination by a politically appointed committee of parent reviewers, and come
out with flying colors. In Utah, the legislature
appointed 10 parent reviewers, and the school
board appointed five, to review the SAGE
items. The committee included several members who vocally opposed the common core,
but nonetheless supported the assessment
after viewing the items. As reported by the

Deseret News (Salt Lake City), one committee
member summarized the committees findings,
I think everyone, based on our comments
this morning, we all feel comfortable with
the test. We dont feel like its skewed, that
theres a political agenda. (Parent
Committee Member Alyson Williams, quoted in Benjamin Wood (2013), Parent committee completes review of new adaptive
testing materials, Deseret News, Nov. 8,
2013.)
The careful construction of the SAGE exam to
be sensitive to the concerns of the Cores political opponents focuses the test clearly on the
important, rigorous skills and content contained in the Core. Of course, community sensitivities in Utah differ from those in Florida,
but the parent review provides one more piece
of evidence that the SAGE items will likely satisfy Floridas community representatives.

Sensitivity Committee
20142015 School Year
As with the Fairness Committee review, the
item pool will have to be reviewed by the
Sensitivity Committee prior to the first operational use in Florida. AIR will have the
Community Sensitivity Committee review the
items selected for the Florida pool. The in-person committee meeting will make use of our
Content Rater System, a system that exists
within ITS. (The Content Rater System is
described in detail in the following section.)
The items selected for the pool (up to 150 per
grade and subject) will be available to the
Community Sensitivity Committee members.
They will review the passages and items and
recommend to the Department if an item
should be accepted or rejected. The system
also will allow for committee members to
respond to specific questions about criteria
relevant to sensitivity reviews. The

Department will make the final determination
about whether to select passages and items
into the pool.
AIR proposes that the Community Sensitivity
Committee take place concurrently with the
combined Bias and Data Review meetings in
late summer of 2014. We will organize the
committees to complete their work within 3
days the first year.

Sensitivity CommitteeOther Years
As with the first year of the contract, AIR proposes using an in-person committee meeting
supported by our Content Rater System both
for EFT items developed for the 2015 test and
for subsequent years of the contract. If the
Department chooses, however, AIR is prepared
to hold online meetings for this review. Again,
the entire pool will be available to the
Community Sensitivity Committee members.
Participants will review the passages and
items, but they also will be able to log comments to the Department recommending
changes to content.
The online review will allow committee members to see how passages and items will appear
to students and how they will function.
Through the Content Rater System, committee
members will be able to access relevant attribute information (e.g., genre, text complexity,
standards, Depths of Knowledge [DOK]) so
that they can reference passage- and item-relevant traits easily. Exhibit 7.3.4.1-1 provides an
example of an item as it appears to participants. The question appears on the right side
of the screen. On the left side, participants
indicate the items acceptability and have the
option to leave comments. This can be configured to the Departments specifications.
Page 7.3-75
Exhibit 7.3.4.1-1: Content Rater System for Online Item Review
As they review the banks content, committee

members will be able to flag items they wish to
revisit, leave comments, and recommend if an
item should be accepted or rejected. AIR proposes that the committee log comments and
recommendations for the Department in the
Content Rater System. The Department can
then review the comments and make final
decisions about whether to accept or reject the
recommendations made. After the Department
reconciles recommendations from the committee, AIR will implement any necessary
changes. Exhibit 7.3.4-2 shows a sample log.
Again, the system will be configured to the
Departments specifications.
After the committee meetings, AIR will generate reports directly from the Content Rater
System. The reports, which export as Excel
documents, will provide the acceptance and
rejection rate of items and also will include the
comments left by each reviewer.
training materials for the review. In advance of
Page 7.3-76
the meeting, AIR will prepare the following

materials:
Meeting agenda
Sensitivity Committee member contract
Ancillary background information the committee member will need to participate
fully in the discussions at the meeting.
AIR proposes assigning batches to committee
members, which will not only allow items to
be easily tracked, but will also ensure that all
items will have the required number of
reviews. Committee members will then simultaneously review these pre-loaded batches,
allowing for thousands of items to be reviewed
in a short window of time.
As stated in the ITN, this annual review will
take place concurrently with the Bias
Committee meetings and will last 2 days, with
approximately 30 participants. AIR will work
with the Department to establish the target
number of passages and items to review each
day.
Exhibit 7.3.4.1-2: Sample Batch Item Review Log
7.3.4.2 Passage and Item Content

Review Committees
The Passage and Item Content Review
Committees, which are made up of Florida
educators and content experts, will be responsible for reviewing passages, items, and scoring
rubrics from a state educators perspective.
Content reviewers will focus on item and scoring-rubric quality, the appropriateness of passages and items for the grade level, the representative nature of the item set, and the match
of the items to Floridas standards and test
specifications. AIR will work with the
Department to determine additional attributes
on which they want the committee to focus
(e.g., DOK).
Passage and Item Content Review

Committees20142015 School Year
For the first year of the contract, participants
will review the items selected for potential
inclusion in the pool to confirm that the items
and passages appropriately reflect Floridas

content standards. Since the items will already
have been field-tested, the committee will not
recommend any changes to content. Rather,
the committee will recommend passages and
items to be selected into the pool. The
Department will make the final decision about
which passages and items are chosen.
For the Passage Content Review Committee
meetings, AIR proposes nine groupsone for
each gradeof eight committee members. For
the Item Content Review Committee meetings,
AIR proposes nine groupsone for each subject and gradeof eight committee members.
We anticipate the meetings will last approximately 5 days. AIR will work with the
Department to establish the target number of
passages and items to review each day. AIR
proposes that Passage Content Review
Committee meetings be held in late summer
2014 and Item Content Review Committee
meetings be held in early fall of 2014.
Page 7.3-77
Passage and Item Content Review

CommitteesOther Years
As with the first year of the contract, reviewers will focus on item and scoring-rubric quality, the appropriateness of passages and items
for the grade level, the representative nature of
the item set, and the match of the items to
Floridas standards and test specifications.
However, for both the EFT passages and items
developed for the 2015 test and for passages
and items developed in subsequent years,
committee members will be able to make recommendations on how to revise content.
AIR will prepare the following materials well in
advance of the meeting:
Meeting agenda
Item Content Review Committee contract
Copy of the Florida Standards
Ancillary background information the
member will need to participate fully in the
discussions at the meeting
AIRs management staff, content team, and
assistants will be onsite to manage the flow of
the meetings. AIR content staff will support
the Department, helping to facilitate orienting
the committee members to the item review
task, presenting the items for review, and
recording all recommended revisions and committee comments.
AIR will be prepared to project each item in
ITS web preview. Committee members will
have the opportunity to review both the content and the online presentation of the each
item and stimulus. Moreover, the committee
will be able to see how students interact with
machine-scored constructed response items
and also be able to review how responses
score. Notes and other comments will be
entered into ITS live, making sure the committees recommendations are fully and accurately
Page 7.3-78
captured. All previous versions of passages and

items are archived in ITS and, if necessary,
can be easily retrieved.
After the committee has concluded its review,
AIR and the Department will have a resolution/reconciliation meeting to go over the committees recommendations. At the meeting,
AIR and the Department will review the committees recommendations to discuss what
revisions are needed and to determine the
final outcome of each item. At the conclusion
of each days meeting, AIR staff will collect and
secure all materials.
As detailed in Section 7.3.3, stimuli and items
will go through additional editorial and content
reviews to ensure changes have been resolved
correctly.
For the Passage Content Review Committee
meetings, we propose nine groupsone for
each gradeof eight committee members. For
the Item Content Review Committee meetings,
we propose nine groups one for each subject
and grade of eight committee members. We
anticipate the meetings will last approximately
5 days. AIR will work with the Department to
establish the target number of passages and
items to review each day.
AIR proposes to hold Passage Review
Committee meetings in August and Item
Review Committee meetings in October, as
described in the ITN.
7.3.4.3 Data Review Committee

Data Review Committee20142015
School Year
As detailed in the section on Passage and Item
Bias Meetings, AIR proposes combining the
Bias and Data Committee meetings for the first
year of the contract. Participants will review
selected items and passages to determine if
passages and items meet Floridas bias and

fairness standards and make recommendations
about which passages and items to be selected
into the pool. AIR proposes to hold the meetings in late summer or early fall of 2014.
Data Review CommitteeOther Years

The Data Review Committee, comprising state
content experts, grade-level teachers, Florida
accessibility and accommodations experts, and
psychometricians, will review field-test item
statistics. AIR will work with the Department
to determine the criteria on which to review
items. Prior to the meeting, AIRs psychometric team will analyze items according to these
criteria. They will then provide to the
Department a Master Data Sheet (MDS) and
access to the ITS item cards. The MDS is a
Microsoft Excel data file that contains a comprehensive summary of the statistics on each
of the field-test items. Both files provide information on the items that the Department can
use to determine if the items will remain in
the pool or whether specific items should be
removed based on some of the field-test statistics. Items that do not meet the Departments
criteria will be flagged. AIR proposes that these
items will then be reviewed by the Data
Review Committee.
AIR proposes to use in-person Data Review
Committees. We will prepare item cards and
have the committee review only the items that
are flagged for a high risk of bias. In general,
this is a relatively small proportion of the
items field-tested. AIR staff will support the
Departments facilitator as the committee
members review items to ensure they behave
as expected. The committee also will see that
items include all correct responses and review
differential item function (DIF) statistics to
ensure the items are fair to all students.
After the meetings, AIR will meet with the
Department to reconcile items reviewed by the
committee. The Department will make a final

recommendation and determine which items
are eligible to move into the operational pool.
For the Data Review meetings, AIR anticipates
the meetings will last 2 days with approximately nine members. As stated in the ITN, the
meetings will be held one time annually in
May or June. AIR will work with the
Department to determine during which month
the meetings will be held.
7.3.4.4 Content Advisory

Committee
With the guidance of the Department, the
Content Advisory Committee (CAC) will
advise the Department about the scope of the
assessments. Specifically, the committee will
provide recommendations about the overall
number of items, the types of items used to
assess each standard, and the standards selected to be assessed at each grade and on each
assessment. AIR will work with the
Department to establish the materials and
process required for the CAC. AIR proposes
using a comprehensive set of materials to help
facilitate the process, as follows:
Passage Specifications
Item specifications
Pool Analyses
Blueprints
As stated in the ITN, the CAC will convene for
3 days, three times per year for the first year
of the contract and two times per year for
each subsequent year of the contract. AIR proposes three groups per subjectone for each
grade band (i.e., grades 35, grades 68, grades
911)of approximately 45 committee members. AIR will work with the Department to
establish the time frame in which the meetings
will be held.
Page 7.3-79
7.3.5 Develop Field-Test Item Sets

and Forms
Schedule A of the ITN shows which tests will
exist online and which will remain as fixed
paper forms until a later transition. The transition to computer-based tests (CBT) is a key
issue of which we are cognizant for the embedded field tests (EFT), and we accommodate the
transition to a different mode in our proposed
EFT approach.
Specifically, the first administration under this
contract will occur during the spring of 2015.
Hence, this will be the first year in which we
could embed field-test items.
However, we do not propose to embed fieldtest items onto a paper form if the state is
transitioning to a CBT in the following year.
Item parameters for an item from a paper form
may not hold when that same item is presented using a different mode of administration.
Therefore, item parameters for an online test
should also come from an online field test.
For this reason, AIR recommends no field testing of items in the year preceding a transition
to a CBT (for instance, grade 4 ELA transitions
to a CBT in 20152016). Hence, the paper
form in 20142015 will not include EFT items
given that the item parameters from the
20142015 paper form are not generalizable to
the online mode of administration.
Exhibit 7.3.5-1 provides a summary of when
AIR proposes to implement an EFT for each
grade and subject and the mode in which the
EFT will occur.
Gap Analysis to Determine Needs of

the Item Bank
Each year, AIR proposes to perform a detailed
operational item pool analysis, identify any
gaps in the item pool, and develop new items
Page 7.3-80
to fill the gaps, as well as to augment the item

pool for the future test administrations. The
gap analysis will proceed by investigating the
item pool with respect to the following criteria:
Number of items in the pool for each
strand relative to the demands of the test
blueprint
Range of item difficulty for items within
each strand
Distribution of item type (e.g., selected
response) in the pool
If the state should choose an assessment to be
delivered adaptively, then the following would
also be applied in the gap analysis:
Examine items that tend to be exposed
often (item exposure is a possible indicator
of an area within the bank that needs
attention)
Simulate testing conditions using AIRs simulator tool (this tool can be used to project
where the algorithm has difficulty meeting
the blueprint given a lack of items in the
pool).
Proposed Number of Field-Test Items

and Forms
The large number of tested students in each
grade in the state makes it possible to field test
many items and grow the item bank quickly.
Our proposed approach is to field test approximately 1.5 times the number of items needed
to build a test form for each grade and subject
in the years in which we conduct the EFT.
This ensures that the item banks each year
will be sufficiently deep to support construction of equivalent test forms even if some
items are dropped during item data review. Of
these items, approximately 30% will be
machine-scored constructed-response (MSCR)
items. For paper administrations, when possible, machine-scored items from the online
Exhibit 7.3.5-1: Summary of Planned Embedded Field Testing

Grade
3
Subject
ELA/L
Mathematics
ELA/L
Mathematics
ELA/L
Mathematics
ELA/L
Mathematics
ELA/L
Mathematics
ELA/L
Mathematics
Algebra I
End-ofCourse
Algebra II
Geometry
20142015
EFT to support future
PBT
PBT
No EFT
PBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
No EFT
CBT
CBT
CBT
20152016
PBT
No EFT
CBT
No EFT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
pool will be rendered for paper administration,

but scored by hand. For machine-scored items
that cannot be rendered faithfully on a paper
test form, AIR will identify replacement constructed-response (CR) items to measure the
same reporting strands.
At Floridas request, AIR would be pleased to
support additional item deliveries (in batches
of 75 items) if the state feels additional item
development is needed to support future test
administrations or alternate forms.
20162017
No EFT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
20172018
CBT
EFT to support for CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
CBT
Exhibit 7.3.5-2 provides the number of items,

by type, and forms that we propose as a part of
our base proposal. Differences in the exhibit
reflect the differences in the expected mode of
field-test delivery as described in Exhibit
7.3.5-1. The paper forms will include 55 selected-response items, 15 scannable constructedresponse items, and 5 hand-scored items.
Online forms will not include hand-scored
items, but instead will include a total of 25
machine-scored constructed-response items.
ELA tests will include up to 10 items to
accommodate either a large number of items
per passage or multiple passages. The paper
Page 7.3-81
form field-test design will reflect the online

design once the transition occurs.
algorithm describes. AIRs online field-test

algorithm as described here will be used to
administer the items. The method applies similarly to items or intact forms.
EFT for Online Forms

Here we describe the sampling methodology
that will be utilized when the forms are administered online. If the CBT is in fact a fixed
form, then 15 forms with 5 items per form will
be constructed with the EFT items placed onto
the same, fixed positions across each form. If
the Department chooses an adaptive option,
then the 75 items will be added into the test
pool and seeded into the test forms as the
The algorithm employed by AIRs field-test

engine ensures that
efficient samples are used for all
items/forms by randomly selecting from
among the least administered items/forms
to ensure that resulting differential item
functioning (DIF) statistics are maximally
efficient across all items;
Exhibit 7.3.5-2: Number and Types of Items and Forms Proposed as Part of Our Base Proposal
Item Type
Grade
EndofCourse
Subject
Mode
Target
Test
Length
50
50
Total
EFT
Items
75
75
Selected
Response
Paper
Paper
Target
Number
of Forms
1.5
1.5
ELA/L
Mathematics
ELA/L
Mathematics
ELA/L
Mathematics
ELA/L
Mathematics
ELA/L
Mathematics
ELA/L
Mathematics
Algebra
I
Online
Paper
50
50
75
75
50
55
25
1.5
Online
Online
50
50
75
75
50
50
25
25
10
5
Online
Online
50
50
75
75
50
50
25
25
10
5
Online
Online
50
50
75
75
50
50
25
25
10
5
Online
Online
50
50
75
75
50
50
25
25
10
5
Online
50
75
50
25
Algebra Online
II
Geometry Online
50
75
50
25
50
75
50
25
55
55
MSCR Scannable Hand- EFT

Number
CR
Scored Items
of
CR per Form Forms
15
5
10
15
15
5
5
15
15
Note: ELA/L excludes the writing performance task, which is discussed in Section 7.3.6.
Page 7.3-82
10
5
15
position effects are averaged out by randomly administering test items across test
positions; and
more robust linkages exist among items,
because each item is linked with every
other item across hundreds of unique test
forms to a degree not manageable
through a set of fixed forms.
The field-test algorithm randomly assigns both
the field-test items and the field-test item position, ensuring that
thus causing the equating between forms to be

less precise.
The design effect, or variance inflation factor,
occurs when student responses arise from
within a similar instructional unit. In such
cases, the nominal sample size (i.e., actual
number of students) can be substantially larger
than the effective sample size (i.e., number of
students providing unique information). Kish
(1965) defines the design effect as:

a random sample of students is administered each item; and

for any given item, the students are sampled with equal probability.
AIRs field-test algorithm yields a representative, randomized sample of student responses
for each item. The field-test algorithm also
leads to randomization of item position and
the context in which items appear. Field-testing each item in many positions and contexts
should render the resulting statistics more
robust to these factors.
EFT for Paper Forms

Exhibit 7.3.5-1 shows when paper-based forms
will include EFT items. On each of the proposed 15 forms, EFT items will be placed onto
the same, fixed slots on each form with each
form containing 5 items.
Additionally, we must consider the sampling
design of these forms to produce the most efficient item statistics. Administering the same
form to students within a similar group, or
cluster, produces what is referred to as a clustered sample design (Kish, 1965). The clustered sampling design, however, yields estimates that are less precise than would be
obtained under a pure random sample (Hedges
& Hedberg, 2007; Kish, 1965). This imprecision also propagates into the equating error,
where m is the number of units in a cluster

and is the intra-class correlation. Holding
m constant, as tends to 0, the design effect
tends to 1, in which case there is no variance
inflation. As becomes larger, the design
effect exceeds 1, causing for variances that
ignore the design effect to be underestimates
of the true sampling variance. Holding
constant, we see that larger clusters yield larger design effects than smaller clusters. For
instance, a district-level cluster will typically
have larger design effects than a school given
that districts are larger than schools.
The loss of information due to clustering can
be insidious. Item response theory (IRT) software designed to estimate simple random-sample standard errors simply report the standard
errors of the item parameters when applied to
a clustered sample design. The reported standard errors overstate the precision of the estimates. Therefore, the lower precision only
becomes apparent when unexpected changes
in performance is observed over time, or items
drift more dramatically than the reported
standard errors would imply. This is critically
important for Florida as the state intends to
examine item/scale drift over time.
Page 7.3-83
For the items with stimuli, such as items associated with a simulation and research/essay
tasks associated with some reading passages, if
an item that is part of a shared stimulus group
is selected, then all items in that group will be
administered.
Variance Inflation in Clustered Designs

Exhibit 7.3.5-3 provides information on the
consequences of a clustered design on the
effective sample size. Each row in the table
assumes four forms and 1,000 students administered each form. For illustration, the table
assumes 200 classrooms, 100 schools, and 50
districts. For planning purposes, we assume an
intra-class correlation of .10 at the district
level, .15 at the school level, and .20 at the
classroom level. The values are on the order of
magnitude recommended by Hedges and
Hedberg (2007), as described in their paper on
power calculations for clustered designs.
The first row in the table assumes a random
sample, in which case the design effect is 1.0.
In this case, the nominal sample size is equal
to the effective sample size and there are
1,000 students providing unique information
for each operational form. Under this design,
the standard errors obtained from commercially available software would be appropriate to
represent the sampling variance of the item
parameters.
The second row shows the consequences on
the effective sample size when forms are spiraled by students within a class. In this scenario, only a subset of students within the
same instructional unit are administered the
same test form. Hence, the covariance
between students within the same group cannot be ignored, resulting in a design effect of
3.6.
Note that in this scenario, the effective sample
size is only 282 even though the nominal samPage 7.3-84
ple size is actually 1,000 students per form. In

the final row of the table, we show the consequences on the effective sample size when
forms are spiraled by district. In this row, the
design effect increases to 11.8 and the resulting effective sample size is 85. In other words,
even though 1,000 students are administered
each form, it is as if only 85 of those students
are providing unique information.
The design effect at the district level is substantially larger than that at other levels
because the design effect is a function of both
the intra-class correlation and also the number
of individuals within a cluster. Holding the
intra-class correlation constant, larger clusters
have larger design effects than small clusters.
Commercially available software computes
standard errors, always assuming there are
1,000 unique responses per form even when
the design effect is greater than 1.0.
Recommendation for Spiraling Paper

Forms
AIRs recommendation is to spiral the EFT
forms by students within a classroom, thus
minimizing the effects of a clustered design by
reducing the number of students within a
given cluster. In addition, AIR proposes to calibrate these items using our iAM software
which computes standard errors consistent
with the sampling design. Hence, our spiraling
proposal minimizes the design effect, but does
not remove it. For this reason, it is still necessary to compute design-consistent standard
errors and AIRs IRT software is the only psychometric software available for this purpose.
7.3.6 Develop Stand-Alone ELA/L

Field-Test Sets and Forms
The ITN indicates a possibility for an independent field test for extended-response items
Exhibit 7.3.5-3: The Effect of Spiraling Design on the Standard Error (SE) of Item Calibrations
Sample Sizes
# Forms
4
# Students
4000
# Classrooms 200
# Schools
100
# Districts
50
Design 1
Design 2
Design 3
Design 4
Design 5
Spiral Design
Forms are administered to a random state-wide sample
Forms are spiraled by students within classrooms
Forms are spiraled by classrooms within schools
Forms are spiraled by schools within districts
Forms are spiraled by districts
in ELA/L. One option AIR is prepared to offer

is for the state to use the extended-response
tasks that were developed for the Utah SAGE
test. Development for that state provided a sufficient number of tasks per grade for which
Florida could use one task per year in each
grade over the course of the contract.
A second option is for the state to develop a
new set of extended-response tasks for a standalone field test. If this option is selected, we
propose to develop 20 extended-response
prompts for each grade to be field tested. With
approximately 200,000 students per grade, this
would yield 10,000 student responses per task.
Sampling Method
If these extended-response tasks are administered on paper, we propose to use the same
spiraling method by students within a class as
described above. If administered in a computer-based environment, the sampling method
will use AIRs online field-test algorithm
described above.
7.3.7 Constructing Test Forms

AIR is proposing a set of options from which
the Department can identify the solution that
works best for students and educators in
Florida. For online assessments, the
Department can elect to stay with fixed-form
assessments, consistent with the paper-based
assessment model, or move to an adaptive
assessment system. Both options draw on large
Design Inflation
Effect
in SE
1.0
1.0
3.6
1.9
4.3
2.1
5.8
2.4
11.8
3.4
Effective
Sample Size
1,000
282
233
172
85
pools of calibrated items designed to support

computer adaptive testing. There are numerous advantages to adaptive testing, including
not only increased measurement precision for
low- and high-achieving students, but also the
capacity to measure a much wider range of the
academic content domain at aggregated levels
of analysis. Although any one student may
only see one or two items per benchmark, just
as in a fixed-form test, that same benchmark
will likely be assessed using a range of items
across the students in a classroom, providing
sufficient information to yield accurate and
stable estimates of benchmark level performing at aggregate units of analysis, and allowing
the Department to significantly enhance the
score reports provided to educators in a psychometrically sound manner.
Although the banks are calibrated to support
computer adaptive testing, the Department is
of course free to adopt any reporting scale it
chooses, or even multiple reporting scales
should the Department elect to maintain separate within-grade and across-grade (developmental) scales. The final reporting scales will
be established immediately following standard
setting activities and adoption of the new
Florida statewide assessment (FSA) standards
by the Department. Should Florida choose to
stay with a fixed-form assessment design, we
anticipate that the Department will wish to
post-calibrate the test forms and establish a
distinct FSA reference scale to which all future
test administrations will be linked.
Page 7.3-85
Regardless of whether the Department decides

to move forward with an adaptive assessment
system or maintain a fixed-form design, AIRs
field test design, described in detail in Section
7.3.5, ensures that the FSA will be an assessment system uniquely designed to meet the
needs of Floridas students and educators.
Floridas large student population (as well as
efficiencies gained by AIRs field-test engine)
ensures that large numbers of Departmentdeveloped items can be calibrated as part of
the first operational test administration to substantially augment the existing adaptive pool or
move to an entirely Florida-exclusive item pool
to support a fixed-form assessment design.
Even if the Department adopts an existing
within-grade scale for purposes of adaptive testing, Florida can still choose to implement its
own vertical linking design to develop a uniquely Floridian developmental scale.
7.3.7.1 Test Construction

Specifications
Test construction proceeds somewhat differently in the context of adaptive versus fixed-form
assessments. In a fixed-form test, the ideal test
information function is decided once. Each
year, a team of content experts and psychometricians makes a sequence of decisions about
items to include. The test construction team
may allow some deviation from the ideal information function to get a better match to the
blueprint. The team may allow reuse of some
items from a prior administration to achieve
the desired balance. The number and types of
compromises made depend, often, on the items
available in the operational item bank.
Participants draw on their experience and professional judgment to make these decisions but
rarely articulate the guiding principles.
In an adaptive, criterion-referenced test, these
decisions take on a different character. The
system does not construct a test but rather
implements an algorithm for constructing
Page 7.3-86
many tests. Calculations and algorithmic logic

take the place of professional judgment. AIRs
adaptive algorithm is highly configurable. This
allows variable weighting of different blueprint
constraints and provides several mechanisms
for controlling the balance between blueprint
match and the precision of measurement. In
addition, the adaptive algorithm provides the
ability to limit item exposure. In a fixed-form
test, all items are overexposed each year.
Blueprint Construction
Sample blueprints are provided in Section
7.3.1. Distribution of item ranges is designed
to accomplish several important goals. First, as
a criterion-referenced test, the FSA must
reflect a representative sample of the knowledge and skills that students are expected to
achieve by the end of each school year or
course. In addition, to support student level
reporting of domain scores, each domain must
be assessed with sufficient items to support
reliable subscale performance level reporting.
Whether Florida chooses adaptive or fixedform tests, the blueprints are designed to
full breadth of the academic content standards, but constrained to ensure a sufficient
number of items in each domain to achieve
reliable subscale reporting. Moreover, item
ranges for depth of knowledge (DoK) levels and
item types ensure that students, regardless of
overall ability, are administered test items representing the full range of cognitive complexity
and are provided the opportunity to demonstrate their knowledge and skills through the
range of item types.
Test Construction
We understand that Florida currently plans to
implement fixed-form tests. Here, we briefly
describe the construction of adaptive tests and
provide a more detailed description of our
approach to constructing fixed-form tests.
Configuration of Adaptive Tests

Constructing forms for adaptive tests consists
of two steps: establishing the item pool and
then optimizing the item selection parameters
for the selected pool. Our adaptive engine is
robust to imbalances in the item pool as long
as there are a sufficient number of items in the
various blueprint categories. Therefore, the
selection of items for inclusion in adaptive
pools is less critical. When items are clustered
into groups, as with reading tests, we must also
pay attention to the composition of the item
sets and the degrees of freedom available to
the algorithm to select from among them. To
configure and evaluate the performance of the
adaptive algorithm, AIR psychometricians use
our simulation tool.
Item Pool. As long as there are sufficient items
to meet each of the blueprint specifications,
the adaptive algorithm is robust with respect
to item pool composition. It is therefore not
necessary that the distribution of content standards within the item pool be proportional to
the test blueprint. Nevertheless, as we describe
in Section 7.3.3 Develop Items, AIR test development and psychometric staff inventory the
item pools prior to each test administration to
identify any gaps in the pool that should be
especially targeted for item development.
Item development in a fixed-form environment
typically aims to concentrate test information
near the critical proficient cut score. When all
students are administered the same form, the
goal is to minimize measurement error near
the location of important cut scores. In adaptive environments, of course, the goal is to
minimize error near each students ability
level. This places a greater premium on developing items that more effectively measure students near the ends of the ability distribution,
increasing the proportion of very easy and
very difficult items. As part of our item pool
inventory, we also evaluate the distribution of
item difficulties within content domains to

guide our item development strategy.
With respect to item pool adequacy, we note
that the evaluation and the selection of items
into the operational pool proceed somewhat
differently between an adaptive and a fixedform assessment. In a fixed-form environment,
the criteria for promoting items to the operational bank may be somewhat relaxed, allowing the inclusion of items that may not have
optimal statistical characteristics but that are
intended only to be available for placement in
a test form when no other suitable items are
available. Thus, item statistics are generally
evaluated twice, once during data review for
inclusion in the operational pool and again at
form construction for placement in operational
test forms. In an adaptive assessment, all items
in the pool are available for test administration. For this reason, the criteria for including
items in the operational bank should be more
stringently applied than might be the case in a
fixed-form environment. All items in the adaptive item pool must meet minimum requirements for item discrimination, be within the
acceptable range of item difficulties, and
demonstrate appropriate fit statistics. The
adaptive algorithm does not recognize items as
being for use only in emergencies.
Simulation Tool. The simulator allows psychometricians to adjust parameters within the
adaptive algorithm to optimize it for use with a
particular item pool. The simulator outputs a
host of indicators including, but not limited to,
estimates of bias at the overall and reporting category levels;
standard errors and distributions of standard errors;
match to blueprint;
number of unique benchmarks administered to each student;
item exposure;
Page 7.3-87
number of item groups administered; and

item reuse across opportunities.
Psychometricians adjust configuration parameters to achieve the optimal mix of outcomes.
For new assessment systems, they may recommend changes to the blueprint, extending or
restricting the range of items on each content
standard or with each affinity group.
Sometimes small changes can improve outcomes, and we encourage our psychometricians to inform our clients when this is the
case. Of course, any substantive blueprint
change remains the Departments decision.
Psychometricians interact with the simulator by
modifying the configuration parameters in a
web-based environment. When the configuration
parameters are finalized, this same configuration
is uploaded to the Item Tracking System (ITS)
to generate a new version of the configuration
file. Simulations are run on the new file as a
final check that nothing has changed and that all
procedures were followed. The Department will
receive a final report on all simulations containing the detailed statistical output.
7.3.7.2 Test Construction System

The process for constructing operational forms
begins after field testing and committee
review. Once the item pool is finalized and
locked down, our content specialists begin the
process of constructing test forms.
Test forms are constructed in pieces. First, we
construct the intact operational forms, then
add to them the field-test items, or (in the
case of our automated field-test algorithm)
specify field-test item locations. Please see
Section 7.3.5 for a discussion of the embedded
field-test items. Operational passages and
items qualified for operational forms are those
that have met all of the criteria established by
the Department in terms of content, fairness
review, and data characteristics.
Page 7.3-88
Each form is built to exactly match the

detailed test blueprint, and match the target
distribution of item difficulty and test information. Together, these constitute the definition
of the instrument. The blueprint describes the
content to be covered, the depth of knowledge
with which it will be covered, the type of items
that will measure the constructs, and every
other content-relevant aspect of the test. The
statistical targets ensure that students will
receive scores of similar precision, regardless
of which form of the test they receive.
AIRs FormBuilder software helps our test developers as they construct operational forms.
FormBuilder interfaces with ITS to extract test
information and interactively creates test characteristics curves (TCCs), test information curves
(TICs), and Standard Error of Measurement
Curves (SEMCs) as test developers build a test
map. This helps our content specialists ensure
that the test forms are statistically parallel, in
addition to ensuring content parallelism.
All test forms for each content area will be constructed so that they are strongly parallel. This
means that the test scales and performance levels will be equivalent from form to form, and
that the raw score to scale score comparisons
for each form will be as similar as possible,
given the characteristics of the item bank.
Using FormBuilder, our content specialists
select test items that match the blueprint and
are of appropriate difficulty. Beginning with
content considerations and supplementing
those considerations with statistical considerations, AIR will create alternate, parallel test
forms by comparing TCCs for the form that is
being created with TCCs from previous forms.
To the degree that the TCC for the total test is
the same as for previous tests, the raw score
required for meeting any performance standard will remain as close to the same as it was
on previous forms.
Not only will AIR construct alternate, parallel

forms at the total test level, we will also attempt
to create alternate forms that have similar subtest psychometric structures, as well. This goal
may be more difficult to attain than equivalence
at the total test level because of the smaller
number of items from which to select.
As test developers build forms, FormBuilder
generates TCCs, TICs, and CSEMs for the reference (previously used) form and the target
(new) form(s) on the screen. The TCCs and
SEM curves are plotted using a different color
trace line for each prototype form. Sample
TCC and SEM curves are provided in Exhibit
7.3.7.2-1.
At this point, the test developer can see the
exact difficulty relationship between the target
and reference forms. If the difficulty and error
structure for the target forms is virtually identical to the reference form, as in the sample TCC
and SEM curves, then the item selection process
concludes with multiple, parallel test forms.
Once the goal of parallel forms is achieved, the
information is entered into ITS, which tracks
item usage and generates bookmaps (test
maps) for use in scoring, forms development,

and other processes.
FormBuilder Output Files

Once operational forms have been defined in
FormBuilder, all bookmaps (test maps), key
files, and conversion tables are produced
directly from FormBuilder to eliminate the
possibility of human error in the construction
of these important files. Bookmaps, key files,
conversion tables, and other critical documents are generated directly from information
maintained in ITS. The information in ITS has
been through rigorous reviews by multiple
skilled reviewers, fortifying it against errors.
Automated production of these critical files
(such as key files) virtually eliminates opportunities for errors.
Bookmaps can include any item attribute
stored in ITS, so that in addition to form-level
attributes such as test administration and item
position, item attributes such as content standard, benchmark, indicator, complexity, item
release status, point value, weight, keyed
response, and more can be included in the
bookmap. The bookmap feature in
FormBuilder can be customized to serve a
Exhibit 7.3.7.2-1: Test Characteristic Curves and Standard Errors of Measurement
Standard Errors of Measurement
1
0.8
0.6
0.4
0.2
0
SEM
Proportion
Test Characteristic Curves

2
1.6
1.2
0.8
0.4
0
-5 -4
-3
-2 -1
-5 -4 -3 -2 -1
Theta
Theta
FORM A Proportional Score
FORM A SEM
FORM B Proportional Score
FORM B SEM
3 4
Page 7.3-89
variety of clients within an assessment project.

Bookmaps used to review and approve operational-test forms require more in-depth item
attribute information. When creating
bookmaps to lay out printed test booklets or to
score test forms, it may be desirable to include
fewer item attributes.
As a further layer of quality assurance for
printed test booklets, both during the blueline
production phase prior to printing and again
following the final printing of all test forms,
two AIR technical team staff members independently take all test forms. Responses to the
test forms are compared to the answer keys for
each form to confirm the accuracy of scoring
keys. In addition, the printed forms are compared against ITS and FormBuilder for content
and item ordering to ensure that no changes to
the form are introduced prior to printing.
7.3.7.3 Construct the English

Language Arts/Literacy, and
Algebra 1, Algebra 2, and
Geometry Retake Forms
As we have described throughout this proposal, whether the Department elects to administer the FSA as an adaptive or fixed-form
assessment, our experience across multiple
states, including those that initially maintained
post-equating, strongly indicates that a move
to online test administration often leads to
pressure from the field for immediate scoring
and reporting of test results. Students and educators alike expect that when test administration is completed, students can expect to
receive feedback on their performance and
educators, from classroom teachers through to
superintendents, expect to be able to gain
immediate access to assessment results that
not only summarize current year performance
but also track student, teacher, school, and
district performance over time.
Page 7.3-90
We therefore recommend that the Department

employ pre-equated item parameter estimates
to score student responses for all online
assessments. In the ITN, while the Department
envisions maintaining a post-equating
approach to scoring census test administrations each spring, the grade 10 retake forms
are designed to be scored using pre-equated
parameter estimates. Pre-equated scoring for
the retake assessments makes sense for the
same reasons described for census test administrations, including much faster return of test
results to students and their families and educators, but also because the retake assessments are administered to highly selected
samples of low-achieving students. Post-equated parameter estimates calibrated on such
highly selected samples of students may not be
accurate, rendering moot any advantages of
post-equating that might typically be gained
when applied to representative samples of students statewide.
An impediment to immediate scoring and
reporting of assessment results has been the
need to hand-score constructed-response item
types. We note that the entire SAGE assessment
is machine scored, with some human backreading of the ELA performance tasks. For Florida,
we recommend human scoring for performance
tasks used in high-stakes individual decisions.
AIR is the industry leader in the development
and implementation of machine-scored item
types that
include real performance-based simulations
and constructed-response items;
can be readily embedded in operational
test forms for field-testing;
are inexpensive to score; and
support immediate reporting of test results.
The variety of machine-scored item types available to the Department is described in Section
7.4.9.9. Most AIR item types are rubric scored,

with rubric validation occurring following fieldtest administration and prior to calibration and
equating activities. Thus, machine-scored items
provide the Department with a rich and relatively inexpensive source of genuine performance items for assessing student achievement
of Floridas academic standards. For essay
items that employ a statistical scoring engine,
and for which we recommend assigning final
scores based on human readers, we can provide preliminary test results immediately following test administration based on machine
scoring, with final scores provided later following hand-scoring or hand-scoring verification of
machine-assigned scores.
Embedded field-testing may not be practical
for some items or item types. There are three
possible approaches for handling these item
types on retake assessments.
As the Department suggests in the ITN, one
approach is simply to exclude such item types
from the retake blueprints. We feel there are
more constructive approaches to this problem.
Even if item parameter estimates are based on
standalone or independent field-testing, such
estimates, especially in the context of rigorous
linking and sampling designs, are quite reliable. Thus, eliminating a class of items simply
because the parameter estimates are not
obtained under ideal circumstances seems an
extreme solution.
A second approach is to populate such item
types in retake tests from previously administered operational test forms when possible.
Especially where there exists a large pool of
operationally administered test items, items
could be migrated from census test administrations to a retake pool and finally targeted for
release based on item exposure rates.
A third approach is to use linked parameter
estimates derived from the standalone field
test. As noted above, when standalone field

tests implement robust linking designs and
employ sound sampling methodologies, linked
parameter estimates are reliable and suitable
for operational scoring. It is often true that
students perform somewhat better in the context of operational test administrations than in
standalone field tests, especially with respect
to more cognitively complex items, but any
differences in motivational factors (or any
other factors) underlying test administration
context are accounted for when items are
linked back to the reference scale.
Decisions about which approach to use must
balance multiple assessment goals. Previously
administered operational test items may have
the advantage of operationally based parameter estimates, but the increased exposure of
such items may pose security risks.
Eliminating the item types from the assessments altogether suggests a change to the
underlying construct and a threat to strong
equating of test scores. Parameter estimates
derived from standalone field tests can provide
valid test scores that are equated with other
census and retake assessments.
Form construction for pre-equated retake tests
follows the same procedures used for the
spring assessments described in the previous
section. As with census assessments, we propose to electronically generate a series of candidate test forms with information targeted at
the critical cut score. Psychometric characteristics of candidate test forms, including test
characteristic curves, and standard error and
test information graphs, will be evaluated by
psychometricians via the FormBuilder component of ITS. Psychometricians will work with
test development staff to make any desired
modifications to test forms prior to submitting
forms to the Department for review and
approval.
Page 7.3-91
7.3.7.4 Prepare Camera-Ready or

Screen-Ready Test Forms
AIR will prepare camera-ready documents or
screen-ready files after the test items have
been selected, composed in forms, and
reviewed per the Departments specifications.
Paper tests go through a traditional production
process. AIRs test booklet production process
starts with the creation of test maps (also
referred to as bookmaps). The test map is built
in ITS and initiates the production of printed
test forms. The process includes the following
five steps:
1. The 11s are generated based on the test
map (bookmap).
2. Blackline 1 is drafted and reviewed internally.
3. Blackline 1 is delivered to the Department
for review and approval.
4. Should any changes be requested in the
blackline 1 review, AIR produces, reviews,
and delivers blackline 2 to the Department.
5. The documents are brought to blueline
(camera-ready copy).
Step 1 is entirely automated within ITS. ITS
houses destination templates that define the
format of the 11s and automatically generates
these documents based on the test map. (AIR
defines 11s as test items printed one per page
according to the test map sequence.) At this
stage, items are proofread by editorial staff and
test development staff at AIR and the
Department. Additionally, they are reviewed to
verify that all edits from previous rounds of
review have been correctly implemented. Any
changes required at this stage are entered
directly into ITS to ensure consistency across
all item uses.
Blackline 1 is a semi-automated process. With
the appropriate destination template defined
Page 7.3-92
and 11 approval, ITS generates a Quark-readable document in the specified format.

Through this integration, items are automatically styled with fonts, graphics, spacing, and
other formatting specifications outlined in the
Departments style guide. Our production staff
may adjust page layout, including instructions,
borders, and other elements, to meet the
Departments guidelines. At this stage, reviewers check the document layout and formatting.
Should any egregious errors be found in the
content of an item, changes must be entered
into ITS and the item must be re-exported to
ensure consistent item use across all test
forms. Changes to blackline 1 require a second
blackline proof. Changes to subsequent blackline proofs require sign-off by senior management.
The final quality assurance step prior to printing is the blueline, or camera-ready copy,
review stage. During this step, AIR and the
Departments staff review proofs from the print
vendor, verifying that the file to be printed
matches the previously approved blackline
proof. At AIR, in addition to reviews by test
development and forms production staff, two
members of the technical teamwho have not
seen the items previouslyindependently take
the tests. This process forces a close look at
the items and gives a final opportunity to verify the keys.
During the production and review process, test
book blacklines are accompanied by answer
document blacklines, which are produced by
our partner, DRC. Answer documents reflect
the demographic fields required by the
Department, as well as fields for pre-code
labels and the scannable marks required for
accurate data collection. The item sequence is
based on test maps and corresponds directly
with test books.
All blacklines in AIRs production queue are
controlled by an electronic version-control
server system that ensures that only the cur-
rent version is immediately available to our

production staff, preventing version-control
errors. Like AIRs ITS, which controls and
tracks all changes to items, this production
system maintains historical records (including
all older versions), which senior production
staff can access if necessary.
Each blackline after blackline 1 and the blueline (camera-ready copy) is automatically
compared with the immediately preceding version using a PDF comparison tool that highlights all changes. This step has proved useful
for identifying unintended changes made during the revision process. Such changes are difficult to detect because they can appear anywhere in a document and may be subtle. The
PDF comparison tool highlights these changes
so differences between versions can be
mapped to an intended revision. All materials
delivered will go through this process, ensuring
that the Department will receive error-free
materials for review and that any changes
requested by the Department are implemented
promptly and accurately.
At each of the review stages, proofs will be
accompanied by proof tickets that identify the
document being reviewed, its review stage, the
scheduled and actual delivery dates, and the
return date. Sign-off by the Department will be
required at each stage before proceeding with
subsequent steps.
Final source files will be made available to the
Department in electronic form no later than
30 days after the close of the test window.
Scannable Documents
Image-scannable test booklets or answer documents will be developed in house by our partner, DRC, using DRCs publishing system.
DRCs Document/Graphic Design Group routinely provides exemplary services to state
testing clients, including the design, layout,
and production of scannable answer documents and other testing materials as well as
the receipt, production, and transfer of electronic print files. They develop effective new
materials and convert existing documents to
meet changing assessment requirements. DRC
uses the latest design and publishing software
and works with multiple file formats and plat
forms so that data file conversion is hassle
free. DRC staff members are committed to
developing accurate and engaging test materials that meet all quality requirements.
AIR and DRC propose that scannable test
booklets be provided for grades 3 and 4, and
separate test book lets and answer documents
be provided for grade 5. After AIR has finalized
the test booklets, DRC will develop and produce customized, scannable answer documents in collaboration with AIR and the
Department. Answer documents will be color
coded to match test booklet and manual colors, as appropriate. DRCs custom-designed
answer documents will feature a light background grid and light background lines on the
response area for constructed-response items,
as well as directions to help students easily
navigate between the test booklet and the
answer document.
The Departments scannable answer documents will be designed and printed to fully
support test administration, scanning, and
scoring processes. All key departments
involved in the scanning and scoring process
will have the opportunity to review and
approve laser proofs and printer proofs.
Requirements and printed documents will be
subjected to strict quality assurance inspections to ensure accuracy. The format and precision of the printed information, including
precode information, will be closely examined
by software quality assurance analysts to make
certain the information is correct. DRC will
provide AIR and the Department with produc-
Page 7.3-93
tion samples of answer documents for an independent quality review.
Production of Online Forms

Preparing online tests for delivery follows a
somewhat different path than preparing a
paper test. These differences allow a workflow
that moves an enormous amount of work off
the critical path, allowing analogs of blackline
reviews to take place even before the forms are
constructed. For example, cover art, layouts,
and other styling are determined in advance
and can be viewed in the Item Tracking
System (ITS) and the test delivery system well
in advance of form construction.
Online items can be reliably reviewed as they
will appear to the student well before the
forms are constructed. AIRs Item Tracking
System incorporates the same rendering
engines that will ultimately render items to
students. Our web preview review actually
locks the item down, ensuring that students
will see the item exactly as the reviewer did.
Another critical difference is that students
interact with items in more diverse ways in an
online test. For examples, items may include
tools, accommodations, session rules, and
other features that are established well in
advance and configured as part of the test
delivery system, rather than as part of a particular form.
As these examples indicate, the preparation of
the screen-ready forms begins well before any
forms are constructed. Items and item groups
can be reviewed as they will appear to the
examinees on the test. AIRs ITS integrates the
same rendering engines included in our test
delivery system, ensuring that items (or item
groups) appear to reviewers as they will appear
to the students.
Operationally, the ability to review items as
they appear before they are assembled onto
Page 7.3-94
forms provides a huge advantage. First, the

Departments reviewers will have more time to
review the items and can spread the review
work out over a longer period. In a typical
paper assessment, blackline reviews come near
the end of the process and compress the work
schedules of reviewers. Using AIR systems, the
online equivalent of blackline reviews can be
conducted on a rolling basis, with each page
being complete and laid out as soon as the
items to appear on that page are finished.
Once the final bookmaps are approved and
accepted, they are published to our test site,
where they undergo platform review, which
looks at each form on many of the supported
computer platforms. Platform reviews ensure
appropriate display of each item on each
platform.
We publish completed forms to the user
acceptance test site. This instance of the test
delivery system is configured to the
Departments specifications.
Unlike the process for paper tests, which
require more lead time to adjust, during the
user acceptance period AIR staff responds to
the Department comments nightly. In most
cases, the Departments comments are
addressed within 24 hours.
The online form development process allows
much of the review work to be done on a flowing basis as items are approved for inclusion
on the test. The integration of the item rendering engine with our ITS allows reviewers to
lock down the specific rendering of an item
long before that form is developed. Contrast
this with the paper-and-pencil process, in
which the layout and format can only be
reviewed during the blackline process, which
is necessarily subject to tight deadlines. By
moving some elements of review off the critical
path, we reduce the likelihood of errors as well
as removing the stress that comes from doing
critical work on tight deadlines.
7.3.7.5 Interpretive Products

Our experience assisting state Departments of
Education in transitioning to new standards,
new assessments, and online tests has taught
us that clear, detailed, interpretive products
are critical to building stakeholder success for
any new assessment system. Understanding
how new item types assess the new standards
and how they are scored is essential if teachers are to provide effective classroom instruction. Similarly, both parents and educators
need to understand how to interpret their students test results so they can identify
strengths and weaknesses early on. AIR will
provide a full suite of interpretive products to
ensure that Florida parents, students, and educators understand the new standards, assessments, and test results including the following:
An Interpretive Guide for Floridas Paper
Score Reports;
A User Guide to help Florida Educators,
Schools, and Districts understand Floridas
Online Reports;
Sample test materials including paper practice tests, practice answer sheets, and
answer keys, including separate Braille,
Large Print, and translated practice test
materials for students who require accommodations or non-English materials;
Computer-based practice tests that use the
same test delivery engine as the operational
tests;
Scoring Guides for the English Language
Arts/Literacy (ELA/L) and Mathematics
Assessments as well as the End-of-Course
(EOC) assessments that include all items
types on the assessments and explain the
alignment to Floridas standards as well as
the Common Core State Standards;
Test Specifications that provide detailed
requirements for writing passages, prompts,
and items.
AIR will produce ADA-compliant documents as

specified in the ITN. In the following sections
we describe a detailed plan for providing each
interpretive product.
Understanding Floridas Assessment

Reports
As Florida adopts new standards and continues
to move its assessment program online, it is
critical for parents, educators, schools, and
districts to understand how to interpret score
information from the new assessments. To
ensure that stakeholders learn how to use new
score reports and score data, AIR will produce
an Interpretive Guide for parents and students
explaining the paper reports that they receive
for the EOC and ELA/L and Mathematics
Assessments. Should the Department decide to
continue providing schools and districts with
PDFs posted in the Online Management
System, AIR will also include information
about the PDF school and district reports and
explanations of each element in this publication. However, should the Department choose
to use AIRs online reporting system to deliver
score data to schools, districts, and educators,
we propose that the Interpretive Guide publication focus solely on the paper reports delivered to parents and students. The User Guide
discussed below will provide information to
educators, schools, and districts on how to
access and interpret reports and data in the
Online Reporting System. (For sample
Interpretive Guides and User Guides, please
see Appendix F.)
In the next two sections we describe the proposed content for both the Interpretive Guide
and the User Guide in more detail.
Interpretive Guide for Floridas Paper
Assessment Reports
We propose to include the following content in
this publication and will work with the
Page 7.3-95
Department to obtain approval on detailed outlines before developing the guide:

Full color screenshots of the EOC, ELA/L,
and Mathematics score reports to students
and families along with explanatory text
Plain language explanations about the content areas and strands tested
An explanation of how to interpret scores
and Standard Error of Measurement
A glossary of terms
Detailed explanations of how released performance tasks are scored, including the
released item, alignment to the standards,
and annotated student responses explaining
why the response was awarded the points
received
Links to more in-depth training materials
described in Section 7.4.7.1
The guide will include an overview of the ELA/L
and Mathematics tests and the EOC Tests will
provide a brief description of the assessment,
the purpose it serves in improving Floridas education system, a Frequently Asked Questions
section that will provide basic information
about why students are taking the assessment,
what happens if they do not score at proficient
level, and who develops the assessment.
A section presenting the Performance Level
Descriptors will provide an explanation of the
expectations of students for each subject and
performance level. This section will allow parents and educators to know what is required for
a student to perform at a particular proficiency
level and will also provide plain-language explanations that include specific examples of what
typical students can do at each level.
The guide will include background information
about the scoring process for both the ELA/L
and Mathematics and EOC assessments. This
section will define the terms used in the ELA/L
Page 7.3-96
and Mathematics and EOC assessment student

reports, such as scale score and subscale
score. The guide may include a description of
the relationship between raw cut-scores, which
represent a students overall score in a subject
area, and sub-scores, which indicate a students strengths and weaknesses in that subjects content standards. We propose that this
section also include a list and definitions of all
possible non-scorable codes that may appear
on a student report.
We propose to include screenshots of each
page of the printed parent report and provide
detailed explanations of each element. For
example, using non-technical language, we can
explain the information presented in graphic
displays, describe how to interpret a students
strengths and weaknesses using a sample
report, and provide a glossary of jargon-free
definitions of any technical terms that may be
included in the reports (e.g.,, constructedresponse question and score levels in the
family guide). Should the Department choose
to include Next Steps text in the parent
reports, this section may also contain an
explanation of how parents can use the customized recommendations to help their child
improve his or her understanding of the standards. This section might also contain a
detailed explanation of how to use the longitudinal data charts to compare student performance from year to year.
Proposed Content for the Online Reporting
System User Guide
As Florida adopts new standards and continues
to move its assessment program online, educators need to become adept at locating score
information quickly and intuitively from multiple assessments. Should the Department
decide to use AIRs online reporting system as
described in Section 7.5.5, it will be critical for
educators to learn how to access and interpret
data from AIRs interactive online reporting
system. In addition to absorbing volumes of

online score data, educators must also be able
to understand and take appropriate actions
based on these data. AIRs User Guide for the
Online Reporting System is designed to provide a conceptual framework that helps users
quickly locate the right level of data, interpret
its impact, and identify the concrete actions
they can take to help students improve.
Over the past five years, we have produced
concise, effective User Guides for the Online
Reporting System explaining to novice and
experienced users alike how to access reports,
create custom rosters, investigate trends in the
data, and generate PDF reports in the Online
Reporting System. The Online Reporting
System User Guides are available in electronic
format as well as PDFs as required. We will
work closely with the Department staff to produce the User Guide that includes step-by-step
instructions on how to use the Online
Reporting System and the following proposed
content:
Color screenshots of different reports and
data views describing key functions
Step-by-step instructions for users on how
to access specific reports and data
Overview of the Online Reporting System
(searching for students in the Online
Reporting System, New Features)
Instructions on how to navigate within the
Online Reporting System
Information on User Roles (who has access
to what data in the Online Reporting
System)
Explanations of Score Report Layouts and
Features (Demographic Filters,
Comparisons, Downloadable Data files,
Legends)
List and explanations of Reporting

Categories
List and explanations of Trend Reports
Explanation of how to create and manage
Custom Roster Reports
Explanation of how to print reports and
export data to Excel
Explanation of available Test Management
Reports and how to use them
The User Guide for the Online Reporting
System begins with an overview of the system,
screen shots, and descriptions of the user
experience while moving through the system.
Next, the guide walks users through scenarios
demonstrating appropriate uses of the data. In
introducing each set of reporting capabilities,
the guide will describe why those reports are
available, how they can help at the school or
in the classroom, and what the limitations of
the data are. The guide also shows users how
to create class and other rosters of students
and how to use these rosters to create aggregate statistics or track groups of students over
time. Step-by-step instructions are provided to
users to view district, institution, teacher, and
individual student reports in table and graph
format; access and use longitudinal reports for
their diagnostic need; filter score reports by
student subgroup (i.e., gender, ethnicity, ELL);
and manage online rosters (groups.)
The User Guide also includes detailed instructions on how authorized users can use Test
Management Reports to monitor student testing status for their school or districts and to
identify students who have and have not tested. Users also learn how to use these reports
to help schedule test activities for their school.
List and explanations of Subject Detail

Reports
Page 7.3-97
Production of Interpretive Guide and Online

Reporting System User Guide
AIR will adhere to Floridas production and
review guidelines as outlined in Section 7.3.10
when developing the Interpretive Guide and
the Online Reporting System User Guide.
Upon approval of final drafts, AIR will post the
Interpretive Guide and User Guide on the
Portal for districts and schools to download.
In producing the User Guide for the Online
Reporting System, AIR will rely on its dedicated team of technical writers, editors, and
graphic designers who are continually involved
in developing, editing, and enhancing documentation for AIRs Online Reporting System.
The goal of the Online Reporting System User
Guide is to motivate teachers and administrators to make the best possible use of the system and the data it provides. AIR will work
closely with the Department to update and
revise the user guide for each school year and
ensure all systems documentation reflects
updates to the system.
AIRs process for developing user guides and
systems documentations is divided into two
phases:
A drafting phase, during which we obtain
agreement on the content and design of the
document.
A production phase, during which we produce the document to the specifications
developed in the first phase, accommodate
any final adjustments, and ensure that the
camera-ready proofs are error free.
The drafting process begins with discussions
with the Department, during which we discuss
issues of both design and content. Once drafts
are approved, we proceed to production. The
first blackline version of the document, which
results from the first iteration through our production process, is designed to reflect the final
Page 7.3-98
layout and formatting. Typically, only format

and layout changes are made at blackline. A
second blackline is produced after incorporating these changes to confirm that they were
made correctly.
Our production processes include multiple editorial reviews, and each document must
receive senior staff sign-off. Exhibit 7.3.7.5-1
provides an overview of our editorial review
process. Each documents content is fully
drafted before the document enters the production system. Camera-ready drafts are called
blackline proofs, and we typically produce up
to three of these, with final edits made on
blackline 1, any missed changes corrected on
blackline 2, and a clean version delivered at
blackline 3.The final blackline proof must have
no changes, and it serves as the master document against which the blueline proof is
evaluated.
Sample Test Materials

Until stakeholders can take a practice test, see
released test questions, and learn how new
item types are scored, many questions about
the new testing program will remain and confidence in the new program will likely remain
low. For this reason it is essential that educators, students, and parents, be able to engage
with the new assessments and standards using
sample test materials in paper and online formats early on. Our proposed strategy for providing paper-based and online practice tests
recognizes that online and paper tests are different and makes the best use of technology
for students taking the test online. Specifically,
we propose to develop paper-based sample test
materials using common items with the online
tests where those items work. For example, we
have found that in many cases, online items
can easily be offered on paper. In those cases
where an online machine-scored constructedresponse item is not well-suited to a paper format, we propose to replace the item with a
Exhibit 7.3.7.5-1: Overview of Our Editorial Review and Revision Process

Process for Publishing Documents (User Guides, Brochures, etc.)
Internal
Review 1
Editor
Review
Final
Draft
Client
Review
(Tech Writers)
(PD/PM)
(Editors)
(TW/PM)
(Black Line 1)
user guides,
manuals, etc.
client-focused
review
Editors with
knowledge
about systems
review docs
Technical
Accuracy
Checks
Work with tech

writer(s) on
suggested edits
Word doc to be
reviewed for
content only
Follow editing
guidelines
(scope)
Approval=no
Draft
Document
Client
Edits
Internal
Review 2
(TW/PM)
(PM)
Incorporate
appropriate
client edits,
suggestions
Approval=yes
inDesign/inCopy
collaboration
(TW/Designer)
Production
Layout
(Designer)
User Guides
may require
1-2 weeks
Return to TW
Publish
Client
Review
Internal
Sign-Off
Internal
Review 3
(Black Line 2)
(PD/PM)
(TW/APE)
Approval Phase
human-scored constructed-response item for

the paper-based practice test (see below for a
more detailed discussion). We propose to provide the following sample test materials:
Paper and online practice tests for each
grade and subject of the ELA/L and
Mathematics Assessments, the EOC
Assessments, and the Writing Assessment;
Practice Answer Sheets for each assessment;
Teacher scoring guides for each practice
test (see section on Scoring Guides);
Large Print and Braille versions of the
paper practice test materials along with
scoring guides;
Translated practice tests and answer sheets
in Spanish and Haitian Creole.
Verification
against
approved content
Our plan for providing each of these materials

is described in more detail in the following
sections.
Plan for Producing Paper Practice Tests
AIR proposes to deliver practice tests consisting of at least 1020 released items that mirror
the operational test form. Items will reflect a
range of standards and skills in order to represent a small-scale version of the actual test.
Computer-based practice tests will provide students with AIRs full suite of online response
mechanisms, thereby giving students a chance
to experiment with the tools and achieve a
level of comfort and proficiency before the
testing window opens. Paper-based practice
tests, including Large Print and Braille accommodated versions, will offer an equivalent preview of testing material so that all students
have an opportunity to become acclimated to
Page 7.3-99
the tests they will take. Both regular print and

ADA-compliant versions of the sample test
materials will be made available each fall to
ensure that students and educators have ample
time to review them.
AIR will provide separate sample test materials
for both paper-based and computer-based tests
as we understand that these have some inherent differences. While computer-based practice
tests will include samples of machine-scored
constructed-response items, these items are
not always translatable to a paper environment. AIR develops machine-scored items with
the intent to measure skills that are simply not
practical or possible to assess otherwise. For
example, our simulation tool allows students to
perform a variety of tasks to identify kinds of
rocks, or use a microscope to differentiate
cells by their properties. These authentic, realword applications are often too complex to
measure on a paper test, but AIRs machinescored item types overcome such limitations
and make them possible to assess in an online
environment. This raises issues for the development of equivalent paper-based practice
items. Since some items cannot be rendered
effectively on paper, AIR proposes a practice
test system that makes the best use of technology for students taking the test online, while
maintaining as much equivalency as possible
for those who require the paper-based practice
test. We will provide common items on the
paper practice test when those items work,
and we will replace complex machine-scored
constructed-response items with humanscored constructed-response items when necessary.
AIR will also ensure that items on the paper
practice tests can be made available in Braille.
Considerations for such items may include
avoiding photographs;
ensuring that graphics or drawings can be
transcribed;
Page 7.3-100
finding appropriate solutions for standard

charts, such as a periodic table of the elements;
ensuring that items do not rely on a visual
experience (e.g., a question that involves
the concept of depth perception); and
avoiding or adapting items that require spatial instructions.
In addition to paper practice tests, we will also
provide a separate answer key booklet for each
practice test (both paper and online). Answer
booklets will provide teachers with the information they need to understand and explain
the items in the sample test materials. Answer
key booklets will contain the full text of the
item from the sample test, along with the following:
The items alignment to the standards.
The correct answer for the item.
A rationale for each multiple choice
response. (We note that Floridas current
answer key booklets contain rationales for
the correct response only.)
A screenshot of the correct response for
machine-scored constructed-response
items.
A complete rubric for each constructed
response with descriptions at each score
point.
Exhibit 7.3.7.5-2 illustrates that AIR provides
rationales for each response. We offer an
explanation of the correct response as well
instructive material teachers can use to
explain why the distractor options are plausible but incorrect.
In Exhibit 7.3.7.5-3 we show a constructedresponse item. The rubric displays the range of
possible answers and an explanation of each
score point. We note that this item can be
used in both online and paper tests. In the
Exhibit 7.3.7.5-2: Sample Test Answer Key
online version, students type their response

into the text box. Test developers use the
propositional rubric tool to create a rubric that
matches the human-readable rubric below.
Scoring Guides
Not only do Floridas scoring guides provide
educators with information about the scoring
of constructed-response items, but they also
provide important information about the test
itself. Beginning with an explanation of the test
design, a summary of the standards and
benchmarks assessed, and a description of the
item types, scoring guides provide the context
for educators to understand the assessment as
a whole. AIR proposes to deliver scoring guides

that provide this critical information as well as
the details of scoring. Scoring guides will
address each released constructed-response
item included on the student reports.
For each item in the scoring guide, AIR will
display the prompt as it appeared to students
on the test. We will provide the complete scoring rubric and samples of student responses at
each score point, not just the top scores. In
addition, we will create detailed annotations of
each student response to clarify for educators
the distinctions scorers used to differentiate
each level of student performance.
Page 7.3-101
Exhibit 7.3.7.5-3: Sample Constructed Response Rubric
The scoring guides for extended-response

items and performance tasks can be particularly challenging as these items assess more complex skills. Through our work with statewide
assessment programs and the Smarter
Balanced Assessment Consortium, AIR staff
have developed experience in creating and
scoring these more complex items. We will collaborate closely with the Department to create
a format for scoring guides that provides the
level of detail Floridas educators need to
understand the scoring of such open-ended
tasks. These guides may provide general information about how released prompts are
scored, an explanation of the rubric, images
Page 7.3-102
of student responses, and annotations at each

score point.
Exhibit 7.3.7.5-4 is an extended-response item
included in an English Language Arts performance task. This item is the culmination of the
task in which students have interacted with,
and responded to, a variety of stimuli around
a central theme. Here, students are asked to
incorporate many elements of the task into
their final writing product. This example is
taken from the Smarter Balanced practice test,
and the scoring guide reflects the Smarter
Balanced scoring guide.
Exhibit 7.3.7.5-4: Sample Extended-Response Item
Page 7.3-103
This item is scored on a 10-point rubric4

points for purpose/organization, 4 points for
evidence/elaboration, and 2 points for
conventions. AIR will work closely with the
Department to ensure educators and stakeholders understand how these items are scored
and have the information they need to use in
the classroom and inform writing instruction.
Plan for Providing Computer-based Practice
Tests
AIR will provide a practice test prior to the
opening of operational or field tests. The ITN
requires such a publicly released test prior to
each live test. AIR practice tests
are delivered using the same system as the
operational test, so that the student experience is the same in every way;
include every item type that students will
encounter on an operational test;
can be accessed using an ordinary browser
(rather than our secure browser), though
some functionality requires the secure
browser; and
can be accessed anonymously for independent practice, or in proctored sessions,
which enables Test Administrators to practice their duties.
The practice tests may be separate subject
tests within each grade or combined subjects,
at the discretion of the Department. For pricing purposes, we assume a total of 20 items
per grade on the practice test. We note that
both SAGE and Smarter Balanced have publicly available practice tests, which we invite
readers to review at
https://utpt.tds.uat.airast.org/student and
https://sbacpt.tds.airast.org/student/.
Practice Test Site
Every item type that students will see on the
operational test will be available on the pracPage 7.3-104
tice site. For example, we will make available

samples of our graphic-response constructedresponse items, including a self-guided interactive tutorial. As new constructed-response
item types become available, tutorials and
practice materials will likewise be deployed
to the practice and training sites.
Only non-secure material will be available
on these sites, including items that the
Department has decided to release. In addition,
all materials used on the site will have appropriate permissions from their copyright holders.
The practice site not only mimics the functionality of the operational test site, it also
runs under the same engine, although it may
be configured differently. Of course, the practice test can be accessed using non-secure
browsers as well as secure ones. Configurable
features include the following:
Test Blueprintthe Department can choose
which content areas and items to include
in this site. Practice tests can have different blueprints than their operational counterparts.
Anonymous Studentsthe Department can
choose whether students who use this site
have to be preregistered in the system.
Anonymous Test Administratorsthe
Department can choose whether Test
Administrators who use this site have to be
preregistered in the system. For example,
parents who are helping their children
prepare for the test may want to play the
role of Test Administrator and create sessions for the tests their children are practicing.
Instant Scorethe Department can choose
what scores or other information to display
to the student at the end of the test.
Accommodations and Modificationsany
accommodation or modification can be
turned off or on by test.
The practice test site can be entered in either

a proctored (educator) or un-proctored (public) mode. In the un-proctored mode, users
simply enter the site and take a test. We make
the proctored mode available so that Test
Administrators can practice the authentication
routines with their students prior to testing if
they choose. In the proctored mode, the students and Test Administrator go through the
usual authentication procedures.
We propose to provide online practice tests for
the ELA/L and Mathematics assessments, the
EOC assessment, and the Writing Assessment
approximately 6-8 weeks before the specific
test window begins. All practice tests will
adhere to the Departments specifications.
Below we discuss the functionality and benefits of the online practice tests in more detail.
Easy-to-Use Online Practice Tests for Test
Administrators and Students
For parents and educators to use the online
practice tests, ease of use is critical. Unlike
Floridas current online practice tests, AIRs
online practice tests do not require any test
content to be downloaded on student
machines prior to their taking a practice test.
AIRs practice test is accessible from any
machine using a standard browser or AIRs
Secure Browser.
The online practice test will be deployed using
the same test delivery engine as the operational test. For this reason, the ancillary materials such as user guides and administration
manuals for the operational test will apply
equally well to the practice test. In addition,
we propose to produce a quick start guide
an online pamphlet of approximately one page
that describes the basics of starting and taking
a practice test. The online practice test will
also include help screens, an integrated users
guide, embedded tutorials, and other supports
to help users learn the system (see Section
7.3.7.5 for more information).
Test Item Specifications

AIR has worked closely with all of our clients
to create thorough item and test specifications.
These documents clearly outline the ways in
which item writers and test developers create
assessments. AIR proposes to use these specifications as a starting point to develop similar
specifications documents for Floridas assessments.
Our item specifications provide the sturdy
scaffolding upon which item writers can take
concepts and build them into clearly defined,
measurable tasks. We have created detailed
specifications documents in each grade and
in each content area that
address the content standard;
define the limits of how that standard
might be measured;
propose acceptable response mechanisms
for measuring the standard (based on the
full suite of item types that AIR offers);
propose ways in which each Depth of
Knowledge category can be achieved; and
provide sample items that address the standard. Sample items include an expected
range of item difficulty.
In addition to these item specifications,
we have also developed extensive passage
specifications.
For a more detailed discussion on the Test
Specifications we propose for Florida, please
see Section 7.3.1. AIR will work with the
Department to ensure that all test item specifications align with the Departments high
expectations for item quality and stand the
test of Content Advisory Committee review.
Page 7.3-105
7.3.8 Production of Printed Test

Materials
AIR has teamed with DRC to provide printing
of all materials for the paper-based tests (PBT),
including accommodated materials, and test
administration and ancillary materials. AIR
has collaborated with DRC in this capacity for
other assessment programs since 2006. AIR
and DRC have extensive experience in producing documents that are attractive and errorfree and take great pride in the quality of testing materials produced on behalf of our
department of education clients. Specific information on DRCs materials production and
printing processes and experience is provided
below.
For the Florida Standards Assessments (FSA),
AIR team member DRC will
print test booklets using print-ready files
provided by AIR;
design, develop, and print scannable
answer documents to correspond with the
appropriate test booklets;
produce and print Braille, large print, and
one-item-per-page (OIPP) special format
materials using the final base test forms
produced by AIR;
print PBT test administration manuals
using print-ready files produced by AIR;
contribute and review content regarding
PBT administration procedures;
print student PreID labels and PreID rosters
using student PreID data files provided by
AIR; and
print other supporting administration materials, including administration record/security checklists, packing lists, pallet lists,
shipping cover memoranda, return shipping
labels, materials return kits and materials,
paper bands, manipulatives, and other misPage 7.3-106
cellaneous administration materials determined by the Department.

Please see Appendix B for AIR and DRCs proposed Materials List and Appendix C for samples of testing-related materials DRC has produced for other large-scale state assessment
programs. DRC will provide all materials in the
appropriate formats and will distribute materials in sufficient quantities to Florida districts
and schools. Please see Section 7.3.11 for a
discussion on producing Braille, large print,
and OIPP materials; Section 7.4.1 for information on DRCs packing and distribution
processes; and Section 7.4.7 for details on
ancillary materials
Producing All Versions of Materials and

Products
AIR understands it is responsible for producing
all versions of the materials and products
developed for this contract, including test
forms. Detailed responses to materials requirements are included in Section 7.3.7, 7.3.8,
7.3.9, and 7.3.10. In Section 7.3.11, we
describe in detail our approach to producing
paper-based accommodated test materials.
Proofreading Documents During All

Stages of Production
As described in our response to 7.4.6, AIR has
developed an editorial review process for materials production that minimizes errors and
allows client review at each appropriate stage
of production. Readers are directed to Section
7.4.6 for a complete description of this
process, including points at which AIR will
obtain Department feedback and approval.
Qualifying persons who will be responsible for
proofreading are included in the organizational
chart in Section 7.7, and their resumes are
included in Appendix A.
Quality of Printing
AIR will develop and publish print-ready files
of all test materials except scannable answer
documents, special format material accommodations, and PBT ancillary materials. DRC
will print all PBT materials using the final
Department-approved print ready files and
produce special format materials using the
final approved base forms provided by AIR.
AIR and DRC understand that any changes
after signoff will require Department approval.
Please see Sections 7.3.77.3.11 and 7.4.6 for
a detailed discussion of AIRs materials publication and production processes. Printing,
publishing, and distribution of test materials
will not begin until final approval has been
provided by the Department. DRCs materials
production and printing quality processes are
described in detail below.
DRCs Materials Production Quality Processes
DRCs Editing Services provides professional
technical editing and copyediting of assessment materials for its department of education
clients. The Document/Graphic Design Group
is staffed with highly experienced desktop publishers, graphic designers, and proofreading
professionals. DRC uses the latest design and
publishing software and works with multiple
file formats and platforms. Please see Section
7.7 for information on DRCs proposed publishing and proofreading staff. Resumes are
included in Appendix A.
DRCs ISO 9001:2008 certified full-service
Document Services produces materials from
typesetting to editing and printing. DRCs
printing presses allow it to print both
scannable and non-scannable documents. DRC
also has variable intelligent printing (VIP)
capability, and provides complete, custom finishing services. DRC has the capacity to
allocate significant and superior printing
resources. In addition to ISO 9001 certifica-
tion, DRC has earned Quality Level II status

from the Government Printing Office (GPO),
which is the second highest status that can be
awarded (Quality Level I status is reserved for
printers who produce bound books, four-color
varnished promotional pieces, or other similar
materials). To obtain this quality status, DRC
underwent a rigorous review and inspection
process, including work samples and process
documentation.
AIR and DRC understand that the Department
requires error-free materials and that we have
the ultimate responsibility for the quality and
accuracy of the testing materials. DRC knows
it is essential that all test materials are in their
intended and technically correct format. To
this end, DRC follows a meticulous set of quality standards to ensure high-quality printed
products for its clients. The Department can
be assured of AIR and DRCs commitment to
produce attractive, error-free materials.
The quality processes AIR and DRC will use to
achieve error-free production include:
Materials Publishing Review and Quality
CheckAIR will conduct word-for-word
proofreading during the electronic publishing stage (typesetting and layout) of all test
materials except scannable answer documents and special format material accommodations. AIR will also conduct a final
review of test booklets, answer documents,
and correlating test administration manuals
and instructions to confirm that all directions work in concert and are accurate and
easy to follow. Prior to bulk printing, AIR
will conduct final reviews of printers proofs
of regular PBT materials.
DRC will conduct word-for-word proofreading
for large print and OIPP materials to ensure
that these materials align with the base form(s)
approved by the Department. DRC will also
conduct a final review of the large print and
Page 7.3-107
OIPP test booklets and correlating test administration manuals and instructions to confirm
that all directions work in concert and are
accurate and easy to follow. Prior to bulk
printing, DRC will conduct final reviews of special format materials printers proofs. DRCs
Braille vendor will conduct similar proofreading and final reviews of Braille materials prior
to reproduction. The Department will review
and approve special format materials, including Braille materials, using the materials development and review processes described in
Section 7.3.77.3.10. Please see Section
7.3.11.1 for more information on DRCs
processes for producing accommodated materials.
Information Systems Review and Quality
CheckDRC will certify that all scannable
documents are designed, developed, and
printed within specified scanning requirements and tolerances. This technical
review ensures error-free processing of
scannable documents that will prevent
delays in the data delivery and reporting
schedule. Please see the next subheading
for more information on scannable document printing.
Printers Reviews and Quality Checks
AIR and DRC will conduct final reviews
of printers proofs prior to bulk printing.
Working closely with print vendors, DRC
will check press proofs to make sure print
quality meets DRC standards (e.g., ink consistency, smudge marks, sequential numbering) and verify adherence to printing
specifications. DRC will require every thousandth document be pulled to check print
quality throughout the print run.
Materials Finishing Quality ChecksDRC
Project Management and Print Procurement
personnel oversee materials finishing,
including the collating, stitching, and trimming phases. Post-printing processes (e.g.,
security barcoding, spiraling, shrinkwrapPage 7.3-108
ping) are also monitored to ensure specifications are being followed consistently and
accurately.
DRC successfully uses these quality procedures for its State Department of Education
clients and is confident that their materials
production protocols and standards will meet
the accuracy requirements of the FSA. Please
see Appendix C, for samples of high-quality
test materials that DRC has produced for largescale state assessment programs. DRCs proposed Operations Project Manager, Mr. Doyle
Kirkeby, will provide the Department with
written notification of any issues encountered
during the materials production and printing
process and will consult with AIR and all
appropriate DRC departments to suggest solutions to any issues. Please see Production
Errors below for more discussion of issue and
error management.
Scannable Document Printing
Scannable document printing requires a mastery of the equipment that only years of training, experience, and dedication to the process
can achieve. Documents printed at DRC are
printed to exacting specifications to guarantee
the highest possible data integrity for DRCs
state-of the-art image scanning systems. DRC
prints 520 million scannable pages and produces 12 million test booklets a year. DRC
stocks only those papers that provide the
proper reflectance required by its leading
imaging scanners. The same holds true for
DRCs inks; all have been tested and formulated to impart a crisp, clean look to the product
for superior aesthetic appeal and also to
dropout in the proper measures for accurate
data capture during the scanning process. With
each document, DRCs skilled press operators
hold their presses to incredible tolerances,
exceeding even the strictest industry standards.
Security Plan for Production of Test

Documents
Because DRCs Document Services Division is
under its direct control, a majority of the
printing for clients is completed by DRC staff
at its fully secure printing facility. If outside
printing vendors are used, they are evaluated
and monitored against DRCs stringent security
and quality standards. All vendors must certify
adherence to these standards before documents are submitted to them for production.
Please see below for additional details regarding DRCs printing vendor certification
process.
All of DRCs secure facilities are designed to
meet the stringent security requirements of
large-scale testing programs. DRC enforces
strict security measures to prohibit unauthorized personnel from access to client materials
through either deliberate or unintentional
action. All materials are handled and stored in
an environment secure from access by the
general public or any unauthorized employee.
All DRC personnel are trained in and required
to adhere to the companys security requirements.
The following corporate-wide facility and personnel security measures are in place at DRC:
Employees use keycard identification
badges to enter the DRC facilities. All
employees, permanent and temporary,
must wear a picture identification badge.
Visitors to all DRC sites must sign in and
are issued badges that must be worn at all
times; visitors are accompanied by a DRC
employee at all times.
The secure access system keeps a log of all
persons entering DRC facilities, including
all after-hours and weekend activity.
All employees receive training regarding
DRCs stringent security procedures. Each
employee must sign a confidentiality and
security agreement and ensure privacy of

all data, at all levels, at all times.
Unauthorized personnel are not allowed in
the materials assembly, receiving, check-in,
or document processing and scanning
areas.
All incoming and outgoing shipments are
logged in and logged out.
Confidentiality of individual and client data
is maintained at all times.
All electronic files are maintained at password-protected workstations and are accessible only to key personnel on the project.
Certification of Printing Vendors
DRC views its supply base as an extension of
its business processes, holding its supply partners to the companys same high internal standards. The DRCs Vendor Management
Program
defines a set of DRC process specifications
and other business requirements that supply partners are required to meet;
utilizes an on-site business process assessment where key DRC Procurement and
Quality staff members audit key supplier
business processes; and
requires monthly and annual business
reviews of supplier performance (reviewing
performance requirements around quality,
delivery, and overall service). When necessary, corrective action plans and follow-up
sessions are also included during these
reviews.
DRC has adopted federal standards such as
DIACAP and HIPAA practices to ensure it
maintains the highest levels of security practices as possible. DRCs security practices are
also documented and embedded into the DRC
Quality Management system, which has been
ISO 9001-compliant for over 6 years. These
Page 7.3-109
practices are audited several times a year by

DRC Internal Audit staff and a third-party ISO
assessor.
DRC standards for printed products include
the selection of reliable printing companies
that can guarantee the level of quality and
security implemented at DRC. DRC selects
only printing vendors that have great sensitivity to state testing requirements and timelines
and that have previous experience with printing educational testing materials. Each external printing vendor is required to maintain the
strictest level of security during the production
of materials and is asked to sign a security affidavit attesting to their commitment to this
security. DRC Program Management, Quality,
and Print Procurement staffs work with printing vendors during the print production phase.
DRC staff routinely visits print vendors sites
during the printing, collating, and finishing of
materials to verify that printers meet all quality standards and individual program specifications. DRC staff will be present during the first
printing runs of each product, and at various
times during the collating, stitching, and trimming phases. DRC staff will conduct checks to
make certain that the print quality meets DRC
standards (e.g., ink consistency, absence of
smudge marks, sequential numbering, correct
collating and binding). DRC requires the vendor to pull every thousandth document to
monitor quality.
Production Errors
It will be AIR and DRCs responsibility to conduct a thorough final proofreading of all printed materials, and to ensure that all materials
are correctly printed, sealed, and collated,
with no errors. In the unlikely event that an
issue occurs, within 30 days of the event, AIR
and DRC will provide the Department with a
root-cause analysis, including information
on the cause and steps for resolution and
improvement. We will also be responsible for
Page 7.3-110
any costs that are incurred for additional

printing and distribution of defective or misprinted materials. Reprinted materials will be
produced and distributed according to a timeline approved by the Department.
DRCs risk and issue management approach is
based on proactive planning to prevent, mitigate, and resolve any project issues. Its issues
management process enables the company to
organize, maintain, and track any issues that
may occur. DRC will use this established
process to identify, address, prioritize, escalate, and resolve problems and issues:
Immediately communicate the issue in
writing to AIR and the Department. DRCs
Project Manager will immediately contact
AIR and the Department whenever a significant issue arises. Significant issues are
defined as those that would impact timeliness of contract deliverables. DRC will not
delay notification because of internal discussion or meetings.
Identify what needs to be done and
resources needed to correct the issue. The
DRC Project Manager will work with internal DRC resource groups along with AIR
and Department staff to determine the
proper steps to resolve the issue.
Receive approval from the Department for
appropriate action. After working closely
with all involved parties, AIR and DRC will
propose a sound and cost-effective solution
and process for improvement to the
Department. AIR and DRC will not proceed
with the solution until approval has been
received from the Department.
Keep the Department informed on the
status of the issue based on a frequency
established by AIR and the Department.
AIR and DRC will communicate with the
Department on a regular basisas determined by the Departmentregarding the
solution for the issue.
At least weekly, provide a listing of all

issues with their current status, deadlines
to correct and improve, and actual dates of
completion that have occurred to AIR and
the Department. These weekly reports will
be submitted according to a schedule
requested by the Department. Issues will
also be documented in the monthly
progress reports to the Department.
AIR is confident that DRCs issue management
process will assure timely delivery and comprehensive materials production issue resolution.
7.3.9 Production of ComputerDelivered Products

Here we describe AIRs process for delivering
computer-delivered products. AIRs well-honed
processes enable us to deliver high-quality,
reliable, and accurate online assessments.
Here, we offer a high-level description of the
key processes that lead to these outcomes.
The production of computer-delivered products involves two divergent types of products,
each of which follows an appropriate quality
process:
1. Content for online delivery shares some
processes with paper versions, but also
requires additional, unique steps.
2. Online test delivery software must deliver
that content reliably, and with the right
tools, accommodations, layouts, etc.
Below, we discuss our production process for
content as well as our process for the production, configuration, and deployment of software.
Our test delivery system also has a real-time
quality monitoring component built in. As students test, data flow through our Quality
Monitor (QM) software. QM conducts a series
of data integrity checks, ensuring, for example,
that the record for each test contains information for each item that was supposed to be on
the test, and that the test record contains no
data from items that have been invalidated.
QM scores the test, recalculates performancelevel designations, calculates subscores, compares item parameters to the reference item
parameters in the bank, and conducts a host of
other checks.
QM also aggregates data to detect problems
that become apparent only in the aggregate.
For example, QM monitors item fit and flags
items that perform differently operationally
than their item parameters predict. This functions as a sort of automated key or rubric
check, flagging items where data suggest a
potential problem. This automated process is
similar to the sorts of checks that are done for
data review, but (a) they are done on operational data and (b) they are conducted in real
time so that our psychometricians can catch
and correct any problems before they have an
opportunity to do any harm.
Production of Content
While the online workflow requires some additional steps, it actually removes a substantial
amount of work from the critical path. Like
a test book, an online system can deliver a
sequence of items; however, the online system
makes the layout of that sequence algorithmic.
A paper form must await final forms construction before blackline proofs can show how the
item will look in the booklet. Online, the
appearance of the item screen can be known
with certainty before the final test form is ever
constructed. This characteristic of online
forms enables us to lock down the final presentation of each item well before forms are
constructed. In turn, this moves the final blueline review of items much earlier in the
process, removing it from the critical path.
Page 7.3-111
The production of computer-based tests

includes five key steps:
freezes the display code that will present the

item to the student.
1. Final content is previewed and approved in

a process that we call web approval. Web
approval packages the item exactly as it
will display to the student.
Web approval functions as an item-by-item

blueline review. It is the final rendering of the
item as the student will see it. Layout changes
can be made after this process in two ways:
2. Forms are finalized using the process

described in Section 7.3.7, and final forms
are approved in our FormBuilder software;
1. Content can be revised and re-approved for

web display.
3. Complete test packages are created with

our test packager, which gathers the content, form information, display information,
and relevant scoring and psychometric
information from the item bank and packages it for deployment.
4. Forms are initially deployed to a test site
where they undergo platform review, a
process during which we ensure that each
item displays properly on a large number of
platforms representative of those used in
the field.
5. The final system is deployed to a staging
environment accessible to our client for
user acceptance testing and final review.
We discuss these steps below.
Web Approval of Content during
Development
AIRs Item Tracking System (ITS) integrates
directly with the Test Delivery System (TDS)
display module, and displays the item exactly
as it will appear to the student. We call this
process web preview, and we tie web preview
to specific review levels. Upon approval at
those levels, the system locks content as it will
be displayed to the student, transforming the
item representation to the exact representation that will be rendered to the student. No
change to the display content can occur without a subsequent web preview. This process
Page 7.3-112
2. Online style sheets can change to revise

the layout of all items on the test.
Both of these processes are subject to strict
change control protocols to ensure that accidental changes are not introduced. Below, we
discuss automated quality control processes
during content publication that raise warnings
if item content has changed after the most
recent web-approved content was generated.
The web approval process offers the benefit of
allowing final layout review much earlier in
the process, reducing the work that must be
done during the very busy period just before
tests go live.
Approval of Final Forms
Section 7.3.7 describes our process for constructing forms, including the approval of 1x1s
by the Department. The forms are built in
FormBuilder, a component of our ITS, and
upon approval they are ready for preliminary
publication. Please refer to Section 7.3.7 for a
description of the quality processes for final
form approval.
Packaging
AIRs test packaging system performs two
simultaneous roles in the preparation of computer-based products: It compiles the form definitions and other information about how the
test is to be administered (e.g., where any
embedded field-test items might be inserted);
and pulls together the content packaged during

web approval.
The test packager assigns form identifiers to
each form, evaluates the form against the blueprint, and performs a quality check against the
content. The content quality check includes
checks to see that every asset (e.g., graphic)
referenced in the item is included in the package, confirms that the item has not changed
since it was web approved, and ensures that
the items have received all the approvals necessary for publication.
Platform Review
Platform review is a process in which we
check that each item displays appropriately on
each platform. A platform is a combination of
a hardware device and an operating system. In
recent years the number of platforms has proliferated, and platform review now takes place
on approximately 15 platforms that are significantly different from one another.
Platform review is conducted by a team. The
team leader projects the item as it was webapproved in ITS, and team members, each
behind a different platform, look at the same
item to see that it renders as expected.
User Acceptance Testing and Final Review
process. While we can support this if timelines

are sufficient, we strongly encourage the
Department to conduct these reviews and
make these edits earlier in the process, ideally
before web approval.
Functionality and Configuration

The items, both in themselves and as configured onto the tests, form one type of online
product. The delivery of that test can be
thought of as an independent service. Here, we
document our process for delivering this service with industry-leading quality.
One area of quality unique to online delivery
is the quality of the delivery system. Three
activities provide for the predictable, reliable,
quality performance of our system:
1. Testing on the system itself to ensure function, performance, and capacity
2. Capacity planning
3. Continuous monitoring
AIRs system is mature. It has been through
many testing and operational cycles. With
each system upgrade, the upgrade undergoes
unit testing and the system undergoes integration testing and regression testing. Each
deployment undergoes a rigorous load and
performance test.
Prior to deployment, the testing system and

content are deployed to a staging server where
they are subject to user acceptance testing
(UAT). UAT of the test delivery system serves
both a software evaluation and content
approval role. We discuss the software aspects
in the next section. The UAT period provides
the Department with an opportunity to interact with the exact test with which the students
will interact.
Our statisticians examine the delivery

demands, including the number of tests to be
delivered, the length of the window, and the
historic state-specific behaviors to model the
likely peak loads. Using data from the load
tests, these calculations indicate the number
of each type of server necessary to provide
continuous, responsive service, and AIR contracts for service in excess of this amount.
We understand that the Department may want

to make authors edits at this stage of the
Once deployed, our servers are monitored at

the hardware, operating system, and software
platform levels with monitoring software that
Page 7.3-113
alerts our engineers at the first signs that trouble may be ahead. Our applications also log
not only errors and exceptions, but latency
(timing) information for critical database calls.
This information enables us to know instantly
whether the system is performing as designed,
or if it is starting to slow down or experience a
problem.
Our monitoring does not stop there. Each student test gathers latency datadata about how
long it takes to load, view, or respond to an
item. All of this information is logged as well,
enabling us to identify automatically schools
or districts experiencing unusual slowdowns,
often before they even notice.
Functionality, Configuration, and User
Acceptance Testing
Each year, AIR implements new features and
capabilities in the test delivery system. These
features are thoroughly tested through a mix of
manual and automated testing, all of which follows a detailed and complete testing plan.
These enhancements are made available to all
of our clients at no additional cost (although
costs may be incurred when content must be
created or modified to exploit a feature). This
has made our system very highly configurable.
AIR has a well-established process for walking
through configuration options and documenting decisions. The configuration covers everything from styles and layouts to the particular
accommodations offered with each test. AIR
project managers will work with Department
representatives to document the required configuration. Our TDS team will deploy the system to a staging site with the configured
options, and that configuration is reviewed by
our project team during a formal specification
review. That staging site becomes the site for
UAT, the process in which the Department will
provide final signoff on the test as it will be
deployed.
Page 7.3-114
Load Testing
The ITN foresees peak loads of approximately
50 percent of the enrollment in the tested
grades. AIRs system is designed to handle
loads that high and much higher. Every function of our system can be scaled outward using
our proprietary private cloud architecture.
Supporting increased loads requires deploying
more servers in the appropriate configuration.
We note that our cloud is a private cloud,
rather than a public cloud, which means that
testing traffic is insulated from any other traffic in the data center, giving us the ability to
fully control the system throughput.
Load testing happens in two stages. The first
stage is a capacity planning stage, in which
small numbers of servers (actually server constellations) are tested to the breaking point.
This provides the information needed to know
the total number of servers required to support
the anticipated load. Capacity planning tests
occur every few years as server technology
improves. We are currently conducting these
tests with new machines.
The second stage is the actual load test. Once
the full set of servers is deployed, we generate
a load exceeding the intended peak load. That
load is generated by a large number of servers
simulating students testing. For example, this
winter we plan a load test of one system that
will include 5,000 servers in a remote facility
simulating the load produced by 500,000 students simultaneously testing.
AIR will be pleased to share load testing results
with the Department.
User Acceptance Testing
Before each live deployment, AIR will deploy
the system to the staging servers for a negotiated period of time for the Department and
your designees to test the system with the
actual test forms deployed as they will be seen

by students.
of every review stage, and we will provide all

final files as well.
AIR will prepare a UAT plan, suggesting a wide

range of cases to test, and will load the system
with fictional students for Department testers
to use while testing. Of course, the Department
is welcome to help craft or expand the UAT
plan, or simply try additional ad hoc cases
during the UAT period.
Our partner, DRC, will print all additional

materials, such as labels and packing sheets as
required to ship materials. We note that our
plan includes a real-time-ID supplement to the
pre-ID process. This allows nearly real-time
registration for students for both paper and
online tests. Late arriving students can be registered for the test immediately before testing,
and schools or districts may print out realtime-ID labels from the Test Information
Distribution Engine (TIDE) on label stock that
we will provide along with the shipments of
books and other materials. Readers are
referred to Section 7.4.1 for a detailed description of this process.
Final approval of UAT triggers final deployment of the test.
7.3.10 Production of Paper-Based

and Computer Test, Style Guide,
and Braille Production Procedures
AIR will be responsible for all aspects of production of the computer-based testing (CBT)
system, secure web-based reporting site, and
printed materials, including formatting, graphics, and key entry. Each process and the
details of the products will be fully documented in a style guide or appropriate specifications document. These documents and specifications are described throughout this project.
Readers are referred to the following sections
for details of the production of the products,
style guides, and specifications documents referenced in this section of the ITN:
CBT system: Section 7.3.9
Secure web-based reporting site:
Section 7.5
Printed materials: Section 7.3.8
AIR has established procedures for selecting
and communicating print plans, including all
necessary specifications. AIR is comfortable
with providing these plans to the Department
as required. AIR will also provide print samples showing type and size for all needed materials. AIR typically provides our clients copies
AIR will submit printing plans showing all

needed specifications for a correct print run.
AIR will also work with the Department to create a style guide that lays out exactly what the
correct look and feel for printed materials is.
This style guide will address form layout, font,
and size, as well as all other needed specifications. AIR is comfortable either adapting the
Departments current style guide for use in an
online assessment, or AIR is also able to completely rewrite the style guide with the
Departments input.
The style guide will include the editorial specifications. Typical information in a style guide
includes appropriate formats for graphics,
illustrations, tables, charts, graphs, and other
style elements. The style guide would also
include appropriate layouts, acceptable uses of
punctuation, bolding, and underlining.
A glossary of preferred and accepted technical
terms and phrases when referencing particular
features of the online assessment should be
specified to allow for consistency across items
as well as student accessibility. Specifying text
for specific guidance to examinees, such as To
Page 7.3-115
remove your selection, click it again, should

be incorporated to ensure that students know
exactly what is required of them in any specific task. The style and color of different item
components should also be outlined in an
updated style guide.
This updated style guide should be regularly
reviewed. AIR has typically gathered comments throughout the year, and then included
all gathered comments in the annual review
cycle. Once the document has been agreed on,
AIR will print revised comments as required.
Our test delivery specifications, described in
Section 7.4.9.7, capture information about
themes, styling, layouts, and other characteristics of the test. While some of the navigation
details are configurable, the navigation elements themselves are integral to the functioning of the system. All text, including labels and
messages, can be customized for the Florida
Standards Assessment (FSA).
AIR will configure all of AIRs online system
access points so they reflect the Departments
preferred look (skins) and color themes.
Additionally, any print materials will be
designed with the same themes so readers
instantly make the connection to Florida
statewide assessments across media. Last, we
recognize that the overall communication plan
will encompass other deliverables; we will
make sure that each piece of information is
coordinated and consistently branded. AIR
will use its practiced review procedures to
ensure error-free and consistent branding
across all Florida systems.
Page 7.3-116
7.3.11 Produce Special Formats and

Accommodations for Students With
Disabilities
In the following sections, we discuss paperbased and computer-based accommodations
AIR will provide for students with disabilities.
7.3.11.1 Paper-Based
Accommodations
AIR and DRC are committed to fair accessibility for all students. AIR team member DRC will
produce, print, and distribute all testing and
administration materials for paper-based special format accommodations, including Braille,
large print, and one-item-per-page (OIPP)
materials. AIR and DRC understand that only
one form for each test will be converted to
Braille, large print, and OIPP versions. AIR will
provide the final Department-approved versions of the selected test forms for conversion
by DRC. AIR and DRC understand that in
addition to test documents, we may also need
to provide large print and/or Braille versions of
the test administration manuals, up to five
each per format per administration, if requested by districts. All paper-based accommodations materials, including test administration
manuals, will be delivered to districts with the
shipments of regular print materials.
DRC has worked with the National Center on
Education Outcomes (NCEO) on many assessment projects, including the design and development of alternate assessments for students
with disabilities and English language learners
(ELL), as well as designing and facilitating
standard settings for alternate assessments.
DRC stays abreast of current legislation, technologies, and methodologies that impact universally designed assessments and retain staff
members that are knowledgeable, trained, and
experienced in developing and administering
assessments that are accessible to all students.
AIR and DRC are ready to assist the

Department with the development and administration of tests that are universally designed,
provide for test accommodations/ modifications, and meet state and Federal legislation
requirements.
Braille Versions
DRC has extensive experience managing the
production of Braille test booklets and materials for many assessment programs, including
those for Alaska, Louisiana, Minnesota,
Pennsylvania, and South Carolina. Each year,
DRC will work with staff from the nationally
recognized American Printing House for the
Blind (APH), to produce Braille test booklets
and other materials for the FSA. APH is highly
experienced with converting assessment materials; DRC has used their services on other
testing programs with excellent results. APH
will also provide transcription of Braille-format
student responses (please see Section 7.5.4.4
for more information on transcription).
APH will ensure that all tests and materials are
modified correctly and that they are accurate.
Codes accepted by the Braille Authority of
North America (BANA) will be followed. Test
documents will be produced in both contracted and uncontracted Braille to accommodate
all Florida students and will be distributed as
requested by districts. Each Braille form for
each test will be produced in two volumes.
DRC and APH will maintain and update an
internal document that identifies which APH
professional, including role and name, will perform each Braille material production task, as
well as detailed information about each task.
This document will be made available as
requested by the Department. In addition, AIR
and DRC will ensure that all Braille materials
are proofed by a certified Braille reader/independent party. DRC and APH will also produce
Braille version test administrator notes and
scripts to supplement the test administration
manuals following the same production

processes described in Section 7.4.6.
Information on packaging and distributing special format tests is provided below, while
Section 7.5.4.4 discusses processing and transcribing special format student responses.
The Braille test booklets and materials will be
provided to the Department for review and
approval before reproduction. AIR and DRC
understand that the Department may also
employ the services of a Braille proofreader.
Braille versions will be available for test coordinators to order with the rest of their materials, and DRC will ensure that a sufficient number of copies are available for any last minute
orders. DRC acknowledges that approximately
50 copies of each Braille material may be
required for each grade level and subject area
per administration. In addition, for each set of
Braille forms distributed to districts, a copy of
the corresponding regular print test booklet
will be included. Each Braille test form will
also be accompanied by six blank pages of
Braille paper.
Large Print Versions
DRC has full, in-house capability to develop,
produce, reformat, and print large print test
materials. They have extensive experience
providing large print materials for many state
assessment clients, including Alaska, Louisiana,
Minnesota, Pennsylvania, and South Carolina.
DRC will provide large print materials formatted to meet the needs of the Florida Standards
Assessments. AIR and DRC will work with the
Department during typesetting and production
to ensure all specifications for the large print
versions are met. DRCs experience, along with
best practice, indicates that type size and line
spacing should be adjusted to the students
reading and grade level. DRC also recommends
providing the 14 x 17 booklets in a spiral
binding that allows the book to open flat. DRC
Page 7.3-117
understands that large print documents will be

printed in a minimum of 18-point type, and
that the paper and binding must be approved
by the Department.
The large print test booklets will be provided
to the Department for review and approval
before reproduction. Large print versions will
be available for test coordinators to order with
the rest of their materials, and DRC will
ensure that a sufficient number of copies are
available for any last minute orders. DRC
acknowledges that up to 400 large print copies
may be required for each grade level and subject area per administration. In addition, for
each set of large print forms distributed to districts, a copy of the corresponding regular
print test booklet will be included. DRC will
also provide large print reference standalone
sheets and large print response grid instructions standalone sheets to accompany applicable assessment forms. Each computer-based
test (CBT) Mathematics form will also be
accompanied by one large print form of the
CBT work folder, and each CBT English
Language Arts/Literacy (ELA/L) assessment
will be accompanied by one large print form of
the CBT worksheet. DRC will also produce
large print test administrator notes and scripts
to supplement the test administration manuals, following the same production processes
described in Section 7.4.6. Information on
packaging and distributing special format tests
is provided below, while Section 7.5.4.4 discusses processing and transcribing special format student responses.
One-Item-Per-Page/One-Sided
Versions
requires an OIPP special format for the FSA.
DRC has extensive experience providing OIPP
versions of testing materials for its assessment
clients, including Louisiana, Nebraska, and
South Carolina. Based on these experiences,
Page 7.3-118
DRC understands the unique issues associated

with this type of accommodation, including
font type and size, paper whiteness, positioning, graphic/stimulus appearance, and layout of
passages/stimuli and corresponding items.
Because of the unique nature of the OIPP
format, AIR and DRC will work with the
Department to develop and clearly define all
specifications prior to typesetting and production, particularly regarding paper type and
quality and font style and size. Department
review and feedback during the typesetting
and production phases will be a key component of producing appropriate OIPP assessment materials. DRC understands that FSA
OIPP materials will be printed on three-hole
punched 8 x 11 paper and collated and
bound using binders. Each OIPP test form will
be accompanied by the appropriate scannable
answer document.
The OIPP test materials will be provided to the
Department for review and approval before
reproduction. AIR and DRC understand that
the Department will provide a list of students
requiring this accommodation, and that
approximately 25 copies of each OIPP version
may be required for each grade level per
administration. DRC will also produce OIPP
version test administrator notes and scripts to
supplement the test administration manuals
following the same production processes
described in Section 7.4.6. Information on
packaging and distributing special format tests
is provided below, while Section 7.5.4.4 discusses processing and transcribing special format student responses.
Packaging and Delivery of Special Print
Materials
All Braille, large print, and OIPP materials will
be packaged separately and delivered to districts with the regular print shipments. DRC is
accustomed to packaging, shipping, and receiv-
ing special format materials. These materials

will be handled with the same quality and
security procedures used with other FSA testing materials (please see Section 7.4.1). DRCs
receipt processes for accommodated materials
require little document preparation by schools
and districts. Unlike many testing companies,
DRC does not require special packaging,
return, and processing procedures for accommodated materials. DRC will provide a clear,
well-documented return process for Braille,
large print, and OIPP materials, increasing the
accuracy and decreasing turnaround time of
return shipments. Please see Section 7.5.4.4
for more information on processing and transcribing special print test materials.
Paper Accommodations for Computer-Based
Reading Tests
AIR and DRC understand that Reading Passage
Booklets will be provided for students that
require paper-based handwritten strategies for
completing computer-based assessments.
These booklets will be provided in regular
print and large print format as required.
7.3.11.2 Computer-Based
Accommodations
AIRs Online Test Delivery System (TDS) is
designed for maximum accessibility.
In Oregon during the 20122013 school year,
all students eligible for the general assessment
tested online. Students requiring a paper
accommodation were tested online using the
secure print-on-demand accommodation.
Students requiring Braille were served using
real-time embossing and refreshable Braille
displays. Other than the Extended Assessment
for students with severe cognitive disabilities,
the only test Oregon makes available to students offline is the writing assessment, which
23,456 students took in paper form last year.
In Hawaii, during the 20122013 school year,
221,552 students sat for assessments. Of that

number, only eight completed a paper version
of the test.
AIRs approach to accessibility favors universal
design over separate accommodations. Each of
our accessibility features can be configured for
universal access (all students can access the
feature) or for access by specific students.
Features that are configured for access by specific students are set remotely by the test
administrator or other authorized adult (such
as a school or district test coordinator or a
state-level administrator) working on a separate workstation. Florida will have the option
to have accessibility features (screen reader,
font size, background color, glossary, etc.) for a
given student set in advance through TIDE,
through either an upload or user interface, or
set in real time by the test administrator
through the TA Interface. The system can be
configured to have each accommodation settable through either mechanism or both mechanisms, and supports different mechanisms for
different accommodations.
In general we recommend that most features,
even those that are universally available, be
set by adults prior to testing, rather than the
student interface. The reason for this recommendation is that the sheer number of features can create a distraction in the student
interface. Students may play around and
change background colors, experiment with
text-to-speech voices, play with masks and
guides, etc. Therefore, too many student
choices can actually reduce access.
Providing these tools through context menus,
rather than through keyboard shortcuts,
results in a usable and accessible student
interface. Students who cannot use a keyboard
often use assistive devices such as switch
arrays that generate a limited number of keyboard commands. Maintaining compatibility
with these devices precludes creating keyboard
Page 7.3-119
shortcuts for each of the many features

offered; many devices cannot generate that
many different codes.
Usability for all students also favors a contextmenu approach. Many tools are available, but
most are relevant only in a limited number of
contexts. For example, the response choice
eliminator (strikethrough) is only relevant for
selected response items, and only for their
response options. Therefore, this option is
only offered when a student brings up the
menu on a response option. The screen would
be cluttered with many, many icons and buttons if all tools were available at all times.
Context menus offer users the capabilities that
are both available and relevant to their current
task.
Section 7.4.9.6 discusses the accessibility
features and embedded supports provided in
the system. Exhibit 7.3.11.2-1 summarizes
those features, followed by a discussion of the
specific features identified in the ITN.
Below, we discuss each of the accommodations
required in Floridas ITN in greater detail.
Screen Reader
The test delivery system supports text-tospeech through built-in, configurable text-tospeech, and through support for third-party
screen readers. AIRs system provides text-tospeech in English and other languages designated by the client. Items are annotated so
that equations, graphics, etc. are read correctly. Students can use the native voice packs
available on their computer system or a variety of commercially available voice packs. AIR
also provides, at no cost, a commercial
Spanish voice pack for Windows. In addition,
AIR offers, at no cost, a high-quality English
language voice pack for Windows machines,
because older Windows machines often have
lower-quality voice packs installed. Students
Page 7.3-120
can set speed, volume, and pitch before starting a test.

AIRs powerful text-to-speech system allows
varying levels of support according to rules
that we can configure in consultation with
Florida. Thus, while some students may have
(for example) only question text read to them,
other students may also need descriptions of
tables and graphics read aloud. These differentiated supports can be configured to meet
Floridas needs within broad boundaries.
Additionally, within the parameters set by
Florida, students can choose to have only sections of text read to them. For example, after
hearing an entire item a student may want to
listen to one of the response options a second
time. Using the context menus built into AIRs
online testing system, the students can choose
to have just one response option or a section
of text read aloud.
For third-party screen-readers AIRs test renders in our streamlined interface which works
with most commercial screen readers. Items
are also tagged for access by students who are
blind as non-visual navigation requires different tags and annotations.
Screen readers can also output text to a refreshable Braille display (RBD). Equations and
graphics cannot be output to an RBD, so realtime embossing is also available and built into
our system. Specifically, we render items to
desktop embossers (which cost between
$5,000 and $10,000 each) that can integrate
Braille and tactile graphics. The items simultaneously render on a reader-accessible screen
using our streamlined interface, and the student will be able to navigate to response
spaces to provide answers.
The process of annotating (tagging) items for
screen readers is an integral part of AIRs item
development procedures as discussed in
Section 7.3.3 of this proposal. After final item
Exhibit 7.3.11.2-1: Summary of Required, Preferred, and Additional Accessibility Features Available through AIRs Test
Delivery System
Feature
Description
Content annotation
Required or
Available in Requires content
available in proposed
requested in RFP
system
annotation
item bank
Screen reader
Built-in text-to-speech, support for

third-party screen readers. Reads text and
describes other media on the screen.
Third-party screen readers often drive
refreshable braille displays.
yes
yes
yes
yes
Increased font size

(large print and zoom)
Font size can be set in advance by a test

administrator or authorized adult. Zoom
buttons are available to students for
individual items.
yes
yes
no
n/a
Font and background

color (color contrast)
Congurable to support any combinations of

font and background color desired including
reverse contrast. Choice of background colors
aects the colors that can be used in
graphics, so we recommend that users stick
with a small number of colors.
yes
yes
no
n/a
Assistive and adaptive

devices
Technologies that mimic keyboard or mouse

work out of box. Technologies that require
the use of on-screen windows require that
student accommodation be set by the test
administrator.
yes
yes
no
n/a
English-to-heritage
language dictionary
Translations can be tagged to individual

words or phrases in passages and questions.
Supported in any language.
yes
yes
yes
English-to-sign language
translation dictionary
American Sign Language supported as a full

translation or tagged to a word/phrase as a
glossary entry.
yes
yes
yes
English language
glossaries
Denitions can be tagged to individual

words or phrases in passages and questions.
no
yes
yes
Line reader
Can be congured to be available for

passage reading.
no
yes
no
n/a
Expandable passages
Available for all passages. Allows students to

expand passage to use complete screen.
Temporarily covers up questions associated
with passage, but minimizes scrolling when
reading long passages.
Congurable for any student for passages
and/or items.
Allows students to mark an item for further
review at the end of the test or testing
session.
no
yes
no
n/a
no
yes
no
n/a
no
yes
no
n/a
Print on request
Mark for review
Smarter Balanced
only; does not
currently include
Haitian Creole
no
Smarter Balanced
only
Page 7.3-121
content is approved by the client, the next

step in our Item Tracking System (ITS) is for
the items to move forward so they can be
tagged for text-to-speech (which is used by
sighted students) and text-to-Braille (which is
used by visually impaired students). AIR has a
team of experienced employees who work
closely with our content staff to ensure that
the items are correctly tagged to conform to
any business rules set by our clients and provide the support that students need.
Increased Font Size (Large Print and Zoom).
AIRs online testing system supports a variety
of print sizes. We generally recommend offering five font sizes with the smallest size being
the equivalent of 12- or 14-point font and the
largest size being approximately five times as
large as the smallest size. We will work with
the Department to determine the specific size
options offered.
The default size may be set prior to testing
through TIDE, or may be set by the administrator at the time of testing through the TA
Interface. Most test administrators are aware
of which students need a larger font and set
the font size in the TA Interface at the time
the student is approved to test. Our student
interface also includes Zoom In and Zoom
Out buttons, so students who need a larger
size for a particular item or graphic can control the print size themselves.
Font and Background Color (Color Contrast).
AIRs testing system supports any combinations of background color and font color,
including reverse contrast (white font on a
black background) desired by our clients.
However, in practice the color combinations
chosen influence the choice of colors that can
be used in graphics. For example, if Florida
were to choose to offer students a blue background, graphics would need to be tested to
ensure they display properly against that background and the use of blue in graphics would
Page 7.3-122
need to be limited. Therefore, we recommend

deciding on a limited number of background/
font color combinations at the start of item
development and then tailoring graphics during item development so that they can be
effectively displayed on the background colors
chosen.
The SAGE item bank was developed to support
the following color combinations:
Black font on white background
White font on black background
Black font on rose background
Yellow font on blue background
Black font on blue background
Black font on light blue background
Smarter Balanced items support these color
combinations:
Black font on white background
White font on black background
Black font on magenta background
Yellow font on dark blue background
Dark grey font on light grey background
Black font on light blue background
We recommend that if Florida uses an existing
item bank as the basis for these tests, Florida
retain the color combinations for which the
items were developed.
Assistive/Adaptive Devices. AIRs online testing system offers two interfaces: the standard
interface, with which many assistive technologies work, and the streamlined interface, with
which even more assistive technologies work.
The streamlined interface is consistent with
the principles of Web Content Accessibility
Guidelines (WCAG) 2.0 and AIR is currently
working toward WCAG 2.0 certification.
Assistive technologies that mimic keyboard or

mouse activities generally work with the
online testing system out of the box. Assistive
technologies that use on-screen windows
require an accommodation to be set because
security protocols built into AIRs online testing system stop other windows from showing
on top of the secure browser used for testing.
Permissive mode, which is built into our
testing system, can be set for a student who
requires an assistive technology that uses an
on-screen window to allow the student access
to the assistive technology while accessing the
test in the secure browser.
English-to-Heritage Language Translation
Dictionary. AIRs system supports glossaries
that can be associated with individual words or
phrases in a question or passages. The glossaries can provide a definition or translation or
the tagged word or phrase. The availability of
this feature is configurable by student.
Translations must be entered individually for
each item, so tagging for glossaries is generally
done at the same time as item development
and tagging for screen readers. This allows a
definition or translation to be tailored to the
exact meaning of the word or phrase as used
in the context of the question or passage. AIRs
system can support glossaries in any language.
English-to-Sign Language Translation
Dictionary. AIRs system is able to deliver sign
language by displaying videos of people signing. While developing our sign language capacity we met with experts on sign language and
advocates for students who are deaf. In all
cases we were advised to use videos of actual
people signing rather than signing avatars. Sign
language stands as an independent language
and the avatars are not currently capable of
translating appropriately from English to
American Sign Language (ASL). Furthermore,
subtler aspects of the communication, such as
facial expressions, are not present in the
avatars. Therefore, we made the decision to

offer our sign language capability through
recorded sign translation.
AIR can provide sign language for an entire
item or passage or for specific words or phrases using a glossary as described above. The
challenge of using a glossary is that ASL is not
a direct word-for-word translation of English.
Therefore, if Florida decides to pursue the use
of an ASL glossary rather than translating a
complete question or passage into ASL, care
will need to be taken to identify segments of
English text that can accurately and meaningfully be translated into ASL.
References
Calisir, F., & Gurel, Z. (2003). Influence of text
structure and prior knowledge of the learner on reading comprehension, browsing
and perceived control. Computers in
Human Behavior, 19, 135145.
Fisher, D., Frey, N., & Lapp, D. (2012). Text
complexity: Raising rigor in reading.
Newark, DE: International Reading
Association.
Freebody, P., & Anderson, R. C. (1983). Effects
of vocabulary difficulty, text cohesion, and
schema availability on reading comprehension. Reading Research Quarterly, 18(3),
277294.
Gillioz, C., Gygax, P., & Tapiero, I. (2012).
Individual differences and emotional inferences during reading comprehension.
Canadian Journal of Experimental
Psychology, 66(4), 239250.
Hedges, L. V., & Hedberg, E. (2007). Intraclass
correlation values for planning group randomized trials in education. Educational
Evaluation and Policy Analysis, 29, 6087.
Kish, L. (1965). Survey sampling (second
printing). New York: John Wiley & Sons.
Page 7.3-123
Kucer, S. B. (2010). Readers tellings:

Narrators, settings, flashbacks and comprehension. Journal of Research in Reading,
33(3), 320331.
Long, D. L., & De Ley, L. (2000).
Understanding anaphors in story dialogue.
Memory & Cognition, 28(5), 731738.
McConaughy, S. H. (1982). Developmental
changes in story comprehension and levels
of questioning. Language Arts, 59(6),
580589.
Mensink, M. C., & Rapp, D. N. (2011). Evil
geniuses: Inferences derived from evidence
and preferences. Memory & Cognition, 39,
11031116.
Rich, S. S., & Taylor, H. A. (2000). Not all narrative shifts function equally. Memory &
Cognition, 28(7), 12571266.
Riding, R. J., & Taylor, E. M. (1976). Imagery
performance and prose comprehension in
seven-year-old children. Educational
Studies, 2(1), 2127.
Rommers, J., Dijkstra, T., & Bastiaansen, M.
(2013). Context-dependent semantic processing in the human brain: Evidence from
idiom comprehension. Journal of Cognitive
Neuroscience, 25(5), 762776.
Sadoski, M., Goetz, E. T., & Fritz, J. B. (1993).
Impact of concreteness of comprehensibility, interest, and memory for text:
Implications for dual coding theory and
text design. Journal of Educational
Psychology, 85(2), 291304.
Schmitt, N., Jiang, X., & Grabe, W. (2011). The
percentage of words known in a text and
reading comprehension. Modern Language
Journal, 95(1), 2643.
Sparks, J. R., & Rapp, D. N. (2011). Readers
reliance on source credibility in the service
of comprehension. Journal of
Experimental Psychology: Learning,
Memory, and Cognition, 37(1), 230247.
Page 7.3-124

AIR understands that the contractor is responsible for all aspects of test administration
described in this section of the ITN. Below we
provide detailed descriptions to our approach
of each of the requirements of this section.
7.4.1 Pack and Distribute Materials

In this section we describe our proposal for
packing and distributing test materials, including cost, data file transfer, and packing specifications.
Packing, Distributing, and Retrieving

Arrangements and Costs
AIR and DRC will be responsible for all distribution and retrieval costs and guarantee that
all test materials will be delivered in a timely
and accurate manner. Accurate packing, shipping, and delivery of test materials are critical
to the successful administration of assessment
programs. AIR team member DRC is proud of
its 30+-year history of providing accurate and
on-time materials packaging, distribution, and
retrieval for assessment programs across the
country. DRC is committed to upholding this
same level of service for the FSA. No school
should experience a delay in its testing or
reporting schedule due the inability of a vendor to meet deadlines or deliver error-free
materials. Please see Packing, Distributing, and
Receiving Materials below for additional information regarding DRCs ISO 9001:2000-certified materials packaging, shipping, delivery,
and materials return processes, including identification of carriers.
Packing and Distribution

Specifications for Each Test
Administration
AIR and DRC will work with the Department
to identify and thoroughly document specifications and business requirements for all
processes, systems, applications, and deliverables that are developed for the FSA program,
including those for materials quantity calculations, packaging, distribution, retrieval, inventorying, and processing. These specifications
will be used by AIR and DRC as the basis for
programming and developing Florida-specific
systems and processes, as well as training
Project Team personnel. A draft specifications
document will be provided at the initial contract management kick-off meeting, along with
samples of packing and inventory lists and
memoranda. All requirements documentation
will be in a Department-approved format and
will be reviewed and approved by the
Department prior to initiation of any project
activity or task. As requirements change for a
project phase or activity change, the documentation will be updated accordingly, so that a
complete record of specifications is consistently available to all AIR and DRC Project Team
members and the Department.
Number of Districts and Special Schools

AIR and DRC have reviewed the information
provided in the ITN, appendices, and addenda
regarding anticipated numbers of students,
schools, and districts. We understand that
these numbers will change throughout the life
of the contract. DRC acknowledges that test
materials will be shipped to approximately 80
separate sites, including districts and remote
school sites, with most materials shipped to
districts for distribution by the districts to
their schools.
Page 7.4-1
Distributing and Receiving Materials

to Remote School Sites
DRC is experienced with working with states
that have unique materials distribution conditions and requirements, including remote
school sites, non-public schools, charter
schools, special testing sites, and detention
facilities. AIR and DRC understand that test
materials may need to be shipped to up to 25
remote school sites not geographically located
within their contracted school districts. The
Department will provide AIR and DRC with a
list of approved remote sites.
District and School List Data File

AIR and DRC are accustomed to working with
states to obtain, receive, and maintain student,
school, and district data files. AIR understands
that the Department will provide us with data
files containing a list of district and school
names and identification numbers at the
beginning of the contract. We will maintain
and update the list throughout the contract
and will provide a copy of the data file to the
Department at the end of the contract, or
upon request. AIR will provide this data file to
DRC for incorporation into their assessment
systems as appropriate. Please see Section
7.5.2 for more information on AIRs enrollment
and student PreID systems and processes.
The real-time-ID process will eliminate the

need for staff to grid student information onto
test booklets, eliminating a big source of error.
In addition, because all demographic and
enrollment data will be captured before testing, district data cleanup can commence while
testing is still going on.
Our real-time-ID process supplements a traditional PreID process, rather than supplanting
it. Using student data files provided by AIR
(see Section 7.5.2), DRC will print student
PreID labels and rosters. Label information will
include all information required by the
Department, such as unique student ID number and barcode, student last and first name,
school name and number, district name and
number, grade and subject area, date of birth,
gender, ethnicity, student classification data.
DRC will use the same data files to produce
student rosters in the same sort order used to
prepare the student labels. DRC will ensure
that the information contained on the student
rosters accurately corresponds with student
label information so that districts and schools
can easily use the rosters to verify the accuracy of student label information.
Exhibit 7.4.1-1: Sample Student PreID Label Information
Production and Delivery of PreID

Labels and Rosters
One critical aspect of our plan to speed the
processing, reduce cleanup time, and improve
data quality involves providing a mechanism
for schools and districts to create real-timeID labels for newly enrolled students. This
process, described in detail in Section 7.5.2
will enable schools and districts to enroll students and print labels in near-real-time on
stock that we will provide.
Page 7.4-2
The PreID barcode will ensure that each PBT

student response document returned to DRC
can be unequivocally associated with only one
record in the master PreID database. Exhibit
7.4.1-1 displays an example of the type of
PreID information that can be printed onto a
student label.
Upon receipt of the student response documents at DRC, the barcode number on the student PreID label will be scanned and validated
against the master PreID database provided by
AIR. The barcode corresponds to the PreID
number and is human- and machine-readable.
Printing of the PreID labels will take place at
DRCs in-house printing facility. Requirements
and printed documents will be subjected to
strict QA inspections throughout label production to ensure printing meets DRCs strict
quality standards. These standards ensure
scanning compatibility and accuracy. Print
quality checks include text alignment and ink
consistency, including the absence of smudging or flaking. The format and precision of the
printed information will also be closely examined throughout production by DRCs software
QA analysts to make certain the information is
correct. AIR and DRC will provide the
Department with production samples of student PreID labels for a quality review. Student
PreID labels and rosters will be packaged and
distributed with test materials during the second wave (TM2) to districts after the PreID
window closes. Supplemental PreID labels and
rosters will be produced and distributed to districts that update their PreID files during subsequent PreID windows established by the
Department.
Student Mobility
AIR and DRC recognize the need to address
student mobility. We propose an on-demand
student label printing process (real-time-ID).
This process will accommodate new students,
permit student updates and corrections, and
eliminate the need to hand grid student information on student response documents, thereby increasing the accuracy of student information.
Engine (TIDE) system to enter and/or update

student information; please see Section 7.4.8
for more information on TIDE. Logic will be in
place to determine whether the student exists
in the main student database and already has
a unique identification number; or will require
a unique statewide ID number. The system will
be set up to allow the user to produce ondemand student PreID labels and print them
on local printers. The labels will be similar in
layout and will contain the same student data
as the student PreID labels printed and distributed by DRC, including unique document
identifier barcodes. As labels are generated,
assigned identifier barcodes will be recorded in
the student database, which will be used to
link the answer document during receiving
and processing at DRC.
To help ensure print quality and scanning
compatibility, DRC will provide blank student
PreID label stock to all districts. AIR and DRC
will work with the Department to determine
blank label stock quantities that will be provided to each district. District and school personnel will be directed to use laser printers to
locally print the student PreID labels to ensure
optimum ink quality and prevent unscannable
student labels due to printer or ink issues. In
the event that DRC receives student response
documents with unscannable student labels,
DRC will access the TIDE system to generate
replacement labels.
Materials and Quantities

AIR and DRC have reviewed the types, quantities, and packaging instructions provided in
Appendix E of the ITN. Please see Appendix B
of our proposal for our Materials List for the
FSA.
District/school test administrators will be able

to access AIRs Test Information Distribution
Page 7.4-3
Packing, Distributing, and Receiving

Materials
AIR team member DRC has fulfilled packaging
and shipping requirements for numerous
large-scale assessment programs, including
Alabama, Alaska, Louisiana, Ohio, Oklahoma,
Pennsylvania, and South Carolina. DRCs ISO
9001:2008 certified distribution process underscores the importance of quality standards;
DRC takes all necessary precautions to ensure
accurate packaging and timely delivery of all
materials. In 2012, DRC packaged and distributed more than 15.6 million secure materials.
Materials Packing
DRC will use its Operations Materials
Management System (Ops MMS) for the packaging of testing materials. This proprietary system provides an accurate and efficient method
for packaging materials. Systematic quality
controls facilitate the tracking of secure materials throughout the packaging and distribution
phase. Using scanners to scan out order-specific materials, Ops MMS provides the flexibility to package secure documents already preassigned to a site-specific level or to use barcode scanning to assign secure ranges at the
point of packaging. All requirements provided
by site-specific packing lists must be satisfied
by these scans in order to complete the packaging process. These systematic controls
ensure that the accurate quantity, material
type, and security range is pulled and packaged for the appropriate site.
Through Ops MMS, DRC staff can view data on
the items scanned into any box and compare
this data to the physical box contents.
Random boxes of packaged material are pulled,
opened, and verified for accuracy against the
contents listed in the system. All data generated in packaging will made available to the
Project Management team, providing an adapt-
Page 7.4-4
able tool for monitoring shipments and satisfying client concerns.

All test materials will be packaged sequentially
in ascending barcode and box order. Each box
will include a packing list that lists all materials in sequential packaging order. Materials will
be shipped to districts and remote sites for distribution. Shipments will be packaged and
shipped by district size, from largest to smallest. DRC understands that district size rankings will be updated each year by the
Department. District shipments will be packaged by school and ordered using school numbers, or route code order followed by school
number, if this packaging order is requested by
districts.
Materials will be packaged and shrink-wrapped
in set and subset quantities specified by the
Department. DRC acknowledges that all secure
materials must be shrink-wrapped, and that
some materials such as test administration
manuals will not require this type of secure
packaging.
As specified in the ITN, distribution quantities
for testing materials will include overages of
15% per district for PBT materials and CBT
ancillary materials. AIR and DRC understand
that districts may select from four distribution
algorithms:
1. Entire 15% overage distributed to the district
2. Entire 15% overage distributed to schools
3. Overage split between district (10%) and
schools (5%)
4. Overage split between district (5%) and
schools (10%)
DRC will ensure that a sufficient number of
copies are available at its facilities for any lastminute orders.
Boxes containing materials for use by districts

will be labeled with the name of the district
assessment coordinator and will be labeled on
the top and all four sides. Boxes will be labeled
with the box number and total number of
boxes, e.g., Box 1 of X. When materials for
multiple administrations are shipped together,
labels will be used to identify which boxes correspond to each administration. Each shipping
box will be preprinted with DRCs return
address and affixed with brightly colored labels
stating TO BE OPENED BY ASSESSMENT
COORDINATOR ONLY. District and school
boxes will be different colors so that they can
be easily distinguished. Colored labels will be
used to differentiate between boxes containing
administration materials (e.g., packing lists,
materials return kits, ancillary materials) and
those containing testing materials. All school
boxes will be labeled with the name, number,
school test coordinators name, address, and
Attention: School Test Coordinator using at
least 16-point font. Boxes containing secure
test materials will never be labeled as such
externally. AIR and DRC will work with the
Department to develop labeling specifications.
For most shipments, DRC uses boxes with a
minimum of 275 pounds bursting weight. Box
size will vary to accommodate specific shipping demands but will be small enough to easily handle. All fully packed boxes will weigh
less than 25 pounds and will conform to the
dimensions provided in the ITN. DRC encourages districts to re-use these boxes for the
return shipment but will send additional boxes
to districts who request them. Return labels
containing school and district information will
be included to simplify the return process.
Please see the subheading below, titled
Materials Collection, Receipt, and Secure
Check-In, for more information on DRCs
materials return procedures.
Materials Distribution
AIR and DRC will be responsible for the cost of
all distribution and retrieval to and from each
district, and will guarantee that all test materials are delivered in a timely and accurate manner. We will also arrange for and cover costs
associated with delivering materials to another
district if delivery errors occur. In the unlikely
event that such an error occurs, DRC will
arrange for overnight delivery service for these
materials. AIR and DRC acknowledge that the
Department will not be responsible for the
costs of schools shipping materials to their district offices, unless shipping is required to correct a packaging error. Distribution of test
materials will not begin until final Department
approval has been received via an Approval to
Ship form.
For each shipment that contains more than
one pallet, DRC will provide hard copy pallet
maps that illustrate the identity and location
of boxes on each pallet. Pallets will be organized to match internal district delivery routes
if provided by districts; route code information
will be collected electronically by AIR during
the enrollment process and transferred to DRC
for use with their packaging and distribution
systems and procedures.
DRC works only with shipping vendors that
provide online tracing and tracking services.
For the Florida Standards Assessments, DRC
will use United Parcel Service (UPS) and
another leading national and global carrier,
King Solutions, Inc. (King), for the distribution
and collection of all assessment materials.
DRC has used both of these carriers for other
assessment programs with excellent results.
Both of these carrier methods will allow costeffective, traceable, and timely distribution of
materials.
Test materials will be delivered in at least two
waves per administration, with test administration manuals shipping in the first wave (TM1),
Page 7.4-5
and test booklets, answer documents, student

pre-ID labels and rosters, and other administration materials shipping in the second wave
(TM2). Additional shipments may be required,
specifically for the distribution of updated student pre-ID labels and rosters and additional
materials. AIR and DRC will work with the
Department to define the actual dates of each
administrations shipping waves. Section 7.1.2
contains a detailed schedule that outlines preliminary delivery dates. DRC will notify all
schools via e-mail regarding all shipments.
DRC will ship all test materials directly to district offices. Delivery of materials will be
scheduled during regular school hours, 9:00
a.m. to 3:00 p.m. Eastern Standard Time
(EST), and will be scheduled to account for
holidays and school breaks. Districts will be
able to contact DRC directly to request alternate shipping schedules; all shipping change
requests will be tracked internally to ensure
timely and accurate completion.
All shipments will be designated as inside
delivery required. Signatures of receipt will
provide proof of delivery and allow DRC and
districts to track all shipments via UPS
QuantumView and Kings online tracking
system. The Department will also have access
to these online tracking tools if desired.
District, school, and Department personnel
will have the complete support of DRCs courteous and professional Customer Service team
to assist with package tracking and the resolution of delivery issues. Please see Section
7.4.10, Customer Service for Paper-Based
Tests, for more information on DRCs
Customer Service function and process.
Upon request, AIR and DRC will provide the
Department with a proof of delivery report,
which will include a description of the materials, date and time of delivery, and the name of
the staff member accepting the delivery, as well
as information on any undeliverable materials.
Page 7.4-6
AIR and DRC understand that proof of 100%

delivery will be used for invoicing purposes.
AIR and DRC maintain our superior customer
service reputations by making sure orders for
additional materials, including additional
accommodated materials, are shipped promptly. Should a schools enrollment change
between the submission of its enrollment and
the time of testing, the assessment coordinator
can easily request these materials via AIRs
online management system, TIDE. Requests
for additional materials received after 3:00
p.m. in DRCs Central time zone will be
processed the following business day. DRC will
ship orders via secure, traceable overnight
service for early-morning delivery to ensure
materials arrive in time for testing. AIR will
maintain a complete additional materials
record and submit the report to the
Department for review if desired.
Distribution of Miscellaneous Materials
DRC will be responsible for the shipment of
any miscellaneous materials requested by the
Department or districts. DRC will ensure that
districts are informed of the carrier service,
and that the carrier service uses properly identified vehicles and delivery persons. All shipments will be designated as inside delivery,
and all large shipments will be unloaded onto
loading docks using carrier-provided equipment. DRC uses reliable carriers that are experienced with delivering assessment materials;
carrier personnel will handle material deliveries and collections in a professional manner
and will be directed to not leave the materials
unattended or unprotected from the elements.
Packing, Delivery, and Receipt of
Accommodations Materials
AIR and DRC understand that all printed special format accommodation materials will be
clearly labeled and packaged separately from
regular print test materials. All special format

materials will be delivered during the same
delivery window as for all other test materials
(TM2). DRCs receipt processes for accommodated materials require little document preparation by schools and districts. Unlike many
testing companies, DRC does not require special packaging, return, and processing procedures for accommodated materials. Special format materials will be handled with the same
quality and security procedures used with regular print testing materials.
Materials Collection, Receipt, and Secure
Check-In
AIR and DRC will be responsible for all costs
associated with the return of materials, including pre-paid return shipping from districts.
DRC will provide all return shipping labels,
prepaid postage labels, forms, instructions, and
other needed materials in a Materials Return
Kit shipped with testing materials in TM2.
Districts will be encouraged to re-use their
original shipment boxes for the return shipment, but DRC will send additional boxes to
districts who request them.
DRC will use UPS Return Service (UPS RS)
and King for the return shipment of Florida
Standards Assessments testing materials, using
the appropriate delivery method necessary to
meet reporting deadlines. UPS and King use
online tracking systems that provide the status
of each shipment from the time it is picked up
until it arrives at DRCs receiving facility. AIR
and DRC will work with the Department to
develop a materials collection schedule that
provides adequate time for districts and
schools to package materials, yet still enables
on-time scoring and reporting. AIR and DRC
understand that some districts will require
additional pickups after their specified pickup
dates, occurring for as many as 15 dis-
tricts/500 boxes for each spring administration

and 10 districts/200 boxes for each retake and
End-of-Course administration.
DRCs materials return process is simple and
straightforward. DRCs return procedures also
offer many advantages to district and school
test coordinators:
AIR and DRC will provide a clear, well-documented return process in the test administration manuals, increasing the accuracy
and decreasing the turnaround time of
return shipments.
DRCs customer service staff offers excellent support and has a history of providing
superior service to school districts around
the country. (Please see Section 7.4.10,
Customer Service for Paper-Based Tests, for
more information about this function.)
DRCs receipt processes require little document preparation by schools and districts.
Unlike many testing companies, DRC does
not require special packaging or return procedures for accommodated materials.
DRCs Ops MMS system enables 100% accuracy in accounting for returned barcoded
materials.
Upon receipt at DRC, all returned boxes will
be scanned in through its automated systems.
Received materials will be compared against
the shippers manifest and testing site counts
to identify any discrepancies, which will be
forwarded immediately to project management
for resolution. Please see Section 7.4.1,
Materials Processing, for more information
on DRCs materials processing systems and
procedures.
Materials Return Errors
During materials receipt, DRC will document
by school and district any errors made in
packaging or returning test materials. The
Page 7.4-7
error log will document any problems noted

during materials return/check-in, based on
Department requirements, and will include the
date, district name, description of error, steps
taken to resolve the issue, and other information as necessary. This error log will be provided to the Department within three weeks after
the scheduled receipt of materials from all districts. DRC will contact districts and the
Department to resolve any issues that cannot
be handled during receiving and processing.
AIR and DRC understand that these incorrectly packaged student response documents will
be processed and scored under extended timelines, and that individual reports will be generated and shipped to the districts.
Materials Processing
Team member DRC achieves accurate and efficient secure materials receipt processes that
require a minimal amount of document preparation on the part of schools and districts. The
companys proprietary materials management
system, Ops MMS, allows DRC to accomplish
this goal. Its advanced automation and barcode
scanners provide fast and accurate data collection with no dependence on materials/document order. This translates into time saved for
assessment coordinators during materials
return. Captured data are organized into userfriendly reports from the start of the secure
materials check-in process, providing valuable
insight into suspected material shortfalls as
early as possible, and mitigating potential consequences of delay. DRC processes up to
250,000 received materials per day (box
receipt, separating and sorting, and scanning
secure barcodes). In 2012, DRC processed
more than 13.1 million secure materials.
As evidenced by its ISO 9001:2008 certification, DRC maintains stringent quality-control
procedures during the document receipt
process. Log-in procedures, developed specifically for the FSA by DRC and approved by the
Page 7.4-8
Department, will provide clerical personnel

with step-by-step instructions, sorting rules,
priority/special processing procedures to be
followed during the log-in process for assessment materials.
Ops MMS provides efficient and accurate
control of all barcoded materials (including
scannable documents) that are shipped from
and returned to DRC for each test administration. The system allows DRC to track documents through all processing steps: Box
Receipt, Materials Receipt, and Quality Control
Rescan. The Box Receipt process is used to
check in all boxes coming back from districts
and schools. This flexible, automated process
helps reduce the processing time. By scanning
box labels, the Box Receipt process can be used
to flag and drive priority, late return, or special
handling processing. The Material Receipt
process uses project-specific processing and
sorting rules are systematically applied to
increase sorting accuracy and efficiency. The
process will begin by hand scanning the DRC
box return label on each box. Each box is then
opened and security codes, and precodes if
applicable, are hand scanned to link each document to the original box. The system, based on
project-specific business requirements, will
guide the user to place the scanned materials
in the correct staging location.
During the next phase, Quality Control
Rescan, using streamfeeder automation, materials will be matched against scan information
obtained during the Materials Receipt process.
These materials will then be staged for further
scanning and scoring processing (please see
Section 7.5.0, for more information on scanning and scoring). Non-scannable materials
also go through a similar scanning process to
verify scan counts and final labeling before
being sent to storage.
Given that materials processing occurs immediately upon receipt of assessment materials,
real-time feedback is provided on actual

receipts versus expected receipts for schools
and districts. In turn, the DRC will be able to
contact any district regarding what appears to
be an anticipated materials receipt shortfall
as soon as the materials for the entire district
are checked in. Secure materials issues can
thus be identified and resolved well in advance
of any reporting. OpsMMS also provides a flexible platform for which materials can be intercepted at the district, school, box, or individual booklet level if needed. DRCs processing
system offers a tremendous advantage to the
Department by providing quality control measures that are specifically related to potential
test security issues. Problems can be caught
early and resolved in a timely manner.
Exhibit 7.4.1-2 illustrates the overall workflow
from receipt of materials through all process-
ing of materials and the presentation of

scanned images for scoring.
Summary Report of Returned Materials
AIR and DRC recognize that assessment security is of the utmost importance to the
Department. DRC will provide the Department
with a report of all secure materials received
and checked in by DRC. AIR and DRC will
work with the Department to define the
requirements for this report including layout,
format, and reporting fields.
AIR and DRC acknowledge that if DRCs barcode verification system is demonstrated to be
unreliable, each document will be required to
be scanned twice, creating independent data
files that can be matched for scanning errors,
and that manual checks of answer documents
Exhibit 7.4.1-2: Scannable Document Processing Workflow
Hewlett
Packard
rd
Materials Receipt
Box Receipt
Image
Scanning
Quality Control Rescan
Document
Storage
Image and
Scan Quality
Control (QC)
Editing/
Editing QC/
Post-Editing
Image Scoring at
Performance
Assessment Sites
Image
Archive
Student Level
Processing
Page 7.4-9
will be required to ensure 100% accuracy of

secure document check-in.
7.4.2 Missing Materials Report and

Inventory
DRC has implemented several processes that
will help ensure the security and tracking of
secure materials. Because each secure material
is imprinted with its own unique test security
number, after booklet check-in for the entire
project is complete, missing materials reports
are accurately and readily produced. After all
returned secure materials have been processed
and scanned, including as many orphan and
late-returned documents as possible, DRC will
provide the Department with a preliminary
missing materials report for review. Following
the 10-day review and approval period, DRC
will produce missing materials reports for
affected schools. These reports will list missing
documents by type, title, barcode/security
numbers, grade, school, district, and other criteria required by the Department. DRC will
ensure that the missing materials reports do
not include materials that have been identified
as soiled/destroyed. School-level reports will be
delivered to district assessment coordinators
along with a district-level summary and memorandum; a state-level summary report will be
delivered to the Department. After district staff
have located missing materials or confirmed
that secure materials are not in the district,
DRC will update the missing materials inventory with this information and produce a final
Missing Materials Summary Report and submit
it to the Department by the date specified in
the Deliverables Schedule.
manuals, and other materials for a period of at

least six months. DRC will also inventory and
store 100 copies of each test and answer book
for each administration throughout the life of
the project. In addition, AIR will store digital
records of the CBT assessments for each
administration. AIR and DRC will store
throughout the life of the project any materials
that may be used for subsequent assessments.
Upon completion of processing, verification of
the individual barcode numbers of all secure
materials returned by the districts, and acceptance accurate results files by the Department,
scannable documents will be boxed for security purposes and final storage. After an initial
120-day period, the used student response
documents will be moved to an off-site secure
facility. The documents, as well as electronic
images and data, will be stored for two years.
The storage system will allow efficient and
easy retrieval of individual student tests within
a short timeframe. Unused and used materials
and electronic images and data will remain
secure until written authorization has been
received from the appropriate Department
contact to ship or securely destroy test booklets or images and data. AIR and DRC will submit a letter to the Department requesting permission to destroy specific materials. AIR and
DRC will submit certificates of destruction that
describe in writing the specific materials and
files that have been destroyed. AIR and DRC
understand that it may be necessary to retain
a small number of used documents for a longer
time period due to Department security investigations and that these documents are expected to be shipped to a new contractor at the
end of the contract.
7.4.3 Disposition and Storage of

Materials
7.4.4 Retrieval of Answer Sheets

and Test Books
After each test administration, DRC will inventory and store unused test and answer books,
AIR and DRC understand that we may periodically be asked to retrieve used test and answer
booklets from storage and individual student
Page 7.4-10
response files from the CBT system, up to

1,200 per grade, per administration, per mode.
These materials and files may be retrieved to
fulfill a variety of purposes, including
Department investigations or audit, manual
rescores, or contractor error resolution. AIR
and DRC will be responsible for all costs associated with the retrieval, review, and delivery
of these materials to the Department.
After a retrieval request is received, DRC staff
will initiate and monitor the retrieval process.
Processed test booklets/answer documents can
be retrieved quickly and efficiently because of
DRCs effective document storage procedures.
Additionally, AIRs CBT system and data file
storage procedures allow for on-demand
retrieval of specified images and files at the
student and school level. The following DRC
storage procedures will ensure the quick
retrieval of used test and answer documents:
Project-specific box labels will be created
containing the following information, as
applicable: unique customer and project
information, materials type, batch number,
pallet/box number, and the number of
boxes for a given batch.
Boxes will be stacked on project-specific
pallets. Each pallet will be labeled with a
list of all the batches it contains.
Before each pallet is stored, a quality check
will be performed to ensure accurate boxing and pallet content labeling.
After an initial 120-day short term storage
period at DRCs processing facilities, used
answer documents and used scorable test
booklets will be stored in an off-site secure
facility that is climate- and pest-controlled,
allowing for the preservation of the documents. The documents will be easily retrievable using the above storage and organization
procedures.
7.4.5 Innovative Product Design

AIRs graphic designers enjoy working with
clients to develop custom designs that provide
a common look and feel across a variety of
media. Sometimes our designs are based on
the natural environment of a state (e.g., we
incorporate graphics of colorful Hawaiian flowers on our website and in our publications for
Hawaii and a lighthouse in our products for
Delaware) or the name of the test (e.g., for the
Oregon OAKS test, we use a graphic of an oak
tree, and for the Utah SAGE tests, we use a
graphic that incorporates a picture of sage).
We work with our clients to develop a style
guide that specifies a range of preferred colors
and fonts that work well with the chosen
graphics. Our style guides also include guidance on formatting of publications (e.g., layouts for tables of contents, charts, graphics, etc.).
AIR can configure all of our online system
access points so they reflect the Departments
preferred look (skins) and color themes.
Additionally, any print materials will be
designed with the same themes so readers
instantly make the connection to Florida
assessments across mediums. We will make
sure that each piece of information is professionally designed, intuitively organized, and
consistently branded.
Our dedicated team of professional technical
writers, editors, and graphic designers will create and secure appropriate permissions and/or
copyrights for all art, graphics, charts, and
illustrations. Our designs will cover such
things as administration manuals, assessment
results reports, information and reference publications, user guides, and other materials. The
design process we employ includes furnishing
conceptual art, design development, and final
cover designs for the Departments review,
input, and final approval.
Page 7.4-11
AIRs ground-breaking work for our user interfaces includes four main objectives around
usability:
1. Backwards browser compatibility (as much
as 10 years ago)
2. Forward browser compatibility including
desktop and mobile support
3. Universal accessibility to people with or
without disabilities
4. Lightweight graphic files and elements for
optimized performance
When designing user interfaces, we allow for
flexibility to accommodate the use of adaptive
technologies. We strive for simple, consistent,
and intuitive design so that the interface is
easy to understand, regardless of the user's
experience, knowledge, language skills, or current concentration level. We consider users
using the slowest available Internet connection
and the lowest screen size possible (such as a
tablet). The systems for which we design need
to be served up quickly, so we trim extra
imagery and heavy graphics to serve up only
relevant site information to the user. Imagery
we use consists of a client ID/logo, a system
name, and universally bound icons to inform
the user and layout strategies to guide the user
through the site. To speed up page load time,
we eliminate extra calls to the server. We use
CSS3 and HTML5 standards for modern
browsers.
In addition, we take care to align the text on
our user interfaces with the graphic design.
Our technical writers always ensure that any
text displayed on our interfaces, as well as
related documentation (such as user guides),
is technically accurate, clear, and concise.
Using the latest technology, AIRs Publications
and Graphic Design Services staff produce a
wide range of materials targeting various audiences. Team members include senior creative
Page 7.4-12
directors, graphic designers, desktop publishers, editors, and copywriters. We have extensive experience developing and delivering
high-quality publications and presentations in
a variety of formats and languages.
From proofreading to copyediting to original
writing to graphic design, spanning production
for both print and broadcast media, translation, and Section 508 compliance, our team
members deliver products appropriate for both
technical and nontechnical audiences. We
have decades of experience in developing
award-winning, research-driven creative strategies, campaign materials, and publications.
Vitae of our design team members are included in Appendix A.
AIR takes as its mission the development of
innovative products and systems that will keep
our clients assessment programs at the forefront of the field. We look forward to partnering with the Department to implement a product design strategy for Florida that ensures
cohesiveness and a common look and feel to
all documents and websites and that synchronizes the flow of information.
7.4.6 Printed Products

Administration
The following section provides detailed information about printed products AIR will develop for test administration.
7.4.6.1 Stages of Development of

Printed ProductsAdministration
AIR has a proven record of developing a structured and effective process for producing
error-free documents, including, but not limited to, test booklets, answer documents, administration manuals, training materials, interpretive guides, user guides, and other ancillary
documents.
AIR will develop test administration materials

that
are fully customizable to include policies
and procedures unique to Florida;
conform to the Departments Test
Administration Manual workflow process;
are scheduled to ensure that there is minimal overlap with other test administration
material under development;
are designed in consultation with layout
and graphic design experts to create a look
and feel that reflects the newly created
Florida Standards Assessment (FSA); and
are reviewed and produced by accessibility
experts to ensure that they are fully 508compliant.
While AIR has extensive experience creating
administration scripts and instructions in support of its Test Delivery System, we recognize
that each state is different. Each state has its
own test administration requirements that
must be communicated to districts and
schools. AIR will work closely with the
Department to include its policies and proven
procedures with which districts, schools,
teachers, and especially students are familiar.
AIR understands and commits to the following
requirements of the ITN for printed administration products:
AIR will produce the first written draft and
all subsequent versions of all materials and
products developed for this project.
Each review period will last a minimum of
seven work days and will begin with the
posting of an electronic version of the document to a secure website by 9:00 a.m.
(EST) on the first day of review.
Development and review of administration
materials will be staggered to minimize the
number of materials under review at the

Department at any one time.
AIR will propose a detailed materials development schedule for Department approval
for managing the workflow and volume of
documents to be reviewed in each sevenday period in a format approved by the
Department.
The production of TAMs will follow the
review schedule of test books/answer documents or computer-based test forms by one
round.
AIR will deliver proofread documents following well-defined procedures to ensure
that all drafts are checked by computer for
spelling and proofread for accuracy by
qualified proofreaders.
Test Administration Manuals

AIR understands and commits to the following
requirements of the ITN:
Administration manuals will be fully customizable to address the Departments
requirements, including, but not limited to,
test security procedures; statutes and regulations; materials handling; test administration scripts; and best practices that have
been established by Department staff and
are familiar to district and school personnel
as well as students.
For each year of the contract ( summer and
fall administrations), AIR will produce in
print and post online the following TAMs:
One (1) English Language Arts/Literacy
(ELA/L) and Mathematics manual for
paper-based tests
One (1) ELA/L and Mathematics manual
for computer-based tests
One (1) ELA/L Retake manual (beginning
with the first retake administration in
2015)
Page 7.4-13
One (1) End-of-Course (EOC) manual
the Departments test administration

policies and procedures
The review sequence is consistent with AIRs

quality assurance (QA) process followed for
printed and electronic materials. AIR will follow the same general process for the development of all materials and adhere to the stages
described in Appendix G of the ITN.
the Departments test security policies

and procedures
We divide the development process into two

phases:
At a minimum, TAMs will address the following topics:
scripts for all assessed grade levels/

subjects
packing and returning materials
receipt and distribution of materials
Instructions for special-format tests will
include scripts necessary for accommodated testing (e.g., Braille, one-item-per-page,
and others) and will be provided as separate addenda to the manuals.
Computer-based manuals will include
scripts for the practice tests.
Test administration manual covers will be
developed separately from the TAMs and
will later be married to the final products.
As mentioned in Section 7.4.5 (Innovative
Product Design), AIR is prepared to support
the Department in its efforts to brand and
promote the FSA by providing personnel
experienced in the areas of visual/graphic
arts and document/software/web design.
TAMs will be packaged (without shrinkwrap) and delivered to districts for each
administration as a part the first test material shipment TM1, according to the specifications in Appendix E, Part 3 of the ITN.
All manuals will be available in a Section508 compliant electronic format within
seven days of Departmental approval of the
web-tagged files.
AIR will follow the specific review stages clearly described in Appendix G of the ITN.
Page 7.4-14
1. A drafting phase, during which we obtain

agreement on the content and design of the
document (this part of the process differs
somewhat for test booklets, which were discussed previously). Appendix G of the ITN
identifies the following two rounds that fall
under the drafting phase:
Department Manuscript Round/TAM
Information Sheet. At this stage, the
Department identifies the prior administrations TAMs to use as a starting point
for the current manual; identifies highlevel changes, additions, and omissions to
the scripts and other sections to be
included; and introduces known or
potential policy updates.
AIR Round 1. AIR uses the markup of the
TAM provided by the Department as well
as the TAM Information Sheet to draft
the manual. AIR updates the TAM with
known annual changes, including new
screenshots, and embeds queries to the
Department for its consideration and
input. While this document may be created using a design program, it is still considered a draft given the outstanding
questions and placeholder text/images
prevalent in this round.
2. A production phase, during which we produce the document to the specifications
developed in the first phase, accommodate
any final adjustments, and ensure that the
camera-ready proofs are error-free.
Appendix G of the ITN identifies seven
rounds that apply to the production phase.
The drafting phase will begin with conversations with the Department, during which we
will discuss issues of both design and content.
Following these discussions and an exchange
of materials, such as the Department manuscript, the TAM Information Sheet, and a
revised style guide, AIR will begin to develop
the draft. The specific form of the draft varies
according to the type of material.
Like the Department, we view the drafting
process as iterative. Especially with new materials, we understand that it is not possible to
make all final decisions prior to a first draft.
Manuals are produced over several weeks and
annual policy changes or clarifications may
not be finalized prior to the beginning of the
production process. Additionally, new reviewers enter the process as documents approach
completion, interactions among design or content elements can sometimes be unexpected,
and sometimes a seemingly good idea at an
early stage simply does not work. Therefore,
we build in time for multiple drafts of each
product prior to its entering the production
process. During the drafting process, the
Department will receive formatted documents,
typically produced in MS Word or a similar
program. These drafts, however, will not reflect
the final formatting. The first blackline version
of the document (Round 1), which results
from the first iteration through our production
process, is designed to reflect the final layout
and formatting. This, too, is an iterative
process, and we work collaboratively with our
clients to ensure an error-free product.
Each documents content is fully drafted
before it enters the production system. The
Department must be assured of error-free production of materials. This is an area where AIR
has had considerable success, though we strive
for continuous improvement. Our production
department is headed by Kara Prezocki, who
brings over a decade of experience in publications management. The staff of the production
department consists of senior editors, editors,
proofreaders, graphic designers, and layout

specialists. Every public assessment document
leaving AIR goes through this department.
Our production processes include multiple
editorial reviews, and each document must
receive senior staff sign-off. For test materials,
this will require sign-off by Sandra Durden, our
proposed test development lead; Kevin Murphy,
our proposed senior program manager or
Meredith Durgin, our proposed project manager for Test Administration.
Exhibit 7.4.6.1-1 provides an overview of our
editorial review process. (NB: Change references to Blackline to Round/Blackline)
When the senior staff member overseeing the
content is satisfied with the content, the document enters the production system; its first
stop is an editorial review.
Upon completion of the editorial review, the
document is formatted in a layout program,
such as Adobe InDesign or QuarkXPress, or
occasionally in other programs requested by
our clients. The document is iteratively proofread and revised until the production department certifies the document to be accurate.
This step is followed by a senior review before
the document is delivered to the client for
review. In all cases, we will schedule at least
seven days for the Departments review.
Camera-ready drafts are called blackline
proofs. While up to four rounds are described
in Appendix G of the ITN, each blackline is
thoroughly proofread and reviewed for content
accuracy. Many development steps described
in the ITN that take place in later rounds
should also take place in earlier rounds (e.g.,
verifying instructions, URLs, key dates) so that
the best document possible is delivered at
each stage. Doing so will lead to higher quality
material that may be able to be approved
ahead of schedule.
Page 7.4-15
Exhibit 7.4.6.1-1: Editorial Review and Revision Process

Client
Review
Draft
Document
Initial Layout
Enter Revision
Process
Senior
Review
YES
No
Change
NO
Blackline 1
Editorial
Review
YES
OK
NO
Revision Process Detail

Layout
NO
Proofreading
and Editorial
YES
OK
The Departments final round (i.e., final blackline) must have no changes, and it serves as
the master document against which the blueline (printer proof) is evaluated. Each blackline after blackline 1 and the blueline are automatically compared with the immediately preceding version using a PDF-comparison tool
that highlights all changes. This step has
proved useful for identifying unintended
changes made during the revision process.
Such changes are difficult for humans to
detect because they can appear anywhere in a
document and may be subtle. The PDF-comparison tool highlights these changes so that
each difference between versions can be
mapped to an intended revision.
Page 7.4-16
As required, all manuals

will be available in a
Section 508-compliant
electronic format within
Senior
seven days of
Review
Departmental approval of
the web-tagged files. AIR
has extensive experience
No
in providing assessment
Change
material in Section 508YES
compliant format.
New
Specifically, for the
Blackline
Minnesota State
Assessment, all test
administration manuals
(TAMs), user guides, technical reports, and several
ancillary materials were
converted to 508-compliant format. AIRs experience is documented in
this section. AIR strives to adhere to the
Section-508 principles from the very first iteration of a document, when possible, and to
reduce the need for retrofitting the document
and encountering unforeseen challenges downstream.
Initial Revision
Process
Senior
Review
NO
Blueline
Proof
All materials delivered will

go through this process,
ensuring that the
Department will receive
error-free materials for
review and that any
changes requested by the
Department are implemented promptly and
accurately.
Section-508 accessibility requirements were

enacted in 1998 as an amendment to the
Rehabilitation Act of 1973. The regulations
seek to ensure that people who have disabilities have equal access to all information provided by the federal government. Compliance
with Section-508 web-accessibility standards
became a requirement for all new and
redesigned federally owned or funded websites

and materials posted or delivered through
these websites launched on or after June 21,
2001. AIR is fully compliant with the provisions of Section 508 of the Rehabilitation Act,
which require AIR to utilize Electronic and
Information Technology (EIT) that is accessible to people with disabilities.
AIR is proud of its reputation for delivering
high-quality, complete, and accurate Section
508-compliant documents to clients. AIR has
developed and refined processes for (a) creating Section 508-compliant websites; applications; e-mails; videos; multimedia; PDF files;
and Microsoft Word, Excel, and PowerPoint
files; (b) testing and validating our work; and
(c) performing QA checks. As a result, AIR is
the go to contractor for creating Section
508-compliant materials and websites for several federal agenciessuch as the Centers for
Disease Control and Prevention (CDC),
National Library of Medicine (NLM), and the
U.S. Department of Education (ED). In several
instances, we have remediated the accessibility work of other contractors, and we have
trained federal clients how to create Section508 accessible products.
AIR understands and has been through the
website for the Section-508 clearance process
for the U.S. Department of Human Services,
Office of the Secretary, Assistant Secretary of
Public Affairs (HHS/OS/ASPA), and many other
government agencies. AIR is in a position to
continue to apply our knowledge and experience with Section-508 requirements for web
applications and documents for the
Department.
Furthermore, AIRs work in usability testing
continues a tradition of highly trained specialists applying their knowledge and skills to solving practical problems for clients. Our usability engineering staff consists of usability engineering professionals with strong training and
experience in user interface design. We have

conducted usability testing on numerous websites for our National Institutes for Health
(NIH) clients, including a usability evaluation
and redesign for the National Library of
Medicine (NLM), usability and redesign of the
National Heart, Lung, and Blood Institute
(NHLBI) website, usability studies on Spanishlanguage websites, and a usability study (with
blind users) for the NHLBI online live chat
application.
7.4.7 Test Administration Ancillary

Materials
The work will be completed by either full-time
AIR or DRC staff under the direction of our
project director, Kevin Murphy. The staff
includes:
Dee Wagner, AIRs corporate director for
operations, will oversee the procedures to
create all test administration. Ms. Wagner
has more than 22 years of experience managing large, high-stakes assessment projects, including the past 13 years at AIR.
Ms. Wagner has overseen test administration, security, and assessment operations,
which include activities related to producing, delivering, receiving, scanning, and
scoring test materials. A Six Sigma Black
Belt and ISO-certified lead assessor, Ms.
Wagner received hold both an M.B.A. and a
bachelors degree in operations management.
Meredith Durgin, project manager, Test
Administration, will lead the development
of ancillaries for test administration, such
as training materials and manuals. Ms.
Durgin has 10 years of management experience in educational assessment, including
managing multimillion-dollar projects. With
AIR, Ms. Durgin has served as a senior
manager for Oregons very successful online
testing project and has led smaller pro-
Page 7.4-17
grams in Florida, Ohio, New Jersey, New

Mexico, Texas, Missouri, Arkansas,
Montana, and Utah.
Peter Tressel, project manager, Paper-andPencil Operations, will lead the development of ancillaries related to handling
paper, which includes kits, forms, and all of
the various labels required for the paper
program. Mr. Tressel has nearly 15 years of
project management experience directly
related to large-scale, statewide assessment
programs, including overall management
and direction for assessment programs in
Minnesota and Ohio. In addition to his
work in the testing industry, Mr. Tressel has
more than 20 years of related experience
with a variety of writing and editing projects and activities.
These lead staff will be supported by operations, editorial, graphic design, and production
staff. Complete information on our staffing
plan is presented in Section 7.7.2.
7.4.7.1 Training Resources for

School and District Technology
Coordinators (CBT)
In this section we discuss proposed materials
and tools specifically for school and district
technology coordinators to support computerbased testing (CBT) administrations. For a
detailed discussion of AIRs proposed training
plan where we outline and list training topics
and timelines for Department and district personnel, please see Section 7.4.9.8.
AIR recognizes the importance of providing
training materials for school and district technology coordinators to familiarize them with
their roles and the online testing system. We
will provide a webinar and PowerPoint presentation for school and district technology coordinators on preparing Florida schools computer labs. This training includes information for
school and district network administrators and
Page 7.4-18
provides guidelines for varied technical setup

and technical support available at the schools.
Technical setups include multiple secure
browser installation methods, such as manual
installation on individual machines, installation on machines through a network, access
from a shared network drive, and thin-client
setup. Minimum hardware requirements will
also be reviewed in detail.
AIR will provide a webinar and PowerPoint
presentation to train school and district technology coordinators on how to use the Test
Information Distribution Engine (TIDE), which
is discussed in greater length in Section 7.4.8.
More specifically, technology coordinators will
learn how to add users at the district and
school levels so they can access appropriate
online systems. The training also covers other
TIDE functionality, including updating student
test settings and setting test restrictions.
Our detailed user guide on system requirements provides information on the hardware
and software needed for the online testing system. More specifically, the user guide discusses
minimum hardware requirements for desktop
computers, supported operating systems and
web browsers, requirements for appropriate
monitor display, and help desk contact information. AIR is committed to having a small
technology footprint at the schools and districts; thus, part of this footprint is to be backwards compatible as well as forward looking.
AIR realizes that inasmuch as schools and districts will have old systems, they will have new
systems too. Thus, AIRs system is able to support a range of operating systems to support
online testing.
AIRs secure browser, described in detail in
Sections 7.4.8.3 and 7.4.9, is a special build of
the Mozilla (Firefox) browser that is modified
to protect the security of tests and support
certain accommodations. Secure testing apps
are available for mobile devices. In order for
students to take tests through the Student

Interface of the online testing system, the
secure browser or app must be installed.
The secure browser installation instructions
include information for downloading and
installing the secure browsers on all computers
that will be used for student testing.
Downloading and installation instructions are
also provided for different operating systems.
All secure browsers can be downloaded from
the Department information portal.
Our system has a built-in diagnostic tool that
reports accurately the number of students who
can be successfully tested. The diagnostic tool
allows technology coordinators to run network
performance tests to assess readiness for
online testing. The diagnostic tool will check
the schools network performance and text-tospeech settings, as well as verify the operating
system. Online practice tests will also be made
available, which use the same applications as
the operational test site. This design ensures
that all users become familiar with the online
testing system before operational testing
begins. User guides and systems documentation will be made available when the online
practice tests open. The online practice tests
will be available throughout each test window.
Technology coordinators will be provided with
a user roles chart, which indicates, at a glance,
the access each user role is permitted in each
online system (please see the sample user role
chart in Appendix D). System access for each
user role will be reviewed in each webinar
(discussed in Section 7.4.9.8) as appropriate.
The user role chart will be posted on the portal; will be available in manuals; and will form
part of the online, self-guided tutorials as
determined in collaboration with the
Department.
For additional information, technology coordinators can access FAQs on the Department
information portal. Examples of questions that

are answered on the FAQ page include:
What do schools need to do technologywise to get ready for Floridas online
assessments?
How many students can be tested at one
time?
How often is the secure browser updated?
Answers to these questions and many others
will be provided to Department personnel on
the FAQ page.
Exhibit 7.4.7.1-1 displays an overview of the
materials AIR proposes to provide for school
and district technology coordinators.
Exhibit 7.4.7.1-1: Proposed Training Materials and Tools
for School and District Technology Coordinators
Proposed Training
Material Topics
Proposed Training
Material Format
Preparing Your Schools Computer Labs:

Learn about guidelines for technical setup
and available technical support.
1. Webinar
2. PowerPoint
How to Use TIDERegistering Users and

Modifying Student Test Settings: Learn
how to register users in TIDE, update student
test settings, and set restrictions.
1. Webinar
2. PowerPoint
System Requirements Document: Learn

about minimum hardware and software
requirements for the online testing system.
1. User Guide
Secure Browser Installation Instructions:

Learn how to download and install the secure
browser on all computers that will be used for
student testing.
1. User Guide
Checking Network Performance and

Becoming Familiar with the Online
Testing System: Learn how to assess
readiness for online testing.
1. Diagnostic Tool
2. Online Practice Tests
3. User Guide
User Roles: Learn about the access each user

role is permitted in each online system.
1. User Roles Chart

2. Component of the
Webinar
FAQs: Learn answers to frequently asked

questions from technology coordinators about
the online testing system.
1. FAQ Page on the

Department
Information Portal
Page 7.4-19
We assume that each webinar will be presented four times, with up to 100 attendees at
each webinar. We propose to record the webinar and replay the recorded webinar during
subsequent presentations. Each webinar will
include a live question-and-answer session at
the end. AIR will provide recordings of the
webinars to the Department and will host
them on the portal at the Departments
request.
uted with test materials during TM2 to districts after the PreID window closes. PreID
labels will be sorted and packaged according to
the order specified by each district, with the
inclusion of page breaks specified by each district and/or the Department. Supplemental
PreID labels and rosters will be produced and
distributed to districts that update their PreID
files during subsequent PreID windows established by the Department.
7.4.7.2 Instructions and Materials

for Braille Tests
AIR and DRC are proposing to address student

mobility by recommending an on-demand student label printing process. This process will
accommodate new students, permit student
updates and corrections, and eliminate the
need to hand grid student information on student response documents, thereby increasing
the accuracy of student information. To help
ensure print quality and scanning compatibility, DRC will provide blank student PreID label
stock to all districts. AIR and DRC will work
with the Department to determine blank label
stock quantities that will be provided to each
district. District and school personnel will be
directed to use laser printers to ensure printing and scanning quality.
DRC and APH will produce Braille-version test

administrator notes and scripts to supplement
the test administration manuals for each subject and grade. If manipulatives are needed for
Braille test items, DRC and APH will also produce these materials; DRC will ensure these
items are distributed with the test materials to
the appropriate districts. The production of
these materials will follow the following the
same production and Department review
processes described in Section 7.4.6. The
Braille notes and scripts will be distributed
with the secure test materials. Information on
APH and packaging and distributing special
format tests is provided in Section 7.3.11.
7.4.7.3 Student PreID Labels

Using PreID files provided by AIR, DRC will
produce and distribute adhesive PreID labels
(3 x 3 ) for each student that will participate in PBT assessments. Local school staff
will affix the labels to student response documents. Please see Section 7.5.2 for more information on AIRs PreID process and Section
7.4.1 for information on DRCs procedures for
producing and distributing PreID labels.
AIR and DRC will provide the Department
with production samples of student PreID
labels for a quality review. Student PreID
labels and rosters will be packaged and distribPage 7.4-20
7.4.7.4 Student PreID Rosters

Using the same PreID files provided by AIR for
producing PreID labels, DRC will produce and
distribute PreID rosters. The information contained on the student rosters will correspond
with student label information so that districts
and schools can easily use the rosters to verify
the accuracy of student label information, with
one row of information provided per student.
AIR and DRC will work with the Department
to determine roster format and layout, with
the understanding that 8 1/2 x 11 paper size
is preferred, with the possibility of increasing
the size to 8 1/2 x 14 for readability.
AIR and DRC will provide the Department
with production samples of PreID rosters for a
quality review. Hardcopies of rosters will be

packaged and distributed with the PreID labels
and test materials during TM2 to districts and
via AIRs online management system, TIDE,
after the PreID window closes. Supplemental
PreID labels and rosters will be produced and
distributed to districts that update their PreID
files during subsequent PreID windows established by the Department.
Please see Section 7.5.2 for more information
on AIRs PreID process and Section 7.4.1 for
information on DRCs procedures for producing and distributing PreID labels and rosters.
7.4.7.5 Document Count Form

(DCF)
AIR and DRC acknowledges the requirement
for a document count form (DCF) (i.e., header
sheets) as outlined in the ITN. However, we
would like to propose an alternate process that
would not require header DCFs and takes
advantage of the efficiencies in DRCs proprietary Ops MMS, while also saving district personnel considerable time. This process is
based on the information contained in the student PreID labels and the ability to link the
labels to school and district.
To provide additional assurance to district and
school personnel that their student response
documents have been received and processed,
DRC will also provide weekly answer document status reports for schools and districts to
verify receipt of student response documents.
The status reports will be available to assist
district assessment coordinators with tracking
returned student response documents and will
be used to identify any discrepancies that
might exist between the district records of
what was returned to DRC and DRCs records
of what was received. Student response documents can be tracked through TIDE by student
name after scanning at DRC. DRC will provide
weekly updates in TIDE throughout the processing and scanning phase.

Based on experience, DRC finds that
DCFs/header sheets can often be inaccurate
sources of important group identifying information. DRC has implemented this alternative
process with many of its state assessment
clients with great success. These clients have
never returned to using DCFs or headers after
the switch to this alternative process. In fact,
DRC finds that testing coordinators are thrilled
with the procedural change as it makes the
process for returning materials much easier
and more efficient. For a full description of
DRCs materials handling process using Ops
MMS, please see Section 7.4.1.
If upon further discussion of our approach following contract award, the Department would
still like to use DCFs for processing, AIR and
DRC are prepared to provide the Department
with the related costs.
7.4.7.6 Administration
Record/Security Checklist
DRC will provide Administration
Records/Security Checklists for schools/districts pre-printed with security barcode numbers for all secure materials, listed in ascending order by material type. The forms will provide space for districts to document materials
assignment and receipt after testing. It will be
the district assessment coordinators responsibility to disseminate the test security information as appropriate and collect completed
forms after testing. The purpose of these forms
will be to ensure compliance to the established
test security procedures. AIR and DRC will
work with the Department to determine the
format of these forms but understand that they
must include box numbers and section breaks
between packages. Forms will also be provided
electronically in an editable format to districts
via AIRs TIDE system.
Page 7.4-21
7.4.7.7 Comment Forms

AIR will work with the Department to gather
feedback on the program from key participants. These data can serve in both a formative role, as we strive to continuously improve
the FSA, and an evaluative role, to inform
judgments about how the program is working
in the field.
Gathering Feedback from All User

Groups
We will largely collect this feedback by conducting surveys of all user groups who will participate in preparing, administering, and taking
the Florida standards assessments:
Test Administrators
We propose to use a survey tool called Key

Survey. It allows us not only to customize the
look and feel and the survey content, but to
also dynamically integrate with AIRs other
systems. Specifically, Key Survey has these
main characteristics:
Customizable look and feelwhich will
match the Florida look and feel chosen
for the Assessment Program
Skip logicwhich lets respondents skip
non-applicable questions
Pipingwhich allows us to pull answers
from one part of a survey into another
Preview and testing capabilitieswhich
allow us to preview and test the survey
before publishing it
School Coordinators
District Coordinators
Technology Coordinators
Students
AIR will create these survey forms in partnership with the Department. They will be customized to type of user as categorized above. A
sample set of questions for technology coordinators is shown in Exhibit 7.4.7.7-1.
In addition to some easily aggregatable closedended questions, each feedback form will
include open-ended questions that will invite
comments from participants.
Particularly for students (who make up the
largest user base), we can encourage respondent feedback by creating survey questions
that are appropriate to their reading age and
length. Thus, we can customize the surveys by
having a different one for elementary (grades
35), middle school (grades 68), and high
school (grades 912) levels. A sample set of
questions for high school students is shown in
Exhibit 7.4.7.7-2.
Page 7.4-22
Building Fully Customizable and

Dynamic Surveys
Question, image, and media file

librarieswhich allow us to access particular images or media files (perhaps for
younger students) to further customize
content
Integrating with TIDE and RTS
Key Survey also has an API service that allows
AIR to interact directly with their platform.
This will allow us to integrate our user management and enrollment database directly with
their survey tool. These include the following:
Having one common source of all user
and student enrollment data, and dynamically getting the latest data from AIRs
Roster Tracking System (RTS)
Using demographic data identical to the
data already in the RTS, such as gender,
date of birth, or ethnicity
Transparently creating and managing
users, including their roles and permissions
Exhibit 7.4.7.7-1: Sample Set of Questions for Technology Coordinators his or her responses when the connec-
tion is restored).
Questions for Technology Coordinators

How did you download and install AIRs Secure Browser on the student machines?
I installed it on each individual computer.
I pushed the browser through the network.
How easy was setting up AIRs Secure Browser?
Very easy
Easy
Difficult
Very difficult
We do not intend to collect any sensitive information on the comment

forms. However, we note that Key
Survey also has state-of-the-art security to ensure data is protected by taking the following precautions:
It hosts data in SAS 70 Type II
certified data centers that provide the
highest level of physical security combined with failover and redundancy
techniques.
Exhibit 7.4.7.7-2: Sample Questions for High School Students

Questions for High School Sstudents
Data reside securely on database

servers behind multiple firewalls.
How often did you use a computer for school learning, for example, working on a
homework assignment or researching a topic?
Every day
1 or 2 times a week
1 or 2 times a month
Not often
Confidential data are secured

through an encrypted connection.
How did you feel about testing on a computer or mobile device?

Very satisfied
Satisfied
Not satisfied
Completely not satisfied
I have done my best quality work on this test.
Yes
No
Possibly creating new tasks based on survey responses (for example, a student
who wrote certain inappropriate words
may trigger an alert in our system)
Fully Accessible and Available
All forms created in Key Survey will be accessible to responders from anywhere and anytime, whether the user uses a desktop, mobile
phone, paper, or is even offline (the tool allows
a user to complete a survey offline and upload
Key Survey is an official licensee

of the distinguished TRUSTe Privacy
Seal Program and certified by the EU
Safe Harbor Privacy Program.
Compiling and Reviewing the Data
The Key Survey tool enables AIR or
authorized users to export the data
into CSV, Excel, or PDF format, or
even directly to a database. AIR can
provide the Department with a complete raw data file (in any of the formats above) containing all responses.
After periodic survey collections (agreed on by

the Department), AIR will produce statistical
reports by user group.
7.4.7.8 Return Shipping Labels

DRC will provide and distribute preprinted,
adhesive, color-coded labels to districts for use
in returning materials.
Page 7.4-23
7.4.7.9 Materials Return Kit

District
DRC will provide and distribute materials
return kits (envelopes) to districts. These kits
will contain return shipping labels, bills of lading, return instructions memorandum, and
Braille notes and scripts (if applicable).
7.4.7.10 Materials Return Kit

School
DRC will provide and distribute materials
return kits (envelopes) to districts for distribution to schools. These kits will contain document count forms (if the Department chooses
to use this method instead of AIR/DRCs proposed alternative), paper bands, and colorcoded return shipping labels.
7.4.7.11 Miscellaneous Return

Materials
DRC will provide and distribute all materials
necessary for packaging and returning all test
materials to DRC for processing and scoring.
Unlike many testing companies, DRC does not
require special packaging, return, and processing procedures for accommodated materials.
Special format materials will be handled with
the same quality and security procedures used
with regular print testing materials. Districts
will be encouraged to re-use their original
shipment boxes for the return shipment, but
DRC will send additional boxes to districts
who request them.
7.4.7.12 Paper Bands

DRC will produce adhesive printed paper
bands (4 x 24) that will be used by schools
and districts to bundle scorable materials.
DRC will ensure that the bands are constructed so that adhesive does not stick to and damage the enclosed documents. AIR and DRC will
work with the Department to define the paperPage 7.4-24
band specifications but understand, at a minimum, the bands will be printed on one side in
order to identify the type of answer document
contained in the bundle and that bands must
be 8-point Carolina paper (or approved equivalent). The paper bands will also be compatible
with multiple pen-and-pencil types so that
required information can be written on the
bands without smearing or smudging.
7.4.7.13
Rulers/Manipulatives/Calculators
Mathematics items developed for SAGE that
require length measurement already have
rulers embedded in the graphics. Standards
3.MD.4 and 4.MD.4, in particular, will include
items in which students must measure items
to the nearest fractional inch. Because the
rulers are embedded in the graphics, teachers
do not need to distribute a supply of rulers to
their students. Paper/pencil forms for these
items will include the rulers embedded in the
graphics to allow students to measure objects
in inches. Furthermore, items developed for
SAGE that require students to measure angles
already have protractors embedded in the
graphics. Standard 4.MD.6 specifically requires
students to measure angles using protractors.
Paper/pencil forms for these items will also
include the protractors embedded in the
graphics to allow students to measure angles to
the nearest degree.
Mathematics items developed for SAGE do not
require the use of any special manipulatives by
students. Instead, graphics have been included
in items where appropriate to the language of
the standard. For example, Standard 6.G.4
requires students to use nets for problem solving related to three-dimensional shapes. These
items will include the net as a graphic in the
stem, or they will require students to construct
(or identify) the net as their response.
The SAGE assessments do not allow calculators for grades 3, 4, and 5. The Common Core
State Standards in these formative grades
depend upon students developing their numeracy and procedural fluency; thus, calculators
are not permitted in these grades. In grade 6, a
basic four-function calculator is allowed for
one segment of the test, as identified by the
Content Standards of the blueprint. In grades
7 and 8, a basic scientific calculator is allowed
for the entire test. In algebra I, geometry, and
algebra II, a scientific calculator with optional
graphing and/or regression features may be
permitted.
For paper/pencil forms, students may bring in
approved calculators based upon the calculators for the equivalent online versions.
[Reference http://www.fldoe.org/ese/pdf/
CBTAccomm.pdf.] The reader is directed to
Section 7.3.0, Tools Available in the Testing
Environment, for further detail on the functionalities provided by AIRs calculators for
online tests.
Using print-ready files provided by AIR, DRC
will produce all required manipulatives, such
as rulers and protractors. DRC will also provide large print and Braille versions of the
manipulatives as necessary.
7.4.7.14 Miscellaneous Memos,

Forms, Labels, and Other
Products
DRC will produce and distribute to districts all
necessary ancillary materials needed to support successful PBT administration, including
miscellaneous letters and memoranda; packaging and shipment cover memoranda; packing
lists; pallet maps; and other miscellaneous
memoranda, labels, forms. AIR will provide
these materials, as appropriate to districts in
electronic format. AIR and DRC will work with
the Department to define the content and for-
mat of these supplemental administration

materials.
7.4.8 Online Management System

AIRs Test Information Distribution Engine
(TIDE) provides an integrated system for
gathering and managing student enrollment
and preID labels;
gathering and adjusting data about students, including student relationships (to
their districts, schools, teachers, classes
and/or programs), demographics, accommodations, or needs; and
ordering materials.
This centralized assessment management system supports both online and paper tests. This
year, AIR and DRC are introducing a new service: real-time ID. Florida schools and districts
will be able to register students only one or
two days in advance of testing for both online
and paper testing to accommodate newly
entering students. For paper tests, schools will
be able to print an individualized, barcoded,
student-specific preID label instead of bubbling information on a form. The scannable
labels can be printed using any laser printer
on label stock to be delivered with the preIDd
answer booklets. This reduces work in the
schools and districts, increases security, and
improves data integrity. Real-time ID speeds
reporting by eliminating most of the post-testing cleanup that occurs when schools must
hand-bubble information for late-arriving
students.
Real-time ID will supplement preID, not
replace it. Real-time ID will only be used for
students without preID books. preID or
enrollment uploads may come directly from
the Department or from individual districts.
This would be at the Departments discretion.
If the Department is able to collect near-real-
Page 7.4-25
time data from districts, central delivery will

help ensure the integrity of their statewide student information system and drive data
improvement. If the mechanisms are not in
place, we can accept uploads directly from
districts.
TIDE, as well as all other AIR systems, will
enforce strict authentication policies that meet
the Departments requirements. TIDE adds,
edits, and deletes users and sets them with different authorization based on roles. Since role
creation is configurable, AIR will work with
the Department to define a sufficient number
of differentiated roles to support workflows at
the schools and districts throughout the state.
Authorized users can upload or enter
user logins, passwords, and other (configurable) attributes;
student information, including demographics, required accommodations for each test,
and student relationships to classes,
schools, and other entities in the system;
and
roster information defining classes, schools,
and other relationships.
TIDE also offers an interface to input information about students or other users directly.
Users have the option of dropping preID files
into a secure AIR directory that is automatically monitored. The system periodically scans
the directory for new files. The system scans
the file to evaluate its format. Errors in the
input file are reported back via a transaction
receipt left in the watched directory.
TIDE allows school, class, and other information to be uploaded by the state, districts, or
schools. This relationship information defines,
updates, or deletes information about how the
various entities in the system (schools, classes,
teachers, students) are related to one another.
The system enables users to create multiple
rosters for each student. The reporting system
Page 7.4-26
reflects the rostering in TIDE and can report

on classes or other rosters defined there.
With our system the tedious creation of
testing sessions is a thing of the past. Test
Administrators (TAs) create a testing session
at testing time with as little as two clicks of a
button. By logging in with a reference to the
session number, students effectively request
admittance into the session. Upon admittance
(which the TA does from his or her workstation), the student is automatically added to the
session roster.
When students log on to a session, they are
offered a choice of test to take. Each student
sees an individualized choice of tests
for which he or she is eligible;
for which the student has not exhausted
his or her opportunities; and
that the Test Administrator has chosen to
offer in the session.
Data collected in TIDE are also used to determine student test eligibility for online and
paper tests and serve as the source of data for
preID labels.
Throughout the testing window, demographic
and other data can be reviewed and corrected
in TIDE, speeding the process of data cleanup,
shaving weeks off the time needed for reporting.
Assessment coordinators can order materials
through TIDE. Like our other systems, TIDE is
highly configurable. For example, TIDE can be
configured to
create default orders based on historical
orders;
flag unusual orders for state review;
provide default overages at the school and
district level; and
allow ordering by school personnel, district

personnel, or both.
TIDE supports multiple ordering windows and
allows tracking of materials after they are
shipped. The system also provides a conduit
for distributing online documents, forms, and
other ancillary materials.
7.4.8.1 Test Staff Contact

Information
TIDE maintains district, school, and staff information by storing user information, which is
configurable, and can be easily searched, filtered, or exported. TIDE stores the users
school, district, and state association, and the
users role based on his or her security authorization level. This information can be exported
and printed.
TIDE also allows easy editing of existing user
information or adding of new users. The system stores information on when a user was
added. Start dates on when a user was added
can be shown on the UI needed. Additionally,
we can tell when a user logged in to the common login system, which allows a user to log in
to any one of the systems (excluding ITS) and
access any other system for which he or she is
authorized. AIR will work with the Department
to define a sufficient number of differentiated
roles to support workflows at the state and in
schools and districts throughout the state.
7.4.8.2 Test Staff Passwords

Login information generated during user addition is communicated securely to the individual users. Passwords are not e-mailed to users.
Instead, upon addition to TIDE, users are sent
a secure link, one that expires after a configurable time, to a site where they set their password. Passwords in the TIDE databases are
encrypted.
7.4.8.3 Test Session Requirements

Creating and Managing Test Sessions
AIR recognizes that the testing process is managed by educators and that every minute spent
on administrative tasks is a minute taken from
educating students. Therefore, we have
designed a testing process that minimizes the
administrative and technical burden on school
personnel. We do this through
implicit registration; and
automatic session construction.
Implicit Registration
Implicit registration means that students do
not need to be explicitly registered for a testing session. We have eliminated the clerical
task of assigning students to sessions. Instead,
at the time of testing, the TA starts a testing
session with as few as two mouse clicks. The
system assigns a session ID, and student login
is simply a request to enter a session. Students
log in with identifying information and the session ID. From his or her workstation, the TA
authenticates the students and admits them to
the session. In this way, the roster of students
to be tested builds itself as students log in.
Nobody has to sit at a computer creating these
rosters.
Implicit registration is supported by our data
exchange process, which is detailed in Section
7.5.2. Through completely automated systems,
student information systems exchange information with our systems, so that each student
is known to the system. Student eligibility to
take a test is determined implicitly through a
series of configurable rules. For example, a
simple eligibility rule might state that students
who (a) are in grade 5 and (b) have not taken
a grade 5 math test are eligible to test.
Page 7.4-27
Automatic Session Construction and

Session Management Tools
Test Administrators use the TA Interface to
create and manage test sessions. The interface
allows authorized TAs to administer test sessions, monitor activity, and respond to testrelated issues in one convenient location. The
secure interface helps ensure that the right
student is taking the right test and lets the
administrator focus on test administration. As
with paper tests, the TAs primary role with
online tests is to ensure the security of the test
and make sure that students have a quiet environment, free of distraction, in which to take
the test. The security of the test is managed
through a simple interaction between the TA
and the students, as summarized in Exhibit
7.4.8.3-1. Students can test only when the TA
is logged on and has a session active. After a
student pauses a test, the TA must re-admit
that student to the session.
Creating a Test Session

Test Administrators create a session simply by
selecting the tests that will be available in the
session. When students join the session, they
can select from among the tests that (1) are
offered in the session and (2) they are eligible
to take.
Students use the session ID to log in to a test
session. Session IDs are unique to each test
session. When a student enters the session ID,
the system links the student requesting access
to the test session to the TA, who then
approves or denies the students access
request. When students join a session by logging in and entering the session ID, the TA
verifies their identity and admits them to the
session.
Once admitted to the session, the student is
given a final chance to confirm that he or she
Page 7.4-28
has requested the correct test. Upon confirmation, the test begins.
Reviewing and Approving Test Settings

During the authentication process, TAs have
an opportunity to review or adjust the test
settings. Test settings include any embedded
supports or accommodations to which the student may require access. These can, and
should, be set in advance in TIDE; however,
we recognize that schools do not always
accomplish this in advance. Hence we make it
possible for TAs to assign some or all accommodations as well (at the discretion of the
Department).
The availability of access features, embedded
supports, and accommodations is completely
configurable. The Department will be able to
determine whether TAs can adjust settings at
the beginning of the session or whether access
to specific features requires higher level
authorization. For example, one state provides
TAs with broad authority to adjust most features, but only authorized state users can
authorize the read-aloud of reading passages.
Exhibit 7.4.8.3-2 shows our online interface
through which TAs review or adjust individual
student accommodations before approving a
student for a test.
Monitoring Students and Other TA Tools

Once testing begins, the TA can monitor student progress and, if necessary, stop one or all
students from testing. The system is designed
to automatically notify a TA of events that
require an action on his or her part. For example, students with the appropriate accommodation may want to print a passage, and such
requests alert the TA to print the passage.
Exhibit 7.4.8.3-3 shows the TA interface during an active test session with five approved
students.
Exhibit 7.4.8.3-1: Starting a Secure Test
TA names a session
and selects the
tests to be given in
that session.
Starting a secure test: The TA tells the system

to start a session, and the students log on to join it.
As with paper tests, the TA confirms the students
identities.
air-176
System assigns a session

ID, which the TA tells the
students.
TA approves students who

have logged in and are
now waiting to take the test.
The TA checks that each
student is taking the right
test and is scheduled to
receive the appropriate
accommodations.
Student logs on using his

or her name, and other
identifying information such
as ID number and session ID.
Student selects test from
among those available in
the session and waits for
approval.
Student confirms the
selected test.
Student begins the test.
Exhibit 7.4.8.3-2: Adjusting Accommodations for Individual Students
This box appears when the TA clicks

on the list of students waiting for test
approval. The TA can approve all
students with one click or can choose
to view individual test settings
before proceeding.
In this example, some student

information, such as test language,
is set at a higher authority level.
These options can be customized
to Florida specifications.
The TA has the authority to adjust

individual student accommodation
codes according to the rules set
by Florida. This list is fully
customizable, and TA authority
can be limited or expanded as
desired.
After adjusting accommodations, the TA

can approve the student in one click or
save or cancel changes before viewing
other student accommodations by
clicking on the student name on the left.
Page 7.4-29
Exhibit 7.4.8.3-3: Test Administrator Interface During an Active Session

The screen clearly
displays the Session ID
for the TA to relay to
students for log in.
During a session, the TA
can add a test by simply
selecting from the
customized list.
Student status
column shows
how many items
have been delivered
to each student.
Student information
shows which students
are testing and other
relevant data for the TA.
The TA can pause

individual tests at
any time.
Test column shows which test

each student is taking. TAs
can proctor multiple tests in a
single session.
During an active test session, the TA can add

tests to offer in the session. This feature is useful, for example, if a student joins a testing session to make up or complete a test that had
been offered at an earlier time.
The list of students currently participating in
the session shows the TA who is actively testing, which test each student is taking, and how
many items have been delivered to each student. The TA can also view approved individual student accommodations by clicking the
binocular icon, and he or she can pause and
restart individual tests or the entire session at
any time. Student actions that require TA
intervention appear in the Requests column.
For example, our secure print-on-demand feature prompts the TA for authorization to fulfill
the request and to retrieve the paper from the
authorized printer.
Page 7.4-30
Click here to view

individual student
settings and
accommodations.
Search Feature
We have built two types of searches to help
Test Administrators look up students who may
have trouble logging in to the test or for other
data-related issues.
1. Quick Search If a student is having trouble logging in, a TA can use the Student
Lookup feature to search for that student
by the statewide student ID.
2. Advanced Search A TA can also search
for students by looking up a district,
school, grade, or name.
Alert Feature
The TA Interface has an alert mechanism that
enables the Department to push messages out
to all TAs currently logged in to the interface.
This is shown in Exhibit 7.4.8.3-4.
Exhibit 7.4.8.3-4: Alert Mechanism
Fully Configurable TA Interface

The Interface is designed to be fully configurable, and we will work with the Department
to set it up according to their specifications.
Exhibit 7.4.8.3-5 shows a list of what the
Department can customize.
Automated Backup, Recovery, and Restart
AIR has developed a system that includes
three innovations not found in other systems:
automated backup, recovery, and restart;
Internet-based solution; and multi-fault tolerant design. AIRs system is based on a realtime, distributed Internet technology, without
the need for caching servers or other burdensome technology in the schools. The system is
fault tolerant, with redundant hardware at
every point. Interrupted tests can be resumed
immediately without additional support or
intervention. Interruptions in Internet service
or computer failures do not lose data, and students can resume testing where they left off
when service is restored. Industry standard
backup and recovery procedures are in place
to ensure the safety, security, and integrity of
Floridas data.
Pure Internet Solution
Our pure Internet solution also reduces the
burden on schools. Our system has no need
for proctor caching or local caching servers.
Such caching mechanisms require extra hardware at schools, intermittent reinstallation,
and a higher level of technical support. Our

system is purely Internet-based, robust, and
nearly maintenance-free. Before the beginning
of the school year, schools install our secure
browser. The secure browser is simply a special build of the Mozilla (Firefox) browser that
is modified to protect the security of the test
and support certain accommodations. It can
be installed from a network on many
machines, or it can be individually installed.
We can even offer installation that does not
require administrative rights. Once the browser is installed, it generally does not change
throughout the school year. Changes or
updates to the test happen on our servers and
are transparent to the schools.
The pure Internet solution is robust. Every
time a student answers a question, the
response is saved to our servers. For longer
responses such as essays, the system can be
configured to save periodically (say, every
minute) or whenever the student presses the
save button. If the system is unable to reach
the server, the student is stopped from testing,
so no work is lost. The system is designed to
make multiple attempts to reach the server, so
even if connectivity is temporarily lost, no
work is lost. Often, the connection will be
reestablished without the student being aware
that it was ever lost. Caching occurs on the
students machines (in memory, not on disk)
in real time. This ensures a seamless testing
experience in which students typically see no
delays between items.
Contrast this approach with systems requiring
caching servers. The caching servers sit in the
schools or districts, providing many additional
points of failure. The likelihood of failure of
such servers is much higher than the likelihood of failure of our industrial, fully redundant servers hosted in professional, secure
hosting facilities. Hence, a caching server
approach requires more technical expertise
within schools or districts, requires more
Page 7.4-31
Exhibit 7.4.8.3-5: Possible Interface Configurations

Name
Look and Feel
Description
Skin
Color theme
Header and footer
Test Administration Windows
Test administration windows (can be specified by

test or a group of tests)
Test Settings
Can the test setting be changed after the student

has begun the test?
Which test settings should be visible?
What is the sort order of displaying these
settings?
Are TAs authorized to change the test setting?
TA Refresh Rate for New Students

TA Refresh Rate for All Else
TA UI Timeout
TA Check-in Time
Indicates how often (in seconds) the TA interface will

refresh for students who are awaiting approval.
Indicates how often the TA interface will refresh the
current status of students testing in the session, using
a variable that is multiplied with the value entered in
TA Refresh Rate for New Students.
Indicates how long (in minutes) a TA and students can
be idle before the proctor is automatically logged out
of the test session.
Indicates how long (in minutes) a TAs test session can
be idle before the system automatically closes the test
session
maintenance, and increases the likelihood of

failure. Our ability to deliver tests without
extra hardware or expertise has contributed to
our record of success wherever we implement
online tests.
Multi-Fault Tolerant Design (with Immediate
Continuation of Testing Activity)
From our online testing experience with multiple states, AIR understands that unforeseen
test interruptions can occur. For example,
there could be an unplanned fire drill, where a
Page 7.4-32
Examples
Combination of blue and green as the
color theme
The Department icon as part of the
header
Grade 38 Math
o 03/02/15 6:59 a.m.
03/13/15 11:59 p.m.
Algebra 1 EOC
o Window 1: 04/07/15 6:59
a.m. 04/25/15 11:59 p.m.
o Window 2: 07/13/15 6:59
a.m. 07/24/15 11:59 p.m.
Some examples of test settings include the
following:
Color Choices
Font Size
Font Type
Print on Request
Others
30 seconds
2 times, resulting in a total proctor refresh

rate of 60 seconds (2 x 30)
20 minutes
20 minutes
TA needs to immediately pause a test session;

a power outage, where testing suddenly stops,
or a student could suddenly get ill and the TA
needs to pause his or her test.
AIRs priority in these circumstances is twofold: (1) to protect the integrity of all test data
impacted by the interruption, and (2) to minimize test interruption by enabling the TA and
students to immediately continue from where
they left the test. Thus, the third innovation
that reduces burden on schools is a fault-tolerant design in multiple aspects of the system
and immediate availability of resuming testing

operations. We have done this in four ways:
1. Designed our hardware and network to be
extremely fault tolerant such that the system can withstand failure of any component with little or no interruption of service
(see sections on automated backup and disaster recovery procedures).
2. Designed our software so it recovers from
interruptions in Internet service without
loss of data and often recovers without
users even becoming aware that the service
was interrupted.
3. Designed our TA interface to enable TAs to
pause a student test at any time, or to
pause the entire test session (effectively
pausing all active tests in that session).
4. Built in real-time participation reports that
inform the Department (and other authorized users) about the testing status of students and test session information. The
reports provide information about current
testing activity and are a powerful tool for
monitoring progress toward getting students tested. See Section 7.5.5 for more
details on reporting.
Disaster Prevention and Recovery
We have designed our system to be extremely
fault tolerant. The system can withstand failure of any component with little or no interruption of service. One way that we achieve
this robustness is through redundancy. Key
redundant systems are as follows:
Our hosting provider has redundant power
generators that can continue to operate for
up to 60 hours without refueling. With the
multiple refueling contracts that are in
place, they can operate indefinitely. They
maintain an n+1 configuration of 16 diesel
generators that, at maximum capacity, can
supply up to 2.0 megawatts each.
Our hosting provider has multiple redundancies in the flow of information to and
from our data centers by partnering with
nine different network providers. Each
fiber carrier must enter the data center at
separate physical points, protecting the
data center from a complete service failure
caused by an unlikely network cable cut.
Every installation is served by multiple web
servers, any one of which can take over for
an individual test upon failure of another.
Active/passive clusters of database servers
are configured so that the passive node
takes over in the event of failure of the
active node.
Each database server in a cluster has dual
connections to the disk arrays containing
the system data.
Each disk array is internally redundant,
with multiple disks containing each data
element. Failure of any individual disk
recovers immediately by accessing the
redundant data on another disk.
Data are further protected by nightly backups.
We complete a full weekly backup and incremental backups nightly. The systems are run
with full transaction logging, enabling us to
restore the system to its state immediately
prior to a catastrophic event.
All backup media are stored in a secure, fireproof, off-site location. Our hosting provider
ensures that all media sent offsite are shipped
in locked, water-resistant and impact-resistant
containers. Our off-site vendor does not have
direct access to individual media containing
customer data at any point or any time during
transport. The offsite storage location is audited on a regular basis to ensure that proper
physical security, media management, and
location tracking are meeting the hosting
providers industry standard guidelines.
Page 7.4-33
7.4.8.4 Security Requirements

AIR has built-in security controls in all of its
data stores and transmissions. Unique user
identification is a requirement for all systems
and interfaces. All of AIRs systems conduct
external data transfers on a Secure FTP connection, and login information generated during user addition is communicated securely to
individual users. All communication with any
of the online websites happens over SSL this
ensures passwords and secure student data are
not passed over the network in clear text.
All of AIRs systems enforce unique identifiers
for all users, including students. The TIDE
database is set up to require an external ID for
every user uploaded and edited in the system,
and it will reject any record if it does not meet
this requirement. Students log in with identifying information and session ID, and the Test
Administrator authenticates the students and
admits them to the session. Please see Section
7.4.8.3 for more information on test administration.
Security of student information is protected at
all times, with security procedures acting at
three levels:
Physical security, preventing access to the
machines on which data reside or are
processed
Network security, including protection of
our networks from infiltration and secure
transmission of data over our networks and
others
Software security, ensuring that only
authorized users access information on our
systems and that their access is limited
only to information that they are authorized to view
Page 7.4-34
Physical Security
Florida Department of Education data will
reside on servers at Rackspace, our hosting
provider. Rackspace maintains 24-hour surveillance of both the interior and exterior of its
facilities. All access is keycard controlled, and
sensitive areas require biometric scanning.
Access credentials are assigned only for
authorized data center personnel, and only
they have access to the data centers. Visitors
identities are verified, and visitors are escorted
at all times while in the facility.
All data center employees undergo multiple
background security checks before they are
hired.
Secure data will be processed at AIR facilities
and will be accessed from AIR machines.
Similar security procedures are in place in our
facilities. Access to our facilities is keycard
controlled. Visitors must sign in and be escorted while in our center. Our servers are in a
secure, climate-controlled location with access
codes required for entry. Access to our servers
is limited to our network engineers, all of
whom, like all AIR employees, have undergone
rigorous background checks.
Staff at both AIR and Rackspace receive formal
training in security procedures to ensure that
they know and implement the procedures
properly.
AIR and Rackspace protect data from accidental loss through redundant storage, backup
procedures, and secure off-site storage.
Network Security
Hardware firewalls protect our networks from
intrusion. They are installed and configured to
prevent access for services other than HTTPS
for our secure sites. Our firewalls provide a
first level of defense against intrusion, backed
up by a capable second line: hardware and

software intrusion detection and remediation.
Our intrusion detection systems constantly
monitor network traffic and raise alerts for
suspicious or unusual network traffic.
pretations. Our system is designed to support

these interpretations flexibly. AIR is committed to working with the Department to maintain data security according to its specifications.
Our systems maintain security and access logs,

which are regularly audited for login failures,
which may indicate intrusion attempts.
Suspicious log entries are investigated and
resolved.
Secure transmission and password-protected

access are basic features of the current system
and ensure authorized data access. All aspects
of the system, including item development and
review, test delivery, and reporting, are
secured by password-protected logins.
All secure data transmitted across the public

Internet are encrypted using SSH (AES) or an
IPSec VPN. Secure websites encrypt data using
128bit SSL public key encryption.
The hosting environment is protected by an
AlertLogic Threat Manager Intrusion
Prevention System (IPS) appliance at the
perimeter and by Symantec Antivirus
Corporate Edition on each individual server.
The AlertLogic IPS appliance combines intrusion protection and vulnerability management
technology into a single integrated solution
that offers both proactive and reactive protection from the latest threats. Symantec
Antivirus offers real-time virus and malware
protection for our servers along with centralized management and administration capabilities.
Software Security
All of our secure websites and software systems enforce role-based security models that
protect individual privacy and confidentiality
in a manner consistent with Floridas privacy
laws, FERPA, and other federal laws.
Our systems implement sophisticated, configurable privacy rules that can limit access to
data to only appropriately authorized personnel. Different states interpret the Family
Educational Rights and Privacy Act (FERPA)
differently, and we support customized inter-
Our systems use role-based security models

that ensure that users access only the data to
which they are entitled and that limit their
ability to change that data according to their
rights. User rights have two dimensions: the
users role and the users data access rights.
The role determines what actions a user can
take, which types of reports he or she can
view, and similar functional limitations. Data
access rights tell, for example, which principal
can view which teacher and student data. Data
access rights are governed by relationships
among entities in our Roster Tracking System
(RTS), along with a configurable set of business rules. The business rules describe which
access rights correspond to which relationships and enforce client-specific policies. We
will work with the Department to maintain
current business rules and adjust business
rules as needed.
7.4.9 Computer-Based Test

Administration Overview
Introduction
The American Institutes for Research (AIR) is
pleased to propose a proven, online testing
solution to assess student proficiency with the
Departments State Standards. Our proposed
solution offers
Page 7.4-35
a robust, online test delivery system that

has proven effective, requiring little equipment, bandwidth, expertise, or support and
including the ability to deliver tests on iOS
and Android tablets and Chromebooks;
a student experience that empowers students with the broadest range of access
technology available in a statewide system,
while also offering a simple, accessible
interface free of distraction;
the industrys richest repertoire of item
types, using technology to probe deeper
into students understanding by asking
them to construct responses graphically
using language and simulations; and
the industrys broadest set of available
machine-scoring (sometimes called artificial intelligence) engines.
AIRs test delivery system currently delivers
statewide assessments in Delaware, Hawaii,
Minnesota, Oregon, and the Ohio Assessment
Pilot and delivered the Smarter Balanced
Assessment Consortium pilot test in spring
2013. All of the features described here are
currently operational.
Student Interface
Students take tests through the Student
Interface, which is essentially a secure website
accessed through AIRs secure browser. This
browser is the only software needed to take a
secure test, and it is simply a secure build of
the Mozilla (Firefox) browser or a secure testing app for tablets and other platforms.
Non-secure tests, such as practice tests and
some formative tests, can be accessed with ordinary browsers. AIRs secure browser operates in
a full-screen mode, disables access to other
applications, and prohibits navigation outside
the test. The browser is designed to intercept all
operating system hot-key combinations and
print capabilities; it enables keyboard combina-
Page 7.4-36
tions specifically designed for test navigation.

The system verifies that the test is being
launched in AIRs secure browser and prevents
the test-taker from continuing if the test is
launched in a normal browser window.
The Student Interface is designed to be intuitive and make it easy for students to track
information on a single screen. The interface
incorporates many tools and accessibility features, and various layouts are available for
items that require reading passages, graphics,
or other item stimuli. While the layouts are
completely configurable, we encourage states
to select layouts that allow students to read
left to right and top to bottom as they naturally would with a paper-and-pencil test.
Test Administrator Interface

Test administrators (TAs) use the TA Interface
to create and manage test sessions. The interface allows authorized TAs to administer test
sessions, monitor activity, and respond to testrelated issues in one convenient location. At
the time of testing, the test administrator
starts a testing session with as few as two
mouse clicks. The system assigns a session ID,
and student login is simply a request to enter a
session. Students log in with identifying information and the session ID. From his or her
workstation, the TA authenticates the students
and admits them to the session. In this way,
the roster of students to be tested builds itself
as students log in. This eliminates the need for
the TA to roster students into sessions, thus
reducing the schools workload. Please see section 7.4.8.3 for more information about the TA
Interface.
In section 7.4.9.6, we discuss the tools and features available to students in the test delivery
system, along with universal design concepts.
In section 7.4.9.7, we present a quick tour of
the interface with which students interact,
highlighting screen layouts for the presentation of computer-delivered test items.
7.4.9.1 Hardware and Software

Requirements
addition, our system supports older versions of

the Windows, Mac, and Linux operating systems. In fact, AIRs online test delivery system
can support nearly three times as many of the
Departments current student computers as the
required specifications allow. This well-proven
capability offers the Department the option of
moving more quickly online and gaining the
benefits of faster turnaround and reporting.
In schools and districts, AIRs test delivery system has succeeded where other systems have
failed. We designed our system to work within
the confines of real-life schools. Schools often
AIR analyzed the results of the Florida
have limited bandwidth, older hardware, and
Innovates Technology Resources Survey
limited or nonexistent technical support. Our
Statistical Report (2012) to produce Exhibit
system is a purely Internet-based system, and
7.4.9.2-1. According to the survey, there are
we support operating systems and browsers
approximately 1 million desktop and mobile
longer than their original manufacturers. This
covers almost all the computers currently found (laptop or netbook) computers available to students in the state. Of those, AIR calculated
in schools. Within the schools, the only software required is AIRs secure browser. It can be that 337,566 meet the operating system
installed with the click of a button, and installa- requirements of the ITN. An additional
678,832 meet the operating system requiretion options are available that do not even
require administrative privileges on the comput- ments of AIRs test delivery system.
er. The secure browser locks down the computer, preventing the student from navigating away AIRs test delivery system is currently supported across older platforms, allowing Florida to
from the test or starting other software. It also
utilize more of its existing computers for testdisables keystrokes that can threaten the secuing. AIRs support for Windows VISTA, Windows
rity of the test. For example, it disables screenXP, and Mac OS 10.5 and above would almost
shots and navigation and prevents test-takers
triple the Departments technology resources for
from viewing the source and opening the
taskbar. It continuously monitors other activi- testing.
ty on the computer for possible threats
Exhibit 7.4.9.2-1: Florida Technology Resources and AIR Support
and terminates testing if a threat is
Number of Student
Number of Student
detected.
Additional Computers
Our system is purely HTML-5- and
Internet-based. To ensure the security of
the test, we require the use of our secure
browser for all operating systems except
Chrome OS (for Chrome, we are able to
secure the native Chrome browser).
Computers Meeting
ITN Specifications
319,974
Computers Meeting
AIR Specifications
920,145
600,170
Mac Desktops
and Laptops
17,592
96,254
78,662
TOTAL
337,566
1,016,398
678,832
Windows Desktops
and Laptops
7.4.9.2 Eligible Device

Specifications
AIR currently provides support for all of the
platforms and ancillary devices identified in
Tables 7.4.9.2A and 7.4.9.2B of the ITN. In
Available for Testing
7.4.9.3 System Performance

Requirements
Our test delivery system does not require any
local caching servers, proctor caching servers,
or other specialized software requiring expert-
Page 7.4-37
ise, support, or maintenance. Such devices

increase the support load at the school,
depend on vulnerable within-school servers,
and historically have required support and
periodic upgrades. Our system does, however,
support local caching and does so without
requiring extra servers or additional software.
The memory of the browsers on the student
machines are used to cache item content (this
cache is never written to disk). This patentpending approach has enabled us to deliver
media-rich content where others have failed.
Our system relies on highly optimized, secure
Internet delivery. Install the secure browser on
the machines that the students will use, and
the rest happens between the student workstations and AIR servers. A typical test requires
only approximately 510 kilobits of bandwidth
per second. This means that a typical school
can easily test hundreds of students simultaneously. In fact, 510 students can be tested on
the bandwidth typically available through a
single smart phone.
Our approach means a smooth and robust
testing experience for students. Results from
2013 show that the typical student waited less
than half a second after having finished one
item before the next item was fully displayed
on his or her screen. The system recovers
from interruptions in Internet service without
loss of data and often recovers without users
even becoming aware that the service
was interrupted.
We prevent data loss by saving responses in
real time to AIR servers. The safe, asynchronous writing system reports the successful
save back to the browser. If the browser does
not receive the response after a configurable
amount of time (usually 3090 seconds), the
system stops the student from testing until
connectivity is restored.
Page 7.4-38
This same process protects data in case of

power outage. Responses are submitted immediately upon student response or automatically
and frequently during longer responses. If the
power goes out, the student responses prior to
the outage are already at AIR servers.
Contrast this with a system that relies on local
caching of student responses. A failure of the
local caching server could lose many responses
from many students. Our efficient use of realtime Internet communication avoids such
calamitous possibilities.
We know that students wait an average of less
than half a second after they press the button
before the next item is logged on the screen
because we log this information. This information helps us to
accurately report information about the
student testing experience;
plan capacity to make sure servers are
never overworked; and
identify schools that may be having local
delivery problems.
The last point is particularly powerful. We
have contacted schools to help troubleshoot
before they were even aware that they were
having a problem. Imagine a help desk that
responds before the user even notices a problem. We believe that this level of information
and transparency is unique within the industry, and we encourage reviewers to seek
item-by-item latency (delay) data from other
offerors who claim that their systems are
responsive and scalable. In addition, our system records an audit trail that tracks every
time a student changes an answer or revisits
a question.
Our architecture sends results back to the server as the student is responding, so there is no
need to cache responses and no risk that failed
machines in a school can lose student data. If a

response cannot be saved due to a lack of connectivity, the student is stopped from testing.
When responding to constructed response
items, student responses can be saved frequently automatically, as well as on-demand
when a student presses the Save button.
Our test delivery system also has an infinitely
scalable architecture. AIRs system is based on
a private-cloud design that allows every function to be distributed across many database
and application servers, while maintaining
responsiveness. In preparation for our Smarter
Balanced field test, we plan a 500,000-simultaneous-user-load test in January 2014 (i.e.,
500,000 students all testing at the same time).
This load is sufficient to prove the scalability
of the systemit will be deployed on distributed hardware for every function. Scaling up
from the 500,000-simultaneous users will only
require deploying more hardware on our private cloud. For the Department, we can either
add to one of the two shared clouds that we
are building (adding capacity to the expected
load where necessary), or if the Department
would prefer (and at a somewhat higher cost),
on a private cloud solely deployed for Florida.
7.4.9.4 Connectivity
Requirements
We understand that schools often do not have
the best or fastest equipment or networks, and
they often have limited technical support
available. AIRs test delivery system has the
smallest possible technology footprint in the
industry and low end-user maintenance. We
accomplish this with an architecture that
is a fully web-based, adaptive test system;
requires no external caching servers to
manage;
requires no extra third-party software;
supports a wide range of OS and HW; and

works on wired and wireless networks.
7.4.9.5 Test Site SetUp/Certification Requirements

As mentioned in sections 7.4.9 and 7.4.9.1, we
require the use of our secure browser to
ensure security of our tests.
We have a web-based tool that performs a
system check to assess the readiness of a
particular computer for testing. One of the
checks is a download and upload speed check
that includes an estimate of the number of
simultaneous examinees who can test at the
same location. This tool has proved useful to
other K12 programs, and AIR will make this
system check tool available to the Department.
The diagnostic readiness tool uses a sophisticated statistical model of the testing process
that models variation in instantaneous demand
(e.g., the number of students who simultaneously press the Next button) and evaluates the
likelihood that peaks will exceed network free
capacity (not used by other processes) with a
frequency likely to cause noticeable delays in
testing. Exhibit 7.4.9.5-1 presents the user
interface for our integrated diagnostic tool.
7.4.9.6 Computer-Based Test

Presentation Requirements
Our test delivery system offers the industrys
most robust set of operational features. In this
section, we describe the tools and features
available and summarize the accessibility features available.
Tools and Features

AIRs system offers a broad array of tools,
including calculators, formula sheets and other
resources, notepads, measurement, and tools.
Exhibit 7.4.9.6-1 shows our straight edge,
Page 7.4-39
Exhibit 7.4.9.5-1: Simple, Effortless, Accurate Diagnostic Tool
Some tools make sense configured by

item, such as the measurement tools in
Exhibit 7.4.9.6-1. Other tools, such as
calculators, need to be in separate
segments. Our segmentation capability allows students to complete items in
one segment, review them, and indicate completion before moving on to
the next segment. Once a segment is
completed, the student cannot return
to it; this capability would prevent the
student from, for example, accessing a
calculator when its use is prohibited.
Accommodations and Embedded

Supports
Exhibit 7.4.9.6-1: Test Delivery System Measurement Tools
AIR has a long standing commitment to

students with disabilities and to students who are English language learners (ELLs). We have been
committed to accessibility
for all students ever since
we conceptualized and
designed our test delivery
system and its items. We
continue to make accessibility a top priority as we
build more innovations
around student testing and
learning.
AIRs test delivery system
offers a robust array of
accessibility features, as
illustrated in Exhibit
7.4.9.6-3.
ruler, compass, and protractor. As with the rest

of our system, these tools are entirely built in
HTML-5 and can be completely controlled by
the keyboard. We believe these HTML-5- and
keyboard-controlled tools are unique within
the industry. Exhibit 7.4.9.6-2 offers a more
complete listing of our available tools.
Page 7.4-40
AIR is proud not only that

we offer these features but that virtually all are
in operational use right now in states for
which we are providing online testing.
In this section, we would like to highlight a few
accessibility features that we have deployed
recently. In 2013, AIR introduced individual-
Exhibit 7.4.9.6-2: Summary of Available Tools

Tools
Description
AIR System Capability

and Comments
Required Features
Calculator
Available calculators include basic, scientific, graphing, regression, and matrices.
Currently Available
Reference Sheet
Students can access a formula sheet and periodic table for Mathematics tests.
Currently Available
Ruler
Students can access a pop-up ruler.
Currently Available
Preferred Features
Straight Edge
Students can access a pop-up straight edge.
Currently Available
Option Eliminator
Students can eliminate any multiple-choice option, whether its in text or graphic. This capability
persists throughout the test.
Currently Available
Bookmark
Bookmark capability allows students to flag an item so they can review it.
Currently Available
Highlighter
Students can select any text to highlight. This capability persists throughout the test.
Currently Available
Notes
Item notes allow students to jot down ideas about items or passages.
Currently Available
Footnote Pop-up
Our word list feature supports footnote pop-ups.
Currently Available
Student Tutorials/
Practice Sessions
A reference feature, practice tests, and tutorials familiarize students with the online testing system.
Our tutorials can vary by item, so items of each type share a tutorial.
Currently Available
ized item pools that take each students access

restrictions and produce a custom filter on the
pool in real time when items are being selected. This feature is most relevant to adaptive
tests and helps demonstrate the depth of our
commitment and capability to serving all of
Floridas students.
We note that we support American Sign
Language (ASL) through recorded video rather
than avatars. Working with representatives of
the deaf community, we learned that ASL is
not simply a representation of English. Rather,
it is a unique language. Avatars are not capable
of translating from English to ASL and instead
provide a word-for-word translation. To speakers of ASL this seems as unnatural as a wordfor-word translation from English to any other
language. Furthermore, ASL is more syntactically dense than the major gestures. Cues such
as facial expression and lip movement form
part of the language. Avatars neglect this
aspect of the communication. Therefore, we
present ASL through video. This is shown in

Exhibit7.4.9.6-4.
We have recently developed a feature that we
call word lists. Word lists support a wide variety of accommodations and embedded supports
through a common, simple interface. Some students require a contextualized dictionary, glossary, or second-language dictionary or translation. Word lists support all of these features.
Any word in any item text can be associated
with word lists of each type. Individual student
profiles determine which types of word lists are
active on each students tests.
The interface is simple, intuitive, and not distracting. Students mouse over words or move
to them using the keyboard, and words associated with entries in the word list are highlighted. This mouse over approach avoids the
distraction of many highlighted words (which
research has shown decreases comprehension). Once a word is highlighted, clicking or
selecting with the keyboard brings up the word
Page 7.4-41
Exhibit 7.4.9.6-3: Accommodations and Accessibility Supports

Accessibility
Feature
Description

and Comments
Accommodation Features (Required)

Variable Font Size
Available calculators include basic, scientific, graphing, regression, and matrices.
Currently Available
AdministratorSelectable Zoom
Default font size can be set in advance through a file upload or user interface or at the time of testing
by the test administrator. Student can zoom or unzoom at any time.
Currently Available
AdministratorSelectable
Large Print font
Default font size can be set in advance through a file upload or user interface or at the time of testing
by the test administrator. Student can zoom or unzoom at any time.
Currently Available
AdministratorAny foreground and background color can be supported.

Currently Available
Selectable Variable
Font and
Background Colors
AdministratorThe system supports an integrated screen-reader that can be configured to provide a variety of support Currently Available
Selectable Screen- levels, each selectable by the administrator.
Reader
Currently Available
AdministratorOur system has a standard and a streamlined interface. Most assistive devices can work with the
Selectable Assistive former, and an even wider group works with the latter. If the use of the device requires relaxation of
Devices Integration certain security features (e.g., if suppression of pop-up windows interferes with on-screen keyboards),
the administrator can select our more permissive mode if the Department chooses to offer
this feature.
Additional Accommodations and Embedded Supports
Additional Time
AIRs system currently does not impose a time limit on the test. It is up to the proctor to stop a students
test or stop the entire session.
Currently Available
However, if there are unforeseen events, such as a fire alarm, that trigger additional testing time, AIRs
system can enable a grace period extension for a single test opportunity or for multiple test opportunities.
Segment Breaks
AIRs system has the capability of adding test segments within a test. A test segment is made up of
multiple item groups and creates a logical break between segments within a test. For example, a
segment break might separate a calculator from a non-calculator segment of a test.
Text-to-Speech Computer reads text and graphics aloud on directions, passages, and items. What is read and how it is
Directions,
read is configurable, and the Department can offer a variety of options.
Passages, Items
Text-to-Speech Computer reads graphics and tables aloud.
Graphic Description
Currently Available
Currently Available
Currently Available
Color Overlay
Any color can be laid on the screen. This persists throughout the Test.
Currently Available
Recorded Audio
Computer efficiently delivers recorded audio. We are able to deliver voice-audio using only about
10Kbps of bandwidth.
Currently Available
Reverse Contrast
Background turns to black, while text turns to white.
Currently Available
Line Reader
This feature will allow a student to track the line he or she is reading.
Currently Available
Page 7.4-42
Exhibit 7.4.9.6-3: Accommodations and Accessibility Supports (Continued)

Accessibility
Feature
Description

and Comments
Refreshable Braille/ Items can be rendered to desktop embossers that can integrate Braille and tactile graphics. The items
Tactile with External will simultaneously render on a reader-accessible screen, and the student will be able to navigate to
Embosser Printer
response spaces to provide answers.
Currently Available
Magnification
Currently Available
Student can zoom and unzoom the entire page. This capability persists throughout the test.
In addition, in the near future, this capability will allow the student to selectively magnify areas on
the screen.
Masking
Students can mask extraneous information on the screen.
Scheduled for Deployment

Winter 2013.
Speech-to-Text
Speech will be converted to text and then saved in the database.
Available Through
Compatibility with ThirdParty Assistive Technology
Sign Language
Directions,
Passages, Items
This capability consists of recorded videos on sign language. Avatars are not recommended by
hearing-impaired experts since they do not translate well to American Sign Language.
Currently Available
Translations
Versions are available in alternate languages.
Currently Available
Glossaries and
Dictionaries
These enable content developers to associate additional content with words or phrases. The content
can be of multiple types, and the content shown to a student can be controlled by his or her
personal profile.
Currently Available
Alternate Language These enable content developers to associate alternate-language content with words or phrases. The
Glossaries and
content can comprise multiple types, and the content shown to a student can be controlled by his or
Dictionaries
her personal profile.
Currently Available
Secure Print Facility A visual accessibility feature, the secure print facility allows the secure printing of items or passages.
A student requests that a passage or item be printed; the request is then encrypted and sent securely
to the proctor; the proctor needs to approve the request before it is sent to the printer.
Currently Available
In addition, this feature also allows for the delivery of real-time paper tests, including large print tests.
Test Pauses and
Restarts
An attention accessibility feature, test pauses and restarts, allows the test to be paused at any time
and restarted and taken over many days. So that security is not compromised, visibility on past items
is not allowed when the test has been paused longer than a specified period of time.
An attention accessibility feature generally for essay items, the writing checklist enables a student
to check off writing guidelines from a checklist.
Currently Available
Review Test
Students can review the test before ending it.
Currently Available
Area Boundaries
An agility accessibility feature, area boundaries for mouse-clicking multiple-choice options allow
students to click anywhere on the selected response text or button.
Currently Available
Language
Any language that is necessary can be supported.
Currently Available
Help Section
A reference feature, the Help Section explains how the system and its tools work.
Currently Available
Writing Checklists
Performance Report A reference feature, a performance report is available at the end of the test for the student.
Currently Available
Currently Available
Page 7.4-43
Exhibit 7.4.9.6-4: American Sign Language
Assignment of Tools and

Supports
Each accommodation can be one of
three different configurations or
combinations of these choices: (a)
available to all students, (b) assigned
to students in advance by the designated state, district, or school administrators or through data upload, or
(c) assigned to students at testing
time by the proctor.
Delivery Choices
We propose to offer a broad array of
features. Furthermore, as we continue to innovate to extend the accessibility of the test, improve measurement, and make testing more engaging, features will proliferate. This
presents a bit of a conundrum:
How will students know what
features are available? Students
only test once or a few times a
year. Therefore, they will generally not become expert users
of the software. Making the
appropriate tools available is
best accomplished by offering
each student the set of tools he
or she needs, without the distraction created by tools that are
not needed.
Exhibit 7.4.9.6-5: Word Lists
Each tool or embedded support

provided by AIRs system, by
design, may be configured to be
available to all students;
list entries indicated by the student profile.

This is illustrated in Exhibit 7.4.9.6-5.
Page 7.4-44
made available to students

by one or more adult roles in the
system (e.g., state, district or
school test coordinator, test
administrator); and
offered to all students but configured for

each student by adult users.
Exhibit 7.4.9.6-6 illustrates some of the configurations other clients have chosen for some of
the offered accommodations.
Some of the accessibility features require special attention and preparation to make them
effective and meaningful for the item and the
student taking the test. These features were
designed to serve their primary purpose in as
meaningful a way as possible by pulling in the
appropriate resources. In addition, each feature is effective only if it interacts seamlessly
with the items it is presented with as well as
other features. Exhibit 7.4.9.6-7 lists these features and their respective requirements for
preparation.
7.4.9.7 Computer-Based Test

Screen Layout Requirements
The look and feel of AIRs TA and Student
Interfaces is customizable. Each test (grade,
subject, etc.) can have its own default font and
layouts, among other options. There is a broad
array of available tools and accessibility features for students that can be configured in
different ways. Additionally, AIR offers a wide
range of tools in the TA Interface for monitoring testing activity and system health.
First, we present a quick tour of the interface
with which students interact. This selection is
not intended to be exhaustive. For example,
the screenshots that follow show very few of
the item types supported and only a handful of
the embedded supports that we offer, many of
which are currently operational in our partner
states.
Student Interface Layout

Our student interface begins with a simple
look and feel. The look and feel is customizable. Each test (grade, subject, etc.) can have
its own default font, layouts, etc. Exhibit

7.4.9.7-1 shows a sample AIR TabletFirst skin.
This skin began with a design for tablets and
brought the same look and feel to other
devices. Notice that the interface is not cluttered with tools or options. In general, we simplify the interface in two ways:
We move the selection of accessibility tools
or accommodations to precede the test. We
usually recommend that the selections be
made at registration time or by the test
administrator at testing time, but the system
can allow the student to make the selections. Some tools are configured to be available to all students taking a particular test.
We use context menus for students to
access the various tools and supports. This
serves two purposes. First, it is part of our
strategy to make sure the entire system is
keyboard navigable, using a very small
number of keyboard commands. This strategy is necessary to support alternate input
devices, such as switch arrays, that provide
access for some students. Second, it supports all students by making all the relevant tools and options available in the context in which they are relevant. This helps
students find what they need when they
need it. Rememberstudents test rarely
and hence cannot be expected to become
expert users of the software.
Exhibit 7.4.9.7-2 illustrates our context-menu
strategy. One click (or keystroke) brings up
the menu. The menu presents all the relevant
options within the context. In this case, the
menu includes highlighting, mark-for-review,
view the tutorial, text-to-speech for the question, or text-to-speech (TTS) for the selection
(which the student had previously selected, as
indicated by the gray background around a
snippet of text).
The system includes a host of small, intuitive
features that help students navigate and use
Page 7.4-45
Exhibit 7.4.9.6-6: Access Configurations for a Sample of Embedded Supports and Accommodations
Accessibility
Feature
Example of Configuration Choice
States Rationale Behind Configuration Choice
Color Choice
Assigned to students at testing time by the proctor
Most of our clients recognize that the proctor is the best person to determine
the appropriate color since the student is with him or her in the room. If this
tool is made available to all students, it would provide a distraction.
Increased Font Size
Default font size (assigned to students in advance)
A students need for this feature is part of his or her school record and
therefore can be determined in advance.
Text-to-Speech Assigned to students in advance

Directions, Passages, OR
Items
Assigned to students at testing time by the proctor
In some states, a students need for this feature is part of his or her school
record and therefore can be determined in advance. In other states, it is
available to all students but requires a proctor to turn it on to ensure that the
student has a headset and so will not disturb anyone else.
Magnification
Available to all students
This feature does not provide a distraction, and the student may need it on
different screens.
Answer Eliminator
Available to all students
This feature does not provide a distraction, and the student may need it on
different screens.
Refreshable Braille
Assigned to students in advance
A students need for this feature is part of his or school record and therefore
can be determined in advance.
Exhibit 7.4.9.6-7: Accessibility Features and Preparation*

Accessibility Feature
Color Choice
Preparation
Colors that will be available on the system require that they do not conflict with the item graphics and text. For
example, if a blue background is one of the background colors, the same blue cannot be used on item graphics.
Thus, items need to be reviewed and may need tweaks if color conflicts arise.
Text-to-Braille and Text-to-Speech Though the computer can automatically speak text rendered on the screen, special tags need to be created on
graphics, directions, passages, and items. These tags would need to comply with the read-aloud principles
determined by the Department.
Large Print and Magnification
Graphics need to be legible in large print and all available magnification levels. Ideally, graphics would be
vector-based so they magnify in a clear and consistent way. However, if vector-based graphics are not possible,
some graphics may need to be enhanced depending on their legibility at specific zoom levels.
Sign Language
Recorded sign language translations need to be created in advance and undergo review levels (ASL or Exact English).
Audio Files
Audio files need to be created or recorded in advance.
Language
Content needs to be properly translated in supported languages other than English.
*We note that the availability of system features does not imply content annotated or prepared for delivery using the feature.
Page 7.4-46
Exhibit 7.4.9.7-1: Simple, Clean Interface (TabletFirst Skin)
Exhibit 7.4.9.7-2: Context Menus Ensure Keyboard

Accessibility and Find-as-Available Functionality
reduces the need to scroll or page.

Exhibit 7.4.9.7-3 shows an example of
this capability.
Our pure HTML-5 Student Interface
supports the industrys widest set of
devices, ranging from desktops and laptops to tablets and Chromebooks. Using
standard HTML-5, we are able to deliver
sophisticated animations, simulations,
multimedia, and a wide range of input
types. Exhibit 7.4.9.7-4 shows an online
performance task that includes an
HTML-5 animation and that asks the
examinee to write equations using our
HTML-5 equation editor. We note that
the student responses can be machinescored using symbolic math logic to
evaluate whether any expression the
student writes is mathematically equivalent to the correct answer. The equation-scoring engine can also evaluate
whether the response has the correct
functional form even though it might
not be correct and whether there are
other properties for which partial credit
might be awarded.
7.4.9.8 Training Delivery

Requirements
the test without getting distracted. For example, the expand-passage function allows the
student to allocate more room to the passage
as he or she reads, while still having immediate access to the test items. While our system
is capable of displaying one item in a set at a
time, good testing practice presents students
with the information for which they will be
reading the passage before (or along with) the
passage itself. The ability to expand the passage reduces the load on working memory and
Providing comprehensive, ongoing training throughout the year on the new

online testing system for Florida educators and administrators is essential to
success. Below we propose an annual training
plan that includes a series of interactive presentations on each of our online systems to be
delivered through a combination of face-toface training, webinars, and webcasts, and selfguided and self-paced online interactive sessions. These training sessions will support
ongoing training for the following audiences:
Page 7.4-47
Exhibit 7.4.9.7-3: Expand-Passage Function
Exhibit 7.4.9.7-4: Integration with HTML-5 Animations and Simulations and MachineScored Equation
Page 7.4-48
Department K12 Student Assessment program team

Other Department staff
District assessment coordinators
School assessment coordinators
School and district hardware/network technical staff
School and district test administrators
Our experience transitioning large-scale
assessment programs in Hawaii, Delaware, and
Minnesota has taught us that clear, early communication about the new online testing system is an important training component.
While training teaches stakeholders the handson skills needed to use each component of the
new online testing system, communication
lays the foundation for what to expect: How
does the new online testing system differ from
the old testing system? What kinds of training
will be provided and when? Will my student
have more than one opportunity to take the
test, and if so, how will his or her final score
be determined? Addressing these questions
early on, in addition to having a comprehensive training plan, has proved effective in
calming concerns and building critical stakeholder support.
In the next section we discuss our proposed
plans for early communication and for ongoing, interactive training in more detail.
Proposed Plan for Early

Communication
Transitioning a large-scale assessment program
to online testing typically produces an enormous amount of information. The objective of
an early communication strategy is to provide
stakeholders with a clear structure for organizing and absorbing the information they will be
receiving. We propose to accomplish this by
collaborating with the Department to cus-
tomize a communication tool for Florida parents, educators, and administrators. We also
propose to work with the Department to develop a customized information portal for all
Florida educators, administrators, families, and
students that functions as a central repository
for all communication and training materials.
The structure of the portal may be organized
according to role (e.g., school assessment coordinator, teacher, technology coordinator, parent, student) or by separate, subject-specific
tabs, such as Secure Browser, Training,
General Resources/Manuals, FAQs, and
Contact Us. The portal tabs function like a
table of contents, helping users organize and
locate information as needed. The portal home
page also offers an easy-to-access location for
important announcements and updates. Aside
from functioning as a central information
repository, the portal also provides each user
role with password-protected access to only
those components of the online testing system
that are relevant to each specific role. AIR staff
will work with the Department to configure the
portal in the most optimal way for Florida educators and administrators.
Proposed Training Plan

In this section we propose an annual training
plan that includes a series of interactive presentations on each of our online systems,
which will be delivered through a combination
of face-to-face trainings, webinars and webcasts, and self-guided, self-paced online interactive sessions.
training for Department and district personnel
to prepare them for online testing. We propose
to conduct 10 half-day face-to-face onsite
training workshops annually. Up to 500 Florida
educators will participate in each onsite faceto-face training, attending a comprehensive
half-day session with breakout groups address-
Page 7.4-49
ing different topics in an interactive manner.

We will work with the Department to identify
training locations throughout Florida on up to
10 dates chosen by the Department in spring
2015.
AIR proposes to conduct a series of three moderated webinar presentations to accommodate
those who could not attend the face-to-face
trainings. We propose each of these three
webinar sessions to be 30 minutes in length,
plus include a question-and-answer period,
and Department and district personnel will
have the capability to call in to webinar sessions. We will post the webinar sessions on the
information portal as webcasts. The webcasts
will provide information about AIRs help desk,
which will be available to answer all questions
about any aspect of our online testing system.
Successful district and school staff training, as
well as thorough system documentation and
support available on the information portal,
has proved effective in minimizing questions
about AIRs online testing system. For those
who may need to ask questions that were not
asked during webinar sessions, our help desk
will provide additional support.
AIRs online interactive sessions are designed
to teach both sophisticated technology users
as well as users new to the system their roles
and responsibilities in context of the new
online systems. While sophisticated users may
not need instruction on all steps of a specific
function and can progress to the point in the
tutorial they find useful, new system users typically find it useful to have instruction at a
very detailed level. The online interactive sessions will also provide follow-up training
throughout the year to educators and administrators who require it. In addition, the online
interactive sessions can be developed to
Page 7.4-50
include quiz questions to provide the users

with a gauge of his or her knowledge after the
training. Each online interactive session will
be up to 15 minutes in length.
Description of Proposed Training Topics

We propose to conduct comprehensive trainings
on the topics displayed in Exhibit 7.4.9.8-1.
The following list provides more detailed
descriptions of the proposed training topics.
1. User Roles Chart. This reference document
indicates, at a glance, the access each user
role is permitted in each online system.
(Please see the sample user role chart in
Appendix D.) System access for each user
role will be reviewed in each webinar as
appropriate. The user role chart will be
posted on the portal, be available in manuals, and form part of the online, self-guided
tutorials as determined in collaboration
2. How to Use TIDE. This training includes
instructions on how to add users at the district and school level so they can access
appropriate online systems. The training
also covers other TIDE functionality,
including updating student test settings and
setting test restrictions.
3. Preparing Your Schools Computer Lab.
This training includes information for
school and district network administrators
and provides guidelines for varied technical
setup and technical support available at the
schools. Technical setups include multiple
secure browser installation methods, such
as manual installation on individual
machines, installation on machines through
a network, access from a shared network
Exhibit 7.4.9.8-1: Proposed Presentations and Formats for Annual Training

Presentation Title
and Primary
Audience
Proposed Training Topics
Proposed Training Formats
Training Phase I: Preparing for Online Testing

1.
2.
User role chart

Will be a component of each
webinar and face-to-face training
How to Use TIDERegistering Users and

Teach schools and districts how to register users in
Modifying Student Test Settings: School and TIDE, update student test settings, and set
restrictions.
District Technology Coordinators, School and
District Assessment Coordinators
1.
2.
3.
Webinar
Face-to-face training
Online interactive session
Preparing Your Schools Computer Labs:

School and District Technology Coordinators,
School and District Hardware/Network
Technical Staff
1.
2.
Instructions on portal
User Roles: Principals, Teachers, School and

District Test Administrators, School and District
Technology Coordinators, School and District
Assessment Coordinators
User roles: Who does what and in which system?
Steps for secure browser installation and minimum

hardware requirements.
Training Phase II: Administering Online Tests

Online Test Administrator Certification
Course: School and District Test Administrators,
Teachers
Learn how to use the new online testing system,

set up a test session, manage and monitor testing,
and assist students with online tools. Certification
earned only by passing a mandatory quiz.
1.
Online Test Administrator

Certification Course
Online Practice Tests: Educators, School and

Practice setting up, pausing, resuming, and ending

live test sessions; setting accommodations; using the
same functionality as the operational test
administrator site.
1.
2.
Online practice tests

Online Practice Tests: Educators, School and

Practice signing in to a live test session using the

same functionality as the operational student
testing site, including using text-to-speech, zoom,
highlighter, and strikethrough and answering all
item types
1.
2.
Online practice tests

How to Start and Monitor Online Testing

and Modify Test Settings: School and District
Test Administrators, Teachers
Learn how to set up a test session, modify test

settings and monitor participation in multiopportunity testing (in addition, learn how to
reset and invalidate tests)
1.
2.
3.
Webinar
1.
2.
3.
Webinar
Training Phase III: Post Testing

How to Use the Online Reporting System:
Teachers, Principals, School and District
Assessment Coordinators
Learn how to access student scores, create class

rosters, and drill down from district and school
results.
Page 7.4-51
drive, and thin-client setup. Minimum

hardware requirements will also be
reviewed in detail.
4. Online Test Administrator Certification
Course. This self-paced course includes
detailed instruction on creating and managing test sessions, monitoring student activity, setting accommodations, and implementing test security measures. The course
is designed to familiarize test administrators with the Student Interface, the
approval process, and the tools students
will use during testing. To ensure that
course-takers have learned the skills necessary to proctor tests using the new online
system, each user must pass a quiz at the
end of the course before being permitted to
sign in to the online testing system. Users
who need to refresh their memory of
course content can retake the course as
many times as needed.
5. Online Practice Tests. Online practice tests
for all assessments will be available prior to
the opening of the test window. Each practice test will be composed of approximately
20 items that are a subset of the blueprint
for the operational test, thus providing
users with exposure to all item types. The
online practice test site uses the same
applications as the operational test site,
such as the TA Interface, the Student
Interface, and the test management
reports. This design ensures that students,
educators, and teachers become familiar
with the online testing system before operational testing begins. In addition, user
guides and systems documentation will be
made available when the online practice
tests open to help educators prepare their
materials prior to testing. The online practice tests will be available throughout the
test window.
Page 7.4-52
6. How to Start and Monitor Testing and

Modify Test Settings. This training includes
information for test administrators on how
to set up a test session, modify test settings, and monitor participation in multiopportunity testing. In addition, test
administrators will learn how reset and
invalidate tests. The content for this training is the same as the Online Test
Administrator Certification Course but is
presented in a webinar and face-to-face
presentation format.
7. How to Use the Online Reporting System.
This training teaches authorized users how
to use two types of reports: test management reports and performance reports.
Users are shown how to use test management reports to monitor student testing status for their schools or districts and to identify students who have and have not tested.
Users also learn how to use these reports to
help schedule test activities for their school.
Turning to performance reports, users learn
how to view district, institution, personnel,
roster, and individual student reports in
table and graph format; access and use longitudinal reports for their diagnostic needs;
filter score reports by student subgroup
(e.g., gender, ethnicity, ELL); and manage
online rosters (groups).
Each year, AIR will collaborate closely with the
Department to update the face-to-face training
presentations, webinars, webcasts, and online
interactive sessions as needed. In the first
year, we propose to focus all presentations on
the basics of the program and the features of
the online system. In subsequent years, we
propose to focus training on the updates to
each system and to provide basic refresher
training on each system. Before the beginning
of each school year, AIR will provide a training
Exhibit 7.4.9.8-2: Proposed Timeline for Year 1 Training

Training Type/Title
1. Conduct Onsite, Face-to-Face Training
2. Webinar on TIDE
3. Webinar on How to Start and Monitor Online
Testing and Modify Test Settings
4. Online Test Administrator Certification Course
5. Online Practice Tests Available
Proposed Timeline
46 weeks before testing window opens
6. Webinar on Online Reporting System

7. System-Specific Online Interactive Sessions
Available for Online Testing System, TIDE, and


plan and schedule for the upcoming year. AIR

will work with Department personnel to identify dates for the trainings that align with the
testing schedule. A proposed timeline for Year
1 training can be found in Exhibit 7.4.9.8-2.
AIR will work with the Department to develop
and finalize a training plan for each upcoming
year.
7.4.9.9 Technology-Supported
Item Types
AIRs test delivery system is able to deliver a
wide variety of item types. Based on a survey
of existing systems and reviews and reviews of
existing statewide tests, we believe that we
offer the largest array of item types, and
machine-scored, constructed-response item
types in particular, in the industry. In this section we present a gallery of item types to illustrate these capabilities.
Rich Repertoire of Item Types

AIRs robust and flexible technology puts the
ability to create a wide variety of item types in
the hands of item developers. While it is typical to discuss items in terms of item types,
our success has derived from a broader, more
generalized understanding. Instead of thinking
about item types, our systems are built

around a broad and growing collection of
response mechanisms. In addition to
multiple-choice items, our system currently supports the following response mechanisms:
Graphic Response, which includes any
item to which students respond by drawing, moving, arranging, or selecting graphic
regions
Hot Text, in which students select or
rearrange sentences or phrases within a
passage
Equation Response, in which students
respond by entering an equation
Word Builder, in which students respond by
entering a single number or word
Proposition Response, in which students
respond in one or more English language
sentences, which may be scored by our
proposition-scoring engine or human
scored, or a mixture of both
Essay Response, in which the student
response is a longer written response to be
scored using an essay-scoring engine or a
mix of machine and human scoring
Simulator Interactions, in which students
interact with a simulator and their
sequence of choices and interactions can
be scored according to an explicit rubric
Spoken Response, which captures student
speech for human scoring
Scratch-Pad Response, which enables students to integrate text, free-hand drawing
and equations to respond to items. These
items will be hand scored.
Page 7.4-53
AIR supports the standard QTI templates;

however, most of the QTI technologyenhanced item types are essentially selectedresponse items. We strongly encourage our
clients to avoid using technology to simply
present selected-response items in a different
format: the interactions are inherently less
accessible than simpler formats, and universal
design principles discourage such choices.
Our technology offers our client the chance to
use technology to measure deeper knowledge
and application of knowledge in a more openended way and to machine score many such
items. We call these applications of the technology machine-scored constructedresponse, or MSCR. Most MSCR items
remain accessible. Whenever accessibility is
sacrificed for some population, we encourage
our clients to weigh the measurement benefit.
For example, recognizing cells under a microscope is an inherently visual task. Our simulation items can measure this ability, but the
task itself cannot be made accessible to students who are blind. In this case, the skill
itself limits accessibility, not the construction
of the item. This is very different than presenting a selected-response item as a spatial
matching or drag-and-drop task.
Our graphic-response mechanism supports
most of the typical technology-enhanced item
types, including sorting, matching, hot-spot,
and drag-and-drop. In addition, it supports
items where students actually draw a machinescorable response and respond by constructing
complex, open-ended diagrams, as well as
many other possibilities. Because they are uni-
Page 7.4-54
formly derived from a single response mechanism, the manipulations and interactions are
consistent across these technology-enhanced
item types, eliminating one possible source of
construct-irrelevant variance.
Hot-text items are effectively selected-response
items, though in some cases the number of
potential selections is quite large. These
machine-scored items can have multiple correct answers and allow for very flexible student responses.
The equation response mechanism asks students to enter one or more equations using a
palette of symbols. Test developers can specify
which symbols are available on an item-byitem basis, or the Department can choose to
have the palette remain consistent across all of
the items within a grade level.
The proposition response mechanism and
essay response mechanism are very similar to
short constructed-response and extended constructed-response items in which the students
type their answers into the space provided.
The response mechanisms differ by the length
of response accepted (both configurable by
item) but, more important, by the scoring
engine used to score the response.
Our system includes a very powerful and flexible mechanism for test developers to define
highly interactive simulation tasks. The test
developers can choose from many standard web
input widgets and can define output that can (a)
drive an animation and (b) populate a table.
These interactive simulations can be offered
as a mechanism for students to explore or

experiment with the simulated phenomenon. If
the Department chooses, student interactions
can be captured and scoredfor example, the
Department can evaluate the quality of an
experiment represented by a sequence of trials.
Our system is capable of capturing speech and
routing it to our scoring system. We have not
included such items in this proposal due to the
logistical requirements, which may be prohibitive. Specifically, each student would need
headphones with a microphone. Testing would
have to be administered on an individual or
small-group level in order to minimize the
noise and distraction in the room (imagine 30
students all speaking at the same time in the
testing setting). However, we mention that we
have this capability in operational use, and we
are happy to negotiate the inclusion of items
that use it if the Department finds the capability attractive.
The availability of tools organized around

response mechanisms creates a very flexible
capability for test developers to create authentic, challenging tasks. The sample items below
illustrate the flexibility that these tools offer
test developers to create authentic, challenging
tasks that probe the full range of depth of
knowledge.
Exhibits 7.4.9.9-1 and 7.4.9.9-10 present sample items illustrating some of the item types
that our system can deliver. Readers may
notice that some of the examples come
from the Smarter Balanced Assessment
Consortiumthese are, in fact, released items
that AIR delivers for Smarter Balanced through
our system. These samples are designed to
illustrate the types of items that we can support. For some examples of the types of items
being proposed for the Department and the
ways that we propose to measure the standards, please see the sample items in section
7.3.1.
As previously mentioned, our scratch-pad

response provides students the tools necessary
to express mathematic reasoning in much the
same way as they might on a paper-and-pencil
test. The very flexible and open-ended nature
of this response mechanism dictates that
responses using it be human scored, at least in
the short term.
Page 7.4-55
Exhibit 7.4.9.9-1: Graphic-Response Item
In this example, students respond by plotting points and connecting lines

to create a triangle. The response is completely machine scored based on
the location of the triangle that is drawn. The scoring assertions allow for
partial credit based on a possible incorrect transformation.
Exhibit 7.4.9.9-2: Equation Response Item
In Exhibit 7.4.9.9-2, students respond by creating an equation using the equation

editor. The functions available to the student on the equation editor are
completely configurable and can include everything from simple operators to
trigonometric functions and integrals. These types of items are machine scored.
Page 7.4-56
Exhibit 7.4.9.9-3: Equation Editor Items
In the constructed-response example in Exhibit 7.4.9.9-3, students respond by

entering the correct number using the physical keyboard or virtual keypad. These
items can be scored either using the equation editor, which requires correct
mathematical input but will match any equivalent form, or using a string-matching
algorithm that enumerates potential correct matches. The latter capability can be
helpful in lower grades where the content may not demand mathematical precision.
Page 7.4-57
Exhibit 7.4.9.9-4: Scratch-Pad Response Item
Scratch-pad responses, as seen in Exhibit 7.4.9.9-4, are generally used for

Mathematics items. Students are given a set of tools that enable them to
integrate drawings, mathematical equations, and words to provide a response.
This very open-ended response mechanism can be effectively used to have
students describe a situation or explain an answer. Scratch-pad responses are
human-scored.
Page 7.4-58
Exhibit 7.4.9.9-5: Matching Item
The item seen in Exhibit 7.4.9.9-5 is a type of selected-response item using the QTI
match interaction. This rendering of the matching item is designed to be accessible to
all students, including those who have limited or no vision. A traditional
representation of a matching item in which students draw a line between matched
elements is not accessible. Here, the relationship is indicated by marking the cell of
the table. The table is accessible to screen readers and is entirely keyboard navigable.
Universal design principles suggest using this rendering for all students.
Page 7.4-59
Exhibit 7.4.9.9-6: Hot-Text Item
In the example seen in Exhibit 7.4.9.9-6, students respond by selecting sentences or

phrases from the paragraph. These items are often used in English Language
Arts/Literacy (ELA/L) to identify evidence for statements within a text. Here, it is used
to express an analysis of an argument in a historical text. These types of items are
machine scored.
Page 7.4-60
Exhibit 7.4.9.9-7: Extended-Response Item
In the example in Exhibit 7.4.9.9-7, students respond by providing a long written

response to be scored using an essay-scoring engine or a mix of machine and human
scoring. Notice that students are offered a variety of writing tools.
Page 7.4-61
Exhibit 7.4.9.9-8: Simulator Interactions Item
AIRs unique simulation technology creates items that enable students to meaningfully
interact with a virtual environment. This interaction can be used as a stimulus for
other items, the interactions can be captured and scored, or both. In the example in
Exhibit 7.4.9.9-8, students are scored on their interactions with a simulator. Students
are presented with several stimuli, which provide the information they need to make
decisions within the task. Students are scored on the sequence of decisions they make
in response to conditions that arise in the simulation.
In this example, students use sliders to indicate allocations of resources. The pie chart
and table to the right are populated in real time based on the students interaction. In
this type of item, students can be scored based on their choices during the simulation,
rather than on the outcome alone.
Page 7.4-62
Exhibit 7.4.9.9-9: Proposition Response
In the example in Exhibit 7.4.9.9-9, students respond by answering a question and

providing details from the text to support their answer. This machine-scorable item uses
our Proposition Scoring engine to recognize correct propositions in the student response.
Page 7.4-63
Exhibit 7.4.9.9-10: Simulator Interactions Item
In this example, students first respond by interacting with a simulator and then by entering an
equation. With the Simulation, students are able to interact with data and provide responses in
varied formats. These responses can be machine-scored. Responses can also be scored based on
the sequence of interactions conducted by the student. Note that this example is entirely
machine-scorable.
Page 7.4-64
7.4.10 Customer Service and

Technical Support
strategic monitoring of all help desk activity through daily, weekly, and monthly
reports;
Easily accessible, prompt, and accurate customer support is essential to success. AIR
commits to providing a toll-free customer support line, chat service, and e-mail for state
users, educators, and administrators Monday
through Friday from 7 a.m. to 8:30 p.m.
Eastern time. AIRs help desk software
(Contactual, 8x8 Inc. software, version 7.0.0)
integrates voice, e-mail, and chat functions. A
user can call the help desk on a dedicated
number, e-mail a dedicated e-mail address, or
initiate a chat with one of our customer service representatives.
a tiered-level approach to escalating and

resolving each query quickly and efficiently; and
The help desk will offer support to Florida educators through e-mail and chat as well as the
toll-free number and will be staffed with agents
who are knowledgeable about the Florida
assessments. The help desk will be available to
answer all questions about Floridas testing
system. Examples of types of support include
installing the secure browser, using the system,
resolving system access issues, and processing
special cases such as test invalidations or test
resets. The help desk staff will be able to identify questions that are matters of policy and
belong to the Department and those that are to
be answered by AIR staff.
Description of Help Desk Operations

Our experience transitioning state departments of education from paper-pencil testing
to online administration has taught us that
easily accessible, prompt, and accurate customer support is essential to success. Our help
desk operations are designed to provide
flexibility to meet peak call times throughout the year;
consistent responses to multiple calls about

the same issue by providing an electronic
repository of FAQ documents, guides, manuals, and other resources that all Florida
help desk agents will refer to. These documents will be developed in consultation
Providing Customer Service during Peak

Times
AIR commits to providing customer service
throughout the year, during peak times and
with additional staff as needed during testing.
From our experience in Hawaii, Delaware,
Oregon, and Minnesota and with the Smarter
Balanced Pilot Test we typically experience
high call volumes three to four weeks before
the test window opens and during the first few
weeks of any test administration window as
test administrators use the system for the first
time. Our preparation for peak times routinely
involves targeted training support for help desk
representatives (this is described in more
detail below) and, when needed, additional
customer representatives are added to support
call volumes.
Monitoring Call Activity

AIR logs all communications to the help desk
regardless of whether an issue is resolved on
the spot by a help desk representative or is
escalated to Tier 2 for further investigation. In
addition to summarizing the communications
in a help desk log, all communications are preserved in their original state. For example,
original e-mails and faxes from the field are
saved and can be retrieved in moments. All
Page 7.4-65
chats are recorded and stored in the case

notes. All phone calls from the field are
recorded, are linked to the help desk case
number, and can be easily retrieved for an
extended period of time if necessary.
ect team generate monitoring reports providing

a range of metrics for the project team to identify issues that may require higher-level expertise to resolve or patterns in the types or frequencies of calls on particular system features.
When the help desk is contacted, a ticket is

opened in the Contactual 8x8 system. The following information is recorded when a ticket is
opened:
For example, in our experience, most help

desk calls are from users who are unfamiliar
with the system and need assistance.
Successful training and the online availability
of thorough system documentation and support in the testing system typically keep these
requests to a minimum. The few help desk
calls that require technical assistance tend to
relate to problems with the users local infrastructure. These include firewall problems and
local network Internet connectivity and are
escalated to AIRs Tier 2 network staff for follow-up and resolution.
A unique, dynamically generated case

number
The callers contact information
The help desk lead
The team member currently assigned (if
the issue is escalated)
Origin (if there are multiple cases regarding
the same topic)
Situation summary
Status of the request (e.g., escalated to Tier
2, resolved)
AIR system (e.g., Test Delivery System,
TIDE)
Local computer information (e.g., hardware, software, network setup, browser)
Supporting documentation is also attached to
the ticket. Additional fields can be collected if
required.
Tiered Approach to Case Escalation and

Resolution
AIR implements a tiered approach for the help
desk to escalate and resolve questions from
callers:
The first tier (Tier 1) consists primarily of
scripted answers approved by the
Department and the resolution of routine
queries by help desk agents. Roughly 85%
of all calls are resolved at Tier 1.
In addition, AIRs Help desk is supported by

highly configurable support center software. In
addition to the case entry and reporting features, the software permits supervisors to
monitor live calls to each representative, monitor the call queue, record calls, review performance metrics in real time, and retrieve
historical case histories sorted by caller.
The second tier consists of a help desk representative escalating a case to a member
of a user support team (Tier 2) for further
investigation. Each team specializes and
has areas of expertise that allow them to
quickly resolve cases assigned to them, e.g.
network support, TDE testing support,
reporting. If the issue cannot be resolved at
the Tier 2 level, it will be escalated to a
senior subject matter expert.
All calls, chats, and e-mails are logged into the

Contactual 8x8 system, where they are monitored daily by help desk management. With this
software, our help desk management and proj-
Tier 3 consists of contacting a subject matter expert, such as a network engineer or a

senior development engineer.
Page 7.4-66
To ensure that cases are resolved quickly and

efficiently at the Tier 2 level, AIR relies on dedicated employees whose function is to coordinate and facilitate the resolution and closure
process for all Tier 2 cases. This position serves
to closely track open cases, working hour to
hour with Tier 2 staff to obtain accurate and
timely information and to secure additional
technical or program management support as
necessary until each case is resolved.
The AIR Help Desk is open throughout the
entire year and will respond to calls received
at any time. AIR will have Tier 1 and Tier 2
staff available beginning not less than four
weeks before and extending one full week after
each test administration. Additional customer
service personnel staff will be available to handle computer-based test administration calls.
The AIR Tier 2 personnel will be skilled and
able to provide advanced technical support
and will be available during the hours that the
help desk is open. The AIR Tier 2 staff will be
immediately available to handle issues that
cannot be resolved by the Tier 1 help desk
staff.
In situations that require information from the
Department, the customer service representative or member of the project team will contact a designated Department staff member by
e-mail and follow up via phone, and this staff
member will locate the answer to the query.
Ensuring Consistent Responses

Ensuring accuracy and consistency in responses from the help desk is critical to establishing
confidence in Florida educators and administrators. To achieve this, all agents are presented with standardized training, tools, and
scripts to provide consistent responses. First,
agents are provided with easy-to-use scripts
that guide these agents to ask each caller the
questions necessary to diagnose the issue
quickly. In situations in which the case
requires escalation the agent will gather the

information necessary and inform the caller
that the issue will be escalated to the next
level of expertise, Tier 2.
Customer service staff will document all communications in a log and furnish it daily to the
Department during each administration or on
request as needed. Under the guidance of the
Department AIR will develop a call topic
matrix consisting of a set of scripts to ensure
that calls are responded to in a consistent and
appropriate manner.
In addition, each agent has access to a statespecific document repository. In responding to
queries from Florida educators and assessment
coordinators, all Florida help desk agents will
access the Florida-specific FAQs, manuals, user
guides, and training materials and will use them
in their responses as appropriate. To ensure
that agents are providing current information,
the FAQs will be updated regularly by the project team, approved by the project director, and
provided to the Department. The four-part
training program completed by all help desk
agents (described in the section below) further
ensures consistency in responses.
Training and Supervision of Customer

Service Representatives at the Help
Desk
AIRs training of customer service representatives at the AIR help desk is based on our
experience in assisting new users in in Hawaii,
Delaware, Minnesota, Oregon, Ohio Online
Pilots, Smarter Balanced Small-Scale Trials,
and Smarter Balanced Pilot Tests with varying
degrees of technical knowledge in the use of an
entirely new online testing system. Our past
experience has shown that callers may not
know enough about the online testing system
to articulate the specific system issue they
have encountered. Therefore, our customer
service agents must be able to guide callers
Page 7.4-67
through a series of questions designed to diagnose the issue quickly.

In addition, our past experience reveals that
many callers simply want someone to walk
them through the step-by-step instructions
provided in the system documentation. AIRs
customer service agents are trained to provide
this level of support to educators and administrators wherever required.
Finally, analysis of past help desk logs shows
that customer queries tend to be divided into
three distinct categories corresponding to
three stages of the online administration:
1. Questions about online testing preparation
(these include questions about scheduling
online test sessions, training test administrators on online procedures, and installing
secure browsers)
2. Questions during the start-up or beginning
of the test window (these include queries
about using the system, setting accommodations, understanding participation
counts, and understanding how to create
class rosters)
3. Questions during the ongoing administration (which tend to include resolving questions about system access, test invalidations, test resets, and resolving issues
involving network latencies)
Based on the type of customer queries we typically receive, AIRs customer service representatives must complete a rigorous four-part
training program that involves the following:
1. Training on the functionality of each component of the online testing systems as well
as the paper testing program including the
policies and procedures set forth in user
guides and manuals
2. Training based on past logs of customer
queries taken from each stage of an online
administration: preparation, startup, and
Page 7.4-68
ongoing administration
3. Training on the use of detailed scripts that
enable the customer service agent to ask
callers a series of questions and thereby
quickly diagnose the system and point of
issue with which the caller is experiencing
difficulty
4. Project-specific training on each aspect of
the states assessment program (All Florida
help desk representatives and Tier 2 staff
will complete Florida-specific, project-specific training sessions covering all Florida
assessments. The sessions will outline commonalities and differences across each
Florida assessment and highlight the major
test administration requirements of each.)
In addition to providing rigorously trained customer service agents, AIRs help desk is supervised and monitored by a full-time management staff responsible for monitoring response
accuracy. Supervisory staff audits calls to the
help desk to ensure that help desk agents are
providing accurate information. When a question of possible inaccuracy arises, the help desk
management staff will obtain a copy of the
phone recording or chat log and can confirm
whether the information provided was correct.
AIRs policy is for the project director to report
any system-wide issues directly to the
Department.
Help Desk Quality Assurance Plan
By closely monitoring the resolution time metrics, the project team can identify how quickly
our agents and Tier 2 support are providing
answers to the field. As explained below, the
daily summary report allows the project team
to pinpoint exactly which cases associated
with which system and team are taking longer
to address.
In addition, the daily help desk report can
function as an early warning system pointing
to an area about which assessment coordinators may need more clarification. For example,
if there are a series of calls about performing a
test reset or obtaining a new password, this
can alert the project team to the need for further explanation. In such situations, the project director will alert the Department of the
issue. Together, the project director and
Department staff can determine whether a
clarifying document or communication is
required.
An integral component of AIRs help desk quality assurance plan centers on the project team
analyzing the metrics provided in detailed daily
and on-demand help desk reports. The
Contactual 8x8 system uses job tickets as the
primary method of tracking issue types, status,
and individual client trouble calls and e-mails,
and the AIR help desk reporting system
includes current aggregate status information
for various resolution types, as described below.
Quality assurance for help desk support
requires daily monitoring of all help desk
activity by AIRs project team, the help desk
manager, and the help desk supervisory staff.
The daily report provides much of the information necessary to make this possible,
including the following:
The length of time the case has been open

Whether the incident caused a student test
session to be interrupted
The date and time the issue was resolved
Status (whether the agent is waiting for a
call back from the caller; whether the case
is under investigation; whether the team is
waiting for additional documentation from
the user)
A sample of the help desk daily report follows
this subsection as Exhibit 7.4.10-1.
The project team also manages the test administration by reviewing the case analysis summary report, which includes the following
information:
The volume of cases open and closed for
the requested time period
The number of cases assigned to specific
systems
The number of cases assigned to specific
Florida tracking categories, which include:
The average and median time it takes to
close a case
The longest time a case has remained
open
The date and time of each call, chat, or

e-mail
The shortest time it has taken to close a

case
The name of the Tier 1 agent taking the

call and/or providing resolution
The percentage of cases that have been

escalated to Tier 2
A case tracking number
The percentage each category contributes to all cases
The name of the team to which the issue

has been escalated to (where appropriate)
A description of the issue
The caller contact information
A detailed description of the activity taken
by the agent and/or the next level of support to resolve or investigate the issue
The specific Tier 2 support team that

currently holds the case, allowing the
project director to see where the bulk of
cases currently reside
A sample of the Case Analyses Summary is
included at the end of this subsection in
Exhibit 7.4.10-2.
Page 7.4-69
In addition our help desk staff and project

director have access to the following:
Average hold times by day and hour
Number of dropped calls
Average time on hold before calls were
dropped
The project team uses this information to
quickly identify the stage or status of open
cases, which system those cases involve, how
long the cases have been open, and the best
way to facilitate speedy resolution. AIR will
provide the Florida staff with detailed help
desk reports daily or at preset time intervals.
The reports will include the information
described above to help inform Florida staff of
the help desk activities.
The project team uses these reports to track
types of calls and ensure timely resolution of
all issues. During AIRs first year transitioning
four separate Minnesota assessment programs
to the new online testing system, our help
desk received approximately 10,000 calls from
Minnesota educators and administrators.
Approximately 80% percent of these cases
were resolved during the initial call or e mail
to the help desk. Our help desk opened and
logged more than 21,000 cases in support of
the Smarter Balanced pilot administration and
approximately 85% were resolved at Tier 1.
In addition, the AIR help desk reports can

function as an early warning system, pointing
to an area in which assessment coordinators
may need more clarification. Together, the
project director and the Florida staff can determine whether a clarifying document or communication should be provided for the field.
References
The Center for Universal Design. (1997). The
principles of universal design, Version 2.0.
Raleigh, NC: North Carolina State
University.
Thompson, S. J., Johnstone, C. J., & Thurlow,
M. L. (2002). Universal design applied to
large scale assessments (NCEO Synthesis
Report 44). Minneapolis: University of
Minnesota, National Center on Educational
Outcomes. Retrieved from http://educaton.umn.edu/NCEO/OnlinePubs/Synthesis4
4.html
Florida Department of Education. (2007).
Technology resources survey statistical
totals report. Retrieved from
http://www.flinnovates.org/survey/homepage-stats-select-12.php
By closely monitoring the resolution time metrics, the project team can identify how quickly
our Tier 1 and Tier 2 support teams are providing answers to the field and resolving open
cases. As previously explained, our summary
reports allow the project team to pinpoint
exactly which cases are associated with which
system and what type of cases are taking
longer to address. The project director and the
help desk manager work closely together to
decide the best way to facilitate an immediate
resolution.
Page 7.4-70
Exhibit 7.4.10-1: Help Desk Daily Report
Page 7.4-71
Exhibit 7.4.10-1: Help Desk Daily Report (continued)
Page 7.4-72
Exhibit 7.4.10-2: Sample Case Analysis Summary
Page 7.4-73
Exhibit 7.4.10-2: Sample Case Analysis Summary (continued)
Page 7.4-74
7.5.0 Scanning, Scoring, and Reporting
7.5.0 Scanning, Scoring, and

Reporting
In this section of our proposal we present our
plans for scanning, hand-scoring, test scoring,
reporting, data cleanup, and delivery. The following individual sections describe the details
of our plans, but we would like to call your
attention to four aspects that will provide significant value to the state of Florida:
We have partnered with Data Recognition
Corporation (DRC) for the scanning and
hand-scoring aspects of this contract.
Among the test processing organizations
with sufficient capacity to process the
Departments tests on its tight timelines,
DRC is the only organization to have
received International Organization for
Standardization (ISO) certification for all of
its testing processes. This certification
ensures the Department that DRC implements replicable processes for every phase
of test processing, offering a level of quality
unsurpassed in the industry.
In addition to PreID processes, we offer a
real-time-ID process for mobile students,
allowing every student to have an identifying
label. This capability offers improvements in
data quality that result in vastly reduced
timelines and enables us to begin data
cleanup even before the tests are complete.
We offer dual-mode reporting, offering
paper to parents and online reports (even
real-time online reports at your discretion)
to educators. Interactive online reports can
convey infinitely more information than
paper or fixed-PDF reports, enabling educators to explore the implications of their
data for their students, slicing and dicing
the data at will.
Our report design has involved cognitive
scientists, psychometricians, and communication experts to ensure that our indus-
try-leading, full-color reports convey information in a way that is clear and appropriately interpreted by the intended audiences. We have appended samples of our
paper reports from Hawaii, including annotated screenshots of our online reporting
system, in Appendix G.
We hope that the Department agrees that our
innovative plan, drawing on technologies and
approaches that have succeeded elsewhere,
will exceed its requirements and help the
Department continue its track record of continuous improvement.
7.5.1 Scoring and Reporting

Specifications
Introduction
The Departments process for documenting
scoring and reporting decisions follows almost
identically with AIRs standard recommended
process. AIR typically develops three documents that are relevant to operational scoring
and reporting:
1. Handscoring specifications, which detail such
processes as backread rates, reliability read
rates, target reliabilities, validity paper relative frequency, and rescore criteria. These
specifications are detailed in Section 7.5.4.5.
2. Analysis specifications, which detail the
specifics of the calibration and equating
process that will yield the item parameters
and the transformations that will place the
scores on the Florida Standards Assessment
(FSA) scale. These specifications, which
also include item analysis details, are discussed under Section 7.6.2.
3. Reporting specifications, which detail all of
the aggregations, transformations, filters,
visual displays, and other characteristics of
the reporting. The reporting specifications
are detailed in this section.
Page 7.5-1
While the ITN refers to scoring and reporting

specifications, we have found that breaking
the document into three parts simplifies
review and maintenance. All three documents
require the attention of project management
and psychometricians. The handscoring document requires input and review by the scoring
team, and the reporting specifications require
input from the reporting team with more
expertise in communication. We hope that the
Department sees the value in this modularization of the specifications document. Of course,
all three parts can be integrated into a single,
three-part document at any point.
AIR has a long history of successfully developing and maintaining scoring and reporting
specifications to ensure that final reports and
data files are produced accurately and on time.
We rely on our specifications documents to
capture detailed business and processing rules
that enable AIR staff to conduct critical quality
assurance processes far in advance of receiving
final data. By the time we receive final data
from online and paper tests, all programming
and system checks have been verified, independently replicated, and are ready to use.
Our many automated procedures, including
quality and data integrity checks to ensure
quality of online and paper tests, scoring
rubrics, data, and other products ensure that
we are able to meet compressed timeframes
and deliver final data files to our clients while
maintaining 100% accuracy.
Should the Department decide to use AIRs
online reporting system to provide data and
reports to Florida schools and districts, AIR
staff will work with the Department to finalize
the online reporting specifications to configure
the online reporting system, as well as specifications for the paper-report and scoring specifications. In this section we describe
our proposed process for completing the
specifications documents;
Page 7.5-2
the detailed rules we propose to include in

the scoring and reporting specifications
along with sample excerpts; and
the quality assurance processes that rely
on detailed scoring and reporting specifications to ensure accurate, on-time data file
deliveries to our clients.
Proposed Process for Producing

Scoring and Reporting Specifications
AIR will work with the Department to draft and
finalize all reporting specifications documents
required for the English Language Arts/Literacy
(ELA/L) and Mathematics assessments and the
End-of-Course (EOC) assessments. To facilitate
this process, we propose conducting an on-site
working session with Department staff to capture detailed business rules for each document.
For example, to prepare for the working session
to finalize the online reporting specifications,
AIR staff will identify the decisions that the
Department is required to make to configure
each part of the online reporting system e.g.,
the Test Management Center, downloadable
data files, definitions of measures, home page
data displays). The project team will send the
list of required decisions to the Department in
advance of the meeting. During the working
session, participants will proceed through the
list to obtain decisions on each point. To provide context to this discussion, AIR staff will
project images of the online reporting system
that correspond to the specific rules being discussed. This will help Department staff visualize the impact of their decisions on the reports
appearing in the online reporting system. Once
the Department approves the specifications,
AIR programmers will use the specifications
document to configure the online reporting system and to conduct quality assurance checks
to ensure the system is functioning as required.
To produce the 201415 scoring and reporting
specifications for paper reports, AIR will use
the 201314 scoring and reporting specifica-
tions provided by the Department along with

the list of the Departments updates outlined in
the ITN.
Proposed Plan for Completing the

Each Spring
The first draft of the scoring and reporting
specifications documents will be produced
each April as required by the Department.
Project staff will be responsible for maintaining
an addendum with updates during the year.
AIR will submit the final specifications addendum at the same time as the last closeout file
of student results for that year.
We recognize that April and May are typically
high-activity months for the scoring and reporting staff and that, if not planned well, work on
updating specifications documents scheduled
for this timeframe can often be delayed. To
mitigate these concerns, AIR will assign separate reporting staff members to the operational
tasks and the planning/specifications tasks:
ELA/L and Mathematics reports for educators must be delivered between the end of
April and mid-May 2015
ELA/L and Mathematics paper reports for
families are to be delivered by June 8th, 2015
Spring EOC educator reports must be delivered by the first week of May, 2015
Spring EOC paper reports for families must
be delivered by mid-May 2015
Despite the consecutive reporting deliverables
scheduled for May, we are confident that project staff will be available to update the scoring
and reporting specifications in April. AIRs integrated quality assurance (QA) systems have
made it possible to automate many of the scoring and reporting QA processes and to conduct
these QA steps far in advance of receiving the
final data (see Section 7.5.5.1 for a detailed
explanation.) As a result, key project staff,

including the reporting manager and the lead
psychometrician, are able to focus on updating
scoring and reporting specifications for the
upcoming year. In addition, AIRs scoring and
reporting specifications process does not
require participation from programmers and
QA staff, who can instead focus exclusively on
conducting last-minute regression tests on system functionality. Additionally, our experience
has been that after the first year of the contract, updating online scoring and reporting
specifications typically requires much less time
than during the first year of the contract when
systems need to be configured from scratch.
Next we present an overview of our scoring
and reporting specifications documents and
samples of the detailed rules that these documents contain.
Overview of Document Content

The online reporting specifications document
defines the rules that govern system configuration, including table layouts, user-role access,
definitions, reporting rules, and content hierarchies. Exhibit 7.5.1-1 provides a summary of
the topics covered in the specifications along
with a description of online reporting rules.
In addition to the online reporting specifications discussed in Exhibit 7.5.1-1, the reporting and scoring specifications specify rules for
aggregating data, merging data, and handling
special populations and discrepant records.
Typically the reporting specifications include
the following sections:
Scope and Project Information (includes a
summary of deliverables by Assessment
and Administration)
Schedule and Key Dates
Schedule of Data Extracts
File Naming Conventions
Page 7.5-3
Exhibit 7.5.1-1: Summary of Topics Covered in Online Reporting Specifications

Topic
Description of Online Reporting Rules
Basic Information
Includes information to configure the home page of the online reporting system, specifies test instrument, grade,
subject/course, and administration labels for the online reporting system.
Opportunities Per Test
For adaptive assessments, indicates the total number of opportunities available for each test to be reported in the online
reporting system.
Definitions of Measures Reported
Includes calculation, aggregation, and exclusion rules for each measure. Examples of measures might be scale scores,
conditional standard errors of measurement (SEM), performance levels, strand scores, strand- and benchmark-level
strengths and weaknesses indicators, Lexile scores.
Reported Measures
Identifies for each grade, subject, and program the measures to be reported.
Reporting Subskills
Specifies all reporting categories/strands, clusters, and benchmarks/standards reported in AIRs online and paper reports.
Reporting Settings
Defines the labels and columns to be configured for each online reporting system report.
Theta to Scale and Cuts
Describes the detailed scoring rules for online and paper reports.
Test Codes and Condition Codes
Specifies all rules governing test and condition codes for processing and reporting.
Attemptedness Rules
Defines rules for attempted, incomplete, and not attempted tests.
Rules for Multiple Records
Specifies the reporting rules for situations where students can potentially take more than the allowed number of tests.
For example, in a single opportunity test, it may be possible for a student to complete two opportunities, one online and
another on paper. These rules specify which score will be selected for final reporting.
Performance Level Descriptors
Defines all PLDs reported in the student report.
Data Processing Rules (including demographic and score data files required to produce paper reports, rules for handling
invalidations and discrepancies, rules for
data cleaning, rules for merging data files,
rules for late-batch data processing)
Rules for Special Populations (includes offgrade testers, home-schooled students,
community schools, parent-exempt students, accelerated students)
Rules for Constructing the Data for Paper
Reports
Aggregation rules
Exclusion rules
Suppression rules
Page 7.5-4
Roster inclusion rules

Address files
Truncation rules
Rules governing variable text (which text
appears for test codes)
Rules for missing data fields
Scoring Rules (includes attemptedness
rules, cut scores, calculation rules)
The scoring and reporting specifications contain
highly detailed calculations and statistical rules
that allow our programmers and statisticians to
set up and test their programs far in advance of
our receiving final data. Exhibit 7.5.1-2 presents
a few excerpts to demonstrate the specificity
Exhibit 7.5.1-2: Excerpted rule detail from some reporting specifications

Computation Rule: Theta Score (from Delaware scoring specifications)
The Delaware Comprehensive Assessment System (DCAS) test is based on the Rasch family of Item Response Theory (IRT) models. Multiple-choice items are scored via the
Rasch model and the polytomous items are scored via Masters Partial Credit Model (Masters, 1982).
Likelihood Function
The likelihood function for generating the maximum-likelihood estimator (MLE) is based on a mixture of items types and can therefore be expressed as:
,
where
exp
exp
where is the lower asymptote of the item response curve (i.e., the guessing parameter), is the slope of the item response curve (i.e., the discrimination parameter),
is the location parameter, is the observed response to the item, i indexes item, and is the kth step for item i with m total categories.
We subsequently find
as the students theta (i.e., MLE) given the set of items administered to the student. Because this test is adaptive, each student
sees a unique set of items; hence, no conversion table is used for scoring. Because this is a Rasch-based model, the following constraints are imposed for Delaware:
.
Standard Error Transformation

Standard errors of the MLEs must also be transformed to be placed onto the same reporting scale.
This transformation is:
The DCAS Reading and Mathematics tests contain test items measuring both on-and off-grade content. Science and Social Studies tests will contain only on-grade items.
Multiple scores are estimated for each student, including two total scores (Reading and Mathematics only) and scaled scores for each reporting category.
Let
represent the complete set of items presented to student s including the on-grade and off-grade items;
Let
represent the subset of test items presented to student s including only the on-grade items; and
Let
represent the subset of items presented to student s in reporting category q including both the on-grade and off-grade items.
Rules Governing Records With Test Codes

The VS test code in the Publication File stands for Valid Score and should not be reported in the test_code field in superdata. For test codes values of INV, ABS, REF, ME,
WG, NC, NA, all fields should remain blank in superdata.
If test_code_r= INV, ABS, REF, ME, WG, NC, NA, or NE, then ensure the following values are blank: achievement_level_r, scale_score_r, raw_score_r,
max_score_r, Lexile_score, Lexile_lower, Lexile_upper.
If test_code_m= INV, ABS, REF, ME, WG, NC, NA, or NE, then ensure the following values are blank: achievement_level_m, scale_score_m, raw_score_m,
max_score_m.
If test_code= WG INV, ABS, REF, ME, NC, NA, then ensure the following values are blank: achievement_level, scale_score, raw_score_test, max_score_test,
Lexile_score, Lexile_lower, Lexile_upper.
If GRAD record test_code_grad= WG, INV, ABS, REF, ME, NC, NA, then ensure the following values are blank: scale_score_grad, raw_score_grad,
max_score_grad.
For high school MCA-II, if MCA test_code= WG, and GRAD test_code= VS Publication File, then:
For MCA record values: ensure subject level values are blank: achievement_level, scale_score, raw_score_test, max_score_test.
For GRAD record values: ensure that in superdata: scale_score_grad, raw_score_grad, max_score_grad.
Page 7.5-5
that allow us to conduct rigorous quality assurance early on. Excerpted rules include:
Computation rules for theta score and likelihood function (from Delaware scoring
specifications)
Standard error transformation rule (from
Delaware scoring specifications);
Rules governing records with test codes
(from Minnesota)
We propose to work with the Department to
capture the same level of detail in developing
scoring and reporting specifications for the
ELA/L and Mathematics Assessments as well as
the EOC Assessments. Next we describe how
these detailed specifications documents are
used to conduct QA procedures early on,
assuring clients of error-free, on-time data files
and reports.
Quality Assurance Processes That Rely

on Scoring and Reporting
Specifications
The purpose of capturing highly detailed specifications is to enable our programmers and
statisticians to set up and test their programs
before we receive final data. We have refined
our specifications and quality assurance
processes significantly so that when operational data is received, the majority of issues
have already been identified and resolved.
Below we describe these QA processes in more
detail, including how they rely on detailed data
layouts and specifications documents.
Quality Assurance Process for Data

Processing and Psychometrics
Our quality assurance processes begin with
our technical staff developing a data layout in
which to receive scored data from our processing partner. AIR has developed an electronic
layout format that drives a variety of pro-
Page 7.5-6
grams, including programs that generate artificial data (used to test analysis programs) and
programs that read or validate the incoming
data. All of our statistical analyses that require
raw data access the data by reading the same
electronic layout file, eliminating any possibility for human error from cutting and pasting
data layouts into different programs or version
control problems created by updates to the
data layouts.
With the data layouts defined, AIRs computer
scientists generate simulated data files that are
designed to model known and anticipated
characteristics of the assessment data, as well
as cases designed to evaluate the performance
of the analysis, scoring, and reporting software
under extreme or unusual conditions. These
simulated data files are used to test all software supporting operational test analysis, scoring, and reporting, including performance of
the data layout generator, data analysis software for computing classical item statistics
and calibrating and equating test items, and
conversion table generation, as well as all scoring and reporting software.
The software is tested and refined until it is
error-free. This process happens well in
advance of any operational data becoming
available. Our test data generator is able to
generate challenge cases, including inconsistent data, missing data, out-of-range data, and
other anomalies that mimic real-world complications. This testing and refinement phase
ensures that we have anticipated any problems
and have plenty of time to deal with them,
should they arise, before the operational data
arrive.
When the operational data arrive, we have all
analyses replicated by two analysts to ensure
that the programs are applied correctly. Any
discrepancy in their results is thoroughly
investigated and resolved. Resulting estimates,
such as item parameters and conversion
tables, are compared with historical expectations in the belief that substantive changes
may signal a problem.
Quality Assurance Process for Score
Reporting
Online reports can appear in real time if the
Department chooses. Therefore, quality assurance does not rely on post-hoc reviews of
reports. Instead, the accuracy of the reporting
system rests on the quality of programming of
the system, the implementation of the
Departments business rules, and the quality of
the algorithms used for aggregating scores.
In building our systems, we undergo an extensive software testing process, and when we
configure the systems for individual clients, we
test them by running simulated or real data
through the full system, allowing the system to
generate reports. Our statistical programming
team simultaneously implements the same
rules and processes the same data. Statistics
from the system are then compared to statistics produced by our statistical team.
Discrepancies are tracked down and resolved.
As described in this section, this entire
process is guided by a set of complete scoring
and reporting specifications. Both the software
team and the independent statistical programming team work from the same specifications,
but each works independently. This process
provides an independent check on the immediate reporting system before it is deployed. Of
course, the Department is welcome to review
the data at any time.
Quality AssuranceStatistical
Programming
The family reports we propose to develop for
the Department contain custom programming
and require rigorous QA processes to ensure
their accuracy. All custom programming is
guided by the detailed and precise specifications in our reporting specifications document
previously discussed in this section. Upon
approval of the specifications, analytic rules
are programmed and each program is extensively tested on test decks and real data from
other programs. The final programs are
reviewed by two senior statisticians and one
senior programmer to ensure that they implement agreed-upon procedures. Custom programming is implemented independently by
two statistical programming teams working
from the specifications. Only when the output
from both teams matches exactly are the
scripts released for production.
7.5.2 Preparing School and Student

Information
Introduction
One of Floridas long-held goals has been to
move the testing window closer to the end of
the year, extending the period during which
students learn before testing. While it may be
hard to imagine, the seemingly mundane
process of gathering student registration data
can have a substantial impact on these timelines. Cleanup of student data and redirecting
reports to the right schools takes weeks under
the current process. AIR and our partner,
DRC, propose a process that will shorten these
timelines while improving data quality. Any
time saved cleaning things up on the back end
can be reallocated to allow testing later in the
year.
Our plan will make paper testing almost as
efficient as online testing, vastly diminishing
end-of-window cleanup and the reconciliation
process. We begin by contrasting typical online
programs from their paper counterparts.
Online programs must authenticate students,
and doing so requires that they have reliable
student information before testing. When
Page 7.5-7
everything is done online, adding a student to

the roster at the last minute is a simple matter.
The Challenge
Until now, this has not been the case with
paper tests, which, of course, must be printed
and shippedmanufacturing activities that
take some time. To pre-identify the booklets
(or labels) with student information requires
that the information be known early enough to
print and ship. Students move around with
some regularity, resulting in approximately 10
percent of the tests not being pre-identified.
These students are accommodated using blank
booklets, with their identifying and demographic information being bubbled by school
or district personnel.
Under a typical system the student information is not discovered until the booklets are
returned to the processing center and scanned.
The weeks that the answer documents are in
transit or being processed are weeks that cannot be used to clean up the data. The bubbling
process ensures that the cleanup required will
be extensive because hand-gridding the demographic information is tedious, labor intensive,
and error prone. These errors must be individually resolved once the data become available.
Our Solution
The delays described above do not occur in
most online programs. The reason for this is
that the online programs must authenticate
students before testing. The need to authenticate students leads to processes in which students are properly and positively identified
before testing.
AIR and our partner, DRC, propose to eliminate bubbling on answer documents. Instead,
our online registration system, the Test
Information Distribution Engine (TIDE), will
allow schools and districts to add students to
Page 7.5-8
the registration system in exactly the same

way that students are added to our online registration system. Once added, the user will be
able to print scannable pre-ID labels, on label
stock provided along with the users booklet
shipment, using any laser printer. These labels,
which might better be called real-time-ID
labels, provide a mechanism to register students who show up at the last minute.
Furthermore, once real-time-ID information is
entered, it is available in cleanup systems, so
that demographics, student-school associations, and other key data can be verified during the testing window rather than afterward.
AIRs TIDE system allows the real-time-ID
information to be entered individually through
an interface or through a file upload matching
the pre-ID file format. The information is subject to the same set of validations as pre-ID
files. From a policy perspective, this process
depends on the state requiring each student
to have a valid statewide ID prior to testing,
which is a typical requirement for online programs as well.
Overview of Proposed Plan

AIR maintains a single, integrated system for
managing test administrations, whether online
or paper. This system, TIDE, is described in
detail in Section 7.4.8.
TIDE interfaces with our Roster Tracking
System database, which maintains information
about each entity in the educational system,
its attributes, and the relationships among
these entities, which include the state, districts, schools, teachers, classes, and students.
We propose to populate the district and school
information directly from Department-supplied
lists.
TIDE accepts inputs or uploads from authorized users. Hence, for example, district users
may upload pre-ID files. Upon upload, the files
undergo both syntactic and semantic validations, which can be customized to Floridas
needs. Typical validations include confirming
valid district and school identifiers associated
with students; valid values for names, grades,
etc.; comparisons to historical data; and other
validations. Our team will work with the
Department to identify and document the
required set of validation rules.
We generally open the TIDE window well
before testing, and leave it open beyond the
testing window. To support the printing of preidentified test booklets, we will work with the
Department to establish one or more dates on
which we will take a snapshot of the data to
compose the pre-ID files.
Throughout the testing window, TIDE offers
the ability to check and revise student demographics, student-school associations, reporting schools, or other information that may
change or require correction. Districts can
view the student information directly through
TIDE, and can implement corrections either
through file uploads or through the interactive
user interface.
As described above, this system enables districts and schools to easily accommodate late
arriving students, even for paper testing.
Where paper tests are in use, users will be able
to print out real-time-ID labels on AIR-provided label stock.
By eliminating the bubble-scan-cleanup loop,
our process can foreshorten the processing
time after the testing window. In fact, most of
the cleanup can be completed online before
the testing window even closes. Many states
establish a date after which cleanup changes
are no longer accepted, and TIDE can enforce
such dates, pulling the final data file immediately thereafter.
We note that TIDE also supports appeals and

invalidations, which can be requested by
authorized users through the same interface.
Specific Requirements
The process described above meets or exceeds
all of the requirements set forth in the RFP. We
discuss each of the enumerated requirements
briefly below, and describe how it is accomplished under our proposed system.
1. Update Student Information between

Scanning and Reporting
The contractor is required to ensure that
school names and identifiers are correct following scanning. Much of this sort of cleanup
is needed as the result of bubbling errors,
which our process eliminates. We solve the
problem of invalid school names and IDs by
preventing their introduction in the first place.
Other cleanup is needed if districts or schools
do not indicate all of the appropriate school
relationships. For example, a student may be
enrolled in one school, but attending an alternative school. In this case, both pieces of
information would have to be entered into
TIDE, either through a file upload or through
an interactive interface. During the cleanup
period (which can happen during the testing
window), schools or districts can check to see
that they are slated to receive the appropriate
reports, and if not, they can add or correct the
necessary data.
AIR staff will be available to assist in this
review and cleanup.
2. MSID Validation
We propose to pre-load TIDE with Floridas
official MSID data. We will accept updates only
from the Department.
Page 7.5-9
3. Initial and Updated Pre-ID

Information
As described above, we propose to allow continuous updating of the pre-ID information by
districts. We will work with the Department to
establish one or more formal pre-ID dates on
which snapshots will be taken to pre-print the
books or labels.
Our real-time-ID process will enable all
schools and districts to immediately obtain
real-time labels through TIDE, eliminating the
need to bubble information on the test booklets.
4. Pre-ID Windows and Business Rules

for Processing File Uploads
AIR will implement the Department-established snapshot dates for pre-identification of
students. As described above, the file uploads
are subject to configurable validation rules.
AIR staff will work with the Department to
document the file validation rules, rules for
accepting or rejecting records or files, and the
final method for pre-identifying the answer
documents.
5. Pre-ID File Layout

AIR will work with the Department to establish
an appropriate file layout and update it as
required for each administration. We note that
we are able to include a wide variety of information in the pre-ID files, including the
required accommodations.
6. Additional Steps to Ensure the

Accuracy of the Pre-ID File
AIR will work with the Department to ensure
that all of the necessary validation checks and
processes are complete. These include the
following:
Page 7.5-10
We will ensure that only valid, active

MSIDs appear, the process for which is discussed above.
We will ensure that the data are valid and
reasonable, which we accomplish through
customized validation rules to be specified
Two days before the snapshot dates we will
contact districts that appear not to have
submitted complete pre-ID data.
The TIDE system provides real-time feedback to help the district user evaluate the
accuracy of the data received, including
frequencies, numbers of errors, and other
configurable information.
We propose to allow districts to amend
their pre-ID information at any time.
AIR will deliver the pre-ID snapshots to the
Department, and the snapshots will include
a pre-ID number for each student. That
number will be unique to the student for
the administration, and will not be repeated in other administrations.
If the pre-ID information is not correct,
school or district administrators will be
able to change them in TIDE, and the information active in TIDE on a designated date
will be merged with the score data prior to
reporting, eliminating the need to grid
school codes or other information on
answer booklets.
Test and answer documents will be sent to
the school of record, or the alternative
school if one is provided. If a pre-identified
booklet is not available for a student,
administrators will be able to correct the
information and print out a new, scannable
label from TIDE.
Pre-ID rosters will be printed and will

accompany the shipped test booklets and
answer documents.
Our real-time-ID and data cleanup processes will ensure that every student testing on
paper has a labeled booklet, and that the
data are accurate in TIDE.
7.5.3 Scanning/Editing PaperBased Tests (PBTs) and Extracting

Computer-Based Test (CBT) Records
AIRs automated data intake, processing, and
extract systems are designed for high-volume
efficiency and accuracy. Results for both CBTs
and PBTs are naturally integrated by the system in real-time as test results are processed.
Extract files that contain student data associated with each test event (administered online
or on paper), test scores, and raw and scored
student item responses, which include the
exact responses entered by students, will be
made available to the Department.
This section describes the scanning and editing procedures for PBTs, the data flows for
both CBT and PBT results, and how PBT
results are naturally integrated with CBT
results. We also present our proposed
approach for delivery of extracted CBT and
PBT results.
DRC Processing and Scanning Statistics

More than 13 million secure materials
received and processed annually
More than 250,000 materials
processed daily
25 in-house IBML Image Trac
scanners
Weekly scanning capacity in excess of
19.3 million sheets
Scanning and Editing Paper-Based

Tests
DRC believes that its combined experience
and expertise in the areas of materials processing, image scanning, and performance assessment scoring are unsurpassed in the industry.
DRCs 30-plus years of experience have resulted in a scannable document process that is
extremely reliable and efficient for its state
testing clients; DRCs processing and scanning
processes are ISO 9001:2008 certified. The
company takes pride in its ability to tailor
processes to meet each of its clients needs.
DRC has successfully processed millions of
scannable test materials for large-scale
statewide assessments, including the following
states: Alabama, Alaska, Louisiana, Nebraska,
Ohio, Oklahoma, Pennsylvania, South
Carolina, and Washington. All of the companys statewide assessment contracts utilize
image scanning and scoring.
Processing and Scanning Resources

and Capacity
One of the most experienced testing contractors in the nation, DRC scores and reports
assessment results for more than 4.5 million
students across the country. All processing and
scanning occurs at fully secure facilities in
Brooklyn Park, Minnesota, conveniently located within a few miles of DRCs corporate headquarters. DRCs materials processing and scanning facilities are custom-designed to support
the companys high-volume operational capacity.
DRCs scanning capabilities include image
scanning and scoring, barcode scanning, and
optical character recognition scanning. The
companys state-of-the-art, proprietary scanning system is highly configurable and fully
scalable, which provides the flexibility needed
to accommodate each of its state clients
needs.
Page 7.5-11
Using a patented process, DRCs 25 in-house

IBML ImageTrac scanners scan both single- or
multiple-color documents. The image scanners
used by DRC utilize high-quality color cameras
capable of capturing images up to 300 dpi.
Grayscale images are captured using 256-level
grayscale (8 bit per pixel) format. Customized
scanning programs selectively read documents
and electronically format scanned information.
All custom scanning programs go through quality review before testing materials arrive.
DRCs image scanning operators have extensive experience performing tasks related to
scanning and the maintenance of image scanning equipment. DRCs proposed operations
lead staff has extensive direct experience with
multiple statewide assessment programs.
Please see Section 2 for complete profiles and
rsums of DRCs scanning and scoring professionals, as well as DRCs Project Manager, Mr.
Peter Tressel.
Providing the capacity for growth is a cornerstone of DRCs business model. The company
is dedicated to funding the latest hardware and
technology to ensure that it meets and exceeds
its operational requirements. Its technical
environment has grown substantially over the
past few years and continues to expand.
Constantly monitoring its usage throughout
the technical environment, DRC routinely
adds application servers, database servers, and
other technical capabilities to stay ahead of
the needs of its clients programs.
DRCs scanning capabilities include image
scanning and scoring, barcode scanning, and
optical character recognition scanning. DRCs
weekly capacity is more than 19.3 million
sheets. DRC has evaluated the scanning capacity required for the Florida Standards
Assessments (FSA) and is confident that its
processing and scoring capacity is more than
adequate to meet the needs of the FSA as well
as the companys current contracts.
Page 7.5-12
Verification of Return Procedures

DRCs state-of-the-art proprietary processing
and scanning systems are highly configurable
and fully scalable, which provides the flexibility needed to accommodate each of its state
clients needs. Quality-control (QC) processes
are embedded into each phase to ensure accurate processing, scanning, and data capture.
DRCs materials management system, Ops
MMS, enables 100% accuracy in processing and
accounting for returned barcoded materials.
Ops MMS uses barcode technology to track
documents through all processing steps: box
receipt, materials check-in and separation, and
data validation. Please see Section 7.4.1 for
more information on processing and accounting for returned student test materials and a
discussion of documenting return errors.
An additional benefit of Ops MMS is that DRCs
procedure for materials processing does not
need or rely on header sheets to accurately
account for all secure materials and track all
data elements needed for reporting. AIR and
DRC have a proposed an accurate and efficient
alternative to using document count forms
(DCFs), which will also save time for district
personnel. Please see Section 7.4.7.5 for more
information.
Document Scanning
DRCs proprietary imaging and hand-scoring
systems maximize performance by accessing
centralized data and images. Storing information centrally improves efficiency and eliminates unnecessary duplication and movement
of data and images between servers and physical locations. Data and images used by both
systems are stored on clustered SQL DB
servers and centralized image file servers,
which are housed in DRCs high-speed SAN
disk storage.
Each IBML image scanner has two dedicated

servers, one for image storage and one for data
storage. Data and images from DRCs scanners
are transferred to centralized locations via a
high-speed network. Copies of the images are
preserved for use during the image scoring
process. Booklet counts and page integrity are
maintained throughout the scanning process
by storing data in a relational database management system (RDMS). A relational database
significantly increases system flexibility and
provides for robust data analysis capabilities.
Scanning programs are customized for each
assessment program and go through quality
review before testing materials arrive.
materials arrive at the companys facilities.

Throughout the scanning process, batches will
be checked for quality and scanning accuracy
by experienced document processing staff.
Using a patented method, all DRC scanners are
calibrated and cleaned regularly to ensure
accurate and consistent scoring. DRC also has
an on-site field service engineer to resolve any
technical issues as they arise.
During the scanning stage, constructedresponse and extended-response items are

captured as images and loaded into DRCs
Image Scoring System. The use of image scanning and scoring technology at DRC mitigates
the inefficiencies of traditional paper-based
packeting of constructed-response and extended-response items and student responses.
DRCs hand-scoring system is based on a
three-tiered architecture. Scoring applications
access data and images via mid-tier applications running on Microsoft IIS servers. Images
and data are accessed throughout the imaging
and hand-scoring systems by proprietary
patent pending software running on multiple
servers and user desktops. The architecture of
its proprietary system allows DRC to adjust
server resources for processing efficiency and
to minimize processing time. Please see
Section 7.5.4.5 for additional detail on DRCs
hand-scoring system.
TIFF images of complete documents; and
A high-level overview of DRCs scannable

materials process is shown in Exhibit 7.5.3.-1.
Image and Scan Quality Control

QC procedures are critical to DRCs document
scanning process. All image scanning programs
will go through quality review before testing
DRCs scanning process produces comprehensive, detailed information, including

student demographic data;
student multiple-choice response data;
identifiers to link the TIFF images to the
student demographic data.
DRCs QC procedures for document scanning
include:
Department-approved scope of work agreements (SOWA) will be established. All processing and scanning staff will adhere to
the requirements contained in the SOWA.
As scanning occurs, a unique serial number
is printed on each sheet of paper. This serial number ties documents together and
maintains sequencing within batches.
The scanners pick up pre-defined processing criteria related to pre-printed barcodes,
multiple-choice items, and student demographic and identification information.
Constructed-response and extendedresponse item student responses are
scanned; images are then separated out for
image-based hand-scoring.
As documents are scanned, the scanner is
monitored to ensure that images meet
DRCs strict quality standards.
Regularly scheduled calibration and scanner cleaning processes will be conducted to
Page 7.5-13
Exhibit: 7.5.3-1 DRCs Scannable Materials Processing
Page 7.5-14
ensure image and read-level quality and

consistency.
All scanned images will pass through a software clean-up program and process. After
image clean-up, a random sample of images
is presented for image quality approval. If
any image fails to meet DRCs quality standards, the entire batch of documents is rescanned.
Page-scan verification is performed to
ensure that all pre-defined portions of a
document were correctly captured. A
flatbed scanner is used to capture responses and images for any missing pages. These
images are then added to the image data
file and merged with the appropriate document.
Editing and Key Entry Processes

After scanning, the documents are processed
through a computer-based editing program to
detect potential errors in specified response
fields. Marks or omissions that do not meet the
pre-defined editing standards are flagged and
routed to the document processing editing staff
for resolution. For the FSA, AIR and DRC will
work with the Department to develop edit
rules specific to the FSA.
Using unique serial numbers printed on the
documents during scanning, the editors compare the actual documents to online data.
Corrections are then made to the data file
according to pre-defined, program-specific
guidelines. Double-key entry verification is
conducted for all keyed items and any changes
made to scanned values by staff personnel
other than the initial keyer/editor. DRC will
maintain a log that documents why scan file
records were edited and by whom, as well as
the name of the keyer(s) and editor(s).
DRCs editing staff follows strict QC procedures to produce clean data files that can be
submitted for scoring and reporting functions.

These procedures include the following:
Edit specifications will be developed mutually by the Department and AIR and DRC.
Experienced editors meticulously review
any potential irregularities detected during
scanning and make necessary corrections
online to the image data file, referring to
the actual document as required.
Editors determine whether the marks are
valid (based on assessment requirements)
or non-correctable.
All items flagged during document scanning
are presented to editing staff for first-time
entry.
Any changes made to scanned values and
all items entered the first time are doublekeyed for verification by editing quality
staff.
Once verification by editing quality staff is
completed, a QC report is generated for
review during post-editing.
After all corrections for a batch have been
entered and verified, the correction file is
stored in a relational database for reference.
Post-Editing Processes
A final edit is performed to confirm that all
requirements for final processing have been
met. Once the demographic information and
multiple-choice data pass all the pre-defined
editing processes, the images of the student
responses to constructed-response and extended-response items are extracted into files for
scoring. These student response images will be
routed through the DRC Imaging Workflow
System to hand-scoring terminals at DRCs
Scoring Center for scoring by qualified readers. Please see Section 7.5.4.5 for a thorough
discussion of DRCs highly regarded hand-scor-
Page 7.5-15
ing process. Images are stored so that they can

be efficiently retrieved on the basis of student
and school identification information, scores,
and item information. Upon completion of processing, scannable documents are boxed for
security purposes and final storage. Please see
Section 7.4.3 for more information on DRCs
storage procedures.
DRCs post-editing QC procedures include the
following:
During this processing step, the actual
number of documents scanned will be compared to the number of scannable documents assigned to the box during materials
receipt. Any count discrepancies between
materials receipt and documents scanned
will be resolved at this time.
Suspect student precodes, district and
school numbers, and documents IDs are
reviewed for additional verification.
All editing QC reports are reviewed to
ensure all changes were processed accurately.
All corrections during post-editing are
made electronically and a new validation
report is generated to confirm the changes
have been processed accurately and the
report is clean.
After all final processing requirements have
been met, the batch is released for constructed-response/extended-response scoring and student-level processing.
Scan Files
Before creating scored files, AIR and DRC will
provide the Department with two versions of
the scan filesoriginal scan files and edited
scan files. We understand that the original
scan files will contain student response information exactly as scanned, while the edited
scan files will contain the final scanning files
Page 7.5-16
that include all edits made during scanning to

ensure that students receive the correct report
during the initial reporting wave. AIR and DRC
will work with the Department to develop edit
rules specific to the FSA. If incorrect edits are
discovered during data verification, AIR and
DRC will update the edit files. DRC will ensure
that all scanning and data file development
and delivery is accurate and timely, adhering
to all timelines necessary for psychometric
tasks and activities to facilitate the on-time
delivery of results reporting.
All original and edited scan files will adhere to
the Departments specifications pertaining to
format and layout, including fields, values, and
content. Any Department-requested changes
will be documented and incorporated. DRC
will utilize stringent quality assurance procedures for the development of all scan files.
DRCs technical and QC production procedures are industry-leading.
DRCs software quality assurance staff will
ensure the quality of the original and edited
scan files and will make certain that each
record is verified for completeness and accuracy. DRC is prepared and accustomed to handling complex programs with multiple forms,
grades, and subjects. The company has solid,
built-in checkpoints and reviews throughout
the entire processing, scanning, editing/postediting, scoring, and data file development
processes. Software quality assurance experts
will conduct extensive quality checks on all
system data. QC checks will be performed
throughout the system-level testing to ensure
the integrity of the data.
Below is an overview of the processes and
methods that DRC will use to ensure the accuracy and completeness of student data and
associated data files:
All student answer documents returned to
DRC will be scored. Multiple-choice items
and demographic information will be image

scanned and the original scanned data will
be converted into a master student file.
Constructed-response and extendedresponse items will also be image scanned
and sent electronically to DRC Scoring
Centers for scoring by trained, qualified
readers. All student information and score
results are kept secure and confidential
throughout the scanning, scoring, and
reporting processes.
Record counts will be verified against the
counts from the document processing staff
to ensure all students are accounted for in
the file. Additionally, a detailed review of
the error-tracking log will be performed to
ensure any discrepancies are resolved
before proceeding with the scoring routines.
The scoring process will include the scoring
of multiple-choice items to the answer key
and the aggregation of raw scores from the
open-ended responses. Quality assurance
staff will compare the processing file
against the answer key source file to ensure
accuracy. Using final, approved answer
keys, edited batches will be processed
through scoring and reporting programs.
Items will be scored as right, wrong, omitted, or double-gridded. After scanning and
scoring, DRC quality assurance staff will
verify values for multiple-choice and openended items.
Computer-Based Test Extracts

Each time a student completes a CBT, the test
delivery system transmits data from that test
event to our Quality Monitor (QM) system.
Transmitted data includes
student, school, and district identifiers;
raw and scored student item responses;
accommodation records; and
initial scores (those reported to the student).

Upon receipt of the student record, the QM
system
verifies scoring of student responses;
validates scores presented to the student;
calculates any other scores that will be
used in the reporting systems along with
standard errors of measurement (SEM);
calculates derivative measures from those
scores (e.g., proficiency designations); and
performs customized validations of incoming data.
In virtually all cases, these checks are all
passed. When they are not, an automated system alerts project team members, who investigate and resolve the issue.
QMs quality checks are typically completed
within a second of receipt of the data. Once
complete, the data are transmitted to the
Database of Record (DoR) and the online
reporting system.
QMs quality reporting interface accesses DoR
to provide reports on a variety of ongoing quality monitoring issues. These statistical reports
include monitoring of item exposure, item
drift, and match to blueprint. With the exception of item drift (which simulations cannot
predict), the content of these reports is well
predicted by the simulations that we conduct
prior to deployment, so surprises are rare.
Data from the system can be extracted and
delivered through an automated system, our
Data Extract Generator (DEG), with whatever
frequency the Department desires, up to once
per day. We will work with the Department to
establish
a customized data delivery format;
Page 7.5-17
a set of data validation rules for the files

sent to the Department; and
a mutually agreed-upon transfer protocol
(we typically use a secure file transfer protocol (SFTP) to monitored directories).
These data delivery files are automatically validated against the format, using the established
business rules prior to delivery. States receiving nightly updates typically retrieve the data
files from the SFTP server programmatically
and automatically update the state data warehouse or other data store. The data can be
delivered with whatever frequency the department desires, up to nightly.
Exhibit 7.5.3-2 shows the data flow for this

process. The two data streams meet in the
automated QM system, which checks results,
gathers statistical data on item and test performance, stores the data in the DoR, and
delivers the data to the online reporting system.
Exhibit 7.5.3-2: Data Flow Process of Integration of CBT
and PBT Results
Scanning/
Editing
Process
If the Department chooses, state personnel as

well as educators throughout the state can
have immediate access to the data through the
online reporting system. That system is
described in detail in Section 7.5.5.
Integration of Paper-Based Test

Results with Computer-Based Test
Results
AIRs system naturally integrates results from
the PBTs and CBTs. Scanned and edited results
from PBTs pass through the Test Integration
System (TIS), which applies a data intake
process to reconcile data that may be inconsistent under a paper program. For example,
unknown or missing student identifiers often
must be resolved. Please see Section 7.5.5.1
for more information about resolving discrepancies.
TIS translates the scanned and edited data to a
common format, which is shared by all other
systems that interact with or store the data,
including the QM system, DoR, and online
reporting system. The data is then sent to the
QM system, where it is processed in the same
way that the CBT results are processed.
Page 7.5-18
Test
Integration
System
(TIS)
Quality
Monitor
(QM)
Test Delivery
System
(TDS)
Online
Reporting
System
9
Paper
Reports
Database of Record
(DoR)
Data Extract
Generator
(DEG)
The
Department
Integrated Data Extracts

AIR recommends extracting and delivering
paper-based testing data, including both
scanned and edited data elements, along with
computer-based testing data in a single, com-
mon layout but can also create different layouts for each type of data, depending on the
Departments needs. A common layout could
be used for the purpose of transmitting participation data without test scores as well as for
the SSR files containing test scores, and derivatives thereof, discussed in Section 7.5.4. As
described previously, AIR will work with the
Department during the specifications process
to fully define the customized data delivery
procedure to transmit all data in the format
and on the timeline that the Department
desires.
production of all data files. AIR will annually

review the data file layouts with the
Department, identify any desired revisions,
and modify code as necessary to implement
revisions, including application of QA procedures to verify that data files and data file layouts are correct.
7.5.4 Scoring
With AIRs online reporting system (ORS), the

Retrieve Student Results tool allows users to
export a raw data file for district, school,
teacher, or roster as the data become available.
For example, as a class or school completes
the Florida Standards Assessment (FSA) tests,
these data can be available in the ORS immediately and users can request a data file for
that class, school, or district, at the
Departments discretion. In addition, these
data files can be configured to include students identifying information, student demographic information, test participation status,
and performance information at the subject
and subscore levels. The files can be exported
into Microsoft Excel or any other delimited file
format so that they can be easily loaded into
other software platforms. The reporting tool
will store the file for a set period of time so
that the user can access it later. Tests administered on paper, or requiring hand scoring, will
appear in the ORS once imaged data passes
through the Quality Monitor (QM) system and
hand scores are integrated via the Test
Integration System (TIS). Although our plan is
to provide downloadable data files through the
ORS, should the Department prefer, AIR is also
prepared to provide the state and districts with
data files outside of the ORS, which can be
securely downloaded.
Accurate scoring and reporting of assessment

results require a robust data management system with built-in quality assurance (QA)
processes as well rigorous external review.
Data files containing background information
about students and users must be collected,
maintained, and updated over time. Student
assessment results, including raw and scored
item responses, as well as ancillary data, such
response latency for online assessments or
mark densities for paper assessments, must be
collected, scored, verified, and merged with
background information to report assessment
results. Previous-year assessment results must
also be maintained to support longitudinal
reporting of student performance over time.
Each year of the contract, AIR will provide the
range of score files required by the
Department. To accomplish this, at the beginning of the contract period, AIR will provide
the Department draft data file layouts for each
required file for review. AIR will work with the
Department to make any revisions to the data
file layouts necessary to meet data analysis
and score reporting requirements. Upon
Department approval of the data file layout,
the statistical programming necessary to automate production of data files will begin, including QA procedures designed to verify accurate
Florida has the option of making early data

available. Currently in Florida, local users can
access downloadable data files only after all of
the scanned data and hand-scored files are
analyzed, prepared, and posted for district
download.
Page 7.5-19
As we describe in Section 7.5.5 on Reporting,

in addition to the raw data files in the Retrieve
Student Results tool, users can export any
table on any report in the reporting system.
An Export button is available on every report,
and the user can immediately export the table
as an Excel file.
scores, are communicated to TDS to guide test

administration and scoring.
It is important to emphasize, however, that

immediate reporting is not a requirement of
the ORS, and the Department may very well
choose to delay reporting of assessment results
for a variety of reasons, including, among others, verification of machine-scored constructed-response items, completion of hand-scoring
activities, application of post-equating activities, or revisions to the student demographics
file. Whether the Department chooses to provide for immediate reporting of test scores or
to delay reporting until critical activities have
been accomplished, the processes used to verify score reports and data files is the same, as
described in detail below.
district, school, teacher, and class data files
Flow of data
The Roster Tracking System (RTS) tracks student enrollment and demographic variables,
relationships among districts, schools, teachers, classes, and students, as well as user
rights. Florida districts can either upload
enrollment data directly or TIDE can accept
nightly updates from the Departments own
enrollment tracking system.
The Departments history files, including alternate passing score eligibility, verification of
testing status, and all previous test scores are
also included as part of the RTS. Student information flows from RTS to the test delivery system (TDS). Information in the RTS is used to
establish student eligibility for test administrations. Students may only access tests for
which they are eligible. In addition, special
scoring rules, including alternate passing
Page 7.5-20
District, school, teacher, class, and student

information also flows from RTS to ORS to
support production of
score reports; and
Raw and scored item responses and other data

collected by the TDS flow to the QM system,
which conduct a series of checks, including
item and test rescore, and then passes scored
items to the Database of Record (DoR) for
storage. The QM system also provides checks
on item functioning and testing irregularities
that psychometricians can evaluate on
demand. Student raw and scored item responses imaged from paper answer documents are
also routed through the QM system before
they are passed to the DoR so that the same
QA checks on item and test scoring are also
applied to paper-based assessments.
Items requiring hand scoring are routed from
the TDS to the hand-scoring system. Upon
completion of hand scoring, scored item
responses are uploaded to the TIS where handscored item responses are attached to the students test record. The TIS merges all the
machine-scored and hand-scored items from a
single test opportunity into a single XML file
for routing to the DoR.
The DoR is AIRs canonical source for all test
data. Outcomes of any item or test score
appeals, test score verifications, item rescores,
invalidations, or any other post-administration
adjustments are reflected in the DoR. Thus, all
test data used to produce data file deliverables
and populate the ORS flow from the DoR.
The Data Extract Generator (DEG) is used to
produce analysis files for classical and IRT
analyses. AIR will work with the Department
Exhibit 7.5.4-1: Systems Summary

System
Description
Quality Monitor (QM)
The QM system receives tests as students complete them, conducts a series of quality checks, stores scored items in the DoR,
and routes items requiring hand scoring to the hand-scoring system. The QM system also conducts a variety of additional
quality and cheating detection checks.
Test Integrated System (TIS)
The TIS is responsible for the automated virtual stitching of the test where there are hand-scored responses. Since handscored items go through a different process than machine-scored items, TIS takes the machine-scored items of a test
opportunity and integrates them with the hand-scored items of the same test in an XML file. This file will then be uploaded
into the DoR and sent to the ORS.
Database of Record (DoR)
The DoR maintains the authoritative records of tests administered and completed. This includes all hand-scored responses
of the test. Data in the DoR reflect appeals, verifications, and other post-administration adjustments to the data.
Data Extract Generator (DEG)
The DEG is responsible for pushing all assessment data into the Departments site on a predetermined schedule.
Online Reporting System (ORS)
The ORS provides a secure interface to assessment data and associated demographic information. It provides educators with
a powerful tool to explore the data and turn them into actionable information.
and their QA contractor to identify the structure of the files transmitted for all statistical
analyses, including calibration and equating,
item and test score verifications, classical item
and differential item functioning (DIF) analyses for field-test items, and mode comparability analyses. Once final specifications have
been approved by the Department, the DEG
will be configured to generate the data file in
the agreed-upon layout, with QA procedures in
place to ensure that the extracted data is accurate and that the file conforms to the agreedupon structure. AIR will apply the same structure to all simulated data file deliveries as well.
Exhibit 7.5.4-1 provides a systems summary
for the flow of data.
7.5.4.1 Types of Scored Files

AIR is committed to providing the Department
with all FSA assessment results accurately, on
time, and in the format necessary to meet the
Departments needs. As noted above, data
file deliverables used for analysis by the
Department and its QA assurance contractor
are generated directly from the DoR. Data files
provided for educator download, including roster reports, and aggregated classroom, teacher,
school, and district reports are designed to be
generated through the ORS. Each users role
will determine which reports can be generated
in accordance with Department-specified
requirements. As described in Section 7.5.5,
all rosters and reports generated in the ORS
can be exported to user-friendly file formats
such as Excel or other file types. Should the
Department also require the data files be available for download through a different secure
site, AIR will provide these files as separate
deliverables.
State Student Results (SSR). This file includes
item responses, test scores, and student enrollment and demographic information for all students participating in the FSAs. This file is
used for all calibration, equating, and scaling
activities, classical and DIF analyses of fieldtest items, item and test rescore verification,
mode comparability analyses, and other psychometric analyses. Data file elements and
structure will be developed by AIR for
Department review and approval. Once data
Page 7.5-21
file specifications are finalized, they will be

programmed into the DEG and extracts will be
verified for accuracy. DEGs can be generated
on demand, on a scheduled basis, or both,
should the Department wish to receive regular
updates from the DoR in addition to milestone
deliverables.
State Aggregated Results (SAR) and District
Aggregated Results (DAR) files. Test data from
the DoR and student enrollment and demographic information, and user information
from the RTS, can be routed to the ORS to
support immediate reporting of assessment
results. Following the Departments aggregation
rules, student results are aggregated to generate classroom, teacher, school, district, and
state level results. The users role applies the
Departments rules on what reports users are
able to generate. At the Departments discretion, online rosters and reports can be exported to user-friendly file formats such as Excel.
Educators can generate reports and data files
immediately upon completion of test administrations that are entirely machine-scored. Test
administrations requiring hand scoring are
not available in the ORS until after results of
hand scoring have been uploaded to the TIS,
attached to the opportunity record containing
machine-scored items, and deposited as a single record in the DoR, at which point they can
be released to the ORS for immediate reporting.
Close-out SAR and DAR. At any point, the
ORS aggregate and roster reports are up to
date. Test records that are embargoed can be
released at any point and upon release will be
automatically included in district and state
aggregated results. Similarly, any answer documents submitted late and processed as a late
batch are available to the ORS for reporting
and data file production once hand-scored
items have been uploaded to the TIS and
stitched to the machine-scored test record,
with the unified test record returned to the DoR.
Page 7.5-22
Concatenated SAR and DAR files across test

administrations each year. Assessments with
retakes can be configured as multiple-opportunity assessments, with eligibility for each
opportunity contingent upon performance in
previous test opportunities. In this way, the
ORS can automatically include all testing
opportunities administered in an assessment
cycle.
Demographic results based on Survey 3 membership file. All aggregate reports in the reporting system can also be disaggregated by subgroups defined by the Department. For example, reports can be disaggregated by demographic characteristics, such as gender, ethnicity, or English language learner (ELL) status.
The reporting system can also be configured to
disaggregate by multiple characteristics at
once (e.g., gender by ethnicity).
In addition to the expected aggregated reports
for a district, school, teacher, and class, the
reporting tool allows users to create custom
rosters of students so that they can be tracked
and reported on separately. Teachers may, for
example, create their own disaggregate groups
to track students who are receiving additional
assistance or participating in a special program. By creating new subgroups, teachers can
generate additional rosters and reports to
monitor the progress of those students independently of classroom at large. The ability to
define special subgroups of interest may help
teachers to address questions about the effectiveness of the extra help or participation in
special programs. All aggregated reports are
calculated from the student level up, so as new
students test, reports for the roster, teacher,
school, and district are updated instantaneously.
Create CDs/DVDs for districts upon request.
Although districts can export data files directly
from the ORS, AIR will also deliver data files
to districts on CDs/DVDs or make data files
available for district download via a secure file

transfer protocol (SFTP) site.
provide the Department with updated history

files for use by the Department and QA contractor to verify subsequent year score reports.
7.5.4.2 Types of History Files

The Department anticipates that three types of
history files will need to be maintained by the
contractor to support implementation of special scoring rules and reporting requirements.
The three history files include:
Alternate Passing Score File. Because some
students may participate in the grade 10
English Language Arts/Literacy (ELA/L)
prior to the Departments adoption of performance standards, these students may be
grandfathered into a lower passing score
requirement. This file will include all students eligible for the alternate passing
score and will be updated to reflect subsequent test administrations for each student.
Testing Status. For the Algebra 1,
Geometry, and Algebra 2 End-of-Course
(EOC) assessments, the Department
requires verification of each students test
status (e.g., first-time tester verses retaker).
ELA/L and Mathematics History File. To
track student performance across test
administrations, beginning with the first
test administration, the history of each students test performance, including test
scores and performance levels, will be
maintained by the contractor, updated following each test administration, and provided to the Department at the end of the
contract period.
AIR will work with the Department to identify
the format of the history files and develop
rules for merging records in the history files
with student test records, all of which will be
submitted to the Department as part of the
scoring and reporting specifications. Should
the Department desire, upon close of test scoring and reporting activities each year, AIR will
7.5.4.3 Types of Scores and Codes

on the SSR/DSRs
As described in Section 7.3.7, test forms are
constructed using the FormBuilder utility in
the Item Tracking System (ITS). This is true of
all forms, whether online or paper, fixed form
or adaptive. In the case of adaptive assessments, the form defines the adaptive item pool
for the administration, which may be a subset
of items available in the operational pool.
Once a test form is defined, FormBuilder generates a series of automated documents that
are critical to test development, scoring, and
reporting, including test maps, key files, and
conversion tables. The automated production
of these documents reduces the likelihood that
errors will be introduced into the production
of these critical documents.
The format of test maps is configurable and
designed to meet the Departments needs. Our
systems directly read the test maps, so some of
the fields are mandatory to support our internal workflow, but we can add fields as necessary to support Department workflows.
Because production of test maps, key files, and
conversion tables is automated, forms can
readily be revised in FormBuilder (e.g., dropping an item), and new test maps, key files,
and conversion tables generate automatically.
The virtue of the automated system is especially apparent when revisions are being made
while working under tight timelines.
Consistent with the Departments traditional
practice, we propose using the maximum likelihood estimator (MLE) pattern scoring rather
than conversion tables to score tests. Pattern
scoring yields more accurate test scores than
number correct scoring, because it utilizes the
Page 7.5-23
item discrimination and guessing parameters

to evaluate student responses to each item. In
addition, dropping items for any reason
becomes less error prone and faster, because
new conversion tables do not need to be generated, verified, and loaded into the scoring
system. Rather, we can simply toggle the item
role switch in the configuration file and regenerate test scores.
Each year AIR will deliver to the Department a
scoring and reporting specifications document.
Scoring specifications will include procedures
for estimating student ability, the linear transformations necessary to place ability estimates
on both the within-grade and developmental
reporting scales, as well as the performance
level classifications based on the cut scores
adopted by the Department. Specifications for
classifying subscale performance, as well as
strength and weakness reports for strand and
benchmarks, will also be included where
appropriate.
We note that the Department anticipates that
ELA/L Retakes may have multiple sets of passing scores based on date of initial assessment.
The RTS maintains each students entire
assessment history, so that test administrations can readily be configured to apply scoring rules based on the date of the students initial test administration.
The RTS maintains all the student and user
information necessary to report assessment
results, including the school type variable that
indicates which schools are included in district
and state aggregate reports and data files (e.g.,
homeschooled students are not included in
aggregated reports).
In addition, some test records are not scored
based on characteristics of the test administration, including, for example, rules governing
whether a test administration was attempted,
invalidations based on cheating, or other irreg-
Page 7.5-24
ularities in the test administration. These rules

are also documented in the scoring and reporting specifications and will be configured into
the QM.
The Department also notes that some codes
are derived from multiple variables. Derived
codes are easily handled by the ORS and will
also be documented in the scoring and reporting specifications for Departmental approval
prior to being programmed into the reporting
system.
all codes to be provided in score report and
data files prior to each test administration
cycle. Rules for computing and applying codes
in the reporting of test results will be documented each year in the scoring and reporting
specifications document. AIR will verify the
accuracy of the code computation and application in score reporting and data file generation.
7.5.4.4 Handling Special Print

Documents
Processing of Accommodated
Materials
AIR team member DRC will provide a clear,
well-documented return process for all special
print test materials (Braille, large print, and
one-item-per-page (OIPP) versions), increasing
the accuracy and decreasing the turnaround
time of special materials return shipments.
Unlike many testing companies, DRC does not
require special packaging or return procedures
for special print test materials, requiring little
document preparation and saving time for
local test administrators involved in materials
return. DRCs automated, barcode-based materials management systemthe Operations
Materials Management System (Ops MMS)
allows for fast and accurate data collection
with no dependence on materials/document
order. DRC will also provide the excellent sup-
port of its highly regarded customer service

staff to assist test administrators with any
questions regarding preparing and packaging
materials for collection and shipment back to
DRC.
DRC will maintain all quality and security procedures that will have been used during processing and scoring of the regular print paperbased test (PBT) materials and student
response documents. DRC will use its standard
operating procedures for checking in and
accounting for all secure materials once they
arrive at DRC (please see Section 7.4.1 for
more information). DRC will ensure that all
special print processing adheres to Department
requirements. Captured data are organized
into user-friendly reports from the start of the
secure materials check-in process, providing
valuable insight into suspected material shortfalls as early as possible and mitigating potential consequences of delay. During materials
receipt, DRC will document in the error log
any errors in packaging or returning test materials made by schools and districts. DRC will
correct any materials preparation issues so
that processing and scoring can proceed.
However, DRC will also contact districts and
the Department to resolve any issues that
require additional resolution.
AIR and DRC understand that scores for students using special print materials will be
included in the initial report releases. DRCs
proven streamlined methods for receiving and
processing special print materials will ensure
that there will be no delay in scoring and
reporting of these materials.
Braille Transcription
After receipt and processing of Braille response
documents received from districts, DRC will
batch these materials for expedited handling
and transcription. DRC will work with staff
from the American Printing House for the
Blind (APH), the same vendor providing Braille
format test materials, to conduct transcription

of Braille answer documents into corresponding, scannable regular print documents. APH
is a nationally recognized Braille publishing
house and experienced provider of Braille
assessment services. DRC and APH will ensure
that all transcribed responses are proofed and
double checked by a qualified professional
other than the initial transcriber.
Transcribed documents will be linked to individual students and their original Brailleresponse documents using unique document
ID numbers and student PreID labels. After
receiving the transcribed materials from APH,
DRC will process, scan, and score these documents using the same processing and scoring
routines it uses for regular print PBT student
response documents. All standard quality and
security procedures will be maintained. After
transcription and processing, both the original
Braille and transcribed response documents
will be securely stored for easy retrieval if
required.
Large Print and One-Item-Per-Page

Transcription
DRC has the full, in-house capability to provide large print and OIPP transcription services for the Florida Standards Assessments
(FSA). DRC offers transcription and key entry
services as alternative processing and response
capture options. With its team of skilled individuals, DRC has the capacity to handle complex transcription and key entry of more than
500,000 complex items during a four-week
processing period, with the ability to increase
capacity as needed.
Customized transcription and key entry relies
heavily on DRCs highly skilled data entry professionals, extensive training and specification
review, internal data validation inputs, and
constant quality control checks throughout a
project. Using the expertise of its data entry
Page 7.5-25
team, DRC will ensure that all transcription is

conducted accurately, efficiently, and within
the processing, scoring, and reporting timelines required for the FSA. All fields will be
keyed twice by two trained personnel and verified prior to final data validation.
After receipt and processing of large print and
OIPP response documents received from districts, DRC will batch these materials for expedited handling and transcription. DRCs data
keyers will transcribe large print and OIPP student responses into corresponding, scannable
regular print documents. Transcribed documents will be linked to individual students and
their original large print or OIPP response documents using unique document ID numbers
and student PreID labels. After transcription,
DRC will process, scan, and score these documents using the same processing and scoring
routines it uses for regular print PBT student
response documents. All standard quality and
security procedures will be maintained. DRC
will ensure that all transcribed responses are
proofed and double checked by a qualified professional other than the initial transcriber. All
transcription will adhere to Department specifications.
If preferred, AIR and DRC large print and OIPP
student responses can be transcribed directly
into AIRs computer-based assessment (CBA)
system. Upon award, if desired, AIR and DRC
will work with the Department to define specifications for a computer-based test (CBT) transcription method.
Special Document Tracking Log

DRC has a great deal of experience meeting
the unique reporting needs of its clients. DRC
routinely provides customized tracking logs
and reports for its clients. For the FSA, DRC
will develop a special document tracking log
for inventorying and tracking special print
documents from receipt and initial processing
Page 7.5-26
through transcription and final processing.

This system will be tailored to meet the specific needs of the FSA; AIR and DRC will work
with the Department to define specifications
and format as well as submission frequency to
the Department.
7.5.4.5 Handscoring
AIR team member DRC brings a tremendous
amount of experience scoring constructed
responses for many large-scale assessments.
AIR has total confidence in DRCs ability to provide superior service to the Florida Standards
Assessments. DRC has an established, successful history of scoring online responses for AIR
for several other assessment programs, including those for Minnesota and Ohio.
DRCs handscoring processes are ISO
9001:2008 certified. The company has 27
years of experience in providing accurate
scores for millions of student responses from
large-scale assessments for numerous clients,
including Alabama, Alaska, Arkansas, Florida,
Kentucky, Louisiana, Minnesota, Nebraska,
New Jersey, North Carolina, Ohio, Oklahoma,
Pennsylvania, the Smarter Balanced
Assessment Consortium, South Carolina, and
Washington. DRC has utilized numerous scoring models to score responses from multiple
content areas, grades, and end-of-course
assessments, and it has significant experience
scoring assessments with a mix of online
responses and pencil-and-paper responses for
clients.
AIR and DRC understand that there will be
handscoring for each of the four assessment
components outlined in the ITN. These
include the computer-based and paper-based
assessments for
English Language Arts/Literacy in Grades
38;
Mathematics in Grades 38;
English Language Arts/Literacy in Grades

911; and
End-of-Course assessments in Algebra 1,
Geometry, and Algebra 2.
DRC understands that the ELA/L grades 311
assessments will include five operational short
constructed responses per student and one
long and one short passage-based essay per
student. DRC further understands that the
assessments for Mathematics grades 38 and
EOC assessments for Algebra 1, Geometry, and
Algebra 2 will include five operational short
constructed responses per student. For online
versions of these assessments, DRC acknowledges that a subset of these items and responses will require human scoring. For paper-based
assessments, DRC will handscore all of these
student responses.
Because of the high stakes associated with the
Reading portion of the Grade 3 ELA/L the
entire grade 10 ELA/L assessments, AIR has
devoted extra handscoring resources to these
tests. Specifically, these assessments will be
double human scored when delivered on
paper, and the responses to the grade 10
Writing prompts delivered online will be both
machine and human scored, with 100% human
resolution of any difference between the
scores. For the high-stakes Algebra I assessment, all constructed responses on paper tests
will be double human scored, with resolution
of nonadjacent scores. For responses writing
prompts in all other grades, 25% of the
machine-scored responses will be routed to
human scorers based on algorithms identifying
potentially anomalous responses. For all other
portions of the assessment, 20% of the
responses will receive a second reading for the
purpose of monitoring and maintaining sufficient inter-rater reliability. AIR and DRC will
work with the Department to establish a specific set of scoring rules for resolving disagreement between first and second readings by utilizing subsequent independent readings performed by team leaders and scoring directors.
DRC has strong performance assessment

expertise in all facets of English Language Arts
stemming from extensive handscoring of ELA
constructed responses for many client states,
including Alaska, Louisiana, Kentucky, Ohio,
Minnesota, Pennsylvania, South Carolina, and
Washington. It has significant experience with
scoring passage-based essay items for
Louisiana, Pennsylvania, and the Smarter
Balanced Assessment Consortium. DRC also
has substantial experience with scoring
Mathematics constructed responses for many
client states, including Alaska, Kentucky,
Louisiana, Pennsylvania, and Washington.
Most of these programs included two- and fourpoint items. For both ELA and Mathematics,
DRC has scored responses for students ranging
from early elementary to high school.
DRC's Performance Assessment Services
knows that the proposed work will require
careful planning, thorough and thoughtful systems designs, and sound execution of agreedupon procedures. DRCs content staff has
many years of experience with large-scale
assessments, and its management staff has an
in-depth knowledge of monitoring scoring sessions, producing accurate results, and meeting
deadlines for various clients. AIR is confident
that DRCs experienced staff will successfully
complete all handscoring associated with the
Florida Standards Assessment and that the
State of Florida Department of Education (the
Department) will be pleased with the results.
Staffing
DRC offers experienced performance assessment personnel who are unsurpassed in the
industry. DRC scorers, team leaders, and scoring directors are dependable, accurate, and
instrumental in meeting strict reporting deadlines. DRCs handscoring personnel have
expertise in scoring many large-scale assessments that have incorporated numerous content areas, scoring models, and procedures.
Page 7.5-27
DRC understands that any scoring

director/assistant scoring director and any
content specialist who is assigned to the
Florida Standards Assessment must have an
appropriate educational background as well as
prior, equivalent scoring experience.
DRC is pleased to propose highly qualified
leaders in the handscoring field, with significant experience working with large-scale ELA
and Mathematics assessments. The DRC handscoring team consists of managers and content
specialists who have overseen handscoring for
many large-scale ELA and Mathematics assessments. Please see Section 2 for complete profiles and rsums of DRCs handscoring professionals in Appendix A.
Handscoring Advisor: Ms. Sherri Woolf, Vice
President of Performance Assessment
Services, will ensure that scoring services for
Florida are of the highest quality. Ms. Woolf is
based out of DRCs Sharonville, Ohio, scoring
center. She brings over 21 years of large-scale
testing experience to DRCs Performance
Assessment Team. Ms. Woolf currently leads
scoring operations for DRC, overseeing more
than 3,000 professional scorers and seven
scoring centers. Over the course of her career,
she has also worked as a grade 7 and 8 Writing
teacher and as a consultant for a state departments scoring center. Ms. Woolf holds an M.S.
in Human Resource Development from Xavier
University in Cincinnati, Ohio and a B.S in
Elementary Education from Miami University
in Oxford, Ohio.
Handscoring Senior Manager: Ms. Wendy
Marik will oversee and coordinate DRCs
Performance Assessment Services to ensure
accurate and timely handscoring for the state
of Florida. Ms. Marik has served in this capacity for the Washington Comprehensive
Assessment Program since 2009. Since 2006,
she has also acted as the Content Specialist for
the Writing and ELA components of the
Pennsylvania System of School Assessment.
Page 7.5-28
Ms. Marik also played an instrumental role in

DRCs Performance Assessment Services ISO
9001:2008 certification. Prior to joining DRC,
Ms. Marik was a kindergarten classroom
teacher, a K12 computer technology teacher,
and an ELA and Mathematics tutor. She holds
a M.Ed. from Xavier University in Ohio as well
as a B.A. in Computer Science from DePauw
University in Indiana.
ELA/Literacy Lead Scoring Director: Ms.
Pamela Ahrens has 10 years of experience
working on large-scale ELA assessment projects. Since 2008, Ms. Ahrens has worked
closely with the Pennsylvania Department of
Education and educators to develop training
materials for the Pennsylvania System of
School Assessment and Keystones exams. She
has similar ELA content experience with the
Smarter Balanced Assessment Consortium,
South Carolina, and Washington. Her responsibilities during this time have included monitoring handscoring projects, supervising handscoring staff, and coordinating rangefinding
activities. Ms. Ahrens has experience with
numerous Writing and Reading assessment
models and rubrics, including the passagebased essay item type. During her time at
DRC, she has served in many handscoring
roles and has applied her expertise in both
Reading and Writing to many large-scale
assessments, including Alaska, Arkansas,
Louisiana, Minnesota, Ohio, Oklahoma,
Pennsylvania, Smarter Balanced Assessment
Consortium, South Carolina, and Washington.
Complementing her performance assessment
background, Ms. Ahrens has teaching experience in both ELA and Spanish in a public
school setting at the middle and high school
levels. She has also managed libraries at two
medical facilities in Minnesota. Ms. Ahrens has
a B.S. in Education for English and Spanish,
and an M.A. in Library Science.
Mathematics Lead Scoring Director: Mr.
Coskun Baysoy has 10 years of experience
working on large-scale Mathematics assess-
ment projects. In his current role as

Mathematics Content Specialist, he manages
all aspects of handscoring operations for the
Washington Comprehensive Assessment
Program end-of-course Mathematics exams.
His responsibilities also include monitoring
handscoring projects, supervising handscoring
staff, and coordinating rangefinding activities.
Mr. Baysoy has experience overseeing automated and computer-aided scoring for a variety of programs and item types. He also played
an integral role in Mathematics item
writing/development for several programs,
including the Smarter Balanced Assessment
Consortium and Pennsylvania Keystone
Exams. Mr. Baysoy holds a Ph.D. and an
M.S.E.E both in Electrical and Computer
Engineering from the University of Minnesota,
as well as a B.S.E.E in Electronics and
Communications Engineering.
Scoring Facilities
DRCs scoring centers are secure facilities.
Access is limited, through the use of electronic
badge access and vigilant front desk staff, to
DRC employees and visitors accompanied by
authorized staff. Student responses are kept
secure at all times. DRC currently operates
scoring centers in six locations: Plymouth,
Minnesota; Woodbury, Minnesota; Sharonville,
DRCs Image Handscoring System
Ohio; Columbus, Ohio; Pittsburgh,

Pennsylvania; and Austin, Texas. Each of
DRCs scoring centers is divided into large,
open scoring rooms. Additionally, there is sufficient office space for conferences and small
group training. DRC also maintains large break
rooms equipped with coffee, tea, hot chocolate, soft drinks, vending machines, and refrigerators. DRC scoring centers are accessible to
the physically challenged and conveniently
located near major highways and amenities.
Upon award of the contract, DRC plans to
open either two or three scoring centers in the
state of Florida. These would be utilized to
score ELA/Literacy and Mathematics constructed response items for the Florida
Standards Assessments. Having a number of
Florida scoring centers and employing hundreds of local Florida scorers will provide significant economic benefits for Florida. DRCs
Florida scoring centers would also be made
available as potential meeting space for use by
the Department. Finally, using local scoring
centers would also allow the Department to
observe firsthand the handscoring activities
associated with the Florida Standards
Assessments, without extensive travel.
Image Handscoring
Now in its thirteenth year of operation, DRCs Image Handscoring
System has proven to be a highly
efficient and accurate platform for
scoring large-scale assessments.
This dynamic system can be used
to score paper-and-pencil responses intermixed with responses created in an online testing platform,
including other vendors testing
engines. The same stringent measures of quality control, including
numerous handscoring reports,
are readily available regardless of
the mode of response. Therefore,
Page 7.5-29
DRCs process for scoring computer-generated

responses is identical to its process for scoring
handwritten responses; this ensures that clients
are provided with the fairest scoring environment possible. DRC also employs a bias training
to ensure that scorers do not view handwritten
responses differently than typed responses.
DRCs Image Handscoring System allows scorers to score online, increasing efficiency by
eliminating both the routing of paper and the
possibility of lost student answer documents.
Instead, student responses are electronically
routed to geographically dispersed DRC Scoring
Centers. As with past projects on which AIR
has partnered with DRC, AIR will securely
transfer computer-based test student responses
to DRC. DRC will then load the responses into
DRCs Image Handscoring System. The
responses will then be allocated to scorers
through a custom dealer program, ensuring
that each scorer is assigned a random workload
and allowing the project to be processed in the
most efficient manner possible.
With the Image Handscoring System, each
open-ended items position within the assessment is defined through the use of the programmatic Item Definition Application. For
each item, the system also requires the definition of possible score values, nonscore values,
and applicable scoring rules (e.g., all responses
receive two independent reads and responses
with non-adjacent scores receive an independent third reading).
The Image Handscoring System provides scorers with the ability to view full-page images
from multiple perspectivessuch as zooming
in/out and image flipping or rotationin order
to correctly interpret written responses.
Images remain intact with the various viewing
capabilities and cannot be modified by the
scorers. Additionally, Image Handscoring functionality applies a set of process rules and
client-defined read-behind criteria.
Page 7.5-30
Each handscoring site is connected to the main

DRC operations facility through a DS3, 45
Mb/S, private circuit. The operations facility
has secure database servers and multiple applications that support the scanning, editing, scoring, and handscoring processes. Database backups and archived images are stored off-site on
tape media for disaster recovery purposes.
Each DRC scoring site has a server and a local
area network (LAN). Scorers, team leaders, and
scoring directors connect to the LAN via hundreds of PC workstations and use locally resident software to view and score student
responses. Authorized on-site DRC personnel
(e.g., content specialists, project managers) can
access the LAN to recall student responses.
Quality Control of the Handscoring

Imaging System
Software Quality Assurance Analysts test the
imaging system to verify that all handscoring
programs are compliant and in place for performance assessment personnel prior to the
transfer of production images. Images produced from test scan files are generated,
processed through the handscoring system
software, and accessed from the handscoring
client software, where Quality Assurance
Analysts score the test images using the handscoring criteria and specifications just as they
will be in the production systems. Throughout
this testing cycle, multiple quality checks are
executed to ensure that the data integrity for
each student record is intact and accurately
reflected in the scoring database.
Additionally, the Image Handscoring functionality applies a set of process rules and clientdefined read-behind criteria. This functionality
includes a 50% rule that ensures that one
scorer cannot score more than 50% of a students responses in any given content area,
which helps to disperse a students responses
across multiple scorers. This quality control
measure ensures that multiple scorers contribute to each students final results, thereby
producing results that are more psychometrically sound.

Rangefinding
The following plan describes DRCs understanding of the rangefinding process described in the
ITN. AIR and DRC look forward to collaborating with the Department in order to fine-tune
this process to meet the Departments needs.
DRC has significant experience with a number
of models of rangefinding and developing training materials. DRC will ensure that comprehensive, high quality training materials are developed with direct input from Florida educators
and the Department.
After receiving student responses, DRC content specialists and scoring directors will prepare for rangefinding by using DRCs Image
Viewer to access student responses for each
open-ended item. They will use the rubrics to
select a representative sampling for each score
point. The rangefinding responses will also
include borderline responses that do not fit
neatly into one of the score levels. These
examples are included in order to address
decision-making challenges that will be faced
by the scorers.
AIR and DRC understand that rangefinding
will occur at a site in Tallahassee and that
each committee will comprise up to 10 Florida
educators with a Department staff person acting as chair. DRC will send grade/subject scoring directors and assistant scoring directors to
all rangefinding meetings. There will be fieldtest rangefinding in the spring, field-test and
operational rangefinding each fall. Each
rangefinding meeting will last 5 days. During
the meetings, the rangefinding committee will
review and consensus-score samples of student
work in order to reach common agreement on
the application of scoring rubrics and provide
responses for scorer training materials. DRC
takes security and confidentiality as seriously
in the rangefinding setting as it does in its
scoring centers. Consequently, no materials

will ever leave the rangefinding rooms, which
will always be locked when not actively in use.
The rangefinding process for each item will
begin with a review and discussion of the
rubric. After an understanding of the scoring
guidelines has been established, the rangefinding committee will read and discuss each
response until a consensus score is reached.
For each item, the committee will score a sufficient number of responses to construct the
anchor, training, qualifying, and validity sets to
be used in field-test scoring. The committee
will move from item to item until all items are
complete. DRCs scoring directors/assistant
scoring directors will maintain a record of all
scores and make careful notes of scoring decisions made by the committees. These notes
will be used to annotate the student responses
in the anchor sets. They will also be used during scorer training to help ensure that the
scorers understand and follow the rangefinding
decisions. Finally, notes taken during
rangefinding will drive longitudinal consistency
of scoring protocol by providing benchmark
points for discussions in subsequent years
rangefinding meetings.
After rangefinding, DRCs handscoring staff will
be available to meet with Department staff to
review the rangefinding responses and compare notes from the meeting. This will ensure
that the Departments understanding of how
the committees viewed the responses matches
DRCs understanding of how the committees
viewed the responses. DRC staff will also work
with the Department to identify responses to
be used as anchors as well as responses to be
used as training, qualifying, and validity
papers. DRCs scoring directors will write
annotations and provide notes necessary for
scoring guides as well as any other criteria
needed to accurately score each constructed
response item. DRC understands that the
Department will approve all selections, annota-
Page 7.5-31
tions, and other materials used in the training

materials.
Developing Training Materials
Once rangefinding is complete, DRC will utilize the committee-scored rangefinding
responses to develop training materials for
scoring the field-test responses. DRC will be
responsible for the creation of all anchor sets
(committee-scored, annotated responses representing each score point) as well as all training, qualifying, and recalibration sets. Scoring
directors will select anchor and training papers
from the sets of committee-scored,
Department-reviewed responses from
rangefinding. All notes generated during the
rangefinding process will remain with each
response selected, either in the annotation (for
anchor papers) or in the scoring directors
notes (for training/qualifying papers). DRC will
submit copies of training materials to the
Department for approval prior to their use.
Any training material created by DRC can be
provided to the Department in PDF format for
archival purposes.
Anchor Sets
Each constructed response item will require
item-specific training materials, including a
scoring guide consisting of the rubric, the passage or scenario (if appropriate), and three to
four annotated anchor responses per score
point. Anchor papers will be selected to illustrate particular scoring concepts. These
responses will help ensure that scorers are
able to make accurate and consistent scoring
decisions for the response types they are likely
to encounter. All anchor papers will be annotated to explain how they exemplify each
score point. The scoring guide/anchor set will
serve as the scorers constant reference. DRC
will provide the Department with copies of
anchor sets for approval before scoring the
field-test responses.
Page 7.5-32
Training/Qualifying Sets
For each constructed response item, DRC will
also develop a minimum of three training sets
and two qualifying rounds of 10 student
responses each. DRC will create more training
and qualifying rounds for more complex item
types, such as passage-based essays. These
training/qualifying papers will further hone
each scorers ability to discern the different
score-point levels in an accurate and consistent manner. When reviewing training papers
from the front of the scoring room, the scoring
director will utilize the notes generated during
rangefinding to ensure that scorers reach scoring decisions in a manner consistent with the
rangefinding committees thinking. DRC will
work with the department to finalize numbers
of training/qualifying sets required for each
item type and to determine a viable qualifying
percentage. DRC will provide the Department
with copies of training and qualifying sets for
approval before scoring.
Operational Materials
DRC understands that operational rangefinding
will take place during the fall. DRC will pull
forward the field-test training materials
(anchor, training, qualifying, validity, and
recalibration sets) for each item selected for
an operational administration. DRC will utilize
committee-scored responses to supplement
these materials (if needed) in order to thoroughly convey scoring criteria as each selected
field-test item transitions to operational status.
These supplemental materials will further scorers understanding of how the rangefinding and
field-test responses were scored in order to
ensure accurate and consistent scoring.
Validity Sets
In addition to anchor, training, and qualifying
sets, DRC will select responses for use as validity sets during both field test and operational
scoring. DRC understands that the Department

will approve selections for validity sets prior to
use. Validity responses can help detect possible room drift as well as individual scorer drift.
Validity reports compare scorers scores to predetermined scores. The validity responses will
be blind to the scorers; scorers will not be
able to distinguish validity responses from live
responses. DRC understands that the frequency distribution of scores within the validity
pool should closely mirror the frequency distribution of scores generated during live scoring. DRC also understands that, for costing
purposes, a validity insertion rate of 1 in 10
should be assumed for extended ELA/L items,
while a validity insertion rate of 1 in 25 should
be assumed for all other constructed items.
Calibration Sets
DRC will also select recalibration responses
throughout the course of scoring operational
assessments. Calibration sets are designed to
help refocus scorers on how to properly use
the rubrics to score responses. These sets will
be selected to help illustrate particular points
and familiarize scorers with the types of
responses commonly seen during the current
operational scoring. DRC will create calibration sets of two to three responses and distribute them to all of the scorers on each item
each day. After the scorers take the recalibration set, the scoring director will go over the
set from the front of the scoring room, using
rubric language and the anchor responses to
explain the reasoning behind each responses
score. DRC will employ extra, or targeted, calibration sets as needed.
Scoring Material Review
DRC understands that each year, at least three
months prior to handscoring training for
spring administrations, there will be a final
materials review meeting to review proposed
operational scoring materials. DRCs content
specialists and scoring directors for each subject/grade will meet with Department staff to
review scoring criteria and field-test training

materials, supplement the anchor papers (if
necessary) and ensure that DRC and the
Department share the same understanding of
the scoring criteria as selected field-test items
become operational. DRC understands that
these will not be committee meetings, but
instead will provide a chance for Department
content staff to give final input into the training materials before their use in an operational
administration.
Selecting and Training Scorers

Recruitment and Selection Process
Because of DRCs highly regarded reputation as
a professional, respectful employer, the company is able to maintain a large pool of experienced scorers from which to staff client projects. As needed, this pool is increased each
spring through a comprehensive, proven hiring
process. DRC selects scorers who are articulate, conscientious, and flexible. DRCs scorers
have strong content-specific backgrounds; they
are educators, writers, editors, and other professionals. DRC scorers are valued for their
professional and academic experience but are
required to set aside any biases concerning
student performance and accept the scoring
standards of each clients program.
With the excellent and well-educated labor
pools available near all DRC scoring facilities
(stemming, in part, from numerous nearby colleges and universities), DRC is able to tailor
scoring staff to each clients program. All DRC
scorers have at least a four-year college degree
in a relevant field and a demonstrated ability
to write. Most scorers have years of experience
with scoring large-scale assessments. DRC will
make a concerted effort to hire scorers for the
Florida Standards Assessments who have prior
handscoring experience for other statewide
testing programs.
DRC has a human resources coordinator dedicated solely to recruiting and retaining hand-
Page 7.5-33
scoring staff. Applications for scorer positions

are screened by the project director, the
human resources coordinator, and recruiting
staff to create a large pool of potential scorers.
During the screening process, preference is
given to candidates with previous experience
scoring large-scale assessments and with
degrees emphasizing appropriate content
areas. During required personal interviews,
scorer candidates are asked to demonstrate
proficiency in program subjects by responding
to a DRC-selected topic. Selections are made
on the basis of previous handscoring experience, previous educational and work experience, as well as subject area ability and expertise demonstrated during the recruiting
process. For the Florida Standards
Assessments, preference will be given to scorers who have prior experience scoring responses in the content areas being assessed. DRCs
dedicated human resources/recruiting staff are
confident in their ability to hire individual
scorers for the Florida Standards Assessments
that best utilize scorer strengths.
Prior to each scoring session, DRCs content
specialists work with human resources/recruiting staff to select scorers based on experience
and background. The content specialists will
divide the scorers into teams and assign each
team a team leader. All of DRCs scorers will be
made aware that no scoring materials are to
leave the scoring center. They must sign legally binding confidentiality agreements stating
that they are aware of the secure nature of the
work before training and scoring begins. DRC
scorers will be frequently reminded of the
importance of confidentiality and security during the course of the project.
DRC insists that scorers work the DRCs full
core hours from 8:30 a.m. to 4:00 p.m.; additionally, scorers must be present for each full
training session or they will not be allowed to
work on that project. This has been a steadfast
policy at DRC. Strict adherence to core hours
allows DRC to plan timelines accurately for its
Page 7.5-34
clients. DRC will take all necessary steps to

anticipate the correct amount of time needed
for scoring; they realize that staying on schedule is of the utmost importance in order for student results to be reported in a timely manner.
DRC understands that all scorers who work on
the FSA will have their degrees verified by an
independent, third party verification organization and that any scorer for whom a degree
cannot be verified before the start of live scoring will be dismissed. Potential scorers in
Florida may include retired Florida teachers
and substitute teachers provided they are not
actively teaching in Floridas public school system for the remainder of that year.
Training Team Leaders
Before scorer training begins, DRC will provide
team leaders with comprehensive training. Team
leader training will last approximately 2 days
and follow the same process used in the scorer
training (detailed below), although it is more
comprehensive due to the training and oversight
responsibilities required of the team leaders.
During their training, team leaders will be
required to annotate all of their training
responses with official Florida annotations. To
promote room-wide scoring consistency, it will
be imperative that each team leader imparts
the same rationale for the score assigned to
each training response. This consistency will
ensure that scorers assign the correct scores
for the correct reasons. Once the team leaders
have qualified for operational scoring, they will
prepare for the arrival of their teams of scorers. Teams typically consist of 1012 scorers
per team, although DRC will utilize smaller
teams if there is a need to increase the number of team leader backreads.
Training and Qualifying Scorers
DRC welcomes the Department to oversee the
training process for the Florida Standards
Assessments through visits to DRCs proposed

in-state Florida scoring centers, conference
calls, and the review of handscoring reports.
DRC will also provide access to handscoring
reports throughout each scoring session.
The training, which is specific to each item,
will begin with a room-wide presentation and
discussion of the rubric which defines each
score point. After reviewing the rubric, the
scoring director will review the anchor set.
The anchor sets contain multiple examples of
each score point, each with an annotation that
explains the responses score using language
from the rubric. Each scorer will be supplied
with a paper copy of the scoring guide, rubric,
and anchor set so that he or she can easily reference these materials while scoring responses. The rubrics and anchors are secure materials that do not leave the scoring room.
After reviewing the rubric and anchor responses, the scoring director will train the scorers
on how to use the Image Handscoring System.
During training, scorers will be seated at imaging stations and grouped by team. Each scorer
will be assigned a unique ID number and password for logging in to DRCs Image
Handscoring System. The scorers will log in to
the Image Handscoring System to access training and qualifying sets in order to continue
their item-specific training.
Once the scoring director has helped the scorers log into the Image Handscoring System,
the scorers will access training sets, which will
serve as an opportunity to practice scoring.
The entire room will work on one training set
at a time. After the scorers complete each
training set, the scoring director will review
reports that detail how each scorer performed
and how each response within the set was
scored. Next, the scoring director and/or team
leaders will use this information to lead a thorough discussion of each set, answering questions to help ensure that the scorers understand the proper way to apply the rubric to
each of the training responses.
For operational assessments, after the scoring

guide and all training sets have been discussed, scorers, just like team leaders, will be
required to demonstrate their ability to apply
the scoring criteria by qualifying (i.e., scoring
with an acceptable agreement rate agreed
upon by the Department) on at least one of
the qualifying sets. The scorers will access the
qualifying sets via the Imaging System so that
their scores can be captured and reported.
Any scorer who does not qualify by the end of
the qualifying process will not be allowed to
score actual student work.
Scoring Procedures
DRC believes that its training, scoring, qualifying, and monitoring processes are the best in
the industry. All of these processes have been
used for years to successfully score large-scale
assessments similar in scope, size, and complexity to the Florida Standards Assessments.
DRC will uphold the same level of dedication
to accuracy and quality for this program.
DRCs general scoring procedures are summarized below:
Scorer Training and Qualifying
Scorers are seated at imaging stations
and are assigned unique ID numbers
and passwords.
The scoring director provides detailed
directions for use of DRCs computerized handscoring system.
The scoring director trains the scorers
using item-specific anchor sets and
training sets.
Scorers must demonstrate scoring proficiency on item-specific qualifying sets
before scoring live responses.
Routing Responses to Ensure Blind
Second Reads
The Image Handscoring System ensures
that responses are routed to qualified
Page 7.5-35
scorers until the prescribed number of

reads is performed for all responses.
Scorers cannot tell if they are the first
or second scorer.
Monitoring Scoring (Handscoring Quality
Control)
Ongoing quality control checks and procedures monitor and maintain the quality of the scoring sessions. If unusual
data are observed, DRC will investigate
and resolve any issues.
Responses can be retrieved on demand
(e.g., specific batch files, specific grades,
specific students) should the need arise
during or subsequent to the handscoring process.
If needed, responses can be rescored
based on item- or response-level information, including item number, date,
score value assigned, or scorer ID.
Handling Unusual Responses
Scorers can forward responses to team
leaders for assistance.
Responses requiring special attention,
including nonscorables and alerts, are
routed to scoring directors for review
and resolution.
Routing Responses to Ensure Blind Second
Reads
DRCs Image Handscoring System separates
responses by item and subject and routes to
qualified scorers who read each response and
enter the score. The process of routing and
scoring responses continues until all responses
have received the prescribed level of readings.
Scorers cannot tell if they are conducting first
or second readings; all readings are blind.
Because of the high stakes associated with the
Reading portion of the Grade 3 ELA/L the
entire grade 10 ELA/L assessments, AIR has
devoted extra handscoring resources to these
Page 7.5-36
tests. Specifically, these assessments will be

double human scored when delivered on
paper, and the responses to the grade 10
Writing prompts delivered online will be both
machine and human scored, with 100% human
resolution of any difference between the
scores. For the high-stakes Algebra I assessment, all constructed responses on paper tests
will be double human scored, with resolution
of nonadjacent scores. For responses writing
prompts in all other grades, 25% of the
machine-scored responses will be routed to
human scorers based on algorithms identifying
potentially anomalous responses. For all other
portions of the assessment, 20% of the
responses will receive a second reading for the
purpose of monitoring and maintaining sufficient inter-rater reliability. DRC will work with
the Department to establish a specific set of
scoring rules for resolving disagreement
between first and second readings by utilizing
subsequent independent readings performed
by team leaders and scoring directors.
Handscoring Quality Control

DRCs scorer quality control monitoring/
reports system is shown in Exhibit 7.5.4.5-1.
Monitoring Scorers
DRC runs handscoring reports on demand in
order to monitor progress and maintain handscoring quality control. The Department will be
given access to all handscoring reports to be
run as needed. Reports will also be automatically generated overnight. DRC will provide the
Department with access to these reports in
PDF format on a daily or on-demand basis.
During the handscoring process, the scoring
directors will meet with their team leaders
each morning to review the statistics generated from the previous days work. If scoring patterns are apparent among individual scorers,
team leaders will deal with these issues on an
Exhibit 7.5.4.5-1: Scorer Quality Control Monitoring/Reports System
individual basis. DRCs imaging system allows

a team leader to determine read-behind rates
(frequency of monitoring) for each scorer. If a
scorer appears to need clarification of the
scoring rules, or is scoring tentatively, DRC
will monitor one out of five readings, making
adjustments to that ratio as needed. The imaging system will randomly select which images
the team leader monitors.
DRC monitors attendance of their scorers closely. DRC understands that any scorer who is
absent for two or more days consecutively while
scoring any portion of the FSA, will have to
retrain/re-qualify before again scoring live student work. DRCs scoring directors will ensure
that this retraining/re-qualifying process occurs
in an orderly fashion. Any scorer who fails to
re-qualify will be released from the project.
DRC will also monitor the inter-rater reliability. If a scorer falls below an acceptable rate of
agreement, the team leader will retrain the
scorer. If a scorer fails to improve after retraining and feedback, DRC will remove the scorer
from the project. In this situation, DRC
removes all scores assigned by the scorer in
question. The responses will then be re-dealt
and rescored.
DRC does not report on scorer performance

after the fact, as some contractors do. DRC
believes that scorers with less-than-acceptable
scoring patterns must be identified immediately and those patterns corrected. DRC has
worked diligently to devise effective monitoring reports and procedures to accomplish both
detection and correction. Accurate and consistent results are the backbone of all handscoring activities. The following methods used by
DRC guarantee scoring quality:
Page 7.5-37
Rigorous training and qualifying for each

item, as detailed earlier in this proposal,
ensures a pool of scorers who will apply
consistent and accurate scores.
Daily review/calibration occurs each day of
the project in the form of a morning review
of selected papers from the Scoring Guide
and an afternoon two- to three-paper calibration set consisting of Departmentapproved responses.
Recalibration sets re-focus scorers on the
scoring standards by comparing the predetermined score to that assigned by the scorer.
Validity responses detect possible room
drift and individual drift. Validity reports
compare scorers scores to predetermined
scores. Validity responses are seeded to
scorers. Scorers cannot distinguish validity
responses from live responses, making this
a powerful measure of quality control.
Team leaders conduct routine read-behinds
to observe, in real time, scorers performance. Team leaders utilize live, scored
responses to provide ongoing feedback and,
if necessary, retraining for scorers.
Inter-rater reliability and score point distribution reports are generated daily or on
demand to monitor scorer reliability and
maintain an acceptable level of scoring
accuracy. The reports compile individual
scorer data, including scorer identification
number, number of responses scored, individual score point distributions, and exact
agreement rates. DRC will investigate any
issues and resolve any problems identified
by the reports. DRC is proud of the scorer
quality control reports that it has developed for handscoring projects.
Quality Control Reports
AIR and DRC look forward to consulting with
the Department to determine the best way to
make program handscoring reports available for
use by the Department. AIR team member DRC
Page 7.5-38
is proud of its suite of quality control reports

developed specifically for handscoring projects.
DRC will provide the Department with annotated samples of all its handscoring reports so that
a determination can be made as to which
reports the Department would like access to
and how best to customize those reports for the
Florida Standards Assessments. The same
reports will be generated during operational and
field test scoring. Because handscoring quality
reports are produced in real time, the
Department can be assured that immediate
action will be taken to resolve scoring discrepancies, within minutes if necessary, of the first
and/or second reading. AIR is confident that
DRC will supply handscoring reports to meet
the Departments needs. Please see Appendix L
for sample DRC quality control reports.
If requested, DRC will convert the handscoring
reports into an agreed-upon format and send
them to the Department on a weekly, bi-weekly, or daily basis. It is also DRCs practice to
provide clients with summary handscoring
reports as well as project technical reports.
DRC plans to continue this practice with
Florida, thereby ensuring that these materials
will be available for review at the annual handscoring debriefing meeting.
DRC currently uses a number of reports to
monitor the quality and/or effectiveness of various aspects of handscoring projects. DRCs
reports have multiple parameter selections, so
often more than one individual report is
included under the same report name. DRC
looks forward to tailoring these reports to meet
the Departments needs. Exhibit 7.5.4.5-2
describes some of the quality control reports
that are provided for many of DRCs assessment clients.
DRC is experienced at producing the handscoring reports identified by the Department
in the ITN. Below, the reports identified in the
ITN are correlated with the DRCs typical
handscoring quality reports described in
Exhibit 7.5.4.5-2:
Exhibit 7.5.4.5-2: Quality Control Reports
Report
Report Specifics
DRCs Scoring Summary Report provides daily and cumulative inter-rater reliability results, score point distribution
data, and production volumes for each reader and item.
Inter-rater Reliability
Monitors how often scorers are in exact agreement with each other and ensures that an acceptable agreement rate is
maintained. This report provides daily and cumulative exact and adjacent inter-scorer agreement and the percentage of
responses requiring resolution (only if required). The calculations for this report are as follows:
Scoring Summary Report
Percent Exacttotal number of responses by scorer where scores are equal divided by the number of responses
that were scored twice.
Percent Adjacenttotal number of responses by scorer where scores are one point apart divided by the number
of responses that were scored twice.
Percent Non-Adjacenttotal number of responses by scorer where scores are more than one score point apart
divided by the number of responses that were scored twice.
Score Point Distribution

Monitors the percentage of responses given to each of the score points. For example, for items on a 04 point scale,
this daily and cumulative report shows how many 0s, 1s, 2s, 3s, and 4s a scorer has given to all the responses he or
she has scored at the time the report is produced. These percentages can be compared to room-wide percentages
to detect individual scoring issues.
Production Volumes
This report also indicates the number of responses read by each scorer each day so that production rates can be
monitored. Additionally, it includes totals for each item, so that progress toward completion can be monitored.
The Item Status Report
(Completion Reports)
Monitors the progress of handscoring. This report tracks each response and indicates the status (e.g., needs a second
reading, complete). This report ensures that all discrepancies are resolved by the end of the project. Information from
this report will be used to create room-specific Daily and Cumulative Completion Reports to compare on a daily and
cumulative basis the number of readings completed in comparison to projected completion targets.
The Read-Behind Log
Used by team leaders/scoring directors to monitor scorer reliability. Team leaders randomly select and read scored
responses from each team member daily. If the team leader disagrees with the scorers score, remediation occurs, with
either the team leader or the scoring director. This has proven to be a very effective form of feedback because it is
implemented with items live scored by individual scorers.
Training and qualifying records are kept on every reader for every set taken. These reports can be run using various
parameters to meet the needs of the Department (by performance task or set, reader, team, by site/grade/subject)
Training/Qualifying Reports and can be run daily or cumulatively. These reports include the information described in the ITN as the Cumulative
Training Report by Reader; Cumulative Training Report by Document; Cumulative Qualifying Report by Reader;
Cumulative Qualifying Report by Document.
These two processes are conducted throughout the entire scoring session. Both processes compare predetermined
Recalibration/
scored responses to readers scores for the same set of responses. Additional responses are given to individuals if the
Validity Reports
scoring director feels it is warranted. These reports can be run at the individual, team, or room level in order to detect
individual, team, or room-wide scorer drift.
Reader Summary Report
This report, which is similar to the Timing Reports, provides, by site, by team, and by item, the daily and cumulative
number of readers engaged in scoring and the daily and cumulative number of responses read by each reader. DRC will
work with the Department to enhance this report so that it provides the information needed from the Timing Reports.
The Responses Read by

Reader Report
Identifies all responses scored by an individual scorer. This report is useful if any responses need
rescoring due to potential scorer drift.
Page 7.5-39
1. Cumulative Training Report by Reader

DRC report name: Training/Qualifying
Reports
2. Cumulative Training Report by
DocumentDRC report name:
Training/Qualifying Reports
3. Cumulative Qualifying Report by Reader
Reports
4. Cumulative Qualifying Report by
DocumentDRC report name:
Training/Qualifying Reports
5. Cumulative Reader Performance Report
Reports, Validity Reports, Scoring
Summary Report
6. Daily and Cumulative Score Point
DistributionDRC report name: Scoring
Summary Report
7. Daily and Cumulative ValidityDRC
report name: Recalibration/Validity Reports
8. Daily and Cumulative Inter-rater
ReliabilityDRC report name: Scoring
Summary Report
9. Score Point Distribution for Standard
Curriculum StudentsDRC report name:
10. Daily and Cumulative Score Point
Distribution and Inter-rater Reliability by
ReaderDRC report name: Scoring
Summary Report
11. Daily and Cumulative Validity by Reader
DRC report name: Recalibration/Validity
Reports
12. Daily and Cumulative Validity by Validity
SetDRC report name:
Recalibration/Validity Reports
13. Daily and Cumulative Read-Behind
ReportsDRC report name: Read-Behind
Reports
Page 7.5-40
14. Daily and Cumulative Timing Report

Similar to DRCs Reader Summary report,
which will be modified with input from the
Department and named Timing Reports
15. Daily and Cumulative Completion
ReportDRC report will be created with
input from the Department using data from
the Item Status Report
Handling Unusual Responses: Alerts and
Nonscorables
DRC will apply a nonscorable code to unusual
or aberrant responses that cannot be assigned a
score. Prior to scoring, AIR and DRC will work
closely with the Department to define what
constitutes a nonscorable response. During
handscoring, DRC will contact the designated
Department representative to obtain a ruling
on any response that cannot be assigned a
score or a nonscorable code based on current
understanding. The Image Handscoring System
has a built-in functionality that can be used to
forward all potential nonscorable responses to
the scoring director. Only the scoring director
will be able to assign the nonscorable code.
To handle possible alerts (responses indicating
potential issues that may require attention at
the state or local level, such as potential security breaches or concerns about a students safety), DRCs Imaging Handscoring System gives
scorers the ability to alert individual student
responses. Alerted images are routed to the
scoring director who will print any responses
deemed to indicate a potential issue. At no
time during scoring do scorers have access to
any demographic information of any students.
Next, these alerts are routed to a handscoring
senior manager who reviews them and, if needed, sends copies of the students responses to
DRC Project Management staff. The DRC project manager will forward copies of the alerts to
the Department. If the Department would prefer a different process for handling alerted
responses, AIR and DRC would be happy to
review and tailor the process with the
Department upon award of the contract.
7.5.5 Reporting
speed the distribution of results to schools,

districts, and families;
AIR is pleased to offer multi-modal delivery of

reports. We recognize the critical need for
paper reports to go home to parents, and AIR
has led the industry in design reports that parents clearly understand and can act on. We
also recognize that printed reports can offer
only a limited view of data. You cannot drill
down into interesting statistics when they are
presented on paper. AIRs mature online
reporting system allows educators to interact
with the data, slicing, dicing, and exploring the
data to uncover implications for their students.
provide secure interactive reporting capabilities that allow educators at all levels to
explore the data and their implications for
instruction; and
All of our reporting systems, paper and online,

bear the responsibility of communicating to
stakeholders (parents and students in particular) who are not technical measurement
experts. While most psychometricians deem
reports valid if the information they present is
true, we recognize a greater validity burden on
our reports.
We strive to have our online reporting system
and our paper reports convey information that
is correct. It is not sufficient to say something
true, we must say (or show) it in such a way
that the typical reader is likely to understand
it in a correct and useful way. Only when this
communication goal is met can the reports
serve their real purposeto help parents and
educators contribute to student learning.
We achieve this goal by applying principles of
cognition to the report design. For example,
we know that the human eye will group
together or equate things that are the same
color. Having different things share a color can
cause confusion. Having things that should be
compared be the same color can cause the
comparison to happen without conscious effort
on the part of the reader.
AIR offers an online reporting system that will
reduce costs by eliminating the need to

produce and ship CDs and paper reports
for schools and districts.
All of these benefits accrue while simultaneously improving data quality and extending the
range of data quality review through automation.
The online reporting system will generate
dynamic, interactive reports for educators and
allow educators to generate data files, speeding
the delivery of these files. The online reporting
system also supports individual student
reports, which are typically available to
schools and teachers. In many states these
reports constitute the parent reports, which
schools print and ship to parents. Here, in
accordance with the ITN requirements, we
propose a separate paper parent report. The
online reports are highly configurable, but the
system design imposes some limits on what
can be displayed. AIRs paper reports are the
most flexible available in the industry.
Given the rapidly changing landscape of online
assessment, reporting solutions based on conventional processes that involve producing
static PDFs and shipping data files on CDs can
be inefficient and costly. The Departments
invitation to submit proposals based on innovative solutions that take advantage of mature
technologies in online student assessment is
especially relevant to reporting student scores.
AIRs plan offers Florida a streamlined, transparent process that will speed reporting while
preserving every aspect of Floridas current
quality processes. In most of the states in
which we work, the data are made available to
Page 7.5-41
educators immediately, but considered preliminary until the final data delivery to the
department at the end of the year. The realtime reporting combined with real-time ID
(see Section 7.5.2) and the continuous data
cleanup period (see Section 7.5.6) will speed
reporting, which will enable Florida to move
the testing window closer to the end of the
year. The continuous cleanup process will
improve transparency, while moving the demographic cleanup off the critical path by making
cleanup possible during the testing window.
Our solution has been successfully used for
years in Oregon, Ohio, Florida, Hawaii,
Delaware, Florida, and Minnesota. We believe
that the solution we outline below for reporting the English Language Arts/Literacy
(ELA/L), Mathematics, and End-of-Course
(EOC) assessments offers several benefits to
Florida educators including
rapid, flexible online reporting, providing
both standard and customized reports,
available immediately (upon completion of
scoring for human-scored tests);
access to the Online Reporting System for
users to download on demand, standard
and custom PDFs, spreadsheets, and data
files;
custom-designed, full-color paper reports
for parents to understand the full meaning
and import of student and school scores;
proven ability to successfully deliver paper
score reports at scale and on time;
benchmark reports for educators that identify strengths and weaknesses at the benchmark level for the class (this feature is psychometrically viable for adaptive tests or
with a sufficient number of operational
fixed forms spiraled within classrooms); and
Page 7.5-42
mobile-compatible online reports that

teachers can access on tablets or even
smartphones.
Should the Department decide to continue
providing static PDF reports for schools and
districts and data files on CDs as is currently
done, AIR commits to producing these deliverables outlined in Section 7.5.5 of the ITN.
However, the solution we propose is able to
more efficiently meet all of the reporting
requirements outlined in the ITN, including
reporting scores for multiple assessments
(EOC and ELA/L and Mathematics assessments); reporting scores for assessments with
handscoring components; delivering multiple
data files; and distributing score results within
exceptionally short timeframes. Most importantly, the Online Reporting System described
in the following pages
provides all of the same data users currently receive;
provides districts, schools, and teachers
with valuable additional data;
offers users the ability to customize their
reports;
enables users to immediately generate their
own data files; and
generates PDFs on demand for use in professional development activities.
To highlight the differences between our proposed solution and Floridas current reports,
we provide a visual comparison of the two
reporting methods in Exhibit 7.5.5-1.
Our Online Reporting System is designed to
support educators as they evaluate the needs
of their students and reflect on their own curricula and practice. When educators enter the
system, they immediately see summary data
about their students: district personnel see
summary data about the whole district, and
teachers see aggregates of the students in their
classes. Clicking on any piece of data enables

the user to explore the data underlying the
aggregate, drilling into progressively smaller
groups of students, disaggregating the content
by strands or other reporting categories, or
tracking students back in time to examine
trends. This interactive approach helps educators at all levels leverage the data to improve
instruction.
To help identify the considerations that the
Department may want to review in making this
decision, we also provide a top-line comparison of our proposed automatic online reporting
solution with Floridas current reporting methods for schools and districts in Exhibit 7.5.5-2.
As illustrated in the Exhibit, moving away
from static PDF reports to an interactive
online reporting system will allow Florida
schools and districts to view exactly the same
data they currently receive while providing
them with valuable added data and functionality including the ability to
download data files delivered immediately
upon completion of handscoring through
ORS;
generate on-demand PDF versions of
reports for stakeholders to distribute and
use in professional development activities;
generate reports that include data on
Floridas subgroups;
access class-level data for teachers (we will
interface with the Departments rostering
system to report by teacher and class);
build customized online reports through a
series of filters (demographic filters, accommodation filters, longitudinal filters, etc.);
build custom rosters for groups of students;
investigate data by drill-down capabilities;
access longitudinal data in the form of

trend reports at the student, class, school,
and district levels;
access benchmark reports at the class level,
if the assessment design supports it; and
access any of the above data on tablets.
The discussion that follows is divided into four
sections:
An overview of the Online Reporting System
A summary of our online capabilities and a
detailed comparison of Floridas current
reports with what schools and districts
would receive using AIRs Online Reporting
System
A description of the processes we use to
produce the online reports and ensure
quality assurance
A description of our paper report development processes and our quality assurance
processes for these reports
Online Reports
Overview of the Online Reporting
System
To help users quickly locate the data they
need, navigation in the reporting system is
designed to mirror the instructional decisionmaking process. The user can intuitively navigate in any of the three dimensions inherent
in the data, and these three dimensions parallel the three kinds of questions that the data
can help the user answer:
1. Who? The data can be displayed at levels of
aggregation anywhere from the individual
level for a specific student up to the entire
state. Demographic breakdowns are immediately available at any level of aggregation.
Page 7.5-43
Page 7.5-44
No teacher or class
level data is provided.
Floridas current reports provide lots

of data, but the presentation does
little to guide users on how to use it.
Districts and schools currently
tly receive
e a report containing
containing
results for all schools in the state, and a separate
demographic report containing sub-group data.
Current PDF reports in Florida contain a static

view of the data; what you see is what you get.
Floridas Current Reports
AIRs online reporting system includes exactly the same data as Floridas PDF
reports and provides users with additional data and functionality...
Exhibit 7.5.5-1: Comparison of AIRs Proposed Reporting Solution and Floridas Current Reports
Inbox (0)
Search Students
Test
T
estt Managem
Management
M
g ent Center

2342
1753
9999956789
Student D
Student E
365
298
371
348
352
Once a custom student

roster is created, users can view
students current year scores as
well as past year scores.
2005
9999934567
9999945678
Student C
2275
2115
9999912345
9999923456
Student A
Student ID
Name
Student B
Developmental
Scale Score
Scale Score
Manage Rosters
M
Scale Score and Achievement Level Summary

Grade 5 Mathematics 2014-2015
Class B
Score Reports
Each report is available for

district, school, teacher, class,
or any special roster of students
that users want to follow.
See the performance

of different sub-groups
of students by using the
filters.
View a roster
of schools in
their districts.
Achievement
Level
This page:
?
Number:
Base Ten
and
Fractions
Help
Export
Geometry and
Measurement
Mea
Meas
M
Measu
Me
eas
ea
easu
eas
asu
asu
surre
remen
reme
emen
eme
e
em
ment
me
n
Print
14
10
11
12
1
2
Expressions,
Expre
p ssions,,
Equations,
Equat
q
ions,,
and
a
nd Statistics
Statistics
Definitions
Our magnifying glass

[
] allows users to
explore any aspect of
the data that is useful
to them.
In AIRs online report system, users can

create custom rosters for teachers by
using the Manage Rosters button.
The magnification
icon can take user to
trend reports containing
historical data.
Color bar charts

help readers
interpret results.
Print reports
or export data to
Excel.
AIRs Online Reporting System
Page 7.5-45
Exhibit 7.5.5-2: Topline Comparison of Current Reporting Method to AIRs Online Reporting System
Features of Current Method of Delivering Static PDFs to
Schools and Districts
Users must wait to receive PDF reports and CD shipments
Teachers do not receive any class-level data
PDF reports contain one static view of the data

Users must synthesize data across multiple PDF reports to
identify a trend
Districts and schools must wait for late batch reports and data
files to arrive on CD
Currently, districts and schools do not receive trend data
Must piece together past year data to identify how a teachers
current class performed in the previous year
Districts must wait to receive their own hard copies of
individual student reports
Must manage and archive multiple PDF reports and CD
shipments for ELA/L and Mathematics assessments, late batch
shipments, and quarterly EOC reports and CDs
2. What? The subject area data can be broken

down into finer or coarser chunks of content. Navigating this dimension allows the
user to travel from subject to content
strand to benchmark and back.
3. When? When data are available over time,
the system allows the user to view a data
trend over time or toggle to a fixed point in
time.
Each navigational step changes the reporting
display, providing richer context when interpreting class performance or an individual students performance. The system, of course,
contains many reports. However, the interface
design encourages users to think about the
substantive, educational questions to which
they need answers and access information
Page 7.5-46
Features of Accessing Interactive Data in AIRs Online

Reporting System (ORS)
All reports, data, and data files are available immediately
upon completion of handscoring and can be batch printed or
exported to Excel
Custom class rosters and class-level data are available for
teachers and can be used for professional development
activities
ORS provides users with the ability to build custom views
through a series of filters and print or download these reports
Users can investigate data trends immediately by using ORSs
drill-down capability
As late batch records are processed, they are populated in ORS
and data files are available immediately for users to download
Trend data at the student, class, school, and district levels are
available in ORS
ORS automatically provides educators with past year data on
their current cohort of students, even if students changed
districts or schools
Districts and schools can print a PDF of the student report from
ORS immediately upon availability
ORS eliminates multiple sources of reports for multiple
assessments (all EOC reports and ELA/L and Mathematics
reports, PDFs, and data files are accessible in a centralized
location)
from that perspective. As test results come

pouring in from the tests, the reporting system
provides stakeholders with immediate access
to the information they need about students
strengths and weaknesses.
Summary of Online Reporting

Capabilities
AIRs Online Reporting System can replace
schools and districts static reports with
dynamic, timely reports that are designed to
complement the instructional decision-making
process. Users can securely access their data
from anywhere, and role-based access ensures
that users have access only to the relevant
data that they have access to. Users have complete control over their reporting experience
nearly every page provides a data display, and
users can change data dimensions, sort data,

disaggregate data, and show/hide data instantly
using a range of filters and features. Next, we
provide a description of these specific features
and capabilities of the Online Reporting
System.
Delivery of District and School Data Files
Currently in Florida, districts and schools
access their data files only after all of the
hand-scored files are loaded onto CDs and into
the Online Management System. With AIRs
Online Reporting System, the Retrieve Student
Results tool allows users to export a raw data
file for district, school, teacher, or roster as the
data becomes available. For example, as a
class or school completes the ELA/L and
Mathematics tests and handscoring is complete, these data will be available in the Online
Reporting System immediately.
In Section 7.4.8, we recommend that the
Department establish a date after which the
data are considered final, allowing authorized
district personnel to update demographic,
enrollment, and other data through TIDE until
that date. These changes appear in the reporting system and data files in near real-time.
The reporting system enables users to request
a data file for that class, school, or district at
any time. This feature proves very helpful during the cleanup process, as many districts can
use this data to automate their data checking
and cleanup.
The specific content of the data files is configurable and will be determined by the
Department. Data files can be configured to
include students identifying information, student demographic information, test participation status, and performance information at
the subject and subscore levels. The files can
be exported into Microsoft Excel or other
delimited file format so that they can be easily
loaded into other software platforms. The

reporting tool will also temporarily store the
file for a set period of time so that the user can
access it later. We propose to deliver all of
Floridas current data files, including the SSR,
SAR, DSRs, DARs, and late batch SSR and DSR
and the demographic SAR, DARs, and DSRs, to
schools and districts using this streamlined
delivery capability.
In addition to the raw data files in the Retrieve
Student Results tool, users can export any
table on any report in the reporting system.
An Export button is available on every
report, and the user can immediately export
the table as an Excel file. Printed tables carry
with them a footnote indicating the data on
which the report was generated.
On-Demand PDF Copies of Reports for
Stakeholders
An individual students report can be printed
or saved as a PDF file. The PDF format makes
electronic distribution easy and ensures that
the report will print properly regardless of the
users browser. Users can batch print or save
multiple students reports in PDF format. The
reporting tool can also PDF versions of students written responses for writing tests. This
capability will allow Florida schools and districts to print their own copies of student
reports instead of waiting for these reports to
be shipped to districts.
parents a printed copy of a report, and this
proposal includes printing and shipping of parent reports. These processes are described
below.
Custom Rosters and Disaggregated Data
In addition to the expected aggregated reports
for a district, school, teacher, and class, the
reporting tool allows users to create custom
Page 7.5-47
rosters of students so that they can be tracked

and reported on separately. For example, a
teacher may teach a group of students in his
or her class who are receiving extra help after
school and may want to track their progress
separately. He or she can create a second roster of only those students to report on their
aggregate performance from the group. This
may help the teacher to begin to answer questions about the effectiveness of the extra help
or the strengths and weaknesses of this subgroup of students. All aggregated reports are
calculated from the student level up, so as new
students test, reports for the roster, teacher,
school, and district are updated instantaneously.
All aggregate reports in the reporting system
can also be disaggregated by subgroups defined
by the Department. For example, reports can
be disaggregated by demographic characteristics like gender, ethnicity, or English language
learner (ELL) status. The reporting system can
also be configured to disaggregate by multiple
characteristics at once (e.g., gender by ethnicity).
Reporting on Multi-Year Student Data
When a user logs in to the reporting system,
he or she is automatically taken to his or her
reports for the current test administration.
Educators can track their current students
back in time, a capability that helps teachers
understand where students may have faced
challenges or exhibited strengths over time.
When reflecting on his or her own practice, a
teacher can review reports for students from
past administrations, but only for the years
leading up to and including the year he or she
taught those students. As long as the user is
authorized to use the reports, the user can
access any of his or her reports from any test
administration with the click of a button, even
Page 7.5-48
if the students from those years have changed

schools or graduated.
Similarly, users can access previous years
scores for their new students. For example, a
middle school teacher may begin teaching a
class of students who have come from multiple
elementary schools. Though the middle school
teacher is not authorized to view the school
reports for the elementary schools, the teacher
can create a report based on the students previous test scores. This allows the teacher to
assess the strengths and weaknesses of the
class and of individual students based on their
previous scores and adjust instruction accordingly.
Customizability
All text and measures in the reporting tool are
configurable, and AIR reporting staff will work
closely with the Department to define all text
and measures as part of the specifications
development (see Section 7.5.1 for more detail
about Reporting Specifications). The reporting
system can accommodate different kinds of
data, and various reporting elements are available (e.g., barrel charts, bar charts, trend
charts) to help best display those data.
The user can also customize his or her reports
with some of the system tools. Users can
choose to show or hide comparison data on
their aggregate reports. Users can also choose
to show or hide individual columns in a table.
In addition, a user can sort most columns in
ascending or descending order to group reports
by categories such as score, performance level,
or demographic subgroup.
Drill Downs
The report navigation is designed to allow the
user to drill down through the different dimensions of their data. For example, a user can
move from the school reports to the teacher
reports to the roster reports to the individual

student reports. At any point, the user may
also want to switch from subject-level reports
and drill down into the subscore reports or
benchmark reports. The user can also switch
from a specific administration to view a trend
report over time. This flexibility allows the
user to drill down to any desired level of granularity in the data and then modify dimensions
of the report with a couple of clicks without
having to go back to the beginning and drill
down again.
Mobile-Compatible Reports
All reports and features of the Online
Reporting System are tablet accessible. For
example, as a teacher walks around the classroom, he or she can use a tablet to access the
Plan and Manage Testing tab to confirm which
students have completed their tests. If a student cannot remember his or her login credentials, the teacher can quickly use the Student
Search feature to find this information. In
addition, all tables and data displays adjust to
the tablet screen size making it easy to view
rosters, request downloadable data files, or
view any number of custom-created reports.
Transfer of Return Files to the Department
Student data files will be provided to the
Department on a regular basis and will be
transferred via a secure FTP site. The
Department can specify how frequently the
data files should be posted (e.g., nightly, weekly), as well as the data file layout.
Report Security and Control of Access
The reporting tool belongs to AIRs suite of
programs and can be accessed through the
Common Login System (CLS), which allows
educators and administrators to seamlessly
move between authorized programs without
having to reenter their login information.
Secure access rules can also be set up to allow

parents to view their childs individual student
report electronically. When a user logs in for
the first time, he or she must agree to the
terms of a non-disclosure agreement before
using the reporting tool. The reporting system
also monitors each users session. The system
will automatically log the user out after a period of inactivity.
Educators and administrators can have different user roles, which are defined in the TIDE
system. These user roles can be defined by the
Department and will specify which reports
users have access to. When a user signs in, he
or she will only see the reports and student
data to which he or she has access. Besides
ensuring that users do not have access to
unauthorized reports, this also helps to
improve the user experience by only displaying data and reports that are relevant to the
user. For example, when a principal logs in, he
or she will only see the information for the
students and teachers in his or her school. As
with all of our systems, student data are protected by the same physical, network, and
software security as our test delivery system.
Please see Section 7.4.8.4 for more details.
Interpretation of the Reports
AIR knows that a report is not useful if it is
not interpreted validly. The reporting tools
design, layout, graphical displays, and use of
color have all been implemented based on rigorous focus groups conducted with educators,
administrators, and parents. Navigation
emphasizes the context of a students performance by relating it to aggregate performance
and trends over time. Colors associated with
performance are used consistently throughout
the report, and graphical displays are chosen
carefully to maximize the amount of information conveyed. For example, bar charts can
illustrate the percentage of a teachers students
at each performance level, but the relative
Page 7.5-49
positioning of each teachers bar chart can also

efficiently communicate which teachers have a
higher performing group of students, even if
there are differences by performance level.
New users or users that require a refresher on
how to use the reporting system, can access an
intuitive user guide that provides step-by-step
instructions for every report, information
about each measure included in the reporting
system, and screenshots to explain how to use
the system. The user guide is accessible from
any page in the reporting system. The reporting system also has a configurable
Definitions tab that provides a quick reference for users who want more information
about how to interpret a report (for detailed
information on the Online Reporting System
User Guide, please see Section 7.3.7.5,
Interpretive Products).
Test Management Center
The Test Management Center allows users to
track test participation in real time to help
plan their testing schedules. For example, the
reports can be filtered by the percentage of
students who have completed a test or whose
tests are started but not yet completed. Highlevel reports are available so that school and
district users can see the percentage of students who are completing each performancebased test and ready to take the end-of-year or
End-of-Course tests.
Student Search
Sometimes a user just wants an individual students report and wants it quickly. The student
search lets users search by student ID, student
name, or any partial student ID or name. The
search results provide a list of any students
who match that criterion, as well as a table
with links to the individual student reports for
any tests that student has completed. As
always, users can only see students that they
Page 7.5-50
have access to, though users can also search

students from previous years. This allows a
user to see an individual students results with
a minimal number of mouse clicks.
Comparison of AIR Proposal and Current
Florida Reports and Data Files
Exhibit 7.5.5-3, on the following pages, summarizes Floridas current data files and PDF
reports and identifies the additional capabilities and data that are available through AIRs
Online Reporting System (ORS).
Next, we describe the process to produce our
online reports along with our quality assurance
processes.
Configuration and Design of Online

Reports
Producing real-time, online reports for
teachers, schools, and administrators
involves the following steps:
Initial skinning of the Online Reporting
System to finalize the look and feel of all
online reports
Working with the Department to gather
requirements and configure the system,
including the specific measures and displays to be presented
Conducting Quality Assurance
that the look and feel of the online reports is
consistent across assessments and with the
Departments paper score reports. Our designers will work with Department staff to select a
skin for the reporting system composed of
colors, logo, headers, and footers that will give
each report and data display a unified style.
To ensure that the online reports meet stakeholder needs, AIR will also work closely with
Exhibit 7.5.5-3: Summary of Floridas Current Data Files and PDF Reports Compared with AIRs ORS Capabilities
Name of Report/Data File
Floridas Current
Reports and Data
Files
Available in ORS
Delivered via data

le in ORS
District Report of Schools (Roster)
District Report of Students (Roster)

Late Reporting District Report of Students (Roster)
State Summary
District Summary
State Report of Districts (Roster)
District Data Files

District Student Results File
State Aggregated Results File
District Aggregated Results File
Demographic District Student Result File
Demographic District Aggregated Results File
Demographic State Aggregated Results File
Late District Student Result File
District Student Results File
District Reports
District Report of Schools (Roster)

Individual Student Reports
Demographic Reports (subgroup analysis)
Aggregated Trend Reports (Longitudinal data of district, school, teacher, and
class performance)
Trend Report of Students (Longitudinal data of student performance)
Not Available
Not Available
Delivered via data

le in ORS
Delivered via data

le in ORS
Not Available
Benchmark Reports (Strengths and Weaknesses by Benchmark)

Accommodation Reports (Performance data by specific accommodations, such as
Braille, large prints, color choice, etc.)
Participation Reports (Reports on testing progress)
Test Completion Rates (number of students who have started/completed each
test)
Download Student Files (File output)
Summary Statistics (Statistics of students who have tested)
Not Available
Not Available
Not Available
Not Available
Not Available
Not Available
Not Available
Not Available
School Data Files

State Aggregated Results File
District Aggregated Results File
School Aggregated Results File
Demographic School Aggregated Results File
Late School Student Result File
Page 7.5-51
Exhibit 7.5.5-3: Summary of Floridas Current Data Files and PDF Reports Compared with AIRs ORS Capabilities
(continued)
School Reports
State Summary
District Summary
School Summary
State Report of Districts (Roster)
Not Available
School Reports of Students (Roster)

Late Reporting School Report of Students (Roster)
Writing Images
Demographic Reports
Accommodation Reports
Aggregated Trend Reports for School, Teachers, and Classes
Trend Report
Benchmark Reports
Participation Reports
Test Completion Rates
Download Student Files
Summary Statistics
Teacher Reports
Not Available
Not Available
Not Available
Not Available
Not Available
Not Available
Not Available
Not Available
Delivered via data

file in ORS
State Summary
District Summary
School Summary
Teacher Summary
Teacher Reports of Classes
Teacher Reports of Students
Late Reporting Teacher Report of Students
Writing Images
Not Available
Not Available
Not Available
Not Available
Not Available
Not Available
Not Available
Demographic Reports
Accommodation Reports
Aggregated Trend Reports for Teachers and Classes
Trend Report of Students
Benchmark Reports
Not Available
Not Available
Not Available
Not Available
Not Available
Not Available
Page 7.5-52
the Department to document requirements

and configure the online reports. We will begin
by working with the Department to draft a
reporting elements document that describes
each stakeholders reporting needs and the
available measures that can be used to meet
those needs. When the reporting elements document is approved, we will work with the
Department to complete the online reporting
specifications which define everything in the
system, including table layouts, user-role
access, definitions, reporting rules, and content hierarchies (for a detailed discussion on
reporting specifications, please see Section
7.5.1.). To facilitate this process, we propose
conducting an on-site working session with
Department staff to capture requirements for
each part of the reporting system and complete all specifications. Once the Department
approves the specifications, AIR programmers
will use these specification documents to configure the Online Reporting System and to
conduct numerous quality assurance checks to
ensure the system is functioning as required.
Below we provide some sample online reports
to demonstrate the kinds of data that educators, schools, and districts will have access to
in the Online Reporting System and how these
displays can be customized.
Exhibit 7.5.5-4: Subject Summary Report for a

Teachers Class
Subscore Summary Reports

Aggregated subscore reports follow the layout
of the subject summary reports, displaying
subscore performance data for the state, districts, schools, teachers, and classes. Exhibit
7.5.5-5 shows a sample subscore summary
report for a grade 5 science class.
Exhibit 7.5.5-5: Subscore Summary Report for a
Teachers Class
Subject Summary Reports

Aggregated subject reports show average performance for the state, schools, teachers, and
classes. Bar chart displays show the distribution of students performance levels. Columns
can be sorted on any table in the system to
allow for easy comparison, and groups can be
disaggregated to report by demographic subgroup (e.g., gender). Exhibit 7.5.5-4 gives a
sample subject summary report.
Page 7.5-53
Exhibit 7.5.5-6 Sample Benchmark Report
Benchmark Reports
The reporting system leverages the information from the online summative tests to provide diagnostic information for a group of students. Educators want very precise information about the bits and pieces of knowledge or
skills that students have or have not mastered.
This sort of fine-grained information is actionable; it tells educators what to teach and lets
them know where they are succeeding.
If the test design supports these reports (either
an adaptive test or a sufficiently large set of
operational fixed forms spiraled among students), we can provide detailed reporting at
the benchmark level for these assessments.
With multiple operational forms, a class of 20
students can respond to a sufficient number of
different items measuring any given benchmark. A mechanism for aggregating them that
accounts for the differences in student overall
proficiency and differences in the difficulty of
the items can provide a robust measure of the
class, school, or district mastery of that benchmark. Here, we propose just such a measure.
Specifically, we propose to report benchmark
information at aggregate levels (e.g., school,
class, group of students). We propose that this
information be presented in terms of relative
strengths and weaknesses so that a teacher
sees whether the class performance on a particular benchmark is better than, worse than,
or about the same as their performance on the
test as a whole. We currently report this information in Delaware, Minnesota, and Oregon
for individual benchmarks.
Trend Reports
Scores for the state, districts, schools, teachers, classes, and students can be plotted on a
trend report to illustrate how performance has
changed over time. Scores can be plotted
across years (e.g., from spring administration
to spring administration) and within a year
(e.g., from fall administration to spring administration). Multiple trend lines can be plotted
at once to see how changes in performance
compare between students or groups of students. Subscore trends can also be added to
see whether there are differences in performance within a subject. Exhibit 7.5.5-7 shows a
sample trend report with two students plotted
on it.
Exhibit 7.5.5-6 shows a sample benchmark

report for a class where relative strengths and
weaknesses are shown for each benchmark.
Page 7.5-54
Exhibit 7.5.5-7: Trend Report
Exhibit 7.5.5-8: Sample Student Report
Individual Student Report

Individual student reports summarize a students performance in an organized, easy-tounderstand document that can be distributed
to educators, parents, and students. The students performance is plotted against cut
scores on a barrel chart that provides detailed
explanations of each performance level.
Exhibit 7.5.5-8 shows a sample student report.
A students subscores and comparison data for
the state, district, and school are provided in
separate tables. The report can be exported as
a PDF document, and users can batch print
multiple students reports simultaneously.
The individual student reports can link to student work, such as the student response to a
performance task. Here, we propose that such
responses be available, and printable, through
this interface. We include these in our proposed paper reports as well, but note that the
paper report page limits specified in the ITN
are not likely to be sufficient to include the
student work.
Participation Reports
Computer-based testing can provide more flexibility to schools in terms of when they administer tests; however, schools still need test participation information to manage their testing
schedule, allocate testing resources, and prioritize testing. The reporting systems Test
Management Center provides multiple tools to
generate up-to-the-minute reports showing students test status. Users can set testing schedules, monitor testing progress across schools
and even track students participation based
on their performance on previous tests.
Exhibit 7.5.5-9 is a sample of the summary
statistics report from the Test Management
Center. This report provides a high-level view
of how many students have completed each
test opportunity and the percentage of students proficient by opportunity and overall.
Page 7.5-55
Exhibit 7.5.5-9: Participation Report with High-Level

Summary Statistics
dents, we propose to enhance the design of

Floridas current student reports by
using full color to aid readers interpretation of the data;
including substantive, customized text recommendations to guide parents and students in their understanding of student
scores and next steps;
including longitudinal data from up to
seven administrations; and
including comparison data from international and other assessments to help students gauge their relative performance.
In addition, we propose that the student
reports contain the following data elements as
outlined in Section 7.5.5 of the ITN:
Scale scores, achievement levels, raw subscores, passing status, points possible, and
comparison data (state means)
AIRs Paper Reports

In addition to providing online reports for
Florida educators, schools, and districts, AIR
commits to developing and distributing customized, full-color score reports for families
and students for all assessments outlined in
the ITN. Should the Department decide not to
use AIRs Online Reporting System to deliver
score data to educators, schools, and districts,
AIR will also develop separate PDF reports for
schools and districts using the same paper
report development process outlined below.
AIR has successfully delivered millions of
pages of error-free, enhanced paper score
reports to families, educators, and administrators across the country. Using state-of-the-art
variable programming technology, our fullcolor paper reports provide clear, customized
text recommendations to help families, teachers, and administrators focus on the actions
that can help their students succeed. In developing new reports for Florida families and stu-
Page 7.5-56
The students actual response for one released

performance task along with the number of
points earned on the item, descriptive information about the question, and the points
earned on all of the performance items
We note that the length specified in the ITN
for the paper reports will likely be insufficient
to include the student performance task
responses. To provide a compatible bid, we
have not expanded the report length, and our
base bid includes the online delivery of the
student performance task responses.
We will also work with the Department to
identify sections of the student report that
must be translated into Spanish and Haitian
Creole. If required, we will provide two copies
of the student report to districts (one to be
mailed to families and the other to be added to
the students official file). Alternatively, if the
Department decides to use AIRs Online
Reporting System, schools and districts can
batch print PDFs of the student report directly

from the system. For a detailed list of paper
reports that AIR proposes to develop along
with the frequency of report distribution for
each assessment, please refer to Exhibit 7.5.511. Below we describe AIRs paper report
development process.
AIRs Paper Report Development

Process
Developing fully customized family paper
reports involve the following steps:
Designing full-color mock-ups
Completing specifications
Developing data-driven text recommendations
Conducting quality assurance
Printing, packaging, and distributing the
reports
Designing Enhanced Color Paper Score
Reports
The initial design phase for paper reports
begins with the consideration of the inferences, actions, and choices that the data
should encourage and then works backward to

the inferences that will lead to these actions,
the data displays that will guide readers to the
appropriate inferences, and the data and measures to be displayed. AIRs reporting team will
work closely with the Department to identify
the actions or behavior the Department would
like to encourage among parents and students
and the data measures and interpretive measures that can help bring about these actions.
For example, if the goal is to increase discussion between parents and teachers about the
new standards, then the reports should include
plain language terminology describing the
standards that is accessible to parents. All data
measures and interpretive elements are captured in the data elements document. Once
the data elements document is approved by
the Department, AIRs designers begin designing report mock-ups that include each of the
approved elements.
AIRs talented design team includes eight fulltime designers who are responsible for designing both paper and online reports. Exhibit
7.5.5-10 presents the qualifications of some of
the graphic designers we can deploy on this
Florida project. Resumes are included in
Appendix A. Given that AIRs paper reporting
Exhibit 7.5.5-10: AIR Graphic Designers and Qualifications

Name/Title
Daniel Schwartz
Senior Designer
Degree/Experience
Relevant Assessment Experience
M.F.A., Computer Design

B.A., Fine Art: Visual Communication
14 years of experience; 8 at AIR
B.A., Writing Seminars; Film & Media
Studies
Lead Designer for the Test Delivery System

(TDS)
Eli Mintzer
Junior Graphic Design Editor
B.F.A., Art and Visual Technology

Creates mock-ups, assets, and materials for student paper

reports using Adobe InDesign, Illustrator, and Photoshop
Adam Clegg
Junior Designer
B.A., Professional Writing

8 years experience in web design; 2 at AIR
Responsible for designing interface changes to web projects,

developing design solutions in HTML/CSS, fixing CSS issues
that occur in projects, developing design mockups, ensuring
code produces compatible designs in most major web browsers
Julie Tucker
Senior Designer
Leads front end development and UI design for AIR student

registration, reporting and other user facing systems. Led
development of design and graphics for AIR variable data
printed score reports.
Page 7.5-57
capability offers unparalleled customizability

(any text, shape, color, or data display on any
part of the page), we have virtually infinite
flexibility in designing the paper reports. The
design process involves our graphic designers
producing numerous iterations of the mockups and working closely with the Department
to ensure that their feedback is reflected in
subsequent iterations. All final colors, designs,
and specifications will be determined and
approved by the Department.
Once mock-ups are approved by the
Department, the next step is to finalize the
reporting specifications document (for a
detailed description of this document, please
see Section 7.5.1). All the data displayed on
paper reports or online reports will conform to
the analysis specifications agreed upon
between the Department and AIR psychometricians and statisticians. These calculation
rules will be documented in a reporting specifications document and approved by the
Department.
Using Color as Interpretive Aid
One of the most important design enhancements we propose to Floridas current student
reports is using color as a positional queue and
interpretive aid. Substantial cognitive research
suggests that appropriate use of color can facilitate accurate interpretations of statistical
graphics in ways that black-and-white graphics
cannot (Cohen & Cohen, 2006). We start by
noting that people can absorb some information without any conscious attention at all.
Cognitive scientists distinguish perceptions
that are processed preattentively, or without
cognitive effort, from those that require effort
and focused attention (Allport, Tipper, &
Chmiel, 1985; Neisser, 1966; Treisman & Sato,
1990). Preattentive processing happens naturally without any cognitive effort.
The correct use of color can lead readers to
compare information that should be compared
and make inferences that should be made.
Research demonstrates that the human eye
perceives differences in color as boundaries
between objects, and preattentively groups
objects of the same color together (Callaghan,
1984, 1989; Kubovy & Cohen, 1991, 1992;
Treisman & Sato, 1990).
Statistical graphics can signal the reader preattentively to compare objects (such as bars in a
bar chart) by making them the same color. For
example, side-by-side graphs of the percentage
of students in each proficiency category for
the state and district can lead most readers to
compare the district and state percent proficient simply by making the proficient sections
the same color in the two graphs (see Exhibit
7.5.5-11 for an example).
AIRs full-color process provides complete freedom in report design, enabling us to use color
to facilitate appropriate inferences and encourage readers to make connections preattentive-
Exhibit 7.5.5-11: Longitudinal Table of Reading Scores
Page 7.5-58
ly. AIRs process goes well beyond the industry

standard, resulting in much more easily interpretable (and therefore more valid) reports, as
well as reports that are more attractive. The
industry standard process for enhanced
reporting begins with preprinted forms, in
which all the color that will appear on a report
appears in fixed locations and configurations.
The only variable information printed on a
report is printed in black ink. While this limited use of color may enhance the attractiveness
of a report, it does not serve to aid the readers
interpretation of data.
Substantive Text Recommendations for
Parents
We also propose to enhance the design of
Floridas current student reports by including
plain-language text for students and parents
that explain the content standards assessed on
each test and what students in each of the
reporting levels typically know and can do. We
further suggest including advice about some
next steps the parents and students might take
at home to practice skills needed to master the
standards. Students performing in different
score ranges receive advice appropriate to
those score ranges. Written in parent-friendly
language, this data-driven text provides a substantive explanation of what parents should
understand from the data and what they can
do as a result of the data.
Interpretations, recommendations, and other
texts are drafted by content specialists or statisticians. Data-driven text appearing on the
paper reports will be developed within the
Item Tracking System (ITS), reviewed by content experts and editors, and submitted to the
Department for review and approval. For
examples of our enhanced family reports that
contain customized recommendations for parents, please see Appendix G.
At the same time that text is being developed,

the paper report development process uses
graphical programming, which takes place in a
Xerox-developed programming language called
VIPP and allows virtually infinite control of the
visual appearance of the reports. After designers at AIR create backgrounds, our VIPP programmers write code that indicates where to
place all variable information (data and text)
on the reports. Programmed output goes
through multiple stages of review and revision
by graphics editors and the Score Reporting
team to ensure that design elements are accurately reproduced and data are correctly displayed. As with each component of this
process, the VIPP programs go through rigorous quality control reviews.
Longitudinal Reporting
AIR proposes to report longitudinal student
performance for parents and students along
with any significant trends evidenced by the
data. This will allow parents to detect patterns
of potential weakness in students at an early
stage and can aid in more precise intervention
to help remedy students weaknesses. In the
section on online reporting above, we describe
the capability to include longitudinal scores for
teachers, schools, and districts. Exhibit 7.5.511 gives an example of the longitudinal data
display that we currently provide to Minnesota
parents and students:
Quality Assurance
Statistical Programming
The family reports we propose to develop for
Florida contain custom programming and
require rigorous quality assurance processes to
ensure their accuracy. All custom programming is guided by detailed and precise specifications in our reporting specifications document (see Section 7.5.1 for more detail). Upon
approval of the specifications, analytic rules
Page 7.5-59
are programmed and each program is extensively tested on test decks and real data from
other programs. The final programs are
reviewed by two senior statisticians and one
senior programmer to ensure that they implement agreed-upon procedures. Custom programming is implemented independently by
two statistical programming teams working
from the specifications. Only when the output
from both teams matches exactly are the
scripts released for production. Quality control, however, does not stop there.
Much of the statistical processing is repeated,
and AIR has implemented a structured software development process to ensure that the
repeated tasks are implemented correctly and
identically each time. We write small programs
(called macros) that take specified data as
input and produce data sets containing derived
variables as output. Approximately 30 such
macros reside in our library for the grades 38
program score reports. Each macro is extensively tested and stored in a central development server. Once a macro is tested and
stored, changes to the macro must be
approved by the Director of Score Reporting
and the Director of Psychometrics, as well as
by the project directors for affected projects.
Each change is followed by a complete retesting with the entire collection of scenarios on
which the macro was originally tested.
The main statistical program is mostly made
up of calls to various macros, including macros
that read in and verify the data and conversion
tables and the macros that do the many complex calculations. This program is developed
and tested using artificial data generated to
test both typical and extreme cases. In addition, the program goes through a rigorous code
review by a senior statistician.
Page 7.5-60
Display Programming
The VIPP code that places the variable text,
graphic displays, and other variable data is
tested using both artificial and real data. AIRs
data generation utilities can read the output
layout specifications and generate artificial
data for direct input into the VIPP programs.
This allows the testing of these programs to
begin before the statistical programming is
complete. In later stages, artificial data are
generated according to the input layout and
run through the psychometric process and the
score reporting statistical programs, and the
output is formatted as VIPP input. This
enables us to test the entire system.
Once we receive final data and VIPP programs,
the AIR Score Reporting team will review
proofs that contain actual data based on quality assurance documentation that is provided
by the Department. In addition, we will compare data independently calculated by AIR
psychometricians with data on the reports. A
large sample of each type of report will be
reviewed by several AIR staff members to
make sure that all data are correctly placed on
reports. This rigorous review typically is conducted over several days and takes place in a
secure location in the AIR building. All reports
containing actual data are stored in a locked
storage area. Staff from the Department are
welcome to visit AIR at any point during this
revision process to oversee our procedures.
Prior to printing the reports, AIR will provide a
live data file and reports with sample districts
as chosen by the Department for the
Departments review. AIR will work closely
with the Department to resolve questions and
correct any problems. The reports will not be
delivered unless the Department approves the
sample reports and data file.
Printing, Packaging, and Distribution

Paper reports for each testing program will be
distributed according to Department specifications. For example, EOC paper reports will be
distributed in summer, fall, winter, and spring,
whereas ELA/L and Mathematics reports are
distributed annually. Once all tests are scored,
AIR will produce data files and print and ship
reports for all students who tested during the
school year.
AIR works with an established network of
print vendors around the country that are
thoroughly trained on our report printing specifications. Our print vendors are ISO certified,
provide fully secure print facilities, and are
capable of printing and packaging high volumes of full-color reports in exceedingly compressed timelines.
Our score reports use cutting-edge digital print
technology. While individual reports can be
printed to a postscript or PDF file, the real
power of VIPP is to directly drive the printer,
dramatically speeding up the printing process
and enabling us to make full use of the speed
and power of full-color Xerox iGen printers.
These machines merge the graphic program
code (VIPP code) with data at print time, drastically reducing the total time needed to print
reports. Printers are able to handle millions of
pages and dynamic color, ensuring that there
will be no late deliveries. During printing,
reports are checked for color against color
samples, and print site auditors review reports
as they are printed to make sure that graphics
are printing properly, all pages are correctly
printed, and there are no printing errors such
as ink smudging or faulty lines.
Our additional quality control procedures during printing including the following:
When the reports are printed, a consecutive number is applied in the gutter of the
page of every report. The number consists
of the district code and consecutive report

number (i.e., 34567-12). This number is
placed on each report in case the reports
are shuffled out of order. This sequence
number allows print site staff to place the
reports back into the correct order before
packaging.
All reports that need to be folded and
stitched together are controlled by the
VIPP code, which tells the booklet maker
on the iGen when and when not to fold and
stitch the job, as well as how to impose the
pages, thus eliminating the possibility of
manual error.
All reports that require folding or stitching
are produced with an automatic inline
stitcher on the iGen, eliminating the possibility of manual errors.
To ensure accurate packaging, the VIPP
code programs divider sheets to appear
between classes, schools, and districts.
Divider sheets are printed on one 1117
page custom label stock paper that moves
through the booklet maker. The dividers
contain information about the reports
between the current and the next divider
sheet including the type of reports/student
labels that will be placed in the package,
the contact name to whom the reports will
ship (if applicable), the name and number
of the district, the name and number of the
school (if applicable), the address to which
the reports will ultimately be delivered, the
package number out of the total number of
packages in the file, and the first and last
serial number of the reports.
Detailed instructions on how to package
reports for special schools such as home
schools, virtual schools, non-public hierarchies, and community schools are provided
to all vendors. Divider sheets convey to
packagers exactly how these reports need
to be packaged and shipped.
Page 7.5-61
Exhibit 7.5.5-12: Process Flow for Paper Score Reports
Interpretive
Guides
Research
Requirements
and Mock-ups
Content
Development
Critical
Document
Development
Text Files
Mock-ups
Design
Backgrounds
Data Layouts,
Data Maps, and
Variable Text
Statistical
Programming
Derived
Data for
Printing
Printing
Score
Reports
Delivery
Item
Parameters
Scored
Student
Data
Customer
Service
VIPP
Divider sheets also contain a removable

label that packagers place on the inside
package containing the reports.
AIR will provide documentation of established
quality checks upon request.
AIRs PrintTracker program manages the printing and shipping process. PrintTracker, a webenabled logistics program, assigns work units
to printers on a flow basis, ensuring that all
printers remain busy by routing work to each
printer when the previous batch is complete.
PrintTracker has multiple, built-in quality
assurance features to ensure that reports are
correctly packaged and shipped. For example,
Page 7.5-62
VIPP Code
an operator at a print site must record the

number of packages and boxes destined for
each school or district. PrintTracker confirms
this number against the database before indicating to the operator that the package may be
sealed and allowing him or her to print shipping labels. Any discrepancies are reported to
and resolved by AIR print management staff to
ensure that every report is shipped to the correct location.
The flow chart in Exhibit 7.5.5-12 summarizes
the process by which we propose to develop
the enhanced paper score reports for Florida.
7.5.5.1 Initial Reporting Guidelines

Introduction
A fundamental purpose of Floridas standardsbased assessment program is to provide accurate information on student achievement. This
requires following rigorous data checking procedures to ensure that score results are provided to the right student in the right school at
the right district. Data cleaning procedures are
typically used to verify three categories of
information:
Student, School, and District associations;
Student, School and District identifiers;
and
Student demographic and enrollment
information.
Up until recently, Departments of Education
have invested significant time and budget
resources to scrutinize and correct their student, school, and district data after testing is
complete and before publicly releasing score
results. Much of this has been unavoidable
because of the time lag between preID and
testing, a period during which many students
change schools or enter the system. These students identifying information has been handgridded onto their answer documentsa
process that inevitably introduces errors.
With the introduction of online testing, much
of the possibility for error has been taken out
of the preID process. Students must be known
to the system before testing. Continuous registration processes eliminate the need for much
of the post-testing cleanup.
AIR and DRC offer a process that brings the
efficiency of continuous registration to paper
testing, extending the benefits in terms of data
quality and reduced cleanup time at the end.
Late arriving students can be added to the sys-
tem at any time, and schools or districts can

print real-time-ID labels using any laser
printer on label stock that we will provide.
There will be no need to bubble student
information onto the cover of an answer document, and the identifying information can be
corrected by school or district personnel at
any time during the testing window.
AIRs student registration system (our Test
Information Distribution Engine, or TIDE) and
the Roster Tracking System (RTS) have made
it possible for districts to verify and update
school and student information throughout the
school yearbefore, during and after testing
allowing Departments of Education to realize
numerous efficiencies including
far greater reliability of school, district, and
demographic data resulting in more accurate distribution of online and paper
reports;
a reduction in the total volume of records
requiring cleanup or reconciliations at the
end of testing; and
speedier production of score reports
despite exceptionally compressed timelines.
Below we outline the challenges that many
Departments of Education face as well as our
proposed solution for ensuring data integrity
through a series of comprehensive, continuous
data verification processes.
The Challenge
When testing large volumes of students, discrepancies in student-school associations, student demographic information, and enrollment
and information are bound to arise. For example, school and district identifiers often change
for a number of reasons. Due to the numerous
types of special schools in Florida (home
schools and virtual schools) and volume of stu-
Page 7.5-63
dents testing in schools that are different from

the schools where their scores must be reported, school numbers often require updating
after scanning. Programs with paper testing
pose even more challenges. The bubbling
process ensures that the data cleanup required
will be extensive, because hand-gridding the
demographic information is tedious, labor
intensive, and error prone.
Our experience working with Departments of
Education across the country has revealed that
there can also be problems with unique student identifiers. Despite multiple checks, it
can and does happen that a student identifier
has been assigned to more than one student in
the same district, causing issues when matching demographic information to test scores.
Other scenarios that weve encountered and
that require adjusting student or school data
include the opening, merging, or closing of
new schools (this requires modifying school
district associations) and a host of test irregularities that include student cheating or interruptions to testing due to fire alarms, etc.
Conventional processes require that all these
kinds of errors be individually resolved and
only after scanning is complete and the score
data become available. As a result, Floridas
current process of cleaning up student, school,
and district information and redirecting
reports to the correct school takes weeks,
delaying the production of final reports.
Our Solution
The solution we propose uses our integrated
online systems to provide districts with a
series of continuous, comprehensive checks to
verify their data throughout the year. The first
series of data verification checks are conducted before or during testing and allow districts
to do all the following:
Page 7.5-64
1. Review and revise student, school, district

information using TIDE (TIDE contains student-school relationship data, school identifiers, as well as demographic and enrollment information, all of which can be
reviewed).
2. Use our real-time ID service to print preID
labels for students testing on paper one or
two days before testing, reducing the possibility of erroneous hand-gridded demographic information.
3. Respond immediately to test irregularities
during the test window instead of after
scanning.
In addition, we propose the following data
cleaning measures after testing is complete to
resolve any remaining data issues:
An automated discrepancy process that
flags for resolution by the Department any
remaining discrepant records after testing
Extension of the data verification window
beyond testing, at the Departments discretion
An opportunity for Department staff to
access the Online Reporting System to verify all data before public release, or before
the results are considered official
One of the Departments concerns as stated
in the ITN, is the ability of reporting and psychometric staff to conduct all spring reporting
activities while also turning their attention to
updating the reporting specifications for the
next year. By moving up critical data cleaning
activities to much earlier in the process,
reporting activities are protected from unnecessary additional time pressure, allowing
reporting and psychometric staff more time
to produce spring reports, conduct regression
testing, and update reporting specification for
the upcoming year.
Next we provide a detailed description of how

TIDE allows districts to revise their student,
school, or district data before and during the
test window.
Real-Time Opportunities for Districts to

Check and Revise School and Student
Information in TIDE
As described above, the Department requires
data verification and cleanup measures before
reports are released, to address situations
where districts or schools do not indicate all
the appropriate school relationships. In most
online testing programs, data verification of
student and school information happens before
testing begins. That is, students must be
authenticated before they test. For example,
if a student does not have a valid student ID,
he or she will not be able to access any online
tests until this is corrected in TIDE through a
file upload or through the user interface.
Online testing requires processes where students are positively and properly identified
before testing. As a result, fewer demographic
and relationship discrepancies exist when conducting tests online than in a conventional
paper-and-pencil testing situation.
The data verification activities mentioned
above are conducted via AIRs Test Information
Distribution Engine (TIDE), which provides an
integrated system for collecting and adjusting
data about students including student relationships (to their districts, schools, teachers, and
classes), demographics, and accommodations.
For a detailed discussion about TIDE please
see Section 7.4.8. In online assessment programs, we typically open the TIDE window
well before testing, and leave it open beyond
the testing window. Throughout the testing
window, TIDE offers the ability to check and
revise student demographics, student-school
associations, reporting schools and other information that may change or require correction.
Districts can view the student information
directly through TIDE, and can implement

corrections either through file uploads or
through the interactive user interface.
For example, during the test window, a district
may need to accommodate a new student who
is enrolled in one school, but attends an alternative school. In this case, both pieces of
information (the students enrolled school and
the school at which the student tested) would
have to be entered into TIDE, either through a
file upload or through an interactive interface.
The seamless integration between our Online
Reporting System and student registration system ensures that all updated information in
TIDE will be used to produce reports and data
files. The information active in TIDE after a
pre-established date will be merged with the
score data prior to reporting, eliminating the
need to conduct downstream data verification
steps after testing is complete. As an additional
check, during the testing window, schools or
districts can confirm the reports they can
expect to receive. If adjustments need to be
made, they can add or correct the necessary
data directly in the TIDE system and the
appropriate reports will be provided.
Reducing Data Discrepancies through

Real-Time PreID
The ITN notes that corrections to school
names and identifiers are often required following scanning. Much of this sort of cleanup
is needed as the result of gridding errors on
paper test booklets, which our proposed Realtime ID process eliminates. (See Section 7.4.8
for a detailed discussion of Real-time ID.)
Real-time ID will allow Florida schools and districts to register students as little as one or two
days in advance of testing for both online and
paper testing to accommodate newly entering
students. For paper tests, schools will be able
to print an individualized, barcoded, student-
Page 7.5-65
specific preID label instead of bubbling information on a form. The scannable labels can be
printed using any laser printer on label stock
to be delivered with the preIDd answer booklets. Real-time ID solves the problem of invalid
school names and IDs by preventing their
introduction in the first place. Our proposed
Real-time ID service also improves data
integrity by ensuring that most of the data
cleanup can be completed online before the
testing window even closes, allowing reporting
activities to begin much earlier than under
Floridas current system.
Opportunities for Districts to Address

Test Irregularities during Test Window
TIDE also supports a series of tools that districts or other authorized users can access to
immediately address test irregularities, including invalidating and resetting student tests.
The term irregularity covers many different
types of events including cheating, fire alarms,
or other emergencies at a school that may stop
testing for the day. Despite multi-layered,
redundant identity checks, students sometimes test under another students identifier,
or individual students may test under multiple
identifiers. Test administrators may misadminister an assessment, students may fall
ill during testing. In a large state like Florida,
almost any anomalous situation that can be
imagined will likely happen at some point.
AIR has developed a suite of tools to enable
districts to take immediate corrective actions
to address testing irregularities. These tools
include the following:
Invalidations, which involve invalidating a

students test so the student cannot resume
testing. In this case, the specific test opportunity has been used. State Departments of
Education typically only allow invalidations
in rare cases.
Page 7.5-66
Resets require a students test to be restarted from the beginning as if the student
never tested. In this situation, the opportunity is preserved by re-setting the test.
Extensions involve an extension of the

pause rule on the students test. The extension allows the student to resume his or
her existing test where the student left off.
Specific extensions include
Extension of the Expiration Date: the
student resumes where he or she left
off but cannot go back to previously
answered items. This is used if the opportunity to resume has expired (and is
implemented by extending the expiration
date).
Grace Period Extensions: the student
resumes where he or she left off but he
or she can also navigate back to questions within the section as needed. This
option is implemented by suspending the
pause rule.
Grace Period Extension with Segment
Permeability: the student resumes where
he or she left off but is permitted back
into previous sections of the test to
review answers. This type of extension
exposes all items within the permitted
segments.
As districts or authorized users enter their

invalidation, extension, or reset requests in
TIDE, their decisions are immediately reflected in the Online Testing System and the
Online Reporting System (ORS), significantly
reducing the need for post-test data cleaning.
For example, if a students test is invalidated,
the results of that test will automatically
be withheld from appearing in the Online
Reporting System. No manual removal of the
record is required from either the downloadable data files in the ORS or the Online
Reports. The ability of districts to address test
irregularities during the test window further
reduces the number of outstanding data issues

that need to be resolved after testing, allowing
reporting activities to begin much earlier.
Automatic Identification and Resolution

of Discrepant Records
While Real-time ID will virtually eliminate the
existence of data discrepancies in paper testing, it is possible that a very few tests might
still fail validation in the Test Integration
System or the QA system. Possible reasons for
validation failure include duplicate identifiers,
schools that have closed, or other unusual
circumstances.
In these cases, we propose to work with the
Department to establish a discrepancy identification and resolution process to correct the
data quickly and efficiently. In online testing
programs, the discrepancy process typically
works as follows: AIR delivers a file to the
Department at pre-established intervals (nightly or weekly) with all records that encountered
an error in the validation process. (We will
work with the Department to establish all validation rules.)
The file includes the student data that was
bubbled on the demographic page, in addition to data from our Roster Tracking
System (RTS), provided the student
appears in the RTS.
The file will conform to a pre-established
layout approved by the Department, preapproved naming conventions, and will be
delivered to an agreed-upon location.
Scanned demographic pages can be included in a separate zip file named with the
timestamp when it was generated.
To resolve identified discrepancies, we propose
that the Department deliver a file back to AIR
at pre-set intervals (nightly or weekly) with
corrections to the test records that failed validation in AIRs systems.

The file will contain agreed-upon variables such as Book ID, and student
identifier.
The file will conform to the agreed-upon
layout, delivered to an agreed-upon location, and use an agreed-upon naming
convention.
The automated discrepancy identification and
resolution process described above ensures
that the few remaining data discrepancies do
not delay reporting activities. Student results
for records with accurate demographic and
SSID information will appear in the online
system automatically. Scores for resolved discrepant records will appear in the Online
Reporting System and the downloadable data
files the day following our receipt of the
Departments correction file.
Real-time ID, continuous registration and
cleanup, and automation of the identification
and resolution of discrepancies eliminate
weeks from the process, ensuring that reports
are produced earlier and use more reliable
data.
Post-Test Data Verification Window

Our proposed solution for ensuring accurate
data to produce the initial wave of reports
includes a continuous cleanup period. Cleanup
can be accomplished during the testing window; however, the cleanup window can extend
beyond the testing window at the Departments
discretion. We will work with the Department
to identify the dates for the verification window. During this window, districts will have
the opportunity to view and revise the following information in TIDE:
Student demographic and enrollment
information;
Page 7.5-67
School and district information;

Student, class, school, and district relationship information; and
Information on test invalidations and
extensions.
During the verification window, districts may
make any required corrections to the above
information in TIDE (or through the state SIS
to be delivered to TIDE). For example, if additional test invalidations need to be implemented during this period, districts can enter these
directly into TIDE. We propose that the timing
of the cleanup period coincide with the dates
that the Department conducts its review of the
draft data file and calibration file.
The seamless integration between our student
registration system and the Online Reporting
System ensures that all updated information in
TIDE will be used to produce reports and data
files. The information active in TIDE after the
close of the post-test verification window, will
be merged with the score data prior to reporting, eliminating the need to conduct extensive
data cleanup steps after the window closes. As
an additional check, during the verification
window schools and districts can confirm the
reports they can expect to receive. If adjustments need to be made, they can add or correct the necessary data directly in the TIDE
system and the appropriate reports will be
provided.
Department Verification of Data before

Reports Released
After the post-test data verification window
closes and before results are publicly released,
we propose the Department review and verify
all student level and aggregate data in the
Online Reporting System.
The Department will receive data files and sole
access to the Online Reporting System. We
Page 7.5-68
recognize that in the past the Department has

required multiple releases of the final data file
before it was correct. We have included this in
our schedule, but believe that you will find initial deliveries to be complete and correct. The
continuous cleanup and quality monitoring
help eliminate many of the traditional challenges that accompany paper testing.
We will work with the Department to identify
dates during which Department staff alone can
access and verify results in the reporting system. During this period, the Department will
be able to confirm all school and district information, student and school associations,
demographic, and enrollment information,
as well as test invalidations. Should there be
additional data discrepancies that need to be
resolved, the Department can provide an
updated data file to be loaded directly into
TIDE.
In addition, we will work with the Department
to identify a sample of reports for the
Departments QC as outlined in the ITN. After
the Departments approval of final results, we
will deploy the Online Reporting System so
that all schools and districts can access their
reports and data files. Upon the Departments
approval, we will also print paper family
reports to arrive in districts on the scheduled
date in Floridas critical dates calendar.
7.5.5.2 Late Reporting Guidelines

We expect that the volume of late reports will
decrease as more schools move to online testing and the data cleaning measures described
above greatly reduce the number of reports
sent to the wrong school. However, for the late
records that do exist, these will flow automatically to the Online Reporting System as handscoring is completed and the scores have
passed all the data validation checks described
in Section 7.5.5. As late batch results flow to
the Online Reporting System, these records
will also be included in the downloadable data

files for schools and districts to generate on
demand. We propose that, like the initial wave
of reporting results, late student results remain
in the Online Reporting System until the
Department approves their release.
Searching For Missing Answer Documents
To facilitate the Departments review of late

batch results, we propose to follow the same
data cleaning processes described above for
the initial wave of reports. Districts will have
an opportunity to view student, school, and
district information in TIDE during the posttest verification window and before the production of late reports. Similarly, we propose
the Department review and verify school, district, and student information in the Online
Reporting System before approving the release
of late results. Our experience in other states
has proven that our data cleaning processes
enable timelier production of late batch
reports, especially for graduating students who
require their results quickly.
All missing document search requests will be

submitted to the Department via the Score
Inquiry System after initial release of the data.
After Department approval is received, DRC
staff will initiate and track the search process.
DRC will follow a series of steps designed to
conduct the most effective search possible:
In summary, we commit to providing three

waves of late batch reporting for each administration and including the following reports and
data files:
Two hard copies of Late Batch Student
reports for each wave of late batch reports.
These reports will include a date on the
front page indicating they are late batch
reports and will be produced using the
same paper reporting process we outline in
7.5.5. (If districts and schools require hard
copy reports immediately, they can print
their own student reports directly from the
Online Reporting System.)
Downloadable data files containing the
District Student Results and the School
Student Results. These files will contain all
late batch records and will be available
immediately in the Online Reporting
System, eliminating the need for districts
to wait for CD shipments.
To address district inquiries about missing

reports or requests to review student scores,
AIRs partner DRC will search for missing
answer documents as required.
During the first step, DRCs document processing team will be requested to identify
the batch with which the document would
have been processed. Using this batch
information and DRCs document records,
a determination will be made whether the
document was scanned, pulled, or missed
for a particular reason or was never sent to
document processing.
If a review of processing records does not
locate the missing document, all test booklets/answer documents received from the
associated district/school will be retrieved
from secure storage and thoroughly
searched to locate the missing materials.
DRC will also search any not-to-be scored
materials in case the document was incorrectly labeled or packaged for return.
As a final attempt to locate the missing
document, DRC will contact the district
test administrator to verify that all materials were sent back to DRC. DRC will utilize
its Missing Score/Document Investigation
Form, which is designed to gather information to assist both DRC and district/school
personnel with locating the missing document. DRC will confirm that the number of
boxes returned to DRC matches the number of boxes received. DRC will also verify
the number of answer documents returned
Page 7.5-69
from the district/school. If district/school

personnel are able to locate the missing
document, DRC will arrange for it to be
sent back to DRC for processing, scanning,
and scoring for inclusion in the late reporting process.
7.5.5.3 Close-Out and Annual

Reporting Guidelines
After the last wave of late batch reports is produced, AIR will produce final end-of-year files
for the Department that contain the final
records for all students. We will work with the
Department to establish a delivery date for
these files.
Our process proposed above for resolving data
discrepancies means that the corrected student records will automatically appear in the
Online Reporting System without the need for
any manual cleanup processes.
We also note that at any point, the ORS aggregate and roster reports are up-to-date in the
Online Reporting System and can be generated
by districts and schools on demand. These
files will contain the same information as the
current the Student Aggregated Results file
(SAR) and the District Aggregated Results file
(DAR).
Allowing districts to view and revise school,
district, and student data throughout the year
means greater data accuracy before initial and
late reports are produced. This in turn results
in our ability to generate end-of-year data files
for the Department which require far less data
cleanup than under Floridas current system.
We commit to providing the Department with
a data extract after all late reports for the
year have been produced that includes the
final records for all students taking the
EOC assessments or ELA/L or Mathematics
assessments; and
Page 7.5-70
concatenated EOC results files that contain

all four administrations of student data.
The above files will include only corrected
records (all erroneous or discrepant records
will have been removed prior to the generation
of this file).
7.5.5.4 Demographic Reporting

Guidelines
In the sections above, we describe our proposed solution for ensuring that districts can
review and revise their demographic information in TIDE before, during, and after the test
window. This ensures that all paper and online
reports, as well as downloadable data files,
contain up-to-the-minute demographic information. Based on this solution, we expect to
produce online and paper reports powered by
real-time demographic information, eliminating the need for producing separate demographic files later in the year.
To meet the Departments need to populate its
interactive demographic databases, we propose
to provide the Department with a data extract
from the Online Reporting System that contains the districts most updated demographic
information. This file will contain disaggregated data for approximately fifty sub-groups as
required by the ITN. We will work with the
Department to finalize a layout for this file
that is compatible with the Departments
demographic websites as well as a delivery
date for this file.
Alternatively, if the Department decides not to
use AIRs online system for distributing school
and district results, we can provide data files
that are based on the Survey 3 demographic
data as outlined in the ITN. As in our first
option above, disaggregated results can be
automatically generated and reported in the
ORS.
7.5.6 Verifying Data

Whether the Department continues to employ
a post-equating approach to scoring and
reporting summative assessments or elects to
adopt a pre-equating approach that will support immediate scoring and reporting, there
are a range of scoring and reporting processes
that must be verified both in preparation for
test deployment and psychometric analyses,
and throughout the scoring and reporting
process.
Because the Department will be supporting a
dual-mode system, a range of mock or simulated data will be used to verify the accuracy of
all processes. Some simulated data is modespecific. For example, to prepare for processing of paper-based assessments, DRC develops
test decks to calibrate and verify accuracy of
imaging equipment and data file uploads to
AIR. Test decks are designed to assess the
range of processing requirements, including
ensuring that scanner settings accurately
implement detection thresholds, including
absolute thresholds for the detection of marks,
and relative thresholds for distinguishing relative intensity between multiple marks.
Similarly, prior to launching online assessments, AIR conducts user acceptance testing
(UAT), which allows the Department and AIR
to verify the accuracy of data capture in the
online system. AIR develops a set of test
decks that are designed to evaluate accuracy
of data capture in a manner quite similar to
the test decks used to calibrate imaging equipment. Test records are generated to simulate a
variety of possible testing scenarios and
response patterns. The Department is also
encouraged to perform its own checks on the
accuracy of data capture in the test delivery
system (TDS).
For all assessments, whether administered on
paper or online, simulated data files are used
to verify the accuracy of item and test scores

and also to ensure that the online reporting
system (ORS) displays test results accurately,
including student-, class-, teacher-, and schoollevel reports. In addition, the same simulated
test records are used to verify the production
of printed score reports and data file deliverables, ensuring that all aggregation rules are
implemented correctly.
7.5.6.1 Checking Mock Data

Mock or simulated data will be generated to
verify the full range of test processes and deliverables. Three kinds of simulated data are
used to verify test processes:
Answer document test decks
Online test decks and UAT
Scoring and reporting data files
Imaging Test Decks. Prior to any Florida
Standards Assessments PBT materials returning to DRC, DRCs Software Quality Assurance
staff will perform extensive tests to ensure all
scanned data (including demographic and multiple-choice responses) are captured and accurately stored in a secure database environment. Each record in the database will be
independently verified against the test decks
for validation.
The analysts will follow a software testing
methodology that thoroughly evaluates and
verifies the scanning and scoring system and
verifies that each scanner is configured and set
up for the Florida Standards Assessments. This
process includes validating test decks, which
will be comprised of answer documents with
and without student and school pre-ID information for each form of the test.
The test decks will be specifically gridded to
include a variety of possible student response
situations, permutations, and combinations,
Page 7.5-71
including erasures, multigrids, demographic

values, score flag codes, school type values,
and scores, including specific/alternating item
response patterns. DRC manually grids all test
decks. Test deck criteria and mock data cases
will be developed to align with Scoring and
Reporting Specifications and/or Department
requirements.
The test decks will be processed completely
through DRCs systems to verify the following:
Readability of security, student, and school
barcodes.
Data capture of pre-ID and barcode information.
Accurate capture of district and school
codes.
Consistent data capture on all scanners.
Accurate scan positions on all documents
and forms.
Scanner calibration and hardware functionality.
The Software Quality Assurance staff will also
perform a validation of all production data
processed through the system. Each student
record will be verified for accuracy to ensure
high quality data file development and reporting. Please see Section 7.5.3 for detailed information on DRCs scanning technology, systems, and processes.
Online Test Decks. Online tests are different
than scanned tests because the items are represented as data, and the programming is constant across items. This minimizes the risk
that a bug will exist for a single item or for
an item on a single form.
For fixed-form tests, AIR will work with the
department to design test decks with a range
of responses to each item and confirm that the
system captures these items correctly. We
write software testing scripts to run through
Page 7.5-72
the test decks on the fixed-form tests. Test

scripts are better than humans in this regard,
because a human might accidentally input an
unintended response, making test results
ambiguous. Test scripts can be verified for
accuracy if an unexpected result appears in
the data file.
If the Department chooses an adaptive model,
a series of simulation studies is used to test
the process. These simulations are described
below.
The test decks are implemented on the production TDS so that final verification is performed on the operational test platform.
Similar to test decks used to calibrate document imagining systems, we implement a
series of test administrations designed to test
all possible responses to selected-response
items, as well as a range of possible responses
to constructed-response items, to ensure that
student responses are captured accurately and
that item responses are scored correctly.
Although AIR is solely responsible for identifying any issues related to item response capture, the Department may also wish to implement its own online test decks during the UAT
window. AIR will provide the Department with
data files containing all Departmental UAT test
records to verify that captured test records
match with data entry records implemented by
the Department.
Simulator Data Files. While we do verify the
scoring of item responses and test scores and
performance levels using the UAT input
records, test deck data is limited with respect
to scoring of test records, especially with pattern scoring where there are thousands of possible test scores. Therefore, we verify the accuracy of the scoring engine using simulated test
administrations. The simulator generates a
sample of students with an ability distribution
that matches that of the state. The ability of
each simulated students is used to generate a

sequence of item responses consistent with the
underlying ability. These simulations provide a
rigorous test of the adaptive algorithm for
adaptively administered tests but also provide
a check of the full range of item responses and
test scores in fixed-form tests. Simulations are
always generated using the production item
selection and scoring engine to ensure that
verification of the scoring engine is based on a
very wide range of student response patterns.
To verify the accuracy of the ORS, we merge
item response data with the demographic
information taken either from previous-year
assessment data, or if current year enrollment
data is available by the time simulated data
files are created, we can verify online reporting
using current-year testing information. As
noted in the ITN, by populating the simulated
data files with real school information, it is
possible to verify that special school types and
special districts are being handled properly in
the reporting system.
Test accommodations, including large print,
text-to-speech (TTS), as well a host of other
available accommodations such as foreign language versions described in Section 7.3.11.1,
will be provided for the online tests. In the
online system, accommodations are provided
as tags to items in the item bank. Because
accommodations are attributes of items, rather
than adaptations of specific test forms, the
accommodations travel with the item and
eliminate the need to produce special accommodated test forms. Moreover, because the
Item Tracking System (ITS) uses the same
item display software used by the TDS, as soon
as items are tagged with an accommodation,
the Department, AIR, and the quality assurance (QA) contractor can verify that the
accommodated version of the test item is
being displayed correctly, allowing verification
of accommodated items to occur well before
the beginning of the test administration win-
dow. In a fixed-form environment, it is more

cost-effective to tag items for accommodations
after the operational test form has been built,
so that the Department does not pay for
accommodations until the item is used.
However, because the items in each of the candidate Florida Stanards Assessment (FSA)
banks were built for use in a Comprehensive
Assessment Test (CAT) environment, all items
in the initial pool have already been tagged
with accommodations. For this reason, even if
the Department chooses to administer fixedform assessments, it may still be cost-effective
to provide accommodation tags annually following item data review for items promoted to
the operational pool.
In addition, because data capture is identical
for accommodated and non-accommodated
items, a separate process for verifying correct
capture of accommodated responses is not
necessary. Test cases for accommodated test
administrations are designed into the online
test deck and can be incorporated into the
Departments UAT. For accommodations provided via paper test forms, test administrators
will use the data entry interface (DEI) to submit student responses. The DEI provides a
user-friendly environment for data entry that
mirrors the students test booklet to reduce the
likelihood of data entry errors. The functionality of the DEI is also verified using the online
test decks.
Specifications for generating simulated data
files, including the test deck designs for both
paper and online assessments, as well as the
design of simulation runs to verify scoring and
reporting, will be included in the scoring and
reporting specifications document submitted
to the Department each year. We understand
the Departments requirement that review of
all simulated data be completed prior to the
opening of the test administration. And in fact,
this requirement is of even greater import in
the context of an online system that supports
Page 7.5-73
immediate scoring and reporting. Before the

test window opens each year, integrity of item
administration, data capture, item and test
scoring and reporting must be verified. Even if
test results initially reported in the ORS areas
are marked as preliminary, the reality is that
there is no room for error in the reporting of
students assessment results.
7.5.6.2 Checking Early District

Return Data
will select certain districts to be part of the
early district return sample for detailed analysis and quality verification for paper-based
tests. DRC will work with selected districts to
collect and return the test materials to DRC
for receiving, processing, and scanning. Prior
to the return of these early district test materials, DRC will put in place mechanisms to
immediately identify and prioritize the boxes
of early return answer documents from these
districts and process them on a first-priority
basis through check-in and scanning. To verify
the accuracy and quality of scanning and scoring, DRC will use the early return student
answer documents to conduct completed runs.
The results from the early returns will be thoroughly checked to ensure that the hardware
and programming specifications are working
correctly and to verify all early return data.
Please see Section 7.5.3 for a detailed discussion of DRCs processing and scanning systems, technology, and processes.
DRC welcomes the opportunity to host representatives of the Department at DRCs secure
scanning facilities for spring scan checks. DRC
understands that the scan checks for other
administrations may occur either at DRCs
facilities or at Department offices. DRC will
provide all early return original and edited
scan files to the Department no later than
three business days prior to the on-site
reviews. For scan checks performed at the
Page 7.5-74
Departments offices, AIR and DRC will also

provide PDF documents of all early return
answer documents via AIRs secure FTP site.
AIR and DRC have proposed an accurate and
efficient alternative to using header
sheets/Document Count Forms (DCFs); please
see Section 7.4.7.5 for more information. Only
after the Department is completely satisfied
with the accuracy of the early return files and
results will AIR and DRC continue with processing, scanning, scoring, and reporting routines for each paper-based test administration.
DRC looks forward to the Departments on-site
visits to its scanning facilities. AIR and DRC
will cover all travel expenses for Department
staff to complete the scan checks within the
reimbursable funds. We understand that the
on-site scan checks may require one or two
consecutive weeks to complete, dependent
upon the availability of Department staff. AIR
and DRC further understand that the
Departments independent contractors may
participate in on-site scan checks, or that they
may visit DRCs scanning facilities prior to the
Departments scan checks.
During the on-site scan checks, DRC will
ensure that Department staff are provided with
computers and access to e-mail and a highspeed printer, as well as access to the CBTs,
SAS (if requested), any key entry verification
logs as applicable, and early return answer
documents, including Braille, large print, and
OIPP documents and updated Special
Document Tracking logs. All answer documents will be batched, organized, and logically
sequenced for easy access. All materials and
systems will remain secure with DRCs fully
secure scanning facilities.
7.5.6.3 Checking Data for

Anomalous Scores
Forensic Analysis of Test
Administration (Online)
Online test administration allows the
Department to track information that is not
possible to track in the context of the paperand-pencil tests. This information includes not
only item responses but also item response
changes, latencies between item responses and
changes, number of revisits to an item or
items, test start and end times, scores in each
opportunity in the current year, scores in the
previous year, and other selected information
in the system (e.g., accommodations). Our
TDS captures all of this information.
Throughout the testing window, AIR routinely
produces a set of QA reports to ensure the
quality of the assessment system. One of the
QA reports aggregates unusual responses at
the student level to detect possible group-level
testing anomalies.
AIR psychometricians monitor testing anomalies throughout the test administration window. Evidence evaluated includes changes in
test scores across administrations, item
response time, and item response pattern
using the person-fit index. The flagging criteria
used for these analyses are configurable and
can be changed by the user. The analyses used
to detect the testing anomalies can be run anytime within the testing window. Analyses are
performed at student-level and summarized for
each aggregate unit, including testing session,
test administrator, and school.
Changes in Student Performance
Score changes are examined within (for multiopportunity assessments) and between years
using a regression model. For within-year comparisons, the most recent opportunity is
regressed on previous performance (2nd most

recent score), controlling for the number of
days between the two administrations, to identify performance gains or losses that are substantially greater than might reasonably be
expected.
For between-year comparisons, the scores
between past and current years are compared,
with the current-year score regressed on the
test score from the previous year. Betweenyear comparisons can be performed starting
with the second year of the test administration.
A large score gain or loss between grades is
detected by examining the residuals for outliers. The residuals are computed as observed
value minus predicted value. To detect unusual
residuals, we compute the studentized t residuals. An unusual increase or decrease in student scores between opportunities is flagged
with studentized t residuals greater or less
than |3|.
The number of students with a large score
gain or loss are aggregated for a testing session, a test administrator, and school. Unusual
changes in an aggregate performance between
opportunities and between years are flagged
based on the average studentized t residuals in
an aggregate unit (e.g., a testing session or a
test administrator). For each aggregate unit, a
critical t value is computed and flagged with t
greater or less than |3|,
where s = standard deviation of residuals in an

aggregate unit; n = number of students in an
aggregate unit (e.g., testing session or test
administrator); and var(e_i )=2 (1-h_ii). The
QA report includes a list of the flagged aggregate units with the number of flagged students
in the aggregate unit.
Page 7.5-75
If the aggregate unit size is 15 students, the

aggregate unit is flagged if the percentage of
flagged students is larger than 50% (%flagged
students > 50%). The aggregate unit size for
the score change is based on the number of
students included in the within- or betweenyear regression analyses in the aggregate unit.
Item Response Latency
Item response latency is captured as the item
page time (the time each item page is presented) in milliseconds. Discrete items appear one
item on the screen at a time. However, for
stimulus-based items selected as part of an
item group, all items associated with the stimulus are selected and loaded as a group. For
each student, the total time taken to complete
the test is computed by summing up the page
time for all items and item groups.
The expectation is that the item response time
will be shorter than the average time if students have prior knowledge of test items. An
example of unusual item response time would
be a test record for an individual who scores
very well on the test even though the average
time spent for each item is far less than that
required of students statewide. If students
already know the answers to the questions, the
response time will be much shorter than the
response time for those items where the student has no prior knowledge of the item content. Conversely, if a test administrator helps
students by coaching them to change their
responses during the test, the testing time
could be longer than expected.
The average and the standard deviation of testtaking time are computed across all students
for each opportunity. Students and aggregate
units are flagged if the test-taking time is
greater than or less than 3 standard deviations
of the state averagei.e., stage average + 3SD
(default criteria). The state average and standard deviation is computed based on all stu-
Page 7.5-76
dents at the time the analysis is performed.

The QA report includes a list of the flagged
aggregate units with the number of flagged students in the aggregate unit.
Inconsistent Item Response Pattern
(Person Fit)
In Item Response Theory (IRT) models, the
person-fit measurement is used to identify
examinees whose response patterns are
improbable given an IRT model. If a test has
psychometric integrity, little irregularity will
be seen in the item responses of the individual
who responds to the items fairly and honestly.
If a test-taker has prior knowledge of some test
items (or is provided answers during the
exam), the student will respond correctly to
those items at a higher probability than indicated by his or her ability as estimated across
all items. In this case, the person-fit index will
be large for the student. However, if a student
has prior knowledge of the entire test content,
this will not be detected based on the personfit index, although the item response latency
index might flag such a student.
The person-fit index is based on all item
responses. An unlikely response to a single test
question may not result in a flagged person-fit
index. Of course, not all unlikely patterns indicate cheating, as in the case of a student who
is able to guess a significant number of correct
answers. Therefore, the evidence of person-fit
index should be evaluated along with other
testing irregularities to determine possible testing irregularities. The number of flagged students is summarized for every testing session
and test administrator.
The person-fit index is computed using the
standardized log-likelihood statistic. Following
Drasgow, Levine, and Williams (1985),
Sotaridona, Pornell, and Vallejo (2003) define
aberrant response patterns as a deviation from
the expected item score model. Snijders

(2001) showed that the distribution of lz is
asymptotically normal (i.e., with an increasing
number of administered items, i). Even at
shorter test lengths of 8 or 15 items, the
asymptotic error probabilities are quite reasonable for nominal Type I error probabilities
of 0.10 and 0.05 (Snijders, 2001).
Sotaridona et al. (2003) report promising
results of using lz for systematic flagging of
aberrant response patterns. Students with lz
values in excess of +3.0 are flagged. Aggregate
units are flagged with t greater or less than |3|.
Sample Report
The QA report can be generated for each subject or all subjects combined. The analysis
report is generated at different aggregate units
(e.g., testing session, test administrator, or
school). The report includes two tabs: the student tab and the aggregate tab. The student
tab includes information on students who are
flagged with at least one of the flag indices in
each flagged aggregate unit. The aggregate tab
includes information on the aggregate units
that are flagged with at least one of the flag
indices.
Exhibit 7.5.6.3-1 presents a sample QA report
for testing irregularity analyses for testing sessions. The table includes a list of the cheating
indices for the flagged aggregate units.
lz
where s = standard deviation of lz values in an

aggregate unit and n = number of students in
an aggregate unit. The QA report includes a
list of the flagged aggregate units with the
number of flagged students in the aggregate
unit (e.g., school, test administrator, test session).
Forensic Analysis of Test

Administration (Paper)
AIR team member DRC provides anomalous
data review for numerous clients, giving the
company the expertise and capability to offer
similar services to the Department. DRCs in-
Exhibit 7.5.6.3-1: Systems Summary

Testing Period: October 17, 2011, to December 31, 2012
Cheating Index (# of Flagged Students)
Score Change
Within Year
Test ID
Score Change
Between Year
Response Time Flag Person Fit Index

(# of Flagged
(# of Flagged
Students)
Students)
Most
Most
Second
Recent Recent
Total # Most
# of
# of
Aggregate School Total #
Most
Students Recent
Students Score in Score in Students Opp1 Opp2 Opp3 Opp1 Opp2 Opp3
Type
ID Students
Recent
Flagged Score
Flagged Current Previous Flagged
Score
Year
Year
HSA_OP_Mathematics-6
Naga-3
100
Naga-3
100
Naga-3
100
Naga-3
100
Naga-3
100
Naga-3
100
Page 7.5-77
house psychometric staff will provide all

anomalous score/test irregularity data review
for paper-based tests for the Florida Standards
Assessment. AIR and DRC will work with the
Department to develop procedures for this
process, with the Department having final
approval over all procedures that will be implemented by DRC.
After scanning of all documents for each
administration has been completed and prior
to report generation, DRC will review all student and school scores for anomalous data and
testing irregularities to identify any data that
would not be considered a valid measure of
student achievement or that might be considered evidence of suspicious activity. If data
irregularities are found, DRC will notify the
Department, which may request that some of
the impacted data be withheld from release,
pending additional investigation. AIR and DRC
will work with the Department to set up rules
regarding the use of Score Status Flags during
reviews between the first and second versions
of the full-state student results files. AIR will
ensure that adequate time for the anomalous
data review process is reflected in the program
schedule, while still providing for on-time
report deliverables.
The following sections provide information on
DRCs mark discrimination technology and
procedures and response data analysis process.
Mark Discrimination
The image scanners used by DRC utilize high
quality color cameras capable of capturing
images up to 300 dpi. Gray-scale images are
captured using 256-level gray-scale (8 bits per
pixel) format. DRCs image scanning system
produces sufficient image quality to permit
distinguishing between various marks and
darkness levels and allows DRC to store electronic responses that emulate the quality of
the original paper response. The flexibility of
Page 7.5-78
DRCs image scanning system allows for erasure marks to be easily captured based on
specified criteria. DRCs scanners are calibrated multiple times daily to ensure that scanners
accurately and consistently capture student
responses.
All marks on a student answer document are
captured at the time of scanning. Marks for
multiple choice responses will be assigned a
read level. Each response bubble is scored
based on two factors: darkness (how darkly the
bubble is marked); and pixel coverage percentage (how much of the bubble is marked).
Using darkness and pixel coverage, each bubble is assigned a value based on industry standards. The value is assigned to one of three
ranges: unmarked or very lightly marked;
lightly marked; or clearly or darkly marked.
Then, each question is assigned a response
based on the value for each of its bubbles:
If only one bubble is marked, then the
response is set to that bubble.
If two bubbles are marked, and one is lightly marked and the other is clearly marked,
then the response is set to the clearly
marked bubble.
If two bubbles are marked, and both are
lightly marked or both are clearly marked,
then the response is marked for review by
a human editor.
If all bubbles are unmarked or only very
lightly marked, the response is marked for
review by a human editor.
Generally, any item for which the scanning
process determined that more than one bubble
was marked will be evaluated for erasures. AIR
and DRC will work with the Department to
define erasure classification criteria. There are
three types of erasures: wrong-to-wrong (WW);
right-to-wrong (RW); and wrong-to-right (WR).
These responses will be aggregated at the state
level to determine what is typical erasing
behavior and the baseline for comparisons of

subgroups, whether district, school, classroom,
or other identified subgroup level.
The average number of WR erasures will be
computed for each subgroup (e.g., school)
statewide. This average is evaluated in terms of
the meaningful statistical difference from the
state-level WR erasure rate. The Department
will receive a report with each subgroups average erasure rates as well as an indication of
how improbable the erasure rates are, given
the state level. These reports can be generated
at the district, school, classroom, or other
identified subgroup level. An illustration of
what this report may look like is shown in
Exhibit 7.5.6.3-2.
In the example, the indication of the improbability of the erasure rate is given by the Outlier
Score. The higher the Outlier Score, the more
different the schools average WR erasure rate
is from the state level. The Outlier Score is a
translation of the probability of occurrence.
The Outlier Scores highlighted in pink are an

indication of unlikely erasure results that may
warrant further investigation. This Outlier
Report would accompany a written summary
of results and methodology.
In addition to analyzing erasure data at a subgroup level, DRC could produce supporting
materials for an erasure analysis. An example
of such supporting materials may be an
enhanced erasure map report. This report has
been a primary tool in reporting answer
change analyses results for many of DRCs
clients. It provides a clear visual summary of
what is happening inside a school. Key aspects
of this report include the following:
Identification of key levels of aggregation
(i.e., district, school, and student)
Color-coded shading indicating type of
answer change (wrong to right, right to
wrong, or wrong to wrong)
Exhibit 7.5.6.3-2: Sample Outlier Score Report


Page 7.5-79
Shading of the field-test item columns to

easily distinguish them from operational
items
Exhibit 7.5.6.3-3 shows a sample erasure map
report. The erasure map is an Excel document,
which allows the user to filter data on a grouping category. It will help provide visual
answers to such questions as these:
Within the school, do the field-test items
have a similar erasure pattern as the operational items?
Within the school, are the same items
being erased?
Are the erasure patterns consistent for a
student across subjects?
tives. The criterion used to identify statistically different situations will be smaller than the
usually expected 0.05 alpha level, thus reducing the chance of falsely identifying schools.
Results will be presented in a manner clarifying that statistical indicators are not proof of
improprieties but rather flags for situations
that could warrant further investigation.
Response Patterns
Student test responses can be analyzed for
unusual testing patterns. DRC has had success
in implementing a modified approach used by
Jacob and Levitt (2003), which incorporates
multiple approaches to pattern analysis.
Suspicious answer strings may be identified by
examining responses for
Usual Increase in School Performance

It is anticipated that performance on the
Florida Standards Assessment will improve
over time due to legitimate reasons such as
changes in the curriculum and improvement
in instruction. However, large and unexpected
score changes may be a sign of suspicious
activity. An approach to identifying improper
behavior is to compare the states level of
change in performance from one year to the
next to a schools change in performance during the same timeframe. This comparison will
determine if a schools average score change is
statistically different from the statewide average score change. Schools, as with erasure
analysis, will be identified with an Outlier
Score. As with any statistical test, there is
Type I error and the possibility of false posi-
unlikely blocks of identical consecutive

answers;
degree of similarity beyond consecutive
items as assessed by the degree of correlation in all student responses; and
cases in which students miss easy items
while answering difficult items correctly.
These methods combine into an index that
can be used to identify schools showing unexpected answer patterns and which may require
further attention. It is possible for unexpected
answer patterns to appear without inappropriate behavior having occurred. For this reason,
the pattern index is combined with an index
measuring unexpected test score fluctuations
across years using a cohort of students.
Exhibit 7.5.6.3-3: Sample Erasure Map

Page 7.5-80

!

"

#

$

%

&

'

!

"

%

&

'

Schools are ranked on both indices and flagged

if in the 95th percentile on both. By ranking
the schools on both indices, Type I error is
lower than if using only one index. Thus,
schools are protected from being incorrectly
identified.
The reporting software systems accurately

report and aggregate the student scores.
Should the Department determine that its data

forensic needs are broader, DRC welcomes an
opportunity to share its expertise. DRC has
extensive experience in assisting departments
of education across the country with their data
forensic studies. Additionally, DRC continues
to research emerging and state-of-the-art
detection methodologies, presents at national
conferences (including the Annual Statistical
Detection of Potential Test Fraud Conference);
and updates developments in security guidelines from the National Council on
Measurement in Education (NCME), Institute
of Education Sciences National Center for
Education Statistics, and Technical Issues in
Large Scale Assessment (TILSA).
Print and packaging quality is maintained

for paper reports.
7.5.6.4 Checking Full-State

DataInitial Reporting
Online Reporting System Verification
Whether tests are administered on paper or
online, scores will be reported back on both
paper and online. Ideally, the online reports
will be delivered in real-time, relying on operational item keys, machine rubrics, and preequated item parameters. As test results come
back from the TDS or scanning-scoring
process, they are routed to our Test
Integration System (TIS) and Quality Monitor
(QM) system and ultimately to our systems for
reporting. The following quality checks are
implemented:
The student response and score data to be
reported are correct.
Paper reports contain accurate data correctly displayed, and any Florida-specific
programming is tested and replicated to
ensure that it is error free.

Each year, AIR will provide the Department
with specifications for scoring and reporting.
In addition to detailing the IRT calibrations
and ability estimation procedures, the specifications document includes rules for validating
test records (i.e., attemptedness and invalidated records) and aggregating test records
(including handling of special schools and districts). Also included are specifications for all
paper and online reports at each level of analysis, including family reports, and classroom-,
teacher-, school-, and district-level roster
reports, summary reports, growth reports, and
reporting of relative strengths and weaknesses
of strand- and benchmark-level standards.
Included as well are specifications for all
downloadable data files from the ORS. Upon
Departmental sign-off on the scoring and
reporting specifications, statistical programming of the ORS will begin, with final verification of the ORS functionality scheduled to be
complete prior to launch of the production
ORS for UAT.
Student Response and Score Data are

Correct
Data entering the reporting process, either
from paper or online tests, flow through our
QM software, described in detail under Section
7.5.4. QM conducts a series of data integrity
checks, ensuring, for example, that the record
for each test contains information for each
Page 7.5-81
item that was supposed to be on the test and

that the test record contains no data from
items that have been invalidated. QM scores
the test, recalculates performance-level designations, calculates subscores, compares item
parameters to the reference item parameters
in the bank, and conducts a host of other
checks.
QM also aggregates data to detect problems
that become apparent only in the aggregate.
For example, QM monitors item fit and flags
items that perform differently operationally
than their item parameters predict. This functions as a sort of automated key or rubric
check, flagging items where data suggest a
potential problem. This automated process is
similar to the sorts of checks that are done for
data review, but (a) they are done on operational data; and (b) they are conducted in real
time so that our psychometricians can catch
and correct any problems before they have an
opportunity to do any harm. Reports on item
quality are available to our psychometricians
on-demand through an automated, online system.
Reporting System Software Accurately

Reports and Aggregates Student
Scores
Online reports appear in real time. Therefore,
QA cannot rely on post-hoc reviews of reports.
Instead, the accuracy of the reporting system
rests on the quality of programming of the system, the implementation of the Departments
business rules, and the quality of the algorithms used for aggregating scores. These
processes are described in Section 7.5.5.
In building our systems, we undergo an extensive software testing process. When we configure the systems for individual clients, we test
them by running simulated or real data
through the full system, allowing the system to
generate reports. Our statistical programming
Page 7.5-82
team simultaneously implements the same

rules and processes the same data. Statistics
from the system are then compared to statistics produced by our statistical team.
Discrepancies are tracked down and resolved.
As described in Section 7.5.5, this entire
process is guided by a set of complete reporting and analysis specifications. Both the software team and the independent statistical programming team work from the same specifications but work independently. This process
provides an independent check on the immediate reporting system before it is deployed. Of
course, the Department is welcome to review
the data at any time.
7.5.6.5 Checking Graduating

Student Scores (Manual Rescore)
For the high stakes grade 10 English Language
Arts/Literacy (ELA/L) and Algebra 1 End-ofCourse (EOC) assessments, which are required
for graduation, the Department requires that
paper test administrations be manually
rescored for students failing by five scale-score
points or 15 developmental scale-score points.
We recognize that the FSA may have a reporting scale that differs from the current FCAT
2.0 scale or the developmental scale. We
assume that the boundaries for rescore will be
translated to the FSA scale to target approximately the same number of students for
rescores.
We note that both the Algebra 1 and Grade 10
ELA/L tests will be delivered primarily online.
Online tests can be reviewed. AIRs has a wellseasoned system for exactly this type of
review: The Secure Item Response Viewing
Environment (SIRVE), which provides a view
for authorized users to step through the students test. SIRVE uses the same code as the
Test Delivery System (TDS) Student Interface
to ensure that the viewing experience is pre-
sented exactly the way it was presented when

the student took the test, and presents the student response exactly as the student saw it.
Once the authorization and identification are
confirmed, the authorized user can then view
all of the student tests items and responses.
Refer to Exhibit 7.5.6.5-1 for a sample screen
capture of this view. The user can navigate
through the test by using the same buttons
and tools as the student such as the Back and
Next buttons, the Mark for Review drop-down
menu, and the End Test button.
When students who took the accommodated
paper form fall within the rescore boundaries,
DRC will initiate and track the student
response document retrieval and rescore
process. Processed answer documents will be
retrieved quickly and efficiently as the need
arises, either during or upon completion of
processing. Original hardcopies of individual
student tests will be easily retrievable because
of DRCs effective document storage procedures. Additionally, DRCs IBML image scanners and Image Scoring System allow for ondemand retrieval of specified images (e.g., spe-
cific batch files, specific grades, specific students); each image is assigned a unique identification number that allows for quick and easy
retrieval at the student and school levels.
After the appropriate student response documents have been retrieved, they will be
reviewed manually by experienced and qualified personnel who will scrutinize the responses to verify student identification information
and confirm that the items have been scored
as the student intended, looking for such
issues as unintentional stray marks in the
answer bubbles, incomplete erasures, or unintended double gridding. In addition, rescore
personnel will look for any irregularities in
editing or processing by comparing all item
responses to the values on the SSR. All applicable security and quality-control procedures
that were implemented by DRC during the
original processing and scoring will be maintained.
If any problems are identified that cause a students score to change to a passing score, the
data will be revised and information will be
reported in the first wave of late reporting,
Exhibit 7.5.6.5-1: SIRVE Sample Screenshot
Page 7.5-83
with a new student report being produced and

distributed to the students district. Data from
this students rescore will also be included on
the School Report of Students sent with the
other first wave of late reporting student data.
In addition, AIR and DRC will provide the
Department with a report that details the
number of student tests reviewed by subject
and the number of rescores that resulted in a
passing status. AIR and DRC understand that if
rescoring does not result in a passing score, no
change will be made in the record.
7.5.6.6 Checking Late Reports

The same procedures used to verify the accuracy of answer documents, test scores, and
score reports received during the on-time
reporting window will be applied to all documents received following the close of the ontime window. As we describe in Section 7.5.5,
all answer documents, including those
received after the on-time window, are imaged,
with machine-scored item responses uploaded
to the QM, where item responses and test
scores are verified prior to being routed to the
Database of Record (DoR). Item responses
requiring human scoring are routed to the performance scoring system and are then
uploaded to the TIS, where hand-scored items
are stitched back to the test opportunity
record. The single, integrated test opportunity
record is again passed through the QM before
being deposited into the DoR. Student item
and test scores are then routed to the ORS,
which incorporates student enrollment and
demographic information, along with user
information, to produce score reports and
downloadable data files. All of these systems
are thoroughly verified prior to launch of the
online testing window, so that all reporting,
whether generated in the on-time window or
later, are accurately produced.
Page 7.5-84
7.5.6.7 Checking Close-Out and

Annual EOC Files
The same QA processes used to verify the
accuracy of item and test scores, and the score
reports and data files produced from them, will
be applied to the generation of close-out and
annual EOC files.
7.5.6.8 Checking Demographic

Files
As we describe in Section 7.5.4, the
Department may choose to manage enrollment
data directly and pass that information to the
Test Information Distribution Engine (TIDE),
or districts may upload and manage enrollment data directly in TIDE. Student and user
data collected in TIDE, or passed through to
TIDE by the Departments system, would be
stored in the Roster Tracking System (RTS)
databasea flexible database system shared
by TIDE, TDS, reporting, and the Data Extract
Generator (DEG) systems. This database
ensures one reliable, common source for all
student and user registrations. Data can be
loaded into RTS (through TIDE) by either an
automated data exchange through a secure file
transfer protocol (SFTP) location or a userbased upload. Because these data control student eligibility for specific tests and can control availability of individual accommodations
and supports, students who are not represented in the RTS are not eligible for test administration. However, we also understand that student data are not static and may need to be
updated prior to final reporting. Moreover,
because some assessments are still paperbased, and schools will continue to use generic
PreID labels on answer documents when student PreID labels are not available, a mechanism is necessary for updating student and
user information in TIDE.
To accomplish this, we propose to open TIDE

to districts or accept updated enrollment
records directly from the Department, for a
period during the testing window. This will
allow districts to make any necessary updates
to student and user data to ensure the most
accurate reporting of assessment results, while
also allowing final results to be generated
immediately following the close of the test
administration window each year.
References
Allport, D. A., Tipper, S. P., & Chmiel, N. R. J.
(1985). Perceptual integration and postcatagorical filtering. In M. I. Posner & O. S.
Martin (Eds.), Attention and performance
XI (pp. 107132). Hillsdale, NJ: Erlbaum.
independent of each other? Paper presented at the annual meeting of the

Psychonomic Society, San Francisco, CA.
Kubovy, M., & Cohen, D. J. (1992, November).
Further tests of feature-module architectures for color and form. Paper presented
at the annual meeting of the Psychonomic
Society, St. Louis, MO.
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2),
149174.
Neisser, U. (1966). Cognitive psychology. New
York: Appleton-Century-Crofts.
Snijders, T.A.B. (2001). The statistical evaluation of social network dynamics.
Sociological Methodology, 31, 361395.
Callaghan, T. C. (1984). Dimensional interaction of hue and brightness in preattentive

field segregation. Perception and
Psychophysics, 36, 2534.
Sotaridona,L.S., Pornel, J.B., & Vallejo, A.

(2003). Some applications of item response
theory to testing. Philippine Statistician,
52(1-4), 81-92.
Callaghan, T. C. (1989). Interference and dominance in texture segregation: Hue, geometric form, and line orientation. Perception
and Psychophysics, 46, 399411.
Treisman, A., & Sato, S. (1990). Conjunction

search revisited. Journal of Experimental
Psychology: Human Perception and
Performance, 16(3), 459478.
Cohen, D. & Cohen, J. (2006). The sectioned

density plot. The American Statistician,
60, 167174.
Drasgow, F., Levine, M.V., & Williams, E.A.
(1985). Appropriateness measurement with
polychotomous item response models and
standardized indices. British Journal of
Mathematical and Statistical Psychology,
38(1), 6786.
Jacob, B. A., & Levitt, S. D. (2003). Rotten
apples: An investigation of the prevalence
and predictors of teacher cheating. The
Quarterly Journal of Economics, 118(3),
843877. doi: 10.1162/
00335530360698441
Kubovy, M., & Cohen, D. J. (1991, November).
Are texture segregation by form and color
Page 7.5-85
7.6.0 Scaling, Equating, Scoring, and Special

Psychometric Studies
7.6.0 Scaling, Equating, Scoring,

and Special Psychometric Studies
One of the principle advantages of online
assessments is that they allow for the possibility of immediate scoring and reporting of test
results, which educators and students actually
expect following online test administration.
Should Florida decide to build on the Student
Assessment of Growth and Excellence (SAGE),
the entire assessment is machine scored, with
the exception of some parts of the writing
assessment.
Immediate scoring and reporting of test results
requires that tests be scored using pre-equated
item-parameter estimates. Given that item
parameters are generally estimated based on
large and representative samples of student
responses to field-test items embedded within
operational test administrations, pre-equated
item-parameter estimates are highly precise and
stable over time. Pre-equating has the additional benefit of eliminating the need to identify a
sample of schools for early return and processing to support post-equating analyses.
The Department has historically used a postequating approach to calibrating and equating
item parameters for scoring paper assessments,
adopting a pre-equating approach only for the
grade 10 retake tests which are administered to
a highly selected sample of students. In a
paper-based assessments system, the time
required to retrieve, process, and hand score
answer documents is such that states, with the
use of early return calibration samples, could
complete post-calibration activities in a timeframe that had little or no impact on the score
reporting timeline. Of course, reports were also
typically distributed in the summer months
when schools were not in session, reducing the
overall value of score reports to families and
educators. With the advent of online testing
and the possibility of immediate scoring and
reporting of test results, the expectation and
valuation of immediate feedback may outweigh

the relatively small gains in measurement precision that may accrue from post-equating
with the possibility of real-time reporting of
test results, there are now costs associated with
post-equating that likely do not outweigh the
benefits of immediate feedback.
For grades and subjects where assessments are
administered in paper format, we propose to
maintain the Departments post-equating
approach to scoring test results. For these
assessments, AIR will provide the Department
with a plan for drawing early return samples
on which to conduct post-equating analyses,
analysis specifications identifying the Item
Response Theory (IRT) models for calibrating
each item type, and the equating procedures
for transforming the post-equated item-parameter estimates back to the reference scale, as
well as simulated data files to support dry runs
of all post-equating activities.
As assessments move online, however, we propose to use a pre-equating design for scoring
and reporting assessment results to support
immediate reporting of test scores. We clearly
recognize that any decision to move away from
post-equating for the online assessments will
be made by the Department, and we are prepared to continue implementing the postequating approach to the scoring of online
assessments if the Department chooses.
Moreover, while the item banks available to
construct the Florida Standards Assessment
(FSA) are already calibrated, the Department
may wish to establish a uniquely Florida reference scale. In this approach, the Department
would re-calibrate the operational test items
following the first test administration and then
identify the linking constants necessary to
bring all other bank items onto the new
Florida scale. All subsequent operational test
administrations could then be scored using
pre-equated item parameters on the Florida
Page 7.6-1
scale. If the Department intends to establish

performance standards following the first operational test administration, then immediate
score reporting will not be possible for the first
test administration in any event, so establishing a new reference scale would not hinder the
reporting of test results.
Calibrating, Scaling, and Equating

Calibration is the process by which we estimate the statistical relationship between item
responses and the underlying trait being measured. AIR proposes to continue to use statistical models based on IRT (Allen & Yen, 1979;
Birnbaum, 1968; Lord & Novick, 1968) to calibrate the items to a common scale. AIR is
experienced with a wide range of IRT models,
extending from the simple one-parameter (or
Rasch) model (Rasch, 1980; Cohen & Kolstad,
2000) to multi-parameter models (Johnson,
1992; Beaton, Johnson, & Ferris, 1987) and
even to cutting-edge multicomponent models
(Cohen & Kolstad, 2000).
The Department has traditionally used the
more general IRT models to describe student
performance on test item. These more complex models (essentially, models that acknowledge the possibility of guessing on multiplechoice items and recognize that some items
measure with less error than others) do not
offer a one-to-one correspondence between
raw scores and scale scores. Although some
psychometricians (Thissen & Orlando, 2000;
Thissen, Nelson, & Swygert, 2000; Yen, personal communication, 1999) advocate mechanisms by which raw-score conversion tables
can approximate the scores that would be
assigned on the basis of these more complex
models. AIR recommends that scores be calculated using maximum likelihood with boundaries for the perfect scores (all correct or
below chance), and this is consistent the
Departments approach to scoring tests.
Page 7.6-2
Overview of Some Common IRT Models

Traditional IRT models assume a single underlying trait and assume that items are independent given that underlying trait. In other words,
the models assume that given the value of the
underlying trait, knowing the response to one
item provides no information about responses
to other items. This basic simplifying assumption allows the likelihood function for these
models to take the relatively simple form of a
product over items for a single student:
where Z represents the pattern of item

responses and represents a students true
proficiency. Traditional IRT models differ only
in the form of the function P(Z). The oneparameter model (1PL, also known as the
Rasch model), which is used for items that are
scored either right or wrong, takes the form

The b parameter is often called the location or

difficulty parameterthe greater the value of
b, the greater the difficulty of the item. The
one-parameter model assumes that the probability of a correct response approaches zero as
proficiency decreases toward negative infinity.
In other words, the one-parameter model
assumes that no guessing occurs. In addition,
the one-parameter model assumes that all
items are equally discriminating. (The typical
formulation of the Rasch model does not
include the arbitrary constant 1.7 factor in the
exponentialthe factor is included here for
consistency with the more general models
described below.)
The two-parameter logistic model (2PL) takes

the form

The b parameter is still called the location or

difficulty parameter. The a parameter is often
referred to as the slope or discrimination
parameter. The slope parameter is essentially
the inverse of the standard deviation of the
measurement error associated with the item.
For multiple-choice models, the three-parameter logistic model (3PL) adds a guessing
parameter, so the probability of a correct
response is given by

The third parameter, c, defines a lower asymptote. In the one- and two-parameter model, the
probability of a correct response approaches
zero as proficiency decreases toward negative
infinity. The third parameter allows the probability to approach some other lower bound.
Given multiple-choice questions, a student
with very little ability on the target trait could
guess a right answer. The guessing parameter
captures the effect of such guessing.
For items that have multiple, ordered response
categories (i.e., partial credit items), we again
have the choice of a simple Rasch family
model (Masters 1982 partial credit model) or a
more general variant such as Murakis generalization of Samejimas (1972) graded response
model. For smaller-sample tests (including, for
example, many alternative assessments), we
recommend the Rasch-family variants because
they can be reliably estimated with fewer
cases. Under Masters model, the probability of
a response in category i for an item with mj
categories can be written as
Murakis generalization adds an item dependent discrimination parameter as follows (again,

Masters formulation does not usually include
the arbitrary constant 1.7):
Returning to the likelihood equation, the contribution of each item to the overall likelihood
function remains independent of all other
items, given . This is convenient for two reasons: mixing models within an analysis (e.g.,
one-parameter and partial-credit items on the
same scale) becomes no more complicated,
and the likelihood of the response pattern may
be calculated as the product of the likelihood
of responses to individual items.
Estimation of IRT Models

AIR will use any software requested by the
Department to calibrate item parameters. We
prefer to use our own software, the IRT Plugin
to our AM statistical software (iAM), which we
distribute without charge (Cohen & AIR,
2002). iAM offers some capabilities that are
not available from other packages, as we discuss below. We note, however, that AIR psychometricians currently use IRTPRO, Multilog,
Parscale, and other software to calibrate item
parameters using the more general IRT models
in other state assessment programs, and we
are therefore prepared to use any software
requested by the Department.
iAM offers a key advantage over other available
software because it provides design-consistent
standard errors and yields accurate estimates
of linking errors under complex designs. Like
Page 7.6-3
Parscale, iAM includes a variety of constructed-response models (including the generalized

partial credit model) that can be simultaneously estimated with two- and three-parameter
logistic models on the same test. Like BilogMG, iAM can simultaneously estimate the proficiency distributions in multiple groups.
Building on its design-consistent estimators,
iAM provides fit statistics that reflect the complex sample design.
Like Parscale and Bilog-MG, iAM estimates item
parameters by using marginal maximum likelihood (MML), which provides consistent estimates under the assumed proficiency distribution. MML estimation numerically integrates
over the population proficiency distribution. In
the absence of analytic solutions to the problem, finding the parameter values that maximize the likelihood of the data requires numerical optimization. The preferred method for finding the maximum likelihood estimate of such
models is the use of the expectation maximization (EM) algorithm (Dempster, Laird, & Rubin,
1977). Bock and Aitkin (1981) propose an EM
algorithm for this class of models, and this is
the approach taken in popular IRT software.
depend on a common set of item parameters

that are themselves estimates. Hence, IRT fit
statistics are notoriously conservative. This
problem is compounded by the fact that item
data almost always come from a complex, clustered sample. The end result is that typical
2
IRT packages report significant
statistics
for most items, regardless of actual fit.
AIR uses a fit statistic that overcomes these
problems, incorporating uncertainty in the
item parameter estimates and uncertainty in
the estimates of the latent trait and taking into
account uncertainty in the item statistics.
When these sources of uncertainty are correct2
ly accounted for, the
becomes an F statistic.
iAM reports fit statistics along with innovative
graphics that characterize the item fit, as
shown in Exhibit 7.6.0-1.
Evaluation of Model Fit

Psychometricians use a variety of measures to
evaluate the fit between models based on IRT
and observed item data (Yen, 1981; McKinley
& Mills, 1985; Fitzpatrick et al., 1996). These
statistics operate by evaluating the variance
between observed and predicted item responses, a task made difficult by the fact that both
the observed and the predicted item responses
are contingent on the value of a latent trait
( ). Typically, some estimate of is used,
and the discrepancy between the true and estimated values of is ignored. Fit statistics typically ignore a second source of error as well:
the estimates of and the predictions both
Page 7.6-4
Exhibit 7.6.0-1: Item Fit Graphics
The top panel in Exhibit 7.6.0-1 shows a sample fit graph for a well-fitting multiple-choice
item, and the bottom panel shows a sample fit
graph for a poor-fitting partial credit item. In
both graphs, the x-axis represents the range of
student ability values, the y-axis represents
probability, the solid line represents the item
characteristic curve (ICC), the dots represent
the empirical estimates, and the bars indicate
the standard error of the distance of the
empirical estimates from the ICC. Note that
the standard error bars are smaller near the
center of the distribution and grow larger
toward the tails. When the standard errors
grow to be virtually infinite, the empirical estimates are represented as an empty circle.
2.2
1.6
0.9
0.3
-0.3
-4.0
-2.0
0.0
2.0
4.0
I_35_7 (F=1.810893, p>F=0.176081
2.2
1.4
0.7
-0.1
-0.9
-4.08
-2.04
0.00
2.04
4.08
I_35_5 (F=36.950400, p>F=0.000000
The item in the first panel of Exhibit 7.6.0-1

shows a well-fitting item as misfit appears only
at the tails of the distribution, where data is
sparse and often not enough to draw valid conclusions. The second graph, however, shows
misfit along the one-point line (represented as
a black ICC). The empirical data points fall
substantially off of the ICC line. Although no
single point is outside of its confidence limit,
the cumulative misfit across points is statistically significant, as depicted by the F printed
at the bottom of the graph. These fit graphics
were designed to provide clearer insights than
the fit graphics typically produced by off-theshelf programs. For example, some graphics
use larger symbols to denote scores with more
casesa practice that is visually misleading.
Estimates from large samples are precise and
should appear at a precise point on the page,
while estimates from small samples should
appear more diffuse. Our graphics achieve this
by graphing standard error bars. Near the center of the proficiency distributions, the estimates are precise, as shown by narrow error
bars; they grow diffuse in the tails where little
data appear, as shown by wide error bars.
Page 7.6-5
Florida Technical Advisory Committee

(TAC)
AIR will coordinate with the Department to
designate the TAC meeting site. AIRs
Conference Services team will support the
logistics and make the lodging and meal
arrangements for TAC meetings. AIR will reimburse TAC members for their services at the
designated rates. AIR will distribute travel
information and reimbursement paperwork
prior to and/or during the meeting to ensure
that committee members are knowledgeable
about logistics, allowances, stipends, and reimbursements.
AIR will process per diems and will reimburse
committee members expenses according to
mutually agreed-upon guidelines. AIR understands that the Department holds two TAC
meetings annually in March and November.
AIR is prepared to participate in both of these
each year. Our proposed lead psychometrician,
Dr. Harold Doran, will attend all TAC meetings.
AIR typically sends additional staff as necessary. Often, this includes our director of psychometrics, Dr. Gary Phillips, and other psychometricians and statisticians who have led
the studies to be presented at the TAC meetings.
Our project team will work with the
Department to identify any required studies,
issues, plans, or results that would benefit
from TAC review. We will work with the
Department to develop the TAC agenda and
prepare the necessary materials. AIR will prepare materials to support the TAC meetings
and deliver them for the Departments review.
We propose to deliver them to the Department
at least two weeks prior to each meeting to
provide time for review and revision so that
the materials may be distributed to the TAC
members no later than one week prior to the
meeting.
Page 7.6-6
7.6.1 Assessments
The following sections provide detailed information about scaling, equating, and scoring for
Florida Standards Summative Assessments and
Retake Assessments.
7.6.1.1 Florida Standards

Summative Assessments (Scaling,
Equating, and Scoring)
As described in Section 7.3.7, AIR will annually provide the Department with test construction specifications consistent with the desired
FSA model. For an adaptive assessment, the
specifications culminate in a simulation technical report, which describes the configuration
of the adaptive algorithm that maximizes
measurement precision and blueprint match
criteria and which documents the performance
of the adaptive algorithm. For fixed-form
assessments, the specifications also include
test blueprints but additionally describe psychometric targets for test information and
measurement precision as well as tolerances
for deviations between test characteristic
curves between forms. Equivalent test forms
will be built to specifications and then submitted to the Department for review and approval.
In addition, AIR will produce a set of analysis
specifications, as described in Section 7.6.2,
detailing the procedures used to calibrate,
equate, and score the FSAs. Although AIR proposes to use pre-equated item-parameter estimates to support immediate scoring and
reporting of online test administrations, the
Department has historically employed a postequating approach to scoring assessment
results, which AIR will certainly support.
IRT Models. The Department has historically
calibrated multiple-choice items using the 3PL
IRT model, the 2PL model for griddedresponse items, and the generalized partial
credit model for constructed-response items.
AIR will continue to calibrate these item types

using the models preferred by the Department.
In addition, AIR will review with the
Department each of the available machinescored item types and identify the most appropriate model for calibrating each item type.
For example, while most machine-scored items
are true constructed-response items, which are
most appropriately scored using the 2PL or
generalized partial credit (GPC) IRT models,
there may also be some hot-spot items, which
are essentially multiple-choice items that may
be most appropriately calibrated under the
3PL IRT model.
Equating Method. To link test items back to
the Florida reference scale, the Department
has traditionally employed the widely used
Stocking-Lord (Stocking & Lord, 1983)
method for minimizing squared differences
between test characteristic curves of linking
items across forms. To verify the robustness of
the linked-parameter estimates, the
Department also prefers to replicate the equating procedure using alternative methods,
including moment-based mean/mean and
mean/sigma transformations, and the Haebara
(1980) method of minimizing the sum of
squared differences between item characteristic curves of linking items across forms. AIR
psychometricians are prepared to apply any
methods the Department wishes to equate test
forms.
Maximum-Likelihood Estimation (MLE)
Scoring. As noted previously, AIR recommends computing test scores using maximum
likelihood pattern scoring, whether tests are
administered adaptively or as fixed forms. This
approach is consistent with the Departments
historical approach to scoring students assessments.
Plan for Early Return Equating Sample. AIR
is the industry leader in sampling designs for
large-scale assessments. The impact of sam-
pling on test results can be substantial, but

most testing companies overlook it. A smaller
sample, randomly selected according to an
appropriate sample design, provides better
estimates of a population than a larger sample
selected using ad hoc methods. Random sampling offers two critical advantages that have
been the pillars of statistical science over the
last century and a half:
Random selection ensures that the probability of selection into the sample is independent of anything that might affect the target
statistics, eliminating potential biases.
Random selection yields estimates with
known sampling properties, enabling statisticians to estimate accurate standard
errors. With ad hoc sampling methods,
accurate estimators for the standard errors
are unavailable, and therefore the resulting
statistics reliability is unknowable.
The National Research Councils (NRCs)
review of National Assessment of Educational
Progress (NAEP) underscored the importance
of appropriate variance estimates in testing. It
determined that the reliability of NAEP estimates was overstated because they neglected
to appropriately consider the standard errors
of the item parameters (Pellegrino, Jones, &
Mitchell, 1999). Shortly thereafter, the
National Center for Education Statistics
(NCES) contracted with AIR to develop estimators for the standard error of IRT item
parameters under complex samples (Cohen,
Jiang, Seburn, & Chan, 2008). These estimators are available through the iAM software,
which AIR distributes without charge (Cohen
& AIR, 2002)
The ability to accurately estimate standard
errors of item parameters, linking constants,
and other key statistics has benefitted AIR
state assessment clients in important ways by
providing
Page 7.6-7
a necessary mechanism to evaluate samples and refine sampling designs over time;
a state-of-the-art basis for evaluating the
equating error of the different forms used
over time;
key measures of linking errors used in evaluating alternate approaches to developing
vertical scales; and
the basis for evaluating (and subsequently
revising) long-standing procedures for dropping linking items used in equating.
AIR will provide sampling plans to the
Department for any activity in which sampling
is required, including early return samples for
equating analyses, as well as stand-alone fieldtest samples to field test items that cannot
appropriately be embedded in operational test
forms. Our proposal to compute test scores
based on pre-equated item-parameter estimates would eliminate the need for drawing an
early return sample, although even if the
Department ultimately transitions to immediate scoring and reporting, an early return
equating sample may still be required for the
first operational test administration.
Concurrent Calibration for Multiple Form
Assessments. For assessments in which multiple forms are administered, including Algebra
1, Algebra 2, and Geometry, rather than separately equating each form to the reference
scale, the Department prefers to concurrently
calibrate items across the multiple forms and
then equate the linking items back to the reference scale. This approach is desirable
because it increases the number of linking
items used to equate test forms, yielding more
reliable linking constants. This approach is
strengthened by a robust common item linking
design for the multiple forms.
Braille Forms. Following principles of universal design, test items will be constructed so
that all accommodated forms, including
Page 7.6-8
Braille, will be as nearly equivalent to items

administered to the general education form as
possible, thereby allowing student scores to be
computed using the same parameter estimates
and scoring rules applied to the general education population. There could be some items
that are deemed critical for the assessments
but which cannot be rendered in Braille. For
those administrations, the item can easily be
dropped or replaced. Because test scores are
computed using MLE, there is no need to produce separate scoring tables for administrations with dropped or replaced items.
7.6.1.2 Retake Assessments

(Scaling, Equating, and Scoring)
Retake assessments are designed to be scored
using pre-equated item-parameter estimates.
As described in Section 7.3.7 Constructing
Test Forms, retake assessments will be constructed following the same test specifications
as census test forms. Should the Department
elect to provide the FSAs as an adaptive
assessment, then retake assessments will be
administered and scored using the same procedures and configurations driving the census
test administrations. The test delivery system
tracks for each student the items previously
administered and will not administer to students previously administered items. Thus,
each test administration, whether census or
retake, will provide an independent assessment of student achievement that meets complex blueprint requirements and targets test
information near each students ability.
Scoring based on pre-equated item parameters
allows for immediate scoring of test results for
tests that do not require hand scoring.
Scoring based on pre-equated parameter estimates supports immediate reporting of test
results for online fixed-form administrations as
well. As discussed previously, there are some
items that are not amenable to embedded field
testing in operational test forms. Because the
Department has traditionally used a postequating approach to scoring test administrations for census administrations, this has not
posed an issue because such items were recalibrated and equated to the reference scale following operational test administration.
However, while it may be the case that itemparameter estimates based on operational test
administrations are preferable, scoring based
on pre-equated item parameters from a welldesigned stand-alone field test, including a
robust linking and sampling designs, results in
accurate and reliable parameter estimates that
provide valid test scores. Modifying test specifications for retake assessments would seem to
pose a greater threat to the validity of test
score interpretations.
As we have described previously, AIR has
developed machine-scored item types that can
support a wide array of constructed responses,
including natural language and essay-scoring
engines. Especially for essay items scored
using statistical algorithms, AIR recommends
that final scores await verification of machine
scoring by human raters or that final scores be
rated by hand with machine scores serving as
a second read. In either event, preliminary
scores can be provided immediately upon
completion of the test administration, with a
note that scores are provisional upon verification by hand scoring.
7.6.2 Scaling, Equating, and

Scoring (Psychometrics)
As we describe throughout this proposal,
although we propose to continue the
Departments post-equating approach to scoring paper-based assessments, AIR recommends
that tests administered online be scored using
pre-equated item-parameter estimates to support immediate reporting of test results to students and educators. The item banks from
which we propose to develop the initial Florida
assessments were designed to support computer-adaptive test administrations, including

items calibrated under operational testing conditions. Should the Department elect to deploy
an adaptive assessment system, the banks are
ready to support an immediate transition. The
Department may also choose to maintain a
fixed-form design for the online assessment
system. In that event, the pre-equated item
parameters could be used to support immediate scoring and reporting of test results in the
first operational test administration, or the
Department could choose to recalibrate the
items following the first operational test
administration to produce a reference scale
unique to Florida, with score reporting in the
first year delayed until after performance standards are identified following the standard-setting workshops. For tests administered online,
we recommend against the continued use of
post-equating following the base-year administration, because the costs of reporting delays
outweigh any minor benefits of post-equating.
Nevertheless, we are prepared to support a
post-equating design should the Department
continue to maintain that tradition.
Even after assessments move online, some students will continue to be administered paper
test forms as an accommodation. For these
assessments, machine-scored items from the
online pool, where possible, will be rendered
as paper items, and students responses will
need to be hand scored. Machine-scored items
that cannot be rendered on paper forms will be
replaced by available constructed-response
item types measuring the same strand, and
responses to these items will also need to be
hand scored. Although the paper-based test
administrations will not be immediately scored
and reported, the sample of students administered the paper-based accommodations will
likely be too restricted to support post-calibration of item parameters. And possible differences in the performance of items across mode
may caution against simply applying bank item
Page 7.6-9
parameters to score paper-accommodated

forms. Therefore, following the first online test
administration, AIR proposes to conduct a
matched samples mode comparability study to
a) identify whether there are significant effects
of mode on item-parameter estimates, and b)
identify the linear transformations necessary
to place the paper item parameters on the
online scale. Should any identified mode
effects be negligible (i.e., within measurement
error), the Department may choose to simply
score responses to paper-accommodated forms
using online bank parameters. In the event of
substantial mode differences, however, the
Department may choose to apply the modebased linking constant to the online item
parameters to create a set of mode-adjusted
item-parameter estimates for scoring responses to paper-accommodated test forms.
AIR will provide to the Department a set of
specifications covering test construction, item
analysis, and test scoring. Descriptions of the
IRT models used to calibrate test items, equating procedures used to link items back to the
reference scale, and scoring computations will
be provided as part of the analysis specifications document that will guide all calibration,
equating, and scoring analyses. AIR will also
generate and provide to the Department and
its quality assurance (QA) contractor(s) simulated data files for use in conducting dry runs
for all calibration, equating, and scoring activities. The simulated data files will be provided
in the same format that operational test data
will be provided to the Department.
The Department intends that all calibration,
equating, and scoring, including dry run and
operational analyses, will be independently
replicated by the Department and its QA contractor(s). AIR works closely with multiple QA
contractors across our state assessment projects to provide the highest quality psychometric services to our clients. In some cases, the
QA contractor is considered an extension of
Page 7.6-10
the client and all deliverables are provided

simultaneously to the state department of education and the QA contractor. Other clients
prefer provide the QA contractor with materials directly. We are prepared to adopt any
strategy the Department wishes to ensure that
all psychometric activities are thoroughly vetted and conducted without error.
Immediately upon completion of post-equating
activities, AIR will produce a technical report
documenting all post-equating procedures and
analysis results for review by the Department
and its Technical Advisory Committee (TAC).
The equating technical report describes the
post-equating sample and the procedures for
analyzing student responses from this sample.
It also presents results comparing classical
item analysis statistics between field-test and
operational-test administrations, operationalitem calibrations and linked item-parameter
estimates, application of cut scores, rounding
rules, and scaled-score transformations to produce student test scores. In addition, test and
subscale-level statistics are presented. The
equating technical report will includes the following sections:
Equating sample, including description of
any exclusion rules
Classical item analysis, documenting statistical procedures and comparing field-test
and operational-test item statistics
Item calibration and equating, describing
procedures for calibrating operational-test
items and placing item parameters on the
base-year reference scale
Documentation of cut scores, rounding
rules, and linear transformations necessary
to produce scale scores from linked operational item parameters
Subscale-score analyses, providing classical
test statistics for content standard scores,
intercorrelations among content standards,

and reporting rules for content standard
performance levels;
Performance outcomes, including test score
means and the percentage of students at
each performance level for each subject
area, for the total population and disaggregated by gender, ethnicity, and district
type;
7.6.2.1 Scaling and Equating

Team
AIR proposes a psychometric team to support
the development and implementation of the
Florida assessments that has extensive experience in statewide assessments with requirements almost identical to those called for in
the INT, including administration of dual mode
assessments, calibration 3PL and 2PL/GPC IRT
models, TCC equating using Stocking-Lord,
and MLE scoring of student tests.
Dr. Ahadi leads AIRs operational psychometricians in performing analyses, writing technical
reports, and conducting validity studies for the
Ohio and Minnesota statewide assessments.
Dr. Ayers has supported Minnesota on the
MCA and Modified assessments for the past 2
years. Drs. Ahadi, Ayers, Stephens, and other
AIR psychometricians use a range of IRT models across assessment projects, including
Rasch/Masters Partial Credit models and the
more general 3PL and 2PL/GPC models. For
the more general IRT models employed, AIR
psychometricians use Multilog and IRTPRO as
the official platform, but AIR psychometricians
also replicate analyses on our own iAM platform.
7.6.2.2 Selection of Calibration

Samples
For assessments that have not yet transitioned
to online, we expect that the Department will
wish to maintain their post-equating approach
to scoring. Although there are many important

advantages to adaptive testing, including not
only increased measurement precision for low
and high ability students but also far greater
coverage of the content domain at aggregate
levels of analysis to support evaluation of
classroom, school, and district accountability,
we recognize that the Department may nevertheless wish to maintain a fixed-form assessment system. Moreover, while Florida may ultimately choose to score tests using pre-equated
item parameters to support immediate reporting of test results, the preference may be to
establish a Florida reference scale in the base
year. Thus, selection of calibration samples
each year remains a critical assessment tasks.
At the beginning of each school year, AIR will
provide the Department with an early return
sampling plan that details the procedures used
to draw the equating samples and documents
the characteristics of the drawn samples.
Although we will draw early return samples for
both paper and online assessments, we propose
to use all available responses to machine-scored
items for equating, since these will be available
immediately upon close of the test window.
Differences between the approach for paper and
online assessments is described below.
Paper assessments. For subjects and grades in
which paper is the primary test administration
mode, post-equating analyses will proceed
along a path consistent with the Departments
past practices. At the beginning of each school
year, AIR will draw a stratified random sample
of schools designated for early processing of
student answer documents. Typically, we
employ color-coded shipping labels to differentiate early return schools, and return shipments from these schools are directed for
immediate processing by DRC. Because the
equating sample drawn by AIR is not a sample
of convenience, accurate and reliable item
parameters can be obtained with representative samples of approximately 10% of the
Page 7.6-11
schools in the state. However, we note that the

Department prefers that approximately 65% of
state school be designated for the equating
sample.
Online assessments. For assessments that are
administered online, there are a variety of
approaches available to the Department,
including tests comprising
machine-scored items only (except for
Writing);
machine-scored items with hand-score verification of target constructed-response
items for a sample of responses;
some constructed-response items that are
hand scored only (Smarter Item Bank).
Each option has implications for the scoring
and reporting of student test scores. Should
the Department elect to administer only
machine-scored items, item and test scores
would be available for analysis immediately
upon close of the test window. Even for those
items requiring rubric validation, samples of
student responses for the rubric validation
process can be pulled prior to the close of the
assessment window, allowing time for rubric
validation committees to meet, revise rubrics
as necessary, and allow for rescore of test
items following the close of the test administration window. Thus, calibration and equating
of all machine-scored items, including selected- and constructed-response, will be based on
all online test administrations.
The Department may also wish to verify the
performance of machine-scored items by hand
scoring a 10% sample of student responses to
targeted constructed-response items. Ideally,
this approach would be implemented as part of
field testing, so that items with unacceptably
low rates of agreement between machine and
human scores could either be dropped or have
the machine-scored rubrics revised. Because
we anticipate that the Department is likely to
Page 7.6-12
recalibrate items banks following the first

operational test administration, this approach
could be used to verify machine-scoring
rubrics for a sample of constructed-response
items prior to the calibration and equating
activities in the first year and then carried forward on field-test items in subsequent years,
allowing for immediate scoring and reporting
of test results should the Department adopt
that approach.
Finally, the Department may elect to administer items that must be hand scored. In this
case, the Department could choose to withhold
any score reporting until after hand-scoring
activities are completed. In the case of delayed
reporting, the Department would likely also
likely maintain a post-equating approach to
scoring student test records. We do note that
immediate reporting of preliminary results
could still be supported, because even though
the test record is not complete, ability estimates would nevertheless be quite stable.
However, because test scores would likely
change, even if only by a small amount, issuing preliminary scores under this approach
might be less desirable.
The early return equating sample for items
administered online will be used only to expedite the processing of a representative sample
of hand-scored items for equating. Because
tests are administered online, with images captured and routed for scoring online, AIR can
draw a simple random sample of students
statewide for early processing. Because simple
random samples are most efficient, immune to
the effects of clustering due to similarities
among students in classrooms, schools, and
districts, the equating samples for hand-scored
items administered online can be much smaller than equating samples based on the selection of schools.
7.6.3 Technical Oversight and

Special Studies
In the following sections, we discuss technical
oversight and assistance AIR will provide for
all aspects of the assessment program
described in the ITN and for the transition to
the Florida Standards Assessment (FSA).
7.6.3.1 Contractors Role

AIR is proposing highly qualified psychometric
staff. The Department will work with the following lead psychometric staff:
Dr. Gary Phillips leads the Psychometrics
and Statistics team, which includes 14 lead
psychometricians with Ph.D. degrees and
more than a dozen statistical programmers,
research associates, and other supporting
staff. Dr. Phillips brings 26 years of experience, including running the National
Assessment of Educational Progress (NAEP)
for the federal government and serving as
acting commissioner of the National Center
for Education Statistics (NCES). Dr. Phillips
holds a Ph.D. from the University of
Kentucky, with an emphasis in statistics
and psychometrics, and has published or
presented more than 200 papers, taught
dozens of advanced graduate-level statistics
courses, and presented hundreds of workshops on advanced statistical and psychometric topics.
Dr. Harold Doran, our proposed lead, will
apply all of the Department-approved technical procedures, will have final approval
over all analyses before submission to the
Department, and also will lead or oversee
the required studies. In addition to supporting the psychometrics of large-scale
assessment programs in multiple states, Dr.
Doran is an expert in value-added models,
growth models, psychometrics, and state
assessment policy. Dr. Doran has directed
psychometrics in Oregon and Delaware and

helped states complete studies needed for a
successful assessment peer review with the
federal government, including work for the
District of Columbia, Delaware, Hawaii,
Indiana, Michigan, Oregon, and Wisconsin.
Dr. Doran was appointed by then-Secretary
of Education Margaret Spellings to serve on
the U.S. Department of Educations
National Technical Advisory Committee.
Dr. Elizabeth Ayers currently runs operational psychometric work for the Minnesota
testing program, where she performs operational and field-test calibrations, conducts
post-equating, and prepares materials for
data review and standard setting. She is
also responsible for updating analysis and
scoring specification files as well as technical reports and summaries. Prior to joining
AIR in early 2012, Dr. Ayers was a postdoctoral researcher at the University of
CaliforniaBerkeley and received her Ph.D.
in statistics from Carnegie Mellon
University.
Drs. Phillips, Doran, and Ayers will be supported by other Ph.D. psychometrians, statistical
programmers, and research staff from both
AIR and DRC. Complete information on our
staffing plan is presented in Section 7.7.2.
7.6.3.2 Technical Report

Technical manuals provide an important window into the inner workings of the testing program. They typically provide transparency that
can assist states in their peer review with the
U.S. Department of Education (DOE) and allow
for users of the test scores to better understand the purpose and methods used for test
construction. AIR routinely helps our state
partners construct technical reports that provide the level of detail needed to respond to
federal or other policy-related requests. Our
typical approach in developing technical manuals is to ensure that all critical elements
Page 7.6-13
found in the DOE Standards and Assessment

peer review document are clearly and completely described in the reports.
2. Summary of simulation studies

a. Bias
b.Match to blueprint
The Florida ITN outlines the volumes that

the vendor must produce; therefore, AIR will
provide technical reports that conform to
this approved outline established by the
Department. Most of the sections are static
and do not change over time, but a few of the
sections present data on the tests administered
for the year, and the statistics will vary with
the compositionof the item pool and the proficiency of the students tested. The variable sections will be updated each year.
The outline provided by Florida is very comprehensive and covers virtually all details of
the testing program. One area in which AIR
would propose to extend the outline found in
the ITN is if the testing program were to use
an adaptive algorithm for any of its online
tests. In such a situation, we would propose
the following to augment the existing outline:
1. Description of the adaptive algorithm
a. Objectives of the item selection algorithm
c. Projected item exposure rates

d.Summary of standard errors
Other than these augmented sections, AIR will
provide technical reports conforming to the
structure of the outline as described in the ITN
as follows:
A. Draft Technical Report Table of Contents
B. Background of Assessments
C. Item Development & Test Construction
Item Development
Item Review
Field Testing
Item Statistics
Classical Statistics
Item Response Theory Statistics:
Difficulty, Discrimination, and Guessing
b.Start/stop rules
Differential Item Functioning (DIF)

Statistics
c. Delivery of field-test items
Model Fit
d.Implementation of any test constraints

AIRs typical practice when implementing an
adaptive test is to use our simulator tool,
which delivers tests to simulated students in
exactly the same way as the operational test to
evaluate the behavior of the algorithm given
the state item pool. This tool can project any
possible issues and allow our psychometricians
time before administering the test to resolve
those issues. This section of the report will
fully describe the results of the simulation performed before the test is operational.
Page 7.6-14
Local Independence
Dimensionality (factor analyses, etc.)
Test Construction
D. Administration
Eligibility
Administration Procedures
Accommodations
Test Security
Data Forensics Program
E. Reports
Linking Assessments During Transitioning

Years
Appropriate Uses for Scores and Reports

Reports Provided
Description of Scores
Linking Assessments to Other Scales for

Performance Comparison
J. Reliability
Appropriate Score Uses

Cautions for Score Use
Estimating Reliability
Standard Error of Measurement
F. Performance Standards
Setting Performance Standards
Interim Performance Standards Florida
Standards Assessments, ITN 2014-47 82
G. Sampling
Student Classification Accuracy and

Consistency
Reliability for sub groups in the population
Reliability of year-to-year changes in school
means
K. Validity
Rationale
Procedure
Perspectives on Test Validity
Representativeness
Criterion Validity
Content and Curricular Validity
H. Scaling
Construct Validity
Rationale
Consequential Validity
Measurement Models
L. Constructed-Response Scoring
Scale Scores
Scale Drift
Scoring Process
Vertical Scaling
Quality Control
Checking IRT Assumptions (Model Fit,

Local Dependence, Dimensionality, etc.)
Handscoring Reliability and Validity
I. Equating
Rater Effects
M. Quality Control Procedures
Rationale
Quality Control for Test Construction
Pre-Equating
Post-Equating
Quality Control for Non-Scannable

Documents
Development Procedure for Future Forms
Quality Control in Data Preparation
Sources of Error Variances to Score

Instability
Quality Control in Production Control

Quality Control in Scanning
Page 7.6-15
Quality Control in Editing and Data Input

Quality Control for Test Form Equating
Quality Control in Scoring and Reporting
Florida Standards Assessments, ITN 201447 83
N. Summary Statistics
Item level statistics
Test level statistics
Review and comparison of raw scores,
sub-scores, scale scores, distributions for
total and sub-populations across years to
monitor the trends across years and subpopulations
Review and comparison of percent proficiency rates across years on different
populations
Frequency distributions of student achievement (all students and by sub-group)
across years by grades and subject
Scale drift indicators
Correlations of student performance across
the various subject areas tested
Item Analysis
This section of the ITN asks for the contractor
to provide item level statistics for field-test
items following the yearly test administration.
Here we list the analyses that AIR will provide
to the Department each year.
Item Analysis
Classical item analyses ensure that the items
function as intended with respect to the
underlying scales. AIRs analysis program computes the required item and test statistics for
each multiple-choice and constructed-response
item to check the integrity of the item and to
verify the appropriateness of the difficulty
Page 7.6-16
level of the item. Key statistics that we compute and examine include the following:
Item Discrimination. The item discrimination index indicates the extent to which
each item differentiates between those
examinees who possess the skills being
measured and those who do not. In general,
the higher the value, the better the item is
able to differentiate between high- and lowachieving students. The discrimination
index for multiple-choice items is calculated as the correlation between the item
score and the students IRT-based ability
estimate. For polytomous items, we compute the mean total number correct for students scoring within each of the possible
score categories. Items are flagged for subsequent reviews if the biserial correlation
for the keyed (correct) response is less
than .05.
Distractor Analyses. Distractor analysis for
the multiple-choice items is used to identify items that may have marginal distractors
or ambiguous correct responses. In the distractor analysis, the correct response
should be the most frequently selected
option among high-scoring examinees. The
discrimination value of the correct
response should be substantial and positive, and the discrimination values for distractors should be lower and, generally,
negative. The biserial correlation for distractors is the correlation between the item
score, treating the target distractor as the
correct response, and the students IRT
ability estimate, restricting the analysis to
those students selecting either the target
distractor or the keyed response. Items are
flagged for subsequent reviews if the biserial correlation for the distractor response is
greater than .05.
Item Difficulty. Items that are either
extremely difficult or extremely easy are
flagged for review but are not necessarily
deleted if they align with the test specifications. For multiple-choice items, we compute the proportion of examinees in the
sample selecting the correct answer (pvalues), as well as those selecting the incorrect responses. For constructed-response
items, item difficulty will be calculated
both as the items mean score and as the
average proportion correct (analogous to
p-value and indicating the ratio of items
mean score divided by the number of
points possible). Items are flagged for
reviews if the p-value is less than .25 or
greater than .95.
Analysis of Differential Item

Functioning
AIR conducts DIF analysis on all field-tested
items to detect potential item bias across
major ethnic and gender groups. The ITN indicates that the following DIF groups will be
included in this analysis:
Caucasian
African-American
Hispanic
Gender
AIR will collaborate with the Department to
identify any additional groups that may be
considered for this analysis, such as students
with disabilities.
Differential item functioning refers to items
that appear to function differently across identifiable groups, typically across different demographic groups. Identifying DIF is important
because sometimes it is a clue that an item
contains a cultural or other bias. Not all items
that exhibit DIF are biased; characteristics of
the educational system may also lead to DIF.
For example, if schools in low-income areas
are less likely to offer geometry classes, students at those schools might perform more
poorly on geometry items than would be

expected, given their proficiency on other
types of items. In this example, it is not the
item that exhibits bias but the curriculum.
However, DIF can indicate bias, so all fieldtested items are evaluated for DIF, and all
items exhibiting DIF are flagged for further
examination by a Fairness and Sensitivity
Committee. Committee members are asked to
reexamine each flagged item, using the statistics as a guide, and to make a final decision
about whether the item should be excluded
from the pool of potential items given its performance in field-testing.
AIR typically uses a generalized MantelHaenszel (MH) procedure to evaluate DIF. The
generalizations include (1) adaptation to polytomous items and (2) improved variance estimators to render the test statistics valid under
complex sample designs. IRT ability estimates
for each student on the test are used as the
ability-matching variable. That score is divided
into five intervals to compute the MH chisquare DIF statistics for balancing the stability
and sensitivity of the DIF scoring category
selection. The analysis program computes the
MH chi-square value, the log-odds ratio, the
standard error of the log-odds ratio, and the
MH-delta for the multiple-choice items, the MH
chi-square, the standardized mean difference
(SMD), and the standard error of the SMD for
the constructed-response items. The purification method described by Holland and Thayer
(1988) is included in the DIF procedure.
Items are classified into three categories (A, B,
or C), ranging from no evidence of DIF to
severe DIF according to the DIF classification
convention illustrated in Exhibit 7.6.3.2-1.
Items are also categorized as positive DIF (i.e.,
+A, +B, or +C), signifying that the item favors
the focal group (e.g., African American/Black,
Hispanic, or female), or negative DIF (i.e., A,
B, or C), signifying that the item favors the
reference group (e.g., white or male). Items are
Page 7.6-17
Exhibit 7.6.3.2-1: DIF Classification Rules for Items

Dichotomous Items
Category
C
B
A
Rule
MH 2 is significant and | | 1
MH
.5
2
MH is significant and | |< 1
MH
MH
.5
is not significant.
Polytomous Items
Category
C
B
A
Rule
MH 2 is significant and | SMD | / | SD | .25.

MH 2 is significant and | SMD | / | SD |< .25.
MH 2 is not significant.
flagged if their DIF statistics fall into the C

category for any group. A DIF classification of
C indicates that the item shows significant
DIF and should be reviewed for potential content bias, differential validity, or other issues
that may reduce item fairness. These items are
flagged regardless of whether the DIF statistic
favors the focal or reference group. A Fairness
and Sensitivity Committee reviews all items
flagged on the basis of DIF statistics.
Committee members are encouraged to discuss these items and are asked to decide
whether each item should be excluded from
the pool of potential items given its performance in field-testing.
7.6.3.3 Transitioning from FCAT

2.0 and EOC Assessments to
(FSA)Linking the Scales
We understand that establishing the connection between the existing scales of measurement supporting the FCAT 2.0 and the new
scales of measurement established for the new
FSA assessment system is a key requirement
specified in this RFP. Florida has the highest
expectations for the technical analyses
required for the development and implementaPage 7.6-18
tion of the FSA. Moreover, we understand and

accept the Departments demand that the technical quality of the FSA meet the Standards
for Educational and Psychological Testing as
well as best practices for high-stakes educational testing.
Given that the link between the scales of measurement defined for the FCAT 2.0 and the FSA
must support student and school accountability
requirements throughout the transition of the
testing programs, the research design and associated methodology must be strong enough to
produce a link between each assessment system across all units of measurement.
To connect the scales of measurement developed
for the FSA and the existing scales of measurement supporting the FCAT, we will implement a
common-item, non-equivalent groups design
wherein different cohorts of students take a common set of linking items. In this case, linking
items from the FCAT will be included within the
initial FSA administration and the Stocking and
Lord procedure (1983) will be used to estimate
the transformation constants that define the link
between the two assessments.
The Stocking and Lord linking method examines the squared difference between the TCCs
of the common items based on the provisional

item parameter estimates and the established
item estimates for those same items. For example, the TCC for a set of multiple-choice items
from the provisional test can be defined as:
1. x
Pi , a xi , bxi , c xi ,
i
Where: x represents the expected raw

score for a person at ability with item
parameters expressed in the scale x metric,
axi , bxi , cxi are the item parameters for item i
on scale x, and is the IRT ability expressed
on scale x,
, .
The TCC for the same set of items but in the
scale y metric can be defined as:
2. x
Pi , a xi , bxi , c xi ,
i
A linear transformation with constants and

can be used to convert the provisional scale
x estimates into the established scale y metric:
3. byi
bxi
4. a yi
axi
5.
c yi
c xi
The transformation constants and are

derived by minimizing the difference between
x and y . Stocking and Lord proposed

the appropriate criterion function as follows:
2
1 N
6. F
x y
N i
1
where N is the number of examinees. This criterion function F is minimized when
7.
F F
0.

The solution for and is obtained iteratively and the transformation constants are used to
transform the item parameter estimates on the
provisional scale x to the existing scale y. This
places all unique items onto the existing scale y.
We will conduct a rigorous evaluation of the
each equating analysis implemented to develop
the link between the FCAT and the FSA.
The Stocking and Lord procedure minimizes
the mean squared difference between the two
test characteristic curves, one based on estimates from the previous calibration and the
other based on transformed estimates from the
current calibration. The parameters for the
linking items were previously estimated in a
preceding FCAT administration, so we can
check to determine whether the parameter
estimates from the current year administration
varies significantly from the prior obtained
parameter estimates. Outlying items can be
identified by plotting the input and estimated
item parameters along with the line of best fit.
We will employ different methods to evaluate
linking items including scatter plot of the item
parameters, t-test of the discrepancy of the
item parameters, and/or Robust-Z statistics.
While the Stocking and Lord procedure is
probably the most common linking method
used in large-scale educational assessments,
we will evaluate the consistency of the equating results based on Stocking and Lord with
other commonly used methods such as:
Mean/Mean (Loyd & Hoover, 1980),
Mean/Sigma (Marco, 1977), Robust
Mean/Sigma (Linn, Levine, Hastings, &
Page 7.6-19
Wardrop, 1981), and another TCC method

developed by Haebara (1980).
Additional Considerations
Selection of the Linking Items. The selection
of linking items to support equating analysis
within the common-items, non-equivalent
groups design must be done carefully. To
ensure that test scores on the new FSA are
equated to the FCAT, and not simply calibrated, linking items should be representative of
the FCAT blueprints and test forms. Ideally,
items would be selected to represent a proportional length blueprint, with psychometric
characteristics consistent with previous FCAT
administrations. In addition, it is especially
important that students have also had the
opportunity to learn the content that is represented within the linking set when assessment
systems are transitioning content standards
such as the FCAT and FSA. Our test development and psychometric team will work closely
with the Department to define the optimal set
of items that will be used to link the FCAT
and FSA.
Size of the Linking Set. Because representation of content is critical to test score equating, we recommend identifying the linking set
as a previous FCAT operational test form. This
ensures that the equating analyses are always
based on a set of items that meets all blueprint
and psychometric specifications for the FCAT,
and in fact passed through all of the
Departments review procedures for operational test administration.
Each student will only respond to a few of
these items, which will be randomly distributed where field-test items might otherwise go.
This matrix design provides a very strong linkage without changing the test length for individual students.
Page 7.6-20
Should the Department choose to identify a

smaller set of linking items, we will work with
the Department to select a subset of items that
represent a proportional length FCAT assessment. However, given Floridas large student
population, embedding a forms worth of FCAT
linking items should not pose a problem.
7.6.3.4 Implement Achievement

Level Standard Setting Meetings
Plan for Setting Performance
Standards
Performance standards in Florida play a central, high-stakes role in the state accountability system. The performance cuts have at least
the following important roles at the state level,
with some other potential high-stakes uses as
chosen by individual districts:
Students must score at or above performance level 2 in Reading to be promoted to
grade 4.
Students must score at level 3 on the high
school Reading and Mathematics tests in
order to graduate.
The A+ school grades are based on the percentage of students meeting high standards
and the percent of students who make
learning gains.
In Section 7.6.3.6 we describe validity studies
that will evaluate the inferences needed to
support these uses. These validity studies will
yield information that might prove useful as
feedback for standard setters. For example, a
validity study occurring for Student
Assessment of Growth and Excellence (SAGE)
will identify the point that best discriminates
between college freshmen in need of remediation from college freshmen taking standard
courses in good standing. The Department
may consider including data from these and
other validity studies as input to the standardsetting process.

We also recognize the timeline challenge faced
by Florida. In May, Florida will need to report
scores on which very high-stakes decisions will
be based. Grade 3 students may be retained,
and high school students may not pass their
graduation test. This poses a critical challenge.
Florida will require valid standards against
which these judgments can be made, but will
have no time to establish these standards after
testing.
The proposed item banks each have the data
that Florida will need to establish fair standards that are suitable for these purposes, so
Florida will be able to conduct standard setting
prior to using the items to build the first form
of the FSA. Through approximate linkages to
other states, standard setters will have a rough
idea of the percent of Florida students who will
fall into each performance category, though
the prediction will not be as precise as it
would have been had the test been given in
Florida. (We do offer an option for Florida to
participate in a SAGE field test this spring to
get direct Florida-specific data.)
Because the standards play a critical role in
both student and school accountability, it is
important that the standards represent rigorous yet realistic levels of achievement for all
students. We propose that there must be at
least three criteria in place to guide the development of standards well suited to the purpose
the Department intends to use them for:
1. Students must be provided with an opportunity to learn the skills being tested. As
such, the performance standards must
reflect achievement relative to the Next
Generation Sunshine State Standardsthe
state enacted curricula.
2. The standards must be attainable such that
well-deserving students are not inadver-
tently retained because the grade 3 or high

school test performance cuts were set to
inappropriately high levels.
3. The standards must be credible and rigorous. Benchmarking against other state and
international tests, inclusion of data generated by the validity studies, and stakeholder reviews of the performance cuts prior to
state board adoption are well suited to this
purpose.
Timeline for Standard Setting

Though our proposal includes options to use
either the SAGE or Smarter Balanced assessment items, our proposal recognizes that
Florida must set its own performance standards given the unique ways in which the performance cuts are used for accountability. For
this reason, our proposal offers an accelerated
approach for standard setting that will allow
the state to fulfill its reporting and accountability objectives with no delay and no impediments to the opportunity to learn for students.
Our recommendation is to complete the standard setting workshops during summer 2014
with ordered item booklets based on the item
parameters from either the SAGE or Smarter
Balanced field-test data. If this option is chosen as we recommend, the state will be able to
immediately score students following the
spring 2015 operational assessment. This proposed approach will provide the state with the
following:
Immediately determine whether grade 3
students meet the level 2 cut for promotion
or retention and inform students of the
decision.
Allow the state to complete analyses needed for its A+ school grades program and
other analyses necessary to satisfy agreements with the U.S. Department of
Educations accountability workbook.
Page 7.6-21
Allow high school students to immediately

learn whether they have received passing
grades on the end-of-course tests. This is
critically important for the subset of students taking the end-of-course tests later in
their high school careers. Some students in
grade 12 take these tests, though this is
rare, and unless performance cuts are
immediately available as we propose, it
would not be possible to know whether
these students scored at level 3 or above
before graduation as required by the state.
If standard setting is delayed until summer
2015, the high school graduation requirement
and the grade 3 reading guarantee will have to
be put on hold for one year. No externally
established standard would likely prove sufficient to support these uses in Florida. AIR will
be pleased to conduct the standard setting at
either time. Florida may consider setting the
standard early and conducting a confirmation
study after the first administration.
An option AIR can also support, and has
implemented in other states, is to perform a
confirmatory standard setting meeting during
summer 2015. If chosen, AIR could use the
post-equated item response data from the
spring 2015 administration and recreate the
ordered item booklets using the updated item
parameters. A subset of the standard setting
committee could reconvene and verify the cuts
previously established during summer 2014.
This option is not included in our base cost.
Technical Approach
Our technical proposal in this section
describes our proposed use of the bookmark
method of standard setting and provides
details on our plan for implementation. The
preceding three criteria are embedded
throughout our proposed technical approach,
allowing for reviews and input from multiple
Page 7.6-22
audiences with varied levels of experience and

expertise.
AIRs plan for recommending performance
standards ensures that cut scores for gradelevel assessments are vertically articulated, so
that any changes in percentage of students
scoring at or above the proficient level can be
meaningfully tied to maturational or instructional differences between grade levels, rather
than an idiosyncratic difference between recommended performance standards across
grade-level panels. Moreover, AIR recommends
that panelists be provided with national and
international benchmark data as well as other
state benchmark data to provide greater context to the meaning of statewide assessment
scores and to encourage panelists to recommend performance standards that are nationally and internationally competitive and competitive with other states with a similar curricular structure.
Our proposed plan for test development would
directly provide for the other state data linkages required if either Smarter Balanced or the
SAGE test is selected. The specifics of our plan
to provide the benchmark data for other state
and international benchmarks are described in
Sections 7.6.3.6.2 and 7.6.3.6.3.
Bookmark Procedure. To ensure vertically
articulated performance standards across
grade-level assessments, AIR proposes to
employ a modified bookmark procedure
(Mitzel, Lewis, Patz, & Green, 2001) to establish cut scores for the Florida statewide assessments. In the modified bookmark approach,
AIR convenes standard-setting panels by grand
bands (e.g., grades 35, 68, and 911). Panels
begin by recommending performance standards in anchor grades (e.g., 4, 7, and 10),
using the standard bookmark method.
Following recommendation of cut scores for
the anchor grades, panelists convene to vertically articulate performance standards across
the anchor grades. Once recommended cuts

scores for the anchor grade have been established, AIR then interpolates the location of
performance standards in the remaining
grades. With interpolated cut scores in hand,
the judgment task for panelists is modified so
that panelists are now asked to judge whether,
for each interpolated cut score, two-thirds (RP
level) of just-barely-meet-the-standard students would be expected to successfully
respond to the item. If the judgment is yes,
panelists are asked to endorse to explicitly
take into consideration the important goal of
recommending vertically coherent performance standards when making bookmark placements. AIR has successfully used this modified
bookmark method to recommend performance
standards for other state assessments.
Augmenting Ordered-Item Booklets. Although
we would ideally like to recommend performance standards on the bank of items that make
up each statewide assessment, such an
approach would be too cumbersome for standard setters. Recommending performance
standards on a single operational test form,
however, is often too restrictive, requiring panelists to recommend performance standards
based on a highly constrained sample of items
that cannot represent the range of content
standards and benchmarks across the spectrum of student achievement. We therefore
recommend using an augmented ordered-item
booklet (OIB) to recommend performance
standards. We propose to construct an OIB
that is proportional to the blueprint but contains more items than any one operational test
administration. Augmenting the OIB in this
way helps ensure that standard setters are
making judgments on a more representative
range of the content standards assessed.
Augmenting the OIB also helps ensure that
there are sufficient items in the higher and
lower ability ranges to support bookmark
placements for cut scores at the higher and
lower performance levels. For example, when
the OIB is constructed from a single operational test form or administration, it is often
the case that moving the bookmark placement
by even one page can have an outsized impact,
and panelists are reluctant to make bookmark
placements based on content considerations
when shifting the placement of a bookmark by
even one page can result in 5% or more additional students not meeting the standard.
Augmenting the OIB modulates the impact of
bookmark placements in the higher and lower
ability regions, freeing panelists to focus on
content considerations when placing bookmarks.
Workshop Panelists and Stakeholder Review
Committees. While it is critically important to
include a range of stakeholders in the standard-setting process, our experience has
shown that it is essential for panelists to have
direct knowledge of academic standards and
student grade-level performance to participate
meaningfully in the bookmarking process. For
this reason, we propose to restrict panel participation to classroom teachers and curriculum specialists with expertise in the subject
area content for each assessment. Additionally,
these experts are the most familiar with the
enacted curricula and can make qualitative
judgments regarding the degree to which students have had the opportunity to learn the
material.
To ensure that the widest range of stakeholders has meaningful input into the standard-setting process, we propose to conduct a reactor review and moderation session with a
more general group comprising table leaders
from the bookmark workshop and representatives from other important stakeholder groups,
including members from the educational community, such as building and district administrators, local school board members, and faculty from higher education. We also propose
including members from outside the educa-
Page 7.6-23
tional community, including parents and business and community representatives.

The reactor meetings will be structured to
allow for stakeholders to review all cuts but to
have a particular emphasis on the review of
cuts that are directly used in high-stakes
accountability (e.g., the Level 2 cut for grade 3
promotion). Stakeholders will be provided with
the impact data as well as written justifications
describing why the high-stakes cuts are attainable and rigorous representations of what students should know and be able to demonstrate
in the affected grades.
Performance-level descriptors (PLDs) define
the content area knowledge, skills, and
processes that test-takers at a performance
level are expected to possess. The standardsetting panelists will base their judgments
about the location of the achievement levels
by using the PLDs as well as the Florida Next
Generation Sunshine State Standards to guide
them in placing their bookmarks.
AIR will work with the Department to draft
appropriate PLDs. Prior to standard setting,
PLDs that are clear, concrete, and reflect the
Departments expectations for proficiency.
This process usually requires multiple rounds
of review, revision, and collaboration.
Particular emphasis will be placed on ensuring
that the PLDs capture knowledge and skills
that students have had the opportunity to
learn prior to the test.
AIR will then convene panels of Florida educators to review and revise the PLDs. The PLDs
jointly authored by the Department and AIR
will be used as input for these meetings, and
we expect that the panels of educators will
fine-tune them. We will capture all comments
and moderate the meetings to achieve consensus on all revisions. The proposed revisions
will be documented and delivered to the
Page 7.6-24
Department. AIR will work with the

Department to resolve all comments and make
the approved changes.
Standard-Setting Materials and

Procedures
Panelist Recruitment. For grade-level assessments, AIR will convene elementary, middle,
and, for ELA, high school panels. For coursebased assessments that require specific content expertise, AIR will convene a separate
panel for each course-based assessment. As
indicated in Exhibit 7.6.3.4-1, AIR will recruit
approximately 15 panelists for each standardsetting panel to recommend performance standards. For grade-level assessments, this represents approximately five panelists per grade in
each grade-band panel. To recruit panelists, we
will actively seek individuals from the diversity
of backgrounds found in Florida. We will seek
the representation of males and females found
in the grade- and course-specific teacher population of Florida. The same principle will be
applied to the geographical representation of
panelists. In addition, we will strive for proportional representation of racial and ethnic backgrounds as well as special education students
and English language learners.
Exhibit 7.6.3.4-1: Standard-Setting Panels and Panelists
Panels
ELA
Math
End-ofCourse
Panelists
Elementary
(38)
High School
(911)
Elementary
(38)
Algebra I
15
15
Algebra II
15
Geometry
15
15
15
Training. An essential element of a standardsetting workshop is thorough training. Training

at the meetings will involve helping panelists
become familiar with the assessment system
and the standard-setting process. The training
will include a review and discussion of the
assessments, the test specifications, the
Florida Next Generation State Sunshine
Standards and the PLDs for each performance
standard.
Table leaders will have the additional responsibility of ensuring that table activities remain
focused on the task at hand, ensuring that
panelists understand the task and alerting
workshop leaders of any issues encountered by
panelists as they engage in their workshop
tasks. Table leaders are not expected to provide training to panelists. Instead, they serve
as liaisons between the panelists and workshop
leaders to ensure that workshop activities are
implemented correctly, alert workshop leaders
of issues that may arise during the course of
conducting workshop activities, and represent
the table in the stakeholder deliberations. We
will convene a table leader orientation meeting
prior to the standard-setting workshop to
familiarize table leaders with their role and
responsibilities, including suggestions on how
to provide leadership at the tables during the
standard-setting process and how to manage
the secure materials.
Central to their training, panelists will use the
PLDs to develop a representation of students
who just barely meet each of the PLDs. During
this training task, panelists learn that although
PLDs are written to characterize typical members of each achievement level, their bookmark placements will be directed toward characterizing and identifying the most minimally
qualified members of each performance level.
Characterizing students who just barely meet
expectations is not an intuitive judgment, and
panelists are encouraged to spend time discussing with one another the minimum char-
acteristics of student achievement for entry

into each performance level.
Finally, panelists will receive training in the
placement of bookmarks in the OIBs. As part
of this training, panelists learn to place a bookmark between the two items that best delineate two achievement levels (e.g., between
items on which students must demonstrate
mastery to meet the minimum requirements
for the proficient level and those items on
which demonstration of mastery is not necessary). In addition to showing panelists how to
place bookmarks in the OIB as part of the
large-group training, AIR will provide sample
OIBs to panelists so that they can practice
placing bookmarks in the OIB.
Bookmark Placement. Following training in
the bookmark procedure, panelists will receive
anchor grade OIBs. For each item in the OIB,
panelists will be instructed to ask what a student must know and be able to do to answer
each question and what makes each item in
the OIB more difficult than the preceding
item. This review of the OIB allows panelists
to gain new perspectives on the knowledge and
skill requirements of items and to share information regarding their thoughts on the location of the threshold region. During this discussion, the workshop leader will circulate
through the room to monitor progress, assist
panelists who are having trouble with the task,
and answer any questions.
Using the PLDs as a guide, panelists will then
be instructed to place a bookmark between the
pages in the OIB that best delineate each of
the performance levels, beginning with the
proficient level performance standard. Once
panelists feel comfortable about their proficient level cut, they will move on to place
bookmarks for the other performance standards. Bookmark placement is conducted in
two rounds, allowing panelists to make independent judgments while still benefiting from
Page 7.6-25
discussion with their fellow panelists. The

placement of the bookmark is illustrated in
Exhibit 7.6.3.4-2. Each panelist uses the PLDs
and the content standards to decide which
item represents the lower bound of each
achievement level. In the example, a panelist
concludes that students who are just barely at
the proficient level would demonstrate mastery
on the item on page 25, whereas students
below the proficient standard would not.
Therefore, the panelist decides that the proficient level performance standard begins on
page 25. The panelist believes that students
below the proficient performance level will not
be able to demonstrate mastery of items
beyond those on page 24 in the OIB.
2. Impact data (scale scores and percentage at

or above each cut score) from the operational test administration
Each table spends time reviewing and discussing cut score placements, focusing on the
lowest and highest recommended cut scores
and the tables median score. Panelists review
the items between the lowest and highest cut
scores for proficiency at their table, discussing
the PLDs, as well as the newly introduced
impact data. Discussion then expands to the
room level, with each table reviewing the basis
for its own recommendations for the group at
large.
After completing these discussions, panelists
again work through the OIB, placing their cut
scores for all three performance levels.
Feedback and Impact Data. Prior to round

two, AIR psychometricians will evaluate
results from round one and provide feedback
Vertical Articulation and Moderation.
forms for each table and for the room as a
Following the recommendation of performance
whole. The forms for each table contain sumstandards for the initial grades, we propose to
mary statistics showing the median, lowest,
and highest cut scores for that table. The room conduct a moderation session comprising table
form contains summary statistics showing the median,
Exhibit 7.6.3.4-2: Example of Bookmark Placement
lowest, and highest cut
scores for each table. In addiMost
tion, an impact data sheet
Difficult
containing scale scores and
54 Item
53
52
the percentage of students at
51
50
49
48
47
or above the level for each
46
45
44
43
possible cut score are provid42
41
40
39
38
ed to panelists for reference
37
36
35
34
and discussion.
33
32
Round two begins with a discussion of two types of information:
1. Feedback on agreement
data, beginning with
table-level feedback and
discussion and progressing to room-level discussion
Page 7.6-26
Ordered
Item
Booklet
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
Consistent with the PLDs

at and above Proficient
Place Proficient
bookmark on page 25
Easiest
Item
Consistent with the PLDs
below Proficient
leaders from each of the panels. Although all

panelists will be allowed to attend the moderation sessions as observers, participation will be
limited to table leaders to effectively manage
moderation session activities. The task of table
leaders in the cross-grade moderation session
will be to evaluate whether the recommended
performance standards are reasonable across
all grade levels assessed and to make modifications to the location of the standards if necessary.
To recommend performance standards in the
intermediate grades, we modify the traditional
bookmark procedures so that panelists are
instructed to determine whether the PLDs support the placement of a specific bookmark on
the interpolated page. If the PLDs do not support the placement of the bookmark on the
interpolated page, panelists are asked whether
they can identify a bookmark placement near
the interpolated page that would be supported
by the PLDs. Panelists are instructed that their
bookmark placements must be guided by content considerations, regardless of whether they
recommend the interpolated page in the OIB
or another bookmark placement.
Stakeholder Reactor Review and Moderation.
To achieve a coherent set of standards across
the system of statewide assessments that
reflects the input of a wide range of stakeholders, both within and outside the education
community, AIR will convene the workshop
table leaders and stakeholders for the purpose
of evaluating the performance standards recommended by the panel and smoothing the
cut scores to achieve a coherent system of
standards across grades. Stakeholders and
table leaders will have the opportunity to offer
additional recommendations on the proposed
standards to the Department.
Workshop Staffing. Each workshop panel will
be staffed by an AIR senior workshop leader, a
content specialist, and a research assistant. In
addition, AIR psychometricians will be available to record panelists decisions, provide

feedback to the workshop participants, and
draft the technical report. The room and all
workshop materials will be kept secure as
determined by a nondisclosure agreement, and
a research assistant will make sure that no
secure materials are taken from the room.
AIR staff will arrive one day early for the meeting to set up equipment and distribute materials. AIR staff will also be on-site for one day
after the workshop to clean up the site, shred
secure documents, and prepare statistical
results for the standard-setting technical
report.
Evaluation. Panelists will have the opportunity
to provide feedback concerning the procedures
and outcomes of the standard-setting workshop through group discussions and a series of
standardized evaluation forms. Panelists will
be asked to evaluate aspects of the standardsetting process throughout the workshop;
these interim evaluations will inform workshop
facilitators about the readiness of the panelists
to recommend performance standards and
allow workshop leaders to address issues that
may arise, including, for example, the need for
additional time or training.
Prior to placing recommended cuts in each
round and for each performance level, panelists will be asked to complete a readiness
form to indicate their preparedness to place
their OIB bookmarks. This form will also ask
panelists to identify any aspects of their task
that are not clear. If any panelists indicate that
they do not feel prepared, the workshop leader
will provide additional training and opportunity for discussion. Panelists must indicate that
they feel ready to move forward before they
recommend a cut.
At the completion of the workshop, panelists
will be asked to complete an evaluation form
Page 7.6-27
designed to elicit feedback on all aspects of the

workshop, including clarity of training and
tasks, appropriateness of the time spent on
activities, and satisfaction with the outcome of
the workshop. Appendix E presents a sample
of a panelist evaluation form.
Results of these evaluations will be summarized and presented in the standard-setting
technical report.
Technical Report. AIR will provide a comprehensive standard-setting technical report that
will be available for review by the Department
and the Technical Advisory Committee (TAC).
It will detail the procedures, participants, and
other relevant elements. The standard-setting
technical report will be specifically oriented
toward providing the type of comprehensive
details required by NCLB peer review. The
report will contain these sections:
1. Executive Summary, which is written for
policymakers and can be used by the
Department for workshops and provided to
the Florida Board of Education
2. Background and history of Floridas
statewide assessments
3. Preparation for the standard-setting workshop, which covers the development of the
PLDs, psychometric details of the field test,
item calibrations and the creation of the
vertical scale, estimation of impact data,
explanation of the articulation process and
international benchmarking procedures,
and the composition and demographics of
the standard-setting panels
4. Conduct of the standard-setting workshop,
which includes agendas, training procedures, details of the round one and round
two estimates of cut scores, stakeholder
review and moderation sessions, security
procedures, and evaluation
5. Results of the standard-setting workshop,
which include summaries of each round,
Page 7.6-28
reliability of the achievement-level standards, changes in cut scores resulting from

the stakeholder review and moderation sessions, final cut scores, and the percentage
at and above each achievement level.
In addition, AIR will assist the Department
with any materials or any other assistance
needed for the Florida Board of Educations
approval of the standards.
7.6.3.5 Vertical Scaling

Vertical Linking Background
AIRs proposed plan for establishing a vertically linked scale in reading and math is
described in this section. Our proposed
approach for Florida is to adopt the vertically
linked scale constructed for either the SAGE
or Smarter Balanced tests. Both of those scales
were established using a representative sample
of items from adjacent grades using a common
item, non-equivalent groups approach (Kolen
& Brennan, 2004). In both instances, items
were administered to students using AIRs distribution algorithm, which ensures a random,
representative sample of responses from students.
If Florida chooses fixed forms for all tested
grades rather than adaptive and also implements post-equating, then those scores will be
linked back to the scale using the StockingLord procedure (1983) as described in this
section.
AIRs Experience with Large-Scale

Vertical Linking
We understand that the vertical scale required
for Florida will be used to measure growth and
value-added as described in the state ITN.
AIR has experience both in the construction of
vertical scales for large-scale state testing programs as well as in developing growth models
for accountability, particularly using the

Florida developmental scaled scores for the
state value-added model.
Exhibit 7.6.3.5-1 lists the states where AIR has
developed vertical scales for large state testing
programs. Our experiences in each state are
similar in that our process begins with a proposed plan that is reviewed by state Technical
Advisory Committees (TACs) and revised as
needed before implementation. Additionally,
AIR typically implements multiple ways to statistically link the results such that our state
clients can evaluate which approach provides
results that are best suited to their reporting
and accountability situations.
Using Growth Scales for Accountability

AIR is currently responsible for the valueadded implementation in the State of Florida,
which makes use of the current developmental
scaled scores (DSS). We have also implemented growth/value-added models in other states
and have identified certain limitations commonly used by psychometricians that restrict
some students from demonstrating growth.
Exhibit 7.6.3.5-2 illustrates this issue, which is

based on Florida FCAT 2.0 data from 201213.
This bar graph shows the average grade 5 math
gain score for students on the Florida DSS
based on their grade 4 performance level. For
instance, students scoring in performance level
1 in grade 4 have larger gains than students
who started at any other performance level.
Conversely, students scoring in performance
level 5 in grade 4 showed negative gains from
one year to the next.
To some degree, this result is related to mean
reversion. The lowest performing students in
the state in grade 4 are most likely to experience a large, negative draw from the error distribution and the highest performing students
in grade 4 are those who are most likely to
experience a large, positive draw from the
error distribution. On a later retest, they are
unlikely to receive the same large negative or
positive draw, resulting in a score that pulls
these students toward the middle of the score
distribution. The result is a growth pattern
that has limitations for use in accountability,
as the negative gains for the highest perform-
Exhibit 7.6.3.5-1: States Where AIR Has Developed Vertical Scales

State
Delaware
Minnesota
Utah
Ohio
Washington
Wisconsin
Vertical
Scale
Reading and
math
Reading and
math
Reading and
math
Reading and
math
Reading and
Math
Reading and
Math
IRT
Model
Rasch
Concurrent calibration
310
For reporting and accountability
3PL
Stocking-Lord
310
3PL
Stocking-Lord
310
Rasch
Mean, Concurrent
calibration
Mean/concurrent
calibration
Stocking-Lord/Concurrent
calibration
310
38
Research and used for standard setting

verification of articulated standards
Research
38
Research
Rasch
3PL
Linking Method
Grades
Purpose
Page 7.6-29
Exhibit 7.6.3.5-2: Average Grade 5 Math Gain Score in

Florida, 201213
Technical Approach to Developing the

Vertical Scale
We propose to base Floridas vertical scale on
the vertical scale established for the assessment with which the FSA will startSAGE or
Smarter Balanced. Both assessments use our
randomized item-embedding algorithm to distribute vertical scaling items to students.
ing students suggest they declined in their academic progress, not improved.
This pattern is almost certainly not a true
reflection of growth, but a reflection of two
inherent issues that might be somewhat mitigated in scale development. First, if test forms
can be constructed to minimize error variance
at all points along the scale, then mean reversion for extreme scoring students would be
reduced. Computer-adaptive testing is one
option that minimizes error variance for students along the score continuum, and can be
supported by both the SAGE and Smarter
Balanced tests.
Second, a common psychometric practice that
restricts growth is to censor the within-grade
score scalesmeaning that the lower and
upper ends of the score scale are arbitrarily
capped at a value. Psychometricians commonly refer to these as lowest obtainable scaled
scores (LOSS) and highest obtainable scaled
scores (HOSS). When the scales have artificial
caps, it is entirely possible that a students true
score can be above the HOSS, but the censored score scales prevent that score from
being reported. This would restrict a student
from showing growth, especially for high-performing students who can score no higher
than the HOSS.
Page 7.6-30
In this section we describe the ways in which

the items from adjacent grades were selected
to build the vertical scale for SAGE and
Smarter Balanced and then describe how postequated fixed forms can be linked back to the
vertical scale if Florida were to choose that
option.
Selection of Linking Items

The common-item non-equivalent groups
design requires that a set of items are embedded into the pool to establish between-grade
linkages. Though the vertical scale establishes
only an approximation of the construct measured over time (e.g., grade 4 math is not just a
harder version of grade 3 math), the linking
set must represent the test blueprints for each
grade to establish a content-relevant set of
linking items.
From within each grade, a sample of items was
chosen by content experts to adequately and
proportionally reflect the common core test
blueprint. Two types of items were chosen
from each grade to serve in the linking set,
which we refer to as upward-linking and backward-linking items. Upward-linking items are
those drawn from the pool in grade g and are
linked on forms in grade g + 1. Backwardlinked items are those items selected from the
pool in grade g that are then used as linking
items in grade g 1.
Stocking-Lord Procedure
The Department may opt for fixed forms for all
grades and subjects using a set of items drawn
from the available items pools established for
SAGE or Smarter Balanced. In this instance,
post-equating methods using the most recent
Florida data would most likely be implemented, thus requiring the post-equated data to be
linked back to the vertical scale. We propose
to use the Stocking-Lord procedure (1983) to
find linking constants that minimize the
squared distance between two test characteristic curves. The method is applied to find linking constants between each grade pair when
items within a grade are separately calibrated.
The common items in this case would have
parameters from the original vertical scale calibration and the new post-equated estimates.
The approach evaluates the following integral
where the indices i denote an item and a and
b denote separate forms:
2
a

SL
p( ; aia , bia , cia ) p( ; ib , Abib B, cib ) f ( )d ( )
A
i
1
i
1

I
Replication
The ITN calls for the vendor to replicate this
study three years after the initial scale is
developed, which may occur in the contract
renewal period. AIR proposes to replicate the
same methods implemented, as described in
this section of our proposal, and compare the
growth patterns observed between adjacent
grade pairs between the initial and replicated
vertical scales. Any new research or significant
findings in the area of vertical linking that
occurs after the initial scale is developed may
be applied to improve on the initial work completed.
7.6.3.6 Scaling and Equating

Studies
Scaling and Equating Studies
This section of the ITN lists several research
studies that must be conducted to support
inferences regarding scores as the state transitions to a new scale and a new mode of test
administration. Here we provide our detailed
responses to the studies listed specifically in
the ITN, including the following:
Effects of scale drift on score interpretation
Comparability of scores using different
devices
Linking the new Florida assessment to
other assessments measuring the same
standards
A study to link the new Florida assessment
to international standards
We also provide as a separate cost option other
validity studies that may be useful to support
the goals of the state. These studies are not
requested in the ITN, and for this reason, we
propose them as optional studies AIR is prepared to implement for Florida if selected.
These studies are centered on the high-stakes
nature of the test scores and their uses in student-level promotion/graduation and the
teacher accountability system using test scores.
The additional validity studies we propose
focus on evidence needed to support the following uses of the test scores in the Florida
accountability system:
Evidence to support the use of the grade 3
level 2 cut score for determining studentlevel promotion decisions
Evidence to support the use of the level 3
cuts as a graduation requirement for the
end-of-course exams
Page 7.6-31
Evidence to support the use of the new scale

scores to support continuity of the state
value-added model program and gain scores
used for the A+ accountability program
In the remainder of this section, we outline
our responses to the studies listed specifically
in the ITN and the additional, optional studies
proposed by AIR.
Investigating the Effects of Scale Drift in

the Presence of Equating Chains
The state accurately describes in the ITN the
potential for multiple equating chains to conflate the meaning and interpretation of test
scores with other factors that may cause the
shift. The accumulation of random and systematic error from multiple equating functions
over time may alter the interpretation of score
inferences.
In the ideal scenario, changes in score distributions can be attributed to true changes in
the ability of students in the population.
Unfortunately, multiple other factors may be
confounded with shifts in the score distributions, potentially impeding this ideal inference
to be supported. Scale drift occurs when a specific score at time t, i,t has a different meaning and interpretation as i,t' when observed
on a later test form.
Minimally, both equating error and changes in
the trait measured may change over time,
leading to differences in the interpretation of
the scores. Haberman and Dorans (2009) provide a summary list of other sources that may
contribute to the scale drift.
Effects of Equating Error on Scale Drift
Equating error tends to have an insidious and
often ignored effect in the interpretation of
linked score scales. While the Standards for
Educational and Psychological Testing (APA,
Page 7.6-32
1999) indicate that equating error should always

be reported, it is rarely estimated and reported
in operational testing programs. This is generally
true because analytic formulae are typically not
available for psychometricians and therefore not
implemented in software programs.
Moreover, equating error, when reported, is
almost always an underestimate of the true
equating variance, as it ignores an important,
and equally important, source of variation in
the estimated linkage. Cohen, Johnson, and
Angeles (2000) demonstrated that equating
error is composed of variation in two dimensions: the sampling of students and the sampling of common items. However, the latter
source of variance due to the sampling of common items is typically ignored when estimating the variance of an equating function. If a
potentially large source of random error is
ignored, then the estimate of the equating variance will be smaller than it truly is.
Students are, of course, a sample of students
tested in the population, and this source of
variation is well-recognized. However, the
common items used in a common-item linking
design are only a sample from the universe of
possible items that could have been used for
the common-item design. Had another
exchangeable set of common items been used,
a small difference in the estimate of the equating function would have been estimated.
Michaelides and Haertel (2004) demonstrated
that the variance due to the sampling of common items is at least as large as the variation
due to the sampling of students. As a result,
any estimates of the equating variance that
ignores the sampling of common items would
underestimate the true variance of the equating
function and potentially claim that scale drift
exists when the observed changes are expected
given the sampling variance of the linkage.
The implication of this missing source of variance is important with respect to scale drift.
Suppose an equating function yields a linkage
and its variance is computed but underestimates the true sampling variance. If changes
in score distributions occur in future years,
one risk is that the change is no larger than
what the true variance of the equating variance would imply. However, the change might
either be attributed to scale drift or true
changes in the population.
This implies there are two important tools necessary to properly study scale drift with
respect to equating variance. First, all sources
of variation must be considered when estimating the variance of the equating function. AIR
has pioneered work in this area (Cohen et al.,
2000), and our work has led to changes in how
the field of psychometrics views the error in
equating functions.
Second, analysis methods must be articulated
and accessible to psychometricians to estimate
the equating variance. AIR has done extensive
work in this area and our iAM item response
theory (IRT) software provides estimates of
the equating error as a by-product of item calibrations. Outside of the software, AIR has
demonstrated with other state operational testing programs how resampling methods, such
as the jackknife method, can be used to estimate equating variance.
Potential Effects of Changes in the Trait on
Scale Drift
In some cases, shifts in a tested blueprint or
changes in the curricular structure (such as
better alignment to the Next Generation
Sunshine State Standards) may allow for scale
drift if a change to the measured trait is substantial. This is particularly important in that
performance cuts are set on the scale in the
initial years and scale drift could affect the
interpretation of the performance cuts if
changes in the interpretation of scores changes

over time.
If the underlying trait is changing, then one
way this could be identified and accommodated is through a confirmatory standard setting
procedure. Panelists could reconvene at a
future point in time (say the summer of 2018)
and revisit the ordered items booklets originally used for standard setting and the performance level descriptors. Panelists would evaluate if the ability estimate corresponding to
important cut scores on the test have the same
meaning and interpretation with respect to the
performance level descriptors as they did previously. If they detect changes in the meaning
and interpretation of the scale, possible
changes to the cuts might be considered.
A Proposal to Study Scale Drift in Florida
The ITN does not call for a specific study to be
proposed at this time, but does call for the
details of that study to be presented to the
Department and its technical advisory committee (TAC) in the spring of 2018. To support the
state in this area, there are two keys areas of
work for AIR to consider. To begin, it is important for the new contractor to understand linking and equating methods that are likely to
mitigate the effects of scale drift over time
(Harris & Kolen, 1994). AIR is cognizant of
these recommendations, and our test development protocols used for large-scale testing programs build linking sets and use linking methods that are known to minimize the effects of
scale drift.
Additionally, AIR will provide the state with a
complete research plan for its review and distribution to the TAC. That review will include
a complete and comprehensive review of the
literature on scale drift and methods for
assessing its impact. Various methods for
assessing drift have been proposed (Gou, Liu,
Dorans, & Feigenbaum, 2011; von Davier,
Page 7.6-33
2009). A technical report following the implementation of the study will be provided to the
state documenting the methods, results, and
implications.
Optional Validity Studies to Support

Uses of Test Scores in Accountability
Systems
The policies surrounding the use of the performance level cuts and the scaled scores are
high-stakes in nature. For this reason, some
validity evidence should be collected to assess
the degree to which the cuts are appropriately
aligned to support the state policy objectives.
As noted, these are not required in the ITN
and therefore AIR is proposing these as optional validity studies that are not included in our
base cost.
Evidence to Support Use of Grade 3 Level 2
Cut score
The state accountability system requires students in grade 3 to score at level 2 or above in
reading to be promoted to grade 4. Some
exemptions to this rule exist, such as students
demonstrating performance on an approved
alternative standardized test or a student portfolio. However, outside of these exemptions the
rule sets a minimally acceptable level of
achievement that students must demonstrate
to be promoted to grade 4.
Clearly this rule exists to provide students
with the support in reading they need before
moving on to higher grades. Hence, the underlying hypothesis is that students who do not
score at level 2 in reading do not possess the
prerequisite skills needed to be successful in
grade 4 and beyond.
The policy objective aims to use a cut score to
differentiate between students who are likely
to succeed in future grades without further
remediation and support in reading. Therefore,
Page 7.6-34
validity evidence should be available to substantiate two issues:

1. Evidence should be available showing
that students who do not score at level
2 or above would in fact continue to
struggle in grade 4 and beyond.
2. Students that are retained in grade 3 via
this policy benefit from that retention
and demonstrate improved levels of
achievement in grade 4 and beyond.
Because this rule is enacted in the state, it is not
possible to collect evidence from within Florida
that provides evidence related to the first issue.
Data from within the state can be collected to
gather evidence on the second issue.
The state of Utah is considering a similar policy and is interested in partnering with AIR to
complete a study on the validity of cut scores
used in a fashion similar to the Florida policy.
Currently, students in grade 3 in Utah are not
retained via the cuts and evidence from that
state can be collected relative to the first
research question.
If this optional study is desired, AIR can facilitate collaboration in this research task with
both states and complete a study to provide
the necessary validity evidence. If the state
selects this proposed optional study, AIR will
provide a complete research plan for its review
and submission to the TAC. Final results will
be provided in presentations to the state and
its TAC and in a technical report documenting
the methods and results.
Evidence to Support Use of Graduation
Requirements
Another area in which the cuts have a studentlevel accountability component is with the
high school end-of-course tests. The tests to be
passed depend on when students entered high
school, but the general principle is the same in
that a passing score on various assessments

has been established as one component of an
overall graduation requirement.
Here the policy objective is to use a cut score
to differentiate between students who are prepared for career and college readiness and
those who are not. Similar to the previous
research questions, this objective requires
validity evidence regarding the following:
1. Evidence should be available showing
that students who do not achieve a
passing score are not as well prepared
for career or college as those who do
achieve a passing score.
2. Students that do not graduate via this
policy subsequently benefit and demonstrate skills commensurate with students who are career and college ready.
AIR and the state of Utah are currently planning a study, to be conducted in the spring of
2014, to evaluate the empirical relevance of
the Common Core as measured by the SAGE
for college readiness. The study will administer
the SAGE to college freshmen in remedial
courses and college freshmen in good standing
in standard courses. Statistical analysis will
evaluate (a) the relationship between SAGE
scores and the need for remediation and (b)
the SAGE score that best discriminates
between students who do and do not require
remediation. The current plan will include a
full range of community colleges, colleges, and
universities in Utah, but the Utah State Office
of Education is interested in extending this
study to other collaborating states. We would
encourage Florida to participate in this study if
the SAGE assessment is given as a base test.
AIR has had informal talks with Smarter
Balanced Assessment Consortium leadership
encouraging a similar study with their test.
If this optional validity study is chosen, AIR
can facilitate collaboration in this research
task with both states and complete a study

that provides the necessary validity evidence.
AIR will provide a complete research plan for
state review and submission to the TAC. Final
results will be provided in presentations to the
state and its TAC and in a technical report
documenting the methods and results.
Evidence to Support Continued Use of Scaled
Scores in Growth and Value-Added Models
AIR is currently serving as the states valueadded contractor, and during this time, we
have provided the state with studies showing
the effects of transitioning from the FCAT 1.0
to the FCAT 2.0 with respect to the valueadded model. Again, a new transition study
will be necessary when a new vertical scale is
introduced. In Section 7.6.3.3, we provide a
detailed response showing how the new scale
can be linked with the FCAT 2.0 scale.
However, a scale must have properties beyond
this linkage that make it useful for a highstakes teacher accountability system.
There are generally two criteria that must be
in place for a test scale to be appropriate for a
value-added model:
1. The test itself must be highly aligned
with the curriculum enacted by classroom teachers.
2. The test scores must be highly related
to the prior test scores used as the control variables in the model.
The first criterion is resolved through a test
development process that aligns forms with
test blueprints matching the Next Generation
Sunshine State Standards. The second criterion evaluates the statistical relationship of the
scores and the degree to which the underlying
trait measured by both scales is related. To
assess this relationship, AIR proposes to evaluate the correlation between the two traits
measured by the new and old tests using mar-
Page 7.6-35
ginal maximum likelihood (MML). In the MML

equation below, we treat the student ability
parameter as a nuisance parameter, as it is
measured with error, and integrate that out of
the likelihood.
where p(zi,a| 1) is the probability of a correct

response for individual i to the items on test
form a and p(zi,b| 2) is the same, but for test
form b, and
Obtaining item
responses from both test forms would require

a small experimental design where a subset of
students take the new Florida Standards
Assessments and additionally take the old
FCAT test in the same school year.
We maximize the function with respect to the
mean and variance covariance matrix. For the
current analysis, the focal parameter is ,
which represents the true correlation between
the underlying traits measured by the two tests.
If this option is selected, AIR will provide a
complete study design for the state and its TAC
to review. Results and details of the study will
be provided in a complete technical report.
7.6.3.6.1 Investigate the

Comparability between Devices
The comparability of scores resulting from different modes of administration has been a central question since the introduction of computers in assessment (Huff & Sireci, 2001). Much
of this research has concluded that paper-andpencil and online assessments can be constructed so that resulting scores are comparable (Paek, 1995; Mead & Drasgow 1993).
However, mode effects have been found when
items require scrolling, such as is typically the
case with reading tests (Bergstrom, 1992; Way,
Page 7.6-36
Davis, & Fitzpatrick, 2006), in situations

where test timing impacted performance (Ito
& Sykes, 2004), for mathematics items requiring graphing and geometric manipulations
(Keng, McClarty, & Davis, 2006), and for the
lower-grade students (Choi & Tinkler, 2002).
Drawing a single conclusion regarding score
comparability is difficult because the technology that underlies test administration is rapidly
evolving. Score comparability research quickly
becomes outdated as test administration
engines are refined and deployed on a variety
of platforms (e.g., desktop, laptop, tablet) using
different operating systems (e.g., iOS,
Windows, Linux) using different devices (e.g.,
iPad, Microsoft Surface, Samsung Galaxy with
Android) of varying shapes and sizes.
Differences in how test administration engines
are deployed can potentially impact the comparability of student scores.
Furthermore, students have much greater
experience in working with different devices
and platforms using different tools at earlier
and earlier ages. The presentation of all
aspects of the test administration (e.g., items,
tools, input) must be natural for a given
device. Changing devices students are accustomed to working with, or changing particular
aspects or features of a given device for test
administration, can further introduce confounding variance. Therefore, ongoing evaluation of score comparability is recommended
when testing programs offer test administration via multiple platforms/devices.
We present two options to address the comparability between devices delivering the assessments. The first option is a randomized trial in
which students are assigned a device. While in
most cases randomization provides assurance
that biasing factors do not affect results, in this
case, randomization may introduce biases. For
example, students who are accustomed to using
a particular device may be distracted by the
challenge of adapting to the new device. The

second option, a matched sample of naturally
occurring device users, suffers from self-selection into the device category but does not introduce the added distraction of switching devices.
In either case, we are proposing to investigate
the comparability of scores administered via
three major categories of devices: desktop, laptop, and tablet. Should the Department be
interested in specific comparisons of particular
operating systems (e.g., iOS versus Windows)
or different devices (e.g., iPad versus Android),
we can readily modify our plan to reflect this
focus. Regardless of the specific platforms
and/or devices ultimately used within the comparability research, our research will be
designed to address the following questions:
To what degree are item-level and test level
scores obtained from various devices comparable?
To what degree can score comparability be
obtained through scaling items onto a single metric, linking, or concordance?
What interactions exist among device and
item/task characteristics and among grade,
subject, and student characteristics?
The evaluation of score comparability for different devices will be documented in a technical report and presented to the Department
and the TAC regardless of which option is ultimately selected.
Option 1 (Not Included in Base Cost):

An Experimental Design Evaluation of
Score Comparability
For the optional score comparability study, we
will select one test form for each grade and
subject that will be administered over different
devices. In order to keep test administration
time for students participating in the study to a
manageable level, the test will be divided into
two parts to form approximately equivalent
subtests in terms of content coverage and item

types. Both subtests will be administered via
different devices and we will counterbalance
the order of administration of each subtest and
device to control for individual differences and
potential item position effects. An illustration
of the design is presented in Exhibit 7.6.3.6.1-1
for evaluating score comparability of assessments delivered via iPad versus Android.
Exhibit 7.6.3.6.1-1: Illustration of Comparability Study
Design for iPad/Android for Single Grade/Content Area
Group
1
2
3
4
Device 1 Subtest Device 2 Subtest

iPad
A
Android
B
iPad
B
Android
A
Android
A
iPad
B
Android
B
iPad
A
For schools that agree to participate in the comparability study, we will randomly assign students in classrooms to one of four test administration conditions. Each subtest of items will be
presented via the iPad to half of the students
participating and via Android to half of the students. Each student will take two subtests, one
via iPad and one via an Android tablet.
Sampling Plan
As described in Section 7.3.5, Embedded Field
Testing, it is essential to address the clustered
nature of the educational assessment data
required for the comparability study.
Depending on the size of design effect, the
impact of the clustering dictates that to obtain
an effective sample size of 1,000, a sample of
approximately 3,000 students may in fact be
required to estimate item parameters and
accurately estimate their associated standard
error for each group. Thus, to address score
comparability for a given grade/subject area for
Exhibit 7.6.3.6.1-1 above, 12,000 students
would need to be sampled for a given
device/platform comparison.
Page 7.6-37
Analyses
Our analysis will be focused on whether the
use of a particular device leads to specific
advantages or disadvantages for students taking the EOC assessments. Ability estimates
will be estimated for each device and summary
statistics will be compared for all students as
well as relevant subgroups using analysis of
variance methods. Comparisons of item
parameters and associated scoring conversion
tables will also be made after calibrating items
within and across conditions. In addition, differential item functioning (DIF) analyses will
be used to examine item-level differences
across devices. Our experience with such
analyses is that test scores are comparable
across different devices.
Option 2 (Included in Base Cost):

Evaluating Score Comparability using
Matched Samples
With classroom testing time at a premium and
the associated impact on administration and
score reporting schedules, we acknowledge
that experimental design work as described in
Option 1 is often impossible to implement in
K12 testing programs. As an alternative, we
are proposing to address score comparability
for the EOC assessments using matched samples that are specifically constructed to form
equivalent groups. While a variety of approaches can be used to construct matched samples
(Karkee, Kim, & Fatica, 2010; Moses, Deng, &
Zhang, 2011; Way, Lin, & Kong, 2008), key
demographic variables such as gender, ethnicity, and prior achievement are typically used as
matching criteria. Once the matched samples
have been developed, they are treated as
equivalent groups that can be used to evaluate
mode comparability. That is, the matched
sample reduces the bias between the groups
when randomization is difficult or impossible.
Page 7.6-38
Propensity score matching (DAgostino, 1998;

Rosenbaum, 1995; Rosenbaum and Rubin,
1983; Rubin 2006) will be implemented to
construct equivalent groups of students who
took the EOC assessments using different
devices/platforms. Propensity score matching
uses covariates to predict group membership
(e.g., device administered) using logistic
regression. Each student who took the test on
a given platform is matched with a student
who the test on another device or platform
based on their propensity scores.
Demographic information such as (gender, ethnicity, income level status, ELL status, IEP,
and prior performance) will be used in developing the propensity scores for the device
comparability analyses. Propensity scores,
expressing the likelihood that each student
was a member of each group taking different
devices, are then used to select the equivalent
groups of students.
Analyses
Once equivalent groups are constructed, our
analysis will be focused on whether the use of
a particular device leads to specific advantages
or disadvantages for students taking the EOC
assessments. IRT calibrations will be performed to evaluate the differences between the
groups of students taking the test using different devices. Ability estimates will be estimated
for each device and summary statistics will be
compared for all students as well as relevant
subgroups. Score comparability over different
devices will be compared using analysis of
variance methods. Comparisons of item
parameters and associated score conversion
tables will also be made after calibrating items
within and across conditions. In addition, DIF
analyses will be used to examine differences
across devices at the item level.
7.6.3.6.2 Linking Florida Standards

Assessments to Other Assessments
Measuring the Same Standards
The Department would like to link the Florida
Standards Assessment (FSA) scale to the
scales in other tests in states that measure the
same content standards. The first step in this
linking process is to conduct a comprehensive
curriculum-matching study to determine content overlap between Florida content standards and content standards in other states.
Curriculum Matching Study

AIR content experts will obtain the English
Language Arts/ Literacy (ELA/L) standards and
Mathematic standards for grades 312 from
other states that have not adopted the Common
Core State Standards (CCSS). We will also survey the states that have adopted the CCSS to
see if they are using a subset of the standards or
have modified the standards. AIR content
experts will map the Florida standards to the
standards of these other states, indicating which
standards for each grade level and content area
are the same and which are different. Part of
the analysis will investigate which states had a
focus for each grade level and content area that
is the same or different. For example, number
and operations may still be the focus for grade 3
even though the exact wording of the standards
may differ. AIR content experts will prepare a
comprehensive report that lists all these findings and present it to the Department. AIR content experts will prepare a list of suggested
states for linking based on these findings and
present them to the Department. We expect this
curriculum-matching study will be completed
by summer 2015.
Linking Plan
Following the curriculum-matching study, AIR
will develop a linking study plan that would be
reviewed by the Technical Advisory
Committee (TAC) and approved by the

Department. AIR will negotiate with target
states to share items that can be used for linking. Such a linkage will benefit both states. If
the states do not wish to share items, then an
anchor test linking design would be used to
put FSA scores on the scale of the target state
with similar content. A two-stage statistical
moderation procedure would be used to link
the FSA to the National Assessment of
Educational Progress (NAEP) and then link
NAEP to the target state. In this case, NAEP is
used as the anchor. This procedure would provide linking data for grades 4 and 8 for
Reading and Mathematics.
In the case of cooperating states, we propose to
select a sample of the states items that matches
the target states blueprint. During the 2014
administration, we will randomly embed these
items in field-test slots. This approach maximizes the information available for the calibration of the target state items onto the FSA scale
through concurrent calibration. The items
would then be linked back to the target states
scale using a Stocking-Lord procedure. We will
estimate the linking error for these linkages, as
described in Section 7.6. This will provide the
Department with (a) an estimate of the correspondence between the FSA scale and each target states scale; and (b) an accurate margin of
error around that correspondence.
One of the options offered in the AIR proposal
is the use of the Smarter Balanced test for the
FSA. If we do not use the Smarter Balanced
test, we can still use Smarter Balanced items
and license it for a sufficient number of students to conduct a common item linking study
between the FSA and Smarter Balanced. We
will include a draft of the Smarter Balanced
policy in Appendix N. To accomplish this,
items would be obtained from Smarter
Balanced and used in a linking study embedded in the FSA. Smarter Balanced is required
to provide this information based on their
Page 7.6-39
cooperative agreements with the U.S.

Department of Education.* This would provide
the Department with linking data for all subjects and grades that overlap between the FSA
and Smarter Balanced.
Statistically linking to the Partnership for
Assessment of Readiness for College and
Careers (PARCC) would not be possible until
after its post-equating in the spring 2015
because PARCC item parameters would not be
available until then.
We will deliver an initial linking study technical report for TAC review and deliver the final,
revised report by July 2015. The technical
report will include concordance tables showing
how the FSA is aligned with other state assessments using the same content standards. If
PARCC items become available, we will update
the technical report when the PARCC item
parameters become available.
7.6.3.6.3 Linking Florida Standards

Assessments
International Benchmarking
Florida would like to know that their performance standards are not only college and career
ready, but that they are also internationally
competitive. To accomplish this, AIR would
develop a plan for international linking using
data from the Trends in International
Mathematics and Science Study (TIMSS), the
Progress in International Reading Literacy
Study (PIRLS), or the Program for
International Student Assessment (PISA). The

plan would be reviewed by the TAC and
approved by the Department.
The international linking plan would include a
review of available international data for linking which depends on the testing schedule of
the international studies.
a. PISA is on a 3-year cycle with the last
administration in 2012, with a release in
December 2013, and the next assessment
in 2015, with a release in December 2016.
b. TIMSS is on a 4-year cycle with the last
c. PIRLS is on a 5-year cycle with the last
Both the SAGE exam and the Smarter
Balanced exam plan to embed PISA items to
establish a link to high school math, and this
information should be available for SAGE by
mid-summer of 2014. This linkage information
will be available to the Department.
Linkages to TIMSS or PIRLS cannot be accomplished through common-item embedding
because they do not allow their items to be
used this way in other assessments. AIR has
obtained permission from the Organisation for
Economic Co-operation and Development
(OECD) to use secure PISA items for purposes
of linking. TIMSS and PIRLS do not permit the
use their items for this purpose (I. Mullis, personal communication, November 21, 2013).
*For example, the PARCC cooperative agreement states unless otherwise protected by law or agreement as proprietary information, make any assessment content (i.e., assessments and assessment items) developed with funds
from this grant category freely available to States, technology platform providers, and others that request it for purposes of administering assessments, provided they comply with consortium or State requirements for test or item
security (Smarter Balanced cooperative agreement with the U.S. Department of Education, January 27, 2011, pp.
1112). Similar language is in the PARCC cooperative agreement.
Page 7.6-40
AIR has already conducted linking studies to

inform the standard setting process in three
states: Delaware, Hawaii, and Oregon (Phillips
& Jiang, 2011).
The following is a brief description of how the
linking study would be conducted. AIR has
obtained 30 representative items for each subject tested in PISA (reading, mathematics, and
science). For the Florida Standards
Assessments, only the items in reading and
mathematics would be used. The 30 items
would be embedded as field-test items in the
online Florida Standards Assessments. The
AIR field-test algorithm would randomly
administer the PISA items just like it would
any other field-test item. For example, lets say
that if the state had F = 400 field-test items, P
= 30 PISA items, and the test length was L =
40 items a student takes in the test administration, then the expected number of PISA
items administered to each student would be
E(p) = 1/F * L * P = 3. If the student population size was N = 200,000, then the expected
sample size per PISA item would be E(n) = 1/F
* L * N = 20,000. The number of items per
student does not provide enough information
to get estimates of PISA scores (i.e., distributions of PISA-plausible values), but it does provide enough students to get good state-based
estimates of the PISA item parameters. This
then permits the state test to be linked to the
PISA scale through common-item linking.
Below is an example of the type of data that
would be obtained from the international linking study between Florida and PISA. The
example in Exhibit 7.6.3.6.3-1 is from an
international benchmarking study between
Hawaii and PISA in mathematics.
The international benchmarking study can be
summarized in the above table. The first column shows the page numbers of an ordereditem booklet with 61 pages used during the
standard setting in mathematics. The second
column shows the item parameters associated

with the page numbers in the ordered-item
booklet, and the third column contains the
thetas associated with the response probability
for each item. The fourth column notes the
standard error of the theta associated with the
response probability. This standard error is
needed because it is a component of error in
the standard error of the PISA-equivalents.
The fifth column includes the state impact
data used during the standard setting. This column tells the panelists what percentage of students is likely to reach the performance standard depending on which page number is
selected. The sixth column shows the PISAequivalent of the theta in the third column.
The seventh column contains the standard
error of the PISA-equivalents. The eighth column shows the inverse cumulative percentage
for the population of 30 OECD countries that
participated in the 2006 PISA. It shows the
approximate percentage of students across the
OECD countries that would reach each PISAequivalent performance standard. The previous proficient performance standard for mathematics in Hawaii was on page 19 of the
ordered-item booklet. The standard setting
panelists decided to place the new Hawaii proficient performance standard on page 26. This
raised the states proficient performance standard from a standard comparable in difficulty
to Level II of PISA to one that is comparable in
difficulty to Level III of PISA and similar to the
international average of the OECD student
performance.
Validity of International Benchmarking

The international linking study plan would also
include strategies for verifying the validity of
the international linking study. This could be
done in several ways. A validity analysis could
be conducted by correlating the PISA item
parameter estimates obtained in Florida with
the estimates of the international item parameters obtained by the OECD. A high correlation
Page 7.6-41
Exhibit 7.6.3.6.3-1: Example of International Benchmarking Using PISA

Results for State Assessment
OrderedItem
Booklet
Page
Item
Theta
Associated
with
RP67
61
60
59
58
57
56
55
54
53
52
51
50
49
48
47
46
45
44
43
42
41
40
39
38
37
36
35
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
1.67
1.56
1.54
1.50
1.44
1.36
1.34
1.29
1.25
1.22
1.21
1.16
1.16
1.14
1.14
1.13
1.06
1.02
0.93
0.89
0.86
0.75
0.75
0.67
0.64
0.56
0.51
0.45
0.39
0.33
0.30
0.22
0.19
0.11
0.08
0.02
-0.06
0.23
-0.15
-0.18
-0.27
-0.32
-0.36
-0.45
2.5
2.3
2.3
2.2
2.1
2.1
2.0
2.0
2.0
1.9
1.9
1.9
1.9
1.8
1.8
1.8
1.8
1.7
1.6
1.6
1.6
1.5
1.5
1.4
1.3
1.3
1.2
1.2
1.1
1.0
1.0
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.4
0.4
0.4
0.3
Page 7.6-42
Standard Error
Percent
Standard
PISAof Theta
Reaching
Error of
equivalent
Associated
State
PISAScaled
with
Cut
Score
equivalent
Score
RP67
0.12
6
618
10
0.11
7
601
9
0.11
7
600
9
0.11
8
597
9
0.11
8
592
8
0.11
9
586
8
0.10
9
584
8
0.10
9
581
8
0.10
10
577
8
0.10
10
575
8
0.10
10
574
8
0.10
11
571
8
0.10
11
571
8
0.10
11
569
8
0.10
11
568
8
0.10
11
568
8
0.10
12
563
8
0.10
13
559
8
0.10
14
553
8
0.10
15
549
8
0.10
15
547
8
0.10
15
539
8
0.10
15
538
8
0.10
19
532
8
0.10
19
530
8
0.10
21
524
8
0.09
22
519
8
0.09
23
515
8
0.10
25
510
8
0.10
26
506
8
0.10
27
504
8
0.10
29
497
8
0.10
30
495
8
0.10
33
489
8
0.09
33
486
8
0.09
34
482
8
0.10
34
475
8
0.08
34
474
7
0.10
37
468
8
0.10
42
466
8
0.10
45
459
8
0.10
47
455
8
0.10
48
452
8
0.10
52
445
8
Percent
Reaching
OECD
PISAequivalent
10
13
13
14
15
17
18
18
20
20
21
22
22
22
22
22
24
25
28
29
30
33
33
36
36
39
41
43
45
47
48
51
51
54
55
57
60
60
63
64
67
68
69
72
Exhibit 7.6.3.6.3-1: Example of International Benchmarking Using PISA (continued)

Results for State Assessment
OrderedItem
Booklet
Page
Item
Theta
Associated
with
RP67
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
-0.45
-0.45
-0.45
-0.45
-0.45
-0.45
-0.45
-0.45
-0.45
-0.45
-0.45
-0.45
-0.45
-0.45
-0.45
-0.45
-0.45
2.5
2.3
2.3
2.2
2.1
2.1
2.0
2.0
2.0
1.9
1.9
1.9
1.9
1.8
1.8
1.8
1.8
Standard Error
Percent
Standard
PISAof Theta
Associated Reaching equivalent Error of
State
PISAScaled
with
Cut Score
equivalent
Score
RP67
0.10
54
618
10
0.10
58
601
9
0.10
59
600
9
0.10
61
597
9
0.10
62
592
8
0.10
65
586
8
0.10
66
584
8
0.11
68
581
8
0.10
69
577
8
0.11
71
575
8
0.11
73
574
9
0.11
77
571
9
0.11
77
571
9
0.11
79
569
9
0.12
81
568
9
0.12
84
568
9
0.12
84
563
9
would indicate that Florida and the OECD are

getting essentially the same estimates.
NAEP and international studies tend to provide
the same ranking for states. For example,
Massachusetts tends to be ranked the number
one state in NAEP, TIMSS, and PISA. A validity
check could be obtained by seeing if the PISA
linking ranks Florida about the same as NAEP.
For example, if the state NAEP for Florida
ranks just below the national NAEP, then we
would expect the state PISA for Florida to rank
just below the national PISA.
References
Allen, M. J., & Yen, W. M. (1979). Introduction
to measurement theory. Monterey, CA:
Brooks/Cole.
American Psychological Association (1999).
Standards for educational and psychological testing (2nd ed.). Washington, DC.
Beaton, A. E., Johnson, E., & Ferris, J. J.
(1987). The assignment of exercises to stu-
Percent
Reaching
OECD
PISAequivalent
74
76
77
78
79
81
82
83
84
85
87
89
89
90
91
93
93
dents. In A. E. Beaton (Ed.), Implementing

the new design: The NAEP 1983-84 technical report (NAEP 15-TR-20). Princeton, NJ:
Educational Testing Service.
Bergstrom, B. A. (1992, April). Ability measure
equivalence of computer adaptive and
pencil and paper tests: A research synthesis. Paper presented at the annual meeting
of the American Educational Research
Association, San Francisco, CA.
Birnbaum, A. (1968). Some latent trait models
and their use in inferring an examinees
ability. In F. M. Lord & M. R. Novick (Eds.),
Statistical theory of mental test scores (pp.
397479). Reading, MA: Addison-Wesley.
Bock, R. D., & Aitkin, M. (1981). Marginal
maximum likelihood estimation of item
parameters: Application of an EM algorithm. Psychometrika, 46, 443459.
Choi, S. W., & Tinkler, T. (2002, April).
Evaluating comparability of paper-andpencil and computer-based assessment in
a K12 setting. Paper presented at the
Page 7.6-43
annual meeting of the National Council on

Measurement in Education, New Orleans,
LA.
Cizek, G. J. (2001). Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Erlbaum.
Cohen, J., & American Institutes for Research.
(2002). AM statistical software (Beta
Version 0.06.00) [Computer software].
Washington DC: American Institutes for
Research.
Cohen, J., & Kolstad, A. (2000). Theory consistent IRT models. Washington, DC:
American Institutes for Research.
Cohen, J., Chan, T., Jiang, T., & Seburn, M.
(2008). Consistent estimation of Rasch
item parameters and their standard errors
under complex sample designs. Applied
Psychological Measurement, 32(4),
289310.
Cohen, J., Johnson, E., & Angeles, J. (2000).
Variance estimation when sampling in
two dimensions via the jackknife with
application to the National Assessment of
Educational Progress. Washington, DC:
American Institutes for Research.
DAgostino, R. B. (1998). Propensity score
methods for bias reduction in the comparison of a treatment to a non-randomized
control group. Statistics in Medicine, 17,
22652281.
Dempster, A. P., Laird, N. M., & Rubin, D. B.
(1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of
the Royal Statistical Society, 39, 138.
Fitzpatrick, A. R., Link, V., Yen, W., Burket, G.,
Ito, K., & Sykes, R. (1996). Scaling performance assessments: A comparison
between one-parameter and two-parameter
partial credit models. Journal of
Educational Measurement, 33, 291314.
Page 7.6-44
Guo, H., Liu, J., Curley, E., & Dorans, N.

(2012). The stability of the score scales for
the SAT Reasoning Test from 2005 to 2010
(ETS Research Report Series [RR-12-15]).
Princeton, NJ: Educational Testing Service.
Haberman, S., & Dorans, N.J. (2009). Scale
consistency, drift, stability: Definitions,
distinctions and principles. Paper presented at the annual meeting of the American
Educational Research Association and
National Council on Measurement in
Education, San Diego, CA.
Haebara, T. (1980). Equating logistic ability
scales by a weighted least squares method.
Japanese Psychological Research, 22(3),
144149.
Haebara, T. (1980). Equating logistic ability
scales by a weighted least squares method.
Japanese Psychological Research, 22,
144149.
Harris, D., & Kolen, M. (1994). Stability
checks in random groups equating. Paper
presented at the annual meeting of the
American Educational Research
Association. New Orleans, LA.
Holland, P. W., & Thayer, D. T. (1988).
Differential item performance and the
Mantel-Haenszel procedure. In H. Wainer &
H. I. Braun (Eds.), Test validity. Hillsdale,
NJ: Erlbaum.
Huff, K. L., & Sireci, S. G. (2001). Validity
issues in computer-based testing.
Educational Measurement: Issues and
Practice, 20(3), 1625.
Ito, K., & Sykes, R. C. (2004). Comparability
of scores from norm-referenced paper and
pencil and web-based linear tests for
grades 412. Paper presented at the annual
meeting of the American Educational
Research Association, San Diego, CA.
Johnson, E. G. (1992). The design of the
National Assessment of Educational
Progress. Journal of Educational

Measurement, 29, 95110.
Mead, A. D., & Drasgow, F. (1993). Equivalence

of computerized and paper-and-pencil cognitive ability tests: A meta-analysis.
Psychological Bulletin, 114, 449458.
Karkee, T., Kim, D. I., & Fatica, K. (2010).

Comparability study of online and paper
and pencil tests using modified internally
and externally matched criteria. Paper
American Educational Research
Association, Denver, CO.
Messick, S. (1995). Validity of psychological

assessment: Validation of inferences from
persons responses and performances as
scientific inquiry into score meaning.
American Psychologist, 50(9), 741749.
Keng, L., McClarty, K. L., & Davis, L. L. (2006).

Item-level comparative analysis of online
and paper administrations of the Texas
Assessment of Knowledge and Skills. Paper
Education, San Francisco, CA.
Michaelides, M., & Haertel, E. (2004).

Sampling of common items: An unrecognized source of error in test equating
(Center for the Study of Evaluation Report
[CSE-636]). Los Angeles, CA: National
Center for Research on Evaluation,
Standards, and Student Testing.
Kolen, M. J., & Brennan, R. L. (2004). Test

equating, linking, and scaling: Methods
and practices (2nd ed.). New York:
Springer-Verlag.
Mitzel, H. C., Lewis, D. M., Patz, R. J., &

Green, D. R. (2001). The Bookmark procedure: Psychological perspectives. In G.
Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Erlbaum.
Linn, R. L., Levine, M. V., Hastings, C. N., &

Wardrop, J. L. (1981). Item bias in a test of
reading comprehension. Applied
Psychological Measurement, 5, 159173.
Lord, F. M., & Novick, M. R. (1968). Statistical
theories of mental test scores. Reading, MA:
Addison-Wesley.
Loyd, B. H., & Hoover, H. D. (1980). Vertical
equating using the Rasch model. Journal of
Educational Measurement, 17, 179193.
Marco, G. L. (1977). Item characteristic curve
solutions to the three intractable testing
problems. Journal of Educational
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2),
149174.
McKinley, R. L., & Mills, C. N. (1985). A comparison of several goodness-of-fit statistics.
Applied Psychological Measurement, 9,
4957.
Moses, T., Deng, W., & Zhang, Y. (2011, July).

Two approaches for using multiple anchors
in NEAT equating: A description and
demonstration. Applied Psychological
Measurement, 35(5), 362379.
Paek, P. (2005). Recent trends in comparability studies (PEM Research Report 05-05).
Retrieved from http://www.pearsonedmeasurement.com/downloads/research/RR_05_
05.pdf.
Pellegrino, J. W., Jones, L. R., & Mitchell, K. J.
(1999). Grading the nations report card:
Evaluating NAEP and transforming the
assessment of educational progress.
Washington, DC: National Academy Press
Phillips, G. W., & Jiang, T. (2011). Using PISA
to internationally benchmark state performance standards. Paper presented at
the annual meeting of the American
Educational Research Association (AERA),
New Orleans, LA.
Page 7.6-45
Rasch, G. (1980). Probabilistic models for

some intelligence and attainment tests.
Chicago: The University of Chicago Press.
(Original work published 1960)
Rosenbaum, P. R. (1995). Observational
Studies. New York: Springer-Verlag.
Rosenbaum, P., & Rubin, D. B. (1983). The
central role of the propensity score in
observational studies for causal effects.
Biometrika, 70, 4155.
Rubin, D. B. (2006). Matched sampling for
causal inference. Cambridge, England:
Cambridge University Press.
trends. Paper presented at the annual

meeting of the National Council on
Measurement in Education, New York, NY.
Webb, N. (2005, April). Issues related to judging the alignment of curriculum standards
and assessments. Paper presented at the
annual meeting of the American
Educational Research Association,
Montreal, Canada.
Yen, W. (1981). Using simulation results to
choose a latent trait model. Applied
Psychological Measurement, 5, 245262.
Samejima, F. (1972). A general model for freeresponse data. Psychometrik (Monograph

Supplement No. 18).
Stocking, M. L., & Lord, F. M. (1983).
Developing a common metric in item
response theory. Applied Psychological
Thissen, D., & Orlando, M. (2001). Item
response theory for items scored in two
categories. In D.Thissen & H. Wainer
(Eds.), Test scoring (pp. 73140). Mahwah,
NJ: Erlbaum.
Thissen, D., Nelson, L., & Swygert, K. A.
(2000). Item response theory applied to
combinations of multiple-choice and constructed response itemsapproximation
methods for scale scores. In D. Thissen &
H. Wainer (Eds.), Test scoring. Hillsdale,
NJ: Erlbaum.
Way, W. D., Davis, L. L., & Fitzpatrick, S.
(2006, April). Score comparability of online
and paper administrations of the Texas
Assessment of Knowledge and Skills. Paper
Education, San Francisco, CA.
Way, W. D., Lin, C., & Kong, J. (2008, March).
Maintaining score equivalence as tests
transition online: Issue, approaches and
Page 7.6-46
4. Management Plan
4. Management Plan
AIR recognizes the key role that the Florida
Standards Assessment (FSA) will play in
Floridas educational system, with high stakes
for schools, districts, teachers, and students.
Floridas commitment to transparency supports these high-stakes uses by enabling stakeholders to see how the tests are created and
deployed.
AIRs management approach is designed to
support such high-stakes, high-visibility programs. Our management approach relies on an
extensive system of planning and specifications documents. It is not enough for all participants in the process to hear the same decisions in a meeting; decisions must be documented in a place where they can be easily
found, referenced, reviewed, and considered.
For this reason, each aspect of the testing program is governed by a specification or planning document. Each document has a designated location with Knowledge Tree, the
secure online service that we use to control
management documents and manage the
change workflow.
Section 7.7.6 describes the collection of documents that, together, support the management
of the program. Each document enlists stakeholders, and as decisions are made or changed,
each stakeholder is alerted to review the
change, so that the ramifications of the decisions can be considered by the technical
experts in all areas. For example, a change to a
draft blueprint may have implications for what
can be reported, what data is transferred
among systems, and possibly even data file formats. Our management communication
ensures that these impacts are understood at
the time that decisions are made.
Managing this complex project requires a project management team that is experienced at
focusing people on the important issues and
helping them reach consensus. AIRs senior
program manager, Kevin Murphy, will have
overall, daily responsibility for all processes
and deliverables under the full scope of this
program. More specifically, the Program team
will be responsible for maintaining the implementation plan and schedule and for ensuring
open and frequent communication with the
Department staff to provide updates on status,
key accomplishments, upcoming activities,
and issues. Section 7.7.2 provides details about
the highly qualified personnel AIR proposes to
work closely with the Department to manage
the contract.
AIR understands that adhering to standard
processes and documentation is critical to the
programs successful operation, and we use a
set of unique management tools and reports to
ensure the program is on target, the schedule
is maintained, and the deliverables are successfully addressed and completed. In Section
7.7.6, we describe our management approach
and management tools for planning, monitoring, and managing changes to the
Departments assessment program and our
quality assurance protocols. We describe how
we will use our management experience and
tools to successfully complete the work under
this contract while collaborating with the
Department. We have successfully applied
these methods on state programs and will
adapt to meet the Departments requirements.
We are proud of our work, and we look forward
to a productive relationship with the
Department that serves Florida students and
advances the assessment field.
Page 4-1
7.7.1 Provisions Governing the

Work of the Contractor
AIR acknowledges and commits to the
provisions governing the general work of this
contract.
a. AIRs project team will work with the
Department to secure approval of all
processes and products at milestone
stages. Applying the management methods
described in Section 7.7.0 and the procedures detailed throughout this proposal will
ensure that communication, transparency,
and decision-making are clear and documented co-efforts between AIR and the
Department. We will make significant
efforts to define the specifications of tasks
and products with the Department. AIRs
project schedules will ensure specifications
are defined in a timely manner for successful implementation.
b. The Department will have design control as
it relates to the assessment logo and color
scheme of AIR systems and AIR-produced
documents. Our systems are designed to be
branded for our clients, and we will work
with the Department to identify its preferred logo and color scheme. Some of our
secure testing apps will have to retain AIR
branding because the distribution outlets
(i.e., Apple App Store) will not let us post
multiple versions of an app with different
branding.
c. AIR and the Department will collaborate on
the creation and maintenance of materials
used to support and administer the Florida
Standards Assessment (FSA). Reasonable
modifications to these materials will be
captured within the scheduled review and
edit phases of the products schedule. AIR
will endeavor to implement those modifications prior to printing, production, or
deployment. Modifications requested outside
Page 4-2
of the review and edit phases may impact

the delivery of subsequent deliverables.
d. AIR has accounted for annual modifications
to products and processes within the scope
of this proposal. As mentioned in Section
7.7.5, annual planning meetings will allow
us to capture new or updated information
for each administration or contract year.
Modifications will also be captured in more
detail in the annual revisions of specifications and materials, which will be reviewed
and approved by the Department. We will
secure approval of print quantities from the
Department prior to production to reduce
costs and security risks.
7.7.2 Contractor Staff and

Responsibilities
In this section, we introduce the staff from
both AIR and DRC.
AIRs Assessment Program includes 380 fulltime and 200 temporary staff members available to support Florida and other state assessment clients.
AIRs assessment work operates under the
direction of Dr. Jon Cohen, AIRs Executive
Vice President and President, Assessment;
Steve Kromer, AIRs Vice President for
Assessment; and Heather Hayes, who directs
all online testing projects at AIR. Dr. Cohen,
Mr. Kromer, and Ms. Hayes have the authority
to make corporate decisions related to this
project. Dr. Cohen and Ms. Hayes, and other
AIR senior managers, are intimately involved
in project operations and attend key planning,
progress, and even technical meetings.
AIRs Assessment Program Structure

The organizational structure of each of AIRs
projects follows the functional organization of
AIRs Assessment Program. We summarize the
organizational structure and the depth of capability of our staff below. AIRs corporate directors work directly on our state projects more
than 90% of the time. AIRs Assessment
Program is organized functionally, and each
functional area is led by a member of AIR
Assessments senior management team. Within
each functional area and for each project,
teams are formed and a team leader is designated. The team leader takes first-line responsibility for the teams products and services for
the project, with ultimate responsibility resting
with the corporate director of the relevant
functional area.
Steve Kromer brings more than 25 years of

assessment experience to his role as AIR
Assessments Vice President. In this role, he
is responsible for ensuring that repeatable
processes are in place in each functional group
and that we follow these processes in delivering each program.
Given the specialized nature of our work and
AIRs unique approach to it, staff are expected
to master their capabilities through an apprenticeship system predicated on the gradual
increase of individual responsibility. Group
training sessions and collaborative interaction
that encourage and promote mentoring and
supervision are at the core of this system.
We achieve this goal by organizing work into
structured teams, each with well-defined specifications and work processes:
Computer and Statistical Sciences Center

(CSSC)
Psychometrics and Statistics
Test Development
Operations and Scoring
Online Testing Systems
Communications and Reporting
Project Management
State assessment
contracts such as
the Florida
Senior Program
Standards
Manager:
Assessments are
Kevin Murphy, M.A.,
large and complex,
P.M.P.
involving many different skills, client
management staff,
school staff, vendors, systems,
datasets, schedules, and other resources and
stakeholders to operate efficiently. The Project
Management team is responsible for orchestrating these resources to deliver and report
on multiple assessments.
Director:
Heather Hayes, J.D.
Team Leadership
Dr. Cohen sets the vision and standards for

the program; recruits, develops, and motivates
staff; and makes sure needed infrastructure
and procedures are in place. AIRs approach
to managing personnel and staff ensures that
there is a team to support each contract
requirement and that required information
never resides in a single person.
Project Management
Ms. Hayes serves as AIR corporate director for

project management for online testing programs and leads a team of 38 project management and operations staff members, 26 of
whom hold masters or Ph.D. degrees. Ms.
Hayes has more than 15 years of experience in
online systems, including managing Oregons
online testing project and documenting and
transferring best practices in online, multipleadministration testing from Oregon to newer
AIR state clients including Delaware, Hawaii,
Minnesota, and Ohio and soon Utah, Missouri,
and Michigan. She received her J.D. from the
University of Iowa.
Page 4-3
For this project, AIR is proposing one of our

most experienced senior program managers,
Kevin Murphy.
Ms. Hayes and Mr. Murphy will be completely

familiar with the entire project and will participate in all meetings. Together they will lead
all internal management meetings with AIR
and DRC staff and all communications with
Florida management staff and serve as a team
for communication with other stakeholders.
Mr. Murphy will take the lead on schedules,
produce minutes and other documentation,
and implement the project management and
quality assurance procedures to ensure fidelity
to the projects goals and policies.
Director:
Selina Tolosa, M.B.A.
Project Manager,
Computer-Based Testing:
Stacy Tsakeris, M.B.A.
Team Leadership
A Project Management Institute (PMI) certified

manager, Mr. Murphy has more than 12 years
of experience directing multi-million dollar,
statewide assessment projects. For 8 years at
AIR, he has successfully led the paper/pencil
testing program in Ohio, managed projects to
transition Hawaii and Delaware from
paper/pencil testing programs to 100% online,
and managed Minnesotas dual-mode program.
He is experienced in all aspects of assessment
projects, including test development,
paper/pencil and computer administration,
psychometrics, scoring, reporting, and professional development. Mr. Murphy also brings 6
years of classroom teaching experience. Mr.
Murphy received his B.A. in Business from
Lehigh University and his M.A. in Secondary
Education from Hunter College.
Computer and Statistical Sciences

Center (CSSC)/Online Testing
CSSC is responsible
for both our software
infrastructure and our
software products,
including the technology and support for
AIRs student registration, online testing, reporting, learning management,
and related systems.
CSSC takes as its mission the development of

innovative products and systems that will keep
AIRs Assessment Program at the forefront of
our field. CSSC combines a structured software development unit (developers and project
directors) with a team of statisticians, psychometricians, and mathematicians. It is organized into groups supporting families of software
systems:
Preproduction systems: Responsible for
systems that support assessments before
they are delivered, including test development, publication, and user management
Test delivery and reporting: Responsible for
our high-demand, public-facing systems
that deliver tests and reports, as well as
other curricular and educational supports
Data analysis: Responsible for test data processing, analysis systems, and data deliveries
Each family of systems has a technical lead,
and together these technical leads form a committee responsible for overseeing peer review
of key documents such as requirements,
design documents, and test plans, as well as
peer review of program code.
CSSCs interdisciplinary team includes more
than 120 staff members, nine of whom have
Page 4-4
Close interdisciplinary collaboration enables

CSSC to develop remarkable technical products that effectively turn raw data into useful
information and manage complex human
processes to efficiently support test development, psychometrics, and score reporting.
For eight years, Selina Tolosa has served as
team leader for the development and implementation of assessment software systems at
AIR including AIRs online test delivery engine
for operational tests (Oregon, Delaware,
Hawaii, and Minnesota). Prior to joining AIR,
Ms. Tolosa was the director of information
services and the financial officer for New
American Schools, a nonprofit organization
funding the development of schoolwide designs
aimed at transforming elementary and secondary schools. Ms. Tolosa also brings experience
in financial forecasting and analysis; quality
assurance of government enterprise systems;
and the application of database skills, statistical sampling, and relational databases to data
organization of assessment systems. Ms. Tolosa
received her M.B.A. from the Wharton School
of Business, University of Pennsylvania.
Reporting directly to Ms. Tolosa, Stacy
Tsakeris will serve as project manager, computer-based testing. Ms. Tsakeris is managing
the quality control for the Smarter Balance
field testing in 24 states and also managing the
test content documentation of the Open
Source software which will be available to the
Smarter Balanced states. Ms. Tsakeris has
more than 10 years of experience in software
project management and has earned her PMP,
her M.B.A, and her B.S. in Computer Science
and Engineering. She will be supported by our
most experienced team of software project
managers and engineers

Vice President and
Chief Psychometrician:
Gary Phillips, Ph.D.
Psychometrics Lead:
Harold Doran, Ph.D.
Team Leadership
Ph.D.s in areas ranging from computer science

to statistics and engineering. Fifty-three of
their software project directors and other engineers have M.B.A.s and other advanced
degrees.
AIRs Assessment
Program offers psychometric and statistical services that
stand alone in terms
of quality and innovation. The integration
of psychometrics with
statistics and sampling sets AIR apart
from the competition.
Although testing firms often bring expertise in

psychometrics, the quality of those services
depends dramatically on the samples on which
the data are based. Typical samples used in
state testing programs can undercut the best
psychometrics, leading to volatile test results
from year to year and inaccurate classification
of examinees. AIR combines expertise in sampling and psychometrics; all of our samples are
optimized, and our statistics accurately reflect
the complexities of the sample designs.
Dr. Gary Phillips leads the Psychometrics and
Statistics team, which includes 14 lead psychometricians with Ph.D. degrees and more
than a dozen statistical programmers, research
associates, and other supporting staff.
Dr. Phillips brings 26 years of experience, with
distinctions such as having run the National
Assessment of Educational Progress (NAEP) for
the federal government and having served as
acting commissioner of the National Center for
Education Statistics (NCES). Prior to that, he
served as the deputy commissioner of NCES.
Previous responsibilities at NCES included
overseeing the NAEP, the National Adult
Literacy Study (NALS), and the Third
International Mathematics and Science Study
(TIMSS). Dr. Phillips was the architect and
served as the executive director of President
Clintons Voluntary National Test (VNT). He
Page 4-5
AIR is proposing Dr. Harold Doran as the lead

psychometrician. Working closely with Dr.
Gary Phillips, Dr. Doran will apply all of the
Department-approved technical procedures,
will have final approval over all analyses before
submission to the Department, and also will
lead or oversee the required studies. In addition to supporting the psychometrics of largescale assessment programs in multiple states,
Dr. Doran is an expert in value-added models,
growth models, psychometrics, and state
assessment policy.
The Psychometrics and Statistics team is
responsible for the following tasks:
Sample design
Field-test design
Item analysis
Analysis of differential item functioning
Calibration, using both classical and IRT
methods
Parallel form equating and vertical linking
of related tests
Design and implementation of standard-setting sessions
Design and implementation of special studies, including validity studies
Randomized field trials
Page 4-6
Cross-form reliability studies

Program and initiative evaluations
Test Development
Developing more
than 10,000 items
annually, our test
Project Manager,
development staff
Test Development:
has been developSandra Durden, M.A.
ing and aligning
existing items to
Common Core
State Standards
since their inception. This team is
led by June Zack. Ms. Zack has been the director of test development for the Assessment
Program at AIR for the past 12 years, growing
the team to include more than 110 staff members (10 Ph.D. degrees, 61 masters degrees).
Director:
June Zack, M.A.
Team Leadership
holds a Ph.D. from the University of Kentucky

with an emphasis in statistics and psychometrics. He has published or presented more than
200 papers, taught dozens of advanced graduate-level statistics courses, and presented hundreds of workshops on advanced statistical and
psychometric topics. His areas of interest are
structural equation models, hierarchical linear
models, item response theory (IRT) models,
and Bayesian inference. He is nationally and
internationally known for his expertise in
large-scale assessments and complex surveys.
Ms. Zack is primarily responsible for the development of items and the assembly of test
forms for all of AIRs assessment projects. She
is responsible for budget tracking, staffing,
product development, and overall coordination
of the development of the assessments. Before
joining AIR, Ms. Zack worked at ETS for 10
years, managing test development for numerous projects, including the GRE, GMAT, LSAT,
SAT, NAEP, and Praxis assessments. Ms. Zack
was also involved in project management for
the reading, writing, and mathematics development for the ISEE and the CTP IV. Ms. Zack
holds an M.A. in Educational Psychology from
New York University.
Our Test Development team is responsible for
critical activities including the development of
assessment frameworks, item and form development, and curricular content development.
Across projects, AIR Test Development staff
are responsible for the following tasks:
Framework development
Test blueprint and item specification
development
Item development and review
Development and review of scoring rubrics
Form construction
Curricular content development
Development of supporting materials for
item release
Client committee support and liaison with
rangefinding services
AIR is proposing multiple senior staff to lead
form building, item development, and related
activities with the Department. Sandra Durden
will serve as the Project Manager, Test
Development, and as AIRs Item Development
Manager. Ms. Durden has more than 20 years
of experience in the field of education, including 10 years in the assessment industry and 10
years as a K8 educator. At AIR, she is working
on performance tasks for the Smarter
Balanced Assessment Consortium and item
development for state programs and K2 formative assessments. Ms. Durden holds an M.A.
in Mathematics Education and a B.A. in
Elementary Education. Ms. Durden will be supported by Diana Reed and Jim McCann, our
content leads for ELA and Math, respectively.
Operations and Scoring
Project Manager,
Test Administration:
Meredith Durgin, B.S.
Team Leadership
Director:
Dee Wagner, M.B.A.
AIRs Operations
and Scoring team
includes 29 fulltime and more than
200 part-time professionals and is
responsible for
print test forms,
production, and
help desk operations and for ware-
housing, distributing, collecting, security processing, scanning, editing, performance scoring, and preparing data files. The supporting
software systems interface directly with operations relating to pre-ID and ordering, student
tracking, print tracking, and other AIR systems.
Dee Wagner, M.B.A., is AIRs Corporate
Director for Operations. Ms. Wagner has more
than 22 years of experience managing large,
high-stakes assessment projects, including the
past 13 years at AIR. Ms. Wagner has overseen
test administration, security, and assessment
operations, which include activities related to
producing, delivering, receiving, scanning, and
scoring test materials. She is also adept at
managing AIRs subcontractors for processing
and has managed DRC on paper/pencil testing
projects in Ohio and Minnesota. A Six Sigma
Black Belt and ISO Certified Lead Assessor,
Ms. Wagner gets the most out of staff and consistently identifies and gives more responsibility to high performers. The results have been
outstanding; AIR projects run on schedule,
within budget, and to high client satisfaction.
Ms. Wagner received her operations management (B.S.B.A.) degree from the University of
Delaware and her M.B.A. from Philadelphia
University.
AIR is proposing multiple project managers
and our major subcontractor on this team.
Meredith Durgin will serve as project manager,
test administration. Ms. Durgin has 10 years of
management experience in educational assessment, including managing multi-million dollar
projects. Ms. Durgin has served as a senior
manager for Oregons very successful online
testing project and has led smaller programs in
Florida, Ohio, New Jersey, New Mexico, Texas,
Missouri, Arkansas, Montana, and Utah.
Cynthia Benis is AIRs help desk manager and
manages AIRs help desk across all assessment
projects, with a focus on support for online
testing. Ms. Benis has 25 years of customer
Page 4-7
service experience, including the last 5 years

at AIR leading our help desk across projects.
DRC will be responsible for paper/pencil processing and all hand-scoring services for the
Florida Standards Assessments. This is a
proven relationship; together AIR and DRC
have delivered high stakes tests in Ohio and
Minnesota without incident for 5m+ tests. We
already have established the data flows and
automated quality assurance checks between
our systems to ensure quality performance.
Communications and Reporting

Team Leadership
AIR goes beyond

simply reporting
scores and other
Project Manager,
numbers to provide
Scoring and Reporting:
deep analyses of
Cathy Kugler, B.A.
the data, reported
in a clear, appealing, and actionable
way. AIR is perhaps
alone in the industry in delivering
information, often in full color, with variable
text and graphics. The reports begin with the
same basic data that have been traditionally
reported, but they extend much further. The
basic data are reported, using tested data displays that facilitate correct interpretation and
impede misinterpretation. The implications
(as well as limitations) of additional analyses,
including the increasingly popular valueadded analyses, are presented both graphically and in text that is customized for each individual report. Our professional development
services focus on assessment administration
and data use, helping our clients develop
assessment literacy in their states.
Director:
Bani Dheer, J.D.
Bani Dheer is the director of our

Communications and Reporting area, bringing
14 years of experience to this role. Ms. Dheer
has led the development and on-time delivery
Page 4-8
of more than 30 million pages of score reports

for AIR clients in Hawaii, New Mexico, Delaware,
Minnesota, Ohio, South Carolina, and various
city assessment systems. Prior to joining AIR,
Ms. Dheer designed and developed the messaging strategies for several national and state
education groups, including the National
Education Association, the Learning First
Alliance, the Civil Society Institute, and the
California Teachers Association. She also
worked with lead Democratic strategist Mark
Mellman to develop the communications strategy for a presidential campaign. Before entering the field of educational assessment and
communications, Ms. Dheer worked for five
years as managing editor for a Vancouverbased publishing group.
AIR is proposing Cathy Kugler to serve as the
project manager, scoring and reporting. Ms.
Kugler, who has seven years of assessment
experience at AIR, brings a unique combination of statistical and management expertise.
She has directly relevant experience managing
data files, team schedules, and deliverables for
both psychometric and reporting activities for
state assessments in Ohio and Minnesota.
Marta Gebrehiwot will lead score reporting for
the Florida Standards Assessments. Ms.
Gebrehiwot brings experience as the reporting
line manager for assessments in Delaware, New
Mexico, South Carolina, Minnesota, and other
states.
Following the guidelines in the ITN, we have
organized the remainder of this section as
follows:
Exhibit 7.7.2-1, on the next page, presents
the project organization chart.
Exhibits 7.7.2-2 and 7.7.2-3, following the
project organziation chart, presents the
required staffing charts as outlined in the
ITN.
Dr. John Selisky, DRC

Content Lead
Chris Peterson, DRC
Content Specialist
Dr. James Bell, DRC

Content Lead
Anne Kirpes, DRC
Content Specialist
Editorial Staff
Katerina Sagris, M.A.

Chris Kincheloe, B.S.
Roshanak Matewere, B.A.
Leads, High School
(Algebra I, Geometry,
and Algebra II)
Mark Lovely, M.F.A.

Sean Redmond, M.A.
Charles Ferry, M.A.T.
Leads, High School
Allen Chong, M.Ed.

Meghan Mulhern, Ed.M.
Austin Buckley, B.A.
Leads, Grades 68
Eileen Heneghan, B.A.

Maureen Font, M.Ed.
Leads, Grades 35
Christine Keller, M.A.T.

Terry Hill, B.A.
Allison Stingley, M.A.E.
Leads, Grades 35
George Ligon, M.Ed.

Kari Thicksten, M.S.
Natalie Morgan, B.S.
Leads, Grades 68
Math
Jim McCann, M.S.
Content Lead
ELA
Diana Reed, Ed.M.
Content Lead
Sandra Durden, M.A.

Project Manager, Test Development
Research Associates
Research Assistants
Christie Plackner, DRC

Manager,
Psychometric Quality
Dr. Yi Du, DRC
Psychometrician
Elizabeth Ayers, Ph.D.

Psychometrician
Harold Doran, Ph.D.

Lead Psychometrician
Heather Hayes, J.D.

Director, Online Testing
June Zack, M.A.

Director,
Test Development
Tier 2/3 Help Desk Staff
Developers, QA Staff,
and Network Engineers
Graham Tapscott
Network Engineering Lead
Paul Van
Wamelen, Ph.D.
Sr. Application
Developer
Dorian Sofiaj, M.S.

Software QA Lead
Aaron
Newman, M.S.
Sr. Application
Developer
Dipesh Mamtora,
M.S. & M.B.A.
Sr. Application
Developer
Brian Kane, Ph.D.

Software Project
Manager
Network
Engineering
Jeremy Fisher, M.S.

Software Project
Manager
Test Delivery
(TDS)
Zorina Alliata, M.B.A.

Software Project
Manager
Item Bank
(ITS)
Analysis Systems:
Quality Monitoring
(QM) and Database
of Record (DOR)
Tier 1 Help Desk Staff
Software Quality
Assurance
Anil Chevuri, B.E.

Sr. Application
Developer
Shailesh Gattewar,
M.A. & M.B.A.
Software Project
Manager
Student
Registration
(TIDE)
Tamonica Jackson
Matthew Hirsch
Administrative Assistants

Project Manager,
Computer-Based Testing
Cynthia Benis, B.S.

Help Desk Manager

Director, Computer and Statistical
Sciences Center
Jon Cohen, Ph.D.

President, AIR Assessment
Steve Kromer, B.B.A.
COO, AIR Assessment
Florida Policy Advisor

and Director, Interstate Relations
Kevin Murphy, M.A.

Senior Program Manager
Project Management
Psychometrics

Vice President
and Chief Psychometrician
Test Development
Exhibit 7.7.2-1: Project Organization Chart
Forms Managers,
Operations and
Production Staff
Scoring Staff
Math
Coskun Baysoy, DRC
Lead Scoring Director
ELA
Wendy Marik, DRC

Project Manager,
Handscoring
Sherri Woolf, DRC

Lead, Handscoring
Key:
Corporate Directors
Project Leads
Graphic Designers
Marta Gebrehiwot, B.A.

Lead, Score Reporting
Cathy Kugler, B.A.

Project Manager,
Bani Dheer, J.D.

Director, Online and
Paper Reporting
Pam Ahrens, DRC

Lead Scoring Director
Pete Tressel, DRC

Project Manager, Paper/Pencil

Project Manager,
Test Administration
Dee Wagner, M.B.A.

Director, Operations
Operations
Page 4-9
Page 4-10
90%
90%
90%

Content Lead, Mathematics
Content Lead, ELA
Project Manager, Computer-based Testing
Project Manager, Test Administration
Project Manager, Paper/Pencil Operations
DRC Administrative Assistants
Project Manager, Help Desk
10%
5%
40%
5%
Harold Doran, Ph.D.
men
t

Florida Policy Advisor/Director, Interstate Relations
Administrative Assistant
Position
Cynthia Benis, B.S.

Scanning, Scoring, and Reporting
Cathy Kugler, B.A.
Project Manager, Scoring and Reporting
Wendy Marik, M.Ed.
Project Manager, Handscoring
Pam Ahrens
Lead Scoring Director, ELA/Literacy
Coskun Baysoy
Lead Scoring Director, Math
Lead, Score Reporting
Scaling, Equating, Scoring, and Special Psychometric Studies
Tamonica Jackson
Matthew Hirsch
Test Development
Sandra Durden, M.A.
Jim McCann, M.S.
Diana Reed, Ed.M.
Test Administration
Pete Tressel, M.B.C.
Name/Position
Program Management
Kevin Murphy, M.A.
Dev
elop
Test
Exhibit 7.7.2-2: Staff Loading Chart
Test
Ad
m
inis
trat
ion
90%
100%
10%
40%
25%
10%
25%
20%
80%
100%
100%
100%
100%
10%
40%
10%
10%
10%
10%
80%
20%
10%
20%
5%
5%
30%
120%
50%
20%
40%
60%
S ca
nnin
g
,
S
cori
ng,
and
R
e
por
S ca
ting
ling
,
Spe Equat
i
c
n
i
a
g
l
P
sych , Scori
n
om
et g, an
r
i
c
Stu d
dies
Pro
gra
mM
a
n
age
men
t
10%
5%
5%
5%
Tota
l
80%
100%
100%
100%
100%
100%
100%
100%
50%
200%
25%
100%
100%
100%
100%
50%
100%
100%
Exhibit 7.7.2-3: Estimated Number of AIR and DRC Staff and Total Person Days by Month by Staff Category by Major Activity,
July 2014-June 2015
Major Activity/Staff Category
Number
of Staff Jul
Aug Sep Oct
Nov
Dec Jan
Feb Mar Apr May
Number
Jun of Staff
Test Development
2
28
2
9
10
4
12
292
4
18
50
45
12
292
4
18
50
45
15
355
4
23
63
52
3
64
4
12
6
3
64
4
12
6
3
64
4
12
3
6
124
9
24
12
6
6
136 136
9
9
24 24
12 12
6
136
9
-
12
292
8
18
-
12
292
8
18
-
96
2,247
28
143
-
24
12
50
25
50
25
395
258
6
8
26
32
6
2
54
50
145
75
19
10
34
40
50
165
14
10
54
145
230
14
10
54
145
230
8
10
34
50
130
16
10
54
145
140
30
10
44
255
120
20
10
54
418
140
4
10
66
418
150
4
10
66
418
135
4
10
66
418
135
4
10
44
145
135
4
10
23
2,752
1,785
141
120
2
2
8
5
34 140
6 20
6 126
4,378 2,120
4
2
2
2
160
30
126
50
40
2
2
140 140
20 20
126 126
434 434
5
5
2
2
40 140
30 20
63 84
25 2,120
2
2
2
410
20
63
100
5
2
5
5
5
- 20 40 40
- 55 55 55
482 482 482 522 430
30 20 20 30 20
84
84 126 126 126
100 17,772 11,130 11,130 510
2
2
2
2
2
23
100
165
3,568
410
1,260
45,925
55
2
Program Management
Test Development
Psychometrics/Statistics
Test Administration
Software
Handscoring
Production/Editorial
Administrative
Test Administration
Program Management
Test Development
Test Administration
Software
Handscoring
Administrative

Program Management
Test Development
Test Administration
Software
Handscoring-full-time staff
Handscoring-scorers and scoring directors
Administrative
Scaling, Equating, Scoring, and Special Psychometric Studies

Program Management
Test Development
Test Administration
Software
Handscoring
Administrative
2
12
8
2
3
2
2
40
40
16
2
2
60
70
8
5
2
2
60
90
2
5
2
2
40
90
2
5
2
2
50
2
2
2
50
2
2
2
50
8
2
2
50
8
2
2
50
8
2
2
50
16
2
2
80
16
2
2
100
16
2
24
200
770
104
5
9
4
2
3
2
4
60
70
10
5
5
3
40
60
62
2
1
1
1
40
60
62
2
1
1
1
40
60
62
2
1
1
1
40
60
62
2
1
1
1
40
60
62
2
1
1
1
40
60
70
10
5
5
3
40
60
62
2
1
1
1
40
60
62
2
1
1
1
40
60
60
60
720
62
2
1
1
1
40
62
2
1
1
1
40
62
2
1
1
1
40
760
40
20
20
16
480
15
24
Program Management
Program Management
Test Development
Test Administration
Software
Handscoring
Administrative
Page 4-11
Below we introduce the senior staff members for each functional team. One-page
resumes can be found in Appendix A.
AIRs Proposed Project Staff

Project Management
Kevin Murphy, M.A., P.M.P., Senior Program
Manager. For this project, AIR is proposing
one of our most experienced senior program
manager, Kevin Murphy. A Project
Management Institute (PMI) certified manager,
Mr. Murphy has more than 12 years of experience directing multi-million dollar, statewide
assessment projects. For 8 years at AIR, he has
successfully led the paper/pencil testing program for 800,000 students in Ohio, managed
the projects to transition Hawaii and Delaware
from paper/pencil testing programs to 100%
online, and managed Minnesotas dual-mode
program. Prior to joining AIR, Mr. Murphy
worked for 4 years on the North Carolina
Testing Program, both at the North Carolina
Department of Public Instruction and at North
Carolina State University. He is experienced in
all aspects of assessment projects, including
test development, paper/pencil and computer
administration, psychometrics, scoring, reporting, and professional development. Mr. Murphy
also brings 6 years of classroom teaching experience (New York State Teacher Certification,
Social Studies, grades 712) and has experience with reporting systems from his work
with the Grow Network. Mr. Murphy received
his B.A. in Business from Lehigh University
and his M.A. in Secondary Education from
Hunter College.
He will be dedicated to work full-time (100%)
on the Florida Standards Assessments. As
Senior Program Manager, Mr. Murphy will have
full responsibility for the development, implementation, and support of the Florida
Standards Assessments, including training and
other human systems required for the system
Page 4-12
to work effectively. The senior program manager will be responsible for managing the dayto-day operations and for providing the
Department with the information required to
effectively monitor progress.
Mr. Murphy will converse directly and frequently with Department staff to ensure that
they are kept informed about project progress.
AIRs management approach to project communication is predicated on close communication with the Department. We fully recognize
the importance of establishing and maintaining
frequent communication with the Department
to ensure prompt and effective responsiveness
to the Departments needs. Mr. Murphy will
serve as the primary point of contact with the
Department and will be available at all times
for telephone consultation and face-to-face
meetings over the course of the project. Mr.
Murphy will maintain an updated project plan
and schedule and will provide regular and frequent status updates to the Department. He
has used Microsoft Project and similar predecessor tools in coordinating current and previous projects. The detailed schedule that he
creates will include both internal AIR timelines and project milestones that involve the
Department. This schedule will be shared with
the Department, and any changes will be discussed in biweekly conference calls. Mr.
Murphy will work with the Department to
design summary reports or calendars that will
conveniently portray delivery dates, review
timelines, and other scheduled events impacting workflow at the Department.
He will take primary responsibility for liaison
with the Department and Florida educators;
communication with the Department; working
with the Department to define requirements
and finalize schedules, planning documents,
and specification documents; coordination of
test delivery, statistical and psychometric services, test development, operations, and score
reporting activities; operational logistics asso-
ciated with rollout of the system; overseeing

the printing, packaging, shipping, receipt, processing, and scoring of the paper-and-pencil
operations; development of appropriate manuals and documentation; and planning and oversight of logistics and materials for training
events.
Florida Policy Advisor and Director,
Interstate Relations. AIR has included
resources for a half-time senior staff member
to serve as a project advisor on policy and to
coordinate communications within Florida.
AIR will work closely with the Department to
identify this person.
Tamonica Jackson, Administrative Assistant,
Meetings. Ms. Jackson has more than 17 years
of experience in planning and managing conferences, including more than five years of
experience in statewide assessment projects in
Hawaii, Delaware, and Minnesota. On these
projects she is responsible for working directly
with the clients, managing conference budgets,
hotel search and selection, contract negotiations, vendor negotiations, and travel arrangements and reimbursements. Ms. Jackson plans
and manages more than 50 meetings per year
for participants ranging in number from 10 to
500. These include TAC, test steering, test
content (item and passage), fairness and sensitivity, rangefinding, rubric validation, data
review, standard setting, contract management, and various advisory committee meetings. Ms. Jackson has convened assessmentrelated meetings in AIR offices, hotels, and
conference facilities. She assists with registration services, including name badges, tent
cards, signs, and participant lists, and ensures
the accuracy of custom and/or individualized
packets. Ms. Jackson maintains and updates
systems for participant travel receipts, itineraries, and reimbursements. She negotiates
with hotels and service vendors, coordinating
facility and audiovisual equipment arrangements. Her other responsibilities include
preparing and collating materials for the distribution of promotional, registration, and other
mailings. Ms. Jackson has completed coursework in Business Management from Morgan
State University.
Matthew Hirsch, Administrative Assistant.
Mr. Hirsch, who has completed three years of
classes in computer science at the University
of Maryland, has supported assessment projects for more than five years at AIR. He has
supported multiple assessments in Ohio,
including the high-stakes graduation test. He
supports the senior program manager in all
project management activities, including meeting agenda, minutes, and associated documentation. He supports integrated scheduling
across all AIR teams and subcontractors, and
ensures that materials move to and from the
client using established protocols. He has
become expert in AIRs student registration
system from the project management perspective, working with the senior program manager
in reviewing specifications, responding to any
district or school upload issues escalated to
the senior program manager, and monitoring
orders for paper testing materials to ensure
they are within expected quantities.
Test Development
Sandra Durden, M.A., Project Manager, Test
Development. Ms. Durden has more than 20
years of experience in the field of education,
including 10 years in the assessment industry
and 10 years as a K8 educator. At AIR, she is
working on performance tasks for the Smarter
Balanced Assessment Consortium and item
development for state programs and K2
formative assessments. Prior to joining AIR,
Ms. Durden worked at Pearson and Harcourt,
where she served most recently as director of
research and development and was responsible
for all phases of new product research, design,
and development for learning assessments. In
her previous roles as senior assessment design
Page 4-13
manager, senior assessment specialist, and

content lead, she helped develop research
plans, content frameworks, test blueprints, test
and scoring designs, and asset development
plans. In addition, she managed content development and construction of operational and
field-test forms for various statewide assessment programs and national educational
assessment products. Ms. Durden taught
kindergarten and grades 3, 4, and 5. She also
worked as a middle school mathematics
teacher for grades 6, 7, and 8. Ms. Durden
holds an M.A. in Mathematics Education from
the University of the Incarnate Word, a B.A.
in Elementary Education from Saginaw Valley
State University, and a B.A. in Government/
Prelaw from Evangel University.
ELA
Diana Reed, Ed.M., ELA Content Lead.
Ms. Reed combines 10 years of experience in
teaching and in developing large-scale assessments. At AIR, Ms. Reed currently serves as
internal ELA content lead for AIRs project
supporting the Smarter Balanced Assessment
Consortium, overseeing AIRs development of
over 1,500 items. Previously at AIR, she was a
grade level lead for the Minnesota State
Assessment System. In that role, she supervised forms development for the middle and
high school grades, working with both psychometrics and client content leads to create
forms. Prior to her work at AIR, Ms. Reed
taught English language arts in the Boston
Public Schools for 6 years. She mentored
teaching interns from Harvards Graduate
School of Education and from the Boston
Teacher Residency. Previous teaching experience came from 4 years as a founding teacher
at the Girls Middle School, where she served
as technology director, computer science
teacher, and language arts teacher. Before
teaching, Ms. Reed worked as a software engineer at Intermetrics, Inc., where she developed software that allowed students to create
Page 4-14
online storybooks and interactive programs.

She holds a B.A. from Brown University, where
she double-concentrated in English and
American Literature and Computer Science.
She received her Ed.M. in Teaching and
Curriculum from Harvard University.
Leads, Grades 35: Christine Keller, Terry Hill,
and Allison Stingley
Christine Keller, M.A.T. Ms. Keller joined AIR
with 9 years of educational experience as a
K6 and ESOL instructor. In addition to teaching elementary school for 7 years in
Pennsylvania, North Carolina, Florida, and the
District of Columbia in a range of socioeconomic schools, she is also certified in English
as a Second Language (ESL) and Reading for
grades K8. Ms. Keller worked as an ESL-inclusive classroom teacher for 5 years before
teaching abroad at the International School of
Krakow in Poland and the Changwon City
Public School system of South Korea. Upon
returning to the United States, she worked as a
collaborative learning ESL instructor with the
District of Columbia Public Schools. At AIR,
Ms. Keller has led ELA diagnostic development
for Common Core State Standards-aligned
assessments in Ohio and also developed new
items for Oregons English Language
Proficiency Assessment (ELPA) and Utahs
Student Assessment of Growth and Excellence
(SAGE) ELA assessments aligned to the Utah
Core Standards. She both develops and
reviews new items ensuring the accuracy of
item content, scoring, keying, fairness, language, and accessibility. She has led item
writer training, constructed forms to blueprints, and facilitated meetings for content,
fairness, bias and sensitivity, rangefinding, and
data review. At AIR, she also directs the development of K2 English/language arts diagnostic
assessment items and forms. She received her
M.A. in Elementary Education and her B.A. in
Liberal Arts.
Terry Hill, B.A. Mr. Hill has over 10 years of

experience in ELA assessment, with the last 5
years concentrating in passage and item development. He has a B.A. in Professional English.
He is currently the content lead for the
Minnesota project and the content lead for
grades 37 in the Utah SAGE project. He has
also worked on the Smarter Balanced project
and assessments for physical education and
visual and performing arts. Prior to joining
AIR, Mr. Hill worked at Measurement
Incorporated, first as a team leader for the
handscoring team, then as an editorial assistant, and finally as an ELA content specialist.
There he worked on projects for the states of
Ohio, New Jersey, and Texas and helped develop an internal item bank. He received his B.A.
in Professional English.
Allison Stingley, M.A.E. Ms. Stingley is currently a test development specialist at AIR and
serves as the ELA content lead for the contract
with the Hawaii Department of Education for
both the Hawaii State Assessments and the
End-of-Course Assessments. Ms. Stingley
serves as the assistant item development manager for the Hawaii assessments. She is also
the writing content lead for the Utah SAGE
assessments. Ms. Stingley directs the development of K12 English/language arts passages,
items, and writing prompts and trains junior
staff members and outside vendors. She has
been responsible for the development and
review of approximately 500 innovative items
and passages (technology-enhanced items and
machine-scored essays) and for facilitating
approximately 20 committee meetings (such
as content review, bias and fairness review,
rubric validation, and data review) with
clients, community members, and teachers.
Ms. Stingley has experience building both
online and paper/pencil forms for Minnesota
and Hawaii. She has experience writing items
and creating blueprints for Minnesota and
Hawaii state standards and Common Core
State Standards. Prior to AIR, Ms. Stingley
worked as a product developer for Pearson in

their curriculum and textbook division and
taught grades 11 and 12. Ms. Stingley holds an
M.A.E. in Literature from DePaul University
and a B.A. in Secondary Education and English
from the University of Illinois at Chicago.
Leads, Grades 68: Allen Chong, Meghan
Mulhern, and Austin Buckley
Allen Chong, M.Ed. Mr. Chong is currently a
test development associate at AIR, primarily
working for the Smarter Balanced Assessment
Consortium project. Prior to his position at
AIR, Mr. Chong served as a test development
specialist at the Center for Applied Linguistics,
in the area of second language acquisition. Mr.
Chong also has two years as a teacher in a full
inclusion, dual immersion (Spanish-English)
classroom in the District of Columbia Public
Schools system. Mr. Chong received his M.Ed.
from the George Washington University and a
B.A. in Political Science from Vassar College.
Meghan Mulhern, Ed.M. Ms. Mulhern has
more than 5 years of experience in the field
of education, including 2 years of classroombased experience and over 3 years in teacher
preparation. As an ELA test developer with
AIR, Ms. Mulhern serves as the listening
development lead for the Smarter Balanced
Assessment Consortium. In this role, Ms.
Mulhern has helped manage the development
of over 330 listening stimuli and over 1,000
associated listening items to assess students
ability to analyze point of view, purpose, and
quality of spoken texts and to draw and support conclusions based on the content of spoken presentations. Ms. Mulhern has also written and reviewed items for projects in
Delaware and Hawaii and assisted in the development of state-approved ELA item specifications for the Common Core State Standardsaligned assessment for Utah. She also serves
as a content team liaison with AIRs item tool
developers and has provided training for both
Page 4-15
internal and external item writers on AIRs

technology-enhanced item types. Ms. Mulhern
holds an Ed.M. in Language and Literacy and a
B.A. in Linguistics.
Austin Buckley, B.A. Ms. Buckley is a test
developer at AIR and serves as the ELA content lead for the Delaware Department of
Education contract where she leads the development of items and passages across grades
212. Additionally, she also serves as grade
level lead for grades 4 and 5 for Smarter
Balanced listening development, serves as
grade level lead for grades 3 and 4 of internal
Common Core State Standards development,
and contributes to Hawaii End-of-Course
development for expository writing. Ms.
Buckley creates innovative items for clients
and proposals and leads committee meetings
with clients and teacher panels. Ms. Buckley
holds a B.A. in English Language and
Literature from the University of Maryland.
Leads, High School: Mark Lovely, Sean
Redmond, and Charles Ferry
Mark Lovely, M.F.A. Mr. Lovely has over
8 years of experience in secondary level
assessment. At AIR, Mr. Lovely is currently
the performance task lead for the Smarter
Balanced Assessment Consortium, overseeing
the development of items and stimuli in accordance with the Common Core State Standards.
Since joining the staff of AIR in 2011, Mr.
Lovely has developed a variety of test materials for Ohio, Delaware, Minnesota, and Hawaii
and has served as the lead developer for the
Ohio Achievement Assessments, grades 68.
Previously, Mr. Lovely worked at Measured
Progress in the departments of Scoring and
Curriculum and Assessment. Before entering
the assessment field, Mr. Lovely taught for 7
years at several New England-area colleges,
specializing in developmental reading and writing and beginning to intermediate composition. He received his M.F.A. in Fiction Writing
Page 4-16
from the University of Iowa and his B.A. in

English.
Sean Redmond, M.A. Mr. Redmond has 9
years of ELA test development experience at
AIR, WestEd, and Riverside Publishing and is
currently ELA content lead for Minnesotas
assessment programs. His expertise includes
writing and reviewing online (including technology-enhanced) and paper-and-pencil test
items and passages for 12 states throughout
the country, resolving client feedback issues,
and building forms. He has also run CAC, bias
and sensitivity, and standards setting meetings
for various states. Immediately prior to joining
AIR, Mr. Redmond developed the verbal portion of the Cognitive Abilities Test 7 (CogAT 7)
for Riverside, working on the National Item
Tryout through Standardization and the current operational forms. Mr. Redmond designed,
orchestrated, and summarized the CogAT
Fairness Review of NIT items, working with a
diverse group of educators from several countries. He has classroom teaching experience
from elementary to college levels. He received
his M.A. in English Literature and his B.A. in
English and Secondary Education from the
University of Michigan.
Charles Ferry, M.A.T. Mr. Ferry brings extensive experience in education and educational
assessment to Florida. At AIR, Mr. Ferry develops, edits, and reviews content for various
English language arts projects. As a member of
AIRs Common Core State Standards team, he
is fluent in the standards and was integral in
the development of the passage and item specifications that are used to guide the development of the assessment aligned to Utahs Core
Standards SAGE ELA assessment. He also
serves as the eighth and ninth grade level lead
for SAGE. He has been involved in developing
and editing a wide range of middle and high
school level content for the Smarter Balanced
Assessment Consortium, as well as numerous
other state projects. He has led content and
fairness committee meetings and has attended

rubric validation meetings. Prior to joining
AIR, Mr. Ferry served as an English language
arts teacher in the District of Columbia Public
Schools, where he developed and implemented
lesson plans and curricula aligned to the
Common Core State Standards. Mr. Ferry
received his M.A.T. in Secondary Education
and his B.A. in English Literature.
Math
Jim McCann, M.S., Mathematics Content
Lead. Mr. McCann has over 8 years of experience in education, including more than
5 years in test development at AIR. During his
time with AIR, he has served as a test developer and grade level lead for Ohio, Hawaii, and
Delaware, as well as content lead and item
development manager for the alternate assessments of South Carolina, New Mexico, and
Delaware. As item development manager for
the Delaware alternate assessment, he supervised the transition of the alternate assessment
from a portfolio-based system to a performance task assessment. This required managing
the development of test specification, blueprints, and a bank of field-test tasks for all
content areas. Prior to joining AIR, he was a
fifth grade teacher in Stafford County, Virginia.
Mr. McCann received his M.S. in Educational
Psychology with a concentration in
Assessment, Testing, and Evaluation from
George Mason University and his B.S. in
Elementary Education with a concentration in
Mathematics from the University of Maine.
Leads, Grades 35: Eileen Heneghan and
Maureen Font
Eileen Heneghan, B.S. Ms. Heneghan has
7 years of mathematics education experience.
At AIR, she serves as a test developer and is
responsible for writing, reviewing, and editing
math items for various AIR projects. For the
Smarter Balanced Assessment Consortium per-
formance tasks, she is the lead developer of

18 classroom activities, which provide the necessary scaffolding for completing the tasks.
Prior to joining AIR, Ms. Heneghan worked at
Pearson Education, working on the creation,
release, and maintenance of the MathXL and
MyMathLab products, a series of online courses which provide students and instructors with
machine-scored constructed-response problems that correlate to textbook problems. She
also has experience working as a teaching
assistant for a variety of university courses,
including intermediate algebra, linear algebra,
and discrete mathematics. Ms. Heneghan is
currently pursuing a M.S. in Pure Mathematics
at DePaul University (expected May 2014).
Maureen Font, M.Ed. Ms. Font began working
for AIR in 2011 after graduating in 2007 from
the University of Illinois with an Elementary
Education degree (along with a concentration
in Mathematics and a minor in Italian). She is
currently AIRs mathematics content lead for
the Smarter Balanced Assessment Consortium,
working on grades 311, for the upcoming field
test. She was responsible for delivering about
1,000 performance task items for the consortium and managing the internal development
process, as well as coordinating many development tasks with other vendors and the client.
As a mathematics content lead for Minnesota,
working on grades 511, Maureen has been
responsible for delivering about 500 items
each year. Maureen was a middle school math
educator for grades 68 in Illinois for 4 years
before becoming a freelance SPSS data analyst
and a program evaluation intern at ChildServ
in Chicago.
Leads, Grades 68: George Ligon, Kari
Thicksten, and Natalie Morgan
George Ligon, M.Ed. Mr. Ligon brings over
8 years of educational experience to AIR,
where he is responsible for writing, reviewing
and editing math items for various AIR proj-
Page 4-17
ects that include state assessments for Hawaii,

Ohio, Minnesota, Delaware, and Smarter
Balanced Assessment Consortium. Prior to
joining AIR, Mr. Ligon worked as a certified
teacher in the areas of middle school mathematics, special education, health, and physical
education. Mr. Ligon has worked with the
School District of Hillsborough County,
Florida; Prince William County Schools,
Virginia; and the District of Columbia Public
Schools (DCPS). During his teaching experience, Mr. Ligon taught grades 68 mathematics, algebra I, and personal fitness. He also
worked as a special educator, developing individual education plans (IEP) and leading IEP
meetings for special education students. Mr.
Ligon was the math department chair for a
K8 educational campus with DCPS and was a
member of the Common Core Math Corp, a
group of DCPS educators who worked with
central office leadership to design, plan, and
implement the Common Core State Standards
math rollout. Mr. Ligon received his B.S. in
Physical Education with a concentration in
Exercise Science from the University of
South Florida and his M.Ed. in Educational
Leadership with a concentration as a
Mathematics Specialist Leader (K8) from
George Mason University.
Kari Thicksten, M.S. Ms. Thicksten has over
10 year of experience in education in grades
K8, specializing in math. Since joining AIR in
2012, Ms. Thicksten has served as a grade
level lead for both the Minnesota and Utah
projects. She has written and reviewed math
items of all grades levels, including 800 previously written items that were realigned to
common core for the Utah project. She lead
several successful development cycles, producing 300 items for Minnesota and 180 items for
Utah in the latest cycles. Ms. Thicksten has led
six content and fairness committee reviews for
various projects. Ms. Thicksten is a National
Board certified teacher in the area of early
adolescent mathematics. She holds a M.S in
Page 4-18
Educational Administration and a B.A. in

Elementary Education with a minor in math
from Carthage College.
Natalie Morgan, B.S. Ms. Morgan is currently a
math test developer at AIR and is responsible
for writing, reviewing, and editing math items
for various AIR projects. With a focus on algebra, functions, expressions, and equations, she
has written and reviewed math items for all
grades levels. She has also assisted with aligning over 800 previously written items to the
common core. Ms. Morgan has served as a
Minnesota grade level lead for grades 78 and
Utah grade level lead for grades 68. She has
managed the development of over 500 items
for grades 68. In addition, Ms. Morgan has
experience leading client facing committee
meetings and resolving client feedback. Prior
to joining AIR, Ms. Morgan worked as an
eighth grade mathematics teacher in the
Baltimore County Public School system and
continued in that position for 6 years. She
has experience teaching a variety of math
courses from eighth grade math to algebra II.
In the Baltimore County Public School system,
Ms. Morgan worked as an eighth grade team
leader and acted as a liaison between
eighth grade staff and administrators; was a
University of Maryland, Baltimore County, student teacher mentor; and was a 24 Club advisor. Ms. Morgan holds a B.S. in Education from
Slippery Rock University and is currently
working on her M.B.A. with an emphasis in
leadership.
Leads, High School (Algebra I, Geometry, and
Algebra II): Katerina Sagris, Chris Kincheloe,
and Roshank Matewere
Katerina Sagris, M.A. Ms. Sagris has more
than 9 years of mathematics education experience, with 3 years of experience in educational
assessment. At AIR, Ms. Sagris currently serves
as the algebra I grade level lead for the Hawaii
End-of-Course Assessment, serves as the grade
68 lead for the Minnesota State Assessment,

and is responsible for all current form building
for Delaware. In her roles at AIR, Ms. Sagris is
responsible for writing, reviewing, and creating
scoring logic for thousands of items. She has
facilitated client meetings for middle school
item reviews in Utah. Prior to joining AIR, Ms.
Sagris worked as a test development specialist
at Riverside Publishing in Illinois. In that position, she served in key roles on many projects,
including Detroit and Missouri. Ms. Sagris
taught high school mathematics and ELA for
5 years with the Chicago Public Schools and
for 1 year at East Leyden High School in
Schiller Park, Illinois. Ms. Sagris received her
B.A. in secondary mathematics and English
education at Concordia University, Illinois,
and her M.A. in Education with an English
language learner teacher certification at
Concordia University, Illinois.
Chris Kincheloe, B.S. Mr. Kincheloe has over
5 years of mathematics education experience.
At AIR, he serves as a test developer and is
responsible for writing, reviewing, and editing
math items for various AIR projects, including
assessments for Utah, Minnesota, and the Ohio
Graduation Test (OGT). He has participated in
a content advisory committee meeting and in
building forms for the OGT paper test. Prior to
joining AIR, Mr. Kincheloe worked at Pearson
Education, where he created, tested, and edited online mathematics tutorials and questions
for MyMathLab and MathXL, two online homework and test platforms. Mr. Kincheloe
received his B.S. in Mathematics from the
University of Illinois at Urbana-Champaign.
Roshanak K. Matewere, B.A. Ms. Matewere
has 5 years of high school mathematics education experience and nearly 3 years working in
education assessment for AIR. In her current
role, she writes, reviews, and revises math test
items; resolves team and client feedback on
test items; and participates in developing
machine-scored constructive-response items.
She has been involved in the management of

the item development for Hawaii, Minnesota,
and Utah and the development and alignment
of items to the Common Core State Standards.
She also has experience leading client facing
committee meetings. Mrs. Matewere came to
AIR from the classroom, having taught for the
Roanoke City Public Schools for 3 years and
volunteered with the Peace Corps for 2 years
in Malawi, Southeast Africa. She received her
B.A. in Mathematics from George Mason
University in Fairfax, Virginia, in 2000 and is
currently working on her masters in Public
Policy with an emphasis in Social Policy from
George Mason University in Arlington, Virginia.

Harold Doran, Ph.D., Lead Psychometrician.
Dr. Doran is the proposed Psychometrics and
Statistics team leader for AIRs support of the
Florida Standards Assessments and is responsible for all work conducted by the psychometricians. Working closely with Dr. Gary Phillips,
AIRs director for psychometrics and chief psychometrician, Dr. Doran will apply all of the
Department-approved technical procedures,
finalize all sampling plans, and have final
approval over all analyses before submission
to the Department. Dr. Doran will also lead
and/or oversee the validity work. He will work
in close collaboration with the Department
and Floridas external psychometric quality
control contractors. In addition to supporting
the psychometrics of large-scale assessment
programs, Dr. Doran is an expert in valueadded models, growth models, psychometrics,
and state assessment policy. Dr. Doran has
helped states complete studies needed for a
successful assessment peer review with the
federal government, including work for the
District of Columbia, Delaware, Hawaii,
Indiana, Michigan, Oregon, and Wisconsin.
Dr. Doran was appointed by then-Secretary of
Education Margaret Spellings to serve on the
U.S. Department of Educations National
Page 4-19
Technical Advisory Committee, which advises

the department on complex technical issues
involving the design and implementation of
state standards, assessments, and accountability systems. He is the author of the MiscPsycho
package in R, which provides miscellaneous
psychometric functions for statistical analysis
including item response theory estimation.
Elizabeth Ayers, Ph.D., Psychometrician/
Statistician. Dr. Ayers runs operational psychometric work for the Minnesota testing program, where she performs operational and
field test calibrations, conducts post-equating,
and prepares materials for data review and
standard setting. She is also responsible for
updating analysis and scoring specification
files as well as technical reports and summaries. Prior to joining AIR in early 2012,
Dr. Ayers was a post-doctoral researcher at the
University of California, Berkeley. While there,
she worked for Mark Wilson as project manager for the IES funded Assessing Data Modeling
and Statistical Reasoning project. The project,
a collaboration with Vanderbilt University,
integrates cognitive theories of learning about
data and chance with psychometric approaches to modeling student knowledge and skill. In
addition, Dr. Ayers worked with Sophia RabeHesketh on extensions of a class of measurement models known as cognitive diagnosis
models. This research included model development, estimation using Markov chain Monte
Carlo, evaluation of estimators in a variety of
scenarios using simulation studies, and application of the methods to real data from the
ASSISTment tutor, an online mathematics
tutor. Elizabeth holds a Ph.D. in Statistics from
Carnegie Mellon University.
Psychometric Research Associates/Assistants
AIR has a team of experienced research associates on our psychometric and statistics team
who provide essential support throughout the
Page 4-20
project life cycle, from forms development and

item analyses to standard setting and reporting. For example, they review forms and actually take the completed test using strict protocols before deployment or distribution of
assessments. They support all aspects of standard setting and production of technical
reports.
Computer and Statistical Sciences

Center (Software)
Selina Tolosa, M.B.A., Director, CSSC. As
described in the Offeror Profile, Ms. Tolosa
leads AIRs Computer and Statistical Sciences
Center and has served as team leader for the
development and implementation of assessment software systems at AIR for 7 years,
including AIRs online test delivery engine and
pre-ID and student tracking for all of AIRs
clients. Prior to joining AIR, Ms. Tolosa was
the director of information services and the
financial officer for New American Schools, a
nonprofit funding the development of schoolwide designs aimed at transforming elementary
and secondary schools. Ms. Tolosa brings experience in financial forecasting and analysis;
quality assurance of government enterprise
systems; and the application of database skills,
statistical sampling, and relational databases to
data organization of assessment systems. Ms.
Tolosa received her M.B.A. from the Wharton
School of Business, University of Pennsylvania.
AIR is proposing to implement our proven test
delivery, item banking/form construction, student registration, and related systems to support the Florida Standards Assessments. These
systems will be configured and deployed for
Florida by the following team, which brings
experience integrating AIR systems with DRC
systems for paper-and-pencil operations and
scoring. Each software system at AIR has a
lead software project manager and a lead or
senior application developer.
Software project manager. In addition to

the technical lead, each system or family
of systems has a software project manager.
The software project manager (SPM) is
responsible for ensuring that appropriate
protocols are followed, schedules are met,
and the software developers have the
resources they need to meet their schedules. The SPM serves as the primary liaison
to clients and the rest of the project team.
The SPM owns the requirements document
and the final test plan, working with the
developers, quality assurance team, clients,
and senior program manager to ensure that
the requirements are clear, accurate, and
sufficient and that the test plan adequately
addresses all requirements. He or she also
manages the development schedule, ensuring that resources are allocated appropriately to meet the schedule, notifying the
senior program manager of variances from
the schedule, and working with the senior
program manager and development team to
remediate any deadlines that are likely to
be missed. Finally, the SPM documents
requested changes to the production systems and the testing of those changes,
receives technical lead approval for the
requested changes, and submits changes
to the Production Control Board.
Technical lead/senior application developer.
This high-level software engineer takes
responsibility for the design and implementation of the systems in his or her area. For
each new system or modification to a system, the technical lead works with the
team to develop the overall architecture;
assign components or modules to senior
developers to design, review, and approve
detailed requirements and design documents; review and approve unit test and
integration test plans; and approve all
requests for deployment changes going to
the Production Control Board.
The senior project manager and senior

application developer for each system are
as follows:
Test Delivery
Jeremy Fisher, Software Project Manager,
Test Delivery, M.S. (systems development
management), B.S. (information systems
and technology), 6+ years of experience
configuring and delivering TDS in in
20 states.
Aaron Newman, Senior Application
Developer, M.S. (Electrical Engineering
and Computer Science), 20+ years systems
engineering experience; leads high-performance, cross-browser (including mobile)
user interfaces for AIR assessments
accounting for accessibility, including keyboard accessibility, text-to-speech, ASL,
visual impairments, and other special
accommodations.
Student Registration
Shailesh Gattewar, Software Project
Manager, Student Registration, M.B.A. and
M.A. (computer science), PMP, 9+ years of
experience as a software project manager
and engineer, including 6 years at AIR.
Anil Chevuri, Senior Application
Developer, B.E. (electrical engineering),
7+ years of programming experience,
including 6 years on AIRs student registration systems for online and paper testing.
Item Banking (ITS)
Zorina Alliata, Software Project Manager,
Item Banking, M.B.A., B.S. (computer science), PMP, 15+ years of experience as a
programmer and software project manager
directing teams of developers and software
support staff.
Page 4-21
Dipesh Mamtora, Senior Application

Developer, M.S. and M.B.A. (electrical
engineering), 10+ years of experience as a
senior software engineer, including 6 years
at AIR managing three developers in delivering online testing for AIR assessment
clients.
Analysis Systems (Quality Monitor and
Database of Record)
Brian Kane, Software Project Manager,
Ph.D., PMP (Astronomy), experience in
mission-critical IT systems and statistical
analysis of large data sets in multi-year longitudinal studies.
Paul Van Wamelen, Senior Application
Developer, Ph.D. (mathematics), AIR senior computational statistical specialist with
15+ years of experience in computational
number theory; manages the analysis team,
which develops and supports the quality
analysis and database of record systems.
Under these two main roles, lead developers
are responsible for the actual creation of
design documents and software, as well as
addressing any issues that arise with the production system. Within a team of developers,
the lead developer distributes work assignments, oversees the work of other developers,
and implements software. The lead developer
is responsible for technical direction and the
production of design documents from requirements documents, distribution and oversight
of work assignments to the development team,
review of software modules developed by the
development team, direct development of
some modules, and contribution to and technical review of test plans.
Finally, each system has data analysts responsible for preparing data in accordance with
analysis specifications. In a primarily online
program, much of the data analysis is automated, and the data analysts two main roles
Page 4-22
involve preparation of data for the field-test

analyses and special studies and quality assurance investigations. For this program, the data
analysts main responsibilities include extracting item tryout data, ensuring their quality,
and preparing them for analysis by the psychometrics and statistics team; providing periodic
quality assurance checks to validate automated quality assurance systems; and preparing
analysis files for special analyses, validity studies, and other data sets that may be required
by the client or the psychometrics and statistics team/contractor.
Software Quality Assurance for Systems
Software quality assurance (QA) is a critical
function. AIR maintains a substantial QA
team, and our QA process involves everyone
from the senior program manager through the
developers and QA engineers and leads. The
QA process begins with requirements definition. As requirements for a system or modification are developed, the SPM works with the
QA engineer and lead to begin development of
the QA design. Once requirements are complete, the SPM, QA lead, QA engineer, and lead
developers work together to produce a multifaceted QA plan, which is then implemented
during development (unit testing) and after
development (integration testing).
Dorian Sofiaj, M.S., Software Quality
Assurance Lead. As the lead of software quality assurance, Mr. Sofiaj is responsible for
ensuring that every AIR system deployment
meets specifications, functions, provides a satisfying user experience, and accurately captures or generates data. He has test responsibility for every system, including AIRs system
for item banking, test delivery, electronic
reporting, and websites. He develops plans for
load testing, interface/functional testing, and
data and calculation model testing. He ensures
that every system in development has a thorough testing plan that includes automated and
thorough regression testing at scale (rather

than on small datasets) and that every system
documents procedures to ensure security and
confidentiality. Mr. Sofiaj joined AIR after 8+
years in software QA at Fannie Mae, where he
progressed to senior technical risk specialist.
At Fannie Mae he had managerial and handson experience with all aspects of testing,
including front end and back end, white box
and black box, system, regression, failover,
shakeout, and performance testing. He has two
M.S. degrees (computer science and physics)
and a B.S. degree (mechanical engineering).
He has extensive additional training in
Oracle/PLSQL, Java, Unix, security, test
methodologies, SDLC, management, culture,
diversity, and other subjects. Mr. Sofiaj leads
AIRs QA team and is responsible for the technical quality of QA plans and their successful
implementation, as well as the management
and scheduling of the QA testers. He designs
the overall architecture of the QA plan, which
is embodied in the QA design document,
reviews and approves detailed QA plans, and
allocates human and hardware resources to
implement the QA plan. Under his direction,
individual QA leads work with the SPM to draft
the QA plan and oversee its implementation.
The QA lead participates in the development
of the QA design, works on the team led by the
SPM to develop the QA plan that implements
the QA design, and implements the QA plan,
writing automation scripts and overseeing the
work of a team of QA testers. Our QA leads on
this team have bachelors degrees (e.g., B.S.,
commerce with statistics and computers; B.E.,
electronics and communications), have 8+
years of experience managing software defecttracking processes, and often hold relevant
Microsoft and Cisco certifications.
Network Engineering
As described in our technical approach, AIR
supports application hosting with both external and internal resources. AIR hosts our ITS,
TDS, reporting, and related critical assessment

systems and websites in a managed hosting
facility, because the scale and investment
available from a dedicated hosting company
far exceed what any one company can provide.
We use several of Rackspaces managed hosting
facilities to host the online testing for Oregon,
Delaware, Hawaii, Minnesota, and other smaller efforts, as well as other core AIR assessment
systems. Rackspace is rated one of the Magic
Quadrant leaders by Gartner Research.
Internally, AIR has our own staff of network
engineers dedicated to our assessment infrastructure. This team is led by Graham
Tapscott, AIRs Network Engineering Lead,
who has 12+ years of experience in network
design and maintenance, including 8 years as
network engineer for AIRs online testing system. His team includes six additional network
engineers, with a combined 60+ years of experience in network engineering, administration,
and management for 24/7 mission-critical
operational environments, and multiple fulltime and part-time tier-2 network help desk
support staff.
Computer-Based Testing
Stacy Tsakeris, M.B.A., Project Manager,
Computer-Based Testing. At AIR, Ms. Tsakeris
is currently managing the quality control for
the Smarter Balance computer-based testing
across all AIR systems for a consortium of 24
states. This effort includes planning, budgeting, and tracking testing activities, as well as
coordinating with AIR software project managers and software engineers for student registration, test delivery, quality assurance, and
related systems to ensure quality control. In
addition, she is managing the documentation
for the Open Source testing software that AIR
is developing and delivering to the Open
Source repository, which will be available to
all Smarter Balanced states. Before joining
AIR, Ms. Tsakeris was a consultant who managed various projects in finance, marketing,
Page 4-23
education, and the public sector. Most recently, she managed the University of Californias
Research and Grants Program Office (RGPO)
Grants Administration and Management software upgrade. She received her M.B.A. from
the University of Maryland and her B.S. in
Computer Science and Engineering from The
Ohio State University.
Test Administration
Meredith Durgin, B.S., Project Manager, Test
Administration. Ms. Durgin has 10 years of
management experience in educational assessment, including managing multimillion-dollar
projects. At AIR, Ms. Durgin has served as a
senior manager for Oregons very successful
online testing project. For example, she led
the development of manuals for the 201213
school year; developed the project implementation plan for the year; oversaw the annual
update of the data exchange interfaces; led
AIRs team in district test coordinator training
on student registration; completed UAT for
both assessments in ELA, math, science, social
studies, and English language proficiency; and
oversaw the launch of testing. Prior to joining
AIR, Ms. Durgin was a senior program manager
for Questar Assessment and Houghton Mifflin
Harcourt. At Questar, Ms. Durgin served for 2
years as the program director for the Missouri
End-of-Course Assessments, an online testing
program in algebra I and II, geometry, biology,
English I and II, and American History and
Government. Her responsibilities included liaison with state departments of education; vendor selection and evaluation; development of
program schedules and system design, quality,
and consistency; and preparation of materials
and data/reporting requirements. At Houghton
Mifflin Harcourt, Ms. Durgin served as senior
program manager for large-scale contracts for
assessment programs in Ohio, New Jersey,
New Mexico, Florida, Texas, Missouri,
Arkansas, Montana, and Utah, as well as shelf
product development. Her responsibilities
Page 4-24
included the supervision of a program management team to ensure that department and
company objectives, goals, and timelines were
met; coordinated communications; program
design; and scheduling activities across all
departments and client organizations. For the
Florida Reading First program, Ms. Durgin
managed the contract between Riverside and
the Florida Department of Education to support districts administration of the MacGinitie
Reading Test to measure growth in different
reading skills to help improve reading instruction in grades K3. Ms. Durgin provided training on administering the test, managing delivery and return of paper/pencil materials, monitoring scoring, and returning results. Ms.
Durgin received her B.S. in Elementary
Education from the University of Toledo.
Cynthia Benis, B.S., Help Desk Manager.
Ms. Benis manages AIRs help desk across all
assessment projects, with a focus on support
for online testing. Ms. Benis has 25 years of
customer service experience, including the last
5 years at AIR leading our help desk across
projects. She manages AIRs help desk knowledge bases, inquiry tracking systems, and telephone and electronic communications systems. She oversees full-day support across five
time zones for 10,000 monthly inquiries. She
manages a team that expands to approximately
50 staff members during peak periods, providing services from 5 a.m. EST to 10 p.m. EST
Mondays through Fridays, with occasional
Saturday support periods. As the help desk
manager, she establishes and enforces processes and best practices, adheres to proper training methodologies and provides staff training,
produces management reports, ensures the
call center adheres to standards, appropriately
communicates project statuses and issues with
program management, oversees our clients
service level agreements, and utilizes standard
help desk metrics to track and manage work
flows. Ms. Benis graduated from The Ohio
State University with a B.S. in Marketing.

Cathy Kugler, B.A., Project Manager, Scoring
and Reporting. Ms. Kugler has seven years of
assessment experience at AIR, primarily managing data files, team schedules, and deliverables for psychometric and reporting activities
on state assessment projects in Ohio and
Minnesota. Working closely with the AIR
Psychometrics and Statistics and Score
Reporting teams, Ms. Kuglers responsibilities
include coordinating analysis specifications
and reporting documents related to the statistical integrity of state assessments, along with
composing and editing technical reports. She
has developed and overseen quality control
procedures for the implementation of test
administration and scoring systems and has
performed quality control activities for test
construction, scoring, analysis, and score
reporting. She also has prepared materials
and coordinated logistics for item data review
meetings and standard setting workshops.
Earlier in her career, Ms. Kugler has served as
a statistical support associate and managed
tasks such as the transition of technical item
banks from the previous testing contractor
to AIR. Ms. Kugler received her B.A. in
Psychology from the University of Maryland.
Marta Gebrehiwot, B.A., Lead, Score
Reporting. AIR proposes that Ms. Gebrehiwot
will lead score reporting for the Florida
Standards Assessments score reports by working closely with Florida managers and with the
AIR Test Development and Psychometrics and
Statistics teams. Ms. Gebrehiwot will be
responsible for overseeing the design and the
printing and delivery of the paper reports.
Since joining AIR, Ms. Gebrehiwot has managed three reporting lines by overseeing the
day-to-day activities in designing, programming, and producing reports. Since 2010, she
has led the production of the state, district,
school, and family reports in South Carolina,
New Mexico, and other states and the develop-
ment and delivery of reports from a school climate survey of high school and middle school
students in Cleveland. Ms. Gebrehiwot, who
has more than seven years of experience in
administering standardized tests, including the
TOEFL and CLEP exams, received her B.A. in
Economics/Management magna cum laude
from Dickinson College.
Graphic Designers. As required in the ITN,
AIR presented the qualifications and experience of our graphic designers.
DRC Proposed Project Staff

AIR has proposed to subcontract with DRC for
the following activities:
All required hand-scoring of constructedresponse items, including associated
rangefinding.
Production, distribution, collection, and
processing of all paper/pencil testing.
Two of the research studies: 7.6.3.3,
Transitioning from FCAT 2.0 and EOC
Assessments to Florida Standards
Assessments (FSA)Linking the Scales,
and 7.6.3.6.1, Investigate the Comparability
between Devices Delivering Assessments.
In addition, AIR has included DRC test development specialists to ensure AIR delivers to
Florida the broadest range of new items possible.
AIRs Senior Program Manager, Kevin Murphy,
will have final responsibility for DRCs work.
As described above, DRCs day-to-day handscoring and paper/pencil operations will be
overseen by AIRs Corporate Director of
Operations, Dee Wagner. Dr. Harold Doran
will manage DRCs work on the research
studies, and our Project Manager for Test
Development, Sandra Durden, will ensure that
DRCs test development specialists are trained
and monitored.
Page 4-25
DRC Project Management

Peter Tressel, M.B.C., PMP, Project Manager,
Paper/Pencil Operations. Mr. Tressel is a
Director, State Assessment Programs at DRC
and will serve as DRCs Project Manager, overseeing and facilitating DRCs work on the
Florida Standards Assessments. Mr. Tressel has
nearly 15 years of project management experience directly related to large-scale, statewide
assessment programs. In his current role at
DRC, Mr. Tressel provides overall management
and direction for assessment programs in
Minnesota and Ohio. In addition to his work
in the testing industry, Mr. Tressel has more
than 20 years of related experience with a
variety of writing and editing projects and
activities, working with public sector clients
(state departments relative to insurance licensure testing) and private sector clients. Mr.
Tressel holds a masters degree in Business
Communications from the University of St.
Thomas in St. Paul, Minnesota, and a Bachelor
of Arts degree in Journalism from the
University of Minnesota. He is certified as a
Project Management Professional (PMP)
through the Project Management Institute.
DRC Hand-Scoring
Sherri Woolf, Vice President of Performance
Assessment Services. Ms. Woolf will oversee
all hand-scoring activities for the project. Ms.
Woolf has more than 20 years of educational
assessment experience. During her career at
DRC, she has managed performance assessment activities for numerous state clients,
including Alabama, Arkansas, Kentucky,
Louisiana, Minnesota, Nebraska, North
Carolina, Ohio, Oklahoma, Pennsylvania, and
South Carolina. In addition, Ms. Woolf leads
DRCs product development team for strategic
planning and execution in innovative items,
automated scoring, and artificial intelligence.
Ms. Woolf has been instrumental in assisting
several of DRCs state clients with transitioning
Page 4-26
to online assessments, and is keenly interested

in the pedagogical implications of online learning and assessment environments. She recently gave a presentation titled ConstructedResponse Scoring and Associated Technologies
to the Technology in Large-Scale Assessment
(TILSA) committee of the Council of Chief
State School Officers (CCSSO). Ms. Woolf
holds an M.S. in Human Resource Development
from Xavier University in Cincinnati, Ohio,
and a B.S. in Elementary Education from
Miami University in Oxford, Ohio. She is a
member of Pi Lambda Theta, the International
Honor Society and Professional Association in
Education.
Wendy Marik, Project Manager, Hand-Scoring.
A Senior Project Manager, Performance
Assessment Services at DRC, Ms. Marik has
14 years of experience working on large-scale
assessment projects. Her primary responsibilities include working closely with state department personnel and educators to develop
training materials for scoring writing assessments, monitoring hand-scoring projects and
supervising hand-scoring staff, and coordinating rangefinding activities. Ms. Marik has experience with numerous writing assessment models and rubrics. She has worked with clients to
implement transition from domain to holistic
scoring guidelines. Her prior experience as a
classroom teacher helped refine her knowledge
of curriculum and standards. During her time
at DRC, she has applied her expertise to many
statewide assessments in a variety of subject
areas, such as writing, mathematics, English
language arts, social studies, science, and practical living skills. Ms. Marik earned an M.Ed.
from Xavier University in Cincinnati, Ohio,
and a B.A. in Computer Science from DePauw
University in Greencastle, Indiana.
Pam Ahrens, ELA/Literacy Lead Scoring
Director. A Content Specialist for Performance
Assessment Services at DRC, Ms. Ahrens has
10 years of hand-scoring experience working
on large-scale assessment projects. Her primary responsibilities include working closely

with state department personnel and educators
to develop training materials for scoring
English language arts and writing assessments,
coordinating rangefinding activities in order to
ensure accurate and efficient scoring, monitoring hand-scoring projects, and supervising
hand-scoring staff. During her time at DRC,
Ms. Ahrens has applied her expertise to the
subject areas of writing and English language
arts, at various elementary and secondary
grade levels, as well as for alternative assessment student populations. She has worked
on testing programs for Alaska, Arkansas,
Louisiana, Minnesota, Ohio, Oklahoma,
Pennsylvania, the Smarter Balanced Assessment
Consortium, South Carolina, and Washington.
Ms. Ahrens prior experience as a classroom
teacher and librarian/information professional
helped refine her knowledge of educational
assessment, curriculum, and standards. She
holds an M.A. in Library Science from the
University of Minnesota, as well as a B.S. in
Teaching English and Spanish from Mankato
State University in Mankato,
Coskun Baysoy, Mathematics Lead Scoring
Director. A Mathematics Content Specialist
for Performance Assessment Services at DRC,
Mr. Baysoy has more than 10 years of progressive leadership experience in large-scale educational testing and performance assessment
operations. In his current role, Mr. Baysoy
manages all aspects of the scoring operations
for the Washington EOC Mathematics Exams.
He works closely with state department assessment specialists and educators during planning, development, and scoring phases of the
projects; manages internal teams responsible
for the development of training materials
through external rangefinding activities; develops and monitors scoring processes; and
ensures project tasks and milestones are successfully completed. Mr. Baysoy holds a Ph.D.,
ABD in Electrical and Computer Engineering
and an M.S.E.E. in Electrical and Computer

Engineering from the University of Minnesota
and a B.S.E.E. in Electronics and
Communications Engineering from Istanbul
Technical University in Istanbul, Turkey.
DRC Paper/Pencil Operations

Doyle Kirkeby, Senior Vice President of
Operations. Mr. Kirkeby will provide executive-level oversight of all operations functions
for the project. Mr. Kirkeby has more than 40
years of experience in corporate operations
management and software development. He
has effectively implemented projects and managed staff for a variety of corporate clients.
Mr. Kirkeby has been a senior leader in DRC
Operations since 1995. He currently serves as
the executive in charge of all phases of production scheduling, printing, packaging, materials
distribution and collection, warehouse operations, document processing (scanning, editing,
and clerical support), and hand-scoring.
Ginny Burnett, Senior Director, Operations
Education. Ms. Burnett will oversee scanning
and editing of all answer documents. Ms. Burnett
currently serves as DRCs Senior Director,
OperationsEducation, providing leadership
for the timely delivery and processing of education materials. With more than 27 years of
experience in operational activities, Ms. Burnett
demonstrates a commitment to efficient and
innovative management of large-scale assessment projects. Ms. Burnett has implemented
quality assurance procedures into the DRC
Operations environment and expanded the use
of technology into education operations quality functions. Her large-scale assessment experience includes projects for Alabama, Alaska,
Georgia, Idaho, Kentucky, Louisiana, North
Carolina, Ohio, Oklahoma, Pennsylvania,
South Carolina, and Washington. Throughout
her career, Ms. Burnett has significantly
improved business processes, integrated new
technical systems, and provided outstanding
project leadership.
Page 4-27
Doug Miller, Senior Director of Materials

Operations and Logistics. Mr. Miller will oversee all of the packaging, distribution, receipt,
and processing of test materials. Mr. Miller
manages all aspects of materials packaging and
logistics within the DRC Operations group. He
is also responsible for the tracking, receipt
processing, and inventory for all secure materials associated with DRCs educational assessment projects. Mr. Miller has been responsible
for, and successfully fulfilled, all DRC scheduled commitments for delivery of materials
and reports since 1998 and has set the bar for
exacting standards in test security. He continuously designs and augments existing QC procedures for materials handling operations
and regularly partners with the Information
Systems group to design and implement technological solutions to complex materials handling issues. Mr. Miller has been instrumental
in the development and continued enhancement of the Operations Material Management
System, DRCs proprietary materials tracking
and security system, and continues his central
role in its phased expansion. Mr. Miller holds
a Bachelor of Arts degree in English and
Psychology from St. Olaf College in Northfield,
Minnesota.
DRC Psychometrics
Yi Du, Psychometrician. Dr. Du will lead DRCs
work on the research studies. She is a highly
respected technical consultant with extensive
experience in all aspects of psychometrics,
specifically, IRT modeling. She has extensive
experience working with a variety of state
assessment programs. In her current role at
DRC, Dr. Du serves as the Lead Psychometrician
for state assessment projects in Louisiana and
Michigan and enhances the algorithms in
DRCs CAT. Prior to joining DRC, Dr. Du was
the director of research and evaluation for the
Edina Public School District and assistant
director and testing assessment specialist for
the Minneapolis Public School District. In
Page 4-28
addition to her work at DRC, Dr. Du teaches

multivariate statistics courses for doctoral programs at the University of Minnesota and the
Saint Marys University as an adjunct professor. She also provides psychometrics consulting for various institutional and licensure/certificate testing companies. Dr. Du holds both a
B.S. and an M.S. in Management Information
Systems from Beijing University and received
her doctorate in Measurement, Evaluation,
and Statistical Analysis (MESA) from the
University of Chicago where she studied with
Dr. Benjamin Wright for Rasch measurement
models, with Dr. Darrell Bock for 2-PL and
3-PL IRT models, and with Dr. Anthony Bryk
for hierarchical linear modeling.
Christie Plackner, Manager, Psychometric
Quality. Ms. Plackner will be responsible for
data forensics and psychometric quality.
Ms. Plackner has 17 years of educational
assessment and research experience. In her
current role at DRC, she directs quality initiatives for psychometric services and develops
psychometric quality plans for clients. Her
experience in large-scale assessment includes:
item analysis, calibration, linking, and sampling; conducting analyses related to technical
reports; contributing to special studies and
reports; facilitating standard setting sessions
and committee reviews; and assisting with the
development of test forms and providing psychometric sign-off. Ms. Plackner holds an M.A.
in Industrial/Organizational Psychology from
St. Marys University in San Antonio, Texas,
and a B.S. in Psychology from Valparaiso
University in Valparaiso, Indiana. She is currently pursuing a Ph.D. in Educational
Psychology focusing on quantitative methods
in education from the University of Minnesota.
DRC Test Development

Patricia McDivitt, Senior Vice President of
Curriculum, Instruction, and Assessment.
Ms. McDivitt will provide executive-level sup-
port and guidance for all test development

activities for the project. She has more than
25 years of direct, hands-on educational curriculum, instruction, and assessment development experience. Ms. McDivitt oversees all
phases of the item and test development
process for K12 large-scale assessment programs in content areas such as reading,
English language arts, science, mathematics,
and civics and government. She also oversees
the development of assessment ancillaries,
including the development of curriculum units
and lesson plans designed to link assessment
results to instruction. Ms. McDivitt has direct
CCSS experience with programs in Alabama,
Alaska, Louisiana, and Pennsylvania, providing
guidance and oversight for all of DRCs work.
Ms. McDivitt earned an M.S. in Education,
School Guidance and Counseling, and
Secondary Teaching in English Language Arts
with a Special Education Endorsement, and a
B.S. in Secondary Education English Language
Arts, both from Indiana University in
Bloomington, Indiana.
James Bell, Senior ELA Test Development
Content Specialist. Dr. Bell will lead ELA
development activities assigned to DRC.
Dr. Bell has more than 25 years of experience
in education and assessment. At DRC, he participates in all aspects of English language arts
test and item development. Dr. Bell has direct
CCSS experience with programs in Louisiana
and for the Smarter Balanced Assessment
Consortium. Prior to taking a full-time position
at DRC, Dr. Bell served as chair and associate
professor in the Department of English, Foreign
Language and Humanities at Northwestern
Oklahoma State University. He has also served
as director of institutional assessment and
associate professor at College of the Ozarks,
and as a freelance English language arts writer,
editor, and facilitator in assessment. He has
extensive experience as a writing consultant
for elementary and secondary schools, conducting workshops and on-site visits related to
standardized test development and review,

secondary writing center design, the writing
process, and authentic analogies for the writing process. Dr. Bell holds a Ph.D. in English
from Texas Tech University in Lubbock, Texas;
an M.A. in English and Drama from Texas
Womans University in Denton, Texas; and a
B.S. in Secondary Education in English and
Psychology from Lamar University in
Beaumont, Texas.
Anne Kirpes, ELA Content Specialist. Ms.
Kirpes will assist Dr. Bell with ELA development activities for DRC. Ms. Kirpes has 15
years of reading and language arts test development experience at all grade levels. As
DRCs Reading Test Development Director, she
provides subject-specific expertise relative to
all stages of the test development process. She
has worked on more than 10 statewide assessments, including providing leadership as team
lead and/or content lead. Ms. Kirpes has direct
CCSS experience working with Alabama,
Alaska, Louisiana, Pennsylvania, and the
Ms. Kirpes assessment experience ranges
from test design to standard setting, including
directing and participating in item/rubric
development and test construction, participating in performance assessment scoring
processes, facilitating and participating in standard setting, and facilitating rangefinding and
data committee reviews. In addition to her
comprehensive experience in large-scale
assessment, Ms. Kirpes has eight years of
teaching experience at the elementary level.
Ms. Kirpes earned an Ed.M. with a concentration in Teaching and Learning from Harvard
University Graduate School of Education. She
also holds a B.A. in Elementary Education,
Reading/Language Arts, from the University of
Northern Iowa.
John Selisky, Mathematics Content Lead.
Dr. Selisky will lead Mathematics development
activities assigned to DRC. Dr. Selisky has 17
Page 4-29
years of test development and educational

measurement experience and more than 25
years of teaching experience. In his current
role as Director, Test Development for
Mathematics, Dr. Selisky oversees all aspects
of mathematics item and test development. He
has written and developed a variety of ancillary products in support of assessment programs; conducted training presentations, committee reviews, and analytical procedures for
educators and measurement professionals; and
trained and mentored assessment professionals
in item writing and development, test construction, and data analysis. He has direct
CCSS and CCRS experience working with
Louisiana, Michigan, Pennsylvania, and the
Dr. Selisky holds an Ed.D. in Mathematics
Education and an M.A. in Instructional
Technology from the Teachers College,
Columbia University in New York, New York;
an M.A. in Secondary Education from Lehigh
University in Bethlehem, Pennsylvania; and
B.S. degrees in Earth and Planetary Sciences
and Mathematics from University of Pittsburgh
in Pittsburgh, Pennsylvania.
Christopher Peterson, Mathematics Content
Specialist. Mr. Peterson will assist Dr. Selisky
with mathematics development activities for
DRC. Mr. Peterson has six years of experience
in large-scale, statewide assessments and is an
important contributor to the test development
process at DRC. He has participated in DRCs
work for assessments in Alabama, Louisiana,
Michigan, Pennsylvania, and South Carolina.
Mr. Peterson has direct CCSS experience with
Louisiana, Pennsylvania, and the Smarter
Balanced Assessment Consortium. Mr. Peterson
helped establish the test development group at
DRC. He was responsible for quality testing
several assessment projects and working with
psychometric data to make informed decisions
about item and test development. Mr. Peterson
recently returned to the large-scale, statewide
assessment industry after spending eight years
Page 4-30
in the middle school mathematics classroom.

His experience there included teaching prealgebra and algebra for grades 6 and 7 students
in both the accelerated and the traditional
classroom. He has also worked extensively
with special needs students in grades 6
through 9 mathematics classrooms, including
teaching geometry curriculum. He holds a
M.Ed. degree in Teaching from the University
of St. Thomas in Minneapolis, Minnesota, and
a B.S in Mathematics from Wheaton College in
Wheaton, Illinois.
7.7.3 Subcontractors
As described throughout this proposal, AIR
will engage DRC as a subcontractor. We have
included documentation of DRCs experience
and personnel (e.g., resumes) alongside AIRs
experience and staff.
AIR understands we assume full responsibility
for all services and communications, including
those delegated to subcontractors. Further, we
understand any proposed changes in subcontractors must be approved by Florida in
advance and that no subcontracted work (or
any work) related to customer service or to
any secure document or data will be provided
by a persons or agencies physically located
outside of the United States.
In addition to DRC as our partner for handscoring and paper-and-pencil operations
(including the printing of all scannable documents), AIR expects to engage subcontractors
for the following activities:
Rackspace, for managed hosting services,
supports AIRs online testing in 20+ states.
Bradford & Bigelow and RR Donnelley, for
the printing of nonscannable materials,
have printed more than 100 million pages
of materials each year for secure testing
programs.
Automated Graphics Solutions (AGS), for

the printing of score reports, has printed
more than 4 million pages of score reports
for AIR each year, using AIRs documented
processes and strict quality control procedures.
United Parcel Service (UPS) and King
Solutions, for shipping, have supported AIR
and DRC in 15+ states.
American Printing House for the Blind
(APH), for Braille-related services, has successfully provided services to AIR and DRC
on multiple state projects.
We have included detailed processes to be
used with each of these subcontractors in the
relevant sections of the technical proposal.
Please see Exhibit 7.7.3-1 for the subcontractor roles and qualifications.
Finally, in addition to these subcontractors,

AIR also engages many vendors for various
services in support of projects. These include
communications (e.g., VOIP, WebEx, cell
phones), routine mailing (e.g., USPS, Federal
Express), internal copying (e.g., Xerox, Oc),
temporary employment agencies, supplies, and
many other services required to support our
staff and project work. Although not specifically procured for this project, AIR treats these
agreements in the same manner described
above at a corporate level. We procure these
services after careful analysis and/or competition, develop formal contractual agreements,
and have designated accountability and review
procedures.
AIRs successful subcontracting is based on

three critical components:
Clear delineation of responsibility and
expectations
Accountability mechanisms
Trust and integrity on the part of each
player
AIRs formal subcontract agreements outline
its commitments, which include an acknowledgment of the required documents, areas of
responsibility, and the products for which the
subcontractor is responsible. AIR staff review
subcontractor work and hold it to the same
high standards as AIR holds itself. This formal,
clear communication of expectations details
the means by which developed products will
be deemed acceptable. Please see Exhibit
7.7.3-1 for the role and qualifications of the
proposed subcontractors.
Page 4-31
Exhibit 7.7.3-1: Proposed Subcontractor Roles and Qualifications

Subcontractor/Role
Qualifications
AGS, part of the CGX family, is the worlds leading provider of digital print solutions. With a growing fleet
of 242 state-of the-art digital presses across North America, and in Prague and Asia, the AGS/CGX
network continues to expand its digital footprint with breakthrough technologies.
AGS has led AIRs printing of full-color digital print score reports for nearly a decade. AGS equipment
includes the largest network of Xerox iGens and five high-speed, inkjet web presses for high-volume,
variable data needs, and easily produces millions of pages of score reports.
Automated Graphics Solutions (AGS)
Printing Score Reports

The American Printing House for the Blind
(APH)
Creating Braille Materials

Bradford and Bigelow (B&B)
Printing Non-Scannable Documents
King Solutions
Shipping Test Materials

Rackspace
Managed Hosting Services
RR Donnelly
Printing Non-scannable documents
United Parcel Service (UPS)
Shipping Test Materials
Page 4-32
APH, founded in 1858, is the world's largest nonprofit organization creating educational, workplace, and
independent living products and services for people who are blind and visually impaired. APHs mission is
to promote the independence of blind and visually impaired persons by providing specialized materials,
products, and services needed for education and life. APH has adapted and produced tests in accessible
media for over 50 years. In 2002, APH became a member of the Association of Test Publishers (ATP). APH
has produced Braille tests or test-related tools for numerous statewide assessment programs in states
such as Alaska, Arizona, California, Florida, Illinois, Kentucky, Massachusetts, Michigan, Minnesota, New
York, South Carolina, and Tennessee.
Bradford & Bigelow (B&B) provides a supply chain solution for publishers. Their market focus and the
innovative technology of the Digital Book Factory with seamless integration into the B&B/Timson Press
Platform provide the speed, capacity, and redundancy for their publishers to significantly reduce lead
times, back orders, obsolescence, and shipping and administrative costs. B&B is a complete,
environmentally responsible supply chain solution. B&B recycles more than 5 MM pounds of paper and
100 K aluminum plates per hour. Their recycling efforts annually preserve 68,000 trees and 321,000 cubic
feet of landfill space. Almost every piece of material that enters their facility is sold as a finished product,
or recycled for future use. Their custom-built recycling system removes essentially all paper dust from the
plant, creating a healthier work environment for their employees.
Over 20 years ago, founders Mike Patterson and Meyer Bolnick saw a need in the transportation industry
and they decided to fill it. So in 1989, they established King Solutions as a values-based organization
committed to a customer-focused approach. Since then, King has become a leading provider of
international freight transportation, retail project management, global logistics, mail consolidation,
fulfillment, warehousing, and more.
Rackspace is rated one of the Magic Quadrant leaders by Gartner Research. Rackspace secured facilities
are designed with redundancy at every potential point of failure. Multiple redundant Tier 1 Internet
connections, router and switch configuration, real-time and backup power supplies are just examples of
the components of Rackspaces Business Continuity Plan (BCP). Rackspace delivers the best network
performance in the industry, using only quality bandwidth from Tier 1 backbone providers and has multigigabytes of high quality provisioned data capacity with virtually unlimited scalability.
Based on AIRs unique system design, AIRs contract with Rackspace guarantees 100% uptime. We
believe this is unique in the assessment industry.
Founded in 1864, RR Donnelley is the world's premier full-service provider of print and related services,
including business process outsourcing. With more than 55,000 employees worldwide, the company
provides solutions in commercial printing, direct mail, financial printing, print fulfillment, forms and
labels, logistics, call centers, transactional print-and-mail, print management, online services, digital
photography, color services, and content and database management to customers in publishing,
healthcare, advertising, retail, technology, financial services, and many other industries. RR Donnelley
has extensive experience working with large-scale assessment vendors around the country to provide
printing services for secure test material.
UPS, founded in 1907, is the world's largest package delivery company and a leading global provider of
specialized transportation and logistics services. UPS has grown into a multi-billion dollar corporation by
clearly focusing on the goal of enabling commerce around the globe. Every day, more than 399,000
employees worldwide manage the flow of goods, funds, and information in more than 200 countries and
territories worldwide. AIR and DRC have developed an excellent working relationship with UPS and have
worked together to guarantee the delivery and pickup of materials for more than 20 educational
assessment programs.

Communication
Program management communication is critical to the success of the project. While meetings and calls are important, the documentation resulting from these events is just as critical. AIR maintains a comprehensive suite of
specifications documents designed to capture
decisions in a systematic way and ensure that
those decisions are reviewed by the technical
experts whose work they may impact. In
Section 7.7.6 we describe these management
communication documents and the tools we
maintain to ensure that all specifications and
decisions are accurately documented.
Here, we focus on the management meetings
themselves. A more complete view of our management communication process is presented
in Section 7.7.5.
Led by the senior program manager, AIR will
host the weekly Department/AIR program
update conference calls to discuss the periods
progress and to identify upcoming deadlines
and possible challenges. During peak intervals
in the project schedule, AIR proposes to host
the meetings more frequently, if necessary.
The project plan and schedule, the policy decisions that need to be made, and the upcoming
tasks and timelines that need to be met will be
reviewed. At each meeting, the Department
will receive general progress updates from
each team leader. The Department will have
the opportunity to speak directly with AIRs
Technical, Systems Engineering, Item
Development, Operations, and Project
Management team leaders.
Decisions made during the meeting will be
documented in the appropriate specifications
document and in the weekly meeting minutes.
Changed specifications will be circulated
among internal and Department stakeholders

before the decision becomes final. In this way,
we ensure that (a) the entire team is aware of
the change and (b) each functional group has
an opportunity to identify risks or issues that
may arise as a result of a change in another
functional area.
AIR maintains a suite of communication tools
and services to support project communication. AIR will provide a toll-free conference
line for all Department/AIR project-related
conference calls. AIR offers web-conferencing
solutions that enhance a virtual meeting.
To conduct virtual meetings, appropriate technology needs to be in place. AIR has made significant investments in the hardware, software,
and connectivity required to support virtual
meetings. We use well-known industry-standard solutions that are easily accessible.
AIR offers all the services listed below in this
section. We have a nimble Conference Services
team and will adapt to existing virtual conferencing systems as needed. Because AIR has
offices around the world and already works
with stakeholders in multiple locations, the
AIR staff are very familiar with these solutions.
The offerings listed below require minimal
support. During the course of the project and
the different types of meetings, AIR will help
select the right tool for the right event.
CISCOs WebEx may provide the best option
and will facilitate meetings involving attendees
in multiple locations. Again, AIR will adapt to
existing conference services solutions as needed. Web-Ex connects us with real-time, online
webinars and telephone conferencing. Whether
we are meeting with the Department staff or
assisting in training state personnel or even
local staff, WebEx will assist us in connecting.
In addition to sharing presentation slides, participating in live video, and recording events,
attendees can interact with each another,
Page 4-33
regardless of the size of the meeting, using a

live Q&A.
KnowledgeTree site. AIR will maintain the

same users across systems.
In addition to WebEx, which offers video and

audio conferencing, instant messaging and
desktop sharing, AIR offers the following virtual conference services solutions:
Annually and upon request, AIR will provide

complete lists of users for the Department to
review and will eliminate or add users on
request. AIR will work with the Department
to configure the system with user roles and
respective file permissions.
The Global Crossing Conference Phone Line

is a traditional audio conference line and may
be suited for smaller Department/AIR meetings.
Increasing in popularity, Skype offers video
and audio conferencing, instant messaging,
and desktop sharing. Skype is easily downloaded and requires little technical support. A
user guide and a tutorial are available to assist
users accessing Skype.
For complete video conferencing between multiple locations, AIR offers Tandberg Video
Conferencing Services. Although this solution
requires local support (e.g., video cameras,
connectivity), Ciscos Tandberg offers screenbased video communication systems that
enhance our ability to communicate objectives
for a particular meeting.
AIR will maintain a KnowledgeTree site for the
Department. Before deploying documents to
the site, AIR will propose a list of users and
their access rights. The Department will have
an opportunity to modify this list before providing final approval.
Additionally AIR proposes to establish a
Secure File Transfer Protocol (SFTP) site for
the transfer of large databases and secure
materials. The SFTP site offers an efficient
mechanism for transferring large-scale data.
An SFTP site will also be available for transfer
of documents and data that do not require the
version control of KnowledgeTree. This will be
available to both the Department and the field
as needed. Such transfers do not require the
workflow or overhead associated with the
Page 4-34
When Department staff visit AIR they will be

assigned a visitors office. These are the same
offices that AIR remote staff use when visiting
the DC office and are lockable. They include
a telephone with a speaker. Fax machines are
available on every floor and can be arranged
for at need. Department staff will have access
to printers.

Meetings
AIR understands that in order for this project
to be successful, we will need to collaborate
with many stakeholders. AIR will make every
effort to be flexible, provide information as
required, and keep this important project on
track. We are prepared to participate in regular
management meetings, either at our own facilities or at a mutually agreed-upon location,
and to send up to 15 of our staff members to
at least two project management meetings per
year. All necessary AIR staff will attend up to
four project-specific meetings. DRC representatives will also attend these meetings, as
appropriate.
AIR will be responsible for all logistics and
facilities as well as the travel cost for AIR and
DRC staff, as required. If electronic participation is required, AIR can facilitate this logistic,
as needed; our remote conferencing solutions
are described in detail in Section 7.7.4.
Additionally, AIR will be responsible for the
travel and lodging cost of up to 10 Department
staff up to 15 times a year, as needed.
AIR believes that open collaboration with the

Department and other stakeholders is critical
to a productive relationship that fosters good
decision making and a common understanding
of program priorities. Below we propose a communications plan that will help achieve those
goals.
Communications Plan
Upon award of the contract, AIR will travel to
Tallahassee to meet with the Department within 30 days after the contract is finalized. At
that time, we will work with the Department to
develop a communications plan that specifies
who needs to receive specific information;
when that information needs to be communicated (on a specific date or at regular
intervals);
the person responsible for communicating
the information;
the method used to communicate the information; and
the processes for documentation that the
information has been communicated and
sign-offs have been received (if applicable).
AIR will create the projects organizational
chart, which will include all key team members, their contact information for immediate
access, and the chain-of-command contact
information. AIR will assist the Department in
designing a comparable project-based organizational chart, if desired, so that contact information is at everyones fingertips.
Additionally, we recognize the importance of
adhering to an agreed-upon escalation protocol. AIR will coordinate with the Department
during the resolution of all escalated issues to
ensure timely resolution and communication
with impacted users of the assessment system.
We will also work closely with the Department
to develop and document an effective strategy
for ensuring the Departments advance notice

of changes that may disrupt service to users of
the online system.
Please see Appendix H for a sample communications plan.
Real-Time Communication with the

Department
AIRs program manager will proactively communicate with the Department staff in advance
of changes that may disrupt service in locations required to administer the state assessment. In addition to the formal communications plans identified in this section, the
program manager will stay in close contact
with the Department staff, informing them of
any project change that might impact schedule
or deliverables. AIR will report all system
errors to the Department within 24 hours.
Weekly Meetings
In Section 7.7.4, AIR detailed our proposal to
host weekly program update conference calls
wherein the Department and AIR will discuss
the project plan and schedule, the policy decisions that need to be made, and the upcoming
tasks and timelines. Decisions made during the
meeting will be documented in the appropriate
specifications document and in the weekly
meeting minutes. Additionally, as discussed in
greater detail in Section 7.7.6, AIR will maintain an issues log to document issues across
the weekly meetings. The issues log will be
updated continuously and will document the
resolution of each issue.
Face-to-Face Meetings
AIR will work with the Department to schedule
monthly in-person meetings between the
Departments program manager and AIRs program manager, as needed.
Page 4-35
Annual Planning Meetings

AIR proposes a two-day, AIR-sponsored annual
planning meeting, during which the Department
and AIR will review schedules for the upcoming year and establish the baseline plan.
Attendees include the program manager and
leads in each area. AIR will draft an agenda for
review and revision by the Department. This
meeting may coincide with one of the comprehensive program management meetings or it
may be scheduled separately.
Teams are not segregated. To ensure message

consistency, staff members will regularly participate in other team or breakout meetings as
well as attend the program update conference
calls between the Department and AIR.
The goals of the project, along with the necessary steps needed to achieve those goals, are
communicated to the staff weekly. The result
is a cohesive project-wide plan communicated
by senior leadership that also contains the
task specificity provided by the staff members
of each team to accomplish the deliverables.
Communication across Teams at AIR

In addition to communication with the
Department, it is essential to have proper
channels of communication internally to
accomplish the described work. AIR conducts
several agenda-driven and efficiently run meetings across teams.
Each team leader also has communication
channels within his or her staff. Team leaders
meet with their staff or senior staff members
and share relevant information from the weekly team leaders meeting, as well as discuss
team-specific issues.
Each team is linked by a project calendar
maintained by AIRs program manager and
team leaders. The calendar is accessible to all
project members.
AIR encourages collaboration among and within teams. Because of their frequent interaction, team members are well-versed in the
responsibilities of other teams. Each teams
processes and procedures are codified and
placed on a shared file directory. The processes and procedures are written so that each
new team member can easily contribute to the
team without having to overcome a steep
learning curve.
Page 4-36

Reports
Key to a successful online assessment administration is proper documentation and tracking.
AIR understands the Department requires five
types of reports to facilitate and document
program management communications, as
follows:
Program Schedule
AIR produces schedules on three levels. At
the Program level, we produce the overall
Program Schedule. The Program Schedule
documents work-start and completion/
delivery dates for key activities, milestones,
and deliverables. We typically maintain the
Program Schedule in Microsoft Project
internally and will deliver the schedule to
the Department in Microsoft Excel format,
as required by the ITN. A preliminary
Program Schedule for this project appears
in Section 7.1.2.
At the next level, we have team schedules. For example, our Test Development
team will work with key content personnel
at the Department to develop detailed committee meeting schedules and batch delivery schedules that document the dates that
batches of items will be delivered for review.
The Psychometrics and Statistics team

develops an analysis schedule, which identifies milestones in the data receipt, analysis, and delivery schedules. Some of these
schedules are entirely internal, designed to
enable us to meet the milestones on the
project schedule.
The third level, task-specific schedules
such as the quality control of scoring data,
include tasks that require interaction with
the Department, and these schedules will
be developed in collaboration with the
appropriate Department staff.
Weekly Management and
Action Item Reports
Our Program Management team will produce and manage the Weekly Management
and Action Item Report. This report will
summarize, on a weekly basis, the current
status of the program, including status on
milestones and critical tasks, change management, risk management, and a review of
the current issues log. We look forward to
working with the Department in formalizing a format and structure of this report.
Technical Report
The Technical Report will be developed
by our Psychometrics and Statistics team.
Details on the Technical Report can be
found in Section 7.6.3.
Annual Program Review and
Accounting Reports
Information pertaining to the Annual
Program Review and Accounting Reports
is summarized following the discussion of
our set of additional program management
reports and documents and our management approach.
AIR is committed to providing each of these
reports. In our experience with other online
assessment programs, we have come to understand that programs with similar size and
scope require a substantial amount of addi-
tional documentation. Below we describe, in

detail, the management documentation that
communicates and documents the decisions
that comprise the overall program design.
Management Documents
AIR has developed a series of tools that enable
our project directors and managers to analyze
the project requirements. These tools form the
basis for clear, open, and accessible communication about the project schedule, deliverables,
and specific requirements. Our Program
Managers function as coordinators among the
technical experts in our other functional areas,
much as a general contractor orchestrates the
work of specialized subcontractors on a construction job.
The suite includes four types of tools that program directors use to orchestrate the work of
the project:
Schedules
Planning Documents
Tracking Documents
Specification Documents
AIRs various scheduling tools have been outlined above.
Planning documents kick off complex activities. They outline the general approach to the
activity, assign responsibilities, and establish
milestones. Some of these milestones may be
further specification documents or elements
needed to complete the plan. For example, the
senior program manager produces an administration summary for each administration, outlining the basics of the administration, including characteristics such as the grades and subjects available for testing, the number of
opportunities offered for each, the number of
field-test items to be included, and the start
and end of the administration period. The Test
Page 4-37
Development team uses this document as a

starting point for their item development plan,
and the Psychometrics and Statistics team
uses this document to begin their field-test
plan and analysis schedule.
include project team members whose work

depends directly or indirectly on the work
described in the document. In almost every
case, client staff members are stakeholders in
these documents as well.
Our Program Management team relies on several tracking documents to monitor project
status, coordinate work among internal teams,
and communicate progress toward milestones
to our clients. For example, to coordinate the
multitude of deliverables exchanged between
AIR and our clients, our operations lead maintains (and shares with our clients) a priority
list of deliverables that are currently in play
(e.g., blackline 1 of the Test Administration
Manual). The priority list identifies the document name, the responsible party, the date the
document is due to be returned, and the date
the document is due to be finalized. This document also assigns a priority level to help
clients manage and prioritize work during
busier times in the schedule.
Exhibits 7.7.6-1 through 7.7.6-4 summarize

our standard schedule documents, planning
documents, tracking documents, and specification documents.
Our team leads work with our clients to

develop detailed specification documents.
Specification documents function as the
detailed requirements documents for each
phase of the program. For example, our
Reporting team will work with the Department
staff to develop a detailed reporting specifications document, which includes lists of reports
to be produced, descriptions of report recipients, packaging specifications for paper
reports, calculation rules for each data element, and other fine details of the reporting
system. This document is usually created in
collaboration with the lead psychometrician
and the technical, content, and leadership staff
at the Department.
Each document has an owner and a list of
stakeholders. The owner is the person responsible for keeping the document up-to-date and
ensuring that all stakeholders sign off on any
changes to the document. Stakeholders
Page 4-38
Repository for Management Documents

Given the complex nature of assessment projects, the multitude of documents alone, not
including the review and approval process of
iterations of these documents, can be a challenge to maintain using traditional share-drive
options found in most organizations.
Therefore, AIR proposes that all major project
documents, in particular those documents
identified in Exhibits 7.7.6-1 through 7.7.6-4
(Management Documents), be stored on
KnowledgeTree.
KnowledgeTree is an online document management system on a secure cloud computing
platform. KnowledgeTree manages document
workflow and file sharing, alerts document followers to changes, and maintains version control, including authors and document history.
Documents are checked in and checked out
for editing. Access to documents can be controlled through a user roles and permissions
system.
This project will start with a clean slate of
project documents. Managing documents
through KnowledgeTree from the kickoff will
increase productivity, increase efficiency,
reduce error and miscommunication, provide
a single reference point for all documents that
house key project decisions, and lead to a
higher degree of collaboration across internal
teams and across our organizations.
KnowledgeTree has three distinguishing features:

1. Ease of use: A document management system can be effective only if it can be sustainably used by all users of varying technical ability. KnowledgeTree is user friendly
and requires little to no training (even for
nontechnical users). For example, creating
files by authorized users is as intuitive as
doing these same tasks on Windows
Explorer.
2. Workflow management: The process of
fully vetting specification and schedule
changes is both complicated and critical.
Our process allows participants to identify
a stake in a document, thereby becoming
stakeholders. Proposed changes are automatically circulated among stakeholders for
review and approval. This provides an
opportunity for those who must accomplish
the work to identify dependencies or implications that may have been initially overlooked.
3. Built-in version control system:
Documents can be checked in and checked
out for editing. Review workflows can also
be built into each document type. AIR proposes to establish a secure FTP (SFTP) site
for the transfer of large, secure databases.
SFTP offers an efficient mechanism for
transferring large-scale data. Such transfers
do not require the workflow or overhead
associated with the KnowledgeTree site.
AIR will maintain the same users across
systems.
AIR will maintain the KnowledgeTree and
SFTP sites. Before deploying documents to the
site, AIR will propose a list of users and their
access rights. The Department will have an
opportunity to modify this list before providing
final approval.
Annually and on request, AIR will provide
complete lists of users for the Departments
review and will eliminate or add users at its

request. AIR will work with the Department
to configure the system with user roles and
respective file permissions. KnowledgeTree can
be programmed to have a hierarchy that is
arbitrarily deep.
Change Management
The purpose of change management is to
ensure that changes are communicated, managed, and controlled. This is accomplished by
working with the Department to clearly define
the change control processes during the planning stage so that when the inevitable change
requests surface, there will be no confusion
about roles, responsibilities, and process
requirements. Otherwise, uncontrolled change
management can negatively impact budget,
scope, schedules, and possibly quality.
We recognize that plans and objectives can
change frequently. Therefore, we have a structured process for gathering, tracking, and disseminating changes to these documents. These
include regular planning meetings, project
updates, issues logs, and other tools.
Weekly conference call meetings between AIR
and the Department cover near-term deliverables and address issues as they arise. In addition, the AIR Program Manager will produce
weekly reports for the Department that will
include status of items such as deliverables,
key accomplishments, upcoming activities, any
changes to the schedule, or any issues. AIR
will deliver the weekly status report via email.
Decisions are documented in the meeting minutes and in the issues log, which is updated
weekly and tracks issues and near-term deliverables. It notes progress, problems, risks, and
assignments.
A potential change will be identified and documented on a change request form and sent to
the Program Manager. The Program Manager
Page 4-39
Exhibit 7.7.6-1: Schedule Documents

Team
Document
Management
Program Schedule
Management
Production Schedule
Psychometrics and
Statistics
Operations
Analysis Schedule
Delivery Schedule
Operations
Receipt and Scoring Schedule
Test Development
Batch Delivery Schedule
Test Development
Committee Meeting Schedule
Computer and Statistical

Sciences Center (CSSC)
Development and
Deployment Schedules
Page 4-40
Description
Defines project milestones; start dates
for activities
Details timelines for review and
approval of items, forms, user guides,
and ancillary materials
Lists schedule of data flows and analysis
activities
Details milestones for packaging and
shipping materials
Defines milestones for document
receipt, scanning, and scoring
Lists dates for delivery of item batches
and item reviews
Lists dates and milestones for the
preparation and convening of annual
committee meetings
Outlines schedules for development and
deployment
Owner
Program Manager
Program Manager
Operations Lead
Operations Lead
Item Development Manager
Software Project Manager
Exhibit 7.7.6-2: Planning Documents

Team
Document
Management
Project implementation plan
Management
Administration summary
Psychometrics and
Statistics
Field-test plan
Psychometrics and
Statistics
Analysis plan
Psychometrics and
Statistics
Standard-setting plan
Operations
Special version summary
Operations
Ancillary abstract
Test Development
Annual pool analysis and

development plan
Test Development
Committee meeting
summary sheet
Description
Summarizes the overall project,
including all deliverables, goals,
milestones, grades, subjects tested, and
modes of testing
Lists characteristics of the
administration, such as grades and
subjects available for testing, the
number of opportunities offered for
each, the number of field-test items to
be included, and the start and end of
the administration period
Describes the sampling methods used to
field test items, the statistics used to
analyze them, the ways in which risks of
item exposure will be mitigated, and
other aspects of the field-test plan
Lists objectives of the analysis and the
specific analysis methods for calibration
and linking for operational
psychometrics; special analyses such as
generalizability analyses, comparability
studies, etc., are described in each
studys individual analysis plan
Details the standard-setting workshops,
including text of instructions to
panelists, statistical analyses, and
construction of item booklets
Summarizes the special versions to be
offered, including translations,
alternate language audio, Braille;
details the requirements for each test
Summarizes ancillary documents to be
developed, including administration
guides, user manuals and other
materials; includes special instructions
or requirements
Describes annual charts summarizing
the state of the item pool relative to the
requirements of the item pool or forms
to be deployed the subsequent year
Summarizes committee meetings,
including the number of items to be
reviewed, number of committee
members, dates
Owner
Program Manager
Program Manager
Lead Psychometrician or
Study Lead
Standard-setting Lead
Program Manager
Program Manager
Page 4-41
Exhibit 7.7.6-2: Planning Documents (continued)
CSSC
Project charter
CSSC
Testing design
CSSC
Test plans
CSSC
Software release plans
CSSC
Production control forms
Reporting
Reporting goals
Page 4-42
Describes the goals, approach, and

nature of new systems or system
modifications
Details the design document describing
the testing strategies to be employed
for a new system or the modifications to
an existing system; lays the groundwork
for the test plan
Details plans for load testing, individual
test cases, and regression tests to be
implemented
Lists step-by-step activities for Software
Development team and Network
Engineering teams to deploy software
systems for scheduled releases
Lists steps required by AIRs Production
Control Board for evaluating and
approving the test plans for a
configuration change in the middle of
the year in production systems
Lists objectives of the elements to be
included in the report; describes the
goals and the types of actions that the
report might motivate, and suggests
data elements that might support those
actions
Reporting Lead
Exhibit 7.7.6-3: Tracking Documents

Team
Document
Management
Issues Log
Management
Priority List
Management
Risk Register
Test Development
Item Release List
Test Development
Batch Delivery Tracking

Sheet
Description
Updated weekly; tracks issues and nearterm deliverables; and notes progress,
problems, risks, assignment, and agreedupon changes to scope or progress
Lists the testing and administration
documents currently being exchanged
between AIR and the Department; assigns
a priority value to each document;
identifies the documents review level and
due dates
Identifies and classifies project risks,
organized by function and area;
summarizes proposed mitigation strategies
Lists specific items to be released each year
for each subject
Tracks the item batches by grade and
subject, including the number of items in
each batch, number of items accepted, and
number of items accepted with revisions;
tracks due dates
Owner
Program Manager
Program Manager
Program Manager
Lead Psychometrician,
Page 4-43
Exhibit 7.7.6-4: Specification Documents

Team
Document
Psychometrics and
Statistics
Analysis specifications
Psychometrics and
Statistics
Psychometrics and
Statistics
QC specifications
Psychometrics and
Statistics and Test
Development
Test Development
Test specifications
Test Development
Item Tracking System (ITS)

specifications
Operations
Packaging specifications
Operations
Receipt specifications
Operations
Scoring specifications
CSSC
Requirement
documents
CSSC
Test Information Distribution

Engine (TIDE) specifications
CSSC
Data file generator

specifications
Learning Point Navigator
specifications
CSSC
Page 4-44
Scoring-engine
specifications
Field-test summary
Description
Owner
Describes the details of analyses, including

software settings, locations of data files,
and many others
Identifies quality control-specific analyses
and comparisons to be undertaken
Summarizes scores to be reported at
various levels, attemptedness rules, and
scoring computations that are performed
by the test-scoring engine for all tests
Includes test blueprints, item
specifications, and style guides
Shows numbers and types of items to be

field-tested each year and form
construction details
Defines the attributes to be collected for
each item, the content review sequences,
and the item layout templates
Describes materials to be shipped, shipping
carrier, expected packing list format,
materials to be packed, and details of
packaging
Details secure and nonsecure documents to
be received, expected quantity, receipt
start and end dates, and special
instructions
Identifies human-scored constructedresponse items, the dimensions and points
on which items are scored, condition codes
for unscored responses, backread and
second-read rates, frequency of validity
papers, and target reliability and validity
statistics
Defines the detailed requirements for the
software or software modification to be
implemented
Defines detailed requirements for materials
ordering, student management, and user
management, including user roles within
the suite of online testing applications
Defines detailed requirements for student
and item-score files delivered after testing
Defines detailed requirements for AIRs
Learning Point Navigator, which provides
instructional resources targeted to
identified areas of strengths and
weaknesses
Lead Psychometrician,
Packaging and Shipping

Manager
Packaging and Shipping

Manager
Scoring Director

Exhibit 7.7.6-4: Specification Documents (continued)

CSSC
CSSC
CSSC
Online Project Portal

specifications
Test Delivery System
specifications
Reporting

specifications
Reporting specifications
Reporting
Report mockups
Defines the layout, skin/theme, user cards,

and content sections
Defines the test settings for the Proctor and
the Student Interfaces and the business
rules that govern test opportunities
Lists the reporting settings for student and
aggregate reports
Details the reports, data elements to be
included, calculation rules for each data
element, special rules for merging and
cleaning data, and other similar items
Presents annotated graphic mockup of
reports, providing the look and feel and
identifying the critical elements of the
visual appearance of the reports
will log all change requests into the Change

Control Log and then assign a subject matter
expert (SME) to analyze the impact of the
change request on the scope/deliverables,
schedule, budget, and overall project quality.
The SME analyzes the request and documents
the impact including the feasibility of making
the change and any impacts and risks associated with the change. In some instances, it
may be necessary to define the impact if the
change is not made. The Program Manager will
review the analysis and provide it to the key
stakeholders, including the Department, to
either authorize or reject the change. A change
will not be implemented until the Department
has signed off on the change in writing.
Whether the change is approved or rejected,
the Program Manager will update the Change
Control Log to reflect the updated status. A
sample Change Control Log can be found in
Appendix K.
Changes are reflected in the appropriate communication documents through a carefully
coordinated process. The Program Manager is
responsible for communicating the changes to
the document owner, who reviews the changes
and amends the document. The document is


Reporting Lead
Reporting Lead
then routed to each stakeholder who has subscribed to the document. Changes to the document are not final until they receive signoff
from each stakeholder. The document owner is
responsible for ensuring that the changes are
accurately drafted, reviewed, and accepted by
each stakeholder, including the Department.
A change will not be implemented until the
Department has signed off on the change in
writing.
Issue Management
AIR understands that these are high-stakes
online administrations and any issue that arises must be dealt with quickly and effectively.
The Program Manager will maintain an issues
log that will be updated regularly and reviewed
at each weekly meeting.
Once an issue is identified, it will be reported
to the Program Manager who will evaluate the
issue with the project team. If the issue is considered valid, it will be added to the issues log
with all pertinent information, and a priority
will be assigned. This log will be reviewed during the weekly conference calls. However, if
the issue is assigned a priority of medium or
Page 4-45
high, the issue will be immediately shared with

AIR management and the Department staff as
needed, according to proper escalation protocol. Program staff will be assigned to determine
the resolution. This information will be shared
with the Department staff, but no action will
be taken until approval is received from the
Department to proceed.
The issues log will be managed by the Program
Manager, and it will be available to all stakeholders via the document management system.
Each issue will receive a unique number when
entered into the log. The entry in the log will
include a description of the issue, the date
identified, the person who identified it, and
the method used to report it. Each issue will
be evaluated to measure the affected program
team members, to define the impacted areas of
the project and to be assigned a priority. Each
issue entry will indicate the owner of the issue,
the action steps to be taken, and the expected
resolution date. Each issue must be approved
by the Department before any action can be
undertaken. Finally, the actual resolution date
and status will be updated. Like most other
project documentation, the issues log will be
reviewed by the project team regularly to
ensure that issues are being resolved. The document will be updated and communicated to
all project stakeholders as updates are made.
A sample issues log can be found in
Appendix I.
Because of the high stakes associated with this

program, it is vital that schedules are monitored closely and that sufficient time is provided for reviews and production of materials.
The risk log will be reviewed at each cycle
kickoff meeting, as well as during weekly
progress meetings throughout the life of the
project.
A sample risk register format can be found in
Appendix J.
Annual Program Review and

Accounting Reports
AIR will provide two copies of the Annual
Program Review each year via CDs/DVDs/jump
drives inside plastic sleeves in loose-leaf
binders. The Annual Program Review will
include a record of major program accomplishments and successes as well as proposed
changes and improvements. The review will
address all aspects of program service, including details of meetings held with educators,
milestones attained/deliverables completed for
each project area, issues encountered, root
cause analysis, corrective actions taken, and
performance metrics reports as required by
the Department.
In addition, AIR will provide electronic copies
of the following:
Risk Management
AIR understands the need to analyze and monitor risks in all our programs. It is critical that
the risk management plan be vetted and complete as early in the project as possible.
AIR looks forward to working with the
Department staff to complete and approve this
process. In conjunction with the Department,
an initial risk management plan defining all
Page 4-46
processes used in risk management will be

developed and delivered to the Department
within a mutually agreeable timeframe.
PDFs of the test answer books

PDFs of all screenshots for computer-based
test approved forms
PDFs of all passage booklets, reference
sheets, large print test books, one-item-perpage test books, large print reference
sheets, large print response grid instruction
sheets, and CBT work folders
PDFs (untagged and tagged, if applicable) of

all interpretive and ancillary products
Native application files of all interpretive
and ancillary products
PDFs, or other appropriate file formats
such as video, of all other products, including memoranda and report templates
Missing materials and resolution reports
Final Program Schedule reflecting planned
vs. actual dates of task completion
Weekly Management and Action Item
reports
File Layouts
Specifications documents
Accounting Reports
The Accounting Reports will provide a summary of the Quarterly Accounting reports. The
Accounting Reports will list expenditures by
program task and level of effort as well as
expenditures and balances of Reimbursable
Funding Categories. See Section 7.7.9 for more
information.
We look forward to working with the
Department to adapt and merge our Program
Management documentation tools and processes
to fit the needs of the Department going forward.
7.7.7 Meetings Involving Florida

Educators and Other Citizens
AIR has conducted thousands of meetings for
Departments of Education over the years. We
are committed to providing on-site meeting
organization, support, and management for all
meetings outlined in the ITN. Our project
management staff will coordinate the flow of
materials, offer on-site meeting assistance,
address the Department staffs needs, and take
care of all post-meeting procedures. In addi-
tion to our project management staff, AIR is

supported by a full-service meeting planning
staff dedicated to supporting the project team
in meetings and exceeding the Departments
and participants needs.
Details on the content of the following specific
meetings can be found in Section 7.3.4 of our
proposal: Bias and Sensitivity Committee
Reviews, Passage and Item Bias Meetings,
Passage and Item Community Sensitivity
Meetings, Passage and Item Content Review
Committees, Data Review Committee, and
Content Advisory Committee. Details on the
content of Rangefinding, Technical Advisory
Committee, and Standard Setting can be found
in Sections 7.5.4.5, 7.6.0, and 7.6.3.4 respectively.
AIR understands that the Department will
facilitate all meetings unless otherwise specified. The Department will designate Florida
educators, Florida citizens, or consultants to
participate in meetings, which AIR staff will
attend, as required. Our project management
staff, comprising the project manager, the
team leaders for the program area, and an
administrative assistant, will be responsible for
arranging, organizing, attending, and paying
the cost of bringing participants to specific
meetings. Appropriate AIR staff will assist and
participate in each meeting. We understand
that for all meetings involving separate subject/grade groups, we will be required to provide the appropriate professional staff to take
part in each of the separate subject/grade
groups. AIR will be responsible for all expenses
associated with our staffs travel and participation. All expenses incurred will be in accordance with the Department of Financial
Services Guidelines for State Expenditures.
We will also arrange, organize, attend, and pay
travel, lodging, and per diem costs required to
bring participants together for all meetings
outlined in Table 7.7.7 of the ITN. AIR understands and is fully prepared to meet the speci-
Page 4-47
fications outlined in Table 7.7.7. regarding

expected meeting durations, approximate
number of attendees, and estimated costs for
each meeting. In this section we provide more
details on our meeting logistics, committee
travel and reimbursements, and contractor
staff travel and expenses.
Meeting Logistics
AIR will make all logistical arrangements and
cover costs for each meeting outlined in Table
7.7.7 of the ITN. AIR has many years of experience providing high-quality, responsive, and
flexible meeting planning services to largescale assessment clients. Our meeting planning
and coordination experts are qualified and prepared to provide first-class meeting services for
Florida.
Well-organized, coordinated meetings, and
associated administrative processes, combined
with appropriate and comfortable meeting
facilities, help promote successful assessment
programs. Over the years, AIR has gained considerable experience coordinating all types of
meetings for multiple assessment clients,
establishing lasting relationships with meeting
facilitators and their staff and ensuring a high
level of customer service.
AIRs meeting coordinators work closely with
other project team members to ensure that all
administrative processes and meeting arrangements exceed client expectations. AIRs staff is
skilled at negotiating contracts with hotels and
meeting facilities to obtain optimal rates and
ensuring that meeting costs are within project
budgets. AIR provides full-service meeting
administration services to clients as well,
including overall organization and coordination for meeting participants; developing and
distributing letters and e-mail communications
to attendees; tracking participant registration;
and processing invoices, travel vouchers,
stipends (for eligible committees), and substitute teacher reimbursement forms to ensure
Page 4-48
proper payment. AIRs meeting coordination

staff excels at providing exceptional customer
service to meeting attendees.
AIR will provide the following logistical services for committee meetings:
Meeting LocationAIR understands that
meetings held in Florida will typically be
organized in a central Florida location, a hub
city such as Jacksonville, or in Tallahassee.
For both in-state and out-of-state locations,
easily accessible and cost-effective locations
for meetings (i.e., capacity to meet the logistical demands of large-scale, simultaneous, or
multi-group meetings; proximity to the airport
for committees attended by members who
travel to the meeting from out of state).
Meetings will be held at a hotel, conference
center, or similar suitable location. Where possible, meetings will be scheduled concurrently
and in the same venue to reduce costs. AIR
will continue to seek meeting locations that
offer a variety of nearby, cost-effective restaurant options.
FacilitiesAIR will make all facility arrangements, including securing appropriate meeting
rooms and supplying all technology, audiovisual equipment, web access, and supplies needed
to conduct the meetings. AIR will also take
measures to ensure that the meeting rooms
are secure.
Meeting Planning DocumentationPrior to
each meeting, AIR will provide information to
the Department that outlines the parameters
of the meeting, participants, logistical information, and the Department-required materials,
resources, and other details. Upon the
Departments approval, AIR will produce meeting schedules, agendas and participant lists for
each meeting. At the conclusion of each meeting, a follow-up report containing meeting
notes, review logs, action items, a summary of
results, copies of committee member paper-
work, and any other requested information

will be provided to the Department. These
materials can be delivered in hardcopy or electronically, depending on the Departments
preference.
Meeting Facilitation and Management
AIRs test development and/or technical staff
(depending on the meeting), along with an AIR
meeting planner/administrative assistant, will
be present at meetings, as needed, to provide
overall meeting management and support to
the Department.
Signed AgreementsAIR requires signed
security agreements from all committee meeting participants and retains the agreements for
the duration of the contract. At the beginning
of all meetings, each participant will be asked
to sign a confidentiality (non-disclosure) letter
specifying the confidentiality agreement and
security regulations. The agreement will also
outline ownership regulations. AIR will ensure
that no confidential materials related to the
project will be released without the Departments
explicit approval. The original copies of these
forms will be retained in secure storage, and
copies will be made available to the
Department upon request.
Secure MaterialsSecure materials will never
be left unattended. AIR staff will monitor the
security of all testing-related materials
throughout the entire process. AIR will continue to prepare secure materials on green paper
for easy identification.
All materials sent to meetings are personally
delivered, ensuring security. Should any materials require shipment, packages will be sent
through a secured mailing process and have
tracking documentation.
AIR will number each set of materials used
during the meetings so that any missing material will be immediately noted when materials
are checked in and out each meeting day.
When materials are not in use, they will be

kept in a locked space. AIR staff is vigilant
about maintaining security at these meetings.
that the Departments preferences and requirements for meetings are understood and met.
All prices associated with meeting planning
services and deliverables have been provided
in the Cost Proposal.
Committee Travel, Communication,

and Reimbursements
Travel ArrangementsAIR will secure hotel
rooms for participants who travel specified distances to the meeting site and therefore meet
eligibility criteria for sleeping rooms (i.e., travel more than 100 miles for a multiday meeting). When possible, AIR will secure sleeping
rooms at hotels that offer complimentary
Internet access and breakfast and/or evening
receptions. AIR will reimburse participants for
travel-related expenses, including airfare, car
rental, hotel accommodations, and other relevant expenses, such as tolls, parking, cab fare,
and vicinity mileage for personal vehicle use.
Communication with ParticipantsAIR
understands that the Department will designate Florida educators, Florida citizens, or
consultants from outside the state to participate in meetings as required. Based on participant lists provided by the Department, AIR
will prepare and distribute meeting notifications, invitations, and logistical information to
committee members. All communications with
districts regarding meeting travel logistics will
be reviewed and approved by the Department
before sending to District personnel. AIR will
also coordinate any follow-up communications
to meeting participants after the meetings are
concluded, as required by the Department.
ReimbursementsWe understand that AIR
will be responsible for paying travel, lodging,
and per diem costs required to bring partici-
Page 4-49
pants together for specific meetings as identified in the ITN.
Contractor Staff Travel and Expenses

AIR will be responsible for all expenses associated with our staffs travel and participation in
all meetings. All expenses incurred will be in
accordance with the Department of Financial
Services Guidelines for State Expenditures.
7.7.8 Communication Between the

Contractor and Districts
that communications with the district are
clear, consistent, and follow appropriate protocols. Our project staff members are sensitive
to the fact that Departments of Education follow specific communication protocols unique
to their programs when communicating with
districts. All communication between Floridas
districts, schools, and AIR will be directed
through or receive prior approval from the
Department. In addition, we will work closely
with the Department to establish communication guidelines that specify the protocols for
drafting, reviewing, and sending all communications. Instructions on the types of communications that should be directed to the District
Coordinator of Assessment will be included,
as well as protocols for communicating with
special schools. These specifications will be
reviewed and approved by the Department and
will be used by all AIR project staff to guide all
district communications.
AIR will also establish a dedicated customer
support line for district and school personnel
as required in Section 7.7.8 of the ITN. This
toll-free customer support line will be used
only for the purpose of
clarifying meeting travel logistics;
correcting computer-based system issues;

and
reporting problems with the delivery or
pickup of test materials
Below we provide more detail on our
proposed methods for communicating with
districts;
proposed process for drafting and reviewing
all communications with the Department
before submission; and
proposed plan for providing additional customer service support for district and
school personnel.
Proposed Methods for Communicating

with Districts
Complex assessment programs serving large
numbers of districts and schools require
extraordinary coordination in all aspects,
especially communication with districts. Our
work supporting Departments of Education
in Oregon, Hawaii, Delaware, Utah, and
Minnesota has helped us refine a number of
effective communication mechanisms to efficiently convey a range of messages. For example, we have found that the portal home page
is an effective mechanism for posting important announcements about PreID windows,
upcoming webinars, or the availability of the
secure browser for download. At the same
time, information on the status of each online
system is typically posted on a separate website so district and school personnel can quickly confirm whether systems are operable
across the state or if they may be experiencing
a local issue. AIR will use the specific mode of
communication as required by the Department
when communicating with the districts including e-mail, fax, or mail. In addition, we propose the following communication mechanisms for specific information:
ordering additional test materials;

Page 4-50
1. Portal announcements
System Status Page
2. Status of online systems

3. E-mail blasts to the districts on a specific
topic requiring immediate attention
4. Memos containing clarifying instructions
5. Memos accompanying shipments of score
reports or test booklets
Portal Home Page

Our portal home page serves as a central location for district and school personnel to access
(please see Exhibit 7.7.8-1, sample portal
home page from the Hawaii portal). For this
reason, the home page provides an optimal
location to post important announcements
regarding a number of topics including:
In addition to announcements housed on the

portal home page, we also propose to create a
separate status page to allow schools and districts to easily confirm whether online systems
are operable or whether they may be experiencing a local issue. Please see Exhibit 7.7.8-2,
a status page from Hawaii.
We will work with the Department to draft all
text for the status page for each online system.
All subsequent text changes and updates to the
status page will require Department approval
prior to being implemented.
date, time, and login information for training webinars;

availability of online training courses and
reminders on which personnel are required
to take these courses;
reminders to districts about the dates they
can expect shipments of test booklets and
score reports;
opening dates for TIDE and the availability
of the TIDE User Guide along with a
reminder about the specific tasks that districts are required to perform in TIDE; and
availability of the training test site for students and Test Administrators and instructions on how to access these sites.
The Department can, of course, use the portal
home page for other information and
announcements. All text for announcements
will be reviewed and approved by the
Department prior to release.
Page 4-51
Exhibit 7.7.8-1: Sample Portal Home Page
announcements posted on the portal home

page, e-mail blasts directly to District
Assessment Coordinators can be effective
(please see sample e-mail to districts below).
We understand that the Department may prefer alternative modes of communication with
districts, including fax, memo, and regular
mail, and are willing to use these mechanisms
as required. As stated above, any communication to districts will be thoroughly vetted and
approved by the Department before sent.
As we enter the final week of online testing, a
few reminders:
All online tests must be completed in the
online testing system by 5:00 p.m. on
Friday, May 18.
Exhibit 7.7.8-2: Sample Systems Status Page
All student responses from accommodated

test materials must be entered into the
Data Entry Interface by 5:00 p.m.
If you have any questions, please contact the
AIR Help Desk at 1877-xxx-xxxx.
Memos
E-mail Communications to Districts

With multiple responsibilities and assessments
administered simultaneously, districts often
need additional reminders to help them complete the tasks required during an administration. We have found that in addition to
Page 4-52
In addition to the above communication mechanisms, we propose to draft memos that contain clarifying instructions or that accompany
shipments of score reports or test booklets.
Memos accompanying shipments typically
summarize the contents of the shipment and
include a link to view updates on the status of
material orders or online reports, as well as
the toll-free customer service number to
address any questions. Memos containing clarifying instructions generally focus on a specific
task and outline the steps required to complete that task (for example, how to access district passwords or request a new password if
your password was lost). All district memos
will follow the proposed review and approval
process outlined below and will require
Department approval prior to release.
Description of Process to Draft,

Review, and Approve All District
Communications
AIRs project teams are well versed in thoroughly vetting all communications with our
clients prior to communicating with districts
or schools. Our standard process for drafting
all district and school communications, such
as portal announcements, updates to the status page, memos, district e-mails, and instructions, include the following review steps:
Senior program manager discusses the proposed communication with the Department
and obtains the Departments preferred
mode of communication and agreement on
the general message
Project team drafts the initial communication
AIR editors review and edit the draft using
the Departments style guidelines
Project team sends the draft communication to the Department for review
Project team implements the Departments
revisions and sends to AIR editors for final
edit check
Project team sends final edited draft back
to Department for approval
Department approves communication to be
sent to districts
We will work closely with the Department to
add additional stages of Department review as
required.
Dedicated Customer Support Line for

Districts
AIR will follow the same process outlined in
Section 7.4.10 to provide additional customer
service and technical support to district coordinators of assessment. We describe our plan
for customer support in Section 7.4.10. In that
section, we commit to providing a dedicated

toll-free line Monday through Friday from 7:00
a.m. to 8:30 p.m. for all district and school
personnel.
In addition to the customer service provided
in Section 7.4.10, we understand that districts
will require a second dedicated customer service line to ask questions regarding the following topics:
Clarifying meeting travel logistics
Ordering additional test materials
Correcting computer-based system issues
Reporting problems with the delivery or
pickup of test materials
In order to answer queries from district coordinators of assessment on the above topics,
we will provide additional customer service
Monday through Friday from 6:00 a.m. to 8:00
p.m. We can provide a second toll-free number
for district assessment coordinators, or we can
add a prompt (press 2 if you are a district
assessment coordinator) on the main toll-free
customer support number.
AIR will also provide web-based, searchable
frequently asked questions (FAQs) on the portal to provide districts and schools with
answers to common questions from the topics
listed above.
7.7.9 Reimburseable Funding

Categories
AIR is completely familiar with support for
reimbursable services in our fixed-price
statewide assessment projects. We have formal
procedures that we will adapt to include all
formal authorizations from the Department,
along with appropriate follow-up documentation to ensure such services were completed
to the Departments satisfaction.
Page 4-53
1. Consultant services. AIR has procured similar services for Ohio, New Mexico, and
other states for tasks such as independent
research studies, TAC presentations, and
specialized training.
4. Provision for meetings with Florida educators
areas and open a new subcode for each authorized assignment. AIRs core financial system is
Delteks Costpoint, which is the industry-leading ERP software designed specifically to meet
the business needs of sophisticated projectdriven organizations. It also holds the largest
market share in financial software for government contractors. With Costpoint, AIR can
track, manage, and report on every aspect of a
project, including budgets, schedules, expenses, purchasing, revenue recognition, business
performance, regulatory compliance, and
more. Because AIR is a major federal government contractor, our accounting system is routinely audited by the federal government.
5. Provision for meetings with Department

staff
7.7.10 Final Delivery of Materials
2. Contingency services. On existing contracts, AIR is often asked to provide cost

estimates for such activities as unexpected
analyses.
3. Authors alterations and design control. We
appreciate Floridas understanding of the
cost impact of last-minute changes.
6. Contractor staff positions. AIR currently

supports multiple editors for Minnesota. We
believe these contractor staff will find AIRs
fringe benefits package most competitive.
The most important control associated with
reimbursable funding is clearly defined purchase authority. AIR will work with the contracting officer immediately upon contract
award to document who is authorized to
expend these funds. With several of AIRs
clients, different client individuals have different authority based on level of cost or work
assignment. We will ensure this document is
kept up to date.
To support Department decision-making, AIR
will present cost estimates whenever possible.
For example, if the Department requests an
additional analysis or a preferential blueline
change, AIR will provide a cost estimate to
enable the Department to do a cost-benefit
analysis.
We can easily capture and report these costs
on the required accounting reports. Within
AIRs cost accounting system, we will establish
separate project codes for each of these six
Page 4-54
Introduction
The successful transition of a testing program
entails transferring all required data, products,
knowledge, and other state-owned assets. The
transfer must
be complete and correct;
avoid disruption of services or support
during the transition period; and
ensure that the tests remain accurately
equated over time.
AIRs management process, described in
Section 7.7.0, relies on detailed specification
documents for virtually every element of the
program. We use these documents to ensure
clear and uncomplicated communication of
specifications within our organization and
between our organization and the Florida
Department of Education. These same documents will clearly communicate the specifications and requirements to the next contractor
working with the Department on this program.
Below, we discuss our experience and processes for ensuring a smooth transition from this
contract to the next, should AIR be selected as

the Contractor.
Transition Experience
During the past 7 years, AIR has successfully
transitioned from other contractors over a
dozen separate testing programs in Ohio,
Oregon, Hawaii, South Carolina, New Mexico,
Delaware, Minnesota, and Utah. When the time
comes to transition this program to another
contractor, AIR will support the Department as
we would hope to be supported when programs
are transitioned to us.
Our most recent experiences demanded
aggressive transition timelines, a massive item
and item bank transition, and the transition of
technical specifications, reporting specifications, and other documentation details to
ensure that a program could run uninterrupted
and consistently after the change in contractors. In the course of each implementation, we
have had the opportunity to establish best
practices for effective transitions from one
contractor to the next.
Ohio
AIR transitioned the OGT program from
Measurement Incorporated in 200607. In this
case, AIR worked with the Ohio Department of
Education to develop a transition matrix, identifying each piece of information, data, or content, along with its preferred source and format. The matrix facilitated a smooth transition
and became the focal point of a series of transition meetings. The success of that transition
model has made it the standard approach recommended by AIR, and we have used it successfully in other programs, as noted below.
Hawaii
When AIR took over Hawaiis entire state
assessment program, we were able to obtain
much of the needed data and documentation

directly from Hawaii Department of Education
(HIDOE) staff and files rather than from the
prior contractor. This information had been
reviewed and verified by HIDOE as part of its
processes. In other cases, we had to rely on
information gathered from the outgoing contractor. This information was not always
shared promptly, and we discovered many
inconsistencies, inaccuracies, and other problems in the data. For example, test item identifiers were sometimes repeated with different
items, or multiple, different versions of items
appeared on different paper forms.
To address this issue, we established a policy
of independent verification of every item and
the linkages among them. For example, our
technical team replicated several years of calibration and linkage information to ensure that
we had the correct data and items. Bookmaps
were compared with physical test booklets and
then compared to items extracted from electronic Quark files to ensure that they were
accurately linked.
In some cases, neither HIDOE nor the prior
contractor had the material we needed. For
example, we did not receive an electronic item
bank. We extracted the items directly from the
Quark files of all prior administrations. We
developed software to extract the item text,
formats, graphics, and location from the books
and loaded them into AIRs Item Tracking
System (ITS). Additionally, we discovered that
the item identifiers embedded in the file did
not always match the test bookmaps (Excel
files). Resolving this discrepancy required an
extensive quality control effort to prepare an
accurate item bank to begin our work. We also
discovered that the previous contractor did
not maintain a list of previously released items
or items that were rejected at data review.
Without this information, we were at risk of
including one of these items on a subsequent
test. This lack of data required us to expand
Page 4-55
our transition list to include all prior practice

tests, parent information booklets, and other
documents that contained released items.
Because of these and other issues, the transition process lasted longer than HIDOE or the
previous contractor anticipated and required
flexibility from all involved.
Utah
More recently, we had the opportunity to again
transition a statewide assessment program
from another contractor to AIR and to learn
valuable lessons in the process. As we entered
into our contract with the Utah State Office of
Education (USOE), we provided a transition
specification document to request key information needed for a successful transition of
their testing program. USOE was only able to
provide a small number of the requested materials. AIR then initiated a process to develop
documentation, based on our many discussions with USOE staff, to obtain the information required and to create a full documentation library for USOE to use moving forward.
The Utah contract included the transfer of
their item pool from their existing vendor to
AIR. The contract called for the items to be
provided in standard QTI format. However, the
prior vendor was not able to provide this format. AIR worked with USOE and their prior
vendor to define a workable format that the
prior vendor could provide. The AIR technology team developed scripts to successfully
import the items into the new item bank.
These examples underscore the need for a
meeting of the minds between the state
department of education, the outgoing contractor and the new contractor on such issues
as the format of the information to be delivered and the availability of required materials
prior to implementation.
Requirements for Successful Transition

We have experienced both successes and challenges in our transitions over the years, and
we have learned that a successful transition
has four core requirements:
Identification: The first step in developing
a transition plan is to clearly identify all
requisite transition information, as well as
the data and format in which the data will
be delivered.
Planning: Highly detailed plans chronicling
specific delivery deadlines for transferred
information facilitate a smooth transition.
Delivery: Once critical transition elements
are identified and delivery timelines are
established, the actual information transfer
must be monitored to ensure that all information delivered is both accurate and timely.
Quality Control: To ensure successful completion, the transitioning contractor must
support the new contractor in replicating
work and run analyses to ensure that data
and trends are accurate.
Successful transition can be facilitated through
an ongoing transfer of program documents,
data, and content from the contractor to the
Department. This will ensure that the
Department has access to, and control of, the
necessary products for your system.
The specific documents, data, and content to be
transferred will be detailed in a specifications
document described later in this section and
may include such materials and products as
test books;
answer documents;
final electronic source files of
interpretive products;
software source code; and
ancillary materials.
Page 4-56
In addition, we recognize that integrating the

Departments data and content with the new
contractors systems will require support from
the outgoing contractor. We have been on both
sides of assessment program transitions, and
we understand the factors that can contribute
to a successful transition. Putting to work the
lessons we have learned, we propose a plan
that follows these five steps:
It is, of course, necessary to negotiate the

specifics of the plan: What information is
already available to the new contractor? In
what format does the new contractor need the
information? When does the new contractor
need each piece of information? In what format is it possible for us to deliver on schedule?
These questions will be addressed in a series of
meetings:
1. An initial meeting is convened between the

Department and AIR to identify the critical
information that needs to be transferred
and to discuss the quality assurance plans
that will ensure accurate transfer.
Initial Planning Meeting: A preliminary meeting to draft a transition specification document

that identifies all relevant data, test items,
testing information, and other materials necessary for a successful transition.
2. AIR completes a draft of the transition

specification document, which includes
details about the format of the information
to be delivered and the delivery schedule.
3. The Department, the new contractor, and
AIR meet to review the transition specification document, refine it as needed and
negotiate any changes.
4. AIR completes the final transition specification document by specifying the exact
approach to be taken to incorporate the
transferred information into our workflow
and systems and to refine the quality assurance plan associated with each item.
5. AIR implements the plan as outlined in the
Department-approved transition specification document.
Below, we provide more details about key transition meetings that will help facilitate the
process.
Transition Meetings
As noted above, we propose beginning with a
discussion of the information to be transferred,
the preferred format, the quality assurance
plan, and the schedule for implementation, all
of which will be captured in the transition
specifications.
Contractor Meeting: A follow-up meeting with

the Department and the new contractor.
During this meeting, both the project leadership and technical/operational staff of the new
contractor will work with AIR and the
Department to define information format and
delivery schedules and to finalize the transition specification document.
Transition Meetings: Smaller meetings specific
to relevant functional areas. Staff from AIR,
the Department, and the new contractor will
monitor transition timelines and delivery compliance as defined in the transition specification document.
We recommend that the initial meeting
include all the technical Department staff
members that have specific knowledge of the
data, items, information, and other materials
needed to transition the program. We propose
a subsequent meeting to include the new contractor, whose representatives will include
both the project leadership and the
technical/operational staff. During this meeting, the transition specifications will be finalized. Based on our experience, we recommend
that these meetings be convened face-to-face.
Smaller working meetings will be necessary
between the new contractors functional team
Page 4-57
leads, the technical/operational staff of AIR

and the Department. Project leadership may
choose to join. In previous transitions, these
smaller meetings have been convened as conference calls on a set schedule (e.g., biweekly,
1-hour calls for 3 to 4 months). We will update
the status of each item on the transition
specifications before each call.
Replication
Replication studies are critical to ensuring that
the key transition goals are achieved, and AIR
recognizes that small psychometric errors can
have vast consequences, such as scores that
are based on inaccurately linked data or inaccurate conversion tables.
We recommend that the receiving contractor
replicate statistical analyses to ensure that the
specifications have been interpreted correctly.
We have included time in our budget to advise
the subsequent contractor on the interpretation of statistical specification documents, if
the Department chooses to have a subsequent
contractor undertake replication activities.
For these reasons, we recommend that a new
contractor replicate
Transfer Materials and Knowledge

During the transition to a new contractor, we
propose to use our transition specification document to guide the process. We use this document to keep track of progress and plan for
contingencies around what may or may not be
available. Our preliminary plan is organized
according to our functional teams, including
test development, psychometrics and statistics, operations, alternate assessment, score
reporting, and websites. The transition specification document captures the information outlined in Exhibit 7.7.10-1.
We note that the final transition specification
document will include quality assurance steps
for every element to ensure that the information delivered is correct. This may include
identifying sources or formats for information that may be less convenient for the new
contractor to use but are more reliable and
accurate.
For each area of activity, we complete the full
transition specification for each detailed item
and track it throughout the transition process.
These specifications are necessarily quite
detailed. For example, some items that might
appear under paper score reports are
2 years of classical item statistics;

2 years of IRT calibrations;
statistical codes;
2 years of linkages; and

2 years of conversion tables.
These replications will help ensure that the
item parameters imported to the new contractors item bank represent the appropriate versions and that they have been appropriately
linked. Even the smallest discrepancies will
raise a red flag, and we commit to advising the
new contractor in their investigation to determine the cause of any differences.
Page 4-58
scoring and reporting processing rules;

technical specification for statistical
calculations;
input data layouts;
reporting static content;
reporting variable content;
training support materials and documents;
printing specifications; and
focus group/research plans and results.
Exhibit 7.7.10-1: Transition Specification Document

Element
Description
Item Number
A number assigned to each element for inventory purposes
Team
The subject matter team that ultimately generates the material (e.g., psychometric/technical, reporting, scoring, customer service)
Item/Material
The data, knowledge, or other information required
Description
Detailed information about the item/material
Origination Entity/Person
The name of the person or entity responsible for ensuring the delivery
Reference or Source
A source of the information known to be accurate
Preferred Format
The new contractors preferred format in which to receive the information
Best Available Format
The format in which we will actually deliver the information
Backup Format
An alternate format in which the data are also available in case a backup is necessary and where the best available format
differs from the preferred format
Quality Control Plan
Where possible, the plan for ensuring the accuracy of the information
Delivery Date
The agreed-on delivery date
Actual Delivery Date
The actual delivery date
Provided By
The name of the person or entity at AIR who delivered the material
Receiving Entity/Person
The name of the person or entity at the new contractor who received the material
Completion Date
The date that the full transition of the material was complete
Status
1 = complete; 0 = incomplete
Page 4-59
Conclusion
Within 30 days following acceptance by the
Department of the final program report, AIR
will transfer the information enumerated in
the transition specifications document to the
Department on an ongoing basis and work
with the incoming vendor to provide the data
in a format that may be integrated with its systems. This will ensure transition materials support the reproduction of key testing program
elements.
Further, as required by the Department and
upon its authorization, AIR will destroy all
data files remaining in our possession once all
handoffs are made to the next contractor, and
provide written verification of destruction to
the Department.
If we are the selected contractor, we will commit to supporting the next vendors transition
as we would hope to be supported as an
incoming contractor.
Page 4-60
Appendix H:
Sample Communications Plan
Schedule updates
Updates to key
information posted
to the online
Portal
Online
Assessment
System
Deliverable/
Description
Project
Management
Phase
ODE and AIR
ODE and AIR
Stakeholder
(Audience)
Posted to ODE
ftp site
Delivery
Method
Based on
upcoming
milestones
and issue
Weekly
Timing/
Frequency
AIR
Project
Director/
ODE
Project
Director
AIR
Project
Director
Author
Describe the communication types, content categories, ownership and dates of communications in the following matrix.
Communications Plan
AIR Project
Director
AIR Project
Director
Responsible
Party
(Sender)
Ongoing
Occurs weekly on time
Status
Appendix G:
Sample Family Reports
Student Name:
School:
Complex Area:
Test Date:
Jane Doe
Aloha Elementary School
Ewa
20122013
FAMILY Report
Dear Doe Family:
Reading &
Mathematics
AF
The Hawaii Department of Education is pleased to send you this report

about Janes performance on the Online Hawaii State Reading and
Mathematics Assessments. The Reading and Mathematics
Assessments are designed to test students on the Hawaii Content and
Performance Standards (HCPS III) learned in the previous school year.
The standards describe what students should know and be able to do in
reading and mathematics.
Assessment Results
Students are permitted to take each assessment up to three times during the
school year. This report shows Janes best performance on the assessments for
each subject, which counts as her official score for that subject.
In addition to showing how well Jane did on the assessments, this report compares
her scores with those of other students in her school, her complex area, and the state.
It explains the different areas of the Reading and Mathematics Assessments. Also,
the report shows whether Jane reached proficiency in these areas and suggests how
you can help her improve her reading and mathematics skills.
The report also includes Janes Lexile measure that indicates her current reading
ability based on the Reading Assessment. The Lexile measure can help you select
appropriate reading materials for your child to read at home and also monitor her
progress as she works to become a better reader over time. More information about
Lexile measure can be found on page 4 of this report.
For additional information, I encourage you to talk to Janes teacher about this report,
what it means, and how you can help.
What is in this report?
Janes scores on the Reading and Mathematics

Assessments
Janes Lexile measure
How Janes scores compare
The areas that make up the Reading and

Mathematics Assessments
Whether Jane reached proficiency in the

different areas of each subject
How you can help Jane improve her Reading

and Mathematics skills
Very truly yours,
Kathryn S. Matayoshi
Superintendent of Education
Grade
20122013
For more information

about this assessment, go to
www.alohahsap.org
Photograph: Lignum Vitae

2007 Selvin Chin-Chance
FAMILY Report
Janes
Score: 376
How does this compare?

Janes Reading score is 376. This score
is higher than the average score of third
graders in her school, higher than that of
third graders in her complex area, and
higher than that of third graders
statewide.
300
266
Your School: 336
Your Complex Area: 320
State Average: 317
A students test score can vary if the test is

taken several times. If your child was tested
again, it is likely that Jane would receive a
score between 358 and 394.
DOES NOT MEET STATE STANDARD
376
324
MEETS STATE STANDARD
Janes Reading Score
Exceeds Proficiency - Students can determine the

meaning of words by using word parts, explain the purpose
of literary and informational texts based on how they are
organized, and compare and contrast parts of texts.
Meets Proficiency - Students can identify the meaning of
difficult words and main ideas in texts and can describe
how different texts are organized.
Approaches Proficiency - Students can recognize the

meaning of new words, make predictions about events and
characters in texts, and give opinions about stories.
Well Below Proficiency - Students at this level

demonstrate skills and understanding of Reading below the
performance needed to reach the Approaches Proficiency
level.
Using a home or library computer, go to www.alohahsap.org
to see a complete listing of skills for each level.
Near
The test does not always provide enough information

to tell if a student has reached the Meets Proficiency
mark for this area of Reading.
AF
Students may recognize sounds and letters, read thirdgrade-level vocabulary, and find the meaning of new words.
They may be able to read different types of grade 3 texts.
Conventions and Skills

Use knowledge of the conventions of
language and text to create meaning for
different literary and informational texts
for several purposes.
Has Jane Reached Proficiency in the Different Areas of Reading?
Yes
Literary Response and Analysis

Respond to literary texts from many
points of view: personal (like giving
opinions), interpretive (like explaining
main ideas or events), and critical (like
explaining the difference between
figurative and literal language).
Yes
The score is at or above the Meets Proficiency range

for this area of Reading.
Reading Comprehension
Use reading strategies (like
understanding how texts are organized;
making predictions; finding main ideas,
problems, or solutions) to create meaning
from different texts.
2
2
Students in this range understand most of what they read.

Students understand main ideas and events and can make
predictions. They can also identify problems and solutions.
The score is at or above the Meets Proficiency range

for this area of Reading.
Students in this range understand literary texts by giving

opinions, figuring out main ideas and events, and
understanding that language can be literal or figurative (for
example, "he was a bear" means that the man was very big
and grumpy). They understand the authors purpose for
using figurative language.
Next Steps
For example, read out loud to your
child, especially books that are too
difficult for her to read on her own.
Discuss new vocabulary by asking
what the words mean (e.g., "What is a
dog doing if it snarls?") and showing
that words have different parts (e.g.,
singing = sing + ing).
For example, have your child read

different types of texts, like chapter
books, nonfiction, and magazine
articles. Encourage her to guess the
end of the story. Ask your child why
some events might take place and
what the authors purpose or goal
might be.
For example, have your child read

different types of literary texts, like
poems, chapter books, and short
stories. While your child reads, ask her
opinion of the text (e.g., Why did a
character act a certain way?). After
reading, ask your child to analyze the
story (why the events happened). Help
your child pick out examples of similes
(e.g., "Rex is like a bear") and idioms
(e.g., "I will take a rain check on the
offer").
Reading and Mathematics Assessment Results

How does this compare?
Janes
Score: 280
Janes Mathematics score is 280. This

score is lower than the average score of
third graders in her school, lower than
that of third graders in her complex area,
and lower than that of third graders
statewide.
343
300
268
A students test score can vary if the test is

taken several times. If your child was tested
again, it is likely that Jane would receive a
score between 268 and 292.
280
Your School: 328

Your Complex Area: 318
State Average: 312
Janes Mathematics Score
Exceeds Proficiency - Students can explain differences

between congruent and similar shapes, generalize rules for
numerical and spatial patterns, interpret data that are
presented in numerous ways, and justify the ordering of
fractions and use them to solve problems.
Meets Proficiency - Students can solve problems with whole
numbers and fractions, understand the concept of area and
volume, classify shapes as similar, create numerical patterns,
and represent data.
Approaches Proficiency - Students can solve basic addition
and subtraction problems, choose correct measurement tools,
recognize basic geometric shapes, and collect and organize
data.
Well Below Proficiency - Students at this level demonstrate
skills and understanding of Mathematics below the
performance needed to reach the Approaches Proficiency
level.
Using a home or library computer, go to www.alohahsap.org
to see a complete listing of skills for each level.
Has Jane Reached Proficiency in the Different Areas of Mathematics?

The score is below the Meets Proficiency range for this
area of Mathematics.
Numbers and Operations

Understand numbers, ways of representing
numbers, and relationships among numbers
and number systems; understand the
meaning of operations and how they relate
to each other; use computational tools (like
calculators) and strategies well; and use
estimation when needed.
No
Measurement
Near
AF

mark for this area of Mathematics.
Students may select and use standard tools to measure

length, capacity, and weight with standard and metric units;
be able to find the elapsed time between two events; and
understand the concepts of area and volume.
Understand attributes, units, and systems of

units in measurement (e.g., metric system)
and develop and use techniques, tools, and
formulas for measuring.
Students may be able to add and subtract 1-digit numbers

but may have trouble adding and subtracting 2- and 3-digit
numbers, may understand place value (for example, 24 is 2
tens and 4 ones), and may be able to recognize number
sentences that have the same value.
Geometry and Spatial Sense

Analyze properties of objects and
relationships among the properties, use
transformation and symmetry to analyze
mathematical situations, use different
representational systems including
coordinate geometry, and use visualizations
and spatial reasoning to solve problems.
No
Patterns, Functions, and Algebra
No
Understand different types of patterns and

functional relationships and use symbolic
forms to represent, model, and analyze
mathematical situations.
Data Analysis, Statistics, and Probability

Ask questions and collect, organize, and
represent data to answer questions; interpret
data through exploratory data analysis; make
and evaluate inferences, predictions, and
arguments that are based on data; and
understand and apply basic notions of
chance and probability.

Students may have difficulty recognizing and remembering

some basic properties of shapes (e.g., triangles have 3
sides), knowing the differences between congruent and
similar shapes (e.g., two squares are similar if one is a larger
version of the other), and recognizing transformations (e.g.,
flips, turns) of shapes.
Students may not be able to recognize numerical patterns

(like 1, 5, 9, ...) or identify situations involving change (like
how the height of a plant changes over time) and may have
difficulty recognizing the model that represents an expression
(like a picture that represents 1 + 3 = 4).
Near

mark for this area of Mathematics.
Students may be able to organize and show data in more

than one way, answer questions about the data set, and
predict how likely it is that an event will take place (e.g., if a
bag contains the numbers 1, 3, and 5, pulling a 2 is not
likely).
Next Steps
For example, help your child add
and subtract money. Talk about
how a 1-dollar bill is 4 quarters, 1
quarter is 2 dimes and 1 nickel, and
1 quarter is 1/4 of a dollar.
For example, work with your child to

estimate how many pictures are
needed to cover the refrigerator
door. Work with her to figure out
how many cups of water are
needed to fill a pitcher.
For example, have your child draw

different triangles. Discuss why they
are different (e.g., a right triangle
has a 90-degree angle). Turn the
paper clockwise a little and discuss
how a triangles size and shape
stay the same, even after the turn.
For example, play a game with your

child by saying, "2, 5, 8, 11." Ask,
"What is the next number?" Help
her describe the pattern (adding 3
to each number).
For example, have your child put 10

red marbles, 8 blue marbles, and 1
green marble in a bag. Ask her how
likely it is to pull a green marble
from the bag. Discuss why it is
unlikely that she will pull a green
marble.
FAMILY Report
Similar Schools
The schools that were compared with your school were

chosen because their third-grade students had backgrounds
most similar to third-grade students in your school. Aloha
Elementary School teaches many disadvantaged students
and many students with disabilities.
Mathematics
2%
6%
57%
63%
41%
31%
About 2% of similar schools did

better than your school.
About 6% of similar schools did

better than your school.
About 57% of similar schools

had about the same average
score as your school.

had about the same average
score as your school.

did not do as well as your
school.
AF
Students come from many different environments. These

differences do not necessarily affect student performance.
Many issues contribute to student performance, such as
administration and oversight, curriculum and content,
teaching and testing, professional development, instructional
materials, parent and community support, and funding. You
may want to contact your parent community networking
coordinator to inquire about parent workshops that support
the schools ongoing effort to improve student performance.
Reading

did not do as well as your
school.
This chart shows how third-grade students in Janes school

did compared with third-grade students in other schools
when tested on concepts and skills that were taught to these
third-grade students last year. School similarity is assessed
by three criteria: percentage of (1) disadvantaged students,
(2) English language learners, and (3) students with
disabilities.
Lexile Measure
Reading Lexile Measure
The Lexile Framework for Reading is a scientific approach to

measuring readers and reading materials. A key component of
the Lexile Framework is a number called the Lexile measure.
A Lexile measure is a standard score that matches a students
reading ability with the difficulty of textual material. A Lexile
can be interpreted as the level of text that a student can
read with 75 percent comprehension. Experts have identified
75 percent comprehension as the level at which students
can read with a certain amount of comfort while still being
challenged.
Both a Lexile text measure and a Lexile reader measure are
indicated using a simple number followed by an L (e.g.,
850L) and are placed on the Lexile scale. The Lexile scale
ranges from below 200L for beginner-level readers and text to
above 1700L for advanced-level readers and text. A readers
recommended Lexile range is 50L above and 100L below his
or her Lexile measure. This is the range between the easiest
kind of reading materials for a student and the most difficult
level at which he or she should be able to read.
Janes
Lexile Measure
985L
The Online Hawaii State Reading Assessment is linked to the
Lexile Framework to provide educators and parents with an
additional indicator of a students reading ability. A student must
take the Reading Assessment and receive a score in order
to have a Lexile measure. Because the Lexile measure is a
measure of textual difficulty, it does not take into account age
appropriateness, student interest, or the content of the text.
For more information on interpreting Lexile measure, please go to www.alohahsap.org.
4
4
FA M I LY S C O R E R E P O RT
Prepared for
The Family of Jane Smith

Birth Date: 03/07/2000
Test Date:
Spring 2013
School:
Abc Es (012345)
District:
Abc City Sd (012345)
hio
Department
Education
of
Grade 5 Reading, Math and Science Achievement Assessments

Dear Family,
In May, Jane took the Grade 5 Reading, Math and Science achievement assessments. These tests
measure how much your student has learned at the end of the school year in relation to what is
expected in the Ohio Academic Content Standards.
This report explains what Janes test scores for spring 2013 mean. The report also includes ideas
for ways that you can help her improve. I urge you to talk to Janes teacher about the report and
what can help keep Janes learning on track.
Next school year, Ohio will move from the 10-year-old Ohio Academic Content Standards to
Ohios New Learning Standards. The new standards will call for your child to learn more in
class than the old standards did. But the Ohio Achievement Assessments next year will be based
on a combination of the old and new standards.
AF
I also want you to know that there will be new State and Local Report Cards. The Local
Report Card grades your school district and school on how well they are doing in educating
their students. The new 2013 Local Report Cards are parent and student friendly and show grades
from A-F on nine measures of student progress in learning. Ask your principal to explain the new
Report Card for your school.
Sincerely,
Dr. Richard A. Ross

Superintendent of Public Instruction
We wish Jane much success in her learning. Thank you for all you do to support her in school.
Quick Facts About Your Child and State Achievement Assessments

Q. Why does my child take achievement
assessments?
The Reading, Math, and Science Achievement Assessments help ensure
that all students have learned the level of reading, math, and science
expected of Ohio students at the end of the fifth grade.
Q. What happens if my child does not pass

the Grade 5 Achievement Assessments?
These test results are one of the factors, but not the sole factor, that
teachers use to evaluate a childs readiness to move to the next grade.
Classroom teachers determine whether your child should move to Grade
6 by considering her class work and school attendance.
If your child does not score in the proficient or above levels, this is a sign
Spring 2013 012345.128
that she is having trouble learning the skills and knowledge expected of
fifth-grade students and may need additional help. This report includes
suggested activities to help your child practice specific skills. Your childs
teacher may be able to recommend additional ways your child can
improve her achievement.
hio
Department
Education
of
Reading Achievement Results
534
Advanced - Students go beyond simply
understanding what they read and can make
good judgments about the authors use of
fact versus opinion.
She has performed at

the proficient level and
meets standards for
Grade 5 Reading.
384
State Average Score: 416
Proficient - Students understand what

they read. They try to explain how an
authors choice of words creates the storys
mood and describes the storys setting.
Basic - Students understand some of what
they read. They use strategies (e.g., authors
definitions and examples) to learn the
meaning of new words.
AF
400
Accelerated - Students understand what

they read (e.g., plays, biographies,
newspapers) and can summarize stated and
unstated themes.
441
Janes score
is 416.
459
District Average Score: 399
School Average Score: 394
247
Limited - Students may struggle with

simple reading tasks (e.g., plays, biographies,
newspapers).
Dierent versions of the test are used each time the test is given. The scores in this report are used to compare performance from year to year.
Scores in dierent subjects cannot be compared.
Has Jane reached proficient in the different areas of Reading?

This chart shows you how well Jane performed in each
standard. She is near proficient in Acquisition of
Vocabulary, near proficient in Reading Process, above
proficient in Informational Text, and near proficient in
Literary Text.
Below
Near
Above
Proficient Proficient Proficient
Acquisition of
Vocabulary
Reading Process
Informational Text
Literary Text
Note: There is some uncertainty in all test scores. Please see the interpretive guide for more information.
E D U C A TION.OHIO.G OV
Spring 2013 012345.128
THE OHIO DEPARTMENT OF EDUCATION
hio
Grade 5 Reading, Math & Science Achievement Assessments
Department
Education
of
OHIO ASSESSMENT SYSTEM
releasedTest
test questions
Teacher Score Report | Grade 3See
Maththe
Achievement
from earlier years achievement
assessments at
What Are Your Childs Strengths and Weaknesses in Reading?
www.success.ode.state.oh.us
Acquisition of Vocabulary
Jane Scored Near Proficient
Students use clues in the text, knowledge of

word parts (e.g., underneath) and knowledge of
where words come from (e.g., autograph; graph
means to write in Greek) to learn new words.
They understand figurative words (e.g., the bus
crawled through the traffic).
WHAT THESE RESULTS MEAN
NEXT STEPS
Your child is learning to use word parts (e.g.,

eraser) and clues in the texts to learn the
meaning of new words. She understands
figurative language (e.g., the saw bit into the
wood). Your child can also abbreviate words
(e.g., St. for Street; Ave. for Avenue).
Have your child hunt for words with a specific

part, such as prefixes (e.g., un-, unhappy; pre-,
preview) and suffixes (e.g., -ing, writing; -less,
careless). Choose other word parts and do the
searches together.
Students make and support predictions about

what they read, and they summarize
information. They can check their
understanding and read in different ways (e.g.,
skimming, re-reading). They can use outlines to
understand what they read.
NEXT STEPS
Your child is learning to gather ideas from

different places in texts (e.g., dramas,
biographies, newspapers). She uses new
vocabulary and an understanding of the storys
events to tell what will happen next in the story.
Have your child read and retell part of a book

for you. Have your child explain how the
characters feel in the story or the reasons certain
events took place in the story. Ask her to use
details from the story for support.
AF
Informational Text
Reading Process
Jane Scored Above Proficient
NEXT STEPS
Your child is able to pull out complex new

information from what she reads. She
understands how the author organizes ideas
(e.g., as they happen, in question-answer
format). This helps her understand the authors
message.
Have your child preview difficult informational

texts, such as articles in magazines or
newspapers. Have your child read a text that
includes a chart and explain the relationship
between the chart and the article.
Students use text features (e.g., subheadings) to

locate important information. Students compare
information from different sources, summarize
main ideas, and distinguish between important
and unimportant details.
Literary Text
Students identify how the main events of a story

affect what happens later in the story. They
analyze setting, character and theme and how
they affect each other. They explain how
authors use figurative words (e.g., metaphors,
similes) to create mood.
NEXT STEPS
Your child is learning to explain how different

elements of a story (e.g., setting, character,
theme) influence the story as a whole. She is
learning to identify stated and unstated themes.
Read the same story your child is reading and

ask her to describe how the literary elements
(e.g., action, character, plot, setting) influence
each other. How does the setting affect the plot?
How do characters affect one another? What is
the theme?
E D U CA T I O N.OHIO.G OV
W WW.O DE.STATE.OH.US
Spring 2013 012345.128

T H E O H I O D EPA RT M ENT OF EDU C AT I ON
hio
Department
Education
of
Math Achievement Results
548
Advanced - Students can solve new,
complex problems (e.g., find the area of an
irregular shape). They explain their reasoning
using math terms (e.g., probability) and
notation.
424
382
Proficient - Students can solve common

problems with fractions, percents and tools
(e.g., protractor). They estimate solutions and
use informal reasoning skills.
Basic - Students can remember math
terms (e.g., mean) and concepts (e.g., using
variables). They carry out routine procedures,
such as finding areas and reading graphs.
AF
400
Accelerated - Students combine math

ideas to solve problems (e.g., use
multiplication to find area). They explain their
reasoning (e.g., use terms and graphs to
describe data).

the proficient level and
meets standards for
Grade 5 Math.
Janes score
is 422.
439
245
Limited - Students struggle with math

terms (e.g., ray) or concepts. They have
trouble adding decimals and negative
numbers and using words to describe
number patterns.
Has Jane reached proficient in the different areas of Math?

Below
Near
Above

standard. She is above proficient in Number, Number
Sense, and Operations, near proficient in Measurement,
below proficient in Geometry and Spatial Sense, near
proficient in Patterns, Functions and Algebra, and near
proficient in Data Analysis & Probability.
Number, Number
Sense and Operations
Measurement
Geometry and
Spatial Sense
Patterns, Functions
and Algebra
Data Analysis and
Probability
Spring 2013 012345.128
hio
Department
Education
of
See the released test questions from earlier years achievement assessments at
Teacher Score Report | Grade 3 Math Achievement Test
What Are Your Childs Strengths and Weaknesses in Math?

Number, Number Sense and Operations
Jane Scored Above Proficient
Students add and subtract fractions and

decimals; show numbers in different ways
(e.g., 1 = 2/2 = 1.0 = 100%); use rules of
operations (+, -, , /); and understand ratios (1
to 2 = 1/2), negative numbers and perfect
squares (e.g., 81 = 9 9). They solve
problems in many ways.
NEXT STEPS
Your child can add, subtract and round

fractions, decimals and percents. She can
create and use equivalent forms of fractions
(e.g., 2/6 = 1/3), decimals (e.g., 1/2 = 0.5) and
percents (e.g., 3/4 = 0.75 = 75%). She can make
estimates and determine their reasonableness.
Have your child find examples of ratios

around the house (e.g., 5 blue plates to 9 total
plates). Help her use ratios to solve related
problems such as, "The ratio of apples to
bananas is 3 to 2. There are 15 apples. How
many bananas are there?"

NEXT STEPS
Your child can estimate, measure and draw

angles and make conversions between units
(e.g., feet to inches). She can find perimeter,
area and volume and the length of paths on a
grid.
When measuring length, capacity, weight, or

time, give your child the measure in one unit
and ask her to give you the measure in a
different unit. For example, ask her to give
you her height in feet and then in inches.
Students draw and measure angles. They

develop formulas for finding perimeter, area
and volume; describe the difference between
surface area and volume; and make
conversions among measurement units (e.g.,
inches to feet).
AF
Measurement
Jane Scored Below Proficient
NEXT STEPS
Your child can use geometric terms but may

have trouble identifying the relationships
among parts of circles. She may not
understand properties of congruent (same size
and shape) figures.
Have your child search for real-world

examples of geometric shapes. For example,
pies are often cut from the center, so the edge
of a slice is a radius. Also, streets may be
examples of parallel and perpendicular lines.
Students identify relationships among parts of

circles (radius, diameter, circumference);
understand congruence; plot negative values
on a grid; and recognize two-dimensional
shapes that can be folded to make a threedimensional object.
Patterns, Functions and Algebra
Students can write rules to represent patterns;

use variables as unknown quantities; create
and interpret equations and inequalities; and
use graphs and tables to draw conclusions and
make predictions.
Data Analysis and Probability

Students read, construct and interpret
frequency tables, circle graphs, line graphs,
and more complex displays of data. They
understand the meaning of range, mean,
median, and mode. They use ratios to
represent the probability of simple events.
Spring 2013 012345.128
NEXT STEPS
Your child can use rules, models, tables, and

graphs to represent and describe patterns. She
can create equations to represent problems, use
variables to represent unknowns and describe
the rate of growth over time as constant or
varying.
Tell your child a rule (e.g., multiply by 5) and

a number, and have her use the rule to tell you
what number the rule would generate. Have
her determine her own rule and use words, a
table or a graph to show how the rule works.

NEXT STEPS
Your child can read and construct graphs. She

can identify the probability of events, list all
possible outcomes in a situation, and identify
the range, mean, median, and mode of the set
of data.
Find a graph in a magazine or newspaper and

ask your child questions such as "What does
the graph tell you?" "Could the data be
displayed in a different kind of graph?" Ask
how the graph might change if additional data
were collected.

hio
Department
Education
of
Science Achievement Results
565
Advanced - Students apply science facts

and recognize relationships. They design
investigations, draw conclusions, and apply
knowledge to new situations.

400
Proficient - Students know science ideas

and use them to make connections. They
plan and carry out investigations and use the
results to draw logical conclusions.
Basic - Students know some science facts
and may know how they are connected. They
may rely on simple ideas and facts to make
obvious predictions.
AF

the basic level and is not
meeting standards for
Grade 5 Science.
Accelerated - Students use science

information to discuss patterns; make
reasonable predictions; make plans to collect
data; and evaluate technological solutions.
Janes score
is 396.
417
448
363
239
Limited - Students may know few science

facts and might have difficulty describing
models or relationships. They may struggle
to plan science investigations.
Has Jane reached proficient in the different areas of Science?

Below
Near
Above

standard. She is below proficient in Scientific Processes,
near proficient in Physical Sciences, near proficient in
Earth & Space Sciences, and near proficient in Life
Sciences.
Scientific Processes:
Inquiry, Technology &
Ways of Knowing
Physical Sciences
Earth & Space
Sciences
Life Sciences
Spring 2013 012345.128
hio
Department
Education
of
releasedTest
test questions
Teacher Score Report | Grade 3See
Maththe
Achievement
What Are Your Childs Strengths and Weaknesses in Science?
from earlier years achievement

assessments at
Scientific Processes: Inquiry,

Technology and Ways of Knowing
Students design and safely conduct
experiments. They know how to share their
results with others and they use observations
to make their own conclusions. Students learn
about the effects of technology. They use a
process (method) to solve problems.
Jane Scored Below Proficient

NEXT STEPS
Your child may not be able to design or make

conclusions from experiments. She may not
be able to describe the effects of technology.
She may be able to use a process to solve
problems, but may not be able to explain that
process.
Review your childs notes from a class

experiment. Ask her questions that help her
explain the experiment to you. Talk with her
about the effects of technology on your lives
and how scientific processes (like record
keeping) are used daily.
Earth and Space Sciences
NEXT STEPS
Discuss common physical changes (like ice

melting) and chemical changes (like iron
rusting). Show her how electricity flows
through a circuit (like a bulb connected to a
battery by wires). Discuss how heat is
transferred between objects.
Your child may be able to describe objects by

their properties, but may struggle to compare
physical changes (like ice melting) and
chemical changes (like iron rusting). She may
not be able to fully describe the properties of
light and sound.
Students compare properties and movement of

objects. They explain physical changes (like
melting ice) and chemical changes (like iron
rusting). Students trace the flow of electricity
through a circuit (like a light bulb attached to
a battery by wires).
AF
Physical Sciences

NEXT STEPS
Your child may know Earths place in the

solar system. She knows some weather
concepts and patterns, but may have difficulty
explaining how weather and other processes
shape Earths surface. She can describe some
ways to save natural resources (like trees).
When traveling with your child, discuss how

wind, water and ice shape the land (for
example, rainfall can erode hills). Also, help
her record local weather on a calendar or map.
Life Sciences
Students compare the life cycles of plants and

animals. They know how the structures of
plants (e.g., flowers) and body parts of
animals (e.g., teeth) help them survive. They
explain how energy flows through an
ecosystem (like a forest).
NEXT STEPS
Your child may be able to describe life cycles

of plants or animals. She may have difficulty
comparing life cycles. She may be able to tell
how survival depends on changes in the
environment, and the purpose of plant and
animal parts (like leaves and wings).
Have your child describe or draw two life

cycles. Ask her to tell you what is the same
and what is different about them. Discuss with
her how plants and animals obtain food. Have
her draw a food chain and ask what eats what
and why.
Students explain Earths cycles (like seasons)

and its movement around the sun. They have
knowledge of weather concepts and patterns,
and can explain how weather and other
processes shape Earths surface. They
describe ways to save natural resources (like
trees).
Spring 2013 012345.128

hio
Department
Education
of
AF
To the Family of Jane Smith
Spring 2013 012345.128
Student Name:
Jane M. Smith
Test Date:
Spring 2013
Grade:
10
School:
ABC School 1 (654321)
District:
ABC District 1 (123456)
Birth Date:
Online Resources for the

12/31/2001

st
s
Ohi
Te
aduation
Gr
education.ohio.gov
Search for OGT
OGT
Resu ts
l
Dear Family,
In March, Jane took the Ohio Graduation Tests (OGT), which measure student progress and are one of the requirements for a high school
diploma. Your students scores are in this report.
The OGT is based on Ohio Academic Content Standards, which are what students should know and be able to do by the time they leave
high school. These standards were developed by Ohio teachers, parents and the community. Educators and state policy makers believe
that Ohio needs to and can rise to high expectations. The Ohio legislature made passing tests a graduation requirement in 1987.
Many resources are available to students on the ODE website. We hope your student is taking advantage of these resources to pass the
OGT in order to earn a high school diploma. These resources are available at www.success.ode.state.oh.us. From the main page, click on
OGT and go to the student section.
AF
If your child did not achieve the proficient level on this test, she should keep trying to pass the OGT. She can go to the ODE website and
use the information and practice tests there to help study.
This report explains what Janes test scores mean and suggests ways that you can help. I encourage you to talk to Janes teachers about
this report and resources that can help to ensure your childs success.
I also want you to know that there will be new State and Local Report Cards. The Local Report Card grades your school district
and school on how well they are doing in educating their students. The new 2013 Local Report Cards are parent and student friendly
and show grades from A-F on nine measures of student progress in learning. Ask your principal to explain the new Report Card for your
school.
We wish Jane much success. Thank you for all you do to support her in school.
Sincerely,
Dr. Richard A. Ross

Superintendent of Public Instruction
Place
postage
here
To the Family of Jane M. Smith
Spring 2013 123456.1
hio
Department
Education
of
How did Jane perform on the Ohio Graduation Tests in Spring 2013?
Reading
Writing
Science
Social Studies
553
579
591
582
448
444
476
445
446
429
425
430
425
429
415
Above
Proficient
Mathematics
554
421
Proficient
400
400
Below
Proficient
383
384
262
254
In Spring 2013, Jane passed the:
Proficient
400
388
Basic
414
Proficient
400
414
400
378
371
382
267
209
231
Reading OGT Writing OGT

Social Studies OGT
In Spring 2013, Jane did not pass the: Mathematics OGT
Proficient
Above
Proficient
Below
Proficient
Science OGT
Janes Overall Summary

Review the Interpretive Guide and the ODE Web site for information about
these score reports and the construction of the OGT, including detailed test
statistics.
AF
Jane has scored proficient or better on the Reading, Writing, Science,

and Social Studies OGT. Jane has not reached the proficient level on the
Mathematics OGT. Talk to Janes teachers about her performance in all
subjects and about how you can help prepare her for future education
and career options. Jane must pass the OGT in all subjects and meet
state curriculum requirements to receive an Ohio high school diploma.
For practice tests, please go to www.success.ode.state.oh.us

Jane passed the Reading OGT with a score of 415 (Proficient).
554
Advanced: Students demonstrate an overall and complete
415
Proficient
400
State: 425
District: 414
383
School: 406
262
429
448
understanding of grade-appropriate text. They are consistently able to

define unfamiliar and complex words, read for understanding and explain
and analyze the various ways authors may influence text.
Accelerated: Students demonstrate an overall understanding of
grade-appropriate text. They are able to define unfamiliar and complex
words, read for understanding, explain main ideas in informational text,
and describe and explain elements of literature.
Proficient: Students are able to define unfamiliar words, read for

understanding, explain main ideas in informational text, and describe and
identify elements of literature.
Basic: Students continue to need help defining unfamiliar words,

reading for understanding, identifying main ideas in informational text,
and describing and identifying elements of literature.
Below
Proficient
Near
Proficient
Above
Proficient
i
j
i
i
j
i
i
j
i
i
j
i
Spring 2013 123456.1
Jane has passed the OGT in reading,

scoring at the proficient level.
Encourage her to build on her success
by taking challenging English courses.
Talk to Jane about her reading interests,
and encourage her to speak with
members of the community to find
careers where reading skills can be
used.
Did you know?
Limited: Students need help understanding grade-appropriate text.

They struggle with defining unfamiliar words, reading for understanding
and recognizing the main idea in text.
Reading Content Standard Performance
Help Jane prepare for posthigh school success in reading
The Public Policy Institute of California

has found that taking above-level
English courses is predicted to increase
future earnings more than taking
average-level English courses.
Acquisition of Vocabulary
Reading Process
Informational Text
Literary Text
hio
Department
Education
of
Jane did not pass the Mathematics OGT. She scored 388 (Basic).
553
Advanced: Students demonstrate deep knowledge and skills across
School: 397
425
400
388
Basic
384
254
444
District: 408
Proficient: Students demonstrate understanding of mathematical

concepts and terms. They use informal reasoning and some problemsolving strategies. They adequately communicate using mathematical
language.
Basic: Students demonstrate recall and recognition of mathematical

concepts and terms. They use routine procedures to solve basic problems.
Students may have difficulty using mathematical language to
communicate.
Limited: Students demonstrate inadequate recall and recognition of
mathematical concepts and terms. They may have difficulty performing
routine computations and communicating ideas using mathematical
language.
Near
Proficient
Above
Proficient
j
j
i
j
i
i
i
j
i
j
i
i
i
i
i
Jane has not passed the OGT in

mathematics, scoring at the basic level.
Discuss Janes strengths and
weaknesses in mathematics with her
teachers. Ask them about exercises you
can encourage Jane to complete to help
her prepare for the next administration
of the Mathematics OGT.
Jane performed below proficient in
Number, Number Sense & Operations,
Measurement and Patterns, Functions &
Algebra. Visit the OGT Student Web site
at www.success.ode.state.oh.us to see
questions from previous
administrations. Encourage Jane to
focus on the problems that address her
weaknesses. Discuss the mathematics
used in the problems with her.
AF
Mathematics Content Standard Performance

Below
Proficient
Help Jane pass the next

Mathematics OGT
State: 429
the mathematical standards. They display abstract thinking in their

approach, analysis and justification to problem solving. They
communicate using precise mathematical language and notation.
Accelerated: Students demonstrate an understanding of
similarities and differences between various mathematical concepts, terms
and procedures. They evaluate and justify solutions in problem situations
and communicate in clear and concise mathematical language.
Number, Number Sense and Operations

Measurement
Patterns, Functions and Algebra
Data Analysis and Probability
Jane passed the Writing OGT with a score of 421 (Proficient).

579
Advanced: Students demonstrate an excellent understanding of the
421
Proficient
400
State: 424
378
District: 413
School: 406
267
430
476
writing process. They demonstrate exceptional control of spelling,

grammar, punctuation, and capitalization as well as revising, editing and
developing writing.
Accelerated: Students demonstrate a complete understanding of
the writing process. They demonstrate very good control of spelling,
grammar, punctuation, and capitalization and very good skill revising,
editing and developing writing.
Proficient: Students demonstrate a satisfactory understanding of the

writing process. They demonstrate adequate control of spelling, grammar,
punctuation, and capitalization and some skill revising, editing and
developing writing.
Basic: Students demonstrate minimal understanding of the writing

process. They demonstrate some control of punctuation, spelling,
grammar, and capitalization and some skill revising, editing and
developing writing.
Limited: Students demonstrate an inadequate understanding of the
writing process. They have difficulty with organizing their writing and
show little evidence of revising content or editing for errors in grammar or
sentence structure.
Help Jane prepare for posthigh school success in writing

Jane has passed the OGT in writing,
by continuing to enroll in writing
classes. Meet with Jane and her
counselor to discuss her interest in
writing. Help Jane research writingrelated careers and extracurricular
activities.
Did you know?
According to the College Board,
government economists expect job
growth for writers to be faster than the
average for all careers through 2014.
Writing Content Standard Performance

Below
Proficient
Near
Proficient
Above
Proficient
i
i
i
i
i
j
j
j
i
Spring 2013 123456.1
Writing Conventions
Writing Applications
Writing Process
hio
Department
Education
of
Jane passed the Science OGT with a score of 414 (Proficient).

591
Advanced: Students show superior knowledge and reasoning
Proficient
400
State: 419
District: 398
371
School: 386
209
Proficient: Students explain scientific concepts, show working

knowledge of investigative processes and technological applications. They
recognize and represent relationships and describe interactions and
changes in living, physical and Earth and space systems.
Basic: Students show some familiarity with valid scientific facts,

concepts and investigative processes, and technological applications.
They show some understanding of basic models and identify and classify
parts in some living and physical systems.
Limited: Students demonstrate less than basic understanding and
skills. Problem areas include: understanding scientific facts, processes and
technological applications, and identifying and classifying parts in living
and physical systems.
414
425
445
abilities in the sciences. They design scientific investigations and evaluate

technological solutions for real-world situations and evaluate models and
predict effects in living and physical systems.
Accelerated: Students show solid knowledge and reasoning
abilities in the sciences. They revise and critique scientific investigations
and technological solutions for problems and analyze and explain
interactions and relationships in living and physical systems.
Below
Proficient
Near
Proficient
Above
Proficient
i
j
i
j
i
i
j
i
j
i
i
i
Jane has passed the OGT in science,

by continuing to take science courses.
Help Jane research science-related
professions and discuss ways that Jane
can continue her science education
after graduation.
Did you know?
According to the U.S. Department of
Labor, seven of the projected top 10
fastest-growing occupations in the
United States for 2004-2014 are science
related.
AF
Science Content Standard Performance
Help Jane prepare for posthigh school success in science
Scientific Processes: Inquiry, Technology and Ways of Knowing

Physical Sciences
Earth and Space Sciences
Life Sciences
Jane passed the Social Studies OGT with a score of 414 (Proficient).
582
Advanced: Students typically can make judgments about social
414
Proficient
400
State: 425
District: 412
382
School: 406
231
429
446
studies issues. They can: discuss causes and effects, choose between roles
of government, analyze effects of changes in the Constitution, and
balance issues of rights and responsibilities.
Accelerated: Students typically can analyze issues across most
areas of social studies content. They can: establish cause and effect, give
details about the roles of government, explain changes to the
Constitution, and analyze responsibilities of citizens.
Proficient: Students typically can explain issues in most areas of

social studies content. They can: place historical events in sequence,
explain the roles of government, show how the Constitution has been
used, and give examples of rights and responsibilities.
Basic: Students do not consistently explain issues of social studies

content. Problem areas may include: identifying the order of events,
explaining the roles of government, identifying how the Constitution is
used, and discussing the rights of citizens.
Limited: Students demonstrate some social studies skills and
understandings. Problem areas may include: knowing events in history
have a sequence, understanding what governments do, recognizing parts
of the Constitution, and identifying rights of citizens.
Social Studies Content Standard Performance

Below
Proficient
Near
Proficient
Above
Proficient
j
i
i
i
i
i
j
j
i
j
i
i
Spring 2013 123456.1
Help Jane prepare for posthigh school success in social

studies
Jane has passed the OGT in social
studies, scoring at the proficient level.
by taking advantage of additional social
studies-related courses. Encourage Jane
to meet with members of the
community to research the role that
social studies skills play in their careers.
Did you know?
According to the U.S. Department of
Labor, about half of the people who
choose careers in social studies-related
fields work for federal, state and local
governments (and most of this work is
for the federal government).
History
People in Societies and Geography
Economics, Government and Citizenship Rights and Responsibilities
Social Studies Skills and Methods
Appendix J:
Sample Risk Register Format
/),),.
-
,+),/),/

,

!!

%'
" !
! !$$
*!
#,
$(
%'*
$#
$#$
&

" !

)

!&
!"
#"
# "!
$#
&(
"$
"!*
!!
!!
$ !(

Appendix K:
Sample Change Control Log
Printed: 8/23/2013
Updated: 02/08/2005
Issues
Page 1 of 1
OAKS Online Assessment

Project Lead: Meredith Durgin Date of Initial Issues Capture: DATE
Status Priority Contract
Description of Need or Issue
Owner Evaluato Detailed
AIR
Proposed Activity
Deliverable
r
Request Functional
Area
Online Reporting will work with
Complete Medium
Online
Austin Oregon Submitt
Yes
Online
Reporting has requested an additional report Fossey Dept. of
Reports PM and ODE to determine the
specifications
of the additional
within the ORD system
Ed.
deliverable.
Pending
High
Item
Oregon Oregon
Yes
Content AIR content staff will develop 20
Development has requested additional item
additional items per grade span.
Dept. of Dept. of
development for the ELPA test
Ed.
Ed.
Change Control Log
3-Jul-13
8-Jul-13
XXX
TBD
Date
Date of Estimated
Planned
Identified Decision Budget
Resolution
Date
3-Jun-13 7-Jun-13 Impact?
none
27-Jun-13
Version 0.1
Riverside Publishing Confidential
Actual
Closure Action
Comments
Resolution
Date
25-Jun-13
UAT will be completed
by AIR and Oregon
Dept. of Ed. Oregon
Dept. of Ed. will sign-off
TBD
Schedule needs to be
adjusted, and additional
resources added to the
project.
Appendix E:
Sample Panelist Evaluation Form
Delaware End of Course (EOC-2)

Standard Setting Workshop
July 30-31, 2013
The purpose of this evaluation form is to secure your feedback about the standard setting process. Your feedback
will provide a basis for evaluating the training, methods, and materials used during this workshop.
Please complete the information below.
PART I. ABOUT YOU

1.
What positions do you hold? (Mark all that apply.)

Regular Education Teacher

ESL Teacher
Special Education Teacher
Resource Teacher
Administrator/Specialist
Higher Education Instructor
Community Member (Non Educator)
Other (Please specify:_________________________)
I have a background in English as a Second

Language
I have a background in Special Education

I am a parent of a child who attends a K-12 Delaware
Public or Charter School
2.
Did you serve as a table leader during the

workshop?
Yes
No
PART II. OVERALL EXPERIENCE

1. Please read each of the following statements carefully and indicate your response.
Strongly
Agree
Agree
Disagree
Strongly
Disagree
a.
I understood the purpose of this standard setting

workshop.
b.
The standard setting method I used during this

workshop results in setting valid standards.
c.
The standard setting method I used during this

workshop was fair and unbiased.
d.
The training on the standard setting process gave

me the information I needed to complete my
assignment.
EOC Standard Setting, July 2013

Workshop Evaluation Form
e.
The training on the Performance Level Descriptors

(PLDs) gave me the information I needed to
complete my assignment.
f.
The feedback data provided gave me the

information I needed to complete my assignment.
g.
The impact data provided gave me the information

I needed to complete my assignment.
2. Please rate the clarity of the following components of the workshop.

Very
Clear
Somewhat Somewhat
Clear
Unclear
Very
Unclear
a.
Instructions provided by the Workshop Leader
b.
Ordered Item Booklet (OIB)
c.
Performance Level Descriptors (PLDs)
3. How important was each of the following factors in your placement of the bookmarks?
Very
Important
Somewhat
Important
Not
Important
a.
Performance Level Descriptors (PLDs)
b.
My perception of the difficulty of the items
c.
My experiences with students
d.
Small group discussions
e.
Large group discussions
f.
Feedback data
g.
Impact data

4. How appropriate was the amount of time you were given to complete the following components of the
standard setting process?
Too
Much
About
Right
Too
Little
a.
Training on the Bookmark method
b.
Reviewing the Ordered Item Booklet (OIB)
c.
Placement of your bookmarks in each round
d.
Round 1 discussion
e.
Round 2 discussion
PART III. STANDARD SETTING OUTCOME

Are you generally satisfied with the placement
of the cut points?
Yes
No
Please explain:
PART IV. YOUR TURN

What was the most effective portion of the training or standard setting process?
What suggestions do you have to improve the training or standard setting process?
Thank you for participating in the Delaware End of Course (EOC) Standard Setting Workshop!

Appendix D:
Sample User Role Chart
20132014 User Roles and Access to AIRs Systems
District Assessment
Coordinator (DAC)
District Administrator
(DA)
District Technology
Coordinator (DTC)
School Principal (PR)
School Assessment
Coordinator (SAC)
Test Monitor (TM)
Test Monitor with

Reporting Access (TMR)
Teacher (TE)
Test Data Entry (TDE)
System or Feature/Task
This table provides a brief overview of AIRs systems and which users can access each system and the features/tasks
within each system. For TIDE, features are referred to by their corresponding tab. The corresponding user guide for each
system contains complete information about each systems features. Note: User roles have not changed for 20132014.
District Assessment Coordinator (DAC)
District Administrator (DA)
District Technology Coordinator (DTC)
School Assessment Coordinator (SAC)
Test Monitor (TM)
Test Monitor with Reporting Access (TMR)
Teacher (TE)
Test Information Distribution Engine (TIDE)

Features
Administration Tasks
My Account (change password)
Contact Information
Edit DAC contact information and district shipping
and mailing addresses
Orders
Supplemental Information
Initial Orders
Additional Orders
Order History
Order Summary
Track Shipments
Student Information
Search/View Student Information
Edit Student Test Settings and Accommodations
- Color Choices
- Print Size
- Accommodated Text-to-Speech (TTS)
Manage Users *
View/Edit Users
Add Users
Upload Users
* DAC, DA, and SAC users can only view, add, or modify the users in their district and/or school who have a lower role
level. DACs cannot add or modify other DAC users, and SACs cannot add or modify other SAC, PR, or any district-level
users.
Page 1 of 2
Posted 8/19/2013
School Assessment
Coordinator (SAC)
Test Monitor (TM)
District Technology Coordinator (DTC)
School Assessment Coordinator (SAC)
Test Monitor (TM)
Test Monitor with Reporting Access

(TMR)
District Technology
Coordinator (DTC)
District Administrator (DA)
Teacher (TE)
District Administrator
(DA)
District Assessment Coordinator (DAC)
Test Monitor with

Reporting Access (TMR)
District Assessment
Coordinator (DAC)
Teacher (TE)
Invalidations
Indicate Invalidations (online assessments only)
Create and Edit Assignments
View Student and Class Assignment Reports
View Invalidations
Upload Class Rosters
General Resources (download NeoSpeech voice pack)
Non-Disclosure (complete electronic agreement)
Test Administration and Data Entry Sites

Online Testing System (Test Monitor Interface)
Data Entry Interface
Test Administration Training Site

Test Monitor Training Site **
Online Reporting System Features

View Score Reports
View Test Management Center Reports
Create Class Rosters
Manage Class Rosters
Learning Point Navigator Features ***

Browse Resources
** The Test Monitor Training Site allows authorized users to create practice test sessions that students can log into using
the Online Item Samplers. For additional information, refer to the Test Monitor User Guide for the Online Testing System,
which is available on the Resources section of the Minnesota Assessments portal.
*** While any user may access Learning Point Navigator, only those who have a school-level role and who are
associated with student rosters can create and edit assignments for those students.
Page 2 of 2
Posted 8/19/2013
Appendix L:
Sample Handscoring Reports
Totals
103,624
364
348
Handscore
Handscore
Handscore
Grade 08 Math Q062
Grade 08 Math Q063
Grade 08 Math Q064
Grade 08 Math Q065 Form Handscore

A
Grade 08 Math Q065 Form Handscore

B
103,624
103,624
Handscore
103,624
79
84
91
85
87
81
86
415,208
Grade 08 Math Q061
Current Autoscore
Current Handscore
%EX
2X
Inter-Rater Reliability
20
15
14
13
16
13
%AD
9:11:31 AM
1 of 1
Time:
Page:
LEAP Spring 2013 - ELA285

Grade 08 | Subject: Mathematics
%NA
1,674
1,682
103,624
103,624
103,624
103,624
417,852
Total
24
37
14
%0
29
29
29
30
22
27
27
%1
23
18
22
26
30
22
%2
19
11
29
17
29
31
27
%3
14
12
12
23
15
%4
%B
Blanks: Include
Item: ALL - Cumulative
Scoring Center: ALL - Team: <No Team>
Priority: ALL
11/27/2013
Date:
Item Summary Report
256
309
376
370
14
17
389
326
558
203
408
316
288
442
449
422
429
438
523
370
639
7249
8610
9165
9221
9312
11871
13005
13625
14548
14990
15113
15118
15738
15842
16949
16962
16963
16990
17037
17165
19350
10,494
1049
Current Handscore
83
84
86
84
83
86
84
84
81
80
77
87
82
88
83
100
100
83
84
85
87
100
84
17
16
14
16
17
14
16
16
19
20
22
13
18
12
17
17
16
15
13
16
2X %EX %AD %NA
Grade 08 Science Q047
Totals
11:35:51 AM
1 of 2
Time:
Page:
LEAP Spring 2013
3,520
2,012
2,910
2,461
2,377
2,449
2,133
2,452
1,621
1,723
2,244
1,150
3,067
1,805
1,887
24
135
2,128
2,077
1,699
1,285
57,009
50
63
49
63
61
63
58
56
53
64
55
58
67
60
49
71
78
59
58
62
60
33
58
29
25
27
25
24
22
24
27
27
24
23
25
21
25
28
13
26
23
24
24
67
25
13
10
15
10
13
10
14
13
15
19
15
11
15
10
15
12
Total %0 %1 %2
25
%B
Blanks: Include | Item Format: All
Item: Grade 08 Science Q047 - Cumulative
Scoring Center: <ALL> - Team: None
Priority: <ALL>
Grade 08 Science
12/2/2013
Date:
379
554
19370
19555
300
19369
33
201
19360
353
319
19359
19439
471
19358
19438
339
10,494
19352
Current Handscore
83
80
79
84
82
80
83
88
87
84
17
19
18
16
18
20
16
12
12
16
2X %EX %AD %NA
Grade 08 Science Q047
Totals
11:35:51 AM
2 of 2
Time:
Page:
LEAP Spring 2013
2,137
1,908
201
3,200
1,415
1,006
1,741
2,425
1,814
57,009
61
59
50
61
56
49
62
61
55
58
25
28
25
24
28
37
23
23
22
25
12
11
24
14
14
11
13
11
12
Total %0 %1 %2
11
%B
Blanks: Include | Item Format: All
Item: Grade 08 Science Q047 - Cumulative
Scoring Center: <ALL> - Team: None
Priority: <ALL>
Grade 08 Science
12/2/2013
Date:
Grade 08 Writing Question Q001 (Qualifying Set #1)
4097
4098
4132
4190
4216
4278
4279
4289
4292
4296
4352
4359
4449
4459
11221
11391
11435
11437
11455
11583
12633
13407
13497
14450
14765
14799
14806
14809
Set Description
3520
Reader
Time:
Page:
PASS Writing Spring 2013

Grade 08 Writing Q001
100
80
100
90
75
95
100
85
100
95
100
100
100
100
100
100
90
100
100
100
100
85
90
80
100
85
100
100
80
20
10
25
15
10
15
10
15
15
20
%EX %AD %NA
Content and
Development
100
80
100
90
75
95
100
85
100
95
100
100
100
100
100
100
90
100
100
100
100
85
90
85
100
85
100
100
80
20
10
25
15
10
15
10
15
15
20
%EX %AD %NA
Organization
75
70
95
75
65
75
90
75
90
90
85
90
85
90
70
85
90
100
95
95
100
100
85
70
80
80
85
90
80
25
30
25
35
25
10
25
10
10
15
10
15
10
30
15
10
15
30
20
20
15
10
20
%EX %AD %NA
Voice
70
70
80
80
55
70
95
70
85
80
70
90
90
90
95
70
70
85
80
80
90
70
75
70
70
80
75
100
80
30
25
20
20
30
20
30
15
15
30
10
10
10
30
30
10
20
20
10
30
25
30
30
20
25
20
15
10
%EX %AD %NA
Conventions
Set: Grade 08 Writing Question Q001 (Qualifying Set #1) Team: <ALL> Reader: <ALL>
Date:
TQR Domain Reader Results Report
1 of 4
3:01:22 PM
11/26/2013
Time:
Page:

Set Description
Reader
15141
15691
15791
16440
16444
16553
16554
16683
16843
16853
16868
17177
17236
17239
17591
17609
17719
17928
17939
17950
18127
18133
18240
18241
18242
18421
18430
18514
18516
100
85
95
90
90
95
80
90
45
90
80
100
95
80
100
100
90
100
100
85
90
90
85
90
90
100
100
80
85
15
10
10
20
10
55
10
20
20
10
15
10
10
15
10
10
20
15
%EX %AD %NA
Content and
Development
100
85
95
90
90
95
80
90
45
90
80
100
95
80
100
100
90
100
100
85
90
90
85
90
90
100
100
80
85
15
10
10
20
10
55
10
20
20
10
15
10
10
15
10
10
20
15
%EX %AD %NA
Organization
75
70
80
70
70
95
75
75
30
85
75
85
100
80
85
95
85
90
90
85
85
80
85
100
70
85
85
70
70
25
30
20
30
30
25
25
70
15
25
15
20
15
15
10
10
15
15
20
15
30
15
15
30
30
%EX %AD %NA
Voice
70
75
70
70
80
70
70
85
45
75
70
75
70
70
80
85
85
80
70
80
75
70
85
90
70
80
70
90
70
20
20
25
30
15
20
20
15
35
25
20
25
25
30
20
15
15
20
25
20
25
25
15
10
25
20
25
10
30
10
10
10
20
10
%EX %AD %NA
Conventions
Date:
2 of 4
3:01:22 PM
11/26/2013
Time:
Page:

Set Description
Reader
18520
18531
18578
18634
18643
18658
18737
18797
18863
19155
19156
19157
19158
19159
19160
19161
19162
19163
19164
19165
19167
19168
19170
19171
19172
19173
19174
19175
19176
65
75
90
85
70
60
75
60
80
95
80
70
75
100
95
80
90
80
90
60
100
90
85
90
95
80
80
100
80
30
25
15
30
35
25
35
20
15
30
25
20
10
20
10
40
10
15
10
20
15
20
%EX %AD %NA
Content and
Development
65
75
95
85
70
65
75
60
80
95
80
70
75
95
95
80
90
80
90
60
100
90
85
90
95
80
85
100
80
30
25
15
30
30
25
35
20
15
30
25
20
10
20
10
40
10
15
10
20
15
20
%EX %AD %NA
Organization
60
80
80
80
75
60
70
45
85
70
55
75
75
100
70
75
70
70
80
55
95
75
75
70
85
75
70
95
75
40
20
20
20
25
40
30
45
15
30
45
20
25
30
25
30
30
20
45
25
25
30
15
25
30
25
10
%EX %AD %NA
Voice
30
75
80
70
75
50
75
30
80
70
30
70
70
80
75
75
70
50
75
35
80
70
80
80
70
70
80
80
70
50
25
20
25
20
30
25
50
15
25
45
20
30
20
25
25
25
35
15
60
15
20
10
20
25
25
20
15
30
20
20
20
25
10
15
10
10
10
%EX %AD %NA
Conventions
Date:
3 of 4
3:01:22 PM
11/26/2013
Time:
Page:

Set Description
Reader
19177
19178
19179
19180
19181
19182
19183
19184
19185
19186
19187
19188
19189
19190
19191
19192
19193
19194
19195
19196
19221
19222
19231
85
75
100
95
65
85
80
100
90
95
90
80
70
70
65
75
85
80
80
75
100
100
75
15
25
35
15
20
10
10
15
30
30
30
25
15
20
20
25
25
%EX %AD %NA
Content and
Development
85
70
100
95
80
85
80
100
90
95
90
80
70
70
65
75
85
80
80
75
100
100
80
15
30
20
15
20
10
10
15
30
30
30
25
15
20
20
25
20
%EX %AD %NA
Organization
85
50
85
70
75
90
75
95
90
80
70
80
50
75
75
80
70
70
80
75
95
90
70
15
50
15
30
25
10
25
10
20
30
20
50
25
20
20
30
30
20
25
10
30
%EX %AD %NA
Voice
70
40
70
70
25
85
75
90
80
70
70
70
40
75
40
75
70
70
75
70
75
90
70
25
45
30
20
60
15
20
10
15
25
30
30
45
25
50
25
25
20
25
15
25
10
10
15
10
15
15
10
10
15
20
%EX %AD %NA
Conventions
Date:
4 of 4
3:01:22 PM
11/26/2013
350
Total %
35
35
35
35
35
35
35
35
Total #
10
35
35
#R
Paper
10
11
TS Lo LO
NA
92
323
91
32
97
34
94
33
89
31
97
34
83
29
89
31
97
34
91
32
94
33
EX
16
14
HI
Hi
NA
Grade 08 English Language Arts Question ELA Q036 (Qualifying Set #1)
Set: Grade 08 English Language Arts Question ELA Q036 (Qualifying Set #1) Team: <ALL>
Page: 1 of 1
Time: 11:45:21 AM
LEAP/GEE Summer 2013

Grade 08 ELA Q036
Date: 11/27/2013
TQR Item Detail Report
10011820
10012956
10012958
10012968
10012970
10012985
10012986
10012990
10012996
10013212
10013219
10019355
10019402
10019425
10019426
10043304
10081272
10081277
10093283
10093382
11/19/13
11/19/13
11/19/13
11/08/13
11/19/13
11/19/13
11/19/13
11/08/13
11/19/13
11/08/13
11/08/13
11/08/13
11/08/13
11/19/13
11/19/13
11/19/13
11/19/13
11/19/13
11/19/13
Lithocode
11/19/13
First Dealt
Grade HS Math Q034
Totals
49
51
49
49
47
51
50
49
49
51
49
49
51
51
51
52
47
50
50
49
TS
#NA
Lo
1,145
#R
#NA
Lo
#R
Cumulative
Team: <ALL>
#EX
#LO
46
48
46
40
41
51
50
40
47
51
42
49
40
47
50
35
43
50
50
45
#EX
58 1,049
#LO
Grade HS Mathematics
EOH315
Validity Item Detail Report
16
#HI
31
#HI
#NA
Hi
#NA
HI
%NA
Lo
%NA
LO
18
13
14
12
14
%LO
%LO
94
94
94
82
87
100
100
82
96
100
86
100
78
92
98
67
91
100
100
92
%EX
92
%EX
Page:
Time:
Date:
31
%HI
%HI
%NA
HI
%NA
HI
1 of 2
12:46:19 PM
11/27/2013
10093383
10103001
10103052
11/19/13
11/19/13
Lithocode
11/19/13
First Dealt
Grade HS Math Q034
Totals
48
52
51
TS
#NA
Lo
1,145
#R
#NA
Lo
#R
Cumulative
Team: <ALL>
#EX
10
#LO
46
41
51
#EX
58 1,049
#LO
Grade HS Mathematics
EOH315
Validity Item Detail Report
#HI
31
#HI
#NA
Hi
#NA
HI
%NA
Lo
%NA
LO
19
%LO
%LO
96
79
100
%EX
92
%EX
Page:
Time:
Date:
%HI
%HI
%NA
HI
%NA
HI
2 of 2
12:46:20 PM
11/27/2013
ELA308 GEE Retest

Grade HS English Language Arts
Validity Reader Detail Report
Date:
12/2/2013
Time:
11:20:00 AM
Page:
1 of 2
Cumulative
Reader: 7840
Item: Grade HS ELA Q022
Lithocode
Score Time
TS
RS
3113625
11-08
3124306
11-08
3132280
11-08
3133069
11-08
3109582
11-08
3111179
11-08
3113014
11-08
3113138
11-08
3113181
11-08
3113196
11-08
3113499
11-08
3113665
11-08
3113675
11-08
3113689
11-08
3113700
11-08
3113730
11-08
3115256
11-08
3115293
11-08
3115315
11-08
3119119
11-08
3119140
11-08
3120115
11-08
3122964
11-08
3124271
11-08
3124311
11-08
3124325
11-08
3124427
11-08
3130990
11-08
3131227
11-08
3132245
11-08
3132385
11-08
3132387
11-08
3136238
11-08
3136553
11-08
ELA308 GEE Retest

Grade HS English Language Arts
Validity Reader Detail Report
Date:
12/2/2013
Time:
11:20:00 AM
Page:
2 of 2
Cumulative
TOTAL
3136721
11-08
3138544
11-08
3139534
11-08
3139679
11-08
3139772
11-08
3141992
11-08
#R
#EX
#HI
#LO
%EX
40
36
90
%HI
%LO
Appendix N:
Smarter Balanced Draft Policy
Availiability of Consortium
Resources to Non-members
Consistent with the priorities of its member states, the Smarter Balanced Assessment Consortium is
committed to maintaining the value of Consortium membership while creating opportunities for
students in non-member states to benefit from the Smarter Balanced Assessment System.
Value Proposition of Consortium M em bership
Governing members in the Consortium have control over:
Cost drivers for assessment implementation: Members control the length of the
test, the emphasis of content on the test, and the types of items that will be presented to
students
The operational budget: Members develop and approve the Consortiums membership
fees and expenditures
The scope of services: Members are able to ensure that the Consortium provides the
services that their members and stakeholders require
Value Proposition for M aintaining and Enhancing the Assessm ent System
Consortium governing members will work together to maintain and enhance the usefulness and
value of the system by:
Developing and calibrating additional secure assessment items to better measure the depth
and breadth of College and Career Readiness Content Standards
Developing additional non-secure assessment items to better allow teachers to measure the
progress of students throughout the school year
Developing additional formative tools and supports to better improve teaching and learning
Supporting the open source implementation of the Smarter Balanced test administration,
data warehouse, item archive and authoring tool, and digital library
Monitoring item exposure, exposure and use of other secure and non-secure assessment
resources
Maintaining existing and securing new copyright agreements
Protecting the security of Consortium materials
Providing communication materials and supports to service providers as required to support
the system
Consortium Approach to Sustainability
The Consortium intends to fund services to its members and enhance the assessment system
through the collection of annual membership fees to be established by the member states.
Availability of Consortium Resources to Non-M em bers
The Consortium intends to make Consortium resources available to non-members under the
following conditions:
Non-members will not have the privileges of membership including but not limited to the:
o Authority to contribute to or approve policy decisions
Availiability of Consortium
Resources to Non-members
Authority to approve the Consortiums annual budget (i.e., membership fee structure
and expenditures)
Non-members will:
o Pay a per-student fee equal to the applicable membership fees. (This fee will provide
the resources required to address the applicable distribution and management
requirements of both secure and non-secure materials
o Agree to maintain the security of Consortium materials and use materials
appropriately based on the Consortium test administration manuals and applicable
copyright agreements
o Agree to Consortium certification requirements if the non-member plans on reporting
results on the Smarter Balanced scale
The Consortium reserves the right to require a security bond contingent on the Consortiums
analysis of the security risks posed by non-member use of Consortium materials
o
Appendix F:
Sample Interpretive Guides and User Guides
Assessment System
Online Reporting System User Guide
20132014
Delaware Department of Education, 2013

Descriptions of the operation of the Test Delivery System and related systems are property of the American
Institutes for Research (AIR) and are used with the permission of AIR.
Delaware Department of Education, 2013

Descriptions of the operation of the Test Delivery System and related systems are property of the American
Institutes for Research (AIR) and are used with the permission of AIR.
Table of Contents
Introduction.............................................................................................................................................................. 1
Additional Resources ......................................................................................................................................................... 1
Computer and System Requirements............................................................................................................................. 1
User Support....................................................................................................................................................................... 1
Section I. Overview of the Online Reporting System................................................................................................ 2
Score Reports (Home)......................................................................................................................................................... 3
Test Management Center................................................................................................................................................... 3
Inbox ................................................................................................................................................................................... 3
Manage Rosters................................................................................................................................................................... 3
Search Students .................................................................................................................................................................. 3
New Features and Changes for 20132014 ...................................................................................................................... 4
Score Reports: Home: Reporting Student Counts......................................................................................................... 4
Score Reports: Test Events Filter ................................................................................................................................... 4
Score Reports: Individual Score Reports: Test Accommodations................................................................................. 4
Score Reports: Print Student Score Reports to PDF...................................................................................................... 4
Test Management Center: Plan and Manage Testing: New Parameters....................................................................... 4
Test Management Center: Plan and Manage Testing: Updated Test Statuses............................................................. 4
Section II. Logging into the Online Reporting System............................................................................................. 5
Navigating between DCAS Applications (CommonLoginSystem)................................................................................. 7
Section III. Understanding the Online Reporting System Layout............................................................................. 8
Online Reporting System General Tools............................................................................................................................ 9
Section IV. Search Students (Student Lookup)....................................................................................................... 10
Searching for Students...................................................................................................................................................... 10
Understanding Student Search Results and Selecting a Test......................................................................................... 11
Viewing a Selected Students Test Results.................................................................................................................... 12
Section V. Score Reports.......................................................................................................................................... 13
User Roles in the Online Reporting System.................................................................................................................... 13
Score Reports: Getting Started........................................................................................................................................ 14
Select Test and Year....................................................................................................................................................... 14
View Scores by All Students or Current Students ....................................................................................................... 15
Understanding the Home Page Report Tables............................................................................................................. 15
About Score Reports Dimensions.................................................................................................................................... 16
Navigating between Score Reports and Dimensions................................................................................................... 16
Exploration Menu...................................................................................................................................................... 17
Score Reports Layout and Features.................................................................................................................................. 18
Breakdown By Demographic Subgroup........................................................................................................................ 18
Breakdown By Test Event.............................................................................................................................................. 19
Comparison (On/Off).................................................................................................................................................... 19
Download Student Data File ........................................................................................................................................ 20
Score Reports Legends.................................................................................................................................................. 20
% Completed Column for DCAS Spring Tests.............................................................................................................. 20
Reports in the Who Dimension: Subject Detail Reports............................................................................................. 21
Aggregated [Grade] [Subject] Scale Score and Performance Levels of Students in [Group]...................................... 21
Sample District-Level Report........................................................................................................................................ 22
Exploring the Data......................................................................................................................................................... 22
Sample Teacher-Level Report........................................................................................................................................ 23
iii
Sample Roster-Level Report.......................................................................................................................................... 23

Sample Student Listing Report..................................................................................................................................... 24
About the Student Listing Report............................................................................................................................. 24
Sample Individual Student Report............................................................................................................................... 25
Reports in the What Dimension: Reporting Categories.............................................................................................. 26
Aggregated Scale Score and Performance Levels for [Grade] [Subject] Reporting Categories for Students
in [Entity]..................................................................................................................................................................... 26
Exploring the Data......................................................................................................................................................... 26
Sample Reporting Categories Report........................................................................................................................... 26
Sample Strengths and Weaknesses by Benchmarks Report........................................................................................ 27
Finding Resources in Learning Point Navigator....................................................................................................... 28
Reports in the When Dimension: Trends..................................................................................................................... 29
Aggregated [Grade] [Subject] Scale Score Trend of Students in [Entity]................................................................... 29
Dropped Students...................................................................................................................................................... 30
Hiding Trend Lines..................................................................................................................................................... 31
Zoom Feature............................................................................................................................................................. 31
Sample District Trend Report (by Reporting Category and SubgroupBreakdown).................................................. 32
Sample Teacher Trend Report ...................................................................................................................................... 33
Sample Roster Trend Report ........................................................................................................................................ 34
Sample Student Listing Trend Report ......................................................................................................................... 35
Section VI. Manage Rosters..................................................................................................................................... 36
Section VII. Test Management Center.................................................................................................................... 39
Summary Statistics........................................................................................................................................................... 39
My Testing Windows..................................................................................................................................................... 41
Retrieve Student Results.................................................................................................................................................. 42
My Inbox........................................................................................................................................................................ 43
Opening and Viewing the Data File........................................................................................................................... 44
Plan and Manage Testing (Updated)................................................................................................................................ 45
Selecting the Parameters for Your Report.................................................................................................................... 45
Understanding Your Participation Report................................................................................................................... 47
Participation Reports Tools........................................................................................................................................... 50
Exporting Participation Data into Excel....................................................................................................................... 51
Test Completion Rates (New Test Management Center Report)................................................................................... 52
Selecting the Parameters for Your Report.................................................................................................................... 52
Appendix: Data Available in Score Reports............................................................................................................. 54
Aggregated Subject Detail................................................................................................................................................ 54
Aggregated Reporting Categories.................................................................................................................................... 55
Aggregated Benchmarks................................................................................................................................................... 55
Student Listing and Individual Student Reports............................................................................................................ 56
iv
Introduction
The Delaware Comprehensive Assessment System
(DCAS) includes all students who participate in
statewide testing (DCAS and DCAS Alt1). The Online
Reporting System provides data for each student who
completes at least one DCAS, DCAS Alt1, or End of
Course (EOC) test, as well as participation reports to
determine who has completed testing and who needs to
either complete testing or take another test opportunity
in a subject. Students who do not take any DCAS,
DCAS Alt1, or EOC tests are not included in the
Online Reporting System.
Additional Resources
User Guide Organization

This document provides information on the major features
of the Online Reporting System, including the Score
Reports, Test Management Center, and Manage Rosters.
Section I.
Section II.
Section III.

Section IV.
Section V.
Section VI.
Section VII.
Appendix:
Overview of the Online Reporting System

Logging into the Online Reporting System
Understanding the Online Reporting
System Layout
Search Students (Student Lookup)
Score Reports
Manage Rosters
Data Available in Score Reports
The DCAS Subjects and Benchmarks document provides

a comprehensive list of all benchmarks by content area
(subject). Administration manuals and other system user
guides for DCAS are also available. All documents are available on the DCAS portal (de.portal.airast.org).
Computer and System Requirements

This user guide does not provide information on computer or system requirements. Only supported operating
systems and web browsers can be used to access DCAS applications. This information is outlined in the System
Requirements document, also available on the DCAS portal.
User Support
If this user guide does not answer your questions,
contact the DCAS Help Desk at the American Institutes
for Research (AIR) or the Delaware Department of
Education.

DCAS Help Desk Contact Information
Phone: 877.560.8331
E-mail: DCASHelpDesk@air.org
If your user account is unavailable, you cannot

otherwise access the Online Reporting System, or
you have other technical questions, contact the
DCAS Help Desk at AIR.
For situations in which student data are
incorrect or missing, please contact Jim Hertzog
or Lisa Alexander at the Delaware Department of
Education.
If you have specific questions about DCAS Alt1
test data, please contact the DCAS Alt1 Help
Desk.
MondaysFridays (except holidays)

Hours: 5:00 a.m. to 10:00 p.m.
DCAS Alt1 Help Desk Contact Information

Phone: 877.206.7039
E-mail: DCASalt1helpdesk@air.org
MondaysFridays (except holidays)
Hours: 7:00 a.m. to 6:00 p.m.
In order to assist with your issue, provide the Help Desk

with detailed information that may include the following:
The SSID of the student who was affected by the issue

Any error messages that appeared
Operating system and browser information (if a functional question)
Section I. Overview of the Online Reporting System

The DCAS Online Reporting System (ORS) contains two major applications: Score Reports and the Test
Management Center.
The Score Reports (previously Performance Reports) provide score data for each of the DCAS, EOC, and
DCAS Alt1 tests. Users can compare score data between individual students and the school, district,
or overall state average scores. Information on benchmark strengths and weaknesses is also available by
reporting category.
(Note: Test Administrator [TA] users cannot access score reports data.)
The Test Management Center provides participation data for students who are taking the DCAS,
DCASAlt1, or EOC tests. Users can determine which students need to complete testing and who may
need to take another opportunity of a test. Users can also view participation summary statistics (count and
percentages) of students who tested in a selected subject and grade level.
All reports and features are accessible at any time from the top of the screen, just below the DCAS banner. The
[Score Reports] button links to the Online Reporting System home page.
Online Reporting System Home Page
Score Reports (Home)

Score reports allow authorized users to view aggregated scores for districts, schools, teachers, and rosters, as well
as individual student performance data for the DCAS, EOC, DCAS Alt1, and Norm-Referenced/TerraNova (2010
2011) tests. Users can take advantage of data analyses to determine what strategies may need to be implemented to
improve teaching and learning.
Score reports features include:
The ability to download rosters as a data file from any aggregation.
Enhanced disaggregation with the Breakdown By tool, which allows users to sort individual students or
rosters by subgroups such as gender.
A % Completed opportunity column, which displays the total percentage of students in the aggregation who
have completed each of the two test opportunities delivered during the spring DCAS testing window (to
date). (This column does not appear for the DCAS Alt1 or EOC tests.)
Benchmark reports, which provide a granular analysis of a rosters relative strengths and weaknesses. These
reports also link to Learning Point Navigator. (These reports are available for DCAS Mathematics, Reading, and
Science tests only.)
A test condition events filter (Test Events), which allows users to view data by accommodation code or type.
This feature is available for the subject detail reports. (Note: This feature is available beginning with the Fall
2013 test administration.)
Reminder: Student performance data in the Online Reporting System are not the final quality checked data used for accountability purposes.
For more information, refer to Section V.

The Test Management Center allows authorized users to generate customized reports showing students activity in
the online testing system, view summary statistics, and retrieve data.
For more information about the Test Management Center and participation report filters, refer to Section VII.
Inbox
The Inbox link in the header row is a quick link to the Test Management Centers Retrieve Student Results section
and allows you to access student performance data files. You can also access batch Individual Score Report files in
PDF format. These files are generated from the Student Listing Report page.
Manage Rosters
You can add or edit rosters in the Online Reporting System. These rosters are also used in Learning Point Navigator.
For more information, refer to Section VI.
Search Students
The student search feature allows you to find students by SSID or by first or last name and directly access their
test results. This feature searches for score report results for students, not their test status(es) for the current test
window. For more information, refer to Section III.
New Features and Changes for 20132014

Score Reports: Home: Reporting Student Counts
On the Home page, student counts will now be displayed in the Percentage of Students Proficient and Percentage of
Students Tested tables.
In the Percentage of Students Proficient table, the count will display the number of students in the selected
school or district who achieved a proficient or better rating out of the total number of students who took the
subject and grade test (e.g., 47/75).
In the Percentage of Students Tested table, the count will display the number of students in the selected school
or district who took the test out of the total number of students eligible for the subject and grade test (e.g.,
35/100).
Score Reports: Test Events Filter

The Test Events filter will allow users to filter score data based on a selected accommodation (e.g., text-to-speech). This
feature is available starting from the current test year. The report data will filter into two groups: those who had the
selected accommodation enabled, and those who did not have the accommodation.
This filter is available on the Subject Detail Report and Student Listing Report. The Individual Student Report will also
display a table listing the accommodation(s) the student had.
Score Reports: Individual Score Reports: Test Accommodations

The Individual Score Reports will now display test accommodations information for the selected subject test. This
information is available beginning with Fall 2013 assessments.
Score Reports: Print Student Score Reports to PDF

The Individual Student Report is now also available as a downloadable PDF. Simply click [Print] to generate the file,
which can be saved.
On the Student Listing Report, the [Print] button will generate a single PDF file containing the Individual Student
Report for each student listed. The generated file will be delivered to the Inbox, where it can be downloaded and saved.
Test Management Center: Plan and Manage Testing: New Parameters

In addition to selecting the test, administration, and test name for the participation reports, you can also filter by
demographic subgroup as well as test condition events.
Test Management Center: Plan and Manage Testing: Updated Test

Statuses
Upon selecting the test type (e.g., DCAS or DCAS Alt1, the status filter will update automatically to only display
the test statuses that apply to the selected test). Statuses that do not apply to the alternate assessments will not be
displayed.
Section II. Logging into the Online Reporting System

All DCAS application sites, including the Online Reporting System, are accessed through DDOEs Identity Management
System (IMS) application.
1. Access the DDOE IMS login page at https://login.doe.k12.de.us. Use your regular IMS ID and password to log in.

2. After you have successfully logged into IMS, you will see a list of applications available to you, including DCAS.
Click the [DCAS] button from the IMS menu. You will be directed to the DCAS portal.
3. Click either the [Test Administrators/Coordinators] or [Administrators and Teachers] button to access DCAS
applications.

4. Click the [Online Reporting System] button. If you are authorized to access this application, you will be
automatically directed to the Online Reporting System home page. You will not have to log in again.
Navigating between DCAS Applications

(CommonLoginSystem)
After you have logged into a DCAS system, you can switch between applications without needing to log out or log in
again. This Common Login System (CLS) integrates the following DCAS applications:

TIDE
Test Administrator Interface
TA Practice and Training Site
Data Entry Interface
DCAS Alt1 Score Entry Interface
CLS is designed to ease the login process and simplify navigation between DCAS applications. After logging into your
account, you will remain signed into DCAS as long as one of your browser windows remains on a DCAS system. (System
timeout rules apply.)
The top left corner of your browser contains a drop-down menu listing all DCAS systems. Your current application is
displayed to the left of this drop-down menu.
Select the application you want to switch to using the drop-down menu. Although navigating to another DCAS system
is easy, it is important to understand how the system operates in order to avoid unintended consequences of switching
systems.
If you are also using the TA Interface application and you log out of the Online Reporting System,
this action will also log you out of the TA Interface. Your test session will stop and all students in the
session will be logged out. You cannot resume your session. You will have to create a new session and
your students will have to log in again to resume testing.
Section III. Understanding the Online Reporting System Layout

All Online Reporting System features are available from any page in the system. Links are visible just below the DCAS
banner.
Online Reporting System Home Page With Test Management Center Drop-Down Menu
The [Score Reports] button links to the score reports.

The [Test Management Center] button provides a drop-down menu containing the four Test Management
Center sections:
oo Summary Statistics and My Testing Windows
oo Retrieve Student Results (Download Data Files)
oo Plan and Manage Testing
oo Test Completion Rates (New!)
The [Inbox] button is a quick link to the Retrieve Student Results section in the Test Management Center.
The [Search Students] button opens a pop-up window that allows you to search for a student by SSID or by first
or last name and access his or her test results from a current or previous test administration.
The [Manage Rosters] button allows authorized users to add and manage their own rosters.
All authorized DCAS users except Test Administrators (TA) can access all features. Test Administrators may access the Test
Management Center. Test Administrators with Reporting Access (TAR) can access all features of the Online Reporting System.
Important: Use the on-screen buttons and tools to navigate within the Online Reporting System. Do
not use your web browsers back button.
Online Reporting System General Tools

At the top of every screen, below the DCAS banner, are three tools that you can use: Help, Print, and Export. The
Definitions tool is also available on many screens in the same location.
The [Help] button links to this Online Reporting System user guide.
The [Print] button lets you print what you see on the screen.
oo On the Student Listing Report, this feature allows you to print either the students report or reports for all
the students in the selected roster (for the selected test).
The [Export] button lets you export the student data that are visible on the screen. The exported file will be a
Microsoft Excel (.xls) file.
The [Definitions] button provides definitions for terms specific to the report you are viewing.
Section IV. Search Students (Student Lookup)

The Online Reporting System includes a feature to look up students by SSID, first name, or last name. This feature is
useful if you need to find a students score reports but do not know the students grade or school. Note: You cannot view
students who are not associated with your school or district.
Searching for Students

Users can access this feature by clicking the [Search Students] link, which is in between the Manage Rosters and Inbox
links. This feature searches for score report results for students, not their test status(es) for the current test window.
If any information for a student is incorrect, the students profile must be updated in eSchool and
DELSIS. DDOE uploads student information to TIDE on a nightly basis. Student records should be
updated within 48 hours. After a students record is updated, he or she can begin testing.
Student Search
1. Click the [Search Students] button from any screen in the Online Reporting System.
2. Select the school year you want to limit your search to (e.g., 20122013; 20112012).
3. Enter the search criteria:
a. Enter up to 20 students full or partial SSIDs. If you are entering multiple queries, you must separate each one
with a comma (e.g., 9999999005, 9999999007).
b. Enter a students first name or last name, but not both.
4. Click the [Search] button. If the search results in a match, those students information will display.
10
Understanding Student Search Results and Selecting a Test

The search results will show a list of students who matched your search.
To view the tests a student has taken, click the [+] icon in the first column. This will expand the student row.
To view the students performance on a test, click the test name button. Be sure to note the test
administration for the test that you are selecting (listed in the left column).
Note: If you are selecting a students test that is from a different administration than the one that was selected on your
home screen, you will be prompted with a message asking you to confirm that you want to change test administrations.
Click [OK] to continue.
11
Viewing a Selected Students Test Results

This page displays the students performance for the test that you selected. As with any individual student score report,
you can compare the selected students performance with your school and districts average performance.
New: This report now includes a table displaying any accommodations that the student used for the selected subject test.
For example, the demo student above earned an Accountability Score of 939 with a conditional standard error of
measurement (SEM) of 39, and an Instructional Score of 953 with an SEM of 39. This students performance falls within
the Advanced category.
Now compare these to the average instructional and accountability scores listed in the Comparison Scores box on the
right side of the page. You can see that this students scores are higher than the average Instructional and Accountability
Scores for the school and district, as well as the average state scores of all students in the state (who have taken the
same test to date).
To return to the search results page, click the [Back to search results] link.
12
Section V. Score Reports

Score reports provide performance data to users at the state, district, school, teacher, and roster levels, as well as data
for individual students. All score reports data, except for individual students reports, can be disaggregated into AYP
categories for detailed analysis.
The Online Reporting System presents data on the aggregate and individual performance of
students as their online tests (DCAS, EOC, and DCAS Alt1) are completed and submitted. It does
not take into account any accountability rules. Data presented in this system should be considered
preliminary.
All score data can be compared with the overall state and district average for the test you are analyzing. Additionally,
you can view performance trends and determine whether overall performance is increasing.
The Online Reporting System allows you to create custom groups of students so that you can analyze
aggregated data based on shared characteristics (e.g., student athletes in an enrolled grade). Refer to
the Manage Rosters section for additional information.
User Roles in the Online Reporting System

Your user role determines the data you can view. For example, district users can view data for all schools, teachers,
classes, and students in their district; school users can view data only for teachers, classes, rosters, and students in their
school.
Table 51. Score Reports: User Roles and Access

Access Levels and Roles
State
District
State Assessment Coordinators (SAC)
District Administrators (DA)
District-Level
Data
(all districts)
School-Level
Data
Teacher-Level
Data
Roster-Level
Data
Student
Data
(all schools in
district)
School
School Administrators (SA)
Test Administrators with Reporting Access
(TAR)
Teachers (TE and TEALT)
Note: Test Administrators can only access the Plan and Manage Testing and Test Completion Rates sections in the Test
Management Center.
13
Score Reports: Getting Started

When you first access the Online Reporting System, you will see the Home page. By default, this page displays the
overall summary of score data and testing progress for your district or school, and is the starting point for data analysis.
To return to this page at any time, click the [Score Reports] button just below the Online Reporting tab in the banner.
Select Test and Year

In this section, you can select the test and administration for which you want to view score data.
Available tests include DCAS, DCAS-EOC, DCAS Alt1, and NR (Norm-Referenced/TerraNova (20102011
school year only)). DCAS is the default option.
Available administrations include the following test windows: fall, winter, and spring, through the 20112012
school year. For the 20122013 school year and beyond, it will be fall and spring administrations only. The
current test window is the default option.
Select a show scores category (see below).
14
View Scores by All Students or Current Students

This feature is particularly helpful for school personnel who want to see how new/current students did previously (e.g.,
students who moved to a school from another school).
Scores for students who were mine during the selected administration (default)
oo This view allows you to see score data for those students who tested in the selected test and
administration.
oo Note: This option enables you to see data only for students who were associated with your school or district
in the selected test and administration.
Scores for my current students

oo This option allows you to immediately view score data for only those students who are associated to your
current rosters.
oo This view also enables you to see your current students previous strengths and weaknesses before they
begin testing this year.
oo For example, if you are a grade 9 teacher and you want to see how your current students performed on
the DCAS Fall 20132014 tests, select DCAS and Fall 20132014 from the Test and Administration
drop-down lists. Data from your current students who tested in that window will be displayed in the score
reports.
Note: If you currently have a student who did NOT test in the selected test and administration, no
data will be displayed for that student.
Understanding the Home Page Report Tables
Percentage and Count of Students ProficientThis table displays the percentage and count of students to
date who have scored proficient (Meets Proficiency or Advanced) on each DCAS, EOC, and DCAS-Alt1 test. The
count is a ratio of the number of students that scored proficient on the test to the number of students that took
the test. Data in this table are by grade level for all DCAS and DCAS Alt1 Science and Social Studies tests and
by grade band for DCAS Alt1 Reading and Math tests.
Note: Data will not load in this table when viewing the 20102011 Norm-Referenced/TerraNova test because
there were no proficiency standards for this test.
Percentage and Count of Students TestedThis table displays the percentage and count of students to date
who have completed and submitted their test for scoring. The count is a ratio of the number of students who
took the test to the number of students who are eligible to take the rest. The percentage of students who tested
is calculated out of the total number of students who are currently associated with your school or district and
eligible to take the specified test.
Note: This table is not available for Show scores for my current students.
Data on this page and all other score report pages are based on the total number of students whose
tests have been scored. Students who have completed their tests but not submitted them for scoring
will not be included in these data.
To access the score reports for a particular test (by subject and grade or grade band), simply click a cell that
contains a percent value. You will be directed to the Subject Detail Report page.
15
About Score Reports Dimensions

Score reports use three dimension categories as the primary means of navigating between report levels: Who, What,
and When. This feature allows users to view data that can be analyzed appropriately, depending on whether the focus is
on testing group (e.g., school or roster), content type (e.g., subjects or reporting categories), or period of time (trend).
Note: Not all dimension categories are available at all times. Available categories are dependent on your user role and the
report type and level you are viewing.
Table 52. Score Reports Dimensions

Dimension
Description
Categories/Reports
Who
Lets you see the data for institutions and groups (district-level,
school-level, teacher-level, roster-level, and individual students).
View Institution
View Personnel
View Roster
View Students
What
Lets you explore the data by reporting categories and benchmarks

within the selected subject/content area.
View Subject
View Reporting Categories
View Benchmarks
When
Lets you see the data either as a snapshot of performance in the

current test window or as a historical trend (for the subject/content
area you are viewing).
View Trend
View Testing Window
Navigating between Score Reports and Dimensions

To navigate up or down a level or change to a different report type, simply click the magnifying glass icon [
an entity in the Name column. This action will bring up the Exploration Menu.
Sample Report Screen for a Selected Test
16
] next to
Exploration Menu
The pop-up Exploration Menu will display the available reports and levels. To view the report you want to see, click an
available cell.
Less Detailed
More Detailed
Who
What
When
Important: Use the Exploration Menu to navigate back to the upper levels of the reports from the
lower levels. Do not use your web browsers back button.
17
Score Reports Layout and Features

All score reports except individual student reports are in table format. Each column header is clickable; you can click
any header to sort the data on the screen by that column. You can also show or hide columns in the table by clicking the
arrow button [
] in the top right corner of the table and selecting the columns you want to include or hide.
Sample Report Screen for a Selected Test
Note: At the bottom of every report is a time stamp that indicates when the report was generated.
At the top of each report are two options for analyzing the report you are viewing:
Breakdown By
Comparison
Breakdown By Demographic Subgroup

The Breakdown By filter allows you to disaggregate the performance data
by a specific AYP category or a cross-section of two subgroups.
Selecting a filter will expand the on-screen report to display the data for
each subgroup. For example, selecting Gender will display the associated
performance data for male and female students. You can also select crosssections of groups, such as Ethnicity by Gender.
Table 53. Demographic Subgroups

CD504
CD504
Not CD504
ELL
ELL
Not ELL
Ethnicity
Gender
African American
American Indian/Alaskan Native
Asian/Pacific Islander
Hispanic
Multi-Racial
White
Male
Female
18
Special Education
Special Education
Not Special Education
Title 1
Title I
Not Title I
Breakdown By Test Event

Similar to the breakdown by demographic subgroup filter, the Test Event filter allows you to disaggregate the
performance data by a specific test accommodation. This feature is available starting from the current test year.
Selecting an accommodation (e.g., Language), will expand the on-screen report to display the data for each category
(e.g., if you selected Language, then the report will display All, English, or Spanish).
Comparison (On/Off)
This feature allows you to show or hide the overall score data of the state, district, and/or school that appears in the top
rows (above the green bar). The Comparison: On status will display these rows. The Comparison: Off status will hide
these rows. Simply click the [Comparison] button to change the setting.
Comparison: On
Comparison: Off
19
Download Student Data File

This feature allows you to download a data file of students who are included in the selected entity (aggregate group).
For example, if you click the download icon [ ] for a school, the data file will include all students associated with the
school, their demographic information, their testing status, and performance data for scored tests. When your file is
saved, go to the Inbox to access it. See the Retrieve Student Results section in this user guide for information on downloading
and viewing data files.
Score Reports Legends

The legend at the top of each report page provides you with the color code that corresponds to each DCAS performance
level. Refer to these legends to ensure that you understand the visual data presented.
Sample Legend DCAS Alt1
Sample Legend DCAS
Notes:
The legends vary between assessments and report types; e.g., Norm-Referenced/TerraNova tests will show
national quartiles instead of performance levels.
Legends do not appear on student-level report pages.
% Completed Column for DCAS Spring Tests

This column will display the percentage of students who have completed opportunities for the
selected test during the spring testing window. This column is displayed only for DCAS spring
Reading and Mathematics tests because students may have up to two opportunities for each test.
This column is not displayed for tests that have only one opportunity in the testing window.
Reminder:
For the DCAS fall administration, students have only one test opportunity for
each DCAS test. (This also applies to the DCAS winter administration, which
was available before the 2012-2013 school year.)
Students have one opportunity each for DCAS Alt1 and EOC tests, in the
Spring test window.
20
Reports in the Who Dimension: Subject Detail Reports

Aggregated [Grade] [Subject] Scale Score and Performance Levels of
Students in [Group]
Note about header: This is the actual name of the report that displays on screen; the bracketed text indicates a variable.
These Subject Detail Reports display overall student performance for the selected test. All data are based on the total
number of students who have taken and completed the test and submitted it for scoring. This report contains the data
shown in Table 54.
Table 54. Subject Report Column Descriptions

Column
Description
Name
The name of the entity/individual you are viewing (e.g., district, school, teacher/personnel, roster,
student).
Download Data File
This feature allows you to download test data for a single test window for a complete list of students that
are assigned to an entity.
Student Count
The number of students to date who have submitted the selected test for scoring.
Average Instructional Score
The mean Instructional Score and standard error of the mean of students who have completed the selected
test; this score includes both on-grade and cross-grade items on the test.
Note: This column is not displayed for EOC, DCAS Alt1, Norm-Referenced/TerraNova, or DCAS Science and
Social Studies tests.
Average Accountability Score
The mean Accountability Score and standard error of the mean of students who have completed the
selected test; this score includes only on-grade items on the test.
Percent (%) at Each DCAS

Performance Level
The distribution of students across the DCAS performance levels based on their Accountability Scores.
Estimated Percent (%) at Each

DSTP Performance Level
Estimate of how your students would have placed in each of the DSTP performance levels based on their
DCAS results.
Note: This column will only be displayed for DCAS Mathematics and Reading tests in the 20102011 reports.
Percent (%) Completed
The overall percentage of students in the group (district, school, roster) who have completed the selected
test and submitted it for scoring (spring testing window only).
This column is not displayed for DCAS Alt1 or DCAS Science and Social Studies tests.
For the spring 2012 and 2013 DCAS administrations, this column will show two opportunities for
Mathematics and Reading tests.
Average TerraNova Score
The mean TerraNova score and standard error of the mean of students who have completed the selected
Norm-Referenced/TerraNova test(20102011 school year only).
Average NCE Score
The mean Normal Curve Equivalent (NCE) score and standard error of the mean of students who have
completed the selected Norm-Referenced/TerraNova test(20102011 school year only).
Percent at Each National

Quartile
The distribution of students in each of the four national quartiles based on their percentile ranks from the
selected Norm-Referenced/TerraNova test (20102011 school year only).
21
Sample District-Level Report

This report is the default for state- and district-level users and shows data for the selected grade and subject. Authorized
users can view the data for each school in the district.
Exploring the Data

To explore the data further, click the magnifying glass [ ] in any of the white rows to bring up the Exploration Menu.
You can continue to lower levels of this report and view data at the teacher (personnel), roster, and student levels. You
can also switch to the Reporting Categories report or the Trend report.
Reminder: Use the Exploration Menu to navigate back to the upper levels of the reports from the lower
levels. Do not use your web browsers back button.
22
Sample Teacher-Level Report

To navigate to this report from the District-Level Report, click the magnifying glass icon next to a school name,
and then click [View Personnel]. This report will display all the teachers in that school who have students who have
completed the selected test.
Sample Roster-Level Report

To navigate to this report from the Personnel-Level Report, click the magnifying glass icon next to a teachers name
and then click [View Roster]. This report will display the teachers rosters; these rosters include students who have
completed the selected test.
Hint: To see all the rosters in the school (above the green bar), click the magnifying glass next to the schools name and
then click [View Roster].
23
Sample Student Listing Report

To navigate to this report from the Roster-Level Report, click the magnifying glass icon next to a roster name and then
click [View Student]. This report will display all the students in the selected roster who have completed the test.
Note that you can see each students scores as well as the average scores for the state, district, school, teacher, and roster.
For Reading assessments, a Lexile Measure column will display. A Lexile measure represents a
students reading ability and can be used to guide the selection of books and other materials at an
appropriate difficulty level. For additional information, visit Lexile.com.
Hint: To view all the students in a school or all the students associated with a teacher, click the magnifying glass next to
the school or teachers name (instead of clicking on a roster name), then click [View Student].
About the Student Listing Report

The Student Listing Report differs from the Subject Detail Reports in several key ways.
The SSID is displayed.

The scores are for individual students and are not mean scores (and the error band is the conditional standard
error of measurement, not the standard error of the mean).
To disaggregate students by subgroup with the Breakdown By feature, you must first choose a category from
the Breakdown By drop-down list, then choose a specific subgroup from the Values drop-down menu and click
[Go]. The new list will only include students in the subgroup you selected.
Reminder: All student data are confidential; if you export or print these student reports, ensure that
they are kept in a secure place where others cannot access them.
24
Sample Individual Student Report

To navigate to this report from the Student Listing Report, click the magnifying glass icon next to a students name and
then click [View Student].
This report displays the breakdown of the students score, the conditional standard error of measurement, and the
Instructional Scores for each reporting category as well as performance-level descriptions for the selected grade and
subject. Reminder: Student score data in this system are preliminary and not official.
If you look at the Student Subject Performance box, you can see that the students Instructional Score is 754 with an
SEM of 24, and that the students Accountability Score is 742 with an SEM of 25. Now compare these to the scores
listed in the Comparison Scores box on the right side of the page. You can see that this students scores are lower than
the average Instructional and Accountability Scores for the roster and school, but higher than the average scores earned
by the teachers students. This student's scores are also similar to scores earned by students in the district and state.
(Reminder: Average scores are compiled from the scores of those students who have taken the same test to date; averaged
computations do not include those students who have not taken or completed the test.)
Reporting category data are not provided for the DCAS Alt1 or Norm-Referenced/TerraNova tests.
The barrel chart and performance-level descriptors are not provided for the Norm-Referenced/
TerraNova test because this assessment does not have any associated performance levels.
25
Reports in the What Dimension: Reporting Categories

Aggregated Scale Score and Performance Levels for [Grade] [Subject]
Reporting Categories for Students in [Entity]
The Reporting Categories reports show the overall performance of your students in each of the reporting categories for
the selected DCAS subject. (These reports are not available for the DCAS Alt1 and Norm-Referenced/TerraNova tests.)
This report displays the number of students who have completed the test (Count) and the mean scale score and standard
error of the mean of these students.
Exploring the Data

To explore the data further, click a magnifying glass in one of the white rows to bring up the Exploration Menu. You can
continue to lower levels of the Subject Detail Report and view data at the teacher (personnel), roster, or student level.
You can also switch to the Benchmarks report or the Trend report.
Sample Reporting Categories Report

This report is the default for state- and district-level users.
26
Sample Strengths and Weaknesses by Benchmarks Report

To navigate to this report from the main Reporting Categories page, click the magnifying glass icon next to a school
name and then click [View Benchmarks]. This report will display all the benchmarks for each of the reporting categories
in the selected DCAS test. The data shown in the Comparison Scores table are the average subject scores for the selected
school.
Note: This report is not available for DCAS Grade 2 Mathematics and Reading, DCAS Social Studies, EOC, DCAS Alt1,
or Norm-Referenced/TerraNova tests.
The performance levels indicated on this report are relative to the test as a whole. Unlike performance levels provided at
the subject level, these strengths and weaknesses do not imply proficiency. Instead, they show how a group of students
performance is distributed across benchmark content relative to their overall subject performance.
For example, a group of students may have performed very well in a subject, but performed slightly lower in several
benchmarks. Thus, the minus sign does not imply a lack of proficiency. Instead, it simply communicates that these
students performed lower than expected given their higher performance across all of the other benchmarks put
together. Though the students are doing well, an educator may want to focus on these areas.
Definitions for each performance level are described in Table 55. Refer to the DCAS Subjects and Benchmarks document
on the Test Administration Resources page of the DCAS portal for a comprehensive list of all benchmarks by content
(subject).
27
Table 55. Benchmark Level Legend Descriptions

Benchmark Level
Description
Strength compared to the test for

the group selected
This benchmark is a relative strength. The group of students performed better on items from this
benchmark than they did on the rest of the test as a whole.
Performance similar to the test for

the group selected
This benchmark is neither a relative strength nor a relative weakness. The group of students
performed about as well on items from this benchmark as they did on the rest of the test as a
whole.
Weakness compared to the test for

the group selected
This topic is a relative weakness. The group of students did not perform as well on items from this
benchmark as they did on the rest of the test as a whole.
Insufficient information
Not enough information is available to determine whether this benchmark is a relative strength
or weakness.
In 20102011, this report was generated at the Topic level instead of the Benchmark level. There is
no difference between Topics and Benchmarks in the DCAS Reading Reports; however, Benchmarks
provide more detailed information than Topics for DCAS Mathematics and Science. In the 20112012
school year, Benchmark reports replaced the Topic-level reports.
Finding Resources in Learning Point Navigator

This Benchmark report also enables you to directly link to Mathematics, Reading, and Science resources for each content
area benchmark in the Learning Point Navigator application. Click the [Find Resources] button for any benchmark to
open a new browser window or tab. You will be automatically directed to the Browse Resources page. The associated
resources for the topic you selected will be automatically displayed.
For information about using Learning Point Navigator and creating assignments using available resources, refer to the
Learning Point Navigator User Guide.
28
Reports in the When Dimension: Trends

Aggregated [Grade] [Subject] Scale Score Trend of Students in [Entity]
The Trend reports show the overall performance of your students throughout the administrations for the selected test.
The average scale score and associated standard error for each testing window is plotted. (For DCAS Mathematics and
Reading, the Trend report is based on students Instructional Scores.) The graph will automatically display the data
available to you. Scores from previous years represent a groups average score or a students individual score from that
years spring testing window.
Table 56. Access Level Descriptions

Access Level
Description
District
The Choose Who to Graph table will initially display the district and all schools in that district.
School/Teacher
The Choose Who to Graph table will initially display the district, school, and all personnel associated with the
school.
Sample Trend Report
These trend reports are interactive. You can use Choose Who to Graph and What to Graph (if available) to plot
specific data on the historical graph.
29
Choose Who to Graph

In this section, you may select up to five* entities. A trend line for each selected entity will appear on the graph and
these will be color-coded.
* Note: Due to space constraints, you can select no more than five trend lines to compare at one time.
Viewing Data by Demographic Subgroup
As with the subject and reporting categories reports, you may also choose to plot data points by subgroup, e.g.,
gender. When a subgroup is selected, the Choose Who to Graph box will display the subgroup options for each
entity. Select the combination entity and subgroup(s) that you want to plot.
Dimensions
Each entity listed in this box also displays the magnifying glass icon. You can elect to drill down and view trend
data at the teacher (personnel), roster, or student levels.
What to Graph (by Subject or Reporting Category)
In this section, you may select the overall test subject data (default) or individual reporting categories to plot.
(Reminder: Reporting Categories are not available for DCAS Alt1 tests.)
Dropped Students
All trend report pages include a column called Dropped Students. If
you see a [View] button in this column, this indicates students were not
included in the trend report. If you see N/A, this means that no students
were dropped from the trend report.
Students in the aggregate grouping are dropped from a trend report if they
have not completed the selected test across all test windows. For example, a
student who took Grade 5 Mathematics in the fall and winter test windows,
but not the spring window, will be excluded from a trend report that
includes all three testing windows.
Click the [View] button to see which students were not included. A pop-up
window will appear showing the names of the students.
Why Students Are Dropped
When looking at data for a group of students over time, it is important to look at the same sample of students. If a
student was not present for all testing windows displayed in your trend report, then that student will be omitted from
the calculation of the mean scores. This helps to ensure that observed changes in mean scores are the result of actual
differences in performance and not the result of the absence of a student during a testing window. Because of this, the
average score on the trend report may not always match what you observe in the Subject Detail Report.
30
Hiding Trend Lines

After you have plotted trend lines, you may want to temporarily hide a trend line to better view data. To do so,
navigate to the box below the graph that contains the legend for the trend lines.
Simply click the trend(s) you want to hide. The trend you have selected will become grayed out and the line will
disappear from the graph.
To display the trend line again, click the trend name again. The line will reappear on the graph.
Zoom Feature
Sometimes data points are plotted very close to one
another, and it can be difficult to discern what points
belong to which entity and/or reporting category. You
can use your computers mouse to zoom in on different
areas of the graph.
To zoom in, click your mouses left button and

drag the mouse over an area of the graph that
has at least one data point. You may repeat this
action until you are satisfied with the view.
To zoom out and return to the full trend graph,
click the [Reset zoom] link at the upper left
corner of the graph.
31
Sample District Trend Report (by Reporting Category and

SubgroupBreakdown)
You can view trends by reporting category by clicking the magnifying glass icon next to the district, school, teacher,
or class name and selecting [View Reporting Categories]. As with reports in the Who dimension, you can also filter
trend reports by a selected subgroup or cross-section of two subgroups.
(Reminder: Reporting Categories are not available for DCAS Alt1 tests.)
Sample District Trend Report by Reporting Category and Ethnicity
32
Sample Teacher Trend Report

To navigate to this report, click the magnifying glass icon next to a school name and then click [View Personnel]. The
data shown are the average scale scores for the selected school.
To plot trends for specific teachers, click the check box next to each teacher whose data points you want to view.
33
Sample Roster Trend Report

To navigate to this report, click the magnifying glass icon next to a teachers name and then click [View Roster]. The
data shown are the average scale scores of all rosters for the selected teacher.
To plot trends for specific rosters, click the check box next to each roster for which you want to view data points.
34
Sample Student Listing Trend Report

To navigate to this report, click the magnifying glass icon next to a class name and then click [View Student]. By
default, the trend line displayed is for the first student shown at the top of the list in the Choose Who to Graph
box.
To plot trends for specific students, click the check box next to each student whose data points you want to view.
(Reminder: You cannot plot more than five trend lines at a time.)
35
Section VI. Manage Rosters

The Manage Rosters section allows you to create custom groupings of students for which you can analyze
aggregate data. This feature is accessible at all times except when the TIDE system is unavailable as a result of
scheduled maintenance. Click [Manage Rosters] to create and edit rosters.
After you click the icon, a pop-up window will appear.
Manage RostersStep 1
State and district users will be able to select the institution. (For school and teacher log-ins, the institution will already
be selected and grayed out.)
Manage RostersStep 2
1. In the drop-down menu, select either an existing roster or ADD NEW ROSTER to create a new roster.
2. When are you are done, click [Next].
If selecting a roster to edit, this button may take some time to load.
36
Manage RostersAdding a New Roster

1. Enter a name for the roster. This roster name will appear in the Online Reporting System and in Learning Point
Navigator. (Note: Roster names must have between five and 100 characters.)
2. Select a teacher to associate with the roster. Be sure that you are selecting an active teacher (one with current
DCAS credentials).
oo Teachers listed are those who have been identified as teacher users in DCAS and are associated with the
selected school.
3. Select the grade level(s) of the students you want to include in this roster. The list of associated students in the
school will populate the Available Students box.
Reminder: The score reports display data only for a single subject and grade (not grade band) at one time. If a
roster includes students from multiple grades, you will see only the roster-level data that correspond to the test
subject and grade you selected on the Home page. (If a roster includes students from grades 3 and 4, and you select
a Grade 3 Math test, you will see data only for the students who tested in grade 3.)
4. Select the students you want to include in the roster.
oo Press the [Ctrl] button on your keyboard to select multiple students at once.
oo Press the [Shift] button on your keyboard to select a range of students.
5. Click [Move In >>] to move the selected students into the Students in This Roster box.

6. When you are done, click [Save] to add the roster to the Online Reporting System. The Manage Rosters pop-up
window will also close and you can return to the report page you were viewing.
Note: You may add students to multiple rosters.
37
Manage RostersEditing an Existing Roster
Please remember to edit or delete rosters that were created by district or school personnel. Do not edit
rosters that were created by the Department.
The following fields are editable:
Class Name
Teacher Name (select another teacher to associate with the roster)
To remove students from the roster:

Note: This will NOT remove students from the Online Reporting System.
1. Select the students you want to remove in the roster.
2. Click [<< Move Out] to move the selected students into the Available Students box.
To add more students to the roster:
1. Select the grade level(s) of the students you want to include in this roster. The list of associated students in the
school will populate the Available Students box.
2. Select the students you want to include in the roster.
3. Click [Move In >>] to move the selected students into the Students in This Roster box.
When you are done editing the rosters attributes and adding/removing students, click [Save].
Deleting a Roster:
Note: This action CANNOT BE UNDONE. Please take care when deleting rosters. (Please delete only district- or schoolcreated rosters.)
1. Follow the steps to open an existing roster.
2. Click the [Delete] button. A pop-up warning will appear, asking you to confirm that you want to take this action.
3. Click [OK] to continue and to remove the roster from the Online Reporting System (this will also remove the
roster from Learning Point Navigator).
38
Section VII. Test Management Center

The Test Management Center allows authorized users to view summary statistics, retrieve student data, and generate
customized reports showing students activity in the online testing system. To access the Test Management Center, click
the [Test Management Center] link, which is available at the top of any page.
Summary Statistics
The Summary Statistics section allows you to view the statistics for students who have tested at each opportunity for a
selected assessment type (DCAS, EOC, or DCAS Alt1), test window (fall, winter, or spring) and test (subject and grade
or grade band).
From the drop-down lists, select the variables you want and then click [Generate Report].
Notes:
Summary statistics are not available for DCAS Alt1-SR. However, the My Testing Windows feature is available
for this test.
Depending on the amount of data being processed, the table may take a few seconds or longer to populate.
39
Table 71. Summary Statistics

Data Column
Description
% Tested at each opportunity and count
This column displays the total percent of students whose tests are in reported
status for each opportunity in the test window.
0 the % of students whose tests have not yet
Reminder: For spring 2012 and 2013 DCAS tests, students have two opportunities
reached Reported status. (This includes students each for Mathematics and Reading.
who have not started the test opportunity.)
1 completed (reported) first opportunity
2 completed (reported) second opportunity
[#] Student count
The student count column shows the number of students who have completed each
opportunity in brackets. The student count displayed for 0 opportunity is the
number of students who have not yet started the first opportunity.
% Proficient by Opportunity
This column displays the percent of students who have met (or exceeded)
proficiency in each opportunity for the selected test.
% Proficient across Opportunity
This column displays the total percent of students who have met (or exceeded)
proficiency in any of the available opportunities in the testing window.
Beginning in the 20122013 school year, the values in the Summary Statistics table will be cumulative
within a test window. For example, the percentage of students who have completed the first
opportunity will include both the students who have only taken their first opportunity as well as those
students who have taken the first opportunity and gone on to take a second opportunity. Similarly, the
percentage of students proficient by opportunity will display the pass rate for all students who took
the opportunity, regardless of whether they went on to take the second opportunity.
40
My Testing Windows
This feature is for school users to create, record, and manage planned testing windows in their school. These testing
windows are independent of the actual DDOE testing windows. School users can use these to plan and manage testing
schedules within the school.
Authorized school users can add and edit testing windows.
Users in a school cannot view testing windows created in other schools, only those created within
their school.
District users can view testing windows that school users have created.
To add a testing window:
1. Click the [Add] button. A pop-up window will appear.
2. Select the school and test from the drop-down lists.
3. Click in the Start Date and End Date fields to bring
up a calendar. Select the start and end dates for the
testing window.
4. Click [Save]. The information will appear in the My
Testing Windows box.
To edit a testing window:

1. Use the horizontal scroll bar to go to the last column of
the table and click [Edit].
2. Edit the fields as necessary, and then click [Save].
To delete a testing window:
1. Use the horizontal scroll bar to go to the last column of
the table and click [Delete].
2. A pop-up will appear asking you if you are sure that you
want to delete this testing window. Click [OK].
41
Retrieve Student Results

Users can download student data for a selected test window by district, school, teacher, or roster. The data will include
students personal information, including enrolled school and district and grade level, and the selected test scores and
reporting category scores (if applicable).
Users can also retrieve generated batch PDF files of individual student reports. These batch files are generated on the
Student Listing Report page. (Note: You cannot generate these PDF files on this page.)
Step 1: Choose What

Select an assessment type (DCAS, EOC, DCAS Alt1, or Norm-Referenced/TerraNova) from the drop-down list.
Select a test window from the drop-down list.
Select the tested grade or grade band and date range (if desired).
oo OR select All to create a separate file for each available tested grade. These files will be provided in a
Zip file.
Select a file format. The default is an Excel (.xls) spreadsheet file.
Optional: Select a demographic subgroup filter. If you select a filter other than All, a Values drop-down will
appear. Select the desired value. The data in the generated report will contain only those students who meet the
selected parameter(s).
Step 2: Choose Who
Confirm that the district is accurate (users with access to multiple districts will need to select a district).
Select a school from the drop-down list.
oo OR select All to generate a report that includes all schools.
If a school was selected, choose a teacher from the drop-down list.
oo OR select All to generate a report that includes all teachers within the selected school.
If a teacher was selected, choose a roster from the drop-down list or select All rosters.
Step 3: Click the [Download Roster] button. An alert window will appear, indicating that the requested data file is
being processed.
As the requested data may take time to generate, you will receive an e-mail when the requested file is ready.
Step 4: Download the data file by clicking the [Completed] link next to the report in the My Inbox section. You can
access the My Inbox section either by clicking the [Inbox] link in the header or by returning to the Test Management
Center and accessing this screen. Your files will remain available for 30 days.
42
My Inbox
This section of the Retrieve Student Results page lists the data files that you requested.
The following information is shown:
The institution name

The type of data file you requested (district, institution, personnel, or roster)
The test type (DCAS, EOC, DCAS Alt1, or Norm-Referenced/TerraNova)
The test window (e.g., fall 20112012, winter 20112012)
The grade level of the students in the data file
The date and time the data file was created
The status of the request
Click the [Completed] link in the Status column to access the data file.
43
Opening and Viewing the Data File

You can download your requested student data file in either .xls or .csv format. You may also choose to download it in
another format (e.g., .txt file type).
If you choose to download the file as a .txt file, then you will need to follow the instructions below to view the file in
Microsoft Excel or a similar spreadsheet program.
1. Save the file to your desktop or other desired location on your computer.
2. Open Microsoft Excel.
3. Navigate to File > Open.
4. A drop-down list displaying file types will appear next to or below the File Name box. Select Text Files (*.prn; *txt;
*.csv). The .txt file that you saved should now appear in the location in which you saved it.
5. Select the file and click [Open].
6. The Text Import Wizard will open. You will have to navigate through three steps:
a. Make sure Delimited is selected and click [Next].
b. Click the check box next to Other, enter the | symbol (this is often on the same key as the forward slash \),
and click [Next].
c. Click [Finish].
7. The data will populate in separate columns in Excel. For ease in opening this file in the future, you may want to resave the file as an Excel Workbook (.xls or .xlsx) file. This will prevent you from having to import the data again.
44
Plan and Manage Testing (Updated)

This section allows you to generate customized reports showing your students activity in the online testing system.
Selecting the Parameters for Your Report

Step 1: Choose What
Select an assessment type (DCAS, DCAS-EOC, DCAS Alt1, DCAS Alt1-SR, DCAS-Survey, or NR).
Select a test window (e.g., Fall 20112012).
Select a test (subject and grade or grade band).
Select an enrolled grade or all grades.
Step 2: Choose Who

For most users, the district or institution (school) you are associated with will already be displayed in the dropdown lists. Verify that it is accurate.
oo For users who are associated with more than one school and/or district, you will need to verify and/
or select the district from the first drop-down menu. Once the district is selected, the list of associated
schools you have access to will populate in the second drop-down menu. Select the school you want to
generate a participation report for.
Optional: Select the teacher (personnel) you would like to generate the report for. The default variable for this is
All.
45
Step 3: Get Specific

Select the Report Filter option that best matches your needs. Click the radio button for the row that has the parameters
you wish to use, and then select the filters from the drop-down menu(s) in that row. For example, you may wish simply
to view all students in your school who have not yet started any tests.
Four Report Filter options are available. Each filter will generate a report containing students whose information
matches the parameters you have selected.
Display [All/Grade #] students who [have/have not] [completed/started] the [1st/2nd/3rd/4th/Any]*

opportunity and [are/are not] proficient. This filter allows you to:
oo determine which students have completed testing and are or are not proficient.
oo determine which students have or have not started testing.
Display students whose current opportunity will expire in [##] days.

oo This filter allows you to determine which students who are currently testing have at least one test that will
expire shortly. For example, you can enter 2 days to determine who needs to complete testing by the end
of the next day.
Display students on their [1st/2nd/3rd/4th]* opportunity and have a status of [student test status]
oo This filter allows you to determine which students have a specified test status for the selected test
opportunity. Please refer to the test status definitions in Table 73 on page 54 or click [Definitions] on
the Plan and Manage Testing page.
Display students whose last [Session ID/Test Administrator] was ________ between [start date] and [end
date]
oo This new filter allows you to search for students who were in a specific session and view their current test
status.
oo To view this report, enter the Test Administrators Session ID OR enter the Test Administrators name,
and then select a Session ID associated with the TA. You will also have to click on the calendar icons to
select the timeframe.
oo This report will return the most recent participation data for students who have taken the selected test.
The Session ID and TA name fields are optional. You can leave these fields blank and simply search for
participation information within a particular date range. If the Session ID or TA name is not specified, you
will be limited to a date range search of 15 days.
When you are done selecting your report parameters from the filter drop-down menus, click [Generate Report] to view
the results on your screen, or click [Export Report] to open the report in Microsoft Excel (.xls).
The information displayed reflects data current as of the time you clicked [Generate Report]. To update the information
displayed in your report, you must generate a new report. Do NOT use your Web browsers Refresh button.
* DCAS has two test windows, plus separate administrations for EOC and DCAS Alt1 tests. For DCAS tests, each student
has one opportunity for Mathematics and Reading in the fall test window. During the spring DCAS test window, students have
up to two opportunities each for Mathematics and Reading tests and one opportunity each for Science and Social Studies tests.
The opportunity number shown in Plan and Manage Testing corresponds to the number of opportunities taken across the entire
school year, not a single testing window.
You may also select any opportunity, which will allow you to see all opportunities for students who meet the selected criteria.
This is only available for the first filter.
46
Understanding Your Participation Report

Upon clicking [Generate Report], the Participation Report displays data based on your selected parameters. Each report
includes a list of all students who meet your selected parameters. (If no students are listed, this means that no students
matched your report criteria.)
Sample Participation Report
Please note that students may be listed multiple times; each opportunity is counted as a single record. For example, if a student
has taken two assessments so far, that students name will appear twice if both records meet your report parameters.
By default, the Participation Report displays 18 columns. To view all the data, you will need to use the horizontal scroll
bar at the bottom of the report or use the Show/Hide Columns tool to select only the columns you wish to view.
Table 72 identifies all the columns in a participation report and provides a description of each one.
Table 73 identifies the definitions for each test status (listed in the status column).
47
Table 72. Column Attribute Definitions

Attribute
Description
Name
Students legal name (Last Name, First Name).
SSID
Students 10-digit State Student Identification Number.
Enrolled Grade
The grade in which a student is enrolled.
Test
Test name for this student record (e.g., Fall-Grade 3 Reading).
Results ID
The unique identifier linked to the students results for that specific opportunity.
Opportunity
The opportunity number for that students specific record.
Date Started
The date when the first test item was presented to the student for that opportunity.
Date Completed
The date when the student submitted the test for scoring.
Restricted Subjects
The subject(s) the student is restricted (blocked) from taking tests in.
Current ELL
Indicates whether the student is categorized as an English language learner.
Assigned DCAS Alt1 Test
If a student was assigned an alternate assessment, this column will display the subject test.
Language
The language setting that was assigned to the student (English or Spanish).
TA Name
The Test Administrator who created the session in which the student is currently testing (or in
which the student completed the test).
Session ID
The Session ID to which the test is linked.
Status
The status for that specific opportunity (see Table 73 for status definitions).
Restarts
The total number of times a student has resumed an opportunity (e.g., if a test has been paused
three times and the student has resumed the opportunity after each pause, this column will
show three Restarts).
(This includes Restarts Within Grace Periodsee below.)
Restarts Within Grace Period
The total number of times a student has resumed an opportunity within 30 minutes after a
test was paused (e.g., if an assessment has been paused three times and the student resumed
the opportunity within 30 minutes of two pauses but 35 minutes after the third pause, then
this column will show two Restarts Within Grace Period).
Last Activity
The date of the last activity for that opportunity/record. A completed test can still have
activity (as it goes through the QA and reporting process).
Expiration Date
The date the test opportunity expires.
48
Table 73. Test Status Definitions

DCAS General Descriptions
(When students take the online tests)
DCAS Paper and DCAS Alt1 Descriptions

(When a TA enters responses or scores)
Approved
The TA has approved the student for the session, but

the student has not yet started or resumed the test.
The TA has completed the login process for the students

test session, but has not yet started or resumed
response or score entry.
Completed
The student has submitted the test for scoring. No

additional action can be taken by the student.
The TA has completed entering the students responses

or scores and submitted the test for scoring. No
additional action can be taken by theTA.
Denied
The TA denied the student entry into the session. If

the student attempts to enter the session again, this
status will change to Pending until the TA approves
or denies the student.
This status does not apply to paper or alternate

assessments.
Expired
The students opportunity has not been completed and cannot be resumed because the test opportunity has
expired. (Opportunities expire after 45 calendar days from the date the student started the test, or at the end of
the test window.)
Invalidated
The test result has been invalidated.
Paused
The students test is currently paused (as a result of one of the following):
Status
The student paused his or her test by clicking the [Pause] button.
The student idled for too long and the test was automatically paused.
The Test Administrator paused the session.
The Test Administrator paused the individual students test.
The students browser or computer shut down or crashed.
Pending
The student is awaiting TA approval for a new test

opportunity.
Reported
The students score for the completed test has been submitted to the reporting system.
Rescored
The test opportunity was rescored.

assessments.
Review
The student has answered all test items and is

currently reviewing his or her answers before
submitting the test for scoring. (A test with a review
status is not considered complete.)
The TA has completed entering the students responses

or scores and is reviewing them before submitting the
test for scoring. (A test with a review status is not
considered complete.)
Scored
The test will display a scored status, followed by the

students score.
The students test score is calculated and no additional

action can be taken by the TA.
Started
The student has started the test and is actively testing.
Submitted
The test has been submitted for quality assurance review and scoring before it is sent to the Online Reporting
System.
Suspended
The student is awaiting TA approval to resume a test

opportunity.
49

assessments.

assessments.
Participation Reports Tools

Once you have generated your Participation Report, a number of on-screen tools can help you sort, organize, and
analyze the data. A description of these tools follows.
Sort Results by Attribute

You may sort Participation Report data in any column. Click the column header to sort data in ascending (AZ;
110) or descending (ZA; 101) order. The default sort is by Student Name, in ascending order by last name.
View Definitions of Columns and Student Statuses

Click the [Definitions] button to view information about column attributes and other definitions relevant to
your report.
Hide/Show Columns
Use the [Hide/Show Columns] drop-down menu to select which attributes display in your report.
Print Report
Click the [Print] button to print your Participation Report.
Note: Set your Print Options to Landscape mode (horizontal) to accommodate the various columns in the report.
You can also try adjusting the scale of the content (using your browsers Print Preview feature) to fit on a
printed page.
FERPA prohibits the release of any personally identifiable information. Printed reports contain
personally identifiable student data and must be securely stored and destroyed in accordance with the
Materials and Test Security section of the DCAS Online Test Administration Manual (available on
the DCAS portal at http://de.portal.airast.org).
Excel files should not be shared electronically and should not be stored on computers accessible to
individuals who are not authorized to view confidential student information.
50
Exporting Participation Data into Excel

If you want to export participation results to an Excel file, click [Export Report] from the Participation Reports filter
page. A pop-up window will appear asking if you would like to open or save your file. You can open the file and work with
it immediately, or you can save it to your computer and open it to examine the data later.
The report data will be exported as a Microsoft Excel (.xls) file.
Note: You can also export results for all students in your school who are participating in online testing; however, this may
take some time, depending on the amount of data involved.
Below is a sample report exported into Microsoft Excel.
Sample Excel Report: Students in Grade 5 who have completed the first opportunity in Grade 5 Mathematics and
are not proficient
51
Test Completion Rates (New Test Management Center Report)

School and district users can create a file showing the number of students who have started and completed each test.
District level users can create a file either for a specific school or for the whole district. The user can also generate
the report for all tests or for one specific test. These files can be viewed in Excel or uploaded into most data analysis
programs. Refer to Table 74 for a description of each column in the file.
Selecting the Parameters for Your Report

Step 1: Select an Entity
Select the report level (district or school).
If you are associated with more than one school,
you can select one school or all schools.
Step 2: Select a Test
Select a single test or all tests.
Step 3: Click [Export]
You can open or save the file and view it in Microsoft Excel.
Sample District Participation Counts Report
52
Table 74. Test Completion Rates Report Column Descriptions

Column
Description
Date
The date and time that the file was generated.
Test Name
The grade, test, and subject that are being reported.
Opportunity
The test opportunity number that is being reported.
District Name
The name of the reported district.
District ID
The ID number of the reported district.
School Name
The name of the reported school.
School ID
The ID number of the reported school.
Total Student
The number of students with an active relationship to the school in TIDE. This may include students with
end-dated SSIDs. (Students with end-dated SSIDS are inactive but still associated with the school in which
they were most recently enrolled.)
Total Student Started
The number of students who have started the test.
Total Student Completed
The number of students who have finished the test and submitted it for scoring.
Percent Started
The percentage of students who have started the test out of the total number of students with an active
relation to the school in TIDE.
Percent Completed
The percentage of students who have completed the test out of the total number of students with an
active relation to the school in TIDE.
53
Appendix: Data Available in Score Reports

Not all data are available for all DCAS, EOC, DCAS Alt1, and Norm-Referenced/TerraNova tests. The following tables
provide an at-a-glance illustration of which reports contain which data.
Aggregated Subject Detail

DCAS
Alt1
DCAS
Mathematics
DCAS
Reading
DCAS
Science
DCAS
Social
Studies
EOC
NormReferenced/
TerraNova
Average Instructional Score
Instructional Score Standard

Error of the Mean
Student Count
Average Accountability Score
Accountability Score Standard

Error of the Mean
Percent at Each DCAS

Performance Level
Average TerraNova Score
(2010-2011 school year only)

TerraNova Score Standard Error
of the Mean

Average NCE Score

Percent at Each National
Quartile
54
Aggregated Reporting Categories

DCAS
Mathematics
DCAS
Reading
DCAS
Science
DCAS Social
Studies
EOC
Student Count
Reporting Category
Average Instructional
Score
Instructional Score
Standard Error of the
Mean
Average Scale Score
Scale Score Standard

Error of the Mean
DCAS Social
Studies
EOC
DCAS Alt1
NormReferenced/
TerraNova
Aggregated Benchmarks
DCAS
Mathematics
DCAS
Reading
DCAS
Science
DE Standard
Performance Level
DCAS Alt1
55
NormReferenced/
TerraNova
Student Listing and Individual Student Reports

DCAS Alt1
DCAS
Mathematics
DCAS
Reading
DCAS
Science
DCAS
Social Studies
EOC
NormReferenced/
TerraNova
Student ID
Accountability Score
Accountability Score
Conditional Standard Error
of Measurement
Instructional Score
Instructional Score
of Measurement
Reporting Category
Instructional Score
Reporting Category
Instructional Score
of Measurement
Reporting Category Scale

Score
Reporting Category Scale

Score Conditional Standard
Error of Measurement
DCAS Performance Level

Estimated DSTP
Performance Level
Percentile Rank
Stanine
TerraNova Scale Score
56
Interpretive Guide for Minnesota Assessment Reports 201221013
TABLE OF CONTENTS
LETTER FROM THE DEPARTMENT OF EDUCATION ............................................................................................................. 1
INTRODUCTION TO THE INTERPRETIVE GUIDE FOR MINNESOTA ASSESSMENT REPORTS ............................ 2
PURPOSE OF THE MINNESOTA ASSESSMENTS ....................................................................................................................... 3
Title I Assessments ................................................................................................................................................................................ 3

Assessments to Meet Graduation Assessment Requirements .............................................................................................................. 3
Test Specifications ................................................................................................................................................................................. 4
DATA SITES AND RESOURCES ......................................................................................................................................................... 4
MDE Data Center ................................................................................................................................................................................... 4

Online Reporting System ....................................................................................................................................................................... 4
Learning Point Navigator ....................................................................................................................................................................... 4
Lexile Website ....................................................................................................................................................................................... 5
TYPES OF FINAL SCORE REPORTS FOR MCA, MCA-MODIFIED, MTAS, AND GRAD ................................................. 5
INTERPRETING SCORES AND ACHIEVEMENT LEVELS ........................................................................................................ 5
Development of the Achievement Level Descriptors (ALDs) ................................................................................................................. 6

How to Use the ALDs ............................................................................................................................................................................. 7
TITLE I ASSESSMENTS ........................................................................................................................................................................ 8
MCA ....................................................................................................................................................................................................... 8
Mathematics Sub-Scores ................................................................................................................................................ 9

Reading Sub-Scores ......................................................................................................................................................... 9
Science Sub-Scores.......................................................................................................................................................... 9
MCA-Modified ..................................................................................................................................................................................... 10
MTAS ................................................................................................................................................................................................... 12
GRAD ASSESSMENTS......................................................................................................................................................................... 13
Written Composition GRAD ................................................................................................................................................................. 13
Reading and Mathematics GRAD......................................................................................................................................................... 13
REPORT DESCRIPTIONS FOR TITLE I ASSESSMENTS........................................................................................................ 14

General Description of the MCA, MCA-Modified, and MTAS Individual Student Report.................................................................... 14
REPORT DESCRIPTIONS FOR ASSESSMENTS MEETING GRADUATION ASSESSMENT REQUIREMENTS... 21

General Description of the Written Composition GRAD Individual Student Report ........................................................................... 22
General Description of the Grade 10 Reading MCA Individual Student Report .................................................................................. 23
General Description of the Grade 11 Mathematics MCA Individual Student Report.......................................................................... 24
General Description of the Reading and Mathematics GRAD Individual Student Report ................................................................... 27
STUDENT LABELS............................................................................................................................................................................... 28
General Description of the Student Labels .......................................................................................................................................... 28
CONTACT INFORMATION ............................................................................................................................................................... 29
Letter from the State of Minnesota Department of Education

Dear Educator,
The state tests administered each year measure student achievement on
Minnesotas Academic Standards and the Minnesota Standards for English
Language Development, developed by the WIDA Consortium. We have developed
this Minnesota Assessment Interpretive Guide to help educators understand the
results from these tests.
The guide contains information on how to read the reports and interpret the data
from these tests. Once you have become familiar with the information, you will be
in a position to gauge the effectiveness of your schools curriculum and instruction.
You will also have some individual student information that can guide your
students instruction.
We encourage you to use this guide to inform parents, students, and other
interested persons in your community about how the Minnesota Assessment
System supports all students in their learning of the Minnesota Academic
Standards and the Minnesota Standards for English Language Development.
Minnesota believes all students can learn and strives to set high standards for
student performance.
State of Minnesota
INTRODUCTION TO THE INTERPRETIVE GUIDE FOR MINNESOTA ASSESSMENT REPORTS

Minnesota has developed an assessment system to measure student proficiency on the Minnesota Academic Standards
and the Minnesota Standards for English Language Development, developed by the World-Class Instructional Design and
Assessment (WIDA) Consortium. This system comprises standardized, criterion-referenced tests that provide individual
and aggregate data on student performance aligned to grade-level standards.
The Minnesota Assessments have multiple uses:
School and district results are used for school and district accountability under the Elementary and Secondary
Education Act (ESEA).
Individual student reports inform parents and students of progress in achieving the grade-level Minnesota
Academic Standards and/or the Minnesota Standards for English language development.
Individual student and aggregate summary results are available to help schools and districts make instructional
and policy decisions.
The assessments administered in high school may also be used to meet graduation assessment requirements.
Many measures of learning are necessary to derive an understanding of a students strengths and weaknesses. Each
performance measure in a comprehensive assessment system requires that users know what the data mean and how to
use the data to make effective decisions.
This Interpretive Guide is intended to help educators understand the results of the Minnesota Assessments. The guide
provides basic information about each assessment, describes each available report, and suggests ways to use the results.
The major topics of this guide are:
The purpose of the Minnesota Assessments
Overview of data sites and resources
Types of score reports
Interpretation of scores and achievement levels
Achievement level descriptors
Descriptions of specific reports
Contact information
Minnesota is part of the WIDA Consortium. As a member of this consortium, Minnesota districts administer the ACCESS
for ELLs assessment to English learners in grades K12. ACCESS for ELLs stands for Assessing Comprehension and
Communication in English State-to-State for English Language Learners. In addition, the Alternate ACCESS for ELLs is
available in grades 112 for English learners with significant cognitive disabilities. The ACCESS for ELLs and Alternate
ACCESS for ELLs fulfill the Title III requirement of the ESEA. The purpose of these assessments is to measure students'
achievement on the Minnesota Standards for English Language Development, developed by the WIDA Consortium.
Information about reports for ACCESS for ELLs and Alternate ACCESS for ELLs is not included in this guide; for more
information, please refer to the Interpretive Guide for Score Reports on the WIDA website (http://www.wida.us). View
the WIDA website (http://www.wida.us/assessment/ACCESS/). In addition, please refer to the Data Sites and Resources
section later in this manual about how to find ACCESS for ELLs and Alternate ACCESS for ELLs results on the Minnesota
Department of Education (MDE) website.
References to additional information on the MDE website (http://education.state.mn.us) and Minnesota Assessments
portal (www.mnstateassessments.org) are referenced throughout this manual. The Minnesota Assessments portal is the
American Institutes for Research (AIR) website; AIR is the service provider for the Title I and GRAD assessments.
No single assessment can comprehensively measure a students learning in a school or environment.
Results on the Minnesota Assessments are only a subset of the data that schools and districts can use
to determine how well students have acquired the knowledge and skills on the Minnesota Academic
Standards and Minnesota Standards for English Language Development and how well the school is
teaching them.
PURPOSE OF THE MINNESOTA ASSESSMENTS

Title I Assessments
Minnesota Comprehensive Assessment in Reading, Mathematics, and Science
In 2013, the Minnesota Comprehensive Assessment (MCA) was administered to students in reading in grades 38 and
10, mathematics in grades 38 and 11, and science in grades 5, 8, and high school. The purpose of the MCA is to
measure Minnesota students achievement on the Minnesota Academic Standards. The MCA results inform curriculum
decisions at the district level; inform instruction at the classroom level; and, in reading and mathematics, demonstrate
student academic progress from year to year.
The Reading and Mathematics MCA are the primary assessments Minnesota uses for ESEA accountability. All students
are required to take these tests or a designated replacement, such as the Reading and Mathematics MCA-Modified for
persistently low-performing students or the Reading and Mathematics Minnesota Test of Academic Skills (MTAS) for
students with significant cognitive disabilities. The test results are used to calculate Adequate Yearly Progress (AYP) and
Multiple Measurement Ratings (MMR) for Minnesota schools and districts. MCA results can be used to compare schools
and districts across the state. Science MCA participation (or Science MTAS, for eligible students) is required under ESEA
but is not included in AYP or MMR calculations at this time.
Minnesota Comprehensive Assessment-Modified (MCA-Modified) in Reading and Mathematics
The MCA-Modified is an alternate assessment in reading and mathematics in grades 58, reading in grade 10, and
mathematics in grade 11 based on the same academic standards as the MCA but with modified achievement standards.
The MCA-Modified is positioned between the MCA and the MTAS. Participation is limited to persistently low-performing
students receiving special education services whose IEP team determines that they meet eligibility requirements for the
MCA-Modified.
Minnesota Test of Academic Skills (MTAS) in Reading, Mathematics, and Science
The Minnesota Test of Academic Skills (MTAS) is an alternate assessment in reading and mathematics in grades 38,
reading in grade 10, mathematics in grade 11, and science in grades 5, 8, and high school that is based on alternate
achievement standards. The MTAS is part of a statewide program that measures the extent to which students with the
most significant cognitive disabilities are making progress in the general education curriculum on standards that have
been reduced in breadth, depth, and complexity.
The MTAS is a performance-based assessment where performance tasks in reading, mathematics, and science are
administered to students in a one-on-one setting. Test administrators score performance tasks using a script and taskspecific scoring rubric.
Assessments to Meet Graduation Assessment Requirements

In order to be eligible for a diploma from a Minnesota public high school, all students must fulfill graduation assessment
requirements. Based on new legislation in 2013, these requirements have changed, and students are no longer required
to meet the Graduation-Required Assessment for Diploma (GRAD) requirements in reading, mathematics, and writing in
order to receive a diploma. However, meeting the GRAD requirements is still one of the options available for students
enrolled in grade 8 through 20112012.
In 20122013, students could meet the GRAD requirements as follows:
Earning a proficient score on a high school reading or mathematics Title I assessment (MCA, MCA-Modified, or
MTAS) or passing the GRAD component embedded in the Mathematics MCA
Earning a passing score on the Written Composition GRAD
Passing Reading and Mathematics GRAD retests
Meeting GRAD alternate routes:
o Meet mathematics alternate pathway requirements (for students in grade 8 through 20092010 only)
o Receive an individual passing score (for students on an IEP or 504 plan)
o
o
Receive an English language learner (ELL) exemption

Pass an accountability assessment from another state approved by MDE
Additional information about the legislative changes to the graduation assessment requirements is available on the Test
Administration page of the MDE website. View the Test Administration page (http://education.state.mn.us/MDE/
SchSup/TestAdmin/index.html). Further information about GRAD alternate routes is available in the Procedures Manual
for the Minnesota Assessments, which is posted to the Minnesota Assessments portal. View the Procedures Manual on
the Minnesota Assessments portal (http://www.mnstateassessments.org/resources/?section=1).
Test Specifications
Test specifications are specific rules and characteristics that guide the development of a tests content and format. They
indicate which strands, sub-strands, standards, and benchmarks will be assessed on the test and in what proportions.
View test specifications for the Title I and GRAD assessments on the Test Specifications section of the MDE website
(http://education.state.mn.us/MDE/EdExc/Testing/TestSpec/index.html).
DATA SITES AND RESOURCES

MDE Data Center
There are two sections of the Data Center on the MDE website where educators can analyze test results and create,
view, and download reports that meet their needs. The Data for Parents and Educators section is open to the public and
allows the user to view and analyze data for any public school or district in the state. The only restriction is that data are
suppressed when a data set consists of fewer than 10 students. To access the Data for Parents and Educators section, go
to the MDE website, choose Data Center, and then choose Data for Parents and Educators. Information about how
to use this section of the website is included on the Data for Parents and Educators pages.
The MDE Secured Reports section is only open to educators who have obtained permission to access secured reports
from their superintendents. This section allows users to download student-level information through the District
Student Results (DSR) or School Student Results (SSR) files, as well as test results summary information for each test. To
access the MDE Secured Reports section, go to the MDE website, choose Data Center, choose Data Reports and
Analytics, and then choose the applicable report from the list under MDE Secured Reports. The user guide is included
on both the DSR and SSR and Test Results Summary pages.

AIRs Online Reporting System (accessed through the Minnesota Assessments portal) allows authorized users to view
preliminary student test data for Title I and GRAD assessments. Depending on a users role, he or she can view
performance data at the student, roster, school, and district levels. Participation data can be viewed to see what
percentage of students have completed testing and who still needs to complete tests.
Please note that the data available in the Online Reporting System should not be considered as official for accountability
purposes; official accountability data are provided to districts by MDE. Districts can use the information in the Online
Reporting System for instructional purposes or to inform about graduation status and may share score information with
students and families. However, only the data provided by MDE should be used for official student record-keeping.

AIRs Learning Point Navigator system (also accessed through the Minnesota Assessments portal) is a tool that allows
teachers to access curricular support resources to assign to students. Teachers can use the site to enhance students
learning by creating assignments that focus on standards and benchmarks in Mathematics, English Language Arts
(Reading) and Science. After students complete assignments, they can see their raw scores. Teachers can also view
individual student and class reports for each assignment given.
Learning Point Navigator is accessed through the Minnesota Assessments portal by selecting either the Teacher/Test
Monitor or Assessment Coordinator icon on the home page and then selecting Learning Point Navigator.
District personnel must define school- and teacher-level users roles in AIRs Test Information
Distribution Engine (TIDE) and then create rosters for them in TIDE or via the Manage Rosters tool in
the Online Reporting System in order for these school-level users to create assignments for students in
Learning Point Navigator.
Lexile Website
The Reading MCA individual student reports include predicted Lexile score ranges. The Lexile Framework is a system
that helps match readers with literature appropriate for their reading skills. When reading a book within the predicted
Lexile range, the reader should comprehend enough of the text to make sense of it, while still being challenged enough
to maintain interest and learn. View the Lexile website for more information about the Lexile Framework
(http://www.lexile.com/).
TYPES OF FINAL SCORE REPORTS FOR MCA, MCA-MODIFIED, MTAS, AND GRAD
Score reports are generated for each district and school. The following table lists the types of reports that are available
for final results.
Report
Name
Student Results Files
District (DSR)
School (SSR)
Summary Files
District
School
Student Reports
Shipment
Home Copy
Student Results Labels
(optional for Title I)
Format
Blank
Online
Online
Blank
Online
Online
Blank
District
Blank
Blank
Blank
Blank
Paper
Paper
Blank
Blank
School
Blank
Blank
Blank
Blank
Blank
See the Data Sites and Resources section for more information about student results files and summary files that are
available through MDE Secured Reports; the student reports are described in detail later in this manual. Schools student
reports shipments are packaged by school and delivered to the districts for distribution. Preliminary results information
is available in the Online Reporting System as described in the Data Sites and Resources section.
INTERPRETING SCORES AND ACHIEVEMENT LEVELS

The following types of information are available on the summary files:
Percent of students proficient
Percent of students at each achievement level
Average scale scores (for the total test)
Average sub-scores (for strands, sub-strands, and extended benchmarks)
For each of these scores, you can compare the results for your school and district to those for schools and districts of
interest to you or to the state through the Data Center section of the MDE website. For example:
Compare average sub-scores. If the number of possible points for a particular sub-score is small, be cautious
when interpreting small differences. Use differences in average sub-scores to guide further investigation of the
curriculum and instruction at the school or district level.
Compare different perspectives, such as average scale scores and percent proficient. For example, your district
or school may have a lower average scale score than the state, but the percent of students who are proficient
may be greater than the state.
Look at the distribution of your students scale scores and sub-scores. Averages can be strongly influenced by
students with very high or very low scores.
The distribution of an entire groups scores may help you better understand the strengths and weaknesses of your
students, especially when the sub-scores distributions are included. The District and School Student Results (DSR and
SSR) files give you the data electronically, which makes it easier to see a distribution of scores. For more information,
refer to the Data Sites and Resources section of this manual.
The average sub-scores for MCA-III assessments are reported on a standardized 1 to 9 scale that is intended to facilitate
comparison of strand performance across strands and years. On other tests (MCA-II, MCA-Modified and MTAS), subscores are reported as raw score points earned, and schools and districts can only be appropriately compared within a
particular year for those assessments. Such comparisons can tell an organization about its strengths or areas needing
improvement relative to other schools or districts. Sub-scores based on raw score points are not equated for differences
in difficulty for a given year; one strand or sub-strand may have items that are more difficult than others. Thus, direct
comparisons between different sub-scores or across multiple years may be misleading. Be cautious when making
comparisons between strands or sub-strands.
Trend data are available for the Minnesota Assessments. However, please use caution when interpreting trend data
because assessments change when academic standards are revised. For example, a new baseline for mathematics was
set in 2011, for science in 2012, and for reading in 2013. For this reason, comparisons between the percentage of
students who scored proficient should be done only when keeping in mind the standards measured from one year to the
next.
Development of the Achievement Level Descriptors (ALDs)

The Achievement Level Descriptors (ALDs) give descriptive information about what typical students are expected to
know of the Minnesota Academic Standards.
The ALDs were developed focusing on the content of the Minnesota Academic Standards. Preliminary drafts of the ALDs
were provided for the standard setting panels as they began their work to determine cut scores for each of the
achievement levels. After standard setting, minor adjustments were made to more accurately reflect the skills
demonstrated by students at each of the achievement level score ranges. View the full ALDs on the MDE website
(http://education.state.mn.us/MDE/EdExc/Testing/AchievLevDesc/index.html).
Performance definitions are the equivalent of the ALDs for the ACCESS for ELLs and the Alternate ACCESS for ELLs English
language proficiency assessments. These descriptors assist families, teachers, and administrators with the
interpretation of the proficiency levels reported on a six point scale. In addition to performance definitions, Can Do
descriptions are available for the levels of performance on the ACCESS for ELLs. Both the performance definitions and
the Can Do statements can be found in the documents listed in the Downloads and Products section of the Can Do
Descriptors page of the website. View the CAN Do descriptors page (http://wida.us/standards/CAN_DOs/). Performance
definitions for the Alternate ACCESS for ELLS are available in the Alternate ACCESS for ELLs section of the WIDA website
under Downloads and Products. View the Performance Definitions on the Alternate ACCESS for ELLs page
(http://wida.us/assessment/alternateaccess.aspx).
How to Use the ALDs

The ALDs can be used to communicate with parents, students, and the public about the basic skills and knowledge
expected of the typical student at each achievement level. The ALDs give concrete meaning to a scale score and its
associated achievement level. They can be used as examples when talking with others about student performance. The
ALDs may be used as a tool to inform parents of the performance expectations for their child and to suggest changes in
skills and knowledge as a student moves from one achievement level to a higher level.
The ALDs can also be used by educators in instructional planning. The ALDs can help teachers develop curriculum maps
to reflect the building of skills on each of the benchmarks. Teachers may also find the ALDs useful as they develop their
school improvement plans. If a school uses Minnesota assessment data with formative assessment to group students for
instruction, the ALDs may be used to provide some cursory information about the skills and knowledge that need
emphasis to move the students to the next achievement level. If a student is involved in supplemental services related
to his or her performance on an assessment, then a service provider might use the ALDs to identify the scaffolding of
skills needed to help the student reach proficiency on skills measured in previous grades so that the student can be
successful in his or her current grade.
When using any of the Minnesota ALDs, it is important to remember that the performance of an individual student at an
achievement level may vary from the descriptors.
TITLE I ASSESSMENTS
MCA
Scale Score
The raw score totals for Science and grade 11 Mathematics MCA are converted to a scale score specific to each test
subject and grade. For the grades 38 Mathematics MCA and all grades of the Reading MCA, the scale score is not based
on the raw score total; it is based on the specific pattern of correct and incorrect responses given by the student. Use
the scale score to determine how the student did on the test. Each year, the test is equated for difficulty with the
previous years test. This means the scale score has equivalent meaning and provides a valid comparison from year to
year for a given grade and subject.
For each Reading, Mathematics and Science MCA, the scale score can range from G01 to G99, with G standing for
Grade. The first digit (i.e., 38) or first two digits (i.e., 10 or 11) represent the students grade when tested. The last
two digits of the number identify the position of the score on the grade scale. For example, a student in grade 4 could
earn a scale score between 401 and 499, while a student in grade 11 could earn a scale score between 1101 and 1199.
Note: While the high school Science MCA can be administered in any grade (912) depending on coursework
completion, grade 10 is used to represent the grade for the high school scores.
For the grades 38 and 10 Reading MCA, please note that the scale score should not be directly compared to scores
from previous years. These tests are new and based on the 2010 revised reading academic standards. Grades 38
Mathematics MCA scores can only be compared to 2011 and 2012 scale scores as 2011 was the first year of the
assessment based on the 2007 revised mathematics academic standards. Grades 5, 8 and high school Science MCA
scores can be compared to 2012 scale scores because 2012 was the first year of the assessment based on the 2009
revised science academic standards.
For assessments that convert raw scores to scale scores (Science MCA and grade 11 Mathematics MCA), more than one
raw score point may be assigned the same scale score, except at the cut scores for each achievement level or at the
maximum possible score of G99. Specific details regarding the raw score to scale score relationship are reported on the
Technical Reports section of the MDE website. View the Technical Reports section of the MDE website
(http://education.state.mn.us/MDE/SchSup/TestAdmin/MNTests/TechRep/index.html).
Achievement Levels
There are four achievement levels for the MCA:
Exceeds the Standards (E)
Meets the Standards (M)
Partially Meets the Standards (P)
Does Not Meet the Standards (D)
Students are assigned an achievement level based
on their scale score. For the MCAs, the diagram to
the right illustrates the commissioner-approved cut
scores used to assign achievement levels. The cut
scores for levels Partially Meets the Standards (P)
and Meets the Standards (M) are G40 and G50,
respectively. The cut score for level Exceeds the
Standards (E) varies by grade and subject.
MCA Scale Scores & Achievement Levels

Each Grade level will have the same score
range (G01 to G99), with G = Grade.
Last two digits of the number identify the
position within the scale range.
The first one or two digits represent the
grade.
The first two cut scores will be constant
over years, G40 and G50. The third cut
score varies by grade and subject.
Sub-Scores
The sub-scores for the mathematics grade 11 strands are the raw score points earned by the student on each strand or
sub-strand. The number of possible points for each sub-score varies by grade, subject, and year. The school, district, and
state averages of these sub-scores are available at the MDE website under Data Center. The sub-scores for the
mathematics grades 38, reading and science strands are the scale scores (reported on a 1 to 9 scale score metric)
earned by the student for each strand.
For more information on sub-scores, reference the applicable test specifications on the MDE website. View the Test
Specifications page of the MDE website (http://education.state.mn.us/MDE/EdExc/Testing/TestSpec/index.html).
Mathematics Sub-Scores
The Mathematics MCA sub-scores reported for grade 11 reflect the four mathematics strands from the Mathematics
Standard of the 2003 Minnesota Academic Standards, which are outlined in the MCA-II test specifications However, the
Number Sense strand is not reported separately for the MCA component of the test; it is embedded in the other three
strands.
Number Sense: understanding numbers, operations, and quantitative reasoning
Patterns, Functions & Algebra: understanding patterns, relationships, and algebraic reasoning (the use of
symbols to represent real-world situations)
Data, Statistics & Probability: understanding probability (the chance that an event will occur) and statistics (the
collection, organization, and interpretation of data)
Spatial Sense, Geometry & Measurement: understanding geometry and spatial reasoning (the location/position
of an object and the amount of space it occupies in the real world)
The Mathematics MCA sub-scores reported for grades 38 represent the four mathematics strands from the
Mathematics Standard of the 2007 Minnesota Academic Standards. The strands are outlined in the test specifications.
Number & Operation: understanding meanings of numbers and operations and how they relate to each other,
computing fluently, and making reasonable estimates
Algebra: models to understand, represent, and analyze patterns, relations, and functions
Geometry & Measurement: analyzing characteristics and properties of two- and three-dimensional geometric
shapes and developing mathematical arguments about geometric relationships; understanding the units,
systems, and processes of measurement
Data Analysis (grades 35) and Data Analysis & Probability (grades 68): organizing and displaying relevant data
to answer questions; understanding and applying basic concepts of probability
The grades 38 mathematics reports have the same four sub-score categories.
Reading Sub-Scores
The Reading MCA sub-scores reflect the sub-strands of Literature and Informational Text from the 2010 Minnesota
Academic Standards in Language Arts, which are outlined in the test specifications. All the reading reportsgrades 38
and 10have the same sub-score categories.
Literature: use strategies to analyze, interpret, and evaluate fiction (such as short stories, fables, poetry, and
drama).
Informational Text: use strategies to analyze, interpret, and evaluate nonfiction (such as expository and
persuasive text, and literary nonfiction).
The ten reading standards are organized under four skill domains. The skill domains are Key Ideas and Details (standards
1-3), Craft and Structure (standards 4-6), Integration of Knowledge and Ideas (standards 7-9), and Range of Reading and
Level of Text Complexity (standard 10). Seven of the ten reading standards are assessed on the Reading MCA-III.
Standards 7, 9, and 10 are best assessed using classroom measures and are not assessed in the MCA-III.
Science Sub-Scores
The Science MCA sub-scores are for the four strands in grades 5 and 8 and two strands in high school from the Science
Standards of the 2009 Minnesota Academic Standards and are outlined in the test specifications
Grade 5 Strands
Nature of Science & Engineering: conducting controlled scientific investigations, constructing explanations
based on evidence, identifying engineering solutions to problems
Physical Science: describing and experimenting with the properties of matter, light, heat, sound, electricity,
magnetism, and force and motion
Earth & Space Science: modeling the positions of Earth, the Sun and the Moon, describing how weathering and
erosion shape Earths surface and how water moves through the water cycle
Life Science: identifying structures and functions of organisms and relationships among organisms, and
understanding individual differences give advantages in survival
Grade 8 Strands
Nature of Science & Engineering: understanding how humans affect scientific investigations, designing and
conducting investigations, communicating results, and refining engineering solutions
Physical Science: differentiating between physical and chemical changes, understanding properties of waves
and force and motion of an object, and describing changes in energy
Earth & Space Science: understanding how forces affect motions of objects in the universe, describing weather
patterns, and understanding the processes that occur on Earth
Life Science: identifying changes in energy within an ecosystem, understanding cell processes and genetic
variation, and describing the effect of humans on ecosystems
High School Strands
Nature of Science & Engineering: analyzing risks and benefits of engineering solutions accurately
communicating scientific results, and testing hypotheses
Life Science: describing cell functions and processes, understanding relationships of organisms in an ecosystem,
and the role of DNA and variation in evolution
CAUTION Use care when interpreting:

Data involving few students or test items: The more students taking the test or more test
items measuring the concepts, the more confident you can be of the results.
Sub-scores that are reported as raw score points: The difficulty of a strand or sub-strand and
points possible will likely vary by grade, subject, and year.
MCA-Modified
Scale Score
The raw score totals for Mathematics and Reading MCA-Modified are converted to a scale score for each test subject
and grade. This scale score represents how the student performed on the test. Each year, the test is equated for
difficulty with the previous years test, which means the scale score permits a valid comparison of achievement from
year to year for a given grade and subject.
Achievement Levels
There are four achievement levels for the MCA-Modified:
Exceeds the Modified Achievement Standards (E)
Meets the Modified Achievement Standards (M)
Partially Meets the Modified Achievement Standards (P)
Does Not Meet the Modified Achievement Standards (D)
Students are assigned an achievement level based on their scale score. The cut scores for levels Partially Meets the
Standards (P) and Meets the Standards (M) for all grades and subjects are 190 and 200, respectively. The cut score for
level Exceeds the Standards (E) varies by grade and subject.
10
Specific details regarding the raw score to scale score relationship are reported on the Technical Reports section of the
MDE website. View the Technical Reports section of the MDE website (http://education.state.mn.us/MDE/SchSup/
TestAdmin/MNTests/TechRep/index.html).
Sub-Scores
The sub-scores are the raw score points earned by the student on the mathematics strands and the reading sub-strands.
The number of possible points for each sub-score varies by grade, subject, and year. The school, district, and state
averages of these sub-scores are available at the MDE website under Data Center.
The Mathematics MCA-Modified sub-scores in grade 11 reflect the four mathematics strands from the Mathematics
Standard of the 2003 Minnesota Academic Standards and are outlined in the grade 11 MCA-Modified test specifications.
of an object and the amount of space it occupies in the real world)
The Mathematics MCA-Modified sub-scores in grades 58 represent the four mathematics strands from the
Mathematics Standard of the 2007 Minnesota Academic Standards. The strands are outlined in the test specifications.
Number & Operation: understanding meanings of numbers and operations and how they relate to each other,
computing fluently, and making reasonable estimates
Algebra: models to understand, represent, and analyze patterns, relations, and functions
Geometry & Measurement: analyzing characteristics and properties of two- and three-dimensional geometric
shapes and developing mathematical arguments about geometric relationships; understanding the units,
systems, and processes of measurement
Data Analysis (grade 5) and Data Analysis & Probability (grades 68): organizing and displaying relevant data to
answer questions; understanding and applying basic concepts of probability
The grades 58 mathematics reports have the same four sub-score categories. The grade 11 report has only three subscores; the Number Sense strand is embedded in the other three stands and is not reported.
Reading Sub-Scores
The Reading MCA Modified sub-scores reflect the sub-strands of Literature and Informational Text from the 2010
Minnesota Academic Standards in Language Arts, which are outlined in the test specifications. All the reading reports
grades 58 and 10have the same sub-score categories.
Reading sub-strands:
Literature: use strategies to analyze, interpret, and evaluate fiction (such as short stories, fables, poetry, and
drama).
Informational Text: use strategies to analyze, interpret, and evaluate nonfiction (such as expository and
persuasive text, and literary nonfiction).
The ten reading standards are organized under four skill domains. The skill domains are Key Ideas and Details (standards
1-3), Craft and Structure (standards 4-6), Integration of Knowledge and Ideas (standards 7-9), and Range of Reading and
Level of Text Complexity (standard 10). Seven of the ten reading standards are assessed on the Reading MCA-ModifiedIII. Standards 7, 9, and 10 are best assessed using classroom measures and are not assessed in the MCA-Modified-III.
11
MTAS
Scale Score
The raw score totals for Mathematics, Reading, and Science MTAS are converted to a scale score for each test subject
and grade. This scale score represents how the student performed on the test. Each year, the test is equated for
difficulty with the previous years test, which means the scale score permits a valid comparison of achievement from
year to year for a given grade and subject.
Achievement Levels
There are four achievement levels for the MTAS:
Exceeds the Alternate Achievement Standards (E)
Meets the Alternate Achievement Standards (M)
Partially Meets the Alternate Achievement Standards (P)
Does Not Meet the Alternate Achievement Standards (D)
Students are assigned an achievement level based on their scale score. The cut scores for levels Partially Meets the
Standards (P) and Meets the Standards (M) for all grades and subjects are 190 and 200 respectively. The cut score for
level Exceeds the Standards (E) varies by grade and subject.
Specific details regarding the raw score to scale score relationship are reported on the Technical Reports section of the
MDE website. View the Technical Reports section of the MDE website (http://education.state.mn.us/MDE/SchSup/
TestAdmin/MNTests/TechRep/index.html).
Sub-Scores
The sub-scores are the raw score points earned by the student on the mathematics, reading, and science tasks identified
by the essence statements described in the MTAS test specifications. The number of possible points for each task is 3.
Each MTAS sub-score may stem from a single or multiple tasks. The sum of an individual students sub-scores is the
students total raw score.
CAUTION Use care when interpreting:

Data involving few students or test items: The more students taking the test or more test
items measuring the concepts, the more confident you can be of the results.
Sub-scores: The difficulty of a strand or sub-strand and raw score points possible will likely
vary by grade, subject, and year.
12
GRAD ASSESSMENTS
Written Composition GRAD
The total score for Written Composition GRAD is the holistic score earned by the student. Students write to one prompt
and the paper is assigned a score between 16 based on the raters overall impression of the writing. A score of 3 or
higher is required to pass the Written Composition GRAD.
Reading and Mathematics GRAD

The raw score totals for Reading and Mathematics GRAD are converted to a scale score and the scale score is used to
determine how the student performed on the test. Each test is equated for difficulty with the previous tests, which
means the scale score provides a valid comparison from test to test for a given grade and subject. The passing score for
the GRAD is 50 in a scale score range of 1585.
Reading Sub-Scores
The Reading GRAD sub-scores align with the three sub-strands of the Reading and Literature strand of the Language Arts
Standards from the Minnesota Academic Standards and are outlined in the Minnesota GRAD Test Specifications for
Reading.
Vocabulary Expansion: using a variety of strategies to expand reading vocabulary (the use of context clues to
understand new words)
Comprehension: showing understanding of the meaning of text and demonstrating literal, interpretive,
inferential, and evaluative comprehension
Literature: demonstrating the ability to read, understand, respond to, analyze, evaluate, and interpret a wide
variety of fiction and nonfiction text
The Mathematics GRAD sub-scores align with Minnesota Graduation-Required Assessments for Diploma (GRAD) Test
Specifications for Mathematics.
of an object and the amount of space it occupies in the real world).
13
REPORT DESCRIPTIONS FOR TITLE I ASSESSMENTS

General Description of the MCA, MCA-Modified, and MTAS Individual Student Report
An Individual Student Report (ISR) is generated for every student who took the assessment. The ISR describes an
individual students performance in terms of scale score, achievement level, and Minnesota Academic Standards for
each subject.
Schools will receive a hard copy of each students ISR to send home with the student or to mail to the students
parent/guardian. For the hard copy paper ISRs, reading and mathematics results are printed on one 4 page report;
science is printed on a separate 1 page report. Districts can also access final student-level information through the DSR
and SSR files provided on the MDE website.
Description of the Reading and Mathematics MCA

Sample Individual Student Report Page 1
1. Subject and Test The subject and test being reported.
2. Students Achievement Level The achievement level earned by
the student in the subject.
3. Student Demographic Information A description of the
demographic information for the student, including: Student
Name, Local Use # (optional number assigned by districts to aid in
sorting data), MARSS Number (unique student number), UIN
(unique identification number assigned by the vendor and MDE),
Grade, Birth Date, School, District.
4. About this Report A brief description of the assessment.
5. Achievement Levels A student can achieve one of four levels for
each subject: Exceeds the Standards, Meets the Standards,
Partially Meets the Standards, or Does Not Meet the Standards.
6. Students Performance A graphical representation of the
relationship between the achievement level and the scale score
the student earned for the subject.
7. Scale Score Range A scale score is a conversion of a students
raw score that equalizes possible differences in test difficulty
from one year to the next.
8. School, District, and State Comparison This section graphically
shows the average scale score for all tested students in the
school, district, and state. It provides a quick comparison of the
students performance to reference groups. MTAS and MCAModified averages are reported only at the state level.
Student results for Science MCA will appear on a separate Individual Student Report.
14
Reading and Mathematics MCA Sample Individual Student Report Pages 2 and 3
2. Students Achievement The achievement level earned by the student in the subject.
3. Achievement Level Description A summary of the expected knowledge and skills of the typical Minnesota student scoring at the
achievement level identified. These descriptors are unique for each grade, subject, and achievement level.
4. Score Analysis by Strand A description of the interpretations of sub-scores.
5. Sub-Scores The strands or sub-strands from the Minnesota Academic Standards.
6. Scale Score A scale score is a conversion of a students raw score that equalizes possible differences in test form difficulty.
7. Scale Score Range This section graphically shows the students score and a band of uncertainty around their score.
8. Percentile Rank A comparison among other students who took this test.
9. Lexile Measure The predicted Lexile measure of a student reading ability and upper and lower range that helps match a reader with
literature appropriate for their reading skills. Available for Reading MCA only.
10. Subject Progress A graphical representation of a students progress from grade to grade. Student scores at or above the blue line
indicate student performance is meeting or exceeding the standards. Student scores below the blue line indicate students performance is
not meeting standards. Progress scores are not reported for science.
15
MCA-Modified Sample Individual Student Report Pages 2 and 3

modified achievement level identified. These descriptors are unique for each grade, subject, and achievement level.
4. Total Points Earned The total points earned out of the total points on the test.
6. Sub-Scores Points Earned and Points Possible The points the student earned and the number of possible points for each strand or
sub-strand.
7. State Average The average number of points earned for all students tested in the state.
8. What Was Measured? A brief description of what is being assessed by each of the strands or sub-strands of the Minnesota Academic
Standards.
16
MTAS Sample Individual Student Report Pages 2 and 3

5. Sub-Scores The strands or extended benchmarks from the Minnesota Academic Standards.
extended benchmark.
Standards.
9. Reading Access During test administration, the test administrator indicates how the student accessed the reading passage. The choices
available for each passage are: the passage was read independently by the student, the student read along with the test administrator,
and the test administrator read the passage to the student.
10. Scoring Rubric The 03 rubric used for scoring MTAS tasks.
17
Reading and Mathematics Sample Individual Student Report Page 4

1. Address Section The school can use this area to print an address for mailing the Student Report to the students home.
The school district return address has been pre-printed. The report must be tri-folded in order to take advantage of this
section.
2. Learn More Information This section contains an overview of the Minnesota Assessments and a link to the MDE
website.
18
Science MCA Sample Individual Student Report Pages 1 and 2

1. Address Section The school can use this area to print an address for mailing the Student Report to the students home. The school
district return address has been pre-printed. The report must be tri-folded in order to take advantage of this section.
2. About this Report A brief description of the assessment and a link to the MDE website.
3. Student Demographic Information A description of the demographic information for the student, including: Student Name, Local Use #
(optional number assigned by districts to aid in sorting data), MARSS Number (unique student number), UIN (unique identification number
assigned by the vendor and MDE), Grade, Birth Date, School, District.
5. Students Performance A graphical representation of the relationship between the achievement level and the scale score the student
earned for the subject.
6. Scale Score Range A scale score is a conversion of a students raw score that equalizes possible differences in test difficulty from one
year to the next.
19
Science MTAS Sample Individual Student Report Pages 1 and 2

1. Address Section The school can use this area to print an address for mailing the Student Report to the students home. The school
district return address has been pre-printed. The report must be tri-folded in order to take advantage of this section.
5. Students Performance - A graphical representation of the relationship between the achievement level and the scale score the student
6. Scale Score Range A scale score is a conversion of a students raw score that equalizes possible differences in test difficulty from one
year to the next.
8. Sub-Scores The strands or extended benchmarks from the Minnesota Academic Standards.
extended benchmark.
Standards.
20
REPORT DESCRIPTIONS FOR ASSESSMENTS MEETING GRADUATION ASSESSMENT

REQUIREMENTS
Minnesota students must satisfy graduation assessment requirements in order to graduate from a Minnesota public
high school. As a result of legislation enacted in 2013, the graduation assessment requirements have changed from
those indicated in the Individual Student Reports (ISRs). While the ISRs refer specifically to the GRAD requirements,
students now have additional options to meet the graduation assessment requirements. Achieving proficiency on a high
school Reading or Mathematics MCA, MCA-Modified, or MTAS, or passing the Written Composition GRAD and Reading
and Mathematics GRAD retests are no longer the only ways to meet graduation assessment requirements.
21
General Description of the Written Composition GRAD Individual Student Report

An Individual Student Report (ISR) is generated for every student who took the assessment. Schools will receive a hard
copy of each students ISR to send home with the student or mail to the students parent/guardian. Districts can also
access final student-level information through the DSR and SSR files provided on the MDE website and an electronic
copy of the students essay through the Online Reporting System. The ISR describes an individual students performance
in terms of passing score.
The Minnesota Alternate Assessment: Writing is the alternate assessment for the Written Composition GRAD. To
complete the Minnesota Alternate Assessment: Writing, the teacher evaluates the student on a set of writing attributes;
grade 9 students who participate in this assessment will receive an ISR showing the average values for each attribute.
View the Statewide Assessment for Students with Disabilities section of the MDE website for further information
(http://education.state.mn.us/MDE/StuSuc/SpecEdProg/StateAssessStuDisab/index.html)
Description of the Written Composition GRAD Sample Individual Student Report Front and Back Page
1. Address Section The school can use this area to print an address for mailing the Individual Student Report to the students home. The
school district return address has been pre-printed. The report must be bi-folded to take advantage of this section.
2. About the Assessment A brief description of the assessment.
3. Student Demographic Information A description of the demographic information for the student, including: Student Name, Local Use
# (optional number assigned by districts to aid in sorting data), MARSS Number (unique student number), UIN (unique identification
number assigned by the vendor and MDE), Grade, Birth Date, School, and District.
4. Student Score and Passing Status The performance level and passing status earned by the student in the subject.
5. Prompt The prompt the student responded to for this assessment.
6. Holistic Scoring Guidelines A graphical representation of the students score and a description of each possible score point on the
rubric.
22
General Description of the Grade 10 Reading MCA Individual Student Report

Students taking the grade 10 Reading MCA receive a single Individual Student Report (ISR). Because MCA Reading does
not include an embedded GRAD component, students taking these assessments will receive only a GRAD status
statement in addition to their MCA score.
Schools will receive a hard copy of each student's ISR to send home with the student or mail to the student's
parent/guardian. Districts can also access final student-level information through the DSR and SSR files provided on the
MDE Secured Reports website.
Description of the Grade 10 Reading Sample Individual Student Report Front and Back Page
1. Address Section The school can use this area to print an address for mailing the Student Report to the students home. The school district
return address has been pre-printed. The report must be tri-folded in order to take advantage of this section.
5. Students Performance - A graphical representation of the relationship between the achievement level and the scale score the student
6. Scale Score Range A scale score is a conversion of a students raw score that equalizes possible differences in test difficulty from one year
to the next.
8. Students GRAD Achievement A summary indicating whether the student has met the GRAD requirement for that subject.
14. Lexile Measure The predicted Lexile measure and upper and lower range for the student. Available for MCA Reading only.
23
General Description of the Grade 11 Mathematics MCA Individual Student Report

Students taking the grade 11 Mathematics MCA receive a single Individual Student Report (ISR) with the results from
both the MCA and GRAD assessments as shown on the ISR below. For students who meet the GRAD requirement based
on their MCA score, only their MCA scores will be reported. All students taking these assessments will see a GRAD status
statement indicating whether they have met the GRAD requirement based on these assessments. Because the MTAS
and MCA-Modified do not include an embedded GRAD component, students taking these assessments will receive only
a GRAD status statement in addition to their MTAS or MCA-Modified score.
Schools will receive a hard copy of each student's ISR to send home with the student or mail to the student's
parent/guardian. Districts can also access final student-level information through the DSR and SSR files provided on the
MDE website.
Description of the Grade 11 Mathematics Sample
Individual Student Report Page 1
2. Students Achievement Level The achievement level earned
by the student in the subject.
3. Student Demographic Information A description of the
demographic information for the student, including: Student
Name, Local Use # (optional number assigned by districts to
aid in sorting data), MARSS Number (unique student number),
UIN (unique identification number assigned by the vendor and
MDE), Grade, Birth Date, School, District.
4. About this Report A brief description of the assessment.
5. Achievement Levels A student can achieve one of four levels
for each subject: Exceeds the Standards, Meets the Standards,
Partially Meets the Standards, or Does Not Meet the
Standards.
6. Students Performance - A graphical representation of the
relationship between the achievement level and the scale
score the student earned for the subject.
7. Scale Score Range A scale score is a conversion of a
students raw score that equalizes possible differences in test
difficulty from one year to the next.
8. School, District, and State Comparison This section
graphically shows the average scale score for all tested
students in the school, district, and state. It provides a quick
comparison of the students performance to reference
groups. MTAS and MCA-Modified averages are reported only
at the state level.
24
Grade 11 Mathematics Sample Individual Student Report Pages 2 and 3

2. Students Achievement The achievement level earned by the student in the subject. Page 3 indicates whether the
student has met the GRAD requirement for that subject.
3. Achievement Level Description A summary of the expected knowledge and skills of the typical Minnesota student
scoring at the modified achievement level identified. These descriptors are unique for each grade, subject, and
achievement level.
6. Sub-Scores Points Earned and Points Possible The points the student earned and the number of possible points for
each strand or sub- strand.
7. Averages The average number of points earned for all students tested in the school, district, and state.
8. What Was Measured? A brief description of what is being assessed by each of the strands or sub-strands of the
Minnesota Academic Standards.
25
Grade 11 Mathematics Sample Individual Student Report Page 4

1. Address Section The school can use this area to print an address for mailing the Student Report to the students home.
The school district return address has been pre-printed. The report must be tri-folded in order to take advantage of this
section.
2. Learn More Information This section contains an overview of the Minnesota Assessments and a link to the MDE
website.
26
General Description of the Reading and Mathematics GRAD Individual Student Report
Students taking the Reading or Mathematics GRAD assessment receive a separate Individual Student Report (ISR) that
will contain the tested subjects results. Schools will receive a hard copy of each students ISR quarterly to send home
with the student or mail to the students parent/guardian. All students taking these assessments will see a GRAD status
statement indicating whether they have met the GRAD requirement based on these assessments.
Electronic copies of student reports for GRAD retests can be printed from the Online Reporting System and shared with
students and families to inform about graduation status. Districts can also access final student-level information through
the DSR and SSR files provided on the MDE website.
Description of the GRAD Retests Sample Individual Student Report Front and Back
1. Address Section The school can use this area to print an address for mailing the Individual Student Report to the students home. The
school district return address has been pre-printed. The report must be bi-folded to take advantage of this section.
(optional number assigned by districts to aid in sorting data), MARSS Number (unique student number), UIN (unique identification
number assigned by the vendor and MDE), Grade, Birth Date, School, District.
3. Subject The subject being reported.
4. Student Score and GRAD Passing Status The scale score and GRAD passing status earned by the student in the subject.
5. Students Performance A graphical representation of the relationship between the achievement level and the scale score the student
Standards.
7. Sub-Scores Points Earned and Points Possible The points the student earned and the number of possible points for each strand or substrand.
27
STUDENT LABELS
General Description of the Student Labels
The student labels provide test score information for every student tested. These labels can be used on the students
hard-copy permanent file. Districts determine whether they want to receive student labels for Title I assessments.
Student labels are automatically sent for the grade 9 Written Composition GRAD and GRAD retests.
Note: For Title I assessments, only grade 11 Mathematics MCA will include a GRAD score on the label. There is not an
embedded GRAD component in any other assessment.
Description of the Sample Student Labels

1.
2.
3.
4.
5.
Assessment, subject, and testing year.

Student name.
District and school where test was taken.
Student demographic data such as MARSS Number, Grade, Date of Birth, Gender, and Local Use ID.
Students scale score and achievement level or pass status for each subject in the assessment; science and writing will each be on
separate labels.
28
CONTACT INFORMATION
MDE Contacts
Area Code
612
651, 763, 952
507, 320
218
AIR Contacts
Contact
Jennifer Burton
Tracy Cerda
Lisa Grasdalen
Julie Nielsen-Fuhrmann
Linda Sams Manager
General Inquiries
Contact
American Institutes for Research Help Desk
Phone Number
651-582-8622
651-582-8692
651-582-8485
651-582-8837
651-582-8431
651-582-8231
E-mail
Jennifer.Burton@state.mn.us
Tracy.Cerda@state.mn.us
Lisa.Grasdalen@state.mn.us
Julie.Nielsen-Fuhrmann@state.mn.us
Linda.Sams@state.mn.us
mde.testing@state.mn.us
Phone Number
877-215-8749
E-mail
MNHelpDesk@air.org
To assist with the report descriptions, we include sample reports in this guide. The names, scores, and
other data displayed in this document are fictitious, used solely for the purpose of demonstrating the
functionality of Minnesota testing and reporting. Any similarity to real persons or assessment results is
coincidental and not intended by MDE or AIR.
29
Appendix A:
Resumes
Resume Table of Contents

Executive Leadership
Jon Cohen, Ph.D.

Chief Operating Officer, AIR Assessment
A-1
A-3
Project Management/Oversight
Heather Hayes, J.D.
Corporate Director, Online Testing
Dee Wagner, M.B.A.
Corporate Director, Operations
Kevin Murphy, M.A., PMP
Cynthia Benis, B.S.
Help Desk Manager
Tamonica Jackson
Matthew Hirsch
A-5
A-7
A-9
A-11
A-13
A-15
A-17
Psychometrics
Harold Doran, Ph.D.
Vice President and Chief Psychometrician

Psychometrician
A-19
A-21
A-23

Stacy Tsakeris, M.B.A., PMP
Corporate Director, Computer and Statistical Sciences Center A-25

Project Manager, Computer-Based Testing
A-27
Student Registration (TIDE)

Shailesh Gattewar, M.B.A., PMP Software Project Manager, Student Registration
A-29
Anil Chevuri, B.E.
Senior Application Developer, Student Registration A-31
Item Bank (ITS)
Zorina Alliata, M.B.A., PMP
Dipesh Mamtora, M.S.
Software Project Manager, Item Bank

Senior Application Developer, Item Bank
A-33
A-35
Test Delivery (TDS)

Jeremy Fisher, M.S.
Aaron Newman, M.S.
Software Project Manager, Test Delivery

Senior Application Developer, Test Delivery
A-37
A-39
Analysis Systems [Quality Monitoring (QM) and Database of Record (DOR)]

Brian Kane, Ph.D. PMP
Software Project Manager, Analysis Systems
A-41
Paul Van Wamelen, Ph.D.
Senior Application Developer, Analysis Systems
A-43
Page i
Software Quality Assurance

Dorian Sofiaj, M.S.
Software Quality Assurance Lead
A-45
Network Engineering
Graham Tapscott
Network Engineer Lead
A-47

Bani Dheer, J.D.
Cathy Kugler, B.A.
Corporate Director, Communications and Reporting

Lead, Handscoring
A-49
A-51
A-53
Test Development
June Zack, M.A.
Sandra Durden, M.A.
Corporate Director, Test Development

A-55
A-57
ELA
Diana Reed, Ed.M.
Terry Hill, B.A.
Allen Chong, M.Ed.
Mark Lovely, M.F.A.
Sean Redmond, M.A.
Content Lead, English Language Arts

Lead, English Language Arts Grades 35
Lead, English Language Arts High School
A-59
A-61
A-63
A-65
A-67
A-69
A-71
A-73
A-75
A-77
Math
Jim McCann, M.S.
Maureen Font, M.Ed.
George Ligon, M.Ed.

Lead, Mathematics Grades 35
Lead, Mathematics High School
A-79
A-81
A-83
A-85
A-87
A-89
A-91
A-93
A-95
Page ii
Processing Operations Staff, Data Recognition Corporation (DRC)

Paper Operations Management
Peter Tressel
Project Manager, Paper/Pencil
A-97
Handscoring
Sherri Woolf
Wendy Marik
Pam Ahrens
Coskun Baysoy
Lead, Hand Scoring

Project Manager, Handscoring
ELA Lead Scoring Director
Math Lead Scoring Director
A-99
A-101
A-103
A-105
Test Development
James Bell
Anne Kirpes
John Selisky
Christopher Peterson
ELA Content Lead

ELA Content Specialist
Math Content Lead
Math Content Specialist
A-107
A-109
A-111
A-113
Psychometrics
Christie Plackner
Yi Du
Manager, Psychometric Quality

Psychometrician
A-115
A-117
Page iii
Jon Cohen, Ph.D.

Education
Ph.D.
M.A.
B.A.
1992, Government (Methodology and American Politics)

University of Texas at Austin, Austin, TX
1988, Government (Methodology and American Politics)
University of Texas at Austin, Austin, TX
1985, Politics and Philosophy
Bennington College, Bennington, VT
Present Position
Executive Vice President, AIR, and President, AIR Assessment Program,
American Institutes for Research (AIR)
Responsible for all aspects of AIRs K12 assessment program, with annual revenues of
$100 million. As Chief Statistician, responsible for ensuring that AIRs statistical research
contributes to our mission of improving society through better social and behavioral
science. This entails responsibility for developing and evaluating new statistical procedures
as well as the software to implement them.
Project Experience at AIR

Successfully directs the Assessment Program, responsible for all aspects of AIRs K12 assessment program, with annual revenues of $100 million (2004Present), and for the statistical
research of the program, as AIRs Chief Statistician (1998Present). Director, Statistical
Services for Secondary Analysis, NAEP, Dr. Cohen leads the development, distribution, and
support of statistical software designed for the analysis of NAEP data (1999Present). Director
of Statistics and Psychometrics, National Assessment of Adult Literacy (1999Present), this
study utilizes a multistage sample design selected with unequal probabilities and a partially
balanced, incomplete block matrix test design. Calibration of the assessment includes both
traditional item response methods and a theory-driven multi-component model developed by
Dr. Cohen. Study Director, Evaluation and Review of Statistical Methods Used in National
Assessment of Educational Progress (NAEP) (20032004). Statistical and Psychometric
Advisor, California High School Exit Exam (CAHSEE) (20002002). Study Director, Models of
Total Survey Error (19962001). Project Director, Technical and Analytical Support for the
Assessment Group of NCES (19962000).
Employment History
2012Present
20042012
1998Present
20032004
19982004
19971998
19881997
19861988
19851988
Executive Vice President, AIR, and President, AIR Assessment Program, AIR,
Washington, DC
Senior Vice President and Director of Assessment Program, AIR, Washington, DC
Chief Statistician, AIR, Washington, DC
Director, Computer and Statistical Sciences Center, AIR, Washington, DC
Associate Director for Research, AIR, Washington, DC
Managing Associate/Senior Statistician, AIR, Washington, DC
Research Associate/Research Analyst/Senior Research Analyst, AIR,
Washington, DC
Systems Analyst, Defense Research Technologies (DRT), Rockville, MD
Teaching Assistant, University of Texas at Austin, Austin, TX
Page A-1

Chief Operating Officer, AIR Assessment
Education
B.B.A.
1985, Management Information Systems

University of Iowa, Des Moines, IA
Present Position
Chief Operating Officer, AIR Assessment, American Institutes for Research (AIR)
Responsible for building on and fulfilling AIRs goal of providing clients with customized
assessments that measure student achievement against the state standards and provide
meaningful score reports that can help students, parents, and educators address any area
of student weakness. Manages and provides leadership to the Assessment Program with a
primary focus on developing and maturing the operational policies and procedures of the
business. Identifies and implements new systems and processes as needed to produce the
quality results required in the high-stakes assessment industry. Communicates directly
with all AIR clients.

Successfully manages and provides leadership to AIRs Assessment Program with a primary
focus on developing and maturing the operational policies and procedures of the business as
the Chief Operating Officer of Assessment (2006Present).
Employment History
2012Present
20062012
19852006
Chief Operating Officer, AIR Assessment, AIR, Washington, DC

Executive Director of Operations, AIR, Washington, DC
Pearson Educational Measurement, Iowa City, IA
Executive Vice President and General Manager (20032006)
Vice President and General Manager (20002003)
Director, Operations (19982000)
Account Executive, Publisher Services (19941998)
Manager State Assessments, Software Development (19891994)
Analyst Programmer/Systems Analyst (19851989)
Page A-3
Heather Hayes, J.D.

Corporate Director, Online Testing
Education
J.D.
B.A.
1996, College of Law

1992, English
Present Position
Corporate Director, Online Testing, American Institutes for Research (AIR)
Responsible for documenting and transferring best practices in online, adaptive, full-year,
multiple-administration testing from Oregon, Minnesota, and new state clients. Mentors
project directors supporting Hawaii and Delaware.

Managed the Oregon Assessment of Knowledge and Skills (OAKS) Online, the first
No Child Left Behind (NCLB)-approved online adaptive state assessment system. As the
Senior Project Director, successfully provided approximately 5 million tests to more than
335,000 Oregon students in grades 312 in math, reading, science, and social science in a
nine-month testing window for the OAKS Online Testing Program (2008Present).
Recently extended OAKS to support Oregons Online Writing and Online English Language
Proficiency Assessments (ELPA). Deployed the writing tests in the OAKS system; set up
AIRs online distributed performance scoring system; and supported multiple in-state scoring
sessions with groups of Oregon educators. Led the configuration and deployment of the
ELPA in the OAKS system along with AIRs performance scoring system and AIR scorers.
Employment History
2008Present
20032008
20032006
20002003
19992000
19971999
19961997
Corporate Director, Online Testing, AIR, Washington, DC

Senior Director/Director, Editorial and Content Development, United States
Legal Markets, LexisNexis, Charlottesville, VA
Director, Online Annotated Statutes Project, LexisNexis, Charlottesville, VA
Editorial and Project Manager, Citations & Case Law, Editorial and Publishing
Operations, North American Legal Markets, LexisNexis, Colorado Springs, CO
Senior Legal Editor, Case Law Enhancement Operations, LexisNexis,
Colorado Springs, CO
Legal Analyst, Professional Publications, LexisNexis, Charlottesville, VA
Tax Examiner, Internal Revenue Service, Austin, TX
Page A-5
Dee Wagner, M.B.A.

Corporate Director, Operations
Education
M.B.A.
B.S.
1993, Business Administration

Philadelphia University, Philadelphia, PA
1980, Business Administration (Operations Management)
University of Delaware, Newark, DE
Professional Credentials and Certifications

International Organization for Standardization (ISO) Certified Lead Assessor,
British Standards Institute, 2002
Six Sigma Black Belt (statistical process control to achieve total customer satisfaction in
services as well as products), Six Sigma Research Institute, Motorola University, 2002
Present Position
Corporate Director, Operations, American Institutes for Research (AIR)
Oversees the work of 42 project management staff members and content, technical, score
reporting, production, and operations team leaders. Manages multiple separate contracts for
Ohio, covering summative assessments grades 38 and graduation tests in five subjects, Ohios
test of English Language Acquisition, Ohios Alternate Assessments for Students with Disabilities,
and other smaller efforts. Manages the activities of more than 80 staff and supervises the
Columbus office, which runs on schedule and within budget and receives high client satisfaction.
Ensures that test development staff meet deadlines. Works to restructure almost every aspect of
the development process, including committee review procedures and staffing, bias reviews,
rangefinding, and item development schedules. Manages subcontracts with AIRs operations
partners and is intimately involved in test operationsfrom production to distributionof regular and special versions. Personally led the redesign of the accommodations for testing in other
languages, increasing the abilities of the state to test students and of AIR to deliver a high-quality
product. Has developed business plans for score reporting, websites, and other new initiatives.

Currently oversees the work of AIRs content, technical, score reporting, production, and operations team leaders as the Corporate Director of Project Management (2003Present). Successfully
maintained responsibility for managing and overseeing all aspects of the Training and Technical
Assistance Center for the Comprehensive Community Mental Health Services for Children and
Their Families program as the Acting Project Director/Director of Operations (20012003). These
responsibilities included working closely with the client, multiple program partners, and subcontractors to deliver relevant and timely technical assistance to 63 federally funded systems of care
community sites. Managed and oversaw all logistical aspects of the School District of Philadelphias
Citywide Proficiency Exams as AIRs Director of Operations (20002001).
Employment History
2003Present
20012003
20002001
19921994
19901992
19851990
Corporate Director, Operations, AIR, Washington, DC

Acting Project Director/Director of Operations, AIR, Washington, DC
Director of Operations, AIR, Washington, DC
Quality Performance Specialist, Asea Brown Boveri, Richmond, VA
Operations Analyst/Financial Controller, Asea Brown Boveri, Richmond, VA
Associate Program Director, Advanced Placement Program, Educational
Testing Service, Princeton, NJ
Page A-7

Education
B.S.
1995, Elementary Education

University of Toledo, Toledo, Ohio
Present Position
Senior Project Manager, American Institutes for Research (AIR)
Responsible for managing statewide assessment contracts, including direct client contact,
program communication, documentation and scheduling, on-time delivery of high-quality
products and services, staff supervision, management of subcontractors, meeting facilitation, training and development, and other areas of planning and implementation necessary
to ensure fulfillment of contract requirements. Oversees the planning and implementation
of new initiatives to establish standards in project management and test development and
operations processes.

Manages Oregons very successful online testing project (2012Present) and serves as Senior
Project Manager to the Ohio Administrative Knowledge System (OAKS) Online Assessment
(2012Present).
Employment History
2012Present
20102012
20022010
19992002
Senior Project Manager, AIR, Chicago, IL

Senior Program Manager, Questar Assessment, Inc., Minneapolis, MN
Senior Program Manager, Houghton Mifflin Harcourt, Boston, MA
Operations/Marketing Manager, COIN Educational Products, Toledo, OH
Page A-9
Kevin Murphy, M.A., PMP

Education
M.A.
B.A.
1999, Secondary Education

CUNYHunter College, New York, NY
1994, International Relations; International Business
Lehigh University, Bethlehem, PA

Project Management Professional (PMP), Project Management Institute, 2008
New York State Teacher Certification, Social Studies, Grades 712, 1999
Present Position
Director, State Programs, American Institutes for Research (AIR)
Responsible for managing statewide assessment contracts, including direct client contact,
program communication, documentation and scheduling, on-time delivery of high-quality
products and services, staff supervision, management of subcontractors, meeting facilitation, training and development, and other areas of planning and implementation necessary
to ensure fulfillment of contract requirements. Oversees the planning and implementation
of new initiatives to establish standards in project management and test development and
operations processes.

Successfully managed all aspects of assessment contracts for the paper-and-pencil testing
program for 800,000 students in the Ohio Achievement Assessments (20042007), the
transition from paper-and-pencil testing programs to 100% online in Hawaii and Delaware
(20082011), and Minnesotas dual-mode program (20112013). Served as Project Director
for Chicago Public Schools High School Transformation Project (2008) and the Chicago
Public Schools Social and Emotional Learning Project (20052006).
Employment History
2008Present
20072008
20042007
20012004
20002001
Director, State Programs, AIR, Washington DC

Program Manager, McGraw-Hill/The Grow Network, New York, NY
Senior Operations Specialist, AIR, Washington, DC
Education Research and Evaluation Consultant, North Carolina Testing
Program (North Carolina Department of Public Instruction and North
Carolina State University)
Teacher, Wake County Public Schools, North Carolina
Professional Affiliations
Institute for Educational Leadership (IEL)
Professional Presentations
Biggs, P., Murphy, K., Thomas, J., & Grammer, B. (2004, November). Making wise decisions
about participation in the statewide testing programunlocking the mysteries of testing.
Paper presented at the 54th Conference on Exceptional Children/NC Department of Public
Instruction, Greensboro, NC.
Page A-11
Cynthia Benis, B.S.

Help Desk Manager
Education
B.S.
1981, Marketing
The Ohio State University, Columbus, OH
Present Position
Help Desk Manager, American Institutes for Research (AIR)
Manages AIRs help desk knowledge bases, inquiry tracking systems, and telephone and
electronic communications systems. Oversees full-day support across five time zones.
Responsible for the recruitment, hiring, and daily activities of the AIR customer help desk
staff. Trains, mentors, and coaches staff on all policies and procedures, including compliance with established service-level agreements. Provides direct oversight and management,
using standard help desk metrics to manage work flow. Prepares customer service quality
plans, surveys, and reports and creates call monitoring standards. Evaluates department
performance and communicate daily operational activity to client and staff. Provides a
technology-based help desk that responds to nontechnical users of AIRs testing systems.
Ensures client agreements are met.

Leads help desk operations as the Help Desk Manager at the Ohio Department of Education
for all state assessments (2008Present); for the Oregon Assessment of Knowledge and Skills
(OAKS), Oregon Department of Education (2008Present); New Mexico Alternate Performance
Assessment (NMAPA), New Mexico Public Education Department (2008Present); South
Carolina Alternate Assessment (SC-Alt), South Carolina Department of Education
(2008Present); Delaware Comprehensive Assessment System (DCAS) and Delaware
Comprehensive Assessment System-Alternate 1 (DCAS-Alt1), Delaware Department of
Education (2009Present); Hawaii State Assessment (HSA) and Hawaii State Alternate
Assessment (HSA-Alt), Hawaii Department of Education (2009Present); Minnesota
Comprehensive Assessments (MCA) and Minnesota Test of Academic Skills (MTAS),
Minnesota Department of Education (2010Present); and the Pilot and Field-Test
Administration, Smarter Balanced Assessment Consortium (2011Present).
Employment History
2008Present
20042007
20012004
20002001
19982000
19921998
19891992
Help Desk Manager, AIR, Columbus, OH

Customer Service Team Manager, Employment Benefit Management
Corporation, Dublin, OH
Customer Service Manager, CompManagement Health Systems, Dublin, OH
MIS Requirements Researcher, Netsmart Technologies, Dublin, OH
Customer Support Manager, Netsmart Technologies, Dublin, OH
Team Leader, MIS Support, Netsmart Technologies, Dublin, OH
Applications Analyst I, MIS Support, Netsmart Technologies, Dublin, OH
Page A-13
Tamonica Jackson
Education
Coursework 19971989, Business Management
Morgan State University, Baltimore, MD
Present Position
Senior Conference Manager, Administrative Services, American Institutes for Research,
(AIR)
Manages daily operations of conference staff. Provides oversight for conference department
staff members in implementing conference support activities, such as managing and coordinating logistical, professional and administrative support. Responsible for the quality, efficiency, and productivity of services provided by the staff that supports contracts. Manages
support for more than 50 meetings, special events, and conferences throughout the year
with an attendance from 25 to 500+. Support includes directing and coordinating site
search and selection. Support also including budget development and tracking, managing and
coordinating logistical support, hotel and vendor negotiations, development and dissemination of promotional material and post meeting evaluations. Oversees and manages web
registration, traveler reimbursement, and meeting and administrative support services.

More than 5 years experience in planning and managing conferences with AIR on statewide
assessment projects in Hawaii, Delaware, and Minnesota. On these projects she is responsible
for working directly with the clients, managing conference budget, hotel search and selection,
contract negotiations, vendor negotiations, and travel arrangements and reimbursements.
Ms. Jackson plans and manages over 50 meetings per year for participants that range in size of
10500, including TAC, test steering, test content (item and passage), fairness and sensitivity,
rangefinding, rubric validation, data review, standard setting, contract management, and various
advisory committees. Convenes assessment-related meetings in AIR offices, hotels, and conference facilities and assists with registration services, including name badges, tent cards, signs, and
participant lists, and ensures the accuracy of custom and/or individualized packets.
Employment History
2003Present
20022003
19992002
19981999
19961998
19941996
Senior Conference Manager, AIR, Silver Spring, MD

Project Manager, J W Associates
Senior Conference Manager, J W Associates
Conference Manager, Laurel Consulting Group
Conference/Exhibit Coordinator, Research Assessment Management
Conference/Council Relations Assistant, American Association of Community
Colleges
Society of Government Meeting Professionals (SGMP)
Professional Convention Management Association (PCMA)
Page A-15
Matthew Hirsch
Education
Coursework 19951997, Electrical Engineering, Computer Science
University of Maryland, College Park, MD
Present Position
Project Associate, American Institutes for Research (AIR)
Supports the Senior Program Manager in all project management activities, including
meeting agendas, minutes, and associated documentation. Supports integrated scheduling
across all AIR teams and subcontractors and ensures that materials move to and from the
client using established protocols. As an expert in AIRs student registration system from
the project management perspective, works with the Senior Program Manager in reviewing specifications, responding to any district or school upload issues escalated to the
Senior Program Manager, and monitoring orders for paper testing materials to ensure they
are within expected quantities.

Assists the Senior Program Manager in all aspects of operations for Ohios Graduation
Tests (2009Present). Assists in score reports projects with the creation of student standardized test reports for Ohio and Hawaii in support of No Child Left Behind (NCLB).
Provides quality control for digital printing software programming and student test score
data processing. Oversees on-site printing and delivery fulfillment operations at printing
vendor (2011Present). Assisted in the National Assessment of Educational Progress
(NAEP) project with creation of a background question database for use by educational
researchers; provided quality control of database entries; and verified accuracy of test
data (20082011).
Employment History
2008Present
19972007
Project Associate, AIR, Washington, DC

Operations Manager, RadioShack, Rockville, MD
Page A-17

Vice President and Chief Psychometrician
Education
Ph.D.
M.S.
B.A.
1983, Applied Statistics and Psychometrics

University of Kentucky, Lexington, KY
1977, School Psychology
West Virginia College of Graduate Studies, Huntington, WV
1971, Psychology and Philosophy
West Virginia State College, Institute, WV
Present Position
Vice President and Chief Scientist, Psychometrics, American Institutes for Research (AIR)
Provides senior-level guidance on all statistical and psychometric techniques and procedures
used in assessment activities, including item response theory calibrating and equating; classical
statistical analysis; differential item functions (DIF); standard-setting, reliability, and validity
studies; generalizability studies; item banking; distributional projections; and value-added
studies involving hierarchical linear models and structural equation models.

As the Vice President and Chief Scientist, Psychometrics, Dr. Phillips provides senior-level
guidance on all statistical and psychometric techniques and procedures used in assessment
activities (2003Present).
Employment History
2003Present
19992002
19981999
19911997
19861990
19801986
19771983
19711975
Vice President and Chief Scientist, Psychometrics, AIR, Washington, DC

Acting Commissioner, National Center for Education Statistics, U.S. Department
of Education
Deputy Commissioner, National Center for Education Statistics, U.S. Department
of Education
Associate Commissioner, Education Assessment Division, National Center for
Education Statistics, U.S. Department of Education
Branch Chief, Education Assessment Branch, U.S. Department of Education
Branch Chief, Research and Evaluation Branch, Maryland State Department of
Education
Adjunct Instructor, West Virginia State College, West Virginia College of
Graduate Studies, University of Kentucky, University of Maryland
Social Worker, Kanawha County Department of Welfare, Charleston, West Virginia
Selected Publications
Peer-reviewed Journals
Phillips, G. W. (2009). Linking NAEP to NAAL: Adult literacy using high school standards in
mathematics and reading. Manuscript submitted for publication.
Phillips, G. W., & Dossey, J. A. (2008, October). Counting on the future: International benchmarks in mathematics for comparing cities and nations. Washington, DC: American
Institutes for Research.
Phillips, G. W. (2007, April). Expressing international educational achievement in terms of
U.S. performance standards: Linking NAEP achievement levels to TIMSS. Washington,
DC: American Institutes for Research.
Page A-19
Harold Doran, Ph.D.

Education
Ph.D.
M.A.
B.S.
2001, Educational Leadership and Educational Psychology

University of Arizona, Tucson, AZ
1997, Educational Leadership
Northern Arizona University, Flagstaff, AZ
Northern Arizona University, Flagstaff, AZ
Present Position
Principal Research Scientist, American Institutes for Research (AIR)
Psychometrics Director for the Delaware state testing program and the Statistical Director for
AIRs value-added program. Responsible for the development and implementation of valueadded models, the development of growth models to support the No Child Left Behind (NCLB)
Act, statistical computing, the implementation of psychometric models to support operational
testing programs, accountability system design, and assessment policy.

Senior Psychometrician, Florida Value-Added Model Development and Implementation
(2009Present); Senior Psychometrician, Baltimore Value-Added Model (2010Present);
Senior Psychometrician, Oregon Testing Program (20052011); Project Director, Oregon
Growth Models (2008); Project Director, Wisconsin Growth Project (20062007); Project
Director, IES Value-Added Model (20062007); Director of Peer Review Studies for Hawaii
State Assessment (20042005); Senior Research Analyst, Design Task to Evaluate the Validity
of Value-Added Models, Institute for Education Sciences (20032005); Project Director,
Oregon Department of Education Psychometric Contracts (2004).
Employment History
2008Present
20042008
2001Present
20022004
19972002
19941997
Principal Research Scientist, AIR, Washington, DC

Senior Research Analyst, AIR, Washington, DC
Statistical Consultant, Amphitheater School District and District of Columbia
Public Schools, Washington, DC
Director of Research and Evaluation, New American Schools, Arlington, VA
School Principal, Winifred Harelson Elementary School, Tucson, AZ
Classroom Teacher, Winifred Harelson Elementary School, Tucson, AZ
Doran, H. C., & Lockwood, J. R. (2006). Fitting value-added models in R. Journal of
Educational and Behavioral Statistics, 31(2), 205230.
Doran, H. C., & Fleischman, S. (2005). Challenges of value-added assessment. Educational
Leadership, 63(3), 8587.
Doran, H. C. (2005). The information function for the one-parameter logistic model: Is it
reliability? Journal of Educational and Psychological Measurement, 65(5), 759769.
Lockwood, J. R., Doran, H. C., & McCaffrey, D. F. (2003). Using R for estimating longitudinal
student achievement models. The Newsletter of the R Project, 3(3), 1723.
Doran, H. C. (2003). Adding value to accountability. Educational Leadership, 61(3), 5559.
Page A-21

Psychometrician
Education
Ph.D.
M.S.
B.S.
2010, Statistics
Carnegie Mellon University, Pittsburgh, PA
2005, Statistics
Carnegie Mellon University, Pittsburgh, PA
2003, Mathematics (summa cum laude)
Shippensburg University, Shippensburg, PA
Honors and Awards

Institute of Education Sciences Postdoctoral Fellowship, Quantitative Methods, Assessment,
and Evaluation, University of California, Berkeley, 2009
Educational Testing Service Travel Award, Educational Testing Service, 2009
Graduate Programs Office Conference Travel Funds, Carnegie Mellon University, 2009
Statistics Department Student of the Year, Carnegie Mellon, 2009
Institute of Education Sciences Predoctoral Fellowship, Program for Interdisciplinary
Educational Research (PIER), Carnegie Mellon University, 2006
National Science Foundation Graduate Fellowship, Carnegie Mellon University, 2003
Present Position
Psychometrician/Statistician, American Institutes for Research (AIR)
Psychometrician and Statistical support for AIRs Assessment program. Runs operational analyses,
conducts post-equating, and prepares for data reviews and standard setting. Conducts quality
control for final scored student data files and oversees quality control of student reports.

Provides psychometrician and statistical support for Minnesota State Assessments. Runs
operational psychometric work for the Minnesota testing program, including performing
operational and field-test calibrations, conducting post-equating, and preparing materials
for data review and standard setting (2012Present).
Employment History
2012Present
20092011
Psychometrician/Statistician, AIR, Washington, DC

Postdoctoral Scholar, University of California, Berkeley
American Educational Research Association (AERA)
Classification Society of North America (CSNA)
Institute of Mathematical Sciences (IMS)
National Council on Measurement in Education (NCME)
Psychometric Society
Technical Skills
Software: R, SPSS, SAS, Stata, WinBUGS, IRTPRO, Multilog, Winsteps, Microsoft applications.
Ayers, E., Rabe-Hesketh, S., & Nugent, R. (2013). Incorporating student covariates in cognitive
diagnosis models. Journal of Classification, 30(2), 195224.
Page A-23

Corporate Director, Computer and Statistical Sciences Center
Education
M.B.A.
B.S.
Strategic Management, Marketing and Public Policy

Wharton School of Business, University of Pennsylvania, Philadelphia, PA
Business Management (cum laude)
Ateneo de Manila University, Quezon City, Philippines
Present Position
Corporate Director, Computer and Statistical Sciences Center (CSSC), Information
Technology Administration Manager, American Institutes for Research (AIR)
Leads, manages, and supervises a software development team of 80 personnel for the Assessment
Division. The team includes software engineers, web graphic designers, testers, project managers,
system engineers, data analysts, and technical writers. Ensures all assessment software can
effectively provide test development, online and paper-based services, and scoring and reporting
of large-scale high-stakes assessments. These systems include the following: a web-based, secure
Test Delivery System that administers fixed or computer-adaptive tests and scores them dynamically (supports Windows, Macs, and Linux platforms); machine-scorable constructed-response
questions; a quality management system that automatically reviews each test before it is reported;
a web-based learning management system that allows teachers and students to organize and use
a wide range of curricular materials, including identifying learning prerequisites; web-based
reports that provide immediate results at every level, track student progress, and tie testing back
to learning; user registration system and web services that regularly extract student data from
district and school systems; and an item banking system that enables test development and
customized reviews and workflows.

Directs the CSSC and manages the Information Technology Administration (2007Present).
Successfully managed software development projects as the Senior Project Manager and
Quality Assurance Lead for Computer and Statistical Sciences Center (20052007).
Employment History
2007Present
20052007
20032004
20032004
20022003
2001
19992000
Corporate Director, Computer and Statistical Sciences Center (CSSC), AIR,

Washington, DC
Senior Assessment Management Specialist, Computer and Statistical Sciences
Center (CSSC), AIR, Washington, DC
Director of Information Services, New American Schools, Alexandria, VA
Financial Officer, New American Schools, Alexandria, VA
Senior Associate, New American Schools, Alexandria, VA
Summer Associate, Ball Foundation, Glen Ellyn, IL
Part-Time Professor, Ateneo de Manila University, Quezon City, Philippines
Co-Chair, International Admissions Committee, Wharton School of Business, University of
Pennsylvania (20012002)
Executive Editor, eBusiness Review, Wharton School of Business, University of Pennsylvania
(20012002)
Pianist/Keyboardist, Wharton Follies, Wharton School of Business, University of Pennsylvania
(20002002)
Technical Skills
Visual Basic, Ruby, Standard Query Language, Java, C
Page A-25
Stacy Tsakeris, M.B.A., PMP

Project Manager, Computer-Based Testing
Education
M.B.A.
B.S.
2010, Business Administration

University of Maryland University College, Adelphi, MD
2002, Computer Science and Engineering (Specializing in Information Systems)
The Ohio State University, Columbus, OH

Project Management Professional (PMP), Project Management Institute, 2011
Present Position
Project Manager, Computer-Based Testing, American Institutes for Research (AIR)
Manages the quality control for the Smarter Balanced Assessment Consortium computer-based
testing across all AIR systems for a consortium of 24 states. Plans, budgets, and tracks testing
activities, as well as coordinates with AIR Software Project Managers and Software Engineers
for student registration, test delivery, quality assurance, and related systems to ensure quality
control. In addition, manages the documentation for the Open Source testing software that
AIR is developing and delivering to the Open Source repository, which will be available to all
Smarter Balanced states.
Additional Project Experience

Lead and coordinated efforts to upgrade the University of Californias Research and Grants
Program Office (RGPO) Grants Administration and Management software from Altum
proposalCENTRAL & GRAIL to Altum Easygrants. This effort included streamlining workflows
while reducing redundant data entry, manual calculations, and paper forms for RGPO staff and
grant applicants. Managed the upgrade and enhancements to the Sheet Metal Occupational
Health Institute Trusts (SMOHIT) Medical Screening Database system, which was used to
facilitate asbestos testing for more than 250,000 building and trade union members
(20112013).
Employment History
2013Present
20112013
20062011
20032006
Project Manager, Computer-Based Testing, AIR, Washington, DC

Project Manager, Compuware Corporation, McLean, VA
Senior Software Engineer/Analyst, Legal & General America, Frederick, MD
Advanced Computer Developer/Analyst, Lynchval Systems Worldwide,
Chantilly, VA
Technical Skills
Computer Languages: Turbo Pascal, Fortran, Cobol, C, C++, HTML, SQL (DB2/Oracle/MS
SQL/Sybase), Delphi, Visual Basic, .Net concepts, XML, ActiveX Controls, Crystal Reports,
SSRS, SSIS. Database Systems: Models, languages, organization, design, integrity, integrating
new software with Legacy Systems. Operating Systems Experience: Concepts, componentbased software, computer architecture, object-oriented components and design, quality assurance testing. Process-Oriented Improvement: Six Sigma and Capability Maturity Model
Integration. Project Management & Development Tools: CMMI Methodologies, MS Project,
JIRA, TeamCity, Onehub, VersionOne, Basecamp, Installshield, PVCS Version Manager and
Tracker, MS Project.
Page A-27
Shailesh Gattewar, M.B.A., PMP

Software Project Manager, Student Registration
Education
M.B.A.
M.A.
B.A.
2008, Specialization in Project Management,

Business Development, Strategy and Finance
Georgetown University, Washington, DC
2002, Computer Science
The George Washington University, Washington, DC
Mumbai University, Mumbai, India

Project Management Professional (PMP), Project Management Institute (PMI), 2008
Present Position
Senior Technical Project Manager, American Institutes for Research (AIR)
For AIRs Assessment Program, manages the development of four different software products that
form the basis for most of the software systems delivered by AIR, with each product customized
for multiple projects used in different states. Collect requirements for each project from its stakeholders and negotiate important features for building the base products so that the systems
designed are cost-effective and easily expandable. Lead the technical efforts to change the K12
testing system for the schools in Oregon from paper-based to a computer-adaptive system that
worked across multiple operating systems. Manage the development and implementation of AIRs
Test Information Distribution Engine (TIDE) that provides administrators the tools to view, add,
and manage users for administrative tasks associated with assessments; view and edit student
demographic information and test settings; and manage orders for paper test materials.

Supports online systems to support assessment projects in Delaware (2009Present), Hawaii
(2007Present), Minnesota (2010Present), New Mexico (2007Present), Ohio (2009Present),
Oregon (2007Present), South Carolina (2007Present), Utah (2012Present), and the
Smarter Balanced Assessment Consortium (2012Present).
Employment History
2007Present
20042007
20032004
20022003
20022003
Senior Technical Project Manager, AIR, Washington, DC

Project Lead, ISIS, Georgetown University Medical Center, Washington, DC
Software Engineer, Division of Telemedicine and Robotics, ISIS, Georgetown
University Medical Center, Washington, DC
Game Programmer, Lyra Studios, Washington, DC
3-D Application Developer, Center for Prostate Disease Research, Bethesda, MD
Technical Skills
Programming: MS.NET 2.0, C#.Net, VB.Net, ASP.Net, C, C++, Java, J2EE, Perl, XML, XSLT,
SOAP, SQL, Ajax, Atlas, JavaScript, OpenGL, DirectX, UML, KML, ESRI libraries, Python,
MFC; IDE: Visual Studio.Net (for architects, developers, and testers), Emacs, Eclipse, Visual
Perl; Web Server: IIS, Tomcat, .NET Web Services, SQLXML; Methodologies: MSF for Agile
software development, eXtreme Programming (XP), Scrum; GIS: ESRI products (ArcIMS,
ArcGIS, ArcSDE, Shapefiles), NASA, WorldWind, Google Maps, MapPoint, ObjectFX
(SpatialFX); Databases: SQL Server 7.0/2000/2005, Oracle 8i/9i, SQL Express, MySQL;
Platforms: Windows (all), RedHat and RealTime Linux, Solaris, SGI, Apple Macintosh
Page A-29
Anil Chevuri, B.E.

Senior Application Developer, Student Registration
Education
B.E.
2002, Electronics and Communication

Visvesvaraya Technological University, Karnataka, India

Microsoft Certified Professional (70-315): Developing and implementing web applications with
Microsoft Visual C# .NET and Microsoft Visual Studio .NET
Microsoft Certified Professional (70-320): Developing XML web services and server components
with Microsoft Visual C# .NET and the Microsoft .NET Framework
Present Position
Senior Application Developer, American Institutes for Research (AIR)
Provides technical design and analyzes different projects for paper and online testing. Works
as part of a pre-processing system that performs all student data import, export, and analysis.
Responsible for planning, designing, implementing, and deploying the databases across all
clients, including the designing, implementing, and deploying of data input/output collection
services and websites using Visual studio 2012 ASP.net/C#/SQL 2005/2008 for both systems
and maintaining the data integrity during extraction, manipulation, processing, analysis, and
storage. Responsible for preparing the deployment documents and assisting with deployment
of all projects from staging to production environment. Assists with preparation and execution
of test cases.

Works on AIRs Roster Tracking Systems (RTS), Common Login System (CLS/OpenAM), and
material ordering system (TIDE). Involved in requirement gathering for RTS and TIDE database and website (2007Present).
Employment History
2007Present
20062007
20032006
20022003
Senior Application Developer, AIR, Washington, DC

Software Engineer, Tellabs, Naperville, IL
Programmer, VisualSoft, Motorola/Brinks System, Hyderabad, India
Programmer, Prakasha Arts, Hyderabad, India
Technical Skills
.Net: .NET Framework 1.1, 2.0, and 3.5; ASP.NET; Web Services (SOA); Windows Forms;
ADO.NET; Remoting; Silverlight. Languages: C#, VB.NET, C, Visual Basic 6.0/5.0, SQL, T-SQL,
Java. Server-Side Programming: ASP.NET, BizTalk, ASP. Web Technologies: XML, XSLT,
XPATH, HTML, DHTML, CSS. Scripting: JavaScript, JQuery. Databases: SQL Server
2000/2005/2008, ORACLE 8i/9i. Tools: Redgate SQL compare, Redgate Data compare, SQL
prompt, Visio, Photoshop, Microsoft FrontPage, Macromedia Dreamweaver Ultradev, Jetbrain
profiler. Distributed System: Web Services, Sockets. Development Environment: Microsoft
Visual Studio .NET 2003, Visual Studio .NET 2005, Visual Studio .NET 2008. Documentation
Tools: Visio, NDoc, MS Office. Version Control Systems: Visual Source Safe, PVCS, WINCVS.
Page A-31
Zorina Alliata, M.B.A., PMP

Software Project Manager, Item Bank
Education
M.B.A.
B.S.
2003, International Marketing

Strayer University, Rockville, MD
Gh. Asachi Technical University of Iasi, Iasi, Romania

Certified ScrumMaster (CSM), Scrum Alliance, 2013
Present Position
Manages a team of developers and help desk operators to support clients and internal
departments as needed. Create and manage project plans, schedules, and resource allocations. Interact with project directors and item development managers to communicate
statuses, milestones, and project risks. Communicate with managers of other projects to
ensure the item banks are correctly set up and to determine and resolve issues with item
types and formats. Review testing procedures and communicate with the quality control
team to effect process improvement. Continually improve the release management,
business analysis/requirement, and change request processes.

Manages the online systems for AIRs Assessment Program (Item Bank System (IBS), Content
Standards Repository (CSR), and Roadmap projects) to support assessment projects in
Delaware (2013Present), Hawaii (2013Present), Minnesota (2013Present), New Mexico
(2013Present), Ohio (2013Present), Oregon (2013Present), South Carolina
(2013Present), Utah (2013Present), and the Smarter Balanced Assessment Consortium
(2013Present).
Employment History
2013Present
20122013
20072012
20032007
20002003

Senior Business Consultant/Project Manager, Aquilent, Laurel, MD
Senior Business Analyst/Project Manager, FINRA , Rockville, MD
Manager Web Development, SHRM, Alexandria, VA
Systems Analyst/Programmer, The NASDAQ Stock Market, Inc., Rockville, MD
Member, Project Management Institute (PMI)
Member, Scrum Alliance
Page A-33
Dipesh Mamtora, M.S.

Senior Application Developer, Item Bank
Education
M.S.
B.E.
2003, Electrical Engineering

University of Maryland, Baltimore County, MD
2001, Electronics and Telecommunications
University of Mumbai, Mumbai, India
Present Position
Senior Application Developer, Online Reporting System, American Institutes for Research (AIR)
Works with the development of the Online Reporting System, which is a web-based
application providing student assessment reports to states, districts, schools, and teachers.
Created the software library using C#.NET 2.0 to support a complex web-based reporting
application created using ASP.NET 2.0, using C#.NET, and Microsoft SQL Server 2005.

As the Senior Software Engineer, responsible for debugging, making upgrades, and maintaining
the application code and database to the Test Information Distribution Engine (TIDE)
(2007Present). As the Lead Software Engineer, works with a team of three developers to
implement the redesigned solution of the Item Tracking System (ITS) (2008Present).
Employment History
2007Present
20032007
2002
20012003
Senior Application Developer, Online Reporting System, AIR, Washington, DC

Lead Software Engineer, Blue Sky Factory, Inc., Baltimore, MD
Software Engineer Intern, University of Maryland, Baltimore, Dental School,
Baltimore, MD
Teaching Assistant, University of Maryland, Baltimore, MD
Technical Skills
Programming languages: Microsoft C# .NET, Visual Basic.NET, C, C++, Java. Web technologies:
ASP.NET 2.0, ASP.NET 3.5, ASP.NET AJAX, AJAX, Macromedia ColdFusion 5/6.1/7, JavaScript,
DHTML, CSS, XML, XML web services using ASP.NET, MATLAB. Other technologies: MATLAB,
Adobe Photoshop, familiar with Macromedia/Adobe Flash. Database: Microsoft SQL Server,
Microsoft Access, MySQL. Platforms: All Microsoft Windows platforms, Linux.
Page A-35
Jeremy Fisher, M.S.

Software Project Manager, Test Delivery
Education
M.S.
B.S.
2009, Systems Development Management

Strayer University, Rockville, MD
2006, Information Sciences and Technology
Pennsylvania State University, University Park, PA
Present Position
Technical Project Manager, American Institutes for Research (AIR)
Responsible for several major applications that facilitate online testing, including requirements
gathering; creating and maintaining system documentation; help desk support; client coordination; facilitating platform item review; conducting onsite and offsite software demonstrations; creating and coordinating project schedules with project directors, team leads, and
other staff; aiding the software development and testing; and integrating with other teams.
Creates custom reports by developing and running database queries, assisting the developers
and testers in uncovering system defects, coding in C#, and assisting the developers with
bringing new system instances online. Conducts research on design principles and investigates
new software to aid in the software development, testing, and management of the project.
Delivers the Test Delivery System (TDS).

Supports online systems for assessment projects in Delaware (2009Present), Hawaii
Employment History
2007Present
Technical Project Manager, AIR, Washington, DC
Page A-37
Aaron Newman, M.S.

Senior Application Developer, Test Delivery
Education
M.S.
B.A.
1996, Electrical Engineering and Computer Science

University of Illinois at Chicago, Chicago, IL
1990, Music
University of Wisconsin, Madison, WI
Present Position
Senior Application Developer Engineer, American Institutes for Research (AIR)
Manages an item development team. This team works with educators, graphic designers,
database analysts, and other technical staff to coordinate creation, delivery, and scoring of
assessment items.
Develops high-performance, cross-browser (including mobile) user interfaces for AIR
assessments. Interfaces need to account for accessibility, including keyboard accessibility,
text-to-speech, American Sign Language, visual impairments, and other special accommodations.
Develops and maintains a Student Geographical Distribution application, which allows
transparent load balancing of assessments across multiple servers or geographical regions.
Employment History
2013Present
19982012
19961998
Senior Application Developer Engineer, AIR, Naperville, IL

Principal Engineer, Tellabs, Naperville, IL
Software Engineer, Tollgrade, Inc. (formerly Teradyne, Inc.), Deerfield, IL
Technical Skills
Programming: MS.NET 4.0, C#.Net, ASP.Net, C, C++, Java, Perl, XML, XSLT, SOAP, SQL,
JavaScript, UML, Python, MFC. IDE: Visual Studio.Net (for architects, developers, and testers),
Eclipse, IIS, Tomcat, .NET Web Services, SQLXML. Methodologies: MSF for Agile software
development, eXtreme Programming (XP), Scrum; Fagan, PSP/TSP. Databases: SQL Server
7.0/2000/2005, SQL Express, MySQL. Platforms: Windows (all), RedHat and Ubuntu Linux,
Solaris, Apple Macintosh, Android, IOS, WindRiver, OSE.
Page A-39
Brian Kane, Ph.D., PMP

Software Project Manager, Analysis Systems
Education
Ph.D.
M.A.
S.B.
1996, Astronomy
Boston University, Boston, MA
1990, Astronomy
Boston University, Boston, MA
1986, Mathematics and Physics
College of William and Mary, Williamsburg, VA

Honors and Awards

Presidents Quality Award, Indus Corporation, 2005
Present Position
For AIRs assessment program, manages a team of seven architects, developers, and testers
supporting the scoring, quality control, reporting, and delivery of data from tests taken during
online educational assessments. Responsibilities include operational and developmental management of projects with a total contract worth of $30 million; planning and coordinating with
content developers and psychometricians; and analysis of functional, technical, and qualitydriven system requirements. Provides expertise in mission-critical IT systems and full-cycle
web software development for large, longitudinal data sets.

Supports online systems to support assessment projects in Delaware (2012Present), Hawaii
Employment History
2012Present
20022012
19972008
20012005
20012002
19992001

Senior Technical Lead/Project Manager, Indus Corporation, Washington, DC
Nonprofit Owner and Developer, Astroplace, Inc., Washington, DC
Faculty Associate, Georgetown University, Washington, DC
Senior Systems Analyst, Signal Corporation, Washington, DC
Senior Developer, Tisani, Bethesda, MD
Member, Project Management Institute (PMI)
Kane, B.D., & Clemens, D.P. (1997). Rotation of starless bok globules. Astronomical Journal,
113, 17991814.
Page A-41
Paul Van Wamelen, Ph.D.

Senior Application Developer, Analysis Systems
Education
Ph.D.
M.S.
B.S.
1994, Mathematics
University of California, San Diego, CA
1992, Mathematics
University of California, San Diego, CA
1988, Mathematics and Physics
University of Pretoria, Pretoria, South Africa
Present Position
Senior Application Developer, American Institutes for Research (AIR)
Leads the Analysis group within the Computer and Statistical Sciences Center (CSSC).
Responsible for the development and implementation of statistical and aggregate reporting of
educational data (10 SAS programmers) and another, smaller group developing .Net applications for data manipulation in support of online reporting.

Successfully leads a team within the Computer and Statistical Sciences Center (CSSC) as the
Senior Application Developer, in support of online reporting (2006Present).
Employment History
2006Present
19942006
Senior Application Developer, AIR, Washington, DC

Associate Professor, Louisiana State University, Baton Rouge, LA
Van Wamelen, P. B. (1999). Examples of genus two CM curves defined over the rationals.
Mathematics of Computation, 68(225), 307320.
Van Wamelen, P. B. (1999). Proving that a genus 2 curve has complex multiplication.
Mathematics of Computation, 68(228), 16631677.
Van Wamelen, P. B. (2000). Poonens question concerning isogenies between Smarts genus 2
curves. Mathematics of Computation, 69(232), 16851697.
Van Wamelen, P. B. (2002). Jacobi sums over finite fields. Acta Arith, 102(1), 120.
Van Wamelen, P. B., Li, Z., & Iyengar, S.S. (2004). A fast expected time algorithm for the 2-D
point pattern matching problem. Pattern Recognition, 37(8), 16991711.
Van Wamelen, P. B. (2007). Enumerating Motzkin-Rabin geometries. Journal of Combinatorial
Designs, 15(3), 179194.
Page A-43
Dorian Sofiaj, M.S.

Software Quality Assurance Lead
Education
M.S.
M.S.
Certificate
B.S.

American University, Washington, DC
2001, Applied Physics
American University, Washington, DC
1994, Mechanical Engineering
University of Tirana, Tirana, Albania
Present Position
Senior Quality Control Specialist, Manager of Software Testing, CSSC,
American Institutes for Research (AIR)
Leads and manages a 12-member Quality Assurance team through efficient and effective testing practices, which ensure that every system deployed meets specifications, functions, provides a satisfying user experience, and accurately captures or generates data. Also responsible
for ensuring that assessment systems and statistical products delivered to clients are properly
tested within the expected timeframes. Supports the efforts toward improving quality assurance and testing in all aspects of AIRs Computer and Statistical Sciences Center.
Employment History
2009Present
20052009
20042005
20022004
20012002
Senior Quality Control Specialist, Manager of Software Testing, CSSC,

AIR, Washington, DC
Senior Technical Risk Specialist, Test Automation Team Lead, Fannie Mae,
Washington, DC
Senior Technical Risk Specialist, Backend Test Automation Team Lead,
Fannie Mae, Washington, DC
IT Analyst, Backend Lead Tester/Automation Engineer, Fannie Mae,
Washington, DC
IT Analyst, Backend Tester, Fannie Mae, Washington, DC
Technical Skills
Programming Language: Java. Database Language: SQL, PL/SQL, TSQL. Scripting: Perl, UNIX
scripting. Tools Scripting Languages: TSL (WinRunner), VBScript (QTP).
Page A-45
Graham Tapscott
Network Engineer Lead
Education
B.S.
(In Progress) Computer and Information Science

University of Maryland University College, Adelphi, MD
Present Position
Network Specialist, American Institutes for Research (AIR)
Responsible for the overall network and system engineering for the assessment divisions
web hosting environments. Additional responsibilities include system and network analysis,
performance tuning, and application support for the Computing and Statistical Sciences
group and its team of software developers, statisticians, and quality assurance specialists.

Serves as the primary system engineer for the initial and subsequent deployments of the
Deployment and System Management of the Web-Based Test Delivery System (TDS) Platform
(2008Present). Responsible for building hardware specifications, system configuration, and
performance tuning to support the transactional and bandwidth requirements that the senior
computer scientists provide. As the Network Specialist, responsibilities include the overall
network and system engineering for the Assessment divisions web hosting environments
(2004Present).
Employment History
2004Present
20022004
19992002
19981999
Network Specialist, AIR, Washington, DC

Network Analyst II, Infopro Inc., Reston, VA
System Engineer, Legg Mason, Wood Walker, Baltimore, MD
Field Technician, US Communications, Annapolis, MD
UsenixSystem Administrator Special Interest Group (SAGE)
The League of Professional System Administrators (LOPSA)
Technical Skills
Software: Windows Server and UNIX/Linux System administration; Microsoft SQL and
MySQL database administration; VB, PHP, Perl, C, C++, and C# programming; ASP.NET
Web development environments; TCP/IP network administration/troubleshooting;
Network/system design and documentation
Page A-47
Bani Dheer, J.D.

Corporate Director, Communications and Reporting
Education
J.D.
B.A.
1995, Juris Doctorate

University of British Columbia, Vancouver, BC
1990, International Relations
University of British Columbia, Vancouver, BC
Present Position
Corporate Director, Communications and Reporting, American Institutes for Research (AIR)
Oversees all AIR assessment activities related to the design, production, deployment, and
delivery of paper and online score reports for the Minnesota Department of Education,
Delaware Department of Education, Hawaii Department of Education, Oregon Department of
Education, Ohio Department of Education, New Mexico Department of Education, and the
South Carolina Department of Education. Built state of the art paper and online reporting
capabilities for AIRs assessment program. Leveraged professional expertise in communications
and market research to design, test, and refine data use and score reporting efforts with
Department of Education officials to produce and distribute test score reports with evidence of
impact on instruction. Scaled AIRs paper test score reporting capabilities from a single-district
pilot in 2004 to seven large-scale assessment programs in 2008. Integrated paper reporting
function with online reporting capabilities in one-and-a-half years. Managed operations involving designing and delivering more than 30 million pages of full-color score reports with variable
text and graphics intended to drive changes in instruction for large-scale assessment programs.

Establishes and manages AIRs relationship with the Program for International Student
Achievement (PISA) of the Organization for Economic Cooperation and Development (OECD)
as the Director, International Benchmarking for State Assessments (2009Present). Oversees
the design, development, production, and delivery of family, class, school, and complex area
reports for Hawaii Department of Education reports for grades 38 and 10 (2006Present).
Oversaw all deployments of the online reporting system for Delawares Department of
Education reports for grades 210 (2009). Oversaw production and delivery of score reports
for Ohios Alternate Assessment for Students with Disabilities (20042008), Ohios assessment
system for grades 38 (20042008), and Ohios Graduation Test (20062009). Oversaw the
design, development, production, and delivery of Chicago Public Schools reports on learning
environment (20062007) and South Carolinas reports for the End-of-Course (EOC) high
school exit exam (20062007).
Employment History
2004Present
20022004
20012004
20002001
19972000
Corporate Director, Communications and Reporting, AIR, Washington, DC

Senior Researcher, The Mellman Group, Washington, DC
Principal, Cook & Dheer, Vancouver, BC
Research Associate, The Strategic Counsel, Toronto, ON
Researcher/Consultant, d-Code, Inc., Toronto, ON
Dheer, B. (1999, 2000). Research findings on youth consumer trends, Marketing.
Page A-49
Cathy Kugler, B.A.

Education
B.A.
2006, Psychology
Present Position
Statistical Support Associate, American Institutes for Research (AIR)
Provides research, development, and logistical support to senior psychometric staff in the
Assessment program. Composes and edits technical reports, analysis specifications, and
reporting documents related to the statistical integrity of standardized testing programs.
Manages cross-project tasks and quality control efforts for multimillion-dollar state projects.
Guides activities of junior statistical support staff.

Manages psychometric team schedules and deliverables for state assessments as a
Psychometric team member. Responsibilities also include facilitating transition of technical
item banks from previous testing contractor to AIR, developing quality control procedures for
implementation of test administration and scoring systems for assessments, including the
Minnesota Department of Education, Minnesota Assessment Program (2011Present); Ohio
Department of Education, (2006Present); Ohio Graduation Test (2006Present); Ohio Test of
English Language Acquisition, (2006Present); Ohio Alternate Assessments for Students with
Disabilities (2006Present); and Ohio Achievement Assessments (2006Present). As a
Research Assistant, provided support and assistance to AIRs Assessment division of
Psychometricians and Statisticians by generating and producing materials for item review and
standard-setting meetings. Provided support for data analysis programs. Performed quality
assurance checks on analysis outputs, preparation and development of technical reports, and
review of all field-test and operational-test forms (20062008).
Employment History
2008Present
20062008
Statistical Support Associate, AIR, Washington, DC

Research Assistant, AIR, Washington, DC
Page A-51

Lead, Handscoring
Education
B.A.
2006, Economics, International Business and Management

(magna cum laude)
Dickinson College, Carlisle, PA
Present Position
Score Reporting Project Coordinator, American Institutes for Research (AIR)
Manages score reporting projects from mockup development to report programming,
testing, and production. Participates in weekly internal and external client meetings to
discuss schedules and reporting specifications. Communicates client-requested reporting
modifications to designers, statistical and graphical programmers, and print vendors.
Identifies and implements process improvements in the production of score reports.

Manages score reporting projects from mockup development to report programming, testing,
and production as the Score Reporting Project Coordinator for assessments including the
South Carolina Alternate Assessment (2010Present), New Mexico Alternate Assessment
(2010Present), and Delaware State Assessments (2011Present). As a Research Assistant
responsibilities included conducting research through focus groups consisting of district test
coordinators, principals, and teachers to identify improvements in the presentation of
assessment data and information for the Hawaii Alternate Assessment (20122013), Ohio
Alternate Assessment (20122013), Minnesota State Assessment (20112012), and the
Cleveland CFT project (20102013).
Employment History
2012Present
20092012
20072009
20062007
20052006
20042005
Score Reporting Project Coordinator, AIR, Washington, DC

Research Assistant, AIR, Washington, DC
Learning Resource Center (LRC) Monitor, Strayer University, Washington, DC
Facilities Asset Analyst, Sightlines LLC, Guilford, CT
Marketing Contractor, Z-Band, Inc., Carlisle, PA
Office Assistant, Dickinson College, Library and Information Services (LIS),
Carlisle, PA
Omicron Delta Epsilon (Economics Honor Society)
Sigma Beta Delta (International Honor Society for Business, Management, and Administration)
Alpha Lambda Delta (Freshman Honor Society)
Page A-53
June Zack, M.A.

Corporate Director, Test Development
Education
M.A.
B.A.
1980, Educational Psychology (coursework included Psychometric Theory,

Educational Psychology, Reading Instruction, Cognitive Psychology,
Psycholinguistics, and Neuropsychology)
New York University, New York, NY
1968, Psychology (coursework included Psychology, Speech Pathology,
Audiology, and Educational Psychology)
Douglass College, Rutgers, the State University of New Jersey
Present Position
Corporate Director, Test Development, Assessment, American Institutes for Research (AIR)
Responsible for overseeing the development of reading, writing, mathematics, science,
social studies, and special education items for all AIR state and national assessment
contracts including Ohios K2 Diagnostic Assessments and Grade 38 Ohio Achievement
Tests in reading, writing, and mathematics, science and social studies.

Responsible for directing the item writing and the assembly of diagnostic and achievement
assessments for grades K5 as the Principal Research Analyst, Manager of Test Development,
Ohio K8 Assessment, State Assessment Projects (2000Present). Item Development Manager,
responsible for managing the development of the blueprints, item specifications, tasks and
items, and test forms for South Carolina Alternate Assessment and New Mexico Alternate
Assessment. Principal Research Analyst, Manager of Test Development, Philadelphia K4
Assessment Program. Principal Research Analyst, Manager of Reading Development, NAEP
Foreign Language Assessment. Principal Research Analyst, Manager of Reading Test
Development, Voluntary National Test Grade.
Employment History
2000Present
1988
19872000
19872000
19851988
19851988
19691988
Corporate Director, Test Development, Assessment, AIR, Washington, DC

Evaluator of Reading Programs, New York City Board of Education, New York, NY
Director, ETS Atlanta Test Development Staff
Team Member, ETS Computer-Delivered Tests Training Modules
Adjunct Instructor, New York University, New York, NY
Adjunct Instructor, Lehman College CUNY, New York, NY
Reading and Language Arts Teacher in the Waldwick Public School System,
Waldwick, NJ
Page A-55
Sandra Durden, M.A.

Education
Ph.D.
M.A.
B.A.
B.A.
(In Progress) Organizational Leadership

University of the Incarnate Word, San Antonio, TX
2006, Mathematics Education
University of the Incarnate Word, San Antonio, TX
1996, Elementary Education (cum laude)
Saginaw Valley State University, Saginaw, MI
1986, Government (cum laude)
Evangel University, Springfield, MO

1997Lifetime
Teaching Certificate: Early Childhood PKKG, Texas State Board for Educator Certification,
Teaching Certificate: Mathematics Grades 18, Texas State Board for Educator Certification
Teaching Certificate: SelfContained Grades 18, Texas State Board for Educator Certification
Teaching Certificate: Social Studies Composite Grades 18, Texas State Board for Educator
Certification
Teaching Certificate: Elementary K5 All Subjects (K8 All Subjects Self Contained),
Michigan Department of Education, 19962002
Teaching Certificate: Mathematics 68, Michigan Department of Education, 19962002
Teaching Certificate: Social Science 68, Michigan Department of Education, 19962002
Honors and Awards

Teacher Quality Grant Awards, University of the Incarnate Word, 20032006
Present Position
Senior Test Developer, American Institutes for Research (AIR)
Development of mathematics performance test items for the Smarter Balanced Assessment
Consortium and development of mathematics items for state programs and formative assessments.

Develops math items as the Senior Test Developer for the Smarter Balanced Assessment
Consortium (2013Present), Utah State Assessment (2013Present) and the web-based assessment, Learning Point Navigator (2013Present).
Employment History
2013Present
20082013
20032008
19992003
19961999
19911996
Senior Test Developer, AIR, Washington, DC

Director of Research and Development, Pearson
Senior Assessment Specialist, Harcourt Assessment
Mathematics Teacher Grade 8, Mathematics Department Chair, Northside
Independent School District
Elementary Teacher, Edgewood Independent School District
Substitute Teacher, Saginaw Public Schools
National Council of Teachers of Mathematics
American Educational Research Association
Page A-57
Diana Reed, Ed.M.

Content Lead, English Language Arts
Education
Ed.M.
B.A.
2003, Teaching and Curriculum

Harvard University, Cambridge, MA
1997, Double Concentration in English and Computer Science
Brown University, Providence, RI

Professional Teaching License in English Levels 58, 912, Massachusetts, 2003
Honors and Awards

Calderwood Writing Fellow, Calderwood Foundation, 20062007
Grant Recipient, Braitmeyer Foundation, 2000
Present Position
English Language Arts Test Development Specialist, American Institutes for Research (AIR)
Writes, reviews, and revises English language arts (ELA) test items based on their fairness and
content. Resolves team and client feedback on test items at individual meetings and group
review sessions. Helps senior staff recruit, train, and manage internal and external item
writers. Drafts training materials for item writers. Assembles test item pools as directed by
senior staff. Attends client committee meetings in which stakeholders review the fairness and
content of test items.

Develops test items as the ELA Content Lead for the Smarter Balanced Assessment
Consortium (2013Present). ELA Item Developer for the Delaware Comprehensive Assessment
(20122013). Assembled test items as the ELA Grade Lead for statewide assessments including
Minnesota (20112012) and Hawaii (2012).
Employment History
2011Present
20032009
19982002
19971998
English Language Arts Test Development Specialist, AIR, Washington, DC

English Language Arts Teacher, John W. McCormack Middle School
Language Arts and Computer Science Teacher; Technology Director, The
Girls Middle School
Software Engineer, AverStar, Inc.
National Council of Teachers of English (20042009)
International Reading Association (20062008)
Computer Using Educators (19992002)
Sloan, J., Veilleux, J., & Reed, D. (2007, February). Shared genre and author studies:
Opportunities to collaborate to improve student learning. Presented at Americas Choice
National Conference, Atlanta, Georgia.
Page A-59

Education
M.A.T.
B.A.

University of Pittsburgh, Pittsburgh, PA
2003, Liberal Arts
University of Pittsburgh, Pittsburgh, PA

Elementary Education K6 teaching license, Office of the State Superintendent of Education,
Washington, DC, 2008
Reading Endorsement K8, Florida State Department of Education, 2008
ESOL Endorsement K8, Florida State Department of Education, 2007
Elementary Education K6 teaching license, Florida State Department of Education, 2005
Honors and Awards

Maxine Roberts Excellence in Education Award, 2004
Present Position
Collaborates on the development of K3 English language arts (ELA) assessment items for
diagnostic assessments for the Ohio Department of Education. Specific responsibilities include
creating reading passages and other stimuli; writing, managing, and editing test items; and
working with clients during item reviews and bias and sensitivity meetings. Other duties
include providing feedback to item writers and supporting other ELA teams on assessment
development and review.

Successfully develops items for the Ohio Department of Educations diagnostic assessments
(2012Present).
Employment History
2012Present
20102012
20092010
20082009
20072008
20052007
20042005
20032004
English Lanugage Arts Test Development Specialist, AIR, Washington, DC

ESL Instructor, District of Columbia Public Schools, Washington, DC
ESL Facilitator, Changwon Public Schools
Grade 1 Instructor, International School of Krakow
Grade 4 Instructor, Seminole County Public Schools
Grades 1 and 4 Instructor, Bay County Public Schools
Grade 1 Instructor, CharlotteMecklenburg Public Schools
Grade 2 Instructor, Pittsburgh Public Schools
WIDA Consortium
National Autism Association
Central and Eastern European Schools Association
Keller, C. J. (2009, April). Integrating the second language learner into the mainstream classroom.
Presented at the conference of the Central and Eastern European Schools Association,
Bucharest, Romania.
Page A-61
Terry Hill, B.A.

Education
B.A.
1995, Professional English

North Carolina A&T State University, Greensboro, NC
Honors and Awards

Golden Key Honors Member, Golden Key International Honors Society, 1992
Present Position
Responsibilities include directing the development of various writing and modified projects.
Selects reading passages and other stimuli, managing and editing items for various clients,
using AIRs Item Bank System. Work with clients during item reviews, bias and sensitivity
meetings, rangefinding, and standard-setting meetings. Provides feedback to item writers and
in-house development staff and creates test forms.

Successfully develops English language arts (ELA) items for projects including the
MinnesotaModified (2013Present), Utah Writing Assessment (2013Present), and Hawaii
State Assessment (20122013).

As ELA Content Specialist at Measurement Incorporated (20102012), responsibilities
included working with item writers for purposes of developing items for the Item Bank project,
acting as the final reviewer of content of items, invoicing item writers, and writing passages
and items for the Item Bank project. As the Editorial Assistant, responsible for leading projects
for the Council of Chief State School Officers (CCSSO)/English Language Development
Assessment (ELDA), the New Jersey Special Review Assessment (SRA), the New Jersey
Biology performance assessment tasks (PATs), and the Michigan Educational Assessment
Program (MEAP) for reading (20032010).
Employment History
2012Present
20102012
20032010
20012003
19992001
19951999

ELA Content Specialist, Measurement Incorporated, Durham, NC
Editorial Assistant, Measurement Incorporated, Durham, NC
News Desk Editor, Total Sports/Quokka Sports, Raleigh, NC
Online Editor, Terra-Lycos
Technical Correspondent, Underwriters Laboratories, Washington, DC
Page A-63

Education
M.A.E.
B.A.
2006, Literature
DePaul University, Chicago, IL
2004, Secondary Education of English

Teaching Certificate, Secondary and Middle School
Present Position
Assists in the development of English language arts (ELA) assessment items and stimuli.
Responsibilities include writing, editing, and reviewing test items, providing feedback to item
writers, resolving client feedback issues, and meeting with clients for test item reviews.

Writes, edits, and reviews ELA items as the Test Development Specialist for Utah State
Assessments (2013Present) and Hawaii State Assessments and End-of-Course (EOC)
Assessments (2012Present). Successfully developed ELA items for Learning Point Navigator
(2013), Minnesota State Assessments (20112013), and Ohio State Assessments (2011).

Designed study plans, scope and sequence, and instruction as Pearsons Editor for onTarget
ELA Database Phases 1 and 2. Edited and wrote manuscript and metadata for all grade-levels.
Worked with CCST-based instruction for literature, informational text, speaking and listening,
and language (20102011). Editor, Texas ELA adoption for 2011 (20092010) and California
ELA adoption for 2010 (20072009). Responsibilities included designing ELA curriculum that
met all state standards; managing and advising a team of freelance writers; creating guidelines;
editing manuscripts; and ensuring lessons were aligned with the states standards.
Employment History
2011Present
20062011
20042005
20032005
English Language Arts Test Development Specialist, AIR, Chicago, IL

Editor, Pearson Education, Chicago, IL
Teacher, Cicero Public Schools, Chicago, IL
Writing Tutor, University of Illinois at Chicago and DePaul University,
Chicago, IL
Page A-65
Allen Chong, M.Ed.

Education
M.Ed.
B.A.
2010, Education and Human Development

George Washington University, Washington, DC
2008, Political Science
Vassar College, Poughkeepsie, NY
Present Position
English Language Arts Test Development Associate, American Institutes for Research (AIR)
Assists in the development of K12 English language arts (ELA) assessment items and stimuli.
Responsibilities include writing and selecting passages and other stimuli; writing, editing, and
reviewing test items; working with clients during item reviews and bias and sensitivity meetings; and responding to client feedback.

Writes, edits, and reviews ELA items as the Grade Level Lead for the Smarter Balanced
Assessment Consortium (2013Present). Successfully assisted in the test development of ELA
items for Oregons English Language Proficiency Assessment (2013), as well as Utahs state
assessments (2013) and Minnesotas state assessments (2013).
Employment History
2013Present
20122013
20112012
20102011
20082010
English Language Arts Test Development Associate, AIR, Washington, DC

Test Development Specialist, Center for Applied Linguistics, Washington, DC
Research Assistant, Center for Applied Linguistics, Washington, DC
Administrative Assistant, Early Head Start National Resource Center,
Washington, DC
Teacher, District of Columbia Public Schools, Washington, DC
Technical Reports
Yanosky, T., Minji L., Chong, A., Cameron, C., MacGregor, D., & Kenyon, D. (2012). A bridge
study between MontCAS ELP and ACCESS for ELLs(r) for the state of Montana.
Washington, DC: Center for Applied Linguistics.
Yanosky, T., Chong, A., Louguit, M., Olson, E., Choi, Y., MacGregor, D., Yen, S.J., Cameron,
C., & Kenyon, D. (2012). Annual technical report for ACCESS for ELLs(r) English language proficiency test, series 202, 20102011 administration (Vol. 1). Washington, DC:
Center for Applied Linguistics.
Yanosky, T., Chong, A., Olson, E., Zhang, Y., MacGregor, D., & Kenyon, D. (2011). A bridge
study between LAS Links(r) and ACCESS for ELLs(r) for the state of Missouri.
Washington, DC: Center for Applied Linguistics.
Chong, A., & Laya, G. (2013, February). Midiendo xito: El desarrollo y la aplicacin de
PODER. Presented at meeting of National Association of Bilingual Educators (NABE), Lake
Buena Vista, FL.
Page A-67

Education
Ed.D.
Candidate
Bilingual Special Education

The George Washington University, Washington, DC
Ed.M.
2009, Language and Literacy

Harvard Graduate School of Education, Cambridge, MA
B.A.
2006, Linguistics, Psychology (University Honors Program)

Present Position
Assists in the development of K12 English language arts (ELA) assessment items and stimuli.
Responsible for selecting reading passages and other stimuli; writing, editing, and reviewing
test items; working with clients during item reviews and bias and sensitivity meetings; and
responding to client feedback.

Assists in the development of K12 ELA assessment items and stimuli as the Test
Development Associate for the Smarter Balanced Assessment Consortium (2012Present),
Hawaii State Assessment (20122013), and the Utah State Assessment (2013).
Employment History
2012Present
20112012
20092011
20092009
20082009
20062007

Visiting Assistant Professor, The George Washington University, Washington, DC
Project Coordinator, The George Washington University, Washington, DC
Research Assistant, The George Washington University, Washington, DC
Graduate Research Assistant, Harvard Graduate School of Education,
Cambridge, MA
Paraeducator, Montgomery County Public Schools, College Park, MD
Council for Exceptional Children
Mulhern, M. (2011, February). Vocabulary instruction for ELLs. Invited guest lecturer for the
graduate education course Teaching Second Language Reading and Writing, Prince
Georges County Public Schools, Riverdale, MD.
Mulhern, M., Llagas, C., & Mazur, A. (2012, April). Teacher perceptions about their English
language learners. Paper presented at the Council for Exceptional Children Annual
Conference, Denver, CO.
Llagas, C., Mulhern, M., & Mazur, A. (2012, April). Teacher Perceptions and Practices of Text
Adaptation. Paper presented at the AERA Annual Meeting, Vancouver, BC.
Page A-69

Education
B.A.
2010, English Language and Literature, Womens Studies

Present Position
English Language Arts Test Development Assistant, American Institutes for Research (AIR)
Creates and enters English language arts (ELA) passages, items, and classification information
in test creation software; monitors item review schedules across testing programs; and checks
item flow in test creation software. Additionally, prepares items for internal reviews and prepares review logs, item booklets, and ancillary materials for client committee meetings.

Serves as the ELA content lead for the Delaware Department of Education contract where she
leads the development of ELA items and passages across grades 212 for Delaware
Compressive Assessment Standards (2011Present). Additionally, serves as Grade Level Lead
for grades 4 and 5 for the Smarter Balanced Assessment Consortium listening development
(2012Present); serves as Grade Level Lead for grades 3 and 4 of internal Common Core State
Standards (CCSS) development (2012Present); and contributes to Hawaii End-of-Course
(EOC) development for Expository Writing (2011Present).
Employment History
2011Present
20092011
20072009
English Language Arts Test Development Assistant, AIR, Washington, DC

Legal Clerk, U.S. Department of Justice, Aviation and Admiralty Litigation
Human Resources Associate, Flippo Construction Company, Inc.
Page A-71
Mark Lovely, M.F.A.

Education
M.F.A.
B.A.
1997, Fiction Writing

University of Iowa, Iowa City, IA
1991, Humanities (summa cum laude)
Bradford College, Bradford, England
Honors and Awards

Teaching-Writing Fellow, University of Iowa, 19961997
Present Position
Responsibilities include writing, editing, and reviewing items, providing feedback to item
writers, and working closely with clients to ensure quality assessments.

As a Test Development Specialist, responsible for writing, editing, and reviewing ELA items for
the Smarter Balanced Assessment Consortium (2013Present). Developed items for the Ohio
Achievement Assessments (20122013), Minnesota Comprehensive and Modified Assessments
(20112012), Hawaii State Assessments (20112012), and Delaware Comprehensive
Assessment System (20112012).

Worked as the ELA Specialist, Curriculum and Assessment, at Measured Progress
(20092011). Responsibilities included development of ELA materials in accordance with
state and district frameworks and guidelines. Development work typically included locating
grade-appropriate reading passages, producing items that measure each states frameworks,
and meeting with advisory committees and departments of education. In addition, responsible for writing annotations for state released items, providing detailed explanations of
score-point exemplars. Also, worked as the Quality Assurance Coordinator at Measured
Progress (20052008). Evaluated standardized tests in the content areas of reading and
writing, monitored scorer performance as a Senior Reader, prepared and conducted scorer
training, monitored progress of scoring project, and evaluated scorer and Senior Reader
performance as a Quality Assurance Coordinator. Assembled anchor and training packs in
accordance with project standards.
Employment History
2011Present
20092011
20052008
20022005
19982005
20022004
20002004

ELA Specialist, Curriculum and Assessment, Measured Progress, Dover, NH
Quality Assurance Coordinator, Measured Progress, Dover, NH
Instructor of English, Merrimack College, North Andover, MA
Reading Instructor, Northern Essex Community College, Haverhill, MA
Writing Instructor, Northern Essex Community College, Haverhill, MA
Reading Center Staff Member, Northern Essex Community College, Haverhill, MA
Page A-73
Sean Redmond, M.A.

Education
M.A.
B.A.
2004, English Literature

San Francisco State University, San Francisco, CA
1999, English and Secondary Education (cum laude)
University of Michigan, Ann Arbor, MI

Michigan Secondary Teaching Certificate in English and Social Studies, 1999
Present Position
Responsibilities include writing, editing, and reviewing online and paper-and-pencil test items;
training and providing feedback to item writers; resolving client feedback issues; and meeting
with clients for item and passage reviews

Writes, edits, and reviews online and paper-and-pencil test items; trains and provides feedback
to item writers; resolves client feedback issues; and conducts meetings with clients for item
and passage reviews as the Test Development Specialist for the Minnesota Comprehensive
Assessments (2011Present).

The Lead Content Editor, Educational Product Development at Riverside Publishing
(20092011). Researched state standards for multiple subjects to guide development of the
next-generation Iowa Tests. Cognitive Abilities Test (CogAT) Content and Verbal Lead, guiding
development and production activities, including building test forms for all verbal test items
through national item tryout, standardization, and operational forms while overseeing all
CogAT content and training team members. As the Editor of Assessment and Standards
Development Services at WestEd (20052009), led ELA test development for several key
states, writing and editing ELA test items and passages, benchmarking and ensuring alignment
to state standards, building forms (Kentucky), and checking for fluency, readability, grammar,
and content.
Employment History
2011Present
20092011
20052009
20042005
2003
2003
20022004
20012002
20002001
English Language Arts Test Development Specialist, AIR, Chicago, IL

Lead Content Editor, Riverside Publishing, Rolling Meadows, IL
ELA Editor, WestEd, Chicago, IL
Editor, Ohana Foundation
Teaching Assistant, San Francisco State University
SAT Verbal Instructor, Monicas Learning Center
Tutor, San Francisco State University Educational Opportunity Program
Teacher, Lincoln Elementary School
Instructional Assistant, Sunnyvale Middle School
Page A-75

Education
M.A.T.
B.A.
2007, Secondary English Education

Piedmont College, Demorest, GA
2004, English
The University of Georgia, Athens, GA

Grades 612 English Teacher Certification, Office of the State of Georgia, Superintendent of
Education, 2009
Grades 612 English Teacher Certification, Georgia Department of Education, 2007
Present Position
Develops, edits, and revises English language arts (ELA) test items for various ELA projects;
develops curriculum and lesson plans for online learning system. Completed the AIR formbuilding training module.

Develops, edits, and reviews content for various ELA projects as the Test Development
Associate. Involved in editing a wide range of middle and high school level content for the
Smarter Balanced Assessment Consortium (2012Present). Fluent in Common Core State
Standards (CCSS) and was integral in the development of the passage and item specifications
that are used to develop the assessment aligned to Utahs Core Standards for the Student
Assessment of Growth and Excellence (SAGE) for ELA (2013Present) and the AIR Common
Core Internal project (2013Present). Has additional experience developing ELA test items for
Delaware State Assessment (2012) and the Hawaii State Assessment (2012).
Employment History
2012Present
20112012
20092011

Literacy Teacher, E. L. Haynes Public Charter School, Washington, DC
English Teacher, Burroughs Education Campus, District of Columbia Public
Schools, Washington, DC
Page A-77
Jim McCann, M.S.

Education
M.S.
B.S.
2010, Educational Psychology (concentration in Assessment, Testing, and Evaluation)

George Mason University, Fairfax, VA
2005, Elementary Education (concentration in Mathematics, minor in Sociology)
(cum laude)
University of Maine, Orono, ME
Present Position
Mathematics Test Development Specialist, American Institutes for Research (AIR)
Supervises content staff during the item development, form construction, and pool management phases of a variety of projects. Item development includes writing and reviewing items
both individually and in a group setting. Form construction involves selecting a set number of
approved items and building a test that is statistically and contextually sound. Pool management includes the analysis of item pools to determine areas of strength and weakness to guide
efficient future development. Works directly with clients and supervises content staff as they
facilitate meetings with clients and classroom teachers.

Oversees the test development for all content areas for the Alternate Assessment for the
state of Delaware as well as the mathematics items for Delawares statewide assessment
(2010Present). Assisted in the oversight of test development for all content and areas for
the alternate assessment for New Mexico, South Carolina Alternate Assessment and the
modified assessments for the states of Ohio, Oregon, and Minnesota (20082010).
Successfully managed the development of items as the Math Grade Level Lead for
statewide assessments in Hawaii and Ohio (20062008).
Employment History
2010Present
20082010
20062008
20052006
Mathematics Test Development Specialist, AIR, Washington, DC

Test Development Associate, AIR, Washington, DC
Test Development Assistant, AIR, Washington, DC
Teacher, Stafford County, Virginia
Page A-79

Education
M.S.
B.A.
Expected 2014, Pure Mathematics

DePaul University, Chicago, IL
2008, Applied Mathematics
Northeastern Illinois University, Chicago, IL
Present Position
Mathematics Test Developer, American Institutes for Research (AIR)
Writes, reviews, edits, and revises mathematics assessment items based on content and
fairness for a range of assessment projects.

Writes, reviews, edits, and revises mathematics assessment items as the Mathematics Test
Developer for the Smarter Balance Assessment Consortium (2013Present) and the Utah
Statewide Computer-Adaptive Assessment (2013Present).

Worked as the Content Developer on MyMathLab and MathXL for Pearson (20082013)
where responsibilities included creating algorithmic math items for developmental mathematics through third-year calculus, which were used in MathXL and MyMathLab in online
homeworks, tutorials, and assessments, establishing new and efficient ways to develop
algorithmically challenging items, and creating templates that could be used in the future
for similar items.
Employment History
2013Present
2013Present
20082013
2005
20042006
Mathematics Test Developer, AIR, Chicago, IL

Mathematics Teaching Assistant, DePaul University, Chicago, IL
MyMathLab/MathXL Content Developer, Pearson Education, Chicago, IL
Mathematics Student Researcher, Louisiana State University, Baton Rouge, LA
Mathematics Teaching Assistant, New College of Florida, Sarasota, FL
Page A-81
Maureen Font, M.Ed.

Education
M.Ed.
B.S.
2011, Measurement, Evaluation, Statistics, and Assessment

2007, Elementary Education and Math
University of Illinois at Urbana-Champaign, Champaign, IL

Type 03 Teaching Certificate, State of Illinois, 2007
Present Position
Mathematics Test Development Associate, American Institutes for Research (AIR)
Writes, reviews, and revises mathematics assessment items based on content and fairness
for a range of assessment projects.

Serves as AIRs Mathematics Content Lead for the Smarter Balanced Assessment
Consortium, for grades 311, for the upcoming field test (2012Present). Responsible for
delivering approximately 1,000 performance task items for Smarter Balanced and managing
the internal development process, as well as coordinating many development tasks with
other vendors and the client. As Mathematics Content Lead for the MinnesotaModified
Assessment (20112013), working on grades 511, responsible for delivering approximately
500 items each year.
Employment History
2011Present
20072011
Mathematics Test Development Associate, AIR, Chicago, IL

Mathematics Educator, Cook County, IL
Chicago Evaluation Association
Page A-83
George Ligon, M.Ed.

Education
M.Ed.
B.S.
2013, Educational Leadership, Math Specialist Leader (K8)

2006, Physical Education, Exercise Science
University of South Florida, Tampa, FL

Professional Educators Certificate: Mathematics (Grades 59), Physical Education (Grades
K12), Exceptional Student Education (Grades K12), Florida Department of Education,
20092014
Collegiate Professional License: Middle Education 68 Mathematics, Health and Physical
Education PreK12, Special Education General Curriculum K12, Virginia Department of
Education, 20092014
Present Position
Works with the Mathematics Content Team on Test Development projects across a variety of
assessment programs. Writes, reviews, and revises test items based on fairness and content.
Resolves team and client feedback on test items at individual meetings and group review
sessions.

Assists in the development of math items for statewide assessments, including Smarter
Balanced Assessment Consortium (2013Present), Utah (2013), Ohio (2013), Minnesota
(2013), Hawaii (2013), and Delaware (2013).
Employment History
2013Present
20112013
20092011
20062009
Mathematics Test Development Specialist, AIR, Washington, DC

Common Core Mathematics Corps, Mathematics, and Special Education
Teacher, District of Columbia Public Schools, Washington, DC
Mathematics Teacher/Wrestling Coach, Prince William County Public Schools,
Woodbridge, VA
Mathematics, Physical Education, and Exceptional Student Education
Teacher/Football, Tennis, and Wrestling Coach, School District of
Hillsborough County, Tampa, FL
Kappa Alpha Psi Fraternity, Inc.
National Education Association
Virginia Council of Teachers of Mathematics
Page A-85

Education
M.S.
B.A.
2005, Educational Administration

Capella University, Minneapolis, MN
2001, Elementary Education (Mathematics)
Carthage College, Kenosha, WI

National Board Certified Teacher Early Adolescent Mathematics, National Board for
Professional Teaching Standards, 2007
Master Level Certified Mathematics Teacher K12; Illinois State Board of Education, 2007
Administration Certification, Illinois State Board of Education, 2005
Elementary Certified Teacher, Illinois State Board of Education, 2002
Present Position
Writes, reviews, and develops test items for statewide assessments in mathematics of all grades
levels, including 800 previously written items that were realigned to the Common Core State
Standards (CCSS) for the Utah project. Has led six content and fairness committee reviews for
various projects.

Develops math items as the Test Development Specialist for the Smarter Balanced Assessment
Consortium (2012Present). Serves as a Grade Level Lead for both the Minnesota State
Assessment (2013Present) and the Utah State Assessment (2013Present). Lead several
successful development cycles, producing 300 items for Minnesota and 180 items for Utah in
the latest cycles. Additionally, has developed math items for the Hawaii State Assessment on
an as-needed basis for the 20122013 school year.
Employment History
2012Present
20102012
20022010
Mathematics Test Development Specialist, AIR, Chicago, IL

Gifted Mathematics Teacher, Lower School, Science & Arts Academy
Educator, Lake Villa Consolidated School District
National Council of Teaching Mathematics
Page A-87

Education
B.S.
2005, Education
Slippery Rock University, Slippery Rock, PA

Graduate Certificate, Business Fundamentals, University of Baltimore, 2012
Standard Professional II Teaching Certification, Maryland State Board of Education, 2011
Standard Professional I Teaching Certification, Maryland State Board of Education, 2006
Instructional I Certification, Pennsylvania State Board of Education, 2005
Present Position
As a Test Development Associate, responsible for writing, reviewing, and editing math items
for various AIR projects for all grade levels. Serves as a Grade Level Lead and manages the
development of items with a focus on algebra, functions, expressions, and equations.
Responsible for leading client-facing committee meetings and resolving client feedback.

Manages the development of items as the Grade Level Lead for state assessment projects
for both Utah (2013Present) and Minnesota (2013Present). Managed the development of
more than 500 items for Utah for grades 68. Participates in the item development process
by writing items, participating in group item reviews, editing items, and performing content
reviews for the as a Test Developer for the Hawaii State Assessment (2012Present) and
the Smarter Balanced Assessment Consortium (2012Present).
Employment History
2012Present
20092012
20062012
Mathematics Test Development Associate, AIR, Washington, DC

Eighth Grade Team Leader, Baltimore County Public Schools, Arbutus, MD
Mathematics Teacher, Baltimore County Public Schools, Arbutus, MD
Page A-89

Education
M.A.
B.A.
2010, Curriculum and Instruction with ESL (Education)

Concordia University, Portland, OR
2004, Secondary Education
Concordia University, Portland, OR
Present Position
Develops, reviews, and edits items for various statewide assessments. Works closely with
national Common Core State Standards (CCSS) to write a variety of test items. Conducts
content reviews on a variety of items for different projects.

Responsible for developing mathematics items for grades 312 as the Item Writer and
Content Review for assessments, including the Utah State Assessment (2013Present),
Ohio Graduation Test (2013Present), Minnesota State Assessment (2012Present), and
Smarter Balance Assessment Consortium (2012Present). Responsible for directing creation and development for online diagnostic and achievement End-of-Course (EOC) exams
for Algebra I as the Grade Level Lead for Hawaii State Assessment (2012Present).

Developed middle-school and upper-level assessments for various state and local projects; as a
Test Development Specialist II, worked on Riverside Publishings CCSS research-based interim
shelf projects in Detroit, MI; Prince Georges County, MD; and Oklahoma City, OK. In addition
to building assessments, worked on item reads written from a variety of vendors and item
writers, edited a variety of items for an item bank, and worked on various state alignments for
both existing and new items (20102012).
Employment History
2012Present
20102012
20092010
20082009
20042008
Mathematics Test Development Specialist, AIR, Chicago, IL

Math Test Development Specialist, Riverside Publishing, Rolling Meadows, IL
ACT Prep/Consumer Education Teacher, Kenwood Academy High School,
Chicago, IL
Secondary Mathematics Teacher, East Leyden High School, Schiller Park, IL
Secondary Mathematics and English teacher, William Howard Taft High
School, Chicago, IL
Page A-91

Education
B.S.
2007, Mathematics
University of Illinois at Urbana-Champaign, Champaign, IL
Present Position
Mathematics Test Developer, American Institutes for Research (AIR)
Writes, reviews, and revises test items based on content and fairness for a range of
mathematics assessments.

As the Mathematics Test Developer, responsibilities include creating, reviewing, and revising
test items based on standard alignment and content for the Ohio Graduation Test
(2013Present) and the Utah State Office of Education (2013Present).

Worked as the Content Developer on MathXL and MyMathLab for Pearson (20082013),
creating algorithmic math items for developmental mathematics through third-year
calculus, which were used in MathXL and MyMathLab in online homeworks, tutorials, and
assessments. Established new and efficient ways to develop algorithmically challenging
items and created templates that could be used in the future for similar items. As part of a
select group of developers, tested new functionality on the assessment and tutorial sites
before public release. As part of a select group of developers, mapped learning objectives in
textbooks, which were used to create adaptive homeworks and assessments for students.
Employment History
2013Present
20082013
Mathematics Test Developer, AIR, Washington, DC

Content Developer, Pearson, Washington, DC
Page A-93

Education
M.A.
Candidate
Expected 2015, Public Policy

George Mason University, Arlington, VA
B.A.
2000, Mathematics (minor in History, with a concentration in U.S. History)

Present Position
Develops mathematics test items for a variety of statewide assessments. Writes, edits, and
reviews math test items; resolves team and client feedback on test items; leads client-facing
committee meetings; and participates in developing machine-scored constructive-response
items. Codes technology-enhanced items for different projects at AIR.

Successfully manages the development of math items for grades 7, 8, and 10 as the Grade
Level Lead for the Hawaii State Assessment (2011Present). Participates in the item
development process as the Test Developer for statewide assessments, including Hawaii
State Assessment (2011Present), Minnesota Assessment System, Delaware Comprehensive
Assessment System, and Ohio K8 Assessment (2011).
Employment History
2011Present
20082011
20052008
2004
2003
20002002
Mathematics Test Development Associate, AIR, Washington, DC

Algebra I and Geometry Teacher, Patrick Henry High School, Roanoke, VA
Administrator and Logistician, Dignitas International, Zomba, Malawi
International Student Seminar Coordinator, World University Service of
Canada, Lilongwe, Malawi
Seminar Coordinator, Ministry of Health, Nutrition Unit, Lilongwe, Malawi
Secondary School Mathematics Teacher, U.S. Peace Corps, Ntchisi, Malawi
Page A-95
PETER TRESSEL
Director, State Assessment Programs
PROFESSIONAL EXPERIENCE
2010Present
Director, State Assessment Programs, Data Recognition Corporation,

Maple Grove, MN
20072010
20022007
Manager, State Assessment Programs, Data Recognition Corporation,

Maple Grove, MN
Senior Project Manager, Data Recognition Corporation, Maple Grove, MN
20012002
Project Manager, Data Recognition Corporation, Maple Grove, MN
19992001
Project Coordinator, Data Recognition Corporation, Maple Grove, MN
19951999
Communications Specialist, Experior Assessments, St. Paul, MN
19931995
Assistant Editor, Intertec Publishing, Malibu, CA
PROJECT-SPECIFIC EXPERIENCE
2011Present
2011Present
Minnesota Graduation-Required Assessments for Diploma
2011Present
Minnesota Test of Academic Skills
2007Present
Ohio Alternate Assessment for Students with Disabilities
2007Present
20072008
Ohio Ninth Grade Proficiency Tests
19992005
South Carolina Basic Skills Assessment Program
20062007
South Carolina English Language Development Assessment
19992007
South Carolina Palmetto Achievement Challenge Tests
EDUCATION AND CERTIFICATIONS

M.B.C.
Business Communications, University of St. Thomas, St. Paul, MN
B.A.
Journalism, University of Minnesota, Minneapolis, MN

Project Management Professional (PMP), Project Management Institute,
Bloomington, MN
PMP
Page A-97
SHERRI WOOLF
Vice President of Performance Assessment Services
2006Present
19931997
Vice President of Performance Assessment Services, Data Recognition Corporation,

Cincinnati, OH
Senior Director of Performance Assessment Services, Data Recognition Corporation,
Cincinnati, OH
Director of Performance Assessment Services, Data Recognition Corporation,
Cincinnati, OH
Educational Research and Evaluation, North Carolina State University, Raleigh, NC
19921993
Scoring Specialist, National Computer Systems, Iowa City, IA
20042006
19972004
2009Present
19992001
Arkansas Comprehensive Testing Assessment & Accountability: Benchmark

Examinations
19992005
Arkansas Comprehensive Testing Assessment & Accountability: End of Course

Examinations
20002002
Kentucky Commonwealth Accountability Testing System Program
2005Present
19992005
Louisiana Educational Assessment Program for the 21st Century
2011Present
Nebraska State Accountability Writing Assessment
2007Present
Ohio Graduation Test
20052010
Oklahoma School Testing Program: Core Curriculum Tests
2010Present
Pennsylvania Keystone Exams
1998Present
2007Present
South Carolina English Language Development Assessment
19982009
2009Present
South Carolina Palmetto Assessment of State Standards

M.S.
Human Resource Development, Xavier University, Cincinnati, OH
B.S.
Elementary Education, Miami University, Oxford, OH
Page A-99
WENDY MARIK
Senior Project Manager, Performance Assessment Services
2010Present
Senior Project Manager, Performance Assessment Services, Data Recognition

Corporation, Woodbury, MN
20062010
Project Manager, Performance Assessment Services, Data Recognition

20002005
Scoring Director/Team Leader, Performance Assessment Services, Data

Recognition Corporation, Woodbury, MN
19982000
Reader, Performance Assessment Services, Data Recognition Corporation,

Minnetonka, MN
19991996
Classroom Teacher
19912004
Private Tutor
20042005
Alaska Comprehensive System of School Assessment: High School Graduation

Qualifying Examination and Standards Based Assessment
19982001
Kentucky Commonwealth Accountability Testing System
2001
20032005
Minnesota Comprehensive Assessment System
2001
North Carolina End-of-Grade Testing Program
2000Present
2012Present
Smarter Balanced Assessment Consortium Pilot Item Development
19982005
2008Present
Washington Comprehensive Assessment Program: High School Proficiency

Exams and Measurements of Student Progress

M.Ed.
Education, Xavier University, Cincinnati, OH
B.A.
Computer Science, DePauw University, Greencastle, IN
Page A-101
PAM AHRENS
Content Specialist, Performance Assessment Services
2013Present
Content Specialist, Performance Assessment Services, Data Recognition

20082013
Scoring Director, Performance Assessment Services, Data Recognition

20052007
Team Leader, Performance Assessment Services, Data Recognition Corporation,

Woodbury, MN
Reader, Performance Assessment Services, Data Recognition Corporation,
Woodbury, MN
Family Information Coordinator, Childrens Hospitals and Clinics Integrative
Medicine Program, Minneapolis/St. Paul, MN
20032004
20002002
19982000
19791997
Library Manager, Fairview University Medical Center, Minneapolis, MN

Library Systems & Services Manager, Allina Hospitals and Clinics, St.
Paul/Minneapolis, MN
20042005;
2010; 2012
Alaska Comprehensive System of Student Assessment: High School Graduation

20112012
20112012
20102012
Ohio Alternate Assessment for Students with Disabilities
20032010;
2012Present
20032008
2011Present
Washington Comprehensive Assessment Program: End of Course Assessments
2009Present
Washington Comprehensive Assessment Program: High School Proficiency

Exams and Measurements of Student Progress

M.A.
Library Science, University of Minnesota, Minneapolis, MN
B.S.
Education, Mankato State University, Mankato, MN
Page A-103
COSKUN BAYSOY
Mathematics Content Specialist, Performance Assessment Services
2011Present
Mathematics Content Specialist, Performance Assessment Services, Data

Recognition Corporation, Plymouth, MN
20092011
Scoring Director, Performance Assessment Services, Data Recognition Corporation,

Plymouth, MN
Mathematics Item Writer, Test Development, Data Recognition Corporation,
Maple Grove, MN
Image Field Definition Specialist, Education Information Systems and Imaging
Information Systems, Data Recognition Corporation, Maple Grove, MN
20062011
20092011
20012009
Project Team Leader/Scorer, Performance Assessment Services, Data Recognition

Corporation, Minnetonka, MN
20002011
AdvisorTechnology Integration and Business Development, Sovereign Grounds,

Inc., Minneapolis, MN
19932000
Research Assistant, Department of Electrical and Computer Engineering, University

of Minnesota, Minneapolis, MN
20022005
20042011
20062011
20012010
20102011
20012009
2008Present
2010Present
Arkansas Comprehensive Testing & Assessment Program:, End-of-Course Examinations

Idaho Standards Achievement Test
Washington Comprehensive Assessment Program: High School Proficiency Exams
and Measurements of Student Progress
Washington Comprehensive Assessment Program: End of Course Assessments

Ph.D., ABD
Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN
M.S.E.E.
Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN
B.S.E.E.
Electronics and Communications Engineering, Istanbul Technical University,

Istanbul, Turkey
Page A-105
JAMES BELL
Senior English Language Arts Content Specialist
2013Present
Senior English Language Arts Content Specialist, Data Recognition Corporation,

Maple Grove, MN
2010Present
Chair, Department of English, Foreign Language and Humanities, Northwestern

Oklahoma State University, Alva, OK
19992010
Director of Institutional Assessment, College of the Ozarks, Point Lookout, MO
1992Present
Freelance Writer, Editor, and Committee Facilitator
2012Present
2012Present
Pennsylvania System of School Assessments
COMMON CORE STATE STANDARDS (CCSS) EXPERIENCE

Developed CCSS-aligned items, passages, and scoring rubrics for LA in English Language
Arts (Grades 38)
Developing CCSS-aligned items for the Smarter Balanced Assessment Consortium
Participated in discussions of PARCC-related issues as a faculty representative to the Course
Equivalency Project Discipline Committee meetings sponsored by the OK State Regents for
Higher Education, and as a university representative to meetings sponsored by the Oklahoma
Association of Colleges for Teacher Education
Ph. D.
English, Texas Tech University, Lubbock, TX
M.A.
English, Texas Womans University, Denton, TX
B.S.
Secondary Education: English and Psychology Lamar University,

Beaumont, TX
Page A-107
ANNE KIRPES
Reading Test Development Director
2002Present
Reading Test Development Director, DRC, Maple Grove, MN
19982002
Sr. Test Development Specialist, The Riverside Publishing Company, Itasca, IL
19891998
Teacher
PROJECT-SPECIFIC EXPERIENCE (current)
2005Present
Alabama High School Graduation Exam
2009Present
2004Present
2012Present
Florida Student Standards Tutorial Project Committee of Experts
2006Present
2012Present
2011Present
Michigan K12 Item Development
2009Present
Nebraska State Accountability
2009Present
Pennsylvania Classroom Diagnostic Tools
2009Present
2002Present
2009Present
Pennsylvania Voluntary Model Curriculum
2008Present
Washington Comprehensive Assessment Program
Reviewed proposed AK alignment to the Common Core and provided feedback
Reviewed AL reading alignment to the Common Core and provided feedback
Develop PA assessment anchors and eligible content and learning progressions aligned to the
Common Core and developed new reading passages, items, and scoring guidelines aligned to
these new PA standards
Developed CCSS-aligned items for the PA CDT and Keystone Exams
Aligned existing LA assessments to the Common Core and oversaw the development of new
reading passages, items, and scoring rubrics for the inclusion in the transitional ELA
assessments
Oversees item development for the Smarter Balanced Assessment Consortium
Ed.M.
B.A.
Education, concentration in Teaching and Learning, Harvard University

Graduate School of Education, Cambridge, MA
Elementary Education, Reading/Language Arts, University of Northern Iowa,
Cedar Falls, IA
State of IllinoisLanguage Arts, Grades K9, Elementary, Grades K6
Page A-109
JOHN SELISKY
Director, Test Development for Mathematics
2012Present
Director, Test Development for Mathematics, DRC, Plymouth, MN
20092012
Senior Project Lead, DRC, Plymouth, MN
20082009
Student Assessment Specialist, Chicago Public Schools, Chicago, IL
20062008
Publishing Project Manager, CTB McGraw-Hill, Burr Ridge, IL
20032006
Mathematics Assessment Specialist, Educational Testing Service, Princeton, NJ
20022003
Head of Special Project Development, Educational Testing Service, Princeton, NJ
19992002
Senior Mathematics Consultant, Riverside Publishing, Itasca, IL
19961999
Mathematics and Science TD Specialist, Riverside Publishing, Itasca, IL
19711996
Teacher and Assistant Professor of Education and Mathematics
2012Present
2010Present
2012Present
Michigan K-12 Item Development
2012Present
2011Present
Nebraska State Accountability
19992002
Oklahoma School Testing Program
2009Present
Pennsylvania Voluntary Model Curriculum
2009Present
Conducted CCSS alignment studies for PA Assessment Anchors and Eligible Content
Developed sample items for PA Keystone Examination
Developed and wrote 114 lesson assessments for the PA Voluntary Model Curriculum
Conducted alignment studies for mathematics standards for the LA LEAP
Reviewed and evaluated mathematics items for the MEAP, MI Access, and MME
Conducted item writing training activities for MI educators
Supervises mathematics item development for Smarter Balanced Assessment Consortium
Mathematics Education Teachers College, Columbia University, New York, NY
Ed.D.
M.A.
Instructional Technology Teachers College, Columbia University, New York, NY
M.A.
Secondary Education, Lehigh University, Bethlehem, PA
B.S.
Earth and Planetary Sciences and Mathematics, University of Pittsburgh,

Pittsburgh, PA
PMP
Project Management Professional, Project Management Institute
Page A-111
CHRISTOPHER K. PETERSON
Senior Mathematics Test Development Specialist
2010Present
Senior Mathematics Test Development Specialist, DRC, Maple Grove, MN
20052010
Mathematics Teacher, South View Middle School, Edina, MN
20032005
Special Education Paraprofessional, Valley View Middle School, Edina, MN
19992001
Research Analyst, DRC, Maple Grove, MN
Quality Assurance Analyst, DRC, Minnetonka, MN
19981999
2011Present
2011Present
19992001
2011Present
Michigan K-12 Item Development
2010Present
2010Present
2010Present
20122013
Smarter Balanced Assessment Consortium Pilot Test Item Development
2013Present
Smarter Balanced Assessment Consortium Field Test Item Development
South
Carolina Palmetto Achievement Challenge Tests
19982001
Led the development of elementary, middle, and high school items for the 2013 SBAC pilot
test and the 2014 SBAC field test
Led the development of elementary items for DRC-facilitated SBAC cognitive labs
Worked on the alignment review of the existing LA assessments to the Common Core in LAs
effort to transition to the CCSS for mathematics and the implementation of PARCC
assessments
Worked on the development of new mathematics items and scoring rubrics for the inclusion in
the transitional LA Mathematics assessments
Worked with committees of PA educators to develop PA assessment anchors and eligible
content and learning progressions aligned to the Common Core
Responsible for the development of new mathematics items and scoring guidelines aligned to
the PA Core Standards
M.Ed.
Teaching, University of St. Thomas, Graduate School of Education,
Minneapolis, MN
B.S.
Mathematics, Wheaton College, Wheaton, IL
State of MinnesotaSecondary Mathematics, Grades 512
Page A-113
CHRISTIE PLACKNER
Manager, Psychometric Quality
2010Present
Manager, Psychometric Quality, Data Recognition Corporation,

Maple Grove, MN
20082010
Psychometrician, Data Recognition Corporation, Maple Grove, MN
20052008
Senior Statistical Analyst, Data Recognition Corporation, Maple Grove, MN
20022005
Statistical Analyst, Data Recognition Corporation, Maple Grove, MN
19992002
Research Associate, Applied Measurement Professionals, Inc., Lenexa, KS
19981999
Test Development Specialist, Applied Measurement Professionals, Inc.,

Lenexa, KS
1998
Test Development Assistant, Applied Measurement Professionals, Inc.,

Lenexa, KS
19961997
Research Assistant, Mei Technology, San Antonio, TX
1997
Intern, United Services Automobile Association (USAA), San Antonio, TX
2009Present
Idaho Standards Achievement Test
2002Present
Louisiana Educational Assessment Program:
2011Present

Ph.D.
Educational Psychology, Quantitative Methods in Education (in progress),

University of Minnesota, Minneapolis, MN
M.A.
Industrial/Organizational Psychology, St. Marys University, San Antonio, TX
B.S.
Psychology, Valparaiso University, Valparaiso, IN
Page A-115
YI DU
Managing Senior, Psychometrics
2004Present
Managing Senior, Psychometrics, Data Recognition Corporation, Maple Grove, MN
2007Present
Adjunct Assistant Professor, Saint Marys University and the University of

Minnesota, Minneapolis, MN
19992004
Director of Research and Evaluation, Edina Public Schools, Edina, MN
19962004
Technical Advisor, Minnesota Department of Education (MDE), Roseville, MN
19951999
Assistant Director & Testing Assessment Specialist II, Department of Research,

Evaluation, and Assessment, Minneapolis Public Schools, Minneapolis, MN
19921995
Psychometrician, Department of Student Assessment, Illinois State Board of

Education, Springfield, IN
1991
Pre-Doctor Fellow, Statistical and Psychometrical Research and Service, Education

Testing Services, Princeton, NJ
19871988
Visiting Scholar, University of California, Berkeley, CA
19841987
Assistant Professor, Beijing University, China
20052006
19921995
2007Present
2007Present
2012Present
19962005
20002005
2006
2011
20062007
Georgia High School Graduation Tests

Illinois Goal Assessment Programs
Integrated Louisiana Educational Assessment Program
Michigan Interim Assessments
Minnesota Basic Skills Test
North Carolina Test of U.S. History and Test of Civics and Economics: Standard
Setting Services
Pennsylvania System of Student Assessment: Standard Setting Services

Ph.D.
Measurement, Evaluation, and Statistical Analysis, University of Chicago, Chicago, IL
M.S.
Management Information SystemsBeijing University, Beijing, China
B.S.
Management Information SystemsBeijing University, Beijing, China
PMP
Certified Project Management Professional Project Management Institute

Page A-117
Appendix B:
Materials List

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
3-8 ELA
Administrati
on Manual ELA/Math
CBT
43,340
43,340
NA
368
3-8 ELA
Administrati
on Manual ELA/Math
CBT
54,395
54,395
NA
3-8 ELA
Administrati
on Manual ELA/Math
CBT
67,045
67,045
3-8 ELA
4,5,6
Administrati
on Manual ELA/Math
CBT
68,090
3-8 ELA
Administrati
on Manual ELA/Math
PP
3-8 ELA
Administrati
on Manual ELA/Math
PP
3-8 ELA
Administrati
on Manual ELA/Math
PP
December 2013
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
Quantity (for Quantity (for Pages Incl
printing/
processing/
Cover
packaging)
scoring)
Finished
Size
Binding
Paper Type Scannable Perforation Litho Code/

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
Cover Color and

black;
Internal Black
8 1/2 x 11
Perfect
Bound
50# White NA
Internal
pages, 65#
White
Cover
None
NA
NA
NA
M barcode Shrink-wrap
in packs of
10; M
Barcode
cover insert
368
Cover Color and

black;
Internal Black
8 1/2 x 11
Perfect
Bound
50# White NA
Internal
pages, 65#
White
Cover
None
NA
NA
NA
in packs of
10; M
Barcode
cover insert
NA
368
Cover Color and

black;
Internal Black
8 1/2 x 11
Perfect
Bound
50# White NA
Internal
pages, 65#
White
Cover
None
NA
NA
NA
in packs of
10; M
Barcode
cover insert
68,090
NA
368
Cover Color and

black;
Internal Black
8 1/2 x 11
Perfect
Bound
50# White NA
Internal
pages, 65#
White
Cover
None
NA
NA
NA
in packs of
10; M
Barcode
cover insert
40,700
40,700
NA
368
Cover Color and

black;
Internal Black
8 1/2 x 11
Perfect
Bound
50# White NA
Internal
pages, 65#
White
Cover
None
NA
NA
NA
in packs of
10; M
Barcode
cover insert
28,919
28,919
NA
368
Cover Color and

black;
Internal Black
8 1/2 x 11
Perfect
Bound
50# White NA
Internal
pages, 65#
White
Cover
None
NA
NA
NA
in packs of
10; M
Barcode
cover insert
16,880
16,880
NA
368
Cover Color and

black;
Internal Black
8 1/2 x 11
Perfect
Bound
50# White NA
Internal
pages, 65#
White
Cover
None
NA
NA
NA
in packs of
10; M
Barcode
cover insert
Page 1 of 18

Gr 3-8 ELA
Program
3-8 ELA
Contract
Year
Material
Type
# of Forms
(versions)
Quantity
per form
4,5,6
Administrati
on Manual ELA/Math
PP
4,054
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
4,054
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
NA
368
Cover Color and

black;
Internal Black
8 1/2 x 11
Perfect
Bound
50# White NA
Internal
pages, 65#
White
Cover
None
NA
NA
NA
in packs of
10; M
Barcode
cover insert
3-8 ELA
1,2,3,4,5,6 Grade 3
Reading
Test
Booklet
Braille
Contracted
40
40
NA
80
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
3-8 ELA
1,2,3,4,5,6 Grade 3
Reading
Test
Booklet
Braille
Uncontract
ed
10
10
NA
88
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
3-8 ELA
Grade 3
Reading
Test
Booklet
Consumabl
e
15
17,585
263,780
239,800
64
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
Yes
S Barcode
Shrink-wrap
in packs of
10; shrinkwrap range
sheets
inside
packs.
3-8 ELA
Grade 3
Reading
Test
Booklet
Consumabl
e
15
18,231
273,460
248,600
64
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
Yes
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
December 2013
Page 2 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
3-8 ELA
Grade 3
Reading
Test
Booklet
Consumabl
e
271,040
271,040
246,400
64
3-8 ELA
4,5,6
Grade 3
Reading
Test
Booklet
Consumabl
e
14,520
14,520
13,200
3-8 ELA
1,2,3,4,5,6 Grade 3
Reading
Test
Booklet
Large-Print
400
400
3-8 ELA
1,2,3,4,5,6 Grade 3
Reading
Test
Booklet
OIPP
Version
1,2,3,4,5,6 Grade 3
Writing
Test
Booklet
Braille
Contracted
25
1,2,3,4,5,6 Grade 3
Writing
Test
Booklet
Braille
Uncontract
ed
3-8 ELA
3-8 ELA
December 2013
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
64
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
NA
72
Black
11 x 17
Spiral on
17" side
Color stock Non65# Cover, scannable

50# white
Internal
pages
NA
NA
Two interior NA
security
seals
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
25
NA
60
Black
8.5 X 11
3-ring
binder
60# White
Off-set
Nonscan
NA
NA
NA
NA
S Barcode
Label
Individually
shrinkwrapped
40
40
NA
72
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
10
10
NA
80
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
Page 3 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
3-8 ELA
Grade 3
Writing
Test
Booklet
Consumabl
e
15
17,585
263,780
239,800
56
3-8 ELA
Grade 3
Writing
Test
Booklet
Consumabl
e
15
18,231
273,460
248,600
3-8 ELA
Grade 3
Writing
Test
Booklet
Consumabl
e
271,040
271,040
3-8 ELA
4,5,6
Grade 3
Writing
Test
Booklet
Consumabl
e
14,520
14,520
3-8 ELA
1,2,3,4,5,6 Grade 3
Writing
Test
Booklet
Large-Print
400
400
3-8 ELA
1,2,3,4,5,6 Grade 3
Writing
Test
Booklet
OIPP
Version
25
25
December 2013
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
56
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
246,400
56
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
13,200
56
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
NA
64
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
Two interior NA
security
seals
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
NA
60
Black
8.5 X 11
3-ring
binder
60# White
Off-set
NA
NA
NA
S Barcode
Label
Individually
shrinkwrapped
Nonscan
Sealing
Spiraling
NA
Security
Packaging
Bar Coding
Ink Jet or
Laser
Page 4 of 18

Gr 3-8 ELA
Program
# of Forms
(versions)
Quantity
per form
1,2,3,4,5,6 Grade 3-8

ELA Test
Administrat
or Notes
and Scripts
for Braille
300
1,2,3,4,5,6 Grade 3-8

ELA Test
Administrat
or Notes
and Scripts
for LargePrint
3-8 ELA
1,2,3,4,5,6 Grade 3-8

Reading
Braille
Blank
Pages
300
300
NA
6 sheets
NA
3-8 ELA
1,2,3,4,5,6 Grade 3-8

Writing
Braille
Blank
Pages
300
300
NA
6 sheets
3-8 ELA
1,2,3,4,5,6 Grade 4
Reading
Test
Booklet
Braille
Contracted
40
40
NA
3-8 ELA
1,2,3,4,5,6 Grade 4
Reading
Test
Booklet
Braille
Uncontract
ed
10
10
NA
3-8 ELA
3-8 ELA
Contract
Year
December 2013
Material
Type
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
300
400
NA
2,400
NA
32
32
Finished
Size
Binding
Color/Black 8.5 X 11
Saddlestitched
Color/Black 8.5 X 11

Image or
s/ Corner
Serial #s
nonscan
Cuts
45# equal
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
Nonscan
None
NA
M barcode Shrinkwrapped w
Braille test
books
Nonscan
None
NA
M barcode Shrinkwrapped w
Large-Print
test books
45# equal
Saddlestitched
45# equal
45# equal
11 x 11.5
NA
80# white
Nonscannable
NA
NA
NA
NA
NA
Each set of
6 pages will
be shrinkwrapped
with Braille
materials
NA
11 x 11.5
NA
80# white
Nonscannable
NA
NA
NA
NA
NA
Each set of
6 pages will
be shrinkwrapped
with Braille
materials
80
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
88
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
Page 5 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
3-8 ELA
Grade 4
Reading
Test
Booklet
Consumabl
e
245,630
245,630
223,300
64
3-8 ELA
2,3,4,5,6
Grade 4
Reading
Test
Booklet
Consumabl
e
13,310
13,310
12,100
3-8 ELA
1,2,3,4,5,6 Grade 4
Reading
Test
Booklet
Large-Print
400
400
3-8 ELA
1,2,3,4,5,6 Grade 4
Reading
Test
Booklet
OIPP
Version
1,2,3,4,5,6 Grade 4
Writing
Test
Booklet
Braille
Contracted
25
1,2,3,4,5,6 Grade 4
Writing
Test
Booklet
Braille
Uncontract
ed
3-8 ELA
3-8 ELA
December 2013
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
64
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
NA
72
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
Two interior NA
security
seals
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
25
NA
60
Black
8.5 X 11
3-ring
binder
60# White
Off-set
Nonscan
NA
NA
NA
NA
S Barcode
Label
Individually
shrinkwrapped
40
40
NA
72
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
10
10
NA
80
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
Page 6 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
3-8 ELA
Grade 4
Writing
Test
Booklet
Consumabl
e
245,630
245,630
223,300
56
3-8 ELA
2,3,4,5,6
Grade 4
Writing
Test
Booklet
Consumabl
e
13,310
13,310
12,100
3-8 ELA
1,2,3,4,5,6 Grade 4
Writing
Test
Booklet
Large-Print
400
400
3-8 ELA
1,2,3,4,5,6 Grade 4
Writing
Test
Booklet
OIPP
Version
1,2,3,4,5,6 Grade 5
Reading
Answer
Booklet
25
25
13,310
13,310
1,2,3,4,5,6 Grade 5
Reading
Answer
Booklet
Large-Print
400
400
3-8 ELA
3-8 ELA
December 2013
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
56
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
NA
64
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
Two interior NA
security
seals
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
NA
60
Black
8.5 X 11
3-ring
binder
60# White
Off-set
Nonscan
NA
NA
NA
NA
S Barcode
Label
Individually
shrinkwrapped
12,100
16
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
NA
24
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
NA
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
NA
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
Page 7 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
3-8 ELA
1,2,3,4,5,6 Grade 5
Reading
Test
Booklet
13,310
3-8 ELA
1,2,3,4,5,6 Grade 5
Reading
Test
Booklet
Braille
Contracted
40
40
3-8 ELA
1,2,3,4,5,6 Grade 5
Reading
Test
Booklet
Braille
Uncontract
ed
10
3-8 ELA
1,2,3,4,5,6 Grade 5
Reading
Test
Booklet
Large-Print
3-8 ELA
1,2,3,4,5,6 Grade 5
Reading
Test
Booklet
OIPP
Version
1,2,3,4,5,6 Grade 5
Writing
Test
Booklet
3-8 ELA
December 2013
13,310
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
48
Color
8 1/2 x 11
Cover,
Black Ink
on Internal
pages
Two wire
saddlestitch
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
NA
64
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
10
NA
72
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
400
400
NA
64
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
Two interior NA
security
seals
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
25
25
NA
60
Black
8.5 X 11
3-ring
binder
60# White
Off-set
Nonscan
NA
NA
NA
NA
S Barcode
Label
Individually
shrinkwrapped
13,310
16
Color
8 1/2 x 11
Cover,
Black Ink
on Internal
pages
Two wire
saddlestitch
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
13,310
12,100
Finished
Size
12,100
Page 8 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
3-8 ELA
1,2,3,4,5,6 Grade 5
Writing
Answer
Booklet
Large-Print
400
400
3-8 ELA
1,2,3,4,5,6 Grade 5
Writing
Answer
Booklet
13,310
13,310
3-8 ELA
1,2,3,4,5,6 Grade 5
Writing
Test
Booklet
Braille
Contracted
40
40
3-8 ELA
1,2,3,4,5,6 Grade 5
Writing
Test
Booklet
Braille
Uncontract
ed
10
3-8 ELA
1,2,3,4,5,6 Grade 5
Writing
Test
Booklet
Large-Print
3-8 ELA
1,2,3,4,5,6 Grade 5
Writing
Test
Booklet
OIPP
Version
December 2013
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
NA
24
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
12,100
40
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
NA
56
NA
11 x 11.5
Plastic
comb
10
NA
64
NA
11 x 11.5
400
400
NA
56
Black
25
25
NA
60
Black
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
NA
NA
NA
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
Two interior NA
security
seals
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
8.5 X 11
3-ring
binder
60# White
Off-set
NA
NA
NA
S Barcode
Label
Individually
shrinkwrapped
Nonscan
NA
Sealing
NA
Page 9 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
3-8 ELA
1,2,3,4,5,6 Grade 6
Reading
Answer
Booklet
13,310
13,310
3-8 ELA
1,2,3,4,5,6 Grade 6
Reading
Answer
Booklet
Large-Print
400
400
3-8 ELA
1,2,3,4,5,6 Grade 6
Reading
Test
Booklet
13,310
13,310
3-8 ELA
1,2,3,4,5,6 Grade 6
Reading
Test
Booklet
Braille
Contracted
40
40
3-8 ELA
1,2,3,4,5,6 Grade 6
Reading
Test
Booklet
Braille
Uncontract
ed
10
10
December 2013
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
12,100
16
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
NA
24
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
12,100
48
Color
8 1/2 x 11
Cover,
Black Ink
on Internal
pages
Two wire
saddlestitch
50# White
NA
64
NA
11 x 11.5
Plastic
comb
NA
72
NA
11 x 11.5
Plastic
comb
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
NA
NA
NA
NA
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
Page 10 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
3-8 ELA
1,2,3,4,5,6 Grade 6
Reading
Test
Booklet
Large-Print
400
400
NA
64
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
Two interior NA
security
seals
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
3-8 ELA
1,2,3,4,5,6 Grade 6
Reading
Test
Booklet
OIPP
Version
1,2,3,4,5,6 Grade 6
Writing
Test
Booklet
25
25
NA
60
Black
8.5 X 11
3-ring
binder
60# White
Off-set
Nonscan
NA
NA
NA
NA
S Barcode
Label
Individually
shrinkwrapped
13,310
13,310
12,100
16
Color
8 1/2 x 11
Cover,
Black Ink
on Internal
pages
Two wire
saddlestitch
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
3-8 ELA
1,2,3,4,5,6 Grade 6
Writing
Answer
Booklet
Large-Print
400
400
NA
24
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
NA
NA
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
3-8 ELA
1,2,3,4,5,6 Grade 6
Writing
Answer
Booklet
13,310
13,310
12,100
40
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
3-8 ELA
1,2,3,4,5,6 Grade 6
Writing
Test
Booklet
Braille
Contracted
40
56
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
3-8 ELA
December 2013
40
NA
NA
Page 11 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
3-8 ELA
1,2,3,4,5,6 Grade 6
Writing
Test
Booklet
Braille
Uncontract
ed
10
10
NA
64
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
3-8 ELA
1,2,3,4,5,6 Grade 6
Writing
Test
Booklet
Large-Print
400
400
NA
56
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
3-8 ELA
1,2,3,4,5,6 Grade 6
Writing
Test
Booklet
OIPP
Version
1,2,3,4,5,6 Grade 7
Reading
Answer
Booklet
25
25
NA
60
Black
8.5 X 11
3-ring
binder
60# White
Off-set
Nonscan
NA
13,310
13,310
12,100
16
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
3-8 ELA
1,2,3,4,5,6 Grade 7
Reading
Answer
Booklet
Large-Print
400
400
NA
24
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
3-8 ELA
1,2,3,4,5,6 Grade 7
Reading
Test
Booklet
13,310
13,310
12,100
48
Color
8 1/2 x 11
Cover,
Black Ink
on Internal
pages
Two wire
saddlestitch
50# White
3-8 ELA
December 2013
Nonscannable
Spiraling
NA
Security
Packaging
Bar Coding
Ink Jet or
Laser
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
Two interior NA
security
seals
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
NA
NA
NA
S Barcode
Label
Individually
shrinkwrapped
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
NA
NA
NA
NA
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
NA
NA
Two interior NA
security
seals
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
Page 12 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
3-8 ELA
1,2,3,4,5,6 Grade 7
Reading
Test
Booklet
Braille
Contracted
40
40
NA
64
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
3-8 ELA
1,2,3,4,5,6 Grade 7
Reading
Test
Booklet
Braille
Uncontract
ed
10
10
NA
72
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
3-8 ELA
1,2,3,4,5,6 Grade 7
Reading
Test
Booklet
Large-Print
400
400
NA
64
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
Two interior NA
security
seals
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
3-8 ELA
1,2,3,4,5,6 Grade 7
Reading
Test
Booklet
OIPP
Version
1,2,3,4,5,6 Grade 7
Writing
Test
Booklet
25
25
NA
60
Black
8.5 X 11
3-ring
binder
60# White
Off-set
Nonscan
NA
NA
NA
NA
S Barcode
Label
Individually
shrinkwrapped
13,310
13,310
12,100
16
Color
8 1/2 x 11
Cover,
Black Ink
on Internal
pages
Two wire
saddlestitch
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
1,2,3,4,5,6 Grade 7
Writing
Answer
Booklet
Large-Print
400
400
NA
24
Black
Spiral on
17" side

50# white
Internal
pages
NA
NA
NA
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
3-8 ELA
3-8 ELA
December 2013
11 x 17
NA
Page 13 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
3-8 ELA
1,2,3,4,5,6 Grade 7
Writing
Answer
Booklet
13,310
3-8 ELA
1,2,3,4,5,6 Grade 7
Writing
Test
Booklet
Braille
Contracted
40
40
3-8 ELA
1,2,3,4,5,6 Grade 7
Writing
Test
Booklet
Braille
Uncontract
ed
10
3-8 ELA
1,2,3,4,5,6 Grade 7
Writing
Test
Booklet
Large-Print
3-8 ELA
1,2,3,4,5,6 Grade 7
Writing
Test
Booklet
OIPP
Version
1,2,3,4,5,6 Grade 8
Reading
Answer
Booklet
3-8 ELA
December 2013
13,310
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
40
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
NA
56
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
10
NA
64
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
400
400
NA
56
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
Two interior NA
security
seals
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
25
25
NA
60
Black
8.5 X 11
3-ring
binder
60# White
Off-set
Nonscan
NA
NA
NA
NA
S Barcode
Label
Individually
shrinkwrapped
13,310
16
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
13,310
12,100
Finished
Size
12,100
Page 14 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
3-8 ELA
1,2,3,4,5,6 Grade 8
Reading
Answer
Booklet
Large-Print
400
400
3-8 ELA
1,2,3,4,5,6 Grade 8
Reading
Test
Booklet
13,310
13,310
3-8 ELA
1,2,3,4,5,6 Grade 8
Reading
Test
Booklet
Braille
Contracted
40
40
3-8 ELA
1,2,3,4,5,6 Grade 8
Reading
Test
Booklet
Braille
Uncontract
ed
10
3-8 ELA
1,2,3,4,5,6 Grade 8
Reading
Test
Booklet
Large-Print
3-8 ELA
1,2,3,4,5,6 Grade 8
Reading
Test
Booklet
OIPP
Version
December 2013
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
NA
NA
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
NA
24
Black
12,100
48
Color
8 1/2 x 11
Cover,
Black Ink
on Internal
pages
Two wire
saddlestitch
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
NA
64
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
10
NA
72
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
400
400
NA
64
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
Two interior NA
security
seals
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
25
25
NA
60
Black
8.5 X 11
3-ring
binder
60# White
Off-set
NA
NA
NA
S Barcode
Label
Individually
shrinkwrapped
Nonscan
NA
Page 15 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
3-8 ELA
1,2,3,4,5,6 Grade 8
Writing
Test
Booklet
13,310
13,310
3-8 ELA
1,2,3,4,5,6 Grade 8
Writing
Answer
Booklet
Large-Print
400
400
3-8 ELA
1,2,3,4,5,6 Grade 8
Writing
Answer
Booklet
13,310
13,310
3-8 ELA
1,2,3,4,5,6 Grade 8
Writing
Test
Booklet
Braille
Contracted
40
40
3-8 ELA
1,2,3,4,5,6 Grade 8
Writing
Test
Booklet
Braille
Uncontract
ed
10
10
December 2013
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
12,100
16
Color
8 1/2 x 11
Cover,
Black Ink
on Internal
pages
Two wire
saddlestitch
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
NA
24
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
NA
NA
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
12,100
40
Color
Cover,
Dropout
and Black
Ink on
Internal
pages
8 1/2 x 11
Two wire
saddlestitch
60# White
Reflective
Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap
in packs of
sheets
inside
packs.
NA
56
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
NA
64
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrinkwrapped
individually
with Braille
script, blank
pages, and
reg test and
answer
booklets
Page 16 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
3-8 ELA
1,2,3,4,5,6 Grade 8
Writing
Test
Booklet
Large-Print
400
400
NA
56
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
Two interior NA
security
seals
S Barcode
Label
Shrinkwrapped
individually
with script
and reg test
and answer
booklets
3-8 ELA
1,2,3,4,5,6 Grade 8
Writing
Test
Booklet
OIPP
Version
1,2,3,4,5,6 Grades 3-8
ELA Braille
Reading
Passage
Booklets for
CBT
25
25
NA
60
Black
8.5 X 11
3-ring
binder
60# White
Off-set
Nonscan
NA
NA
NA
NA
S Barcode
Label
Individually
shrinkwrapped
50
300
NA
32
NA
11 x 11.5
Plastic
comb
80# white
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Shrinkwrapped
individually
1,2,3,4,5,6 Grades 3-8

ELA Large
Print and
Braille
Materials
Return
Envelope
1,2,3,4,5,6 Grades 3-8
ELA LargePrint
Reading
Passage
Booklets for
CBT
900
900
NA
NA
NA
12 x 16 x 2" NA
Tyvek
Nonscan
Expandable
Envelope
MODEL S5620
NA
NA
NA
NA
HandNA
applied
label with
instructions
and M
barcode
400
2,400
NA
24
Black
11 x 17
Spiral on
17" side

50# white
Internal
pages
NA
NA
NA
NA
S Barcode
Label
Shrinkwrapped in
packs of 5
with range
sheet
3-8 ELA
3-8 ELA
3-8 ELA
3-8 ELA
3-8 ELA
December 2013
Grades 3-8
ELA
Regular
Size
Reading
Passage
Booklets for
CBT
Grades 3-8
ELA
Regular
Size
Reading
Passage
Booklets for
CBT
6,570
39,420
NA
24
Black
8.5 x 11
Saddlestitched
45# equal
Nonscan
None
NA
NA
NA
S barcode
Shrinkwrapped in
packs of 5
with range
sheets
8,242
49,450
NA
24
Black
8.5 x 11
Saddlestitched
45# equal
Nonscan
None
NA
NA
NA
S barcode
Shrinkwrapped in
packs of 5
with range
sheets
Page 17 of 18

Gr 3-8 ELA
Program
Contract
Year
Material
Type
# of Forms
(versions)
3-8 ELA
8,383
50,300
NA
24
3-8 ELA
4,5,6
Grades 3-8
ELA
Regular
Size
Reading
Passage
Booklets for
CBT
Grades 3-8
ELA
Regular
Size
Reading
Passage
Booklets for
CBT
Sample
Test
Materials
(STM) Test
Booklets Large-Print
Version
10,317
61,900
NA
400
2400
Sample
Test
Materials
(STM) Test
Booklets Braille
Version
50
300
3-8 ELA
1,2,3,4,5,6
3-8 ELA
1,2,3,4,5,6
December 2013
Quantity
per form
Total
Total
8.5 x 11 Ink Colors
printing/
processing/
Cover
packaging)
scoring)
Finished
Size
Binding

Image or
s/ Corner
Serial #s
nonscan
Cuts
Black
8.5 x 11
Saddlestitched
45# equal
Nonscan
None
NA
NA
NA
S barcode
Shrinkwrapped in
packs of 5
with range
sheets
24
Black
8.5 x 11
Saddlestitched
45# equal
Nonscan
None
NA
NA
NA
S barcode
Shrinkwrapped in
packs of 5
with range
sheets
NA
24
Black
11 x 17
Spiral on
17" side
65# Cover, Nonscan

50# white
Internal
pages
NA
NA
NA
NA
NA
Shrinkwrapped
individually
NA
32
N/A
11 x 11.5
Plastic
comb
80# white
NA
NA
NA
NA
NA
Shrinkwrapped
individually
Nonscan
Sealing
Spiraling
Security
Packaging
Bar Coding
Ink Jet or
Laser
Page 18 of 18

Gr 3-8 Math
Program
Contract
Year
Material Type
# of Forms
(versions)
3-8 Math
Grade 3 Math Test

Booklet Consumable
15
3-8 Math
Grade 3 Math Test

Booklet Consumable
3-8 Math
3,4,5,6
Grade 3 Math Test

Booklet Consumable
3-8 Math
Grade 4 Math Test

Booklet Consumable
15
3-8 Math
Grade 4 Math Test

Booklet Consumable
3-8 Math
3,4,5,6
Grade 4 Math Test

Booklet Consumable
3-8 Math
1,2,3,4,5,6 Grade 5 Math Answer

Booklet
3-8 Math

Booklet
3-8 Math

Booklet
3-8 Math
Grade 8 Math Answer

Booklet
3-8 Math
2,3,4,5,6
Grade 8 Math Answer

Booklet
Quantity
per form
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
packaging) processing/
scoring)
17,585
263,780
239,800
64
Color Cover,
Dropout and Black
Ink on Internal
pages
273,460
273,460
248,600
64
Color Cover,
Dropout and Black
Ink on Internal
pages
14,520
14,520
13,200
64
Color Cover,
Dropout and Black
Ink on Internal
pages
16,375
245,630
223,300
64
Color Cover,
Dropout and Black
Ink on Internal
pages
251,680
251,680
228,800
64
Color Cover,
Dropout and Black
Ink on Internal
pages
13,310
13,310
12,100
64
Color Cover,
Dropout and Black
Ink on Internal
pages
13,310
13,310
12,100
16
Color Cover,
Dropout and Black
Ink on Internal
pages
13,310
13,310
12,100
16
Color Cover,
Dropout and Black
Ink on Internal
pages
13,310
13,310
12,100
16
Color Cover,
Dropout and Black
Ink on Internal
pages
258,940
258,940
235,400
16
Color Cover,
Dropout and Black
Ink on Internal
pages
13,310
13,310
12,100
16
Color Cover,
Dropout and Black
Ink on Internal
pages
40
40
NA
80
NA
Finished
Size
Binding
Paper Type
Scannable Perforation Litho Code/

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Bar Coding
Ink Jet or
Laser
8 1/2 x 11
Two wire saddlestitch
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
Shrink-wrap in packs of
10; shrink-wrap range
sheets inside packs.
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
NA
Label
Shrink-wrapped
individually with Braille
script, blank pages, and
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
individually with script
and reg test and answer
booklets
Each set of 6 pages will
be shrink-wrapped with
Braille materials
Shrink-wrapped
reg test and answer
booklets
3-8 Math
1,2,3,4,5,6 Grade 3 Math Test

Booklet Braille
Contracted
3-8 Math

Booklet Braille
Uncontracted
10
10
NA
88
NA
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
3-8 Math

Booklet Large-Print
400
400
NA
72
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
3-8 Math
1,2,3,4,5,6 Grade 3-8 Math Braille

Blank Pages
150
NA
6 sheets
NA
11 x 11.5
NA
80# white
Nonscannable
NA
NA
NA
NA
NA
3-8 Math

Booklet Braille
Contracted
40
NA
80
NA
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
December 2013
300
40
Packaging
Page 1 of 4

Gr 3-8 Math
Program
Contract
Year
Material Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
10
10
NA
88
NA
3-8 Math

Booklet Braille
Uncontracted
3-8 Math

Booklet Large-Print
400
3-8 Math

Booklet Large-Print
400
400
3-8 Math

Booklet
13,310
3-8 Math

Booklet Braille
Contracted
3-8 Math

Booklet Braille
Uncontracted
3-8 Math
Binding
Paper Type

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
Spiraling
NA
Security
Bar Coding
Ink Jet or
Laser
Label
NA
72
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
NA
24
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
13,310
12,100
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
40
40
NA
72
NA
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
10
10
NA
80
NA
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label

Booklet Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
3-8 Math

Booklet Large-Print
400
400
NA
24
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
3-8 Math

Booklet
13,310
13,310
12,100
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
3-8 Math

Booklet Braille
Contracted
40
40
NA
72
NA
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
3-8 Math

Booklet Braille
Uncontracted
10
10
NA
80
NA
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
3-8 Math

Booklet Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
3-8 Math

Booklet Large-Print
400
400
NA
24
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
3-8 Math

Booklet
13,310
13,310
12,100
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
3-8 Math

Booklet Braille
Contracted
40
40
NA
72
NA
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
3-8 Math

Booklet Braille
Uncontracted
10
10
NA
80
NA
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
December 2013
400
Finished
Size
Packaging
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Page 2 of 4

Gr 3-8 Math
Program
Contract
Year
Material Type
# of Forms
(versions)
Quantity
per form
3-8 Math

Booklet Large-Print
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
400
400
NA
56
Black
3-8 Math

Booklet Large-Print
400
400
NA
Finished
Size
Binding
Paper Type

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Bar Coding
Ink Jet or
Laser
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
24
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Packaging
Shrink-wrapped
booklets
Shrink-wrapped
booklets
3-8 Math
Grade 8 Math Test

Booklet
258,940
258,940
235,400
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
3-8 Math
2,3,4,5,6
Grade 8 Math Test

Booklet
13,310
13,310
12,100
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
3-8 Math

Booklet Braille
Contracted
40
40
NA
72
NA
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
3-8 Math

Booklet Braille
Uncontracted
10
10
NA
80
NA
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
3-8 Math

Booklet Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Shrinkwrapped in packs
of 30 with cover sheet
3-8 Math
ELA CBT Work Folder
1,406,000
1,406,000
NA
1 sheet
black
8.5 x 11
NA
Light green #45
nonscan
NA
NA
NA
NA
M barcode
3-8 Math
ELA CBT Work Folder
1,838,000
1,838,000
NA
1 sheet
black
8.5 x 11
NA
Light green #45
nonscan
NA
NA
NA
NA
M barcode
3-8 Math
ELA CBT Work Folder
1,853,000
1,853,000
NA
1 sheet
black
8.5 x 11
NA
Light green #45
nonscan
NA
NA
NA
NA
M barcode
3-8 Math
4,5,6
ELA CBT Work Folder
1,947,000
1,947,000
NA
1 sheet
black
8.5 x 11
NA
Light green #45
nonscan
NA
NA
NA
NA
M barcode
3-8 Math
Math CBT Work Folder
1,345,000
1,345,000
NA
1 sheet
black
NA
Yellow #45
nonscan
NA
NA
NA
NA
M barcode
3-8 Math
1,789,000
1,789,000
NA
1 sheet
black
NA
Yellow #45
nonscan
NA
NA
NA
NA
M barcode
3-8 Math
2,279,000
2,279,000
NA
1 sheet
black
NA
Yellow #45
nonscan
NA
NA
NA
NA
M barcode
3-8 Math
4,5,6
2,315,000
2,315,000
NA
1 sheet
black
NA
Yellow #45
nonscan
NA
NA
NA
NA
M barcode

Booklet OIPP Version
1,2,3,4,5,6 Grade 3-8 Math Large
Print and Braille
Materials Return
Envelope
25
25
NA
60
Black
11 x 17
folded to
8.5 by 11
11 x 17
folded to
8.5 by 11
11 x 17
folded to
8.5 by 11
11 x 17
folded to
8.5 by 11
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
900
NA
NA
NA
12 x 16 x 2" NA
Tyvek Expandable Nonscan

Envelope MODEL S5620
NA
NA
NA
NA
Individually
shrinkwrapped
NA

25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
S Barcode
Label
Handapplied
label with
instructions
and M
barcode
S Barcode
Label
S Barcode
Label
S Barcode
Label
S Barcode
Label
3-8 Math
3-8 Math
3-8 Math
3-8 Math
3-8 Math
3-8 Math
December 2013
900
Individually
shrinkwrapped
Individually
shrinkwrapped
Individually
shrinkwrapped
Individually
shrinkwrapped
Page 3 of 4

Gr 3-8 Math
Program
3-8 Math
3-8 Math
3-8 Math
3-8 Math
3-8 Math
3-8 Math
3-8 Math
Contract
Year
Material Type

1,2,3,4,5,6 Grade 3-8 Math Test
Administrator Notes and
Scripts for Braille
1,2,3,4,5,6 Grade 3-8 Math Test
Scripts for Large-Print
1,2,3,4,5,6 Sample Test Materials
(STM) Test Booklets Large-Print Version
(STM) Test Booklets Braille Version
1,2,3,4,5,6 Math Reference Sheets Large-Print Version
1,2,3,4,5,6 Math Reference Sheets Braille Version
December 2013
# of Forms
(versions)
1
1
Quantity
per form
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
25
25
NA
60
Black
300
300
NA
32
Color/Black
Finished
Size
Binding
8.5 X 11
3-ring binder
8.5 X 11
Saddle-stitched
Paper Type
60# White Off-set
45# equal
6
400
2,400
NA
32
Color/Black
8.5 X 11

Image or
s/ Corner
Serial #s
nonscan
Cuts
Nonscan
Spiraling
NA
NA
NA
NA
Nonscan
None
NA
Nonscan
None
Security
Bar Coding
Ink Jet or
Laser
Packaging
NA
S Barcode
Label
M barcode
Individually
shrinkwrapped
Shrinkwrapped w Braille
test books
NA
NA
M barcode
Shrinkwrapped w LargePrint test books
45# equal
Saddle-stitched
45# equal
Sealing
45# equal
400
2,400
NA
24
Black
11 x 17
Spiral on 17" side
65# Cover, 50#

Nonwhite Internal pages scannable
NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped
individually
50
300
NA
32
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrink-wrapped
individually
400
2,400
NA
Black
11 x 17
NA
50# white
NA
NA
NA
NA
NA
50
300
NA
N/A
11 x 11.5
NA
80# white
Nonscannable
Nonscannable
NA
NA
NA
NA
NA
Shrink-wrapped in
packs of 20
Shrink-wrapped
individually
Page 4 of 4

Gr 9-11 ELA
Contract
Year
2,3,4,5,6
Material Type
# of Forms Quantity per

Total
Total
8.5 x 11
(versions)
form
printing/ processing/
Cover
packaging)
scoring)
Administration Manual Retake Gr 10 ELA PP &

CBT
Fall Gr 10 ELA Retest
Answer Book
13,500
13,500
NA
148
8,800
8,800
8,000
16
2,3,4,5,6

Braille Blank Pages
50
50
NA
2,3,4,5,6

Test Book
8,800
8,800
8,000
48
2,3,4,5,6

Test Book - Braille
Contracted
40
40
NA
2,3,4,5,6

Test Book - Braille
Uncontracted
10
10
2,3,4,5,6

Test Book - Large-Print
400
2,3,4,5,6

Test Book - OIPP Version
1,4,5,6
Administration Manual ELA Grades 9-11 CBT
2,3
Ink Colors
Finished
Size
Binding
Cover - Color and

black; Internal Black
Color Cover,
Dropout and Black
Ink on Internal
pages
NA
8 1/2 x 11
Perfect Bound
8 1/2 x 11
Paper Type
Scannable Perforations Litho Code/

Image or
/ Corner
Serial #s
nonscan
Cuts
NA
50# White Internal

pages, 65# White
Cover
60# White
Reflective Read
11 x 11.5
NA
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
64
N/A
NA
72
400
NA
25
25
34,000
Administration Manual ELA Grades 9-11 CBT
1,2,3,4,5,6 Administration Manual ELA Grades 9-11 PP
Spiraling
Security Bar
Coding Ink
Jet or Laser
NA
NA
NA
M barcode
Image Scan NA
Yes
NA
NA
S Barcode
80# white
Nonscannable
NA
NA
NA
NA
NA
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
S Barcode
Label
34,000
NA
184
8 1/2 x 11
Perfect Bound
None
NA
NA
NA
M barcode
34,500
NA
184
8 1/2 x 11
Perfect Bound
NA
None
NA
NA
NA
M barcode
2,057
2,057
NA
184
8 1/2 x 11
Perfect Bound
NA
None
NA
NA
NA
M barcode
1,2,3,4,5,6 Grade 10 Reading

Answer Booklet
13,310
13,310
12,100
16
8 1/2 x 11
50# White Internal

pages, 65# White
Cover
50# White Internal
pages, 65# White
Cover
50# White Internal
pages, 65# White
Cover
60# White
Reflective Read
NA
34,500
Image Scan NA
Yes
NA
NA
S Barcode

Answer Booklet LargePrint
400
400
NA
24
Cover - Color and

Cover - Color and
Cover - Color and
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
1,2,3,4,5,6 Grade 10 Reading Test

Booklet
13,310
13,310
12,100
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode

Booklet Braille
Contracted
40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label

Booklet Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label

Booklet Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
2,3,4,5,6
December 2013
6 sheets
None
Sealing
S Barcode
Label
Packaging
10; M Barcode cover
insert
Each set of 6 sheets will
Braille materials
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
10; M Barcode cover
insert
10; M Barcode cover
insert
10; M Barcode cover
insert
Shrink-wrapped
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Page 1 of 5

Gr 9-11 ELA
Contract
Year
Material Type

Total
Total
8.5 x 11
(versions)
form
Cover
packaging)
scoring)
Ink Colors
Finished
Size
Binding
Paper Type

Image or
/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling

1,2,3,4,5,6 Grade 10 Writing Answer
Booklet
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
13,310
13,310
12,100
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA

Booklet Large-Print
400
400
NA
24
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
1,2,3,4,5,6 Grade 10 Writing Test

Booklet
13,310
13,310
12,100
40
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA

Booklet Braille
Contracted
40
40
NA
56
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable

Booklet Braille
Uncontracted
10
10
NA
64
N/A
11 x 11.5
Plastic comb
80# white

Booklet Large-Print
400
400
NA
48
Black
11 x 17
Spiral on 17" side

Answer Booklet
25
25
NA
60
Black
8.5 X 11
13,310
13,310
12,100
16

Answer Booklet LargePrint
400
400
NA
24
Color Cover,
Dropout and Black
Ink on Internal
pages
Black

Booklet
13,310
13,310
12,100
48

Booklet Braille
Contracted
40
40
NA

Booklet Braille
Uncontracted
10
10

Booklet Large-Print
400

Booklet

Booklet Large-Print
December 2013
Security Bar
Coding Ink
Jet or Laser
Packaging
S Barcode
Label
S Barcode
Individually
shrinkwrapped
NA
S Barcode
Label
NA
Two interior NA
security
seals
S Barcode
Shrink-wrapped
booklets
NA
NA
NA
NA
Label
Nonscannable
NA
NA
NA
NA
Label
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
Label
S Barcode
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
13,310
13,310
12,100
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
400
400
NA
24
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
S Barcode
Label
S Barcode
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Shrink-wrapped
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Shrink-wrapped
booklets
Page 2 of 5

Gr 9-11 ELA
Contract
Year
Material Type

Total
Total
8.5 x 11
(versions)
form
Cover
packaging)
scoring)
Ink Colors
Finished
Size
Binding
Paper Type

Image or
/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security Bar
Coding Ink
Jet or Laser
Packaging

Booklet
13,310
13,310
12,100
40
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode

Booklet Braille
Contracted
40
40
NA
56
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label

Booklet Braille
Uncontracted
10
10
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label

Booklet Large-Print
400
400
NA
48
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label

1,2,3,4,5,6 Grade 9 Reading Answer
Booklet
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
14,520
14,520
13,200
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
1,2,3,4,5,6 Grade 9 Reading Answer

Booklet Large-Print
400
400
NA
24
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
S Barcode
Label
S Barcode
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label

Booklet
14,520
14,520
13,200
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode

Booklet Braille
Contracted
40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label

Booklet Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label

Booklet Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label

Booklet
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
14,520
14,520
13,200
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA

Booklet Large-Print
400
400
NA
24
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
S Barcode
Label
S Barcode
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label

Booklet
14,520
14,520
13,200
40
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode

Booklet Braille
Contracted
40
40
NA
56
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
Label
December 2013
NA
Shrink-wrapped
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Shrink-wrapped
booklets
Shrink-wrapped
reg test and answer
booklets
Page 3 of 5

Gr 9-11 ELA
Contract
Year
Material Type

Total
Total
8.5 x 11
(versions)
form
Cover
packaging)
scoring)
Ink Colors
Finished
Size
Binding
Paper Type

Image or
/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
10
10
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA

Booklet Large-Print
400
400
NA
48
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label

1,2,3,4,5,6 Grade 9-11 ELA Test
Scripts for Braille
1,2,3,4,5,6 Grade 9-11 ELA Test
1,2,3,4,5,6 Grade 9-11 ELA Test
Scripts for OIPP
1,2,3,4,5,6 Grade 9-11 Reading
Braille Blank Pages
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
150
150
NA
32
Color/Black
8.5 X 11
Saddle-stitched
Nonscan
None
NA
S Barcode
Label
M barcode
45# equal
45# equal
Nonscan
None
NA
M barcode
45# equal
45# equal
Nonscan
None
NA
M barcode
45# equal
45# equal
Shrinkwrapped w OIPP
test books
150
150
NA
6 sheets
NA
11 x 11.5
NA
80# white
Nonscannable
NA
NA
NA
NA
NA
1,2,3,4,5,6 Grade 9-11 Writing

Braille Blank Pages
150
150
NA
6 sheets
NA
11 x 11.5
NA
80# white
Nonscannable
NA
NA
NA
NA
NA
1,2,3,4,5,6 Grades 9-11 Braille

Reading Passage
Booklets for CBT
1,2,3,4,5,6 Grades 9-11 ELA LargePrint Reading Passage
Booklets for CBT
1,2,3,4,5,6 Grades 9-11 ELA
Regular Size Reading
Passage Booklets for
CBT
1,2,3,4,5,6 Grades 9-11 ELA Retest
Large Print and Braille
Materials Return
Envelope
50
150
NA
32
NA
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
S Barcode
Label

Braille materials
Braille materials
Shrink-wrapped
individually
400
1,200
NA
24
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped in packs
of 5 with range sheet
31,000
NA
24
Black
8.5 x 11
Saddle-stitched
Nonscan
None
NA
NA
NA
S barcode
of 5 with range sheets
Handapplied
label with
instructions
and M
barcode
Handapplied
label with
instructions
and M
barcode
S Barcode
Label
NA
150
400
10,333
150
1,200
NA
NA
32
32
Color/Black
Color/Black
8.5 X 11
8.5 X 11
Saddle-stitched
Saddle-stitched
NA
Label
Packaging

Booklet Braille
Uncontracted
NA
Security Bar
Coding Ink
Jet or Laser
45# equal
450
450
NA
NA
NA
12 x 16 x 2" NA

NA
NA
NA
NA
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
test books
2,3,4,5,6
Gr 10 ELA Retest Large

Print and Braille Materials
Return Envelope
300
300
NA
NA
NA
12 x 16 x 2" NA

NA
NA
NA
NA
2,3,4,5,6
Grade 10 ELA Braille

Reading Passage
Booklets for CBT
Grade 10 ELA Retest
Large-Print Reading
CBT
Grade 10 ELA Retest
Regular Size Reading
CBT
Grade 10 ELA Retest
Test Administrator Notes
and Scripts for Braille
50
100
NA
32
NA
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
400
800
NA
24
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped in packs
of 5 with range sheet
NA
24
Black
8.5 x 11
Saddle-stitched
Nonscan
None
NA
NA
NA
S barcode
of 5 with range sheets
Nonscan
None
NA
M barcode
test books
2,3,4,5,6
2,3,4,5,6
2,3,4,5,6
December 2013
5,750
11,500
45# equal
50
50
NA
32
Color/Black
8.5 X 11
Saddle-stitched
45# equal
45# equal
NA
Shrink-wrapped
individually
Page 4 of 5

Gr 9-11 ELA
Contract
Year
Material Type
2,3,4,5,6
Grade 10 ELA Retest

Test Administrator Notes
and Scripts for LargePrint
Spring Gr 10 ELA Retest
Answer Book
2,3,4,5,6

Total
Total
8.5 x 11
(versions)
form
Cover
packaging)
scoring)
1
400
400
NA
5,500
5,500
5,000
32
16
Color/Black
Finished
Size
8.5 X 11
Binding
Paper Type

Image or
/ Corner
Serial #s
nonscan
Cuts
Saddle-stitched
Nonscan
45# equal
45# equal
Sealing
None
NA
Spiraling
Security Bar
Coding Ink
Jet or Laser
Color Cover,
Dropout and Black
Ink on Internal
pages
NA
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
11 x 11.5
NA
80# white
Nonscannable
NA
NA
NA
NA
NA
Each set of 6 sheets will

Braille materials

Braille Blank Pages
50
50
NA
2,3,4,5,6

Test Book
5,500
5,500
5,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
2,3,4,5,6

Test Book - Braille
Contracted
40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
2,3,4,5,6

Test Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
2,3,4,5,6

Test Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
2,3,4,5,6

Test Book - OIPP Version
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
S Barcode
Label
2,3,4,5,6

Answer Book Large-Print
400
400
NA
24
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
2,3,4,5,6

400
400
NA
24
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
1,200
1,200
NA
24
Black
11 x 17
Spiral on 17" side
65# Cover, 50#

NA
NA
NA
NA
S Barcode
Label
150
150
NA
32
N/A
11 x 11.5
Plastic comb
80# white
NA
NA
NA
NA
Label
December 2013
Packaging
M barcode
2,3,4,5,6

(STM) Test Booklets Large-Print Version
(STM) Test Booklets Braille Version
6 sheets
Ink Colors
Nonscannable
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Shrink-wrapped
individually
Shrink-wrapped
individually
Page 5 of 5

EOC Math
Program
Contract
Year
Material Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
44,000
44,000
NA
352
Cover - Color and
57,500
57,500
NA
352
Cover - Color and
61,100
61,100
NA
352
Cover - Color and
62,200
62,200
NA
352
Cover - Color and
300
300
NA
6 sheets NA
Finished
Size
Binding
8 1/2 x 11
Perfect Bound
8 1/2 x 11
Perfect Bound
8 1/2 x 11
Perfect Bound
8 1/2 x 11
Perfect Bound
11 x 11.5
EOC Math
Administration Manual EOC Math PP & CBT
EOC Math
EOC Math
EOC Math
4,5,6
EOC Math
EOC Braille Blank Pages
EOC Math
2,3,4,5,6
EOC Braille Blank Pages
500
500
NA
6 sheets
NA
11 x 11.5
NA
M barcode
NA
None
NA
NA
NA
M barcode
NA
None
NA
NA
NA
M barcode
NA
None
NA
NA
NA
M barcode
NA
Nonscannable
NA
NA
NA
NA
NA
NA
80# white
Nonscannable
NA
NA
NA
NA
NA
NA
NA
NA
NA
Nonscan
None
NA
Handapplied
label with
instructions
and M
barcode
M barcode Shrinkwrapped w Braille
test books
Nonscan
None
NA
NA
NA
NA
12 x 16 x 2" NA
EOC Math
1,2,3,4,5,6 EOC Math Test

Scripts for Braille
1,2,3,4,5,6 EOC Math Test
2
Fall Algebra I Answer
Book
150
150
NA
32
Color/Black
8.5 X 11
EOC Math

Saddle-stitched
45# equal
32
4,400
4,400
4,000
16
Color/Black
8.5 X 11
Packaging
NA
450
NA
Security
Bar Coding
Ink Jet or
Laser
NA
450
1,200
Spiraling
None
400
Sealing
NA
1,2,3,4,5,6 EOC Math Large Print

and Braille Materials
Return Envelope

Image or
s/ Corner
Serial #s
nonscan
Cuts
50# White Internal

pages, 65# White
Cover
50# White Internal
pages, 65# White
Cover
50# White Internal
pages, 65# White
Cover
50# White Internal
pages, 65# White
Cover
80# white
EOC Math
EOC Math
Paper Type
45# equal
Saddle-stitched
45# equal
45# equal
10; M Barcode cover
insert
10; M Barcode cover
insert
10; M Barcode cover
insert
10; M Barcode cover
insert
Braille materials
Braille materials
NA
M barcode
Color Cover,
Dropout and Black
Ink on Internal
pages
Color Cover,
Dropout and Black
Ink on Internal
pages
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped
booklets
EOC Math

Book
5,500
5,500
5,000
16
EOC Math
4,5,6

Book
6,600
6,600
6,000
16
EOC Math
2,3,4,5,6

Book Large-Print
400
400
EOC Math
Fall Algebra I Test Book
4,400
4,400
4,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math
5,500
5,500
5,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math
4,5,6
6,600
6,600
6,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math
Fall Algebra I Test Book Braille Contracted
40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
December 2013
NA
24
Page 1 of 10

EOC Math
Program
Contract
Year
Material Type
# of Forms
(versions)
EOC Math
4,5,6
EOC Math
EOC Math
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Fall Algebra I Test Book Braille Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
4,5,6
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
Fall Algebra I Test Book Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
4,5,6
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
4,5,6
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
2,3,4,5,6
Fall Algebra I Test Book OIPP Version

Fall Algebra II Answer
Book
1,100
1,100
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
EOC Math
2,3,4,5,6
Fall Algebra II Answer

Book Large-Print
400
400
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
S Barcode
Label
S Barcode
Label
S Barcode
Label
S Barcode
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
EOC Math
2,3,4,5,6
Fall Algebra II Test Book
1,100
1,100
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math
2,3,4,5,6
Fall Algebra II Test Book Braille Contracted
40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
2,3,4,5,6
Fall Algebra II Test Book Braille Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
2,3,4,5,6
Fall Algebra II Test Book Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
2,3,4,5,6
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
2,3,4,5,6
Fall Algebra II Test Book OIPP Version

Fall Geometry Answer
Book
1,100
1,100
16
Color Cover,
Dropout and Black
Ink on Internal
pages
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
Label
S Barcode
December 2013
Quantity
per form
1,000
NA
1,000
1,000
24
Finished
Size
Binding
Paper Type

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Bar Coding
Ink Jet or
Laser
Packaging
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Individually
shrinkwrapped
Individually
shrinkwrapped
Shrink-wrapped
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Page 2 of 10

EOC Math
Program
Contract
Year
Material Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
400
400
NA
24
Black
EOC Math
2,3,4,5,6
Fall Geometry Answer

Book Large-Print
EOC Math
2,3,4,5,6
Fall Geometry Test Book
1,100
1,100
EOC Math
2,3,4,5,6

- Braille Contracted
40
40
EOC Math
2,3,4,5,6

- Braille Uncontracted
10
EOC Math
2,3,4,5,6

- Large-Print
EOC Math
2,3,4,5,6
EOC Math

- OIPP Version
Winter Algebra I Answer
Book
EOC Math
EOC Math
Binding
Paper Type

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Bar Coding
Ink Jet or
Laser
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Packaging
Shrink-wrapped
booklets
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
3,300
3,300
3,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA

Book
4,400
4,400
4,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
4,5,6

Book
4,400
4,400
4,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
EOC Math
2,3,4,5,6

Book Large-Print
400
400
Color Cover,
Dropout and Black
Ink on Internal
pages
Color Cover,
Dropout and Black
Ink on Internal
pages
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
S Barcode
Label
S Barcode
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
EOC Math
Winter Algebra I Test

Book
3,300
3,300
3,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Shrink-wrapped
booklets
EOC Math

Book
4,400
4,400
4,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math
4,5,6

Book
4,400
4,400
4,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math

Book - Braille Contracted
40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math

40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
4,5,6

40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
December 2013
1,000
Finished
Size
NA
24
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Page 3 of 10

EOC Math
Program
Contract
Year
EOC Math

Book - Braille
Uncontracted
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
10
10
NA
72
N/A
EOC Math

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
4,5,6

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
4,5,6

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
4,5,6
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
2,3,4,5,6

Book - OIPP Version
Book - OIPP Version
Book - OIPP Version
Winter Algebra II Answer
Book
1,100
1,100
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
EOC Math
2,3,4,5,6
Winter Algebra II Answer

Book Large-Print
400
400
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
S Barcode
Label
S Barcode
Label
S Barcode
Label
S Barcode
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
EOC Math
2,3,4,5,6
Winter Algebra II Test

Book
1,100
1,100
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math
2,3,4,5,6

40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
2,3,4,5,6

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
2,3,4,5,6

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
2,3,4,5,6
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
2,3,4,5,6

Book - OIPP Version
Winter Geometry Answer
Book
1,100
1,100
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
EOC Math
2,3,4,5,6
Winter Geometry Answer

Book Large-Print
400
400
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
S Barcode
Label
S Barcode
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
December 2013
Material Type
# of Forms
(versions)
Quantity
per form
1,000
NA
1,000
1,000
NA
24
24
Finished
Size
Binding
Paper Type

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Bar Coding
Ink Jet or
Laser
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
NA
S Barcode
Label
Packaging
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Individually
shrinkwrapped
Individually
shrinkwrapped
Shrink-wrapped
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Shrink-wrapped
booklets
Page 4 of 10

EOC Math
Program
Contract
Year
EOC Math
2,3,4,5,6
Winter Geometry Test

Book
EOC Math
2,3,4,5,6

40
40
NA
64
EOC Math
2,3,4,5,6

Book - Braille
Uncontracted
10
10
NA
EOC Math
2,3,4,5,6

Book - Large-Print
400
400
EOC Math
2,3,4,5,6
25
25
EOC Math

Book - OIPP Version
Spring Algebra I Answer
Book
11,000
EOC Math
2,3,4,5,6
Spring Algebra I Answer

Book
1,2,3,4,5,6 Spring Algebra I Answer

Book Large-Print
EOC Math
Material Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
1,100
1,100
1,000
48
Color Cover, Black
Ink on Internal
pages
Finished
Size
Binding
Paper Type

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Bar Coding
Ink Jet or
Laser
Packaging
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
11,000
10,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
14,300
14,300
13,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
400
400
NA
24
Color Cover,
Dropout and Black
Ink on Internal
pages
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
S Barcode
Label
S Barcode
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped
booklets
EOC Math
Spring Algebra I Test

Book
11,000
11,000
10,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math
2,3,4,5,6

Book
14,300
14,300
13,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math

40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
2,3,4,5,6

40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
2,3,4,5,6

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
2,3,4,5,6

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
2,3,4,5,6

Book - OIPP Version
Book - OIPP Version
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
S Barcode
Label
S Barcode
Label
Shrink-wrapped
script and reg test and
answer booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Individually
shrinkwrapped
December 2013
Page 5 of 10

EOC Math
Program
Contract
Year
Material Type
# of Forms
(versions)
EOC Math
Spring Algebra II Answer

Book
EOC Math
2,3,4,5,6
Spring Algebra II Answer

Book
1,2,3,4,5,6 Spring Algebra II Answer

Book Large-Print
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
11,000
11,000
10,000
16
Color Cover,
Dropout and Black
Ink on Internal
pages
12,100
12,100
11,000
16
Color Cover,
Dropout and Black
Ink on Internal
pages
400
400
NA
24
Black
EOC Math
Quantity
per form
Finished
Size
Binding
Paper Type

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Bar Coding
Ink Jet or
Laser
Packaging
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped
booklets
EOC Math
Spring Algebra II Test

Book
11,000
11,000
10,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math
2,3,4,5,6

Book
12,100
12,100
11,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math

40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
2,3,4,5,6

40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
2,3,4,5,6

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
2,3,4,5,6

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
2,3,4,5,6
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math

Book - OIPP Version
Book - OIPP Version
Spring Geometry Answer
Book
11,000
11,000
10,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
EOC Math
2,3,4,5,6
Spring Geometry Answer

Book
12,100
12,100
11,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
1,2,3,4,5,6 Spring Geometry Answer

Book Large-Print
400
400
NA
24
Color Cover,
Dropout and Black
Ink on Internal
pages
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
S Barcode
Label
S Barcode
Label
S Barcode
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Individually
shrinkwrapped
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
11,000
11,000
10,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Shrink-wrapped
booklets
EOC Math
EOC Math
December 2013
Spring Geometry Test

Book
Page 6 of 10

EOC Math
Program
Contract
Year
EOC Math
2,3,4,5,6
EOC Math
Material Type
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
12,100
12,100
11,000
48
Color Cover, Black
Ink on Internal
pages

Book

40
40
NA
64
EOC Math
2,3,4,5,6

40
40
NA
EOC Math

Book - Braille
Uncontracted
10
10
EOC Math
2,3,4,5,6

Book - Braille
Uncontracted
10
EOC Math

Book - Large-Print
EOC Math
2,3,4,5,6

Book - Large-Print
EOC Math
EOC Math
2,3,4,5,6
EOC Math
EOC Math
EOC Math
EOC Math
Finished
Size
Binding
Paper Type

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Bar Coding
Ink Jet or
Laser
Packaging
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label

Book - OIPP Version
Book - OIPP Version
Summer Algebra I
Answer Book
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
5,500
5,500
5,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
Summer Algebra I
Answer Book
7,700
7,700
7,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
3,4,5,6
Summer Algebra I
Answer Book
8,800
8,800
8,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
400
400
Color Cover,
Dropout and Black
Ink on Internal
pages
Color Cover,
Dropout and Black
Ink on Internal
pages
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
S Barcode
Label
S Barcode
Label
S Barcode
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Individually
shrinkwrapped
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped
booklets
1,2,3,4,5,6 Summer Algebra I

NA
24
EOC Math
Summer Algebra I Test

Book
5,500
5,500
5,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math

Book
7,700
7,700
7,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math
3,4,5,6

Book
8,800
8,800
8,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math

40
40
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
Label
Shrink-wrapped
script and reg test and
answer booklets
December 2013
NA
NA
Page 7 of 10

EOC Math
Program
Contract
Year
Material Type
# of Forms
(versions)
EOC Math

EOC Math
3,4,5,6
EOC Math
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label

40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
3,4,5,6

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
3,4,5,6

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
3,4,5,6
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math

Book - OIPP Version
Book - OIPP Version
Book - OIPP Version
Summer Algebra II
Answer Book
3,300
3,300
3,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
EOC Math
2,3,4,5,6
Summer Algebra II
Answer Book
3,300
3,300
3,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
400
400
Color Cover,
Dropout and Black
Ink on Internal
pages
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
S Barcode
Label
S Barcode
Label
S Barcode
Label
S Barcode
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped
booklets
EOC Math
1,2,3,4,5,6 Summer Algebra II

Quantity
per form
NA
24
Finished
Size
Binding
Paper Type

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Bar Coding
Ink Jet or
Laser
Packaging
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Individually
shrinkwrapped
Individually
shrinkwrapped
EOC Math
Summer Algebra II Test

Book
3,300
3,300
3,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math
2,3,4,5,6

Book
3,300
3,300
3,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math

40
40
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
Label
Shrink-wrapped
reg test and answer
booklets
December 2013
NA
NA
Page 8 of 10

EOC Math
Program
Contract
Year
Material Type
# of Forms
(versions)
EOC Math
2,3,4,5,6

EOC Math
EOC Math
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
2,3,4,5,6

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
2,3,4,5,6

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
EOC Math
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math
2,3,4,5,6
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
EOC Math

Book - OIPP Version
Book - OIPP Version
Summer Geometry
Answer Book
3,300
3,300
3,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
EOC Math
2,3,4,5,6
Summer Geometry
Answer Book
3,300
3,300
3,000
16
8 1/2 x 11
60# White
Reflective Read
Image Scan NA
Yes
NA
NA
S Barcode
400
400
Color Cover,
Dropout and Black
Ink on Internal
pages
Color Cover,
Dropout and Black
Ink on Internal
pages
Black
S Barcode
Label
S Barcode
Label
S Barcode
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped
booklets
EOC Math
1,2,3,4,5,6 Summer Geometry

Quantity
per form
NA
24
Finished
Size
Binding
Paper Type

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Bar Coding
Ink Jet or
Laser
Packaging
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
Shrink-wrapped
booklets
Individually
shrinkwrapped
Individually
shrinkwrapped
EOC Math
Summer Geometry Test

Book
3,300
3,300
3,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math
2,3,4,5,6

Book
3,300
3,300
3,000
48
Color Cover, Black

Ink on Internal
pages
8 1/2 x 11
50# White
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
EOC Math

40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
2,3,4,5,6

40
40
NA
64
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math
2,3,4,5,6

Book - Braille
Uncontracted
10
10
NA
72
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
EOC Math

Book - Large-Print
400
400
NA
56
Black
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
reg test and answer
booklets
Shrink-wrapped
booklets
December 2013
S Barcode
Label
Page 9 of 10

EOC Math
Program
Contract
Year
EOC Math
2,3,4,5,6
EOC Math
EOC Math
EOC Math
EOC Math
EOC Math
EOC Math
Material Type

Book - Large-Print
# of Forms
(versions)
Quantity
per form
Total
Total
8.5 x 11
Ink Colors
Quantity
Quantity
Pages Incl
(for printing/
(for
Cover
scoring)
400
400
NA
56
Black
Finished
Size
Binding
Paper Type

Image or
s/ Corner
Serial #s
nonscan
Cuts
Sealing
Spiraling
Security
Bar Coding
Ink Jet or
Laser
11 x 17
Spiral on 17" side
Color stock 65#

Cover, 50# white
Internal pages
Nonscannable
NA
NA
Two interior NA
security
seals
S Barcode
Label
S Barcode
Label
S Barcode
Label
S Barcode
Label
Packaging
Shrink-wrapped
booklets
Individually
shrinkwrapped
Individually
shrinkwrapped
Shrink-wrapped
individually

Book - OIPP Version
2,3,4,5,6 Summer Geometry Test
Book - OIPP Version
1,2,3,4,5,6 Spring Sample Test
Materials (STM) Test
Booklets - Large-Print
Version
1,2,3,4,5,6 Spring Sample Test
Booklets - Braille Version
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
25
25
NA
60
Black
8.5 X 11
3-ring binder
60# White Off-set
Nonscan
NA
NA
NA
NA
400
1,200
NA
24
Black
11 x 17
Spiral on 17" side
65# Cover, 50#

NA
NA
NA
NA
50
150
NA
32
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrink-wrapped
individually
1,2,3,4,5,6 Summer Sample Test

Version
1,2,3,4,5,6 Summer Sample Test
400
1,200
NA
24
Black
11 x 17
Spiral on 17" side
65# Cover, 50#

NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped
individually
50
150
NA
32
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrink-wrapped
individually
Fall Sample Test

Version
Fall Sample Test
400
1,200
NA
24
Black
11 x 17
Spiral on 17" side
65# Cover, 50#

NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped
individually
50
150
NA
32
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrink-wrapped
individually
Winter Sample Test

Version
Winter Sample Test
400
1,200
NA
24
Black
11 x 17
Spiral on 17" side
65# Cover, 50#

NA
NA
NA
NA
S Barcode
Label
Shrink-wrapped
individually
50
150
NA
32
N/A
11 x 11.5
Plastic comb
80# white
Nonscannable
NA
NA
NA
NA
Label
Shrink-wrapped
individually
Spring Math Reference

Sheets - Large-Print
Version
Spring Math Reference
Sheets - Braille Version
Summer Math Reference
Version
Summer Math Reference
400
1200
NA
Black
11 x 17
NA
50# white
Nonscannable
NA
NA
NA
NA
NA
Shrink-wrapped in
packs of 20
50
150
NA
N/A
11 x 11.5
NA
80# white
NA
NA
NA
NA
NA
400
1200
NA
Black
11 x 17
NA
50# white
Nonscannable
Nonscannable
NA
NA
NA
NA
NA
Shrink-wrapped
individually
Shrink-wrapped in
packs of 20
50
150
NA
N/A
11 x 11.5
NA
80# white
Nonscannable
NA
NA
NA
NA
NA
Shrink-wrapped
individually
Fall Math Reference

Version
Fall Math Reference
Winter Math Reference
Version
Winter Math Reference
400
1200
NA
Black
11 x 17
NA
50# white
Nonscannable
NA
NA
NA
NA
NA
Shrink-wrapped in
packs of 20
50
150
NA
N/A
11 x 11.5
NA
80# white
NA
NA
NA
NA
NA
400
1200
NA
Black
11 x 17
NA
50# white
Nonscannable
Nonscannable
NA
NA
NA
NA
NA
Shrink-wrapped
individually
Shrink-wrapped in
packs of 20
50
150
NA
N/A
11 x 11.5
NA
80# white
Nonscannable
NA
NA
NA
NA
NA
EOC Math
2,3,4,5,6
EOC Math
2,3,4,5,6
EOC Math
2,3,4,5,6
EOC Math
2,3,4,5,6
EOC Math
1,2,3,4,5,6
EOC Math
1,2,3,4,5,6
EOC Math
1,2,3,4,5,6
EOC Math
1,2,3,4,5,6
EOC Math
1,2,3,4,5,6
EOC Math
1,2,3,4,5,6
EOC Math
1,2,3,4,5,6
EOC Math
1,2,3,4,5,6
December 2013
Shrink-wrapped
individually
Page 10 of 10
Appendix I:
Sample Issues Log

2*3*.1
0*/.*.1

.
/
$
# '
$
# '

%
!!#!
'
+
"

'
+
"
)! "

"!"
!#"%
"

!""!#!!"
"$ "
%"'!"
"!%%
&) "! "!
""$"
# "'
!
#(
!# !

%
'&
&
(

*
*

#
0*/.)!
"
"!!"'

"# ""
% "
""!"
%"
!"
#
"'
"
!#
2*/-*.1

!%
"#
"

$
!

!
!
"
%

!$
!
!
!
!

2*4*.1
Contract #14-652, Attachment I

Price Response
Price Reply
The American Institutes for Research (AIR) has based our proposal and fixed prices on the terms,
conditions, and schedule of the Florida Standards Assessments, ITN 2014-47, as amended by:
Addendum #1 issued on or about October 31, 2013
Addendum #2 issued on or about November 12, 2013
Addendum #3 issued on or about November 18, 2013
Excel spreadsheet version of 7.8 Deliverables received from Fran Shewan
Assumptions Underlying Our Fixed Prices

We have documented our assumptions throughout this proposal and have relied on the Q&As in
every case where they clarify or further define the requirements of the ITN. We have also based our
proposal on the following assumptions:
AIR is proposing to leave Florida with a completely Florida-developed and Floridaowned assessment at the end of the contract.
AIR is proposing licensing of Utah Student Assessment of Growth and Excellence

(SAGE) and/or the Smarter Balanced Assessment Consortium item banks to enable
Florida to build and administer an assessment beginning in the 201415 school year.
This is discussed in more detail under Licenses below.
AIR is proposing significant new item development to allow Florida to build an

assessment that does not contain any SAGE or Smarter Balanced licensed materials.
Florida will have approved, and will own, all of these items.
AIR will provide copyright permissions for any third-party passages for the length of the
contract.
AIR and Florida will agree on terms whereby AIR will retain ownership of all software
and other AIR intellectual property (IP), as well as any derivations of this IP, developed
before entering into the contract.
AIR understands the Department reserves the right to request alternative intellectual
property arrangements during negotiations in order to obtain a reply that provides the best
value to the state.
The format and general content for the online score reporting, data file formats, and other
reports and documents will be finalized with the first operational use.
In the event of a reduction or change in the scope of work or partial termination for
convenience, AIR assumes that any adjustment in contract price will account for potential
Page 1
increases in the price of the remaining tasks occasioned by such a reduction or change in the
scope of work.
Any conflicts between the provided documents (e.g., the ITN, Terms and Conditions, and
Proposal) will be resolved during negotiations. We assume that the final contract will
include an order of precedence in which AIRs proposal is read together with the ITN and
Q&As to determine the contract requirements.
Section 5.19, Liquidated Damages, refers to interim deliverables, final deliverables, critical
work tasks, and extremely critical work tasks. We did not identify a clear designation of
interim and final deliverables in Section 7.8 of the ITN, nor a clear relationship between the
critical work tasks in Appendix B of the ITN and the deliverables on Section 7.8 of the ITN.
We assume we will clarify a final list of tasks subject to liquidated damages as part of the
contract, with the specific amounts associated, during negotiations. AIR agrees to the 10%
cap on liquidated damages proposed by the Department in Section 5.19.c, and we assume
the total contract amount in this section means the total contract amount for that year.
Per Section 5.15 of the ITN, Performance Bond, AIR will deliver the required performance
bond within 10 calendar days prior to beginning work under the contract. We assume this
section means a one-year bond (or adjusted period) for 10% of the total annual contract
amount and that the Department will return the prior-year bond as AIR delivers each new
annual bond.
We do not believe any of these assumptions represent exceptions to the Terms and Conditions of the
ITN.
Specific Deliverable Assumptions
Per Section 7.2.1, we have based our quantities of materials on the expected number of
students, shipping destinations, and schools provided in Appendix C, with the required 10%
overage. We based our handscoring estimates on these actual quantities before materials
overage.
In Section 7.3.7.3, AIR has not proposed to exclude performance tasks but rather to base
retakes on machine-scored items to support immediate scoring and reporting. As an option,
we could extend this capability to the essays in the retake writing assessment. As a further
option, we could offer either 100% handscoring as a second read for all machine-scored
items on the retakes or a less expensive verification of machine scores on a sample (e.g., 10
20%) of student responses.
In Sections 7.3.11.2.5 and 7.3.11.2.5.6, we have included functionality to support Englishto-Heritage Language Translation Dictionary through glossaries that can be associated with
individual words or phrases in a question or passage and to deliver sign language by
displaying videos of people signing. Each of these functions depends on item-level tagging
that is not included (or required) in the base scope of this ITN.
We assume the requirement to support simultaneous online users under Section 7.4.9.2 of
the ITN refers to the 50% of the current membership eligible to test rather than the overall
membership.
Page 2
In Section 7.6.2.1, we have included $40,000 per year in our budget to engage an
independent subcontractor for SES replication and will work with Florida to select the
appropriate individual or organization and review and approve the subcontract scope of
work.
Licenses
Our budget includes the license to use the Utah SAGE items for all years of census paper testing and
for the first three years of online testing in each grade/subject and the End-of-Course (EOC)
assessment. After that point for each assessment, AIR will have developed sufficient items so that
Florida will have transitioned to a completely Florida-developed and Florida-owned assessment.
The Utah State Office of Education (USOE) has agreed to license these items to Florida under one of
two models:
Florida may select one or more fixed forms from the bank for a license fee of $2 per student
per form. Hence, if Florida chooses to select one form per grade/subject, the license fee will
be $4 per student. This is the license in our proposed budget. As an option to this model, but
not required in the ITN, Florida could choose to select a primary form and a breach form for
a license fee of $8 per student.
Florida may elect to participate in Utahs adaptive version of the test, using Utahs
blueprint, at a license fee of $2 per test administered.
Our budget assumes the first model, without the breach form.
USOE recognizes that Florida may need to, or may unintentionally, release items. In this case,
USOE has agreed to allow release of the items selected by Florida for a price of $5,000 per released
item to reimburse USOE for the cost of replacing these items in the pool.
Our budget also includes the costs for a license for the summative Smarter Balanced assessments to
embed sufficient items during Year 1 to link the Florida Standards Assessments (FSA) to the
Smarter Balanced scale and for the integration of the Programme for International Assessment
(PISA) items to link FSA to the PISA scale. As an option, Florida could chose to use the Smarter
Balanced assessments instead of the SAGE assessments as the base assessment for the FSA. If
Florida is interested in this option, we would be happy to discuss costs during negotiations.
Completing Exhibit 7.8

Pricing by Component
AIR has submitted prices for each of the four components (A, B, C, and D) and for the overall
contract (Component E).
In pricing by individual Components AD, the sum of the component prices are much higher than
our combined price (Component E). The Department will recognize there are significant initial setup
costs for administering the first online or the first paper-and-pencil test as well as for some overall
project management and other costs (e.g., Technical Advisory Committee support) that do not vary
with the number of subjects or tests. Some tasks (e.g., studies) are virtually duplicated across
components. Similarly, there are savings from managing multiple programs together. Therefore,
Page 3
adding any combination of Components AD would result in an incorrect high price.

If the Department is interested in some other combination other than any one or all four components
(e.g., grades 38 English Language Arts/Literacy (ELA/L) plus grades 38 Mathematics), we would
be happy to provide such pricing.
Matching ITN Requirements to Specific Deliverables

Spreading our costs across all of the specific deliverables in Exhibit 7.8 is not an exact science.
Many deliverables are interrelated and/or are completed in combination with other deliverables. In
addition, we have found that clients prefer to make larger payments when specific milestone tasks
are accomplished, such as the delivery of items, test materials, score reports, and technical reports.
We have placed a price in virtually every cell of the pricing sheet, and we tried to match all of the
ITN requirements to specific deliverables on Exhibit 7.8. We made assumptions where the match
was not entirely clear to us. Some examples of these assumptions include:
We were not clear where to place costs for new item development in Section 7.3.3 of the
ITN. We placed these costs in the deliverable titled Periodic: Field Test Item Sets and
Forms Review in Year 1 and the deliverable titled Periodic: Content Item Review
Cleans in Years 26. We also placed the licensing fee for the Utah SAGE and Smarter
Balanced item banks in these lines. Also, AIR proposes to develop items in Year 1, so we
included the Item Development Plan and Item Writer Training costs in the Field Test Item
Sets and Forms Review deliverable for Year 1.
Exhibit 7.8 did not include a line for training development and delivery before August 2015.
We included the costs to develop and deliver training in advance of spring and summer 2015
testing under the deliverable titled Training Plan for Year 1.
Test construction is broken across three deliverables on the Exhibit 7.8 Pricing Schedule:
Test Development: Periodic: Field Test Item Sets and Forms Review
Test Development: Periodic: Onsite Test Construction (Meeting)
Scaling, Equating, Scoring, and Special Psychometric Studies: Periodic: Development of

Test Forms for Field Test and/or Operational Administrations
AIR has proposed a significant number of machine-scored constructed-response items and

has included associated rubric validation activities under the deliverable titled Handscoring
of Performance Tasks.
We have assumed that student reports (two printed copies) will contain the performance task
reports within the four pages AIR has budgeted per report and we have included those costs
together in the deliverable titled Initial Reporting Student Reports and CD/DVDs or Jump
Drives.
We understand we must convene the Florida Technical Advisory Committee (TAC) in

Section 7.6 and have placed these associated costs under the deliverable titled Scaling,
Equating, Scoring, and Special Psychometric Studies: Creating and Finalizing Technical
Page 4
Reports. We have assumed the payments to and travel for TAC members will be covered
under reimbursable expenses.
As indicated in ITN Section 7.3.6, there is no place for a price for a stand-alone ELA/L field
test on the Exhibit 7.8 Pricing Schedule, and thus we did not include this cost. Per the ITN,
if it is determined during negotiations that a stand-alone field test will take place, all test
administration and scoring schedules and deliverables will be agreed upon prior to contract
execution.
We have proposed standard setting for Year 1 (2014), but Exhibit 7.8 includes places for
this cost in Year 2. Similarly, the linking study may also occur sooner than shown in the
deliverable schedule. We would expect we would determine the final placement of these
costs during negotiations.
Within our total proposed annual prices, we will be happy to move costs between line items if any of
our assumptions are incorrect and/or to negotiate with the Department on others changes to specific
payments.
Printing Exhibit 7.8

As directed by the ITN instructions, we have placed our fixed prices in Exhibit 7.8, ITN 2014-47
Attachment 1b Deliverables Pricing. However, we note both the PDF and Excel versions of this
form provided by the Department print on legal-sized paper, which conflicts with the ITN Section
6.2.4 requirement to deliver the proposal on letter-sized paper. Considering this conflict, we printed
Exhibit 7.8 directly from the Excel file provided by the Department (i.e., on legal-sized paper). If
this was incorrect, we would be happy to redeliver Exhibit 7.8 in letter size (obviously with no
changes to the prices). Please note we provided the PDF versions in both letter and legal size, along
with the completed Excel version.
As directed by the ITN, we have placed our fixed prices in the deliverables column of Exhibit 7.8.
Rather than printing additional copies of each component, we added columns for Components AE.
We hope the reviewers find this easier to review our line item and total prices across components.
Page 5

Specify Component A, B, C, D or E (A-D): _____A_____
A
1
3
4
5
6
7
8
Test Development
Document
Test Development

Specifications
9
10
11
12
13
C
7.3.10
via SFTP.
Due Date(s)
June 1, 2014
June 1, 2015
June 1, 2016
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5
F
Component A
3-8 ELA
$38,237.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
14
Test Development
15
16
17
18
19


7.5.4.5

SFTP.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

7.5.4.5

SFTP.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
20
Test Development
21
22
23
24
25

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$20,191.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
Submitted by AIR 12/13/13
Test Development
A
1

B
The contractor should incorporate the handscoring (1) Provide Department email reflecting approval of
Specify Component A, B, C, D or
E (A-D): _____A_____
specifications will be a detailed
guide to conducting SFTP.
C
D
E
handscoring managers
andDescription
the Department. The
Deliverable
Due Date(s)
(Renewal 3 - 2020)
7.5.4.5
F
Component A
3-8 ELA
$2,000.00
26
Test Development
27
28
29
30
31

7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
Spring EOC: Handscoring Specifications The contractor should incorporate the handscoring
7.5.4.5

SFTP.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
32
Test Development
33
34
35
36
37
$0.00
$0.00
$0.00
$0.00
$0.00
38
Test Development
39
40
41
42
43
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
44

A
1
Test Development
45
46
47
48
49
50
51 Test Development
52
53
54
55
56
B
Specifications
C
7.5.4.5
D
SFTP.
7.3.10
Test Development


Test Development

each scoring guide.
educators understand the scoring of constructed- (2) Final PDF files provided via SFTP.
response items included on student reports. These
F
Component A
3-8 ELA
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
June 1, 2014
June 1, 2015
June 1, 2016
(1) Provide Department email reflecting approval of November 1, 2015
$15,797.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$35,812.00
57
58
59
60
61
62

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$31,444.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
Test Development
A
1

B
each scoring guide.
Attachment
1b - DELIVERABLES
PRICING
educators understand the scoring
of constructed(2) Final PDF
files provided via SFTP.
Specify
Component
A, These
B, C, D or E (A-D): _____A_____
response items included
on student
reports.
C
D
E
Due Date(s)
(Renewal
3
2020)
F
Component A
3-8 ELA
$2,000.00
63
Test Development

Assessment Reports
64
65
66
67
68
This publication describes the various reports of

test results distributed to students, schools, and
Reference 7.3.7.5

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$67,402.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
69
Test Development
70
71 Test Development
72
73
74
75
76 Test Development

(1) Provide Department email reflecting approval of June 1, 2015

$147,745.00

management report
Reference 7.3.4.2

February 1, 2017
management report
$3,956,087.00
$4,036,987.00
$2,137,072.00
$1,867,100.00
$1,584,816.00
Review
(1) Provide Department email reflecting completion May 1, 2015

$4,034,877.00

management report

A
Development
1 Test Deliverable
Category
77
78
79
80
81 Test Development
82
83
84
85
86
87 Test Development
88
89
90
91 Test Development
92
93
94
95 Test Development
96
97
98
99
Periodic:Contractor
Field Test Item
Sets and Forms Whether an embedded
or Description
stand-alone model is
Deliverable
Deliverable
Review
(1) Provide Department

email
reflecting completion
Evidence of
Completion
Due Date(s)
May 1, 2016
management report
1) Provide Department email reflecting completion July 15, 2014
of the test construction process
July 15, 2015
July 15, 2016
grades/courses and subjects. To the extent possible, (2) Appearance of completed schedule in weekly
the Department expects to be involved in any test management report
F
Component A
3-8 ELA
$32,066.00
$32,066.00
$32,066.00
$32,066.00
$33,668.00
$48,762.00
$48,762.00
$48,762.00
$48,762.00
$48,762.00

Reference 7.3.0
1) Provide Department email reflecting completion

of the Item Development Plan.
management report.
December 1, 2015
December 1, 2016
$23,304.00
$23,304.00
$23,304.00
$23,304.00

The contractor, in conjunction with the
Department, will conduct an annual training session
for all Item Writers. ITN Reference 7.3.3

of Item Writer Training.
management report.
December 1, 2015
December 1, 2016
$19,359.00
$9,109.00
$9,109.00
$9,109.00


previous invoice.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$111,390.00
$112,882.00
$114,410.00
$117,566.00
$119,720.00
Test Development
A
1
100
102
103
104
105
106
108
109
110
111
112
Test Administration
B

Component
A, B, C, D or
E (A-D):
_____A_____
convening meetingsSpecify
for the purpose
of reviewing
previous
invoice.
assessment items and otherwise
supporting the
(2) Provide the meeting costs,
C
D a final agenda, and a




Once all materials are scanned in and inventoried,

contractors facilities, the contractor provides a
final report listing any outstanding missing
materials. ITN Reference 7.4.2
113
114
115
116
Due Date(s)
Component A
3-8 ELA
(Renewal 3 - 2020)
$120,058.00

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
(1) Provide evidence of Department acceptance and (TBD - Part of negotiation) 2015
approval (email is acceptable) of the English-to-Sign 2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1) Final missing materials report.
2) Provide Department email reflecting approval of administration dates. Finalize during negotiations.
the report.
2015
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00

2016
$0.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
$0.00
$0.00
$0.00
117
Test Administration
118
119
120
121

the report.
Test Administration
A
1

B
1b - DELIVERABLES
return any secure materialsAttachment
identified as missing.
the report.PRICING
Specify
Component
A, B,
After a thorough search
has been
conducted
by C, D or E (A-D): _____A_____
C
D
Due Date(s)
Component A
3-8 ELA
(Renewal 3-2020)
$0.00
122
Test Administration
123
124
125
126
127

a report.
the report.

2015
$2,000.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

the report.

2015
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
the report.

2015
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
128
Test Administration
129
130
131
132
133
Test Administration
134
135
136
137
138

A
1
Test Administration
139
140
141
142
143
F
Component A
3-8 ELA
the report.
Due Date(s)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00

the report.

2015
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00
the Specifications.

2015.
$0.00
$0.00
$0.00
144
Test Administration
145
146
147
148
149
150
152
153
154
155
156

2016
$0.00
(Renewal 1- 2017)
$0.00
(Renewal 2-2018)
$0.00
(Renewal 3 - 2019)
$0.00
$0.00
and distribution specifications for each requirements for the packaging and distribution of (2) Provide Department email reflecting approval of packaging requirements of the selected contractor
the Specifications.
2016

A
1
Test Deliverable
Administration
Category
158
159
160
161
162
164
165
166
167
168
169
171
172
173
174
175
Spring ELA/L
Retake:Deliverable
Develop packaging A document detailing
all processes,
procedures, and (1) Digital copyEvidence
of the approved
specifications.
Contractor
Deliverable
Description
of Completion
the Specifications.
E
Prior to each test administration,
Due Date(s) TBD based on
2016
F
Component A
3-8 ELA
$0.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
packaging and distribution
requirements for the packaging and distribution of (2) Provide Department email reflecting approval of packaging requirements of the selected contractor
specifications for each test
the Specifications.
administration.
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$79,909.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
$81,640.00
$67,932.00
$72,745.00
$75,477.00
$76,384.00
$0.00
$0.00

distribution specifications for each test requirements for the packaging and distribution of (2)Provide Department email reflecting approval of
administration.
the Specifications.
2016
$0.00
(Renewal 1- 2017)
$0.00
(Renewal 2-2018)
$0.00
(Renewal 3 - 2019)
$0.00
$0.00
distribution specifications for each test requirements for the packaging and distribution of (2) Provide Department email reflecting approval of packaging requirements of the selected contractor
administration.
the Specifications.
2015.

A
1
Test Deliverable
Administration
Category
177
178
179
180
181
183
184
185
186
187
188
190
191
192
193
194
195
Winter EOC:
DevelopDeliverable
packaging and
A document detailing
all processes,
of the approved
specifications.
Contractor
Deliverable
Description
of Completion
distribution specifications for each test requirements for the packaging and distribution of (2) Provide Department email reflecting approval of
administration.
the Specifications.

2015.
Component A
3-8 ELA
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
administration.
the Specifications.
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
administration.
the Specifications.
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
197
198
199
200
201
Test Administration
202
203
204
205
206
207
208
Test Administration
209
210
211
212
213
214
215
Test Administration
216
217
218
219
220
221
222
B
memos, etc.
districts.

memos.


districts.

memos.


districts.

memos.


districts.
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$257,340.00
$305,950.00
$350,882.00
$377,733.00
$389,093.00
$390,647.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$259,914.00
$227,780.00
$219,031.00
$184,386.00
$190,454.00
$190,756.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
224
225
226
227
228
Test Administration
229
230
231
232
233
234
Test Administration
235
236
237
238
239
240
241
Test Administration
242
243
244
245
246
247
and accommodations.
ancillary materials must be successfully developed
and delivered to districts by the critical or extremely
critical date listed in Appendix B. This includes
paper-based accommodations for computer-based
assessments. ITN Reference 7.3.8, 7.4.1, and 7.4.7
districts.

and accommodations.


districts.



districts.

accommodations.


districts.
Due Date(s)
negotiations.)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$2,388,868.00
$1,457,411.00
$652,271.00
$694,605.00
$717,158.00
$720,334.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
248
249
250
251
252
253
Test Administration
254
255
256
257
258
259
260
Test Administration
261
262
263
264
265
266
267
Test Administration
268
269
270
271
272
273
accommodations.
districts.

accommodations.


districts.

accommodations.


districts.


and 7.4.7.4

districts.
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
B
districts.

and 7.4.7.4

districts.
and rosters.
and 7.4.7.4

districts.

labels and rosters.

and 7.4.7.4

districts.

and 7.4.7.4

districts.

280
281
282
283
284
285
286
Test Administration
287
288
289
290
291
292
Test Administration
293
294
295
296
297
298
Test Administration
299
and 7.4.7.4
274
275
276
277
278
279
Test Administration

labels and rosters.
Due Date(s)
negotiations.)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Test Administration
A
1
300
301
302
303
304
305
Test Administration
306
307
308
309
310
311
312
314
315
316
317
318
320
321
322
323
324
326
327
Spring EOC: Deliver B

preidentification
labels and rosters.

labels and rosters.
For each administration, all

C preidentification labels
districts by theDeliverable
critical or extremely
critical date
Description
and 7.4.7.4
(1) Provide proof of delivery

D spreadsheet listing the
who received materials,
destination
address,
Evidence of
Completion
districts.

and 7.4.7.4

districts.
Due Date(s)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
7.4.8

November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00


7.4.8

May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00


7.4.8

(or no) malfunctions or unavailability of the site that May 2015
May 2016
$0.00
$139,096.00
$262,891.00

Test Administration
A
1
328
329
330
331
333
334
335
336
337
339
340
341
342
343
345
346
347
348
349
350
352
353
354
355

Deliver
B
system for
each assessment.
Contractor
Deliverable
Provide evidence that theConline management

was operable for
the invoice
period. ITN Reference
Deliverable
Description
7.4.8
(1) Provide a report listingDmet requirements and any

E
occurred during
the period,
signed by the
Evidence
of Completion
Due Date(s)
Department team. Delivered at the initiation of the May 2017

assessment.

7.4.8

November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

February 2016
February 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
Summer EOC: Deliver and maintain the Provide evidence that the online management
assessment.
7.4.8

August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
F
Component A
3-8 ELA
$262,891.00
$262,891.00
$262,891.00
$262,891.00
Test Administration
A
1
356
Specify Component
A, B,
C, D or
(A-D):
_____A_____
system met the requirements
of the contract
and
(orEno)
malfunctions
or unavailability of the site that
assessment.
was
operable
for
the
invoice
period.
ITN
Reference
occurred
during
the
period,
B
C
D signed by the
E
7.4.8
Due Date(s)
F
Component A
3-8 ELA
$0.00
$0.00
357
359
360
361
362



November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
363
365
366
367
368

May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
369
371
372
373
374
375

administration.


May 2015
May 2016
May 2017
$0.00
$3,749,054.00
$4,294,013.00
$5,097,356.00
$5,602,608.00
$5,506,854.00
$5,331,649.00
376
378
379
380
381



November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
Test Administration
A
1
382
384
385
386
387
388
390
391
392
393
394
395
397
398
399
400
401
402
404
405
406
407
408

B
Attachment 1b - DELIVERABLES
(1) ProvidePRICING
a report listing met requirements and any
A,ofB,the
C, D or
(A-D):
_____A_____
assessment platformSpecify
met theComponent
requirements
(orEno)
malfunctions
C
D
E
Due Date(s)









PERIODIC: Develop and deliver

approved practice tests and tutorials for
the computer-based and paper-based

administration. All practice tests (computer-based
and paper-based) must contain 10-20 sample items.
For the paper-based practice materials, there
should be a separate test booklet and a separate
answer key booklet for teachers. These paper-based
practice materials are produced in ADA-compliant
PDF files for posting. Practice materials must also
be produced for all accommodated versions

and subjects, and computer-based practice tests for
all grades and subjects that will be offered as
computer-based subjects the first year. As grades
practice tests must also be released. This delivery
includes the annual delivery of requested paperbased accommodated versions, which are the only
F
Component A
3-8 ELA
$0.00
February 2016
February 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
May 2015
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
November 2014
November 2015
November 2016
$63,181.00
$3,000.00
$3,000.00
$3,000.00
$3,000.00
$3,000.00
Test Administration
A
1

B

Attachment
1bon
- DELIVERABLES
PRICING
materials must be developed
and approved
a
the practice
tests and tutorials that reflect the
Specifythat
Component
A, B, C, D or
E (A-D): _____A_____
timeline to be established
offers students
requirements
for each year of the assessment. For
C
D
answer key booklet for teachers. These paper-based in Appendix A), the appropriate computer-based
includes the annual delivery of requested paperbe produced for all accommodated versions
based accommodated versions, which are the only
(computer-based accommodations and paperversions that are shipped to districts. (All others are
based accommodations, such as braille and large
print). ITN Reference 7.3.7.5 and 7.4.9.6
Due Date(s)
Component A
3-8 ELA
$0.00
409
Test Administration
410
412
413
414
415
417
418
419
420
421
423
424
425
426
PERIODIC: Develop and deliver a

training plan for the initial training and
supply materials to support ongoing
training for the audiences identified in
the contract.


training.
September 2014
7.4.9.8
administration.
August 2015
August 2016
November 2016
November 2017
signed by the Department team. Invoice at the end (Renewal 1 - November 2018)
of each assessment year (end of Summer EOC
previous year.
November 2015
November 2016
November 2017
$242,481.00
$143,014.00
$143,014.00
$143,014.00
$143,014.00
$143,014.00
$537,928.00
$445,587.00
$456,410.00
$475,277.00
$486,865.00
$490,413.00
$45,445.00
$33,796.00
$16,362.00
$17,002.00
$17,389.00
Test Administration
A
1
PERIODIC: Inventory, store, and destroy The contractor will inventory and store unused test (1) Provide Department approval to destroy the
Specify Component
A, B, C,for
D or
E (A-D):and
_____A_____
and answer books, manuals,
and other materials
materials
evidence of destruction. Invoice at the
a
period
of
at
least
six
months,
and
used
documents
end
of
each
assessment
year
B
C
D (end of Summer EOC
E
Due Date(s)
F
Component A
3-8 ELA
$17,411.00
427
Test Administration
428
429
430
431
432
433
Test Administration
434
435
436
437
438
439
Test Administration
440
441
442
443
444
445
446
Test Administration
447
448
449
450
451

comment forms.

comment forms.

recommended actions, with summary of selected the comment form summaries and analyses.

negotiations.)
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
(1) Provide Department email reflecting approval of See Appendix B (Contingent upon final, approved
negotiations.)
2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
reports and recommendations from all recommended actions, with summary of selected the comment form summaries and analyses.
negotiations.)
2015

comment forms.
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
Test Administration
A
1
452
Test Administration
453
454
455
456
457
458
Test Administration
459
460
461
462
463
464
465
Test Administration
466
467
468
469
470
471
472
Test Administration

comment forms.
B
comment forms.

comment forms.

comment forms.

Trial Test

1b -forms.
DELIVERABLES PRICING
response questions from allAttachment
online comment
Component A, B, C, D or E (A-D): _____A_____
ITN Reference 7.4.1 Specify
and 7.4.7.7
C
Due Date(s)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Develop and make available an infrastructure test, (1) Provide Department email reflecting approval of August 1, 2014
consisting of mock items, that replicates the size of the final product and its availability. Product must be
platform (i.e., users log in to the testing platform as administration. Updates to the test may be
if they were logging into a live test). ITN References requested during the life of the contract, depending
7.4.8 and 7.4.9.5
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$58,275.00
473

A
1
Test Administration
474
475
476
477
478
479
Test Administration
480
481
482
483
484
485
Test Administration
486
487
488
489
490
491
492
Test Administration
493
494
495
496
497
498
B



versions)
F
Component A
3-8 ELA
Reference 7.3.11.2
Due Date(s)
negotiations.)

Reference 7.3.11.2
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Reference 7.3.11.2
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Reference 7.3.11.2
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$132,091.00
$78,071.00
$78,071.00
$78,071.00
$78,071.00
$78,071.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
499
500
501
502
503
504
Test Administration
505
506
507
508
509
510
511
Test Administration
512
513
514
515
516
517
518
Test Administration
519
520
521
522
523
524
versions)
Reference 7.3.11.2
D
Due Date(s)
negotiations.)
Component A
3-8 ELA
$0.00
versions)
Reference 7.3.11.2
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

versions)
Reference 7.3.11.2
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00


Reference 7.4.1.
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
525
526
527
528
529
530
Test Administration
531
532
533
534
535
536
537
Test Administration
538
539
540
541
542
543
Test Administration
544
545
546
547
548
549
Reference 7.4.1.
Due Date(s)
negotiations.)
testing
Reference 7.4.1.


negotiations.)
Reference 7.4.1.


Reference 7.4.1.
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$505,593.00
$286,840.00
$165,562.00
$158,857.00
$165,227.00
$167,286.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
B
550
551
552
553
554
555
556
Test Administration
557
558
559
560
561
562
563
565
566
572
573
574
576
577
578
Reference 7.4.1.
Reference 7.4.1.

567
568
569
571

ITN Reference 7.5.1
D
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016

management system.
(2) Provide Department email approval of the
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
April 2014
April 2015
April 2016

management system.


management system.

F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
September 2015
September 2016
$0.00
$0.00
$0.00

January 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
January 2017
A
1
579
581
582
583
584
585
587
588
589
590
592
593
594
595
597
598
599
600
601
603
604
605
606
607
609

districts to review. B

The Contractor will collect
monitor district
(1)EPreID
frequency
reports posted to the online
Specifyand
Component
A, B, C, D or
(A-D):
_____A_____
C reports of issues
Frequency Reports and the
Deliverable
Description
assessments. ITN
Reference
7.5.2
management system.
D reflecting receipt of
(2) Provide Department email
management system.
E
Due Date(s)
January 2015
January 2016
January 2017

review.

management system.
review.

management system.
Spring EOC: Development of PreID freq The Contractor will collect and monitor district
review.


Fall ELA/L Retake: Development of the The Contractor will work with the Department to
file layout and Full-State PreID data file create a PreID layout for each administration for
in .txt format
the district to use to upload student demographic
and testing information. The contractor will post
the layout to online management system. After
each PreID window closes, the contractor will
create and send to the Department a full state

September 2015
September 2016
October 2015
F
Component A
3-8 ELA
$0.00
$2,500.00
$1,250.00
$1,250.00
$1,250.00
$1,250.00
$1,250.00
$0.00
$0.00
$0.00
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

management system.

January 2015
January 2016
January 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

management system.

June 2015
June 2016
June 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Protocol (SFTP).

September 2015
September 2016

A
1
610
611
612
614
615
616
617
619
620
621
622
623
625
626
627
file layout
and Full-State
PreID data file create a PreIDDeliverable
layout for each
administration for
Contractor
Deliverable
Description
in .txt format
PreID file that includes a record for each student
and test assigned. ITN Reference 7.5.2

management system
andofSecure
File Transfer
Evidence
Completion
Protocol (SFTP).

the file layout and Full-State PreID data create a PreID layout for each administration for
file in .txt format

Protocol (SFTP).
January 2016

Development of the file layout and Full- create a PreID layout for each administration for

Protocol (SFTP).
January 2015
January 2016
January 2017
create a PreID layout for each administration for
format

Protocol (SFTP).
September 2015
Due Date(s)
January 2017
September 2016
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
A
1
628
630
631
632
633
635
636
637
638
639
641
642
643
644

Specify
Component
A, B,toC, D or
(A-D):
_____A_____
Fall EOC: Development of the file layout The Contractor will work
with
the Department
(1)EFinal
PreID
file layout posted to the online
and Full-State PreIDBdata file in .txt
format
create a PreID layout for each

C administration for
management system and Secure

File Transfer
D
Protocol (SFTP).

.txt format


Protocol (SFTP).
October 2015
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00

.txt format


Protocol (SFTP).
January 2015
January 2016
January 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

.txt format


Protocol (SFTP).
June 2015
June 2016
June 2017
$0.00
$0.00
$0.00
$0.00
$0.00
E
Due Date(s)
F
Component A
3-8 ELA
$0.00
A
1
645
647
648
649
650
652
653
654
655
657
658
659
660
661
663
664
665
666
668
669
670
671

.txt format

Attachment
1b - DELIVERABLES
PRICING
The Contractor will work with
the Department
to
(1) Final PreID
Component
A, B,
E (A-D): _____A_____
create a PreID layoutSpecify
for each
administration
forC, D or
management
system and Secure File Transfer
the district to use to upload student demographic Protocol (SFTP).
C contractor will post
D the SFTP.
and testing information. The
(2) PreID data file posted to
Deliverable
of Completion
each PreID window
closes,Description
the contractor will
and approval ofEvidence
the file layout
and data file.
The Contractor will use a Special Document

Tracking log to inventory and track the processing
of all special documents. The contractor will
provide the final transcription log to Department.
(1) Final transcription log posted to SFTP.

the special documents transcription log.

the special documents transcription logs Tracking log to inventory and track the processing

E
Due Date(s)

$0.00
$0.00
$0.00

May 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
May 2017
May 2015
May 2016
May 2017



Winter EOC: Development of the special The Contractor will use a Special Document

Spring EOC: Development of the special The Contractor will use a Special Document

$0.00
October 2015
October 2016

Development of the special documents Tracking log to inventory and track the processing
transcription logs
F
Component A
3-8 ELA

October 2015
October 2016
December 2015
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

May 2015
$0.00
$0.00

A
and Reporting
1 Scoring
Deliverable
Category
673
674
675
676
677
679
680
681
682
683
685
686
687
688
690
691
692
Spring EOC:
Development
of the special The ContractorDeliverable
will use a Special
Document
Contractor
Deliverable
Description
(1) Final transcription

logofposted
to SFTP.
Evidence
Completion




scoring and creation of the State Student Results
file during the editing process. ITN References 7.5.3,
7.5.6.2, and 7.5.6.4

scan/edit, and CBT extract data files, posted to
online management system and SFTP.


7.5.6.2, and 7.5.6.4

E
Due Date(s)
May 2016
May 2017
July 2015
July 2016
July 2017
October 2015
October 2016
March 2016
March 2017
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
A
1
693
695
696
697
698
699
701
702
703

Specify
A, B,toC, D or
(A-D):
_____A_____
The contractor will work
withComponent
the Department
(1)EFinal
approved
file layouts, early/full state

full-state scan/edit and
and CBT extract
B CBT extract data develop the layouts for scan/edit
C
7.5.6.2, and 7.5.6.4

data files

scan/edit, and CBT extractDdata files, posted to


7.5.6.2, and 7.5.6.4


7.5.6.2, and 7.5.6.4

E
Due Date(s)
March 2015
March 2016
March 2017
October 2015
October 2016
F
Component A
3-8 ELA
$0.00
$2,660,273.00
$1,660,242.00
$1,681,235.00
$393,406.00
$409,265.00
$414,159.00
$0.00
$0.00
$0.00
$0.00
A
1

scan/edit and CBT extract
data files:
B

Specify
A, B,toC, D or
(A-D):
_____A_____
withComponent
the Department
(1)EFinal
approved
develop the layouts for scan/edit
and CBT extract
C
7.5.6.2, and 7.5.6.4

E
Due Date(s)
F
Component A
3-8 ELA
$0.00
704
706
707
708


7.5.6.2, and 7.5.6.4

December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
709
711
712
713
714


7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017
$0.00
$0.00
$0.00
$0.00
$0.00
A
1
715
717
718

Development of theBfile layouts,
Summer EOC: Early district and fullstate scan/edit and CBT extract data
719
720
721
723
724
725
726
728
729
730
Fall ELA/L Retake: Anomalous score



Component
B, C, D or
E (A-D):and
_____A_____
develop the layouts Specify
for scan/edit
and CBT A,
extract
scan/edit,
CBT extract data files, posted to
data files. For each administration,
the
contractor
online
management
system
C
D and SFTP.
E
unpacks and scans the documents to be scored and (2) Provide Department email reflecting receipt and
Due Date(s)
7.5.6.2, and 7.5.6.4

7.5.6.2, and 7.5.6.4


the data files.

the data files.
F
Component A
3-8 ELA
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00

$0.00
$0.00
October 2015
October 2016
$0.00
$0.00
$0.00

April 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2017
A
1
731
733
734
735
736
737
739
740
741
742
744
745
746
747
749
750
751
752
753
755
756
757
758
759
761

data files: Development
B of the
data files

Specifythe
Component
A, and
B, C, D or
(A-D): _____A_____
The contractor will analyze
full state SSR
(1)EAnomalous
score data files posted to SFTP.
return a Suppression fileC with student invalidation (2) Provide Department email
the data files.

the data files.
E
Due Date(s)
April 2015
April 2016
April 2017

data files

the data files.
data files
the data files.
data files
the data files.

score data files

the data files.


Reference 7.5.5

October 2015
October 2016
December 2015
December 2016
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017

October 2015
October 2016
F
Component A
3-8 ELA
$0.00
$64,841.00
$37,420.00
$28,074.00
$18,115.00
$18,115.00
$18,115.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
762
763
764
766
767
768
769
771
772
773
774
775
777
778
779
780
782
783
B
and DSRsContractor
Deliverable of


DSR data files.

and DSR data files.

C
develop the State
Student Description
Results (SSR) and District
Deliverable
Reference 7.5.5

posted to SFTPEvidence
and online
of management
Completion system.
Due Date(s)
(2) Provide Department email reflecting receipt and (Renewal 1 - October 2017)

April 2016
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

Reference 7.5.5

April 2017
(2) Provide Department email reflecting receipt and (Renewal 1 - April 2018)

Reference 7.5.5


Reference 7.5.5

October 2015
October 2016
$0.00
$0.00
$0.00

Reference 7.5.5

December 2015
December 2016
(2) Provide Department email reflecting receipt and (Renewal 1 - December 2017)
$0.00
$0.00
$0.00
$0.00
$0.00

A
Winter EOC: Initial reporting

SSRs and
B
SSR/DSR Contractor
file layouts Deliverable
and multiple
784
785
787
788
789

790
791
793
794
795
796
797
799
800
801
802
804
805



The contractor will work with

C the Department to
Student Results
(DSR) file layouts
and use the layout
Deliverable
Description
Reference 7.5.5
(1) Final initial reporting SSR

D and DSR data files
E
email
reflecting receipt and
Evidence of
Completion
Due Date(s)

Reference 7.5.5


Reference 7.5.5


students tested in the State. A DSR file contains
data for the students in that district. ITN Reference
7.5.5

October 2015
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00


April 2016
April 2017
F
Component A
3-8 ELA
$0.00
$0.00
April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
Spring ELA/L Retake:BInitial reporting

of SAR/DAR
file layouts
and multiple
Contractor
Deliverable

C Department Scoring
Results (SAR) and
District Aggregated
Deliverable
DescriptionResults (DAR)
7.5.5
(1) Final initial reporting SAR

D and DAR data files
E
email
Evidence of
Completion
Due Date(s)

7.5.5


and DAR data files.

7.5.5

October 2015
October 2016
$0.00
$0.00
$0.00


7.5.5

December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
806
807
809
810
811
812
813
815
816
817
818
820
821
822

DAR data files.
F
Component A
3-8 ELA
$0.00
$0.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
December 2015
$0.00
$0.00
823

A
1
825
826
827
B
828
829
831
832

833
834
835
837
838
839
840
842
843
844

data files.

data files.
E
Due Date(s)
April 2015
April 2016
April 2017
F
Component A
3-8 ELA
7.5.5

7.5.5


7.5.5.2 and 7.5.6.6
November 2016
$0.00
$0.00
$0.00

$0.00
$0.00

7.5.5.2 and 7.5.6.6
June 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
June 2017
A
1
845
847
848

B
data files.

849
850
851
853
854
855
856
858
859
860
Attachment
- DELIVERABLES
PRICING
Department1b
Scoring
(1) Final late
reporting SSR and DSR data files posted
and Reporting Staff Specify
to develop
the
State
Aggregated
to
online
management
system and SFTP.
Results (SAR) and District Aggregated Results (DAR) (2) Provide Department email reflecting receipt and
C to develop the
E
file layouts and use the layout
approval of the data files. D
Deliverable
Due Date(s)
scores and codes
describedDescription
in the SSR layout for all
7.5.5.2 and 7.5.6.6
7.5.5.2 and 7.5.6.6
F
Component A
3-8 ELA
$0.00
June 2015
June 2016
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
7.5.5.2 and 7.5.6.6
November 2016
$0.00
$0.00
$0.00

$0.00
$0.00

data files.
February 2015
February 2016
$0.00
$0.00
$0.00
$0.00
$0.00

7.5.5.2 and 7.5.6.6
861

A
1
863
864
865
B
data files.
866
7.5.5.2 and 7.5.6.6
867
869
870
871
872

data files.

7.5.5.2 and 7.5.6.6
873
875
876
877

for each administration. The Department will
review and approve these data files. ITN Reference
7.5.5.3 and 7.5.6.7
878
880
881
882
7.5.5.3 and 7.5.6.7
E
Due Date(s)
June 2015
June 2016
June 2017
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00
October 2017
883

A
1
885
886
887
888
889
891
892
893
894
896
897
898
899
901
902
903
904
905
907
908
F
Component A
3-8 ELA
assessments.
7.5.5.3 and 7.5.6.7
Due Date(s)
October 2015
October 2016
October 2017

7.5.5.3 and 7.5.6.7
December 2015

7.5.5.3 and 7.5.6.7
March 2015
March 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

7.5.5.3 and 7.5.6.7
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

October 2015
(2) Provide Department email reflecting receipt and October 2016
October 2017
7.5.5.3 and 7.5.6.7
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
909
910
911
913
914
915
Summer EOC: CloseBout SSR data files:

files for all
assessments.
Contractor
Deliverable


918
919
920
921
922
923
924
925
926


reporting
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
September 2015
September 2016
September 2017
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

2015
$1,000.00
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2017
reporting
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
916
917
The contractor will generate

C close out files after all (1) Final closeout SSR filesDposted to SFTP.
E
administrationDeliverable
of all assessments.
For the EOC, the approval of theEvidence
data files.
Description
of Completion
Due Date(s)
7.5.5.3 and 7.5.6.7
927
928
929
930
931
932
933

initial reporting


2015
$0.00
$0.00
$0.00
$0.00
$70,000.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
(Renewal 3 - 2020)
$1,000.00

A
1
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955


D
E
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Summer EOC: Initial reporting files and The contractor will post the data files and reports to (1) Final data files and reports posted to online
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
956
957
958
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
Student Reports and CD/DVDs or jump Student Reports and CD/DVDs or jump drives. The the samples of the student reports and CD/DVD or contract negotiation.
jump drives.
2015
Development of the student reports and Department after all reports and media are
$0.00
(Renewal 1 - 2017)
$0.00
districts.
A
1
959

Development of theBstudent reports and
- DELIVERABLES
The contractor will print andAttachment
deliver initial1b
reporting
(1) ProvidePRICING
Department email reflecting approval of
Student Reports andSpecify
CD/DVDs
or
jump
drives.
The
the
samples
of the student reports and CD/DVD or
jump drives.
C and media are
E
Department after all reports
Evidence of
Completion
Due Date(s)
destination
address,
(Renewal 3 - 2019)
districts.
F
Component A
3-8 ELA
$0.00
$0.00
960
961
962
963


964

jump drives.
districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00

jump drives.
districts.

2015
$969,849.00
2016
2017
(Renewal 1 - 2018)
$888,930.00
$896,784.00
$989,759.00
(Renewal 2 - 2019)
$989,759.00
(Renewal 3 - 2020)
$989,759.00
$0.00
$0.00
$0.00
965
966
967
968


969
970
971
972
973
974
975

initial reporting


jump drives.
districts.

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
976
977

initial reporting

$0.00
the samples of the student reports and CD/DVD or contract negotiation.
jump drives.
2015

A
1
978
979

Student
B
Submit proof
of delivery:
Development
Contractor
Deliverable
initial reporting
The contractor will print and

C deliver initial reporting
contractor willDeliverable
submit proof
of delivery to the
Description
980

D reflecting approval of
jump drives. Evidence of Completion
districts.
Due Date(s)
Component A
3-8 ELA
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
(Renewal 3 - 2019)
$0.00
981
982
983
984
985
986

initial reporting


jump drives.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00

initial reporting


jump drives.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00

reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
987
988
989
990
991
992
993
994
995
996
997


performance task. The responses will be
reproduced on a separate page that includes an
explanation of the tasks and the scoring criteria.
The contractor will print and deliver performance
task reports. Submit proof of delivery to the
A
1

B
For each test administration, the student report will (1) Provide Department email reflecting receipt and
PRICING
Specify
Component
A, B, C,
E (A-D): _____A_____
performance tasks in
each subject
that contains
theD or
reports.
performance task. The responses
will be
C
E
Due Date(s)
The contractor will print and deliver performance quantity and type of material delivered. Must show
(Renewal 3 - 2019)
districts.
F
Component A
3-8 ELA
$0.00
998
999
1000
1001
1002



reports.
districts.

2016
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00



reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00


reports.
districts.

2015
2016
(Renewal 1 - 2017)
$0.00
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012

reports
$0.00
$0.00
A
1
1013

hard copy of the Performance
Tasks
B
reports
performance tasks in eachCsubject that contains the reports.
D
E
reproduced onDeliverable
a separate page
that includes an
following for each
shipment:
date/time delivered,
Description
Evidence
of Completion
Due Date(s)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
districts.
F
Component A
3-8 ELA
$0.00
$0.00
1014
1015
1016
1017
1018

reports


reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00

reports


reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
Summer EOC: Performance Task



reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
A
1
1030

B
include a students written response
to selected
approval of
the samples of the performance task
Attachment
1b - DELIVERABLES
PRICING
each
subject
that
contains
the
reports.
D date/time delivered,
E
reproduced on a separateCpage that includes an
following for each shipment:
The contractorDeliverable
will print and
deliver performance quantity and type
of material
delivered. Must show
Description
Evidence
of Completion
Due Date(s)
(Renewal 2 - 2019)
districts.
(Renewal 3 - 2020)
F
Component A
3-8 ELA
$0.00
$0.00
1031
1032
1033
1034
1035

reporting


districts.

2015
$0.00
2016
(Renewal 1 - 2017)
$0.00
$0.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00

districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00

districts.

2015
$2,400.00
2016
2017
(Renewal 1 - 2018)
$2,400.00
$2,400.00
$2,400.00
(Renewal 2 - 2019)
$2,400.00
1036
1037
1038
1039

reporting

1040
$0.00
$0.00
$0.00
1041
1042
1043
1044
1045
1046

reporting Student Reports and
CD/DVDs. Submit proof of delivery:

A
1
1047
1048
1049

delivered to the districts. ITN
Attachment
Reference 7.5.5
1b - DELIVERABLES
following for
PRICING
each shipment: date/time delivered,
who
received
materials, destination address,
B
C
D to all participating
proof of successful delivery
districts.
1050
1051

Due Date(s)
(Renewal 3 - 2020)
2015
(Renewal 1 - 2017)
(Renewal 3 - 2019)
districts.
Component A
3-8 ELA
$2,400.00
$0.00
$0.00
$0.00
$0.00
$0.00
1052
1053
1054
1055


1056

districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
(Renewal 3 - 2019)
$0.00

districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00

districts.

2015
$0.00
2016
$0.00
1057
1058
1059
1060
1061


1062
1063
1064
1065


A
1
1066
1067
1068
1069
1071
1072
1073
1074
1076
1077
1078
1079
1081
1082
1083
1084
1085
1087
1088
1089

Reports and CD/DVDs.
B Submit proof of
reports and
media for
the late reporting
Contractor
Deliverable


Reports and CD/DVDs or jump
C drives. The
Department after
all reports
and media are
Deliverable
Description

approval of the samples ofDthe late reporting student
(2) Provide proof
of delivery
spreadsheet listing the
Evidence
of Completion
districts.
Due Date(s)
Component A
3-8 ELA
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00



reports delivered to the Department or posted to
the secure file transfer protocol (SFTP).
September 2015
September 2016
$0.00
$0.00
$0.00
$0.00
$0.00



March 2016
$0.00
$0.00
$0.00
$0.00
$0.00

mock PreID label/Roster, Mock data
files and reports/medias for mock data.


March 2015
March 2016
March 2017



March 2017

September 2015
September 2016
$4,400.00
$4,400.00
$4,400.00
$4,400.00
$4,400.00
$4,400.00
$0.00
$0.00
$0.00
$0.00
$0.00
1090

A
1
1092
1093
1094
1095
1097
1098
1099
1100


1101
1103
1104
1105
1106
1107



Performance Tasks

1108
1109
1110
1111
Due Date(s)
Component A
3-8 ELA
November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00

March 2015
March 2016
March 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

June 2015
June 2016
June 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
A
1

Performance Tasks
B
Handscoring refers to the processes necessary for (1) Provide Department email reflecting approval of
determining the rating of a students response on the final handscoring report.
Component
A, B, C, D or E (A-D): _____A_____
the writing promptsSpecify
or on the
performance-based
tasks in ELA/L and Mathematics
that respondents
C
D
E
Due Date(s)
(Renewal 3 - 2019)
F
Component A
3-8 ELA
$0.00
1112
1113
1114
1115
1116
1117

Performance Tasks

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00

2015
$7,062,855.00
$0.00
$0.00
$0.00
$0.00
$0.00
1118
1119
1120
1121
1122
1123

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$5,926,541.00
$6,149,176.00
$5,606,000.00
$5,472,025.00
A
1

B
Attachment
1b - DELIVERABLES PRICING
the writing prompts or on the
performance-based
Specify Component
A, B, C, D or E (A-D): _____A_____
that respondents
may propose. Both paper-based and computerC
D
E
based tests will contain performance tasks, and so
Due Date(s)
(Renewal
3
2020)
F
Component A
3-8 ELA
$5,442,603.00
1124
1125
1126
1127
1128

Tasks

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
$0.00
$0.00
1129
1130
1131
1132
1133

Performance Tasks
$0.00
$0.00
$0.00
A
1

Performance Tasks
B
Specify
Component
A, B,on
C, D or
(A-D):
_____A_____
determining the rating
of a students
response
theEfinal
handscoring
report.
the writing prompts or onCthe performance-based
D
E
may propose. Both paper-based and computerDeliverable Description
Due Date(s)
(Renewal 3 - 2019)
F
Component A
3-8 ELA
$0.00
1134

Performance Tasks
1135
1136
1137
1138
1139


2015
1140
1141
1142
1143
1144

Performance Tasks
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
A
1
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167

Performance Tasks
B
Component
A, B, C, D or E (A-D): _____A_____
or on the
performance-based
that respondents
C
D
E
Due Date(s)
(Renewal 2 - 2019)
(Renewal 3 - 2020)


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5
June 2015
June 2016
January 2016
January 2017
January 2015
January 2016
January 2017
June 2015

the SFTP.
June 2016
(2) Provide Department email reflecting receipt and (Renewal 1 - June 2017)
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$28,226.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
all assessments
Reference 7.5.5

all assessments
Reference 7.5.5

all assessments
Reference 7.5.5

ELA/Mathematics

ELA/Mathematics
D
the SFTP.
Due Date(s)
Component A
3-8 ELA
October 2015
October 2016
(1) Final set of approved report mockups posted to January 2015
the SFTP.
January 2016
(2) Provide Department email reflecting receipt and January 2017
the SFTP.

April 2015
April 2016
April 2017

December 2015
(2) Provide Department email reflecting receipt and December 2016
Reference 7.5.4.2
Reference 7.5.4.2

October 2016
October 2017
October 2015
and ELA/Mathematics
October 2017
Reference 7.5.4.2
December 2015
Reference 7.5.4.2
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$6,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00

A
1
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
Scoring
and Reporting
Deliverable
Category
will create and
update history files (1) History filesEvidence
posted toofthe
SFTP.
Description
Completion
Due Date(s)
Reference 7.5.4.2
March 2016
(2) Provide Department email reflecting receipt and March 2017
Reference 7.5.4.2
October 2015
October 2017
Reference 7.5.4.2

ELA/Mathematics
The contractor will create and update history files

requirements specified in the SR specifications. ITN
Reference 7.5.4.2

Retake
The contractor will create and update an APS file for

Retake test after each test administration using the
rule specified in the SR specifications. NOTE: This is
Fall EOC:Contractor
Development
of the history
Deliverable
1225
1226
1227
1229
1230

(1) History files posted to the SFTP.
October 2015
approval of the history data files.
October 2017
(1) APS file posted to the SFTP.
November 2015
(2) Provide Department email reflecting receipt and November 2016
approval of the APS file.

May 2016

the APS (Alternate Passing Score) file for Retake test after each test administration using the (2) Provide Department email reflecting receipt and May 2017
Retake
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1231
1232
1234
1235
1236
1237
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1255
1256
Spring ELA/L Retake:BDevelopment of

The contractor will createCand update an APS file for (1) APS file posted to the SFTP.
D
E
Retake Contractor Deliverable
rule specified in
the SR specifications.
APS file. of Completion
Deliverable
DescriptionNOTE: This is approval of theEvidence
Due Date(s)

rescoring.

Reference 7.5.6.5

management system.
log.

manual rescoring.

Reference 7.5.6.5

management system.
log.


Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

October 2015
October 2016
April 2016
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2017
April 2015
$0.00
$0.00
$0.00
$0.00
$0.00
$2,000.00
April 2016
April 2017
October 2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
October 2016
December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

1
1257
1258
1260
1261
1262
1263
1264
1266
1267
1268

B
rescoring.Contractor Deliverable

C
Deliverable
Description
requirements specified
in the
SR Specification. ITN
Reference 7.5.6.5

D
management system.
log.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.
Periodic: Writing Image CD/DVD or

jump drives and proof to delivery:

or jump drives for each grade tested. The
Department after all of the Writing Image CD/DVD
or jump drives are delivered to districts. ITN
Reference 7.5.5

districts.


therefore the Contractor shall provide a report
which indicates the completed status of
Specifications posted to Secure File Transfer
Protocol (SFTP).

rescoring.
1269
1270
1271
1272
1273
1274
1275
1276
1278
Studies
1279
1280
1281

E
Due Date(s)
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

July 2015
$0.00
$28,620.00
July 2016
July 2017
$28,620.00
$28,620.00

$28,620.00
$28,620.00
$28,620.00
May 2014
May 2015
$96,473.00
$2,000.00
May 2016
$0.00
$0.00

$0.00
$0.00
1282

A
1
1284 Studies
1285
Administrations
1286
1287
1288
Studies
1290
1291
1292
1293
1294
Studies
1296
1297
1298
1299
1300
1303 Studies
1304
1305
D
deliverable.
Periodic: Development of the Sampling Detailed plan for sampling of students from the
Plans
Administrations

Specifications


Due Date(s)
Component A
3-8 ELA
July 2014
July 2015
$39,726.00
$39,726.00
July 2016
$39,726.00
$39,726.00

$39,726.00
$39,726.00
July 2014
July 2015
$6,000.00
$2,000.00
July 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$6,000.00
$2,000.00

August 2014
August 2015
August 2016
December 2014
December 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$40,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00

Studies
A
1306
1307
1308
1309
1310
1311
1312
1313

Studies

1314 Studies
1315
1316
1317
1318
1321
1322
1323
Studies

Specifications
B
Specify
Component
A,plans
B, C, D or
(A-D): Department
_____A_____
and equating plans and
procedures,
scoring
(2)EProvide
email reflecting receipt and
and procedures. The plan Cshould include a
D
E
Due Date(s)


posted to SFTP.
May 2015
May 2016
May 2017
2016

via SFTP.
2017

(1) Submit final Calibration Sampling Specifications (Dates to be negotiated) 2015

via SFTP.
2016
(Renewal 1 - 2018)
(Renewal 2 - 2019)

(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Component A
3-8 ELA
$2,000.00
$200,000.00
$200,000.00
$200,000.00
$200,000.00
$200,000.00
$200,000.00
$3,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00

Studies
1324
1327

B

Technical Reports


posted to SFTP.


Reference 7.6.3.3

November 2014
posted to SFTP.

Collecting legal opinion, subject to the

Departments approval, about the defensibility of
linking methodology. ITN Reference 7.6.3.3
(1) Submit final version of the legal opinion

October 2014
SFTP.
Studies
1328
1329
1330
Studies
Attachmentmust
1b - DELIVERABLES
The contractors sampling specifications
(1) SubmitPRICING
final Calibration Sampling Specifications
include analyzing the
most
recent
SSR
files
and
via
SFTP.
establishing a road map for gathering the required (2) Provide Department email reflecting receipt and
D
E
data proven to represent C
the demographic
Description
Due Date(s)
performance. Deliverable
The specifications
will also address
(Renewal 3 - 2020)
F
Component A
3-8 ELA
$0.00
December 2015
December 2016
December 2017
$125,905.00
$202,094.00
$202,094.00
$202,094.00

$202,094.00
$202,094.00
$20,412.00
$0.00
1332
1333 Studies
1334
Studies


May 2015
$697,578.00
$0.00
$85,970.00
$0.00
1336

A
1
C
7.6.3.4
Studies
Setting Work Plan

Studies
approval of the technical report and excel tables
with interactive impact data and graphs.

Studies

Technical Report
(1) Submit the final version of the Standard Setting November 2015
SFTP.
Due Date(s)
Component A
3-8 ELA
$2,000.00
1337
$161,561.00
1338
1339
Studies


7.6.3.4.
Department.
renewal)
(1) Post the plan for Vertical Scaling (secure file) via November 2014
SFTP.
$30,761.00
$9,000.00
$5,000.00
1341

A
1
Studies
Report
Due Date(s)
August 2015
Files (secure files), including vertically aligned scale (Renewal 2 - August 2018)




Reference 7.6.3.6
parameters (secure files) to the Department via
SFTP.
F
Component A
3-8 ELA
$18,407.00
$5,000.00
1343
Studies
1344
1345
Studies

Report
$17,819.00
$0.00
$45,291.00
$0.00
1347
1348 Studies

across devices.
input types.
concordance.
characteristics.
$5,400.00

Scaling, Equating,
A Scoring,
Studies
1349
Studies
Periodic: Development
for conducting
B of Specifications Preparing a detailed plan C
Delivering
Assessments
comprehensiveDeliverable
and detailed
timeline for all tasks
Contractor
Deliverable
Description
across devices.
input types.
concordance.
characteristics.
(1) Submit the plan for conducting

the comparability November 2014
D
email
Evidence of
Completion

with TAC Committee

(1) Submit Final Report and Recommendations to

August 2015
the Department. These materials are secure, and



November 2015
SFTP.
Due Date(s)
Component A
3-8 ELA
$0.00
$29,752.00
$0.00
1351
Studies
1352
$900.00

Studies
A
1353
Studies
1355
Studies

Assessments to other
B assessments
Preparing a detailed plan for conducting the linking (1) The plan for conducting comparability study is
November 2015
study. The plan should include a comprehensive and secure, and therefore contractor shall post them to
detailed timeline for all tasks
SFTP.
C and events, as
D
The plan mustDeliverable
be vetted byDescription
TAC members.
approval of theEvidence
study specifications.
of Completion


7.6.3.6.2

August 2016

Assessments

technical feasibility of each method given the
nature and design of the Florida Standards
Assessments and those of international
assessments such as PISA and TIMSS, propose
method(s) for linking the scales of assessments,
propose alternative ways for collecting data, explain
the process of establishing the linking
relationship(s) in detail, and provide a

November 2015
SFTP.
Due Date(s)
Component A
3-8 ELA
$0.00
$9,868.00
$0.00
$900.00
$0.00
1357

A
1
Studies
B
Assessments
1358
C
7.6.3.6.3
D
August 2016
Due Date(s)
Component A
3-8 ELA
$900.00
$0.00
1359
1361
1362
1363
1364
1365
1367
1368
1369
1370
1371
1373
1374
1375
1376
1377
1379
1380
1381
1382

7.7.2

report

Meetings


previous invoice.

ITN.


Action Item Report

work. ITN Ref 7.7.6

April 15, 2014

April 15, 2015
April 15, 2016
$66,198.00
$27,930.00
$27,930.00
$27,930.00
$27,930.00

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$27,930.00
$121,879.00
$122,426.00
$122,851.00
$123,260.00
$124,154.00
$124,566.00
$1,139,500.00
$1,276,463.00
$1,287,729.00
$1,102,105.00
$1,122,579.00
$1,129,158.00
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$112,459.00
$112,459.00
$112,459.00
$112,459.00
$112,459.00
Program Management
A
1
1383
1385
1386
1387
1388
1389
1391
1392
1393
1394

Action Item Report B

The Weekly Management Report and Action Item (1) For each invoice period, provide one
Specify
E (A-D): _____A_____
Report will report the
status Component
of recent andA, B, C, D or
management
report from each month since the
upcoming tasks, highlighting
those
leading
up
to
previous
invoice
period. (Quarterly
invoices to occur
C
D
E
Due Date(s)
(Renewal 3 - 2019)
work. ITN Ref 7.7.6
F
Component A
3-8 ELA
$112,459.00

improvements or changes for subsequent years.
The annual review will be based on information
November 2015
November 2016
November 2017
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00

ITN Ref 7.7.6
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$69,917.00
$69,797.00
$69,797.00
$69,797.00
$69,797.00
Program Management
A
1

B

Specify
Component
A,level
B, C,ofD or
E (A-D): reports.
_____A_____
including expenditures
by program
task and
accounting
(Quarterly invoices to occur in
effort as well as expenditures
and
balances
of
February,
May,
August,
and
C
D November)
ITN Ref 7.7.6
Due Date(s)
Component A
3-8 ELA
(Renewal 3 - 2019)
$69,797.00

whether the base is extended into one, two, or
three one-year renewals]
$43,167.00
1395
Program Management
Ref. 7.7.10
1396

Specify Component A, B, C, D or E (A-D): _____B_____
A
1
3
4
5
6
7
8
Test Development
Document
Test Development

Specifications
9
10
11
12
13
C
7.3.10
via SFTP.
Due Date(s)
June 1, 2014
June 1, 2015
June 1, 2016
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5
F
Component B
3-8 Math
$38,237.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
14
Test Development
15
16
17
18
19


7.5.4.5

SFTP.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

7.5.4.5

SFTP.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
20
Test Development
21
22
23
24
25

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$20,191.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
Test Development
A
1

B
E (A-D): _____B_____
C
D
E
andDescription
the Department. The
Deliverable
Due Date(s)
(Renewal 3 - 2020)
7.5.4.5
F
Component B
3-8 Math
$2,000.00
26
Test Development
27
28
29
30
31

7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
7.5.4.5

SFTP.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
32
Test Development
33
34
35
36
37
$0.00
$0.00
$0.00
$0.00
$0.00
38
Test Development
39
40
41
42
43
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
44

A
1
Test Development
45
46
47
48
49
50
51 Test Development
52
53
54
55
56
B
Specifications
C
7.5.4.5
D
SFTP.
7.3.10
Test Development


Test Development

each scoring guide.
F
Component B
3-8 Math
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
June 1, 2014
June 1, 2015
June 1, 2016
$15,797.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$35,812.00
57
58
59
60
61
62

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$31,444.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
Test Development
A
1

B
each scoring guide.
Attachment
1b - DELIVERABLES
PRICING
Specify
Component
A, These
B, C, D or E (A-D): _____B_____
on student
reports.
C
D
E
Due Date(s)
(Renewal
3
2020)
F
Component B
3-8 Math
$2,000.00
63
Test Development

Assessment Reports
64
65
66
67
68

Reference 7.3.7.5

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$67,402.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
69
Test Development
70
71 Test Development
72
73
74
75
76 Test Development


$147,745.00

management report
Reference 7.3.4.2

February 1, 2017
management report
$3,845,421.00
$3,926,321.00
$2,026,406.00
$1,756,434.00
$1,474,150.00
Review

$3,924,211.00

management report

A
Development
1 Test Deliverable
Category
77
78
79
80
81 Test Development
82
83
84
85
86
87 Test Development
88
89
90
91 Test Development
92
93
94
95 Test Development
96
97
98
99
Periodic:Contractor
Field Test Item
or Description
Deliverable
Deliverable
Review

email
Evidence of
Completion
Due Date(s)
May 1, 2016
management report
July 15, 2015
July 15, 2016
F
Component B
3-8 Math
$32,066.00
$32,066.00
$32,066.00
$32,066.00
$33,668.00
$48,762.00
$48,762.00
$48,762.00
$48,762.00
$48,762.00

Reference 7.3.0

management report.
December 1, 2015
December 1, 2016
$23,304.00
$23,304.00
$23,304.00
$23,304.00


management report.
December 1, 2015
December 1, 2016
$19,359.00
$9,109.00
$9,109.00
$9,109.00


previous invoice.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$111,390.00
$112,882.00
$114,410.00
$117,566.00
$119,720.00
Test Development
A
1
100
102
103
104
105
106
108
109
110
111
112
Test Administration
B

Component
A, B, C, D or
E (A-D):
_____B_____
for the purpose
of reviewing
previous
invoice.
supporting the
C





113
114
115
116
Due Date(s)
Component B
3-8 Math
(Renewal 3 - 2020)
$120,058.00

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
the report.
2015
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00

2016
$0.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
$0.00
$0.00
$0.00
117
Test Administration
118
119
120
121

the report.
Test Administration
A
1

B
1b - DELIVERABLES
the report.PRICING
Specify
Component
A, by
B, C, D or E (A-D): _____B_____
has been
conducted
C
D
Due Date(s)
Component B
3-8 Math
(Renewal 3-2020)
$0.00
122
Test Administration
123
124
125
126
127

a report.
the report.

2015
$2,000.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

the report.

2015
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
the report.

2015
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
128
Test Administration
129
130
131
132
133
Test Administration
134
135
136
137
138

A
1
Test Administration
139
140
141
142
143
F
Component B
3-8 Math
the report.
Due Date(s)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00

the report.

2015
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00
the Specifications.

2015.
$0.00
$0.00
$0.00
144
Test Administration
145
146
147
148
149
150
152
153
154
155
156

2016
$0.00
(Renewal 1- 2017)
$0.00
(Renewal 2-2018)
$0.00
(Renewal 3 - 2019)
$0.00
$0.00
the Specifications.
2016

A
1
Test Deliverable
Administration
Category
158
159
160
161
162
164
165
166
167
168
169
171
172
173
174
175
Spring ELA/L
Retake:Deliverable
all processes,
of the approved
specifications.
Contractor
Deliverable
Description
of Completion
the Specifications.
E
2016
F
Component B
3-8 Math
$0.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
the Specifications.
administration.
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$79,909.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
$81,640.00
$67,932.00
$72,745.00
$75,477.00
$76,384.00
$0.00
$0.00

administration.
the Specifications.
2016
$0.00
(Renewal 1- 2017)
$0.00
(Renewal 2-2018)
$0.00
(Renewal 3 - 2019)
$0.00
$0.00
administration.
the Specifications.
2015.

A
1
Test Deliverable
Administration
Category
177
178
179
180
181
183
184
185
186
187
188
190
191
192
193
194
195
Winter EOC:
DevelopDeliverable
packaging and
all processes,
of the approved
specifications.
Contractor
Deliverable
Description
of Completion
administration.
the Specifications.

2015.
Component B
3-8 Math
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
administration.
the Specifications.
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
administration.
the Specifications.
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
197
198
199
200
201
Test Administration
202
203
204
205
206
207
208
Test Administration
209
210
211
212
213
214
215
Test Administration
216
217
218
219
220
221
222
B
memos, etc.
districts.

memos.


districts.

memos.


districts.

memos.


districts.
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$257,340.00
$305,950.00
$350,882.00
$377,733.00
$389,093.00
$390,647.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$259,914.00
$227,780.00
$219,031.00
$184,386.00
$190,454.00
$190,756.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
224
225
226
227
228
Test Administration
229
230
231
232
233
234
Test Administration
235
236
237
238
239
240
241
Test Administration
242
243
244
245
246
247
and accommodations.
districts.

and accommodations.


districts.



districts.

accommodations.


districts.
Due Date(s)
negotiations.)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$3,232,373.00
$2,356,484.00
$2,140,787.00
$1,285,869.00
$1,326,868.00
$1,331,988.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
248
249
250
251
252
253
Test Administration
254
255
256
257
258
259
260
Test Administration
261
262
263
264
265
266
267
Test Administration
268
269
270
271
272
273
accommodations.
districts.

accommodations.


districts.

accommodations.


districts.


and 7.4.7.4

districts.
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
B
districts.

and 7.4.7.4

districts.
and rosters.
and 7.4.7.4

districts.

labels and rosters.

and 7.4.7.4

districts.

and 7.4.7.4

districts.

280
281
282
283
284
285
286
Test Administration
287
288
289
290
291
292
Test Administration
293
294
295
296
297
298
Test Administration
299
and 7.4.7.4
274
275
276
277
278
279
Test Administration

labels and rosters.
Due Date(s)
negotiations.)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Test Administration
A
1
300
301
302
303
304
305
Test Administration
306
307
308
309
310
311
312
314
315
316
317
318
320
321
322
323
324
326
327

preidentification
labels and rosters.

labels and rosters.

critical date
Description
and 7.4.7.4

destination
address,
Evidence of
Completion
districts.

and 7.4.7.4

districts.
Due Date(s)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
7.4.8

November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00


7.4.8

May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00


7.4.8

May 2016
$0.00
$139,096.00
$262,891.00

Test Administration
A
1
328
329
330
331
333
334
335
336
337
339
340
341
342
343
345
346
347
348
349
350
352
353
354
355

Deliver
B
system for
each assessment.
Contractor
Deliverable

was operable for
the invoice
Deliverable
Description
7.4.8

E
occurred during
the period,
signed by the
Evidence
of Completion
Due Date(s)

assessment.

7.4.8

November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

February 2016
February 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
assessment.
7.4.8

August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
F
Component B
3-8 Math
$262,891.00
$262,891.00
$262,891.00
$262,891.00
Test Administration
A
1
356
Specify Component
A, B,
C, D or
(A-D):
_____B_____
of the contract
and
(orEno)
malfunctions
assessment.
was
operable
for
the
invoice
period.
ITN
Reference
occurred
during
the
period,
B
C
D signed by the
E
7.4.8
Due Date(s)
F
Component B
3-8 Math
$0.00
$0.00
357
359
360
361
362



November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
363
365
366
367
368

May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
369
371
372
373
374
375

administration.


May 2015
May 2016
May 2017
$0.00
$3,749,054.00
$4,294,013.00
$5,097,356.00
$5,602,608.00
$5,506,854.00
$5,331,649.00
376
378
379
380
381



November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
Test Administration
A
1
382
384
385
386
387
388
390
391
392
393
394
395
397
398
399
400
401
402
404
405
406
407
408

B
(1) ProvidePRICING
A,ofB,the
C, D or
(A-D):
_____B_____
met theComponent
requirements
(orEno)
malfunctions
C
D
E
Due Date(s)












F
Component B
3-8 Math
$0.00
February 2016
February 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
May 2015
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
November 2014
November 2015
November 2016
$63,181.00
$3,000.00
$3,000.00
$3,000.00
$3,000.00
$3,000.00
Test Administration
A
1

B

Attachment
1bon
- DELIVERABLES
PRICING
and approved
a
the practice
Specify
Component
A, B, C, D or
E (A-D): _____B_____
that
offers students
requirements
C
D
Due Date(s)
Component B
3-8 Math
$0.00
409
Test Administration
410
412
413
414
415
417
418
419
420
421
423
424
425
426

the contract.


training.
September 2014
7.4.9.8
administration.
August 2015
August 2016
November 2016
November 2017
previous year.
November 2015
November 2016
November 2017
$242,481.00
$143,014.00
$143,014.00
$143,014.00
$143,014.00
$143,014.00
$537,928.00
$445,587.00
$456,410.00
$475,277.00
$486,865.00
$490,413.00
$61,492.00
$54,645.00
$53,700.00
$31,475.00
$32,172.00
Test Administration
A
1
Specify Component
A, B, C,for
D or
E (A-D):
_____B_____
and other materials
materials
and
a
period
of
at
least
six
months,
and
used
documents
end
of
each
assessment
year
B
C
E
Due Date(s)
F
Component B
3-8 Math
$32,195.00
427
Test Administration
428
429
430
431
432
433
Test Administration
434
435
436
437
438
439
Test Administration
440
441
442
443
444
445
446
Test Administration
447
448
449
450
451

comment forms.

comment forms.


negotiations.)
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015

comment forms.
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
Test Administration
A
1
452
Test Administration
453
454
455
456
457
458
Test Administration
459
460
461
462
463
464
465
Test Administration
466
467
468
469
470
471
472
Test Administration

comment forms.
B
comment forms.

comment forms.

comment forms.

Trial Test

1b -forms.
online comment
Component A, B, C, D or E (A-D): _____B_____
and 7.4.7.7
C
Due Date(s)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
7.4.8 and 7.4.9.5
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$58,275.00
473

A
1
Test Administration
474
475
476
477
478
479
Test Administration
480
481
482
483
484
485
Test Administration
486
487
488
489
490
491
492
Test Administration
493
494
495
496
497
498
B



versions)
F
Component B
3-8 Math
Reference 7.3.11.2
Due Date(s)
negotiations.)

Reference 7.3.11.2
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Reference 7.3.11.2
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Reference 7.3.11.2
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$132,091.00
$78,071.00
$78,071.00
$78,071.00
$78,071.00
$78,071.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
499
500
501
502
503
504
Test Administration
505
506
507
508
509
510
511
Test Administration
512
513
514
515
516
517
518
Test Administration
519
520
521
522
523
524
versions)
Reference 7.3.11.2
D
Due Date(s)
negotiations.)
Component B
3-8 Math
$0.00
versions)
Reference 7.3.11.2
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

versions)
Reference 7.3.11.2
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00


Reference 7.4.1.
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
525
526
527
528
529
530
Test Administration
531
532
533
534
535
536
537
Test Administration
538
539
540
541
542
543
Test Administration
544
545
546
547
548
549
Reference 7.4.1.
Due Date(s)
negotiations.)
testing
Reference 7.4.1.


negotiations.)
Reference 7.4.1.


Reference 7.4.1.
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$684,116.00
$463,790.00
$543,383.00
$294,080.00
$305,700.00
$309,332.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
B
550
551
552
553
554
555
556
Test Administration
557
558
559
560
561
562
563
565
566
572
573
574
576
577
578
Reference 7.4.1.
Reference 7.4.1.

567
568
569
571

ITN Reference 7.5.1
D
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016

management system.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
April 2014
April 2015
April 2016

management system.


management system.

F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
September 2015
September 2016
$0.00
$0.00
$0.00

January 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
January 2017
A
1
579
581
582
583
584
585
587
588
589
590
592
593
594
595
597
598
599
600
601
603
604
605
606
607
609


monitor district
(1)EPreID
frequency
Specifyand
Component
A, B, C, D or
(A-D):
_____B_____
C reports of issues
Deliverable
Description
assessments. ITN
Reference
7.5.2
management system.
management system.
E
Due Date(s)
January 2015
January 2016
January 2017

review.

management system.
review.

management system.
review.


in .txt format

September 2015
September 2016
October 2015
F
Component B
3-8 Math
$0.00
$2,500.00
$1,250.00
$1,250.00
$1,250.00
$1,250.00
$1,250.00
$0.00
$0.00
$0.00
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

management system.

January 2015
January 2016
January 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

management system.

June 2015
June 2016
June 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Protocol (SFTP).

September 2015
September 2016

A
1
610
611
612
614
615
616
617
619
620
621
622
623
625
626
627
file layout
and Full-State
layout for each
administration for
Contractor
Deliverable
Description
in .txt format

management system
andofSecure
File Transfer
Evidence
Completion
Protocol (SFTP).

file in .txt format

Protocol (SFTP).
January 2016


Protocol (SFTP).
January 2015
January 2016
January 2017
format

Protocol (SFTP).
September 2015
Due Date(s)
January 2017
September 2016
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
A
1
628
630
631
632
633
635
636
637
638
639
641
642
643
644

Specify
Component
A, B,toC, D or
(A-D):
_____B_____
with
the Department
(1)EFinal
PreID
format


File Transfer
D
Protocol (SFTP).

.txt format


Protocol (SFTP).
October 2015
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00

.txt format


Protocol (SFTP).
January 2015
January 2016
January 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

.txt format


Protocol (SFTP).
June 2015
June 2016
June 2017
$0.00
$0.00
$0.00
$0.00
$0.00
E
Due Date(s)
F
Component B
3-8 Math
$0.00
A
1
645
647
648
649
650
652
653
654
655
657
658
659
660
661
663
664
665
666
668
669
670
671

.txt format

Attachment
1b - DELIVERABLES
PRICING
the Department
to
(1) Final PreID
Component
A, B,
E (A-D): _____B_____
for each
administration
forC, D or
management
D the SFTP.
Deliverable
of Completion
each PreID window
closes,Description
the contractor will
the file layout
and data file.




E
Due Date(s)

$0.00
$0.00
$0.00

May 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
May 2017
May 2015
May 2016
May 2017





$0.00
October 2015
October 2016

transcription logs
F
Component B
3-8 Math

October 2015
October 2016
December 2015
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

May 2015
$0.00
$0.00

A
and Reporting
1 Scoring
Deliverable
Category
673
674
675
676
677
679
680
681
682
683
685
686
687
688
690
691
692
Spring EOC:
Development
will use a Special
Document
Contractor
Deliverable
Description

logofposted
to SFTP.
Evidence
Completion




7.5.6.2, and 7.5.6.4



7.5.6.2, and 7.5.6.4

E
Due Date(s)
May 2016
May 2017
July 2015
July 2016
July 2017
October 2015
October 2016
March 2016
March 2017
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
A
1
693
695
696
697
698
699
701
702
703

Specify
A, B,toC, D or
(A-D):
_____B_____
withComponent
the Department
(1)EFinal
approved

and CBT extract
C
7.5.6.2, and 7.5.6.4

data files



7.5.6.2, and 7.5.6.4


7.5.6.2, and 7.5.6.4

E
Due Date(s)
March 2015
March 2016
March 2017
October 2015
October 2016
F
Component B
3-8 Math
$0.00
$1,658,766.00
$1,512,186.00
$217,182.00
$218,756.00
$227,573.00
$230,294.00
$0.00
$0.00
$0.00
$0.00
A
1

data files:
B

Specify
A, B,toC, D or
(A-D):
_____B_____
withComponent
the Department
(1)EFinal
approved
and CBT extract
C
7.5.6.2, and 7.5.6.4

E
Due Date(s)
F
Component B
3-8 Math
$0.00
704
706
707
708


7.5.6.2, and 7.5.6.4

December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
709
711
712
713
714


7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017
$0.00
$0.00
$0.00
$0.00
$0.00
A
1
715
717
718

719
720
721
723
724
725
726
728
729
730



Component
B, C, D or
E (A-D):and
_____B_____
for scan/edit
and CBT A,
extract
scan/edit,
the
contractor
online
management
system
C
D and SFTP.
E
Due Date(s)
7.5.6.2, and 7.5.6.4

7.5.6.2, and 7.5.6.4


the data files.

the data files.
F
Component B
3-8 Math
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00

$0.00
$0.00
October 2015
October 2016
$0.00
$0.00
$0.00

April 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2017
A
1
731
733
734
735
736
737
739
740
741
742
744
745
746
747
749
750
751
752
753
755
756
757
758
759
761

B of the
data files

Specifythe
Component
A, and
B, C, D or
(A-D): _____B_____
full state SSR
(1)EAnomalous
the data files.

the data files.
E
Due Date(s)
April 2015
April 2016
April 2017

data files

the data files.
data files
the data files.
data files
the data files.

score data files

the data files.


Reference 7.5.5

October 2015
October 2016
December 2015
December 2016
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017

October 2015
October 2016
F
Component B
3-8 Math
$0.00
$64,841.00
$37,420.00
$28,074.00
$18,115.00
$18,115.00
$18,115.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
762
763
764
766
767
768
769
771
772
773
774
775
777
778
779
780
782
783
B
and DSRsContractor
Deliverable of


DSR data files.

and DSR data files.

C
develop the State
Student Description
Deliverable
Reference 7.5.5

and online
of management
Completion system.
Due Date(s)

April 2016
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

Reference 7.5.5

April 2017

Reference 7.5.5


Reference 7.5.5

October 2015
October 2016
$0.00
$0.00
$0.00

Reference 7.5.5

December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00

A

SSRs and
B
SSR/DSR Contractor
and multiple
784
785
787
788
789

790
791
793
794
795
796
797
799
800
801
802
804
805




C the Department to
Student Results
(DSR) file layouts
and use the layout
Deliverable
Description
Reference 7.5.5

E
email
Evidence of
Completion
Due Date(s)

Reference 7.5.5


Reference 7.5.5


7.5.5

October 2015
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00


April 2016
April 2017
F
Component B
3-8 Math
$0.00
$0.00
April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A

of SAR/DAR
file layouts
and multiple
Contractor
Deliverable

Results (SAR) and
District Aggregated
Deliverable
7.5.5

E
email
Evidence of
Completion
Due Date(s)

7.5.5


and DAR data files.

7.5.5

October 2015
October 2016
$0.00
$0.00
$0.00


7.5.5

December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
806
807
809
810
811
812
813
815
816
817
818
820
821
822

DAR data files.
F
Component B
3-8 Math
$0.00
$0.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
December 2015
$0.00
$0.00
823

A
1
825
826
827
B
828
829
831
832

833
834
835
837
838
839
840
842
843
844

data files.

data files.
E
Due Date(s)
April 2015
April 2016
April 2017
F
Component B
3-8 Math
7.5.5

7.5.5


7.5.5.2 and 7.5.6.6
November 2016
$0.00
$0.00
$0.00

$0.00
$0.00

7.5.5.2 and 7.5.6.6
June 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
June 2017
A
1
845
847
848

B
data files.

849
850
851
853
854
855
856
858
859
860
Attachment
- DELIVERABLES
PRICING
Department1b
Scoring
(1) Final late
to develop
the
State
Aggregated
to
online
management
system and SFTP.
C to develop the
E
Deliverable
Due Date(s)
scores and codes
7.5.5.2 and 7.5.6.6
7.5.5.2 and 7.5.6.6
F
Component B
3-8 Math
$0.00
June 2015
June 2016
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
7.5.5.2 and 7.5.6.6
November 2016
$0.00
$0.00
$0.00

$0.00
$0.00

data files.
February 2015
February 2016
$0.00
$0.00
$0.00
$0.00
$0.00

7.5.5.2 and 7.5.6.6
861

A
1
863
864
865
B
data files.
866
7.5.5.2 and 7.5.6.6
867
869
870
871
872

data files.

7.5.5.2 and 7.5.6.6
873
875
876
877

7.5.5.3 and 7.5.6.7
878
880
881
882
7.5.5.3 and 7.5.6.7
E
Due Date(s)
June 2015
June 2016
June 2017
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00
October 2017
883

A
1
885
886
887
888
889
891
892
893
894
896
897
898
899
901
902
903
904
905
907
908
F
Component B
3-8 Math
assessments.
7.5.5.3 and 7.5.6.7
Due Date(s)
October 2015
October 2016
October 2017

7.5.5.3 and 7.5.6.7
December 2015

7.5.5.3 and 7.5.6.7
March 2015
March 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

7.5.5.3 and 7.5.6.7
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

October 2015
October 2017
7.5.5.3 and 7.5.6.7
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
909
910
911
913
914
915

files for all
assessments.
Contractor
Deliverable


918
919
920
921
922
923
924
925
926


reporting
F
Component B
3-8 Math
$0.00
$0.00
$0.00
September 2015
September 2016
September 2017
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

2015
$1,000.00
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2017
reporting
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
916
917

E
of all assessments.
data files.
Description
of Completion
Due Date(s)
7.5.5.3 and 7.5.6.7
927
928
929
930
931
932
933

initial reporting


2015
$0.00
$0.00
$0.00
$0.00
$70,000.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
(Renewal 3 - 2020)
$1,000.00

A
1
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955


D
E
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
956
957
958
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
jump drives.
2015
$0.00
(Renewal 1 - 2017)
$0.00
districts.
A
1
959

- DELIVERABLES
deliver initial1b
reporting
(1) ProvidePRICING
CD/DVDs
or
jump
drives.
The
the
samples
jump drives.
C and media are
E
Evidence of
Completion
Due Date(s)
destination
address,
(Renewal 3 - 2019)
districts.
F
Component B
3-8 Math
$0.00
$0.00
960
961
962
963


964

jump drives.
districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00

jump drives.
districts.

2015
$969,849.00
2016
2017
(Renewal 1 - 2018)
$888,930.00
$896,784.00
$989,759.00
(Renewal 2 - 2019)
$989,759.00
(Renewal 3 - 2020)
$989,759.00
$0.00
$0.00
$0.00
965
966
967
968


969
970
971
972
973
974
975

initial reporting


jump drives.
districts.

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
976
977

initial reporting

$0.00
jump drives.
2015

A
1
978
979

Student
B
Submit proof
of delivery:
Development
Contractor
Deliverable
initial reporting

submit proof
of delivery to the
Description
980

districts.
Due Date(s)
Component B
3-8 Math
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
(Renewal 3 - 2019)
$0.00
981
982
983
984
985
986

initial reporting


jump drives.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00

initial reporting


jump drives.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00

reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
987
988
989
990
991
992
993
994
995
996
997


A
1

B
PRICING
Component
A, B, C,
E (A-D): _____B_____
performance tasks inSpecify
each subject
that contains
theD or
reports.
will be
C
E
Due Date(s)
(Renewal 3 - 2019)
districts.
F
Component B
3-8 Math
$0.00
998
999
1000
1001
1002



reports.
districts.

2016
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00



reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00


reports.
districts.

2015
2016
(Renewal 1 - 2017)
$0.00
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012

reports
$0.00
$0.00
A
1
1013

Tasks
B
reports
D
E
a separate page
that includes an
following for each
shipment:
Description
Evidence
of Completion
Due Date(s)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
districts.
F
Component B
3-8 Math
$0.00
$0.00
1014
1015
1016
1017
1018

reports


reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00

reports


reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00



reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
A
1
1030

B
to selected
approval of
Attachment
1b - DELIVERABLES
PRICING
each subject
that
contains
the
reports.
E
will print and
of material
Description
Evidence
of Completion
Due Date(s)
(Renewal 2 - 2019)
districts.
(Renewal 3 - 2020)
F
Component B
3-8 Math
$0.00
$0.00
1031
1032
1033
1034
1035

reporting


districts.

2015
$0.00
2016
(Renewal 1 - 2017)
$0.00
$0.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00

districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00

districts.

2015
$2,400.00
2016
2017
(Renewal 1 - 2018)
$2,400.00
$2,400.00
$2,400.00
(Renewal 2 - 2019)
$2,400.00
1036
1037
1038
1039

reporting

1040
$0.00
$0.00
$0.00
1041
1042
1043
1044
1045
1046


A
1
1047
1048
1049

Attachment
Reference 7.5.5
1b - DELIVERABLES
following for
PRICING
who
received
B
C
districts.
1050
1051

Due Date(s)
(Renewal 3 - 2020)
2015
(Renewal 1 - 2017)
(Renewal 3 - 2019)
districts.
Component B
3-8 Math
$2,400.00
$0.00
$0.00
$0.00
$0.00
$0.00
1052
1053
1054
1055


1056

districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
(Renewal 3 - 2019)
$0.00

districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00

districts.

2015
$0.00
2016
$0.00
1057
1058
1059
1060
1061


1062
1063
1064
1065


A
1
1066
1067
1068
1069
1071
1072
1073
1074
1076
1077
1078
1079
1081
1082
1083
1084
1085
1087
1088
1089

B Submit proof of
reports and
media for
the late reporting
Contractor
Deliverable


C drives. The
Department after
all reports
and media are
Deliverable
Description

(2) Provide proof
of delivery
Evidence
of Completion
districts.
Due Date(s)
Component B
3-8 Math
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00



September 2015
September 2016
$0.00
$0.00
$0.00
$0.00
$0.00



March 2016
$0.00
$0.00
$0.00
$0.00
$0.00



March 2015
March 2016
March 2017



March 2017

September 2015
September 2016
$4,400.00
$4,400.00
$4,400.00
$4,400.00
$4,400.00
$4,400.00
$0.00
$0.00
$0.00
$0.00
$0.00
1090

A
1
1092
1093
1094
1095
1097
1098
1099
1100


1101
1103
1104
1105
1106
1107



Performance Tasks

1108
1109
1110
1111
Due Date(s)
Component B
3-8 Math
November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00

March 2015
March 2016
March 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

June 2015
June 2016
June 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
A
1

Performance Tasks
B
Component
A, B, C, D or E (A-D): _____B_____
or on the
performance-based
that respondents
C
D
E
Due Date(s)
(Renewal 3 - 2019)
F
Component B
3-8 Math
$0.00
1112
1113
1114
1115
1116
1117

Performance Tasks

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00

2015
$2,633,087.00
$0.00
$0.00
$0.00
$0.00
$0.00
1118
1119
1120
1121
1122
1123

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$2,151,059.00
$1,038,406.00
$1,109,820.00
$1,151,323.00
A
1

B
Attachment
performance-based
Specify Component
A, B, C, D or E (A-D): _____B_____
that respondents
D
E
Due Date(s)
(Renewal
3
2020)
F
Component B
3-8 Math
$1,096,250.00
1124
1125
1126
1127
1128

Tasks

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
$0.00
$0.00
1129
1130
1131
1132
1133

Performance Tasks
$0.00
$0.00
$0.00
A
1

Performance Tasks
B
Specify
Component
A, B,on
C, D or
(A-D):
_____B_____
of a students
response
theEfinal
handscoring
report.
D
E
Due Date(s)
(Renewal 3 - 2019)
F
Component B
3-8 Math
$0.00
1134

Performance Tasks
1135
1136
1137
1138
1139


2015
1140
1141
1142
1143
1144

Performance Tasks
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
A
1
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167

Performance Tasks
B
Component
A, B, C, D or E (A-D): _____B_____
or on the
performance-based
that respondents
C
D
E
Due Date(s)
(Renewal 2 - 2019)
(Renewal 3 - 2020)


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5
June 2015
June 2016
January 2016
January 2017
January 2015
January 2016
January 2017
June 2015

the SFTP.
June 2016
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$28,226.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
all assessments
Reference 7.5.5

all assessments
Reference 7.5.5

all assessments
Reference 7.5.5

ELA/Mathematics

ELA/Mathematics
D
the SFTP.
Due Date(s)
Component B
3-8 Math
October 2015
October 2016
the SFTP.
January 2016
the SFTP.

April 2015
April 2016
April 2017

December 2015
Reference 7.5.4.2
Reference 7.5.4.2

October 2016
October 2017
October 2015
and ELA/Mathematics
October 2017
Reference 7.5.4.2
December 2015
Reference 7.5.4.2
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$6,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00

A
1
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
Scoring
and Reporting
Deliverable
Category
will create and
posted toofthe
SFTP.
Description
Completion
Due Date(s)
Reference 7.5.4.2
March 2016
Reference 7.5.4.2
October 2015
October 2017
Reference 7.5.4.2

ELA/Mathematics

Reference 7.5.4.2

Retake

Fall EOC:Contractor
Development
of the history
Deliverable
1225
1226
1227
1229
1230

October 2015
October 2017
November 2015

May 2016

Retake
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1231
1232
1234
1235
1236
1237
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1255
1256

D
E
rule specified in
Deliverable
Due Date(s)

rescoring.

Reference 7.5.6.5

management system.
log.

manual rescoring.

Reference 7.5.6.5

management system.
log.


Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

October 2015
October 2016
April 2016
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2017
April 2015
$0.00
$0.00
$0.00
$0.00
$0.00
$2,000.00
April 2016
April 2017
October 2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
October 2016
December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

1
1257
1258
1260
1261
1262
1263
1264
1266
1267
1268

B

C
Deliverable
Description
in the
Reference 7.5.6.5

D
management system.
log.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.


Reference 7.5.5

districts.


Protocol (SFTP).

rescoring.
1269
1270
1271
1272
1273
1274
1275
1276
1278
Studies
1279
1280
1281

E
Due Date(s)
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

July 2015
$0.00
$28,620.00
July 2016
July 2017
$28,620.00
$28,620.00

$28,620.00
$28,620.00
$28,620.00
May 2014
May 2015
$96,473.00
$2,000.00
May 2016
$0.00
$0.00

$0.00
$0.00
1282

A
1
1284 Studies
1285
Administrations
1286
1287
1288
Studies
1290
1291
1292
1293
1294
Studies
1296
1297
1298
1299
1300
1303 Studies
1304
1305
D
deliverable.
Plans
Administrations

Specifications


Due Date(s)
Component B
3-8 Math
July 2014
July 2015
$39,726.00
$39,726.00
July 2016
$39,726.00
$39,726.00

$39,726.00
$39,726.00
July 2014
July 2015
$6,000.00
$2,000.00
July 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$6,000.00
$2,000.00

August 2014
August 2015
August 2016
December 2014
December 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$40,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00

Studies
A
1306
1307
1308
1309
1310
1311
1312
1313

Studies

1314 Studies
1315
1316
1317
1318
1321
1322
1323
Studies

Specifications
B
Specify
Component
A,plans
B, C, D or
(A-D): Department
_____B_____
procedures,
scoring
(2)EProvide
D
E
Due Date(s)


posted to SFTP.
May 2015
May 2016
May 2017
2016

via SFTP.
2017


via SFTP.
2016
(Renewal 1 - 2018)
(Renewal 2 - 2019)

(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Component B
3-8 Math
$2,000.00
$200,000.00
$200,000.00
$200,000.00
$200,000.00
$200,000.00
$200,000.00
$3,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00

Studies
1324
1327

B

Technical Reports


posted to SFTP.


Reference 7.6.3.3

November 2014
posted to SFTP.



October 2014
SFTP.
Studies
1328
1329
1330
Studies
Attachmentmust
1b - DELIVERABLES
(1) SubmitPRICING
include analyzing theSpecify
most recent
SSR
files
and
via
SFTP.
D
E
the demographic
Description
Due Date(s)
The specifications
will also address
(Renewal 3 - 2020)
F
Component B
3-8 Math
$0.00
December 2015
December 2016
December 2017
$125,905.00
$202,094.00
$202,094.00
$202,094.00

$202,094.00
$202,094.00
$20,412.00
$0.00
1332
1333 Studies
1334
Studies


May 2015
$697,578.00
$0.00
$85,970.00
$0.00
1336

A
1
C
7.6.3.4
Studies
Setting Work Plan

Studies

Studies

Technical Report
SFTP.
Due Date(s)
Component B
3-8 Math
$2,000.00
1337
$161,561.00
1338
1339
Studies


7.6.3.4.
Department.
renewal)
SFTP.
$30,761.00
$9,000.00
$5,000.00
1341

A
1
Studies
Report
Due Date(s)
August 2015




Reference 7.6.3.6
SFTP.
F
Component B
3-8 Math
$18,407.00
$5,000.00
1343
Studies
1344
1345
Studies

Report
$17,819.00
$0.00
$45,291.00
$0.00
1347
1348 Studies

across devices.
input types.
concordance.
characteristics.
$5,400.00

Scaling, Equating,
A Scoring,
Studies
1349
Studies
for conducting
Delivering
Assessments
and detailed
Contractor
Deliverable
Description
across devices.
input types.
concordance.
characteristics.

D
email
Evidence of
Completion

with TAC Committee


August 2015



November 2015
SFTP.
Due Date(s)
Component B
3-8 Math
$0.00
$29,752.00
$0.00
1351
Studies
1352
$900.00

Studies
A
1353
Studies
1355
Studies

B assessments
November 2015
SFTP.
C and events, as
D
TAC members.
of Completion


7.6.3.6.2

August 2016

Assessments


November 2015
SFTP.
Due Date(s)
Component B
3-8 Math
$0.00
$9,868.00
$0.00
$900.00
$0.00
1357

A
1
Studies
B
Assessments
1358
C
7.6.3.6.3
D
August 2016
Due Date(s)
Component B
3-8 Math
$900.00
$0.00
1359
1361
1362
1363
1364
1365
1367
1368
1369
1370
1371
1373
1374
1375
1376
1377
1379
1380
1381
1382

7.7.2

report

Meetings


previous invoice.

ITN.


Action Item Report

work. ITN Ref 7.7.6

April 15, 2014

April 15, 2015
April 15, 2016
$66,198.00
$27,930.00
$27,930.00
$27,930.00
$27,930.00

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$27,930.00
$121,879.00
$122,426.00
$122,851.00
$123,260.00
$124,154.00
$124,566.00
$1,139,500.00
$1,276,463.00
$1,287,729.00
$1,102,105.00
$1,122,579.00
$1,129,158.00
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$112,459.00
$112,459.00
$112,459.00
$112,459.00
$112,459.00
Program Management
A
1
1383
1385
1386
1387
1388
1389
1391
1392
1393
1394


Specify
E (A-D): _____B_____
status Component
management
those
leading
up
to
previous
invoice
period. (Quarterly
invoices to occur
C
D
E
Due Date(s)
(Renewal 3 - 2019)
work. ITN Ref 7.7.6
F
Component B
3-8 Math
$112,459.00

November 2015
November 2016
November 2017
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00

ITN Ref 7.7.6
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$69,917.00
$69,797.00
$69,797.00
$69,797.00
$69,797.00
Program Management
A
1

B

Specify
Component
A,level
B, C,ofD or
E (A-D): reports.
_____B_____
by program
task and
accounting
and
balances
of
February,
May,
August,
and
C
D November)
ITN Ref 7.7.6
Due Date(s)
Component B
3-8 Math
(Renewal 3 - 2019)
$69,797.00

$43,167.00
1395
Program Management
Ref. 7.7.10
1396

Specify Component A, B, C, D or E (A-D): _____C_____
A
1
3
4
5
6
7
8
Test Development
Document
Test Development

Specifications
9
10
11
12
13
C
7.3.10
via SFTP.
Due Date(s)
June 1, 2014
June 1, 2015
June 1, 2016
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5
F
Component C
9-11 ELA
$38,237.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
14
Test Development
15
16
17
18
19


7.5.4.5

SFTP.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

7.5.4.5

SFTP.
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
20
Test Development
21
22
23
24
25

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$20,191.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
Test Development
A
1

B
E (A-D): _____C_____
C
D
E
andDescription
the Department. The
Deliverable
Due Date(s)
(Renewal 3 - 2020)
7.5.4.5
F
Component C
9-11 ELA
$2,000.00
26
Test Development
27
28
29
30
31

7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
7.5.4.5

SFTP.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
32
Test Development
33
34
35
36
37
$0.00
$0.00
$0.00
$0.00
$0.00
38
Test Development
39
40
41
42
43
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
44

A
1
Test Development
45
46
47
48
49
50
51 Test Development
52
53
54
55
56
B
Specifications
C
7.5.4.5
D
SFTP.
7.3.10
Test Development


Test Development

each scoring guide.
F
Component C
9-11 ELA
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
June 1, 2014
June 1, 2015
June 1, 2016
$15,797.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$35,812.00
57
58
59
60
61
62

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$15,722.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
Test Development
A
1

B
each scoring guide.
Attachment
1b - DELIVERABLES
PRICING
Specify
Component
A, These
B, C, D or E (A-D): _____C_____
on student
reports.
C
D
E
Due Date(s)
(Renewal
3
2020)
F
Component C
9-11 ELA
$2,000.00
63
Test Development

Assessment Reports
64
65
66
67
68

Reference 7.3.7.5

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$67,402.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
69
Test Development
70
71 Test Development
72
73
74
75
76 Test Development


$73,872.00

management report
Reference 7.3.4.2

February 1, 2017
management report
$1,985,716.00
$2,026,166.00
$1,076,209.00
$941,222.00
$800,081.00
Review

$2,025,111.00

management report

A
Development
1 Test Deliverable
Category
77
78
79
80
81 Test Development
82
83
84
85
86
87 Test Development
88
89
90
91 Test Development
92
93
94
95 Test Development
96
97
98
99
Periodic:Contractor
Field Test Item
or Description
Deliverable
Deliverable
Review

email
Evidence of
Completion
Due Date(s)
May 1, 2016
management report
July 15, 2015
July 15, 2016
F
Component C
9-11 ELA
$16,033.00
$16,033.00
$16,033.00
$16,033.00
$16,834.00
$24,381.00
$24,381.00
$24,381.00
$24,381.00
$24,381.00

Reference 7.3.0

management report.
December 1, 2015
December 1, 2016
$11,652.00
$11,652.00
$11,652.00
$11,652.00


management report.
December 1, 2015
December 1, 2016
$9,679.00
$4,555.00
$4,555.00
$4,555.00


previous invoice.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$55,695.00
$56,441.00
$57,205.00
$58,783.00
$59,860.00
Test Development
A
1
100
102
103
104
105
106
108
109
110
111
112
Test Administration
B

Component
A, B, C, D or
E (A-D):
_____C_____
for the purpose
of reviewing
previous
invoice.
supporting the
C





113
114
115
116
Due Date(s)
Component C
9-11 ELA
(Renewal 3 - 2020)
$60,029.00

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
the report.
2015
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$1,000.00
$1,000.00
$1,000.00
$1,000.00

2016
$1,000.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
$1,000.00
$1,000.00
$1,000.00
117
Test Administration
118
119
120
121

the report.
Test Administration
A
1

B
1b - DELIVERABLES
the report.PRICING
Specify
Component
A, by
B, C, D or E (A-D): _____C_____
has been
conducted
C
D
Due Date(s)
Component C
9-11 ELA
(Renewal 3-2020)
$1,000.00

a report.
the report.

2015
$2,000.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

the report.

2015
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
the report.

2015
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
122
Test Administration
123
124
125
126
127
128
Test Administration
129
130
131
132
133
Test Administration
134
135
136
137
138

A
1
Test Administration
139
140
141
142
143
F
Component C
9-11 ELA
the report.
Due Date(s)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00

the report.

2015
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00
the Specifications.

2015.
$0.00
144
Test Administration
145
146
147
148
149
150
152
153
154
155
156

$0.00
$98,074.00
2016
$81,868.00
(Renewal 1- 2017)
$87,228.00
(Renewal 2-2018)
$91,416.00
(Renewal 3 - 2019)
$92,033.00
$0.00
the Specifications.
2016

A
1
Test Deliverable
Administration
Category
158
159
160
161
162
164
165
166
167
168
169
171
172
173
174
175
Spring ELA/L
Retake:Deliverable
all processes,
of the approved
specifications.
Contractor
Deliverable
Description
of Completion
the Specifications.
E
2016
F
Component C
9-11 ELA
$1,000.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
the Specifications.
administration.
2015.
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$39,955.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
$40,820.00
$33,966.00
$36,372.00
$37,738.00
$38,192.00
$0.00
$0.00

administration.
the Specifications.
2016
$0.00
(Renewal 1- 2017)
$0.00
(Renewal 2-2018)
$0.00
(Renewal 3 - 2019)
$0.00
$0.00
administration.
the Specifications.
2015.

A
1
Test Deliverable
Administration
Category
177
178
179
180
181
183
184
185
186
187
188
190
191
192
193
194
195
Winter EOC:
DevelopDeliverable
packaging and
all processes,
of the approved
specifications.
Contractor
Deliverable
Description
of Completion
administration.
the Specifications.

2015.
Component C
9-11 ELA
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
administration.
the Specifications.
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
administration.
the Specifications.
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
197
198
199
200
201
Test Administration
202
203
204
205
206
207
208
Test Administration
209
210
211
212
213
214
215
Test Administration
216
217
218
219
220
221
222
B
memos, etc.
districts.

memos.


districts.

memos.


districts.

memos.


districts.
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component C
9-11 ELA
$0.00
$188,019.00
$192,792.00
$196,825.00
$203,368.00
$202,985.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$257,340.00
$305,950.00
$350,882.00
$377,733.00
$389,093.00
$390,647.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$259,914.00
$227,780.00
$219,031.00
$184,386.00
$190,454.00
$190,756.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
224
225
226
227
228
Test Administration
229
230
231
232
233
234
Test Administration
235
236
237
238
239
240
241
Test Administration
242
243
244
245
246
247
and accommodations.
districts.

and accommodations.


districts.



districts.

accommodations.


districts.
Due Date(s)
negotiations.)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component C
9-11 ELA
$0.00
$68,567.00
$71,314.00
$75,323.00
$77,282.00
$77,969.00
$2,000.00
$80,380.00
$81,889.00
$86,585.00
$89,717.00
$89,655.00
$0.00
$511,157.00
$527,368.00
$544,683.00
$578,592.00
$597,470.00
$600,208.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
248
249
250
251
252
253
Test Administration
254
255
256
257
258
259
260
Test Administration
261
262
263
264
265
266
267
Test Administration
268
269
270
271
272
273
accommodations.
districts.

accommodations.


districts.

accommodations.


districts.


and 7.4.7.4

districts.
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A
1
Test Administration
B
districts.

and 7.4.7.4

districts.
and rosters.
and 7.4.7.4

districts.

labels and rosters.

and 7.4.7.4

districts.

and 7.4.7.4

districts.

280
281
282
283
284
285
286
Test Administration
287
288
289
290
291
292
Test Administration
293
294
295
296
297
298
Test Administration
299
and 7.4.7.4
274
275
276
277
278
279
Test Administration

labels and rosters.
Due Date(s)
negotiations.)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component C
9-11 ELA
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Test Administration
A
1
300
301
302
303
304
305
Test Administration
306
307
308
309
310
311
312
314
315
316
317
318
320
321
322
323
324
326
327

preidentification
labels and rosters.

labels and rosters.

critical date
Description
and 7.4.7.4

destination
address,
Evidence of
Completion
districts.

and 7.4.7.4

districts.
Due Date(s)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
7.4.8

November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00


7.4.8

May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00


7.4.8

May 2016
$0.00
$139,096.00
$262,891.00

Test Administration
A
1
328
329
330
331
333
334
335
336
337
339
340
341
342
343
345
346
347
348
349
350
352
353
354
355

Deliver
B
system for
each assessment.
Contractor
Deliverable

was operable for
the invoice
Deliverable
Description
7.4.8

E
occurred during
the period,
signed by the
Evidence
of Completion
Due Date(s)

assessment.

7.4.8

November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

February 2016
February 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
assessment.
7.4.8

August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
F
Component C
9-11 ELA
$262,891.00
$262,891.00
$262,891.00
$262,891.00
Test Administration
A
1
356
Specify Component
A, B,
C, D or
(A-D):
_____C_____
of the contract
and
(orEno)
malfunctions
assessment.
was
operable
for
the
invoice
period.
ITN
Reference
occurred
during
the
period,
B
C
D signed by the
E
7.4.8
Due Date(s)
F
Component C
9-11 ELA
$0.00
$0.00
357
359
360
361
362



November 2015
November 2016
$0.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
363
365
366
367
368

May 2016
May 2017
$0.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
369
371
372
373
374
375

administration.


May 2015
May 2016
May 2017
$0.00
$1,874,527.00
$2,147,007.00
$2,548,678.00
$2,801,304.00
$2,753,427.00
$2,665,824.00
376
378
379
380
381



November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
Test Administration
A
1
382
384
385
386
387
388
390
391
392
393
394
395
397
398
399
400
401
402
404
405
406
407
408

B
(1) ProvidePRICING
A,ofB,the
C, D or
(A-D):
_____C_____
met theComponent
requirements
(orEno)
malfunctions
C
D
E
Due Date(s)












F
Component C
9-11 ELA
$0.00
February 2016
February 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
May 2015
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
November 2014
November 2015
November 2016
$31,591.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
Test Administration
A
1

B

Attachment
1bon
- DELIVERABLES
PRICING
and approved
a
the practice
Specify
Component
A, B, C, D or
E (A-D): _____C_____
that
offers students
requirements
C
D
Due Date(s)
Component C
9-11 ELA
$0.00
409
Test Administration
410
412
413
414
415
417
418
419
420
421
423
424
425
426

the contract.


training.
September 2014
7.4.9.8
administration.
August 2015
August 2016
November 2016
November 2017
previous year.
November 2015
November 2016
November 2017
$207,841.00
$122,583.00
$122,583.00
$122,583.00
$122,583.00
$122,583.00
$268,964.00
$222,793.00
$228,205.00
$237,638.00
$243,432.00
$245,206.00
$9,724.00
$12,229.00
$13,663.00
$14,163.00
$14,487.00
Test Administration
A
1
Specify Component
A, B, C,for
D or
E (A-D):
_____C_____
and other materials
materials
and
a
period
of
at
least
six
months,
and
used
documents
end
of
each
assessment
year
B
C
E
Due Date(s)
F
Component C
9-11 ELA
$14,507.00
427
Test Administration
428
429
430
431
432
433
Test Administration
434
435
436
437
438
439
Test Administration
440
441
442
443
444
445
446
Test Administration
447
448
449
450
451

comment forms.

comment forms.


negotiations.)
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015

comment forms.
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$2,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
Test Administration
A
1
452
Test Administration
453
454
455
456
457
458
Test Administration
459
460
461
462
463
464
465
Test Administration
466
467
468
469
470
471
472
Test Administration

comment forms.
B
comment forms.

comment forms.

comment forms.

Trial Test

1b -forms.
online comment
Component A, B, C, D or E (A-D): _____C_____
and 7.4.7.7
C
Due Date(s)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
7.4.8 and 7.4.9.5
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$58,275.00
473

A
1
Test Administration
474
475
476
477
478
479
Test Administration
480
481
482
483
484
485
Test Administration
486
487
488
489
490
491
492
Test Administration
493
494
495
496
497
498
B



versions)
F
Component C
9-11 ELA
Reference 7.3.11.2
Due Date(s)
negotiations.)

Reference 7.3.11.2
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$12,735.00
$12,735.00
$12,735.00
$12,735.00
$12,735.00
$0.00

Reference 7.3.11.2
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$13,045.00
$13,045.00
$13,045.00
$13,045.00
$13,045.00
$0.00

Reference 7.3.11.2
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$66,045.00
$39,035.00
$39,035.00
$39,035.00
$39,035.00
$39,035.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
499
500
501
502
503
504
Test Administration
505
506
507
508
509
510
511
Test Administration
512
513
514
515
516
517
518
Test Administration
519
520
521
522
523
524
versions)
Reference 7.3.11.2
D
Due Date(s)
negotiations.)
Component C
9-11 ELA
$0.00
versions)
Reference 7.3.11.2
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

versions)
Reference 7.3.11.2
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00


Reference 7.4.1.
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$17,315.00
$17,315.00
$18,801.00
$18,801.00
$19,882.00

A
1
Test Administration
525
526
527
528
529
530
Test Administration
531
532
533
534
535
536
537
Test Administration
538
539
540
541
542
543
Test Administration
544
545
546
547
548
549
Reference 7.4.1.
Due Date(s)
negotiations.)
testing
Reference 7.4.1.


negotiations.)
Reference 7.4.1.


Reference 7.4.1.
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component C
9-11 ELA
$0.00
$13,344.00
$13,344.00
$14,226.00
$15,191.00
$15,191.00
$0.00
$108,184.00
$103,794.00
$138,253.00
$132,325.00
$137,652.00
$139,388.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
B
550
551
552
553
554
555
556
Test Administration
557
558
559
560
561
562
563
565
566
572
573
574
576
577
578
Reference 7.4.1.
Reference 7.4.1.

567
568
569
571

ITN Reference 7.5.1
D
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016

management system.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
April 2014
April 2015
April 2016

management system.


management system.

F
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
September 2015
September 2016
$1,000.00
$1,000.00
$1,000.00

January 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
January 2017
A
1
579
581
582
583
584
585
587
588
589
590
592
593
594
595
597
598
599
600
601
603
604
605
606
607
609


monitor district
(1) EPreID
frequency
Specifyand
Component
A, B, C, D or
(A-D):
_____C_____
C reports of issues
Deliverable
Description
assessments. ITN
Reference
7.5.2
management system.
management system.
E
Due Date(s)
January 2015
January 2016
January 2017

review.

management system.
review.

management system.
review.


in .txt format

September 2015
September 2016
October 2015
F
Component C
9-11 ELA
$1,000.00
$1,000.00
$500.00
$500.00
$500.00
$500.00
$500.00
$0.00
$0.00
$0.00
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

management system.

January 2015
January 2016
January 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

management system.

June 2015
June 2016
June 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00

Protocol (SFTP).

September 2015
September 2016

A
1
610
611
612
614
615
616
617
619
620
621
622
623
625
626
627
file layout
and Full-State
layout for each
administration for
Contractor
Deliverable
Description
in .txt format

management system
andofSecure
File Transfer
Evidence
Completion
Protocol (SFTP).

file in .txt format

Protocol (SFTP).
January 2016


Protocol (SFTP).
January 2015
January 2016
January 2017
format

Protocol (SFTP).
September 2015
Due Date(s)
January 2017
September 2016
F
Component C
9-11 ELA
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
A
1
628
630
631
632
633
635
636
637
638
639
641
642
643
644

Specify
Component
A, B,toC, D or
(A-D):
_____C_____
with
the Department
(1) EFinal
PreID
format


File Transfer
D
Protocol (SFTP).

.txt format


Protocol (SFTP).
October 2015
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00

.txt format


Protocol (SFTP).
January 2015
January 2016
January 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

.txt format


Protocol (SFTP).
June 2015
June 2016
June 2017
$0.00
$0.00
$0.00
$0.00
$0.00
E
Due Date(s)
F
Component C
9-11 ELA
$0.00
A
1
645
647
648
649
650
652
653
654
655
657
658
659
660
661
663
664
665
666
668
669
670
671

.txt format

Attachment
1b - DELIVERABLES
PRICING
the Department
to
(1) Final PreID
Component
A, B,
E (A-D): _____C_____
for each
administration
forC, D or
management
D the SFTP.
Deliverable
of Completion
each PreID window
closes,Description
the contractor will
the file layout
and data file.




E
Due Date(s)

$1,000.00
$1,000.00
$1,000.00

May 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
May 2017
May 2015
May 2016
May 2017





$0.00
October 2015
October 2016

transcription logs
F
Component C
9-11 ELA

October 2015
October 2016
December 2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

May 2015
$0.00
$0.00

A
and Reporting
1 Scoring
Deliverable
Category
673
674
675
676
677
679
680
681
682
683
685
686
687
688
690
691
692
Spring EOC:
Development
will use a Special
Document
Contractor
Deliverable
Description

logofposted
to SFTP.
Evidence
Completion




7.5.6.2, and 7.5.6.4



7.5.6.2, and 7.5.6.4

E
Due Date(s)
May 2016
May 2017
July 2015
July 2016
July 2017
October 2015
October 2016
March 2016
March 2017
F
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$101,604.00
$87,016.00
$93,459.00
$97,415.00
$97,415.00
$9,000.00
$9,000.00
$10,000.00
$10,000.00
A
1
693
695
696
697
698
699
701
702
703

Specify
A, B,toC, D or
(A-D):
_____C_____
withComponent
the Department
(1) EFinal
approved

and CBT extract
C
7.5.6.2, and 7.5.6.4

data files



7.5.6.2, and 7.5.6.4


7.5.6.2, and 7.5.6.4

E
Due Date(s)
March 2015
March 2016
March 2017
October 2015
October 2016
F
Component C
9-11 ELA
$10,000.00
$181,870.00
$187,806.00
$175,622.00
$176,894.00
$184,024.00
$186,225.00
$0.00
$0.00
$0.00
$0.00
A
1

data files:
B

Specify
A, B,toC, D or
(A-D):
_____C_____
withComponent
the Department
(1) EFinal
approved
and CBT extract
C
7.5.6.2, and 7.5.6.4

E
Due Date(s)
F
Component C
9-11 ELA
$0.00
704
706
707
708


7.5.6.2, and 7.5.6.4

December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
709
711
712
713
714


7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017
$0.00
$0.00
$0.00
$0.00
$0.00
A
1
715
717
718

719
720
721
723
724
725
726
728
729
730



Component
B, C, D or
E (A-D):and
_____C_____
for scan/edit
and CBT A,
extract
scan/edit,
the
contractor
online
management
system
C
D and SFTP.
E
Due Date(s)
7.5.6.2, and 7.5.6.4

7.5.6.2, and 7.5.6.4


the data files.

the data files.
F
Component C
9-11 ELA
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00

April 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2017
A
1
731
733
734
735
736
737
739
740
741
742
744
745
746
747
749
750
751
752
753
755
756
757
758
759
761

B of the
data files

Specifythe
Component
A, and
B, C, D or
(A-D): _____C_____
full state SSR
(1) EAnomalous
the data files.

the data files.
E
Due Date(s)
April 2015
April 2016
April 2017

data files

the data files.
data files
the data files.
data files
the data files.

score data files

the data files.


Reference 7.5.5

October 2015
October 2016
December 2015
December 2016
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017

October 2015
October 2016
F
Component C
9-11 ELA
$1,000.00
$32,421.00
$18,710.00
$14,037.00
$9,057.00
$9,057.00
$9,057.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00

A
762
763
764
766
767
768
769
771
772
773
774
775
777
778
779
780
782
783
B
and DSRsContractor
Deliverable of


DSR data files.

and DSR data files.

C
develop the State
Student Description
Deliverable
Reference 7.5.5

and online
of management
Completion system.
Due Date(s)

April 2016
F
Component C
9-11 ELA
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

Reference 7.5.5

April 2017

Reference 7.5.5


Reference 7.5.5

October 2015
October 2016
$0.00
$0.00
$0.00

Reference 7.5.5

December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00

A

SSRs and
B
SSR/DSR Contractor
and multiple
784
785
787
788
789

790
791
793
794
795
796
797
799
800
801
802
804
805




C the Department to
Student Results
(DSR) file layouts
and use the layout
Deliverable
Description
Reference 7.5.5

E
email
Evidence of
Completion
Due Date(s)

Reference 7.5.5


Reference 7.5.5


7.5.5

October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00

$1,000.00
$1,000.00


April 2016
April 2017
F
Component C
9-11 ELA
$0.00
$0.00
April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00

A

of SAR/DAR
file layouts
and multiple
Contractor
Deliverable

Results (SAR) and
District Aggregated
Deliverable
7.5.5

E
email
Evidence of
Completion
Due Date(s)

7.5.5


and DAR data files.

7.5.5

October 2015
October 2016
$0.00
$0.00
$0.00


7.5.5

December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
806
807
809
810
811
812
813
815
816
817
818
820
821
822

DAR data files.
F
Component C
9-11 ELA
$1,000.00
$1,000.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
December 2015
$0.00
$0.00
823

A
1
825
826
827
B
828
829
831
832

833
834
835
837
838
839
840
842
843
844

data files.

data files.
E
Due Date(s)
April 2015
April 2016
April 2017
F
Component C
9-11 ELA
7.5.5

7.5.5


7.5.5.2 and 7.5.6.6
November 2016
$1,000.00
$1,000.00
$1,000.00

$1,000.00
$1,000.00

7.5.5.2 and 7.5.6.6
June 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
June 2017
A
1
845
847
848

B
data files.

849
850
851
853
854
855
856
858
859
860
Attachment
- DELIVERABLES
PRICING
Department1b
Scoring
(1) Final late
to develop
the
State
Aggregated
to
online
management
system and SFTP.
C to develop the
E
Deliverable
Due Date(s)
scores and codes
7.5.5.2 and 7.5.6.6
7.5.5.2 and 7.5.6.6
F
Component C
9-11 ELA
$1,000.00
June 2015
June 2016
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
7.5.5.2 and 7.5.6.6
November 2016
$0.00
$0.00
$0.00

$0.00
$0.00

data files.
February 2015
February 2016
$0.00
$0.00
$0.00
$0.00
$0.00

7.5.5.2 and 7.5.6.6
861

A
1
863
864
865
B
data files.
866
7.5.5.2 and 7.5.6.6
867
869
870
871
872

data files.

7.5.5.2 and 7.5.6.6
873
875
876
877

7.5.5.3 and 7.5.6.7
878
880
881
882
7.5.5.3 and 7.5.6.7
E
Due Date(s)
June 2015
June 2016
June 2017
F
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
December 2015
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
October 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
October 2017
883

A
1
885
886
887
888
889
891
892
893
894
896
897
898
899
901
902
903
904
905
907
908
F
Component C
9-11 ELA
assessments.
7.5.5.3 and 7.5.6.7
Due Date(s)
October 2015
October 2016
October 2017

7.5.5.3 and 7.5.6.7
December 2015

7.5.5.3 and 7.5.6.7
March 2015
March 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

7.5.5.3 and 7.5.6.7
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

October 2015
October 2017
7.5.5.3 and 7.5.6.7
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
909
910
911
913
914
915

files for all
assessments.
Contractor
Deliverable


918
919
920
921
922
923
924
925
926


reporting
F
Component C
9-11 ELA
$0.00
$0.00
$0.00
September 2015
September 2016
September 2017
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

2015
$1,000.00
$1,000.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2017
reporting
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
916
917

E
of all assessments.
data files.
Description
of Completion
Due Date(s)
7.5.5.3 and 7.5.6.7
927
928
929
930
931
932
933

initial reporting


2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$50,000.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
(Renewal 3 - 2020)
$1,000.00

A
1
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955


D
E
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
956
957
958
F
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$177,489.00
(Renewal 3 - 2020)
jump drives.
2015
$177,489.00
(Renewal 1 - 2017)
$175,378.00
districts.
A
1
959

- DELIVERABLES
deliver initial1b
reporting
(1) ProvidePRICING
CD/DVDs
or
jump
drives.
The
the
samples
jump drives.
C and media are
E
Evidence of
Completion
Due Date(s)
destination
address,
(Renewal 3 - 2019)
districts.
F
Component C
9-11 ELA
$175,378.00
$175,378.00
960
961
962
963


964

jump drives.
districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$323,266.00
(Renewal 3 - 2020)
$122,230.00

jump drives.
districts.

2015
$581,910.00
2016
2017
(Renewal 1 - 2018)
$533,358.00
$538,070.00
$593,856.00
(Renewal 2 - 2019)
$593,856.00
(Renewal 3 - 2020)
$593,856.00
$123,266.00
$121,908.00
$122,230.00
965
966
967
968


969
970
971
972
973
974
975

initial reporting


jump drives.
districts.

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
976
977

initial reporting

$0.00
jump drives.
2015

A
1
978
979

Student
B
Submit proof
of delivery:
Development
Contractor
Deliverable
initial reporting

submit proof
of delivery to the
Description
980

districts.
Due Date(s)
Component C
9-11 ELA
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
(Renewal 3 - 2019)
$0.00
981
982
983
984
985
986

initial reporting


jump drives.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00

initial reporting


jump drives.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00

reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
987
988
989
990
991
992
993
994
995
996
997


A
1

B
PRICING
Component
A, B, C,
E (A-D): _____C_____
each subject
that contains
theD or
reports.
will be
C
E
Due Date(s)
(Renewal 3 - 2019)
districts.
F
Component C
9-11 ELA
$0.00
998
999
1000
1001
1002



reports.
districts.

2016
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00



reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00


reports.
districts.

2015
2016
(Renewal 1 - 2017)
$0.00
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012

reports
$0.00
$0.00
A
1
1013

Tasks
B
reports
D
E
a separate page
that includes an
following for each
shipment:
Description
Evidence
of Completion
Due Date(s)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
districts.
F
Component C
9-11 ELA
$0.00
$0.00
1014
1015
1016
1017
1018

reports


reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00

reports


reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00



reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
A
1
1030

B
to selected
approval of
Attachment
1b - DELIVERABLES
PRICING
each subject
that
contains
the
reports.
E
will print and
of material
Description
Evidence
of Completion
Due Date(s)
(Renewal 2 - 2019)
districts.
(Renewal 3 - 2020)
F
Component C
9-11 ELA
$0.00
$0.00
1031
1032
1033
1034
1035

reporting


districts.

2015
$1,000.00
2016
(Renewal 1 - 2017)
$1,000.00
$1,000.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$1,000.00
$1,000.00

districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$1,000.00
(Renewal 3 - 2020)
$1,000.00

districts.

2015
$1,200.00
2016
2017
(Renewal 1 - 2018)
$1,200.00
$1,200.00
$1,200.00
(Renewal 2 - 2019)
$1,200.00
1036
1037
1038
1039

reporting

1040
$1,000.00
$1,000.00
$1,000.00
1041
1042
1043
1044
1045
1046


A
1
1047
1048
1049

Attachment
Reference 7.5.5
1b - DELIVERABLES
following for
PRICING
who
received
B
C
districts.
1050
1051

Due Date(s)
(Renewal 3 - 2020)
2015
(Renewal 1 - 2017)
(Renewal 3 - 2019)
districts.
Component C
9-11 ELA
$1,200.00
$0.00
$0.00
$0.00
$0.00
$0.00
1052
1053
1054
1055


1056

districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
(Renewal 3 - 2019)
$0.00

districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00

districts.

2015
$0.00
2016
$0.00
1057
1058
1059
1060
1061


1062
1063
1064
1065


A
1
1066
1067
1068
1069
1071
1072
1073
1074
1076
1077
1078
1079
1081
1082
1083
1084
1085
1087
1088
1089

B Submit proof of
reports and
media for
the late reporting
Contractor
Deliverable


C drives. The
Department after
all reports
and media are
Deliverable
Description

(2) Provide proof
of delivery
Evidence
of Completion
districts.
Due Date(s)
Component C
9-11 ELA
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00



September 2015
September 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00



March 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00



March 2015
March 2016
March 2017



March 2017

September 2015
September 2016
$2,200.00
$2,200.00
$2,200.00
$2,200.00
$2,200.00
$2,200.00
$0.00
$0.00
$0.00
$0.00
$0.00
1090

A
1
1092
1093
1094
1095
1097
1098
1099
1100


1101
1103
1104
1105
1106
1107



Performance Tasks

1108
1109
1110
1111
Due Date(s)
Component C
9-11 ELA
November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00

March 2015
March 2016
March 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

June 2015
June 2016
June 2017

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$263,097.00
$251,567.00
$264,773.00
$272,510.00
A
1

Performance Tasks
B
Component
A, B, C, D or E (A-D): _____C_____
or on the
performance-based
that respondents
C
D
E
Due Date(s)
(Renewal 3 - 2019)
F
Component C
9-11 ELA
$278,476.00
1112
1113
1114
1115
1116
1117

Performance Tasks

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

2015
$0.00
$221,275.00
$229,456.00
$241,985.00
$248,763.00
$253,075.00
1118
1119
1120
1121
1122
1123

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$3,311,908.00
$3,459,155.00
$3,570,675.00
$3,761,255.00
$3,882,573.00
A
1

B
Attachment
performance-based
Specify Component
A, B, C, D or E (A-D): _____C_____
that respondents
D
E
Due Date(s)
(Renewal
3
2020)
F
Component C
9-11 ELA
$3,896,135.00
1124
1125
1126
1127
1128

Tasks

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
$0.00
$0.00
1129
1130
1131
1132
1133

Performance Tasks
$0.00
$0.00
$0.00
A
1

Performance Tasks
B
Specify
Component
A, B,on
C, D or
(A-D):
_____C_____
of a students
response
theEfinal
handscoring
report.
D
E
Due Date(s)
(Renewal 3 - 2019)
F
Component C
9-11 ELA
$0.00
1134

Performance Tasks
1135
1136
1137
1138
1139


2015
1140
1141
1142
1143
1144

Performance Tasks
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
A
1
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167

Performance Tasks
B
Component
A, B, C, D or E (A-D): _____C_____
or on the
performance-based
that respondents
C
D
E
Due Date(s)
(Renewal 2 - 2019)
(Renewal 3 - 2020)


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5
June 2015
June 2016
January 2016
January 2017
January 2015
January 2016
January 2017
June 2015

the SFTP.
June 2016
F
Component C
9-11 ELA
$0.00
$0.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$28,226.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
all assessments
Reference 7.5.5

all assessments
Reference 7.5.5

all assessments
Reference 7.5.5

ELA/Mathematics

ELA/Mathematics
D
the SFTP.
Due Date(s)
Component C
9-11 ELA
October 2015
October 2016
the SFTP.
January 2016
the SFTP.

April 2015
April 2016
April 2017

December 2015
Reference 7.5.4.2
Reference 7.5.4.2

October 2016
October 2017
October 2015
and ELA/Mathematics
October 2017
Reference 7.5.4.2
December 2015
Reference 7.5.4.2
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$4,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$3,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00

A
1
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
Scoring
and Reporting
Deliverable
Category
will create and
posted toofthe
SFTP.
Description
Completion
Due Date(s)
Reference 7.5.4.2
March 2016
Reference 7.5.4.2
October 2015
October 2017
Reference 7.5.4.2

ELA/Mathematics

Reference 7.5.4.2

Retake

Fall EOC:Contractor
Development
of the history
Deliverable
1225
1226
1227
1229
1230

October 2015
October 2017
November 2015

May 2016

Retake
F
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$5,000.00
$1,000.00
$1,000.00

A
1231
1232
1234
1235
1236
1237
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1255
1256

D
E
rule specified in
Deliverable
Due Date(s)

rescoring.

Reference 7.5.6.5

management system.
log.

manual rescoring.

Reference 7.5.6.5

management system.
log.


Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

October 2015
October 2016
April 2016
F
Component C
9-11 ELA
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2017
April 2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2016
April 2017
October 2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
October 2016
December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

1
1257
1258
1260
1261
1262
1263
1264
1266
1267
1268

B

C
Deliverable
Description
in the
Reference 7.5.6.5

D
management system.
log.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.


Reference 7.5.5

districts.


Protocol (SFTP).

rescoring.
1269
1270
1271
1272
1273
1274
1275
1276
1278
Studies
1279
1280
1281

E
Due Date(s)
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017
F
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

July 2015
$0.00
$12,266.00
July 2016
July 2017
$12,266.00
$12,266.00

$12,266.00
$12,266.00
$12,266.00
May 2014
May 2015
$41,346.00
$2,000.00
May 2016
$0.00
$0.00

$0.00
$0.00
1282

A
1
1284 Studies
1285
Administrations
1286
1287
1288
Studies
1290
1291
1292
1293
1294
Studies
1296
1297
1298
1299
1300
1303 Studies
1304
1305
D
deliverable.
Plans
Administrations

Specifications


Due Date(s)
Component C
9-11 ELA
July 2014
July 2015
$59,589.00
$59,589.00
July 2016
$59,589.00
$59,589.00

$59,589.00
$59,589.00
July 2014
July 2015
$6,000.00
$2,000.00
July 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$6,000.00
$2,000.00

August 2014
August 2015
August 2016
December 2014
December 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$35,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00

Studies
A
1306
1307
1308
1309
1310
1311
1312
1313

Studies

1314 Studies
1315
1316
1317
1318
1321
1322
1323
Studies

Specifications
B
Specify
Component
A,plans
B, C, D or
(A-D): Department
_____C_____
procedures,
scoring
(2) EProvide
D
E
Due Date(s)


posted to SFTP.
May 2015
May 2016
May 2017
2016

via SFTP.
2017


via SFTP.
2016
(Renewal 1 - 2018)
(Renewal 2 - 2019)

(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Component C
9-11 ELA
$2,000.00
$150,000.00
$150,000.00
$150,000.00
$150,000.00
$150,000.00
$150,000.00
$2,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00

Studies
1324
1327

B

Technical Reports


posted to SFTP.


Reference 7.6.3.3

November 2014
posted to SFTP.



October 2014
SFTP.
Studies
1328
1329
1330
Studies
Attachmentmust
1b - DELIVERABLES
(1) SubmitPRICING
most recent
SSR
files
and
via
SFTP.
D
E
the demographic
Description
Due Date(s)
The specifications
will also address
(Renewal 3 - 2020)
F
Component C
9-11 ELA
$0.00
December 2015
December 2016
December 2017
$94,429.00
$151,570.00
$151,570.00
$151,570.00

$151,570.00
$151,570.00
$20,412.00
$0.00
1332
1333 Studies
1334
Studies


May 2015
$697,578.00
$0.00
$85,970.00
$0.00
1336

A
1
C
7.6.3.4
Studies
Setting Work Plan

Studies

Studies

Technical Report
SFTP.
Due Date(s)
Component C
9-11 ELA
$2,000.00
1337
$115,401.00
1338
1339
Studies


7.6.3.4.
Department.
renewal)
SFTP.
$30,761.00
$0.00
$0.00
1341

A
1
Studies
Report
Due Date(s)
August 2015




Reference 7.6.3.6
SFTP.
F
Component C
9-11 ELA
$0.00
$0.00
1343
Studies
1344
1345
Studies

Report
$17,819.00
$0.00
$45,291.00
$0.00
1347
1348 Studies

across devices.
input types.
concordance.
characteristics.
$5,400.00

Scaling, Equating,
A Scoring,
Studies
1349
Studies
for conducting
Delivering
Assessments
and detailed
Contractor
Deliverable
Description
across devices.
input types.
concordance.
characteristics.

D
email
Evidence of
Completion

with TAC Committee


August 2015



November 2015
SFTP.
Due Date(s)
Component C
9-11 ELA
$0.00
$29,752.00
$0.00
1351
Studies
1352
$900.00

Studies
A
1353
Studies
1355
Studies

B assessments
November 2015
SFTP.
C and events, as
D
TAC members.
of Completion


7.6.3.6.2

August 2016

Assessments


November 2015
SFTP.
Due Date(s)
Component C
9-11 ELA
$0.00
$9,868.00
$0.00
$900.00
$0.00
1357

A
1
Studies
B
Assessments
1358
C
7.6.3.6.3
D
August 2016
Due Date(s)
Component C
9-11 ELA
$900.00
$0.00
1359
1361
1362
1363
1364
1365
1367
1368
1369
1370
1371
1373
1374
1375
1376
1377
1379
1380
1381
1382

7.7.2

report

Meetings


previous invoice.

ITN.


Action Item Report

work. ITN Ref 7.7.6

April 15, 2014

April 15, 2015
April 15, 2016
$47,285.00
$19,950.00
$19,950.00
$19,950.00
$19,950.00

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$19,950.00
$121,879.00
$122,426.00
$122,851.00
$123,260.00
$124,154.00
$124,566.00
$1,139,500.00
$1,276,463.00
$1,287,729.00
$1,102,105.00
$1,122,579.00
$1,129,158.00
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$112,459.00
$112,459.00
$112,459.00
$112,459.00
$112,459.00
Program Management
A
1
1383
1385
1386
1387
1388
1389
1391
1392
1393
1394


Specify
E (A-D): _____C_____
status Component
management
those
leading
up
to
previous
invoice
period. (Quarterly
invoices to occur
C
D
E
Due Date(s)
(Renewal 3 - 2019)
work. ITN Ref 7.7.6
F
Component C
9-11 ELA
$112,459.00

November 2015
November 2016
November 2017
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00

ITN Ref 7.7.6
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$69,917.00
$69,797.00
$69,797.00
$69,797.00
$69,797.00
Program Management
A
1

B

Specify
Component
A,level
B, C,ofD or
E (A-D): reports.
_____C_____
by program
task and
accounting
and
balances
of
February,
May,
August,
and
C
D November)
ITN Ref 7.7.6
Due Date(s)
Component C
9-11 ELA
(Renewal 3 - 2019)
$69,797.00

$43,167.00
1395
Program Management
Ref. 7.7.10
1396

Specify Component A, B, C, D or E (A-D): _____D_____
A
1
3
4
5
6
7
8
Test Development
Document
Test Development

Specifications
9
10
11
12
13
C
7.3.10
via SFTP.
Due Date(s)
June 1, 2014
June 1, 2015
June 1, 2016
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5
F
Component D
EOC
$38,237.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
14
Test Development
15
16
17
18
19


7.5.4.5

SFTP.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00

7.5.4.5

SFTP.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
20
Test Development
21
22
23
24
25

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
Test Development
A
1

B
E (A-D): _____D_____
C
D
E
andDescription
the Department. The
Deliverable
Due Date(s)
(Renewal 3 - 2020)
7.5.4.5
F
Component D
EOC
$0.00
26
Test Development
27
28
29
30
31

7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5

SFTP.
$0.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
32
Test Development
33
34
35
36
37
$0.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
38
Test Development
39
40
41
42
43
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$8,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
44

A
1
Test Development
45
46
47
48
49
50
51 Test Development
52
53
54
55
56
B
Specifications
C
7.5.4.5
D
SFTP.
7.3.10
Test Development


Test Development

each scoring guide.
F
Component D
EOC
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
June 1, 2014
June 1, 2015
June 1, 2016
$15,797.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$35,812.00
57
58
59
60
61
62

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$15,722.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
Test Development
A
1

B
each scoring guide.
Attachment
1b - DELIVERABLES
PRICING
Specify
Component
A, These
B, C, D or E (A-D): _____D_____
on student
reports.
C
D
E
Due Date(s)
(Renewal
3
2020)
F
Component D
EOC
$2,000.00
63
Test Development

Assessment Reports
64
65
66
67
68

Reference 7.3.7.5

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$67,402.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
69
Test Development
70
71 Test Development
72
73
74
75
76 Test Development


$73,872.00

management report
Reference 7.3.4.2

February 1, 2017
management report
$1,915,038.00
$1,955,488.00
$1,005,531.00
$870,544.00
$729,403.00
Review

$1,954,433.00

management report

A
Development
1 Test Deliverable
Category
77
78
79
80
81 Test Development
82
83
84
85
86
87 Test Development
88
89
90
91 Test Development
92
93
94
95 Test Development
96
97
98
99
Periodic:Contractor
Field Test Item
or Description
Deliverable
Deliverable
Review

email
Evidence of
Completion
Due Date(s)
May 1, 2016
management report
July 15, 2015
July 15, 2016
F
Component D
EOC
$16,033.00
$16,033.00
$16,033.00
$16,033.00
$16,834.00
$24,381.00
$24,381.00
$24,381.00
$24,381.00
$24,381.00

Reference 7.3.0

management report.
December 1, 2015
December 1, 2016
$11,652.00
$11,652.00
$11,652.00
$11,652.00


management report.
December 1, 2015
December 1, 2016
$9,679.00
$4,555.00
$4,555.00
$4,555.00


previous invoice.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$55,695.00
$56,441.00
$57,205.00
$58,783.00
$59,860.00
Test Development
A
1
100
102
103
104
105
106
108
109
110
111
112
Test Administration
B

Component
A, B, C, D or
E (A-D):
_____D_____
for the purpose
of reviewing
previous
invoice.
supporting the
C





113
114
115
116
Due Date(s)
Component D
EOC
(Renewal 3 - 2020)
$60,029.00

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
the report.
2015
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00

2016
$0.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
$0.00
$0.00
$0.00
117
Test Administration
118
119
120
121

the report.
Test Administration
A
1

B
1b - DELIVERABLES
the report.PRICING
Specify
Component
A, B,
has been
conducted
by C, D or E (A-D): _____D_____
C
D
Due Date(s)
Component D
EOC
(Renewal 3-2020)
$0.00

a report.
the report.

2015
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00

the report.

2015
$1,000.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
the report.

2015
$1,000.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
122
Test Administration
123
124
125
126
127
128
Test Administration
129
130
131
132
133
Test Administration
134
135
136
137
138

A
1
Test Administration
139
140
141
142
143
F
Component D
EOC
the report.
Due Date(s)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

the report.

2015
$1,000.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
the Specifications.

2015.
$1,000.00
144
Test Administration
145
146
147
148
149
150
152
153
154
155
156

$0.00
$0.00
2016
$0.00
(Renewal 1- 2017)
$0.00
(Renewal 2-2018)
$0.00
(Renewal 3 - 2019)
$0.00
$0.00
the Specifications.
2016

A
1
Test Deliverable
Administration
Category
158
159
160
161
162
164
165
166
167
168
169
171
172
173
174
175
Spring ELA/L
Retake:Deliverable
all processes,
of the approved
specifications.
Contractor
Deliverable
Description
of Completion
the Specifications.
E
2016
F
Component D
EOC
$0.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
the Specifications.
administration.
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$97,889.00

administration.
the Specifications.
2016
$82,307.00
(Renewal 1- 2017)
$87,490.00
(Renewal 2-2018)
$91,271.00
(Renewal 3 - 2019)
$91,645.00
$0.00
administration.
the Specifications.
2015.

A
1
Test Deliverable
Administration
Category
177
178
179
180
181
183
184
185
186
187
188
190
191
192
193
194
195
Winter EOC:
DevelopDeliverable
packaging and
all processes,
of the approved
specifications.
Contractor
Deliverable
Description
of Completion
administration.
the Specifications.

2015.
Component D
EOC
$97,687.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
administration.
the Specifications.
2015.
$82,012.00
$87,493.00
$91,272.00
$91,567.00
$0.00
$94,480.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
administration.
the Specifications.
2015.
$97,780.00
$82,469.00
$87,089.00
$90,639.00
$92,072.00
$0.00
$1,000.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$97,538.00
$82,199.00
$87,200.00
$87,200.00
$92,017.00

A
1
197
198
199
200
201
Test Administration
202
203
204
205
206
207
208
Test Administration
209
210
211
212
213
214
215
Test Administration
216
217
218
219
220
221
222
B
memos, etc.
districts.

memos.


districts.

memos.


districts.

memos.


districts.
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component D
EOC
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$296,765.00
$370,887.00
$402,573.00
$433,362.00
$445,877.00
$446,576.00

A
1
224
225
226
227
228
Test Administration
229
230
231
232
233
234
Test Administration
235
236
237
238
239
240
241
Test Administration
242
243
244
245
246
247
and accommodations.
districts.

and accommodations.


districts.



districts.

accommodations.


districts.
Due Date(s)
negotiations.)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component D
EOC
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$151,508.00
$158,840.00
$170,746.00
$176,281.00
$177,318.00

A
1
Test Administration
248
249
250
251
252
253
Test Administration
254
255
256
257
258
259
260
Test Administration
261
262
263
264
265
266
267
Test Administration
268
269
270
271
272
273
accommodations.
districts.

accommodations.


districts.

accommodations.


districts.


and 7.4.7.4

districts.
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component D
EOC
$0.00
$147,868.00
$155,280.00
$163,992.00
$168,576.00
$170,205.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$311,355.00
$238,334.00
$243,691.00
$259,417.00
$266,943.00
$268,296.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$171,752.00
$164,770.00
$172,527.00
$183,147.00
$188,649.00
$188,693.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
B
districts.

and 7.4.7.4

districts.
and rosters.
and 7.4.7.4

districts.

labels and rosters.

and 7.4.7.4

districts.

and 7.4.7.4

districts.

280
281
282
283
284
285
286
Test Administration
287
288
289
290
291
292
Test Administration
293
294
295
296
297
298
Test Administration
299
and 7.4.7.4
274
275
276
277
278
279
Test Administration

labels and rosters.
Due Date(s)
negotiations.)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component D
EOC
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00

Test Administration
A
1
300
301
302
303
304
305
Test Administration
306
307
308
309
310
311
312
314
315
316
317
318
320
321
322
323
324
326
327

preidentification
labels and rosters.

labels and rosters.

critical date
Description
and 7.4.7.4

destination
address,
Evidence of
Completion
districts.

and 7.4.7.4

districts.
Due Date(s)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
7.4.8

November 2015
November 2016


7.4.8

May 2016
May 2017


7.4.8

May 2016
Component D
EOC
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Test Administration
A
1
328
329
330
331
333
334
335
336
337
339
340
341
342
343
345
346
347
348
349
350
352
353
354
355

Deliver
B
system for
each assessment.
Contractor
Deliverable

was operable for
the invoice
Deliverable
Description
7.4.8

E
occurred during
the period,
signed by the
Evidence
of Completion
Due Date(s)

assessment.

7.4.8

November 2015
November 2016
$0.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00

assessment.

7.4.8

February 2016
February 2017
$0.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00

assessment.

7.4.8

assessment.
7.4.8

F
Component D
EOC
$0.00
$0.00
$0.00
$0.00
August 2015
August 2016
August 2017
$0.00
$14,611.00
$14,659.00
$14,908.00
$14,978.00
$15,368.00
$14,973.00
August 2015
August 2016
August 2017
$0.00
$5,000.00
$4,000.00
$4,000.00
$4,000.00
Test Administration
A
1
356
Specify Component
A, B,
C, D or
(A-D):
_____D_____
of the contract
and
(orEno)
malfunctions
assessment.
was
operable
for
the
invoice
period.
ITN
Reference
occurred
during
the
period,
B
C
D signed by the
E
7.4.8
Due Date(s)
F
Component D
EOC
$4,000.00
$4,000.00
357
359
360
361
362



November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
363
365
366
367
368

May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
369
371
372
373
374
375

administration.


May 2015
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
376
378
379
380
381



November 2015
November 2016
$0.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
Test Administration
A
1
382
384
385
386
387
388
390
391
392
393
394
395
397
398
399
400
401
402
404
405
406
407
408

B
(1) ProvidePRICING
A,ofB,the
C, D or
(A-D):
_____D_____
met theComponent
requirements
(orEno)
malfunctions
C
D
E
Due Date(s)












February 2016
February 2017
May 2015
May 2016
May 2017
F
Component D
EOC
$10,000.00
$0.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$0.00
$1,874,527.00
$2,147,007.00
$2,548,678.00
$2,801,304.00
$2,753,427.00
$2,665,824.00
August 2015
August 2016
August 2017
$0.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
November 2014
November 2015
November 2016
$31,591.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
Test Administration
A
1

B

Attachment
1bon
- DELIVERABLES
PRICING
and approved
a
the practice
Specifythat
Component
A, B, C, D or
E (A-D): _____D_____
offers students
requirements
C
D
Due Date(s)
Component D
EOC
$0.00
409
Test Administration
410
412
413
414
415
417
418
419
420
421
423
424
425
426

the contract.


training.
September 2014
7.4.9.8
administration.
August 2015
August 2016
November 2016
November 2017
previous year.
November 2015
November 2016
November 2017
$207,841.00
$122,583.00
$122,583.00
$122,583.00
$122,583.00
$122,583.00
$268,964.00
$222,793.00
$228,205.00
$237,638.00
$243,432.00
$245,206.00
$9,724.00
$12,229.00
$13,663.00
$14,163.00
$14,487.00
Test Administration
A
1
Specify Component
A, B, C,for
D or
E (A-D):and
_____D_____
and other materials
materials
a
period
of
at
least
six
months,
and
used
documents
end
of
each
assessment
year
B
C
E
Due Date(s)
F
Component D
EOC
$14,507.00
427
Test Administration
428
429
430
431
432
433
Test Administration
434
435
436
437
438
439
Test Administration
440
441
442
443
444
445
446
Test Administration
447
448
449
450
451

comment forms.

comment forms.


negotiations.)
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015

comment forms.
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
Test Administration
A
1
452
Test Administration
453
454
455
456
457
458
Test Administration
459
460
461
462
463
464
465
Test Administration
466
467
468
469
470
471
472
Test Administration

comment forms.
B
comment forms.

comment forms.

comment forms.

Trial Test

1b -forms.
online comment
Component A, B, C, D or E (A-D): _____D_____
ITN Reference 7.4.1Specify
and 7.4.7.7
C
Due Date(s)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
7.4.8 and 7.4.9.5
Component D
EOC
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$4,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$58,275.00
473

A
1
Test Administration
474
475
476
477
478
479
Test Administration
480
481
482
483
484
485
Test Administration
486
487
488
489
490
491
492
Test Administration
493
494
495
496
497
498
B



versions)
F
Component D
EOC
Reference 7.3.11.2
Due Date(s)
negotiations.)

Reference 7.3.11.2
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Reference 7.3.11.2
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Reference 7.3.11.2
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$19,947.00
$19,947.00
$19,947.00
$19,947.00
$19,947.00

A
1
Test Administration
499
500
501
502
503
504
Test Administration
505
506
507
508
509
510
511
Test Administration
512
513
514
515
516
517
518
Test Administration
519
520
521
522
523
524
versions)
Reference 7.3.11.2
D
Due Date(s)
negotiations.)
Component D
EOC
$0.00
versions)
Reference 7.3.11.2
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$20,131.00
$20,131.00
$20,131.00
$20,131.00
$20,131.00
$0.00

versions)
Reference 7.3.11.2
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$65,378.00
$39,565.00
$39,565.00
$39,565.00
$39,565.00
$39,565.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
$20,077.00
$20,077.00
$20,077.00
$20,077.00
$20,077.00
$20,077.00
$0.00


Reference 7.4.1.
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
525
526
527
528
529
530
Test Administration
531
532
533
534
535
536
537
Test Administration
538
539
540
541
542
543
Test Administration
544
545
546
547
548
549
Reference 7.4.1.
Due Date(s)
negotiations.)
testing
Reference 7.4.1.


negotiations.)
Reference 7.4.1.


Reference 7.4.1.
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component D
EOC
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$9,000.00
$11,336.00
$13,369.00
$13,834.00
$13,834.00
$0.00
$8,000.00
$9,000.00
$10,000.00
$10,000.00
$10,000.00

A
1
Test Administration
B
550
551
552
553
554
555
556
Test Administration
557
558
559
560
561
562
563
565
566
572
573
574
576
577
578
Reference 7.4.1.
Reference 7.4.1.

567
568
569
571

ITN Reference 7.5.1
D
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016

management system.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
April 2014
April 2015
April 2016

management system.


management system.

F
Component D
EOC
$0.00
$38,448.00
$45,512.00
$47,748.00
$51,006.00
$53,126.00
$54,189.00
$0.00
$15,149.00
$17,646.00
$19,697.00
$20,938.00
$22,200.00
$21,892.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
September 2015
September 2016
$0.00
$0.00
$0.00

January 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
January 2017
A
1
579
581
582
583
584
585
587
588
589
590
592
593
594
595
597
598
599
600
601
603
604
605
606
607
609


monitor district
(1)EPreID
frequency
Specifyand
Component
A, B, C, D or
(A-D):
_____D_____
C reports of issues
Deliverable
Description
assessments. ITN
Reference
7.5.2
management system.
management system.
E
Due Date(s)
January 2015
January 2016
January 2017

review.

management system.
review.

management system.
review.


in .txt format

September 2015
September 2016
October 2015
F
Component D
EOC
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
October 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

management system.

January 2015
January 2016
January 2017
$1,000.00
$3,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

management system.

June 2015
June 2016
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00

Protocol (SFTP).

September 2015
September 2016

A
1
610
611
612
614
615
616
617
619
620
621
622
623
625
626
627
file layout
and Full-State
layout for each
administration for
Contractor
Deliverable
Description
in .txt format

management system
andofSecure
File Transfer
Evidence
Completion
Protocol (SFTP).

file in .txt format

Protocol (SFTP).
January 2016


Protocol (SFTP).
January 2015
January 2016
January 2017
format

Protocol (SFTP).
September 2015
Due Date(s)
January 2017
September 2016
F
Component D
EOC
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
A
1
628
630
631
632
633
635
636
637
638
639
641
642
643
644

Specify
Component
A, B,toC, D or
(A-D):
_____D_____
with
the Department
(1)EFinal
PreID
format


File Transfer
D
Protocol (SFTP).

.txt format


Protocol (SFTP).
October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

.txt format


Protocol (SFTP).
January 2015
January 2016
January 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

.txt format


Protocol (SFTP).
June 2015
June 2016
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
E
Due Date(s)
F
Component D
EOC
$1,000.00
A
1
645
647
648
649
650
652
653
654
655
657
658
659
660
661
663
664
665
666
668
669
670
671

.txt format

Attachment
1b - DELIVERABLES
PRICING
the Department
to
(1) Final PreID
Component
A, B,
E (A-D): _____D_____
for each
administration
forC, D or
management
D the SFTP.
Deliverable
of Completion
each PreID window
closes,Description
the contractor will
the file layout
and data file.




E
Due Date(s)

$0.00
$0.00
$0.00

May 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
May 2017
May 2015
May 2016
May 2017





$1,000.00
October 2015
October 2016

transcription logs
F
Component D
EOC

October 2015
October 2016
December 2015
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

May 2015
$1,000.00
$1,000.00

A
and Reporting
1 Scoring
Deliverable
Category
673
674
675
676
677
679
680
681
682
683
685
686
687
688
690
691
692
Spring EOC:
Development
will use a Special
Document
Contractor
Deliverable
Description

logofposted
to SFTP.
Evidence
Completion




7.5.6.2, and 7.5.6.4



7.5.6.2, and 7.5.6.4

E
Due Date(s)
May 2016
May 2017
July 2015
July 2016
July 2017
October 2015
October 2016
March 2016
March 2017
F
Component D
EOC
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
A
1
693
695
696
697
698
699
701
702
703

Specify
A, B,toC, D or
(A-D):
_____D_____
withComponent
the Department
(1)EFinal
approved

and CBT extract
C
7.5.6.2, and 7.5.6.4

data files



7.5.6.2, and 7.5.6.4


7.5.6.2, and 7.5.6.4

E
Due Date(s)
March 2015
March 2016
March 2017
October 2015
October 2016
F
Component D
EOC
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$103,694.00
$91,006.00
$97,582.00
$101,850.00
A
1

data files:
B

Specify
A, B,toC, D or
(A-D):
_____D_____
withComponent
the Department
(1)EFinal
approved
and CBT extract
C
7.5.6.2, and 7.5.6.4

E
Due Date(s)
F
Component D
EOC
$103,262.00
704
706
707
708


7.5.6.2, and 7.5.6.4

December 2015
December 2016
$103,054.00
$90,396.00
$95,924.00
$98,680.00
$101,385.00
709
711
712
713
714


7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017
$128,071.00
$138,552.00
$126,216.00
$134,120.00
$139,589.00
A
1
715
717
718

719
720
721
723
724
725
726
728
729
730



Component
B, C, D or
E (A-D):and
_____D_____
develop the layoutsSpecify
for scan/edit
and CBT A,
extract
scan/edit,
the
contractor
online
management
system
C
D and SFTP.
E
Due Date(s)
7.5.6.2, and 7.5.6.4

7.5.6.2, and 7.5.6.4


the data files.

the data files.
F
Component D
EOC
$140,592.00
July 2015
July 2016
July 2017
$18,212.00
$111,086.00
$97,779.00
$104,332.00
$108,183.00
$109,915.00
October 2015
October 2016
$0.00
$0.00
$0.00

April 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2017
A
1
731
733
734
735
736
737
739
740
741
742
744
745
746
747
749
750
751
752
753
755
756
757
758
759
761

B of the
data files

Specifythe
Component
A, and
B, C, D or
(A-D): _____D_____
full state SSR
(1)EAnomalous
the data files.

the data files.
E
Due Date(s)
April 2015
April 2016
April 2017

data files

the data files.
data files
the data files.
data files
the data files.

score data files

the data files.


Reference 7.5.5

October 2015
October 2016
December 2015
December 2016
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017

October 2015
October 2016
F
Component D
EOC
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$8,000.00
$8,000.00
$8,000.00
$8,000.00
$8,000.00
$8,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00

A
762
763
764
766
767
768
769
771
772
773
774
775
777
778
779
780
782
783
B
and DSRsContractor
Deliverable of


DSR data files.

and DSR data files.

C
develop the State
Student Description
Deliverable
Reference 7.5.5

and online
of management
Completion system.
Due Date(s)

April 2016
F
Component D
EOC
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Reference 7.5.5

April 2017

Reference 7.5.5


Reference 7.5.5

October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00

Reference 7.5.5

December 2015
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A

SSRs and
B
SSR/DSR Contractor
and multiple
784
785
787
788
789

790
791
793
794
795
796
797
799
800
801
802
804
805




C the Department to
Student Results
(DSR) file layouts
and use the layout
Deliverable
Description
Reference 7.5.5

E
email
Evidence of
Completion
Due Date(s)

Reference 7.5.5


Reference 7.5.5


7.5.5

October 2015
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00


April 2016
April 2017
F
Component D
EOC
$1,000.00
$1,000.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
July 2015
July 2016
July 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00

A

of SAR/DAR
file layouts
and multiple
Contractor
Deliverable

Results (SAR) and
District Aggregated
Deliverable
7.5.5

E
email
Evidence of
Completion
Due Date(s)

7.5.5


and DAR data files.

7.5.5

October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00


7.5.5

December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
806
807
809
810
811
812
813
815
816
817
818
820
821
822

DAR data files.
F
Component D
EOC
$0.00
$0.00
April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
December 2015
$1,000.00
$1,000.00
823

A
1
825
826
827
B
828
829
831
832

833
834
835
837
838
839
840
842
843
844

data files.

data files.
E
Due Date(s)
April 2015
April 2016
April 2017
F
Component D
EOC
7.5.5

7.5.5


7.5.5.2 and 7.5.6.6
November 2016
$0.00
$0.00
$0.00

$0.00
$0.00

7.5.5.2 and 7.5.6.6
June 2016
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
July 2015
July 2016
July 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
June 2017
A
1
845
847
848

B
data files.

849
850
851
853
854
855
856
858
859
860
Attachment
- DELIVERABLES
PRICING
Department1b
Scoring
(1) Final late
to develop
the
State
Aggregated
to
online
management
system and SFTP.
C to develop the
E
Deliverable
Due Date(s)
scores and codes
7.5.5.2 and 7.5.6.6
7.5.5.2 and 7.5.6.6
F
Component D
EOC
$0.00
June 2015
June 2016
June 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
7.5.5.2 and 7.5.6.6
November 2016
$1,000.00
$1,000.00
$1,000.00

$1,000.00
$1,000.00

data files.
February 2015
February 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

7.5.5.2 and 7.5.6.6
861

A
1
863
864
865
B
data files.
866
7.5.5.2 and 7.5.6.6
867
869
870
871
872

data files.

7.5.5.2 and 7.5.6.6
873
875
876
877

7.5.5.3 and 7.5.6.7
878
880
881
882
7.5.5.3 and 7.5.6.7
E
Due Date(s)
June 2015
June 2016
June 2017
F
Component D
EOC
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
October 2015
October 2016
October 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00
October 2017
883

A
1
885
886
887
888
889
891
892
893
894
896
897
898
899
901
902
903
904
905
907
908
F
Component D
EOC
assessments.
7.5.5.3 and 7.5.6.7
Due Date(s)
October 2015
October 2016
October 2017

7.5.5.3 and 7.5.6.7
December 2015

7.5.5.3 and 7.5.6.7
March 2015
March 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

7.5.5.3 and 7.5.6.7
October 2015
October 2016
October 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

October 2015
October 2017
7.5.5.3 and 7.5.6.7
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A
1
909
910
911
913
914
915

files for all
assessments.
Contractor
Deliverable


918
919
920
921
922
923
924
925
926


reporting
F
Component D
EOC
$1,000.00
$1,000.00
$1,000.00
September 2015
September 2016
September 2017
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

2015
$1,000.00
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2017
reporting
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00

initial reporting

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00
916
917

E
of all assessments.
data files.
Description
of Completion
Due Date(s)
7.5.5.3 and 7.5.6.7
927
928
929
930
931
932
933

$0.00
$0.00
$0.00
$0.00

A
1
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955


D
E
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
956
957
958
F
Component D
EOC
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$50,214.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$2,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
(Renewal 3 - 2020)
jump drives.
2015
$0.00
(Renewal 1 - 2017)
$0.00
districts.
A
1
959

- DELIVERABLES
deliver initial1b
reporting
(1) ProvidePRICING
CD/DVDs
or
jump
drives.
The
the
samples
jump drives.
C and media are
E
Evidence of
Completion
Due Date(s)
destination
address,
(Renewal 3 - 2019)
districts.
F
Component D
EOC
$0.00
$0.00
960
961
962
963


964

jump drives.
districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00

jump drives.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
965
966
967
968


969
970
971
972
973
974
975

initial reporting


jump drives.
districts.

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$125,171.00
$153,502.00
$163,525.00
$163,525.00
$163,525.00
976
977

initial reporting

$78,266.00
jump drives.
2015

A
1
978
979

Student
B
Submit proof
of delivery:
Development
Contractor
Deliverable
initial reporting

submit proof
of delivery to the
Description
980

districts.
Due Date(s)
Component D
EOC
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$99,964.00
$106,693.00
$106,693.00
(Renewal 3 - 2019)
$106,693.00
981
982
983
984
985
986

initial reporting


jump drives.
districts.

2015
$947,532.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$801,521.00
$819,401.00
$824,298.00
$824,298.00
$824,298.00

initial reporting


jump drives.
districts.

2015
$60,218.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$64,687.00
$69,765.00
$69,765.00
$69,765.00
(Renewal 3 - 2020)
$69,765.00

reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
987
988
989
990
991
992
993
994
995
996
997


A
1

B
PRICING
Specify
Component
A, B, C,
E (A-D): _____D_____
each subject
that contains
theD or
reports.
will be
C
E
Due Date(s)
(Renewal 3 - 2019)
districts.
F
Component D
EOC
$0.00
998
999
1000
1001
1002



reports.
districts.

2016
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00



reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00


reports.
districts.

2015
2016
(Renewal 1 - 2017)
$0.00
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012

reports
$0.00
$0.00
A
1
1013

Tasks
B
reports
D
E
a separate page
that includes an
following for each
shipment:
Description
Evidence
of Completion
Due Date(s)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
districts.
F
Component D
EOC
$0.00
$0.00
1014
1015
1016
1017
1018

reports


reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00

reports


reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00



reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
A
1
1030

B
to selected
approval of
Attachment
1b - DELIVERABLES
PRICING
each
subject
that
contains
the
reports.
E
will print and
of material
Description
Evidence
of Completion
Due Date(s)
(Renewal 2 - 2019)
districts.
(Renewal 3 - 2020)
F
Component D
EOC
$0.00
$0.00
1031
1032
1033
1034
1035

reporting


districts.

2015
$0.00
2016
(Renewal 1 - 2017)
$0.00
$0.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00

districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00

districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
(Renewal 2 - 2019)
$0.00
1036
1037
1038
1039

reporting

1040
$0.00
$0.00
$0.00
1041
1042
1043
1044
1045
1046


A
1
1047
1048
1049

Attachment
Reference 7.5.5
1b - DELIVERABLES
following for
PRICING
who
received
B
C
districts.
1050
1051

Due Date(s)
(Renewal 3 - 2020)
2015
(Renewal 1 - 2017)
(Renewal 3 - 2019)
districts.
Component D
EOC
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
1052
1053
1054
1055


1056

districts.

2015
$1,000.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$1,000.00
$1,000.00
$1,000.00
(Renewal 3 - 2019)
$1,000.00

districts.

2015
$2,000.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$2,000.00
$2,000.00
$2,000.00
$2,000.00
(Renewal 3 - 2020)
$2,000.00

districts.

2015
$1,000.00
2016
$1,000.00
1057
1058
1059
1060
1061


1062
1063
1064
1065


A
1
1066
1067
1068
1069
1071
1072
1073
1074
1076
1077
1078
1079
1081
1082
1083
1084
1085
1087
1088
1089

B Submit proof of
reports and
media for
the late reporting
Contractor
Deliverable


C drives. The
Department after
all reports
and media are
Deliverable
Description

(2) Provide proof
of delivery
Evidence
of Completion
districts.
Due Date(s)
Component D
EOC
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00



September 2015
September 2016
$0.00
$0.00
$0.00
$0.00
$0.00



March 2016
$0.00
$0.00
$0.00
$0.00
$0.00



March 2015
March 2016
March 2017



March 2017

September 2015
September 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
1090

A
1
1092
1093
1094
1095
1097
1098
1099
1100


1101
1103
1104
1105
1106
1107



Performance Tasks

1108
1109
1110
1111
Due Date(s)
Component D
EOC
November 2015
November 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

March 2015
March 2016
March 2017
$4,000.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00

June 2015
June 2016
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
A
1

Performance Tasks
B
Component
A, B, C, D or E (A-D): _____D_____
or on the
performance-based
that respondents
C
D
E
Due Date(s)
(Renewal 3 - 2019)
F
Component D
EOC
$0.00
1112
1113
1114
1115
1116
1117

Performance Tasks

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00

2015
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
1118
1119
1120
1121
1122
1123

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
A
1

B
Attachment
performance-based
Specify Component
A, B, C, D or E (A-D): _____D_____
that respondents
D
E
Due Date(s)
(Renewal
3
2020)
F
Component D
EOC
$0.00
1124
1125
1126
1127
1128

Tasks

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$95,466.00
$106,355.00
$118,305.00
$122,497.00
$123,453.00
1129
1130
1131
1132
1133

Performance Tasks
$119,170.00
$76,212.00
$82,935.00
$85,725.00
A
1

Performance Tasks
B
Specify
Component
A, B,on
C, D or
(A-D):
_____D_____
of a students
response
theEfinal
handscoring
report.
D
E
Due Date(s)
(Renewal 3 - 2019)
F
Component D
EOC
$85,801.00
1134

Performance Tasks
1135
1136
1137
1138
1139


2015
1140
1141
1142
1143
1144

Performance Tasks
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
$782,724.00
$833,429.00
$862,996.00
$913,019.00
$941,635.00
$914,768.00
$143,899.00
$159,651.00
$169,211.00
$181,194.00
A
1
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167

Performance Tasks
B
Component
A, B, C, D or E (A-D): _____D_____
or on the
performance-based
that respondents
C
D
E
Due Date(s)
(Renewal 2 - 2019)
(Renewal 3 - 2020)


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5
June 2015
June 2016
January 2016
January 2017
January 2015
January 2016
January 2017
June 2015

the SFTP.
June 2016
F
Component D
EOC
$187,426.00
$188,691.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$3,000.00
$0.00
$0.00
$0.00
$0.00

A
1
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
all assessments
Reference 7.5.5

all assessments
Reference 7.5.5

all assessments
Reference 7.5.5

ELA/Mathematics

ELA/Mathematics
D
the SFTP.
Due Date(s)
Component D
EOC
October 2015
October 2016
the SFTP.
January 2016
the SFTP.

April 2015
April 2016
April 2017

December 2015
Reference 7.5.4.2
Reference 7.5.4.2

October 2016
October 2017
October 2015
and ELA/Mathematics
October 2017
Reference 7.5.4.2
December 2015
Reference 7.5.4.2
$0.00
$0.00
$0.00
$0.00
$0.00
$9,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00

A
1
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
Scoring
and Reporting
Deliverable
Category
will create and
posted toofthe
SFTP.
Description
Completion
Due Date(s)
Reference 7.5.4.2
March 2016
Reference 7.5.4.2
October 2015
October 2017
Reference 7.5.4.2

ELA/Mathematics

Reference 7.5.4.2

Retake

Fall EOC:Contractor
Development
of the history
Deliverable
1225
1226
1227
1229
1230

October 2015
October 2017
November 2015

May 2016

Retake
F
Component D
EOC
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$4,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1231
1232
1234
1235
1236
1237
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1255
1256

D
E
rule specified in
Deliverable
Due Date(s)

rescoring.

Reference 7.5.6.5

management system.
log.

manual rescoring.

Reference 7.5.6.5

management system.
log.


Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

October 2015
October 2016
April 2016
F
Component D
EOC
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2017
April 2015
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2016
April 2017
October 2015
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
October 2016
December 2015
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

1
1257
1258
1260
1261
1262
1263
1264
1266
1267
1268

B

C
Deliverable
Description
in the
Reference 7.5.6.5

D
management system.
log.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.


Reference 7.5.5

districts.


Protocol (SFTP).

rescoring.
1269
1270
1271
1272
1273
1274
1275
1276
1278
Studies
1279
1280
1281

E
Due Date(s)
April 2015
April 2016
April 2017
F
Component D
EOC
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

July 2015
July 2016
July 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

July 2015
$1,000.00
$12,266.00
July 2016
July 2017
$12,266.00
$12,266.00

$12,266.00
$12,266.00
$12,266.00
May 2014
May 2015
$41,346.00
$2,000.00
May 2016
$0.00
$0.00

$0.00
$0.00
1282

A
1
1284 Studies
1285
Administrations
1286
1287
1288
Studies
1290
1291
1292
1293
1294
Studies
1296
1297
1298
1299
1300
1303 Studies
1304
1305
D
deliverable.
Plans
Administrations

Specifications


Due Date(s)
Component D
EOC
July 2014
July 2015
$79,452.00
$79,452.00
July 2016
$79,452.00
$79,452.00

$79,452.00
$79,452.00
July 2014
July 2015
$6,000.00
$2,000.00
July 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$6,000.00
$2,000.00

August 2014
August 2015
August 2016
December 2014
December 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$35,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00

Studies
A
1306
1307
1308
1309
1310
1311
1312
1313

Studies

1314 Studies
1315
1316
1317
1318
1321
1322
1323
Studies

Specifications
B
Specify
Component
A,plans
B, C, D or
(A-D): Department
_____D_____
procedures,
scoring
(2)EProvide
D
E
Due Date(s)


posted to SFTP.
May 2015
May 2016
May 2017
2016

via SFTP.
2017


via SFTP.
2016
(Renewal 1 - 2018)
(Renewal 2 - 2019)

(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Component D
EOC
$2,000.00
$150,000.00
$150,000.00
$150,000.00
$150,000.00
$150,000.00
$150,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

Studies
1324
1327

B

Technical Reports


posted to SFTP.


Reference 7.6.3.3

November 2014
posted to SFTP.



October 2014
SFTP.
Studies
1328
1329
1330
Studies
Attachmentmust
1b - DELIVERABLES
(1) SubmitPRICING
include analyzing the
most
recent
SSR
files
and
via
SFTP.
D
E
the demographic
Description
Due Date(s)
The specifications
will also address
(Renewal 3 - 2020)
F
Component D
EOC
$1,000.00
December 2015
December 2016
December 2017
$94,429.00
$151,570.00
$151,570.00
$151,570.00

$151,570.00
$151,570.00
$20,412.00
$0.00
1332
1333 Studies
1334
Studies


May 2015
$697,578.00
$0.00
$85,970.00
$0.00
1336

A
1
C
7.6.3.4
Studies
Setting Work Plan

Studies

Studies

Technical Report
SFTP.
Due Date(s)
Component D
EOC
$2,000.00
1337
$115,401.00
1338
1339
Studies


7.6.3.4.
Department.
renewal)
SFTP.
$30,761.00
$0.00
$0.00
1341

A
1
Studies
Report
Due Date(s)
August 2015




Reference 7.6.3.6
SFTP.
F
Component D
EOC
$0.00
$0.00
1343
Studies
1344
1345
Studies

Report
$17,819.00
$0.00
$45,291.00
$0.00
1347
1348 Studies

across devices.
input types.
concordance.
characteristics.
$5,400.00

Scaling, Equating,
A Scoring,
Studies
1349
Studies
for conducting
Delivering
Assessments
and detailed
Contractor
Deliverable
Description
across devices.
input types.
concordance.
characteristics.

D
email
Evidence of
Completion

with TAC Committee


August 2015



November 2015
SFTP.
Due Date(s)
Component D
EOC
$0.00
$29,752.00
$0.00
1351
Studies
1352
$900.00

Studies
A
1353
Studies
1355
Studies

B assessments
November 2015
SFTP.
C and events, as
D
TAC members.
of Completion


7.6.3.6.2

August 2016

Assessments


November 2015
SFTP.
Due Date(s)
Component D
EOC
$0.00
$9,868.00
$0.00
$900.00
$0.00
1357

A
1
Studies
B
Assessments
1358
C
7.6.3.6.3
D
August 2016
Due Date(s)
Component D
EOC
$900.00
$0.00
1359
1361
1362
1363
1364
1365
1367
1368
1369
1370
1371
1373
1374
1375
1376
1377
1379
1380
1381
1382

7.7.2

report

Meetings


previous invoice.

ITN.


Action Item Report

work. ITN Ref 7.7.6

April 15, 2014

April 15, 2015
April 15, 2016
$47,285.00
$19,950.00
$19,950.00
$19,950.00
$19,950.00

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$19,950.00
$121,879.00
$122,426.00
$122,851.00
$123,260.00
$124,154.00
$124,566.00
$1,139,500.00
$1,276,463.00
$1,287,729.00
$1,102,105.00
$1,122,579.00
$1,129,158.00
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$112,459.00
$112,459.00
$112,459.00
$112,459.00
$112,459.00
Program Management
A
1
1383
1385
1386
1387
1388
1389
1391
1392
1393
1394


Specify
E (A-D): _____D_____
statusComponent
management
those
leading
up
to
previous
invoice
period. (Quarterly
invoices to occur
C
D
E
Due Date(s)
(Renewal 3 - 2019)
work. ITN Ref 7.7.6
F
Component D
EOC
$112,459.00

November 2015
November 2016
November 2017
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00

ITN Ref 7.7.6
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$69,917.00
$69,797.00
$69,797.00
$69,797.00
$69,797.00
Program Management
A
1

B

Specify
Component
A,level
B, C,ofD or
E (A-D): reports.
_____D_____
by program
task and
accounting
and
balances
of
February,
May,
August,
and
C
D November)
ITN Ref 7.7.6
Due Date(s)
Component D
EOC
(Renewal 3 - 2019)
$69,797.00

$43,167.00
1395
Program Management
Ref. 7.7.10
1396

Specify Component A, B, C, D or E (A-D): _____E_____
A
1
3
4
5
6
7
8
Test Development
Document
Test Development

Specifications
9
10
11
12
13
C
7.3.10
via SFTP.
Due Date(s)
June 1, 2014
June 1, 2015
June 1, 2016
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5
F
Component E
All Components
$47,796.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
14
Test Development
15
16
17
18
19


7.5.4.5

SFTP.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

7.5.4.5

SFTP.
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
20
Test Development
21
22
23
24
25

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$25,239.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
Test Development
A
1

B
E (A-D): _____E_____
C
D
E
andDescription
the Department. The
Deliverable
Due Date(s)
(Renewal 3 - 2020)
7.5.4.5
F
Component E
All Components
$2,000.00
26
Test Development
27
28
29
30
31

7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5

SFTP.
$0.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
32
Test Development
33
34
35
36
37
$0.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
38
Test Development
39
40
41
42
43
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$8,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
44

A
1
Test Development
45
46
47
48
49
50
51 Test Development
52
53
54
55
56
B
Specifications
C
7.5.4.5
D
SFTP.
7.3.10
Test Development


Test Development

each scoring guide.
F
Component E
All Components
Due Date(s)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
June 1, 2014
June 1, 2015
June 1, 2016
$19,746.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$59,687.00
57
58
59
60
61
62

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$67,380.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
Test Development
A
1

B
each scoring guide.
Attachment
1b - DELIVERABLES
PRICING
Specify
Component
A, These
B, C, D or E (A-D): _____E_____
on student
reports.
C
D
E
Due Date(s)
(Renewal
3
2020)
F
Component E
All Components
$2,000.00
63
Test Development

Assessment Reports
64
65
66
67
68

Reference 7.3.7.5

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$84,252.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
69
Test Development
70
71 Test Development
72
73
74
75
76 Test Development


$316,596.00

management report
Reference 7.3.4.2

February 1, 2017
management report
$11,472,807.00
$11,710,747.00
$6,122,763.00
$5,328,726.00
$4,498,480.00
Review

$11,704,540.00

management report

A
Development
1 Test Deliverable
Category
77
78
79
80
81 Test Development
82
83
84
85
86
87 Test Development
88
89
90
91 Test Development
92
93
94
95 Test Development
96
97
98
99
Periodic:Contractor
Field Test Item
or Description
Deliverable
Deliverable
Review

email
Evidence of
Completion
Due Date(s)
May 1, 2016
management report
July 15, 2015
July 15, 2016
F
Component E
All Components
$89,906.00
$89,906.00
$89,906.00
$89,906.00
$72,146.00
$104,489.00
$104,489.00
$104,489.00
$104,489.00
$104,489.00

Reference 7.3.0

management report.
December 1, 2015
December 1, 2016
$49,938.00
$49,938.00
$49,938.00
$49,938.00


management report.
December 1, 2015
December 1, 2016
$41,483.00
$19,520.00
$19,520.00
$19,520.00


previous invoice.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$278,476.00
$282,206.00
$286,025.00
$293,914.00
$299,300.00
Test Development
A
1
100
102
103
104
105
106
108
109
110
111
112
Test Administration
B

Component
A, B, C, D or
E (A-D):
_____E_____
for the purpose
of reviewing
previous
invoice.
supporting the
C





113
114
115
116
Due Date(s)
Component E
All Components
(Renewal 3 - 2020)
$300,145.00

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
the report.
2015
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$1,000.00
$1,000.00
$1,000.00
$1,000.00

2016
$1,000.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
$1,000.00
$1,000.00
$1,000.00
117
Test Administration
118
119
120
121

the report.
Test Administration
A
1

B
1b - DELIVERABLES
the report.PRICING
Specify
Component
A, by
B, C, D or E (A-D): _____E_____
has been
conducted
C
D
Due Date(s)
Component E
All Components
(Renewal 3-2020)
$1,000.00

a report.
the report.

2015
$2,000.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

the report.

2015
$1,000.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
the report.

2015
$1,000.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
122
Test Administration
123
124
125
126
127
128
Test Administration
129
130
131
132
133
Test Administration
134
135
136
137
138

A
1
Test Administration
139
140
141
142
143
F
Component E
All Components
the report.
Due Date(s)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

the report.

2015
$1,000.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
the Specifications.

2015.
$1,000.00
144
Test Administration
145
146
147
148
149
150
152
153
154
155
156

$0.00
$98,074.00
2016
$81,868.00
(Renewal 1- 2017)
$87,228.00
(Renewal 2-2018)
$91,416.00
(Renewal 3 - 2019)
$92,033.00
$0.00
the Specifications.
2016

A
1
Test Deliverable
Administration
Category
158
159
160
161
162
164
165
166
167
168
169
171
172
173
174
175
Spring ELA/L
Retake:Deliverable
all processes,
of the approved
specifications.
Contractor
Deliverable
Description
of Completion
the Specifications.
E
2016
F
Component E
All Components
$1,000.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
the Specifications.
administration.
2015.
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$199,773.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
$204,101.00
$169,829.00
$181,862.00
$188,692.00
$190,961.00
$0.00
$97,889.00

administration.
the Specifications.
2016
$82,307.00
(Renewal 1- 2017)
$87,490.00
(Renewal 2-2018)
$91,271.00
(Renewal 3 - 2019)
$91,645.00
$0.00
administration.
the Specifications.
2015.

A
1
Test Deliverable
Administration
Category
177
178
179
180
181
183
184
185
186
187
188
190
191
192
193
194
195
Winter EOC:
DevelopDeliverable
packaging and
all processes,
of the approved
specifications.
Contractor
Deliverable
Description
of Completion
administration.
the Specifications.

2015.
Component E
All Components
$97,687.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
administration.
the Specifications.
2015.
$82,012.00
$87,493.00
$91,272.00
$91,567.00
$0.00
$94,480.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
administration.
the Specifications.
2015.
$97,780.00
$82,469.00
$87,089.00
$90,639.00
$92,072.00
$0.00
$1,000.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$97,538.00
$82,199.00
$87,200.00
$87,200.00
$92,017.00

A
1
197
198
199
200
201
Test Administration
202
203
204
205
206
207
208
Test Administration
209
210
211
212
213
214
215
Test Administration
216
217
218
219
220
221
222
B
memos, etc.
districts.

memos.


districts.

memos.


districts.

memos.


districts.
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component E
All Components
$0.00
$188,019.00
$192,792.00
$196,825.00
$203,368.00
$202,985.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$367,629.00
$437,072.00
$501,260.00
$539,618.00
$555,847.00
$558,067.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$371,306.00
$325,400.00
$312,902.00
$263,409.00
$272,077.00
$272,508.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$296,765.00
$370,887.00
$402,573.00
$433,362.00
$445,877.00
$446,576.00

A
1
224
225
226
227
228
Test Administration
229
230
231
232
233
234
Test Administration
235
236
237
238
239
240
241
Test Administration
242
243
244
245
246
247
and accommodations.
districts.

and accommodations.


districts.



districts.

accommodations.


districts.
Due Date(s)
negotiations.)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component E
All Components
$0.00
$68,567.00
$71,314.00
$75,323.00
$77,282.00
$77,969.00
$2,000.00
$80,380.00
$81,889.00
$86,585.00
$89,717.00
$89,655.00
$0.00
$6,132,398.00
$4,341,263.00
$3,337,741.00
$2,559,066.00
$2,641,496.00
$2,652,530.00
$0.00
$151,508.00
$158,840.00
$170,746.00
$176,281.00
$177,318.00

A
1
Test Administration
248
249
250
251
252
253
Test Administration
254
255
256
257
258
259
260
Test Administration
261
262
263
264
265
266
267
Test Administration
268
269
270
271
272
273
accommodations.
districts.

accommodations.


districts.

accommodations.


districts.


and 7.4.7.4

districts.
Due Date(s)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component E
All Components
$0.00
$147,868.00
$155,280.00
$163,992.00
$168,576.00
$170,205.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$311,355.00
$238,334.00
$243,691.00
$259,417.00
$266,943.00
$268,296.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$171,752.00
$164,770.00
$172,527.00
$183,147.00
$188,649.00
$188,693.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A
1
Test Administration
B
districts.

and 7.4.7.4

districts.
and rosters.
and 7.4.7.4

districts.

labels and rosters.

and 7.4.7.4

districts.

and 7.4.7.4

districts.

280
281
282
283
284
285
286
Test Administration
287
288
289
290
291
292
Test Administration
293
294
295
296
297
298
Test Administration
299
and 7.4.7.4
274
275
276
277
278
279
Test Administration

labels and rosters.
Due Date(s)
negotiations.)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component E
All Components
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00

Test Administration
A
1
300
301
302
303
304
305
Test Administration
306
307
308
309
310
311
312
314
315
316
317
318
320
321
322
323
324
326
327

preidentification
labels and rosters.

labels and rosters.

critical date
Description
and 7.4.7.4

destination
address,
Evidence of
Completion
districts.

and 7.4.7.4

districts.
Due Date(s)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
Component E
All Components
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
7.4.8

November 2015
November 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00


7.4.8

May 2016
May 2017
$0.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00


7.4.8

May 2016
$0.00
$231,827.00
$438,152.00

Test Administration
A
1
328
329
330
331
333
334
335
336
337
339
340
341
342
343
345
346
347
348
349
350
352
353
354
355

Deliver
B
system for
each assessment.
Contractor
Deliverable

was operable for
the invoice
Deliverable
Description
7.4.8

E
occurred during
the period,
signed by the
Evidence
of Completion
Due Date(s)

assessment.

7.4.8

November 2015
November 2016
$0.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00

assessment.

7.4.8

February 2016
February 2017
$0.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00

assessment.

7.4.8

assessment.
7.4.8

F
Component E
All Components
$438,152.00
$438,152.00
$438,152.00
$438,152.00
August 2015
August 2016
August 2017
$0.00
$14,611.00
$14,659.00
$14,908.00
$14,978.00
$15,368.00
$14,973.00
August 2015
August 2016
August 2017
$0.00
$5,000.00
$4,000.00
$4,000.00
$4,000.00
Test Administration
A
1
356
Specify Component
A, B,
C, D or
(A-D):
_____E_____
of the contract
and
(orEno)
malfunctions
assessment.
was
operable
for
the
invoice
period.
ITN
Reference
occurred
during
the
period,
B
C
D signed by the
E
7.4.8
Due Date(s)
F
Component E
All Components
$4,000.00
$4,000.00
357
359
360
361
362



November 2015
November 2016
$0.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
363
365
366
367
368

May 2016
May 2017
$0.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
369
371
372
373
374
375

administration.


May 2015
May 2016
May 2017
$0.00
$8,520,577.00
$9,759,121.00
$11,584,901.00
$12,733,199.00
$12,515,577.00
$12,117,384.00
376
378
379
380
381



November 2015
November 2016
$0.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
Test Administration
A
1
382
384
385
386
387
388
390
391
392
393
394
395
397
398
399
400
401
402
404
405
406
407
408

B
(1) ProvidePRICING
A,ofB,the
C, D or
(A-D):
_____E_____
met theComponent
requirements
(orEno)
malfunctions
C
D
E
Due Date(s)












February 2016
February 2017
May 2015
May 2016
May 2017
August 2015
August 2016
August 2017
November 2014
November 2015
November 2016
F
Component E
All Components
$10,000.00
$0.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$0.00
$185,702.00
$185,702.00
$185,702.00
$185,702.00
$185,702.00
$185,702.00
$0.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$121,502.00
$8,000.00
$8,000.00
$8,000.00
$8,000.00
$8,000.00
Test Administration
A
1

B

Attachment
1bon
- DELIVERABLES
PRICING
and approved
a
the practice
Specify
Component
A, B, C, D or
E (A-D): _____E_____
that
offers students
requirements
C
D
Due Date(s)
Component E
All Components
$0.00
409
Test Administration
410
412
413
414
415
417
418
419
420
421
423
424
425
426

the contract.


training.
September 2014
7.4.9.8
administration.
August 2015
August 2016
November 2016
November 2017
previous year.
November 2015
November 2016
November 2017
$346,402.00
$204,305.00
$204,305.00
$204,305.00
$204,305.00
$204,305.00
$1,222,563.00
$1,012,697.00
$1,037,296.00
$1,080,174.00
$1,106,511.00
$1,114,574.00
$126,385.00
$112,900.00
$97,388.00
$76,803.00
$78,534.00
Test Administration
A
1
Specify Component
A, B, C,forD or
E (A-D):
_____E_____
and other materials
materials
and
a
period
of
at
least
six
months,
and
used
documents
end
of
each
assessment
year
B
C
E
Due Date(s)
F
Component E
All Components
$78,621.00
427
Test Administration
428
429
430
431
432
433
Test Administration
434
435
436
437
438
439
Test Administration
440
441
442
443
444
445
446
Test Administration
447
448
449
450
451

comment forms.

comment forms.


negotiations.)
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015

comment forms.
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$10,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
Test Administration
A
1
452
Test Administration
453
454
455
456
457
458
Test Administration
459
460
461
462
463
464
465
Test Administration
466
467
468
469
470
471
472
Test Administration

comment forms.
B
comment forms.

comment forms.

comment forms.

Trial Test

1b -forms.
online comment
Component A, B, C, D or E (A-D): _____E_____
and 7.4.7.7
C
Due Date(s)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
7.4.8 and 7.4.9.5
Component E
All Components
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$4,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$58,275.00
473

A
1
Test Administration
474
475
476
477
478
479
Test Administration
480
481
482
483
484
485
Test Administration
486
487
488
489
490
491
492
Test Administration
493
494
495
496
497
498
B



versions)
F
Component E
All Components
Reference 7.3.11.2
Due Date(s)
negotiations.)

Reference 7.3.11.2
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$12,735.00
$12,735.00
$12,735.00
$12,735.00
$12,735.00
$0.00

Reference 7.3.11.2
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$13,045.00
$13,045.00
$13,045.00
$13,045.00
$13,045.00
$0.00

Reference 7.3.11.2
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$330,227.00
$195,177.00
$195,177.00
$195,177.00
$195,177.00
$195,177.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$19,947.00
$19,947.00
$19,947.00
$19,947.00
$19,947.00

A
1
Test Administration
499
500
501
502
503
504
Test Administration
505
506
507
508
509
510
511
Test Administration
512
513
514
515
516
517
518
Test Administration
519
520
521
522
523
524
versions)
Reference 7.3.11.2
D
Due Date(s)
negotiations.)
Component E
All Components
$0.00
versions)
Reference 7.3.11.2
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$20,131.00
$20,131.00
$20,131.00
$20,131.00
$20,131.00
$0.00

versions)
Reference 7.3.11.2
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$65,378.00
$39,565.00
$39,565.00
$39,565.00
$39,565.00
$39,565.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
$20,077.00
$20,077.00
$20,077.00
$20,077.00
$20,077.00
$20,077.00
$0.00


Reference 7.4.1.
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$17,315.00
$17,315.00
$18,801.00
$18,801.00
$19,882.00

A
1
Test Administration
525
526
527
528
529
530
Test Administration
531
532
533
534
535
536
537
Test Administration
538
539
540
541
542
543
Test Administration
544
545
546
547
548
549
Reference 7.4.1.
Due Date(s)
negotiations.)
testing
Reference 7.4.1.


negotiations.)
Reference 7.4.1.


Reference 7.4.1.
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component E
All Components
$0.00
$13,344.00
$13,344.00
$14,226.00
$15,191.00
$15,191.00
$0.00
$1,297,893.00
$854,424.00
$847,198.00
$585,262.00
$608,579.00
$616,006.00
$0.00
$9,000.00
$11,336.00
$13,369.00
$13,834.00
$13,834.00
$0.00
$8,000.00
$9,000.00
$10,000.00
$10,000.00
$10,000.00

A
1
Test Administration
B
550
551
552
553
554
555
556
Test Administration
557
558
559
560
561
562
563
565
566
572
573
574
576
577
578
Reference 7.4.1.
Reference 7.4.1.

567
568
569
571

ITN Reference 7.5.1
D
E
Due Date(s)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016

management system.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
April 2014
April 2015
April 2016

management system.


management system.

F
Component E
All Components
$0.00
$38,448.00
$45,512.00
$47,748.00
$51,006.00
$53,126.00
$54,189.00
$0.00
$15,149.00
$17,646.00
$19,697.00
$20,938.00
$22,200.00
$21,892.00
$45,765.00
$45,765.00
$45,765.00
$45,765.00
$45,765.00
$45,765.00
September 2015
September 2016
$1,000.00
$1,000.00
$1,000.00

January 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
January 2017
A
1
579
581
582
583
584
585
587
588
589
590
592
593
594
595
597
598
599
600
601
603
604
605
606
607
609


monitor district
(1) EPreID
frequency
Specifyand
Component
A, B, C, D or
(A-D):
_____E_____
C reports of issues
Deliverable
Description
assessments. ITN
Reference
7.5.2
management system.
management system.
E
Due Date(s)
January 2015
January 2016
January 2017

review.

management system.
review.

management system.
review.


in .txt format

September 2015
September 2016
October 2015
F
Component E
All Components
$1,000.00
$6,000.00
$3,000.00
$3,000.00
$3,000.00
$3,000.00
$3,000.00
$1,000.00
$1,000.00
$1,000.00
October 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

management system.

January 2015
January 2016
January 2017
$1,000.00
$3,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

management system.

June 2015
June 2016
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

Protocol (SFTP).

September 2015
September 2016

A
1
610
611
612
614
615
616
617
619
620
621
622
623
625
626
627
file layout
and Full-State
layout for each
administration for
Contractor
Deliverable
Description
in .txt format

management system
andofSecure
File Transfer
Evidence
Completion
Protocol (SFTP).

file in .txt format

Protocol (SFTP).
January 2016


Protocol (SFTP).
January 2015
January 2016
January 2017
format

Protocol (SFTP).
September 2015
Due Date(s)
January 2017
September 2016
F
Component E
All Components
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
A
1
628
630
631
632
633
635
636
637
638
639
641
642
643
644

Specify
A, B,toC, D or
(A-D):
_____E_____
withComponent
the Department
(1) EFinal
PreID
format


File Transfer
D
Protocol (SFTP).

.txt format


Protocol (SFTP).
October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

.txt format


Protocol (SFTP).
January 2015
January 2016
January 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

.txt format


Protocol (SFTP).
June 2015
June 2016
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
E
Due Date(s)
F
Component E
All Components
$1,000.00
A
1
645
647
648
649
650
652
653
654
655
657
658
659
660
661
663
664
665
666
668
669
670
671

.txt format

Attachment
1b - DELIVERABLES
PRICING
the Department
to
(1) Final PreID
A, B,
E (A-D): _____E_____
for eachComponent
administration
forC, D or
management
D the SFTP.
Deliverable
of Completion
each PreID window
closes,Description
the contractor will
the file layout
and data file.




E
Due Date(s)

$1,000.00
$1,000.00
$1,000.00

May 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
May 2017
May 2015
May 2016
May 2017





$1,000.00
October 2015
October 2016

transcription logs
F
Component E
All Components

October 2015
October 2016
December 2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

May 2015
$1,000.00
$1,000.00

A
and Reporting
1 Scoring
Deliverable
Category
673
674
675
676
677
679
680
681
682
683
685
686
687
688
690
691
692
Spring EOC:
Development
will use a Special
Document
Contractor
Deliverable
Description

logofposted
to SFTP.
Evidence
Completion




7.5.6.2, and 7.5.6.4



7.5.6.2, and 7.5.6.4

E
Due Date(s)
May 2016
May 2017
July 2015
July 2016
July 2017
October 2015
October 2016
March 2016
March 2017
F
Component E
All Components
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$101,604.00
$87,016.00
$93,459.00
$97,415.00
$97,415.00
$9,000.00
$9,000.00
$10,000.00
$10,000.00
A
1
693
695
696
697
698
699
701
702
703

Specify
A, B,toC, D or
(A-D):
_____E_____
withComponent
the Department
(1) EFinal
approved

and CBT extract
C
7.5.6.2, and 7.5.6.4

data files



7.5.6.2, and 7.5.6.4


7.5.6.2, and 7.5.6.4

E
Due Date(s)
March 2015
March 2016
March 2017
October 2015
October 2016
F
Component E
All Components
$10,000.00
$4,500,909.00
$3,360,234.00
$2,074,039.00
$789,056.00
$820,862.00
$830,678.00
$103,694.00
$91,006.00
$97,582.00
$101,850.00
A
1

data files:
B

Specify
A, B,toC, D or
(A-D):
_____E_____
withComponent
the Department
(1) EFinal
approved
and CBT extract
C
7.5.6.2, and 7.5.6.4

E
Due Date(s)
F
Component E
All Components
$103,262.00
704
706
707
708


7.5.6.2, and 7.5.6.4

December 2015
December 2016
$103,054.00
$90,396.00
$95,924.00
$98,680.00
$101,385.00
709
711
712
713
714


7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017
$128,071.00
$138,552.00
$126,216.00
$134,120.00
$139,589.00
A
1
715
717
718

719
720
721
723
724
725
726
728
729
730



Component
B, C, D or
E (A-D):and
_____E_____
for scan/edit
and CBT A,
extract
scan/edit,
the
contractor
online
management
system
C
D and SFTP.
E
Due Date(s)
7.5.6.2, and 7.5.6.4

7.5.6.2, and 7.5.6.4


the data files.

the data files.
F
Component E
All Components
$140,592.00
July 2015
July 2016
July 2017
$18,212.00
$111,086.00
$97,779.00
$104,332.00
$108,183.00
$109,915.00
October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00

April 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2017
A
1
731
733
734
735
736
737
739
740
741
742
744
745
746
747
749
750
751
752
753
755
756
757
758
759
761

B of the
data files

Specifythe
Component
A, and
B, C, D or
(A-D): _____E_____
full state SSR
(1) EAnomalous
the data files.

the data files.
E
Due Date(s)
April 2015
April 2016
April 2017

data files

the data files.
data files
the data files.
data files
the data files.

score data files

the data files.


Reference 7.5.5

October 2015
October 2016
December 2015
December 2016
April 2015
April 2016
April 2017
July 2015
July 2016
July 2017

October 2015
October 2016
F
Component E
All Components
$1,000.00
$124,695.00
$71,961.00
$53,989.00
$34,836.00
$34,836.00
$34,836.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$8,000.00
$8,000.00
$8,000.00
$8,000.00
$8,000.00
$8,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A
762
763
764
766
767
768
769
771
772
773
774
775
777
778
779
780
782
783
B
and DSRsContractor
Deliverable of


DSR data files.

and DSR data files.

C
develop the State
Student Description
Deliverable
Reference 7.5.5

and online
of management
Completion system.
Due Date(s)

April 2016
F
Component E
All Components
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

Reference 7.5.5

April 2017

Reference 7.5.5


Reference 7.5.5

October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00

Reference 7.5.5

December 2015
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A

SSRs and
B
SSR/DSR Contractor
and multiple
784
785
787
788
789

790
791
793
794
795
796
797
799
800
801
802
804
805




C the Department to
Student Results
(DSR) file layouts
and use the layout
Deliverable
Description
Reference 7.5.5

E
email
Evidence of
Completion
Due Date(s)

Reference 7.5.5


Reference 7.5.5


7.5.5

October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00

$1,000.00
$1,000.00


April 2016
April 2017
F
Component E
All Components
$1,000.00
$1,000.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
July 2015
July 2016
July 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A

of SAR/DAR
file layouts
and multiple
Contractor
Deliverable

Results (SAR) and
District Aggregated
Deliverable
7.5.5

E
email
Evidence of
Completion
Due Date(s)

7.5.5


and DAR data files.

7.5.5

October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00


7.5.5

December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
806
807
809
810
811
812
813
815
816
817
818
820
821
822

DAR data files.
F
Component E
All Components
$1,000.00
$1,000.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
December 2015
$1,000.00
$1,000.00
823

A
1
825
826
827
B
828
829
831
832

833
834
835
837
838
839
840
842
843
844

data files.

data files.
E
Due Date(s)
April 2015
April 2016
April 2017
F
Component E
All Components
7.5.5

7.5.5


7.5.5.2 and 7.5.6.6
November 2016
$1,000.00
$1,000.00
$1,000.00

$1,000.00
$1,000.00

7.5.5.2 and 7.5.6.6
June 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
July 2015
July 2016
July 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
June 2017
A
1
845
847
848

B
data files.

849
850
851
853
854
855
856
858
859
860
Attachment
- DELIVERABLES
PRICING
Department1b
Scoring
(1) Final late
and Reporting Staff to
develop
the
State
Aggregated
to
online
management
system and SFTP.
C to develop the
E
Deliverable
Due Date(s)
scores and codes
7.5.5.2 and 7.5.6.6
7.5.5.2 and 7.5.6.6
F
Component E
All Components
$1,000.00
June 2015
June 2016
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
7.5.5.2 and 7.5.6.6
November 2016
$1,000.00
$1,000.00
$1,000.00

$1,000.00
$1,000.00

data files.
February 2015
February 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

7.5.5.2 and 7.5.6.6
861

A
1
863
864
865
B
data files.
866
7.5.5.2 and 7.5.6.6
867
869
870
871
872

data files.

7.5.5.2 and 7.5.6.6
873
875
876
877

7.5.5.3 and 7.5.6.7
878
880
881
882
7.5.5.3 and 7.5.6.7
E
Due Date(s)
June 2015
June 2016
June 2017
F
Component E
All Components
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
October 2015
October 2016
October 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
December 2015
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
October 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
October 2017
883

A
1
885
886
887
888
889
891
892
893
894
896
897
898
899
901
902
903
904
905
907
908
F
Component E
All Components
assessments.
7.5.5.3 and 7.5.6.7
Due Date(s)
October 2015
October 2016
October 2017

7.5.5.3 and 7.5.6.7
December 2015

7.5.5.3 and 7.5.6.7
March 2015
March 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

7.5.5.3 and 7.5.6.7
October 2015
October 2016
October 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

October 2015
October 2017
7.5.5.3 and 7.5.6.7
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A
1
909
910
911
913
914
915

files for all
assessments.
Contractor
Deliverable


918
919
920
921
922
923
924
925
926


reporting
F
Component E
All Components
$1,000.00
$1,000.00
$1,000.00
September 2015
September 2016
September 2017
$6,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00

2015
$2,000.00
$1,000.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2017
reporting
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
916
917

E
of all assessments.
data files.
Description
of Completion
Due Date(s)
7.5.5.3 and 7.5.6.7
927
928
929
930
931
932
933

initial reporting


2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$100,101.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
(Renewal 3 - 2020)
$1,000.00

A
1
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955


D
E
Due Date(s)
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
956
957
958
F
Component E
All Components
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$50,214.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$2,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$177,489.00
(Renewal 3 - 2020)
jump drives.
2015
$177,489.00
(Renewal 1 - 2017)
$175,378.00
districts.
A
1
959

- DELIVERABLES
deliver initial1b
reporting
(1) ProvidePRICING
CD/DVDs
or
jump
drives.
The
the
samples
jump drives.
C and media are
E
Evidence of
Completion
Due Date(s)
destination
address,
(Renewal 3 - 2019)
districts.
F
Component E
All Components
$175,378.00
$175,378.00
960
961
962
963


964

jump drives.
districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$123,266.00
(Renewal 3 - 2020)
$122,230.00

jump drives.
districts.

2015
$2,204,203.00
2016
2017
(Renewal 1 - 2018)
$2,222,324.00
$2,241,960.00
$2,474,398.00
(Renewal 2 - 2019)
$2,474,398.00
(Renewal 3 - 2020)
$2,474,398.00
$123,266.00
$121,908.00
$122,230.00
965
966
967
968


969
970
971
972
973
974
975

initial reporting


jump drives.
districts.

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$125,171.00
$153,502.00
$163,525.00
$163,525.00
$163,525.00
976
977

initial reporting

$78,266.00
jump drives.
2015

A
1
978
979

Student
B
Submit proof
of delivery:
Development
Contractor
Deliverable
initial reporting

submit proof
of delivery to the
Description
980

districts.
Due Date(s)
Component E
All Components
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$99,964.00
$106,693.00
$106,693.00
(Renewal 3 - 2019)
$106,693.00
981
982
983
984
985
986

initial reporting


jump drives.
districts.

2015
$747,532.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$801,521.00
$819,401.00
$824,298.00
$824,298.00
$824,298.00

initial reporting


jump drives.
districts.

2015
$60,218.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$64,687.00
$69,765.00
$69,765.00
$69,765.00
(Renewal 3 - 2020)
$69,765.00

reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
987
988
989
990
991
992
993
994
995
996
997


A
1

B
PRICING
Component
A, B, C,
E (A-D): _____E_____
each subject
that contains
theD or
reports.
will be
C
E
Due Date(s)
(Renewal 3 - 2019)
districts.
F
Component E
All Components
$0.00
998
999
1000
1001
1002



reports.
districts.

2016
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00



reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00


reports.
districts.

2015
2016
(Renewal 1 - 2017)
$0.00
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012

reports
$0.00
$0.00
A
1
1013

Tasks
B
reports
D
E
a separate page
that includes an
following for each
shipment:
Description
Evidence
of Completion
Due Date(s)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
districts.
F
Component E
All Components
$0.00
$0.00
1014
1015
1016
1017
1018

reports


reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00

reports


reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00



reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
A
1
1030

B
to selected
approval of
Attachment
1b - DELIVERABLES
PRICING
each subject
that
contains
the
reports.
E
will print and
of material
Description
Evidence
of Completion
Due Date(s)
(Renewal 2 - 2019)
districts.
(Renewal 3 - 2020)
F
Component E
All Components
$0.00
$0.00
1031
1032
1033
1034
1035

reporting


districts.

2015
$1,000.00
2016
(Renewal 1 - 2017)
$1,000.00
$1,000.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$1,000.00
$1,000.00

districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$1,000.00
(Renewal 3 - 2020)
$1,000.00

districts.

2015
$5,000.00
2016
2017
(Renewal 1 - 2018)
$5,000.00
$5,000.00
$5,000.00
(Renewal 2 - 2019)
$5,000.00
1036
1037
1038
1039

reporting

1040
$1,000.00
$1,000.00
$1,000.00
1041
1042
1043
1044
1045
1046


A
1
1047
1048
1049

Attachment
Reference 7.5.5
1b - DELIVERABLES
following for
PRICING
who
received
B
C
districts.
1050
1051

Due Date(s)
(Renewal 3 - 2020)
2015
(Renewal 1 - 2017)
(Renewal 3 - 2019)
districts.
Component E
All Components
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
1052
1053
1054
1055


1056

districts.

2015
$1,000.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$1,000.00
$1,000.00
$1,000.00
(Renewal 3 - 2019)
$1,000.00

districts.

2015
$2,000.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$2,000.00
$2,000.00
$2,000.00
$2,000.00
(Renewal 3 - 2020)
$2,000.00

districts.

2015
$1,000.00
2016
$1,000.00
1057
1058
1059
1060
1061


1062
1063
1064
1065


A
1
1066
1067
1068
1069
1071
1072
1073
1074
1076
1077
1078
1079
1081
1082
1083
1084
1085
1087
1088
1089

B Submit proof of
reports and
media for
the late reporting
Contractor
Deliverable


C drives. The
Department after
all reports
and media are
Deliverable
Description

(2) Provide proof
of delivery
Evidence
of Completion
districts.
Due Date(s)
Component E
All Components
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00



September 2015
September 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00



March 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00



March 2015
March 2016
March 2017



March 2017

September 2015
September 2016
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
1090

A
1
1092
1093
1094
1095
1097
1098
1099
1100


1101
1103
1104
1105
1106
1107



Performance Tasks

1108
1109
1110
1111
Due Date(s)
Component E
All Components
November 2015
November 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

March 2015
March 2016
March 2017
$4,000.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00
$4,000.00

June 2015
June 2016
June 2017

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$263,097.00
$251,567.00
$264,773.00
$272,510.00
A
1

Performance Tasks
B
A, B, C, D or E (A-D): _____E_____
or on theComponent
performance-based
that respondents
C
D
E
Due Date(s)
(Renewal 3 - 2019)
F
Component E
All Components
$278,476.00
1112
1113
1114
1115
1116
1117

Performance Tasks

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

2015
$0.00
$221,275.00
$229,456.00
$241,985.00
$248,763.00
$253,075.00
1118
1119
1120
1121
1122
1123

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$13,007,850.00
$11,536,755.00
$10,758,257.00
$10,477,075.00
$10,505,920.00
A
1

B
Attachment
performance-based
Specify Component
A, B, C, D or E (A-D): _____E_____
that respondents
D
E
Due Date(s)
(Renewal
3
2020)
F
Component E
All Components
$10,434,988.00
1124
1125
1126
1127
1128

Tasks

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$95,466.00
$106,355.00
$118,305.00
$122,497.00
$123,453.00
1129
1130
1131
1132
1133

Performance Tasks
$119,170.00
$76,212.00
$82,935.00
$85,725.00
A
1

Performance Tasks
B
Specify
Component
A, B,on
C, D or
(A-D):
_____E_____
of a students
response
theEfinal
handscoring
report.
D
E
Due Date(s)
(Renewal 3 - 2019)
F
Component E
All Components
$85,801.00
1134

Performance Tasks
1135
1136
1137
1138
1139


2015
1140
1141
1142
1143
1144

Performance Tasks
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
$782,724.00
$833,429.00
$862,996.00
$913,019.00
$941,635.00
$914,768.00
$143,899.00
$159,651.00
$169,211.00
$181,194.00
A
1
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167

Performance Tasks
B
A, B, C, D or E (A-D): _____E_____
or on theComponent
performance-based
that respondents
C
D
E
Due Date(s)
(Renewal 2 - 2019)
(Renewal 3 - 2020)


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.


Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5
June 2015
June 2016
January 2016
January 2017
January 2015
January 2016
January 2017
June 2015

the SFTP.
June 2016
F
Component E
All Components
$187,426.00
$188,691.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$35,283.00
$0.00
$0.00
$0.00
$0.00
$0.00
$3,000.00
$0.00
$0.00
$0.00
$0.00

A
1
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
all assessments
Reference 7.5.5

all assessments
Reference 7.5.5

all assessments
Reference 7.5.5

ELA/Mathematics

ELA/Mathematics
D
the SFTP.
Due Date(s)
Component E
All Components
October 2015
October 2016
the SFTP.
January 2016
the SFTP.

April 2015
April 2016
April 2017

December 2015
Reference 7.5.4.2
Reference 7.5.4.2

October 2016
October 2017
October 2015
and ELA/Mathematics
October 2017
Reference 7.5.4.2
December 2015
Reference 7.5.4.2
$0.00
$0.00
$0.00
$0.00
$0.00
$9,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$4,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$10,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A
1
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
Scoring
and Reporting
Deliverable
Category
will create and
posted toofthe
SFTP.
Description
Completion
Due Date(s)
Reference 7.5.4.2
March 2016
Reference 7.5.4.2
October 2015
October 2017
Reference 7.5.4.2

ELA/Mathematics

Reference 7.5.4.2

Retake

Fall EOC:Contractor
Development
of the history
Deliverable
1225
1226
1227
1229
1230

October 2015
October 2017
November 2015

May 2016

Retake
F
Component E
All Components
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$4,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$5,000.00
$1,000.00
$1,000.00

A
1231
1232
1234
1235
1236
1237
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1255
1256

D
E
rule specified in
Deliverable
Due Date(s)

rescoring.

Reference 7.5.6.5

management system.
log.

manual rescoring.

Reference 7.5.6.5

management system.
log.


Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

October 2015
October 2016
April 2016
F
Component E
All Components
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2017
April 2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$4,000.00
April 2016
April 2017
October 2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
October 2016
December 2015
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

1
1257
1258
1260
1261
1262
1263
1264
1266
1267
1268

B

C
Deliverable
Description
in the
Reference 7.5.6.5

D
management system.
log.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.


Reference 7.5.5

districts.


Protocol (SFTP).

rescoring.
1269
1270
1271
1272
1273
1274
1275
1276
1278
Studies
1279
1280
1281

E
Due Date(s)
April 2015
April 2016
April 2017
F
Component E
All Components
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

July 2015
July 2016
July 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

July 2015
$1,000.00
$40,886.00
July 2016
July 2017
$40,886.00
$40,886.00

$40,886.00
$40,886.00
$40,886.00
May 2014
May 2015
$137,819.00
$5,000.00
May 2016
$0.00
$0.00

$0.00
$0.00
1282

A
1
1284 Studies
1285
Administrations
1286
1287
1288
Studies
1290
1291
1292
1293
1294
Studies
1296
1297
1298
1299
1300
1303 Studies
1304
1305
D
deliverable.
Plans
Administrations

Specifications


Due Date(s)
Component E
All Components
July 2014
July 2015
$218,492.00
$218,492.00
July 2016
$218,492.00
$218,492.00

$218,492.00
$218,492.00
July 2014
July 2015
$14,911.00
$5,000.00
July 2016
$5,000.00
$5,000.00
$5,000.00
$5,000.00
$14,537.00
$5,000.00

August 2014
August 2015
August 2016
December 2014
December 2016
$5,000.00
$5,000.00
$5,000.00
$5,000.00
$52,264.00
$5,000.00
$5,000.00
$5,000.00
$5,000.00

Studies
A
1306
1307
1308
1309
1310
1311
1312
1313

Studies

1314 Studies
1315
1316
1317
1318
1321
1322
1323
Studies

Specifications
B
Specify
Component
A,plans
B, C, D or
(A-D):Department
_____E_____
procedures,
scoring
(2) EProvide
D
E
Due Date(s)


posted to SFTP.
May 2015
May 2016
May 2017
2016

via SFTP.
2017


via SFTP.
2016
(Renewal 1 - 2018)
(Renewal 2 - 2019)

(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
F
Component E
All Components
$5,000.00
$390,462.00
$390,462.00
$390,462.00
$390,462.00
$390,462.00
$390,462.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

Studies
1324
1327

B

Technical Reports
$1,000.00

posted to SFTP.


Reference 7.6.3.3

November 2014
posted to SFTP.
$20,412.00
$0.00



October 2014
SFTP.
$99,654.00
1328
1329

Studies
F
Component E
All Components

Studies
1330
Attachmentmust
1b - DELIVERABLES
(1) SubmitPRICING
most recent
SSR
files
and
via
SFTP.
D
E
the demographic
Description
Due Date(s)
The specifications
will also address
(Renewal 3 - 2020)
December 2015
December 2016
December 2017
$157,381.00
$252,617.00
$252,617.00
$252,617.00

$252,617.00
$252,617.00
1332
1333 Studies
1334
Studies


May 2015
$0.00
$122,814.00
$0.00
1336

A
1
C
7.6.3.4
D
Due Date(s)
Component E
All Components
Studies
Setting Work Plan

Studies

Studies

Technical Report
SFTP.
$34,179.00
SFTP.
$10,000.00
$5,000.00
$2,000.00
1337
$230,802.00
1338
1339
Studies


7.6.3.4.
Department.
renewal)
1341

A
1
Studies
Report
Due Date(s)
August 2015




Reference 7.6.3.6
SFTP.
F
Component E
All Components
$20,452.00
$5,000.00
1343
Studies
1344
1345
Studies

Report
$19,799.00
$0.00
$50,323.00
$0.00
1347
1348 Studies

across devices.
input types.
concordance.
characteristics.
$6,000.00

Scaling, Equating,
A Scoring,
Studies
1349
Studies
for conducting
Delivering
Assessments
and detailed
Contractor
Deliverable
Description
across devices.
input types.
concordance.
characteristics.

D
email
Evidence of
Completion

with TAC Committee


August 2015



November 2015
SFTP.
Due Date(s)
Component E
All Components
$0.00
$33,058.00
$0.00
1351
Studies
1352
$1,000.00

Studies
A
1353
Studies
1355
Studies

B assessments
November 2015
SFTP.
C and events, as
D
TAC members.
of Completion


7.6.3.6.2

August 2016

Assessments


November 2015
SFTP.
Due Date(s)
Component E
All Components
$0.00
$10,964.00
$0.00
$1,000.00
$0.00
1357

A
1
Studies
B
Assessments
1358
C
7.6.3.6.3
D
August 2016
Due Date(s)
Component E
All Components
$1,000.00
$0.00
1359
1361
1362
1363
1364
1365
1367
1368
1369
1370
1371
1373
1374
1375
1376
1377
1379
1380
1381
1382

7.7.2

report

Meetings


previous invoice.

ITN.


Action Item Report

work. ITN Ref 7.7.6

April 15, 2014

April 15, 2015
April 15, 2016
$94,569.00
$39,900.00
$39,900.00
$39,900.00
$39,900.00

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$39,900.00
$243,757.00
$244,851.00
$245,701.00
$246,520.00
$248,308.00
$249,132.00
$2,532,223.00
$2,836,585.00
$2,861,619.00
$2,449,123.00
$2,494,619.00
$2,509,241.00
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$321,310.00
$321,310.00
$321,310.00
$321,310.00
$321,310.00
Program Management
A
1
1383
1385
1386
1387
1388
1389
1391
1392
1393
1394


Specify
E (A-D): _____E_____
status Component
management
those
leading
up
to
previous
invoice
period. (Quarterly
invoices to occur
C
D
E
Due Date(s)
(Renewal 3 - 2019)
work. ITN Ref 7.7.6
F
Component E
All Components
$321,310.00

November 2015
November 2016
November 2017
$78,060.00
$78,060.00
$78,060.00
$78,060.00
$78,060.00
$78,060.00

ITN Ref 7.7.6
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$99,882.00
$99,710.00
$99,710.00
$99,710.00
$99,710.00
Program Management
A
1

B

Specify
Component
A,level
B, C,ofD or
E (A-D): reports.
_____E_____
by program
task and
accounting
and
balances
of
February,
May,
August,
and
C
D November)
ITN Ref 7.7.6
Due Date(s)
Component E
All Components
(Renewal 3 - 2019)
$99,710.00

$61,667.00
1395
Program Management
Ref. 7.7.10
1396

A
1
3
4
5
6
7
8
Test Development
Document
Test Development

Specifications
9
10
11
12
13
C
7.3.10
via SFTP.
Due Date(s)
June 1, 2014
June 1, 2015
June 1, 2016
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5
F
Component A
3-8 ELA
$38,237.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
14
Test Development
15
16
17
18
19


7.5.4.5

SFTP.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$20,191.00

7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
20
Test Development
21
22
23
24
25

$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
26
Test Development
27
28
29
30
31
$0.00
$0.00
$0.00
$0.00
$0.00
32

A
1
Test Development
33
34
35
36
37
F
Component A
3-8 ELA
7.5.4.5
Due Date(s)
SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
38
Test Development
39
40
41
42
43
$0.00
$0.00
$0.00
$0.00
$0.00
44
Test Development
45
46
47
48
49
50
51 Test Development
52
53
54
55
56
Test Development

Specifications
7.3.10



June 1, 2014
June 1, 2015
June 1, 2016
November 1, 2015
$0.00
$0.00
$0.00
$0.00
$0.00
$15,797.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$35,812.00
57

A
1
Test Development
B
58
59
60
61
62
each scoring guide.
educators understand the scoring of constructed(2) Final PDF files provided via SFTP.
it appeared on the test and responses for each score
point, images of student responses, and annotations
explaining why the response received a particular
score are provided. ITN Reference 7.3.10
Due Date(s)
Reference 7.3.7.5


June 1, 2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Component A
3-8 ELA
$31,444.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
63
Test Development

Assessment Reports
64
65
66
67
68
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$67,402.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
69
Test Development
70
71 Test Development
72
73
74
75
76 Test Development
77
78
79
80

$147,745.00

management report
Reference 7.3.4.2

February 1, 2017
management report
$3,956,087.00
$4,036,987.00
$2,137,072.00
$1,867,100.00
$1,584,816.00
Review

$4,034,877.00
$32,066.00
$32,066.00
$32,066.00
$32,066.00

management report
May 1, 2015
May 1, 2016

A
1
81 Test Development
82
83
84
85
86
87 Test Development
88
89
90
91 Test Development
92
93
94
95 Test Development
96
97
98
99
100
102
103
104
105
106
108
109
110
111
112
B
To ensure the quality and number of items/passages

developed is sufficient to create future operational
test forms, the contractor will submit an item
development plan delineating the number and
types of items to be developed. ITN Reference 7.3.0

management report.
December 1, 2015
December 1, 2016
$23,304.00
$23,304.00
$23,304.00
$23,304.00


management report.
December 1, 2015
December 1, 2016
$19,359.00
$9,109.00
$9,109.00
$9,109.00


previous invoice.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$111,390.00
$112,882.00
$114,410.00
$117,566.00
$119,720.00
$120,058.00




management report
Due Date(s)
July 15, 2014
July 15, 2015
July 15, 2016
F
Component A
3-8 ELA

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$33,668.00
$48,762.00
$48,762.00
$48,762.00
$48,762.00
$48,762.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A
1
Test Administration
B
F
Component A
3-8 ELA
the report.
Reference 7.4.2
Due Date(s)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
the report.
ITN Reference 7.4.2

2016
$0.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00

a report.
the report.
ITN Reference 7.4.2

2015
$2,000.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

the report.
ITN Reference 7.4.2

2015
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
the report.
ITN Reference 7.4.2

2015
$0.00
2016
$0.00
113
114
115
116
$0.00
117
Test Administration

118
119
120
121
122
Test Administration
123
124
125
126
127
128
Test Administration
129
130
131
132
133
Test Administration
134
135

Test Administration
A
1
136
137
Specify
Component
A, B, C, D or
E (A-D):
_____A_____
Winter EOC: Inventory missing materials Once all materials are
scanned
in and inventoried,
1) Final
missing
materials report.
return any secure materials
the report.
B
C identified as missing.
D
districts, materials
have been
accounted for, and
Deliverable
Description
ITN Reference 7.4.2
Due Date(s)
Component A
3-8 ELA
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
the report.
ITN Reference 7.4.2

2015
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00

the report.
ITN Reference 7.4.2

2015
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00
the Specifications.

2015.
$0.00
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2017
$0.00
138
Test Administration
139
140
141
142
143
144
Test Administration
145
146
147
148
149
150
152
153
154
155
156
158
159

the Specifications.
Test Administration
A
1
160
161
162
164
165
166
167
168
169
171
172
173
174
175
177
178
179
180
181
183
184
Attachment
1b - DELIVERABLES
PRICING
test materials, ancillary materials,
and student
the Specifications.
reports. ITN Reference
7.4.1 Component A, B, C, D or E (A-D): _____A_____
Specify
B
C
D
the Specifications.

administration.

administration.

administration.
the Specifications.
the Specifications.
the Specifications.
Due Date(s)
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$79,909.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
$81,640.00
$67,932.00
$72,745.00
$75,477.00
$76,384.00
$0.00
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
$0.00
Test Administration

administration.
and student
the Specifications.
Attachment
1b - DELIVERABLES
PRICING
A
1
185
186
187
188
190
191
192
193
194
195
197
198
199
200
201
Test Administration
202
203
204
205
206
207
208
Test Administration
209
210
211
212
213
214
215
Test Administration
216
217
B

administration.

memos, etc.
C
D
the Specifications.


districts.


districts.
PERIODIC: ELA/L and Mathematics Paper- For each administration, all test administration
Based Test: Deliver test administration manuals and ancillary materials must be
manuals, including packing lists, work
folders, applicable memos.

districts.

memos.

districts.

memos.

Due Date(s)
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$257,340.00
$305,950.00
$350,882.00
$377,733.00
$389,093.00
$390,647.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$259,914.00
$227,780.00
$219,031.00
$184,386.00
$190,454.00
$190,756.00
$0.00
2015
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Test Administration
A
1
218
219
220
221
222
224
225
226
227
228
Test Administration
229
230
231
232
233
234
Test Administration
235
236
237
238
239
240
241
Test Administration
242
243
244
245
246
247
Test Administration
248
249
250
251
252
253

administration manuals,
B including
memos. Contractor Deliverable

manuals and ancillary materials
must be
C
the critical or extremely
critical
date listed in
Deliverable
Description

quantity and type
of material
Evidence
of Completion
districts.

and accommodations.


and accommodations.




accommodations.


accommodations.

Due Date(s)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$2,388,868.00
$1,457,411.00
$652,271.00
$694,605.00
$717,158.00
$720,334.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
254
255
256
257
258
259
260
Test Administration
261
262
263
264
265
266
267
Test Administration

accommodations.



and 7.4.7.4

districts.

and 7.4.7.4

districts.

and 7.4.7.4

districts.
and rosters.
and 7.4.7.4

districts.

274
275
276
277
278
279
Test Administration
280
281
282
283
284
285
286
Test Administration
287
288
accommodations.
268
269
270
271
272
273
Test Administration

Due Date(s)
negotiations.)
Component A
3-8 ELA
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Test Administration
A
1
289
290
291
292
Test Administration
B
C
and rosters.
and rosters must be successfully
delivered to
listed in Appendix
B. ITN References
7.5.2, 7.4.7.3,
Deliverable
Description
and 7.4.7.4

quantity and type
of material
Evidence
of Completion
districts.

labels and rosters.

and 7.4.7.4

districts.

and 7.4.7.4

districts.

and 7.4.7.4

districts.
Fall ELA/L Retake: Deliver and maintain

the online management system for the

7.4.8



7.4.8

293
294
295
296
297
298
Test Administration

labels and rosters.
299
300
301
302
303
304
305
Test Administration
306
307
308
309
310
311
312
314
315
316
317
318
320
321
322
323

labels and rosters.
Due Date(s)
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00

Test Administration
A
1
324
326
327
328
329
330
331
333
334
335
336
337
339
340
341
342
343
345
346
347
348
349
350
352
353
354
355
356
357
359

Specify
Component
A, B, C, D or
(A-D):a_____A_____
Provide evidence that
the online
management
(1) EProvide
report listing met requirements and any
was operable for the invoice period. ITN Reference occurred during the period, signed by the
C
D
E
7.4.8
Due Date(s)
F
Component A
3-8 ELA
$0.00


7.4.8


assessment.

7.4.8

November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

February 2016
February 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00



(or no) malfunctions or unavailability of the site that November 2015
May 2015
May 2016
May 2017
$0.00
$139,096.00
$262,891.00
$262,891.00
$262,891.00
$262,891.00
$262,891.00
$0.00
$0.00

A
Test Administration

assessment platform
met the
requirements of the
Deliverable
Description

(or no) malfunctions
or unavailability
of the site that
Evidence
of Completion
Due Date(s)
November 2016
363
365
366
367
368

May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
369
371
372
373
374
375

administration.


May 2015
May 2016
May 2017
$0.00
$3,749,054.00
$4,294,013.00
$5,097,356.00
$5,602,608.00
$5,506,854.00
$5,331,649.00
376
378
379
380
381



November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
382
384
385
386
387



February 2016
February 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00



May 2015
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
1
360
361
362
388
390
391
392
393
394
Fall ELA/L Retake: Deliver a computerbased assessment

platform
that meets
Contractor
Deliverable
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
Test Administration
A
1
395
397
398
399
400
401
402
404
405
406
407
408

(1) ProvidePRICING
met theComponent
requirementsA,ofB,the
(orEno)
malfunctions
C, D or
(A-D):
_____A_____
C
D
Evidence
ofdate.
Completion
the current invoice
billing

E
Due Date(s)




November 2014
November 2015
November 2016

materials to support ongoing training
for the audiences identified in the
contract.

September 2014
August 2015
August 2016
August 2017
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$63,181.00
$3,000.00
$3,000.00
$3,000.00
$3,000.00
$3,000.00
$0.00
409
Test Administration
410
412
413
414
415
417
418
419
420

training.
7.4.9.8
administration.
August 2015
August 2016
November 2016
November 2017
previous year.
$242,481.00
$143,014.00
$143,014.00
$143,014.00
$143,014.00
$143,014.00
$537,928.00
$445,587.00
$456,410.00
$475,277.00
$486,865.00
$490,413.00
421

A
1
423
424
425
426
D
F
Component A
3-8 ELA
Due Date(s)
November 2015
November 2016
November 2017
$45,445.00
$33,796.00
$16,362.00
$17,002.00
$17,389.00
$17,411.00
427
Test Administration
428
429
430
431
432
433
Test Administration
434
435
436
437
438
439
Test Administration
440
441
442
443
444
445
446
Test Administration
447
448
449
450
451
452
Test Administration
453
454
455

comment forms.

comment forms.



comment forms.

comment forms.



negotiations.)
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
$0.00
$0.00

Test Administration
A
1
456
457
458
Test Administration
459
460
461
462
463
464
465
Test Administration
466
467
468
469
470
471
472

comment forms. B

comment forms.

comment forms.
Component
B, C, D or
(A-D):Department
_____A_____
Includes data file of Specify
all responses,
analysis,A,
and
(1) EProvide
email reflecting approval of
response questions from all
C online comment forms.
D


Test Administration

Trial Test

7.4.8 and 7.4.9.5

Test Administration


Reference 7.3.11.2

Due Date(s)
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
August 1, 2014
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$58,275.00
473
474
475
476
477
478
479
Test Administration
480
481
482
483
484
485


Reference 7.3.11.2


negotiations.)
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
486
487
488
489
490
491
492
Test Administration
493
494
495
496
497
498
Test Administration
499
500
501
502
503
504
Test Administration
505
506
507
508
509
510
511
Test Administration
512
513
514
515
516
517
518
Test Administration
519
B

versions)
C
Reference 7.3.11.2

Reference 7.3.11.2
versions)
Reference 7.3.11.2
Spring EOC: Computer-Based Test form

versions)

Reference 7.3.11.2

versions)
Reference 7.3.11.2


Reference 7.4.1.
D
Due Date(s)
negotiations.)
Component A
3-8 ELA
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$132,091.00
$78,071.00
$78,071.00
$78,071.00
$78,071.00
$78,071.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00





Test Administration
A
1
520
521
522
523
Fall ELA/L Retake: Retrieve

used and
B
After each test administration,

the contractor will
C
the ITN for scanning,
scoring,
and storing, as
Deliverable
Description
Reference 7.4.1.
(1) Provide a proof of retrieval

D spreadsheet listing
E
quantity and type
of material
received, and the
Evidence
of Completion
Due Date(s)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)

Reference 7.4.1.


negotiations.)
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
testing
Reference 7.4.1.


negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
524
Test Administration
Spring ELA/L Retake: Retrieve used and

525
526
527
528
529
530
Test Administration
531
532
533
534
535
536
537
Test Administration
538
539
540
541
542
543
Test Administration
Reference 7.4.1.

544
545
546
547
548
549
Test Administration
550
551
552
553
554


Reference 7.4.1.

Reference 7.4.1.
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
$505,593.00
$286,840.00
$165,562.00
$158,857.00
$165,227.00
$167,286.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
Test Administration
A
1
555
556
Test Administration
557
558
559
560
561
562
563
565
566

B
572
573
574
576
577
578
579
581
582
583
584
585
587
588
589
590
592
593
Reference 7.4.1.

567
568
569
571

1b - DELIVERABLES
PRICING
retrieve test materials fromAttachment
the sites identified
in
the following
for each shipment: date/time received,
the ITN for scanning,Specify
scoring,Component
and storing, as
A, B, C, D or
quantity
E (A-D):
and_____A_____
type of material received, and the
described in the ITN. This includes accommodated associated district. Must show proof of successful
C
D
Reference 7.4.1.

ITN Reference 7.5.1
Due Date(s)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016

management system.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
April 2014
April 2015
April 2016

management system.



management system.



management system.

management system.
review.

management system.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$0.00
$0.00
$0.00

January 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
January 2017
January 2015
January 2016
January 2017

$0.00
$0.00
$0.00
September 2015
September 2016

review.
F
Component A
3-8 ELA

September 2015
September 2016
October 2015
October 2016
$0.00
$2,500.00
$1,250.00
$1,250.00
$1,250.00
$1,250.00
$1,250.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
594
595
597
598
599
600
601
603
604
605
606
607
609
610
611
612
614
615
616
617
619
620
621
622
623
Winter EOC: Development

of PreID freq The Contractor will collectCand monitor district
B
review. Contractor Deliverable
Frequency Reports
and theDescription
reports of issues
Deliverable
(1) PreID frequency reports

D posted to the online
management system.
reflecting receipt of
Evidence email
of Completion
E
Due Date(s)

review.


management system.

January 2015
January 2016
January 2017



management system.

June 2015
June 2016
June 2017

in .txt format


Protocol (SFTP).

file in .txt format

Protocol (SFTP).
January 2016


Protocol (SFTP).
January 2015
January 2016
January 2017
format

Protocol (SFTP).
September 2015

September 2015
September 2016
January 2017
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00

A
and Reporting
1 Scoring
Deliverable
Category
625
626
627
628
630
631
632
633
635
636
637
638
639
641
642
648
649
650
F
Component A
3-8 ELA
Fall EOC:Contractor
Development
of the file layout The ContractorDeliverable
will work with
the Department to
Deliverable
Description
format
(1) Final PreID Evidence

file layoutof
posted
to the online
Completion
Protocol (SFTP).
Due Date(s)
September 2016

.txt format


Protocol (SFTP).
October 2015
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00

.txt format


Protocol (SFTP).
January 2015
January 2016
January 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

.txt format


Protocol (SFTP).
June 2015
June 2016
June 2017
$0.00
$0.00
$0.00
$0.00

$0.00
$0.00
October 2015
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00
643
644
645
647

7.5.4.4
$0.00
$0.00
$0.00
$0.00

A
1
652
653
654
655
657
658
659
660
661
663
664
665
666
668
669
670
671
673
674
675
676
677
679
680
681
682
683
685
686
687
7.5.4.4
Due Date(s)
Component A
3-8 ELA
May 2016
May 2017

transcription logs
7.5.4.4
May 2015
May 2016
May 2017

7.5.4.4
7.5.4.4
7.5.4.4

7.5.4.4

7.5.6.2, and 7.5.6.4


October 2015
October 2016
December 2015
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

May 2015
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

October 2015
October 2016

A
1
Fall ELA/L Retake: Early district and fullstate scan/edit and CBT
B extract data

and CBT extract
C
7.5.6.2, and 7.5.6.4

scan/edit, and CBT extractDdata files, posted to online
E
Due Date(s)
688
690
691
692


7.5.6.2, and 7.5.6.4

March 2016
693
695
696
697
698

data files

7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017


699
701
702
703

March 2017
October 2015
October 2016
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$2,660,273.00
$1,660,242.00
$1,681,235.00
$393,406.00
$409,265.00
$414,159.00
$0.00
$0.00
$0.00
$0.00

A
1

data files:
B

and CBT extract
C
7.5.6.2, and 7.5.6.4

E
Due Date(s)
F
Component A
3-8 ELA
$0.00
704
706
707
708


7.5.6.2, and 7.5.6.4

December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
709
711
712
713
714


7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00



July 2015
scan/edit, and CBT extract data files, posted to online July 2016
July 2017
715
717
718
$0.00
$0.00
$0.00

A
Summer EOC: Early B

district and full-state
Development
of the Deliverable
file layouts,
Contractor
719
720
721
723
724
725
726
728
729
730
731
733
734
735
736
737
739
740
741
742
744
745
746
747
749
750
751
752
753

C the Department to
data files. For each
administration,
the contractor
Deliverable
Description
7.5.6.2, and 7.5.6.4
(1) Final approved file layouts,

D early/full state
E
management system
andofSFTP.
Evidence
Completion
Due Date(s)
(2) Provide Department email reflecting receipt and (Renewal 1 - July 2018)
score data files
the data files.


the data files.

data files

the data files.

the data files.
data files
the data files.
data files
the data files.
$0.00
$0.00
$0.00
October 2015
October 2016
$0.00
$0.00
$0.00

April 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$64,841.00
$37,420.00
$28,074.00
$18,115.00
April 2017
April 2015
April 2016
April 2017

data files
F
Component A
3-8 ELA

October 2015
October 2016
December 2015
$18,115.00
$18,115.00
$0.00
$0.00
$0.00
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
755
756
757
758
759
761
762
763
764
766
767
768
769
771
772
773
774
775
777
778
779
780
782
783
784
score data files
the data files.


Reference 7.5.5


DSR data files.

and DSR data files.



Reference 7.5.5


Reference 7.5.5


Reference 7.5.5


Reference 7.5.5

E
Due Date(s)
July 2015
July 2016
July 2017
October 2015
October 2016

April 2016
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2017
October 2015
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00
December 2015
$0.00
$0.00
$0.00
$0.00
December 2016
A
1
785
787
788
789

SSR/DSR file layoutsBand multiple

790
791
793
794
795
796
797
799
800
801
802
804
805



806
807
809
810
811
812

DAR data files.


Student Results (DSR) file C
layouts and use the layout
Deliverable
contain all students
testedDescription
in the State. A DSR file
Reference 7.5.5

D reflecting receipt and

Reference 7.5.5


Reference 7.5.5







E
Due Date(s)
F
Component A
3-8 ELA
$0.00
April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00

$0.00
$0.00
October 2015
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
A
1
813
815
816
817
818
820
821

B of the SAR and
and multiple versions
DAR data files.
838
839
$1,000.00

October 2015








7.5.5.2 and 7.5.6.6
November 2016
$0.00
$0.00
$0.00
$0.00


833
834
835
837
Due Date(s)
F
Component A
3-8 ELA

828
829
831
832

and DAR data files.
822
823
825
826
827
The contractor will work with Department Scoring (1) Final initial reporting SAR and DAR data files
and Reporting Staff to develop the State Aggregated posted to online management system and SFTP.
C to develop the
scores and codes
Deliverable

data files.
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00
December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
840
842
843

SSR/DSR and multiple
B versions of DAR
data files.

data files.
844
845
847
848

849
850
851
853
854
855
856
858
859
F
Component A
3-8 ELA
$0.00

7.5.5.2 and 7.5.6.6
(1) Final late reporting SSR and DSR data files posted June 2016
June 2017
$0.00
$0.00
$0.00
$0.00
$0.00

7.5.5.2 and 7.5.6.6
June 2015
June 2016
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
7.5.5.2 and 7.5.6.6
November 2016
$0.00
$0.00
$0.00

$0.00
$0.00

data files.

7.5.5.2 and 7.5.6.6
(1) Final late reporting SSR and DSR data files posted February 2015
February 2016
(2) Provide Department email reflecting receipt and (Renewal 1 - February 2017)
$0.00
$0.00
$0.00
$0.00
$0.00

June 2016
860
861
863
Specify
B, C, D or
(A-D):
withComponent
Department A,
Scoring
(1) EFinal
late_____A_____
and Reporting Staff to develop the State Aggregated to online management system and SFTP.
Results (SAR) and District C
Aggregated Results (DAR) (2) Provide Department email
E
SAR/DAR data Deliverable
files. The SSR
files contain all of the
Description
Due Date(s)
7.5.5.2 and 7.5.6.6

data files.
$0.00
$0.00

A
1
864
865
B
DSRs dataContractor
files: Development
of
Deliverable
data files.
866
to online management
and SFTP.
Evidencesystem
of Completion
Due Date(s)
(2) Provide Department email reflecting receipt and June 2017
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
and 7.5.6.7
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00

assessments.
and 7.5.6.7
October 2015
October 2016
October 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

December 2015
and 7.5.6.7

7.5.5.2 and 7.5.6.6
873
875
876
877

and 7.5.6.7
889
891
892
$0.00
$0.00
$0.00
$0.00

data files.
883
885
886
887
888
F
Component A
3-8 ELA
867
869
870
871
872
878
880
881
882

and Reporting Deliverable
Staff to develop
the State Aggregated
Description
7.5.5.2 and 7.5.6.6
$0.00
$0.00
$0.00

A
893
894
896
897
898
899
901
902
903
904
905
907
908
909
910
911
913
914
915
Fall EOC: Close out SSR

B data files:
files for all
assessments.
Contractor
Deliverable
918
919
920
921
922
923
924
F
Component A
3-8 ELA
$0.00

and 7.5.6.7
March 2015
March 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

and 7.5.6.7
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

and 7.5.6.7
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00


September 2015
September 2016
September 2017
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

reporting

2015
$1,000.00
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2017
reporting
$0.00
$0.00
$0.00
$0.00
$0.00
916
917

C close out files after all (1) Final closeout SSR files Dposted to SFTP.
E
of all assessments.
data files.
Description
of Completion
Due Date(s)
and 7.5.6.7

$0.00

A
1
925
926

files and PDFs of reports
for districts and
B for districts and the online management system
C
and PDFsContractor
of reports for
the initial
Deliverable
reporting

D
approval of files
and reports.
Evidence
of Completion

initial reporting


Due Date(s)
Component A
3-8 ELA
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
927
928
929
930
931
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955


$1,000.00
$1,000.00
$1,000.00
$1,000.00
(Renewal 3 - 2020)
2015
$1,000.00
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
$0.00
$0.00
$0.00
$0.00
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
956
957

$70,000.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
932
933

2015


jump drives.
districts.
(Renewal 3 - 2020)
2015
2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1

Student Reports andBCD/DVDs or jump
Development
of the Deliverable
student reports and
Contractor

Student Reports and CD/DVDs
C or jump drives. The
Department after
all reports
and media are
Deliverable
Description
958
959

the samples of the studentDreports and CD/DVD or
jump drives.
(2) Provide proof
of delivery
Evidence
of Completion
districts.
Due Date(s)
Component A
3-8 ELA
(Renewal 1 - 2017)
$0.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00

jump drives.
districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00

jump drives.
districts.

2015
$969,849.00
2016
2017
(Renewal 1 - 2018)
$888,930.00
$896,784.00
$989,759.00
(Renewal 2 - 2019)
$989,759.00
(Renewal 3 - 2020)
$989,759.00
960
961
962
963


964
$0.00
$0.00
$0.00
965
966
967
968


969
970
971
972
973
974
975

initial reporting


jump drives.
districts.

2015
2016
(Renewal 1 - 2017)

jump drives.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
(Renewal 3 - 2019)
$0.00

jump drives.
districts.

2015
$0.00
2016
$0.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
976
977
978
979

initial reporting

980
981
982
983

initial reporting


A
1
984
985
986

Reports and CD/DVDs
B or jump drives.
of the student
reports
and media for the
Contractor
Deliverable
initial reporting

Department after
all reports
and media are
Deliverable
Description

jump drives.
(2) Provide proof
of delivery
Evidence
of Completion
districts.

initial reporting

Due Date(s)
Component A
3-8 ELA
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00

jump drives.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00
987
988
989
990
991
992
993
994
995
996
997


Reference 7.5.5

reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00


Reference 7.5.5

reports.
districts.

2016
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00


Reference 7.5.5

reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
998
999
1000
1001
1002
1003
1004
1005
1006
1007
A
1

of the PerformanceBTasks reports
1008
Attachment
1b - DELIVERABLES
For each test administration,
the student report
will (1) ProvidePRICING
Department email reflecting receipt and
Specifyresponse
Component
A, B, C, D or
E (A-D):
_____A_____
include a students written
to selected
approval
of the
samples of the performance task
performance tasks in each subject that contains the reports.
will be reproduced (2) Provide proof of delivery
C
on a separate page that includes an explanation of following for each shipment: date/time delivered,
Submit proof of delivery to the Department after all proof of successful delivery to all participating
reports and media are delivered to the districts. ITN districts.
Reference 7.5.5
Due Date(s)
Component A
3-8 ELA
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00
1009
1010
1011
1012
1013

reports

Reference 7.5.5

reports.
districts.

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00

reports

Reference 7.5.5

reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00

reports

Reference 7.5.5

reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00

reports

Reference 7.5.5

reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
A
1

reports
B
Specify
Component
A,
B,
C,
D
or
(A-D):proof
_____A_____
performance task. The responses will be reproduced (2) EProvide
of delivery spreadsheet listing the
the tasks and the scoring criteria.
The contractor
who received materials, destination
address,
C
D
Submit proof of
delivery toDescription
the Department after all proof of successful
delivery
to all participating
Deliverable
Evidence
of Completion
Reference 7.5.5
Due Date(s)
Component A
3-8 ELA
(Renewal 3 - 2020)
$0.00

districts.

2015
$0.00
2016
(Renewal 1 - 2017)
$0.00
$0.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00

districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00

districts.

2015
$2,400.00
2016
2017
(Renewal 1 - 2018)
$2,400.00
$2,400.00
$2,400.00
(Renewal 2 - 2019)
$2,400.00
$2,400.00
$0.00
1031
1032
1033
1034
1035

reporting

1036
1037
1038
1039

reporting

1040
$0.00
$0.00
$0.00
1041
1042
1043
1044

late reporting

1045
1046
1047
1048
1049
1050
1051


(Renewal 3 - 2020)
2015
(Renewal 1 - 2017)
(Renewal 3 - 2019)
districts.
$0.00
$0.00
$0.00
$0.00
1052

A
1
1053
1054
1055
B
C
1056
districts.
Due Date(s)
2015
Component A
3-8 ELA
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
(Renewal 3 - 2019)
$0.00

districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00

districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
1057
1058
1059
1060
1061


1062
1063
1064
1065
1066
1067
1068
1069
1071
1072
1073
1074
1076
1077
1078
1079





September 2015
September 2016
$0.00
$0.00
$0.00
$0.00
$0.00



March 2016
$0.00
$0.00
$0.00
$0.00
$0.00



March 2015
March 2017
$4,400.00

A
and Reporting
1 Scoring
Deliverable
Category
1081
1082
1083
1084
F
Component A
3-8 ELA
will submit Description
approved final mock
(1) Mock PreIDEvidence

Label/Roster,
data files, media and
of Completion
Due Date(s)
reports delivered to the Department or posted to the March 2016
March 2017
(2) Provide Department email reflecting receipt and (Renewal 1 - March 2018)



September 2015



November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
1095
1097
1098
1099
1100



March 2015
March 2016
March 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
1101
1103
1104
1105
1106
1107



June 2015
June 2016
June 2017

Performance Tasks

$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
1085
1087
1088
1089
1090
1092
1093
1094
1108
1109
1110
1111
Spring ELA/L
and Mathematics:
Contractor
DeliverableFinal
$4,400.00
$4,400.00
$4,400.00
$4,400.00
$4,400.00
$0.00
$0.00
$0.00
$0.00
$0.00
September 2016

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
A
1

Performance Tasks
B
Attachment
1b - DELIVERABLES
PRICING
Handscoring refers to the processes
necessary
for
(1) Provide
Specify
Component
A, B,on
C, D or
(A-D):
_____A_____
of a students
response
theEfinal
handscoring
report.
that respondents
C
D
E
Due Date(s)
(Renewal 3 - 2019)
F
Component A
3-8 ELA
$0.00
1112
1113
1114
1115
1116
1117

Performance Tasks

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00

2015
$7,062,855.00
$0.00
$0.00
$0.00
$0.00
$0.00
1118
1119
1120
1121
1122
1123

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$5,926,541.00
$6,149,176.00
$5,606,000.00
$5,472,025.00
A
1

B

Attachment
1b - DELIVERABLES
PRICING report.
determining the rating of a students
response
on
the final handscoring
or on the
Component
performance-based
A, B, C, D or E (A-D): _____A_____
D
E
Due Date(s)
(Renewal 3 - 2020)
F
Component A
3-8 ELA
$5,442,603.00
1124
1125
1126
1127
1128

Tasks

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
1129
1130
1131
1132
1133

Performance Tasks
$0.00
$0.00
$0.00
$0.00
1134

A
1
B
Performance Tasks
1135
1136
1137
1138
1139

Performance Tasks
1141
1142
1143
1144
1145
1146
1148
1149
1150
1151
1153
1154
1155
1156
Due Date(s)

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1140
Fall ELA/L Retake: Final Report Mockups: The contractor will post the final approved report
all assessments
Reference 7.5.5

the SFTP.


the SFTP.

Reference 7.5.5
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
June 2015
June 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

January 2016
January 2017

A
1
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
Reference 7.5.5
the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

ELA/Mathematics

ELA/Mathematics
Reference 7.5.4.2
Reference 7.5.4.2

and ELA/Mathematics
Reference 7.5.4.2

Reference 7.5.4.2
E
Due Date(s)
January 2015
January 2016
January 2017
June 2015
June 2016
October 2015
October 2016
January 2015
January 2016
January 2017
April 2015
April 2016
April 2017
December 2015
December 2016
October 2016
October 2017
October 2015
October 2016
October 2017
December 2015
F
Component A
3-8 ELA
$28,226.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$6,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00

A
1
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
Scoring
and Reporting
Deliverable
Category
B
Fall EOC:Contractor
Development
of the history
Deliverable
will create and
posted toofthe
SFTP.
Description
Completion
Reference 7.5.4.2
Reference 7.5.4.2
Reference 7.5.4.2

ELA/Mathematics
Reference 7.5.4.2

Retake
1225
1226
1227
1229
1230
1231
1232
1234
1235
1236
1237

Retake

rescoring.

Reference 7.5.6.5

management system.
log.

manual rescoring.

Reference 7.5.6.5

management system.
log.
E
Due Date(s)
December 2016
March 2016
March 2017
October 2015
October 2016
October 2017
October 2015
October 2016
October 2017
November 2015
November 2016

May 2016
May 2017

October 2015
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2016
$0.00

A
and Reporting
1 Scoring
Deliverable
Category
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1255
1256
1257
1258
1260
1261
1262
1263
1264
1266
1267
1268
Spring ELA/L
Retake:Deliverable
Manual Rescore
Contractor
manual rescoring.
will create aDescription
log to identify students
Reference 7.5.6.5
(1) Manual rescore

log posted
to the online
Evidence
of Completion
management system.
log.


Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.



districts.
1269
1270
1271
1272
1273
1274
1275
E
Due Date(s)
April 2017
April 2015
April 2016
April 2017
October 2015
October 2016
December 2015
F
Component A
3-8 ELA
$0.00
$0.00
$0.00
$0.00
$2,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

July 2015
$0.00
$28,620.00
July 2016
July 2017
$28,620.00
$28,620.00

$28,620.00
$28,620.00
$28,620.00
1276

A
1
1278
Studies
1279
B
1280
1281
1282
1284 Studies
1285
C
Administrations

Studies
1290
1291
1292
1293
1294
Studies
1296
1297
1298
1299
1300
1303 Studies
1304
1305

Plans

Administrations

Specifications
Due Date(s)
Component A
3-8 ELA
May 2014
May 2015
$96,473.00
$2,000.00
May 2016
$0.00
$0.00

$0.00
$0.00

deliverable.
July 2014
July 2015
$39,726.00
$39,726.00
July 2016
$39,726.00
$39,726.00

$39,726.00
$39,726.00

July 2014
July 2015
$6,000.00
$2,000.00
July 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$6,000.00
$2,000.00
1286
1287
1288



August 2014
August 2015
August 2016
December 2014
December 2015
December 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$40,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00

Studies
1306
1307
1308
1309
1310
1311
1312
1313

Studies

1314 Studies
1315
1316
1317
1318
1321
1322
1323
B
Studies
1328
1329
B, C, D or
(A-D):
_____A_____
Preparing a detailedSpecify
plan forComponent
conducting allA,scaling
(1) ESES
Specifications
(secure) posted to SFTP.
C
D


posted to SFTP.
E
Due Date(s)
May 2015
May 2016
May 2017
2016
F
Component A
3-8 ELA
$2,000.00
$200,000.00
$200,000.00
$200,000.00
$200,000.00
$200,000.00
$200,000.00
$3,000.00
$1,000.00

via SFTP.



via SFTP.


posted to SFTP.
December 2015
December 2016
December 2017
$125,905.00
$202,094.00
$202,094.00
$202,094.00
$202,094.00
Studies
1324
1327

Specifications

Technical Reports
2017
$1,000.00
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$1,000.00
$1,000.00
$1,000.00
2016
2017
$0.00
$0.00
$0.00
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00

Studies
1330

Technical Reports
B
Preparing TechnicalSpecify
Reports,Component
as described in
(1) ESubmit
versions and the final versions of
A,section
B, C, D or
(A-D):initial
_____A_____
7.6.3.2. Note that Table of contents listed in 7.6.3.2 Technical Reports (secure) to the Department,
includes the minimum requirements for the report. posted to SFTP.
Collecting Feedback from Cthe Department and
Technical Advisory Committee (TAC), finalizing the approval of the technical reports.
reports after incorporating
the feedback, and
E
Due Date(s)
F
Component A
3-8 ELA
$202,094.00


Reference 7.6.3.3

posted to SFTP.
November 2014



SFTP.
October 2014


May 2015

Studies

Setting Work Plan
$2,000.00

Studies
$161,561.00

Studies

Technical Report
SFTP.

Studies
$20,412.00
$0.00
1332
1333 Studies
1334
Studies
$697,578.00
$0.00
$85,970.00
$0.00
1336
7.6.3.4
1337
1338
1339

7.6.3.4.
Department.
$30,761.00

A
1
Studies
1341
Studies
Due Date(s)
Component A
3-8 ELA
renewal)
SFTP.

Report


August 2015
$18,407.00
$5,000.00



$17,819.00

Reference 7.6.3.6
November 2014
$9,000.00
$5,000.00
1343
Studies
1344
1345
Studies

Report
$0.00
$45,291.00
$0.00
1347
1348 Studies

across devices.
input types.
concordance.
characteristics.
$5,400.00

Scaling, Equating,
A Scoring,
Studies
1349
Studies
B of Specifications Preparing a detailed plan for
C conducting
Delivering
Assessments
and detailed
Contractor
Deliverable
Description
across devices.
input types.
concordance.
characteristics.

D
Evidence email
of Completion

with TAC Committee





November 2015
SFTP.


7.6.3.6.2

August 2016

Assessments

Assessments and those of international assessments
such as PISA and TIMSS, propose method(s) for
linking the scales of assessments, propose
alternative ways for collecting data, explain the

SFTP.
Due Date(s)
Component A
3-8 ELA
$0.00
$29,752.00
$0.00
1351
Studies
1352
1353
Studies
1355
Studies
$900.00
$0.00
November 2015
$9,868.00
$0.00
$900.00

A
1
Scaling,
Equating,Category
Scoring,
Deliverable
Studies
Periodic:Contractor
Development
of a study plan
Deliverable
Assessments
The Contractors
plan will Description
include, review of the
Deliverable
process of establishing the linking relationship(s) in
detail, and provide a comprehensive and detailed
timeline for all tasks and events, as described in
section 7.6.3.9.
(1) The plan forEvidence

conducting
the linking study is
of Completion
SFTP.
November 2015

Studies

Assessments

7.6.3.6.3

Due Date(s)
Component A
3-8 ELA
$0.00
1357
1358
$900.00
$0.00
1359
1361
1362
1363
1364
1365
1367
1368
1369
1370
1371
1373
1374
1375
1376

7.7.2

report

Meetings


previous invoice.

ITN.

April 15, 2014

April 15, 2015
April 15, 2016
$66,198.00
$27,930.00
$27,930.00
$27,930.00
$27,930.00
$27,930.00
$121,879.00
$122,426.00
$122,851.00
$123,260.00
$124,154.00
$124,566.00
$1,139,500.00
$1,276,463.00
$1,287,729.00
$1,102,105.00
$1,122,579.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)

Program Management
A
1
1377
1379
1380
1381
1382
1383
1385
1386
1387
1388
1389
1391
1392
1393
1394
B
Component
B, C, D or
(A-D):support
_____A_____
The work described Specify
in the ITN
constitutes aA,large,
(1) EInclude
for program management staff
complex, challenging program that requires yearoperational for the invoice period.
round activity and close attention to overlapping
C
D
tasks for different years and the impact on project appropriate staff indicated for each invoice period.
resources. The contractor is expected to provide a (Quarterly invoices to occur in February, May,
ITN.
Due Date(s)
Component A
3-8 ELA
(Renewal 3 - 2020)
$1,129,158.00
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$112,459.00
$112,459.00
$112,459.00
$112,459.00
$112,459.00
$112,459.00

Action Item Report

work. ITN Ref 7.7.6

November 2015
November 2016
November 2017
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00

ITN Ref 7.7.6
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$69,917.00
$69,797.00
$69,797.00
$69,797.00
$69,797.00
$69,797.00


1395

A
1
Program Management
B
Ref. 7.7.10
Due Date(s)
one-year renewals]
Component A
3-8 ELA
$43,167.00
1396

A
1
3
4
5
6
7
8
Test Development
Document
Test Development

Specifications
9
10
11
12
13
C
7.3.10
via SFTP.
Due Date(s)
June 1, 2014
June 1, 2015
June 1, 2016
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5
F
Component B
3-8 Math
$38,237.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
14
Test Development
15
16
17
18
19


7.5.4.5

SFTP.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$20,191.00

7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
20
Test Development
21
22
23
24
25

$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
26
Test Development
27
28
29
30
31
$0.00
$0.00
$0.00
$0.00
$0.00
32

A
1
Test Development
33
34
35
36
37
F
Component B
3-8 Math
7.5.4.5
Due Date(s)
SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
38
Test Development
39
40
41
42
43
$0.00
$0.00
$0.00
$0.00
$0.00
44
Test Development
45
46
47
48
49
50
51 Test Development
52
53
54
55
56
Test Development

Specifications
7.3.10



June 1, 2014
June 1, 2015
June 1, 2016
November 1, 2015
$0.00
$0.00
$0.00
$0.00
$0.00
$15,797.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$35,812.00
57

A
1
Test Development
B
58
59
60
61
62
each scoring guide.
Due Date(s)
Reference 7.3.7.5


June 1, 2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Component B
3-8 Math
$31,444.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
63
Test Development

Assessment Reports
64
65
66
67
68
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$67,402.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
69
Test Development
70
71 Test Development
72
73
74
75
76 Test Development
77
78
79
80

$147,745.00

management report
Reference 7.3.4.2

February 1, 2017
management report
$3,845,421.00
$3,926,321.00
$2,026,406.00
$1,756,434.00
$1,474,150.00
Review

$3,924,211.00
$32,066.00
$32,066.00
$32,066.00
$32,066.00

management report
May 1, 2015
May 1, 2016

A
1
81 Test Development
82
83
84
85
86
87 Test Development
88
89
90
91 Test Development
92
93
94
95 Test Development
96
97
98
99
100
102
103
104
105
106
108
109
110
111
112
B


management report.
December 1, 2015
December 1, 2016
$23,304.00
$23,304.00
$23,304.00
$23,304.00


management report.
December 1, 2015
December 1, 2016
$19,359.00
$9,109.00
$9,109.00
$9,109.00


previous invoice.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$111,390.00
$112,882.00
$114,410.00
$117,566.00
$119,720.00
$120,058.00




management report
Due Date(s)
July 15, 2014
July 15, 2015
July 15, 2016
F
Component B
3-8 Math

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$33,668.00
$48,762.00
$48,762.00
$48,762.00
$48,762.00
$48,762.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A
1
Test Administration
B
F
Component B
3-8 Math
the report.
Reference 7.4.2
Due Date(s)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
the report.
ITN Reference 7.4.2

2016
$0.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00

a report.
the report.
ITN Reference 7.4.2

2015
$2,000.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

the report.
ITN Reference 7.4.2

2015
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
the report.
ITN Reference 7.4.2

2015
$0.00
2016
$0.00
113
114
115
116
$0.00
117
Test Administration

118
119
120
121
122
Test Administration
123
124
125
126
127
128
Test Administration
129
130
131
132
133
Test Administration
134
135

Test Administration
A
1
136
137
Specify
Component
A, B, C, D or
E (A-D):
_____B_____
scanned
in and inventoried,
1) Final
missing
materials report.
the report.
B
D
have been
accounted for, and
Deliverable
Description
ITN Reference 7.4.2
Due Date(s)
Component B
3-8 Math
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
the report.
ITN Reference 7.4.2

2015
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00

the report.
ITN Reference 7.4.2

2015
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00
the Specifications.

2015.
$0.00
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2017
$0.00
138
Test Administration
139
140
141
142
143
144
Test Administration
145
146
147
148
149
150
152
153
154
155
156
158
159

the Specifications.
Test Administration
A
1
160
161
162
164
165
166
167
168
169
171
172
173
174
175
177
178
179
180
181
183
184
Attachment
1b - DELIVERABLES
PRICING
and student
the Specifications.
7.4.1 Component A, B, C, D or E (A-D): _____B_____
Specify
B
C
D
the Specifications.

administration.

administration.

administration.
the Specifications.
the Specifications.
the Specifications.
Due Date(s)
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$79,909.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
$81,640.00
$67,932.00
$72,745.00
$75,477.00
$76,384.00
$0.00
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
$0.00
Test Administration

administration.
and student
the Specifications.
Attachment
1b - DELIVERABLES
PRICING
A
1
185
186
187
188
190
191
192
193
194
195
197
198
199
200
201
Test Administration
202
203
204
205
206
207
208
Test Administration
209
210
211
212
213
214
215
Test Administration
216
217
B

administration.

memos, etc.
C
D
the Specifications.


districts.


districts.

districts.

memos.

districts.

memos.

Due Date(s)
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$257,340.00
$305,950.00
$350,882.00
$377,733.00
$389,093.00
$390,647.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$259,914.00
$227,780.00
$219,031.00
$184,386.00
$190,454.00
$190,756.00
$0.00
2015
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Test Administration
A
1
218
219
220
221
222
224
225
226
227
228
Test Administration
229
230
231
232
233
234
Test Administration
235
236
237
238
239
240
241
Test Administration
242
243
244
245
246
247
Test Administration
248
249
250
251
252
253

B including

must be
C
critical
date listed in
Deliverable
Description

quantity and type
of material
Evidence
of Completion
districts.

and accommodations.


and accommodations.




accommodations.


accommodations.

Due Date(s)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$3,232,373.00
$2,356,484.00
$2,140,787.00
$1,285,869.00
$1,326,868.00
$1,331,988.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
254
255
256
257
258
259
260
Test Administration
261
262
263
264
265
266
267
Test Administration

accommodations.



and 7.4.7.4

districts.

and 7.4.7.4

districts.

and 7.4.7.4

districts.
and rosters.
and 7.4.7.4

districts.

274
275
276
277
278
279
Test Administration
280
281
282
283
284
285
286
Test Administration
287
288
accommodations.
268
269
270
271
272
273
Test Administration

Due Date(s)
negotiations.)
Component B
3-8 Math
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

Test Administration
A
1
289
290
291
292
Test Administration
B
C
and rosters.
delivered to
listed in Appendix
B. ITN References
7.5.2, 7.4.7.3,
Deliverable
Description
and 7.4.7.4

quantity and type
of material
Evidence
of Completion
districts.

labels and rosters.

and 7.4.7.4

districts.

and 7.4.7.4

districts.

and 7.4.7.4

districts.


7.4.8



7.4.8

293
294
295
296
297
298
Test Administration

labels and rosters.
299
300
301
302
303
304
305
Test Administration
306
307
308
309
310
311
312
314
315
316
317
318
320
321
322
323

labels and rosters.
Due Date(s)
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00

Test Administration
A
1
324
326
327
328
329
330
331
333
334
335
336
337
339
340
341
342
343
345
346
347
348
349
350
352
353
354
355
356
357
359

Specify
Component
A, B, C, D or
(A-D):a_____B_____
the online
management
(1) EProvide
C
D
E
7.4.8
Due Date(s)
F
Component B
3-8 Math
$0.00


7.4.8


assessment.

7.4.8

November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

February 2016
February 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00



May 2015
May 2016
May 2017
$0.00
$139,096.00
$262,891.00
$262,891.00
$262,891.00
$262,891.00
$262,891.00
$0.00
$0.00

A
Test Administration

assessment platform
met the
requirements of the
Deliverable
Description

or unavailability
of the site that
Evidence
of Completion
Due Date(s)
November 2016
363
365
366
367
368

May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
369
371
372
373
374
375

administration.


May 2015
May 2016
May 2017
$0.00
$3,749,054.00
$4,294,013.00
$5,097,356.00
$5,602,608.00
$5,506,854.00
$5,331,649.00
376
378
379
380
381



November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
382
384
385
386
387



February 2016
February 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00



May 2015
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
1
360
361
362
388
390
391
392
393
394

platform
that meets
Contractor
Deliverable
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
Test Administration
A
1
395
397
398
399
400
401
402
404
405
406
407
408

(1) ProvidePRICING
met theComponent
(orEno)
malfunctions
C, D or
(A-D):
_____B_____
C
D
Evidence
ofdate.
Completion
the current invoice
billing

E
Due Date(s)




November 2014
November 2015
November 2016

contract.

September 2014
August 2015
August 2016
August 2017
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$63,181.00
$3,000.00
$3,000.00
$3,000.00
$3,000.00
$3,000.00
$0.00
409
Test Administration
410
412
413
414
415
417
418
419
420

training.
7.4.9.8
administration.
August 2015
August 2016
November 2016
November 2017
previous year.
$242,481.00
$143,014.00
$143,014.00
$143,014.00
$143,014.00
$143,014.00
$537,928.00
$445,587.00
$456,410.00
$475,277.00
$486,865.00
$490,413.00
421

A
1
423
424
425
426
D
F
Component B
3-8 Math
Due Date(s)
November 2015
November 2016
November 2017
$61,492.00
$54,645.00
$53,700.00
$31,475.00
$32,172.00
$32,195.00
427
Test Administration
428
429
430
431
432
433
Test Administration
434
435
436
437
438
439
Test Administration
440
441
442
443
444
445
446
Test Administration
447
448
449
450
451
452
Test Administration
453
454
455

comment forms.

comment forms.



comment forms.

comment forms.



negotiations.)
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
$0.00
$0.00

Test Administration
A
1
456
457
458
Test Administration
459
460
461
462
463
464
465
Test Administration
466
467
468
469
470
471
472

comment forms. B

comment forms.

comment forms.
Component
B, C, D or
(A-D):Department
_____B_____
all responses,
analysis,A,
and
(1) EProvide
D


Test Administration

Trial Test

7.4.8 and 7.4.9.5

Test Administration


Reference 7.3.11.2

Due Date(s)
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
August 1, 2014
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$58,275.00
473
474
475
476
477
478
479
Test Administration
480
481
482
483
484
485


Reference 7.3.11.2


negotiations.)
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
486
487
488
489
490
491
492
Test Administration
493
494
495
496
497
498
Test Administration
499
500
501
502
503
504
Test Administration
505
506
507
508
509
510
511
Test Administration
512
513
514
515
516
517
518
Test Administration
519
B

versions)
C
Reference 7.3.11.2

Reference 7.3.11.2
versions)
Reference 7.3.11.2

versions)

Reference 7.3.11.2

versions)
Reference 7.3.11.2


Reference 7.4.1.
D
Due Date(s)
negotiations.)
Component B
3-8 Math
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$132,091.00
$78,071.00
$78,071.00
$78,071.00
$78,071.00
$78,071.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00





Test Administration
A
1
520
521
522
523

used and
B

the contractor will
C
scoring,
and storing, as
Deliverable
Description
Reference 7.4.1.

E
quantity and type
of material
received, and the
Evidence
of Completion
Due Date(s)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)

Reference 7.4.1.


negotiations.)
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
testing
Reference 7.4.1.


negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
524
Test Administration

525
526
527
528
529
530
Test Administration
531
532
533
534
535
536
537
Test Administration
538
539
540
541
542
543
Test Administration
Reference 7.4.1.

544
545
546
547
548
549
Test Administration
550
551
552
553
554


Reference 7.4.1.

Reference 7.4.1.
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
$684,116.00
$463,790.00
$543,383.00
$294,080.00
$305,700.00
$309,332.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
Test Administration
A
1
555
556
Test Administration
557
558
559
560
561
562
563
565
566

B
572
573
574
576
577
578
579
581
582
583
584
585
587
588
589
590
592
593
Reference 7.4.1.

567
568
569
571

1b - DELIVERABLES
PRICING
in
the following
scoring,Component
and storing, as
A, B, C, D or
quantity
E (A-D):
and_____B_____
C
D
Reference 7.4.1.

ITN Reference 7.5.1
Due Date(s)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016

management system.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
April 2014
April 2015
April 2016

management system.



management system.



management system.

management system.
review.

management system.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$0.00
$0.00
$0.00

January 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
January 2017
January 2015
January 2016
January 2017

$0.00
$0.00
$0.00
September 2015
September 2016

review.
F
Component B
3-8 Math

September 2015
September 2016
October 2015
October 2016
$0.00
$2,500.00
$1,250.00
$1,250.00
$1,250.00
$1,250.00
$1,250.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
594
595
597
598
599
600
601
603
604
605
606
607
609
610
611
612
614
615
616
617
619
620
621
622
623

B
Frequency Reports
and theDescription
reports of issues
Deliverable

management system.
Evidence email
of Completion
E
Due Date(s)

review.


management system.

January 2015
January 2016
January 2017



management system.

June 2015
June 2016
June 2017

in .txt format


Protocol (SFTP).

file in .txt format

Protocol (SFTP).
January 2016


Protocol (SFTP).
January 2015
January 2016
January 2017
format

Protocol (SFTP).
September 2015

September 2015
September 2016
January 2017
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00

A
and Reporting
1 Scoring
Deliverable
Category
625
626
627
628
630
631
632
633
635
636
637
638
639
641
642
648
649
650
F
Component B
3-8 Math
Fall EOC:Contractor
Development
will work with
the Department to
Deliverable
Description
format

file layoutof
posted
to the online
Completion
Protocol (SFTP).
Due Date(s)
September 2016

.txt format


Protocol (SFTP).
October 2015
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00

.txt format


Protocol (SFTP).
January 2015
January 2016
January 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

.txt format


Protocol (SFTP).
June 2015
June 2016
June 2017
$0.00
$0.00
$0.00
$0.00

$0.00
$0.00
October 2015
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00
643
644
645
647

7.5.4.4
$0.00
$0.00
$0.00
$0.00

A
1
652
653
654
655
657
658
659
660
661
663
664
665
666
668
669
670
671
673
674
675
676
677
679
680
681
682
683
685
686
687
7.5.4.4
Due Date(s)
Component B
3-8 Math
May 2016
May 2017

transcription logs
7.5.4.4
May 2015
May 2016
May 2017

7.5.4.4
7.5.4.4
7.5.4.4

7.5.4.4

7.5.6.2, and 7.5.6.4


October 2015
October 2016
December 2015
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

May 2015
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

October 2015
October 2016

A
1
B extract data

and CBT extract
C
7.5.6.2, and 7.5.6.4

E
Due Date(s)
688
690
691
692


7.5.6.2, and 7.5.6.4

March 2016
693
695
696
697
698

data files

7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017


699
701
702
703

March 2017
October 2015
October 2016
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,658,766.00
$1,512,186.00
$217,182.00
$218,756.00
$227,573.00
$230,294.00
$0.00
$0.00
$0.00
$0.00

A
1

data files:
B

and CBT extract
C
7.5.6.2, and 7.5.6.4

E
Due Date(s)
F
Component B
3-8 Math
$0.00
704
706
707
708


7.5.6.2, and 7.5.6.4

December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
709
711
712
713
714


7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00



July 2015
July 2017
715
717
718
$0.00
$0.00
$0.00

A
Summer EOC: Early B

Development
of the Deliverable
file layouts,
Contractor
719
720
721
723
724
725
726
728
729
730
731
733
734
735
736
737
739
740
741
742
744
745
746
747
749
750
751
752
753

C the Department to
administration,
the contractor
Deliverable
Description
7.5.6.2, and 7.5.6.4

D early/full state
E
management system
andofSFTP.
Evidence
Completion
Due Date(s)
score data files
the data files.


the data files.

data files

the data files.

the data files.
data files
the data files.
data files
the data files.
$0.00
$0.00
$0.00
October 2015
October 2016
$0.00
$0.00
$0.00

April 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$64,841.00
$37,420.00
$28,074.00
$18,115.00
April 2017
April 2015
April 2016
April 2017

data files
F
Component B
3-8 Math

October 2015
October 2016
December 2015
$18,115.00
$18,115.00
$0.00
$0.00
$0.00
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
755
756
757
758
759
761
762
763
764
766
767
768
769
771
772
773
774
775
777
778
779
780
782
783
784
score data files
the data files.


Reference 7.5.5


DSR data files.

and DSR data files.



Reference 7.5.5


Reference 7.5.5


Reference 7.5.5


Reference 7.5.5

E
Due Date(s)
July 2015
July 2016
July 2017
October 2015
October 2016

April 2016
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2017
October 2015
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00
December 2015
$0.00
$0.00
$0.00
$0.00
December 2016
A
1
785
787
788
789


790
791
793
794
795
796
797
799
800
801
802
804
805



806
807
809
810
811
812

DAR data files.


Deliverable
testedDescription
Reference 7.5.5


Reference 7.5.5


Reference 7.5.5







E
Due Date(s)
F
Component B
3-8 Math
$0.00
April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00

$0.00
$0.00
October 2015
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
A
1
813
815
816
817
818
820
821

B of the SAR and
DAR data files.
838
839
$1,000.00

October 2015








7.5.5.2 and 7.5.6.6
November 2016
$0.00
$0.00
$0.00
$0.00


833
834
835
837
Due Date(s)
F
Component B
3-8 Math

828
829
831
832

and DAR data files.
822
823
825
826
827
C to develop the
scores and codes
Deliverable

data files.
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00
December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
840
842
843

B versions of DAR
data files.

data files.
844
845
847
848

849
850
851
853
854
855
856
858
859
F
Component B
3-8 Math
$0.00

7.5.5.2 and 7.5.6.6
June 2017
$0.00
$0.00
$0.00
$0.00
$0.00

7.5.5.2 and 7.5.6.6
June 2015
June 2016
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
7.5.5.2 and 7.5.6.6
November 2016
$0.00
$0.00
$0.00

$0.00
$0.00

data files.

7.5.5.2 and 7.5.6.6
February 2016
$0.00
$0.00
$0.00
$0.00
$0.00

June 2016
860
861
863
Specify
B, C, D or
(A-D):
withComponent
Department A,
Scoring
(1) EFinal
late_____B_____
E
files. The SSR
Description
Due Date(s)
7.5.5.2 and 7.5.6.6

data files.
$0.00
$0.00

A
1
864
865
B
DSRs dataContractor
files: Development
of
Deliverable
data files.
866
and SFTP.
Evidencesystem
of Completion
Due Date(s)
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
and 7.5.6.7
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00

assessments.
and 7.5.6.7
October 2015
October 2016
October 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

December 2015
and 7.5.6.7

7.5.5.2 and 7.5.6.6
873
875
876
877

and 7.5.6.7
889
891
892
$0.00
$0.00
$0.00
$0.00

data files.
883
885
886
887
888
F
Component B
3-8 Math
867
869
870
871
872
878
880
881
882

Staff to develop
Description
7.5.5.2 and 7.5.6.6
$0.00
$0.00
$0.00

A
893
894
896
897
898
899
901
902
903
904
905
907
908
909
910
911
913
914
915

B data files:
files for all
assessments.
Contractor
Deliverable
918
919
920
921
922
923
924
F
Component B
3-8 Math
$0.00

and 7.5.6.7
March 2015
March 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

and 7.5.6.7
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

and 7.5.6.7
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00


September 2015
September 2016
September 2017
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

reporting

2015
$1,000.00
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2017
reporting
$0.00
$0.00
$0.00
$0.00
$0.00
916
917

E
of all assessments.
data files.
Description
of Completion
Due Date(s)
and 7.5.6.7

$0.00

A
1
925
926

for districts and
C
and PDFsContractor
of reports for
the initial
Deliverable
reporting

D
approval of files
and reports.
Evidence
of Completion

initial reporting


Due Date(s)
Component B
3-8 Math
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
927
928
929
930
931
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955


$1,000.00
$1,000.00
$1,000.00
$1,000.00
(Renewal 3 - 2020)
2015
$1,000.00
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
$0.00
$0.00
$0.00
$0.00
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
956
957

$70,000.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
932
933

2015


jump drives.
districts.
(Renewal 3 - 2020)
2015
2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1

Development
of the Deliverable
student reports and
Contractor

Department after
all reports
and media are
Deliverable
Description
958
959

jump drives.
(2) Provide proof
of delivery
Evidence
of Completion
districts.
Due Date(s)
Component B
3-8 Math
(Renewal 1 - 2017)
$0.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00

jump drives.
districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00

jump drives.
districts.

2015
$969,849.00
2016
2017
(Renewal 1 - 2018)
$888,930.00
$896,784.00
$989,759.00
(Renewal 2 - 2019)
$989,759.00
(Renewal 3 - 2020)
$989,759.00
960
961
962
963


964
$0.00
$0.00
$0.00
965
966
967
968


969
970
971
972
973
974
975

initial reporting


jump drives.
districts.

2015
2016
(Renewal 1 - 2017)

jump drives.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
(Renewal 3 - 2019)
$0.00

jump drives.
districts.

2015
$0.00
2016
$0.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
976
977
978
979

initial reporting

980
981
982
983

initial reporting


A
1
984
985
986

Reports and CD/DVDs
B or jump drives.
of the student
reports
and media for the
Contractor
Deliverable
initial reporting

Department after
all reports
and media are
Deliverable
Description

jump drives.
(2) Provide proof
of delivery
Evidence
of Completion
districts.

initial reporting

Due Date(s)
Component B
3-8 Math
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00

jump drives.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00
987
988
989
990
991
992
993
994
995
996
997


Reference 7.5.5

reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00


Reference 7.5.5

reports.
districts.

2016
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00


Reference 7.5.5

reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
998
999
1000
1001
1002
1003
1004
1005
1006
1007
A
1

1008
Attachment
1b - DELIVERABLES
the student report
Specifyresponse
Component
A, B, C, D or
E (A-D):
_____B_____
to selected
approval
of the
C
Reference 7.5.5
Due Date(s)
Component B
3-8 Math
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00
1009
1010
1011
1012
1013

reports

Reference 7.5.5

reports.
districts.

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00

reports

Reference 7.5.5

reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00

reports

Reference 7.5.5

reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00

reports

Reference 7.5.5

reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
A
1

reports
B
Specify
Component
A,
B,
C,
D
or
(A-D):proof
_____B_____
The contractor
address,
C
D
Submit proof of
delivery
Deliverable
Evidence
of Completion
Reference 7.5.5
Due Date(s)
Component B
3-8 Math
(Renewal 3 - 2020)
$0.00

districts.

2015
$0.00
2016
(Renewal 1 - 2017)
$0.00
$0.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00

districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00

districts.

2015
$2,400.00
2016
2017
(Renewal 1 - 2018)
$2,400.00
$2,400.00
$2,400.00
(Renewal 2 - 2019)
$2,400.00
$2,400.00
$0.00
1031
1032
1033
1034
1035

reporting

1036
1037
1038
1039

reporting

1040
$0.00
$0.00
$0.00
1041
1042
1043
1044

late reporting

1045
1046
1047
1048
1049
1050
1051


(Renewal 3 - 2020)
2015
(Renewal 1 - 2017)
(Renewal 3 - 2019)
districts.
$0.00
$0.00
$0.00
$0.00
1052

A
1
1053
1054
1055
B
C
1056
districts.
Due Date(s)
2015
Component B
3-8 Math
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
(Renewal 3 - 2019)
$0.00

districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00

districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
1057
1058
1059
1060
1061


1062
1063
1064
1065
1066
1067
1068
1069
1071
1072
1073
1074
1076
1077
1078
1079





September 2015
September 2016
$0.00
$0.00
$0.00
$0.00
$0.00



March 2016
$0.00
$0.00
$0.00
$0.00
$0.00



March 2015
March 2017
$4,400.00

A
and Reporting
1 Scoring
Deliverable
Category
1081
1082
1083
1084
F
Component B
3-8 Math
approved final mock

Label/Roster,
of Completion
Due Date(s)
March 2017



September 2015



November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
1095
1097
1098
1099
1100



March 2015
March 2016
March 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
1101
1103
1104
1105
1106
1107



June 2015
June 2016
June 2017

Performance Tasks

$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
1085
1087
1088
1089
1090
1092
1093
1094
1108
1109
1110
1111
Spring ELA/L
and Mathematics:
Contractor
DeliverableFinal
$4,400.00
$4,400.00
$4,400.00
$4,400.00
$4,400.00
$0.00
$0.00
$0.00
$0.00
$0.00
September 2016

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
A
1

Performance Tasks
B
Attachment
1b - DELIVERABLES
PRICING
necessary
for
(1) Provide
Specify
Component
A, B,onC, D or
(A-D):
_____B_____
of a students
response
theEfinal
handscoring
report.
that respondents
C
D
E
Due Date(s)
(Renewal 3 - 2019)
F
Component B
3-8 Math
$0.00
1112
1113
1114
1115
1116
1117

Performance Tasks

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00

2015
$2,633,087.00
$0.00
$0.00
$0.00
$0.00
$0.00
1118
1119
1120
1121
1122
1123

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$2,151,059.00
$1,038,406.00
$1,109,820.00
$1,151,323.00
A
1

B

Attachment
1b - DELIVERABLES
PRICING report.
response
on
or on theComponent
performance-based
A, B, C, D or E (A-D): _____B_____
D
E
Due Date(s)
(Renewal 3 - 2020)
F
Component B
3-8 Math
$1,096,250.00
1124
1125
1126
1127
1128

Tasks

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
1129
1130
1131
1132
1133

Performance Tasks
$0.00
$0.00
$0.00
$0.00
1134

A
1
B
Performance Tasks
1135
1136
1137
1138
1139

Performance Tasks
1141
1142
1143
1144
1145
1146
1148
1149
1150
1151
1153
1154
1155
1156
Due Date(s)

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1140
all assessments
Reference 7.5.5

the SFTP.


the SFTP.

Reference 7.5.5
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
June 2015
June 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

January 2016
January 2017

A
1
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
Reference 7.5.5
the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

ELA/Mathematics

ELA/Mathematics
Reference 7.5.4.2
Reference 7.5.4.2

and ELA/Mathematics
Reference 7.5.4.2

Reference 7.5.4.2
E
Due Date(s)
January 2015
January 2016
January 2017
June 2015
June 2016
October 2015
October 2016
January 2015
January 2016
January 2017
April 2015
April 2016
April 2017
December 2015
December 2016
October 2016
October 2017
October 2015
October 2016
October 2017
December 2015
F
Component B
3-8 Math
$28,226.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$6,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00

A
1
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
Scoring
and Reporting
Deliverable
Category
B
Fall EOC:Contractor
Development
of the history
Deliverable
will create and
posted toofthe
SFTP.
Description
Completion
Reference 7.5.4.2
Reference 7.5.4.2
Reference 7.5.4.2

ELA/Mathematics
Reference 7.5.4.2

Retake
1225
1226
1227
1229
1230
1231
1232
1234
1235
1236
1237

Retake

rescoring.

Reference 7.5.6.5

management system.
log.

manual rescoring.

Reference 7.5.6.5

management system.
log.
E
Due Date(s)
December 2016
March 2016
March 2017
October 2015
October 2016
October 2017
October 2015
October 2016
October 2017
November 2015
November 2016

May 2016
May 2017

October 2015
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
April 2016
$0.00

A
and Reporting
1 Scoring
Deliverable
Category
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1255
1256
1257
1258
1260
1261
1262
1263
1264
1266
1267
1268
Spring ELA/L
Retake:Deliverable
Manual Rescore
Contractor
manual rescoring.
Reference 7.5.6.5
(1) Manual rescore

log posted
to the online
Evidence
of Completion
management system.
log.


Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.



districts.
1269
1270
1271
1272
1273
1274
1275
E
Due Date(s)
April 2017
April 2015
April 2016
April 2017
October 2015
October 2016
December 2015
F
Component B
3-8 Math
$0.00
$0.00
$0.00
$0.00
$2,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

July 2015
$0.00
$28,620.00
July 2016
July 2017
$28,620.00
$28,620.00

$28,620.00
$28,620.00
$28,620.00
1276

A
1
1278
Studies
1279
B
1280
1281
1282
1284 Studies
1285
C
Administrations

Studies
1290
1291
1292
1293
1294
Studies
1296
1297
1298
1299
1300
1303 Studies
1304
1305

Plans

Administrations

Specifications
Due Date(s)
Component B
3-8 Math
May 2014
May 2015
$96,473.00
$2,000.00
May 2016
$0.00
$0.00

$0.00
$0.00

deliverable.
July 2014
July 2015
$39,726.00
$39,726.00
July 2016
$39,726.00
$39,726.00

$39,726.00
$39,726.00

July 2014
July 2015
$6,000.00
$2,000.00
July 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$6,000.00
$2,000.00
1286
1287
1288



August 2014
August 2015
August 2016
December 2014
December 2015
December 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$40,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00

Studies
1306
1307
1308
1309
1310
1311
1312
1313

Studies

1314 Studies
1315
1316
1317
1318
1321
1322
1323
B
Studies
1328
1329
B, C, D or
(A-D):
_____B_____
plan forComponent
(1) ESES
Specifications
C
D


posted to SFTP.
E
Due Date(s)
May 2015
May 2016
May 2017
2016
F
Component B
3-8 Math
$2,000.00
$200,000.00
$200,000.00
$200,000.00
$200,000.00
$200,000.00
$200,000.00
$3,000.00
$1,000.00

via SFTP.



via SFTP.


posted to SFTP.
December 2015
December 2016
December 2017
$125,905.00
$202,094.00
$202,094.00
$202,094.00
$202,094.00
Studies
1324
1327

Specifications

Technical Reports
2017
$1,000.00
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$1,000.00
$1,000.00
$1,000.00
2016
2017
$0.00
$0.00
$0.00
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00

Studies
1330

Technical Reports
B
Preparing Technical Specify

Reports,Component
as described in
(1) ESubmit
A,section
B, C, D or
(A-D):initial
_____B_____
the feedback, and
E
Due Date(s)
F
Component B
3-8 Math
$202,094.00


Reference 7.6.3.3

posted to SFTP.
November 2014



SFTP.
October 2014


May 2015

Studies

Setting Work Plan
$2,000.00

Studies
$161,561.00

Studies

Technical Report
SFTP.

Studies
$20,412.00
$0.00
1332
1333 Studies
1334
Studies
$697,578.00
$0.00
$85,970.00
$0.00
1336
7.6.3.4
1337
1338
1339

7.6.3.4.
Department.
$30,761.00

A
1
Studies
1341
Studies
Due Date(s)
Component B
3-8 Math
renewal)
SFTP.

Report


August 2015
$18,407.00
$5,000.00



$17,819.00

Reference 7.6.3.6
November 2014
$9,000.00
$5,000.00
1343
Studies
1344
1345
Studies

Report
$0.00
$45,291.00
$0.00
1347
1348 Studies

across devices.
input types.
concordance.
characteristics.
$5,400.00

Scaling, Equating,
A Scoring,
Studies
1349
Studies
C conducting
Delivering
Assessments
and detailed
Contractor
Deliverable
Description
across devices.
input types.
concordance.
characteristics.

D
Evidence email
of Completion

with TAC Committee





November 2015
SFTP.


7.6.3.6.2

August 2016

Assessments


SFTP.
Due Date(s)
Component B
3-8 Math
$0.00
$29,752.00
$0.00
1351
Studies
1352
1353
Studies
1355
Studies
$900.00
$0.00
November 2015
$9,868.00
$0.00
$900.00

A
1
Scaling,
Equating,Category
Scoring,
Deliverable
Studies
Periodic:Contractor
Development
of a study plan
Deliverable
Assessments
The Contractors
Deliverable
section 7.6.3.9.

conducting
of Completion
SFTP.
November 2015

Studies

Assessments

7.6.3.6.3

Due Date(s)
Component B
3-8 Math
$0.00
1357
1358
$900.00
$0.00
1359
1361
1362
1363
1364
1365
1367
1368
1369
1370
1371
1373
1374
1375
1376

7.7.2

report

Meetings


previous invoice.

ITN.

April 15, 2014

April 15, 2015
April 15, 2016
$66,198.00
$27,930.00
$27,930.00
$27,930.00
$27,930.00
$27,930.00
$121,879.00
$122,426.00
$122,851.00
$123,260.00
$124,154.00
$124,566.00
$1,139,500.00
$1,276,463.00
$1,287,729.00
$1,102,105.00
$1,122,579.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)

Program Management
A
1
1377
1379
1380
1381
1382
1383
1385
1386
1387
1388
1389
1391
1392
1393
1394
B
Component
B, C, D or
(A-D):support
_____B_____
in the ITN
(1) EInclude
C
D
ITN.
Due Date(s)
Component B
3-8 Math
(Renewal 3 - 2020)
$1,129,158.00
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$112,459.00
$112,459.00
$112,459.00
$112,459.00
$112,459.00
$112,459.00

Action Item Report

work. ITN Ref 7.7.6

November 2015
November 2016
November 2017
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00

ITN Ref 7.7.6
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$69,917.00
$69,797.00
$69,797.00
$69,797.00
$69,797.00
$69,797.00


1395

A
1
Program Management
B
Ref. 7.7.10
Due Date(s)
one-year renewals]
Component B
3-8 Math
$43,167.00
1396

A
1
3
4
5
6
7
8
Test Development
Document
Test Development

Specifications
9
10
11
12
13
C
7.3.10
via SFTP.
Due Date(s)
June 1, 2014
June 1, 2015
June 1, 2016
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5
F
Component C
9-11 ELA
$38,237.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
14
Test Development
15
16
17
18
19


7.5.4.5

SFTP.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$20,191.00

7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
20
Test Development
21
22
23
24
25

$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
26
Test Development
27
28
29
30
31
$0.00
$0.00
$0.00
$0.00
$0.00
32

A
1
Test Development
33
34
35
36
37
F
Component C
9-11 ELA
7.5.4.5
Due Date(s)
SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
38
Test Development
39
40
41
42
43
$0.00
$0.00
$0.00
$0.00
$0.00
44
Test Development
45
46
47
48
49
50
51 Test Development
52
53
54
55
56
Test Development

Specifications
7.3.10



June 1, 2014
June 1, 2015
June 1, 2016
November 1, 2015
$0.00
$0.00
$0.00
$0.00
$0.00
$15,797.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$35,812.00
57

A
1
Test Development
B
58
59
60
61
62
each scoring guide.
Due Date(s)
Reference 7.3.7.5


June 1, 2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Component C
9-11 ELA
$15,722.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
63
Test Development

Assessment Reports
64
65
66
67
68
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$67,402.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
69
Test Development
70
71 Test Development
72
73
74
75
76 Test Development
77
78
79
80

$73,872.00

management report
Reference 7.3.4.2

February 1, 2017
management report
$1,985,716.00
$2,026,166.00
$1,076,209.00
$941,222.00
$800,081.00
Review

$2,025,111.00
$16,033.00
$16,033.00
$16,033.00
$16,033.00

management report
May 1, 2015
May 1, 2016

A
1
81 Test Development
82
83
84
85
86
87 Test Development
88
89
90
91 Test Development
92
93
94
95 Test Development
96
97
98
99
100
102
103
104
105
106
108
109
110
111
112
B


management report.
December 1, 2015
December 1, 2016
$11,652.00
$11,652.00
$11,652.00
$11,652.00


management report.
December 1, 2015
December 1, 2016
$9,679.00
$4,555.00
$4,555.00
$4,555.00


previous invoice.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$55,695.00
$56,441.00
$57,205.00
$58,783.00
$59,860.00
$60,029.00




management report
Due Date(s)
July 15, 2014
July 15, 2015
July 15, 2016
F
Component C
9-11 ELA

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$16,834.00
$24,381.00
$24,381.00
$24,381.00
$24,381.00
$24,381.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A
1
Test Administration
B
F
Component C
9-11 ELA
the report.
Reference 7.4.2
Due Date(s)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
the report.
ITN Reference 7.4.2

2016
$1,000.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00

a report.
the report.
ITN Reference 7.4.2

2015
$2,000.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

the report.
ITN Reference 7.4.2

2015
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
the report.
ITN Reference 7.4.2

2015
$0.00
2016
$0.00
113
114
115
116
$1,000.00
117
Test Administration

118
119
120
121
122
Test Administration
123
124
125
126
127
128
Test Administration
129
130
131
132
133
Test Administration
134
135

Test Administration
A
1
136
137
Specify
Component
A, B, C, D or
E (A-D):
_____C_____
scanned
in and inventoried,
1) Final
missing
materials report.
the report.
B
D
have been
accounted for, and
Deliverable
Description
ITN Reference 7.4.2
Due Date(s)
Component C
9-11 ELA
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
the report.
ITN Reference 7.4.2

2015
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00

the report.
ITN Reference 7.4.2

2015
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00
the Specifications.

2015.
$0.00
$98,074.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2016
$81,868.00
$87,228.00
$91,416.00
$92,033.00
$0.00
$1,000.00
138
Test Administration
139
140
141
142
143
144
Test Administration
145
146
147
148
149
150
152
153
154
155
156
158
159

the Specifications.
2017
$1,000.00
Test Administration
A
1
160
161
162
164
165
166
167
168
169
171
172
173
174
175
177
178
179
180
181
183
184
Attachment
1b - DELIVERABLES
PRICING
and student
the Specifications.
7.4.1 Component A, B, C, D or E (A-D): _____C_____
Specify
B
C
D
the Specifications.

administration.

administration.

administration.
the Specifications.
the Specifications.
the Specifications.
Due Date(s)
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
Component C
9-11 ELA
$1,000.00
$1,000.00
$1,000.00
$0.00
$39,955.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
$40,820.00
$33,966.00
$36,372.00
$37,738.00
$38,192.00
$0.00
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
2015.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
$0.00
Test Administration

administration.
and student
the Specifications.
Attachment
1b - DELIVERABLES
PRICING
A
1
185
186
187
188
190
191
192
193
194
195
197
198
199
200
201
Test Administration
202
203
204
205
206
207
208
Test Administration
209
210
211
212
213
214
215
Test Administration
216
217
B

administration.

memos, etc.
C
D
the Specifications.


districts.


districts.

districts.

memos.

districts.

memos.

Due Date(s)
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
2015.
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$188,019.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$257,340.00
$305,950.00
$350,882.00
$377,733.00
$389,093.00
$390,647.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$259,914.00
$227,780.00
$219,031.00
$184,386.00
$190,454.00
$190,756.00
$0.00
2015
$192,792.00
$196,825.00
$203,368.00
$202,985.00
$0.00
$0.00

Test Administration
A
1
218
219
220
221
222
224
225
226
227
228
Test Administration
229
230
231
232
233
234
Test Administration
235
236
237
238
239
240
241
Test Administration
242
243
244
245
246
247
Test Administration
248
249
250
251
252
253

B including

must be
C
critical
date listed in
Deliverable
Description

quantity and type
of material
Evidence
of Completion
districts.

and accommodations.


and accommodations.




accommodations.


accommodations.

Due Date(s)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015.
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$68,567.00
$71,314.00
$75,323.00
$77,282.00
$77,969.00
$2,000.00
$80,380.00
$81,889.00
$86,585.00
$89,717.00
$89,655.00
$0.00
$511,157.00
$527,368.00
$544,683.00
$578,592.00
$597,470.00
$600,208.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
Test Administration
254
255
256
257
258
259
260
Test Administration
261
262
263
264
265
266
267
Test Administration

accommodations.



and 7.4.7.4

districts.

and 7.4.7.4

districts.

and 7.4.7.4

districts.
and rosters.
and 7.4.7.4

districts.

274
275
276
277
278
279
Test Administration
280
281
282
283
284
285
286
Test Administration
287
288
accommodations.
268
269
270
271
272
273
Test Administration

Due Date(s)
negotiations.)
Component C
9-11 ELA
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00

Test Administration
A
1
289
290
291
292
Test Administration
B
C
and rosters.
delivered to
listed in Appendix
B. ITN References
7.5.2, 7.4.7.3,
Deliverable
Description
and 7.4.7.4

quantity and type
of material
Evidence
of Completion
districts.

labels and rosters.

and 7.4.7.4

districts.

and 7.4.7.4

districts.

and 7.4.7.4

districts.


7.4.8



7.4.8

293
294
295
296
297
298
Test Administration

labels and rosters.
299
300
301
302
303
304
305
Test Administration
306
307
308
309
310
311
312
314
315
316
317
318
320
321
322
323

labels and rosters.
Due Date(s)
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00

Test Administration
A
1
324
326
327
328
329
330
331
333
334
335
336
337
339
340
341
342
343
345
346
347
348
349
350
352
353
354
355
356
357
359

Specify
Component
A, B, C, D or
(A-D):a_____C_____
the online
management
(1) EProvide
C
D
E
7.4.8
Due Date(s)
F
Component C
9-11 ELA
$0.00


7.4.8


assessment.

7.4.8

November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

February 2016
February 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

assessment.

7.4.8

August 2015
August 2016
August 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00



May 2015
May 2016
May 2017
$0.00
$139,096.00
$262,891.00
$262,891.00
$262,891.00
$262,891.00
$262,891.00
$0.00
$10,000.00

A
Test Administration

assessment platform
met the
requirements of the
Deliverable
Description

or unavailability
of the site that
Evidence
of Completion
Due Date(s)
November 2016
363
365
366
367
368

May 2016
May 2017
$0.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
$10,000.00
369
371
372
373
374
375

administration.


May 2015
May 2016
May 2017
$0.00
$1,874,527.00
$2,147,007.00
$2,548,678.00
$2,801,304.00
$2,753,427.00
$2,665,824.00
376
378
379
380
381



November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
382
384
385
386
387



February 2016
February 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00



May 2015
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
1
360
361
362
388
390
391
392
393
394

platform
that meets
Contractor
Deliverable
Component C
9-11 ELA
$10,000.00
$10,000.00
$10,000.00
$10,000.00
Test Administration
A
1
395
397
398
399
400
401
402
404
405
406
407
408

(1) ProvidePRICING
met theComponent
(or Eno)
malfunctions
C, D or
(A-D):
_____C_____
C
D
Evidence
ofdate.
Completion
the current invoice
billing

E
Due Date(s)




November 2014
November 2015
November 2016

contract.

September 2014
August 2015
August 2016
August 2017
F
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$31,591.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
409
Test Administration
410
412
413
414
415
417
418
419
420

training.
7.4.9.8
administration.
August 2015
August 2016
November 2016
November 2017
previous year.
$207,841.00
$122,583.00
$122,583.00
$122,583.00
$122,583.00
$122,583.00
$268,964.00
$222,793.00
$228,205.00
$237,638.00
$243,432.00
$245,206.00
421

A
1
423
424
425
426
D
F
Component C
9-11 ELA
Due Date(s)
November 2015
November 2016
November 2017
$9,724.00
$12,229.00
$13,663.00
$14,163.00
$14,487.00
$14,507.00
427
Test Administration
428
429
430
431
432
433
Test Administration
434
435
436
437
438
439
Test Administration
440
441
442
443
444
445
446
Test Administration
447
448
449
450
451
452
Test Administration
453
454
455

comment forms.

comment forms.



comment forms.

comment forms.



negotiations.)
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$2,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
$0.00
$0.00

Test Administration
A
1
456
457
458
Test Administration
459
460
461
462
463
464
465
Test Administration
466
467
468
469
470
471
472

comment forms. B

comment forms.

comment forms.
Component
B, C, D or
(A-D):Department
_____C_____
all responses,
analysis,A,
and
(1) EProvide
D


Test Administration

Trial Test

7.4.8 and 7.4.9.5

Test Administration


Reference 7.3.11.2

Due Date(s)
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
August 1, 2014
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$58,275.00
473
474
475
476
477
478
479
Test Administration
480
481
482
483
484
485


Reference 7.3.11.2


negotiations.)
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$12,735.00
$12,735.00
$12,735.00
$12,735.00
$12,735.00
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$13,045.00
$13,045.00
$13,045.00
$13,045.00
$13,045.00

A
1
Test Administration
486
487
488
489
490
491
492
Test Administration
493
494
495
496
497
498
Test Administration
499
500
501
502
503
504
Test Administration
505
506
507
508
509
510
511
Test Administration
512
513
514
515
516
517
518
Test Administration
519
B

versions)
C
Reference 7.3.11.2

Reference 7.3.11.2
versions)
Reference 7.3.11.2

versions)

Reference 7.3.11.2

versions)
Reference 7.3.11.2


Reference 7.4.1.
D
Due Date(s)
negotiations.)
Component C
9-11 ELA
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$66,045.00
$39,035.00
$39,035.00
$39,035.00
$39,035.00
$39,035.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00





Test Administration
A
1
520
521
522
523

used and
B

the contractor will
C
scoring,
and storing, as
Deliverable
Description
Reference 7.4.1.

E
quantity and type
of material
received, and the
Evidence
of Completion
Due Date(s)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)

Reference 7.4.1.


negotiations.)
testing
Reference 7.4.1.


negotiations.)
524
Test Administration

525
526
527
528
529
530
Test Administration
531
532
533
534
535
536
537
Test Administration
538
539
540
541
542
543
Test Administration
Reference 7.4.1.

544
545
546
547
548
549
Test Administration
550
551
552
553
554


Reference 7.4.1.

Reference 7.4.1.
F
Component C
9-11 ELA
$17,315.00
$17,315.00
$18,801.00
$18,801.00
$19,882.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
negotiations.)
2015
2016
2017
(Renewal 1 - 2018)
$0.00
$13,344.00
$13,344.00
$14,226.00
$15,191.00
$15,191.00
$0.00
$108,184.00
$103,794.00
$138,253.00
$132,325.00
$137,652.00
$139,388.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
Test Administration
A
1
555
556
Test Administration
557
558
559
560
561
562
563
565
566

B
572
573
574
576
577
578
579
581
582
583
584
585
587
588
589
590
592
593
Reference 7.4.1.

567
568
569
571

1b - DELIVERABLES
PRICING
in
the following
scoring,Component
and storing, as
A, B, C, D or
quantity
E (A-D):
and_____C_____
C
D
Reference 7.4.1.

ITN Reference 7.5.1
Due Date(s)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
negotiations.)
2015
2016

management system.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
April 2014
April 2015
April 2016

management system.



management system.



management system.

management system.
review.

management system.
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$27,459.00
$1,000.00
$1,000.00
$1,000.00

January 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
January 2017
January 2015
January 2016
January 2017

$0.00
$0.00
$0.00
September 2015
September 2016

review.
F
Component C
9-11 ELA

September 2015
September 2016
October 2015
October 2016
$1,000.00
$1,000.00
$500.00
$500.00
$500.00
$500.00
$500.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
594
595
597
598
599
600
601
603
604
605
606
607
609
610
611
612
614
615
616
617
619
620
621
622
623

B
Frequency Reports
and theDescription
reports of issues
Deliverable

management system.
Evidence email
of Completion
E
Due Date(s)

review.


management system.

January 2015
January 2016
January 2017



management system.

June 2015
June 2016
June 2017

in .txt format


Protocol (SFTP).

file in .txt format

Protocol (SFTP).
January 2016


Protocol (SFTP).
January 2015
January 2016
January 2017
format

Protocol (SFTP).
September 2015

September 2015
September 2016
January 2017
F
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00

A
and Reporting
1 Scoring
Deliverable
Category
625
626
627
628
630
631
632
633
635
636
637
638
639
641
642
648
649
650
F
Component C
9-11 ELA
Fall EOC:Contractor
Development
will work with
the Department to
Deliverable
Description
format

file layoutof
posted
to the online
Completion
Protocol (SFTP).
Due Date(s)
September 2016

.txt format


Protocol (SFTP).
October 2015
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00

.txt format


Protocol (SFTP).
January 2015
January 2016
January 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

.txt format


Protocol (SFTP).
June 2015
June 2016
June 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
643
644
645
647

7.5.4.4
$0.00
$0.00
$0.00
$0.00
October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00

$1,000.00
$1,000.00

A
1
652
653
654
655
657
658
659
660
661
663
664
665
666
668
669
670
671
673
674
675
676
677
679
680
681
682
683
685
686
687
7.5.4.4
Due Date(s)
Component C
9-11 ELA
May 2016
May 2017

transcription logs
7.5.4.4
May 2015
May 2016
May 2017

7.5.4.4
7.5.4.4
7.5.4.4

7.5.4.4

7.5.6.2, and 7.5.6.4


October 2015
October 2016
December 2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
$0.00
$0.00
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

May 2015
May 2016
May 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

July 2015
July 2016
July 2017
October 2015
October 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$101,604.00
$87,016.00
$93,459.00
$97,415.00

A
1
B extract data

and CBT extract
C
7.5.6.2, and 7.5.6.4

E
Due Date(s)
688
690
691
692


7.5.6.2, and 7.5.6.4

March 2016
693
695
696
697
698

data files

7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017


699
701
702
703

March 2017
October 2015
October 2016
F
Component C
9-11 ELA
$97,415.00
$9,000.00
$9,000.00
$10,000.00
$10,000.00
$10,000.00
$181,870.00
$187,806.00
$175,622.00
$176,894.00
$184,024.00
$186,225.00
$0.00
$0.00
$0.00
$0.00

A
1

data files:
B

and CBT extract
C
7.5.6.2, and 7.5.6.4

E
Due Date(s)
F
Component C
9-11 ELA
$0.00
704
706
707
708


7.5.6.2, and 7.5.6.4

December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
709
711
712
713
714


7.5.6.2, and 7.5.6.4

March 2015
March 2016
March 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00



July 2015
July 2017
715
717
718
$0.00
$0.00
$0.00

A
Summer EOC: Early B

Development
of the Deliverable
file layouts,
Contractor
719
720
721
723
724
725
726
728
729
730
731
733
734
735
736
737
739
740
741
742
744
745
746
747
749
750
751
752
753

C the Department to
administration,
the contractor
Deliverable
Description
7.5.6.2, and 7.5.6.4

D early/full state
E
management system
andofSFTP.
Evidence
Completion
Due Date(s)
score data files
the data files.


the data files.

data files

the data files.


the data files.
data files
the data files.
data files
the data files.
$0.00
$0.00
$0.00
October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00

April 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$32,421.00
$18,710.00
$14,037.00
$9,057.00
April 2017
April 2015
April 2016
April 2017

data files
F
Component C
9-11 ELA

October 2015
October 2016
December 2015
$9,057.00
$9,057.00
$0.00
$0.00
$0.00
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
755
756
757
758
759
761
762
763
764
766
767
768
769
771
772
773
774
775
777
778
779
780
782
783
784
score data files
the data files.


Reference 7.5.5


DSR data files.

and DSR data files.



Reference 7.5.5


Reference 7.5.5


Reference 7.5.5


Reference 7.5.5

E
Due Date(s)
July 2015
July 2016
July 2017
October 2015
October 2016

April 2016
F
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2017
October 2015
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00
December 2015
$0.00
$0.00
$0.00
$0.00
December 2016
A
1
785
787
788
789


790
791
793
794
795
796
797
799
800
801
802
804
805



806
807
809
810
811
812

DAR data files.


Deliverable
testedDescription
Reference 7.5.5


Reference 7.5.5


Reference 7.5.5







E
Due Date(s)
F
Component C
9-11 ELA
$0.00
April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
October 2015
October 2016
$1,000.00
$1,000.00
$1,000.00

$1,000.00
$1,000.00
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2015
April 2016
April 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
A
1
813
815
816
817
818
820
821

B of the SAR and
DAR data files.
838
839
$1,000.00

October 2015








7.5.5.2 and 7.5.6.6
November 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00


833
834
835
837
Due Date(s)
F
Component C
9-11 ELA

828
829
831
832

and DAR data files.
822
823
825
826
827
C to develop the
scores and codes
Deliverable

data files.
October 2016
$0.00
$0.00
$0.00

$0.00
$0.00
December 2015
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

A
1
840
842
843

B versions of DAR
data files.

data files.
844
845
847
848

849
850
851
853
854
855
856
858
859
F
Component C
9-11 ELA
$1,000.00

7.5.5.2 and 7.5.6.6
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

7.5.5.2 and 7.5.6.6
June 2015
June 2016
June 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
7.5.5.2 and 7.5.6.6
November 2016
$0.00
$0.00
$0.00

$0.00
$0.00

data files.

7.5.5.2 and 7.5.6.6
February 2016
$0.00
$0.00
$0.00
$0.00
$0.00

June 2016
860
861
863
Specify
B, C, D or
(A-D):
withComponent
Department A,
Scoring
(1) EFinal
late_____C_____
E
files. The SSR
Description
Due Date(s)
7.5.5.2 and 7.5.6.6

data files.
$0.00
$0.00

A
1
864
865
B
DSRs dataContractor
files: Development
of
Deliverable
data files.
866
and SFTP.
Evidencesystem
of Completion
Due Date(s)
867
869
870
871
872

data files.

7.5.5.2 and 7.5.6.6
873
875
876
877

and 7.5.6.7
878
880
881
882
883
885
886
887
888
889
891
892

Staff to develop
Description
7.5.5.2 and 7.5.6.6
F
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
December 2015
December 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
and 7.5.6.7
October 2016
October 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

assessments.
and 7.5.6.7
October 2015
October 2016
October 2017
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

December 2015
and 7.5.6.7
$0.00
$0.00
$0.00

A
893
894
896
897
898
899
901
902
903
904
905
907
908
909
910
911
913
914
915

B data files:
files for all
assessments.
Contractor
Deliverable
918
919
920
921
922
923
924
F
Component C
9-11 ELA
$0.00

and 7.5.6.7
March 2015
March 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

and 7.5.6.7
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

and 7.5.6.7
October 2015
October 2016
October 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00


September 2015
September 2016
September 2017
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

reporting

2015
$1,000.00
$1,000.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2017
reporting
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
916
917

E
of all assessments.
data files.
Description
of Completion
Due Date(s)
and 7.5.6.7

$1,000.00

A
1
925
926

for districts and
C
and PDFsContractor
of reports for
the initial
Deliverable
reporting

D
approval of files
and reports.
Evidence
of Completion

initial reporting


Due Date(s)
Component C
9-11 ELA
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$1,000.00
$1,000.00
$1,000.00
927
928
929
930
931
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955


$1,000.00
$1,000.00
$1,000.00
$1,000.00
(Renewal 3 - 2020)
2015
$1,000.00
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
2016
$0.00
$0.00
$0.00
$0.00
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
956
957

$50,000.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
932
933

2015


jump drives.
districts.
(Renewal 3 - 2020)
2015
2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$177,489.00
$177,489.00

A
1

Development
of the Deliverable
student reports and
Contractor

Department after
all reports
and media are
Deliverable
Description
958
959

jump drives.
(2) Provide proof
of delivery
Evidence
of Completion
districts.
Due Date(s)
Component C
9-11 ELA
(Renewal 1 - 2017)
$175,378.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$175,378.00
$175,378.00

jump drives.
districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$323,266.00
(Renewal 3 - 2020)
$122,230.00

jump drives.
districts.

2015
$581,910.00
2016
2017
(Renewal 1 - 2018)
$533,358.00
$538,070.00
$593,856.00
(Renewal 2 - 2019)
$593,856.00
(Renewal 3 - 2020)
$593,856.00
960
961
962
963


964
$123,266.00
$121,908.00
$122,230.00
965
966
967
968


969
970
971
972
973
974
975

initial reporting


jump drives.
districts.

2015
2016
(Renewal 1 - 2017)

jump drives.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
(Renewal 3 - 2019)
$0.00

jump drives.
districts.

2015
$0.00
2016
$0.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
976
977
978
979

initial reporting

980
981
982
983

initial reporting


A
1
984
985
986

Reports and CD/DVDs
B or jump drives.
of the student
reports
and media for the
Contractor
Deliverable
initial reporting

Department after
all reports
and media are
Deliverable
Description

jump drives.
(2) Provide proof
of delivery
Evidence
of Completion
districts.

initial reporting

Due Date(s)
Component C
9-11 ELA
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00

jump drives.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00
987
988
989
990
991
992
993
994
995
996
997


Reference 7.5.5

reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00


Reference 7.5.5

reports.
districts.

2016
$0.00
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00


Reference 7.5.5

reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
998
999
1000
1001
1002
1003
1004
1005
1006
1007
A
1

1008
Attachment
1b - DELIVERABLES
the student report
Specifyresponse
Component
A, B, C, D or
E (A-D):
_____C_____
to selected
approval
of the
C
Reference 7.5.5
Due Date(s)
Component C
9-11 ELA
(Renewal 2 - 2019)
$0.00
(Renewal 3 - 2020)
$0.00
1009
1010
1011
1012
1013

reports

Reference 7.5.5

reports.
districts.

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00

reports

Reference 7.5.5

reports.
districts.

2015
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00

reports

Reference 7.5.5

reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00
$0.00
$0.00

reports

Reference 7.5.5

reports.
districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
$0.00
$0.00
$0.00
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
A
1

reports
B
Specify
Component
A,
B,
C,
D
or
(A-D):proof
_____C_____
The contractor
address,
C
D
Submit proof of
delivery
Deliverable
Evidence
of Completion
Reference 7.5.5
Due Date(s)
Component C
9-11 ELA
(Renewal 3 - 2020)
$0.00
1031
1032
1033
1034
1035

reporting


districts.

2015
$1,000.00
2016
(Renewal 1 - 2017)
$1,000.00
$1,000.00
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$1,000.00
$1,000.00

districts.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$1,000.00
(Renewal 3 - 2020)
$1,000.00

districts.

2015
$1,200.00
2016
2017
(Renewal 1 - 2018)
$1,200.00
$1,200.00
$1,200.00
(Renewal 2 - 2019)
$1,200.00
$1,200.00
$0.00
1036
1037
1038
1039

reporting

1040
$1,000.00
$1,000.00
$1,000.00
1041
1042
1043
1044

late reporting

1045
1046
1047
1048
1049
1050
1051


(Renewal 3 - 2020)
2015
(Renewal 1 - 2017)
(Renewal 3 - 2019)
districts.
$0.00
$0.00
$0.00
$0.00
1052

A
1
1053
1054
1055
B
C
1056
districts.
Due Date(s)
2015
Component C
9-11 ELA
$0.00
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
(Renewal 3 - 2019)
$0.00

districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
(Renewal 3 - 2020)
$0.00

districts.

2015
$0.00
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
1057
1058
1059
1060
1061


1062
1063
1064
1065
1066
1067
1068
1069
1071
1072
1073
1074
1076
1077
1078
1079


(Renewal 3 - 2020)



September 2015
September 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00



March 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00



March 2015
March 2017
$2,200.00

A
and Reporting
1 Scoring
Deliverable
Category
1081
1082
1083
1084
F
Component C
9-11 ELA
approved final mock

Label/Roster,
of Completion
Due Date(s)
March 2017



September 2015



November 2015
November 2016
$0.00
$0.00
$0.00
$0.00
$0.00
1095
1097
1098
1099
1100



March 2015
March 2016
March 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
1101
1103
1104
1105
1106
1107



June 2015
June 2016
June 2017

Performance Tasks

1085
1087
1088
1089
1090
1092
1093
1094
1108
1109
1110
1111
Spring ELA/L
and Mathematics:
Contractor
DeliverableFinal
$2,200.00
$2,200.00
$2,200.00
$2,200.00
$2,200.00
$0.00
$0.00
$0.00
$0.00
$0.00
September 2016

2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$263,097.00
$251,567.00
$264,773.00
$272,510.00
A
1

Performance Tasks
B
Attachment
1b - DELIVERABLES
PRICING
necessary
for
(1) Provide
Specify
Component
A, B,onC, D or
(A-D):
_____C_____
of a students
response
theEfinal
handscoring
report.
that respondents
C
D
E
Due Date(s)
(Renewal 3 - 2019)
F
Component C
9-11 ELA
$278,476.00
1112
1113
1114
1115
1116
1117

Performance Tasks

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)

2015
$0.00
$221,275.00
$229,456.00
$241,985.00
$248,763.00
$253,075.00
1118
1119
1120
1121
1122
1123

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
$3,311,908.00
$3,459,155.00
$3,570,675.00
$3,761,255.00
$3,882,573.00
A
1

B

Attachment
1b - DELIVERABLES
PRICING report.
response
on
or on theComponent
performance-based
A, B, C, D or E (A-D): _____C_____
D
E
Due Date(s)
(Renewal 3 - 2020)
F
Component C
9-11 ELA
$3,896,135.00
1124
1125
1126
1127
1128

Tasks

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00

2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
1129
1130
1131
1132
1133

Performance Tasks
$0.00
$0.00
$0.00
$0.00
1134

A
1
B
Performance Tasks
1135
1136
1137
1138
1139

Performance Tasks
1141
1142
1143
1144
1145
1146
1148
1149
1150
1151
1153
1154
1155
1156
Due Date(s)

2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
1140
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
all assessments
Reference 7.5.5

the SFTP.


the SFTP.

Reference 7.5.5
June 2015
June 2016
January 2016
January 2017
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$0.00
$0.00
$0.00
$0.00

A
1
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
Reference 7.5.5
the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

all assessments
Reference 7.5.5

the SFTP.

ELA/Mathematics

ELA/Mathematics
Reference 7.5.4.2
Reference 7.5.4.2

and ELA/Mathematics
Reference 7.5.4.2

Reference 7.5.4.2
E
Due Date(s)
January 2015
January 2016
January 2017
June 2015
June 2016
October 2015
October 2016
January 2015
January 2016
January 2017
April 2015
April 2016
April 2017
December 2015
December 2016
October 2016
October 2017
October 2015
October 2016
October 2017
December 2015
F
Component C
9-11 ELA
$28,226.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$4,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$3,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00

A
1
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
Scoring
and Reporting
Deliverable
Category
B
Fall EOC:Contractor
Development
of the history
Deliverable
will create and
posted toofthe
SFTP.
Description
Completion
Reference 7.5.4.2
Reference 7.5.4.2
Reference 7.5.4.2

ELA/Mathematics
Reference 7.5.4.2

Retake
1225
1226
1227
1229
1230
1231
1232
1234
1235
1236
1237

Retake

rescoring.

Reference 7.5.6.5

management system.
log.

manual rescoring.

Reference 7.5.6.5

management system.
log.
E
Due Date(s)
December 2016
March 2016
March 2017
October 2015
October 2016
October 2017
October 2015
October 2016
October 2017
November 2015
November 2016

May 2016
May 2017

October 2015
F
Component C
9-11 ELA
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
October 2016
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
April 2016
$1,000.00

A
and Reporting
1 Scoring
Deliverable
Category
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1255
1256
1257
1258
1260
1261
1262
1263
1264
1266
1267
1268
Spring ELA/L
Retake:Deliverable
Manual Rescore
Contractor
manual rescoring.
Reference 7.5.6.5
(1) Manual rescore

log posted
to the online
Evidence
of Completion
management system.
log.


Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.

rescoring.

Reference 7.5.6.5

management system.
log.



districts.
1269
1270
1271
1272
1273
1274
1275
E
Due Date(s)
April 2017
April 2015
April 2016
April 2017
October 2015
October 2016
December 2015
F
Component C
9-11 ELA
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$0.00
December 2016
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

April 2015
April 2016
April 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

July 2015
July 2016
July 2017
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00

July 2015
$0.00
$12,266.00
July 2016
July 2017
$12,266.00
$12,266.00

$12,266.00
$12,266.00
$12,266.00
1276

A
1
1278
Studies
1279
B
1280
1281
1282
1284 Studies
1285
C
Administrations

Studies
1290
1291
1292
1293
1294
Studies
1296
1297
1298
1299
1300
1303 Studies
1304
1305

Plans

Administrations

Specifications
Due Date(s)
Component C
9-11 ELA
May 2014
May 2015
$41,346.00
$2,000.00
May 2016
$0.00
$0.00

$0.00
$0.00

deliverable.
July 2014
July 2015
$59,589.00
$59,589.00
July 2016
$59,589.00
$59,589.00

$59,589.00
$59,589.00

July 2014
July 2015
$6,000.00
$2,000.00
July 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$6,000.00
$2,000.00
1286
1287
1288



August 2014
August 2015
August 2016
December 2014
December 2015
December 2016
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$35,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00

Studies
1306
1307
1308
1309
1310
1311
1312
1313

Studies

1314 Studies
1315
1316
1317
1318
1321
1322
1323
B
Studies
1328
1329
B, C, D or
(A-D):
_____C_____
plan for Component
(1) ESES
Specifications
C
D


posted to SFTP.
E
Due Date(s)
May 2015
May 2016
May 2017
2016
F
Component C
9-11 ELA
$2,000.00
$150,000.00
$150,000.00
$150,000.00
$150,000.00
$150,000.00
$150,000.00
$2,000.00
$1,000.00

via SFTP.



via SFTP.


posted to SFTP.
December 2015
December 2016
December 2017
$94,429.00
$151,570.00
$151,570.00
$151,570.00
$151,570.00
Studies
1324
1327

Specifications

Technical Reports
2017
$1,000.00
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$1,000.00
$1,000.00
$1,000.00
2016
2017
$0.00
$0.00
$0.00
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00
$0.00
$0.00

Studies
1330

Technical Reports
B
Preparing Technical Specify

Reports,Component
as described in
(1) ESubmit
A,section
B, C, D or
(A-D):initial
_____C_____
the feedback, and
E
Due Date(s)
F
Component C
9-11 ELA
$151,570.00


Reference 7.6.3.3

posted to SFTP.
November 2014



SFTP.
October 2014


May 2015

Studies

Setting Work Plan
$2,000.00

Studies
$115,401.00

Studies

Technical Report
SFTP.

Studies
$20,412.00
$0.00
1332
1333 Studies
1334
Studies
$697,578.00
$0.00
$85,970.00
$0.00
1336
7.6.3.4
1337
1338
1339

7.6.3.4.
Department.
$30,761.00

A
1
Studies
1341
Studies
renewal)
SFTP.

Report


Due Date(s)
Component C
9-11 ELA
November 2014
$0.00
$0.00

August 2015
$0.00
$0.00



Reference 7.6.3.6
1343
Studies
1344
1345
Studies

Report
$17,819.00
$0.00
$45,291.00
$0.00
1347
1348 Studies

across devices.
input types.
concordance.
characteristics.
$5,400.00

Scaling, Equating,
A Scoring,
Studies
1349
Studies
C conducting
Delivering
Assessments
and detailed
Contractor
Deliverable
Description
across devices.
input types.
concordance.
characteristics.

D
Evidence email
of Completion

with TAC Committee





November 2015
SFTP.


7.6.3.6.2

August 2016

Assessments


SFTP.
Due Date(s)
Component C
9-11 ELA
$0.00
$29,752.00
$0.00
1351
Studies
1352
1353
Studies
1355
Studies
$900.00
$0.00
November 2015
$9,868.00
$0.00
$900.00

A
1
Scaling,
Equating,Category
Scoring,
Deliverable
Studies
Periodic:Contractor
Development
of a study plan
Deliverable
Assessments
The Contractors
Deliverable
section 7.6.3.9.

conducting
of Completion
SFTP.
November 2015

Studies

Assessments

7.6.3.6.3

Due Date(s)
Component C
9-11 ELA
$0.00
1357
1358
$900.00
$0.00
1359
1361
1362
1363
1364
1365
1367
1368
1369
1370
1371
1373
1374
1375
1376

7.7.2

report

Meetings


previous invoice.

ITN.

April 15, 2014

April 15, 2015
April 15, 2016
$47,285.00
$19,950.00
$19,950.00
$19,950.00
$19,950.00
$19,950.00
$121,879.00
$122,426.00
$122,851.00
$123,260.00
$124,154.00
$124,566.00
$1,139,500.00
$1,276,463.00
$1,287,729.00
$1,102,105.00
$1,122,579.00
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)

Program Management
A
1
1377
1379
1380
1381
1382
1383
1385
1386
1387
1388
1389
1391
1392
1393
1394
B
Component
B, C, D or
(A-D):support
_____C_____
in the ITN
(1) EInclude
C
D
ITN.
Due Date(s)
Component C
9-11 ELA
(Renewal 3 - 2020)
$1,129,158.00
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$112,459.00
$112,459.00
$112,459.00
$112,459.00
$112,459.00
$112,459.00

Action Item Report

work. ITN Ref 7.7.6

November 2015
November 2016
November 2017
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00
$54,642.00

ITN Ref 7.7.6
2014
2015
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$69,917.00
$69,797.00
$69,797.00
$69,797.00
$69,797.00
$69,797.00


1395

A
1
Program Management
B
Ref. 7.7.10
Due Date(s)
one-year renewals]
Component C
9-11 ELA
$43,167.00
1396

A
1
3
4
5
6
7
8
Test Development
Document
Test Development

Specifications
9
10
11
12
13
C
7.3.10
via SFTP.
Due Date(s)
June 1, 2014
June 1, 2015
June 1, 2016
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
7.5.4.5
F
Component D
EOC
$38,237.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
14
Test Development
15
16
17
18
19


7.5.4.5

SFTP.
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$0.00

7.5.4.5

SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
$0.00
$0.00
20
Test Development
21
22
23
24
25

$0.00
$0.00
$0.00
$0.00
$0.00
26
Test Development
27
28
29
30
31
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
32

A
1
Test Development
33
34
35
36
37
F
Component D
EOC
7.5.4.5
Due Date(s)
SFTP.
2016
(Renewal 1 - 2017)
(Renewal 2 - 2018)
(Renewal 3 - 2019)

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$8,000.00

7.5.4.5

SFTP.
2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$1,000.00
$0.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
38
Test Development
39
40
41
42
43
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
44
Test Development
45
46
47
48
49
50
51 Test Development
52
53
54
55
56
Test Development

Specifications
7.3.10



June 1, 2014
June 1, 2015
June 1, 2016
November 1, 2015
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$15,797.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$35,812.00
57

A
1
Test Development
B
58
59
60
61
62
each scoring guide.
Due Date(s)
Reference 7.3.7.5


June 1, 2015
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
Component D
EOC
$15,722.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
63
Test Development

Assessment Reports
64
65
66
67
68
2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$67,402.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
$2,000.00
69
Test Development
70
71 Test Development
72
73
74
75
76 Test Development
77
78
79
80

$73,872.00

management report
Reference 7.3.4.2

February 1, 2017
management report
$1,915,038.00
$1,955,488.00
$1,005,531.00
$870,544.00
$729,403.00
Review

$1,954,433.00
$16,033.00
$16,033.00
$16,033.00
$16,033.00

management report
May 1, 2015
May 1, 2016

A
1
81 Test Development
82
83
84
85
86
87 Test Development
88
89
90
91 Test Development
92
93
94
95 Test Development
96
97
98
99
100
102
103
104
105
106
108
109
110
111
112
B


management report.
December 1, 2015
December 1, 2016
$11,652.00
$11,652.00
$11,652.00
$11,652.00


management report.
December 1, 2015
December 1, 2016
$9,679.00
$4,555.00
$4,555.00
$4,555.00


previous invoice.

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$55,695.00
$56,441.00
$57,205.00
$58,783.00
$59,860.00
$60,029.00




management report
Due Date(s)
July 15, 2014
July 15, 2015
July 15, 2016
F
Component D
EOC

2016
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
2017
(Renewal 1 - 2018)
(Renewal 2 - 2019)
(Renewal 3 - 2020)
$16,834.00
$24,381.00
$24,381.00
$24,381.00
$24,381.00
$24,381.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$5,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00
$1,000.00

A
1
Test Administration
B
F
Component D
EOC
the report.
Reference 7.4.2
Due Date(s)
2015
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$0.00
$0.00
$0.00
$0.00
the report.
ITN Reference 7.4.2

2016
$0.00
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00

a report.
the report.
ITN Reference 7.4.2

2015
$0.00
2016
2017
(Renewal 1-2018)
(Renewal 2-2019)
(Renewal 3-2020)
$0.00
$0.00
$0.00
$0.00
$0.00

the report.
ITN Reference 7.4.2

2015
$1,000.00
2016
(Renewal 1- 2017)
(Renewal 2-2018)
(Renewal 3 - 2019)
$1,000.00
$1,000.00
$1,000.00
$1,000.00
the report.
report listing

AIR Contract

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

AIR Contract

Uploaded by

Copyright:

Available Formats

Contract No.

Contract No.: 14-652

Contract No.: 14-652

Contract No.: 14-652

Contract No.: 14-652

Contract No.: 14-652

Contract #14-652 Attachment F

DEADLINE FOR TECHNICAL QUESTIONS: November 8, 2013 @ 2:00 PM

MAIL OR DELIVER REPLIES TO:

Florida Standards Assessments, ITN 2014-47

AUTHORIZED TO DO BUSINESS IN THE STATE OF FLORIDA

Florida Standards Assessments, ITN 2014-47

SECTION 4 SPECIAL INSTRUCTIONS

SECTION 5 SPECIAL CONDITIONS

SECTION 6 SPECIAL INSTRUCTIONS REPLY FORMAT & CONTENT

MAIL OR DELIVER REPLIES TO: (DO NOT FAX OR E-MAIL)

SECTION 7 SCOPE OF SERVICES

Florida Standards Assessments, ITN 2014-47

CONTRACTOR STAFF AND RESPONSIBILITIES

SECTION 8 OPENING, EVALUATION AND AWARD

Florida Standards Assessments, ITN 2014-47

INVITATION TO NEGOTIATE REGISTRATION

Internet E-Mail Address:

Florida Standards Assessments, ITN 2014-47

GENERAL INSTRUCTIONS TO RESPONDENT

Florida Standards Assessments, ITN 2014-47

SECTION 2 CONTRACT CONDITIONS

GENERAL CONTRACT CONDITIONS

Florida Standards Assessments, ITN 2014-47

Florida Standards Assessments, ITN 2014-47

Florida Standards Assessments, ITN 2014-47

CRITICAL EVENT DATES

October 25, 2013 ITN Released

November 8, 2013 @ 2:00 PM, ET - Deadline for Technical Questions

November 15, 2013 - Department Response to Technical Questions received by deadline

Florida Standards Assessments, ITN 2014-47

February 13, 2014 BAFOs due on or about

February 25, 2014 ITA Meeting on or about

SECTION 4 SPECIAL INSTRUCTIONS

VISITORS PASS TO THE TURLINGTON BUILDING

REPLY QUESTIONS & ANSWERS

A Site Inspection will not be held.

PROCUREMENT PROTESTS / NOTICE OF RIGHTS

Florida Standards Assessments, ITN 2014-47

ORAL INSTRUCTIONS / CHANGES TO THE INVITATION TO NEGOTIATE (ADDENDA)

MODIFICATIONS, RESUBMITTAL AND WITHDRAWAL

RESTRICTIONS ON COMMUNICATIONS WITH DEPARTMENT STAFF

Florida Standards Assessments, ITN 2014-47

CONFIDENTIAL, PROPRIETARY, OR TRADE SECRET MATERIAL

POOR PERFORMANCE NOTICE

4.10 WITHDRAWAL OF A REPLY

Florida Standards Assessments, ITN 2014-47

SECTION 5 SPECIAL CONDITIONS

AUTHORIZED TO DO BUSINESS IN THE STATE OF FLORIDA

LICENSED TO CONDUCT SERVICES IN THE STATE OF FLORIDA

IDENTICAL EVALUATION OF REPLIES

Florida Standards Assessments, ITN 2014-47

CONVICTED VENDOR LIST

Florida Standards Assessments, ITN 2014-47

Florida Standards Assessments, ITN 2014-47

5.16 PARTICIPATION IN FUTURE STAGES OF THIS PROJECT

THE REMAINDER OF PAGE INTENTIONALLY LEFT BLANK

Florida Standards Assessments, ITN 2014-47