Standard Setting Report

Standard Setting Report
JUNE 28–30, 2010

PORTLAND, ME
Prepared for the Maine Department of Education by:
100 Education Way, Dover, NH 03820 (800) 431-8901

TABLE OF CONTENTS
CHAPTER 1. OVERVIEW OF STANDARD SETTING ..........................................................................................................2
CHAPTER 2. TASKS COMPLETED PRIOR TO STANDARD SETTING ..................................................................................3
2.1 Creation of Achievement Level Descriptors .................................................................................................3
2.2 Selection of Student Portfolios (Bodies of Work)..........................................................................................3
2.3 Preparation of Materials for Panelists .........................................................................................................3
2.4 Preparation of Presentation Materials .........................................................................................................4
2.5 Preparation of Instructions for Facilitators Document ................................................................................4
2.6 Preparation of Systems and Materials for Analysis During the Meeting .....................................................4
2.7 Selection of Panelists ....................................................................................................................................4
CHAPTER 3. TASKS COMPLETED DURING STANDARD SETTING ....................................................................................6
3.2 Orientation................................................................................................................... .................................6
3.3 Review of Assessment Materials ...................................................................................................................6
3.4 Review of Achievement Level Definitions .....................................................................................................7
3.5 Training Evaluation ............................................................................................................................. .........7
3.6 Round 1 Judgments .......................................................................................................................................7
3.7 Tabulation of Round 1 Results ......................................................................................................................7
3.8 Round 2 Judgments ............................................................................................................................. ........10
3.9 Tabulation of Round 2 Results ....................................................................................................................10
3.10 Round 3 Judgments .................................................................................................................. ...................13
3.11 Tabulation of Round 3 Results ....................................................................................................................13
3.12 Repeat Process for Second Grade and Content Area (Mathematics and Reading Only) ........................... 21
3.13 Evaluation............................................................................................................................. ......................21
CHAPTER 4. TASKS COMPLETED AFTER STANDARD SETTING ....................................................................................22
4.1 Analysis and Review of Panelists’ Feedback ..............................................................................................22
4.2 Calculation of Recommended Cutpoints.....................................................................................................22
4.3 Preparation of this Standard Setting Report...............................................................................................27
REFERENCES .................................................................................................................................................................28
APPENDICES...................................................................................................................................................................29
APPENDIX A. ACHIEVEMENT LEVEL DEFINITIONS
APPENDIX B. AGENDAS
APPENDIX C. NONDISCLOSURE FORM
APPENDIX D. SAMPLE RATING FORM
APPENDIX E. SAMPLE EVALUATION
APPENDIX F. OPENING SESSION POWERPOINT
APPENDIX G. FACILITATORS’ SCRIPTS
APPENDIX H. PANELIST AFFILIATIONS
APPENDIX I. EVALUATION RESULTS
APPENDIX J. POLICY ADJUSTMENTS
Overview of Standard Setting Process 1 2010 Maine PAAP Standard Setting Report
Chapter 1. OVERVIEW OF STANDARD SETTING
The purpose of this report is to summarize the activities of the standard setting meeting for Maine’s
Personalized Alternate Assessment Portfolio (PAAP) in reading and mathematics (grades 2–7, 10, and 11),
science (grades 5, 8, and 11) and writing (grades 4, 7, and 11). The PAAP standard setting meeting was held
between June 27 and 29, 2010. In all, there were 14 panels with 70 panelists participating in the process.
Eight panels met for two days, and each panel established cuts for two grade level combinations (either two
reading grades or two mathematics grades). The remaining six panels met for one day and established cuts for
a single grade and content area combination. The configuration of the panels is shown in Table 1-1. Note that
some panelists participated in multiple content areas.
Table 1-1. 2010 PAAP Standard Setting: Configuration of Standard Setting Panels
Number of June 28 June 29 June 30
Panel
panelists
Panel 1 8 Math 2 Math 3
Panel 5 11 Reading 2 Reading 3
Panel 9 7 Science 5
Panel 10 7 Science 8
Panel 11 7 Science 11
Panel 12 10 Writing 4
A modified version of the body of work method was used for setting standards for the PAAP. The
body of work standard setting method was developed specifically for use with assessments that are designed to
allow for a range of student responses, such as portfolio- or performance-based assessments (Kingston, Kahl,
Sweeney, & Bay, 2001). A modified version of the method has been in use for a number of years that
substantially reduces the logistical burden of the procedure and has been found to yield reasonable and
defensible cutpoints. In the body of work method, panelists are presented with samples of actual student work
(in this case, student portfolios) and make their judgments based on those samples. Specifically, panelists
examine each student portfolio and determine which achievement level best matches the particular
knowledge, skills, and abilities the student exhibits through his or her performance on the work sample. This
report is organized into three major sections, describing tasks completed prior to, during, and after the
standard setting meeting.
Overview of Standard Setting Process 2 2010 Maine PAAP Standard Setting Report
Chapter 2. TASKS COMPLETED PRIOR TO STANDARD SETTING
2.1 Creation of Achievement Level Descriptors
The Achievement Level Descriptors (ALDs) describe the set of knowledge, skills, and abilities that
students in each achievement level are expected to display. Staff at the Maine Department of Education
(MDOE) created these draft descriptors prior to the standard setting meeting, where they were presented to
the panelists. The draft ALDs are provided in Appendix A.
2.2 Selection of Student Portfolios (Bodies of Work)

The goal in selecting student portfolios to use for the standard setting was to select a total of 30 to 40
bodies of work, spread as evenly as possible across the range of possible total raw scores. For the PAAP,
teachers select from a series of standardized tasks. For future administrations, the teachers will be required to
sample tasks that match a predefined blueprint. Consequently, an additional goal was to sample portfolios that
matched the blueprint as best as possible. However, because the blueprint was not implemented in 2010, it
was not possible to find portfolios that represented the blueprint at each total score. A list of portfolios was
generated with approximately three times the target number of portfolios at each score point, and the
portfolios were reviewed and selected by Measured Progress special education staff. The final numbers of
portfolios selected ranged from 29 to 38.
2.3 Preparation of Materials for Panelists

The following materials were assembled for presentation to the panelists at the standard setting
meeting:
▪ Meeting agenda
▪ Nondisclosure agreement
▪ ALDs
▪ Samples of student portfolios
▪ Rating forms
▪ Evaluation forms
Copies of the ALDs, meeting agenda, nondisclosure form, a sample rating form, and evaluation forms
are included in Appendices A through E.
Tasks Prior to Standard Setting 3 2010 Maine PAAP Standard Setting Report
2.4 Preparation of Presentation Materials
The PowerPoint presentation used in the opening session was prepared prior to the meeting. The
presentation was designed to give panelists an overview of the assessment and how it is scored as well as a
preview of what to expect throughout the standard setting process. A copy of the presentation is included in
Appendix F.
2.5 Preparation of Instructions for Facilitators Document

A script was created for the group facilitators to refer to while working through each step of the
standard setting process. This document is included in Appendix G. The facilitators also attended a training
session, led by a Measured Progress psychometrician, approximately one week before the standard setting
meeting. The purpose of the training was to prepare the facilitators for the panel activities and to ensure
consistency in the implementation of procedures.
2.6 Preparation of Systems and Materials for Analysis During the

Meeting
The computational programming used to carry out all analyses during the standard setting meeting
was completed and thoroughly tested prior to the standard setting meeting.
2.7 Selection of Panelists

As was emphasized in Cizek and Bunch (2007), regardless of the method used, the selection of
panelists is an important factor in determining standard setting outcomes and maximizing the validity of the
standard setting process. The guidance provided by Standards for Educational and Psychological Testing
(1999) states “a sufficiently large and representative group of judges should be involved to provide reasonable
assurance that results would not vary greatly if the process were repeated.” Consistent with the above guidance,
as well as practical considerations regarding the maximum size of group that can be successfully managed by
group facilitators, the goal was to recruit standard setting panels of eight members representing different
stakeholder groups to set standards for each grade and content area. Targets for the size and composition of the
panels were also consistent with federal guidelines as described in Standards and Assessment Peer Review
Guidance: Information and Examples for Meeting Requirements of the No Child Left Behind Act of 2001 (U.S.
Department of Education, 2009).
Panelists were selected by the MDOE and Measured Progress prior to the standard setting meeting.
The goal was for each panel to consist of approximately eight participants: two or three special education
teachers experienced in working with students who have significant disabilities; two or three content area
teachers (representative of a range of grade level experiences); and one or two school administrators, higher
education personnel, general education teachers, or stakeholders from interest groups related to significant
disabilities. In addition, to the extent possible, panels were assembled so as to reflect a balance of gender,
race/ethnicity, and geographic location. A list of the panelists and their affiliations is included in Appendix H.
Chapter 3. TASKS COMPLETED DURING STANDARD SETTING
3.1 Overview of Body of Work Method
The body of work standard setting method was developed specifically for use with assessments that
are designed to allow for a range of student responses, such as portfolio- and performance-based assessments.
For a number of years a modified version of the method has been in use that substantially reduces the logistical
burden of the procedure and has been found to yield reasonable and defensible cutpoints. Panelists were asked
to evaluate each work sample from a holistic perspective before classifying it into a single achievement level.
3.2 Orientation
With regard to panelist training, Standards for Educational and Psychological Testing states the
following:
Care must be taken to assure that judges understand what they are to do. The
process must be such that well-qualified judges can apply their knowledge
and experience to reach meaningful and relevant judgments that accurately
reflect their understanding and intentions. (AERA/APA/NCME, 1999, p. 54)
The training of the panelists began with a general orientation at the start of the standard setting
meeting. The purpose of the orientation was to ensure that all panelists received the same information about
the need for and goals of standard setting and about their part in the process. First, the MDOE provided some
pertinent context about the PAAP program and an introduction to the issues of standard setting. Second, the
PAAP program manager provided an overview of the assessment, including alternate grade level expectations,
task bank selection, administration, and scoring. Next, a Measured Progress psychometrician presented a brief
overview of the body of work procedure and the activities that would occur during the standard setting
meeting. Once the general orientation was complete, each panel convened in a breakout room, where the
panelists received more detailed training and completed the standard setting activities.
3.3 Review of Assessment Materials

The first step after the opening session was for the panelists to become familiar with the PAAP. The
purpose of this step was to make sure the panelists thoroughly understood how the portfolio is administered
and scored. Panelists reviewed the Alternate Grade Level Indicators (AGLIs) and administration manuals. In
addition, panelists individually reviewed every fifth portfolio and discussed the knowledge, skills, and
abilities associated with each work sample as a group.
Tasks During Standard Setting 6 2010 Maine PAAP Standard Setting Report
3.4 Review of Achievement Level Definitions
The second step in the process, once the panelists convened into their content area and grade level
groups, was to discuss the Achievement Level Descriptors (ALDs). This important step was designed to ensure
that panelists thoroughly understood the knowledge, skills, and abilities needed for students to be classified
into achievement levels (Substantially Below Proficient, Partially Proficient, Proficient, and Proficient
With Distinction). Panelists first reviewed the ALDs on their own and then participated in group discussion
of the ALDs, clarifying the description for each achievement level. The discussions focused on the knowledge,
skills, and abilities that differentiated adjacent achievement levels. Bulleted lists of characteristics for each
level were generated based on the group discussion and were posted in the room for panelists to refer to during
the rounds of ratings.
3.5 Training Evaluation

Prior to beginning the Round 1 ratings, the panelists anonymously completed a training evaluation
form. The purpose of the evaluation was to ensure that panelists were comfortable with the process and ready
to move on to the rating task. Any issues or problems that came up in the training evaluations were addressed
before the facilitator proceeded to Round 1.
3.6 Round 1 Judgments

In the first round, panelists worked individually with the ALDs, the student portfolios, and the rating
form. The work samples consisted of 29 to 38 portfolios, with scores covering the full range of possible total
scores. For each portfolio, the panelists considered the skills and abilities demonstrated in the work sample,
and panelists decided which achievement level was the best match. The panelists worked their way through
the portfolios, making a rating for each one, and recorded their ratings on the rating form. While the portfolios
were presented in order of total score, panelists were not required to rate them strictly in increasing order.
Instead, panelists were encouraged to take a holistic look at the knowledge, skills, and abilities demonstrated
in the portfolio, rather than making a judgment based primarily on the total raw score.
3.7 Tabulation of Round 1 Results

After all panelists had completed their individual ratings, the Measured Progress data analysis team
calculated the average cutpoints for the group based on the Round 1 ratings. Cutpoints were calculated using
SAS statistical software. Logistic regression was used to determine each panelist’s individual cutpoints, and
then the cutpoints were averaged across the group. In addition, impact data were calculated, which reflected
the percentage of students who would fall into each achievement level based on the group average Round 1
ratings. The Round 1 results are outlined in Tables 3-1 through 3-4.
Table 3-1. 2010 PAAP Standard Setting: Round 1 Mathematics
Average Standard Raw score Percent of
Grade Achievement level
cut error Min Max students
PD 62.6 1.6 63 69 3.4
2 Proficient 45.9 2.1 46 62 21.0
Partially Proficient 24.9 2.5 25 45 50.0
SBP NA NA 0 24 25.6
PD 70.2 1.6 NA NA NA
Proficient 52.9 2.8 53 69 35.6
3
SBP NA NA 0 25 22.9
PD 65.4 1.1 66 69 12.0
Proficient 42.8 2.3 43 65 50.9
4
SBP NA NA 0 20 8.5
PD 65.5 1.3 66 69 36.4
Proficient 52.0 2.4 53 65 27.3
5
SBP NA NA 0 25 15.5
PD 85.5 1.6 86 99 10.1
Proficient 57.0 2.1 58 85 49.8
6
SBP NA NA 0 25 14.1
PD 91.4 1.0 92 99 19.8
Proficient 51.1 1.5 52 91 58.5
7
SBP NA NA 0 24 8.2
PD 120.3 0.8 121 129 7.9
Proficient 83.6 4.3 84 120 33.5
10
SBP NA NA 0 36 18.8
PD 125.9 5.8 126 129 9.5
Proficient 83.2 4.3 84 125 34.4
11
SBP NA NA 0 24 13.7
Table 3-2. 2010 PAAP Standard Setting: Round 1 Reading

PD 35.5 5.9 36 46 17.2
Proficient 31.0 6.4 32 35 11.1
2
SBP NA NA 0 11 2.5
PD 45.7 0.1 46 46 7.7
Proficient 30.2 1.6 31 45 33.7
3
SBP NA NA 0 18 25.5
4
Proficient 29.3 1.7 30 46 65.7
continued
Grade Average Standard Raw score Percent of
Achievement level
4
SBP NA NA 0 14 8.4
PD 43.3 1.0 44 46 42.3
Proficient 24.5 1.1 25 43 38.9
5
SBP NA NA 0 12 3.4
PD 61.9 0.9 62 66 5.1
Proficient 50.7 1.6 51 61 31.2
6
SBP NA NA 0 31 30.3
PD 63.1 0.6 64 66 24.8
Proficient 32.8 1.6 33 63 51.8
7
SBP NA NA 0 17 8.7
PD 85.5 0.3 86 86 1.6
Proficient 65.8 5.5 66 85 27.7
10
SBP NA NA 0 21 17.0
Proficient 56.2 2.3 57 86 45.9
11
SBP NA NA 0 32 23.6
Table 3-3. 2010 PAAP Standard Setting: Round 1 Science

PD 64.7 1.5 65 69 19.3
Proficient 44.2 1.4 45 64 44.8
5
SBP NA NA 0 23 13.8
PD 91.9 1.5 92 99 9.1
Proficient 57.5 2.2 58 91 58.9
8
SBP NA NA 0 29 8.7
PD 128.2 4.3 129 129 2.6
Proficient 88.6 6.9 89 128 30.7
11
SBP NA NA 0 46 22.8
Table 3-4. 2010 PAAP Standard Setting: Round 1 Writing

PD 23.0 0.2 23 23 16.2
4 Proficient 16.2 0.7 17 22 29.9
continued
4 SBP NA NA 0 8 29.0
PD 32.8 0.1 33 33 11.7
Proficient 15.7 0.6 16 32 60.2
7
SBP NA NA 0 11 13.6
PD 42.8 2.2 43 43 4.3
Proficient 19.5 3.9 20 42 56.0
11
SBP NA NA 0 8 15.5

The purpose of Round 2 was for panelists to discuss their Round 1 judgments as a group and determine
whether any revisions were necessary. A psychometrician shared the average cutpoint locations with the
panelists to help inform their group discussion and Round 2 ratings. It is important to note that although the
impact data and raw score ranges are presented in Tables 3-1 through 3-4, they were not shared with the
panelists after Round 1. Prior to the group discussion, the facilitator asked for a show of hands to determine the
number of panelists who had placed each portfolio into each achievement level; the facilitator then recorded
the results on chart paper. Starting with the first portfolio they disagreed on, the panelists began discussing the
categorization of the portfolios according to their initial ratings in the context of the classifications made by
other members of the group. Panelists were encouraged to share their own points of view as well as listen to
the comments of their colleagues. Facilitators made sure the panelists knew that the purpose of the discussion
was not to reach consensus; at every point throughout the standard setting process, panelists were asked to
provide their own best judgment. Once the discussions were complete, the panelists completed the Round 2
rating form.

When Round 2 ratings were complete, the Measured Progress data analysis team calculated the
average cutpoints for the room and associated impact data. The results of the panelists’ Round 2 ratings are
outlined in Tables 3-5 through 3-8.

Grade Achievement level error students
cut Min Max
PD 62.6 1.2 63 69 3.4
Proficient 44.7 1.2 45 62 22.7
2
SBP NA NA 0 23 19.9
continued
Achievement level
Proficient 51.5 1.2 52 69 37.6
3
SBP NA NA 0 25 22.9
PD 68.1 0.0 69 69 7.3
Proficient 45.7 1.6 46 68 51.7
4
SBP NA NA 0 21 9.0
PD 66.3 0.9 67 69 33.2
Proficient 51.2 0.4 52 66 31.0
5
SBP NA NA 0 26 16.6
PD 85.0 1.2 86 99 10.1
Proficient 55.5 1.8 56 85 52.0
6
SBP NA NA 0 25 14.1
PD 91.3 0.9 92 99 19.8
Proficient 49.9 1.0 50 91 58.5
7
SBP NA NA 0 24 8.2
PD 119.2 0.9 120 129 7.9
Proficient 87.9 2.3 88 119 31.9
10
SBP NA NA 0 37 19.4
PD 122.4 1.8 123 129 10.8
Proficient 77.1 1.0 78 122 39.0
11
SBP NA NA 0 31 16.2

PD 45.2 0.1 46 46 4.5
2 Proficient 33.0 0.3 33 45 23.7
SBP NA NA 0 18 22.2
PD 45.5 0.1 46 46 7.7
Proficient 29.1 0.7 30 45 39.9
3
SBP NA NA 0 18 25.5
PD 42.8 0.8 43 46 21.5
Proficient 23.7 0.7 24 42 55.8
4
SBP NA NA 0 11 2.4
PD 41.5 0.4 42 46 42.3
5
Proficient 23.9 0.3 24 41 41.3
continued
Achievement level
5 Partially Proficient 13.0 0.3 13 23 13.0
SBP NA NA 0 12 3.4
PD 60.6 0.6 61 66 7.7
Proficient 46.6 0.8 47 60 35.5
6
SBP NA NA 0 28 26.5
PD 62.9 0.7 63 66 25.7
Proficient 34.5 1.0 35 62 49.1
7
SBP NA NA 0 16 8.3
PD 85.5 0.0 86 86 1.6
Proficient 54.7 0.8 55 85 48.4
10
SBP NA NA 0 29 22.3
PD 84.5 0.0 85 86 5.0
Proficient 57.4 0.4 58 84 40.9
11
SBP NA NA 0 35 26.9

Achievement level
PD 65.0 1.2 65 69 19.3
5 Proficient 44.0 0.7 45 64 44.8
SBP NA NA 0 23 13.8
PD 92.6 0.7 93 99 8.7
Proficient 58.4 0.4 59 92 58.9
8
SBP NA NA 0 32 9.1
PD 126.1 1.2 127 129 3.5
Proficient 86.5 1.3 87 126 32.0
11
SBP NA NA 0 49 30.3

PD 22.9 0.0 23 23 16.2
Proficient 14.6 0.2 15 22 32.4
4
SBP NA NA 0 9 29.0
PD 32.9 0.1 33 33 11.7
7 Proficient 15.9 0.3 16 32 60.2
continued
Grade Achievement level error students
cut Min Max
7 SBP NA NA 0 11 13.6
PD 42.1 0.7 43 43 4.3
Proficient 24.3 0.5 25 42 37.5
11
SBP NA NA 0 11 15.9

The purpose of Round 3 was for panelists to discuss their Round 2 ratings as a whole group and, if
necessary, to revise their judgments. Prior to the group discussion, the facilitator once again asked for a show
of hands to determine the number of panelists who had placed each portfolio into each achievement level; the
facilitator recorded the results on chart paper. The group average cuts based on the Round 2 results were
presented. In addition, in this round, the group was presented with the impact data. The psychometrician
presented the group average cuts and impact data to the group and explained how to use the information as
they completed their Round 3 discussions. Panelists were encouraged to discuss whether the percentage of
students classified in each performance level seemed reasonable, given their perceptions of the students and
the knowledge, skills, and abilities demonstrated in the portfolios. As in Round 2, starting with the first
portfolio for which there was disagreement, the panelists discussed their ratings, with the impact data
considered as additional context for the discussion. Finally, after the discussions were complete, panelists were
given a final opportunity to revise their ratings. Once again, the facilitator reminded the panelists that they
should use their individual best judgment and that it was not necessary for them to reach consensus.

When Round 3 ratings were complete, the Measured Progress data analysis team once again
calculated the average cutpoints for the room and associated impact data. The results of the panelists’ Round
3 ratings are outlined in Tables 3-9 through 3-12 and in Figures 3-1 through 3-4.

PD 64.0 0.9 64 69 3.4
Proficient 43.5 0.9 44 63 26.7
2
SBP NA NA 0 23 19.9
3 Proficient 51.1 1.2 52 69 37.6
continued
3 SBP NA NA 0 26 24.4
PD 61.1 0.4 62 69 20.5
Proficient 41.2 0.3 42 61 42.3
4
SBP NA NA 0 21 9.0
PD 66.9 0.6 67 69 33.2
Proficient 51.7 0.3 52 66 31.0
5
SBP NA NA 0 26 16.6
PD 84.8 1.2 85 99 10.1
Proficient 55.9 1.5 56 84 52.0
6
SBP NA NA 0 24 13.2
PD 91.3 0.9 92 99 19.8
Proficient 49.9 1.0 50 91 58.5
7
SBP NA NA 0 24 8.2
PD 120.1 1.3 121 129 7.9
Proficient 85.6 1.6 86 120 32.5
10
SBP NA NA 0 35 18.3
PD 122.4 1.8 123 129 10.8
Proficient 77.5 0.6 78 122 39.0
11
SBP NA NA 0 29 15.8
PD 45.2 0.1 46 46 4.5
Proficient 31.7 0.7 32 45 23.7
2
SBP NA NA 0 17 22.2
PD 45.5 0.1 46 46 7.7
Proficient 28.8 0.6 29 45 42.3
3
SBP NA NA 0 17 25.0
PD 42.2 0.2 43 46 21.5
Proficient 24.4 0.3 25 42 54.2
4
SBP NA NA 0 11 2.4
PD 41.6 0.4 42 46 42.3
Proficient 23.9 0.2 24 41 41.3
5
SBP NA NA 0 13 4.3
PD 59.1 1.0 60 66 8.5
Proficient 30.8 0.8 31 59 62.8
6
SBP NA NA 0 18 13.7
PD 64.0 0.5 64 66 24.8
Proficient 37.3 1.5 38 63 44.0
7
SBP NA NA 0 17 8.7
PD 85.5 0.0 86 86 1.6
Proficient 55.6 0.0 56 85 45.7
10
SBP NA NA 0 29 22.3
PD 84.5 0.0 85 86 5.0
Proficient 56.8 0.0 57 84 40.9
11
SBP NA NA 0 36 28.9
PD 65.9 1.3 66 69 19.3
Proficient 44.6 0.7 45 65 44.8
5
SBP NA NA 0 23 13.8
PD 92.3 0.7 93 99 8.7
Proficient 58.0 0.5 58 92 59.4
8
SBP NA NA 0 32 9.1
PD 126.1 1.2 127 129 3.5
Proficient 86.5 1.3 87 126 32.0
11
SBP NA NA 0 49 30.3

PD 23.0 0.0 23 23 16.2
Proficient 14.4 0.1 15 22 32.4
4
SBP NA NA 0 9 29.0
PD 32.9 0.1 33 33 11.7
Proficient 16.2 0.4 17 32 59.2
7
SBP NA NA 0 11 13.6
PD 40.9 0.7 41 43 8.2
Proficient 23.8 0.6 24 40 33.6
11
SBP NA NA 0 12 16.4
Figure 3-1. 2010 PAAP Standard Setting: Mathematics: Round 3 Results
0
0
.....-
0
co
C"
0
0)
Q.)
+""'
ro 0
u CD
c
CJ)
+""'
c
Q.)
""C
:::J 0
+""'
CJ) "¢
'+-
0
+""'
c
Q.)
u
I-
Q.)
a_ N0
02 03 04 05 06 07 10 11
Proficient with Distinction
Proficient
Partially Proficient
Substantially Below Proficient
Mathematics: Round 3 Results
Figure 3-2. 2010 PAAP Standard Setting: Reading: Round 3 Results
0
0
........
0
ro
0
0)
Q)
.......
co 0
u ID
c
·CJ) -
........
c
Q)
"0
..:..:.:J. 0
CJ) "¢
'+-
0
........
c
Q)
u
Q)
a_ 0
N
02 03 04 05 06 07 10 11
Proficient
Reading: Round 3 Results
Figure 3-3. 2010 PAAP Standard Setting: Science: Round 3 Results
0
0
......
0
co
0
0)
Q)
.......
co
(.)
0
CD
c
·CJ) -
........
c
Q)
"0
..:..:.:J. 0
CJ) "¢
'+-
0
........
c
Q)
(.)
Q)
0
11.. N
05 08 11
Proficient
Science: Round 3 Results
Figure 3-4. 2010 PAAP Standard Setting: Writing: Round 3 Results
0
0
.....--
0
CX)
0
0)
Q)
+""'
co
(.) 0
<D
c
(/)
+""'
c
Q)
""0
::J
+""' 0
(/) "¢
'+-
0
+""'
c
Q)
(.)
I-
Q)
a_ 0C'\1
04 07 11
Proficient
Writing: Round 3 Results
3.12 Repeat Process for Second Grade and Content Area
(Mathematics and Reading Only)
As mentioned above, eight panels each recommended cutpoints for two grade levels in mathematics
and reading. For those panels, once they had completed the entire process for the first grade level, they (1)
completed a process evaluation, giving their perceptions of the standard setting process and results thus far,
and (2) repeated the entire process (except for the training evaluation) for the second grade level.
3.13 Evaluation
The measurement literature sometimes considers the evaluation process to be another product of the
standard setting process (e.g., Reckase, 2001). To provide evidence of the participants’ views of the standard
setting process, panelists were asked to complete questionnaires throughout the process. The results of the
evaluations are presented in Appendix I.
Chapter 4. TASKS COMPLETED AFTER STANDARD SETTING
Upon conclusion of the standard setting meeting, several important tasks were completed. These tasks
centered on reviewing the standard setting process and addressing anomalies that may have occurred in the
process or in the outcomes, presenting the results to MDOE and making any final revisions or adjustments.
4.1 Analysis and Review of Panelists’ Feedback

Upon completion of the evaluation forms, panelists’ responses were reviewed. In general, this review
did not reveal any anomalies in the standard setting process or indicate any reason that a particular panelist’s
data should not be included when the final cutpoints were calculated. The one exception was a panelist in
grade 10 mathematics who did not appear to adequately understand the standard setting task. The panelist
continually placed the Proficient cut at a higher achievement than the Proficient With Distinction cut.
Although, the panelist self-corrected this issue for grade 11 mathematics, the panelist’s ratings were removed
from the grade 10 mathematics calculations and are not reflected in the results presented in this report. It
appeared that all remaining panelists understood the rating task and attended to it appropriately.
4.2 Calculation of Recommended Cutpoints

At the end of the standard setting meeting, the Round 3 cuts were presented to the Department of
Education as the final results of the standard setting meeting. Following the standard setting, a few concerns
were raised about the cuts. Most important of these was that higher standards (from a raw score cut
perspective) were established by the panelists in grades 2 and 3 than in grades 4 and 5 despite the fact that
these grades are based on the same task banks, level of complexity, and raw score scale. In theory, the grade 2
and grade 3 cuts should represent similar achievement requirements as grades 4 and 5. In addition, because a
single task bank is used for High School and because of the population of interest, a single set of cuts for
these students seemed more appropriate. Finally, the Proficient cut was set at a higher level in grade 6
mathematics than in grade 7 mathematics. Given that these two grades are based on the same task banks, level
of complexity, and raw score scale, it was felt that proficiency in the two grades should represent similar
achievement requirements. Consequently, the following adjustments were made to the Round 3 raw score cuts
and are presented in Tables 4-1 through 4-2 and Figures 4-1 through 4-2.
Grade 4 raw score cuts were applied to grade 2 and grade 3.

A single set of cuts was established for High School by averaging the grade 10 and grade 11 raw
score cuts.
The grade 7 Proficient cut in mathematics was raised to equal the grade 6 Proficient cut in
mathematics.
Tasks After Standard Setting 22 2010 Maine PAAP Standard Setting Report
Table 4-1. 2010 PAAP Standard Setting: Mathematics: Adjusted Results
PD 61.1 NA 62 69 3.4
Proficient 41.2 NA 42 61 30.7
2
Partially Proficient 21.5 NA 22 41 58
SBP NA NA 0 21 8
PD 61.1 NA 62 69 14.1
3
Partially Proficient 21.5 NA 22 41 40
SBP NA NA 0 21 11.2
PD 61.1 0.4 62 69 20.5
Proficient 41.2 0.3 42 61 42.3
4
Partially Proficient 21.5 0 22 41 28.2
SBP NA NA 0 21 9
PD 66.9 0.6 67 69 33.2
Proficient 51.7 0.3 52 66 31
5
SBP NA NA 0 26 16.6
PD 84.8 1.2 85 99 10.1
Proficient 55.9 1.5 56 84 52
6
SBP NA NA 0 24 13.2
PD 91.3 0.9 92 99 19.8
7
SBP NA NA 0 24 8.2
PD 121.25 NA 122 129 9
Proficient 81.55 NA 82 121 35.4
HS
Partially Proficient 32.45 NA 33 81 38.4
SBP NA NA 0 32 17.1
Table 4-2. 2010 PAAP Standard Setting: Reading: Adjusted Results
PD 42.2 NA 43 46 7.6
Proficient 24.4 NA 25 42 53
2
SBP NA NA 0 11 2.5
PD 42.2 NA 43 46 17.3
3
SBP NA NA 0 11 2.9
PD 42.2 0.2 43 46 21.5
Proficient 24.4 0.3 25 42 54.2
4
SBP NA NA 0 11 2.4
PD 41.6 0.4 42 46 42.3
Proficient 23.9 0.2 24 41 41.3
5
Partially Proficient 13 0.3 14 23 12
SBP NA NA 0 13 4.3
PD 59.1 1 60 66 8.5
Proficient 30.8 0.8 31 59 62.8
6
Partially Proficient 18.9 0.7 19 30 15
SBP NA NA 0 18 13.7
PD 64 0.5 64 66 24.8
Proficient 37.3 1.5 38 63 44
7
SBP NA NA 0 17 8.7
PD 85 NA 85 86 4.2
HS
Partially Proficient 33 NA 33 56 30.9
SBP NA NA 0 32 24.7
Figure 4-1. 2010 PAAP Standard Setting: Mathematics: Adjusted Results
0
0
......
0
co
C"
0
0)
Q.)
+""'
ro 0
u c.o
c
CJ)
+""'
Q.)
""C
:::J 0
+""'
CJ) 'V
'+-
0
+""'
Q.)
u
I-
Q.)
a_ 0 N
2 3 4 5 6 7 HS
Proficient
Mathematics: Adjusted Results
Figure 4-2. 2010 PAAP Standard Setting: Reading: Adjusted Results
0
0
.....-
0
CX)
C"
0
0)
Q.)
+""'
ro 0
u CD
c
CJ)
+""'
c
Q.)
""C
:::J 0
+""'
CJ) "¢
'+-
0
+""'
c
Q.)
u
I-
Q.)
a_ N0
2 3 4 5 6 7 HS
Proficient
Reading: Adjusted Results
After carefully considering the above information, the MDOE remained concerned about a few of the
cut scores in mathematics and writing. Consequently, the department decided to make a final policy adjustment
to some of the cut scores. The resulting DOE-approved operational cut scores, and a more detailed explanation
outlining how the final cut scores were established can be found in Appendix J.
4.3 Preparation of this Standard Setting Report

Following final compilation of standard setting results, Measured Progress prepared this report, which
documents the procedures and results of the 2010 standard setting meeting in order to establish performance
standards for the Maine PAAP in mathematics, reading, science, and writing.
REFERENCES
American Educational Research Association, American Psychological Association, & National Council on
Measurement in Education (1999). Standards for educational and psychological testing. Washington,
DC: American Educational Research Association.
Cizek, G. J., & Bunch, M. B. (2007). Standard setting: Establishing and evaluating performance standards
on tests. Thousand Oaks, CA: Sage Publications.
Kingston, N., Kahl, S., Sweeney, K., & Bay, L. (2001). Setting performance standards using the body of
work method. Setting performance standards: concepts, methods, and perspectives (pp. 219–
248). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.
Reckase, M.D. (2001). Innovative methods for helping standard-setting participants to perform their task:
The role of feedback regarding consistency, accuracy, and impact. In G. J. Cizek (Ed.) Setting
performance standards: concepts, methods, and perspectives (pp. 159-173). Mahwah, NJ: Lawrence
Erlbaum.
U.S. Department of Education (2009). Standards and assessments peer review guidance: Information and
examples for meeting requirements of the No Child Left Behind Act of 2001. Washington, DC: U.S.
Department of Education Office of Elementary and Secondary Education. Retrieved June 10, 2010
from the World Wide Web: www2.ed.gov/policy/elsec/guid/saaprguidance.pdf.
References 28 2010 Maine PAAP Standard Setting Report

APPENDICES
Appendices 29 2010 Maine PAAP Standard Setting Report

Appendix A—ACHIEVEMENT LEVEL DEFINITIONS
Appendix A—Achievement Level Definitions 1 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 2
Reading achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate reading standards. These definitions serve as the
foundation for achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
• student initiated communication • reading aloud sight words

through signs, symbols, pictures, • putting key events in correct
gestures, and/or oral language sequence
• relating of symbols to the • identifying setting or characters
objects/ideas they represent • comprehension of information
• using phonemic awareness skills gained through listening or
• using context clues to determine viewing
meaning of words • retelling events in a story
Appropriate performance at grade two is clarified by the level of complexity within the standard.

The student’s performance demonstrates the skillful ability to identify and decode unfamiliar
vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the
consistent ability to read, comprehend, and interpret literary OR informational texts appropriate
to the student’s instructional level.
Proficient
The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary
and/or demonstrate understanding of word meaning. The work also generally demonstrates the
ability to read, comprehend, and interpret literary OR informational texts appropriate to the
student’s instructional level.
The student’s performance demonstrates the inconsistent ability to identify and decode
unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also
demonstrates the incomplete or inconsistent ability to read, comprehend, and interpret literary
OR informational texts appropriate to the student’s instructional level.

The student’s performance demonstrates the limited ability to identify and decode unfamiliar
vocabulary and/or demonstrates incorrect understanding of word meaning. The work also
demonstrates minimal or limited ability to read, comprehend, and interpret literary OR
informational texts appropriate to the student’s instructional level.

through signs, symbols, pictures, • distinguishing parts of a book
gestures, and/or oral language (e.g., front, top, bottom, title,
• relating of symbols to the author)
objects/ideas they represent • using explicitly stated information
• using phonemic awareness skills from the text to answer questions
• using context clues to determine • recognizing a central idea from
meaning of words text when presented with pictures
Appropriate performance at grade three is clarified by the level of complexity within the
standard.

Proficient


through signs, symbols, pictures, • putting key events in correct
gestures, and/or oral language sequence
• relating of symbols to the • identifying setting or characters
objects/ideas they represent • comprehension of information
• using phonemic awareness skills gained through listening or
• using context clues to determine viewing
meaning of words • retelling events in a story
Appropriate performance at grade four is clarified by the level of complexity within the standard.

Proficient


through signs, symbols, pictures, • distinguishing parts of a book
gestures, and/or oral language (e.g., front, top, bottom, title,
• relating of symbols to the author)
objects/ideas they represent • using explicitly stated information
• using phonemic awareness skills from the text to answer questions
• using context clues to determine • recognizing a central idea from
meaning of words text when presented with pictures
Appropriate performance at grade five is clarified by the level of complexity within the standard.

Proficient

Reading achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate reading standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in reading.
• relating of symbols to the objects/ideas • using explicitly stated information

they represent from the text to answer questions
• using phonemic awareness skills • recognizing a central idea from text
• using context clues, a dictionary and/or when presented with pictures
glossary to determine meaning of words • identifying or describing characters,
• using word parts, phonics, knowledge setting, problems, solutions, events
of sounds, syllable types and/or word and/or plot
patterns to decode • making logical predictions
• identifying synonyms, antonyms and/or • making basic inferences
categorizing words • paraphrasing or summarizing
• reading aloud sight words
Appropriate performance at grade six is clarified by the level of complexity within the standard.

The student’s performance demonstrates the skillful ability to identify and decode unfamiliar vocabulary
and/or demonstrate understanding of word meaning. The work also demonstrates the consistent ability to
read, comprehend, and interpret literary OR informational texts appropriate to the student’s instructional
level.
Proficient
The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or
demonstrate understanding of word meaning. The work also generally demonstrates the ability to read,
comprehend, and interpret literary OR informational texts appropriate to the student’s instructional level.
The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar
incomplete or inconsistent ability to read, comprehend, and interpret literary OR informational texts
appropriate to the student’s instructional level.

The student’s performance demonstrates the limited ability to identify and decode unfamiliar vocabulary
and/or demonstrates incorrect understanding of word meaning. The work also demonstrates minimal or
limited ability to read, comprehend, and interpret literary OR informational texts appropriate to the
• relating of symbols to the objects/ideas • recognizing a central idea from text

they represent when presented with pictures
• using phonemic awareness skills • obtaining information from the table
• using context clues, a dictionary and/or of contents, glossary, table of contents
glossary to determine meaning of words • connecting information within a text
• using word parts, phonics, knowledge • identifying parts of a book (e.g.,
of sounds, syllable types and/or word author, title, beginning, end)
patterns to decode • Paraphrasing or summarizing
• identifying synonyms, antonyms and/or • Drawing conclusions or making
categorizing words inferences
• reading aloud sight words
• using explicitly stated information from
the text to answer questions
Appropriate performance at grade seven is clarified by the level of complexity within the standard.

level.
Proficient

Achievement Level Definitions for Reading ~ High School
• using phonemic awareness skills • identifying author’s basic message

• using context clues, dictionary, glossary and/or purpose
and/or thesaurus to determine meaning • identifying narrator
of words • paraphrasing, summarizing, and/or
• using word parts or phonics to decode comparing/contrasting
• reading aloud sight words • making logical predictions
• identifying synonyms, antonyms and/or • making inferences
categorizing words • recognizing a central idea from text
• using explicitly stated information from when presented with pictures
the text to answer questions • obtaining information from the table
• recognizing a central idea from text of contents or glossary
when presented with pictures • connecting information within a text
• identifying or describing characters, • synthesizing information
setting, problems, solutions and/or
events
Appropriate performance at high school level is clarified by the level of complexity within the standard.

level.
Proficient

Achievement Level Definitions for Mathematics ~ Grade 2
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
• matching small collections of equivalent sets

• matching simple 2-D shapes
• comparing two items based on multiple attributes
• identifying measurement tools
• copying simple patterns
• collecting data
• identifying the outcome of an event
Appropriate performance at grade 2 is clarified by the level of complexity within the standard.

The student’s work demonstrates an understanding of essential concepts in mathematics,
including the ability to make connections among central ideas. The student’s responses
demonstrate the ability to utilize information and solve problems including implementing
strategies, accurately performing procedures and providing solutions.
Proficient
The student’s work demonstrates an understanding of basic concepts in mathematics and
connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
The student’s work demonstrates an incomplete understanding of basic concepts in mathematics
and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.

The student’s work demonstrates limited understanding of basic concepts in mathematics and
inaccurate connections among central ideas. The student’s responses demonstrate minimal ability
to solve problems. Problem solving strategies may be flawed or inappropriate and there may be
many omissions.
• indicating or labeling collections of • identifying measurement tools

equivalent sets • copying and/or extending patterns
• reading, writing, and counting • collecting data
numbers • identifying the outcome of an
• recognizing place value event
• skip counting
• comparing two items based on
multiple attributes
Appropriate performance at grade three is clarified by the level of complexity within the
standard.

Proficient

many omissions.
• ordering and comparing numbers • identifying congruent figures

• identifying two-digit numbers • identifying and using measurement
• solving addition and subtraction tools
problems involving one-digit • copying and extending simple
numbers patterns
• identifying 2-D shapes • collecting, organizing, and/or
• comparing two items based on interpreting data
multiple attributes • identifying the outcome of an
event

Proficient

many omissions.
• matching and/or identifying coins

• decimal notation
• comparing two items based on multiple attributes
• identifying and using measurement tools
• copying and extending patterns

Proficient

many omissions.
• indicating, comparing, and/or ordering rational numbers limited to fractions with

denominators of 2, 3, 4, and/or 5
• identifying and/or classifying 2-D shapes and/or angles
• matching quantities that are equal
• finding the value that will make an open sentence true
Appropriate performance at grade six is clarified by the level of complexity within the standard.

Proficient

many omissions.
• ordering and comparing whole numbers

• using measurement tools and estimating outcomes
• computing equivalencies
• using more, less, equal, and/or other comparisons to analyze data or solve problems
Appropriate performance at grade seven is clarified by the level of complexity within the
standard.
Proficient with Distinction:

Proficient

many omissions.
Achievement Level Definitions for Mathematics ~ High School
Mathematics achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate mathematics standards. These definitions serve as the
foundation for achievement level definitions for the alternate grade level expectations in mathematics.
• solving multi-step addition, subtraction, • simplifying and writing linear

multiplication and/or division problems algebraic expressions
involving whole numbers, fractions, • collecting, arranging, interpreting,
decimals, percents and/or ratios and/or analyzing data to formulate or
• describing or illustrating the justify conclusions, make predictions
relationships between the four or solve problems
operations • using more, less, equal, and/or other
• copying, extending, and describing comparisons to analyze data or solve
patterns problems,
• writing rules for finding specific cases • using measures of central tendency or
of a linear or nonlinear relationship range to analyze situations or solve
• finding the value that will make an open problems
sentence true
• representing unknown quantities with
letters
Appropriate performance at the high school level is clarified by the level of complexity within the
standard.

The student’s work demonstrates an understanding of essential concepts in mathematics, including the
ability to make connections among central ideas. The student’s responses demonstrate the ability to utilize
information and solve problems including implementing strategies, accurately performing procedures and
providing solutions.
Proficient
The student’s work demonstrates an understanding of basic concepts in mathematics and connections
among central ideas. The student’s responses demonstrate basic ability to solve problems, including
performing procedures and providing solutions. The student’s work may contain minor errors.
The student’s work demonstrates an incomplete understanding of basic concepts in mathematics and
inconsistent connections among central ideas. The student’s responses demonstrate limited ability to solve
problems, including performing procedures and providing solutions. Problem solving strategies may be
flawed and procedures preformed inaccurately.

The student’s work demonstrates limited understanding of basic concepts in mathematics and inaccurate
connections among central ideas. The student’s responses demonstrate minimal ability to solve problems.
Problem solving strategies may be flawed or inappropriate and there may be many omissions.
Achievement Level Definitions for Science ~ Grade 5
Science achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate science standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in science.
• identifying night and day • matching weather to the effects it can

• identifying the Sun have on the Earth’s surface
• identifying the Earth’s Moon • indentifying pictures or descriptions
• identifying the position of the sun at of given animals and plants
different times • indentifying plants and animals, and
• identifying or drawing different phases components of the environments that
of the Moon animals depend on for food and
• identifying weather through observation shelter
• identifying different forms that water
can take in the weather

The student’s work demonstrates an understanding of essential concepts in science, including the
ability to make connections among central ideas. The student’s response demonstrates the ability
to utilize information and solve problems and explain central concepts with clarity and accuracy.
Proficient
The student’s work demonstrates a general understanding of essential concepts in science and
connections among central ideas. The student’s response demonstrates the ability to utilize
information and solve problems and explain central concepts. Student work may contain minor
errors.
The student’s work demonstrates incomplete understanding of essential concepts in science and
inconsistent connections among central ideas. The student’s response demonstrates some ability
to utilize information and solve problem. The quality of the responses is inconsistent. Explanation
of concepts may be incomplete or unclear.

The student’s work demonstrates limited understanding of essential concepts in science and
infrequent or inaccurate connections among central ideas. The student’s responses demonstrate
minimal ability to solve problems. Explanations are illogical, incomplete, or missing. There are
many inaccuracies.
Achievement Level Definitions for Science ~ Grade 8
• identifying or demonstrating ways • identifying parts that allow living

objects can move things to meet basic needs
• identifying that an object’s motion can • identifying that some living things are
be changed by pushing or pulling made of one cell and some are made
• identifying or describing wave motions, of many cells
earthquakes, vibrations, and/or water • identifying parents and offspring
waves • demonstrating an understanding of
• identifying human body parts life cycles, and/or identifying similar
• matching animals and/or plants to their and different characteristics of
parts offspring and parents
Appropriate performance at grade eight is clarified by the level of complexity within the standard.

Proficient
errors.

many inaccuracies.
Achievement Level Definitions for Science ~ High School
• sorting objects into groups using • describing ways in which the needs of
physical properties a plant and/or animal are met by its
• describing physical properties of objects environment
and materials • sorting living things based on external
• using observable characteristics to features
describe physical changes • matching organisms to the
• identifying chemical and physical environment in which they live
changes • identifying organisms that once lived
• identifying organisms that are similar on Earth but no longer exist
and different based on external features • identifying examples of fossils and/or
• describing how plants and/or animals explaining how fossils are used to
look help us understand the past
Appropriate performance at the high school level is clarified by the level of complexity within the
standard.

Proficient
errors.

many inaccuracies.
Achievement Level Definitions for Writing ~ Grade 4
Writing achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate writing standards. These definitions also serve
as the foundation for achievement level definitions for the alternate grade level expectations in
writing.
• identifying pictures or symbols to relate an experience, event, or idea

• composing responses
• using pictures to create an understandable story line with a beginning and end
• using pictures to identify and/or create characters

The student’s performance demonstrates understanding of essential concepts in writing. The
student’s work demonstrates the ability to compose a response that is well-organized, accurate
and focused.
Proficient
The student’s performance demonstrates an understanding of basic concepts in writing. The
student’s work demonstrates the ability to compose a response that is organized, accurate and
focused. Some errors may occur but do not interfere with meaning.
The student’s performance demonstrates an incomplete understanding of basic concepts in
writing. The student’s work demonstrates the ability to compose a response that may be
inconsistent and/or limited in its organization, accuracy and/or focus. Some errors may occur that
interfere with meaning.

The student’s performance demonstrates limited understanding of basic concepts in writing. The
student’s work demonstrates the ability to compose a response that is unorganized, inaccurate
and unfocused. Frequent errors may occur that interfere with meaning.
Achievement Level Definitions for Writing ~ Grade 7
writing.
• composing and sharing related • including details or information

responses to convey needs relevant to topic
• representing facts through pictures • using a given organizational
• using pictures to illustrate details or structure for grouping facts and/or
information related to topic ideas
• sorting or grouping facts and/or • using sufficient details or pictures
ideas within a given category to illustrate facts
• using pictures and/or words to create • using basic transition words
meaning • providing a concluding sentence
Appropriate performance at grade seven is clarified by the level of complexity within the
standard.

and focused.
Proficient

Achievement Level Definitions for Writing ~ High School
writing.
• using phonemic awareness and letter • applying basic capitalization and

knowledge to represent initial or punctuation rules
final consonant sounds, • correctly spelling high frequency
• writing recognizable phrases or short words
sentences to show understanding of • writing a variety of simple
text sentences
• using prior knowledge or references • recognizing or applying English
to text to respond to questions or spelling rules
when reading • stating and maintaining focus
• using a beginning and an ending to when responding to questions
organize ideas
Appropriate performance at the high school level is clarified by the level of complexity within
the standard.

and focused.
Proficient

Appendix B—AGENDAS
Appendix B—Agendas 1 2010 Maine PAAP Standard Setting Report

MAINE ALTERNATE ASSESSMENT STANDARD SETTING
June 28 & 29, 2010
AGENDA
Day 1: Monday, June 28th
1st Grade
All panelists together

8:00 – 8:30 Registration & continental breakfast
8:30 – 8:45 Welcome from Maine’s Department of Education (MDOE)
Introduction of MDOE staff
Introduction of Measured Progress staff
8:45 – 10:00 Overview of the Maine Alternate Assessment
Overview of Standard Setting Process
Panelists break into their respective workgroup rooms
10:00 – 10:15 Break; panelists move to their grade level/content area workgroup rooms
10:15 – 12:00 Introductions of facilitator and panelists
Review of PAAP materials (AGLEs and student samples)
Review and discuss Achievement Level Descriptors
Training Evaluation
12:00 – 12:45 Lunch
Panelists return to their respective workgroup room
12:45 – 4:00 Rounds 1, 2 & 3
Procedural Evaluation
Break as needed (approximately 2:00 pm or between rounds)
Day 2: Tuesday, June 29th

2nd Grade

7:30 – 8:00 Continental breakfast
8:00 – 8:45 Review process, answer questions and check for understanding
8:45 – 12:00 Review of PAAP materials (AGLEs and student samples)
Round 1
Begin Round 2

12:00 – 12:45 Lunch
12:45 – 3:30 Continue with Rounds 2 & 3

3:30 – 4:00 Cross grade panel
Final Evaluation

MAINE ALTERNATE ASSESSMENT STANDARD SETTING
June 30, 2010
AGENDA
7:30 – 8:00 Registration/room assignments & continental breakfast
8:00 – 8:45 Introductions of facilitator and panelists

Review process, answer questions and check for understanding
8:45 – 12:00 Review of PAAP materials (AGLEs and student samples)
Round 1
Begin Round 2
12:00 – 12:45 Lunch
12:45 – 4:00 Continue with Rounds 2 & 3

Final evaluation

Appendix C—NONDISCLOSURE FORM
Appendix C—Nondisclosure Form 1 2010 Maine PAAP Standard Setting Report

Maine Personalized Alternate Assessment Portfolio
Nondisclosure Agreement for Standard Setting Panelists
The student work associated with the Maine Personalized Alternate Assessment
Portfolio (PAAP) is confidential material. As such, the student portfolio may not
be copied, shared, or discussed for any reason other than to score the student
work. It is the policy of the Maine Department of Education that student portfolios
be treated as private and secure material.
The undersigned is a PAAP In-State Standard Setting participant authorized to

view PAAP material and hereby agrees to be bound to the terms of this
agreement restricting the disclosure of said materials.
Name (printed)
Name (signature)
Date
Appendix C—Nondisclosure Form 3 2010 Maine PAAP Standard Setting Report

Appendix D—S AMPLE RATING FORM
Appendix D—Sample Rating Form 1 2010 Maine PAAP Standard Setting Report
Round: ID Number:
Maine PAAP Reading, Grade 2

SBP PP P PWD
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Appendix D—Sample Rating Form 3 2010 Maine PAAP Standard Setting Report
Appendix E—S AMPLE EVALUATION
Appendix E—Sample Evaluation 1 2010 Maine PAAP Standard Setting Report

Content Area:
Grade:
Standard Setting Training Evaluation

The purpose of this evaluation form is to obtain your feedback about the training you have received.
Please complete the information below. Do not put your name on the form. We want your feedback
to be anonymous.
Please mark the appropriate box for each statement.
Undecided
Disagree
Disagree
Strongly
Strongly
Agree
Agree
I understand the goals of the standard setting meeting. □ □ □ □ □
I understand the procedures we are using to set standards. □ □ □ □ □
I understand how to use the standard setting materials. □ □ □ □ □
I understand the differences between the achievement levels. □ □ □ □ □
I understand how to make the cut score judgment. □ □ □ □ □
I know what tasks to expect for the remainder of the meeting. □ □ □ □ □
I am confident in my understanding of the standard setting task. □ □ □ □ □
Please indicate any areas in which you would like more information before you continue.
Please indicate any questions you may have about the remainder of the standard setting meeting.

Content Area:
Grade:
Standard Setting Procedural Evaluation
Undecided
Disagree
Disagree
Strongly
Strongly
Agree
Agree
I understood how to make the cut score judgments. □ □ □ □ □
I understood how to use the materials provided. □ □ □ □ □
I understood how to record my judgments. □ □ □ □ □
I think the procedures make sense. □ □ □ □ □
I am sufficiently familiar with the assessment. □ □ □ □ □
I understand the differences between the achievement levels. □ □ □ □ □
Please rate the influence of the following when setting standards.
Extremely
influential
influential
Not at all
The achievement level descriptors. □ □ □ □ □

The state content standards. □ □ □ □ □
My perception of the difficulty level of the assessment. □ □ □ □ □
The student responses. □ □ □ □ □
My experience working with students. □ □ □ □ □
What materials, information, or procedures were most influential in your placement of the cut scores?
Why?

Content Area:
Grade:
Standard Setting Final Evaluation

Please complete the information below. Your feedback will provide a basis for evaluating the training,
methods, and materials. Do not put your name on the form. We want your feedback to be anonymous.
Gender: Male □ Female□
Race/ethnicity: White □ Black □ Hispanic □ Asian □ Pacific Islander □ □
American Indian
Years of experience in education: 0-5 □ 5-10 □ 10-15 □ More than 15 □
Area of Expertise (Check all that apply): Students with Disabilities □
Students with Limited English Proficiency □
Economically Disadvantaged Students □
Gifted and Talented Students □
General Education □
Undecided
Disagree
Disagree
Strongly
Strongly
Agree
Agree
I understood the goals of the standard setting meeting. □ □ □ □ □
I understood the procedures we used to set standards. □ □ □ □ □
The facilitator helped me understand the process. □ □ □ □ □
The materials contained the information needed to set standards. □ □ □ □ □
I understood how to use the materials provided. □ □ □ □ □
The achievement level descriptors were clear. □ □ □ □ □
I understood how to make the cut score judgments. □ □ □ □ □
I understood how to use the feedback provided after each round. □ □ □ □ □
I understood how to use the impact data. □ □ □ □ □
I understood how the cut scores were calculated. □ □ □ □ □
The facilitator was able to get answers to my questions. □ □ □ □ □
Sufficient time was allotted for training on the standard setting tasks. □ □ □ □ □
Sufficient time was allotted to complete the standard setting tasks. □ □ □ □ □

Content Area:
Grade:
The facilitator helped the standard setting process run smoothly. □ □ □ □ □
Please rate the usefulness of each of the following:
Extremely
Not at all
useful
useful
The opening session. □ □ □ □ □
The small group activities. □ □ □ □ □
Becoming familiar with the assessment. □ □ □ □ □
Articulating the differences between the achievement levels. □ □ □ □ □
Discussions with other participants. □ □ □ □ □
Please rate the influence of the following when setting standards.
Extremely
influential
influential
Not at all
The achievement level descriptors. □ □ □ □ □

My expectations of students. □ □ □ □ □
The difficulty of the test materials. □ □ □ □ □
The student responses. □ □ □ □ □
My experience in the field. □ □ □ □ □
Discussions with other participants. □ □ □ □ □
Cut scores of other participants. □ □ □ □ □
Impact data. □ □ □ □ □
Please provide any additional comments about the standard setting process or suggestions as to how the
training and process could be improved.

APPENDIX F—OPENING SESSION POWERPOINT
Appendix F—Opening Session PowerPoint 1 2010 Maine PAAP Standard Setting Report
Slide 1
Maine’s Personalized
Alternate Assessment
Portfolio (PAAP)
Standard Setting
Slide 2
Introductions
Maine’s Department of Education
- Susan Fossett, PAAP and Accommodations
Coordinator
- Peter Bernard, PAAP Assistant
Measured Progress (Program Mgt.)

- Susan Izard, Division Director
- Sharon Houle, PAAP Program Manager
- Stephanie Arroyo, PAAP Specialist
- Michelle Couture, PAAP Program Assistant
Slide 3
Introductions (cont.)
Data
- Liz Burton, Psychometrician

- Jennifer Dunn, Psychometrician
- Kevin Froton, Data Processing
Slide 4
Introductions (cont.)
Facilitators (Monday and Tuesday)
Grade 2/3 Reading: Amanda Breitmaier

Grade 2/3 Mathematics: Alicia Cuttle
Grade 4/5 Reading: Jaime Alford
Grade 4/5 Mathematics: Stephanie Arroyo
Grade 6/7 Reading: Tim Greenlaw
Grade 6/7 Mathematics: Susan Izard
Grade 10/11 Reading: Jake Goldsmith
Grade 10/11 Mathematics: Kristen Cole
Slide 5 Introductions (cont.)

Facilitators (Wednesday):
Science
Grade 5: Alicia Cuttle
Grade 8: Stephanie Arroyo
Grade 11: Kristen Cole
Writing
Grade 4: Amanda Breitmaier
Grade 7: Susan Izard
Grade 11: Jake Goldsmith
Slide 6
Agenda
8:30 – 10:00 am General Session

10:00 am – 12:00 pm Breakout Rooms
12:00 – 12:45 pm Lunch
12:45 – 4:00 pm Breakout Rooms
Breaks as needed in Breakout Rooms

Slide 7
What is the PAAP?
The PAAP is
- designed for students with significant cognitive
disabilities who meet participation criteria;
- administered during the “teaching year” at the
same grade levels as students in general
education are assessed in October via NECAP;
- a collection of student work;
- assessed during the PAAP assessment window
December 1st through April 30th;
Slide 8 What is the PAAP?

The PAAP is
- tasks that teachers download to assess
students at an instructionally appropriate time;
- tasks selected by teachers that best fit their
students (using the test blueprint as a guide);
- scored using the PAAP Scoring Rubric to
obtain student achievement levels which are
then used to determine reportable scores; and
- reflective of input from an instructional team.
Slide 9
2009-10 Test Blueprint
Slide 10
Visual Guide to PAAP Requirements
Slide 11 Steps to Administer a PAAP

STEP 1: The teacher/IEP team determines a student is
eligible to participate for each content area.
STEP 2: The teacher reviews the Alternate Grade Level
Expectations (AGLE) document to determine which
AGLE/Indicator and Level of Complexity (LoC) is
appropriate from the eight LoCs.
STEP 3: The teacher selects/downloads tasks from the
PAAP Task Bank (including graphics/passages). STEP
4: The teacher administers the tasks to the student. STEP
5: The teacher corrects/grades the student work and
provides details regarding the level of assistance
provided to the student.
Slide 12
Slide 13
Slide 14
Slide 15
Slide 16
Slide 17
Slide 18 What does a Task look like?
1. Entry Slip
2. Task Description
3. Student Work (template)

and graphics
4. Task Summary
Slide 19 The Entry Slip
• Teacher fills in Student
Name and Grade.
• The Content Area,
AGLE, Level of
Complexity (LoC), and
the AGLE/Indicator is
pre-populated.
• By design, text provided
in gray shaded cells is
information for the
teacher and is pre-
populated.
• Areas with white or no
shading indicates the
teacher should fill in the
missing information.
Slide 20 Task Description Page

• The Task Description
page repeats the
Content Area,
AGLE/Indicator, LoC
information.
• The Task Title, AGLE Page
#, Task # has been added
to the top identifying
information.
• Prior Knowledge,
Description of Task,
Directions for Task
Administration, and
Responses Expected are
provided to ease teacher
administration.
Slide 21 Work Template Page

• LoC 1 Work
Template Pages
are always a chart
similar to this one.
• LoC 1 for ELA

(Reading & Writing)
always includes
color images.
How the Item 1 is Presented to the
Slide 22 Student
Slide 23 Option 2 – Always Available at LoC 1

• Option 2 allows for fewer
of the item sets to be used
multiple times when
necessary to match the
level of student knowledge.
• The items must be selected
from those already provided
in the Task.
• Option 2 allows for the use of
a combination of selected
sets that total six items (e.g.,
two sets each used on 3
different occasions…or
they can use the same set
on 6 different occasions).
Slide 24 Liam 12/11/10
C
sleeping
sleeping sleeping
smiling
X
sleeping
sleeping smiling
s miling
Option 2
C
sleeping
sleeping sl eeping
smiling
C
eating
person laughing eating eating
C
eating
eating
person laughing eating
eating
eating X
laughing
person laughing
Slide 25 The Completed Work Template
1. Teacher fills in Liam 12/11/10
student name, date,

and student
C
responses.
X
C
2. Teacher corrects
C
student responses
C
using the Responses
X
Expected from
Student section on
the Task Description.
Slide 26 Scoring Level of Accuracy

1. Determine the Level of Accuracy based on
the corrected student work and Data Key.
– Point values are predetermined. NO PARTIAL POINTS
unless noted.
– % are pre-calculated.
Task 1 Summary
Liam 12/11/10
67
Slide 27 Scoring Level of Assistance
Unscorable:
● Hand-over-Hand
● Altering Items/Tasks (task no longer connects to
the AGLE.)
Level of Assistance Score of 1:

● Modeling
● Demonstrating a response similar to the desired
response

• Use of Option 2 (LoC 1 only) to use fewer of the
item sets multiple times to match student
knowledge
• Limiting a student’s response (outside of LoC 1 at

Option 2) by removing one response option
• Use of clarifying questions to stimulate student

thought to the specific task without providing clues
to specific answers
• Independent
• Providing encouragement
• Completing Task by using
augmentative/alternative means of
communication
• Repeating directions
• Reacting to a student
• Rereading a passage (except for required
reading)
• Reminding a student to stay focused
2. Determine Level of Assistance based

upon criteria outlined in the PAAP
Administration Handbook.
X .
Liam required modeling of Item 1 to understand what was being

asked of her. He proceeded without further assistance.
3.The bottom of the Task Summary is a reminder to
the teacher if the Student Work, Level of Accuracy,
and Level of Assistance boxes are not completed
and submitted together by Task, the Task becomes
UNSCORABLE.
Slide 32 Tasks 2 and 3

• Usually at LoC 1, Tasks 2 and/or 3 are a repeat
of Task 1 with different pictures, with the same
concept being assessed.
• As you move to higher LoCs, Tasks 2 and 3

build off Task 1, getting progressively more
difficult/complex. The skills may be different
depending on what is required of the LoC.
• Reading and Writing require 3 Tasks per AGLE

Entry.
• Mathematics and Science require 2 Tasks per

AGLE Entry.
Slide 33 What does the PAAP Assess?
√ Level of Complexity
√ Level of Accuracy
√ Level of Assistance
Slide 34 Who scored the PAAPs?
● Measured Progress scorers scored 2,024
PAAPs in early May using the PAAP Task
Scoring Rubric. Each PAAP was scored twice
and sometimes a third time if scores between
Scorer #1 and #2 were not an exact match.
● Measured Progress provided training.
● All scorers passed a qualifying test.
● MDOE staff were present at scoring and
available to answer questions.
Slide 35 PAAP Task

Scoring Rubric
Slide 36 A final reminder…
☺ PAAPs were scored by two scorers,

sometimes three.
☺ Do not attempt to rescore the samples
your reviewing!
☺ You may disagree about the order of the
PAAPs; that’s fine.
☺ You need to stay focused on the task at
hand!
Slide 37
Now the process…
Slide 38 Today’s Training

In today’s session we will cover:
1. an overview of standard setting;
2. details of the Body of Work process as it
will be implemented for the PAAP; and
3. your role in this process.
Note:
This session is intended to be an overview.
Your facilitator will give you more details and

will guide you through the process step by
step.
Slide 39 Logistical Overview

Monday/Tuesday Wednesday
Reading 2/3 Science 5
Reading 10/11
Writing 4
Math 2/3 Writing 7
Math 4/5 Writing 11
Math 6/7
Math 10/11
Slide 40
Overview of Standard Setting
Slide 41
Content Achievement
vs.
Standards Standards
Content standards (AGLE) = “What”
Describe the knowledge and skills students are expected to
demonstrate by content area and grade span
Achievement standards (e.g., Proficient) =

“How well”
Describe attributes of student performance based on Achievement
Level Descriptors
Slide 42
What is Your Job?
To recommend cut scores for each of the

achievement levels that will be used to report
PAAP results:
Proficient
Slide 43 We are trying to determine
What knowledge, skills, and abilities (KSAs)

need to be demonstrated to be classified in
each achievement level?
How much is enough?
What portfolio evidence corresponds to:

Proficient
Slide 44
Achievement Continuum
Substantially Partially Proficient Proficient

Below Proficient with
Proficient Distinction
Slide 45
Based on Achievement Level
Descriptors, you will recommend cut
scores…
Cut score Cut score Cut score

needed needed needed
Substantially Partially Proficient Proficient

Below Proficient with
Proficient Distinction
Achievement Continuum
Slide 46
General Phases of Standard Setting
Data-collection
Policy-making/Decision-making
Slide 47
Final Recommendations
Your recommendations will be reviewed and

presented to the policy makers, who are
responsible for final adoption of the cut
scores.
The recommendations may be accepted or
modified by
the Maine Department of Education including the Commissioner of
Education and the Technical Advisory Committee.
Slide 48
Overview of Standard Setting

Method
Slide 49
Cut Score Recommendations
Provide data to establish the

following cut scores:
Cut Score
Cut Score
Proficient
Cut Score
Slide 50
How: The Body of Work Method
Examine student work and make a judgment

regarding the achievement level to which the
student work most closely corresponds.
Student Work Samples (portfolios or PAAPs)
30 to 40 student PAAPs
Your job is to classify each portfolio into the

achievement level in which you feel it
belongs.
Slide 51 Why the Body of Work?
Allows panelists to use samples of actual student

work to make their determinations
Is especially useful for assessments that consist
primarily or entirely of performance-based
items
Has been used successfully for setting standards
on similar assessments in the past
Has resulted in defensible cut points
Slide 52
General Process
Classify each portfolio into

one of 4 achievement
levels based on the
following:
Achievement Level Descriptors

KSAs measured by the portfolios
How the students performed on the portfolios
Slide 53 Before you start classifying portfolios….
You will need to become familiar with:

√ Alternate Grade Level Expectations (AGLEs)
√ Achievement Level Descriptors
What each level means
Identify the knowledge, skills and abilities
necessary to be classified in each level
√ Student PAAPs/portfolios
Understand the knowledge, skills and
abilities demonstrated in the work samples
Slide 54
Achievement Level Descriptors
Individual review of Achievement Level Descriptors
Group discussion of what performance in each

achievement level looks like
Create bulleted lists of

The knowledge, skills, and abilities a student must demonstrate to be
classified in each achievement level
The knowledge, skills, and abilities that distinguish one achievement
level from another
You must reach consensus as a group about

the KSAs that define student performance at
each achievement level.
Slide 55
Student Portfolios
You will classify 30 to 40 student portfolios.

The portfolios cover the range of possible total
scores and are presented in order from lowest
(e.g., Sample #1) to highest (e.g., Sample #35)
total raw score.
Each portfolio has been selected because it
shows typical types of evidence submitted for
students who received a given total score.
Slide 56 Your Task
Think about a student who demonstrates the

KSAs for each level.
Classify each portfolio into the level you feel it
belongs:
Proficient
Slide 57
Slide 58
Slide 59
Please Note:
You may disagree about the order of the

portfolios; that’s fine.
You will categorize the portfolios as you see

fit, whether your ratings agree with the order
or not.
However, it is not your job to rescore the

portfolios; you need to stay focused on the
task at hand.
Slide 60
Round 1
Working Individually:
Review each portfolio
Focus on the knowledge, skills and abilities being
demonstrated in the portfolio
Determine which Achievement Level Descriptor best
matches the knowledge, skills and abilities
Classify the portfolio into the appropriate
achievement level
Complete the rating form
Slide 61
Round 2
Working as a Group:
Discuss your portfolio classifications in relation to
The average round 1 results
The other panelists
The knowledge, skills and abilities
achievement level
Slide 62
Round 3
Working as a Group:
Discuss your portfolio classifications in relation to
The round 2 results & impact data
The other panelists
The knowledge, skills and abilities
achievement level
Slide 63 A few reminders

It is not necessary for panelists to reach
consensus as to how the portfolios should be
categorized.
You should be open-minded when listening to
your colleagues’ rationales for their ratings.
You may or may not change your mind as a result
of the discussions.
We want each panelist to use his or her own best
judgment in each round of rating.
Slide 64
Cross-Grade Policy Forum
(Tuesday afternoon)
After all groups have completed Round 3 for the

second grade level, the groups for each
content area (reading and mathematics) will
meet together to look at results across grades
and provide feedback.
Slide 65
Evaluation
At several different points in the process, we

will ask you to complete an anonymous
evaluation of the standard setting procedures.
Your honest feedback is important for

improving future standard settings, and for
evaluating the results of this one.
Slide 66
Questions about the Body of Work

Method?
Slide 67
Before you break into groups…
Slide 68 Top 10 Misconceptions About

Standard Setting
10. Standard setting is a great opportunity to
review and revise the PAAP.
9. Standard setting is the same thing as scoring.
8. This is a good time to discuss PAAP

administration policy.
7. This is a good time to revise the content

standards.
6. This is a good time to revise the PAAP
Achievement Level Descriptors.
Slide 69
Top 10 Misconceptions About Standard
Setting
5. This is a good time to discuss effective

teaching strategies.
4. Only scholars and researchers are qualified
to do this work.
3. Only educators are qualified to do this work.
2. The process is rigged.
1. Disagreement is bad.
Slide 70
What Next?
☺ Some meeting logistics
☺ After this session, you will break into grade
groups and complete the standard setting
process!
– First grade/content
• Review the portfolios
• Discuss the Achievement Level Descriptors
• Round 1, 2, 3
– Repeat for second grade/content
– Cross grade policy forum
– Evaluation
☺ Some lucky folks will

– Repeat for third grade/content
Slide 71
Room Assignments
Please refer to your hotel map with

room assignments in your folder.
Appendix G—F ACILITATORS’ SCRIPTS
Appendix G—Facilitators’ Scripts 1 2010 Maine PAAP Standard Setting Report

GENERAL INSTRUCTIONS FOR GROUP FACILITATORS
MAINE PAAP STANDARD SETTING
READING AND MATHEMATICS
JUNE 28-30, 2010
Overview
The Reading and Mathematics groups will each be setting standards for two grade levels. The
panels will complete the standard setting activities for the first grade level – discussing the
Achievement Level Descriptors and completing the three rounds of ratings – then will repeat the
entire process for the second grade level. For the first grade level, the panelists will complete two
evaluation forms: a training evaluation before starting round 1 and a procedural evaluation after
round 3. For the second grade level, the panelists will not need to fill out either of these
evaluations; instead, the panelists will complete the final evaluation after the Cross Grade Policy
Forum, which occurs at the very end of the process.
Introductions
1) Welcome group, introduce yourself (name, affiliation, a little selected background

information).
2) Have each participant introduce him/herself.
3) Ask each participant to sign a nondisclosure form. Do not proceed until a signed
nondisclosure form has been collected from each participant.
Complete Standard Setting Activities for First Grade Level
Review Materials
1) Have the panelists take a few minutes to briefly look through the AGLEs
2) Have the panelists briefly review about every fifth portfolio, noting the increasing level of
performance.
3) When they are done, allow a minute or two for comments or questions.
4) Familiarize the panelists with the rating sheet and explain how to complete it:
a. Place one (and only one) “X” in each row
b. They can place the X’s at the low or high end of the box, but they must clearly be
within one box: no straddling!
Discuss Achievement Level Descriptors
Overview: In order to establish a thorough understanding of the expected performance of students

on the test, panelists must have a clear understanding of:
1) The definitions of the four achievement levels, and

2) what the key characteristics are that distinguish students in adjacent achievement level
categories.
The purpose of this activity is for the panelists to come to consensus about what characterizes
students in each of the four achievement level categories. This activity is critical since the ratings
panelists will be making in Rounds 1 through 3 will be based on these understandings.
Activities:
1. Introduce task. In this activity they will:
a. Individually review the Achievement Level Descriptors that describe the main
characteristics that define students in each achievement level category; and
b. discuss Descriptors as a group.
2. Have panelists individually review the Achievement Level Descriptors for each level. They
can make notes if they like. The goal here is for the panelists to come to a common
understanding of what it means to be in each achievement level. It is not unusual for
panelists to disagree with the Descriptors they will see; almost certainly there will be some
panelists who will want to change them. However, the task at hand is for panelists to have a
common understanding of what knowledge, skills, and abilities are described by each
Achievement Level Descriptor.
3. After individually reviewing the Descriptors, have the panelists discuss each achievement
level as a group, starting with Partially Proficient. The panelists will discuss the
characteristics a student must demonstrate in order to be classified in the Partially Proficient
category. Or, put another way, the most important characteristics that distinguish a
Substantially Below Proficient student from a student in the Partially Proficient category. They
will then repeat this process for the Proficient and Proficient with Distinction categories. The
purpose of this step is to have a collegial discussion in which to bring up/clarify any issues
or questions that any individual may have and to reach consensus on an understanding of
the Descriptors.
4. Have the panelists identify the most important characteristics describing students at each
achievement level and record those as bulleted lists on chart paper. These should be posted
on the walls for panelists to refer to as they complete the three rounds of rating.
Training Evaluation (First Grade Level Only)
After completing the discussion of the Achievement Level Descriptors for the first grade level, have
panelists fill out the training evaluation form before proceeding to Round 1. Before you start the
Round 1 activities, scan the completed evaluations to see if there are any problems or concerns that
need to be addressed before proceeding. Return the completed evaluations to the data analysis
room at the next convenient opportunity. It is not necessary to complete the training evaluation
form for the second grade level.
Round 1 Ratings
Overview of Round 1: The primary purpose of Round 1 is to ask the panelists to make their initial
determination as to which achievement level category each portfolio should be classified into. In
this round, panelists will be working individually, without discussion with their colleagues.
Activities:
1. Make sure panelists have the following materials:
a. Rating form
b. set of portfolios
c. Achievement Level Descriptors
2. Orient panelists to the set of portfolios. Point out that the portfolios are presented in order,
from lowest scoring to highest. Make sure panelists understand that, even though the
portfolios are presented from lowest- to highest-scoring, their own ratings do not need to be
in strictly increasing order.
3. Starting with the first portfolio, the panelists will review each portfolio in turn. As they are
reviewing the portfolios, the panelists should keep in mind the Achievement Level
Descriptors. They should consider the knowledge, skills, and abilities demonstrated by
each and how they relate to the definitions of the achievement levels. The purpose of this
step is for panelists to make their initial determinations as to how the portfolios should be
categorized into the four achievement levels. The panelists are free to make notes on the
portfolios, sort them into piles, use sticky notes, or use whatever system helps them to keep
track of their categorizations.
4. Panelists may want to take notes as they work if there are particular points they would like
to discuss with their colleagues in Round 2.
a. Have panelists write their ID and round number on the rating form. The ID number
is on their name tags.
b. Briefly remind them how to fill in the rating form.
c. Answer questions the panelists may have about the work in Round 1.
d. Once everyone understands what they are to do in Round 1, tell them to begin.
5. As panelists complete the task, ask them to carefully inspect their rating forms to ensure
they are filled out properly.
a. The ID and round number must be filled in.
b. Each portfolio must be assigned to one and only one achievement level.
c. Reiterate that although the portfolios are presented in order from lowest- to highest-
scoring, the panelists’ category assignments do not need to be in strictly increasing
order.
6. Facilitators should bring all the completed rating forms together to the data analysis work
room for tabulation. Prior to submitting them, however, using a show of hands, indicate on
a piece of chart paper how many panelists assigned each portfolio to each achievement level
category. This chart will be used for the Round 2 discussions.
Tabulation of Round 1 Results

Tabulation of Round 1 results will be completed as quickly as possible after receipt of the rating
forms. While the tabulation occurs, the group may take a break.
Round 2 Ratings
Overview of Round 2: In Round 2, the panelists will have an opportunity to discuss their Round 1
placements and to revise their ratings on the basis of that discussion. Prior to beginning the Round
2 discussions, the psychometrician will share the group average cut points based on the Round 1
ratings.
Focusing on any portfolios for which there was disagreement as to how they should be categorized,
the panelists will discuss why they categorized each portfolio as they did, making sure that all
different points of view are included in the discussion.
Once panelists have reviewed and discussed the Round 1 categorizations, the panelists will make
their Round 2 ratings.
Activities:
a. Rating form
2. Have panelists write their ID number and Round 2 on the rating form.
3. Provide an overview of Round 2. Paraphrase the following:
a. As in Round 1, the primary purpose is to categorize each portfolio into the
achievement level category where you believe it belongs.
b. Each panelist needs to base his/her judgments on his/her experience with the
content area, understanding of students, the definition of each achievement level
category, discussions with other panelists, and the knowledge, skills, and abilities
required to answer each item.
4. Show the panelists how the portfolios would be categorized based on the room average
Round 1 cut point placements.
5. Remind panelists that they will be discussing each portfolio with their colleagues, but that
they will be categorizing the portfolios individually. It is not necessary for the panelists to
reach consensus about how to categorize each portfolio.
6. Give panelists an opportunity to ask questions about the task for Round 2.
7. Beginning with the first portfolio for which there is disagreement as to how it should be
categorized, the panelists should begin discussing the categorization of the portfolios
according to the Round 1 ratings.
a. Panelists only need to discuss those portfolios for which there was disagreement as
to how they should be categorized.
b. Panelists should be encouraged to listen to their colleagues as well as express their
own points of view.
c. If the panelists hear a logic/rationale/argument that they did not consider and that
they feel is compelling, then they may adjust their ratings to incorporate that
information.
d. On the basis of the discussions and the feedback presented, panelists should make
their round 2 ratings.
e. The group does not have to achieve consensus. If panelists honestly disagree, that is
fine. We are trying to get the best judgment of each panelist. Panelists should not feel
compelled or coerced into making a rating with which they disagree.
Encourage the panelists to use the discussion and feedback to assess how stringent or
lenient a judge they are. If a panelist is categorizing portfolios consistently higher or
lower than the group, he or she may have a different understanding of the Achievement
Level Descriptors than the rest of the group. It is acceptable for panelists to disagree,
but that disagreement should be based on a common understanding of the
8. When the group has completed their Round 2 ratings, collect the rating forms. When you
collect the rating forms, carefully inspect them to ensure they are filled out properly.
b. Each portfolio must have one (and only one) rating.
Round 3 Ratings
Overview of Round 3: In Round 3, the panelists will have a final opportunity to discuss their
Round 2 placements and to revise their ratings on the basis of that discussion. Prior to beginning
the Round 3 discussions, the psychometrician will share the Round 2 results with the group,
including the group average cut points and impact data, i.e., the approximate percentage of
students who would be classified into each achievement level category based on the room average
cut points from Round 2.
Once panelists have reviewed and discussed the Round 2 categorizations, they will make their final
ratings.
Activities:
a. Rating form
4. Review the feedback information with the panelists.
a. Show the panelists how the portfolios would be categorized based on the room
average Round 2 cut point placements.
b. Go over the impact data, explaining that if the Round 2 ratings were to be used to set
the final cut points, these are the approximate percentages of students who would be
classified into each achievement level category.
6. Give panelists an opportunity to ask questions about the feedback information or about the
task for Round 3.
a. Panelists should discuss the portfolios for which there was disagreement as to how
they should be categorized, focusing in particular on thoe portfolios around the cuts.
own points of view.
information.
their final ratings.
8. When the group has completed their final ratings, collect the rating forms. When you collect
the rating forms, carefully inspect them to ensure they are filled out properly.
b. Each portfolio for Round 3 must have one (and only one) rating.
c. Return the completed rating forms to the data analysis work room.
Complete Procedural Evaluation

After the panelists have completed the standard setting activities for the first grade level, have them
complete the procedural evaluation. Submit the completed evaluations to the data analysis work
room at the earliest convenient opportunity.
Complete Standard Setting Activities for Second Grade Level
After the panelists have completed the three rounds of ratings and filled in the procedural
evaluation, they will then repeat the standard setting activities (except the training and procedural
evaluations) for the second grade level: discussing the Achievement Level Descriptors and the
three rounds of ratings.
Complete Final Evaluation Form

Following the cross grade panel, have panelists fill out the final evaluation form. Emphasize that
their honest feedback is important.
GENERAL INSTRUCTIONS FOR GROUP FACILITATORS
MAINE PAAP STANDARD SETTING
SCIENCE AND WRITING
JUNE 30, 2010
Introductions
4) Welcome group, introduce yourself (name, affiliation, a little selected background

information).
5) Have each participant introduce him/herself.
6) Ask each participant to sign a nondisclosure form. Do not proceed until a signed
nondisclosure form has been collected from each participant.
Review Materials
5) Have the panelists take a few minutes to briefly look through the AGLEs
6) Have the panelists briefly review about every fifth portfolio, noting the increasing level of
performance.
7) When they are done, allow a minute or two for comments or questions.
Discuss Achievement Level Descriptors
Overview: In order to establish a thorough understanding of the expected performance of students

on the test, panelists must have a clear understanding of:
3) The definitions of the four achievement levels, and

4) what the key characteristics are that distinguish students in adjacent achievement level
categories.
The purpose of this activity is for the panelists to come to consensus about what characterizes
students in each of the four achievement level categories. This activity is critical since the ratings
panelists will be making in Rounds 1 through 3 will be based on these understandings.
Activities:
5. Introduce task. In this activity they will:
c. Individually review the Achievement Level Descriptors that describe the main
characteristics that define students in each achievement level category; and
d. discuss Descriptors as a group.
6. Have panelists individually review the Achievement Level Descriptors for each level. They
can make notes if they like. The goal here is for the panelists to come to a common
understanding of what it means to be in each achievement level. It is not unusual for
panelists to disagree with the Descriptors they will see; almost certainly there will be some
panelists who will want to change them. However, the task at hand is for panelists to have a
common understanding of what knowledge, skills, and abilities are described by each
Achievement Level Descriptor.
7. After individually reviewing the Descriptors, have the panelists discuss each achievement
level as a group, starting with Partially Proficient. The panelists will discuss the
characteristics a student must demonstrate in order to be classified in the Partially Proficient
category. Or, put another way, the most important characteristics that distinguish a
Substantially Below Proficient student from a student in the Partially Proficient category. They
will then repeat this process for the Proficient and Proficient with Distinction categories. The
purpose of this step is to have a collegial discussion in which to bring up/clarify any issues
or questions that any individual may have and to reach consensus on an understanding of
the Descriptors.
8. Have the panelists identify the most important characteristics describing students at each
achievement level and record those as bulleted lists on chart paper. These should be posted
on the walls for panelists to refer to as they complete the three rounds of rating.
Round 1 Ratings
Overview of Round 1: The primary purpose of Round 1 is to ask the panelists to make their initial
determination as to which achievement level category each portfolio should be classified into. In
this round, panelists will be working individually, without discussion with their colleagues.
Activities:
a. Rating form
8. Orient panelists to the set of portfolios. Point out that the portfolios are presented in order,
from lowest scoring to highest. Make sure panelists understand that, even though the
portfolios are presented from lowest- to highest-scoring, their own ratings do not need to be
in strictly increasing order.
9. Starting with the first portfolio, the panelists will review each portfolio in turn. As they are
reviewing the portfolios, the panelists should keep in mind the Achievement Level
Descriptors. They should consider the knowledge, skills, and abilities demonstrated by
each and how they relate to the definitions of the achievement levels. The purpose of this
step is for panelists to make their initial determinations as to how the portfolios should be
categorized into the four achievement levels. The panelists are free to make notes on the
portfolios, sort them into piles, use sticky notes, or use whatever system helps them to keep
track of their categorizations.
10. Panelists may want to take notes as they work if there are particular points they would like
to discuss with their colleagues in Round 2.
a. Have panelists write their ID and round number on the rating form. The ID number
is on their name tags.
b. Briefly remind them how to fill in the rating form.
c. Answer questions the panelists may have about the work in Round 1.
d. Once everyone understands what they are to do in Round 1, tell them to begin.
11. As panelists complete the task, ask them to carefully inspect their rating forms to ensure
they are filled out properly.
b. Each portfolio must be assigned to one and only one achievement level.
c. Reiterate that although the portfolios are presented in order from lowest- to highest-
scoring, the panelists’ category assignments do not need to be in strictly increasing
order.
Tabulation of Round 1 Results

Tabulation of Round 1 results will be completed as quickly as possible after receipt of the rating
forms. While the tabulation occurs, the group may take a break.
Round 2 Ratings
Overview of Round 2: In Round 2, the panelists will have an opportunity to discuss their Round 1
placements and to revise their ratings on the basis of that discussion. Prior to beginning the Round
2 discussions, the psychometrician will share the group average cut points based on the Round 1
ratings.
Focusing on any portfolios for which there was disagreement as to how they should be categorized,
the panelists will discuss why they categorized each portfolio as they did, making sure that all
different points of view are included in the discussion.
Once panelists have reviewed and discussed the Round 1 categorizations, the panelists will make
their Round 2 ratings.
Activities:
d. Rating form
e. set of portfolios
f. Achievement Level Descriptors
13. Show the panelists how the portfolios would be categorized based on the room average
Round 1 cut point placements.
15. Give panelists an opportunity to ask questions about the task for Round 2.
f. Panelists only need to discuss those portfolios for which there was disagreement as
to how they should be categorized.
g. Panelists should be encouraged to listen to their colleagues as well as express their
own points of view.
h. If the panelists hear a logic/rationale/argument that they did not consider and that
information.
i. On the basis of the discussions and the feedback presented, panelists should make
their round 2 ratings.
j. The group does not have to achieve consensus. If panelists honestly disagree, that is
17. When the group has completed their Round 2 ratings, collect the rating forms. When you
collect the rating forms, carefully inspect them to ensure they are filled out properly.
c. The ID and round number must be filled in.
d. Each portfolio must have one (and only one) rating.
Round 3 Ratings
Overview of Round 3: In Round 3, the panelists will have a final opportunity to discuss their
Round 2 placements and to revise their ratings on the basis of that discussion. Prior to beginning
the Round 3 discussions, the psychometrician will share the Round 2 results with the group,
including the group average cut points and impact data, i.e., the approximate percentage of
students who would be classified into each achievement level category based on the room average
cut points from Round 2.
Once panelists have reviewed and discussed the Round 2 categorizations, they will make their final
ratings.
Activities:
a. Rating form
12. Review the feedback information with the panelists.
a. Show the panelists how the portfolios would be categorized based on the room
average Round 2 cut point placements.
b. Go over the impact data, explaining that if the Round 2 ratings were to be used to set
the final cut points, these are the approximate percentages of students who would be
classified into each achievement level category.
14. Give panelists an opportunity to ask questions about the feedback information or about the
task for Round 3.
a. Panelists should discuss the portfolios for which there was disagreement as to how
they should be categorized, focusing in particular on thoe portfolios around the cuts.
own points of view.
information.
their final ratings.
16. When the group has completed their final ratings, collect the rating forms. When you collect
the rating forms, carefully inspect them to ensure they are filled out properly.
b. Each portfolio for Round 3 must have one (and only one) rating.
c. Return the completed rating forms to the data analysis work room.
Complete Final Evaluation Form

After they complete Round 3, have panelists fill out the final evaluation form. Emphasize that their
honest feedback is important.
APPENDIX H—PANELIST AFFILIATIONS
Appendix H—Panelist Affiliations 1 2010 Maine PAAP Standard Setting Report

Panelist Affiliations
Name Title District/school June 28-29 June 30
Science
Adams, Lynne Asst SPED Director Augusta/Cony Math 2/3
8
Armstrong, Linda Special Educator RSU 1/Fisher Mitchell Reading, 2/3 Writing 4
Writing
Barron, Carla Special Education Tchr Old Orchard Beach/OOB High School Math HS
11
Belanger, Amanda Union 107/Woodland Elementary School Reading 4/5 Writing 4
Belisle, Mary Math Tchr/Content Specialist SAD 51/Greely Middle School Math 6/7
Science
Boucher, Anne Special Education Tchr RSU 34/Southern Penobscot Regional Program Math 6/7
8
Science
Butler, Frances Special Education Tchr Brewer/Brewer High School Math HS
11
Writing
Carr, Deb English Teacher SAD 55/Sacopee Valley High School Reading HS
11
Clark, Jill Special Education Tchr RSU 2/Richmond Middle School Math 6/7 Writing 7
Writing
Clemons, Janet Teacher MSAD #15/Gray New Gloucester High School Reading 6/7
11
Cobb, Patricia Special Education Tchr RSU #11/River View Elementary School Reading 4/5 Writing 4
Cole, Emily Elementary Teacher RSU 29/Wellington School Math 2/3
Cole, Sally Reading Interventionist RSU 29/Houlton Elementary School Reading, 2/3
Writing
Coleman, B David English Teacher MSAD 15/Gray-New Gloucester High School Reading HS
11
Science
Connolly, Kathy Special Education Tchr RSU # 26/Glenburn School Math 6/7
8
Science
Connolly, Shelley Special Education Tchr SAD #4/SAD #4 Elementary School Math 4/5
5
Science
Corbett, Terras Ed Tech 3 Brewer/Brewer High Math HS
11
Cornett, Marla Special Education Tchr Lewiston Public Schools/Farwell Math 2/3 Writing 4
Science
Dawson, Daryl Ed Tech 111 Brewer/Brewer High Math HS
11
Dock, Heidi Special Education Tchr SAD #17/Oxford Elementary Reading, 2/3 Writing 4
continued

Drysdale, Rebekah Special Education Tchr Auburn/Walton Reading 4/5 Writing 7
Science
Dunn, Julie HS SPED Resource Rm MSAD#29/Houlton Reading HS
11
Earnhardt, Marge Teacher Gov. Baxter School f/t Deaf Reading 6/7
Frati, Alice Special Education Tchr Bangor School Department/Mary Snow School Math 4/5
Science
Genovese, Katie School Counselor MSAD 15/Gray-New Gloucester Reading HS
11
Granger, Sheree Special Education Tchr Sweetser Reading, 2/3
Science
Hargrove, Jesse Special Education Tchr A.O.S. 92/Winslow High School Reading HS
11
Hartley, Julie Resource Room Teacher RSU 11/River View Community School Math 4/5 Writing 7
Hayes, Priscilla Special Education Tchr Auburn/Fairview Reading, 2/3
Writing
Hayes, Steve Coordinator Special Srvcs Easton School Dept./Easton schools Math 6/7
11
Herrick, Janet General Education Tchr SAD 4/Carroll L. McKusick Elementary Reading, 2/3 Writing 7
Hodgkins, Susan Special Services Director RSU #37/MSAD #37/N/A Reading 6/7
ME Educational Center f/t Deaf & Hard of Science
Howard, Deborah Teacher of the Deaf Reading 4/5
Hearing/Governor Baxter School for the Deaf 5
Science
Howard, Linda MSAD 41/Milo Elementary Math 2/3
5
Lisbon School Department/Lisbon Community
Inman, Penny Reading, 2/3
School
Science
Kelley, Debbie Special Education Tchr MSAD 37/Narraguagus High School Math 6/7
8
Science
Lavalle-Rivera, Juan RSU Science 11 Math 6/7
8
Writing
Lessard, Robyn Special Education Tchr RSU 24/Ellsworth High School Reading HS
11
Luginbuhl, Ann Resource Room Teacher Union 104/Charlotte School Math 4/5
MSAD 14/Union 108/Vanceboro Elementary
Malone, Sean Grade 3-8 Teacher Reading 6/7 Writing 7
School
McCormick, Kelly Asst Professor of Math University of Southern Maine Math 6/7
continued

Medway School Department/Medway Middle Science
Mitchell, Barbara Special Education Tchr Math 4/5
School 5
Calais School Dept - Union 106/Calais High
Moody, Lyndon John Math HS
School
Science
Mullis, Deborah Math 2/3
5
Science
Oceipka, Gail Reeds Brook Middle School Reading 6/7
8
O'Neill, Kathryn Speech Language Pathologist RSU 1/ Fisher-Mitchell Reading 4/5 Writing 4
Peaslee, Kimberly Special Education Tchr MSAD 15/Gray and New Gloucester Reading HS Writing 7
Pelletier, Deborah Special Educator Acton Elementary School Reading 6/7 Writing 4
Science
Penner, Nancy Ed Tech Brewer/Brewer Middle School Math HS
11
Perry, Heather Plant, Special Educator - Autism Lisbon Community School Reading, 2/3 Writing 4
Narda Pomerleau, Special Education Tchr SAD 20/Fort Fairfield Elem. School Math 2/3
Rosemarie Pulkkinen, Special Education Tchr Scarborough/Blue Point Reading, 2/3 Writing 7
Kerri Randall, Lenora First Grade Teacher SAD 4/Carroll L. McKusick Elementary Reading, 2/3 Writing 4
Reed, Paula Special Education Tchr SAD 4/PCHS Math HS
Title 1 Teacher Portland/East End Community School Reading, 2/3
Science
Rehill, Kathy Math 2/3
5
Writing
Robbins, Barbara District Evaluator - SPED MSAD 74/Solon Elementary Reading 6/7
11
Robert, Cheryl Special Education Tchr Lewiston/Farwell Math 2/3 Writing 4
Saponara, Diana First Grade Teacher SAD 4/Carroll L. McKusick Elementary Reading 4/5 Writing 4
Sawyer, Jane Special Education Admin. NA/Spring Harbor Academy Math 4/5 Writing 7
Seiler, Scott Special Education Tchr MSAD #42/Central Aroostook Jr/Sr High Reading 6/7
Sewell, Jill Special Education Tchr MSAD #70/Mill Pond School Reading 4/5
Writing
Shardlow, Naomi Retired SPED Director Reading HS
11
continued

Science
Skillin, Sarah Special Education Tchr Lewiston/Farwell Math 4/5
5
RSU #15/MSAD #!5/Gray New Gloucester High
Stokes, Kelly Special Educator Math HS
School
Science
Thurber, Jacqueline Special Education Tchr RSU #24/Sumner Memorial High School Math HS
8
Totman, Alice SPED Tchr - Case Mgr RSU 52/Tripp Middle School Math 6/7 Writing 7
Technology Integration
Tucker, Amy RSU 16 Math 6/7 Writing 7
Specialist
Writing
Viere, Janet Special Educator Auburn School Department/Walton School Reading 4/5
11
Writing
Vigneault, Rita RSU #19/Nokomis High School Math 4/5
11
Portland Public Schools/East End Community
Winslow, Susan Title 1 Teacher Reading 4/5
School

APPENDIX I—EVALUATION RESULTS
Appendix I—Evaluation Results 1 2010 Maine PAAP Standard Setting Report

Mathematics Grades 2 and 3
Mathematics Grade 2 Training Evaluation
N Mean % SD %D %U %A % SA
I understand the goals of the standard
8 4.38 0 0 0 62 38
setting meeting.
I understand the procedures we are using
8 3.88 0 0 25 62 12
to set standards.
I understand how to use the standard
8 4.12 0 0 0 88 12
setting materials.
I understand the differences between the
7 3.71 0 0 29 71 0
performance levels.
I understand how to make the cut score
7 3.43 0 0 57 43 0
judgment.
I know what tasks to expect for the
7 3.29 0 14 43 43 0
remainder of the meeting.
I am confident in my understanding of the
8 3.62 0 0 38 62 0
standard setting task.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Cut Scores? Have we discussed that?
I think I understand – not sure
Mathematics Grade 2 Procedural Evaluation

I understood how to make the cut score
8 4.12 0 0 0 88 12
judgments.
I understood how to use the materials
8 4.25 0 0 0 75 25
provided.
I understood how to record my judgments. 8 4.25 0 0 0 75 25
I think the procedures make sense. 8 3.62 0 25 12 38 25
I am sufficiently familiar with the
8 4.25 0 0 12 50 38
assessment.
8 4 0 12 12 38 38
performance levels.
 Very few of the tasks lend themselves to establishing extension to show PWD.
 Still struggle with PAAPs incorrectly scored either inaccuracy or in levels of assistance.
N Mean %Low %High

The performance level descriptors. 8 4 0 0 12 75 12
The state content standards. 8 4.25 0 0 0 75 25
My perception of the difficulty level of the
7 4.14 0 0 14 57 29
assessment.
The student responses. 8 4.62 0 0 0 38 62
My experience working with students. 8 4.5 0 0 12 25 62

What materials, information, or procedures were most influential in your placement of the cut scores?
Why?
 Examples included PAAPs with 4 standards- 4th was counted in standard setting and shouldn’t have
been.
 Examples included PAAPs with only 1 task force for a standard- how can you determine
growth/knowledge with only 1 task?
 Note: please check Level of Accuracy Data Key, many were incorrect which made the teacher %
incorrect. For instance no 2/3 choice.
 The PAAPs themselves. My knowledge of PAAPs. Panel discussions.
 Rubric created by our group.
 Student work vs. Actual score listed as there were many with incorrectly scored items.

Mathematics Grades 2 and 3 Final Evaluation
Panelist Demographics N=8
Male 0
Gender
Female 8
White 8
Black 0
Hispanic 0
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 2
Years of
5–10 0
experience in
10–15 2
education
More than 15 4
Students with Disabilities 8
Students with Limited English
0
Area of Expertise Proficiency
(check all that Economically Disadvantaged
2
apply) Students
Gifted and Talented Students 0
General Education 2
I understood the goals of the standard
8 4.38 0 0 0 62 38
setting meeting.
I understood the procedures we used to
8 4 0 0 25 50 25
set standards.
The facilitator helped me understand the
8 4.25 0 0 12 50 38
process.
The materials contained the information
8 3.38 0 38 12 25 25
needed to set standards.
8 4.12 0 0 12 62 25
provided.
The performance level descriptors were
8 4.25 0 0 12 50 38
clear.
8 3.88 0 12 25 25 38
judgments.
I understood how to use the feedback
8 4.12 0 12 0 50 38
provided after each round.
I understood how to use the impact data. 7 4 0 0 14 71 14
I understood how the cut scores were
8 3.62 12 12 0 50 25
calculated.
The facilitator was able to get answers to
8 4.38 0 0 12 38 50
my questions.
Sufficient time was allotted for training on
8 4.25 0 0 12 50 38
the standard setting tasks.
Sufficient time was allotted to complete
8 4.12 0 12 0 50 38
The facilitator helped the standard setting
8 4.62 0 0 0 38 62
process run smoothly.

Usefulness of N Mean %Low %High
The opening session. 8 3.38 12 0 38 38 12
The small group activities. 8 4.38 0 0 0 62 38
Becoming familiar with the assessment. 8 4.12 0 0 12 62 25
Articulating the differences between the
8 3.88 0 0 25 62 12
performance levels.
Discussions with other participants. 8 4.62 0 0 0 38 62
Influence of N Mean %Low %High

My expectations of students. 8 3.75 0 0 25 75 0
The difficulty of the test materials. 8 4.12 0 0 12 62 25
My experience in the field. 8 4.12 0 0 25 38 38
Cut scores of other participants. 8 3.62 0 12 25 50 12
Impact data. 7 3.14 0 29 29 43 0
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
 Examples should include those with correctly scored items and correctly identified level of
accuracy marked because we were told they were corrected when Measured Progress scored
but we didn’t have that info.Examples shouldn’t have missing tasks or zeros ( at the upper
limits) makes more difficult to give accurate reading limits) makes more difficult to give
accurate reading. It was unique to see I know that no highest level will create the need for
policy level discussion.
 Samples had numerous errors-scoring errors, level of assistance errors, task with scores
below 33 on LOC 4 tasks at “top” of samples (30+) at gr. 3
 Better quality of examples. Felt pressured to raise some students’ standings for better cut
scores.
 Use tests that are scored correctly. Provide raw scores.

Mathematics Grade 4 Training Evaluation
8 4.75 0 0 0 25 75
setting meeting.
8 4.75 0 0 0 25 75
to set standards.
8 4.62 0 0 0 38 62
setting materials.
8 4.12 0 0 0 88 12
performance levels.
8 4.12 0 0 0 88 12
judgment.
8 4.62 0 0 12 12 75
8 4.5 0 0 0 50 50
Please indicate any questions you may have about the remainder of the standard setting
meeting.

Mathematics Grade 4 Procedural Evaluation
16 4.38 0 0 0 62 38
judgments.
16 4.56 0 0 0 44 56
provided.
16 4.62 0 0 0 38 62
assessment.
16 4.44 0 0 0 56 44
performance levels.
meeting.
N Mean %Low %High

The performance level descriptors. 13 4.54 0 0 15 15 69
My perception of the difficulty level of the 13 3.69 15 0 23 23 38
assessment.
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
 None were more influential than others
 Level of complexity, Level of accuracy, level of assistance
 Student responses +my experience
 Student work & core skills assessed for grade level

Male 0
Gender
Female 8
White 8
Black 0
Hispanic 0
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 0
Years of
5–10 1
experience in
10–15 0
education
More than 15 7
1
3
apply) Students
General Education 3
8 4.62 0 0 0 38 62
setting meeting.
8 4.62 0 0 0 38 62
set standards.
8 4.88 0 0 0 12 88
process.
8 4.62 0 0 0 38 62
8 4.62 0 0 0 38 62
provided.
8 4.62 0 0 0 38 62
clear.
8 4.62 0 0 0 38 62
judgments.
8 4.75 0 0 0 25 75
I understood how to use the impact data. 8 4.5 0 0 0 50 50
8 4.5 0 0 0 50 50
calculated.
8 4.88 0 0 0 12 88
my questions.
8 4.62 0 0 0 38 62
8 4.25 0 12 12 12 62
8 4.88 0 0 0 12 88

8 4.62 0 0 0 38 62
performance levels.

Impact data. 8 4.12 0 0 25 38 38
 Sample sizes leading to accuracy was different and impacted decisions.
 Super Facilitator

Mathematics Grades 6 and 7 Training Evaluation

10 4.8 0 0 0 20 80
setting meeting.
10 4.8 0 0 0 20 80
to set standards.
10 4.7 0 0 10 10 80
setting materials.
10 4.8 0 0 0 20 80
performance levels.
10 4.1 0 0 30 30 40
judgment.
9 4.67 0 0 0 33 67
10 4.5 0 0 0 50 50
 Cut score further explained
 Just need to put process to work.
meeting.
 I believe I will learn what further questions I have from doing the actual work.
 None at this time.

Mathematics Grades 6 and 7 Procedural Evaluation
10 4.6 0 0 10 20 70
judgments.
10 4.8 0 0 0 20 80
provided.
I understood how to record my judgments. 10 5 0 0 0 0 100
10 4.5 0 0 10 30 60
assessment.
10 4.5 0 0 0 50 50
performance levels.
meeting.
N Mean %Low %High

The state content standards. 10 4 0 0 20 60 20
10 4.2 0 0 10 60 30
assessment.
My experience working with students. 10 4 0 0 20 60 20
scores? Why?
 The criteria the group set for the steps of the rubric. This help keep me focus when personal
bias came in to play
 The combination of LOC & LoA w/ & of accuracy as well as overall & outcome.
 Achievement level descriptors helped me to better see subtle difference between levels.
 Descriptors
 The student PAAP samples and seeing their work (Loc, level of accuracy and level of
assistance). Combining that info with the descriptors helped me place each PAAP.
 Defining more specifically the levels of proficiency using the Maine PAAP definitions.

Panelist Demographics N = 10
Male 2
Gender
Female 8
White 9
Black 0
Hispanic 1
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 0
Years of
5–10 3
experience in
10–15 0
education
More than 15 7
0
4
apply) Students
General Education 6
10 4.9 0 0 0 10 90
setting meeting.
10 4.8 0 0 0 20 80
set standards.
10 4.9 0 0 0 10 90
process.
10 4.8 0 0 0 20 80
10 4.8 0 0 0 20 80
provided.
10 4.6 0 0 0 40 60
clear.
10 4.7 0 0 0 30 70
judgments.
10 4.7 0 0 0 30 70
10 4.6 0 0 0 40 60
calculated.
10 4.8 0 0 0 20 80
my questions.
10 4.9 0 0 0 10 90
10 4.9 0 0 0 10 90
10 4.9 0 0 0 10 90

The small group activities. 10 5 0 0 0 0 100
10 4.8 0 0 0 20 80
performance levels.

The student responses. 9 5 0 0 0 0 100
Impact data. 10 4.1 0 10 10 40 40
 Maybe the intro sessions with lots of repetition + “you’ll learn more about this in small
groups later” could be shorter
 Enjoyed the process very much

Mathematics Grades 10 and 11 Training Evaluation
9 4.22 0 0 11 56 33
setting meeting.
9 4.22 0 0 11 56 33
to set standards.
9 4.22 0 0 11 56 33
setting materials.
9 4.11 0 0 11 67 22
performance levels.
9 4 0 0 22 56 22
judgment.
9 3.89 0 11 11 56 22
9 3.67 0 11 22 56 11
 Not a question, but a comment, I would have began the achievement levels discussion with
the proficient level instead of the substantially below level proficiency.
 How to apply these cuts to the tasks
 A bit more time with the materials and I’ll have it I’m almost there.
 Let’s go through one together.
 Have learned much so far
meeting.
 I don’t know if I will be able to do this successfully. It seems a little confusing.
 I think if we did the first one together, everyone would be on the same page about what we’re
looking for.
 Am visual – will feel more confident as one is actually done.

Mathematics Grades 10 and 11 Procedural Evaluation
9 4.11 0 0 11 67 22
judgments.
9 4.22 0 0 0 78 22
provided.
9 4.22 0 0 0 78 22
assessment.
9 4.22 0 0 0 78 22
performance levels.
meeting.
Would have liked to do practice task first.
N Mean %Low %High

9 4.33 0 0 0 67 33
assessment.
The student responses. 9 4 0 0 22 56 22
scores? Why?
 Level descriptions
 The descriptions of skills (prior knowledge) in the task descriptions. How specific do the ach.
level descriptions need to be followed when determining the standard?
 Difficulty level and content standards
 The grade level complexities and how the students performed on those tasks were most
influential to me.
 LOC mainly as it loosely corresponded to the skills in the 4 levels.

Male 2
Gender
Female 7
White 9
Black 0
Hispanic 0
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 0
Years of
5–10 2
experience in
10–15 3
education
More than 15 4
2
3
apply) Students
General Education 2
9 4.11 0 11 0 56 33
setting meeting.
9 4 0 11 0 67 22
set standards.
9 4.22 0 11 0 44 44
process.
9 3.78 0 0 33 56 11
9 4.44 0 0 0 56 44
provided.
9 3.89 0 0 33 44 22
clear.
9 4.33 0 0 0 67 33
judgments.
9 4.33 0 0 0 67 33
9 3.78 0 11 11 67 11
calculated.
9 4.33 0 0 0 67 33
my questions.
9 3.89 0 22 0 44 33
9 3.89 0 11 22 33 33
9 4.44 0 0 0 56 44

9 3.78 0 11 11 67 11
performance levels.

Impact data. 9 3.56 0 11 33 44 11
 I didn’t like the SBP PP P PD skills we listed. I found it was easier to go by LOC’s and
student scores along with the need for assistance.
 I think a sample task would be very helpful to go over before starting the evaluations within
your own smaller group.
 Process reversed to have open discussion for example of standard setting expectations.

Reading Grades 2 and 3
Reading Grades 2 and 3 Training Evaluation
18 4.17 0 0 11 61 28
setting meeting.
18 3.94 0 0 22 61 17
to set standards.
18 3.78 0 0 39 44 17
setting materials.
18 3.83 0 0 33 50 17
performance levels.
18 3.5 0 0 61 28 11
judgment.
18 3.94 0 0 28 50 22
18 3.56 0 0 56 33 11
 Is Proficiency based on how well a student did on this particular assessment or upon where
he should be?
 Difference between achievement levels. More about cut scores
 Not sure how the indicators we established are helpful when scoring level ½ assessments.
meeting.
 I just wanted to have the conversation with group to see if we are seeing it in a similar
manner.
 Note: It is helpful to have done standard setting before.
 How does substantially below proficient and partially proficient affect AYP?
 Can a student be considered proficient if they received 100/83 at locs?

Reading Grades 2 and 3 Procedural Evaluation
11 4.18 0 0 0 82 18
judgments.
11 4.36 0 0 0 64 36
provided.
11 3.91 0 9 18 45 27
assessment.
11 4.09 0 0 9 73 18
performance levels.
meeting.
N Mean %Low %High

11 4.18 0 0 9 64 27
assessment.
scores? Why?
 Understanding of the student’s body of work, level of complexities. I know how difficult it
is for students to complete these tasks and try to give them the benefit of the doubt.
 Level of complexity
 Level of assistance and achievement level descriptors.
 Great facilitators- questions were answered.

Reading Grades 2 and 3 Final Evaluation
Male 0
Gender
Female 11
White 11
Black 0
Hispanic 0
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 0
Years of
5–10 1
experience in
10–15 4
education
More than 15 6
1
9
apply) Students
General Education 5
11 4.36 0 0 0 64 36
setting meeting.
11 4.45 0 0 0 55 45
set standards.
11 4.36 0 0 9 45 45
process.
11 4.45 0 0 0 55 45
11 4.36 0 0 0 64 36
provided.
11 4.36 0 0 0 64 36
clear.
11 4.27 0 9 0 45 45
judgments.
11 4.36 0 0 9 45 45
11 4 0 9 18 36 36
calculated.
11 4.55 0 0 0 45 55
my questions.
11 4.45 0 0 0 55 45
11 4.45 0 0 0 55 45
11 4.64 0 0 0 36 64

10 4.5 0 0 0 50 50
performance levels.

My experience in the field. 11 4 0 0 27 45 27
Impact data. 11 4 0 0 18 64 18

9 4.22 0 0 11 56 33
setting meeting.
9 4.11 0 0 11 67 22
to set standards.
9 4 0 0 11 78 11
setting materials.
8 4.12 0 0 0 88 12
performance levels.
9 3.33 0 0 78 11 11
judgment.
9 4.11 0 0 11 67 22
9 3.78 0 0 33 56 11
 As a reg. ed Teacher I wish I had prepared myself more on PAAPs because I have realized
that I really knew very little. Thank you for the brief introduction.
 The training provided was sequential and with a clearly stated outcome. At this point, I need
to get my feet wet with the process. Maybe ?’s will arise as I am completing the task.
 Example of how to score demonstrated would be helpful on first task as a group.
meeting.

9 4.11 0 0 11 67 22
judgments.
9 4.33 0 0 0 67 33
provided.
9 3.89 0 0 33 44 22
assessment.
9 4.11 0 0 11 67 22
performance levels.
meeting.
The procedures make sense but I feel like we could have had more time. Explanations of
different ability levels would have been helpful for Regular Ed. Teachers.
N Mean %Low %High

9 3.44 0 11 33 56 0
assessment.
scores? Why?
 Discussion- as a reg. ed teacher I expected the students to be able to read for the reading
section but they didn’t have to. Hard not to compare to the NECAP.
 Discussion among facilitator and group; more time should be allowed for this process- felt
very rushed.
 The more info we have about student achievement, the better decisions we can make.
 Discussions were helpful; did feel rushed at times to get through the stack!
 LOC; level of assistance; for successive rounds, listening to others’ feedback
 LOCs and the student work with score sheets; more info (specific info) would have helped
with regards to the level of assistance and actual students responses, not just correct or
incorrect.

 Having the handouts which defined the Achievement Levels as well as the charts with
additional descriptors were useful because I didn’t have to rely on my memory for the info- I
had a reference(s) to go back to.
 I would have appreciated more level of assistance descriptions from teachers. As a teacher
who has given PAAPs I was never made aware of the formula in scoring the different levels
of the LOC and the weight given to the higher levels in scoring.
 The LOC Indicators were very helpful with this process. I also would have like to have more
info for LO Assistance from teachers.

Male 0
Gender
Female 9
White 9
Black 0
Hispanic 0
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 1
Years of
5–10 2
experience in
10–15 1
education
More than 15 5
2
3
apply) Students
General Education 5
9 4.22 0 0 0 78 22
setting meeting.
9 4.22 0 0 11 56 33
set standards.
9 4.11 0 0 11 67 22
process.
9 4 0 0 11 78 11
8 4.25 0 0 0 75 25
provided.
9 4 0 0 11 78 11
clear.
9 3.67 0 0 33 67 0
judgments.
9 4.33 0 0 0 67 33
8 3.12 0 25 38 38 0
calculated.
9 4.44 0 0 0 56 44
my questions.
9 3.89 0 11 11 56 22
9 3.89 0 11 11 56 22
9 4.44 0 0 0 56 44

9 4 0 0 22 56 22
performance levels.

The difficulty of the test materials. 9 4 0 0 11 78 11
Impact data. 9 3 11 0 67 22 0
 Devise some sort of graphic organizer to record notes as we revised the portfolios (to keep
track of our reactions + inputs)

8 4.38 0 0 0 62 38
setting meeting.
8 4.12 0 12 0 50 38
to set standards.
8 4.12 0 0 12 62 25
setting materials.
8 4.38 0 0 0 62 38
performance levels.
8 4.12 0 0 0 88 12
judgment.
8 4.12 0 0 0 88 12
8 3.88 0 12 0 75 12
Cut score
meeting.

8 3.62 0 0 38 62 0
judgments.
8 3.5 0 12 25 62 0
provided.
I understood how to record my judgments. 8 4 0 0 12 75 12
7 4.29 0 0 0 71 29
assessment.
8 3.62 0 12 25 50 12
performance levels.
meeting.
N Mean %Low %High

assessment.
scores? Why?
 Using the LOCs and TLC evidence of growth influences my cut score rating.
 The LOC levels
 It appears as if the Loc’s do not match the tasks. Ex an A1 LOC the student must be using
phoenic awareness/word parts and context clues, yet the task does not require the student
read. I feel this needs to be examined more closely.
 It became obvious during/prior to Round 3 what Measured Progress was “looking for” this
influenced the decisions made causing me to question the validity of this entire process.
 Difficulty of the task along with student responses.

Male 2
Gender
Female 6
White 8
Black 0
Hispanic 0
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 1
Years of
5–10 1
experience in
10–15 0
education
More than 15 6
2
5
apply) Students
General Education 4
8 4 0 0 0 100 0
setting meeting.
8 3.75 0 0 25 75 0
set standards.
8 3.88 0 0 12 88 0
process.
8 3.62 0 0 50 38 12
8 3.88 0 0 12 88 0
provided.
8 3.38 0 0 62 38 0
clear.
7 3.71 0 0 29 71 0
judgments.
8 3.88 0 0 12 88 0
8 3.75 0 0 25 75 0
calculated.
8 3.75 0 0 25 75 0
my questions.
8 4.25 0 0 0 75 25
8 4.25 0 0 0 75 25
8 3.88 0 0 25 62 12

Articulating the differences between the 8 3.75 0 0 25 75 0
performance levels.
Providing additional details to the 2 4 0 0 0 100 0
performance level descriptors.

Discussions with other participants. 8 4 0 0 12 75 12
Impact data. 8 3.62 0 12 12 75 0
 The achievement level definitions made this process difficult. They do not match the tasks,
therefore made this process mute, in my opinion.

7 4.57 0 0 0 43 57
setting meeting.
7 3.86 0 0 14 86 0
to set standards.
6 4 0 0 0 100 0
setting materials.
7 4 0 0 14 71 14
performance levels.
7 3.71 0 0 29 71 0
judgment.
7 4.29 0 0 0 71 29
7 3.86 0 0 29 57 14
meeting.

7 4.43 0 0 0 57 43
judgments.
7 4.71 0 0 0 29 71
provided.
7 4.57 0 0 0 43 57
assessment.
7 4.43 0 0 0 57 43
performance levels.
meeting.
 (referring to “I think the procedures make sense”; subject checked “agree”): This seems
counter!
 Subjectivity?
N Mean %Low %High

assessment.
scores? Why?
 Group discussion about student work. Initial setting of achievement indicators.
 Fantastic process.
 Group discussions and skills at each level.
 Achievement Level Descriptors, definitions of the four levels- handout

7 4.57 0 0 0 43 57
judgments.
7 4.71 0 0 0 29 71
provided.
7 4.43 0 0 0 57 43
assessment.
7 4.29 0 0 0 71 29
performance levels.
meeting.
N Mean %Low %High

7 4.57 0 0 0 43 57
assessment.
scores? Why?
 Excellent Process
 Description of 4 levels SBP,PP,P,PWD + achievement level descriptors
 The discussions and standards
 Student work and discussion about it. Achievement indicators

Science Grade 5
Science Grade 5 Final Evaluation

Male 0
Gender
Female 7
White 7
Black 0
Hispanic 0
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 0
Years of
5–10 1
experience in
10–15 2
education
More than 15 4
1
1
apply) Students
General Education 3
7 4.57 0 0 0 43 57
setting meeting.
7 4.43 0 0 14 29 57
set standards.
7 4.57 0 0 0 43 57
process.
7 4.57 0 0 0 43 57
7 4.57 0 0 0 43 57
provided.
7 4.57 0 0 0 43 57
clear.
7 4.43 0 0 14 29 57
judgments.
7 4.57 0 0 0 43 57
7 4.14 0 0 29 29 43
calculated.
7 4.71 0 0 0 29 71
my questions.
7 4.57 0 0 0 43 57
7 4.57 0 0 0 43 57
7 4.57 0 0 0 43 57

7 4.71 0 0 0 29 71
performance levels.

Impact data. 7 4.29 0 0 14 43 43
 Many teacher still are not following the process-fair amount of teacher error rather than
student error
 It was more difficult than need to due to lack of higher LOC’s at some indicators
 Much better samples than for math!
 ( Referring to “I understood how to make cut score judgments.”) Seriously, I don’t feel I
make cut score judgments
 If this were to be done again, it would be helpful to have a summary sheet for each students
with task , LOL,LOA for all tasks on 1 sheet. This would save a lot of time, also ever
consider going paperless. Some of this could definitely be paperless

Science Grade 8

Male 1
Gender
Female 6
White 6
Black 0
Hispanic 1
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 0
Years of
5–10 0
experience in
10–15 0
education
More than 15 7
0
2
apply) Students
General Education 2
7 4.71 0 0 0 29 71
setting meeting.
7 4.71 0 0 0 29 71
set standards.
7 4.43 0 0 14 29 57
process.
7 4.86 0 0 0 14 86
7 4.86 0 0 0 14 86
provided.
7 4.57 0 0 14 14 71
clear.
7 4.57 0 0 0 43 57
judgments.
7 4.57 0 0 0 43 57
7 4.43 0 0 14 29 57
calculated.
7 4.86 0 0 0 14 86
my questions.
7 4.86 0 0 0 14 86
7 4.86 0 0 0 14 86
7 4.71 0 0 0 29 71

The opening session. 7 4 14 0 0 43 43
7 4.86 0 0 0 14 86
performance levels.
Providing additional details to the
4 5 0 0 0 0 100

My experience in the field. 7 4 0 14 0 57 29
Impact data. 7 4 0 0 14 71 14
 I enjoyed the process very much.
 Hope the blue print and fall trainings can give clarity to next years grade level expectations
and that a full task bank will provide more evaluation options.

Science Grade 11

Male 2
Gender
Female 5
White 7
Black 0
Hispanic 0
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 1
Years of
5–10 1
experience in
10–15 4
education
More than 15 1
1
1
apply) Students
General Education 2
7 4.29 0 0 0 71 29
setting meeting.
7 4 0 0 14 71 14
set standards.
7 4.14 0 0 0 86 14
process.
7 4 0 0 29 43 29
7 4.14 0 0 14 57 29
provided.
7 3.71 0 0 29 71 0
clear.
7 4 0 0 14 71 14
judgments.
7 4.14 0 0 14 57 29
7 4.29 0 0 0 71 29
calculated.
7 4.14 0 0 0 86 14
my questions.
7 4.29 0 0 14 43 43
7 4.43 0 0 0 57 43
7 4.29 0 0 0 71 29

Becoming familiar with the assessment. 7 4 0 0 29 43 29
7 3.86 0 0 14 86 0
performance levels.
1 4 0 0 0 100 0

My expectations of students. 7 4 0 0 29 43 29
The difficulty of the test materials. 7 4 0 0 14 71 14
Impact data. 7 3.86 0 0 29 57 14
 Process between facilitators seemed at times different. Possible that this is due to different
groups? Content areas.

Writing Grade 4
Writing Grade 4 Final Evaluation
Male 0
Gender
Female 10
White 10
Black 0
Hispanic 0
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 1
Years of
5–10 3
experience in
10–15 2
education
More than 15 4
0
6
apply) Students
General Education 3
9 4.44 0 0 0 56 44
setting meeting.
9 4.67 0 0 0 33 67
set standards.
9 4.56 0 0 0 44 56
process.
9 4.56 0 0 0 44 56
9 4.67 0 0 0 33 67
provided.
9 4.67 0 0 0 33 67
clear.
9 4.67 0 0 0 33 67
judgments.
9 4.56 0 0 0 44 56
10 4.2 0 0 20 40 40
calculated.
9 4.67 0 0 0 33 67
my questions.
9 4.56 0 0 0 44 56
9 4.67 0 0 0 33 67
9 4.78 0 0 0 22 78

8 4.88 0 0 0 12 88
performance levels.
4 5 0 0 0 0 100

Impact data. 10 3.9 0 0 30 50 20
 Since I was unfamiliar with what the student was expected to do, it would have been more
helpful to see the tasks before we designed our rubrics.

Writing Grade 7
Male 1
Gender
Female 8
White 9
Black 0
Hispanic 0
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 1
Years of
5–10 2
experience in
10–15 0
education
More than 15 5
0
3
apply) Students
General Education 5
9 4.67 0 0 0 33 67
setting meeting.
9 4.67 0 0 0 33 67
set standards.
9 4.67 0 0 0 33 67
process.
9 4.67 0 0 0 33 67
9 4.67 0 0 0 33 67
provided.
9 4.56 0 0 0 44 56
clear.
9 4.56 0 0 0 44 56
judgments.
9 4.56 0 0 0 44 56
9 4.56 0 0 0 44 56
calculated.
9 4.67 0 0 0 33 67
my questions.
9 4.67 0 0 0 33 67
9 4.67 0 0 0 33 67
9 4.67 0 0 0 33 67

9 4.56 0 0 0 44 56
performance levels.
1 4 0 0 0 100 0

Cut scores of other participants. 9 4 0 11 11 44 33
Impact data. 8 4.25 0 0 12 50 38
 Awesome 3 days! This is the best I have been to in 16 yrs!

Writing Grade 11
Male 2
Gender
Female 6
White 8
Black 0
Hispanic 0
Race/ethnicity
Asian 0
Pacific Islander 0
American Indian 0
0–5 0
Years of
5–10 0
experience in
10–15 1
education
More than 15 7
1
6
apply) Students
General Education 4
8 4.25 0 0 0 75 25
setting meeting.
8 4.25 0 0 0 75 25
set standards.
8 4.38 0 0 0 62 38
process.
8 4 0 0 25 50 25
8 4.25 0 0 0 75 25
provided.
8 4.12 0 0 12 62 25
clear.
8 4.25 0 0 0 75 25
judgments.
8 4.25 0 0 0 75 25
I understood how to use the impact data. 8 4 0 0 25 50 25
8 4 0 12 0 62 25
calculated.
8 4.38 0 0 0 62 38
my questions.
8 4.25 0 0 0 75 25
8 4 0 12 0 62 25
8 4.5 0 0 0 50 50

Becoming familiar with the assessment. 8 4 0 0 12 75 12
8 4.12 0 0 12 62 25
performance levels.
1 4 0 0 0 100 0

Impact data. 8 3.62 0 12 12 75 0
 Would like guidelines of LOA to be present in future.
 Efficiently done; leadership/guidance great help and support
 Jake was great to work with!
 Need more clarity on level of assistance- Scribe/- more info provide on how each teacher did
it.
 I believe that we needed more information regarding the level of assistance. Some teachers
were not clear. Training for this may need to be more specific.

APPENDIX J—POLICY ADJUSTMENTS
Appendix J—Policy Adjustments 1 2010 Maine PAAP Standard Setting Report

MDOE Policy:
Students in grades 2 through 5 have the opportunity to access tasks in levels of complexity 1–4.
Adjusted cuts were made based on the grade 4 cuts as stated above. However, when looking at the adjusted
cuts for grades 2 and 3, it was felt that the adjusted cuts at Proficient were too high.
MDOE reviewed 4–6 student samples around the Partially Proficient /Proficient cut points along with
participant rater sheets and determined that cut scores between Partially Proficient and Proficient for students
in grade 2 should be minimally lowered. The same process occurred for students in grade 3 keeping in mind
that grade 3 should be slightly higher than grade 2 but still minimally lower than grade 4. These new cuts are
reflected in the chart titled “Mathematics Policy Results.”
Writing grade 7 cut scores at Proficient was minimally lowered following the same procedures as
outlined above for mathematics.
Mathematics: Policy Results

Achievement level
PD 61.1 NA 62 69 3.4
Proficient 35 NA 35 61 51.7
2
SBP NA NA 0 21 8.0
PD 61.1 NA 62 69 14.1
Proficient 39 NA 39 61 41.5
3
SBP NA NA 0 21 11.2
PD 61.1 0.4 62 69 20.5
Proficient 41.2 0.3 42 61 42.3
4
Partially Proficient 21.5 0 22 41 28.2
SBP NA NA 0 21 9
PD 66.9 0.6 67 69 33.2
Proficient 51.7 0.3 52 66 31
5
SBP NA NA 0 26 16.6
PD 84.8 1.2 85 99 10.1
Proficient 55.9 1.5 56 84 52
6
SBP NA NA 0 24 13.2
PD 91.3 0.9 92 99 19.8
7
SBP NA NA 0 24 8.2
PD 121.25 NA 122 129 9
Proficient 81.55 NA 82 121 35.4
HS
SBP NA NA 0 32 17.1

Writing: Policy Results
PD 22.0 NA 22 23 19.9
Proficient 14.4 0.1 15 21 28.6
4
SBP NA NA 0 9 29.0
PD 32.0 NA 32 33 15.0
7
SBP NA NA 0 11 13.6
PD 40.9 0.7 41 43 8.2
Proficient 23.8 0.6 24 40 33.6
11
SBP NA NA 0 12 16.4

100
80
Percent of students in category
60
40
20
0
2 3 4 5 6 7 HS
Proficient
Substantially Below Proficient Mathematics: Policy Results
100
80
Percent of students in category
60
40
20
0
4 7 11
Proficient
Writing: Policy Results

Standard Setting Report

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Standard Setting Report

Uploaded by

Copyright:

Available Formats

Standard Setting Report

JUNE 28–30, 2010

Prepared for the Maine Department of Education by:

100 Education Way, Dover, NH 03820 (800) 431-8901

2.2 Selection of Student Portfolios (Bodies of Work)

2.3 Preparation of Materials for Panelists

2.5 Preparation of Instructions for Facilitators Document

2.6 Preparation of Systems and Materials for Analysis During the

2.7 Selection of Panelists

3.3 Review of Assessment Materials

3.5 Training Evaluation

3.6 Round 1 Judgments

3.7 Tabulation of Round 1 Results

Table 3-2. 2010 PAAP Standard Setting: Round 1 Reading

Table 3-3. 2010 PAAP Standard Setting: Round 1 Science

Table 3-4. 2010 PAAP Standard Setting: Round 1 Writing

3.8 Round 2 Judgments

3.9 Tabulation of Round 2 Results

Table 3-5. 2010 PAAP Standard Setting: Round 2 Mathematics

Table 3-6. 2010 PAAP Standard Setting: Round 2 Reading

Table 3-7. 2010 PAAP Standard Setting: Round 2 Science

Table 3-8. 2010 PAAP Standard Setting: Round 2 Writing

3.10 Round 3 Judgments

3.11 Tabulation of Round 3 Results

Table 3-9. 2010 PAAP Standard Setting: Round 3 Mathematics

Table 3-12. 2010 PAAP Standard Setting: Round 3 Writing

4.1 Analysis and Review of Panelists’ Feedback

4.2 Calculation of Recommended Cutpoints

Grade 4 raw score cuts were applied to grade 2 and grade 3.

4.3 Preparation of this Standard Setting Report

References 28 2010 Maine PAAP Standard Setting Report

Appendices 29 2010 Maine PAAP Standard Setting Report

• student initiated communication • reading aloud sight words

Proficient with Distinction

Substantially Below Proficient

• student initiated communication • reading aloud sight words

Proficient with Distinction

Substantially Below Proficient

• student initiated communication • reading aloud sight words

Proficient with Distinction

Substantially Below Proficient

• student initiated communication • reading aloud sight words

Proficient with Distinction

Substantially Below Proficient

• relating of symbols to the objects/ideas • using explicitly stated information

Proficient with Distinction

Substantially Below Proficient

• relating of symbols to the objects/ideas • recognizing a central idea from text

Proficient with Distinction

Substantially Below Proficient

• using phonemic awareness skills • identifying author’s basic message

Proficient with Distinction

Substantially Below Proficient

• matching small collections of equivalent sets

Proficient with Distinction

Substantially Below Proficient

• indicating or labeling collections of • identifying measurement tools

Proficient with Distinction

Substantially Below Proficient

• ordering and comparing numbers • identifying congruent figures

Proficient with Distinction

Substantially Below Proficient

• matching and/or identifying coins

Proficient with Distinction