OCTOBER 2015

The purpose of assessment in staffing is to determine whether the job candidate

possesses the knowledge, skills, abilities or competencies necessary to perform the
job in an effective way. To accomplish that, assessment instruments must be valid,
reliable, fair and defensible.

This document presents best practices on the use, development, administration

and scoring of unsupervised tests to protect the integrity, merit and fairness of
assessment in appointment processes. The primary focus is on unsupervised
knowledge tests taken at a location of the candidate’s choosing, often referred
to as “take-home exams.” Much of the content may apply to other types of
unsupervised assessment and may also vary by test type.
 interpret information from various sources
What is a “take-home exam”?
and generate and communicate thoughts and
> It is a test that is given to candidates ideas.
to complete on their own time,
without supervision, over a specified
period
of time. It is typically assumed to
be open-book.
> It is taken in an unsupervised
environment, usually with
access to reference texts and
materials.
> It can take many forms (including
computer-scored, subjectively
scored, on-line or paper and pencil)
and can
assess a range of different qualifications.
> It typically consists of essay or
constructed response questions. There
are two types of questions:
− Extended response – allows for
lengthier responses from candidates
and permits candidates lots of flexibility
in their responses (e.g., essay); and
− Restricted response – limits the
form and content of candidates’
responses (e.g., short answer,
problem sets).
> Depending on the time limit set,
candidates may be able to work at
their own pace, thus permitting longer
and more elaborate responses to
questions.

Advantages of take-home exams

> The main benefit is that they provide an
opportunity for candidates to
demonstrate their ability to reason,
organize and synthesize knowledge and
convey judgement and higher-order
critical thinking skills in writing.
> They reflect realistic job tasks in which
employees are required to organize and

BEST PRACTICES FOR UNSUPERVISED TESTING 3

> They increase accessibility as information, construct arguments, give
candidates can complete the test at explanations, evaluate ideas and
a time and location of their develop original thoughts/ opinions.
choosing and in their own Take-home exams also provide a way
workspace that is set up to best of testing a candidate’s ability to
meet their needs. By using their communicate in writing and to present
own computers, candidates have their responses in a logical manner.
access to special hardware, > To assess how > Formulate the
software or adaptive technology, as candidates apply question so that
required, such as ergonomic mice their knowledge the task is clearly
and keyboards, screen readers and to a specific defined for
speech-to-text software. situation, context candidates.
> They reduce operational costs or problem (e.g., > Situate the
a case study),
as candidates do not need to question within
rather than
travel and organizations do not a context or
regurgitating
need to reserve testing problem/situati
facts. Take-home
facilities, ship test booklets or on.
exams are not
administer the test. > Use directive verbs
recommended to
assess to provide an
Disadvantages of take-home exams knowledge where indication of the
candidates can types of thinking
> For assessments involving subjective
simply copy and and content
scoring (i.e., ratings) such as essay
paste the correct expected in
questions or case studies, the
answer from responding to the
nature of the scoring can be
the Internet. question. The
unreliable and subject to a number
following directive
of biases (e.g., context effect, halo > To assess
verbs can help to
effect). small groups
focus the candidates
> For extended response formats, of candidates
on the desired
it is time-consuming to because they
approach (click here
appropriately read and score can take an
for additional
candidate responses. extended
directive verbs):
amount of
> The unsupervised testing format time to − Agree/disagree
allows the opportunity for score. − Analyze
unauthorized
collaboration and consultation with − Compare/contras
Guidelines t
others, plagiarism and access to
unauthorized materials. for − Describe
constructing − Illustrate
Best use of take-home exams essay − Justify
questions − Summarize
> To assess analysis, synthesis and
evaluation skills by requiring > Clearly define − Note: Avoid
candidates to organize and what is to be using the term
integrate information, interpret assessed. “Discuss” as

2 Public Service Commission

 this can > Consider words), essay
lead to using several expected questions.
responses restricted format
that are response (i.e., Test security
too broad questions complete
guidelines
or vague. rather than sentence
one or two s) and It is essential to
> It may also
extended point provide explicit
be helpful to
response values instructions about
identify the
questions to for what is and what is
intended
obtain each not permitted.
audience
greater question; The following points
(e.g., general
public, senior sampling of − Indica are provided as
management content. te examples of
). This how information that can
> Avoid giving be included in the
provides candidates gram
a mar, instructions to
a choice as increase the security
context to which sente
to help nce and defensibility of
questions a take-home exam.
guide they should struct
candidat ure, > Inform
answer.
es’ etc. candidates of
This
response will be the
decreases
s. handl confidential
the validity
ed;
and nature of the
and
reliability of test and their
the test − Provi responsibility
because de a to respect it
each time and to take
candidate is limit the test
taking a to honestly.
potentially com
> If access to
different plete
reference
test. the
materials is
work
> Provide clear deemed
(e.g.
directions and appropriate,
, 48
expectations: specify the
hour
types of
− Indicate s, 5
days materials that
the
). may and may
length of
not be
desired > For more consulted
response information, (e.g.,
s (e.g., consult the resources,
less than Checklist for reports and
500 constructing

BEST PRACTICES FOR UNSUPERVISED TESTING 3

 referen
ce
materi
als)
and
the
expect
ation
that all
materi
als
must
be
cited
accordi
ngly.
> If
appropriate
, inform
candidates
that
consultatio
n or
collaboratio
n with
other
candidates
or
individuals
is not
permitted.

2 Public Service Commission

> Have candidates confirm and sign compared to the expected response and
that they have read and given credit or
understood the instructions. partial credit when their responses include
desired elements from the expected
> Include consequences for breach of
guidelines (e.g., invalidation of test
results, termination of job application,
fraud investigation).
The following is an example of take-home
exam instructions given to students at
Harvard University:
> “You may consult your books or other
reference material. You may NOT
consult any person other than (the
instructor) about any aspect of this
exam. The rules of attribution apply to
take-home exams: All sources must be
cited.”
> Students are also required to sign
a compliance affidavit: “I affirm that
I have had no conversation regarding
this exam with any persons other
than (the instructor) and other
authorized persons. Further, I certify
that the attached work represents
my own thinking. Any information,
concepts or
words that originate from other sources
are cited in accordance with (university)
guidelines. I am aware of the serious
consequences that result from improper
discussions with others or from the
improper citation of work that is not
my own.”

Guidelines for scoring essay

questions
> Choose a scoring method with explicit
criteria to increase fairness and ensure
consistency in scoring. One approach is
to prepare an expected response to the
questions in advance, where the major
components of the response are
defined and assigned a point value.
Candidates’ responses can then be

BEST PRACTICES FOR UNSUPERVISED TESTING 5

 response. An overall score for a conducted after the unsupervised
question can then be generated by assessment for those candidates
adding the points received for each who passed the unsupervised
required element. version.
Notifying candidates that their scores on
> Decide in advance how elements of the unsupervised exam will be verified
written communication will be through another independent assessment
handled, such as grammar, method can be a disincentive to
punctuation, organization and flow, unauthorized test-taking behaviour.
use of vocabulary/ terminology and
irrelevant/inaccurate information. > It is tips/ developing-
recommended assignments/exams/exa
> Allow plenty of time for scoring. that essay m- preparation,
Often, responses need to be read questions only accessed 27 August
more than once for accurate be used once, 2014.
scoring. since they are
easily shared Criswell, J.R. &
> Score all responses to one question Criswell, S.J. (2004).
before moving to the next question. and easy to
remember. Asking essay
> Occasionally change the order in questions: Answering
which the candidates’ tests are > A plagiarism check contemporary needs.
scored to avoid any order effects may be performed Education, 124 (3),
(e.g., score all question 1 by reviewing likely 510-516.
responses, shuffle the exams and documents that
candidates would Evaluation and
then score all of question 2
responses, etc.). be consulting. Examination Service,
University of Iowa
> Score responses without looking at (n.d.). Preparing and
the candidates’ names to avoid References
evaluating essay test
possible bias in scoring due to Center for questions, Technical
factors such as gender, familiarity Innovation in bulletin 36. From
with the candidate, etc. Teaching & Learning http://www.uiowa.
> Have at least two qualified (2009). Improving edu/~examserv/resourc
individuals score candidates’ your test questions. es_fees/Technical_
responses independently and From Bulletins/Preparing_an
discuss/negotiate their ratings. http://cte.illinois.edu/t d_Evaluating_Essay_
esting/exam/test_que Tests_Questions.html,
> For more information, consult s. html, accessed 27 accessed 27 August
the Checklist for scoring essay August 2014. 2014.
questions.
Centre for Teaching Harvard Kennedy
Additional recommendations Excellence, University School of Government
of Waterloo (n.d.). (2014). Exam
> Given the potential for Preparing tests and procedures and
unauthorized collaboration and exams. From policies. From
consultation during unsupervised https://uwaterloo.ca/ce http://www.hks.harvard.e
assessment, the Public Service ntre-for-teaching- du/degrees/registrar/
Commission recommends that excellence/teaching- faculty/exams-and-
supervised verification testing be resources/teaching- grading/exams,

4 Public Service Commission

accessed 27 Jacobs, L.C. Measurement
August 2014. (2004). How to for People
write better tests: Who (Think
Instruction They) Hate
al A handbook for
Tests &
Assessme improving test
Measurement
nt construction skills.
, 2nd edition.
Resources, From
Sage Publications.
University http://www.indiana.
of Texas edu/~best/pdf_ University of
(2001). docs/better_tests.p New South
Rubrics. df, accessed 27 Wales (2013).
From August 2014. Open book
https://www.utex and take home
Piontek,
as.edu/academic/ exams. From
M.E.
ctl/ https://student.
(2008).
assessment/iar/st unsw.edu.au/op
Best
udents/report/rub en-book-and-
practices
rics.php, take-home-exams,
for
accessed 27 accessed 27 August
designing
August 2014. 2014.
and grading
exams.
Center for
Research
on Learning
and
Teaching,
University
of Michigan.
Reiner, C.M.,
Bothell, T.W.,
Sudweeks,
R.R., & Wood,
B. (2002).
Preparing
effective essay
questions: A
self-directed
workbook for
educators.
New Forums
Press.
Salkind, N.J.
(2013).
Essay items.
Tests &

BEST PRACTICES FOR UNSUPERVISED TESTING 5

List of directive verbs Checklist for constructing
(from Reiner, Bothell, Sudweeks essay questions
& Wood, 2002) (adapted from Reiner, Bothell,
Sudweeks & Wood, 2002)
ǃ Analyze ǃ Explain
ǃ Apply ǃ Evaluate > Could the competency/ability be
better assessed with a different
ǃ Attribute ǃ Generate kind of assessment?
ǃ Classify ǃ Identify > Is the essay question aligned with
ǃ Compare ǃ Illustrate the competency or ability being
assessed?
ǃ Compose ǃ Infer
> Is the essay question too long and
ǃ Contrast ǃ Interpret should it be split up into several relatively
ǃ Create ǃ Justify short essay questions?
ǃ Critique ǃ List > Does the essay question contain a
clear and delimited task and a specific
ǃ Defend ǃ Predict
problem/situation?
ǃ Define ǃ Propose
> Is the question worded and structured
ǃ Describe ǃ Recognize in such a way that it will be clear to
Recall the
ǃ Design ǃ
candidates what they are expected to do?
ǃ Develop ǃ Summarize
> Is the task presented to the
ǃ Differentiate ǃ Trace candidates reasonable?
> Is the problem/situation included in the
essay question a novel situation?
Checklist for scoring essay
> Do the candidates know the time
questions frame for completing their responses?
(adapted from McMillan, 2001)
> Do the candidates know how many
> Is the model response outlined points the essay is worth?
prior to testing candidates?
> Has the use of optional questions
> Is the scoring method appropriate? been avoided?
> Has the role of elements of > Has an expected response or an outline
written communication been of major points that should be included in
clarified? the response been written and
> Are questions scored one at a time? discussed? Is the expected response
aligned with the essay question and the
> Is the order in which the tests are competency or ability being assessed?
scored changed?
> Is the identity of the
candidate anonymous?

6 Public Service Commission