Process notes for Assignment 01

Your task in the first assignment is to act as a reviewer of the questionnaire items in the
psychological questionnaire used to screen applicants for the position of game ranger in the
specialist anti-poaching unit for the South African Parks Board. The criteria by which you review the
questionnaire items are contained in the review scale (see the resource “Review scale” in
Assignment 01 Procedural Booklet pp. 6 - 10). In other words, you have to ask yourself whether the
questionnaire items meet the criteria in the review scale. You therefore have to consider each of the
questionnaire items (see Assessment 01 on myModules for the questionnaire items) and reflect on
the criteria contained in the Review scale to ascertain whether the items meet the criteria or not.

There are 3 sets of review criteria:

- Criteria related to the formulation of the item (1st set of criteria)

- Criteria related to whether the item is correctly grounded in theory (2nd set of criteria) and
- Criteria related to whether the item qualifies for practical use (3rd set of criteria)

Each set of criteria contains a number of sub criteria (in red) with a set of rating rules (rules for
providing a rating):

1. Criteria related to the formulation of the item (1st set of criteria)

a. the item is not using psychological jargon, and is formulated in everyday language
b. the item is not leading
c. the item is not ambiguous

Rating rules:

Rate 1 if: none of a, b, c (give a rating of 1 if none of the sub criteria have been met)

Rate 2 if: one of a, b, c (give a rating of 2 if any one of the sub criteria have been met)

Rate 4 if: two of a, b, c (give a rating of 4 if any two of the sub criteria have been met)

Rate 5 if: a + b + c (give a rating of 5 if all the sub criteria have been met)

NB: Although a 5-point scale offers five possible ratings, namely 1, 2, 3, 4 and 5, all
ratings are not always used. The lowest value (in this case 1) and the highest value (in
this case 5) are always defined, but some of the values in between may be left
undefined. For example, if no rule is provided for rating 3 (as is the case with this set of
criteria), the only possible ratings would be 1, 2, 4 or 5. No rating of 3 can be given.

2. Criteria related to whether the item is grounded in theory (2nd set of criteria)
a. the theory the item is based on is correct
b. the item implements the theory it is based on correctly
c. the item operates correctly in terms of the theory it is based on

Rating rules: (The rating rules here are different to the rating rules for the formulation
of the item: the sub-criteria here are hierarchical in nature, i.e., sub-criterion a must be
met before sub criterion b can be met and sub criteria a AND b have to be met before
sub criterion c can be met).
 Rate 1 if: not a (give a rating of 1 if sub-criterion ‘a’ has not been met – note that if
sub criterion ‘a’ has not been met sub criteria ‘b’ and ‘c’ cannot be met
because sub criterion ‘a’ is a prerequisite for sub criterion ‘b’ and sub
criterion ‘a’ AND ‘b’ are prerequisites for sub criterion ‘c’ – this means that if
the theory the item is based on is not correct, the item cannot implement
the theory correctly or operate correctly in terms of the theory).

Rate 2 if: a (give a rating of 2 if ONLY sub criterion ‘a’ has been met)

Rate 4 if: a + b (give a rating of 4 if BOTH sub criterion ‘a’ and ‘b’ have been met)

Rate 5 if: a + b + c (give a rating of 5 if ALL of sub criteria ‘a’, ‘b’ and ‘c’ have been met)

3. Criteria related to the practical use of the item (3rd set of criteria)
a. the item is comprehensible by those it is intended for
b. the item serves the purpose of the interview
c. the item is valid in practice
d. the item/question is reliable in practice

Rating rules:

Rate 1 if: not a + not b (give a rating of 1 if neither sub criterion ‘a’ nor sub criterion
‘b’ have been met)

Rate 2 if: a or b (give a rating of 2 if either sub criterion ‘a’ OR ‘b’ has been met)

Rate 3 if: a + b (give a rating of 3 if both sub criterion ‘a’ AND ‘b’ have been met)

Rate 4 if: a + b + c (give a rating of 4 if sub criteria ‘a’, ‘b’ AND ‘c’ have been met)

Rate 5 if: a + b + c + d (give a rating of 5 if all the criteria have been met)

Your task as reviewer of the questionnaire items is to determine whether the items meet the criteria
outlined above.

IMPORTANT NOTE:

In order to judge whether an item meets the criteria in the rating scale, you first need to know what
each sub criterion in the scale refers to – in this respect you have to study the ‘Notes on the review
criteria’ (See Assignment 01 Procedural Booklet pp. 11-20) – you cannot judge whether an item is
leading if you do not know what a leading item looks like. Similarly, you cannot judge whether an
item is ambiguous if you don’t know what ambiguity refers to – all the criteria in the rating scale
have been unpacked on p. 11-20 of Assignment 01 Procedural Booklet. It is VITAL to understand
what the criteria refer to if you are going to provide correct ratings.

Let us look at an example:

Complete the following statement by selecting A or B as your response:

“In challenging situations, I mostly…”

A: rely on the support of my colleagues to help me deal with the situation.

B: am confident that I can deal with whatever challenges I am faced with.

Justification**

In order to determine if respondents have confidence in their abilities to succeed in specific

situations, the item employs Bandura’s concept of self-efficacy. According to the theory, self-efficacy
refers to people’s beliefs about their capabilities to function effectively in a given situation. The item
implements the theory that people with high self-efficacy do not have faith in their capabilities to
function effectively and tend to rely on others when faced with challenging situations (Statement A).
The item implements the theory that individuals with low self-efficacy feel they can handle most
situations and challenges they are confronted with (Statement B). The item operates to distinguish
between respondents who display high self-efficacy (those who choose Statement A) and
respondents who display low self-efficacy (those who choose Statement B).

[** Side note: Note that respondents (those who complete the questionnaire item) only see the item
itself and NOT the justification – the justification is provided for you as reviewer to assess whether
the item is correctly grounded in theory (the second set of review criteria)].

First, to determine whether the item is formulated correctly, reflect on the 1st set of review criteria:

- Is the item formulated using psychological jargon or is it formulated in everyday language?

[What does psychological jargon refer to – see the Notes on the review criteria in Assignment 01
Procedural Booklet p. 11]. If you believe the item is formulated in everyday language without
the use of psychological jargon, then this sub criterion is met. If, however, you believe it to
contain psychological jargon, then this sub criterion will not have been met. (Remember, when
assessing the formulation of the item, you look at the item itself and not at the justification! You
only assess the justification in the 2nd set of criteria pertaining to whether the item is correctly
grounded in theory.)
- Is the item leading in nature? [What does a leading item look like? – see the Notes on the
review in Assignment 01 Booklet p. 11] Does the item lead the respondent to answer the
question as a result of some prompt/suggestion contained in the question about the correct
response? If you believe the item does not lead a respondent to a particular answer, this sub
criterion is met. If, however, you believe that the item does lead the respondent to a particular
answer, this sub criterion will not have been met.
- Is the item ambiguous in nature? [What makes an item ambiguous? – see the Notes on the
review criteria in Assignment 01 Procedural Booklet p. 11.] If you believe the item is ambiguous
in nature, this sub criterion will not have been met. If, however, you believe that the item is not
ambiguous, this sub criterion will be met.

If you believe that all 3 of the sub criteria pertaining to the formulation of the item have been met,
then you will give a rating of 5. If you believe that the item only meets 2 of the sub criteria in the
rating scale you will give a rating of 4. If you believe none of the criteria have been met, you will give
a rating of 1.

Second, to determine whether the item is correctly grounded in theory, reflect on the sub criteria in
the 2nd set of criteria: (Remember, to judge this set of criteria you interrogate the information
contained in the justification of the item.)

- Is the theory on which the item is based correct? (See p. 12 of Assignment 01 Procedural
booklet). Here you need to know what the theory says about self-efficacy [Study social cognitive
learning theory in the prescribed book]. You then look at the theoretical information offered in
the justification. In the present example the justification indicates:
 “According to the theory, self-efficacy refers to people’s beliefs about their capabilities to
function effectively in a given situation.”
Reflect on whether this is correct- does self-efficacy refer to people’s beliefs about their
capabilities to function effectively in a given situation? If you believe this to be correct, this sub
criterion is met. If you believe this is incorrect this sub criterion is not met.
- Does the item implement the theory correctly? [What does implementation refer to? – see the
Notes on the review criteria in Assignment 01 Procedural Booklet pp. 12 - 14. Implementation
refers to how the theory has been represented in the item, how it has been translated into
ordinary language in the item]. The justification states that:
“The item implements the theory that people with high self-efficacy do not have faith in
their capabilities to function effectively and tend to rely on others when faced with
challenging situations (Statement A). The item implements the theory that individuals with
low self-efficacy feel they can handle most situations and challenges they are confronted
with (Statement B).”
Is this implementation correct? Does high self-efficacy translate to people who do not have faith
in their capabilities to function effectively and who tend to rely on others when faced with
challenging situations? And does low self-efficacy translate to people who can handle most
situations and challenges they are confronted with? If you believe that self-efficacy has been
correctly represented/translated into ordinary language in the item, this sub criterion is met. If
you believe that self-efficacy has not been correctly represented/translated in the item, then
this sub criterion will not have been met.
- Does the item operate correctly in terms of the theory it is based on? [What does the operation
of an item refer to? – see the Notes on the review criteria in Assignment 01 Procedural Booklet
p. 14.] In the present example, the justification states that:
“The item operates to distinguish between respondents who display high self-efficacy
(those who choose Statement A) and respondents who display low self-efficacy (those who
choose Statement B).”
Is it correct to say that respondents who choose Statement A (those who rely on the support of
their colleagues to help them deal with challenging situations) display high self-efficacy? And is
it correct to say that respondents who choose Statement B (those who are confident they can
deal with whatever challenges they are faced with) display low self-efficacy? If you believe the
item functions correctly to distinguish between respondents with high self-efficacy (those who
rely on the support of colleagues to help deal with the situation) and low self-efficacy (those
who believe the can handle most challenges they are faced with) this sub criterion is met. If,
however, you believe the item does not correctly distinguish between respondents with high
and low self-efficacy, the item does not operate correctly in terms of the theory on which it is
based.

Say you believe that the theory the item is based on is correct but that the item does not implement
the theory correctly or operate correctly in terms of the theory, only sub criterion ‘a’ is met. The
rating rules for this set of criteria says that if only sub criterion ‘a’ is met, you provide a rating of 2. If
you believe that both sub criterion ‘a’ and ‘b’ have been met, you will offer a rating of 4. If you
believe all the sub criteria have been met, you will offer the highest rating of 5.

Lastly, to determine whether the item qualifies for practical use, reflect on the sub criteria
contained in the 3rd set of criteria:

- Is the item comprehensible by those it is intended for? [When is an item NOT comprehensible
by those it is intended for? – see the Notes on the review criteria in Assignment 01 Procedural
 Booklet p. 15-17.] If the item is not comprehensible (for example, if the item contains jargon)
then this sub criterion will not have been met but if you believe the item can be understood
without difficulty this sub criterion will be met.
- Does the item serve the purpose of the questionnaire? [When does an item NOT serve the
purpose of the questionnaire - see the Notes on the review criteria in Assignment 01 Procedural
Booklet pp. 15 – 17]. For an item to serve the purpose of the questionnaire it must 1) operate
correctly in terms of the theory on which it is based (this refers to the 3rd sub criterion in the 2nd
set of review criteria) and 2) it must align with the purpose of the questionnaire and generate
information that contributes to the purpose of the questionnaire]. If you believe the item
operates correctly in terms of the theory and if it generates information that aligns with the
purpose of the questionnaire, then this sub criterion is met. If you believe that the item does
not operate correctly in terms of the theory or if it does not elicit information that aligns with
the purpose of the questionnaire (which is indicated in the scenario, that is the psychological
characteristics/traits the questionnaire has been designed to assess) then this sub criterion will
not have been met.
- Is the item valid in practice? [What conditions must be satisfied for a question to be valid in
practice? – see the Notes on the review criteria in Assignment 01 Procedural Booklet p. 16-18.]
If you believe that the item is relevant for the context in which it is used and if you believe it is
fair to all respondents (i.e., it is not biased towards some respondents but not others) then this
sub criterion will be met. If you believe the item is not relevant for the context in which it is
used or if the item is biased in some way, then this sub criterion will not have been met. (Note
the rating rules here: For one to consider validity in practice, BOTH sub criterion ‘a’ AND ‘b’ have
to be met – if either sub criterion ‘a’ or ‘b’ is not met the item cannot be considered to be valid
in practice).
- Is the item reliable in practice? [When is an item reliable in practice? - see the Notes on the
review criteria in Assignment 01 Booklet p. 16-20.]. If you believe the item will yield information
that is consistent across respondents (i.e., all respondents understand the item in the same way
and therefore respond in the same manner to the item) and, if the item is likely to elicit the
same answer from a respondent when asked at some point later in time, then this sub criterion
will be met. However, if respondents are unlikely to respond consistently to the item or if it is
not likely to elicit the same answer from a respondent when asked at some point later in time
then this sub criterion will not have been met. (Again, note the hierarchical nature of the rating
rules here: For one to consider reliability in practice all three sub criteria ‘a’, ‘b’ and ‘c’ have to
be met – if any of these are not met the item cannot be considered to be reliable in practice).

If that you believe the item to be comprehensible for those it is intended for but that it does not
serve the purpose of the questionnaire (because it does not operate correctly in terms of the
theory) then you will offer a rating of 2 (which requires that either sub criterion ‘a’ or ‘b’ must be
met). If you believe the item is comprehensible, it does serve the purpose of the questionnaire and
that it is valid in practice but not reliable you will offer a rating of 4 (which requires that sub
criterion ‘a’, ‘b’ and ‘c’ are met). If you believe the item is comprehensible, that it serves the
purpose of the questionnaire, that it is valid in practice and reliable in practice you will offer the
highest rating of 5.

***********************************
End of resource