PYC2606

Revision for the examination

Please make sure that you have worked through the lessons and resources in the study guide a
number of times. You can use the assignment questions and feedback (Tutorial letters 201 and
202) to test your knowledge and understanding. The mock examination and online feedback on
how you did will also serve as preparation for the examination. In addition, we provide a few notes
below that you can use together with the assignment feedback.

Content domain and scope of the questionnaire

The aim of a questionnaire determines the content domain of the questionnaire. In the assignment
scenario in Tutorial letter 101, the aim of the School Bullying Questionnaire is to measure
experience of bullying behaviour and bullying is conceptualised in terms of three factors, namely
the form of bullying, the realm of bullying and the impact of bullying. These three factors are
therefore the content domain. If the questionnaire does not include items to measure all three
factors, it does not adequately measure the proposed construct (bullying) and it will lack construct
validity.

The scope of the questionnaire, on the other hand, refers to what is actually included in the
questionnaire. If the questionnaire includes items on all three factors, the scope is adequate. If it
does not include items on all three factors, the scope is insufficient. If it includes items on factors
that are not relevant to the construct of bullying, the scope is overdetermined.

Item format and rating questionnaire items

Make sure that you can identify different types of item formats. This is explained in your study
guide under the resource Item format. Check if you can identify the different types of items in your
assignment scenario.

Lesson 7: Review a questionnaire and the resource Writing questionnaire items will help you with
the questions on rating the formulation of items in the questionnaire. Examples are given of the
following practices that should be avoided: ambiguity, two inherent issues, negatives (including
implicit negatives), passive statements, leading questions, and social desirability. The item should
furthermore deal with the topic covered by the questionnaire (relevance of the item) and it should
be formulated correctly (configuration of the item). For example, if you have a filter question, the
‘yes’ ‘no’ should logically lead to the next question. That is, you cannot ask if the respondent
experiences bullying and then the ‘yes’ skips a question on the types of bullying that he/she
experiences.

Regarding the suitability of items, you must ask yourself if the item refers to something that forms
part of the respondents’ world (you cannot ask primary school children about their marriage) and
that they have the knowledge to answer (the general public will probably not be able to comment
on research results on the mental and physical effects of bullying). Lesson 7 contains a detailed
exercise on rating that you can use and you can also refer back to Assignment 02.

Scatterplots (Item variance and item discrimination)

In the assignment scenario, respondents can obtain a score from 1 to 5 (disagree entirely to agree
entirely) in Section 1: Direct forms of bullying. In this instance we measure experiences of direct
forms of bullying. Each item should have a variety of answers between 1 and 5 to show up
differences between respondents in terms of their experiences of direct forms of bullying (i.e. it
should not all be 1s or 5s, etc.). This is referred to as the variance of the item.

In Table 2 in the assignment scenario you can see that the responses for the 10 learners to Item
1.6 varied between 1 and 5. You can see this even better in scatterplot 3 where the item scores
were spread out over the graph (as opposed to scatterplot 1 where the item scores are grouped
together). Item 1.6 therefore has high item variance.

Each item should also measure the same construct as the other items in the section (i.e.
experiences of direct forms of bullying). This means that learners who score high on the item
(they agree with the statement and scores a 5) should score high on the section (they experience
direct forms of bullying). A high item-total correlation shows good discrimination value. We
measure the relationship between the items by means of a correlation between the scores on an
item and the scores on the test as a whole. Below are the scatterplots from your study guide that
shows a high positive correlation (we can draw an ascending straight line through the dots), a
high negative correlation (we can draw a descending straight line through the dots) and no
correlation. Remember a positive correlation means that high scores on the item are associated
with high scores on the section whereas a negative correlation means that high scores on the
item are associated with low scores on the section.

scatterplot a

scatterplot b

2
scatterplot c

Figure 1: Scatterplots for different degrees of correlations

In Figure 3 in the scenario you will see that the scatterplot for Item 1.6 resembles the high positive
correlation in the example above. Please note that it need not be a perfect correlation as in
example a, but as long as the shape resembles an ascending line, the item-total correlation is
high.

Reliability

Reliability refers to consistency – consistency in terms of the items in the questionnaire and
consistency over time. Consistency in terms of the items of the questionnaire is called internal
consistency and it means that the items in the questionnaire measures the same attribute. One
way to measure this is by means of split-half reliability. The questionnaire is split into two halves
(e.g. odd and even items) and a score is calculated on each half for each respondent. These two
sets of scores are then correlated.

To determine the consistency/stability of the results on the questionnaire over time, test-retest
reliability is calculated. It means that the same questionnaire is administered to the same group
of people on two different occasions and the two sets of scores are correlated.

If you have a table with two sets of scores for respondents (either split-half or test-retest), you can
arrange the first set of scores in ascending order. Write the second score for each person next to
that, it should also be ascending if the reliability is high. This will show a high, positive correlation.
It need not be a perfect correlation, but the trend should be clear.

Example 1 Set of scores 1 Set of scores 2

Person 1 13 18
Person 2 8 7
Person 3 4 6
Person 4 20 25
Person 5 35 30
Arrange first set in ascending order and write the corresponding score for the second set
Person 3 4 6
 Person 2 8 7
Person 1 13 18
Person 4 20 25
Person 5 35 30

Example 1 is a perfect positive correlation: Ascending scores in set 1 are associated with
ascending scores in set 2. We do not expect to find a perfect correlation when we calculate the
reliability of a questionnaire. (Note that it is not about getting the same score, but having the same
order of scores.)

Example 2 Set of scores 1 Set of scores 2

Person 1 13 18
Person 2 8 5
Person 3 4 6
Person 4 20 35
Person 5 35 30
Arrange first set in ascending order and write the corresponding score for teh second set
Person 3 4 6
Person 2 8 5
Person 1 13 18
Person 4 20 35
Person 5 35 30

Example 2 is a more realistic scenario. The scores in set 2 show an ascending (but not perfect)
trend of ascending scores. This will be a moderate to high positive correlation.

Example 3 Set of scores 1 Set of scores 2

Person 1 13 5
Person 2 8 13
Person 3 4 4
Person 4 20 25
Person 5 35 18
Arrange first set in ascending order and write the corresponding score for teh second set
Person 3 4 4
Person 2 8 13
Person 1 13 5
Person 4 20 18
Person 5 35 16

Example 3 indicates that there is not really a pattern of scores in the second set. This will be a
low positive correlation. (A negative correlation implies that an ascending order in the first set is
associated with a descending order in the second set.)

Please go back to Tables 3 and 4 in the scenario and apply the above to the tables to evaluate
the reliability. In table 3 set 1 is even items and set 2 is odd items and in Table 4 set 1 is the first
application and set 2 is the second application.
4
 Validity

The resource in the study guide on validity refers to three types of validity, namely content validity,
criterion-related validity and construct validity. Content validity is ensured during the development
of the questionnaire. This implies that the aim, content domain and scope of the questionnaire
should be alligned (see the first topic above). Criterion validity and construct validity both involve
the calculation of a correlation between sets of scores. The correlation coefficient varies between
-1 (a perfect negative correlation – ascending scores set 1 and descending scores set 2) and +1
(a perfect positive correlation – ascending scores set 1 and set 2) with 0 indicating no correlation
or relationship between the sets of scores. Remember that in the case of validity, a negative
correlation can also be accepted if this makes sense (e.g. correlating emotional intelligence and
bullying behaviour).

In the assignment scenario two types of construct validity are illustrated. Construct validity means
that the questionnaire measures what it is supposed to measure (in this case experience of
bullying behaviour). You would expect a relationship between scores for the three factors that
make up bullying behaviour, namely the form of bullying, the realm of bullying and the impact of
bullying. But remember our questionnaire only included forms of bullying and impact of bullying
and in Table 5 (intercorrelations between the sections of the questionnaire) you see that these
two are indeed related (e.g. a correlation coefficient of 0.80 between indirect and direct bullying).
This supports the construct validity, but because the realm of bullying was not included, the
construct validity remains problematic.

Table 6 illustrates convergent and discriminant validity that are both forms of construct validity.
Scores on a questionnaire should correlate significantly with scores on another questionnaire that
measures a similar construct. On the other hand, scores on a questionnaire should not correlate
significantly with scores on a questionnaire that it is not theoretically related to. Refer to the
feedback in Assignment 02 (Tutorial letter 202) where the information in Table 6 is unpacked.