TEACHING AND ASSESSMENT OF LITERATURE

MODULE STUDIES

CHAPTER 6: Qualitative Analysis

Objectives:
a) Explain categories of approaches in preparing qualitative data.
b) Describe how teacher monitor his/her class
c) Explain different types of qualitative data

A. PURPOSE
Not all topics in language nor literature can be measured statistically. Viewpoints,
actions and characteristics can’t always be represented numerically and so need a
qualitative approach. In test analysis interviews, usability test and close reading are
very useful tools to identify tests and close reading are very useful tolls to identify the
strengths and weaknesses of a task.
Because of its approach, qualitative research may reveal data that would not
emerge from quantitative analysis. There are various way in which tests can be
analyzed qualitatively.

B. CATEGORIES OF APPROACHES

1. Reflection
This approach is aimed at gaining an insight into the thinking processes and
opinions of the test taker.

1|Page
 TEACHING AND ASSESSMENT OF LITERATURE
MODULE STUDIES

2. Verbal reports
Verbal reports or verbal protocols are a way of collecting qualitative data. They
offer and insight into the thought processes of informants.

A number of variables can be distinguished:

 Talk aloud: informants voice their thoughts
 Think aloud: informant voice her thoughts as well as other information
 Concurrent: the verbal report is given in real time
 Retrospective: the verbal report is given afterwards
 Mediated: the researcher occasionally intervene
 Non-mediated: the researcher does not intervene

3. Diary studies
Informant keep a diary which allows researchers to get an insight into their
thoughts. Diaries are not often used in test validation research, but they have
proven their worth in research into their processes.

There are a number of varieties:

Unstructured: the informant is free to write what he/she wants in whatever format.
Guided: the researcher gives the informant guidelines.
Structured: the researcher offers the informant a diary form with closed and open-
form questions.
When using diaries in test analysis, note that the data can be hard to process. It can
be very hard to draw conclusions from specially unstructured diaries. Very structured
diaries on the other hand only offer the information you specifically ask for.

2|Page
 TEACHING AND ASSESSMENT OF LITERATURE
MODULE STUDIES

C. TECHNIQUES FOR MONITORING STUDENT PROGRESS

Some effective techniques for monitoring student progress in the areas of oracy and
literacy include the following:

For more knowledge:

1. https://www.youtube.com/watch?v=2ADAY9AQm54
2. https://www.youtube.com/watch?v=Vq9_HOv2W2g

Self-assessment promotes students abilities to assume more responsibility to identify

where they believe they have been successful and where they believe they require
assistance.

Peer assessment allows students to collaborate and learn from others. Through
discussion with peers, high school students can verbalize their concerns and ideas in a
way that helps them clarify their thoughts and decide in which direction to proceed.

D. TYPES OF QUALITATIVE ANALYSIS

1. Conversation analysis (CA)

The assumption is that every interaction is contextual and has a stable and
predictable nature.
a. From a transcript, the power can be research by analyzing
 The structure adjacency pairs
 Turn-taking

3|Page
 TEACHING AND ASSESSMENT OF LITERATURE
MODULE STUDIES

b. Some possible pitfalls to look out for with CA

 Data quality – the speech recording has to be clear
 Loss of data – the original recording should be kept for reference
 Time – transcripts are very time-consuming to compose and to research.
 Limited generalizability – each interaction is different, so it is hard to
draw general conclusion.

2. Discourse analysis (DA)

DA is the analysis of “text and talk as social practices” and is mainly
concerned with power relations, gender inequalities ect. In DA the transcript of
an interaction is analyzed for adjacency pairs, turn-taking and repair. Special
attention is paid to:
 The effect of examiner behavior on test taker performance;
 The effect of test taker characteristics on performance;
 The effect of task type on performance, and
 Comparing test taker language ability outside of test to test performance.
The potential pitfalls are similar to choose of CA
3. Test language analysis
An analysis of test input or
test taker response for lexical
richness, rhetoric, genre,
discourse makers, grammatical
complexity, ect.
When using this type of
sample analysis, pay attention to:
 Time-test language
analysis is very time-
consuming.
 Data-processing data is
more efficient if you know what you are looking for.

4|Page
 TEACHING AND ASSESSMENT OF LITERATURE
MODULE STUDIES

Task characteristics
This type of validation research helps examine the test tasks and to
determine to which extent they respond to the test goal.
The analysis is performed by a number of expert judges who determine the
task quality.

Bachman and Palmer (1996) suggest a framework of analysis which

considers the following:
 The setting (physical setting, participants, time ect.)
 The rubrics (language, complexity and clarity of evaluation
procedure)
 The test input (length and characteristics of language)
 The expected response
 Relationship between expected and actual response

FEEDBACK METHOD

1. Questionnaires
Questionnaires gather data such as
opinions and views that can also be
gathered through interviews.

Two kinds of questionnaire

a. Closed – the informant replies to
make questions by ticking boxes
or by making a scale
b. Open- the informant replies in his/her own words

https://www.youtube.com/watch?v=61aFHm81pas

The question is one of the main decisions to ponder on when designing a questionnaire.
Keep in mind that closed questions are easy to process but the information from them is
limited to the questions that are explicitly asked.
Power of effective questioning

5|Page
 TEACHING AND ASSESSMENT OF LITERATURE
MODULE STUDIES

Checklists

 a way determining whether all procedures have been gone through, whether
all necessary features are present, ect.
 Can literally be a list of boxes to be checked.
 Can be used in test design to make sure nothing has been forgotten.

2. Interviews
They are a flexible way of gathering data. There are various kind of interview,
depending on the structure and the number of informants interviewed at the same
time.
(1). Unstructured – there is no fixed interview schedule, but rather a number of
themes that are to be addressed.
(2).Semi structured – the researcher follows a preset schedule it is possible to
deviate from this when interesting issues rises.
(3).Structured – the interviewer goes through a fixed series of written questions
without deviating. This type of interview closely resembles a questionnaire.

6|Page
 TEACHING AND ASSESSMENT OF LITERATURE
MODULE STUDIES

(4).One on one – this kind of interview allows the researcher to zoom in on the
views of individual respondents.
(5).Group – the advantage interviewing larger numbers at once id that
interactions might spark observations that would have gone unnoticed.

REFERENCING
Think about the following before the interview:
a) Interviews should get the chance to practice their interview
skills prior to the collection
b) Ideally, the pilot settings resembles the actual
conditions as accurately as possible
c) During the interview, it’s useful to take note of
the interview situations.
d) The success of an interview largely
depends on the interviewer-respondent
interaction.
e) Interviews are time consuming.
Kinds of referencing:
Norm-referencing is the placement of learners in
rank order, their assessment and ranking in relation to their ability of his/her peers.
Criterion-referencing is reaction against norm-referencing in which the learner is
assessed purely in terms of his/her ability in the subject, irrespective of the ability of
his/her peers.
Criterian and Norm Referencing
The mastery criterion-referencing approach is one in which a single ‘minimum
competence standard’ or ‘cut off point’ is set to divide learners into ‘masters’ and ‘non-
masters’ with no degrees of quality in the achievement of the objective being recognized.
The continuum criterion-referencing approach is an approach in which and
individual ability is referred to a defined continuum of all relevant degrees of ability in the
area of question.

References:
Language and Literature Assessment –A comprehensive Guide /pages 50-57
Authors: Mildred B. Go, Ph.D., Ofelia T. Pocesion, Ph.D.

7|Page