Test Manual

English Comprehension

1 Copyright © 2014-15 Aspiring Minds. All Rights Reserved

Contents

1. Introduction ....................................................................................................................................... 3
2. Number of Questions and Test duration ........................................................................................... 4
3. Test Delivery..................................................................................................................................... 5
4. Language English.............................................................................................................................. 5
5. Reading Level ................................................................................................................................... 5
6. Test Scoring ...................................................................................................................................... 5
7. Rubric................................................................................................................................................ 6
8. Reliability.......................................................................................................................................... 6
9. Criterion Validity .............................................................................................................................. 7
9.1 Mid-sized Manufacturing Company ............................................................................................. 7
9.2 Business Process Outsourcing Company ...................................................................................... 8
10. Importance of Local Validation .................................................................................................... 9
11. Reference ...................................................................................................................................... 9

2 Copyright © 2014-15 Aspiring Minds. All Rights Reserved

1. Introduction
Familiarity with English Language in its various nuances is an essential skill, especially in the
current climate of global networking. Ideally, any recruitment should involve a test of skills in
handling the Language in ways that promote the objectives of a company and establish desired
rapport. Needless to state, an appropriate test is necessary. Our English test uses a variety of
internationally standardized resources for framing questions aimed at determining the
candidate’s ability to understand (a) the written text and (b) communicate effectively through
written documents. This test broadly covers the following areas:

1. A wide-ranging vocabulary to cope with general and specific terminology.

2. Syntax and sentence structure, the incorrect use of which distorts meaning and becomes a
communication hurdle.

3. Comprehension exercises designed to test a candidate’s ability to read fluently and understand
correctly.

4. The ability to understand and use suitable phrases, which enrich the meaning of what, is
conveyed.

A scientific design approach has been followed to construct the items. The vocabulary section
has a selection of words based on their frequency of occurrence, context, type and meaning. The
section tests whether the test-taker knows the meaning of words used in day-to-day English by
adults and could she/he infer the meaning of words in a given context. The sentence structure
and grammar section follows a detailed taxonomy where different grammar areas are detailed
into sub-topics and each of them are mapped to CEFR levels. Items have been generated based
on this detailed taxonomy and item formats include fill in the blanks, error identification and
correction of sentences. The English comprehension section comprises of two passages with four
questions each, which asks questions primarily around overall understanding of the text, being
attentive to detail, understanding of implied meaning and ability to understand words in context.

The skills measured by the test map to the following O*NET™i knowledge, skills, ability and
other characteristics: English language, reading comprehension, written comprehension and
expression.

3 Copyright © 2014-15 Aspiring Minds. All Rights Reserved

Based on O*NET™ database list top jobs mapped to English Language and Literature Teachers,
Postsecondary (25-1123.00), Editors (27-3041.00), Reporters and Correspondents (27-3022.00),
Proofreaders and Copy Markers (43-9081.00), General and Operations Managers (11-1021.00)
and Historians (19-3093.00).

2. Number of Questions and Test duration

The test contains 20 questions that have to be completed in 20minutes.

4 Copyright © 2014-15 Aspiring Minds. All Rights Reserved

3. Test Delivery
The test is administered online through a computer. The system should be connected to
broadband internet service. Based on requirements, multiple delivery features can be provided.
These features include:

1. Auto-proctor: The candidate’s web camera is used to click the images of candidate at
random intervals. These images are provided as part of the final report.
2. Print Screen Lock: Prevents candidates from storing (copy/paste) the information related
to assessment. Thus, reducing probability of storage and sharing of assessment artifacts.
3. Geographic Tracking: This feature helps to track the location of candidate.
4. Off-Focus: This feature helps partner to define the number of times a candidate is
allowed to be out of the frame. In case user is away from the frame on more instances
than defined. The assessment is immediately annulled.
5. IP Address Binding: This feature ensures that user is able to deliver test from a single
computer system.

Note:
1. Additional requirement for the additional delivery feature is webcam on candidate’s
computer system.
2. The additional delivery features are non-intrusive features that do not have any impact on
candidate’s assessment experience.
3. The additional delivery features help in maintaining the integrity of the assessment.

4. Language
English

5. Reading Level
Flesch Reading Ease: 56.5

Flesch-Kincaid Grade Level: 9.3

6. Test Scoring
The test is scored using Item Response Theory. The score is generated on a range of 100-900.

5 Copyright © 2014-15 Aspiring Minds. All Rights Reserved

7. Rubric

Skill
Rubric
Level
1 Doesn't demonstrate any understanding of basic English language.
May understand basic words but cannot read, understand or write simple sentences
2
in the English language.
Can read and understand simple sentences, short paragraphs and construct short
3 sentences using basic words and simple grammatical constructs. Can be trained in
understanding and writing English text, letters and emails in short period of time.
Can read, understand and write proper sentences, paragraphs, letters/emails with
4 appropriate choice of words and correct sentence construction. Can interpret facts,
information, context and implied meaning from written text.
Can understand difficult words and comprehend complex information by sifting
5 through paragraphs of text and relate between different pieces of text, can write
elegant English with rich vocabulary, concision and correct sentence structure.

8. Reliability
The English test is an adaptive test based on Item Response Theory (IRT).

Standard error: 48 points on a 100-900 scale.

The standard error is based on theoretical estimates arrived at by computer simulations. The IRT
parameters of the items in the test are set according to the attempt of AMCAT by graduates with
sample size greater than 50,000.

The module shows the following correlations with the scores of other AMCAT modules and
school and college GPAs.

Module Correlation

AMCAT Logical Ability 0.41

0.34
AMCAT Quantitative Ability

10th Grade GPA (High School) 0.31

12th Grade GPA (High School) 0.26

College GPA 0.13

6 Copyright © 2014-15 Aspiring Minds. All Rights Reserved

* The correlations are calculated on a sample of 50,000 AMCAT takers, final year undergraduate students, who
attempted the test in 2014-15. All correlation values are significant.

The module shows the following correlations with the AMPI traits

Trait Correlation

Extraversion 0.09

Conscientiousness -0.02

-0.01
Neuroticism

Openness to Experience -0.03

Agreeableness 0.16

Polychronicity 0.02

The correlation of scores of the English comprehension test with Logical ability and Quantitative
ability test is moderate, as per expectation. The correlation of the English comprehension score is
higher with high school grades as compared to college grades. This is explainable, since college
tests a higher degree of functional knowledge and are also not standardized.

Note: Reliability is a measure of the consistency in scores of the same test taker on an
assessment instrument under the ceteris paribus conditions.

9. Criterion Validity
Criterion related validity provides support to inference that individuals performing better on
assessment will perform on some criterion of interest.

9.1 Mid-sized Manufacturing Company

Scenario: A mid-sized Manufacturing company employed content writers. The content writers
wrote, reviewed and corrected descriptions of the products sold by the company. The company
wanted to use a hiring test for these content writers as a first step to screen candidates in
recruitment process.

Objective: Establish hiring criteria based on objective measures

7 Copyright © 2014-15 Aspiring Minds. All Rights Reserved

Methodology: A job analysis was conducted to understand the knowledge, skills, ability and
other characteristics required for the profile. Based on the analysis, AMCAT English, AMCAT
Critical Reasoning, AMCAT Logical Ability and AMCAT IGS (Information Gathering and
Synthesis) were delivered to 154 current employees in the company with job experience of 1 to 2
years. These candidates were rated by their managers on three parameters: English quality, Daily
work output and Work quality (correctness and lack of errors). The AMCAT test correlations
with performance parameters were as follows:

Information
Critical Gathering and Logical Ability
English Reasoning Synthesis
English Quality 0.48 - 0.29 -
Work Quality - - 0.32 0.24
Work Output - 0.30 0.28 0.34
*Only significant indices reported; Indices in bold are those that showed maximum validity for a performance
parameter

Inference: The results were in line with the analysis of SMEs (Small Medium Enterprises). The
total predictive validity of AMCAT modules with English quality was 0.49. The total predictive
validity of AMCAT modules with Work quality was 0.39. The total predictive validity of
AMCAT modules with Work Output was 0.47. English Quality showed maximum correlation
with English scores, Work Quality with Information Gathering and Synthesis and Work Output
showed maximum correlation with Logical Ability scores. For each of the output parameters, a
regression analysis was performed using only the test scores which showed a significant
correlation with the respective output. As expected, IGS did not add any incremental value in
predicting English quality over and above AMCAT English scores. On the other hand, Work
Quality was influenced by Logical Ability scores other than IGS scores. Critical Reasoning,
Logical Ability and IGS scores all significantly predicted Work Output. Without critical
reasoning scores, the total r reduced to 0.40.The validity exercise showed support for using all
the four AMCAT modules for this job profile.

9.2 Business Process Outsourcing Company

Scenario: A large ITeS/BPO company was looking for standardized benchmarks to hire
candidates at the entry level.

Objective: To find a criteria-set to filter out low performers and reduce the type one error (false
positive), whereas accept those who are mid and high performers.

Methodology: Employees belonging to different profiles: collection agents (54 agents),

customer servicing agents (48 agents) and sales agents (52 agents) were tested on AMCAT
English, Logical Ability, Quantitative Ability and AMPI (personality assessment). All the

8 Copyright © 2014-15 Aspiring Minds. All Rights Reserved

employees had a vintage of between 1 year and 1.5 years. The incentive-linked monthly
performance data was shared for a period of six months. The data classified each candidate as a
high, mid or low performer for each month based on objective work outcome. Consistency of the
rating after each month was checked through inter-month correlations. Average of these ratings
was taken as the output variable for the analysis. The AMCAT test correlations with
performance parameters were as follows:

English Logical Quantitative AMPI

Ability Ability

Validity 0.29 0.20 0.17 A: 0.19

ES: 0.21
(no other trait showed significant
correlation)

Inference: The total predictive validity of the tests achieved by a regression was 0.46 with the
performance. English Comprehension showed a significant correlation of r=0.29 with on the
performance. Our in house proprietary statistical tool recommended cut-offs that optimized for
type 1(bad selects) and type 2 (good rejects) errors while it increased the good performers by
25% to 35% for various processes. This exercise showed support for using all four AMCAT test
for hiring of entry level employees in BPO sector.

10.Importance of Local Validation

The validity information shared in this Test Manual is not intended to serve as a substitute for
locally obtained validity data. The local validity studies on standardized sample or local norm
group can be utilized for determining appropriate use of English comprehension scores. Thus,
based on feasibility the user should conduct the local (within organization or particular location)
validity of the assessment.

11.Reference

i
https://www.onetonline.org/

9 Copyright © 2014-15 Aspiring Minds. All Rights Reserved