2013 International Conference on Asian Language Processing
Evaluating the Difficulty of Concepts on Domain knowledge
Using Latent Semantic Analysis
Tao-Hsing Chang
Dept. of Computer Science and Information Engineering
National Kaohsiung University of Applied Sciences
Kaohsiung, Taiwan, R.O.C
changth@kuas.edu.tw
Yao-Tung Lee
Dept. of Educational Psychology and Counseling
National Taiwan Normal University
Taipei, Taiwan, R.O.C
ddt0320@ntnu.edu.tw
đối với
Abstract— As for the field of educational research and its
applications, evaluating concept difficulty is necessary but
difficult to carry out the work. Two major approaches are
employed in previous research to evaluate the difficulty of
concepts. Both of two approaches do not take into account
whether concepts are acquired by learners and readers or
not. This paper will be focused on constructing basic concept
list of domain knowledge with latent semantic analysis (LSA),
and use the age of acquisition of concept for representing the
difficulty of concept. This paper will utilize natural science
texts of elementary school in Taiwan as experimental
materials to verify the validity of using our proposed method
for evaluating the difficulty of concept.
Keywords- concept difficulty; LSA; age-of-acquisition.
I. INTRODUCTION
As for the field of educational research and its
applications, evaluating concept difficulty is necessary but
difficult to carry out the work. For example, students’
comprehension is often affected by the difficulty of
concepts on reading. Hence, teachers should employ
concepts which students have acquired to design reading
materials for teaching new concepts. In addition, domain-
specific concepts are also the fundamental elements of the
domain knowledge, such as social studies and science
whereas the sequence in concept learning has to take into
account the difficulty of concepts for learning effectively.
Therefore, to construct an effective and method for
evaluating the difficulty of concepts is a crucial and critical
issue.
Two major approaches are employed in previous
research to evaluate the difficulty of concepts. One is
ontology-based approach; the other is word-based
approach. Both of these approaches intend to discover lists
of domain concepts and then evaluate the difficulty of each
concept. Ontology in the first approach refers to a
hierarchical tree structure which is utilized to address the
relation between domain concepts. Such approach applies
the location of the concepts in the ontology to determine
the difficulty of concepts. For instance, Yan, Song & Li
(2006) treated Medical Subject Headings (MeSH)
978-0-7695-5063-3/13 $26.00 © 2013 IEEE
DOI 10.1109/IALP.2013.58
Yao-Ting Sung
Dept. of Educational Psychology and Counseling
National Taiwan Normal University
Taipei, Taiwan, R.O.C
sungtc@ntnu.edu.tw
hierarchical database of medical descriptor as a conceptual
database. Discovering each concept in medical texts from
MeSH, the method then computed the distance from the
concept to the bottom of this hierarchical tree. Using the
distance, the depth of the concept in hierarchical tree, the
document scope (DS) of each text could be achieved.
Furthermore, Zhao & Kan [2] employed Math World
Encyclopedia to establish a list of hierarchical domain
concepts and to compute the difficulty of each concept by
using Flesch-Kincaid Reading Ease formula.
Although the ontology-based method took into account
the hierarchal relationship among concepts in domain
knowledge to evaluate the difficulty of concept, this
approach had two limits. Firstly, there is no any available
hierarchical concept list in some fields. Because of the lack,
it was difficult to evaluate the difficulty of concepts.
Secondly, the researches based on hierarchical concepts
only took into account the location of concepts in the
hierarchical tree to evaluate the difficulty of concepts. The
researches do not take into account whether domain
concepts are acquired by the reader or not.
The second method utilized word-list to evaluate
domain specific texts. The method applied the familiarity
and rarity of domain words to evaluate the difficulty of
each word in domain specific texts. The assumption of the
method was that the less the word occur in a language, the
complexity of that word will be and more difficult to
understand. Therefore, the concept referred by the word
may be more difficult to understand. For instance, Borst,
Gaudiant, Grabar & Boyer [3] utilized word frequency and
the complexity of the word within domain as two variables
to compute the complexity of words. Furthermore, by
these two variables, it was able to evaluate the difficulty of
sentences and texts in medical online documents. The
methods using word-list have two limits. The first is that
the assumption of this method has several counterexamples.
For example, word “tooth brush” is rare in a large-scale
balanced corpus, but “tooth brush” is an easy and simple
concept. Secondly, this method also does not take into
account whether domain concepts are acquired by the
reader or not.
193
 These previous studies indicated that a concept should
be appeared in form of word. A word introduced to express
an abstract concept is called conceptual word. If a pupil
learned a concept, it suggested the pupil understood the
meaning of the concept and the conceptual word at the
same time. For example, concept “fruit” contains apple,
pineapple, banana and tomato. A pupil might know the
common property of these individual fruit, but the nominal
word of this property, fruit, was not perceived by a pupil.
Oppositely, a pupil might have heard of “fruit”, but the
meaning was not understood. Therefore, it was not
concluded and inferred the pupils have learned concept
“fruit”.
The age of acquisition of conceptual words was
supposed to be an important component of the concept
difficulty. Previous studies have proved the age of
acquisition of words was an impact for processing words.
The earlier the word was acquired, the faster the word was
processed in the brain. On the contrary, the word was
being processed later if the word was learned afterwards.
This was termed as age-of-acquisition effect [4]. Some
researches [5][6] showed that the age of acquisition of
words was a crucial variable to affect the cognitive process
of words. Moreover, a study [7] proved the age of
acquisition of words had an impact on understanding
important concept of domain texts for children. In other
words, how fast the pupil could process the word in the
brain was an evidence of concept difficulty, as a result, the
age of acquisition of conceptual words represented as
concept difficulty was a reasonable point of view. Since
the knowledge was all learned from textbooks for most
learners, the age of acquisition of fundamental conceptual
words on domain knowledge could be determined as the
time the student learned from textbooks.
Summarizing from the above discussion, this paper
will be focused on constructing basic concept list of
domain knowledge with latent semantic analysis (LSA),
and use the age of acquisition of concept for representing
the difficulty of concept. LSA [8] was an algorithm for
semantic retrieval. It first generated a matrix consisting of
the frequency of each word in different documents. The
Stage 1 Stage 2
Domain
Documents
Text set
Text set of
grade k
Generating
term-document
matrix Transferring to
semantic vector
SVD
VSM
Semantic similarity
Latent semantic
between words and
space
text sets
Fig. 1 The framework for computing the difficulty of concept
194
matrix was transformed to a matrix called latent semantic
space by the algorithm. Words and texts could all be
transferred to semantic vectors on the semantic space. If
the semantic vectors were closer, then the two words or
texts were more similar. Using LSA, this paper will collect
the words which are related to semantics on domain to be
domain concepts. This paper also proposes an algorithm to
estimate the age of acquisition of conceptual words as the
difficulty of the concepts.
The rest of this paper is organized as follows. Section 2
explains how the list of conceptual words and age of
acquisition can be determined. Section 3 examines the
validity of methodology in this study. The final section
displays discussions and conclusions.
II. METHODOLOGY
Our framework can be divided into three stages as
shown in Figure 1. The first stage is to establish the latent
semantic space of domain knowledge. The second stage
utilizes the latent semantic space to compute semantic
vectors of each word occurring in textbooks and semantic
vectors of each grade level textbooks respectively. The
third stage utilizes semantic vectors in the second stage to
calculate semantic similarity of each level textbook
corresponding to each word. The following subsections
illustrate technical details of each stage.
A. Constructing Latent Semantic Space of Domain
Knowledge
First stage of the proposed method is to construct latent
semantic space of the domain. Two procedures are
contained in the stage. The first is using domain texts to
generate term-document matrix. The second is that using
SVD to transforms the term-document matrix into the
latent semantic space of the domain. The purpose of term-
document matrix, a two-dimension matrix as shown in
Figure 2, aims to address the relation between documents
and words in documents. In Figure 2, d1 to dn represent as
the first document to the nth document; T1 to Tm indicate m
words occurring in all documents; ei,j stands for the
Stage 3
Word 1
Word n
Retrieving
conceptual words
of each grade
Transferring to
semantic vector
Evaluating the
difficulty of
conceptual word
Difficulty of
concept w
frequency of word Ti occurring in the document dj. The
matrix A is so called term-document matrix. 
d1 " d n
T1 ª e1,1 " e1, n º
Amu n « »
# « # % # »
Tm «em,1 " em, m »
¬ ¼
Fig. 2. Example of Term-Document Matrix
Due to huge term-document matrix, SVD, a well-
known operation in linear algebra, is applied to transform
term-document matrix into term-semantic matrix. The
transformation is shown as in formula (1).
Amun U mur u ¦r ur uVrTun (1)
where r represents as the number of semantic dimensions
and usually far less than n.
Using formula (1), term-document matrix A can be
divided into three smaller matrix, U,  and V. Matrix U
refers to the relation between m words and the dimensions.
Matrix, called singular value matrix, is a diagonal matrix,
where all entries on the diagonal are called singular values
which represent the importance of each semantic
dimension to term-document matrix. Matrix V stands for
the importance of each text corresponding to each
dimension. The result of matrix multiplication Uu is so
called latent semantic space of matrix A.
B. Computing Semantic Vectors
Through latent semantic space, any word, text or set of
texts can be transformed to a semantic vector for
representing the association within domain knowledge. A
semantic vector of a word, text or set of texts can be
obtained by using formula (2). Assuming column vector W
is the word vector and entry wi,1 in W refers to the
frequency of ith word occurring in the text set. Semantic
vector Ŝ of the text set is
Sˆ1ur W1Tum u U mur u ¦ r ur (2)
th
For a word viewed as the i word in domain
knowledge, the entry wi,1 in vector W of the word refers to
the frequency of the word occurring in all documents and
other entries in vector W is 0. For a set of texts
corresponding to particular grade, moreover, the semantic
vector of the set is also generated by using formula (2).
The entry wj,1 in semantic vector W of the set refers to the
frequency of the jth word occurring on the text set.
Assume a semantic vector Ŝ w of word w and a
semantic vector Ŝ d of a text set corresponding to a grade.
If both of vectors Ŝ w and Ŝd obtain higher values on the
same dimensions and lower values on other dimensions, it
means both of the word and set may express the same
subtopic. In other words, the word is the key term on texts
of the grade. Hence, higher semantic similarity between
the word and the set should be measured. Based on above
observation, vector space model (VSM) [16] shown as
formula (3) is employed to compute the similarity between
semantic vectors Ŝw and Ŝd .
195
cosine Sˆ w , Sˆd  Sˆ w ˜ Sˆ d (3)
Sˆ Sˆ
w d
The cosine value calculated by VSM falls between -1
to 1. The value closer to 1 indicates more similar semantic,
but that closer to 0 refers more irrelevant semantic.
C. Computing the Difficulty of Conceptual Words
Using formula (3), the semantic similarity between
each word and texts of each grade can be measured. The
words which strongly related to the texts of a grade will be
regarded as conceptual words on domain knowledge
corresponding to the grade. Therefore, given a grade k, the
proposed method computes semantic similarities between
the text set of grade k and each of words. All words,
furthermore, are ranked by the similarities and then to
select the top n percent of the words to be conceptual word
for grade k.
Words in domain texts can be classified into three
categories. First is that a word is not the conceptual word
in texts of any grade. It suggests the word is an auxiliary
term for learning concepts. The word is not a fundamental
concept in the domain even though it is a significant
concept for adults or basic concept in other domains.
Therefore, age of acquisition and difficulty are not herein
discussed. Second is that the word is identified as a
conceptual word of unitary grade. It means the concept of
the word is well formed at the grade even it is first learned.
Thus, the grade is assigned to the age of acquisition of the
concept while it is also the difficulty of the concept.
Third is that the word is identified as a conceptual
word in the texts of different grades. For example, a certain
word is treated as conceptual words for grade 3 and grade
5. The difficulty of the conceptual word will be assigned
with grade 3 based on the following two reasons. First, the
word would have been acquired at grade 3. At grade 5, the
concept is used to explore new concepts. Hence, the age of
acquisition for the word should be grade 3 not grade 5.
Second, the word would be composed of multiple concepts.
As a result, at grade 3 and grade 5, pupils would learn
various concepts derived from the word. This paper
assigns grade 3, the first grade of acquisition of the
conceptual word, to the difficulty of the conceptual word.
Summarizing the discussion mentioned above, the
concept difficulty CpDi of a conceptual word w is
determined as follows.
(4)
where G represents set of grades; Ti represents set of the
conceptual words in texts of grade i.
III. EXPERIMENTS
In the section, textbooks of science used in elementary
school in Taiwan are employed to analyze the performance
of different methods. The textbooks contain 261
documents. The statistical description of the textbooks for
different grades is listed in Table 1. This study randomly
selected 56 conceptual words on the textbooks to be
experimental materials. An expert, who taught sciences
from grade 3 to 6 several years, was asked to evaluate the
difficulty of the 56 conceptual words. The difficulty of
each word was identified as class 3, 4, 5 or 6. Class 3
represents the conceptual word could be cognized and
learned by the third graders while class 6 represents the
conceptual word could be cognized and learned by the
sixth graders.
Table 1. Statistical description for experimental materials
Grade 3 4 5 6
Number of documents 70 67 64
Average number of words 310 397 578
In order to contrast the performance of previous
methods and the proposed method, the difficulty of the 56
conceptual words are also computed by the following
word-list method. First, 1398 words in the textbooks are
ranking by their frequency in Sinica balance corpus. For
example, the frequency of word T1 is highest while that of
word T1398 is lowest. Second, these words are quartered
by their order. For example, T1 and T349 falls first quarter,
T350 and T699 falls second quarter, and so on. Finally, the
difficulty of a conceptual word is determined by the label
of the quarter in which the word falls. For instance, the
difficulty of word T1 is tagged as class 3 while T1398 is
tagged as class 6. Table 2 shows the number of conceptual
words classified into different grades by word list and the
proposed methods respectively.
Table 2. The number of conceptual words
Grade
3 4 5 6
Word list 9 18 15 14
The proposed method 12 11 15 18
This study adopts Spearman rank correlation
coefficient to measure the difference between the
difficulties estimated by expert and various methods. Table
3 shows the comparison between the evaluations of word
list method and the proposed method. In Table 3, the
results support that there is significant correlation between
the evaluations of the expert and the proposed method. On
the other hand, the results also show that there is no
significant correlation between the evaluations of the
expert and word list method
Table 3. The comparison between the performances of word list
method and the proposed method
methods spearman p-value
Word list -.009 .949
The proposed method .598 .000
IV. DISCUSSION AND CONCLUSION
Experimental results shown that the proposed method
can retrieve conceptual words and evaluate the difficulty of
the words. It overcomes the limitations of previous
methods because it explores conceptual words from
domain knowledge automatically. In addition, the results
also shown that the difficulty determined by the proposed
method is closer to the cognition of experts than previous
methods. This may due to the fact that ontology and
wordlist do not take into account the age of acquisition of
conceptual words, thereby having gaps between previous
methods and the cognition of experts.
In the future, many researches can be developed based
on this work. Using the proposed method can further
analyze different domain knowledge, such as social, and
explore the difference of conceptual words between
different domains. In addition, this study only explored the
properties of fundamental conceptual words in the series
textbooks of grades 3-6. The proposed method can be used
to evaluate the concept difficulty of the words in grades 7-
12 textbooks.
ACKNOWLEDGMENT
This work is particularly supported by "Aim for the
60 Top University Project" of the National Taiwan Normal
593
University and the Ministry of Education, Taiwan, R.O.C.
REFERENCES
[1] X. Yan, D. Song, and X. Li, ”Concept-based document readability
in domain specific information retrieval,” Proceedings of the 15th
ACM international conference on Information and knowledge
management, 2006.
[2] J. Zhao and M. Y. Kan, ”Domain-specific iterative readability
computation,” Proceedings of the 10th annual joint conference on
Digital libraries, 2010.
[3] A. Borst, A. Gaudinat, C. Boyer, and N. Grabar, ”Lexically based
distinction of readability levels of health documents,“ Poster at
MIE, 2008.
[4] J. B. Carroll and M. N. White, ”Word frequency and age-of-
acquisition as determiners of picture-naming latency,” Quarterly
Journal of Experimental Psychology, vol. 25, pp.85-95, 1973.
[5] C. M. Morrison and A. W. Ellis, ”The roles of word frequency and
age of acquisition in word naming and lexical decision,” Journal of
Experimental Psychology: Learning, Memory, and Cognition, vol.
21, pp. 116-133. 1995.
[6] C. M. Morrison, T. D. Chappell, and A. W. Ellis, ”Age of
acquisition norms for a large set of object names and their relation
to adult estimates and other variables,“ Quarterly Journal of
Experimental Psychology: Human Experimental Psychology, vol.
50A, pp.528-559, 1997.
[7] A. Biemiller and N. Slonim, ”Estimating Root Word Vocabulary
Growth in Normative and Advantaged Populations: Evidence for a
Common Sequence of Vocabulary Acquisition,” Journal of
Educational Psychology, vol. 93, pp.498-520, 2001.
[8] S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R.
Harshman, ”Indexing by latent semantic analysis,” Journal of the
American Society for Information Science, vol. 41, pp.391-407,
1990.
[9] T. K. Landauer, P. W. Foltz, and D. Laham, ”Introduction to latent
semantic analysis,” Discourse Processes. 25, 259-284, 1998.
[10] K. S. Jones, ”A statistical interpretation of term specificity and its
application in retrieval,” Journal of Documentation, vol. 28, pp.11-
21, 1972.
[11] G. Salton and C. Buckley, ”Term-weighting approaches in
automatic text retrieval,” Information Processing and Management,
vol. 24, pp.513-523, 1988.
[12] K. Kireyev and T. K. Landauar, ”Word Maturity: Computational
Modeling of Word Knowledge,” Proceedings of the 49th Annual
Meeting of the Association for Computational Linguistics, pp. 299-
308, 2011.
[13] J. Y. Yeh, H. R. Ke, W. P. Yang, and I. H. Meng, ”Text
summarization using a trainable summarizer and latent semantic
analysis,” Information Processing and Management, vol. 41, pp.75-
95, 2005.
[14] Y. T. Sung, J. L. Chen, Y. T. Lee, Y. S. Lee, C. Y. Peng, H. C.
Tseng, and T. H. Chang, ”Constructing and Validating a
Readability Modal with LSA : A Case Study of Chinese and Social
Science Textbooks,” Proceedings of the 22th Annual Meeting of
Society for Text and Discourse Process, Montreal, Canada, 2012.
[15] A. C. Graesser, D. S. Mcnamara, M. M. Louwerse, and Z.
CAI, ”Coh-Metrix: Analysis of text on cohesion and language.
Behavior Research Methods,” Instruments, & Computers, vol. 36 ,
pp. 193-202, 2004.
[16] T. K. Landauer, D. S. McNamara, S. Dennis, and W. Kintsch, ”
Handbook of Latent Semantic Analysis,” Lawrence Erlbaum
Associates, Publishers, Mahwah, New Jersey, 2007.
196