Computers in Human Behavior 61 (2016) 522e531

Contents lists available at ScienceDirect

Computers in Human Behavior

journal homepage: www.elsevier.com/locate/comphumbeh

Full length article

Vocational college students' acceptance of web-based summative

listening comprehension test in an EFL course
Harun Cigdem a, *, Mustafa Ozturk b, Abdullah Topcu a
a
Turkish Land Forces Non-Commissioned Officer Vocational College, Balikesir, Turkey
b
Hacettepe University, School of Foreign Languages, Ankara, Turkey

a r t i c l e i n f o a b s t r a c t

Article history: There is a substantial increase in the utilization of web-based assessment procedures to assist teaching
Received 9 December 2015 and learning processes in higher education institutions. Despite their benefits, institutions use them to a
Received in revised form limited extent due to a number of factors influencing both instructors' and students' behaviors. This
20 March 2016
study examines students' acceptance of a web-based summative listening test administered in the 2014
Accepted 22 March 2016
Available online 28 March 2016
e2015 academic year within an ‘English as a Foreign Language' course at a two-year post-secondary
military school in Turkey. The participants consisted of 602 military students. As a model, Computer
Based Assessment Acceptance Model based on Technology Acceptance Model was adopted in order to
Keywords:
Behavioral intention
analyze the participants' perceptions on a web-based summative listening comprehension test. The data
Web based listening test were collected via an online questionnaire. A structural equation modeling analysis was utilized to
Technology acceptance analyze the relationships among factors. The general results showed that perceived ease of use and
Military vocational college perceived playfulness had a direct influence on the participants' behavioral intentions to use web-based
EFL course tests. In other words, web-based tests are expected to be used if they are easy to use and playful enough.
Online testing © 2016 Elsevier Ltd. All rights reserved.

1. Introduction Jeong, 2014).

Seeing that students' frequent use of computers and the Internet
is increasing rapidly, web-based testing (WBT) is also becoming a
prevalent system as an alternative to traditional assessment prac-
tices in all educational settings around the world. WBT is related to
the conceptualization and administration of assessments as a so-
phisticated way of using web technologies (Cigdem & Oncu, 2015)
with the aim of expanding educational assessments in universities,
schools or other industry. WBT's becoming widespread resides in
the fact that an increasing number of faculty members have started
to realize the convenience of creating, implementing, and man-
aging assessment as parts of learning management systems
(Llamas-Nistal, Ferna ndez-Iglesias, Gonzalez-Tato, & Mikic-Fonte,
2013). WBT is seen as a noteworthy method for instructors, espe-
cially regarding high-stake tests, because it aims to optimize the
goals and techniques of teaching and testing in shorter times (Pino-
Silva, 2008) and the delivery or administration of the assessment is
not supposed to be at a fixed time or place (Cigdem & Oncu, 2015;
* Corresponding author.
E-mail addresses: hcigdem@gmail.com (H. Cigdem), mustafaozturk@hacettepe.
edu.tr (M. Ozturk), abdullah.topcu@boun.edu.tr (A. Topcu).
There have been numerous advantages of WBT for both test-
developers and test-takers such as flexibility of time and place;
enhanced resource use; immediate and real-time feedback; high
interaction with test-takers; quick results and real-time score re-
ports; automated grading and reporting; easier data management;
cost reduction; more productive managing, organizing, and
deploying of exams; time-saving evaluation of learners' strengths
and weaknesses; and learners' self-evaluation (Abedi, 2014; Bull &
McKenna, 2004; Chou, Moslehpour, & Huyen, 2014; Cigdem & Tan,
2014; Llamas-Nistal et al. 2013; Morris, 2008; Terzis, Moridis, &
Economides, 2013; Zakrzewski & Steven, 2000). Along with such
advantages, an effective WBT system is required to provide
authentic assessment activities and meaningful feedback as well as
to support multidimensional perspectives (Gikandi, Morrow, &
Davis, 2011).
As test developers and test takers are being immersed in WBT
systems, a greater number of educational researchers tend to work
on the acceptance of WBT systems with the purpose of defining the
variables that might explain the acceptance issue. Users' perceived
acceptance of a technology or behavioral intentions to use a tech-
nology have been studied and described by many researchers
previously as in the Theory of Reasoned Action (TRA) (Ajzen &

http://dx.doi.org/10.1016/j.chb.2016.03.070
0747-5632/© 2016 Elsevier Ltd. All rights reserved.
 H. Cigdem et al. / Computers in Human Behavior 61 (2016) 522e531
Fishein, 1980); the Theory of Planned Behavior (TPB) (Ajzen, 1991);
the Technology Acceptance Model (TAM) (Davis, Bagozzi, &
Warshaw, 1989); the Unified Theory of Acceptance and Use of
Technology (UTAUT) (Venkatesh, Morris, Davis, & Davis, 2003); and
the Computer Based Assessment Acceptance Model (CBAAM)
(Terzis & Economides, 2011). Considering the scope of the current
research, the CBAAM was adopted and studied with the aim of
determining the students' intentions to use a WBT system in the
case of a summative assessment of Listening Comprehension in
English.
2. Literature review
As the researchers have been driven to examine the factors that
affect user's perceptions on the adoption and acceptance of WBT
systems, there has been a considerable amount of literature por-
traying the application of the CBAAM in various educational con-
texts. To exemplify, Dermo (2009) surveyed 130 undergraduate
students participating in an online (formative/summative) assess-
ment system during the 2007e2008 academic year and found that
the reliability, security, validity, and accessibility of the system were
accepted by the participants coming from different academic pro-
grams like management, informatics and engineering, life sciences,
social sciences and education, and therefore learning was promoted
among them. As another example, Terzis and Economides (2011)
worked on the perceptions of introductory informatics course
students on the acceptance of WBT. Their results indicated that
perceived ease of use and perceived playfulness had a direct effect
on the use of computer-based assessments, whereas perceived
usefulness had only an indirect effect. In Sorensen's (2013) survey
on students' perceptions about e-assessments in a college chem-
istry course, the participants thought that e-assessment facilitated
their learning. With that study, more frequent implementations of
e-assessment in other courses was also suggested. In a study con-
ducted with military students at a Mathematics course, it was seen
that students having computers and the Internet at home as well as
prior web-based exam experiences were more optimistic about the
computer-based assessments than the other students (Cigdem &
Tan, 2014). In another survey done with military students,
Cigdem and Oncu (2015) investigated perceptions on WBT in a
computer networks course and found that the contents of the
questions significantly affected perceived usefulness and that
perceived usefulness had a great influence on users' behavioral
intentions.
All those attempts reveal that an increasing number of educa-
tional settings adopt or use WBT systems to deliver and manage
their assessment procedures more efficiently and more conve-
niently. Since language testing is one of those settings, this study
investigates the factors that are likely to influence learners’ in-
tentions to use WBT systems in the assessment of their listening
comprehension. Even though WBT systems are thought to be useful
in the realm of language testing, whether such systems are
preferred over paper-pencil assessment procedures is not clear
enough (Pino-Silva, 2008). Therefore, the current study is thought
to be an important step to look into the perceptions of the students
on WBT systems when assessing a fundamental skill of language
education, which is listening comprehension.
3. Significance of the study
Turkish educational system is trying to raise generations who
can speak at least one foreign language, which is frequently English,
to be able to internationalize its institutions and citizens, and thus
promote language teaching at all levels of education, particularly in
higher education. Having graduates who can understand and speak
523
English is one of the primary policies of many higher education
institutions. This fact makes language teaching and testing in
higher education a difficult and demanding task. Although teaching
the skills of reading and writing are managed to some extent in
current implementations, there is still a limited competence in
teaching and testing of listening and speaking skills. An important
attempt is thought to carry out a computer assisted system to
expand teaching and testing of such skills to be able to make it
more practical and widespread. From this point forth, this study
will be a good basis for the upcoming value of teaching and testing
of listening comprehension through technology in higher educa-
tion. Although the major contribution of the study was to look into
the variables influencing the students' intentions to use a WBT
system, this study aimed to contribute to the field through an
attempt to implement a WBT system within the context of a foreign
language course, in particular for an underestimated sub-skill,
which is Listening Comprehension. Various forms of WBT systems
were adopted previously in various content areas; yet, such studies
were limited in number and scope for language teaching contexts.
Apart from this, the current research setting, which is a unique
post-secondary higher education institution among its types, adds
to the significance of the study.
4. Research model and hypotheses
Computer Based Assessment Acceptance Model (CBAAM) was
proposed by Terzis and Economides (2011) on the basis of previous
acceptance models such as TAM, TPB and UTAUT as well as with the
inclusion of additional variables in order to describe behavioral
intention to use a WBT system. Eight variables were included
within the model: perceived usefulness, perceived ease of use,
computer self-efficacy, social influence, facilitating conditions,
perceived playfulness, content, and goal expectancy. On the whole,
Perceived Usefulness and Perceived Ease of Use were adopted from
the TAM (Davis, 1989); Computer Self Efficacy was adopted from
Social Cognitive Theory (Compeau & Higgins, 1995); Perceived
Playfulness was adopted from an extended TAM version by Moon
and Kim (2001); and Facilitating Conditions and Social Influence
were adopted from the UTAUT (Venkatesh et al., 2003). Apart from
those, two more variables, Goal Expectancy, which is rooted in Self-
Management of Learning (Wang, Wu, & Wang, 2009) and Content
were introduced by Terzis and Economides (2011) to the model.
As illustrated in Fig. 1, the CBAAM suggests that users' intentions
to use a WBT system is directly linked to perceived ease of use,
perceived playfulness, perceived usefulness, and content of web
based test. Apart from those links, perceived ease of use is defined
by computer self-efficacy and facilitating conditions; whereas
perceived playfulness is defined by perceived usefulness, goal ex-
pectancy, and content of web based test. On the other hand,
perceived usefulness is also influenced by social influence, goal
expectancy, and content of web-based test.
4.1. Perceived playfulness (PP)
Moon and Kim (2001) offered Perceived Playfulness (PP) in the
TAM, as a key belief constructed through an individual's personal
experience with a system, Perceived Playfulness (PP) is defined as
“the pleasure the individual feels objectively when committing a
particular behavior or carrying out a particular activity”. PP, as a
significant variable, is attributed to have a positive impact on
behavioral intention to accept and use the Internet or a WBT system
(Moon & Kim, 2001; Terzis & Economides, 2011) and determined
by the aspects of concentration, curiosity, and enjoyment (Terzis &
Economides, 2011). In this framework, it is about the arousal of an
individual's enjoyment, cognitive curiosity, and concentration
524 H. Cigdem et al. / Computers in Human Behavior 61 (2016) 522e531
Fig. 1. Computer based assessment acceptance model.
during an activity. Our hypothesis for the current concept was as
follows:
H1. PP would have a positive effect on the Behavioral Intention to
use WBT.
4.2. Perceived usefulness (PU)
As one of the important factors introduced within the TAM,
Perceived Usefulness (PU) is defined as “the level that a person
thinks using a certain system would enhance his/her job perfor-
mance within an organizational content” (Davis, 1989, p. 320) and
as the degree to which an innovation is perceived as being better
than its precursor (Rogers, 2003). PU was found to be a strong factor
directly influencing the behavioral intention to use a technology in
various educational research contexts (Cigdem & Oncu, 2015; Ong
& Lai, 2006; Sun, Tsai, Finger, Chen, & Yeh, 2008; Van Raaij &
Schepers, 2008; Venkatesh & Davis, 2000). In this sense, a useful
WBT is attributed to increase perceived playfulness (Terzis &
Economides, 2011). Therefore, we hypothesized:
H2. PU would have a positive effect on the Behavioral Intention to
use WBT.
H3. PU would have a positive effect on PP.
4.3. Perceived ease of use (PEU)
Perceived Ease of Use (PEU), which is defined as “the degree to
which a person believes that using a particular system would be
free of effort” (Davis, 1989; Rogers, 2003), is the second major
variable of the TAM (Davis, 1989). A lot of research put forward
evidences on the positive impact of the PEU on behavioral inten-
tion, perceived usefulness and perceived playfulness (Terzis &
Economides, 2011; Venkatesh, 1999; Venkatesh & Davis, 1996). In
this context, as the level of PEU regarding an e-learning system
increases, the acceptance and use of that system by the participants
is most likely to increase (Teo, Lee, & Chai, 2008). In this line, we
had the following hypotheses:
H4. PEU would have a positive effect on the Behavioral Intention
to use WBT.
H5. PEU would have a positive effect on PU.
H6. PEU would have a positive effect on PP.
4.4. Computer self-efficacy (CSE)
Explained as an individual's feelings of his/her ability to use
computers (Compeau & Higgins, 1995), Computer Self Efficacy
(CSE) was claimed to have a direct effect on PEU and an indirect link
to the behavioral intention (Terzis & Economides, 2011). Building
on this claim, we also hypothesized:
H7. CSE would have a positive effect on PEU.
4.5. Social influence (SI)
Integrated into the CBAAM as a variable from the UTAUT (Teo,
2009; Teo et al., 2008) and used in the LMS acceptance models
frequently (Van Raaij & Schepers, 2008; Wang et al., 2009), Social
Influence (SI) is related to individuals' beliefs of how they are
influenced by the opinions and judgments of their colleagues,
friends, family members, and superiors (Fishbein & Ajzen, 1975;
Taylor & Todd, 1995; Venkatesh et al., 2003). SI was measured
through three key variables: Subjective Norm, Image and Volun-
tariness (Karahanna & Straub, 1999; Venkatesh et al., 2003). SI was
claimed to be one of the major determinants explaining behavioral
intention within certain acceptance models like TAM2 and UTAUT
and to have a positive effect on PU in LMS and WBT contexts (e.g.
Cigdem & Topcu, 2015; Terzis & Economides, 2011; Wang et al.,
2009). Although a significantly positive effect of SI only on PU
was found within the CBAAM, SI was said to determine users'
behavioral intentions indirectly through PU (Terzis & Economides,
2011). Our hypothesis for this concept was as follows:
H8. SI would have a positive effect on PU.
4.6. Facilitating conditions (FC)
Facilitating Conditions (FC) was also adopted from the UTAUT
(Teo, 2009; Teo et al., 2008) and used in the LMS acceptance models
(Van Raaij & Schepers, 2008; Wang et al., 2009). FC represents a
variety of factors that possibly influence a person's perceptions to
execute a course of action and depends on the system itself and its
providers. FC might stand for technical support such as helpdesks
and online support services; resource factors such as time and
money (Lu, Liu, Yu, & Wang, 2008); policies, regulations, and legal
environment of a system; communication activities and active
participation of organizational staff (Bueno & Salmeron, 2008).
 H. Cigdem et al. / Computers in Human Behavior 61 (2016) 522e531
Within the scope of the current research, FC was set to cover all the
support provided by the staff during the WBT and hypothesized to
have a positive effect on PEU.
H9. FC would have a positive effect on PEU.
4.7. Goal expectancy (GE)
The significance of the concepts like self-direction, self-man-
agement, self-discipline, goal orientation, and personal outcome
expectations (Smith, Murphy, & Mahoney, 2003; Yi & Hwang,
2003; Shih, 2008) has been mentioned in previous studies. Based
on Self-Management of Learning (Wang et al., 2009), Goal Expec-
tancy (GE) occurs as a variable influencing individuals' beliefs that
they are prepared appropriately to use a WBT through two di-
mensions: learners' satisfaction with their preparation for the WBT
and their desirable level of success. Goals, either specific or difficult,
serve a purpose towards high performance. Sometimes, greater
achievements come with more difficult goals. Therefore, it might
not be a good strategy for instructors to include easy assignments to
support learning experiences as loss of interest in the course or
even dropouts might occur as a result (Locke, 1996; Locke &
Latham, 1990). In a summative assessment within the scope of
the CBAAM, a positive effect of GE on PU and on PP was demon-
strated. In this framework, our hypotheses were:
H10. GE would have a positive effect on PU.
H11. GE would have a positive effect on PP.
4.8. Content (C)
Students are able to learn concepts and improve their perfor-
mances by answering sample practice questions on each concept
and taking continuous feedback on their responses (Moreno &
Mayer, 2007; Tennyson, 1980; Tennyson & Buttrey, 1980).
Learning concepts could depend on learners' attributes and their
prior knowledge (Tennyson & Park, 1980). Referring to the learners'
conceptualizations of the content of the WBT, Content (C) was also
adapted as a variable from previous studies to fit in the WBT context
(Shee & Wang, 2008; Wang, 2003). The items of WBT system were
based on EFL course content. If items in WBT were clear, under-
standable and relevant to the course content, then it was more
likely to expect utility and satisfaction by students (Terzis &
Economides, 2011). It was indicated that C could influence
perceived usefulness (Cigdem & Oncu, 2015), goal expectancy,
perceived playfulness, and behavioral intention to use a WBT sys-
tem (Terzis & Economides, 2011). Besides, it appeared as a signifi-
cant variable to determine e-learners' satisfaction (Wang, 2003). In
the light of all those points, we had the following hypotheses
regarding the Content:
H12. C would have a positive effect on PU.
H13. C would have a positive effect on PP.
H14. C would have a positive effect on GE.
H15. C would have a positive effect on the Behavioral Intention to
use WBT.
5. Method
5.1. Research context and participants
This study was conducted in a compulsory EFL course taught at a
525
post-secondary military school and with 1986 military students
from various academic programs. Although all the students
participated in the assessments, only 602 of them responded to the
online questionnaire.
Among those 602 students, 65 of them were from the depart-
ment of Computer Technology, 265 from the department of Elec-
tronics and Communication Technologies, 133 from the department
of Business Administration, 33 from the department of Electrics, 70
from the department of Mechatronics, and 36 from the department
of Construction. The participants' age ranged from 17 to 23 with an
average of 19.65.
The participants had two distinct listening tests. In the first one,
they were provided with 20 multiple-choice items and the mean
value for all the student groups was 62 out of 100. The second one
included a variety of tasks through 15 items such as drag and drop,
fill-in-the gaps, matching, and multiple-choice types (see Fig. 2).
The overall score of this test was not as high as the first one (M ¼ 45
out 100).
5.2. Data collection instrument
The data were collected quantitatively through an online
questionnaire uploaded on the learning management system
where the students are supposed to take their end-of-semester
listening comprehension exams. The questionnaire included 24
five-point Likert-type items which were adapted from the TAM
literature and administered in Turkish. Table 1 displays the cate-
gories within the scale and the items within each category except
for the items excluded from the analyses as they indicated poor
factorial loadings (i.e. <0.70). One item related to computer self-
efficacy (CSE4), one item related to perceived playfulness (PP3),
two items related to social influence (SI3 and SI4), and two items
related to content (C3 and C4) were excluded from the list.
5.3. Data analysis
As an initial step the sample size was checked. Taking a refer-
ence from the suggestion that a sample of more than 300 was set to
be satisfactory (Hair, Black, Babin, & Anderson, 2010); the sample
size of the current study was far beyond the requirements. As for
the actual analyses, a two-step strategy of structural equation
modeling (SEM) was performed in this study. Firstly, the method of
maximum likelihood (ML) was employed in the Analysis of
Moment Structures (AMOS) software. Next, a confirmatory factor
analysis (CFA) was performed to validate the quality of proposed
measures, and then a SEM was conducted to validate the entire
model in the AMOS.
6. Results
6.1. Validation of the data collection tool
In order to check the suitability of the dataset for factor analysis,
KaisereMeyereOlkin (KMO) coefficients on the scale were
computed as a value of 0.92 which indicated that factors would be
reliable and the scale would be suitable for factor analysis (Field,
2000). Later, a CFA based on the AMOS 21 was performed to vali-
date and confirm the research model. According to Hair et al.'s
(2010) recommendations, c2/df < 5 was set as the acceptable
level for good model fit. Besides, other multiple indicators were
used to obtain a more objective conclusion with the intention of
avoiding power problems by using the Chi-square test in a large
sample. Based on the present data, the modification indices for
covariances, which are available through the AMOS, suggested a
linkage between the two observed variables as shown in Fig. 3. The
526 H. Cigdem et al. / Computers in Human Behavior 61 (2016) 522e531

Fig. 2. A screenshot of computer based listening exam.

Table 1
Items and constructs.

Constructs Items Mean SD Factor loading

Perceived Usefulness (PU) (Davis, 1989), a ¼ .921 1.78 0.97

PU1 Using the Computer Based Assessment (CBA) will improve my work. 1.74 0.990 0.91
PU2 Using the CBA will enhance my effectiveness. 1.78 1.025 0.90
PU3 Using the CBA will increase my productivity. 1.82 1.103 0.88

Perceived Ease of Use (PEOU) (Davis, 1989), a ¼ .882 2.34 1.23

PEOU1 My interaction with the system is clear and understandable. 2.14 1.336 0.90
PEOU2 It is easy for me to become skillful at using the system. 2.18 1.357 0.89
PEOU3 I find the system easy to use. 2.70 1.425 0.75

Computer Self Efficacy (CSE) (Compeau & Higgins, 1995), a ¼ .853 3.42 1.08
CSE1 I could complete a job or task using the computer. 3.40 1.241 0.87
CSE2 I could complete a job or task using the computer if someone showed how to do it first. 3.59 1.265 0.84
CSE3 I can navigate easily through the Web to find any information I need. 3.29 1.210 0.72

Social Influence (Venkatesh et al., 2003), a ¼ .861 2.27 1.20

SI1 People who influence my behavior think that I should use CBA. 2.27 1.264 0.84
SI2 People who are important to me think that I should use CBA. 2.27 1.310 0.90

Facilitating Conditions (Thompson, Higgins, & Howell, 1991), a ¼ .803 2.56 1.27
FC1 When I need help to use the CBA, someone is there to help me. 2.54 1.407 0.81
FC2 When I need help to learn to use the CBA, system's help support is there to teach me. 2.58 1.387 0.83

Content (Terzis & Economides, 2011), a ¼ .824 1.58 0.91

C1 CBA's questions were clear and understandable. 1.66 1.055 0.84
C2 CBA's questions were easy to answer. 1.50 0.923 0.84

Goal Expectancy (Terzis & Economides, 2011), a ¼ .791 2.61 1.177

GE1 Courses' preparation was sufficient for the CBA 2.35 1.338 0.74
GE2 My personal preparation for the CBA. 2.67 1.383 0.80
GE3 My performance expectations for the CBA. 2.83 1.481 0.72

Perceived Playfulness (Moon & Kim, 2001), a ¼ .896 1.69 1.33

PP1 Using CBA keeps me happy for my task. 1.68 1.004 0.92
PP2 Using CBA gives me enjoyment for my learning. 1.68 1.010 0.90
PP4 Using CBA will lead to my exploration. 1.72 1.041 0.77

Behavioral Intention to Use CBA (Davis, 1989), a ¼ .925 2.00 1.66

BI1 I intend to use CBA in the future. 1.89 1.242 0.80
BI2 I predict I would use CBA in the future. 2.09 1.290 0.94
BI3 I plan to use CBA in the future. 2.05 1.244 0.96

final model returned c2 ¼ 549.378, df ¼ 215; c2/df ¼ 2.555 with (RMSEA) ¼ 0.051, goodness of fit index (GFI) ¼ 0.932, adjusted GFI
probability level p ¼ 0.00 < 0 .05. Several fit indices are reported (AGFI) ¼ 0.905, in which GFI and AGFI were greater than 0.80 and
here, including the root mean square error of approximation RMSEA was lower than 0.08. All these values suggested that the
 H. Cigdem et al. / Computers in Human Behavior 61 (2016) 522e531 527

Fig. 3. Confirmatory factor model.

measurement model fitted the data set well. The final model with Table 2
the estimated factor loadings is illustrated in Fig. 3. Composite reliability, average variance extracted, maximum shared squared vari-
ance, average shared square variance of constructs.
In CFA, Composite Reliability (CR), Average Variance Extracted
(AVE), Maximum Shared Squared Variance (MSV) and Average Constructs CR AVE MSV ASV
Shared Square Variance (ASV) were examined. According to Hair Perceived Usefulness 0.92 0.80 0.61 0.31
et al. (2010), high (above 0.70) CR values indicate a good reli- Perceived Ease of Use 0.89 0.73 0.38 0.26
ability. Table 2 shows that the CR of all the constructs of CBAAM was Computer Self Efficacy 0.86 0.67 0.28 0.15
Social Influence 0.86 0.76 0.27 0.18
found to be higher than 0.70 and the AVE of all the constructs of
Facilitating Conditions 0.80 0.67 0.27 0.16
CBAAM was found to be higher than 0.50. Next, convergent validity Content 0.83 0.71 0.58 0.27
and discriminant validity were examined. AVE being 0.50 or higher Goal Expectancy 0.80 0.57 0.15 0.12
indicated a good convergent validity. By using the methods rec- Perceived Playfulness 0.91 0.78 0.61 0.32
ommended by Fornell and Larcker (1981), the discriminant validity Behavioral Intention 0.93 0.82 0.45 0.24

was also tested; by imputing AMOS's correlations and standardized

528 H. Cigdem et al. / Computers in Human Behavior 61 (2016) 522e531

regression tables into the validity testing tool within the ‘Stats Tools Table 3
Package' (Gaskin, 2012). Results of hypotheses.

Table 2 shows that discriminant validity thresholds (MSV < AVE, Hypothesis Path Path coefficient Results
ASV < AVE), are realized for all constructs, indicating acceptable H1 PP [ BI 0.446* Supported
discriminant validity. H2 PU [ BI 0.192 Not Supported
H3 PU [ PP 0.466* Supported
H4 PEU [ BI 0.167* Supported
6.2. Test of the structural model H5 PEU [ PU 0.300* Supported
H6 PEU [ PP 0.020 Not Supported
H7 CSE [ PEU 0.362* Supported
After ensuring the validity of the constructs within the mea- H8 SI [ PU 0.241* Supported
surement model, the structural model was evaluated. To determine H9 FC [ PEU 0.390* Supported
the relationship of the constructs in the proposed model, the H10 GE [ PU 0.044 Not Supported
structural equation model was tested using the AMOS 21 with the H11 GE [ PP 0.069 Not Supported
H12 C [ PU 0.447* Supported
default maximum likelihood estimation method. The model H13 C [ PP 0.418* Supported
returned c2 ¼ 733.100, df ¼ 230; c2/df ¼ 3.187 with probability H14 C [ GE 0.346* Supported
level p ¼ 0.00 < 0 .05; and RMSEA ¼ 0.060, GFI ¼ 0.908, and H15 C [ BI -0.014 Not Supported
AGFI ¼ 0.880. All fit indices obtained in the present study showed
good structural model fit to the data for the proposed research
model. The resulting parameters of the research model are dis- content seemed to have a direct positive effect on perceived play-
played in Fig. 4. fulness; on the other hand, the effects of perceived ease of use and
Table 3 summarizes the results of our hypotheses. Inspecting the goal expectancy on perceived playfulness were not strong enough
model derived from the analyses, it could be clarified that there to exert statistically significant points. On the side of the perceived
appeared a direct positive effect of perceived playfulness and usefulness, goal expectancy did not create any positive effect; yet,
perceived ease of use on the construct of behavioral intention to use social influence, perceived ease of use, and content each contrib-
the WBT systems; whereas perceived usefulness did not exert a uted to perceived usefulness through a strong impact. Apart from
direct influence. Along with these points, perceived usefulness and

Fig. 4. Result of SEM (standardized estimates).

 H. Cigdem et al. / Computers in Human Behavior 61 (2016) 522e531
these, computer self-efficacy and facilitating conditions both had a
direct impact on perceived ease of use. Finally, goal expectancy was
directly and positively influenced by content.
7. Discussion
The major contribution of the current study was to look into the
variables influencing the students' intentions to use a WBT system
in the context of the foreign language education at a two-year post-
secondary higher education institution. During the study, it was
attempted to implement a WBT system specifically for an under-
estimated sub-skill, which is Listening Comprehension. Including
audio files made the design of the listening comprehension test
through the WBT system even much more difficult and time-
consuming, compared to other computer-based tests. Students'
perceptions on the WBT systems were also investigated within the
scope of this study. To illustrate, the factors within the CBAAM were
analyzed descriptively and it was seen that a great majority of the
mean scores were below the value of 3 and ranged from 1.50 to
3.59, except for the dimension of computer self-efficacy (see
Table 1). All those values descriptively indicated that the partici-
pants had generally negative attitudes towards the WBT systems,
which is apparently an inconsistent finding with some previous
studies (Cigdem & Oncu, 2015; Cigdem & Tan, 2014; Dermo, 2009;
Sorensen, 2013). It was deduced that students' getting low marks
from the listening comprehension tests administered through the
WBT system might have led them to perceive the process
negatively.
When the relationships between the variables were investi-
gated through the SEM, an explanatory model (see Fig. 5) was ob-
tained. The general results emerging in the model highlighted
perceived playfulness as the most important determinant of the
behavioral intention to use the WBT systems and perceived use-
fulness had only an indirect impact through perceived playfulness.
This finding was in line with the fundamental proposition of the
CBAAM.
As a parallel finding with Terzis and Economides (2011), the
results of the current study also indicated that perceived usefulness
did not exert a direct impact on behavioral intention; however, this
finding was controversial with the TAM literature, because some
prior studies (Cigdem & Oncu, 2015) suggested a very strong effect
of perceived usefulness on behavioral intention. This could mean
that participants' intention to use a computer-based listening
comprehension test would hardly be affected by their perceptions
Fig. 5. Depicted model for students' behavioral intention to use WBA.
529
on the usefulness of the system. This finding could have been
resulted from two other reasons. One is that the WBT aimed to test
listening comprehension. The students did not have sufficient
awareness of foreign language learning, because they are expected
to be low rank military personnel and not to have duties in inter-
national platforms. Another reason might be that students were
novice in using computer as a tool for an exam. Presenting a
consistent result with Terzis, Moridis, and Economides (2012), the
current study did not reveal any significant influence of perceived
ease of use on perceived playfulness, which might be discussed
through the characteristics of the contemporary users of such
systems; because they were novice in using computer as an
assessment tool although they took computer-related courses
previously. The difficulties experienced during the administration
of the computer-based listening assessment might cause a decrease
in the perceived ease of use and perceived playfulness.
In line with the previous studies such as Cigdem and Topcu
(2015), Terzis and Economides (2011); Terzis et al. (2012);
Venkatesh and Davis (2000), social influence was attributed to be
a critical determinant of perceived usefulness. Similar findings
regarding the previous studies might be explained by the social
sharing related to the exam format and item types before the exam
sessions.
In addition to those points, it was seen that the content of the
items in the test seemed to affect significantly the perceived use-
fulness of the system as in the study of Cigdem and Oncu (2015) and
Terzis and Economides (2011), who suggested that users would
tend to consider the system as useful when their perceptions about
the content of the questions in a WBT system were positive. At this
point, low scores on the content meant that students perceived the
test items harder than they expected and more time-consuming
than they imagined. However, preparing items that are easier to
comprehend and handle could probably have a positive impact on
the use of a computer-based listening comprehension test.
8. Conclusion
This study examined the constructs that influence students'
behavioral intentions to use WBT in a military vocational college
and concluded that perceived playfulness and perceived ease of use
exerted direct effects on behavioral intention to use WBT. Accord-
ingly, when the students feel that WBT is playful enough and easy
to use, their behavioral intention to use such a system seems
stronger. Bearing this in mind, certain techniques adopted to
530 H. Cigdem et al. / Computers in Human Behavior 61 (2016) 522e531
improve the dimensions of perceived playfulness and perceived
ease of use with other means would have a positive impact on the
systems' being accepted. Besides, some game-based or real life-
oriented themes might be embedded into the items so that they
could facilitate the comprehension and thus the students could find
the procedures more playful and easy to use.
Perceived usefulness had no direct impact on behavioral
intention to use WBT which contradicted with the TAM literature.
Goal expectancy was positively impacted by content of questions.
Furthermore, computer self-efficacy and facilitating conditions had
direct effect on perceived ease of use. Perceived usefulness was
positively impacted by content of test items, perceived ease of use
and social influence. Finally, perceived usefulness and content of
test items had direct impact on perceived playfulness.
To sum up, WBT is more likely to be playful when it is useful and
the content of WBT items tends to be easier to understand. Most
practical implications for foreign language educators might be that
the content of the audio items should be based on a problem or a
case or a real life situation. This might increase the intention to use
such an application as a result of an increase in the perceived ease
of use and playfulness.
9. Limitations
A wide range of web-based testing systems exist across different
educational settings such as the ones that incorporate videos,
graphics, animations and simulations. Yet in the current study, the
students were tested on their listening comprehension and audios
were incorporated into the assessment procedures. Listening
comprehension is among the most challenging skills both in
teaching and testing procedures; thus, it is possible that test-takers
have difficulties in all types of listening comprehension tests
whether it is paper-based or computer-based. It is also acceptable
that test-takers reflect different interpretations of and attitudes
towards various types of computer-based assessment, but their
perceptions on a computer-based listening comprehension test is
expected to be different from the ones in other subjects. Therefore,
the findings ought to be interpreted through the lenses of the
subject of ‘listening comprehension' and of the domain of ‘military
students' having no powerful targets related to learning English.
The study on the whole limited to the research context, which is
a two-year post-secondary military vocational college. Broadening
the scope with other post-secondary institutions across various
disciplines would add more value to similar studies. Gender issue
was another limitations as the male military students outnumbered
the females. A future study could consider eliminating the imbal-
ance of gender by including more female students.
References
Abedi, J. (2014). The use of computer technology in designing appropriate test ac-
commodations for English language learners. Applied Measurement in Education,
27(4), 261e272.
Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human
Decision Processes, 50(2), 179e211.
Ajzen, I., & Fishein, M. (1980). Understanding attitudes and predicting social behav-
iour. Englewood Cliffs, NJ: Prentice-Hall.
Bueno, S., & Salmeron, J. L. (2008). TAM-based success modeling in ERP. Interacting
with Computers, 20(6), 515e523.
Bull, J., & McKenna, C. (2004). Blueprint for computer-assisted assessment. London:
Routledge-Falmer.
Chou, C., Moslehpour, M., & Le Huyen, N. T. (2014). Concurrent and predictive
validity of computer-adaptive freshman English test for college freshman En-
glish in Taiwan. International Journal of English Language Education, 2(1),
143e156.
Cigdem, H., & Oncu, S. (2015). E-assessment adaptation at a military vocational
college: student perceptions. Eurasia Journal of Mathematics, Science & Tech-
nology Education, 11(5), 971e988.
Cigdem, H., & Tan, S. (2014). Students' opinions on administering optional online
quizzes in a two-year college Mathematics course. Journal of Computer and
Education Research, 2(4), 51e73.
Cigdem, H., & Topcu, A. (2015). Predictors of instructors' behavioral intention to use
learning management system: a Turkish vocational college example. Computers
in Human Behavior, 52, 22e28.
Compeau, D. R., & Higgins, C. A. (1995). Computer self-efficacy: development of a
measure and initial test. MIS Quarterly, 19(2), 189e211.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance
of information technology. MIS Quarterly, 13(3), 319e340.
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer
technology: a comparison of two theoretical models. Management Science, 35,
982e1003.
Dermo, J. (2009). E-assessment and the student learning experience: a survey of
student perceptions of e-assessment. British Journal of Educational Technology,
40(2), 203e214.
Field, A. (2000). Discovering statistics using SPSS for windows. London, UK: Sage.
Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention and behavior: An intro-
duction to theory and research. Reading, MA: Addison-Wesley.
Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with un-
observable variables and measurement error. Journal of Marketing Research,
18(1), 39e50.
Gaskin, J. (2012). Stats wiki and stats tools package. http://statwiki.kolobkreations.
com/.
Gikandi, J. W., Morrow, D., & Davis, N. E. (2011). Online formative assessment in
higher education: a review of the literature. Computers & Education, 57(4),
2333e2351.
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis
(7th ed.). Englewood Cliffs: Prentice Hall.
Jeong, H. (2014). A comparative study of scores on computer-based tests and paper-
based tests. Behaviour & Information Technology, 33(4), 410e422.
Karahanna, E., & Straub, D. W. (1999). The psychological origins of perceived use-
fulness and ease of use. Information and Management, 35, 237e250.
Llamas-Nistal, M., Fern andez-Iglesias, M. J., Gonza lez-Tato, J., & Mikic-Fonte, F. A.
(2013). Blended e-assessment: migrating classical exams to the digital world.
Computers & Education, 62, 72e87.
Locke, E. A. (1996). Motivation through conscious goal setting. Applied and Pre-
ventive Psychology, 5, 117e124.
Locke, E. A., & Latham, G. P. (1990). A theory of goal setting & task performance.
Englewood Cliffs, NJ: Prentice Hall.
Lu, J., Liu, C., Yu, C., & Wang, K. (2008). Determinants of accepting wireless mobile
data services in China. Information & Management, 45(1), 52e64.
Moon, J., & Kim, Y. (2001). Extending the TAM for a world-wide-web context. In-
formation and Management, 38(4), 217e230.
Moreno, R., & Mayer, R. (2007). Interactive multimodal learning environments.
Special issue on interactive learning environments: contemporary issues and
trends. Educational Psychology Review, 19, 309e326.
Morris, D. (2008). Economics of scale and scope in e-learning. Teaching in Higher
Education, 33(3), 331e343.
Ong, C., & Lai, J. (2006). Gender differences in perceptions and relationships among
dominants of e-learning acceptance. Computers in Human Behaviour, 22(5),
816e829.
Pino-Silva, J. (2008). Student perception of computerized tests. ELT Journal, 62(2).
Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Free Press.
Shee, D. Y., & Wang, Y.-S. (2008). Multi-criteria evaluation of the web-based e-
learning system: a methodology based on learner satisfaction and its applica-
tions. Computers & Education, 50(3), 894e905.
Shih, H. (2008). Using a cognitive-motivation-control view to assess the adoption
intention for web-based learning. Computers & Education, 50(1), 327e337.
Smith, P. J., Murphy, K. L., & Mahoney, S. E. (2003). Towards identifying factors
underlying readiness for online learning: an exploratory study. Distance Edu-
cation, 24(1), 57e67.
Sorensen, E. (2013). Implementation and student perceptions of e-assessment in a
chemical engineering module. European Journal of Engineering Education, 38(2),
172e185.
Sun, P., Tsai, R. J., Finger, G., Chen, Y., & Yeh, D. (2008). What drives a successful e-
learning? an empirical investigation of the critical factors influencing learner
satisfaction. Computers & Education, 50, 1183e1202.
Taylor, S., & Todd, P. (1995). Assessing it usage: the role of prior experience. MIS
Quarterly, 19(4), 561e570.
Tennyson, R. D. (1980). Instructional control strategies and content structure as
design variables in concept acquisition using computer-based instruction.
Journal of Educational Psychology, 72, 525e532.
Tennyson, R. D., & Buttrey, T. (1980). Advisement and management strategies as
design variables in computer-assisted instruction. Educational Communication
and Technology Journal, 28, 169e176.
Tennyson, R. D., & Park, D. C. (1980). The teaching of concepts: a review of
instructional design literature. Review of Educational Research, 50, 55e70.
Teo, T. (2009). Modelling technology acceptance in education: a study of pre-service
teachers. Computers & Education, 52(1), 302e312.
Teo, T., Lee, C. B., & Chai, C. S. (2008). Understanding pre-service teachers' computer
attitudes: applying and extending the technology acceptance model. Journal of
Computer Assisted Learning, 24(2), 128e143.
Terzis, V., & Economides, A. A. (2011). The acceptance and use of computer based
assessment. Computers & Education, 56(4), 1032e1044.
Terzis, V., Moridis, C. N., & Economides, A. A. (2012). How Student's personality
 H. Cigdem et al. / Computers in Human Behavior 61 (2016) 522e531
traits affect computer based assessment acceptance: integrating BFI with
CBAAM. Computers in Human Behavior, 28(5), 1985e1996.
Terzis, V., Moridis, C. N., & Economides, A. A. (2013). Continuance acceptance of
computer based assessment through the integration of user's expectations and
perceptions. Computers & Education, 62, 50e61.
Thompson, R., Higgins, C., & Howell, J. (1991). Personal computing: toward a con-
ceptual model of utilization. MIS Quarterly, 15(1), 124e143.
Van Raaij, E. M., & Schepers, J. J. L. (2008). The acceptance and use of a virtual
learning environment in China. Computers & Education, 50(3), 838e852.
Venkatesh, V. (1999). Creation of favorable user perceptions: exploring the role of
intrinsic motivation. MIS Quarterly, 23, 239e260.
Venkatesh, V., & Davis, F. D. (1996). A model of the antecedents of perceived ease of
use: development and test. Decision Sciences, 27, 451e481.
Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology
acceptance model: four longitudinal field studies. Management Science, 46,
531
186e204.
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of
information technology: toward a unified view. MIS Quarterly, 27(3), 425e478.
Wang, Y. (2003). Assessment of learner satisfaction with asynchronous electronic
learning systems. Information and Management, 41(1), 75e86.
Wang, Y.-S., Wu, M.-C., & Wang, H.-Y. (2009). Investigating the determinants and
age and gender differences in the acceptance of mobile learning. British Journal
of Educational Technology, 40(1), 92e118.
Yi, M. Y., & Hwang, Y. (2003). Predicting the use of web-based information systems:
self-efficacy, enjoyment, learning goal orientation, and the technology adoption
model. International Journal of Human Computer Studies, 59(4), 431e449.
Zakrzewski, S., & Steven, C. (2000). A model for computer-based assessment: the
catherine wheel principle. Assessment & Evaluation in Higher Education, 25(2),
201e215.