Professional Documents
Culture Documents
www.emeraldinsight.com/0951-354X.htm
IJEM
24,2 Harnessing ICT potential
The adoption and analysis of ICT systems for
enhancing the student learning experience
116 Shane Dawson
Graduate School of Medicine, University of Wollongong, Wollongong, Australia
Liz Heathcote
Heathcote Consulting, Wollongong, Australia, and
Gary Poole
Centre for Teaching and Academic Growth, University of British Columbia,
Vancouver, Canada
Abstract
Purpose – This paper aims to examine how effective higher education institutions have been in
harnessing the data capture mechanisms from their student information systems, learning
management systems and communication tools for improving the student learning experience and
informing practitioners of the achievement of specific learning outcomes. The paper seeks to argue
that the future of analytics in higher education lies in the development of more comprehensive and
integrated systems to value add to the student learning experience.
Design/methodology/approach – Literature regarding the trend for greater accountability in
higher education is reviewed in terms of its implications for greater “user driven” direction. In
addition, IT usage within higher education and contemporary usage of data captured from various
higher education systems is examined and compared to common commercial applications to suggest
how higher education management and teachers can gain greater understanding of the student cohort
and personalise and enhance the learning experience much as commercial entities have done for their
client base. A way forward for higher education is proposed.
Findings – If the multiple means that students engage with university systems are considered, it is
possible to track individual activity throughout the entire student life cycle – from initial admission,
through course progression and finally graduation and employment transitions. The combined data
captured by various systems builds a detailed picture of the activities students, instructors, service
areas and the institution as a whole undertake and can be used to improve relevance, efficiency and
effectiveness in a higher education institution.
Originality/value – The paper outlines how academic analytics can be used to better inform
institutions about their students learning support needs. The paper provides examples of IT
automation that may allow for student user-information to be translated into a personalised and
semi-automated support system for students.
Keywords Communication technologies, E-learning, Higher education
Paper type Conceptual paper
HE and ICTs
The emphasis on the demonstration of quality services comprises only one component
of the multiple demands placed on the contemporary HE institution. While universities
are challenged with the need for streamlining data capture and reporting processes
they are also struggling to cater to the educational needs of current and future students
entering a changing and uncertain workforce. In this business-oriented climate, the
adoption of ICT has received considerable attention within the education literature as a
primary instrument in the strategy for responding to the demands associated with
consumerism whilst also facilitating the student learning experience (Hosie and
Mazzarol, 1999; Mazzarol et al., 2003). In essence, the scalable adoption of ICT
embraces both commercialisation and pedagogical agendas. For instance, as
previously noted the reductions in government financial support for the HE sector
has forced universities to seek alternate sources of revenue. One such source has been
IJEM the international student market (Mazzarol et al., 2003). The large-scale adoption of
24,2 online technologies has enabled the sector to expand its reach into new markets while
minimising operational costs (Hanna, 1998; Mazzarol et al., 2003). This expansion has
not only impacted upon the delivery of education resources to off-campus students, but
also on the design and accessibility of learning activities available for on-campus
cohorts (Coates et al., 2005). The rising costs of education and the subsequent
120 requirement for student part-time employment have required the blending of more
traditional modes of education delivery with online ICTs, such as learning
management systems (LMS) (Hosie and Mazzarol, 1999). The adoption of ICTs to
provide increased accessibility to discipline content and contact with peers and
teaching staff is no longer a luxury, but a necessity for all institutions. While this
presents a very simplified explanation for the rapid expansion of ICTs in education, it
does serve to illustrate that the vast majority of universities have at their disposal a
tool that not only provides flexible access and contact with the organisation but also
tracks and records the patterns of behaviour undertaken by particular demographics.
It is this latter affordance that is currently gaining increasing interest amongst the
academic community (Campbell et al., 2007; Dawson, 2007b; Goldstein and Katz, 2005).
Academic analytics
HE institutions are now beginning to acknowledge the additional operational benefits
derived from data mining, analysis and reporting. Campbell and Oblinger (2007)
discuss the heightened level of accountability placed upon universities and the
capacity for ICT systems to provide alternate data sets by which to measure quality
and assist the organisation in their decision making processes. For instance the authors
state that:
As the demand for accountability grows, institutions are being asked to present data that
document their accomplishments. Data can also be used to guide internal improvement. This
culture of accountability puts new pressures on higher education, but these pressures also
create new opportunities for colleges and universities to reexamine the processes and tools
they use for decision making (Campbell and Oblinger, 2007, p. 2).
Although universities have mined enterprise ICT systems for reporting purposes, the
emphasis has been predominantly associated with transactions and operations in lieu
of more predictive modeling and informing teacher-student and student-student
interactions (Goldstein and Katz, 2005). As Goldstein and Katz (2005) report:
Academic analytics in higher education remains in its infancy, or perhaps in early childhood.
The potential, however, is great, and it is likely that the times will demand more of our data
and the systems that manage it (p. 9).
In essence, the focus of the data collation and aggregation has been on refining and
enhancing the processes for reporting, marketing and attracting additional sources of
revenue. Figure 1 illustrates the relationship between data captured within the adopted
LMS and the range of potential information consumers – from institutional
administration using the data for strategic decision making, to students having access
to just-in-time support resources based on their past academic performance. Despite
the depth and range of data available and its capacity to inform a diversity of end-users
there have been minimal and localised applications of this data within HE circles. For
example, the shaded component of Figure 1 illustrates the type of commonly utilised
Harnessing ICT
potential
121
Figure 1.
Example ICT data from
institutional LMS, the
derived information and
possible end-consumers
and reported usages of the available LMS data (adapted from Heathcote and Dawson,
2005; Heathcote and Prakash, 2004).
Given the increasing importance for HE institutions to rapidly respond to the
student cohort social and academic needs, and the high degree of investment in
researching student learning, it is surprising that the information on student behaviour
captured by the LMS has been so rarely interrogated and adopted beyond basic load
and tool usage. The quantity and diversity of data available regarding student online
learning behaviour, interactions with peers and teaching staff and access to other
institutional ICT systems (student services, library, etc.) for example, affords an
opportunity for integrating automated student learning support and services. For
example, early identification of student disengagement in the online environment may
indicate an “attrition-risk” thereby triggering various learning support mechanisms
(such as: academic writing support; tutoring, library resources, staff timetables and
calendar) while also notifying teaching staff of the need for developing and
implementing additional learning interventions.
To date there has been a lack of research regarding the application of academic
analytics to inform the design, delivery and future evaluations of individual teaching
practices. Although the majority of commercial LMS have the in-built capacity to
report on specific types of user activity, e.g. unique users, session login times, resource
usage and potentially, navigational pathways, this data has been commonly only
reported at the request of individual instructors monitoring class activity. For instance
WebCT Vista reports individual student activity (Figure 2), yet comparisons and
averages with fellow class peers or with courses operating in the same program are not
readily available. While data relating to class activity provides information to assist
instructors in the ongoing design and implementation of teaching practices, access to
comparative data sets accessible by all staff and students provides a rich composite of
information to benchmark performance and establish predictors related to the
implemented learning activities, not only within an individual course, but across
multiple courses, programs and faculties.
Goldstein and Katz (2005) and more recently Campbell and Oblinger (2007) have
commenced discussions and research with educational communities such as
EDUCAUSE regarding the broader systemic use of ICT data for both accountability
IJEM
24,2
122
Figure 2.
Individual student reports
available from WebCT
Vista
reporting and enhancing the learning and teaching agenda. In a similar vein Dawson
(2006a, 2006b) has also emphasised the value of accessing ICT data for establishing
potential lead indicators for evaluating various learning and teaching outcomes.
Dawson (2006a, 2006b) demonstrated that ICT data such as discussion forum postings
can be used as a predictor of student community, thereby assisting staff with the
ongoing refinement of learning activities designed to foster a learning community
through access to just-in-time formative ICT data. Similarly, Morris et al. (2005)
identified a correlation between session length of individual online user engagement
and final assessment scores.
An important aspect of this reportage data is the ability to identify patterns of user
behaviour. Although currently lacking in most commercial LMS, the capacity to
present individual user activity within a broader context provides a more robust
analysis and pattern recognition. Anomalies in these behaviours may serve to provide
a catalyst for proactive instructor intervention. For example, Student A actively
engages in the online learning environment on a regular basis. However, the system
informs the instructor of an absence of login activity. The instructor has the option to
personally contact the individual student, offer assistance, and potentially avoid
attrition from the course. Although this may be an extreme example, the power of the
system to focus on students at highest risk and for the teaching team to harness
resources to respond to the cohort dynamics can be seen to offer a greater suite of
personalized learning resources that were previously not feasible in large class sizes.
Campbell et al. (2007) report on the use of academic analytics at Purdue University in
order to identify potential “at risk” students. The predictive model is based on past
academic results (aptitude) and the analysis of student behaviour within the course
management system (effort). The authors note that the next phase of development will
be to implement early indicators of at risk students in order to provide them with
additional resources to facilitate progression towards patterns of behaviour that Harnessing ICT
represent a low risk category (e.g. participation in group discussions, regular access of potential
discipline content and review of assessment criteria).
An additional example lies in the ability to aggregate LMS data around central
themes to identify and target areas requiring additional learning support through staff
development agencies. Burnett and Dawson (2005) argued that staff uptake of LMS
resources is often impeded by an inability for teaching staff to translate technical 123
resources to pedagogical functions. The inclusion of data visualisation of educator and
faculty benchmarks affords teaching staff the opportunity to monitor and track the
impact of designed learning activities – both individual and at a faculty level. Thus,
practitioners can identify the impact of designed learning activities on student
behaviour (engagement) in a timely and on going manner. The visualisation of student
engagement data can assist staff in linking the function of technical resources with
implemented pedagogy.
These examples illustrate the types of data captured and the value of its analysis
that is generally inaccessible to more traditional modes of teaching practice (e.g.
face-to-face instruction). This is not to say that online learning practices should replace
learning activities such as lectures and tutorials, but that the data captured by LMS
offers new insights into the scholarship of teaching and learning. However, the current
reportage capabilities of commercial LMS tends to present what is perceived as
disconnected quantitative data sets (e.g. page hits, total postings, time online, etc.) in
lieu of more learning oriented data visualisation methods. In a recent presentation
Dawson (2007a) demonstrated the interpretative value when grouping BlackBoard
Vista tools into categories of teaching administration (e.g. calendar, tracking, file
manager), assessment (e.g. assignment, my grades), content (e.g. file, content page,
weblinks, notes, folder) and student interaction (e.g. chat, discussion, mail, who is
online) for the purposes of analysis. While these groupings are subjective they are
based on the tool’s primary designed function. The presentation of these grouped data
sets provides teaching staff with a more readily interpretable interface for evaluating
their teaching practice.
125
Figure 3.
IJEM “advertising links” for further information and research. Similarly, areas of student
24,2 and staff support could be tagged for inclusion in questions concerning assessment
processes, academic writing, plagiarism, etc. These features can be seen to assist
students with developing individual literacy skills and fostering independent learning
and self-evaluation, while raising awareness of university support services.
The inclusion of network visualisation tools can be used to assist staff in the rapid
126 identification of students disconnected from the learning community (Figure 4).
Teaching staff can also use the network mapping to reflect on the design, timing and
evaluation of implemented learning activities designed to achieve a set pedagogical
agenda such as the generation of a learning community. Tightly clustered networks
can be seen to be indicative of a learning community (Dawson, 2008).
Surveillance
This paper has presented a rather optimistic view regarding the adoption and
integration of various student data sources. Despite this optimistic presentation, the
inclusion of any added surveillance technologies must be undertaken with caution and
reservation. In relation to the implementation and adoption of surveillance
technologies David Lyon (2001) explores the tension between discourses of societal
safety and societal control, or what Lyon has termed, the “two faces of surveillance”.
The author argues that increasingly sophisticated and often overt surveillance
measures are commonly adopted and subsequently tolerated through justifications of
public safety. For example, the inclusion of video surveillance in urban shopping malls.
In this paper we present a case for enhancing the student learning experience through
the integration of currently disparate data sources. While the data discussed are
currently available, the integration and usage of these data requires consideration of:
student and staff consent; ethics; implementation of governance policies; and ongoing
stakeholder participation.
Conclusion
The business sector has commonly used the data captured through daily customer and
client interactions in order to better inform strategic decisions and to enhance the
end-users experience. Although HE institutions have invested large sums in various
Figure 4.
Example student social
network (sociogram)
derived from ICT
discussion interactions
enterprise systems, US$5 billion in North America alone (Goldstein and Katz, 2005), Harnessing ICT
there remains a great deal of untapped potential in integrating currently disparate potential
institutional systems to better inform teaching practice and enhance the student
learning experience. While student and instructor data is currently readily available for
extraction from these disparate ICT systems, the next step in the development of
academic analytics is to compile these sources into one domain (data warehouse) where
the specific data queries and applications can be developed. A core premise underlying 127
the extraction of ICT data is the linkage from data interpretation of results back to
informing teaching and learning practice. The development of academic analytics and
the extraction of student usage data will continue to evolve at a rapid pace, in part due
to the increasing level of sophistication of the technologies adopted. However, an
understanding of the relationship between the analytical data and the implemented
pedagogical models is still in its infancy. The impact of specific teaching and learning
practices on student behaviour (observed through ICT interactions) requires further
research and investment to ensure that the interpretation of extracted data is
appropriately aligned with the desired pedagogical action.
References
Bates, T. (2005), Technology, E-learning and Distance Education, 2nd ed., Routledge, Abingdon.
Burnett, B. and Dawson, S. (2005), “Aligned and resistant communities: exploring new conduits
for online engagement”, paper presented at the OLT 2005: Beyond delivery, Brisbane,
29 September.
Campbell, J. and Oblinger, D. (2007), “Academic analytics”, available at: http://connect.educause.
edu/library/abstract/AcademicAnalytics/45275 (accessed 25 October 2007).
Campbell, J., De Blois, P.B. and Oblinger, D. (2007), “Academic analytics: a new tool for a new
era”, EDUCAUSE Review, Vol. 42 No. 4, pp. 42-57.
Chalmers, D. (2007), “A review of Australian and international quality systems and indicators of
learning and teaching”, available at: www.carrickinstitute.edu.au/carrick/webdav/site/
carricksite/users/siteadmin/public/t&lindicators_report_nov07.pdf (accessed 20 February
2008).
Cho, Y.H., Kim, J.K. and Kim, S.H. (2002), “A personalized recommender system based on web
usage mining and decision tree induction”, Expert Systems with Applications, Vol. 23 No. 3,
pp. 329-42.
Coates, H., James, R. and Baldwin, G. (2005), “A critical examination of the effects of learning
management systems on university teaching and learning”, Tertiary Education and
Management, Vol. 11 No. 1, pp. 19-36.
Dawson, S. (2006a), “Online forum discussion interactions as an indicator of student
community”, Australasian Journal of Educational Technology, Vol. 22 No. 4, pp. 495-510.
Dawson, S. (2006b), “Relationship between student communication interaction and sense of
community in higher education”, Internet and Higher Education, Vol. 9 No. 3, pp. 153-62.
Dawson, S. (2007a), “Evaluating behind closed doors: the application of student online user
behaviour to inform and guide teaching practice”, paper presented at the Teaching and
Learning with Technology Series, University of British Columbia, 15 November.
Dawson, S. (2007b), “Juxtaposing community with learning: the relationship between learner
contributions and sense of community in online environments”, unpublished PhD thesis,
Queensland University of Technology, Brisbane.
IJEM Dawson, S. (2008), “A study of the relationship between student social networks and sense of
community”, Educational Technology and Society, Vol. 11 No. 3, pp. 224-38.
24,2 Edelstein, H. (2000), “Building profitable customer relationships with data mining:
white paper-executive briefing”, SPSS, Woking.
Goldstein, P.J. and Katz, R.N. (2005), “Academic analytics: the uses of management information
and technology in higher education”, available at: www.educause.edu/ir/library/pdf/
128 ers0508/rs/ers0508w.pdf (accessed 25 October 2007).
Hanna, D. (1998), “Higher education in an era of digital competition: emerging organizational
models”, Journal of Asynchronous Learning Networks, Vol. 2 No. 1, pp. 66-95.
Heathcote, E. and Dawson, S. (2005), “Data mining for evaluation, benchmarking and reflective
practice in an LMS”, paper presented at the E-Learn 2005: World Conference on E-learning
in Corporate, Government, Health and Higher Education, Vancouver, 24-28 October.
Heathcote, E. and Prakash, S. (2004), “What your learning management system is telling you
about supporting your teachers: monitoring system information to improve support for
teachers using educational technologies at Queensland University of Technology”, paper
presented at the International Conference on Information Communication Technologies in
Education, Samos Island.
Hosie, P. and Mazzarol, T. (1999), “Using technology for the competitive delivery of education
services”, Journal of Computer Assisted Learning, Vol. 15 No. 2, pp. 118-28.
Lyon, D. (2001), Surveillance Society: Monitoring Everyday Life, Open University Press,
Buckingham.
Marginson, S. and van der Wende, M. (2007), “To rank or to be ranked? The impact of global
rankings in higher education”, Journal of Studies in International Education, Vol. 11
Nos 3/4, pp. 306-29.
Mazzarol, T., Soutar, G.N. and Seng, M.S.Y. (2003), “The third wave: future trends in
international education”, International Journal of Educational Management, Vol. 17 No. 3,
pp. 90-9.
Mobasher, B., Dai, H., Luo, T., Sun, Y. and Zhu, J. (2000), “Integrating web usage and content
mining for more effective personalization”, paper presented at the 1st International
Conference on Electronic Commerce and Web Technologies, Greenwich.
Morris, L.V., Finnegan, C. and Wu, S. (2005), “Tracking student behavior, persistence, and
achievement in online courses”, Internet and Higher Education, Vol. 8 No. 3, pp. 221-31.
New Media Consortium and EDUCAUSE Learning Initiative (2007), Horizon Report, 2007
Edition, The New Media Consortium, Austin, TX.
Nguyen, N.D., Yoshinari, Y. and Shigeji, M. (2005), “University education and employment in
Japan: students’ perceptions on employment attributes and implications for university
education”, Quality Assurance in Education, Vol. 13 No. 2, pp. 202-18.
Pyle, D. (1999), Data Preparation for Data Mining, Morgan Kaufmann, San Francisco, CA.
van Dyk, L. and Conradie, P. (2007), “Creating business intelligence from course management
systems”, Campus-Wide Information Systems, Vol. 24 No. 2, pp. 120-33.
Corresponding author
Shane Dawson can be contacted at: sdawson@uow.edu.au