This article was downloaded by: [Heriot-Watt University]

On: 06 October 2014, At: 08:20

Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer
House, 37-41 Mortimer Street, London W1T 3JH, UK

International Research in Geographical and

Environmental Education
Publication details, including instructions for authors and subscription information:
http://www.tandfonline.com/loi/rgee20

Development of Evaluation Tools for GIS: How Does

GIS Affect Student Learning?
a b c
Sophia Linn , Joseph Kerski & Sarah (Sally) Wither
a
Fort Collins , Colorado, USA
b
US Geological Survey , Denver, Colorado, USA
c
Orton Family Foundation Community Mapping Program , Steamboat Springs, Colorado,
USA
Published online: 22 Dec 2008.

To cite this article: Sophia Linn , Joseph Kerski & Sarah (Sally) Wither (2005) Development of Evaluation Tools for GIS:
How Does GIS Affect Student Learning?, International Research in Geographical and Environmental Education, 14:3,
217-224, DOI: 10.1080/10382040508668354

To link to this article: http://dx.doi.org/10.1080/10382040508668354

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained
in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of
the Content. Any opinions and views expressed in this publication are the opinions and views of the authors,
and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied
upon and should be independently verified with primary sources of information. Taylor and Francis shall
not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other
liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or
arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematic
reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any
form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://
www.tandfonline.com/page/terms-and-conditions
 Color profile: Disabled
Composite Default screen IRGEE 178d

Development of Evaluation Tools for GIS:

How Does GIS Affect Student Learning?

Sophia Linn
Fort Collins, Colorado, USA

Joseph Kerski
US Geological Survey, Denver, Colorado, USA

Sarah (Sally) Wither

Downloaded by [Heriot-Watt University] at 08:20 06 October 2014

Orton Family Foundation Community Mapping Program, Steamboat

Springs, Colorado, USA
Although Geographic Information Systems (GIS) are viewed as increasingly
valuable for educational use, few tools have been developed to assess the effects
of GIS on student learning (Barstow et al., 1994; Kerski, 2003). A joint project of
the Colorado Geographic Alliance (COGA) and the USGS focuses on the devel-
opment of evaluation tools to determine what, if anything, students gain from
using GIS in a classroom setting. Anecdotal evidence suggests that there exist
tangible benefits for students – whether in content understanding or in attitudes
– but it is necessary for tools to be created to measure such effects. Teachers, prin-
cipals, and other district administrators need to see measurable benefits in using
new technologies in order to determine whether investments in equipment and
staff training is justified. How can results be measured unless there are reliable
tools that can measure them?
A grant from the Colorado Geography Education Fund provided support to
initiate the development of such tools. As part of the grant, scholarships were
awarded to 10 teachers to participate in a week-long, intensive GIS institute for
educators during the summer of 2002. In addition to providing the support for
introductory training, the grant supported a follow-up session in which these
teachers, along with others who had used GIS in their classrooms, gathered to
reflect upon their observations of students using GIS in their classes. Seventeen
geography and science teachers were guided in their discussion by an external
evaluator from the University of Northern Colorado in an effort to develop eval-
uation instruments that would extract what students are gaining in their use of
GIS. A lively and sometimes impassioned discussion ensued that resulted in a
rather substantive list of areas in which teachers ‘suspect’ students are being
impacted by the technology.
The group worked through a structured process in order to build a model of
their collective thoughts, with the goal of describing and quantifying the end
result of using GIS in the classroom. Throughout the discussion, participants
reflected on their own classroom experiences while being guided and focused by

1038-2046/05/03 0225-06 $20.00/0 © 2005 S. Linn et al.

International Research in Geographical and Environmental Education Vol. 14, No. 3, 2005

217

C:\Documents and Settings\Anne\My Documents\Irgee\14-3\irgee14-3.vp

Forum
16 March 2006 10:14:28

IRGEE 178d
 Color profile: Disabled
Composite Default screen
218 International Research in Geographical and Environmental Education
Table 1 Quantitative rubric
3 2
Questions Student asks geographic
questions, such as ‘where?’
and ‘why there?’ on his/her
own. Questions are related to
the problem. Able to revise
questions as new data comes
in. Research questions are
appropriate to the tools, time,
and task.
Information Student determines data
needed to answer question.
Student selects or creates
primary, secondary, and
Downloaded by [Heriot-Watt University] at 08:20 06 October 2014
tertiary sources of data
appropriately. Student is able
to justify data collection
methods and the data chosen
to be collected. Student
observes and systematically
records information. Student
reads and interprets maps,
and conducts interviews as
needed to collect data.
Organising Student organises and
Data displays geographic data in
ways that help with the
analysis and interpretation of
it. Organising the data may
include using and developing
maps, graphs, tables, and
timelines with available tools.
Interpretation Student is able to see patterns,
and Analysis relationships and connections
among the collected and
organised data. Student is able
to describe the data and ask
alternative questions about it.
Student notes associations and
similarities between areas,
recognise patterns in the data,
and draw inferences. Student
uses simple statistics to
identify trends, sequences,
correlations, and relationships.
Conclusions or Student formulates
Solutions explanations or proposes
solutions based on the data.
Student is able to identify
biases or flaws in the data
and/or conclusion. Student
supports conclusions with
evidence in map form, table
form, graphic forms, and
timelines. Student connects
current knowledge to new
conclusions or solutions.
Student makes inferences
based on the information in
geographic form.
C:\Documents and Settings\Anne\My Documents\Irgee\14-3\irgee14-3.vp
16 March 2006 10:14:28
IRGEE 178d
Student asks geographic
questions, such as ‘where?’
and ‘why there?’ with some
adult prompting. Questions
are usually related to the
problem. The student is
sometimes able to revise
questions as new data comes
in. Questions are appropriate
to the tools, time, and task.
Student determines data
needed to answer the
question with adult
assistance. Student selects or
creates primary, secondary,
and tertiary sources of data
appropriately. Student is able
to explain data collection
methods and the data chosen
to be collected. Student
usually observes and
systematically records
information. Student reads
and interprets maps, and
conducts interviews as
needed to collect data with
some adult help.
Student organises and
displays geographic data in
ways that help with the
analysis and interpretation of
it, with some adult help.
Organising the data may
include using and developing
maps, graphs, tables, and
timelines with available tools.
Student is able to see some of
the patterns, relationships and
connections among the
collected and organised data.
Student is usually able to
describe the data and ask
alternative questions about it.
Student notes associations and
similarities between areas,
recognise ‘patterns in the data,
and draw’ inferences. Student
may use simple statistics to
identify trends, sequences,
correlations, and relationships.
Student can usually formulate
explanations or propose
solutions based on the data.
Student is sometimes able to
identify biases or flaws in the
data and/or conclusion.
Student usually supports
conclusions with evidence in
map form, table form, graphic
forms, and timelines. Student
can sometimes connect current
knowledge to new conclusions
or solutions. Student can
sometimes make inferences
based on the information in
geographic form.
1
Student formulates questions
only with teacher help.
Questions are not revised to
connect with new
information. Questions are
either too broad, too narrow,
or irrelevant to the problem.
Student may be able to tell
what information was
gathered but does not
understand how it relates to
the research question. Student
does not understand how
data was gathered. Student’s
data was not systematically
collected nor recorded.
Student is unable to read or
interpret geographic
information.
Student is not able to organise
geographic information in a
way that contributes to
analysis and interpretation.
Student is not able to develop
maps, graphs, tables or
timelines with or without
tools.
Student is unable to see
patterns, relationships or
connections among collected
and well organised data.
Student is not able to describe
the data nor ask alternative
questions. Student does not
use simple statistics to
identify trends, sequences,
correlations, and
relationships.
Student is unable to formulate
explanations or solutions
using the data. Student is
unaware of biases or flaws in
the data or the conclusion.
Student is unable to support
conclusions with data from
maps, tables, graphs, or
statistics. Student is not able
to make connections between
the new information and
present information or
knowledge. Student cannot
make inferences based on the
information in geographic
form.
 Color profile: Disabled
Composite Default screen IRGEE 178d

Forum 219

Table 1 (contd) Quantitative rubric

3 2 1

Presentation Student uses maps, tables,
graphs, timelines and
statistics to support the
conclusion or solution.
Visuals are large enough to be
read easily, simple and easy
to understand, and pleasantly
designed. If GIS is used, the
presentation runs smoothly,
the maps are clear, and
queries are done quickly and
efficiently. Presentation
employs but does not depend
upon the technology to
support the conclusion. The
Downloaded by [Heriot-Watt University] at 08:20 06 October 2014
Student may use maps, tables,
graphs, timelines and
statistics to support the
conclusion or solution.
Usually, the visuals are large
enough to be read easily,
simple and easy to
understand, and pleasantly
designed. If GIS is used, the
presentation usually runs
smoothly, the maps are
mostly clear, and queries can
be done. Presentation may
depend upon the technology
to support the conclusion. The
Student’s presentation does
not include maps, tables,
graphs, timelines, or statistics
to support the conclusions or
solution. Visuals are absent or
are poorly constructed.
Student tends to depend on
technology, GIS, to make the
presentation. If it fails, the
student is unable to proceed.
The argument is not
supported by a presentation
of the research process.

student is able to continue student is able to continue

with the presentation even if
the technology fails. The
argument is supported by the
presentation of the question,
evidence, addressing of faulty
reasoning, poor data,
explanation of data collection
methods, and analysis and
interpretation of data.
with difficulty with the
presentation if the technology
fails. The argument includes
summaries of the question,
evidence, addressing of faulty
reasoning, poor data,
explanation of data collection
methods, and analysis and
interpretation of data.

the facilitator. They discussed the utility of GIS and stressed that as a classroom
tool, GIS must not make teaching more cumbersome. Teachers were prompted to
ask themselves: ‘Does GIS facilitate or improve something? Does it make
teaching more effective or efficient?’ Teachers observed that GIS makes lessons
more student-centred, that students using GIS are using higher-order thinking
skills, abstract thinking, inference, and prediction to a greater degree, and they
become more critical about what they are learning.
The evaluator then asked the teachers, ‘What does it look like when students do
these things?’ The teachers responded that the students are interpreting data,
maps, and graphs, observing patterns at multiple scales, questioning the data,
looking at issues holistically, and transferring their conceptual knowledge from
local to global scales. Next, the group examined whether the students could do
these things without GIS. The general response was that while students can do
many of these things with paper maps and tables, GIS enabled students to acquire
data and create and alter maps more quickly. For example, instead of spending
80% of the time on data acquisition and representation and 20% for analysis and
interpretation, the percentages were switched when using GIS. With GIS, students
can spend the bulk of the time on projects in analysing data, asking questions,
observing patterns, linkages, and trends, and making connections. As the discus-
sion continued, it became clear that the effects of GIS on students could be divided
into three general categories: quantitative, as measured by a pre-determined
rubric; qualitative, as observed by student behaviour; and student perceptions, as
observed in student reflections through journals and interest surveys.
To measure the quantitative effect of GIS on students, a rubric was created that
assesses the questions students pose, the information they gather, the way they
organise data, how they interpret or analyse the data, how they come to their
conclusions or solutions, and how they present their results (Table 1). To assess

C:\Documents and Settings\Anne\My Documents\Irgee\14-3\irgee14-3.vp

16 March 2006 10:14:28
 Color profile: Disabled
Composite Default screen IRGEE 178d

220 International Research in Geographical and Environmental Education

Table 2 Student engagement chart

66% 50% 33% 0 33% 50% 66%
Students are . . . Very Seldom Less than Normal More than Frequently Very
Seldom normal normal Frequently
Making comments or
asking questions about
the project whenever they
see you (hallways,
before/after school, etc)
Asking relevant questions
in class
Working on their own
Helping each other
Asking relevant questions
Downloaded by [Heriot-Watt University] at 08:20 06 October 2014

of each other
Integrating new material
with previous knowledge
Monitoring their own
work
Solving task-related
problems
Discussing project/
problem issues with you
(teacher)
Discussing project/
problem issues with peers
Coming to class
Coming to class ready to
work
Staying on task during the
class time on their own
Taking risks
Initiating action when the
opportunity is available
Suggesting new ideas and
methods to answer
questions/issues
Coming late to class
Staying after class to work
on projects
Following directions
Passive
Giving up easily
Not trying hard
Bored
Interested

Compared to your regular class times with this group (‘Normal’ column) rate the class-
room atmosphere on the listed student engagement indicators on the continuum from
Very Seldom to Very Frequently during the time you are using GIS in the classroom.

the impact of GIS in the classroom, a quasi-experimental design is needed where

students using GIS are compared to students using traditional methods and

C:\Documents and Settings\Anne\My Documents\Irgee\14-3\irgee14-3.vp

16 March 2006 10:14:28
 Color profile: Disabled
Composite Default screen IRGEE 178d

Forum 221

tools. Kerski (2003) used this model to examine students who used GIS to study
the regional geography of Africa, patterns of global oil production, consumption,
and reserves, and demographic change in local neighbourhoods. Half of the
students used GIS, and the other half investigated these same topics using paper
maps, atlases, tabular data, and textbooks.
To assist teachers in gathering qualitative data about student engagement
when using GIS in a classroom setting, a second tool was developed. The indica-
tors are derived from ‘Alternative Approaches to Assessing Student Engagement
Rate,’ (Chapman, 2003), which provides working definitions of student engage-
ment. The findings can be analysed by creating an average of all the charts
received and showing the averages on a blank chart. Percentage suggestions
have been added to each column to help guide the teachers and the researchers as
to what the words mean. Teachers could also fill out a chart for a class before using
Downloaded by [Heriot-Watt University] at 08:20 06 October 2014

GIS and then complete an additional chart for the same class while using GIS. This
method would provide pre-scores and post-scores for each area of student
engagement. The gain scores could then be analysed using t-tests.
Porter (2002) justifies the use of teacher self-report surveys because teachers
are willing to complete them. He has found that agreement between the informa-
tion in a survey and in a teacher’s log has correlations of 0.7 to 0.8, which is
considered high (Porter & Smithson, 2001). The use of surveys also is cost effec-
tive. Some limitations of using surveys include the fact that the surveys are
limited to what the researcher asks, they are subject to a self-report bias, and the
surveys do not capture all of the instructional complexity that exists in a
classroom.
To evaluate the effects of GIS on students’ perception and attitudes, students’
self-reporting techniques were discussed. This can be done in journal format
where students reflect upon various aspects of their experience using GIS. What
have they learned? What have they enjoyed? What caused them frustration or
difficulty? Do students enjoy using these tools? Teachers can periodically gather
these journals and summarise student responses. Alternatively, interest surveys
could be developed at the classroom level that would ask students these or
similar questions.
This project enabled the creation of an initial array of tools that teachers can use
to evaluate the utility of GIS. The fundamental purpose for evaluating new tech-
nologies such as GIS as teaching tools is for teachers to determine for themselves
whether or not using GIS is worthwhile. Future research projects will seek to
gather and compile data from teachers in order to find more conclusive evidence
regarding the effectiveness of GIS as a classroom teaching tool.

Correspondence
Any correspondence should be directed to Sophia Linn, Fort Collins, Colo-
rado, USA (sophia@frii.com).

References
Barstow, D., Gerrard, M.D., Kapisovsky, P.M., Tinker, R.F. and Wojtkiewicz, V. (1994)
Report from the First National Conference on the Educational Applications of Geographic Infor-
mation Systems. 27–29 January 1994. Cambridge, Massachusetts: TERC Communications.

C:\Documents and Settings\Anne\My Documents\Irgee\14-3\irgee14-3.vp

16 March 2006 10:14:28
 Color profile: Disabled
Composite Default screen IRGEE 178d

222 International Research in Geographical and Environmental Education

Chapman, E. (2003) Alternative approaches to assessing student engagement rates. Prac-

tical Assessment, Research and Evaluation: On WWW at http://edresearch.org/pare/.
Kerski, J. (2003) The implementation and effectiveness of geographic information systems
technology and methods in secondary education. Journal of Geography 102 (3), 128–37.
Jacksonville, AL: National Council for Geographic Education. May-June.
Porter, A.C. (2002) Measuring the content of instruction: Uses in research and practice.
Educational Researcher 31 (7), 3–14.
Porter, A.C. and Smithson, J.L. (2001) Are content standards being implemented in the
classroom? A methodology and some tentative answers. In S.H. Fuhrman (ed.) From the
Capital to the Classroom: Standards-based Reform in the States, 100th Yearbook of the National
Society for the Study of Education, Part II (pp. 60–80). Chicago, IL: National Society for the
Study of Education.
Downloaded by [Heriot-Watt University] at 08:20 06 October 2014

C:\Documents and Settings\Anne\My Documents\Irgee\14-3\irgee14-3.vp

16 March 2006 10:14:28