Programmatic Assessment: Turning Process into
Practice by Teaching for Learning
by Dorothy Anne Warner
In order to improve our library
instruction program by
identifying and responding to
learning problems, a formal
pilot assessment was
conducted with a group of 48
students enrolled in a summer
2001 pre-freshmen program.
The group chosen as a base
line group provided us with
the opportunity to work with
students in four sequential
library instruction sessions,
allowing us to observe, reflect
and respond to learning needs
between each session. In the
process of designing and
executing the assessment
system we changed and
improved our teaching
methods to address the
learning problems we
discovered. Our intent was to
improve the library instruction
program in our medium-sized
university library by applying
what we learned from the pilot
to the more typical “one shot”
library instruction sessions.
Dorothy Anne Warner is Associate
Professor-Librarian at Rider University
Libraries, Moore Library, 2083
Lawrenceville Road, Lawrenceville, New
Jersey, 08648-3099 ⬍warner@rider.edu⬎.
The Journal of Academic Librarianship, Volume 29, Number 3, pages 169 –176
W e have recognized for some
time the need to make im-
provements to our library in-
struction program and realized that as a
first step we needed to identify the learn-
ing problems. In the past we had made
casual and subjective assumptions about
student learning. Librarians had depended
upon feedback from the faculty members
and had made informal observations
about student progress in their work with
students in the hands-on portion of in-
struction sessions and on the reference
desk. Although the library instruction pro-
gram is quantitatively assessed each year,
this type of assessment does not include
results of student learning. The need to
develop a formal, objective process to as-
sess learning was clear. Also clear was the
recognition that improvements needed in
the teaching process would become more
obvious. While designing the assessment,
we took advantage of the opportunity to
reflect on what we were teaching and how
we were teaching it and began to make
improvements to our teaching. We were
conscientious about planning the instruc-
tion tied to formal learning goals and ob-
jectives. Based on a literature review, we
realized that our assessment process
would have to be custom-designed, in-
cluding the development of multiple
methods or tools. Our process of assess-
ing actual student learning became formal
with a pilot project developed in summer
2001.
Assessment takes time, but the time
expenditure is very worthwhile. Assump-
tions about learning are either confirmed
or dispelled. The opportunity to observe
both the students’ learning and one’s own
teaching is now formal, rather than infor-
mal. Because the process is formal, the
reality becomes clearer.
LITERATURE REVIEW: RECOGNIZING
PROBLEMS WITH ASSESSMENT
The decision to custom-design our assess-
ment was based on the result of the liter-
ature review to determine if there were
any models for qualitative programmatic
assessment of library instruction. While
some worthy case studies had been con-
ducted, no one had solved the problem of
moving beyond primarily subjective as-
sessments to a sustained objective assess-
ment of the library instruction program
with an interest in programmatic im-
provement. Absent, too, was an objective
focus on the effect of teaching on student
learning of information literacy skills.
From a 20-year literature review of
assessment methods used in libraries, we
found serious limitations to the instru-
ments most typically used, primarily that
they failed to adequately assess student
learning.1 Rather, more attention was
given to impressions of learning, presen-
tation style, and affect, including student
attitude. Ragains corroborated this au-
thor’s concern, emphasizing that “subjec-
tive data alone are inadequate to measure
student learning, guide programmatic im-
provements in library instruction, or be
used as a basis for librarians’ performance
appraisals”.2 Very few tools measured ac-
tual knowledge or application of that
knowledge following library instruction
and none measured learning over a sub-
stantial time period. While recognizing
that a positive student attitude can affect a
student’s learning performance,3 our con-
cern is that an emphasis has been on as-
sessing student attitude rather than assess-
ing student learning. Another review of
the evaluation literature supported our
concern that “the quality of evaluations
conducted will have to improve”.4 We
were more concerned about assessing ac-
tual learning rather than assessing the ex-
May 2003 169
perience of learning and the purpose was
for summative evaluation, or “an ap-
praisal of the outcome of a program in
order to determine its fate”.5
Although there can be many different
purposes for evaluating library instruction
programs, the primary focus for our as-
sessment was, “Where can we make
changes or improve?”6 Also of interest
was addressing the following guideline
given to librarian evaluators by the Mid-
dle States Association of Colleges and
Schools: “Is the staff providing the in-
structional, reference, and bibliographical
services that students, faculty, and admin-
istrative staffs need in order to identify
their informational needs and to access,
evaluate, and effectively utilize the infor-
mation they obtain through the library’s
resources?”7
DISCUSSION
Of concern were the absence of a model
for programmatic assessment and the ab-
sence of examples that used assessment in
order to improve teaching. Noting that
“the purpose of assessment is supposedly
self knowledge which leads to improve-
ment,”8 a distinction of our process was
its focus on using what we learned from
the assessment to build in improvements
or changes to our teaching, thus we made
a conscious decision against a pre/post-
test approach.
We understood the need for an audi-
ence that would be representative of those
we serve. The audience for the pilot as-
sessment was a group of 48 students who
took part in a summer 2001 pre-freshmen
year preparation program. This summer
program, which our library instruction
program has participated in for over ten
years, has three sequential library instruc-
tion sessions over the summer and one
library instruction session in the fall. All
of the librarians are involved with helping
the students, either by teaching during the
instruction component or teaching on the
reference desk, so the opportunity existed
for all librarians to immediately benefit
from the assessment process. We sought
to apply what we learned from the pilot to
the more typical “one shot” library in-
struction sessions, and felt that this group
of students would serve well as a base line
group. The needs of this group are similar
to those of all freshmen we serve. If any-
thing, this particular group of students
tends to be very enthusiastic and eager to
learn all that they can in a very short
window of time, and that is not necessar-
ily typical of most freshmen students. The
170 The Journal of Academic Librarianship
opportunity to have four instruction ses-
sions with one group allowed for more
substantial feedback and the opportunity
to have more of a certainty about the
strengths and weaknesses of our instruc-
tion program.
Each of our library instruction sessions
is unique and designed to be integrated
with specific assignments. With that in
mind we needed to thoroughly examine
the assignments and the library skills
needed for these students to fulfill the
assignment requirements. The library in-
struction component of each session is
developed in collaboration with two pro-
fessors of the Communication Depart-
ment, coordinating the research skills to
be learned with their specific assign-
ments, and including improvements based
on observations from the previous year.
Ianuzzi stresses the shared responsibility
between libraries and teaching faculty
with the teaching and measuring of infor-
mation literacy skills.9
“. . .each instruction session
introduces new library skills
and new resources appropriate
to the assignments.”
There are three assignments over the
four instruction sessions, and each in-
struction session introduces new library
skills and new resources appropriate to
the assignments. The first assignment (in
three parts) is to develop a speech about a
prominent person, who may be living or
dead, using information found in period-
ical and newspaper articles and in bio-
graphical sources in the library’s refer-
ence collection. The following resources
are introduced: Readers’ Guide to Peri-
odical Literature to access information
about those who lived prior to the past ten
years, electronic periodical and newspa-
per indexes to access information about
those currently living, and several bio-
graphical print reference sources. Stu-
dents then need to know how to access the
actual articles (in print, microfilm, or
electronic format).
The second part of the assignment, in-
tended to emphasize the time limitations
of electronic periodical indexes, asks all
students, regardless of the time in which
their topic person lived, to use Reader’s
Guide to Periodical Literature to look for
a periodical article written approximately
fifty years ago. To acquaint students with
the location and use of periodicals and
newspapers available only on microfilm,
all students are given a third part of the
assignment (to be completed with part
two during evening study hours) to locate
a specific topic requiring the use of news-
papers available on microfilm.
The second assignment, covered over
two instruction sessions, is to develop a
speech about a discrimination topic, pre-
paring at least three types of attention
steps in the introduction (e.g., illustra-
tions, statistics, quotes, examples, rhetor-
ical questions, or shocking statements)
and later forming the body of the speech.
At this stage, subject indexes are intro-
duced (e.g., Social Sciences Index, ERIC)
and there is an emphasis on the recogni-
tion and use of controlled vocabularies.
The teaching of free web sources con-
cludes the summer instruction sessions
with an emphasis on the evaluation of
World Wide Web sites. At this stage, stu-
dents are directed to specific web sites
that provide information on discrimina-
tion (e.g., the American Civil Liberties
Union web site) and search techniques for
locating this quality of web site are dem-
onstrated.
The third assignment, in the fall, is to
examine a culture and to develop a speech
that discusses at least three specific as-
pects of the culture. Resources taught in-
clude several print reference sources, ad-
ditional subject periodical indexes, and
selected web sites. The instruction session
for this assignment also provides an op-
portunity to review the learning that has
taken place over the summer.
The measures and assessment tools
were developed in coordination with the
assignments, and the library skill expec-
tations. The learning goals and objectives
for the summer program students, based
on the assignments, were determined in
compliance with the Association of Col-
lege and Research Libraries (ACRL) In-
formation Literacy Competency Stan-
dards for Higher Education.10 The
objectives were determined based on the
specific learning outcomes that we expect
from the summer program participants.
The goals and objectives were linked to
the Library Instruction Program Objective
and Philosophy, to the Library’s Mission
Statement, and to the University’s Mis-
sion Statement. The assessment process
followed the recommendations of the
Rider University Assessment Committee,
which the author is a member of. In ad-
dition, all levels of Bloom’s Taxonomy of
Cognitive Objectives were incorporated
into the assessment objectives. (See the
Appendix).
Assessment Tools
Multiple examples of evaluation in-
struments designed for specific purposes
are noted by Adams11 and by Freedman
and Bantly.12 The assessment tools used
for our program were adapted from such
examples to coordinate with the specific
speech assignments and to assess the
learning objectives determined above.
Three assessment tools were introduced: a
journal tool (integrated into the assign-
ments), a faculty reflection tool (used by
all of the librarians and by the professors),
and an assessment of the summer pro-
gram upperclassmen (all of whom partic-
ipated in similar assignments during their
freshman year). Both the journal tool and
the faculty reflection tool assessed all of
the standards. The upperclassmen assess-
ment tool assessed all but the fourth and
fifth standards.
By using several assessment tools, the
opportunity exists to compare the results
of each. The journal tool was designed to
elicit personalized answers and it pro-
vided a clear documentation of student
learning. It allowed us to easily deter-
mine, for example, the effectiveness of
the students’ search statements. The re-
flection tool was most effective in
prompting a dialogue between teaching
faculty and librarians that resulted in a
revision of the librarians’ teaching strate-
gies. It does not serve well as a documen-
tation of overall student learning, but does
provide observed examples of the learn-
ing experience taking place for individu-
als within the hands-on sessions. The up-
perclassmen assessment tool provided an
indication of learning as applied to other
courses. The observation of the students’
speeches in the fall allowed the librarians
to determine the incorporation of the in-
formation from the resources into their
speeches.
Journal Tool
The multi-page assignments include
the speech assignment and information
about research steps for locating the in-
formation to complete those assignments.
Information literacy terms and their defi-
nitions are introduced and search exam-
ples are included. In order to assess learn-
ing as it was taking place, questions were
integrated into the assignments so that the
students could maintain a journal of their
research steps. For example, in the section
of the assignment that introduces search-
ing in an electronic periodical index, the
following was asked of the students:
“Write the online search statements that
you tried on the lines below. Put a star
next to the most useful search state-
ments.” After several indexes had been
introduced, the students were then asked,
“Write the index/database that you
searched and the search statements that
you tried on the lines below.” Also asked
was, “What controlled vocabulary terms
(descriptors, subjects) did you notice that
helped you develop your search state-
ments?” Students must keep track of all
required citation elements for use in their
bibliography. This includes writing down
the information needed for their web ci-
tations. Following that was the question,
“How did you know that the information
at the web site was reliable (e.g., What do
you know about the author or organiza-
tion; is the information current)?” Other
requests included asking why one would
choose Readers’ Guide rather than an
electronic index, and asking students to
note the availability of a periodical at the
library in microfilm, print or electronic
form. The first set of journals is collected
in time for the librarians to review the
search statements and recognize problems
that need to be addressed. At the end of
the summer, the second set of students’
journals is collected for the librarians to
review and respond to in the fall session.
“The first set of journals is
collected in time for the
librarians to review the search
statements and recognize
problems. . .”
Reflection Tool
Since all of the librarians teach the
students both in formal instruction ses-
sions and on the reference desk, all the
librarians were included in the assessment
with the faculty reflection tool (a directed
observation method). A discussion of li-
brarians’ and professors’ observations
took place at the end of each session and
a final discussion of the professors’ ob-
servations took place at the conclusion of
the fall semester. This formalized a pro-
cess that had been performed informally
in the past and forced those teaching to
immediately recognize areas that needed
improvement and to act immediately (for
the next instruction session) on those
needs. The tool asked of the librarians and
professors, “What successes and prob-
lems did you notice the students having in
any of the following areas?” Several spe-
cific areas were then given with specific
observation requests. For example, “Us-
ing the electronic databases: selecting the
appropriate databases, composing search
statements, using controlled vocabularies
(subjects/descriptors), selecting appropri-
ate periodical articles, selecting appropri-
ate web sites, understanding the differ-
ence between the information found via a
commercially available electronic data-
base versus a free web site.” The profes-
sors were specifically asked to observe
the incorporation of the information
within the speeches. Directed observa-
tions included: “By the end of the sum-
mer, was it obvious to you while the stu-
dents were giving their speeches that they
were drawing information from multiple
sources?” and “Were the statistics used in
their speeches up-to-date?” and “Had the
information at the web sites been care-
fully selected (evaluated) by the students
(e.g., was the author or organization rep-
utable)?” and “Did the students include
all of the appropriate citation information
in their bibliographies (volume numbers,
page numbers, etc.)?”
Upperclassmen Assessment Tool
This tool gave an idea of the types of
information resources that are utilized for
other courses taken during subsequent
years at the university. It is the hope, of
course, that the library instruction ses-
sions will have an application to future
courses for the students. We asked,
“Which library skills learned (during li-
brary instruction sessions) were most
helpful in your research (e.g., search strat-
egies for using indexes/databases, the
web, or the online catalog; finding peri-
odicals in the library, etc.)?” In the effort
to assess individual performance and
learning, we then asked, “What is a recent
topic for your major that you have re-
searched?” Specifically tied to this ques-
tion were the following questions:
“Which indexes/databases/online catalog/
Websites did you select?” and “What was
your search statement (please include a
detailed search statement used in an elec-
tronic database utilizing Boolean opera-
tors (AND, OR, NOT) and truncation)?”
and “What descriptors/subject terms did
you discover that helped you to focus
your search?” and “How did you know
May 2003 171
that the information that you found was
reliable?”
Observation of Student Speeches
The librarians are invited to the fall
speeches and this is our opportunity to
assess the incorporation of information
from the students’ research into the com-
pleted product, the speech.
ANALYSIS AND RESULTS
The assessment had much to teach us and
we responded immediately by making
changes to the teaching process. Gener-
ally, we realized that the students were
not fully making connections within the
research process and were not making the
connection between the assignment and
the skills needed for completing the as-
signment. To address this, the teaching
process has been changed.
“. . .we realized that the
students. . . were not making
the connection between the
assignment and the skills
needed for completing the
assignment.”
Formerly, students rotated from one
hands-on learning station to the next, each
station taught by a different librarian (for
example, one learning station taught the
use of Reader’s Guide to Periodical Lit-
erature and another learning station
taught locating of periodicals), but the
assessment results made us realize that
this was actually confusing for the stu-
dents. The revised teaching process is less
chaotic, with one introduction by one li-
brarian to all of the students, followed by
a more focused hands-on learning session
in which all librarians participate, and
concluded with a summary/reflection pe-
riod with the students, teaching faculty
and all of the librarians. The introduction
during the first instruction session now
stresses the connection between the as-
signment and the skills being taught to
complete the assignment. The entire pro-
cess, from determining the most appropri-
ate index to use, to locating citations, to
locating the actual article, and to properly
citing the article is now carefully outlined.
During the summary/reflection at the end
of the session, concepts are reviewed with
the aid of a checklist of research skills
taught. This checklist, handed out and
172 The Journal of Academic Librarianship
collected at the end of each instruction
session, serves to reinforce the learning
that has taken place while providing an
opportunity for students to indicate on the
checklist those concepts that require fur-
ther reinforcement.
At the conclusion of the fourth instruc-
tion session, the results of each assess-
ment tool were summarized in a report,
with the results divided into “teaching”
and “learning” sections, and teaching rec-
ommendations given within each section.
As illustrations, both positive and nega-
tive student examples were included in
the report. An indication was made of the
objectives met after each instruction ses-
sion. The following is condensed from
those summaries.
Assessing Standard One
The information literate student deter-
mines the nature and extent of the infor-
mation needed.
Objective A
Students will identify a variety of types
and formats of potential sources of infor-
mation.
This may appear to be an obvious,
easily achieved objective. However, the
job of the instruction librarians has be-
come more difficult as students have de-
veloped a false confidence in their re-
searching abilities from using the free
web primarily for personal information
needs. In spite of the first instruction ses-
sion, many students went directly to the
free web to conduct their research. This
occurs throughout the year in our instruc-
tion program. Now, the librarians and
professors emphasize that the free web
will be utilized in subsequent sessions,
and students are to wait until those ses-
sions before using it for their research.
And a stronger attempt is made in all
instruction sessions now to make a dis-
tinction between commercial databases
providing academic resources and the in-
formation available for free on the web.
Possibly one of the most “telling” ses-
sions was the fourth (early fall) instruc-
tion session that began with an oral re-
view of the resources taught during the
summer and observations made from the
students’ journals. Students were asked
why they would select Readers’ Guide
rather than an electronic index. There was
a review of the terms that were taught in
the summer, stressing Boolean logic,
truncation and controlled vocabulary. Stu-
dents were asked how they would evalu-
ate a web site. During the oral review
session only a few students appeared to be
aware of the answers. Although nearly
everyone had given a clear reason for
selecting Readers’ Guide on the summer
journal tool, no one would offer a reason
in the fall. No one demonstrated knowl-
edge of evaluation criteria for a web site.
One advantage of assessing this particular
group is the opportunity to develop a
closer librarian/student relationship due to
the frequency of instruction sessions.
Thus, a reticence that may exist with a
“one-time” session is not an issue with
this group. There was not a reluctance to
participate, but, rather, the skills taught in
the summer sessions needed to again be
reinforced in the fall. From this experi-
ence, the recommendation has been given
to all librarians teaching in the instruction
program to incorporate multiple repetition
methods into their library instruction ses-
sions.
“. . .skills taught in the summer
sessions needed to again be
reinforced in the fall.”
In preparation for the summer 2001
students, there was a teaching decision
ahead of time to concentrate on repetition.
We did this by providing information in
several modes in order to appeal to more
than one learning style. We provided
printed handouts, some of which were
referred to in all four sessions. We said it
and we said it again and we said it again.
We showed “live” examples using a
large-screen projection. We involved the
students in the teaching process by asking
them to be a part of the demonstration and
we asked questions of them. In the future,
more questions need to be asked to both
encourage the thought processes of the
students toward higher levels of Bloom’s
taxonomy and to assess the knowledge
level of the students. Too many assump-
tions are made of student learning rather
than realizing early enough that the stu-
dents are not grasping the information.
We have become more aware of the
teaching methods used in the process of
providing our reference service. The as-
sessment process has influenced us to
slow down dramatically both in instruc-
tion sessions and while assisting students
on the reference desk because of the
awareness of the students’ lack of confi-
dence in doing research, even though the
student attitude may be projecting a com-
pletely different impression. Teaching
takes time and it is much easier to simply
provide the information for the student
than to take the time to teach the student
how to find the information and to assist
with their interpretation of what they have
found.
During the observation of the speeches
in the fall, the students appeared to have
selected from among several sources for
their research, although very few specific
references were given. Students referred
several times to individuals as references
(“according to [a person]”) but the source
for that reference was not given. Judging
by the information given and a great deal
of current detail in the speeches, it ap-
peared that the students had used care-
fully selected sources. In a discussion
with the students at the end of the class
periods, several print reference sources,
including Culturgrams and the Encyclo-
pedia of World Cultures, were preferred
resources for the culture assignment.
The upperclassmen survey demon-
strated that this objective was met mini-
mally as the variety of sources listed was
small, including an encyclopedia, atlas,
several Gale author and criticism series,
and a general electronic periodical index.
Assessing Standard Two
The information literate student ac-
cesses needed information effectively and
efficiently.
Objective A
Students will recognize controlled vo-
cabularies.
Some students demonstrated a begin-
ning understanding of controlled vocabu-
laries by using a “subject search” ap-
proach in an electronic periodical index.
But even after the fourth session, the skill
was not observed to be occurring at even
the Knowledge level of Bloom’s taxon-
omy and there was no indication from the
upperclassmen survey that this objective
was met. All of the librarians teaching in
the library instruction program have been
asked to emphasize the effectiveness of
using controlled vocabularies in a search.
Assessing Standard Two
The information literate student ac-
cesses needed information effectively and
efficiently.
Objective B
Students will illustrate search strate-
gies that incorporate appropriate key-
words and synonyms, controlled vocabu-
laries (when appropriate), Boolean
operators, nesting of terms, and trunca-
tion, refining the search statement when
necessary.
These are skills that need to be rein-
forced in each instruction session. After
the first session, only a few students wrote
well-constructed search statements utiliz-
ing Boolean logic appropriately. Al-
though there were some very good search
statements given in the student journals
(e.g., “[person] and Black nationalism,”
and “[person] and objectivism”), some of
the search statements alerted us to the
need for more focus, for example: “[per-
son] and biography,” and “[person] and
life.” Only one student attempted trunca-
tion. A few students attempted nesting of
terms, but for the most part the entire
search statement was constructed improp-
erly.
“. . .there is a need to focus
the session on yet a smaller
amount of information.”
There was a realization during the fac-
ulty reflection that, as has been observed
in the past, there is a need to focus the
session on yet a smaller amount of infor-
mation. Because of the stated learning
objectives the teaching has become more
focused. It has been recommended that
for all library instruction sessions, the li-
brary faculty (together with the teaching
faculty) determine 3-5 objectives for each
session in order to more clearly focus the
sessions. What do we really want the stu-
dents to know once they have left the
instruction session? What do we really
want the students to know by the end of
the research process over the semester?
It was also noted that less time should
be devoted to formal instruction by the
librarians and more hands-on instruction
time as the students must have more time
for “learning by doing,” recognizing that
“learning and practicing what is taught is
essential to mastering materials and infor-
mation resources”.13 During this instruc-
tional time, many useful learning situa-
tions occur. For example, a student used
the search statement “AIDS discrimina-
tion work” in an electronic index and the
librarian suggested that the student submit
the search request. When no results ap-
peared, the librarian asked how the search
statement could be fixed (e.g., reinforcing
the need to use Boolean operators). A
similar example (“discrimination against
women in the workplace”) was then used
during the following instruction session
as a way to reinforce the need for Boolean
operators in a search. The librarian em-
phasized the searching tips listed on one
of the handouts that give examples of
using truncation and Boolean logic. The
students were asked as a group to recom-
mend ways to improve the search state-
ment. The students were then referred to
the assignment and asked to note the re-
quest for controlled vocabulary terms and
asked how these terms might be useful
with the creation of a search statement.
This type of group learning experience
makes more sense after the students have
had a hands-on opportunity. Once there
has been a trial-and-error approach during
the hands-on sessions, the students are
more alert to discovering ways to improve
their search statements.
In spite of constant repetition of this
objective in each instruction session, stu-
dents continued to have difficulty meeting
this objective. The recommendation for
summer 2002 was to emphasize this ob-
jective more thoroughly during the initial
instruction session, using both “good” and
“poor” examples and inviting student par-
ticipation in developing search state-
ments.
There was no indication from the up-
perclassmen survey that this objective
was met.
Assessing Standard Two
The information literate student ac-
cesses needed information effectively and
efficiently.
Objective C
Students will determine the most ap-
propriate resources for accessing needed
information.
After the first session, the student jour-
nal answers did not indicate that students
were yet in a position to make distinctions
between types of sources. Following the
second session, students used additional
resources, but at the Knowledge level of
Bloom’s taxonomy only. It was hoped
that the students would learn to make
distinctions about sources and the reasons
and appropriateness of selecting one
source over another and after the fourth
session, the students appeared to under-
stand that some resources would answer
questions better than others.
This objective was met minimally,
judging from the upperclassmen survey.
May 2003 173
The following was of concern: both a
Biology major and an American Studies
major only mentioned the use of a general
electronic periodical index and none of
the science or social sciences periodical
indexes.
Assessing Standard Three
The information literate student evalu-
ates information and its sources critically
and incorporates selected information into
his or her knowledge base and value sys-
tem.
Objective A
Students will judge the value of a re-
source by noting its reliability, validity,
accuracy, authority, timeliness, point of
view or bias.
The journal responses included the fol-
lowing in response to a request to evalu-
ate the quality of a web site: “The author
was a ‘well-known’ author and has writ-
ten many other pieces;” “Current infor-
mation; many notable quotes were found
inside the piece;” “Current information/
well-known school/looks pretty reliable
[the Website was a link from an Ohio
State URL].”
It was observed after the session in
which web sources were introduced that
the students were having difficulty with
this objective even at the Knowledge
level of Bloom’s taxonomy. We noted,
while following up with the students in
the study/library evening hours that the
students’ selections of web sources were
arbitrary. The opportunity was used by
the library staff to emphasize the quality
and appropriateness of the information by
following the links to determine the cred-
ibility of the author/organization taking
responsibility for the information. To help
the students understand the quality of web
information expected by the professor,
both a “good” example of a web site is
now demonstrated and contrasted with a
“poor” example. This practice has now
been incorporated into most library in-
struction sessions.
The above experience also reinforced
the need for more active reference out-
reach to the students during their evening
study hours. In our instruction program,
we have tended to concentrate most on
the actual half-hour/hour instruction ses-
sion rather than the entire research pro-
cess. In order to reinforce learning and
incorporate more repetition into our
teaching, the follow-up that takes place by
the reference staff is a crucial component
of the instruction program and thus there
174 The Journal of Academic Librarianship
can never be too much communication
between the instruction and reference li-
brarians. We have recommended for both
the summer program and for all library
instruction sessions that each librarian
communicate even more frequently with
the rest of the librarians about his/her
instruction session with recommendations
for collaborative follow-through by the
entire reference staff. This is done via
email and with an assignment notebook at
the reference desk that contains all of the
current assignments and handouts. When
necessary, librarians provide instruction
sessions for each other about the use of
specialized databases. We each need to
continue to take the responsibility for ap-
propriate follow-up to ensure that the ref-
erence support is adequate, and, in partic-
ular, we need to alert each other to
specific reference needs and to areas
where students may hit snags. In our ex-
perience, most questions arise once the
students have embarked on their search
rather than immediately following an in-
struction session and that is the time when
the librarians must be present to assist
with those questions. Kuhlthau defines as-
pects of this stage as the “uncertainty
principle,” or “uncertainty due to a lack of
understanding, a gap in meaning, or a
limited construct [initiating] the process
of information seeking”.14
“. . . most questions arise once
the students have embarked on
their search rather than
immediately following an
instruction session. . .”
During the observed speeches, this ob-
jective appeared to have been met. Infor-
mation given was current and if web sites
had been selected, they appeared to have
been selected carefully judging from the
information that was given in the
speeches. This was one area where
progress had been made by the fourth
instruction session, judging by the quality
of the information given in the speeches.
According to the upperclassmen sur-
vey, this objective was not met by most,
although there were some good examples
of web sites given, such as a university
site (although it is important to continue
to stress checking the authority at the uni-
versity site), “from groups or sources with
high accreditation,” and a “reliable” news
station (CNN.com).
Assessing Standard Four
The information literate student, indi-
vidually or as a member of a group, uses
information effectively to accomplish a
specific purpose.
Objective A
Students will assemble the information
gathered and create a product.
As noted previously, the observed stu-
dent speeches were well-informed and ev-
idence of substantial research.
Assessing Standard Five
The information literate student under-
stands many of the economic, legal, and
social issues surrounding the use of infor-
mation and accesses and uses information
ethically and legally.
Objective A
Students will appropriately cite their
sources.
Several students left the section blank
that requested citation elements or were
missing elements such as the author, date,
or name of the periodical. It was also
noted during the observation of the
speeches that progress needs to be made
with this objective. The professors re-
marked that the students were not provid-
ing complete bibliographic citations on
their reference lists. At the request of the
professors, increased attention is now
given to developing a proper biblio-
graphic citation utilizing the citation in-
formation requested in the assignment.
Examples of how to cite the information
are given (again, both a “good” example
and a “poor” example, e.g., an isolated
“url”) and the students are asked as a
group to develop a proper bibliographic
citation. In this way a connection can be
made to reinforce the information being
collected on their assignment sheets with
the bibliographic citations required by the
professor.
CONCLUSION
The pilot assessment study focused on
learning outcomes, not on the subjective
impressions of learning nor on presenta-
tion style. The summary of the actual
results of student learning from the pilot
study influenced necessary changes for
both the summer program and for the
entire library instruction program. The
initial results of the summer 2002 assess-
ment indicate an improvement in learning
outcomes. Further recommendations for
improvement to the summer 2003 pro-
gram have been made based on the early
assessment results, thus building the as-
sessment process into the teaching and
learning process.
“. . .the library instruction
component of the required
English composition course, is
now being assessed in
collaboration with the English
faculty.”
Assessment in our library has not
stopped with the pilot project. Building
the assessment process in to the entire
library instruction program is developing
naturally. The library instruction portion
of the library’s policy manual has been
revised to include objectives for the entire
library instruction program adapted from
the ACRL Information Literacy Compe-
tency Standards for Higher Education.
The “bread and butter” of the library in-
struction program, the library instruction
component of the required English com-
position course, is now being assessed in
collaboration with the English faculty. In
our university, each of these courses is
taught in a distinctive way according to
the preferences of the faculty member, so
our challenge is to be as consistent as
possible about our research skill objec-
tives in order to assure particular learning
outcomes on the part of all the students
taking the composition course. Recom-
mendations are given to each librarian
teaching a library instruction component
of a composition course for specific re-
search skills to be interwoven with the
requirements of the professor’s specific
assignment. The assessment tool, there-
fore, must be able to assess each of these
situations. The journal method is being
utilized with questions that ask each stu-
dent to document the research process
while the research is taking place. Stu-
dents are asked to list databases and
search statements used for those data-
bases. They are asked to list controlled
vocabulary terms identified in the process
of performing their research. Librarians
reviewing the search statements will note
the inclusion of controlled vocabulary
terms, the use of Boolean logic, trunca-
tion and nesting. Students are also asked
to evaluate the free web sites used. The
faculty has been asked to return the jour-
nals together with copies of at least three
randomly chosen (and anonymous) re-
search papers. The librarians will then
have the opportunity to note the incorpo-
ration of researched information into the
actual paper.
As we now begin to build assessment
in to the library instruction program as a
whole, we need to continue to be alert to
providing as much continuity as possible
throughout the program and multiple op-
portunities for continued growth for stu-
dent learning of information literacy
skills. The importance of working col-
laboratively with teaching faculty and
maintaining strong communication be-
tween the library instruction and refer-
ence librarians cannot be over-empha-
sized. All must combine their strengths to
reinforce the teaching of information lit-
eracy skills to students.
Acknowledgments: I am grateful for the
continued support and professional collab-
oration with Professor Harriett Schwartz
and Professor Howard Schwartz. I am also
grateful to those who supported the pilot
assessment of the summer program: Rubin
Joyner, Carinne Ramos, and Amber Craw-
ford. The participation and enthusiasm of
the Library Faculty at Rider University Li-
braries, Moore Library, in particular that of
Robert Lackie, the Co-Consultant of the
Library Instruction Program, make all of the
efforts to improve the Library Instruction
Program possible.
APPENDIX
Assessment Objectives Based on
ACRL Standards: (Including the
Primary Elements of Locating,
Evaluating and Using information)
Standard One
The information literate student deter-
mines the nature and extent of the infor-
mation needed.
Objective A
Students will identify a variety of types
and formats of potential sources of infor-
mation.
(Knowledge level of Bloom’s Taxon-
omy of Cognitive Objectives)
Standard Two
The information literate student accesses
needed information effectively and effi-
ciently.
Objective A
Students will recognize controlled vo-
cabularies.
(Understanding level of Bloom’s
Taxonomy of Cognitive Objectives)
Objective B
Students will illustrate search strate-
gies that incorporate appropriate key-
words and synonyms, controlled vocabu-
laries (when appropriate), Boolean
operators, nesting of terms, and trunca-
tion, refining the search statement when
necessary.
(Application level of Bloom’s Taxon-
omy of Cognitive Objectives)
Objective C
Students will determine the most ap-
propriate resources for accessing needed
information.
(Analysis level of Bloom’s Taxonomy
of Cognitive Objectives)
Standard Three
The information literate student evaluates
information and its sources critically and
incorporates selected information into his
or her knowledge base and value system.
Objective A
Students will judge the value of a re-
source by noting its reliability, validity,
accuracy, authority, timeliness, point of
view or bias.
(Evaluation level of Bloom’s Taxon-
omy of Cognitive Objectives)
Standard Four
The information literate student, individ-
ually or as a member of a group, uses
information effectively to accomplish a
specific purpose.
Objective A
Students will assemble the information
gathered and create a product.
(Synthesis level of Bloom’s Taxonomy
of Cognitive Objectives)
Standard Five
The information literate student under-
stands many of the economic, legal, and
social issues surrounding the use of infor-
mation and accesses and uses information
ethically and legally.
Objective A
Students will appropriately cite their
sources.
(Application level of Bloom’s Taxon-
omy of Cognitive Objectives)
NOTES AND REFERENCES
1. John Buschman & Dorothy Warner, “A
Slip Between the Cup and the Lip: Prac-
tical and Intellectual Problems of Market-
May 2003 175
 ing U.S. Academic Libraries,” Education
and Research for Marketing and Quality
Management in Libraries, In: edited by
Rejean Savard (Munchen: K. G. Saur,
2002), 269 –278.
2. Patrick Ragains, “Evaluation of Academic
Librarians’ Instructional Performance:
Report of a National Survey,” Research
Strategies 153 (1997): 159.
3. Wen-Hua Ren, “Library Instruction and
College Student Self-Efficacy in Elec-
tronic Information Searching,” Journal of
Academic Librarianship 26 (September
2000): 323–328.
4. Christopher Bober, Sonia Poulin, & Lui-
gina Vileno, “Evaluating Library Instruc-
tion in Academic Libraries: A Critical Re-
view of the Literature, 1980 –1993,” The
Reference Librarian 65 (1995): 51–52.
5. ACRL: Association of College & Re-
search Libraries, “Evaluating Biblio-
graphic Instruction: A Handbook (Chica-
176 The Journal of Academic Librarianship
go, IL: American Library Association,
1983), p. 108.
6. Diana D. Shonrock, ed., Evaluating Li-
brary Instruction (Chicago, IL: American
Library Association, 1996 p. 1.
7. Commission on Higher Education: Mid-
dle States Association of Colleges and
Schools, “Guidelines for Librarian Eval-
uators (Philadelphia, PA: Commission on
Higher Education, 1997), p. 6.
8. Susan Griswold Blandy, “The Librarians’
Role in Academic Assessment and Ac-
creditation: A Case Study,” Reference Li-
brarian 38 (1992): 83.
9. Patricia Ianuzzi, “We are Teaching, But
are They Learning: Accountability, Pro-
ductivity, and Assessment,” Journal of
Academic Librarianship 25 (July 1999):
304.
10. ACRL: Association of College & Re-
search Libraries, “Information Literary
Competency Standards for Higher Educa-
tion” [Online]. Available: http://www.
ala.org/acrl/ilintro.html (accessed May 29,
2001).
11. Mignon S Adams, “Evaluation,” in
Sourcebook for Bibliographic Instruction,
edited by Katherine Branch (Chicago, IL:
Association of College and Research Li-
braries/American Library Association,
1993), p. 45–57.
12. Janet Freedman & Harold Bantly, “Tech-
niques of Program Evaluation,” in Teach-
ing Librarians to Teach: On-the-Job
Training for Bibliographic Instruction Li-
brarians, edited by Alice S Clark & Kay F
Jones (Metuchen, NJ: Scarecrow Press,
1986), p. 188 –204.
13. Ibid: Diana D. Shonrock, p. 62.
14. Carol Collier Kuhlthau,” Seeking Mean-
ing: A Process Approach to Library and
Information Services (Norwood, NJ:
Ablex, 1993), pp. 108 –127.