See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/344301019

Curriculum assessment practices that incorporate learning outcomes in higher

education – a systematic literature review

Article · September 2020

CITATIONS READS

2 578

6 authors, including:

Roland Ebel Alexandra Thornton

Montana State University Montana State University
54 PUBLICATIONS 177 CITATIONS 9 PUBLICATIONS 3 CITATIONS

SEE PROFILE SEE PROFILE

Charlie Watt Colin Dring

Montana State University Royal Roads University
2 PUBLICATIONS 6 CITATIONS 25 PUBLICATIONS 108 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Advancing an Inclusive Food Systems Curriculum based on a Signature Pedagogy View project

Extensive production of habanero pepper (Capsicum chinense Jacq.) View project

All content following this page was uploaded by Roland Ebel on 21 September 2020.

The user has requested enhancement of the downloaded file.

 Curriculum Assessment Practices That
Incorporate Learning Outcomes in Higher
Education: A Systematic Literature Review
R. Ebel1, S. Ahmed2 , A. Thornton3, and C. Watt4
Montana State University Bozeman
Bozeman, MT

C. Dring5
University of British Columbia
Vancouver, BC

A. Sames6
University of Minnesota Twin Cities
Minneapolis, MN

Abstract

As new higher education programs emerge to prepare
students for being part of a well-equipped workforce
to meet the challenges of our times, and established
programs revise their curricula, clear program learning
outcomes (PLOs) are required. The process of curriculum
(or program) assessment serves to test whether programs
are effective in facilitating their PLOs. The objective of our
systematic review was to identify best practices in curriculum
assessment. We examined peer-reviewed literature from
1995 to 2019. The Preferred Reporting Items for Systematic
Reviews and Meta-Analyses (PRISMA) protocol was
applied to collect evidence, reduce bias, and synthesize
evidence. We screened 535 abstracts and conducted full-
text screening of 43 publications of which 20 were selected
for final extraction. We found that the number of articles
providing empiric evidence about effective curriculum
assessment is low. We identified promising practices based
on the frequency of their mention and their assessment by
the authors. These practices include an assessment plan;
assessment implementation through a faculty committee;
using both, direct and indirect assessment indicators (for
example, student performance in specific assignments
and student exit or alumni interviews); and an ongoing
reevaluation of PLOs. Our findings sustain the development
of an adaptable curriculum assessment procedure for
higher education.
Introduction
Institutions of higher education are responding to the
multifaceted societal challenges of our times by training
students in innovative and complex skills towards preparing
a well-equipped workforce. For example, human activities
are having unprecedented impacts on the Earth and its
systems with implications for human and planetary health
(Eichinger, 2019; Steffen et al., 2015; Tilman and Clark,
2014). These challenges require the development and
implementation of social innovations to transform society
(Olsson et al., 2017). As a result, new program offerings
emerge in institutions of higher education while long-
standing programs revise their curriculum to prepare a
workforce to meet societal needs. Consequently, these
programs need to articulate a clearly defined set of program
learning outcomes (PLOs) for students to demonstrate with
regards to the acquisition of certain knowledge, skills, and
attitudes (Killen, 2000). A PLO describes what is essential
for all students to successfully demonstrate in a measurable
way at the end of their learning experience within a given

1
Department of Health & Human Development, Reid Hall 345, MSU Campus, Bozeman, MT, 59717. roland.ebel@gmx.com. 406-994-5640
2
Department of Health & Human Development, Reid Hall 345, MSU Campus, Bozeman, MT, 59717. selena.ahmed@montana.edu. 406-994-5640
3
Department of Health & Human Development, Reid Hall 345, MSU Campus, Bozeman, MT, 59717. alithrntn@gmail.com. 406-994-5640
4
Plant Sciences & Plant Pathology Department, Plant BioScience Building 232, MSU Campus, Bozeman, MT, 59717. charleswatt@montana.edu. 406-994-5147
5
Centre for Sustainable Food Systems, 2357 Main Mall, Vancouver, BC, V6T 1Z4. colind@mail.ubc.ca. 778-859-1148
6
Department of Fisheries, Wildlife and Conservation Biology, 135 Skok Hall, 2003 Upper Buford Circle, St. Paul, MN 55108. same0057@umn.edu. 612-624-3600
238 NACTA Journal • Nov 2019 - Oct 2020
program (Knight, 2001). There is further need for curriculum
assessment to ensure that new and existing programs are
effective in meeting specific PLOs for preparing students as
professionals to address the grand challenges of our times.
Outcome-based education (OBE) has gained
momentum over the past five decades to prepare a
professional workforce that meets societal needs.
Curriculum assessment (often synonyms with ‘program
assessment’) is a core procedure in OBE. The identification
of suitable PLOs to assess how a curriculum meets these
PLOs is one of the core purposes of OBE (Barman et al.,
2014).
The way we assess our curricula, and ultimately our
students, are recognized as game-changers to improve
student outcomes for future success (Bryan and Clegg,
2019). There is thus a need to identify best practices for
curriculum assessment suitable in different contexts, which
can be applied towards the development of an adaptable
curriculum assessment tool for higher education institutions.
Apart from increasing the alignment of curricula and their
PLOs, such a tool would allow for the communication of
common educational goals, experiences, and methods
across instructors, programs, and departments. To our
knowledge, there is a lack of a published review identifying
best practices for curriculum assessment as well as of a
comprehensive adaptable curriculum assessment tool or
protocol.
The objective of our paper is to address the best
practices and protocol gap and advance the field of OBE
through conducting a systematic literature review on
practices for curriculum assessment that incorporate
PLOs. Specifically, this systematic review examines peer-
reviewed literature from 1995 to 2019 that reports outcomes
of concrete curriculum assessment interventions in higher
education with an emphasis on assessment practices based
on PLOs. The Preferred Reporting Items for Systematic
Reviews and Meta-Analyses (PRISMA) protocol (Moher et
al., 2009) was applied to collect evidence on the closed-
frame study question: In OBE higher education curricula,
what are practices for curriculum assessment that
incorporate PLOs? The Population, Intervention/Exposure,
Comparator, Outcome (PICO) framework (Schardt et al.,
2007) elements were applied to structure this systematic
review which examines curriculum assessment as the
intervention/exposure with the assessment procedure as
the outcome. The PRISMA and PICO protocols were applied
for the review to reduce bias and synthesize all evidence in
a replicable way. Findings from the review will inform the
development of an adaptable curriculum assessment tool
to evaluate the effectiveness of higher education programs.
Background on Outcome-based education
(OBE)
Overview of OBE
A curriculum is an attempt to communicate the
principles and features of an educational proposal to
effectively translating it into practice (Stenhouse, 1985).
The goal of OBE is to align curriculum, including content
NACTA Journal • Nov 2019 - Oct 2020
and activities, to PLOs, which describe what is essential
for all students to successfully demonstrate at the end of
their learning experience (Spady, 1994). While curriculum
development was historically relatively subjective and
unstructured, during the second half of the 20th century,
systematic curriculum development emerged on the agenda
of academic discussion and educational administration
(Knight, 2001). For the last fifty years, OBE has been the
prevailing approach to systematic curriculum development
(Barman et al., 2014).
The idea of OBE has its origins in behaviorist learning
theories (Morcke et al., 2013). In the mid-20th century, Tyler
[(1949) in (Morcke et al., 2013)] argued that curriculum
design should be determined by explicit objectives. Tyler’s
student Bloom (Bloom, 1956) built on the approach of
designing curriculum based on explicit objectives through
the development of a taxonomy of educational goals which
he classified into knowledge, skills, and attitudes (Morcke
et al., 2013). While Bloom´s taxonomy was reviewed
and updated to fit contemporary assessment standards
(Krathwohl, 2002), the differentiation between knowledge,
skills, and attitudes persists (Harden, 2001).
In the 1970s, constructivist approaches to curriculum
design opposed OBE. Particularly Stenhouse (1985)
emphasized curricula based on the students’ learning
process instead of the intended outcomes of this process.
During the 1990s, OBE regained popularity, especially in
designing curricula for pre-university education. Spady
(1994), whose work is seen as a turning point in curriculum
development, conserved the constructivist principle that the
students’ knowledge and skills are a consequence of learning
experiences but subordinated this principle to measurable
and observable PLOs. Precisely, Spady considered such
learning experiences, which he calls learning goals, a
requirement for obtaining PLOs. In contrast, he defines
PLOs as an explicit demonstration of learning. Therefore,
in modern OBE, PLOs serve as the reference point for the
assessment and regulation of the students’ proficiency
(Morcke et al., 2013) and the implicit consequences for
course and curriculum design (Barman et al., 2014). At the
end of the 1990s, OBE approaches became increasingly
frequent in the development of higher education curricula,
particularly in medicine, nursery, and engineering programs
that expect graduates to have specific demonstrable
knowledge, skills, and attitudes (Keating, 2014; Knight,
2001). Today, OBE is almost omnipresent, from language
(Barrette and Paesani, 2018) to nutrition (Joyner-Melito,
2016) programs.
The OBE framework systematically and deductively
facilitates and structures the development of a curriculum
by first defining the final product, the PLOs. Since OBE is
performance-oriented, it relates curriculum design to the
students’ professional careers after graduation. Accordingly,
OBE can increase graduates’ adaptation to a new job
(Harden, 2001; Hartel and Gardner, 2003; McGourty et al.,
1998). Some of the pioneers in OBE emphasized that it also
seeks to strengthen students’ role in the broader society
(Mitchell and Spady, 1978).
The adoption of OBE has not been without resistance
(Harden, 2007). The main criticism is that both, the
239
 Core stakeholders and processes in outcome-based education based on input from Harden (2002) and Koppang
(2004).

conceptual background of OBE and the empirical evidence real curriculum development but has maintained influence
to prove its efficiency, are relatively weak (Barman et al., on certain pedagogies, such as feminist teaching practices
2014). Others criticize that since most of the work is usually (Carillo, 2007) and transformational teaching (Slavich and
done by program coordinators or one specific scholar, Zimbardo, 2012).
those most affected by OBE, students, and instructors, are
not sufficiently involved (Keating, 2014). Another critique Core concepts and methods of OBE: learning
refers to the fact that personal and professional attributes outcomes and curriculum mapping
of graduates are more complex than the skills expressed Learning outcomes (rather than learning goals, learning
in PLOs (Barman et al., 2014). In the 1970s, Stenhouse’s objectives, or student competencies) are the basis of OBE.
constructivist approach was the basis for the development They refer to measurable skills and knowledge students
of an alternative curriculum development approach, the master after completing their studies (Cumming et al., 2007).
process-based curriculum. The model emphasizes the Learning goals describe attitudes, emotions, and values
pedagogy of a curriculum rather than which skills are students are expected to develop as a result of their learning
facilitated (Knight, 2001). Contrary to OBE, process-based experiences. Since they are not measurable, learning goals
curriculum development involves flexible content but a are not included in the specification of curricula in OBE
determined pedagogy throughout the program (Table 1). (Morcke et al., 2013; Spady, 1994). Learning objectives
The process-based model has never played a strong role in describe discrete but not necessarily quantifiable results
of learning. Though they are sometimes developed at
Table 1. Core differences between outcome-based and process- the program level, learning objectives usually refer to a a
based curriculum development (Knight, 2001). course (Cumming et al., 2007). While a learning objective
is characterized as an intention, the learning outcome is the
Out- Process-based
projected realization of this intention (Burke, 1995). Different
come-based curriculum
from PLOs, competencies are acquired by students or
education development
graduates, rather than the program and its instructors.
Curriculum content Defined and Flexible Furthermore, competencies build on learning experience
constantly made before post-secondary education (Morcke et al.,
updated 2013). Although there are punctual differences between
Curriculum pedagogy Commonly Largely defined outcome- and competency-based education (Morcke et al.,
flexible 2013), when referring to the point of graduation, identical
Curriculum No clear descriptors can be used for PLOs and competencies
Tools (Cumming et al., 2007).
mapping methodology
Curriculum mapping is a standard tool for curriculum
Key assess- Real PLOs are design and assessment in OBE (Harden, 2001). Following
ment parame- too complex to Plaza et al. (2007), it is most commonly used to visualize
Position on PLOs
ters be objectively the intended curriculum (what the program officers intend
assessed. to facilitate), which can be distinguished from the delivered
240 NACTA Journal • Nov 2019 - Oct 2020
 Table 2. Initial search terms. Search terms combinations were tested in the following databases: Web of Science, ScienceDirect, Wiley
Online Library, ProQuest, ERIC, Educators Reference Complete, Explora Educator's Edition.

Term 1 Term 2 Term 3

Learning Outcome AND Curriculum Assessment AND Higher education
OR OR OR
Program Learning Outcome Assessment University
Program Assessment Undergraduate
Curriculum Evaluation Graduate
Program Evaluation Major
Table 3. Systematic Review Search Terms. A combination of terms referring to curriculum assessment and evaluation in higher education
(and five synonyms) were searched in the following databases: Web of Science, ScienceDirect, Wiley Online Library, ProQuest, ERIC,
Educators Reference Complete, Explora Educator's Edition.

Term 1 Term 2 Term 3

Curriculum AND Assessment AND Higher education
OR OR OR
Program Evaluation University
Undergraduate
Graduate
Major
Food Systems

(what is effectively taught), hidden (what is communicated Methods

but not a part of the intended curriculum), the learned
(what the students take out of the program), and the
assessed curriculum (what is considered for grading). A
curriculum map generates a graphic portrayal such as a
matrix or template (Cuevas et al., 2010) that presents the
sequence of courses in the context of PLOs (Arafeh, 2016).
Curriculum mapping serves as a navigational, data-based
process tool to visualize curricular goals, including course
scopes and sequences and their relation, emphasizing the
nexus between PLOs and instructional activities as well
as highlighting at which level of proficiency the diverse
PLOs are addressed (Cuevas et al., 2010; Gluga et al.,
2012; Harden, 2002). To operationalize OBE, curriculum
mapping helps identify curriculum gaps and redundancies
(Freeman et al., 2008), discern whether different curriculum
components align (Kopera-Frye et al., 2008), and to
determine which adjustments can be made to increase this
alignment (Harden, 2001).
The role of curriculum assessment in OBE
The concept of OBE is the foundation for an entire
framework to create and improve curricula (Figure 1).
The framework involves a series of tools to develop and
improve curricula, one of which is curriculum assessment.
Apart from program design and assessment, OBE supports
the connectivity between academic learning activities
(Harden, 2007; Tonni and Oliver, 2013), commonly labeled
as curriculum alignment or “scaffolding”.
PRISMA protocol
This systematic literature review (SLR) was carried out
to collect and analyze published evidence on the following
closed-frame study question: In outcomes-based education
in higher education curricula, what are practices for
curriculum assessment that incorporate learning outcomes?
We followed the Preferred Reporting Items for Systematic
Reviews and Meta-Analyses (PRISMA) protocol (Moher
et al., 2009). The procedure involves four essential steps:
(1) search term definition and article search, (2) title and
abstract screening, (3) full-text screening, and (4) data
systematization and analysis.
In addition to the PRISMA protocol, the Population,
Intervention/Exposure, Comparator, Outcome (PICO)
framework (Schardt et al., 2007) was applied to structure our
review. The measured outcome of the studies we reviewed
was curriculum assessment criteria and procedures
applied to higher education curricula. The components of
the PICO framework were: (1) the population equivalent is
higher education curricula; (2) the intervention/exposure
is curriculum assessment; (3) the comparator is pre- and
post- curriculum assessment based on PLOs and; (4) the
outcomes are specific assessment criteria and procedures.
Identification of search terms
The overall research question of the SLR determined
the search terms following the PRISMA protocol. A
multidisciplinary team with expertise in food systems
education helped determine the initial search terms (Table
2) and the final set of search terms (Table 3). Our goal was
to identify all publications sharing outcomes of concrete

NACTA Journal • Nov 2019 - Oct 2020 241

 Table 4a. Appearance of search terms in articles (out of seven reviewed databases).

Term 1 Term 2 Term 3 Quantity of publications Rank**

(Median)*
Learning Outcome Curriculum Assessment Higher education 14 22
University 19 18
Undergraduate 13 24
Graduate 9 27
College 15 20
Assessment Higher education 137 5
University 202 1
Undergraduate 173 3
Graduate 173 3
College 194 2
Program Assessment Higher education 65 9
University 63 10
Undergraduate 51 14
Graduate 54 13
College 60 12
Curriculum Evaluation Higher education 20 16
University 31 15
Undergraduate 20 16
Graduate 15 20
College 17 19
Program Evaluation Higher education 83 7
University 106 6
Undergraduate 62 11
Graduate 68 8
College 14 22

curriculum assessment interventions in higher education
with an emphasis on assessment experiences based on
PLOs. The result of this discussion was to select all possible
combinations of three search terms, where there were two
options for the first term, as well as five options for the
second and third term, respectively (Appendix 1).
To specify our search, we tested all fifty resulting search
term combinations of terms listed in Table 2 in seven different
publication databases (Web of Science, ScienceDirect,
Wiley Online Library, ProQuest, ERIC, Educators Reference
Complete, and Explora Educator's Edition). We selected
these databases due to their relevance for identifying peer-
reviewed publications in all scientific fields as well as being
sources of high-quality publications in the field of education.
We revised the search terms through an iterative
process. First, we ranked the search term combinations
based on the number of publications associated with each
combination considering a median number of publications
throughout the seven databases. Therewith, we identified
the quadrille of the least frequent combinations and discard
them. Second, we examined relevant articles resulting from
the search process and refined the remaining search terms.
Hence, we guaranteed a balanced and systematic review
based on all significant search term combinations.
The search term testing of the initial 50 search term
242
combinations) in the seven selected databases showed that
the term combinations including “Curriculum Evaluation” (as
combined words) as the second search term resulted in the
lowest number of associated articles (Table 4). Based on
the results of the initial search term testing, we selected 24
word combinations for our final list of search terms (Table 3).
Based on this list, we identified 525 articles for our review
process. After removing 58 duplicates, we finally reviewed
467 publications.
The final set of search terms was processed in the same
seven databases as used in the initial process of identifying
search terms. We searched for the appearance of the final
set of search terms in the title, abstract, or keywords of peer-
reviewed articles in these databases. Additionally, in the
more topic-specific databases (ERIC, Educators Reference
Complete, Explora Educator's Edition), we expanded
our search to the appearance of the respective search
terms in the full text. We emphasized articles about case
studies regarding implemented curriculum assessment
processes, which we classified as primary data. During
the initial process of identifying search terms, it became
apparent that publications on such case studies are rare.
Thus, we also included reviews, opinion papers, and other
communications about curriculum assessment, which we
classified as secondary data.
NACTA Journal • Nov 2019 - Oct 2020
 Table 4b. Appearance of search terms in articles (out of seven reviewed databases).

Term 1 Term 2 Term 3 Quantity of publications Rank**

(Median)*
Program Learning Curriculum Assessment Higher education 0 40
Outcome University 0 40
Undergraduate 0 40
Graduate 0 40
College 0 40
Assessment Higher education 11 25
University 11 25
Undergraduate 4 30
Graduate 8 28
College 8 28
Program Assessment Higher education 3 32
University 4 30
Undergraduate 3 32
Graduate 3 32
College 3 32
Curriculum Evaluation Higher education 0 40
University 0 40
Undergraduate 0 40
Graduate 0 40
College 0 40
Program Evaluation Higher education 2 36
University 2 36
Undergraduate 0 40
Graduate 2 36
College 2 36

* Median of identified publications among publications databases

** Based on median, descending

Inclusion criteria and initial article screening
For the abstract review process, the a-priori inclusion
criteria were: (i) peer-reviewed publications between 1995-
2019; (ii) articles published in the English language; (iii)
the title, abstract, or keywords include one of our search
terms; and (iv) the abstract addresses the research
question directly. The exclusion criteria include (i) books,
book chapters, reports, maps, newspaper articles, and non-
print media; (ii) articles without an abstract; (iii) abstracts
written in poor English or in a language other than English;
(iv) articles published before 1995; (v) articles about
curriculum assessment in countries other than the USA,
the UK, Japan, Canada, Switzerland, France, Denmark,
Israel, Germany, and Sweden (the ten leading countries in
the Center for World University Rankings (CWUR) index
(CWUR, 2019); (vi) articles that do not discuss curriculum
assessment criteria and procedure; (vii) articles which
are not about higher education (for example, elementary,
middle, secondary education); (ix) articles about student
assessment (for example, exams or assignments); (x)
articles about instructor or course assessment; (xi) articles
about self-assessment; and, (xii) articles about program
development without an assessment component.
The title and abstract of each article were screened
for their relevance to addressing the study question. Two
review panel members screened each article to minimize
reviewer bias. Discrepancies on whether to exclude or
include a specific article were discussed by at least three
reviewers to resolve conflicts between reviewers. All articles
that met the inclusion criteria were included in the full-text
screening; all other articles were excluded.
The identified articles from the initial screening that met
the inclusion criteria were critically appraised by the review
panel to ensure they provided evidence to address the study
question and met the a-priori inclusion criteria. Two panel
members screened each full text. Again, discrepancies were
discussed and resolved by at least three reviewers. After
excluding inappropriate articles, the final set of publications
was included in the quantitative synthesis of our systematic
review.
The specific parameters and evidence extracted from
each article during the synthesis included: (i) article title,
(ii) article author(s) and publication year, (iii) journal, (iv)
observed program(s), (v) definition (understanding) of

NACTA Journal • Nov 2019 - Oct 2020 243

curriculum assessment, (vi) assessment subjects (what
is assessed), (vii) levels of assessment, (viii) frequency
of assessment, (ix) components of assessment, (x)
assessment follow-up, (xi) advantages of assessment,
(xii) assessment of limitations, and (xiii) ways to improve
curriculum assessment. For case studies, we additionally
extracted the respective PLOs, assessment coordination
and execution, the assessment procedure, and the
assessment timeline. Finally, we examined primary and
secondary data for reasons for executing curriculum
assessment, preconditions and planning for assessment,
assessment methods, and tools, assessment follow-
up, assessment frequency, as well as advantages and
challenges of curriculum assessment in OBE.
Using the software NVivo 11 (Leech and Onwuegbuzie,
2011), one assigned panel member per article extracted the
parameters and evidence from each text to systematically
synthesize all remaining articles. Afterward, articles were
split into two groups according to the type of data used in
the study: (1) articles using primary data (case studies) and
(2) articles using secondary data (reviews or methodological
discussions). We decided to discuss certain contents only
based on primary data, and others based on both primary
and secondary data.
Figure 2. PRISMA Flow Diagram (Moher et al., 2009) generated by the software Covidence resuming the literature screening process
from the number of articles identified through the database searching to the final number of studies included in the qualitative data
extraction.
244
Abstract & title screening and full-text
screening
Using the systematic review platform Covidence (Götz,
2018) and applying inclusion criteria to article titles and
abstracts, 43 out of 525 articles were identified as appropriate
for full-text screening. The review panel conducted a full-
text screening of the 43 articles, of which, 23 articles were
determined to not meet the inclusion criteria and were
removed. This resulted in a total of 20 articles that met the
inclusion criteria and addressed the study question. Figure
2 outlines the PRISMA Flow Diagram of the screening
process used in the systematic review (Moher et al., 2009).
Categorization of the articles based on data type (primary
versus secondary data) resulted in seven articles that used
primary data and 13 articles based on secondary data.
The seven primary data papers dealt with 13 different case
studies, each of which about the assessment of a specific
undergraduate program. Of these 13 majors, six regarded
Science Technology Engineering and Mathematics (STEM)
programs, and seven were about programs in humanities,
social sciences, or art (Table 5).
NACTA Journal • Nov 2019 - Oct 2020
Table 5. Publications reviewed and synthesized in the systematic literature review that address the study question, “In outcomes-based
education in higher education curricula, what are practices for curriculum assessment that incorporate learning outcomes”.

Reference Article Title Primary (P) Study site (only Assessed

or Secondary P) Programs (only P)
(S) data
Barrette and Conceptualizing cultural literacy through student P Wayne State Foreign Language
Paesani (2018) learning outcomes assessment University
Beasley et al. Systematic plan of evaluation part I: Assessment of S
(2018) end-of-program student learning outcomes
Benson and Systemic decision application: linking learning outcome S
Dresdow assessment to organizational learning
(1998)
Dunning (2014) Developing a Competency-Based Assessment P Troy University Public Administration
Approach for Student Learning
Felder and Designing and Teaching Courses to Satisfy the ABET S
Brent (2003) Engineering Criteria
Feldhaus Assessment of Undergraduate Risk Management and S
(2002) Insurance Programs
Gosselin et al. Integration of sustainability in higher education: three P Northern AZ Environmental
(2013) case studies of curricular implementation University, Studies, Earth
University of Sciences and
Nebraska- Environmental
Lincoln, Kean Sustainability,
University Sustainability
Science
Hartel and Making the Transition to a Food Science Curriculum S
Gardner (2003) Based on Assessment of Learning Outcomes
Hartel and A Report from the Higher Education Review Board S
Iwaoka (2016) (HERB): Assessment of Undergraduate Student
Learning Outcomes in Food Science
Houston (2005) Outcomes Assessment for Beginning and Intermediate P Saint Louis Spanish
Spanish: One Program's Process and Results University
Joyner-Melito Curriculum Mapping: A Method to Assess and Refine P University of ID Food Science
(2016) Undergraduate Degree Programs
Mathews and Ongoing Assessment of a University Foreign Language S
Hansen (2004) Program
McGourty et al. Developing a Comprehensive Assessment Program for S
(1998) Engineering Education
McGourty Four Strategies to Integrate Assessment into the S
(1999) Engineering Educational Environment
Metzler et al. The Aspirational Curriculum Map: A Diagnostic Model S
(2017) for Action-Oriented Program Review
Myers et al. Assessing an Economics Programme: Hansen S
(2009) Proficiencies, ePortfolio, and Undergraduate Research
Swarat et al. How Disciplinary Differences Shape Student Learning P California State Physics, history,
(2017) Outcome Assessment: A Case Study University civil engineering,
child and adolescent
studies
Thompson and The Management Curriculum and Assessment S
Koys (2010) Journey: Use of Baldrige Criteria and the Occupational
Network Database
Tonni and A Delphi approach to define learning outcomes and S
Oliver (2013) assessment
Wright et al. Assessment from an educational development P University of Gender and
(2017) perspective Iowa, University Sexuality Studies,
of WI Madison, Business, Art
University of
Michigan, Duke
University
NACTA Journal • Nov 2019 - Oct 2020 245
 Figure 3. Frequency of use of assessed sources for curriculum assessment in 13 reviewed case studies.
Results and Discussion
A. Case study (primary data) results
Following the reviewed literature, individual faculty
are most commonly in charge of the administration
and coordination of curriculum assessment, followed
by assessment committees comprising of core faculty
members, university administrators, and independent
consultants. Further assessment coordinators include
deans and department heads whose primary role is to
appoint the assessment coordinator or committee.
In all reviewed case studies, the curriculum assessment
is based on specific documents, which we term as the
assessed source. Six out of 13 case studies use one single
document as assessed source, three case studies use
two or three documents, three base their assessment on
more than three documents, and in one case, the assesses
sources are not specified. Figure 3, which illustrates the
assessed sources and their frequency of occurrence
across the thirteen reviewed case studies, shows that the
most common sources are student assignments, followed
by student surveys and interviews. Table 6 (following page)
presents specifications regarding the diverse sources
for assessment. Accredited survey tools, developed and
assessed by external certifiers, are common when curricula
246
aim for accreditation by third-party institutions. Although
the reviewed studies indicate which sources were used for
assessment, they do not discuss their effectiveness and
adequacy in the actual assessment process.
Joyner-Melito (2016) provides insight into curriculum
assessment based on student assignments, one of the
most common procedures: using a curriculum map, the
assessment coordinator identifies a course which addresses
a particular PLO; then, the course instructor reviews their
course syllabus and identifies assignments, readings, or
other activities addressing this PLO; the instructor reports
the students’ performance in this assignment, usually
the average grading, to the assessment coordinator; the
coordinator then contrasts the performance with certain
threshold values for each PLO indicating its successful or
unsuccessful facilitation through the curriculum.
The assessment criteria used in each of the case
studies depend on the program and level of assessment.
Of the seven case studies that specify their assessment
criteria, six state that their goal is to measure how efficiently
programs addressed their PLOs.
While assessment execution varies greatly among the
case studies, most were systematic in their assessment
procedures. For example, Swarat et al. (2017) note that
the assessment of PLOs in a physics B.S. program was
NACTA Journal • Nov 2019 - Oct 2020
Table 6. Documents and data (assessed sources) used as the basis for curriculum assessment.

Assessed sources Specification

Student assignments Student scores of selected course assignments, exams (or specific exam
questions), (not further specified) narratives, oral presentations, poster
presentations, research proposals, or diverse writings (Dunning, 2014;
Joyner-Melito, 2016; Swarat et al., 2017; Wright et al., 2017), in some case
limited to the evaluation of senior students (Dunning, 2014).
Case analyses emphasizing the appli- Commonly flexible
cation of analytical skills and knowledge
gained from curriculum courses (Dun-
ning, 2014)
Assessment of capstone course projects Curriculum mapping
and oral presentations (Gosselin et al.,
2013).
Student scores of assignments purpose- Key assessment parameters
ly designed for curriculum assessment
(Swarat et al., 2017).
Exit interviews, not specified (Gosselin et al., 2013; Houston, 2005; Swarat
Student surveys and interviews
et al., 2017).
Adaptation of external exam instrument to program needs (Dunning, 2014;
Accredited survey tool
Swarat et al., 2017)
Certified placement exam results (Hous-
ton, 2005).
Application of external exam in capstone
course (Gosselin et al., 2013).
Purposeful collection of student work (essays, presentations, projects,
creative products) during their time as students; may include videotaped pre-
Student portfolios
sentations and audio-recorded interviews; assessment different from course
grading (Barrette and Paesani, 2018; Houston, 2005).
Student satisfaction survey, usually course- and instructor-specific (Houston,
Course / instructor evaluations
2005; Wright et al., 2017).
Congruence of course syllabi and course PLOs with PLOs (Barrette and
Course syllabi
Paesani, 2018; Joyner-Melito, 2016).
Student performance (average grading) in junior and senior level courses
Registrar data
(Gosselin et al., 2013).
Student experience and demographics,
both not specified (Wright et al., 2017).

Table 7. Example of a calibrated assessment rubric. Troy University, Master of Public Administration, PLO 3 (“Evaluate the primary sources
of revenue at all levels of government based on the principles of taxation”). Example for the assessment of an assignment regarding the
discussion about the potential for implementing transaction-based taxes in a determined case. Resumed and modified from Dunning
(2014).

Does not meet expectations Meets expectations Exceeds expectations

The discussion does not fully identify The discussion fully identifies the The discussion provides a complete
the primary sources of revenues at primary sources of revenues at all description of the primary sources of
all levels of government or evaluate levels of government and evaluates revenues at all levels of government
their effectiveness. their effectiveness.

NACTA Journal • Nov 2019 - Oct 2020 247

administered by reviewing the students’ performance in
specific assignments (selected based on their alignment to
PLOs), such as capstone presentations and exam scores.
Assessment results were then consulted to modify and
improve the overall program.
In most case studies, curriculum mapping allows
the assessment administrators to adequately select the
appropriate assessment tools. One case study highlights
how the assessment administrator went so far as to
complete a gap/redundancy analysis from the curriculum
map to illustrate if any PLOs were not being addressed, and/
or if they were becoming redundant by being introduced in
too many courses (Joyner-Melito, 2016).
To ensure reliability within those who were assessing
the sources of documents evaluated during curriculum
assessment, four out of 13 reviewed studies utilize
calibrated assessment rubrics to decrease the variability
among assessment results. These rubrics resemble
rubrics used for the assessment of student performance
in exams or assignments (see Table 7 for an example).
In none of the 13 reviewed case studies, data about an
evidenced measurement of the quality and functioning of
the assessment process was identified.
B. Primary and secondary data results
Regarding motives for implementing curriculum
assessment, six of twenty reviewed articles point to an
assessment process mandated and directed by external
accrediting agencies like Accreditation Board for Engineering
and Technology (ABET), an institution which certifies
natural science, computing, and engineering programs, or
the Commission on Collegiate Nursing Education (CCNE),
which certifies nursing programs in response to industry
demand (ABET, 2020; CCNE, 2019; McGourty, 1999).
Three of 20 publications mention explicit preconditions
for initiating an assessment process including (i) agreement
on PLOs, (ii) development of a coordinated intended
curriculum based on meeting these PLOs, (iii) mapping of
the intended curriculum, (iv) development of a standardized
syllabus template, (v) implementation of the curriculum, (vi)
assessment training for faculty and staff (the training may
continue during assessment), and, (vii) development of a
coordinated assessment plan.
A quarter of the reviewed papers suggests developing an
assessment plan before initiating a curriculum assessment
process (Beasley et al., 2018; Felder and Brent, 2003;
Joyner-Melito, 2016; McGourty, 1999; Myers et al., 2009;
Wright et al., 2017). Following these sources, a systematic
assessment plan should involve the assessment frequency,
ways to integrate faculty and staff, assessment methods,
preparation of an assessment toolbox (templates and forms
for all assessors), expected levels of achievement, a data
analysis strategy, and rules about the decision making
based on the data.
Assessment implementations commonly begin with
developing clear learning goals or outcomes and then
selecting the best assessment tools to gather information
about student achievement of the goals (Hartel and
Gardner, 2003). A few reviewed articles emphasize that
the appropriate selection of assessment methods (direct or
248
indirect assessment indicators, or a mix of both) depends
on the nature of the assessed program. Direct indicators
of learning evaluate how the curriculum impacts student
learning; they measure the students’ mastering of the PLOs
(Benson and Dresdow, 1998). In contrast, indirect indicators
serve to continually improve the curriculum by measuring
the professional performance of graduated students
(based on their self-assessment or the opinion of external
stakeholders such as employers). Indirect indicators may
also include the perception of instructors as in the case
of syllabus analysis (Feldhaus, 2002; Hartel and Gardner,
2003; Joyner-Melito, 2016). Direct indicators are more
frequently applied in curriculum assessment than indirect
ones (Table 8).
Two articles (Felder and Brent, 2003; Hartel and
Gardner, 2003) suggest that once the assessment process
concludes, the following documents and processes should
be developed as part of the follow-up process:
• Program outcome assessment matrix, which
indicates which courses might be modified,
• Course assessment matrix for each of those courses,
suggesting areas that require strengthening,
• Teaching plan suggesting pedagogical strategies to
increase alignment with PLOs,
• Updated curriculum map,
• Identification, review, and (if necessary) modification
of key institutional practices to ensure alignment
with educational outcomes and the assessment
process,
• Scheduling of a process to re-evaluate PLOs, and
• Documentation of all decision making
based on the use of the assessment data.
In terms of frequency of assessment, two publications
recommend initial baseline studies right after the creation of
a new program (Gosselin et al., 2013; Houston, 2005). Of all
reviewed publications, about a third (seven out of 20) gives
explicit advice about the ideal interval between assessment
processes: Three suggest curriculum assessments every
second or third year, two recommend yearly assessment,
and two recommend assessment in a five-year interval.
Seven out of 20 papers discuss the advantages of
curriculum assessment employing PLOs. These advantages
arise in three main areas: clarification of expectations,
faculty and staff engagement, and program improvement.
They can be categorized into advantages for three distinct
audiences: students, faculty, and the broader institution.
One paper identifies clarity of expectations and self-
assessment of the learning process as an advantage
for student learning. Six out of twenty articles identified
advantages for faculty, which include advantages at the
program level. These advantages included:
• Facilitation of evidence-based teaching (two of 20
publications reviewed),
• Improvement in designing and applying an
appropriate pedagogy (2/20),
• Facilitation of student assessment and grading
linked to PLOs (2/20),
NACTA Journal • Nov 2019 - Oct 2020
Table 8. Direct and indirect tools for curriculum assessment highlighted in the reviewed articles combining primary and secondary data
(n=20).
Assessment tool Number of reviewed articles (n=20) Direct (D) or indirect (I) assessment
recommending the assessment
tool
Courses with embedded additional
assessment
Tests and evaluations
Student portfolio of issue papers
Capstone course evaluation (or only
final capstone presentations)
Thesis evaluation
Exit competency exam
Employer surveys
Alumni surveys (after 5 years)
Salary surveys
Internship evaluations
Student/graduate surveys
External reviewers
Curriculum / syllabus analysis
• Constant update of course syllabi based on PLOs
(1/20),
• Consistency between program mission (educational
objectives) and teaching (2/20),
• Development and examination of the optimum
sequence of courses within a curriculum (2/20),
• Identification of gaps between the intended PLOs
and actual student learning (2/20),
• Identification of gaps between the intended PLOs
and student learning opportunities (2/20),
• Increased collaboration and alignment between
faculty around courses (1/20), and
• Improvement in program development
and curriculum in general (4/20).
Six out of 20 articles refer to advantages for the
program and the institution, especially thanks to an explicit
link between PLOs and the assessment process. These
advantages include:
• Coordination of curricular activities (2/6),
• Enhanced dialogue among faculty groups teaching
the same course (1/6),
• Communication of learning goals (1/6),
• Improvement of relations with broader stakeholders,
students, alumni and employers (1/6), and
• Generally improved faculty and staff engagement
(1/6).
None of the reviewed articles provides data about the
effectiveness of curriculum assessment in general and the
functionality of their proposed approaches. However, twenty
percent of the reviewed publications discuss the challenges
and disadvantages of curriculum assessment in OBE based
NACTA Journal • Nov 2019 - Oct 2020
indicators (Hartel and Gardner,
2003)
3 D
3 D
2 D
2 D
1 D
1 D
2 I
2 I
1 I
1 I
2 I
1 I
1 I
on subjective impressions of the authors. In this context,
two publications (Barrette and Paesani, 2018; Hartel and
Gardner, 2003) report that implementing PLO-based
assessment can be difficult, particularly if the commitment
of the faculty is weak. Accordingly, few instructors respond
well to being told what to do in their classes. Hence, top-
down approaches to curriculum assessment may create
more resistance than support among the faculty needed to
carry out and respond to the assessment. This is especially
true when the value of the assessment process is not clear
to faculty and instead it is perceived as a burden on time
and energy (Barrette and Paesani, 2018). Resistance is
spurred by fears that PLO-based assessment may lead to
unwanted change or even retraining requirements when
faculty feel their current courses and programs are working
well enough (Hartel and Gardner, 2003).
Occasionally, PLOs lack specificity, and prove difficult
to be measured in practice (Beasley et al., 2018; Dunning,
2014), especially when PLOs originate at the institution
level but are implemented at the program level without
sufficient attention to adapting the PLOs to the discipline
being assessed (Barrette and Paesani, 2018).
Synthesis of findings
Our review highlights that the literature about program
assessment in higher education is scarce and that existing
publications lack evidence about the effectiveness of applied
assessment practices. Furthermore, we identified notable
variations in the applied practices themselves. Both, the
scope and the applied practices at 13 different institutions
we covered in our review, are widely inconsistent. As
similar programs sometimes apply fundamentally different
assessment practices, the explanation of this variation
cannot be limited to the diversity of the programs.
249
 The nature of a specific assessment procedure is most holistic approach to curriculum assessment.
strongly defined by which assessment indicators (direct Consequently, a 360 Degree Assessment process was
versus indirect) are used. Direct indicators evaluate how developed, considering
the curriculum impacts student performance as related to
program LOs whereas indirect indicators evaluate how the • how students master the skills related to the eight
students’ learning process impacts external stakeholders adaptable PLOs;
and the students’ perception of learning achievements. • how senior students perceive their learning process
There is no explicit correlation between the assessed regarding each PLO;
majors and the recommended assessment practice. • how instructors perceive the students’ learning
However, we observed the tendency that STEM and achievements regarding each PLO;
economics programs rely stronger on direct assessment • how SFBS alumni assess the utility of the SFBS
indicators (e.g. average student scores from assignments), curriculum for their professional careers;
while language programs favor indirect indicators (e.g. • and how employers of SFBS alumni assess what
student surveys and interviews). Mixed (combined direct and their employees learned in the SFBS program.
indirect) indicators are common in social sciences programs
(yet we also identified STEM and language programs using Our preliminary assessment plan includes different
mixed indicators). Swarat et al. (2017) suggest that applied procedures for each of the five components (Table 9). The
assessment practices in a program differ based on the program assessment will be guided by a committee including
epistemological norms and culture in the respective science. faculty and students, directed by the program coordinator.
This circumstance may explain the preference of STEM and We plan to assess the SFBS program in an interval of two
economics programs for direct assessment indicators, as to three years and based on a regular assessment of the
the analysis methods (such as analysis of variance) used functionality of our PLOs.
to process assessment data are common in these sciences
(Jenner et al., 2004). On the other hand, indirect indicators Summary and Conclusions
such as interviews require qualitative methods, which are
prevalent in linguistics (Abbuhl et al., 2014).
Institutions of higher education prepare their students
Among the direct indicators, average student scores
for becoming impactful members of their communities
in selected assignments are most frequent. Numerous
and of a well- and workforce prepared for facing the
institutions emphasize assignments that are part of regular
complex and global social, economic, and environmental
course activities, while other institutions design assignments
challenges of our times. To increase the impact, accuracy,
exclusively for curriculum assessment or use assignments
and objectivity of their educational efforts, an increasing
developed by external certifying entities.
number of academic programs have defined explicit
The reviewed case studies highlight the importance of
and specific program learning outcomes (PLOs) for their
having a designated group of individuals, commonly faculty
curricula (Caspersen et al., 2017). The PLOs ensure that
or program coordinators, to administer and coordinate
all stakeholders involved in the teaching of a determined
curriculum assessment. Occasionally, external stakeholders
program (students, instructors, researchers, coordinators,
and alumni are involved in curriculum assessment.
administrators, and external collaborators) contribute to the
The core assessment criteria in most case studies
achievement of the same outcomes in student learning,
are PLOs. A few case studies note specific protocols to
while simultaneously maintaining a diversity of teaching
ensure the reliability of the assessment procedure including
styles and pedagogical approaches (Harden, 2007; Knight,
assessment rubrics similar to those used for grading student
2001).
assignments. Yet, the provided evidence is not strong
Only defining outcomes is not enough; there is a need
enough to ensure that this is the most appropriate format.
to ensure that new and existing programs are effective in
Several reviewed articles recommend an assessment
meeting their PLOs: Outcome-based education (OBE) is a
plan before as well as follow-up activities after assessment.
framework that systematically serves to align a program’s
Follow-up activities include a teaching plan suggesting
curriculum, mainly course offer, and sequence as well as
pedagogical strategies to increase the alignment of all
core pedagogical activities, to achieve its PLOs (Spady,
courses with the PLOs. Additionally, a constant PLO
1994). The framework offers guidelines to develop
reevaluation is recommended. According to several
curricula of new programs and to analyze existing ones.
publications, assessment should be carried out in an interval
For programs already being implemented and where clear
of two or three years.
PLOs are defined, OBE provides the process of curriculum
assessment to measure the effectiveness of the design,
Proposal for a curriculum assessment
delivery, and direction of an educational program based on
procedure
its PLOs (McGourty et al., 1998).
As of 2020, the Sustainable Food and Bioenergy
Curriculum assessment (sometimes also labeled
Systems Major (SFBS) at Montana State University will be
program assessment) serves to verify whether the intended
implementing an extensive curriculum assessment process
curriculum of a program is congruent with what students
based on the findings of the present review. Especially,
are learning, the learned curriculum (Plaza et al., 2007). In
we considered the diversity of assessment tools that were
other words, it puts the rule to the test: Is the curriculum
applied in those of the reviewed articles which provided the
250 NACTA Journal • Nov 2019 - Oct 2020
Table 9. Set of assessment tools applied in the assessment of the Sustainable Food and Bioenergy Systems Major, Montana State
University.
Target group Method
Students (skill mastering) Assignments
Students (perception)
Instructors Semi-structured interviews
Alumni Focus groups
Employers
facilitating all the PLOs it pretends to generate?
Consequently, numerous programs have made
curriculum assessment an essential component of their
efforts to improve their curricula. Critics of OBE claim that
both, the conceptual background of OBE and the evidence
to prove its efficiency, are weak and insufficient for assessing
contemporary programs with an increasing emphasis on
global and holistic thinking (Barman et al., 2014; Bryan and
Clegg, 2019). Through our systematic literature review, we
can confirm that the literature on curriculum assessment
is not as abundant and specific as, for example, literature
about curriculum mapping, another OBE process (Cuevas
et al., 2010).
The objective of our systematic review was to identify
the best practices for curriculum assessment in OBE based
on evidence of successful program assessment. We could
not find such evidence in the existing literature as even case
studies do not provide an analysis of the effectiveness of
the applied assessment processes. However, we identified
commonly used assessment strategies and we also
considered the subjective appraisal of these strategies by
the diverse authors. In this regard, we identified the following
basic recommendations for curriculum assessment:
•Existence of an assessment plan (including
assessment frequency, faculty and staff
integration, and assessment methods, among
NACTA Journal • Nov 2019 - Oct 2020
Specification
For each PLO, we will develop an
assignment to prove whether the
students master the PLO at advanced
level of proficiency. These (ungraded)
assignments will be implemented in
junior and senior courses.
Survey Students will take the survey in their
capstone course. They will be asked
how they assess the facilitation of
each of the PLOs throughout the
SFBS curriculum.
After a faculty meeting, we will
interview each SFBS instructor who
teaches a course that addresses a
PLO at advanced level of proficiency
about how they perceive the SFBS
curriculum in terms of facilitating the
PLOs.
During an online conference or
alumni meeting, we will have focus
groups discussing the effectivity of
the PLOs (and how they were taught)
for their application in a professional
workspace.
Survey Selected employers will take online
surveys regarding the preparedness
of SFBS alumni for their jobs.
other components) before initiating curriculum
assessment process (Myers et al., 2009);
• Assessment implementation through a committee
of involved individuals, usually faculty, to guarantee
commitment and active participation (Feldhaus,
2002) and inclusion of students into the assessment
process (Keating, 2014);
• Prior curriculum mapping to select the appropriate
assessment tools for each course (Metzler et al.,
2017);
• A mix of direct and indirect assessment indicators
including student scores of selected course
assignments (preferably purposely designed for
curriculum assessment) as well as exit and alumni
interviews (Gosselin et al., 2013; Swarat et al.,
2017);
• Calibrated assessment rubrics to decrease the
variability among assessment results (Dunning,
2014);
• Follow-up activities including a teaching plan
including pedagogical strategies to increase
program alignment with PLOs and a process of
reevaluation of PLOs (Felder and Brent, 2003;
Hartel and Gardner, 2003);
• Curriculum assessment every second or third year
(Swarat et al., 2017).
251
 Curriculum assessment helps to avoid arbitrariness
and subjectivity in curricula. Therefore, we consider it
an essential contribution to a constant improvement of
higher education curricula. However, we are aware of
the fact that curriculum assessment has been historically
pushed by the neoliberal “assessment movement”, which
sought to examine the effectiveness of funds invested
in higher education institutions by assessing learning
upon completion of their degrees (Tam, 2014). We do not
advocate for program assessment to limit funding but to
improve education.
The outcome of curriculum assessment depends on
the used PLOs, the lens under which assessment occurs.
Although we consider curriculum assessment a per se
neutral process, we also know that the influence of faculty
on defining PLOs has decreased while the impact of external
stakeholders such as industry leaders has increased (Millar,
2016). In contrast, we favor curriculum assessment based
on PLOs that contemplate far-reaching societal needs
instead of short-term job market demands.
The process of curriculum assessment is worth being
sustained with coherent and evidence-based guidelines. We
hope that with our literature review we prepared the ground
for comprehensive and adaptable curriculum assessment.
Acknowledgments
We are grateful to the students, colleagues, staff, and
education stakeholders who influenced our understanding
of curriculum assessment. Funding for this project is
provided by the United States Department of Agriculture
Higher Education Challenge Grant, National Institute of
Food and Agriculture (Award Number: 2018-70003-27649).
Literature Cited
Abbuhl, R., S. Gass and A. Mackey. 2014. Experimental
research design. In: Podesva, R. J. and D. Sharma,
(eds). Research methods in linguistics. Cambridge, UK:
Cambridge University Press,
ABET (Accreditation Board for Engineering and Technology).
2020. About ABET. https://www.abet.org/about-abet/.
ABET. April 10, 2020.
Arafeh, S. 2016. Curriculum mapping in higher education: a
case study and proposed content scope and sequence
mapping tool. Journal of Further and Higher Education
40: 585-611.
Barman, L., C. Silen and K. Laksov. 2014. Outcome based
education enacted: teachers' tensions in balancing
between student learning and bureaucracy. Advances
in Health Sciences Education 19: 629-643.
Barrette, C.M. and K. Paesani. 2018. Conceptualizing
Cultural Literacy through Student Learning Outcomes
Assessment. Foreign Language Annals 51: 331.
252
Beasley, S.F., S. Farmer, N. Ard and K. Nunn-Ellison. 2018.
Systematic Plan of Evaluation Part I: Assessment of
End-of-Program Student Learning Outcomes. Teaching
and Learning in Nursing 13: 3-8.
Benson, J. and S. Dresdow. 1998. Systematic Decision
Application: Linking Learning Outcome Assessment to
Organizational Learning. Journal of Workplace Learning
10: 301.
Bloom, B.S. 1956. Taxonomy of educational objectives (Vol.
1: Cognitive domain). New York City, NY: McKay.
Bryan, C. and K. Clegg. 2019. Innovative Assessment
in Higher Education: A Handbook for Academic
Practitioners. London, UK: Routledge.
Burke, J.W. 1995. Outcomes, learning, and the curriculum:
implications for NVQs, GNVQs, and other qualifications.
London, UK: Psychology Press.
Carillo, E.C. 2007. Feminist" teaching/teaching" feminism.
Feminist Teacher 18: 28-40.
Caspersen, J., N. Frølich and J. Muller. 2017. Higher
education learning outcomes–Ambiguity and change in
higher education. European Journal of Education 52:
8-19.
CCNE (Commission on Collegiate Nursing Education).
2019. CCNE Accreditation. https://www.aacnnursing.
org/CCNE. Commission on Collegiate Nursing
Education. April 10, 2020.
CWUR (Center for World University Rankings). 2019.
World's Top Universities. https://cwur.org/2019-2020.
php. CWUR. November 7, 2019.
Cuevas, N.M., A.G. Matveev and K.O. Miller. 2010. Mapping
general education outcomes in the major: Intentionality
and transparency. Peer Review 12: 10.
Cumming, A., A. Cumming and M. Ross. 2007. The
Tuning Project for Medicine–learning outcomes for
undergraduate medical education in Europe. Medical
teacher 29: 636-641.
Dunning, P.T. 2014. Developing a Competency-Based
Assessment Approach for Student Learning. Teaching
Public Administration 32: 55.
Eichinger, M. 2019. Transformational change in the
Anthropocene epoch. The Lancet Planetary Health 3:
116-117.
Felder, R.M. and R. Brent. 2003. Designing and teaching
courses to satisfy the ABET engineering criteria. Journal
of Engineering Education 92: 7-25.
Feldhaus, W.R. 2002. Assessment of undergraduate
NACTA Journal • Nov 2019 - Oct 2020
 risk management and insurance programs. Risk
Management and Insurance Review 5: 155-171.
Freeman, M., P. Hancock, L. Simpson, C. Sykes, P. Petocz,
I. Densten and K. Gibson. 2008. Business as usual:
A collaborative and inclusive investigation of existing
resources, strengths, gaps and challenges to be
addressed for sustainability in teaching and learning in
Australian university business faculties. ABDC Scoping
Report 1: 1-54.
Gluga, R., J. Kay and T. Lever. 2012. Foundations for
modeling university curricula in terms of multiple
learning goal sets. IEEE Transactions on Learning
Technologies 6: 25-37.
Gosselin, D., R. Parnell, N.J. Smith-Sebasto and S. Vincent.
2013. Integration of sustainability in higher education:
three case studies of curricular implementation. Journal
of Environmental Studies and Sciences 3: 316-330.
Götz, S. 2018. Supporting systematic literature reviews
in computer science: the systematic literature
review toolkit. In: Proceedings of the 21st ACM/IEEE
International Conference on Model Driven Engineering
Languages and Systems. Copenhagen, Denmark: 14-
19 Oct.
Harden, R., M. 2001a. AMEE Guide No. 21: Curriculum
mapping: a tool for transparent and authentic teaching
and learning. Medical teacher 23: 123-137.
Harden, R.M. 2002. Developments in outcome-based
education. Medical teacher 24: 117-120.
Harden, R.M. 2007. Outcome-based education: the future
is today. Medical teacher 29: 625-629.
Hartel, R. and D. Gardner. 2003. Making the transition to
a food science curriculum based on assessment of
learning outcomes. Journal of Food Science Education
2: 32-39.
Hartel, R. and W. Iwaoka. 2016. A report from the Higher
Education Review Board (HERB): Assessment of
undergraduate student learning outcomes in food
science. Journal of Food Science Education 15: 56-62.
Houston, T. 2005. Outcomes Assessment for Beginning
and Intermediate Spanish: One Program's Process and
Results. Foreign Language Annals 38: 366.
Jenner, B., U. Flick, E. von Kardoff and I. Steinke. 2004. A
companion to qualitative research. Newbury Park, Ca:
Sage.
Joyner-Melito, H.S. 2016. Curriculum Mapping: A Method to
Assess and Refine Undergraduate Degree Programs.
Journal of Food Science Education 15: 83.
NACTA Journal • Nov 2019 - Oct 2020
Keating, S.B. 2014. Curriculum development and evaluation
in nursing. Springer New York.
Killen, R. 2000. Outcomes-based education: Principles and
possibilities. Unpublished manuscript. Newcastle, UK:
University of Newcastle, Faculty of Education.
Knight, P.T. 2001. Complexity and curriculum: a process
approach to curriculum-making. Teaching in Higher
Education 6: 369-381.
Kopera-Frye, K., J. Mahaffy and G.M. Svare. 2008. 2.
The Map to Curriculum Alignment and Improvement.
Collected Essays on Learning and Teaching 1: 8-14.
Koppang, A. 2004. Curriculum mapping: Building
collaboration and communication. Intervention in
School and Clinic 39: 154-161.
Krathwohl, D.R. 2002. A Revision of Bloom's Taxonomy: An
Overview. Theory into Practice 41: 212.
Leech, N.L. and A.J. Onwuegbuzie. 2011. Beyond constant
comparison qualitative data analysis: Using NVivo.
School Psychology Quarterly 26: 70.
Mathews, T.J. and C.M. Hansen. 2004. Ongoing Assessment
of a University Foreign Language Program. Foreign
Language Annals 37: 630.
McGourty, J. 1999. Four strategies to integrate assessment
into the engineering educational environment. Journal
of Engineering Education 88: 391-395.
McGourty, J., C. Sebastian and W. Swart. 1998. Developing
a comprehensive assessment program for engineering
education. Journal of Engineering Education 87: 355-
361.
Metzler, E., G. Rehrey, L. Kurz and J. Middendorf. 2017.
The Aspirational Curriculum Map: A Diagnostic Model
for Action-Oriented Program Review. To Improve the
Academy 36: 156-167.
Millar, V. 2016. Interdisciplinary curriculum reform in the
changing university. Teaching in Higher Education 21:
471-483.
Mitchell, D.E. and W.G. Spady. 1978. Organizational
contexts for implementing outcome based education.
Educational Researcher 7: 9-17.
Moher, D., A. Liberati, J. Tetzlaff and D.G. Altman. 2009.
Preferred reporting items for systematic reviews and
meta-analyses: the PRISMA statement. Annals of
internal medicine 151: 264-269.
Morcke, A.M., T. Dornan and B. Eika. 2013. Outcome
(competency) based education: an exploration of its
origins, theoretical basis, and empirical evidence.
253
 Advances in Health Sciences Education 18: 851-863.
Myers, S.C., M.A. Nelson and R.W. Stratton. 2009. Assessing
an Economics Programme: Hansen Proficiencies,
ePortfolio, and Undergraduate Research. International
Review of Economics Education 8: 87-105.
learning outcomes and assessment. Eur. J. Dent. Educ.
17: e173-e180.
Tyler, R.W. 1949. Tyler, Ralph W., Basic Principles of
Curriculum and Instruction. Chicago, IL: The University
of Chicago Press.

Olsson, P., M.-L. Moore, F.R. Westley and D.D.P. McCarthy.
2017. The concept of the Anthropocene as a game-
changer a new context for social innovation and
transformations to sustainability. Ecology and Society
22:31.
Wright, M.C., M. Goldwasser, W. Jacobson and C. Dakes.
2017. Assessment from an educational development
perspective. To Improve the Academy 36: 39-49.

Plaza, C.M., J.R. Draugalis, M.K. Slack, G.H. Skrepnek

and K.A. Sauer. 2007. Curriculum mapping in program
assessment and evaluation. American Journal of
Pharmaceutical Education 71: 1-8.

Schardt, C., M.B. Adams, T. Owens, S. Keitz and P. Fontelo.

2007. Utilization of the PICO framework to improve
searching PubMed for clinical questions. BMC medical
informatics and decision making 7: 1-6.

Slavich, G.M. and P.G. Zimbardo. 2012. Transformational

teaching: Theoretical underpinnings, basic principles,
and core methods. Educational psychology review 24:
569-608.

Spady, W.G. 1994. Outcome-Based Education: Critical

Issues and Answers. Airlington, VA: American
Association of School Administrators.

Steffen, W., K. Richardson, J. Rockström, S.E. Cornell, I.

Fetzer, E.M. Bennett, R. Biggs, S.R. Carpenter, W. De
Vries and C.A. De Wit. 2015. Planetary boundaries:
Guiding human development on a changing planet.
Science 347: 736.

Stenhouse, L. 1985. Research as a basis for teaching:

readings from the work of Lawrence Stenhouse.
Portsmouth, NH : Heinemann Educational Books.

Swarat, S., P.H. Oliver, L. Tran, J.G. Childers, B. Tiwari and

J. Babcock. 2017. How Disciplinary Differences Shape
Student Learning Outcome Assessment. AERA Open
3: 1-12.

Tam, M. 2014. Outcomes-based approach to quality

assessment and curriculum improvement in higher
education. Quality Assurance in Education 22: 158-168.

Thompson, K.R. and D.J. Koys. 2010. The management

curriculum and assessment journey: Use of Baldrige
criteria and the occupational network database. Journal
of Leadership & Organizational Studies 17: 156-166.

Tilman, D. and M. Clark. 2014. Global diets link environmental

sustainability and human health. Nature 515: 518.

Tonni, I. and R. Oliver. 2013. A Delphi approach to define

254 NACTA Journal • Nov 2019 - Oct 2020
View publication stats