Professional Documents
Culture Documents
Abstract – The Canadian Engineering Accreditation curriculum and related processes, and finally managing and
Board (CEAB) requires engineering programs to assess implementing change [2]. In fact, as part of the consent
twelve high-level educational graduate attributes. For renewal process for engineering degrees, colleges should
evidence of teaching and learning of these engineering regularly show evidence that programs are meeting the
attributes, we developed a framework founded on academic standards, and that students are achieving the
pedagogical practices from backwards design, program and course level outcomes.
constructive alignment, outcome-based education, and Currently, for the consent renewal process, analysis is a
student-centred learning. This quantifiable framework manual endeavour that is mostly qualitative in nature. It
offers a rigorous and tunable approach to systematically entails collecting sample work, assignments, and exam
aggregate performance results over a specified duration. papers from students, reviewing documents, and sharing
Alongside evaluation tools to measure graduate attributes, anecdotes. It may also include focus group discussions,
we created a comprehensive common skills rubric to questionnaires, and surveys administered by the respective
evaluate performance-based learning activities. Overall, provincial Ministry for post-secondary education. For data-
the proposed framework offers an accurate and informed decision making, quantitative process is surely
sustainable solution for the assessment of CEAB graduate more appealing. A systematic and more sustainable
attributes used for Bachelor of Engineering programs. approach to measure and explicitly reveal data points of the
With high precision, results from this assessment process graduate attributes is therefore needed.
can shed light on a set of factors that impact students’ In this paper, we develop a framework to aggregate
performance in achieving outcomes of a particular results incoming from diverse data sources to derive a
engineering program. The results can also build a substantiated conclusion on the assessment of CEAB
foundation that triggers continual improvement in graduate attributes. Empowered by pedagogical practices
teaching and learning courses in engineering programs. of backwards design, constructive alignment, outcome-
based education, and student-centred learning, a
Keywords: Engineering Accreditation, Outcome-Based comprehensive common skills rubric is developed. The
Metric, Graduate Attributes, Common Skills Rubric, rubric is used to evaluate performance-based activities
Engineering Education. such as projects, presentations, work-integrated learnings,
and laboratory assignments using defined indicators.
1. INTRODUCTION Educators and subject matter experts in the field
determined where there were relevant indicators of the
The Canadian Engineering Accreditation Board graduate attributes in each course within the engineering
(CEAB) requires programs to evaluate twelve graduate program. Using formal evaluation tools and the common
attributes (the list is shown in Fig. 2). The accreditation skills rubric, the indicators measure the related graduate
board demands that programs assess graduate attributes at attributes across the span of the program. The significance
different levels across the curriculum, namely, as either of each evaluation based on the course evaluation plan and
introduced, developed, or applied/advanced [1]. To further the design of various assessment tools is quantified. A
expand on the process, the Engineering Graduate discussion of the mechanics of implementing the proposed
Attribution Development (EGAD) project suggested a six- framework and algorithm to assess graduate attributes is
step approach for mapping indicators to the curriculum, provided.
i.e., defining program objectives and indicators, mapping Remarks on how this pedagogical quantification
indicators across the curriculum, collecting data, approach can be used as an effective tool to identify
analyzing, and interpreting the data, improving the teaching and learning gaps is also presented in this work.
Indicator GA#
Graduate Attribute
6. Aggregating the Assessment
Results to evaluate Graduate
Attributes
Indicator GA#
assessment tools, and application of them to evaluate to five courses. In the Bachelor of Engineering (Electrical
students' learning activities. Engineering) program, each graduate attribute is evaluated
Ultimately, the assessment results of indicators are through three to five indicators, represented by variable
aggregated to measure the graduate attributes. The 𝐺𝐺𝐴𝐴𝑖𝑖 , where 1 ≤ 𝑖𝑖 ≤ 𝑁𝑁, such that 𝑁𝑁 represents the number
assessment results of graduate attributes are then collected of indicators that are defined for a specific GA. These
throughout the program to create a repository of the indicators are assessed using various evaluation tools
outcomes. This repository serves as evidence of the including exams, tests, and performance-based evaluations
effectiveness and cohesion of the program curriculum. This such as projects, laboratory assignments, and case studies.
procedure is an essential step for obtaining accreditation The use of the common skills rubric ensures a high
from CEAB. degree of uniformity in the evaluation of performance-
based evaluations. For our assessment process, the results
4. FRAMEWORK FOR ASSESSING GRADUATE are distributed across four different levels of learning
ATTRIBUTES denoted by 𝐿𝐿 = 1, 2, 3, or 4. These learning levels are
defined for the grade ranges shown in Table 1.
The evaluation of graduate attributes is performed using
a variety of assessment tools and in the context of learning
activities. Subject matter experts analyze the content of Table 1: Levels of learning.
each course to identify the most relevant graduate
Level Grade Range
attributes. For this process a repository is created that
𝐿𝐿1 49% and below
contains all curriculum mappings. It also ensures that all
graduate attributes are evaluated adequately throughout the 𝐿𝐿2 50% to 64%
program. 𝐿𝐿3 65% to 79%
The process of assessing graduate attributes is described 𝐿𝐿4 80% and above
in Fig. 2. As noted earlier, graduate attributes are measured
through the indicators at different levels of learning, either The assessment tools used to evaluate the indicators are
at an introductory, developed, or applied/advanced. To not equally significant. Their weights in the assessment
determine the level of learning, we developed a rubric that process should carefully be determined. The contribution
examines learning outcomes of courses, level of of evaluation tools to students’ final grades are clearly
scaffolding, and year and semester in which the course is defined in the evaluation plan of each course. When
offered. The rubric provides consistency in allocating the evaluations assess multiple indicators, their weights are
levels of learning to each course. adjusted proportionally. For instance, if a final exam
In our metric, each graduate attribute is represented by contributes 40% to the final grade of a course and only 25%
variable GA. This variable is generally assessed over three of the exam questions evaluate a specific indicator, the
weight of evaluation for this indicator and through the final recommended for each course. For exams and theoretical
exam should be adjusted to (0.4 × 0.25 × 100) = 10%. In tests, each question is mapped to a course learning
other words, the distribution of the assessment results for outcome. In the Bachelor of Engineering (Electrical
that specific indicator will contribute to the overall Engineering) program, the learning outcomes in each
assessment of the graduate attribute with the weight of course have been mapped to indicators. This means that the
10%. We should also remark that the indicators are mark that students achieve for each question represents
assessed at the question level in theoretical exams and their performance in the associated indicator.
students’ grade distribution in the exam is not necessarily On the other hand, for performance-based evaluations
equivalent to the assessment results of the associated such as projects, case studies, and laboratory assignments,
indicators. We begin aggregating the results when we have including non-traditional assessment tools such as peer-
obtained the assessment results for various indicators of the evaluations or surveys, the common skills rubric is utilized
graduate attribute 𝐺𝐺𝐺𝐺 and determined the weight or to assess the indicators directly. The common skills rubric
significance of the assessment tools for each result. Each is also recommended for assessment of the complementary
indicator such as 𝐺𝐺𝐴𝐴𝑖𝑖 can be assessed by taking the skills such as communication, teamwork, and leadership.
weighted average of all 𝑀𝑀 assessment results. It should be When we unpack the significant skills that emerge from
noted that the value of M may be considerable since the the graduate attribute indicators, we can develop a common
indicator could potentially be assessed several times across skills rubric targeting these discrete skills. This tool
multiple courses and through various tools. The overall enables proper consistency in evaluating skills scaffolded
assessment of the indicator and in the level of learning 𝐿𝐿 across the curriculum from course-to-course and from
can be calculated as follows: year-to-year. This comprehensive rubric provides a
reference for the best academic practices whereby we can
∑𝑀𝑀
𝑗𝑗=1 𝑤𝑤𝑖𝑖,𝑗𝑗 𝑛𝑛𝑗𝑗 (𝐿𝐿) parse the criteria for various learning activities.
𝐺𝐺𝐴𝐴𝑖𝑖 (𝐿𝐿) = (1) The student-centred language of the rubric provides
∑𝑀𝑀
𝑗𝑗=1 𝑤𝑤𝑖𝑖,𝑗𝑗 students with constructive and actionable feedback that
leads them toward academic success. The rubric is
Where 𝐿𝐿 could be either 1, 2, 3 or 4, and 1 ≤ 𝑖𝑖 ≤ 𝑁𝑁. In this designed to evaluate the indicators throughout the program
equation, the variable 𝑛𝑛𝑗𝑗 (𝐿𝐿) represents the number of and can be modified according to requirements of each
grades in level 𝐿𝐿 assessed through the 𝑗𝑗𝑡𝑡ℎ assessment and learning activity. Instructors could modify the rubric by
𝑤𝑤𝑖𝑖,𝑗𝑗 is the weight of the 𝑖𝑖 𝑡𝑡ℎ indicator for this assessment. removing the indicators that are not relevant and assigning
Despite being defined at different pedagogical levels, all numerical grades to each section of evaluations. The rubric
indicators make an equal contribution to the assessment of
the associated graduate attribute. The indicators cover
various aspects and qualities defined by the graduate Input
attributes. Therefore, the overall assessment of a graduate Course Evaluation Tools
attribute can be calculated as the average of all its
indicators with equal weights.
Theoretical Tests Performance-based
and Exams Type of Projects and Labs
𝑁𝑁 Evaluations
1 (2) ?
𝐺𝐺𝐺𝐺(𝐿𝐿) = � 𝐺𝐺𝐴𝐴𝑖𝑖 (𝐿𝐿)
𝑁𝑁
𝑖𝑖=1 Customize the
Common-skills Rubric
Assign Grade of Each
where 𝐿𝐿 is either 1, 2, 3 or 4, and 𝑁𝑁 represents the number
Question to the
of indicators that are defined for the graduate attribute 𝐺𝐺𝐺𝐺. Associated Learning Assess the Indicators
This method can be easily adjusted to evaluate different Outcome or Indicator using the Customized
levels of content, introductory, developed, or applied/ Rubric
advanced. This can be performed by calculating the
weighted average, as described in (1), on the assessment
results that are specific to a particular level of content.
Output
5. ASSESSING THE INDICATORS THROUGH Assessment Results for all Indicators
LEARNING ACTIVITIES Assigned to a Course
could evaluate different criteria of learning activity and describes the different levels of learning, L1 to L4, for the
assess the indicators quantitively. This process is captured design indicators, DE.1 to DE.5. The descriptors are
by the flowchart of Fig. 3, which describes a developed using a student-centred language and they
straightforward method for assessing indicators through provide students with constructive feedback. To help
various course evaluations. readers gain a better understanding of the implementation
of our assessment methodology, we show in Appendix A
6. IMPLEMENTING THE GRADUATE ATTRIBUTE an excerpt of the common-skills rubric. In fact, the excerpt
ASSESSMENT PROCESS of the rubric refers exclusively to the graduate attribute of
design.
To further elaborate the process, we discuss the
assessment of the design graduate attribute in the sixth
100 Embedded App. Dev.
semester of the Bachelor of Engineering (Electrical
(ELEE-30893D)
Engineering) program at Sheridan College. This graduate 90 DE1 DE2 DE3 DE4 DE5
Percentage (%)
60
(ELEE31186D), and Design of Digital Systems
(ELEE31436D). In the evaluation plan of each course, we 50 DE4 DE4 DE4
DE4 DE4 DE4
explicitly connect the assessment tools (e.g., project, 40 Control Systems
DE4
midterm test, and final exam) to possible learning 30
(ELEE-31186D)
DE3 DE3 DE3
outcomes. With this first mapping, we were able to build a DE4
20
comprehensive evaluation matrix for each course. DE3 DE3 DE3
DE3 DE4
As a second mapping, we associated the learning 10
DE3
DE4 DE1 DE1 DE1
Level 4 (80 - 100%) Level 3 (65 - 79%) Level 2 (50 - 64%) Level 1 (below 49%)
DE. 1 Define design • Processes the design Analyzes the problem Explains the design Needs to identify the
requirements, assignments as an and provides an requirements. Needs design requirements
specifications, and open-ended problem accurate list of design to address the and constraints.
constraints. with possible solutions. requirements. constraints.
• Connects the solution
and constraints.
DE. 2 Consider external • Considers all external • Considers most of • Considers some • Needs to consider the
factors including factors including external factors external factors effect of external
environmental, environmental, social, including including factors in the
social, economic economic impacts, environmental, social, environmental, social, engineering design.
impacts as well as health, and safety in economic impacts, economic impacts, • Needs to evaluate the
health and safety the engineering health, and safety in health, and safety in importance of each
in an engineering design. the engineering the engineering factor and clearly
design. • Prioritizes the design. design. prioritizes them.
contributing factors. • Lists the contributing
factors.
DE. 3 Generate Generates multiple Presents some design Needs to present a Needs to study the
divergent solutions design alternatives that solutions that address complete design design problem and
Design
to a design address the design some of key design solution. identify multiple
problem. problem. requirements. solutions considering
the constraints.
DE. 4 Develop a refined • Evaluates several Develops one solution Develops one solution Needs to consider the
design to design alternatives. that meets the design that meets some of the design requirements
implement an Chooses the solution requirements and design requirement. and constraints.
engineering that meets the design constraints.
project requirements and
constraints.
• Modifies the chosen
solution following a
systematic approach.
DE. 5 Evaluate the • Evaluates the Validates the design by • Assesses the Needs to evaluate the
performance of an performance of the comparing the performance of the design solution.
engineering design accurately. performance of the solution with general
design • Addresses the solution with the statements.
practicality of the design requirements. • Needs to build the
solutions. Connects statement based on
the discrepancies to the measurements or
the design constraints. analysis results.