Medical Teacher

ISSN: 0142-159X (Print) 1466-187X (Online) Journal homepage: http://www.tandfonline.com/loi/imte20

Comparability of outcome frameworks in

medical education: Implications for framework
development

Stefanie C. Hautz, Wolf E. Hautz, Markus A. Feufel & Claudia D. Spies

To cite this article: Stefanie C. Hautz, Wolf E. Hautz, Markus A. Feufel & Claudia D. Spies (2015):
Comparability of outcome frameworks in medical education: Implications for framework
development, Medical Teacher

To link to this article: http://dx.doi.org/10.3109/0142159X.2015.1012490

Published online: 20 Feb 2015.

Submit your article to this journal

Article views: 154

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=imte20

Download by: [University of California Santa Barbara] Date: 05 November 2015, At: 21:56
 2015, 1–9, Early Online

Comparability of outcome frameworks

in medical education: Implications for
framework development
STEFANIE C. HAUTZ1, WOLF E. HAUTZ1, MARKUS A. FEUFEL1,2 & CLAUDIA D. SPIES1
1
Charité-Universitätsmedizin Berlin, Germany, 2Max Planck Institute for Human Development, Germany

Abstract
Background: Given the increasing mobility of medical students and practitioners, there is a growing need for harmonization of
Downloaded by [University of California Santa Barbara] at 21:56 05 November 2015

medical education and qualifications. Although several initiatives have sought to compare national outcome frameworks, this task
has proven a challenge. Drawing on an analysis of existing outcome frameworks, we identify factors that hinder comparability and
suggest ways of facilitating comparability during framework development and revisions.
Methods: We searched MedLine, EmBase and the Internet for outcome frameworks in medical education published by national or
governmental organizations. We analyzed these frameworks for differences and similarities that influence comparability.
Results: Of 1816 search results, 13 outcome frameworks met our inclusion criteria. These frameworks differ in five core features:
history and origins, formal structure, medical education system, target audience and key terms. Many frameworks reference other
frameworks without acknowledging these differences. Importantly, the level of detail of the outcomes specified differs both within
and between frameworks.
Conclusion: The differences identified explain some of the challenges involved in comparing outcome frameworks and medical
qualifications. We propose a two-level model distinguishing between ‘‘core’’ competencies and culture-specific ‘‘secondary’’
competencies. This approach could strike a balance between local specifics and cross-national comparability of outcome
frameworks and medical education.

Introduction Practice points

With the increasing mobility of medical students and practi-
tioners (WFME/AMSE International Task Force 2007), there is a
growing need for international harmonization and quality
assurance of medical education to guarantee quality care. For
420 years, and especially since the first CanMEDS publications
in 2000 (Frank et al. 2005), many national associations and
governmental bodies have engaged in developing outcome
frameworks for medical education. Their aim is to foster
quality of medical education by specifying exactly what
medical students need to learn. These national standards
inform curriculum development, assessment, and/or licensing
procedures.
To the outside world, training in no other profession seems
to be as well defined as that of doctors (Wijnen-Meijer et al.
2013). But while the demand for clearly defined standards has
spread, the core of medical knowledge has become less clear
than it used to be (Ten Cate 2014). Furthermore, although
‘‘medical knowledge and research have traditionally crossed
national boundaries’’ (Core Committee of Institute for
International Medical Education 2002), a recent international
comparison noted that models of medical education differ in
terms of their stages of education (undergraduate versus
postgraduate training), time of licensing, degrees awarded,
 A comparison of international outcome frameworks in
medical education provides insights into communal-
ities and differences.
 Outcome frameworks differ in terms of history and
origins, formal structure, medical education system,
target audience and key terms.
 These factors should be considered when comparing,
revising, transferring or merging frameworks.
 A two-level model for outcome frameworks distin-
guishing between ‘‘core’’ competencies and culture-
specific ‘‘secondary’’ competencies could strike a
balance between local specifics and cross-national
comparability.
options for continuing medical education, and underlying
educational theories and terminology (Wijnen-Meijer et al.
2013).
Because such differences hinder the comparability and,
ultimately, quality of education, there have been attempts to
unify educational systems. For instance, in 1999, the Bologna
Agreement (European Higher Education Area 2014) began
to harmonize higher education across 49 countries by

Correspondence: Stefanie C. Hautz, Charité, Prodekanat für Studium und Lehre, Universitätsmedizin Berlin, Charitéplatz 1, 10117 Berlin, Germany.
Tel: +49 0 30 450 576 149; Fax: +49 0 30 450 7 576 149; E-mail: stefanie.hautz@charite.de
ISSN 0142-159X print/ISSN 1466-187X online/15/000001–9 ß 2015 Informa UK Ltd. 1
DOI: 10.3109/0142159X.2015.1012490
 S. C. Hautz et al.
standardizing the formal structure of study programs (e.g.
modules and credit points systems) and final degrees
(Bachelor’s and Master’s). However, unifying structures alone
does not suffice to guarantee quality education.
Against this background, the Tuning Project, another EU
initiative, was initiated in 2000 to promote harmonization of
higher education by aligning the contents (learning outcomes/
competencies) of degree programs across European countries.
Although the Project Committee in Medicine managed to agree
on a European outcome framework for primary medical
degree qualifications (Cumming & Ross 2007), the Committee
members could not agree on some aspects, such as the
definition of the doctor’s role as scholar, or the respective
competencies required. Instead, the Committee ‘‘leave[s] it
open to individual countries, schools or students to decide
Downloaded by [University of California Santa Barbara] at 21:56 05 November 2015
how to prioritize practical research experience, in keeping
with their profile, educational philosophy or career intentions’’
(Cumming & Ross 2008).
Apart from comparability problems related to educational
structure and content, more general phenomena in medical
education also hinder harmonization. For instance, researchers
have acknowledged the need for a common taxonomy of
technical terms (Eva & Regehr 2008). Although glossaries of
terms in medical education have been compiled, such as the
MedEdWorld Glossary (MedEdWorld 2014) or the Glossary of
the Institute for International Medical Education (Wojtczak
2012), consistent definitions of basic terms, such as outcome or
competence1, are still lacking. Currently, the same terms are
often used to refer to different things and different terms are
used interchangeably, making it difficult to harmonize and
implement educational practice and research.
A related problem was raised by Ringsted and colleagues,
who described the problems of transferring the Canadian
CanMEDS framework to Denmark. The authors concluded
that an existing framework cannot just be transferred from
one cultural background to another, but needs to be adap-
ted to the local culture of medical specialists (Ringsted et al.
2006).
In this article, we examine publicly available national
outcome frameworks to investigate in detail whether and to
what extent outcome frameworks are comparable. In so doing,
we aim to foster curriculum development, revision of outcome
frameworks, and attempts at harmonization such as the Tuning
Project. Specifically, we address two questions: (1) which
factors influence the comparability of outcome frameworks?
and (2) given those factors, what can be done to improve
comparability across countries?
Methods
We employed a four-step approach to identify and analyze
relevant outcome frameworks, consisting of search for frame-
works, in- and exclusion, data extraction and analysis, and
search for additional background information.
(1) Search for frameworks: In June 2013, a biomedical
librarian and one of the authors (S.H.) searched electronic
databases (PubMed and EmBase) and Google for out-
come frameworks in medical education published in
2
German or English since January 1980, using permuta-
tions of the keywords medical education, outcome
framework, learning objective, government * (*used as
wildcard to include government, governmental, etc) and
curriculum (Figure 1). Four separate Google searches
with those keywords were run to identify frameworks not
indexed in PubMed or EmBase. In each search, all results
from the electronic databases were retrieved (for Google,
the first 100 results). We also searched the authors’
bookshelves as well as the references of the retrieved
frameworks that met the inclusion criteria to identify
further relevant frameworks.
(2) In- and exclusion: We included all outcome frameworks
at undergraduate or postgraduate level that were based
on a national consensus process and endorsed or
published by a national society or governmental body.
We excluded all outcome frameworks from disciplines
other than medicine (e.g. nursing or physiotherapy) or
from medical sub disciplines (e.g. continuing education in
anesthesia), as well as inter-professional frameworks. We
did not restrict our search to specific countries, but we did
limit our results by including only outcome frameworks
published in English or German. Based on these criteria,
inter-rater agreement (S.H. and W.H.) for the identifica-
tion of relevant outcome frameworks was 97.68%. We
then eliminated doubles from the set of frameworks
identified (100% agreement).
(3) Data extraction and analysis: To analyze and compare the
outcome frameworks, we developed a data extraction
sheet (data sheet) covering five content categories: history
and origins, target audience (undergraduate versus post-
graduate), formal structure (e.g. domains or roles),
medical education system (e.g. duration of undergraduate
medical studies in the respective country), and key terms
(e.g. outcomes or competencies) – factors that had been
identified as important in a previous attempt to identify
differences and similarities between frameworks. Two
of us (W.H. and S.H.) independently extracted content
relating to each of these five categories from each
outcome framework with a 95% agreement. We also
noted the source of publication (e.g. website) and the
date of search. All extracted data are available upon
request.
(4) Search for additional background information: Not all of
the information recorded in the data sheet was available
directly from the publications themselves. Thus, in some
cases we sought additional background information (e.g.
to fully understand the framework’s history and origins or
to find background information on the medical education
system in question). To this end, two of us (S.H. and A.R.)
conducted a Google search aiming to fill in the gaps in the
data sheet.
Results
Our search yielded 1816 results. Of these, 13 met our inclusion
criteria and were further analyzed (Table 1 and Figure 1). For
the sake of simplicity, we refer to each framework by its
 Comparability of outcome frameworks

1812 search results Addional records included

Idenficaon
(Titles/Abstracts) idenfied through from other sources:
database search:
• authors’ bookshelves (n = 2)
• Google* (n = 400) • from references (n = 2)
• PubMed** (n = 712)

Records screened Records excluded because

(n = 1816) they did not meet the
Screening

inclusion criteria (n = 1772)

Records aer exclusion Records excluded for being

(n = 44) doubles (n = 27)
Downloaded by [University of California Santa Barbara] at 21:56 05 November 2015

Full-text arcles assessed for Full-text arcles excluded

Eligibility

eligibility because
(n = 17)
• not published by
governmental body
(n = 2)
• not a medical

Studies included in qualitave

Included

synthesis
(n = 13)

Search terms:
* outcome framework medical educaon; outcome framework medical educaon government*;
learning objecves medical educaon; learning objecves medical educaon government*
** outcome framework AND medical educaon; outcome framework AND medical educaon AND
government (+/– *); learning objecves AND medical educaon; learning objecves AND medical
educaon AND government (+/– *)
*** PubMed search adjusted to EmBase syntax

Figure 1. Prisma flow diagram (Moher et al. 2009).

common abbreviation (listed in Table 1). The following results
are structured by the five content categories considered for
each framework.
History and origins
The development of many of the frameworks was initiated by
medical education councils and/or governmental organiza-
tions. For instance, one framework states: ‘‘Australian
Government Department of Health and Ageing provided
funding to Medical Deans Australia and New Zealand Inc
(Medical Deans) to document competencies for the medical
graduate outcomes required by the Australian Medical Council
(AMC). The project evolved from priorities identified from the
National Clinical Training Review prepared by Medical Deans
for the Medical Training Review Panel in 2008’’ (Carmichael &
Hourn 2011).
Importantly, many of the outcome frameworks reference
each other or are based on frameworks that are either not
publicly available (e.g. ACGME, see below) or did not meet
our inclusion criteria. Figure 2 shows the network of cross-
references across the frameworks analyzed. For example, the
arrow from Tomorrow’s Doctors to Report I indicates that
Report I references Tomorrow’s Doctors. Figure 3 provides a
timeline of the publication dates of the outcome frameworks,
including prior versions. Whereas some outcome frameworks
reference at least one other framework (e.g. Medical Deans,
Tuning, Scottish Doctors and SCLO), others do not (CanMEDS,
Tomorrow’s Doctors and Visions). The two most cited
outcome frameworks are CanMEDS (cited by, e.g. Junior
Doctors, Medical Deans and Tuning), and Tomorrow’s Doctors
(cited by, e.g. Scottish Doctors, Report I and Blueprint); see
Figure 2.
3
 Downloaded by [University of California Santa Barbara] at 21:56 05 November 2015

4
Table 1. Outcome frameworks included in the analysis.

Number of sections
S. C. Hautz et al.

Target Key term(s) used to (chapter, role Number of items

Country Name1 Source audience2 describe learning targets and domain) (approx.)3
Australia and New Zealand Accreditation Standard for AMC (2013) UG Outcomes 4 40
Primary Medical Education
Providers and their Program of
Study and Graduate Outcome
Statements
Australia and New Zealand Developing a Framework of Carmichael & Hourn (2011) UG Combination of Attributes, – 185
Competencies for Medical Learning Outcomes and
Graduate Outcomes (Medical Competencies
Deans)
Europe The Tuning Project: Learning Cumming & Ross (2008) UG Combination of Learning 12 90 (on level 2)
Outcomes/Competences for Outcomes and
Undergraduate Medical Competences
Education in Europe
The Netherlands Blueprint 2001: Training of doctors Metz et al. (2001) UG General objectives 4 525
in The Netherlands
Scotland The Scottish Doctors: Learning Scottish Deans’ Medical Education UG Outcomes 12 290
Outcomes for the Medical Group (2008)
Undergraduate in Scotland: A
Foundation for Competent and
Reflective Practitioners
Switzerland Swiss Catalogue of Learning Working Group under a Mandate of UG General objectives 7 2000
Objectives for Undergraduate the Joint Commission of the
Medical Training (SCLO) Swiss Medical Schools (2008)
UK Tomorrow’s Doctors: Outcomes General Medical Council (2009) UG Outcomes 3 190
and Standards for
Undergraduate Medical
Education
USA Report I: Learning Objectives for MSOP Report Writing Group (1998) UG Learning objectives 4 30
Medical Student Education,
Guidelines for Medical Schools
Australia Australian Curriculum Framework ACF Revision Working Group (2009) PG Outcomes 6 240
for Junior Doctors
UK Good Medical Practice (GMP) General Medical Council (2013) PG – 4 110
USA Guide to Good Medical Practice National Alliance for 39 Physician PG Competencies 6 240
(Guide to GMP) Competence (2009)
Canada CanMEDS Framework Frank et al. (2005) UG/PG (since the Competencies 7 130
2005 version)
India Visions 2015 Medical Council of India (2011) UG/PG Competencies 5 35

UG, undergraduate; PG, post graduate; Items, outcomes/competencies.

1
Abbreviations shown in bold.
2
UG ¼ undergraduate, PG ¼ post graduate.
3
Items ¼ outcomes, competencies, etc.
 Comparability of outcome frameworks
Downloaded by [University of California Santa Barbara] at 21:56 05 November 2015

Figure 2. The network chart of cross-referencing of the analyzed outcome frameworks (for a better clarity, chart shows only the
references among the analyzed outcome frameworks. 1Version 1993 and 2009; in version 2009 Tomorrow’s Doctors refers to
Tuning and Scottish Doctors as ‘‘related documents’’; 2Version 1996 and 2005; 3Version 2001 and 2008; 4Version 2006 and 2013;
5
Version 2002 and 2008 and 6Version 2001 and 2009).

Figure 3. Publication dates of the outcome frameworks analyzed (in bold) and their prior version.

Target audience Accreditation Standard, Tomorrow’s Doctors and Visions), and

The target audiences of the outcome frameworks differ. Most
frameworks (8 out of 13) target undergraduate students (UG);
three, postgraduate students (PG), and two, both UG and PG
students (Table 1). Interestingly, some frameworks refer to
other frameworks that have clearly distinct target audiences.
For instance, Medical Deans, an UG framework, refers to
Junior Doctors, a PG framework. We could not tell whether
and how these differences in target population were
accounted for during framework development.
Formal structure
The formal structure of outcome frameworks also varies. Three
frameworks structure their content by chapters called domains
(GMP, Guide to GMP and Scottish Doctors). Five use roles to
cluster global educational requirements (CanMEDS, SCLO,
five use themes, areas or other categories to structure
outcomes or competencies (Blueprint, Junior Doctors,
Medical Deans, Tuning and Report I).
The number of these sections ranges from 3 (Tomorrow’s
Doctors) to 12 (Tuning, Scottish Doctors) (Table 1). Table 2
provides a detailed overview of the formal structure of each
framework analyzed. The marked differences in structuring
elements could imply very different levels of detail between
outcome frameworks. Indeed, the level of detail of the
outcomes or competencies specified varies considerably. For
instance, outcomes/competencies for the Clinical Application
aspect of the scholar role are detailed and related to practice
(e.g. ‘‘Applies the concept of specificity, sensitivity, pre- and
post-test probability to the interpretation of common diagnos-
tic procedures’’ (Carmichael & Hourn 2011), whereas most
content assigned to the aspect of Teaching and Evaluation is
5
 S. C. Hautz et al.

Table 2. Formal structure of the content of the outcome frameworks.

Structure Outcome framework

Domains: Knowledge, skills and performance; safety and quality; communication, partnership and teamwork; GMP
maintaining trust
Domains: Patient care, medical knowledge and skills, practice-based learning and improvement, interpersonal and Guide to GMP
communication skills, professional behavior, systems-based practice
Domains: Clinical skills; practical procedures; patient investigation; patient management; communication; health Scottish Doctors
promotion and disease prevention; medical informatics; basic, social and clinical sciences and underlying principles;
attitudes, ethical understanding and legal responsibilities; decision making skills and clinical reasoning and judgment;
the role of the doctor within the health service; professional development
Roles: Medical expert, communicator, collaborator, health advocate, manager, professional, scholar CanMEDS
Roles: Medical expert, communicator, collaborator, professional, scholar, health advocate, manager SCLO
Roles: Scientist and scholar, practitioner, health advocate, professional, leader Accreditation Standard
Roles: Scholar and scientist, practitioner, professional Tomorrow’s Doctors
Roles: Clinician who understands and provides preventive, promotive, curative, palliative and holistic care with Visions
Downloaded by [University of California Santa Barbara] at 21:56 05 November 2015

compassion; Leader and member of the health care team and system with capabilities to collect analyze and
synthesize health data; Communicator with patients, families, colleagues and community; Lifelong learner committed
to continuous improvement of skills and knowledge; Professional, who is committed to excellence, is ethical,
responsive and accountable to patients, community and profession.
Themes: Medical aspects, personal aspects, scientific aspects, aspects related to the society and the health care Blueprint
system
Areas: Clinical management, communication, professionalism Junior Doctors
Knowledge, skills and attitudes Medical Deans
Detailed list of Level 1 Outcomes/Competences, further subdivided into Level 2 1 Outcomes/Competences Tuning
N/A Report I

Table 3. Medical education systems (by country) (adapted and extended from Wijnen-Meijer et al. 2013).

Postgraduate
Bachelor’s/college Medical school internship3 Residency4 Point at which unrestricted
Country (in years) (in years) (in years) (in years) practice is allowed
Australia1 – 5; Universities of 2 3–5 After residency
New South Wales 6
2
Australia 4 4 2 3–5 After residency
Canada 4 4 – 2–7 After residency
India – 5.5 – 3–6 After medical school
The Netherlands – 6 – 3–6 After medical school
New Zealand ‘‘Health Science’’ 1 5 1–2 6–9 After 1st year of postgraduate internship
Switzerland – 6 – 45 After residency
UK – 5 2 3–6 After 1st year of postgraduate internship
USA 4 4 – 3–5 After 1st year of residency

1
Universities of New South Wales, University of Newcastle, University of New England.
2
University of Melbourne, University of Adelaide, University of Western Australia, University of Sydney, University of Queensland, Flinders University, Griffith University,
Australian National University.
3
Also called ‘‘foundation program’’, ‘‘house officer training’’ or ‘‘pre-vocational training’’.
4
Duration of residency varies depending on specialty chosen: the number of years specified should be seen as an orientation.

less detailed (e.g. ‘‘Recognizing the important role of all
doctors as mentors and teachers’’, Scottish Deans’ Medical
Education Group 2008).
Medical education system
The medical education system is part of the context that
outcome frameworks must respect in order to be effectively
implemented. For instance, the (number of) required learning
outcomes or competencies should differ between frameworks
depending on the duration of studies. As shown in Table 3,
whereas some medical education systems stipulate four years
of undergraduate medical education (e.g. Canada and USA),
6
other require up to six years (e.g. UK, Switzerland and The
Netherlands). In Australia the number of years of undergradu-
ate studies differs by state between four and six.
Similarly, the postgraduate foundation program takes
between one (e.g. New Zealand) and two years (e.g.
Australia and UK). The duration of residency may even vary
within countries, depending on subspecialty, from two to
nine years.
In our sample, the differing durations of educational phases
across countries are not reflected in the number of outcomes
or competencies defined in the respective outcome frame-
works (see Table 1). For medical education systems with four
years of undergraduate studies, the number of outcomes or

competencies varies from 30 (Report I) to up to 185 (Medical
Deans). In systems with up to six years of undergraduate
education, the number of outcomes or competencies varies
from between 35 (Visions) to up to 2000 (SCLO). In
postgraduate education, the number of outcomes or compe-
tencies per framework varies between 110 (GMP) and 240
(Guide to GMP).
Key terms
Within outcome frameworks, various terms are used to
describe the educational targets, such as competencies,
outcomes, learning objectives, and general objectives. Some
frameworks combine two or three of those terms, usually
without a clear definition or distinction (Table 1). Some
frameworks, such as Medical Deans, provide a glossary
Downloaded by [University of California Santa Barbara] at 21:56 05 November 2015
of terms.
Discussion
The increasing global mobility of medical personnel (WFME/
AMSE International Task Force 2007) has led to a growing
need for harmonization of medical education and for mutual
recognition of medical degrees across borders. Cross-border
recognition of degrees is currently hindered by differences
in medical education systems (Wijnen-Meijer et al. 2013). In
Europe, the Bologna process has sought to harmonize the
length and structure of degree programs (European Higher
Education Area 2014), but currently disregards differences in
the content and context of the curricula leading to such
degrees.
Content
Attempts such as the Tuning Project (Cumming & Ross 2008)
to harmonize the content of several national frameworks at the
international level have encountered difficulties in reaching
consensus, at least in some areas (Cumming & Ross 2008). Our
study’s finding that most frameworks reference other frame-
works suggests that the need to harmonize content across
frameworks is largely recognized by their authors.
Interestingly, some frameworks refer to other frameworks
with very different target audiences. If and how the authors
account for these differences remains an open question.
Furthermore, two originally similar frameworks that reference
each other may develop in different directions. How those
differences become noticed by the authors of the referencing
frameworks remains unclear, which presents another obstacle
to content harmonization.
Our results further suggest that differing levels of detail both
within and between frameworks may hinder harmonization of
content – both the number of outcomes specified and the
number of sections per framework differ considerably,
implying differing levels of detail across frameworks and
sections. Taking account of such differences in framework
structures and the medical education systems for which they
were developed may facilitate comparisons and content
harmonization.
Comparability of outcome frameworks
Context
Ringsted and colleagues drew attention to contextual differ-
ences between frameworks in their account of the problems
of transferring the CanMEDS framework from the Canadian to
the Danish context. The authors concluded that an existing
framework cannot simply be transferred from one cultural
background to another, but needs contextual adaption
(Ringsted et al. 2006). Our results support this conclusion, as
frameworks seem to be specific to the respective medical
education system (as one important contextual factor). The
number of learning outcomes required from graduates obvi-
ously needs to be compatible with the amount of time
dedicated to those outcomes in the curriculum. Note, however,
that Ringsted and colleagues did not consider it necessary to
adapt the ‘‘core’’ structure (e.g. roles) of the CanMEDS
framework in culturally adapting its contents.
Frameworks also use different terminology (as a second
important contextual factor) to describe educational targets
such as outcomes or competencies. Yet ‘‘the language we use
both makes possible and constrains the thoughts we can have’’
(Lingard & Haber 1999). At least for key terms such as
outcomes, this highlights the need for use of a common
taxonomy in outcome frameworks (Eva & Regehr 2008).
Currently, the same terms are used to describe different
objects, and different terms are used interchangeably – as
exemplified by the use of the terms ‘‘outcome’’ and ‘‘compe-
tence’’ in the frameworks analyzed. It has been argued that
outcome-based education (OBE) and competency-based edu-
cation (CBE), the two key educational models encountered
throughout frameworks, are conceptually similar in that
medical education should be guided by predetermined
outcomes (Morcke et al. 2013). Morcke has suggested that
OBE and CBE differ merely in the perspective taken (Morcke
et al. 2013): while CBE describes the competencies that
students should possess after their studies, OBE describes what
institutions teach and assess. Although some frameworks
included in our study at least provide definitions of key
terms (Medical Deans), others largely leave their mean-
ing open to interpretation (which, in turn, depends on the
cultural context).
Approaches to harmonization
Content that depends only in part on a particular culture may
readily be compared across frameworks and medical educa-
tion systems. The physician’s role as scholar, for example, is
predisposed for such comparisons because ‘‘research has
traditionally crossed boundaries’’ (Core Committee of Institute
for International Medical Education 2002) and there is a
growing trend towards greater integration of science and
research in undergraduate medical education (Abu-Zaid &
Alkattan 2013; Sheikh et al. 2013). But even for culturally
depended roles such as the physician as a communicator,
medicine as a globalizing profession requires internationally
agreed standards to guarantee implementation and evaluation
of quality medical education and, ultimately, quality care (Core
Committee of Institute for International Medical Education
2002; WFME/AMSE International Task Force 2007).
7
 S. C. Hautz et al.
Our results suggest that attempts to foster mutual recogni-
tion of medical degrees across borders may be facilitated by a
thorough comparison and possibly harmonization of relevant
global aspects of outcome frameworks, such as their over-
arching structure and definition of key terms and concepts,
together with a clear identification of the target audience.
Researchers aiming to transfer part of a European curriculum
to the Arabic world have previously suggested approaching
the cultural adaption of curricula by defining a group of ‘‘core’’
competencies alongside ‘‘secondary’’ competencies that are
culturally appropriate to each site (Whitford & Hubail 2014).
Based on our results, we propose that this two-level solution be
extended to outcome frameworks, which should be harmo-
nized on a global level (e.g. in terms of common roles, key
terms, level of detail and core competencies), while allowing
Downloaded by [University of California Santa Barbara] at 21:56 05 November 2015
for differences on a local level (e.g. cultural aspects, target
audience, medical education system and ‘‘secondary compe-
tences’’, Whitford & Hubail 2014). This approach would be in
line with the idea that harmonization is achievable without
rigid uniformity (Ross et al. 2014).
Given that most frameworks included in the present
analysis reference CanMEDS, Tomorrow’s Doctors, or both,
these two frameworks may provide a widely accepted
reference standard against which core learning outcomes,
framework structure and key terms can be aligned. Adopting
such an approach may facilitate comparability and transfer-
ability of outcome frameworks in medical education and thus
promote mutual degree recognition.
Limitations
Although we took a systematic approach to the literature
search and in- and exclusion of publications, we used
predefined categories rather than conducting a formal quali-
tative analysis or synthesis of the resulting frameworks.
Our search method identified 13 national outcome frame-
works, but may have missed other frameworks eligible accord-
ing to the inclusion criteria. Most of the frameworks identified
were not directly accessible using classic literature searches and
were only found through web searches. The Google search
algorithm is less transparent than a search of literature databases
that use predefined and publicly available criteria to categorize
or sort articles. The use of web searches thus limits repro-
ducibility. Furthermore, although we did not limit our search
to specific countries, we did limit our results by including
only outcome frameworks published in English or German.
Finally, we did not include the ACGME outcome frame-
work, because it was unavailable at the time of our literature
search and is currently being replaced by the Milestone Project
(Eric et al. 2014).
Conclusion
The questions we asked at the outset can be answered as
follows:
(1) Which factors influence comparability of outcome frame-
works? The five content components that we considered –
history and origins, target audience, formal structure,
8
medical education system, and key terms – appear
relevant to the comparison and harmonization of out-
come frameworks. However, they may not be the only
aspects worthy of consideration. Future research may
identify further factors that limit the comparability or
transferability of outcome frameworks in order to facilitate
the harmonization of future curriculum development.
(2) Given those factors, what can be done to improve
comparability across countries? Outcome frameworks
are not simply transferable to other national contexts,
but need to be customized. Based on our analysis of
existing outcome frameworks, we propose a two-level
solution with harmonization on a global level (e.g. in
terms of common roles, taxonomy, level of detail and core
competencies), and opportunities for case-specific adap-
tions on a local level (e.g. cultural aspects, target
audience, medical education system and ‘‘secondary
competences’’, Whitford & Hubail 2014).
Notes on contributors
STEFANIE C. HAUTZ, is a research scientist, educationalist and teacher by
training with a professional interest in science education and outcome-
based education, working in medical education research at Charité Berlin,
Germany.
WOLF E. HAUTZ, MD, MME, is an Anesthesiologist by training whose
professional interests span clinical reasoning, self-monitoring and scholar-
ship training, at Charité Berlin, Germany.
MARKUS A. FEUFEL, PhD, is a research scientist and assistant to the Vice
Dean of Education at Charité Berlin with interests in decision making
under uncertainty, risk communication and science education.
CLAUDIA D. SPIES, Professor, MD, is Director of the Department of
Anesthesiology at Charité Berlin, Germany, with numerous publications in
curriculum development, quality management and cognitive dysfunction.
Acknowledgments
We thank Claudia Kiessling for revising the manuscript. We
also thank Anne-Katrin Reinsch for her help during data
collection and data extraction.
Declaration of interest: The authors report no declarations
of interest. The authors alone are responsible for the content
and writing of the article. S.H. and W.H. contributed equally.
Berlin Senate Department of Education, Youth and Science in
the context of the Berliner Qualitätsoffensive für die Lehre
2012 bis 2016.
Note
1. There are various definitions for terms as competence and outcome.
To have a common understanding of the terms, we share the
following definitions:
Competence: ‘‘The array of abilities [knowledge, skills, and attitudes
or KSA] across multiple domains or aspects of performance in a
certain context. Statements about competence require descriptive
qualifiers to define the relevant abilities, context and stage of training.
Competence is multi-dimensional and dynamic. It changes with time,
experience and setting.’’ (Frank et al. 2010).
Outcome: ‘‘All possible demonstrable results that stem from casual
factors or activities. In medical education, outcome refers to a new
skill, knowledge or stimulus to improve the quality of patient care.

Setting outcomes can be very useful for developing a framework of
various results expected from various educational activities [. . .].’’
(MedEdWorld 2014).
References
Abu-Zaid A, Alkattan K. 2013. Integration of scientific research training into
undergraduate medical education: A reminder call. Med Educ Online
18:22832.
ACF Revision Working Group. 2009. Australian curriculum framework for
junior doctors. Confederation of Postgraduate Medical Education
Councils (CPMEC). [Accessed 24 October 2013] Available from http://
www.cpmec.org.au/files/Brochure%20final.pdf [Version 2.2].
AMC. Accreditation standards for primary medical education providers and
their program of study and graduate outcome statements. [Accessed
18 October 2013] Available from https://members.aamc.org/eweb/
upload/Learning%20Objectives%20for%20Medical%20Student%20Educ%
Downloaded by [University of California Santa Barbara] at 21:56 05 November 2015
20Report%20I.pdf.
Carmichael A, Hourn M. 2011. Developing a framework of competencies
for medical graduate outcomes. Final report. Medical Deans Australia
and New Zealand Inc. [Accessed 24 October 2013] Available from
http://www.medicaldeans.org.au/wp-content/uploads/Competencies-
Project-Final-Report1.pdf.
Core Committee of Institute for International Medical Education. 2002.
Global minimum essential requirements in medical education. Med
Teach 24(2):130–135.
Cumming AD, Ross MT. 2007. The tuning project for medicine – Learning
outcomes for undergraduate medical education in Europe. Med Teach
29(7):636–641.
Cumming AD, Ross MT. 2008. The tuning project for medicine – Learning
outcomes/competences for undergraduate medical education in
Europe. Edinburgh: The University of Edinburgh. [Accessed 24
October 2013] Available from www.tuning-medicine.com.
Eric SH, Laura E, Stanley H. 2014. ACGME milestones. [Accessed 26
Decmber 2014] Available from https://www.acgme.org/acgmeweb/
tabid/430/ProgramandInstitutionalAccreditation/NextAccreditation
System/Milestones.aspx.
European Higher Education Area. 2014. Bologna process – European
Higher Education Area. Romanian Bologna Secretariat. [Accessed 7
April 2014] Available from http://www.ehea.info/article-details.aspx?
ArticleId¼3.
Eva KW, Regehr G. 2008. ‘‘I’ll never play professional football’’ and other
fallacies of self-assessment. J Contin Educ Health Prof 28(1):14–19.
Frank JR, Jabbour M, CanMEDS Working Groups. CanMEDS framework
2005. Ottawa: Royal College of Physicians and Surgeons. [Accessed 24
October 2013] Available from http://www.collaborativecurriculum.ca/
en/modules/CanMEDS/CanMEDS-intro-background-01.jsp.
Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR, Harris P,
Glasgow NJ, Campbell C, Dath D, et al. 2010. Competency-based
medical education: Theory to practice. Med Teach 32(8):638–645.
General Medical Council. 2009. Tomorrow’s doctors. Outcomes and
standards for undergraduate medical education. London: General
Medical Council. [Accessed 6 November 2013] Available from http://
www.gmc-uk.org/TomorrowsDoctors_2009.pdf_39260971.pdf.
General Medical Council. Good medical practice. London: General Medical
Council. [Accessed 4 November 2013] Available from http://www.gmc-
uk.org/static/documents/content/GMP_2013.pdf_51447599.pdf.
Lingard L, Haber RJ. 1999. Teaching and learning communication in
medicine: A rhetorical approach. Acad Med 74(5):507–10.
MedEdWorld. MedEdWorld glossary. The Association for Medical
Education in Europe (AMEE). [Accessed 7 April 2014] Available from
http://www.mededworld.org/Glossary.aspx.
Comparability of outcome frameworks
Medical Council of India. 2011. Visions 2015. New Delhi: Medical Council
of India. [Accessed 6 November 2013] Available from http://
www.mciindia.org/tools/announcement/MCI_booklet.pdf.
Metz JCM, Verbeek-Weel AMM, Huisjes HJ. 2001. Blueprint 2001: Training
of doctors in The Netherlands. Ajusted objectives of undergraduate
medical education in The Netherlands. Disciplinary Board of Medical
Sciences of the Association of Universities in The Netherlands.
Nijmegen: University Publication Office. [Accessed 24 October 2013]
Available from http://www.bibliosgam.ch/pdf/blueprint.pdf.
Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. 2009.
Preferred reporting items for systematic reviews and meta-analyses:
The PRISMA statement. Int J Surg 8(5):366–41.
Morcke A, Dornan T, Eika B. 2013. Outcome (competency) based
education: An exploration of its origins, theoretical basis, and empirical
evidence. Adv Health Sci Educ 18(4):851–863.
MSOP Report Writing Group. 1998. Report I – Learning objectives for
medical student education guidelines for medical schools. Medical
School Objectives Project. Association of American Medical Colleges.
[Accessed 6 November 2013] Available from https://members. aamc.
org/ eweb/upload/Learning%20Objectives%20for% 20Medical%
20Student% 20Educ%20Report%20I.pdf.
National Alliance for 39 Physician Competence. 2009. A Guide to Good
Medical Practice – USA. National Alliance for Physician Competence.
[Accessed 6 Nopvember 2013] Available from http://gmpusa.org/Docs/
GoodMedicalPractice-USA-V1-1.pdf [Version 1.1].
Ringsted C, Hansen TL, Davis D, Scherpbier A. 2006. Are some of the
challenging aspects of the CanMEDS roles valid outside Canada? Med
Educ 40(8):807–815.
Ross MT, Nikoli-ç N, Peeraer G, Murt A, Kroi-ı̀a J, Elcin M,
Hope D, Cumming AD. 2014. Report of the MEDINE2 Bachelor
of Medicine (Bologna First Cycle) tuning project. Med Teach
36(4):314–321.
Scottish Deans’ Medical Education Group. 2008. The Scottish Doctor:
Learning outcomes for the medical undergraduate in Scotland: a
foundation for competent and reflective practitioneers. Dundee:
Association for Medical Education in Europe. [Accessed 4 November
2013] Available from http://www.scottishdoctor.org/resources/scot-
doc3.pdf [3rd edition].
Sheikh AS, Sheikh SA, Kaleem A, Waqas A. 2013. Factors contributing to
lack of interest in research among medical students. Adv Med Educ
Pract 4:237–243.
Ten Cate O. 2014. What is a 21st-century doctor? Rethinking the signifi-
cance of the medical degree. Acad Med 89(7):966–9.
WFME/AMSE International Task Force. 2007. WFME global standards for
quality improvement in medical education. European specifications.
For basic and postgraduate medical education and continuing
professionel development. Copenhagen: World Federation for
Medical Education. [Accessed 24 October 2013] Available from http://
www.amse-med.eu/documents/EuropeanSpecifications.pdf.
Whitford DL, Hubail AR. 2014. Cultural sensitivity or professional
acculturation in early clinical experience? Med Teach 36(11):
951–957.
Wijnen-Meijer M, Burdick W, Alofs L, Burgers C, Ten Cate O. 2013. Stages
and transitions in medical education around the world: Clarifying
structures and terminology. Med Teach 35(4):301–307.
Wojtczak A. 2012. Glossary of medical education terms. Institute for
International Medical Education. [Accessed 7 April 2014] Available from
http://www.iime.org/glossary.htm.
Working Group under a Mandate of the Joint Commission of the Swiss
Medical Schools. 2008. Swiss catalogue of learning objectives for
undergraduate medical training. Joint Commission of the Swiss Medical
Schools (SMIFK/CIMS). [Accessed 6 November 2013] Available from
http://sclo.smifk.ch/downloads/sclo_2008.pdf [2nd Version].