Accepted Manuscript

Title: From Design to Dissemination: Conducting Quantitative Medical

Education Research

Author: Erika L. Abramson, Caroline R. Paul, Jean Petershack, Janet Serwint,

Janet E. Fischel, Mary Rocha, Meghan Treitz, Heather McPhillips, Tai
Lockspeiser, Patricia Hicks, Linda Tewksbury, Margarita Vasquez, Daniel J.
Tancredi, Su-Ting T. Li

PII: S1876-2859(17)30561-2
DOI: https://doi.org/10.1016/j.acap.2017.10.008
Reference: ACAP 1112

To appear in: Academic Pediatrics

Received date: 24-1-2017

Revised date: 3-10-2017
Accepted date: 26-10-2017

Please cite this article as: Erika L. Abramson, Caroline R. Paul, Jean Petershack, Janet Serwint,
Janet E. Fischel, Mary Rocha, Meghan Treitz, Heather McPhillips, Tai Lockspeiser, Patricia
Hicks, Linda Tewksbury, Margarita Vasquez, Daniel J. Tancredi, Su-Ting T. Li, From Design to
Dissemination: Conducting Quantitative Medical Education Research, Academic Pediatrics
(2017), https://doi.org/10.1016/j.acap.2017.10.008.

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service
to our customers we are providing this early version of the manuscript. The manuscript will
undergo copyediting, typesetting, and review of the resulting proof before it is published in its
final form. Please note that during the production process errors may be discovered which could
affect the content, and all legal disclaimers that apply to the journal pertain.
 1 From Design to Dissemination: Conducting Quantitative
2 Medical Education Research
3
4 Erika L. Abramson, MD MSa, Caroline R. Paul, MDb, Jean Petershack, MDc, Janet Serwint, MDd,
5 Janet E. Fischel, PhDe, Mary Rocha, MDf, Meghan Treitz, MDg, Heather McPhillips MD MPHh,
6 Tai Lockspeiser MD MHPEi, Patricia Hicks, MD MPHEj, Linda Tewksbury, MDk, Margarita
7 Vasquez, MDl, Daniel J. Tancredi, PhDm, Su-Ting T. Li, MD MPHn
8
9
10 Corresponding Author Information:
11 Erika L. Abramson, MD MS
12 Weill Cornell Medicine
13 Departments of Pediatrics and Healthcare Policy & Research
14 525 E. 68th Street, Rm M610A
15 New York, NY 10065
16 Phone: 212 746-3929
17 Fax: 212-746-3140
18 Email: err9009@med.cornell.edu
19
20 b Caroline R. Paul, MD
21 University of Wisconsin School of Medicine and Public Health
22 Department of Pediatrics
23 Email:crpaul@wisc.edu
24
25 c Janet E. Fischel, PhD
26 Stony Brook University School of Medicine
27 Department of Pediatrics
28 Email: Janet.Fischel@stonybrookmedicine.edu
29
30 d Jean Petershack, MD
31 University of Texas Health Science Center at San Antonio
32 Department of Pediatrics
33 Email: Petershack@uthscsa.edu
34
35 e Janet Serwint, MD
36 Johns Hopkins University School of Medicine
37 Department of Pediatrics
38 Email: jserwint@jhmi.edu
39
40
41 f Mary Rocha, MD
42 Baylor College of Medicine
43 Department of Pediatrics
44 Email: Mary.Rocha@bcm.edu
45
46 g Meghan Treitz, MD

Page 1 of 38
 1 University of Colorado School of Medicine
2 Department of Pediatrics
3 Email: meghan.treitz@childrenscolorado.org
4
5 h Heather McPhillips, MD MPH
6 University of Washington, Seattle Children’s Hospital
7 Email: heather.mcphillips@seattlechildrens.org
8
9 i Tai Lockspeiser MD MHPE
10 University of Colorado School of Medicine
11 Department of Pediatrics
12 Email: tai.lockspeiser@childrenscolorado.org; tai.lockspeiser@ucdenver.edu
13
14 j Patricia Hicks, MD, MHPE
15 Perelman School of Medicine at the University of Pennsylvania
16 Department of Pediatrics
17 Email: pjhpeds@gmail.com
18
19 k Linda Tewksbury, MD
20 New York University School of Medicine
21 Department of Pediatrics
22 Email: Linda.Tewksbury@nyumc.org
23
24 l Margarita Vasquez, MD
25 University of Texas Health Science Center at San Antonio
26 Department of Pediatrics
27 Email: LONDONOM@uthscsa.edu
28
29 m Daniel J. Tancredi, PhD
30 University of California, Davis
31 Department of Pediatrics and the Center for Healthcare Policy and Research
32 Email: djtancredi@ucdavis.edu
33
34 n Su-Ting T. Li, MD, MPH
35 University of California, Davis
36 Department of Pediatrics
37 Email: sutli@ucdavis.edu
38
39 Key Words: medical education, research
40
41 Running title: Conducting quantitative medical education research
42
43 Abstract Word Count: 221
44
45 Main Text Word Count: 4969
46

Page 2 of 38
1 Funding: This work was not supported by any funding source.
2
3 Conflicts of Interest: The authors have no conflicts of interest to disclose.
4
5
6

Page 3 of 38
1 Abstract

2 Rigorous medical education research is critical to effectively develop and evaluate the training

3 we provide our learners. Yet, many clinical medical educators lack the training and skills needed

4 to conduct high quality medical education research. This paper offers guidance on conducting

5 sound quantitative medical education research. Our aim is to equip readers with the key skills

6 and strategies necessary to conduct successful research projects, highlighting new concepts and

7 controversies in the field. We utilize Glassick’s criteria for scholarship as a framework to

8 discuss strategies to ensure that the research question of interest is worthy of further study and

9 how to use existing literature and conceptual frameworks to strengthen a research study.

10 Through discussions of the strengths and limitations of commonly used study designs, we expose

11 the reader to particular nuances of these decisions in medical education research and discuss

12 outcomes generally focused upon, as well as strategies for determining the significance of

13 consequent findings. We conclude with information on critiquing research findings and

14 preparing results for dissemination to a broad audience. Practical planning worksheets and

15 comprehensive tables illustrating key concepts are provided in order to guide researchers through

16 each step of the process. Medical education research provides wonderful opportunities to

17 improve how we teach our learners, to satisfy our own intellectual curiosity and ultimately, to

18 enhance the care provided to patients.

19

Page 4 of 38
1 Introduction

2 Rigorous medical education research provides evidence for approaches to improve the

3 education of our learners, and shares with clinical, basic and translational research the ultimate

4 goal of improved patient outcomes. Medical education research can provide data to affirm that

5 we are training competent physicians. Evidence-based decisions, grounded in rigorous medical

6 education research, can guide changes in the delivery of medical education in order to assure that

7 the educational “product,” the trainee, is best prepared for the practice of medicine.1

8 At the same time, conducting a well-designed quantitative medical education research

9 study requires attention to factors quite different from traditional clinical research. For example,

10 the effectiveness of educational interventions often results from a complex interplay between the

11 learner, the educator, and the educational and clinical learning environment. As these factors

12 may vary significantly across programs, conducting large-scale studies and demonstrating

13 generalizability of findings can be challenging. Within individual programs, bleed or cross

14 contamination across learners who have and have not received an intervention can threaten one’s

15 ability to demonstrate an intervention’s effectiveness, as can the natural maturation that tends to

16 occur as learners gain experience and knowledge over time. Also important are institutional

17 review board (IRB) considerations when administering an educational intervention that may

18 impact only a subset of learners, or when trainees are the study subjects and may fear negative

19 consequences for declining consent.

20 Unfortunately, many clinical medical educators lack the training needed to conduct high-

21 quality medical education research and clear reporting of experimental medical education

22 research has been modest at best.2-4 It is imperative to fill this skill gap because quality medical

23 education research has the potential to improve several facets of the complex educational

Page 5 of 38
1 process: teaching and learning strategies, curriculum development and evaluation methods,

2 health care delivery, and, ultimately, patient outcomes.

3 To address these challenges, we have developed this paper on how to conduct meaningful

4 and rigorous quantitative medical education research. We use Glassick’s criteria for scholarship,

5 which includes clear goals, adequate preparation, appropriate methods, significant results,

6 effective presentation, and reflective critique, as essential components of a framework that can

7 be used to answer any research question.5 This paper fills important gaps through its discussion

8 of conceptual frameworks, focus on the nuances and challenges that govern methodologic and

9 ethical considerations in medical education research, and provision of practical tools and

10 suggestions for those embarking on a research project.6, 7 Qualitative research and studies with

11 mixed methodologies are increasingly being recognized for their considerable value in medical

12 education scholarship for many reasons, including with regard to identifying worthwhile

13 hypotheses or most relevant measures for further studies. Coverage of both quantitative and

14 qualitative research methods in a single paper would be prohibitively expansive. This paper

15 therefore focuses on frequently utilized quantitative methods. The reader is referred to an

16 excellent primer on qualitative medical education research for guidance with such designs.8

17

18 Before you Begin: Acquiring Skills to Conduct Medical Education Research

19 Prior to beginning any research endeavor, every researcher must reflect on his or her own

20 skill set and that of potential collaborators. Consider working with content and methodologic

21 mentors within and outside your institution to facilitate planning and conducting your work.

22 Medical education scholarship workshops at national meetings such as Pediatric Academic

23 Societies (PAS), Association for Pediatric Program Directors (APPD), or Council on Medical

Page 6 of 38
1 Student Education in Pediatrics (COMSEP) offer practical approaches to designing and

2 conducting medical education research. Programs such as the Academic Pediatric Association

3 (APA) Educational Scholars Program, the Association of American Medical Colleges (AAMC)

4 Medical Education Research Certificate Program, and the APPD Leadership in Educational

5 Academic Development program offer more formal training.9-11 People interested in more

6 rigorous training may benefit from pursuing a Masters or a PhD in medical education, which can

7 often be completed remotely while working full-time.

9 Clear Goals – What do you want to Accomplish?

10 Sound scholarship starts with a sound idea. We outline below an approach for developing an

11 idea into a compelling research argument, transforming it into a research question guided by

12 conceptual frameworks, and developing a study design and analysis guided by the question.

13 While we present this as a linear process for the sake of clarity, this process is highly iterative.

14

15 Crafting a Compelling Research Argument

16 To be worthy of dissemination, research must address a compelling and widely shared or

17 recognized problem. For example, suppose you want to improve the rates of successful neonatal

18 intubation by trainees. Articulation of a compelling problem might include: 1) residents are

19 mostly unsuccessful when attempting neonatal intubation, potentially jeopardizing patient

20 outcomes 2) this is true despite conventional neonatal resuscitation program participation; and 3)

21 few residents have opportunities to intubate patients.12 Therefore, an intervention that

22 successfully addresses these issues will likely be of interest to many residency programs.

23

Page 7 of 38
1 Conceptual Frameworks

2 With a compelling problem and idea in hand, a key step is to consider the educational

3 philosophies which may illuminate the approach to addressing this problem. Georges Bordage

4 describes these philosophies, known as conceptual frameworks, as “ways of thinking about a

5 problem or a study, or ways of representing how complex things work the way they do.”13

6 Appropriate use of conceptual frameworks in educational research is increasingly being

7 recognized as key to advancing the rigor of quantitative studies.14 One or multiple conceptual

8 frameworks may be used and should be made explicit in order to afford readers greater

9 understanding of what guided particular decisions and how findings might translate to other

10 educational contexts. See Table 1 for examples of conceptual frameworks often used to frame

11 teaching interventions.

12 Consider the neonatal resuscitation example. The theory of deliberate practice may

13 inform an intervention to improve rates of successful neonatal intubation.15 Deliberate practice

14 is the individualized training activities designed to improve specific aspects of an individual’s

15 performance through repetition, feedback, and successive refinement.15 Applying this

16 conceptual model may help you refine and frame your question – What is the optimal frequency

17 and duration of practice? What is the optimal frequency of feedback?

18 As an example from the literature, authors Hunt et al were interested in using a novel

19 simulation approach to improve pediatric resident performance of cardiopulmonary resuscitation

20 (CPR). In traditional simulation interventions, learners are given a scenario, progress through

21 the entire scenario, and then debrief afterwards about what was done effectively or ineffectively.

22 In this study, however, the authors explicitly drew upon the theory of deliberate practice to test

23 the idea that a simulation-based intervention would be more effective if mistakes were corrected

Page 8 of 38
1 in real-time in order to maximize the time spent deliberately practicing a skill performed in the

2 correct way. Thus, they designed their educational simulation intervention as “rapid cycle

3 deliberate practice training” in which residents were given a scenario, performed a hands-on

4 skill, got immediate coaching to correct mistakes, and continued performing the hands-on skill

5 correctly to create muscle memory. This novel approach, guided by the use of a common

6 educational conceptual framework, was associated with improvement in performance compared

7 to traditional simulation methods.

9 Formulating the Research Question

10 Adapting the PICOTS format that has been successfully used in evidence-based

11 medicine can help you develop a clear and focused research question. PICOTS includes

12 population, intervention, relevant comparison group(s), outcome(s) assessed, timing and setting

13 (of the outcome assessment).16 For our example, the research question might be: Among

14 pediatric residents from a medium-sized academic residency program, does participation in

15 weekly 30-minute neonatal intubation simulation during neonatology rotations, compared with

16 participation in an annual standard neonatal resuscitation training program, result in higher rates

17 of successful intubation of patients at the end of the academic year? Our hypothesis may be:

18 implementing weekly 30-minute neonatal intubation simulation sessions during neonatology

19 rotations will lead to mastery of neonatal intubation more effectively than annual standard

20 neonatal resuscitation programs. Hypothesis testing, a hallmark of clinical and bench research,

21 is equally important for rigorous quantitative educational scholarship.

22 The strength of a research question can be measured with the I-SMART criteria (an

23 acronym for important, specific, measurable, achievable, relevant, and timely), which connects

Page 9 of 38
1 well to the first Glassick criterion: Clear Goals (Table 2).17 Given rigorous publication

2 standards, it is prudent to consider the I-SMART question as the “outcome-driven I-SMART

3 question.” A classic paradigm often utilized for educational outcomes is Kirkpatrick’s Learning

4 Evaluation Model, which consists of 4 levels of educational outcomes: Level 1: reaction (learner

5 satisfaction), Level 2: learning (attitude, knowledge, and skill acquisition), Level 3: behavior

6 change, and Level 4: patient impact (Table 3).18 Aim as high on the pyramid as possible.

7 Research with lower outcome levels may be publishable if highly innovative, while less

8 innovative studies generally require higher outcome levels. Importantly, one cannot assume that

9 changes in knowledge, skills, and attitudes in a training or research context will have a direct

10 impact on behavior in the clinical setting or on patient outcomes. Quality improvement, medical

11 education, and perspectives from behavioral economics all support this assertion.19-22 Measuring

12 behavior change in situ and measuring relevant patient outcomes are challenging but critical for

13 advancing the field of medical education research.

14

15 Adequate Preparation– Are you ready to do this project?

16 Glassick’s second criterion, adequate preparation, ensures that the research question

17 satisfies the relevant element of I-SMART. Adequate preparation includes: (1) understanding

18 what has already been studied and where the gaps lie through literature review; (2) acquiring

19 necessary skills/resources/collaborators; and (3) ensuring institutional review board (IRB)

20 approval/exemption.

21

22 Literature Review

10

Page 10 of 38
1 Before embarking on your study, you must understand what is known and unknown

2 about a particular problem and how well your question addresses these gaps. A medical librarian

3 is a valuable resource to assist with this process. Typical databases include MEDLINE,

4 PubMed, Google Scholar, Scopus, Cumulative Index to Nursing and Allied Health Literature

5 (CINAHL), and PsychINFO. Education-specific resources include Educational Resource

6 Information Center (ERIC) [articles, books, theses, guidelines in education], EdIT Library

7 [articles about education and information technology], MedEdPORTAL [a repository of peer

8 reviewed educational tools and curricular materials], MERLOT [a repository of peer reviewed

9 educational materials for all of higher education], Association of Pediatric Program Directors

10 (APPD) Share Warehouse [available to APPD members], the British Educational Index, and Best

11 Evidence Medical Education Collaboration. Critical appraisal of the literature follows to

12 determine whether existing studies are current and comprehensive, methodologically sound, and

13 broadly generalizable. If your idea has already been studied, consider if your project offers a

14 different perspective, such as a different level of learner, an expanded sample size, newer and

15 more accurate measurement tools, or an innovative teaching technique.

16

17 Acquire Necessary Skills and Resources

18 Building a research team with the necessary skills and expertise is key to any project’s

19 success. Statisticians, content and methodological mentors, physician scientists, and non-clinical

20 research faculty can all serve as outstanding partners. Trainees or junior faculty who can

21 contribute in a substantial manner and, in turn, benefit from collaborative efforts, can also be

22 valuable. It is also important to identify the costs to complete your project and potential funding

23 sources if necessary. These sources may include local funding opportunities, educational grants

11

Page 11 of 38
1 available through national organizations such as the American Academy of Pediatrics (AAP),

2 APA, APPD, COMSEP or larger foundation and federal grants (particularly for projects

3 assessing patient outcomes).

5 Institutional Review Board (IRB) Considerations

6 It is essential to receive formal approval through the IRB prior to beginning any study.

7 However, there are several unique considerations for educational research.23 First, educational

8 research is often considered exempt if it is conducted in established or commonly accepted

9 educational settings, involves normal educational strategies, and the participant information is

10 de-identified.24 Thus comparing different instructional techniques and curricula through test

11 scores, performance evaluations, or de-identified surveys is often IRB exempt. Some institutions

12 may even have a blanket IRB exemption for all activities considered education research.

13 On the other hand, learners are a vulnerable population at risk for coercion for many

14 reasons including that they often interact with or are evaluated by those conducting the research

15 study. Therefore, protocols must clearly describe the process of informed consent and detail how

16 learners can refuse to participate or feel free to give honest opinions without fear of retribution,

17 while still being able to participate in all other aspects of any educational intervention that will

18 be implemented broadly. Consideration must also be given as to who will be collecting data so

19 that learners do not feel unfairly scrutinized by those in a position of authority. In addition,

20 concerns about fairness and equity in education may prompt some IRBs to have difficulty

21 approving studies where only some trainees receive an educational intervention, and may be

22 more likely to approve studies utilizing historical controls, cluster-randomized controls by

23 program, or crossover study designs.25 An additional challenge may occur when multiple

12

Page 12 of 38
1 institutions participate in the same educational research study. IRB approval at one institution

2 may suffice; however, often most institutions require their own individual approval.

4 Appropriate Methods – How will you design your project to achieve your specific aims?

5 The saying “begin with the end in mind” applies to the selection of methods for a

6 research study. Some projects aim to generate hypotheses; others to test them. In this section,

7 we will introduce some of the more common terminology and quantitative designs used in

8 educational research, using our neonatal resuscitation example (Table 4). We will focus

9 primarily on methodologies used for hypothesis testing where the goal is to obtain measurable

10 outcomes to answer a question (often described as a positivist paradigm). While this paradigm

11 tends to be predominant in medical education, other paradigms, such as constructivist paradigms,

12 in which subjective experiences are emphasized, are also well suited to medical education

13 research.

14 Let us begin with this scenario: A program director, Dr. Smith, is concerned about the

15 effectiveness of the training her residents receive in neonatal resuscitation. While eager to

16 design an actual intervention to address this concern, she recognizes that she does not have a

17 good understanding of the true nature of the underlying problem and wants to explore the

18 problem through a descriptive study. Her research question might be: What are the facilitators

19 and barriers affecting pediatric residents’ comfort with and skill in performing neonatal

20 resuscitation? To answer this question, she may perform a cross-sectional study in which all data

21 are collected at one time point, such as through a survey with close-ended questions. Her a priori

22 hypotheses can be explored by having residents self-assess comfort and indicate agreement or

23 disagreement with pre-identified barriers and facilitators listed in the survey instrument.

13

Page 13 of 38
1 The results of a descriptive study often provide useful background to generate hypotheses

2 for conducting further explanatory studies. Perhaps the descriptive research revealed that

3 residents are predominantly struggling with the inter-professional teamwork aspect of conducting

4 a neonatal resuscitation. To address this challenge, Dr. Smith decides to introduce a new

5 curriculum that focuses on facilitating the teamwork of residents and inter-professional staff

6 during high-fidelity simulation training with mannequins. Her research question might be: Does

7 introduction of a curriculum to facilitate inter-professional teamwork during high fidelity

8 neonatal resuscitation training improve resident resuscitation skills and ability to work in inter-

9 professional teams in the delivery room?

10 Several different explanatory study designs could be utilized to answer this question,

11 each with distinct strengths and limitations. Cohort studies follow one or more populations of

12 learners longitudinally with respect to an outcome of interest. They are observational rather than

13 experimental – in other words, there is no intervention implemented to assess the outcome of

14 interest. Cohort studies cannot show causality, and thus, in this case, a cohort study would be

15 unlikely to successfully answer Dr. Smith’s research question.

16 In this case, there are quasi-experimental or experimental designs that Dr. Smith can

17 consider. Experimental designs are characterized by the investigator being able to assign

18 participants to specific interventions or conditions, ideally using rigorous randomization methods

19 to ensure a statistically equivalent comparison group. Quasi-experimental designs share

20 attributes of experimental designs, but typically lack a randomized control group, which make

21 them vulnerable to internal validity threats. One such quasi-experimental design is the pre-post

22 design, in which a baseline pre-intervention assessment and a post-intervention assessment of the

23 learner are conducted. The main limitation to this design is the lack of a concurrent comparison

14

Page 14 of 38
1 group. While you may demonstrate changes within each learner with a pre-post design, it is

2 difficult to attribute these to the intervention, because other confounding factors, such as natural

3 professional maturation, may impact learner performance at the post-intervention assessment.

4 Another quasi-experimental approach is the pre/post design with nonequivalent parallel

5 groups. In this approach, pre-intervention and post-intervention data are collected on

6 “intervention” and “control” groups respectively. However, these groups are naturally existing,

7 such as residents in different academic years or at different institutions. They are not randomly

8 allocated. Again returning to our example, if Dr. Smith opts for this quasi-experimental

9 approach, she might have all interns complete the new curriculum, and compare their

10 performance at baseline and the end of intern year to the performance of external NICU rotators

11 who experience the same resuscitation training but are not exposed to the new curriculum. To

12 address one of the main limitations of this study design – that the two groups are fundamentally

13 different from each other -- she might collect information on potential confounding factors such

14 as baseline comfort with and exposure to intubations and resuscitations, which could be used in

15 restricting the sample or in statistical analysis procedures to minimize bias from measured

16 confounders. Another significant limitation of this quasi-experimental study design is

17 “bleeding” between the control and intervention groups that often occurs when residents work

18 together, which can be limited by choosing historical controls or controls at separate sites that

19 have limited interaction with the intervention group.

20 A third option is the randomized control trial, in which subjects are randomly allocated

21 to an “intervention” or “control” arm. While this design is often considered the gold standard in

22 clinical research, it can be particularly challenging in medical education as sample sizes are

23 typically small and random assignment may be difficult or impossible due to deliberate

15

Page 15 of 38
1 placement of trainees on specific rotations. Randomization in a small cohort of students working

2 closely together may also result in exchange of information, which can be difficult to prevent but

3 important to note in any limitation section. In addition, with all experimental study designs,

4 educators may need to consider how to offer an educational intervention to all learners if proven

5 beneficial; one can consider designing a study that includes a plan to train or teach the “control”

6 participants the newly studied skill at the conclusion of the trial, or, with IRB approval, train the

7 “control” participants at the conclusion of the trial and measure their performance at some end

8 point once again on the outcomes of interest.

9 Sample Size Planning. Sample size justifications typically consider the necessary

10 precision to satisfactorily estimate the parameters of interest or the sample size necessary to

11 provide good statistical power (i.e. a high probability of detecting meaningful effects). Ensuring

12 you will have adequate sample size is critical to obtaining meaningful results. One of the major

13 challenges in educational research is that settings such as residency programs are often limited in

14 terms of size and representativeness. As a result, research projects should have built in, from the

15 beginning, a long-term plan for scaling up beyond a simple pilot to extend either to multiple sites

16 or be conducted across multiple time points.

17 Study Registration. Registration of a study prior to data collection in a well-recognized

18 study registry (e.g. clinicaltrials.gov) can be an essential element to promote the rigor and

19 transparency of the study and for some study designs (e.g. clinical trials, including educational

20 randomized controlled trials, and systematic reviews) is required by publishers.

21

22 Are your Results Meaningful?

16

Page 16 of 38
1 Meaningful results are both internally valid (include appropriate research design and analysis

2 to answer the important research question) and externally valid (replicable and generalizable to

3 other populations and settings). Design your analysis to answer your research question. For

4 quantitative studies, choose your study design, outcome variables (often called dependent

5 variables), measurement approaches, and analysis methods in collaboration with your statistician

6 (a partnership that cannot be emphasized enough!). Remember that lack of statistical

7 significance may have multiple explanations: 1) there truly is no difference between the groups;

8 2) there is a difference between the groups but it cannot be detected with your sample’s size and

9 the variability on the outcome measure of interest (Type 1 and Type 2 errors); and 3) there are

10 confounding factors that are not controlled or not equivalent across groups. To properly interpret

11 your results, your study must be appropriately powered to detect a difference between groups.

12 Collect outcome variables and potential confounding variables appropriate to your research

13 question. Confounding variables are variables that could correlate with both the outcomes you

14 are trying to measure (dependent variables) and your intervention/exposure (independent

15 variable). For example, if you are studying whether a just-in-time neonatal intubation simulation

16 is associated with increased success in intubations, you may want to include potential

17 confounders such as size and gestational age of the infant.

18 Measure data using instruments with validity evidence to address your research question.26

19 Validity evidence includes content validity, response process, internal structure, relationship to

20 other variables, and consequences.26 If you conduct a research project that does not yield the

21 outcomes you are seeking, it could be that the outcome you sought did not actually occur in your

22 study. Alternatively, it could be that your measurement instrument was insensitive to those

23 outcomes or changes that did indeed take place. You will not be able to untangle those two

17

Page 17 of 38
1 explanations, so it is important to use instruments with validity evidence.

2 When a measurement instrument with validity evidence does not already exist, you might

3 consider developing and validating a measurement instrument to disseminate for further use. For

4 example, if designing a survey, consider whether the survey questions are based on the literature

5 and agreed upon by content experts in the field (content validity), examined via cognitive

6 interviews to ensure the survey items were clear, relevant, and correctly interpreted (response

7 process), and piloted to ensure survey scores measuring the same construct are correlated

8 (internal structure) and relate to other measures in ways that make sense (relationship to other

9 variables)27 These concepts should be assessed for pre-existing as well as de novo surveys.

10 Returning to the example detailed in the methods section, it is key to note that Dr. Smith could

11 only perform her research effectively if she had a valid instrument with which to assess resident

12 performance in resuscitation skills and resident ability to work effectively as part of an inter-

13 professional team.

14 Utilize data analysis appropriate for your research (Table 5).28 In most instances, instead of

15 or in addition to reporting p-values, one should report parameter estimates (including treatment

16 effect estimates) along with 95% confidence intervals or other measures of uncertainty.29, 30

17 When making multiple comparisons, consider the possibility that as the number of comparisons

18 increase, so too does the risk of incorrectly concluding that a comparison is statistically

19 significant (Type 1 error). Hence, it may be desirable to use methods for simultaneous statistical

20 inference that can limit the overall probability of such an error for a family of comparisons.31

21 The more the study population represents the population at large, the more generalizable the

22 results of the study. Single site studies can be published if they add new information to the

23 literature and address important innovative questions. They may lead to multi-institutional

18

Page 18 of 38
1 studies, which can enhance generalizability of the results. The APPD Longitudinal Educational

2 Assessment Research Network (LEARN) and APA Continuity Research Network (CORNET),

3 are examples of collaborative research networks that enable generalizability of pediatric

4 educational research through supporting multi-institutional studies.32

5 Describe your sample in sufficient detail so that others can determine whether your study

6 sample is similar to their learners in their setting. If you have information about the population

7 at large, compare your study sample to that population. For example, if you are surveying

8 pediatric program directors, compare your study sample (respondents) to non-respondents

9 nationally in terms of program size and location.33 When making inferences about a population,

10 sample the population in a manner that promotes generalizability and minimizes selection bias.

11 For example, if you are studying whether graduating pediatric residents perceive themselves to

12 be prepared to counsel patients on obesity prevention, sample from a population of all graduating

13 pediatric residents, such as the AAP Annual Survey of Graduating Residents.34 However, if your

14 study question is to determine whether resident characteristics and experiences are related to

15 practicing in underserved areas and your hypothesis is that their background may affect their

16 desire to practice in underserved areas, consider oversampling underrepresented minority

17 residents.35

18

19 Effective Presentation –How will you disseminate your findings?

20 Effective presentation and dissemination are critical to advancing the science of medical

21 education. Furthermore, academic success and promotion are typically measured, in part, by the

22 quality and quantity of peer-reviewed presentations and publications. Dissemination of your

23 work builds a track record and creates a niche reflective of your research interests and expertise.

19

Page 19 of 38
1 There are a number of ways to share your research findings. Initially you may present your

2 work at the departmental level as research-in-progress. Once you have results to share, consider

3 presenting at local, regional or national meetings. Through these venues, the feedback and

4 potential mentoring you receive can help frame your writing plan and forge potential multisite

5 collaborations that propel your work to a higher level of importance. Also, consider if your

6 study lends itself to workshop development and delivery. Evaluations from such national

7 workshops that demonstrate the effectiveness of the presentation are critical to include for

8 promotional capital.

9 Identification of the optimal journal for publication requires a critical look into the

10 educational literature. Which journals are interested in this type of work? What journals have

11 published articles using similar methods? Which audience do you wish to reach? What is the

12 impact factor of the journal? The impact factor is a proxy measure of a journal’s relative

13 importance in a field and measures the yearly average number of citations to recent articles

14 published in the journal. In addition, if you have developed an innovative curriculum or an

15 assessment tool, you can consider submitting your work to a resource portal such as

16 MedEdPORTAL or APPD Share Warehouse. Recognize that it may take several attempts to

17 publish a single paper. Incorporating reviewer comments can strengthen a paper prior to

18 resubmission. (See Appendix 2, checklist for manuscript submission, to facilitate the writing

19 process).36, 37 Also of note, for a variety of study designs, there are evidence-based reporting

20 guidelines available from the EQUATOR NETWORK (www.equator-network.org). We strongly

21 recommend that these guidelines be used both in the planning and reporting of studies.

22

23 Reflective Critique – How will You Improve upon Your Work?

20

Page 20 of 38
1 Glassick’s final criterion requires scholars to critically evaluate their own work. This

2 involves reflecting on the literature review, critiquing the findings, and considering next steps to

3 advance your research. Your results should be viewed in the context of the gap identified for the

4 study and should be carefully linked to previous studies and/or the conceptual framework(s) that

5 you utilized. You must display a clear understanding not only of the results of the study but also

6 of the strengths and limitations. Simply listing limitations does not suffice. It is important to

7 consider how the limitations impact your results and their implications. Finally, a primary goal

8 of educational scholarship is to create the foundation for future scholarship. Since no single

9 study solves or resolves all issues around a topic, reflective critique must discuss the implications

10 for educational practice as well as future research to move the field forward.

11

12 Conclusions: How do we Continue to Advance the Field?

13 Quality educational research benefits our trainees as well as the faculty who teach them,

14 our institutions, the medical education community at large and ultimately, the patients we serve.

15 When educators identify and disseminate best educational practices, they have the opportunity to

16 greatly influence the next generation of physicians. Because of the continually changing

17 healthcare environment, and the challenges that educational institutions face as they prepare,

18 promote and assess the knowledge, skills and attitudes of their learners, educators need to

19 continually evaluate educational programs and share best practices among institutions and across

20 the training continuum.

21 Building the skills of clinical medical educators to conduct high quality quantitative

22 research is therefore critical to advancing education at the pace we need. Educational research

23 has made great advances in the past 10 years, as demonstrated in part by an increase in the

21

Page 21 of 38
1 number of articles published as well as journals devoted specifically to medical education.14

2 However, there remain important gaps. Indeed, weak study questions and designs were the

3 primary reasons for manuscripts not being sent for review by the editors of Academic

4 Medicine.38

5 How do we continue to elevate the rigor of medical education research? First,

6 educational studies must be carefully designed prior to implementing educational interventions

7 so that the rigor of the study can be optimized at the outset. Second, conceptual frameworks

8 should guide the development of educational interventions and their assessments. Quality

9 checklists have been proposed that can be used by authors, editors, and consumers of medical

10 education research to help ensure the rigor of research studies.14, 39 In the past these did not

11 explicitly include conceptual frameworks, while more current checklists do. Third, in addition to

12 assessing the effects of interventions on learner knowledge, skills, attitudes, and behaviors, more

13 research must focus on the processes that facilitate application of learning and behavior, as well

14 as the impact of interventions on patients and organizational goals and outcomes. These

15 principles have recently been endorsed as part of the New World Kirkpatrick Model.40 Fourth,

16 within pediatrics, there is growing infrastructure to allow for multi-site studies, such as the

17 Association of Pediatric Program Directors Longitudinal Educational Assessment and Research

18 Network (APPD LEARN) and CORNET.32 Researchers should take advantage of such

19 resources in order to ensure that research findings are generalizable across settings. Lastly,

20 journal readers, authors, and editors alike must continue to demand these high standards of

21 medical education research in order to ensure that these standards become routine and that we

22 continue to innovate within our profession.

23

22

Page 22 of 38
1 References

2 1. Klein MD, Li ST. Building on the Shoulders of Giants: A Model for Developing Medical
3 Education Scholarship Using I-PASS. Academic pediatrics. 2016;16:499-500.
4 2. Levinson W, Rubenstein A. Integrating clinician-educators into Academic Medical
5 Centers: challenges and potential solutions. Acad Med. 2000;75:906-912.
6 3. Cook DA, Beckman TJ, Bordage G. Quality of reporting of experimental studies in
7 medical education: a systematic review. Medical education. 2007;41:737-745.
8 4. Castiglioni A, Aagaard E, Spencer A, et al. Succeeding as a Clinician Educator: useful
9 tips and resources. J Gen Intern Med. 2013;28:136-140.
10 5. Glassick CE HM, Maeroff GI. Scholarship assessed – evaluation of the professoriate.
11 San Francisco, CA: Jossey-Bass; 1997.
12 6. Yarris LM, Deiorio NM. Education research: a primer for educators in emergency
13 medicine. Acad Emerg Med. 2011;18 Suppl 2:S27-35.
14 7. Beckman TJ, Cook DA. Developing scholarly projects in education: a primer for
15 medical teachers. Medical teacher. 2007;29:210-218.
16 8. Hanson JL, Balmer DF, Giardino AP. Qualitative research methods for medical
17 educators. Academic pediatrics. 2011;11:375-386.
18 9. Jerardi KE, Mogilner L, Turner T, Chandran L, Baldwin CD, Klein M. Investment in
19 Faculty as Educational Scholars: Outcomes from the National Educational Scholars
20 Program. J Pediatr. Vol 171. 2016/03/29 ed2016:4-5 e1.
21 10. Medical Education Research Certificate Program. In: Colleges AoAM, ed.
22 11. Trimm F, Caputo G, Bostwick S, et al. Developing leaders in pediatric graduate
23 medical education: the APPD LEAD Program. Academic pediatrics. 2015;15:143-146.
24 12. Campbell DM, Barozzino T, Farrugia M, Sgro M. High-fidelity simulation in neonatal
25 resuscitation. Paediatrics & child health. 2009;14:19-23.
26 13. Bordage G. Conceptual frameworks to illuminate and magnify. Medical education.
27 2009;43:312-319.
28 14. Sullivan GM, Simpson D, Cook DA, et al. Redefining Quality in Medical Education
29 Research: A Consumer's View. Journal of graduate medical education. 2014;6:424-
30 429.
31 15. Ericsson KA KR, Tesch-Römer C. The role of deliberate practice in the acquisition of
32 expert performance. Psychology Review. 1993;100:363-406.
33 16. Medicine CfEB. Asking Focused Questions. Nuffield Department of Primary Care
34 Health Science, University of Oxford.
35 17. Doran GT. There’s a SMART way to write management’s goals and objectives.
36 Manage Rev. 1981;70:35-36.
37 18. Kirkpatrick D. Great ideas revisited. Techniques for evaluating training programs.
38 Revisiting Kirkpatrick's four-level model. . Technology and Development.
39 1996;50:54-59.
40 19. Hostetter M, and Klein S;. In Focus: Using Behavioral Economics to Advance
41 Population Health and Improve the Quality of Health Care Services. Quality Matters
42 Archive: The Commonwealth Fund; 2013.
43 20. Davis D, O'Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact
44 of formal continuing medical education: do conferences, workshops, rounds, and

23

Page 23 of 38
 1 other traditional continuing education activities change physician behavior or
2 health care outcomes? Jama. 1999;282:867-874.
3 21. Team; IE. In: Exchange AHI, ed. Sustaining and Spreading Quality Improvement. Vol
4 20172014.
5 22. Mayer-Mihalski NaD, M;. Effective Education Leading to Behavior Change. In:
6 ParagonRx, ed. Vol 20172009.
7 23. Keune JD, Brunsvold ME, Hohmann E, Korndorffer JR, Jr., Weinstein DF, Smink DS.
8 The ethics of conducting graduate medical education research on residents. Acad
9 Med. 2013;88:449-453.
10 24. Miser WF. Educational research--to IRB, or not to IRB? Family medicine.
11 2005;37:168-173.
12 25. Kraus CK, Guth T, Richardson D, Kane B, Marco CA. Ethical considerations in
13 education research in emergency medicine. Acad Emerg Med. 2012;19:1328-1332.
14 26. Downing SM. Validity: on meaningful interpretation of assessment data. Medical
15 education. 2003;37:830-837.
16 27. Rickards G, Magee C, Artino AR, Jr. You Can't Fix by Analysis What You've Spoiled by
17 Design: Developing Survey Instruments and Collecting Validity Evidence. Journal of
18 graduate medical education. 2012;4:407-410.
19 28. Windish DM, Diener-West M. A clinician-educator's roadmap to choosing and
20 interpreting statistical tests. J Gen Intern Med. 2006;21:656-660.
21 29. du Prel JB, Hommel G, Rohrig B, Blettner M. Confidence interval or p-value?: part 4
22 of a series on evaluation of scientific publications. Deutsches Arzteblatt international.
23 2009;106:335-339.
24 30. Cummings P, Rivara FP. Reporting statistical information in medical journal articles.
25 Arch Pediatr Adolesc Med. 2003;157:321-324.
26 31. Gelman A, Hill J, Yajima, M. . Methodological Studies: Why we (usually) don’t have to
27 worry about multiple comparisons. Journal of Research on Educational Effectiveness.
28 2012;5:189-211.
29 32. Schwartz A, Young R, Hicks PJ. Medical education practice-based research networks:
30 Facilitating collaborative research. Medical teacher. 2016;38:64-74.
31 33. Abramson EL, Naifeh MM, Stevenson MD, et al. Research training among pediatric
32 residency programs: a national assessment. Acad Med. 2014;89:1674-1680.
33 34. Frintner MP, Liebhart JL, Lindros J, Baker A, Hassink SG. Are Graduating Pediatric
34 Residents Prepared to Engage in Obesity Prevention and Treatment? Academic
35 pediatrics. 2016;16:394-400.
36 35. Laraque-Arena D, Frintner MP, Cull WL. Underserved Areas and Pediatric Resident
37 Characteristics: Is There Reason for Optimism? Academic pediatrics. 2016;16:401-
38 410.
39 36. AAMC Review Criteria for Research Manuscripts. In: Durning SJ CJ, ed2015.
40 37. Li ST KM, Gusic M, Vinci R, Szilagyi P. . Crossing the Finish Line: Getting your
41 Medical Education Work Published. . Pediatric Academic Societies Workshop2016.
42 38. Meyer HS, Durning SJ, Sklar D, Maggio LA. Making the First Cut: An Analysis of
43 Academic Medicine Editors' Reasons for Not Sending Manuscripts Out for External
44 Peer Review. Acad Med. 2017.
45 39. Reed DA, Beckman TJ, Wright SM, Levine RB, Kern DE, Cook DA. Predictive validity
46 evidence for medical education research study quality instrument scores: quality of

24

Page 24 of 38
 1 submissions to JGIM's Medical Education Special Issue. J Gen Intern Med.
2 2008;23:903-907.
3 40. Moreau KA. Has the new Kirkpatrick generation built a better hammer for our
4 evaluation toolbox? Medical teacher. 2017;39:999-1001.
5 41. Bandura A. Social foundations of thought and action: A social cognitive theory.
6 Englewood Cliffs, NJ1986.
7 42. Young HN, Schumacher JB, Moreno MA, et al. Medical student self-efficacy with
8 family-centered care during bedside rounds. Acad Med. 2012;87:767-775.
9 43. Ericsson KA. Acquisition and maintenance of medical expertise: a perspective from
10 the expert-performance approach with deliberate practice. Acad Med.
11 2015;90:1471-1486.
12 44. Hunt EA, Duval-Arnould JM, Nelson-McMillan KL, et al. Pediatric resident
13 resuscitation skills improve after "rapid cycle deliberate practice" training.
14 Resuscitation. 2014;85:945-951.
15 45. Kolb DA. Experiential Learning: Experience as the Source of Learning and
16 Development. Upper Saddle River, NJ: Prentice Hall, Inc.; 1984.
17 46. Klein M, Vaughn LM. Teaching social determinants of child health in a pediatric
18 advocacy rotation: small intervention, big impact. Medical teacher. 2010;32:754-
19 759.
20
21

22

25

Page 25 of 38
1

26

Page 26 of 38
 1 Table 1: Examples of Conceptual Frameworks Used in Medical Education
2
Conceptual Description Example of use of conceptual framework
Framework

Used cross-sectional surveys to explore factors that

Bandura’s Social People learn from one another by
supported self-efficacy with family centered care
Cognitive Theory41 observing and imitating others’
among third year medical students during their
behavior.
pediatric clerkship42
Self-efficacy is an important pre-
requisite guiding behavior
Self-efficacy can be supported by
observing role models, having
opportunities to practice a
behavior, and receiving feedback
on performance

Prospective pre-post intervention study of pediatric

Ericsson’s Theory of Individualized training activities
resident rapid cycle deliberate practice of
Deliberate designed to improve specific
43 resuscitation skills with immediate feedback and
Practice aspects of an individual’s
opportunity to “pause, rewind 10 s and try it again”
performance through repetition,
in simulated cardiopulmonary arrest scenarios.44
immediate feedback, and
successive refinement.
Development of a curriculum to teach social
Kolb’s Experiential Learning happens through
determinants of health using experiential learning
Learning Cycle45 transforming experience through a
(e.g., “field trip” to food bank), followed by
4-stage learning cycle:
reflection on the experience and development of
Concrete Experience: Do theory through abstract conceptualization (residents
Something asked to reflect on learning experience and how it
Reflective Observation: Think will influence their clinical practice through
About What You Did “Memo-to-Myself” exercise), and testing their
hypotheses (apply what they learned to their care of
Abstract Conceptualization: Make underserved patients in clinic).46
Sense of What You Experienced
Through Developing Theories
Active Experimentation: Put What
You Learned Into Practice
3
4
5

27

Page 27 of 38
Table 2: Elements of an I-Smart Research Question

Important  Is the question important to you and others in

your field?
Specific  Is the question specific?
 Can it be distilled down further?
 Will it stand on its own?
Measurable  Is there a measurable outcome (or outcomes) for
the study?
Achievable  Can you collect the data variables necessary to
study the outcome you wish to measure?
 Do you have the resources (research team,
mentorship, funding, time, etc.) to successfully
complete your project?
Relevant (not rehashing)  Will the results add new information to the
literature?
 Will the results add to the depth and breadth of
the current literature, or will it simply restate
what is already known?
Timely  Can the study be completed in a time frame that
is reasonable for you? For the audience? For
granting agencies (if applicable)?

28

Page 28 of 38
Table 3. Examples of Kirkpatrick’s Pyramid of Educational Outcomes
Level Description
1 Reaction
2 Learning (attitudes,
knowledge, skills)
3 Behavior
4 Patient outcomes
Question
“Did they like it?”
“What do they plan to do differently based
on what they learned?”
“What did they learn?”
“How much did they learn?”
“Did it change behavior?”
“Did the behavior change affect patients?”
29
Intubation Simulation Example
 Survey participants on the usefulness of the
intubation simulation session using a 5-point
Likert scale.
 Survey participants on self-assessed comfort
with intubation using a 5-point Likert scale.
 Attitudes: Survey participants on whether they
feel all pediatric residents should be competent
at intubating neonates prior to graduation.
 Knowledge: Test participants on medical
knowledge of indications, contraindications,
risks, benefits, and mechanics of intubation
(choosing appropriate sized tube, blade;
landmarks, etc.) before and after intubation
simulation sessions.
 Skills: Compare rates of successful first-attempt
intubation on mannequins for participants in
intubation simulation sessions compared to
nonparticipants.
 Determine rates of successful first-attempt
intubation on neonates for participants in
intubation simulation sessions compared to
nonparticipants.
 Determine differences in morbidity/mortality of
neonates intubated by participants compared to
nonparticipants.
Page 29 of 38
Table 4: Common Quantitative Study Designs Used in Medical Education Research
Study Goal Study Type Example Research Question Example Methods Study Advantages Study Disadvantages
Describe/Explore Descriptive, What are the most common  Cross-sectional survey  Provides  As data collected at
a group or Quantitative facilitators and barriers affecting administered to residents descriptive data one point in time,
phenomenon pediatric resident comfort with  Presents preset choices of barriers from one point in often relies on recall
performing neonatal resuscitation and facilitators which residents time of information which
from the point of view of the have to select or rank  Less demand on is subject to bias
residents?  Answers will include percentages, resources as does  Cannot demonstrate
rank order lists, counts not require causality
follow-up

Test a hypothesis Cohort Study How likely is it that residents exposed to  Follow one group of residents  Takes advantage  Can only demonstrate
(Explanatory) a neonatal simulation-based curriculum longitudinally after exposure to of naturalistic associations, not
will chose a career in neonatology? the simulation-curriculum to see setting causality
how many go on to choose a  Avoids learning
career in neonatology from pretest
Can compare rates to a cohort of
residents not exposed to the
curriculum

Intervention only, Does introduction of a novel,  Conduct a pretest of residents  No need for  Cannot attribute any
pre/posttest design (a simulation-based neonatal resuscitation prior to exposure to new control change/gains to
quasi-experimental program involving residents and curriculum assessing resuscitation  Can demonstrate intervention alone
design because of the interprofessional staff improve resident skills (such as number of attempts change/gains (time, practices, and
absence of a resuscitation skills and ability to work in per successful intubation, time to other educational
randomized comparison interprofessional teams in the delivery successful intubation) and experiences may also
group) room? teamwork skills (using a validated factor in)
teamwork assessment scale)
 Deliver curriculum
 Administer same assessments
post-curriculum exposure to
assess change in performance

Pre/post design with Does introduction of a novel,  Pre-intervention and Post-  Able to compare  Need to control for
nonequivalent parallel simulation-based neonatal resuscitation intervention outcome data on two to control; yet baseline differences
groups (a quasi- program involving residents and groups: Control and still more feasible  Impact of different
experimental design interprofessional staff improve resident Intervention non-randomized than randomized sites/times
because of the absence resuscitation skills and ability to work in groups. study  Subject to immersion
of assignment of interprofessional teams in the delivery  Conduct a pretest of all interns and ecological effects
individuals to study room better than a traditional neonatal assessing resuscitation skills
groups) resuscitation program? (such as number of attempts per
successful intubation, time to

30

Page 30 of 38

Does introduction of a novel,
simulation-based neonatal resuscitation
Equivalence parallel program involving residents and
groups pre-post design interprofessional staff improve resident
(an Experimental resuscitation skills and ability to work in
design characterized by interprofessional teams in the delivery
Random Assignment of room better than a traditional neonatal
individuals to study resuscitation program?
groups)
successful intubation) and
teamwork skills (using a validated
teamwork assessment scale)
 First three blocks of interns get
novel curriculum, second three
blocks of interns get traditional
neonatal resuscitation program
 Conduct similar post-curriculum
assessments and compare results
for the two groups
 Random allocation for control  Reduces  Difficult to do
and intervention group. allocation bias  Resource intensive
 Conduct a pretest of all interns (minimizes  Does not address non-
assessing resuscitation skills baseline uniform intervention
(such as number of attempts per differences)  Intervention “bleed”
successful intubation, time to   Subject to immersion
successful intubation) and and ecological effects
teamwork skills (using a  Education Ethics
validated teamwork assessment concerns
scale)
 Use a random number generator
to allocate interns to two
groups: one group receives
traditional neonatal resuscitation
program at orientation, the other
group receives the novel
simulation-based curriculum
 Conduct similar post-curriculum
assessments and compare results
for the two groups

31

Page 31 of 38
 Table 5. Common Statistical Tests Used in Medical Education Research

What are you trying to Example Question Statistical How should results
determine? Methods be reported?
Summary values for a random What is the average number Estimated Mean (95% CI) and
variable with a bell-shaped of patients that third-year mean and standard deviation
distribution medical students take care standard estimates
of on the general inpatient deviation
wards? from sample
Midpoint value of a rank- What is the median medical Median Median (often
ordered list –minimizes school educational debt of accompanied by the
influence of extreme values; pediatric residents? minimum and
useful for ordinal data maximum value
and/or the 25th and
75th percentiles)
Most common value – useful for What is the most common Mode Mode
nominal or ordinal data subspecialty pediatric
residents enter after
residency?
Compare observed vs. expected Is the study population of Chi-Square p-values
values – categorical variables pediatric residents similar tests
to all pediatric residents in
the United States in regards
to gender?
Compare means of 2 Does an interactive web- Unpaired T- Differences in
independent groups – data based module on EKG test means (95% CI) and
normally distributed interpretation improve p-value, possible
residents’ ability to adjusted for multiple
accurately interpret EKGs comparisons
compared to a lecture on
EKG interpretation?
Compare means of 2 paired Does an EKG module Paired t-test Differences in
groups (e.g., pre- and post-test) improve residents’ ability to means (95% CI) and
– data normally distributed accurately interpret EKGs? p-value, possibly
adjusted for multiple
comparisons
Compare means of 3 or more Is there a difference in Analysis of (Adjusted) mean
groups medical school debt for Variance differences (95% CI)
residents who choose to (ANOVA) or and p-value,
practice in rural, urban, or multiple possibly adjusted for
suburban areas? regression multiple
comparisons
Correlation – data normally How well do resident self- Pearson Correlation
distributed (parametric) assessments of intubation product- coefficient and 95%
skills correlate with faculty moment CI
assessment? correlation

32

Page 32 of 38
 coefficient
Correlation – data not normally How well does resident Spearman’s Correlation
distributed (non-parametric) performance on the In- rank coefficient with 95%
Training-Examination correlation CI
correlate with their coefficient
performance on the
American Board of
Pediatrics certifying exam?
Association – interval and What are factors associated Linear Regression
ordinal data with USMLE Step 1 regression coefficient with 95%
scores? CI
Association – binary data What are factors associated Logistic Odds Ratio with
with passing the American regression 95% CI
Board of Pediatrics
certifying exam on the first
attempt?
CI: Confidence Interval

Interval – Data where the difference between two values is meaningful (i.e., Age)
Ordinal – Data where there is a sense of order, but consecutive values may not be
equally spaced (i.e., Likert scales: Strongly Disagree – 1; Disagree – 2; Neither agree nor
disagree – 3; Agree – 4; Strongly Agree – 5)
Nominal – Categorical - Data in which there is no inherent order (i.e., cardiology,
pulmonary, general pediatrics)

33

Page 33 of 38
Appendix 1: Educational Research Scholarship Guide/Timeline

1. Clear Goals (I-SMART) - Specific Aim: What are you trying to do?
a. Important, interesting
b.Specific, simple to
understand
c. Measurable outcome
d.Achievable
e. Relevant and not
rehashing
f. Timely
What conceptual
framework(s) are you
utilizing?
2. Adequate preparation – Are you ready to do this project? Deadline
a. Literature review –
Sources, keywords

b.Acquire necessary skills

c. Acquire necessary
resources
(i.e. collaborators,
statistical support, etc.)
d.IRB Considerations and
Submission
e. Registration in study
registry (e.g.
clinicaltrials.gov)
f. Selection of relevant
reporting guidelines
(from EQUATOR-
NETWORK)
3. Appropriate methods – How are you going to do it?
a. Study Design, including
sample size justification
and selection of
measurement approaches
and observation schedule

b.Analysis/Evaluation

4. Significant results – So

34

Page 34 of 38
 what?

5. Effective presentation – How will you disseminate your work?

a. Public dissemination –
publication, workshop,
presentation? Where?
b.Peer review
c. Platform on which others
can build
6. Reflective critique – How will you improve upon your work? Plan Do Study Act
(PDSA)
a. Critically evaluate your
work
b.Compare your findings
with prior scholarship
c. Discuss limitations of
your work
d.Discuss next steps

35

Page 35 of 38
Appendix 2: Checklist for Authors Prior to Educational Manuscript Submission
Title/Abstract
1. Title is clear and representative of content
2. Abstract concisely describes study and key findings
3. Conclusions in abstract are justified given information provided in abstract
4. All information provided in abstract are presented in text
5. All information in abstract/ text/figures/tables are consistent
Introduction
1. Builds a convincing case why this problem is important with literature
review
2. Identifies gaps in literature and addresses how this study will fill the gaps
3. Conceptual framework is explicit and justified (and/or in Discussion)
4. Specific aim of the study (and hypothesis where applicable) is clearly
stated
Methods
For ALL Studies
1. Research design appropriate to address research question
2. Research design clearly stated (i.e., cross-sectional cohort study)
3. Methods clearly described in sufficient detail to permit study to be
replicated
3a. Study population (sampling, selection bias)
3b. Study intervention (objectives, activities, time allocation, training)
3c. Study instrument validity evidence (instrument development, content,
preparation of observers/interviewers/raters, scoring method, psychometric
properties)
3d. Study outcomes clearly defined (and high on Kirkpatrick’s pyramid –
may be inversely related to level of innovation, with less innovative ideas
requiring higher outcome levels)
4. Data analysis appropriate for research design and research question
5. Data analysis procedures clearly described in sufficient detail to be
replicated
6. IRB approval/exemption and consent clearly stated
For Quantitative Studies:
1. Study is generalizable due to selection of participants, setting, educational
intervention/materials (external validity – less innovative studies require
higher generalizability with more sites, etc.)
2. Potential confounding variables addressed and adjusted for in analysis
(internal validity)
3. Statistical tests appropriate. Effect size, functional significance discussed
when appropriate. When making multiple comparisons, adjustment for
significance level for multiple tests/comparisons are considered.
4. Power issues are considered in studies that make statistical inferences
(particularly if results not significant)
For Qualitative Studies:

36

Page 36 of 38
 1. Study offers concepts or theories that are transferable to other settings and
methods described in sufficient detail (setting, sample)
2. Philosophical framework clearly stated (i.e., grounded theory)
3. Study design incorporates techniques to ensure trustworthiness (i.e.,
triangulation, prolonged observation)
4. Characteristics of the researchers that may influence the research are
described and accounted for during data collection/analysis
5. Describe how members of the research team contribute to coding,
identifying themes, and/or drawing inferences (dependability,
confirmability)
For Mixed-Methods (Quantitative and Qualitative) Studies:
1. Justify use of mixed-methods (Study must do justice to both methodology)
2. Justify order of quantitative and qualitative study
Results
1. All results are presented and align with study question and methods. All
results are presented in Results section (and not in other sections)
2. Sufficient data is presented to support inferences/themes
3. Tables, graphs, figures used judiciously to illustrate main points in text
Discussion
1. Key findings clearly stated. Conclusions follow from design, methods,
results
2. Findings placed in context of relevant literature, including conceptual
framework. Alternative interpretations of findings are considered as
needed
3. Study Limitations and Study Strengths discussed
4. Practical significance or implications for medical education are discussed.
Guidance for future studies is offered.
References
1. Literature review is comprehensive, relevant, and up-to-date
2. Ideas and materials of others are appropriately attributed (No plagiarism)
Final Journal Check
1. Study is relevant to mission of journal and journal audience
2. Author guidelines are followed (including word count)
3. Prior publication(s) by author(s) of substantial portions of the data are
appropriately acknowledged
4. Conflicts of interest are disclosed
5. Text is well written and easy to follow
6. Manuscript is well organized
Note: This table was adapted from AAMC Review Criteria for Research Manuscripts, 2nd Edition. Eds:
Durning SJ, Carline JD. 2015.

37

Page 37 of 38
38

Page 38 of 38