ORIGINAL CONTRIBUTIONS ARTICLES What We Say and What We Do: Self-reported Teaching Behavior versus Performances in Written Simulations among Medical School Faculty SANDEE L. HARTMAN and MARC S. NELSON, M.D., M.A. ‘Abstract—Much of the research in medical education is per- formed through the use of self-reporting questionnaires. Al- though this can be a valid way to conduct research, little thas been made to examine the fit between what teachers report and what they actually do. In an attempt to investigate he authors interviewed 47 preclinical faculty members at shools in 1990. The faculty members were lf-eport in four ie knowledge or abil cred important for students to develop; factors their curriculum development; and sources from which they sought pedagogical assistance. This was followed by four written simulations that ex: discussions, course design, lecturing, and test construction. In addition, the authors specifically sought to identify any differ- ences in teaching philosophy and practice between those pre- clinical faculty who were physicians and those who were not, a8 ‘well as any interinstitutional differences. Although in certain instances there was a strong correlation between self-reporting ‘and performance as measured by a simulated teaching scenario, ‘in most instances the correlation was quite low. Consequently, ‘the authors suggest that researchers must be careful in their use -reporting as a means of assessing teaching behavior, and that whenever possible, researchers should observe the teachers they are studying. Acad. Med. 67(1992):522-527. Pedagogy is complicated, yet re- searchers who study medical educa- tion and teaching in medical schools seem to have spent little time think- ing about how we know what we know in medical education. The way in which we obtain our information about teachers and teaching hat portant implications for the way in which we use that information and for the credibility we assign to it, Prior research on teaching has em- Phasized self-reporting question- naires and surveys as a means of learning about educational practices ‘among medical school teachers.2-* Al- Ms, Hartman is a fourth-year medical stu- dent; and Dr. Neleon is esistant professor, Di- vision of Emergency Medicine; both are at ‘Stanford University School of Medicine, Stan- ford, California. ‘Correspondence and requests for reprints should be addreteed to Dr. Nelson, Division of ‘Emergency Medicine, Department of Surgery, H-129, Stanford University Hospital, Stan: small-group though useful information may be ob- tained from such data, how this infor- mation relates to actual performance in a classroom or other teaching be- havior is unclear. Prior research in other branches of education has dem- onstrated that there is frequently lit- tle correlation between what teachers self-report and what they do. ‘An exhaustive search of the medi- cal literature failed to uncover any empirical study examining this issue. This prompted us to study whether self-reporting by medical school fac- ulty correlated with their behaviors. We asked faculty members to com- plete a self-assessment questionnaire and then work through four teaching simulations that we hoped would re- flect actual behaviors in a classroom setting, What we have done is not wholly adequate. Teaching simulations may or may not reflect reality. We have tried to evaluate a relationship be- ‘tween a widely used method of as- sessing teaching in medical education and one that is infrequently used. The latter has many limitations, but nevertheless provides insight that will hopefully be useful to those people in- terested in better understanding the process of medical education. In addition to studying self-report- ing and behavior, we sought specific differences between those faculty members who were physicians and those who were not, because histori- cally research has suggested that this basic division between faculty is one of the fow variables that will reliably distinguish the characteristics and instructional practices of medical school faculty. We also examined two different types of academic insti- tutions to see whether there were any interinstitutional differences. ‘Method In 1990 we selected 50 preclinical fac- ulty members at two large western [ACADEMIC MEDICINEmedical schools to interview. One ‘medical school was a Carnegie-classi- fied Research University I; the other was a Research University Il. (The Carnegie classification groups insti- tutions into categories based on the levels of degrees they offer and the comprehensiveness of their programs. Research I universities receive at least $33.5 million in federal support annually; Research II universities re- ceive between $12.5 and $33.5 mil- lion.) All faculty were randomly cho- sn from anatomy, neurobiology, physiology, microbiology, and immu- nology departments. All interviewees were guaranteed anonymity. Three faculty members declined to be interviewed because of time constraints. Of the remaining 47 faculty, 17 were professors, 16, ass0- ciate professors, 11, assistant profes- sors, and three, instructors. Bight were women and 39 were men. Ten were 55 years old or older, 15 were 45-54 years old, 21 were 35-44 years old, and one was 25-34 years old. Ap- proximately half the faculty provided direct patient care: 51% had M.D.s, 49% had Ph.D.s, and 13% had both. ‘Twenty-seven faculty were from the Research I institution and 20 were from the Research I institution. Each interview lasted —approxi- mately one hour and was conducted in a single session. Bach faculty mem- ber was interviewed separately. The interviews consisted of two parts. The first part was a questionnaire with ‘multiple-choice questions that was developed by the authors (and is available on request). Several ques- tions were used to cover each issue. Demographic information about the teachers was collected. Each faculty ‘member was then asked to rank items in four categories that the authors considered important for evaluating teaching philosophy and perform- ance: (1) knowledge or abilities that the faculty member considered im- portant for the preclinical medical student to develop during his or her class, (2) interactive skills that the faculty member used effectively in teaching, (3) factors that influenced the faculty member in developing a curriculum, and (4) sources from which the faculty member sought as- Volume 67 + Number & # AUGUST 1992 Written Simulations Course Segment Design In this simulation, a course director asks a teacher to design and deliver nine hours of instruction dealing with his or her specialty. This module explores faculty members’ approaches to constructing a seg- ment for an introductory course. It examines instructors’ levels of understanding of the values of medical problem solving and student, self-directed learning. It looks at how teachers value student feedback and recommendations from colleagues. Small-group Discussion ‘This simulation focuses on the teacher of a small discussion group who is approached by three students who are concerned they are not learn- ing efficiently. It addresses how the teacher deals with and reflects on decisions he o she faces when responsible for a series of instructional decisions with a small group of students. Itexplores teachers’ attitudes toward group process, their views of their role as facilitator, and their ‘emphases on student interaction and peer respect. Lecturing In this simulation, a newly appointed faculty member asks a more senior instructor to critique and advise him on his lecturing style. This module explores the presentation of a subject area by a teacher to a large group. It looks at the components of effective delivery and at teacher-student interaction in large classroom settings. Tt examines sources teachers consult in analyzing lecture content and style. ‘Test Construction In this simulation, a teacher takes responsibility, with the assistance of two other faculty members, for designing a final examination. This module explores the kinds of decisions medical school teachers make when designing a final examination. It looks at who should be involved in writing test questions and the value of consulting an educational specialist. It examines the option of developing an evaluation instru- ment that, as opposed to emphasizing competition, encourages a col- laborative learning environment among students. tance in and advice about estab- lishing teaching objectives and methods. The faculty were asked to rate the value of each teaching strat- egy and the frequency and effective- ness of use of the strategy. A five- point scale was used, with a score of 1 indicating very important, frequent, or effective; 3, a neutral response; and 5, not at all important, frequent, or effective. ‘The second part of the in consisted of writte i simulations were originally created as part of a project to assess the ways in which teaching faculty approached problems in instruction and instruc tionally related issues. They were later used as self-assessment tools by medical school faculty to aid in learn- ing about effective pedagogy. We used them as models to assess behavior. Although direct teacher observation would have been the ideal way to study behavior, it was not logistically possible to observe 47 faculty mem- bers for long enough periods of time, and in similar enough situations, to explore all the issues examined here. ‘The simulations, which are further explained in the boxed list, were used to examine four areas of teaching: course segment, design, small-group discussions, lecturing, and test con- 320 *% Respondents sateepates tame SSS sever Figure 1. Teachers’ self-reported behaviors versus their behaviors in four simulations, In 1990 the authors interviewed 47 preclinical faculty at two ‘medical schools: the teachers self-reported their beliefs and actions on a ques- tionnaire using a five-point scale, and in each simulation they chose strategies at selected decision points. For each simulation, the left bar shows the mean percentage of teachers who self-reported scores of 1 or 2 (very important or important, very offective or effective, very frequent or frequent) on the various questionnaire items related to the simulation, and the right bar shot the mean percentage of teachers whose strategy choices at the various de sion points corresponded to their scores of 1 or 2 on the questionnaire items, struction, In each simulation, in- structors’ decision-making strategies were determined from their choices at, selected decision points in an instruc- tional problem. ‘The teachers were asked to choose strategies at the deci- sion points that reflected what they ‘would actually do in these situations, not what they thought the ideal teacher should do. Statistical analysis was performed using the Mantel-Haenszel test.° This, test provides a measure of association between self-reporting and behavior of teachers. It computes a different quantity for each question separately ina 2 X2contingency table and then combines the results for a cumulative analysis of each simulation. All data were analyzed using both positive and negative self-reporting. 5m Results Al 47 faculty members who were in- terviewed completed both the ques- tionnaire and all the simulated teach- ing scenarios. Our data show little relation between the teachers’ self- reporting and their behaviors in the simulations. It should be noted that this is a descriptive, analytic study ‘with a focus on qualitative, not quan- titative, analysis, Although we used statistical analysis, where appropri- ate, to complement the qualitative aspect of the study, our sim was to stimulate thoughtful discussion. No significant interinstitutional differences were found when compar- jing the correlation between self- reporting and behavior in each simulation. Figure 1 displays the general pat- tems of the data: for each simulation, the left bar shows the mean percent- age of teachers who self-reported soores of 1 or 2 (very important or important, very effective or effective, very frequent or frequent) on the various questionnaire items _re- lated to the simulation and the right bar shows the mean percentage of teachers whose strategy choices at the various decision points corresponded to their scores of 1 or 2 on the ques- tionnaire items. Detailed analysis of the data is presented in the following four sections. Course Segment Design Of the 47 teachers, 42 (89%) chose to address the issue of how the indivi ual classes would be taught: (eg, lee- ture versus discussion) rather than to address solely the content of the classes. Of those 42 teachers who chose a particular instructional fo mat, 41 self-reported that “skill in problem solving” is very important; however, only five of the 41 chose to pursue a problem-based instructional approach in the simulation. Of the 42 teachers who addressed the issue of instructional format, 39 self-reported that ‘skill in self-directed learning” is very important, but only six of the 39 chose to incorporate self-directed learning in their course designs. Of the 32 (68%) of the 47 instruc- tors who self-reported that they very frequently seek advice from other fac- ulty, 14 chose to consult with other faculty during the simulation. Of the 18 (28%) of the 47 instructors who said that they very infrequently seek advice from colleagues, seven did not consult other faculty in the simula- tion, Of the 47 instructors overall, 21 (45%) consulted with other faculty. In the simulation, 13 (28%) of the 47 teachers chose to meet with prac- ticing physicians to identify concepts and information they thought clini cally essential for medical pract ‘The numbers were similar for the instructors who were themselves phy- sicians (7, 29%) and for those 23 in- structors who were not physicians (6, [ACADEMIC MEDICINE26%). Of the faculty who were not physicians, four (17%) self-reported that they’ very frequently consult physicians in setting course objec- tives; of these four, three actually de- cided to meet with physicians during the simulation. Ten (42%) of the 24 instructors who were physicians said that they very frequently seek out other physicians; however, of these only six de r physicians during the module. ‘Small-group Discussion Of the 47 instructors, 27 (57%) chose to talk in depth with the dissatisfied students (67% of these teachers were physicians, while 38% were not). Of the 27 teachers who chose to talk in depth with the dissatisfied students, 17 self-reported that they are very effective in facilitating small-group discussions; however, of these 17 teachers, only ten asked the students about their prior experience in small groups, and only 11 asked the stu- dents about their attitudes toward the group proces: Of the 47 teachers, only five (11%) chose to talk in depth with the “‘satis- fied” remainder of the class. Of those five, two self-reported they were fective in facilitating small-group dis cussions, yet in the simulation nei- ther of the two asked the satisfied students about their attitudes toward small-group leaming, and only one asked about prior experience in small groups. Of 27 teachers who chose to talk in depth with the three dissatisfied stu- dents, 13 self-reported that the “abil- ity to respect other students” is a very important attribute for their students to develop. However, only five of these 13 teachers asked the students in the simulation about their feelings of identity with other students. Of the five teachers who chose to talk with the satisfied students, three noted that the ability to respect other stu- dents is very important, but only one asked these students about their feel- ings of identity with respect to the other students. ‘About one-fourth of the 47 teachers Volume 67 + Number 8 # AUGUST 1992 ended this simulation by recommend- ing that the dissatisfied students transfer out, while half recommended hat they stay and offered to exist em. Lecturing In this simulation a teacher asks for help in improving his lectures. The person giving advice has four options: attend the next lecture, talk in depth with the teacher requesting help, talk with the students, and/or read over the lecture notes. Of the 47 teachers, 19 (40%) chose only one option (most commonly, attending the lecture). Only two of the faculty chose all four options. It is interesting that al- though the reason the colleague came for help was that the students were complaining, only 17 (36%) of the faculty chose to talk with the students. Of the 24 physicians, 16 (67%) chose to talk in depth with their col- Teague. Of these 16, four self-reported they very frequently speak with fac: ulty who are not physicians about et tablishing teaching objectives and methods. However, only two of the four faculty members asked the lec- turer if he had consulted with other faculty. Of the 23 teachers who were not physicians, seven (30%) decided to talk in depth with their colleague. Of the seven, one self-reported he very frequently speaks with col- eagues who are physicians about teaching methods and course objec- tives. None of these teachers asked the colleague if he had talked to other faculty. Overall, nine (39%) of the faculty who were not physicians self- reported that they very frequently talk with physicians about course ob- Jectives; however, only six of the nine asked their colleague in the simula- tion how the subjects he taught re- lated to the knowledge and skill needed by practicing physicians, It is interesting thet regardless of 5 reporting, only eight (38%) of all 24 physicians and only three (13%) of the 23 other teachers asked how the lectures that were being given related to actual medical practice, OF the 45 (96%) of the 47 instruc- tors who decided to attend their col- league's next lecture, 17 self-reported they are very effective in noticing student comprehension and adjusting the pace of their presentations. Yet, only nine of the 17 teachers took no- tice during the simulation of their col- league's use of formal and informal feedback from the students on the progress of the lecture. Of the 47 teachers, 41 (87%) self-reported they are effective in maintaining students? interest and attention; of the 41 teachers, 39 monitored such behavior during the simulation. ‘Test Construction In this simulation, the teachers were asked to develop a final examination, Of all 47 teachers, 36 (77%) chose a norm-referenced approach; the re- mainder chose a criterion-referenced approach. Only 13 (28%) of the 47 instructors chose a grading plan before construc ing the examination. Of these, eight self-reported that “ability to learn in a noncompetitive manner” is a very important. skill for their students. Yet, all of these instructors decided ‘on @ competitive structure for their test during the simulation. Only four (8%) of the 47 faculty I-reported that they very fre- ‘quently seek advice from an educa- tional specialist; however, only two of these actually did so during the simu- lation. Overall, 17 (36%) of all 4 structors sought assistance from an educational specialist, Of the 17 in- structors, ten were physicians, Discussion Before focusing on the individual sce- narios, we think it important to cuss the general issue of self-report- ing versus behavior in the study of teaching and teachers. Although the validity of self-reporting as a means to study teaching has been questioned 335‘Traditionally, information about teaching has come from one of three areas. The first source is the students, either in the form of direct observa- tion," or indirectly, by monitoring such things as student achievement. Second, the information may come from an observer. Finally, the data may come from the teachers them- selves. Although self-reporting is widely ‘used in research on medical educa- tion, we are unaware of any study that has focused on the accuracy of self-reporting in assessing teachers! behavior. In addition, studies that have been performed outside medical education tend to have focused on ac- tual observation as a means of vali- {-reporting. We opted for a different approach (the use of simul tions) because the use of observers is laden with problems, Prior studies often counted on only two or three observations, which may or may not be typical. It is frequently unclear how qualified the observers are, and rarely are data presented that show the degree of interobserver agree- ‘ment, Finally, practical reasons, such as time, money, and personnel, often make observation impossible. ‘Our data show little relation bi tween self-reporting and behaviors in simulated teaching scenarios. This is important because even though our teaching simulations may not accu- rately portray reality, we believe they are one step closer’ than question- naires or surveys. If, in fact, there is a large discrepancy between self- reporting and behaviors in simula- tions, we wonder how much greater the discrepancy would be between self-reporting and actual behaviors. ‘Thus, one must question any research that relies solely on self-reporting as the means of assessing teaching. Still, because of its ease, this par- ticular research methodology is un- likely to disappear; so rather than abandon self-reporting, future re- search should probably emphasize why reports are not accurate and how we can increase their accuracy. This type of research has been performed By decreasing the social desirability 526 of responses, for example, and em- phasizing accuracy rather than the manifestation of certain types of be- havior, it might be possible to in- crease the validity of self-reporting, Space precludes a lengthy discussion of other techniques, although some merit brief mention. By increasing self-awareness" and by repeated use of the same self-reports," itis possible to achieve a higher correlation be- ‘tween self-reporting and behavior. We tun now to a more specific analysis of each case scenario to fur- ther elucidate the large discrepancy botween what the medical faculty said ‘they did and their behaviors in the simulations. Course Segment Design ‘To us the most significant discrep- ancy between self-reporting and be- havior centered on problem solving and self-directed learning. Most teachers thought that problem solv- ing and self-directed learning were very important, yet only a very small percentage of these instructors inte- grated these skills into their course designs, Phrases such as “problem solving” and “self-directed learning” are part of the popular pedagogical vocabulary. Perhaps, although these words are in vogue and, on the sur- face, highly respected, most teachers do not have the educational back- ground to understand and implement them. Quite frequently, eatchphrases are tossed around without any true understanding of their meanings. Therefore, it makes sense that teachers identify the phrases “prob- Jem solving” and “self-directed learn- ing” as very important; yet, when confronted with practical situations in which these practices are not ex- plicity labeled, they are unable to use One ‘teacher commented, ‘Aren't homework, problems ‘prob- lem-based’ learning?” Although it seems logical and pro- ductive to consult other faculty, as supported by the self-reported data, ‘only about half of the faculty actually id so in the simulation. This is un- fortunate; the reasons are no doubt complex, but probably relate to time constraints, ego barriers, physician- researcher “turf” battles, and the general sense of in which most teachers operate. It is especially disturbing that only about one-fourth of the teachers ‘chose to consult practicing doctors. In our opinion, medical school faculty need to consult practicing physicians, even if they are “experts” in the ma- terial being taught. ‘Small-group Discussion ‘The use of small-group discussions as an adjunct to or in place of regular lectures has become popular, al- though for a variety of reasons it has yet to receive widespread acceptan Dissatisfaction with this type of teaching modality may come from the students, the teachers, or both groups, We were struck by the small number of the faculty members (63%) who chose to talk with the dissatisfied students and by the fact that almost no-one chose to talk with the rest of the class. Students, especially adults, provide valuable feedback, and al- though most of the faculty’ members self-reported that, such feedback is important, when given an appropriate opportunity to seek it, they rarely did. Lecturing Although itis well known that skill in teaching has never been a require- ment for joining or remaining on a al school faculty, we were non theless disappointed at how the f ulty members chose to assist: their troubled colleague. Tt was especially disheartening that, although the reason the col- league sought assistance was that stu- dents had complained, only one-third of them chose to talk with the stu- dents. Even among those reporting that they very frequently sought ad- vice from students, only one-third asked their colleague if he had talked with the students, ‘Test: Construction ‘This ecenario interested us because of the frequently expressed dosire to [ACADEMIC MEDICINE.‘make medical school a less competi- tive environment. In many schools, this has been partly achieved by adopting a pass—fail method of grad- ing. It is clear, however, that faculty still seek a means of comparing stu- dents with each other rather than de- cciding whether there is a minimum ‘amount of information that is needed by all students. This was reflected in our study by the overwhelming choice of a norm-referenced test as opposed to a criterion-referenced test as the means of assessing students. Ina sense, this is not surprising, as most: teachers lean how to teach from their experiences as students. ‘Having grown up in a competitive en- vironment and with no one available to guide them, and with little incen- tive to change, it is logical that the faculty would choose @ norm-refer- enced approach, Few, if any, faculty members have had any formal training in psycho- metrics, so we were ly pointed that less than 10% of them said they very frequently sought vice from an educational special Although it is possible that they think educators would be unable to h them, research does not support this belief Most likely they remain in ignorant bliss of how difficult it is to make up a good examination. Caveats ‘A number of sources of error that are pertinent to our study have been identified in the self-assessment- behavior literature." First, the ques- tionnaire itself may have been flawed, If the respondent does not, under- stand what is being assessed, he or she may not be able to respond accu- rately. Our questions were straight- forward and the same information was elicited in a variety of ways so to minimize this potential problem. Another source of error can be the respondent him- or herself. Most peo- ple are somewhat unreliable, depend- ing on the task. In addition, there may be a scale-respondent mis- match. The scale may, for example, Volume 67 + Number 8 # AUGUST 1992 ‘assess something the respondent con- siders unimportant, or the respon- dent may not be motivated or may not take the situation seriously. In addition, self-evaluation is not emphasized in medical training,? and this lack of practice may partially ex- plain why our faculty members did such a poor job of self-reporting. In ‘our study we hoped to obviate these problems by our relatively large sam- ple size (for a qualitative study), al- though it should be noted that even though we interviewed nearly 60 fac- tulty members, in some cases the sam- ple size was still not large enough for us always to be able to do appropriate statistical analysis. We may have been unable to detect certain differ- ences, for example, between faculty who were practicing physicians and those who were not, because the study lacked sufficient power (beta error). ‘We chose the sample size based on the largest number of faculty mem- bers time and resources would allow us to interview. Finally, and perhaps most impor- tant, we’ chose simulations as our representation of reality. If these teachers had actually been observed, they might have been found to per- form quite differently. We would maintain, however, that simulations are closer to reality than question- naires. We intend, and we would en- courage others, to do further research to investigate how closely such simu- ions represent actual practice, Conelusion Because we found a poor correlation between what medical school faculty self-report and how they perform in simulated teaching scenarios, we be- lieve that our results tend to cast doubts on the accuracy of any re- search that relies solely on self- reporting as a means of obtaining in- formation about teaching practices. Our results also provide a powerful argument for the importance of addi- tional data, such as observation and student feedback, in the assessment of teaching. study was fonded in part by the Stanford Mont Sttolare Propane The sathor Che De. Abba Sharma of the Division of Biostatie: tice, Stanford University School of Medicine, for hor holp with th statistical analy, References 1. Creswell, J. W., and Kuster, C. G. Survey Practices ‘in Dental Education. J. Dent Beduc, 47(1989)678- 680, 2. Arnold L, Willoughhy,. I. and Calkins EV. SelfRvaluation’ in Undergraduate ‘Medical Education: A Longitudinal Per- ‘Reports of Behavior Changes by Partic- pants after a CME Course. J. Med. Béue 61(3986):579 584, 4, Jason, Hand West 5. Hook,’ C. M, and Rosenshine, B. V. Accuracy of ‘Teacher Reports of Their Chasstom Behavior. Rev. Educ, Res. 49(1979)1-12. 6. Mantel, Nand Haensel, W. Statistical Aspects ofthe Analysis of Data from Ret- roopective Stay of Disease Natl Cancer Int, 22050): 710- 148, 1. Kena 8M, and Burns, P. Teacher’ Ac- curacy’ in Self Reporting. about. Insos- fiona’ Practices Using Focused Se por Tnventon. J. Bue ex 79 (a310):208-200, ‘8 Manfredo, Mand Shalby, B. The Baer of Using Salt Report Measures in Test of ‘Atitode-Eshavior Relationahipe, J. Sox Pryohol 126(1986/791-743, 9, MeKeachia, W. J, and Lin, YG. A Note‘on Vality of Student Ratings of ‘Teachers Bdue. Res. @ 411878)45-d7- 10, Meighan, 1. Children’s Judgements of the Teaching Performance of Stent ‘Teachers Bde. Rex. 27(1074582-€0, 1, Gittons, F. ¥, Solf-Atention and Sele Reports "The “Weridcalty” Hypothesis J. Person, (1989517 842 1, Newfeldy 3. Ascuracy of Teacher Selt aports: Reports and Observations of Spe- cite Classroom Behaviers. J. Educ. Re ‘ra(iseo 78-82 13, Nelson, M.S, Clayton, BL, and Moreno R. How Metical Sehool Faculty Regard Educational Revearch and Make, Peda- foaial "Decisions. Acad. Med. 65 G500) 222-128, In Behavior. Paychol Rev. B1(19%4): 506-520. 16, Rosenberg, M, Society and the Adolescent ‘Self-Image. Princeton, New Jersey: Prince ton University Pres, 1968. su