European Journal of Dental Education ISSN 1396-5883

Introduction of an e-portfolio in clinical dentistry: staff and student views

C. Vernazza1, J. Durham1, J. Ellis1, D. Teasdale2, S. Cotterill2, L. Scott2, M. Thomason1, P. Drummond2 and J. Moss2
1 2

School of Dental Sciences, Newcastle University, Newcastle upon Tyne, UK, School of Medical Education Development, Newcastle University, Newcastle upon Tyne, UK

Keywords portfolio; assessement; undergraduate; reection; feedback. Correspondence Christopher Vernazza School of Dental Sciences Newcastle University Framlington Place Newcastle upon Tyne NE2 4BW, UK Tel: +44 19 1222 8396 Fax: +44 19 1222 5928 e-mail: c.r.vernazza@ncl.ac.uk Accepted: 27 January 2010 doi:10.1111/j.1600-0579.2010.00631.x

Abstract Introduction: An electronic portfolio was introduced for undergraduate students in the School of Dental Sciences at Newcastle University. Its introduction was evaluated in terms of both staff and student response. Methods: A quantitative-qualitative methodology was adopted. Student views were examined quantitatively using a Likert scale based questionnaire both pre- and postintroduction of the eportfolio. Staff views were examined qualitatively by the use of focus groups. Results and discussion: Findings included that the system was easy to use and it provided a large quantity of high quality data. The aim of the system to improve reection and feedback was not perceived as a benet by staff or students. The need for training was highlighted and a major disadvantage of the system was its time consuming nature. The evaluation has lead to further development of the system and continued evaluation will be important. The early, initial development of portfolios was centred around paper based systems. Increasingly now, however, electronic versions are being used, which bring the benets of customisation, better data manipulation and increased ability to share and transfer information over paper based systems (9, 10). There are, however, disadvantages associated with electronic portfolios: the time and cost of development; technical difculties in accessing and using electronic resources; and the increased time and effort required to complete electronic based records (11). There are mixed views from healthcare students on using portfolios (1216). The differences in opinion centre around: the nature of the development of reective skills (1214); and the time pressures and environments inconducive to their use (15, 16). Factors inuencing the success of portfolios include: making the purpose of the portfolio and its procedures explicit; fully integrating the portfolio with the curriculum and assessment; having a exible structure and proper support for the portfolio; and making the portfolio easy and efcient to use, were all key factors in the success of a portfolio (17). Staff perceptions of portfolio use have been the subject of much less research. This is somewhat surprising given that staff input is vital to the success of a portfolio (17). One paper (18) examining staff perceptions showed that they felt that portfolios in medical education should and could stimulate reection, but

Introduction
The exact denition of a portfolio and therefore its content and characteristics will depend on its purposes. Many portfolios have several purposes (1), including: formative and/or summative assessment; professional accreditation or revalidation; presentation, promotion or for job application; personal appraisal and personal development planning (2, 3). As the Dearing Report (4), there has been increasing interest in portfolios in higher education in the UK. This report recommended the introduction of a progress le for all students in higher education in the UK. It gave the reasons for this suggestion as: rst to provide an ofcial record of achievement; and second as a means by which students can monitor, build and reect upon their own personal development. The General Dental Council (GDC) also requires dental courses to use assessment methods which encourage reection and self-assessment and recommend that students have a portfolio of experience and achievements (5). Aside from the impetus created by the Dearing Report in the UK, the use of portfolios has increased in healthcare education, in particular because of the growing emphasis on nurturing and assessing professionalism (6, 7). Development of reective skills is a key element in the engendering of professionalism, but these skills are difcult to teach. Portfolios can be an important tool in developing a reective nature (8).
36

Eur J Dent Educ 15 (2011) 3641 2011 John Wiley & Sons A/S

Vernazza et al.

Introduction of an e-portfolio in clinical dentistry

only if the students were motivated. Others found that one of the major barriers to use by staff was the time taken to use it, particularly in the early stages following introduction of a system where users were familiarising themselves with portfolio (19, 20).

Data collection
A priority sequence model was adopted in a quantitative-qualitative order (24). First, student views were captured using questionnaires (Appendix S1). The results from these questionnaires then informed focus groups held with staff. The focus groups allowed an in depth exploration of student perceptions with the staff and an exploration of staff perspectives of the eportfolio. Student questionnaire The initial student questionnaire was administered to the 78 students in the year group studied immediately before students began using the eportfolio. A follow up questionnaire asking the same questions as well as additional questions related to the eportfolio was then administered at the end of the rst term of using the eportfolio. Both questionnaires were administered whilst students attended practical teaching sessions not related to the eportfolio pilot speciality by staff not linked to the pilot speciality. The questionnaires used a mixture of 5 point Likert and free text responses drawing on questions from a number of existing portfolio related questionnaires (9, 13, 25), as well as including new questions relevant to this specic context. The questionnaires are shown in Appendix S1. The questionnaire was piloted with a group of students in other years not using the eportfolio at the time and minor changes were made to aid understanding following this piloting. All of the students were issued a random anonymous number, which was included in both pre- and post-pilot questionnaires to allow comparison of changes in matched questions on an individual basis. Focus groups All staff involved in undergraduate clinical teaching (dental school and outreach based) except those involved in the development of the eportfolio were invited to take part in focus groups during the second term of its use. After initial responses were gained staff were selected based on fullling the purposive sampling requirements (Table 1). Four focus groups were held with two trained facilitators. A exible topic guide was developed based on ndings from the quantitative student aspect (Table 2) of the project and this evolved as the focus groups progressed. Informed consent was taken from all participants who included assurances of anonymity and an agreement to be quoted verbatim. Focus group constitution was decided upon on the basis of a priori knowledge of personality to ensure all participants would be able to contribute. In addition, facilitators encouraged participation from all members of each group. The sessions were recorded digitally and transcribed verbatim. Further focus groups were deemed unnecessary as saturation (no new themes arising) had been reached after the rst four groups.

Aim
The aim of this project was to examine the student and staff perceptions of a new eportfolio during the rst two terms of its introduction and to detail their observations regarding reection and feedback with a change from paper to electronic portfolio.

Methods
The Newcastle ePortfolio
A consortium of dental education providers formed with the purpose of developing and implementing an electronic portfolio for dentistry (9, 21). In the Newcastle undergraduate setting, the eportfolio was initially based on an existing paper version previously described (22, 23). The eportfolio was developed as a web based system with access provided via laptops connected to a wireless network to allow use in the clinical teaching environment. The most frequently used section, and therefore the rst section trialled with the eportfolio is the record of clinical activity and formative assessment. Students typically use this section several times a day, generating a large amount of data. The students and their supervising clinical tutors use it immediately after each patient contact. The student enters details of the patient seen, procedures undertaken and then grades themselves on their own performance in several domains. The tutor then discusses the self-assessment with the student, gives the student feedback on the episode and enters the nal agreed grades. The eportfolio was initially piloted in one clinical speciality, for one academic term with third year dental undergraduates attending the clinic longitudinally. This was selected as a practical contained unit which gave a representative experience of how the eportfolio would be used. The student results are based on their experience of the pilot. Minor necessary technical changes were made and the eportfolio was then rolled out to all of the major clinical specialities across all clinical undergraduate years. Staff views were gained after one academic term of this full use.

Sample
The student year initially using the eportfolio and therefore involved in this study (third year) contained 78 students. For the qualitative arm, a purposive maximum variation sample was drawn from the staff to ensure a depth and breadth of views were captured. Staff were sampled on the basis of their hierarchical position, teaching speciality and gender. The sample represented all those staff involved in teaching undergraduates and who therefore used the eportfolio, excluding those involved in the development of the eportfolio.

Data analysis
Quantitative analysis of the student results consisted of descriptive statistics and paired tests (Wilcoxon) to analyse changes
37

Eur J Dent Educ 15 (2011) 3641 2011 John Wiley & Sons A/S

Introduction of an e-portfolio in clinical dentistry

Vernazza et al.

TABLE 1. Composition of staff focus groups Number Group of staff Grade of staff 1 4 1 1 1 1 2 1 1 1 1 2 1 1 1 1 Lecturer Senior lecturer Associate specialist Staff grade Clinical fellow Lecturer Visiting practitioner Clinical fellow Senior lecturer Visiting practitioners Specialist registrar Lecturer Senior lecturer Professor

Results
Quantitative student views
Sixty of the year group surveyed (77%), completed the prepilot questionnaire and 72 (92%) the post-pilot questionnaire. 55 (71%) students completed both pre- and post-pilot questionnaires. These 55 students form the study population for the quantitative analysis with all post-pilot responses (72) considered for the free text analysis. Table 3 shows mean pre- and post-pilot scores and the mean change between these for questions reproduced in both surveys. In the post-pilot questionnaire, students found the eportfolio had good security with a mean score of 1.83 (SD 0.87) (scale is 1 security is good to 5 security is poor), although security was not a concern anyway with a mean score of 4.28 (SD 1.05) (scale is 1 security is a concern for me to 5 security does not worry me). The training given had been only slightly useful with a mean score of 2.89 (SD 1.13) (scale is 1 training given was useful to 5 training given was irrelevant). Of the complete set of 72 responders, 71 (99%) made at least one free text comment, and most students made multiple comments that fell into different themes. These themes could be broadly grouped into three key positive and three key negative areas which are shown in Table 4.

Speciality 2 Restorative dentistry 2 Child dental health

2 1 1 3 1

Restorative dentistry Oral surgery Outreach (community dentist) Restorative dentistry Child dental health

1 Restorative dentistry 1 Child dental health

TABLE 2. Outline of principal themes included in initial topic guide Theme Experience of clinical teaching Experience and comfort of using computers Technical difculties with eportfolio Specic prompts Speciality Sessions per week Computer use at home/work Use of other technologies Difculties with registration Difculties with hardware Difculty logging in Difculty navigating software Condentiality Time issues Effect on feedback./reection Security of data Changes in reective practice Changes in feedback Student monitoring Quality of record keeping Hardware changes (e.g. PDAs) Software enhancement Recording of additional data in clinical data Other uses of eportfolio system

Qualitative staff views

Four main themes emerged from the framework analysis: l Logistical considerations and application of the eportfolio. l Eportfolios inuence on learning and reection. l Eportfolios use in assessment. l Potential improvements to eportfolio. Each of these will be discussed in turn with representative quotations from the data used to support the theory developed or proposed.

Disadvantages of eportfolio

Benets of eportfolio

Application of eportfolio
Generally the system was well accepted and even those staff who were not comfortable with computers found the system intuitive and easy to use, with one such quote being If I can get it to work and it works for me then anybody can use it (Visiting General Dental Practitioner, Focus Group 3). One issue that was described was the intermittent or patchy nature of the wireless network coverage necessary to log on to the system. This limited the exibility in placement of the laptops and so caused issues with condentiality when giving feedback where laptops could only be placed in busier public areas. I dont mind the confrontation between the student and the work but it can suddenly become very public domain that this is all being discussed because it is a big chunk of machinery you cant move freely and its an open clinical area (Clinical Lecturer, Focus Group 2) Another issue raised was being able to get back into the system after logging out by using the back button of the web browser. There was a misperception that details could be changed, but even though this was not true it did allow other users

Changes to eportfolio wanted

between pre and 6-month scores. Free text was analysed by content analysis to identify major themes. For the qualitative analysis of staff views, transcripts were coded independently by two researchers and emerging recurrent themes were identied following immersion in the data. Once the themes had been agreed by the two researchers, they were used as the basis for a framework analysis (26). This type of analysis is used to organise and facilitate the inductive and iterative analysis of the data, with coded comments being tabulated by respondent under each thematic heading. Data can then be interrogated by theme or by respondent allowing further interpretation.

38

Eur J Dent Educ 15 (2011) 3641 2011 John Wiley & Sons A/S

Vernazza et al.

Introduction of an e-portfolio in clinical dentistry

TABLE 3. Mean and standard deviation (SD) of perceptions (pre- and post-pilot scores and mean change) (n = 55) Anchor points on Likert scale Easy (1) difcult (5) Useful (1) irrelevant (5) Quick (1) time consuming Easy (1) difcult (5) Useful (1) irrelevant (5) Quick (1) time consuming Easy (1) difcult (5) Useful (1) irrelevant (5) Quick (1) time consuming Useful (1) irrelevant (5) Quick (1) time consuming Mean pre-pilot score (SD) 2.67 2.37 3.28 2.50 1.94 3.02 3.22 2.19 2.74 1.28 2.25 (1.10) (1.26) (0.92) (0.93) (0.81) (0.92) (1.04) (1.03) (1.07) (0.60) (0.94) Mean post-pilot score (SD) 2.33* (0.93) 1.94* (1.07) 3.20 (1.11) 2.44 (1.09) 1.89 (1.08) 3.11 (1.06) 2.33* (0.87) 1.91 (0.98) 2.83 (1.16) 1.87* (1.05) 2.94* (1.28) Mean change (pre post) (SD) 0.33 0.43 0.07 0.06 0.06 )0.09 0.89 0.28 )0.09 )0.59 )0.70 (1.21) (1.22) (1.16) (1.41) (1.46) (1.35) (1.16) (1.38) (1.55) (1.27) (1.31)

Question I nd reection on my clinical experiences I nd keeping a record of my clinical activity I nd grading my clinical work

(5)

(5)

(5) (5)

I nd my clinical tutors grades and feedback

*Signicant difference (P < 0.05) (Wilcoxon signed rank test).

TABLE 4. Positive and negative free text student responses grouped by theme Theme Physical Positive No paper le to carry around Secure Organised data Good overview of data Better tutor feedback Ease of use Negative Availability of laptops Fears about what would happen if system broke Level of detail recordable Poorer tutor feedback Time consuming Technical problems Training insufcient

Assessment
Staff felt that monitoring of clinical activity and assessment were the main strengths of the system. They were impressed with the amount of data it was possible to record. Limited access to the data in the system and the lack of ability to generate reports, however, was seen as a major problem. Several staffs were concerned about the lack of audit of the data and the potential for under or over-recording of clinical activity. As a teacher Id like to be able to see how many stainless steel crowns have been done or what are the spread of grades over a clinical group. Im sure all the data are there but its not at that stage of development yet. Its still a bit of a sleeping giant to be honest (Clinical Senior Lecturer, Focus Group 3)

Quality

Logistics

to look at screens that the previous user had seen, showing potentially condential patient or student assessment details.

Improvements
One of the main improvements suggested was a move to Personal Digital Assistants (PDAs), especially for staff. I think PDAs would be fantastic. A PDA for each [teacher] would solve the main issues with this, because its there and its instantaneous (Clinical Fellow, Focus Group 2) Suggestions for a better induction and a protocol for the system were made on the basis that staff and students were not fully aware of the aims of the system. Id like to see a written protocol, not only of how (to use it) but also this is why were doing it. Is it for reection and development or just to feed into progress checks? (Clinical Fellow, Focus Group 3) Staff also wanted the ability to record comments on the system thereby increasing its role in feedback. They also requested to be able to record the results of summative assessments on the system.

Learning and reection

One of the aims of the eportfolio was to encourage reection. Staff felt that this aim was not being met. I would say the reection is extremely important, [but Im not sure] whether or not the electronic reection is enough or its even more articial than paper (Clinical Senior Lecturer, Focus Group 4) It is interesting to note that staff were either unaware of the aims of the eportfolio, or had different perceptions of the aims. I thought Well yeah there is a vast amount of data there but I hadnt really thought how it could be used. I actually saw it more as a kind of updated paper [data] base that would facilitate [reviews] and I nished there (Clinical Fellow, Focus Group 3) Staff felt that students were probably unaware, at this early stage, of the eportfolios aim of encouraging reection. The portfolio was thought to facilitate feedback and staff felt that the introduction of the system had raised the prole of feedback. They felt, however, that the system itself was inconducive to good feedback. It raises the banner again about feedback; my concern is that were squeezing it into a smaller hole than weve been given (Clinical Lecturer, Focus Group 1)

Discussion
The introduction of the clinical logbook and grading elements of the eportfolio evaluated here have been successfully introduced with both staff and students nding the system intuitive and easy to use. The perceived benets of the eportfolio are the quantity, quality and potential for manipulation of the data in
39

Eur J Dent Educ 15 (2011) 3641 2011 John Wiley & Sons A/S

Introduction of an e-portfolio in clinical dentistry

Vernazza et al.

terms of log book and grades, although staff felt the system was not being fully exploited in this respect. There were mixed views on the auditing of the data but most staff agreed there was a need to check student input in some form. One of the key aims of the system was, however, to aid reection and feedback and this was not perceived as a benet by students or staff during the early stages of its use. This ts with similar negative ndings of an evaluation of a similar portfolio in nal year undergraduate and rst year postgraduate settings (27). Several reasons were given for this feeling by staff: problem of feeding back to the computer rather than the student; losing feedback time because of the need to input the electronic data; problems of condentiality where the feedback had to be done at a xed point (the computer) where students tended to congregate. Several staff recognised, however, that reection can take different forms and students with different learning styles will reect in different ways and to different extents. Security was felt to be an important issue by staff, but was generally perceived to be good by both students and staff, aside from a few small misperceptions. The major problem was being able to use the back button. Although it was impossible to eliminate this problem, staff has been advised to close the web browser window when logging out which prevents this from happening. This issue shows both the importance staff place on security but also that training needs to be thorough and clear. There were several other misperceptions from the student point of view relating to the backing up of the data and the capabilities of the system. These misunderstandings reect a need for more extensive training, also noted by the students and staff. The importance of training in eportfolio acceptance has been found elsewhere (28). Alongside this training need, staff suggested that a protocol for the use of the eportfolio would help clarify the aims, purposes and potential uses of the system as well as providing a written source of training in its use. The data also highlight that logistical issues need to be carefully considered in introducing an eportfolio, in particular, the need for contextual chairside feedback. This was highlighted early in the development of the eportfolio, which is why a web based system with wireless access was developed. Even having made these considerations there were still issues of condentiality of feedback and limited wireless access. Staff suggested the use of PDAs rather than laptops as a potential solution to these problems. It was felt this would reduce time needed and overcome the condentiality problems described above. The major concern reected in both groups was the time consuming nature of using the eportfolio. There is therefore a balance between the extra time that the system takes to use on a day to day basis and the extra benets it brings in terms of better reection, more accurate and easily available records and the time saved longer term on compiling data. The issues raised formed an action plan for further development of the eportfolio and as further changes are made it is important that staff and student views continue to be canvassed so that real and perceived concerns can be addressed in order to develop a better system but also to engender a feeling of ownership.
40

Conclusions
The introduction of the log book and grading system as part of an electronic portfolio was successful. In particular, it seems that its function in data collection was positively received by both staff and students. The inuence of the eportfolio on feedback and reection is unclear and staff and students seem to fail to recognise this as an explicit aim of the system. There were several technical and practical difculties as would be expected when introducing any such system and the importance of training was highlighted. The major concern was that the system took more time, but this must be weighed against the increased benets. The evaluation has lead to further changes and ongoing evaluation is important.

Acknowledgements
The ePortfolio has been developed by the School of Medical Sciences Education Development and the School of Dental Sciences at Newcastle University. Recent developments of the ePortfolio and this evaluation have been funded by CETL4HealthNE.

References
1 Challis M. AMEE Medical Education Guide No. 11 (revised): portfolio-based learning and assessment in medical education. Med Teach 1999: 21: 370386. 2 Cotterill S, McDonald A. Principles of electronic portfolios (workshop at Ottawa Conference). 2004. Available at: http:// www.eportfolios.ac.uk/docs/fdtl4_docs/Ottawa_2004_ePortfolio_Workshop_part_A.ppt#285,1,Slide 1. [Last accessed: 22 November 2007]. 3 Butler P. A review of the literature on portfolios and electronic portfolios. 2006. Available at: http://eduforge.org/docman/view.php/ 176/1111/ePortfolio%20Project%20Research%20Report.pdf. [Last accessed: 7 January 2008]. 4 The National Committee of Inquiry into Higher Education. Higher education in the learning society. London: Her Majestys Stationary Ofce, 1997. 5 General Dental Council. The rst ve years: a framework for undergraduate dental education. London: General Dental Council, 2002. 6 Ben David MF, Davis MH, Harden RM, Howie PW, Ker J, Pippard MJ. AMEE Medical Education Guide No. 24: portfolios as a method of student assessment. Med Teach 2001: 23: 535551. 7 Brown G, Manogue M, Rohlin M. Assessing attitudes in dental education: is it worthwhile? Br Dent J 2002: 193: 703707. 8 Goldie J, Dowie A, Cotton P, Morrison J. Teaching professionalism in the early years of a medical curriculum: a qualitative study. Med Educ 2007: 41: 610617. 9 Cotterill S, McDonald AM, Drummond P, Hammond GR. Design, implementation and evaluation of a generic ePortfolio: the Newcastle experience. 2004. Available at: http://www.eportfolios.ac.uk/ FDTL4/docs/fdtl4_docs/ePortfolios_2004_Newcastle_Paperl.doc. [Last accessed: 22 November 2007]. 10 Driessen EW, Muijtjens AMM, van Tartwijk J, van der Vleuten CPM. Web- or paper-based portfolios: is there a difference? Med Educ 2007: 41: 10671073. 11 Wetzel K, Strudler N. Costs and benets of electronic portfolios in teacher education: student voices. Journal of Computing in Teacher Education 2006: 22: 99108.

Eur J Dent Educ 15 (2011) 3641 2011 John Wiley & Sons A/S

Vernazza et al.

Introduction of an e-portfolio in clinical dentistry

12 Pee B, Woodman T, Fry H, Davenport ES. Practice-based learning: views on the development of a reective learning tool. Med Educ 2000: 34: 754761. 13 Rees C, Sheard C. Undergraduate medical students views about a reective portfolio assessment of their communication skills learning. Med Educ 2004: 38: 125128. 14 Gordon J. Assessing students personal and professional development using portfolios and interviews. Med Educ 2003: 37: 335340. 15 Pearson DJ, Heywood P. Portfolio use in general practice vocational training: a survey of GP registrars. Med Educ 2004: 38: 8795. 16 Dornan T, Carroll C, Parboosingh J. An electronic learning portfolio for reective continuing professional development. Med Educ 2002: 36: 767769. 17 Driessen EW, van Tartwijk J, van der Vleuten C, Wass V. Portfolios in medical education: why do they meet with mixed success? a systematic review Med Educ 2007: 41: 12241234. 18 Driessen EW, van Tartwijk J, Overeem K, Vermunt JD, van der Vleuten CPM. Conditions for successful reective use of portfolios in undergraduate medical education. Med Educ 2005: 39: 12301235. 19 Kjaer NK, Maagaard R, Wied S. Using an online portfolio in postgraduate training. Med Teach 2006: 28: 708712. 20 Snadden D, Thomas ML. Portfolio learning in general practice vocational training - does it work? Med Educ 1998: 32: 401406. 21 Davenport E. Transferable electronic portfolios for the dental curriculum. In: Proceedings 30th Meeting of the Association for Dental Education in Europe. Eur J Dent Educ 2005: 9: 2996. 22 Anziani H, Durham J, Moore U. The relationship between formative and summative assessment of undergraduates in oral surgery. Eur J Dent Educ 2008: 12: 233238. 23 Ellis JS, Hobson RS, Waterhouse PJ, et al. Tutor perceptions of the use of a reective portfolio within a pastoral tutor system to facili-

24

25

26 27

28

tate undergraduate personal development planning. Eur J Dent Educ 2006: 10: 217225. Morgan DL. Practical strategies for combining qualitative and quantitative methods: applications to health research. Qual Health Res 1998: 8: 362376. Cotterill S, Jones S, Walters R, Horner P, Moss J, McDonald A. Evaluating the use of hand-held computers to access electronic portfolios and clinical guidelines in a wireless environment for undergraduate medical education.. Available at: http://www.eport folios.ac.uk/fdtl4_docs/PDA_CETL4HealthNE_study_for_ASME_ 2006.ppt#256,1, (accessed: 23 March 2010). Ritchie J, Lewis J. Qualitative research practice: a guide for social science students and researchers. London: Sage, 2003. Davenport E, Roberts B, Orr G, Ellis JS. Does an electronic portfolio facilitate reection as effectively as a paper based version In: Proceedings of 31st Annual Meeting of the Association for Dental Education in Europe. Athens: Eur J Dent Educ, 2005: 10: 107121. Duque G, Finkelstein A, Winer L, Gold S. Immediacy and efcacy of giving feedback and evaluating medical students during their rotations in geriatric medicine: the McGill electronic portfolio. J Am Geriatr Soc 2003: 51: S197S197.

Supporting information
Additional Supporting information may be found in the online version of this article: Appendix S1. Questionnaires Please note: Wiley-Blackwell are not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.

Eur J Dent Educ 15 (2011) 3641 2011 John Wiley & Sons A/S

41