Education for Primary Care (2010) 21: 14964

# 2010 Radclie Publishing Limited

What is a good general practitioner (GP)? The development and evaluation of a multi-source feedback instrument for GP appraisal
Annabel Shepherd MRCGP
GP and Researcher

Murray Lough MD FRCGP

Director of Educational Research NHS Education for Scotland, Glasgow, UK

WHAT IS ALREADY KNOWN IN THIS AREA . Multi-source feedback (MSF) is a method of providing feedback to doctors about observable behaviours in the workplace. Whilst general practitioners participating in GP appraisal in Scotland have been encouraged to submit MSF reports, no questionnaire developed in primary healthcare in the UK is currently available. WHAT THIS WORK ADDS . In order to develop a multi-source feedback questionnaire this study addressed the question: What do primary healthcare teams consider to be a good GP?. The resulting process was found to be feasible and acceptable and there was evidence that GPs may consider improvements in practice as a result of MSF. SUGGESTIONS FOR FUTURE RESEARCH . Future research should assess the reliability, construct validity and consequential validity of this MSF questionnaire.

Keywords: appraisal, continuing medical education, feedback, multi-source feedback, primary care, revalidation

SUMMARY
Background
Although multi-source feedback (MSF) has been used in primary healthcare, the development of an MSF instrument specic to this setting in the UK has not been previously described. The aims of this study were to develop and evaluate an MSF instrument for GPs in Scotland taking part in appraisal.

GP?. The data were reduced and coded by two researchers and questions were devised. Following content validity testing the MSF process was evaluated with volunteers using face-to-face interviews and a postal survey.

Results
Thirty-seven statements covering the six domains of communication skills, professional values, clinical care, working with colleagues, personality issues and duties and responsibilities were accepted as relevant by ten primary healthcare teams using a standard of 80 percent agreement. The evaluation found the MSF process to be feasible and acceptable and participants provided some evidence of educational impact.

Methods
The members of ten primary healthcare teams in the west of Scotland were asked to provide comments in answer to the question, What is a good

150 A Shepherd and M Lough

Conclusion
An MSF instrument for GPs has been developed based on the concept of the good GP as described by the primary healthcare team. The evaluation of the resultant MSF process illustrates the potential of MSF, when delivered in the supportive environment of GP appraisal, to provide feedback which has the possibility of improving working relationships between GPs and their colleagues.

ary evaluation of the feasibility, acceptability and educational impact of this process was conducted.

METHODS
Content generation and content validity
The primary healthcare teams (PHCTs) of ten GP practices in the west of Scotland were asked to provide comments in response to the question What is a good GP?. PHCT members from each of the following groups were invited to participate: general practitioners, practice managers, district nurses, health visitors, practice-based nurses and administration and clerical sta. GPs were also recruited from an English primary care trust support group. Statements were analysed for content by two researchers who reduced and coded them to identify themes. The wording of questions, based on each theme, was agreed between the researchers and ML. Content validity was established by asking participants to rate questions for relevance on a 1 to 4 content validity index scale and comment on the wording of statements:14 1 not relevant 2 unable to assess relevance without major changes 3 relevant but needs minor alteration 4 very relevant. No change needed. Agreement was dened as a score of 3 or 4 and the accepted standard was 80% agreement.14

INTRODUCTION
Peer multi-source feedback (MSF) relies on a doctor and his/her colleagues completing questionnaires which ask them to rate the doctors performance against a series of statements about observable behaviours.1 MSF provides information about doctors level of insight into how colleagues perceive them and can give suggestions for potential self-improvement.2 Evidence generated by MSF has been proposed as a requirement for medical revalidation in the UK.3 Ramsey published the rst data concerning the reliability of MSF in healthcare in the USA.4 Experience in MSF has grown, not only in the USA but also in Alberta, Canada, where the method has been used in a re-certication process since 1999.5 In the UK, MSF instruments are obligatory for Foundation year and specialty training programmes.611 Despite evidence that a single generic MSF tool may be able to reliably assess doctors in the UK for the purposes of revalidation,3 a recent study recommended specialtyspecic MSF questionnaires12 and the use of specialty-specic questions has been raised as an issue by the Academy of Medical Royal Colleges MSF Working Group.13 In order to develop an MSF questionnaire for general practice appraisal in the UK we sought primary healthcare teams descriptions of the good GP. Using the resulting questionnaire we established an MSF process (Box 1). A preliminBox 1 The MSF process GP completes self-rating questionnaire online GP nominates colleagues and appraiser by email address | Raters receive email inviting them to rate GP online | GP and appraiser are emailed after 3 weeks and asked to download MSF report | GP and appraiser discuss report contents and formulate action plan for development/change
The online system collates data anonymously

Evaluation
Scottish GP Appraisal Conference delegates were invited to participate in part one of the evaluation. Appraisers for the volunteers had been allocated by the Scottish GP appraiser system and these appraisers were invited by letter to participate. GPs and appraisers were provided with written instructions and a pilot MSF training compact disc (CD) which provided information about delivering MSF feedback. In-depth semi-structured interviews with participants were undertaken by AS. A topic guide was compiled using information from the literature.1517 Interviews were audio-taped and transcribed verbatim. The data were analysed by AS concurrently and relevant emerging issues were incorporated into subsequent interviews using an iterative approach. Following data reduction, data from each interview were coded and categorised. Content from relevant questionnaires was combined with the results of part one of the evaluation to generate items for an evaluation question-

What is a good GP? The development and evaluation of a MSF instrument for GP appraisal

151

naire.18,19 A pilot of the questionnaire with ve GPs who were educationalists with NHS Education for Scotland allowed minor comments to be incorporated. The questionnaire was amended for appraiser participants. 160 GPs in Scotland were invited to participate in part two of the evaluation. These GPs were registered GP appraisers and the Scottish GP appraisal operational manager provided their contact details. Appraisers for the volunteers from this group were subsequently invited by letter to participate. Volunteers who had previously participated in the study were excluded. Questionnaires and a pre-paid return envelope were sent to volunteers and their appraisers. Reminder letters and further questionnaires were sent to non-respondents on two further occasions at fortnightly intervals. Data entered into a Microsoft Excel spreadsheet were exported to Mini-tab software for analysis. The number of participants scoring four or more for each question determined the level of agreement with each statement.

The themes generated from interviews with the eight GP volunteers were:
. . . . . .

the MSF process involving raters practical considerations the MSF questionnaire the MSF report and reaction to MSF discussing feedback and perceived educational impact . MSF and GP appraisal.

MSF process
Although the majority of participants found the process easy to undertake, one participant was confused at rst about whether to self-rate. A problem with internet rewalls initially led to some raters not receiving the email inviting them to complete the questionnaire. Three participants felt that this reduced the number of responses they received. Its really basic and straightforward you do. So . . . gosh is that all there is to it? (GP 5)

RESULTS
Content generation and content validity

Involving raters
Fifty-one primary healthcare team members took part in the questionnaire content generation exercise. The number of participants from each sta group was: 12 administration and clerical sta, eight practice managers, 16 GPs, six district nurses, four health visitors and two practicebased nurses. The number of individual statements gathered was 1588. Analysis generated six themes which formed the basis for questionnaire domains. The content related to each theme generated 37 questions. All items for nal inclusion met the standard of 80% agreement when tested for relevance. The six domains were: 1 2 3 4 5 6 communication skills professional values clinical care working with colleagues personality issues duties and responsibilities. All participants found involving, educating and asking permission from raters the most dicult part of the process. The majority approached all eligible raters but some left out particular colleagues such as those who were family members. It probably is quite a big ask for folk, if theyre going to take a bit of time to think about it. To get the value out of it. (GP 2) Participants reected that lack of response may be inuenced by the following factors:
. . . .

lack of time lack of enthusiasm for the process lack of IT aptitude poor relationship with the GP. Everybody that didnt respond would have their own reasons, some of them would be time, some of them would be they just forgot . . . Some of my older partners dont really like these sorts of things. (GP 3)

The questionnaire can be found in Appendix 1 and selected relevant quotes can be found in Box 2.

Evaluation
In part one of the evaluation eight volunteers were recruited from ve health board areas in Scotland. Interviews lasting 30 to 45 minutes with eight volunteer GP interviewees and six appraisers were carried out.

All participants found that work done to increase raters understanding of the process improved response rates. Strategies involved reminding colleagues verbally, allocating protected time and giving written instructions. I managed to get some protected time for the girls (administration and clerical sta) where I had to come down and physically say well

152 A Shepherd and M Lough

Box 2 Selected quotes from primary healthcare team members in response to the question What is a good GP? Communication skills It is essential that a GP is a good listener; patients need to feel that their concerns and fears are being taken seriously. (Practice Manager 3) As well as having legible handwriting, todays GP is computer literate, keeps accurate records of consultations and has good IT and audit skills. (Practice Manager 1) Professional values Probity, openness in all dealings, especially nancial, and no inappropriate relationships or friendships with patients or sta. (GP 2) He or she should respect the patients dignity, be polite and considerate of their personal views. A good GP should be honest, trustworthy and respect condentiality. (Treatment Room Nurse 1) Should be able to switch o from the demands of the job and have a good support network and coping mechanisms. (Health Visitor 1) Clinical care Keeping up to date, having a willingness to learn, identifying educational needs and acting on them and participating in professional development. (GP 9) As a GP but also as a patient myself I think a good GP should be up to date, knowledgeable in all relevant areas, a good listener, be a good history taker, examine adequately, make a good diagnosis and not miss serious illness. (GP 10) Be able to cope with the stress of the job stress comes from the pressure of work, complaints, dealing with uncertainty coping with these requires dierent skills. (Practice Nurse 2) A good GP takes an empathetic and supportive approach to all aspects of terminal care. (District Nurse 6) Working with colleagues A good communicator at all levels, having open discussions regarding care management with all disciplines of the healthcare team, with respect for their views and judgements. (District Nurse 3) Personality issues A good GP should have an excellent rapport with colleagues and patients registered with the practice but at the same time be able to maintain a professional stance. (Practice Manager 3) A good GP can deal with emergency situations calmly and eectively. (GP 4) A good GP should be smart and clean. (Practice Manager 5) Duties and responsibilities A good GP responds to requests from team members in relation to patients needs no matter how they feel or how busy they are. (Clerical 6) Being well organised and able to cope with the heavy workload, having good time management. (GP 9) A good GP is continually self-evaluating. Capable of admitting errors and learning from them. (Health Visitor 1)

listen stay o the phones right now, set it up here, its going to take you ten minutes. (GP 7) I showed him how to log in and things like that and I just left the room. Hes not very internet aware. (GP 1) The majority of participants felt that the process had improved working relationships with colleagues. We are actually thinking of a quality practice award. This has given people condence to

take this farther in terms of team working and inter-personal relationships. (GP 8)

Practical considerations
The majority of participants found that access to email had to be organised to allow some raters to participate. A few voiced concern about the lack of privacy some raters had when completing questionnaires in busy administrative areas. Similarly, a few GPs felt they could work out who had submitted particular comments. The majority of participants were concerned about the impact

What is a good GP? The development and evaluation of a MSF instrument for GP appraisal

153

of this problem on future relationships. You get the written comments. I know all of these people quite well. Im trying to think, I think I know who that person was. I can never know for sure. (GP 3)

The majority of GPs felt no need to make changes as a result of the feedback. A few described planning changes such as:
. improving systems used for communication . changing behaviour in interactions with colleagues . improving systems involving delegation of work.

MSF questionnaire
The majority of participants thought that the questionnaire was fair, comprehensive and quick to ll in. No concerns were raised about specic items. You maybe would lose some of the feedback if you compressed (the questions) more. (GP 1) You cant change your personality but you can modify behaviours if youre given feedback, if youre open enough to take that. (GP 3)

MSF and GP appraisal

The vast majority of participants found that although MSF enhanced the appraisal interview, they had concerns about how any dicult issues raised should be tackled in the context of appraisal. Its a really nice bit of evidence to have. I was not only pleased with the content, I was actually pleased with having something solid to take to my appraisal. (GP 6) The themes generated from interviews with the six appraisers were:
. preparation and training . feedback meeting . MSF in the context of GP appraisal.

MSF report and reaction to MSF

All participants found the report easy to download and interpret. Whilst the majority had no apprehension about participating in MSF, a few felt a degree of anxiety. A few came across comments or ratings they didnt expect and a few were initially annoyed by aspects of the feedback. I was quite apprehensive about what they would say. Youd like to think that they would say very nice things about you; you dont actually know. (GP 2) You always get that moment of butteries thinking have I really oended someone without knowing it. (GP 6)

Discussing feedback and perceived educational impact

A few participants discussed the feedback informally with their spouse or a friend. A few found the time between receiving the report and discussing its contents useful to emotionally process the feedback. Something negative you need a little bit of time to think about because the natural reaction is to kind of put the barriers up and sometimes you need a bit of time to put them down again. (GP 2) The majority of participants felt that discussing feedback with an appraiser brought an objective perspective. Issues raised were delegating, timekeeping, practice management, colleague relationships, communication, the importance of the perceptions of others, team working and practice systems. You kind of read something and you take your spin on it. (The appraiser) made me look at other options, other ways of interpreting it. (GP 1)

Preparation and training

Although the majority of appraisers spent around 20 to 30 minutes preparing for the feedback meeting, one appraiser spent no time preparing. All appraisers felt that they had many of the skills required to discuss MSF. The vast majority found that although the training CD was useful the content could be rationalised. I think a lot of us have skills. I think we are learning those skills; the more appraisals we do, we do get better at that. (Appraiser 3)

Feedback meeting
All appraisers found the report useful in providing them with ideas to discuss further during appraisal. I think when you rst look at it its a little bit daunting because theres a lot of information there, it does take a while to go through it. But theres a lot of depth there, theres a lot of things that you could potentially pick up and talk about. (Appraiser 1)

154 A Shepherd and M Lough

The majority of appraisers felt that their appraisees managed to identify areas for development during feedback meetings. A few found that MSF allowed them to open up a discussion about their appraisees relationships with colleagues. A lot of group practices have issues but some of them are under the carpet, they dont come out. So I think MSF opens that up a little bit more. (Appraiser 4)

MSF we discussed a lot of the setting and the practice as an initiation of appraisal. (Appraiser 3) In part two of the evaluation there were 20 volunteer GP respondents and eight appraiser respondents. Results can be found in Tables 1 and 2. In contrast to part one of the evaluation, the task of nominating raters online was perceived to be the most dicult part of using the website, although 14 out of 20 volunteers agreed that this was straightforward. In contrast to previous participants, respondents to the survey responded positively to questions about ease of nding ten colleagues who were willing to participate in MSF. The majority also felt they had access to the necessary IT facilities. Although the majority of respondents felt the purpose of meeting for feedback was clear, some felt that receiving the feedback from the appraiser rst instead of as a paper copy beforehand would have been better. The majority had identied areas for improvement and felt that the process had been valuable.

MSF in the context of GP appraisal

The majority of appraisers felt that honesty on the part of raters, appraisees and appraisers was central to making MSF meaningful. The majority had concerns about MSF reports which raised important or dicult issues for GPs. However, the vast majority felt that MSF had the potential to provide insight into a GPs practice. Im quite positive about it. It can enhance an appraisal. Because I had spent time on the

Table 1 (n=20)

Participants views and experiences of the feasibility, acceptability and educational gain of MSF selected questions

Please estimate your level of agreement with the following statements about the feasibility, acceptability and educational impact of Scottish GP MSF Attitudinal statement

Level of agreement Continuous rating scale 17 (1=strongly disagree, 7=strongly agree) Median Mean (SD) Number of rating scores =4

Website I was able to complete the MSF questionnaire easily It was straightforward to nominate raters on the website Most GPs should be able to use the MSF website Questionnaire The MSF questionnaire had too many questions The questions seemed fair The MSF questions were relevant to the qualities a good GP should have It was dicult to nd ten individuals who were willing/able to complete the MSF questionnaire The colleagues/sta I asked to participate had the necessary computer access to complete the questionnaire The colleagues/sta I asked to participate had the necessary privacy to complete the questionnaire My colleagues told me they had diculty nding time to complete the questionnaire Report I disagreed with the content of the MSF report I was upset by the content of the MSF report It would be better if the feedback was face-to-face verbal feedback Feedback meeting The purpose of the MSF feedback meeting was clear I was able to identify areas where I will be able to make changes/ improvements to my practice I have made changes/improvements to my practice as a result of MSF Taking part in MSF has improved my relationships with my colleagues Most GPs would benet from participating in MSF

6 6 6

6.2 (1.1) 4.9 (1.7) 5.6 (1.5)

19 14 17

2 6 6 2 6 6 3

2.9 (1.5) 5.6 (1.1) 5.4 (1.5) 2.9 (1.8) 5.9 (1.5) 5.5 (1.5) 3.2 (1.9)

5 18 18 5 18 17 8

2 2 2

1.8 (0.9) 2.1 (1.2) 3.3 (2.2)

2 2 7

5.5 4 4 4 5.5

4.3 (2.4) 3.9 (2.1) 3.6 (1.7) 3.9 (1.2) 4.9 (2.0)

13 13 11 16 15

What is a good GP? The development and evaluation of a MSF instrument for GP appraisal

155

Table 2 Appraisers views and experiences of the feasibility, acceptability and educational gain of MSF selected questions (n=8) Please estimate your level of agreement with the following statements about the feasibility, acceptability and educational impact of Scottish GP MSF Attitudinal statement Level of agreement Continuous rating scale 17 (1=strongly disagree, 7=strongly agree) Median Mean (SD) Number of rating scores =4

Report I felt well prepared for the feedback meeting Feedback meeting I have the necessary skills to facilitate MSF feedback MSF feedback could be facilitated as part of the GP appraisal meeting The purpose of the feedback meeting was clear The appraisee was able to identify areas where they will be able to make changes/improvements to their practice MSF feedback enhances GP appraisal

5.6 (1.2)

5 5.5 5.5 5.5 5

5.4 (0.9) 5.5 (1.2) 5.1(1.4) 4.5 (2.7) 5.3 (1.0)

8 8 7 6 8

DISCUSSION
Summary of main ndings
We have developed a multi-source feedback tool for GP appraisal in the UK which drew on primary healthcare teams experience of what constitutes the good GP. GPs and appraisers who used the tool found it to be feasible and acceptable. There were verbal reports of change in practice as a result of MSF.

Strengths and limitations of the study

The proposed GP MSF tool has been developed by members of the primary healthcare team rather than by GPs alone. Restricting the sample that generated content to the west of Scotland may limit the applicability of the instrument. The continually changing nature of the qualities needed for a good GP in the UK may impact on the face validity of the questionnaire. Evaluation participants are prone to selection bias.

Comparison with existing literature

Previous MSF tools in medicine in the UK have been developed by mapping questions3,20 to established General Medical Council (GMC) standards for clinical practice. Although this MSF tool has overlapping themes with the GMC document Good Medical Practice21 and the GMCs working framework for appraisal and assessment,22 a number of items do not fall into the categories described. It is possible that primary healthcare teams insight into qualities important for the good GP raise supplementary standards which reect the role of the GP as a member of a multidisciplinary team.

Between eight and ten raters are required to ensure MSF reliability20 and diculties achieving this have previously been described.3 This may be especially relevant for small practices and certain groups such as sessional GPs. One solution is to provide rater information and training. This may not only improve response rates2 but could also improve feedback quality whilst helping to maintain anonymity.23 Training may also militate against raters using MSF to provide upward feedback about the organisation.23 However, there is concern that the validity of results could be compromised by training and we consider this to be an area requiring further work.3 This study found that narrative comments may be problematic for rater anonymity. Herson observes that participants may become preoccupied by narrative comments at the expense of other signicant issues raised by the process.24 However, there is evidence to suggest that narrative comments are particularly powerful, especially when they are behaviour orientated.25 In contrast with some MSF processes developed for doctors in the UK, this process makes the MSF report available to the GP before the facilitated feedback meeting with their appraiser. This allows the GP time for self-reection, which is known to vary in timing and depth between individuals.26 The impact of this approach on the GP and practice team is an area requiring further work. There is conicting evidence about whether MSF can bring about improvements in doctors practice.2729 Although both self and rater MSF scores for physicians have a tendency to improve over time, resulting changes in behaviour are likely to be small to moderate.30 A range of barriers to change are recognised,3133 including strong negative reactions to the feedback. There is evidence that doctors often lack the ability to assess their own performance using MSF,34 implying that the process can often reveal a lack

156 A Shepherd and M Lough

of insight. It has been suggested that facilitation by a mentor or appraiser could help doctors to overcome barriers to change and increase the chances of improvements in practice in response to MSF.29,30,33 The use of trained facilitators may account for some of the positive responses to the process that we encountered in our evaluation.

Suggestions for future research

In order to ensure the overall utility of our MSF tool for primary healthcare teams, further work to establish certain aspects of the construct validity and reliability of the instrument should now take place. Additionally, it will be necessary to explore aspects of the consequential validity of MSF as a process and to ascertain which aspects of the feedback report and subsequent appraiser facilitation meeting increase the likelihood of achieving improvements in practice.

CONCLUSION
We have developed a GP specialty-specic MSF tool with the aim of encouraging team working and development by promoting the concept of the good GP. Our evaluation of the tool has found it to be feasible and acceptable to GPs. There is some evidence of educational impact for participants when MSF is delivered in a supportive environment such as GP appraisal.

Acknowledgements
We are grateful to the GPs and their colleagues who contributed to this study. Many thanks to Dr C Reid (NHS Education for Scotland, UK) and Dr L Pope (University of Glasgow, UK) who contributed to the data analysis.

Funding
This study was funded by NHS Education for Scotland

Ethical approval
Ethical approval was not required for this study.

Conicts of interest
None.

References
1 Davies H and Archer J (2005) Multi source feedback: development and practical aspects. The Clinical Teacher 2: 7781.

2 Gray A, Lewis A, Fletcher C et al (2007) 360 Degree Feedback Best Practice Guidelines: the British Psychological Society 2007. www.psychtesting.org.uk/the-ptc/guide linesandinformation.cfm (accessed 20 January 2010) 3 Campbell JL, Richards SH, Dickens A, Greco M, Narayanan A and Brearley S (2008) Assessing the professional performance of UK doctors: an evaluation of the utility of the General Medical Council patient and colleague questionnaires. Quality and Safety in Health Care 17: 18793. 4 Ramsey P, Carline JD, Inui TS, Larson EB, LoGerfo J and Wenrich MD (1989) Predictive validity of certication by the American Board of Internal Medicine. Annals of Internal Medicine 110: 71926. 5 Lockyer J (2003) Multisource feedback in the assessment of physician competencies. Journal of Continuing Education in the Health Professions 23: 412. 6 Archer J, Norcini J, Southgate L, Heard S and Davies H (2008) mini-PAT (Peer Assessment Tool): a valid component of a national assessment programme in the UK? Advances in Health Sciences Education: Theory and Practice 13: 18192. 7 Carr S (2006) The Foundation Programme assessment tools: an opportunity to enhance feedback to trainees? Postgraduate Medical Journal 82: 5769. 8 Hrisos S, Illing JC and Burford BC. (2008) Portfolio learning for foundation doctors: early feedback on its use in the clinical workplace. Medical Education 42: 21423. 9 Whitehouse A, Hassell A, Bullock A, Wood L and Wall D (2007) 360 degree assessment (multisource feedback) of UK trainee doctors: eld testing of team assessment of behaviours (TAB). Medical Teaching 29: 1716. 10 Wilkinson JR, Crossley JG, Wragg A, Mills P, Cowan G and Wade W (2008) Implementing workplace-based assessment across the medical specialties in the United Kingdom. Medical Education 42: 36473. 11 Royal College of General Practitioners (2006) Multisource Feedback. Royal College of General Practitioners: London. www.rcgp-curriculum.org.uk/nmrgcp/ wpba/multi-source_feedback.aspx (accessed 18/12/09). 12 Davies H, Archer J, Bateman A, et al (2008) Specialtyspecic multi-source feedback: assuring validity, informing training. Medical Education 42: 101420. 13 The Academys MSF Working Group (2009) Multi-source Feedback, Patient Surveys and Revalidation: report and recommendations. Academy of Medical Royal Colleges, London. 14 Waltz CW and Bausell RB (1981) Nursing Research: design, statistics and computer analysis. Philadelphia, PA: FA Davis. 15 McKay J, Shepherd A, Bowie P and Lough M (2008) Acceptability and educational impact of a peer feedback model for signicant event analysis. Medical Education 42: 121017. 16 Crossley J, Humphris G and Jolly B (2002) Assessing health professionals. Medical Education 36: 8004. 17 Rees C and Shepherd M (2005) The acceptability of 360degree judgements as a method of assessing undergraduate medical students personal and professional behaviours. Medical Education 39: 4957. 18 Sargeant J, Mann K and Ferrier SN (2005). Exploring family physicians reactions to multisource feedback: perceptions of credibility and usefulness. Medical Education 39: 497504. 19 Brett JF and Atwater LE (2001) 360 degree feedback: accuracy, reactions, and perceptions of usefulness. Journal of Applied Psychology 86: 93042.

What is a good GP? The development and evaluation of a MSF instrument for GP appraisal

157

20 Archer JC, Norcini J and Davies HA (2005). Use of SPRAT for peer review of paediatricians in training. BMJ 330: 12513. 21 UK General Medical Council (2006) Good Medical Practice. General Medical Council: London. 22 General Medical Council (2008) Working Framework for Appraisal and Assessment. www.gmc-uk.org/about/ reform/gmp_framework.asp 23 Bracken D, Timmreck CW and Church AH (2001) The Handbook of Multi-source Feedback: the comprehensive resource for designing and implementing MSF processes. Jossey-Bass, San Francisco. 24 Hersen M and Thomas J (2004). Comprehensive Handbook of Psychological Assessment: industrial and organizational assessment. John Wiley and Sons, Hoboken: New Jersey. 25 Smither JW and Walker AG (2004) Are the characteristics of narrative comments related to improvement in multirater feedback ratings over time? Journal of Applied Psychology 89: 57581. 26 Sargeant JM, Mann KV, van der Vleuten CP and Metsemakers JF (2009) Reection: a link between receiving and using assessment feedback. Advances in Health Sciences Education: Theory and Practice 14: 399410. 27 Smither JW, London M and Reilly RR (2005) Does performance improve following multi-source feedback? A theoretical model, meta-analysis and review of empirical ndings. Personnel Psychology 58: 3368. 28 Lockyer J, Violato C and Fidler H (2003) Likelihood of change: a study assessing surgeon use of multisource feedback data. Teaching and Learning in Medicine 15: 16874.

29 Overeem K, Faber MJ, Arah OA et al (2007) Doctor performance assessment in daily practise: does it help doctors or not? A systematic review. Medical Education 41: 103949. 30 Violato C, Lockyer JM and Fidler H (2008) Changes in performance: a ve-year longitudinal study of participants in a multi-source feedback programme. Medical Education 42: 100713. 31 Fidler H, Lockyer JM, Toews J and Violato C (1999) Changing physicians practices: the eect of individual feedback. Academic Medicine 74: 70214. 32 Sargeant JM, Mann KV, Ferrier SN et al (2003) Responses of rural family physicians and their colleague and coworker raters to a multi-source feedback process: a pilot study. Academic Medicine 78: S424. 33 Sargeant J, Mann K, Sinclair D, Van der Vleuten C and Metsemakers J (2008) Understanding the inuence of emotions and reection upon multi-source feedback acceptance and use. Advances in Health Sciences Education: Theory and Practice 13: 27588. 34 Violato C and Lockyer J (2006) Self and peer assessment of pediatricians, psychiatrists and medicine specialists: implications for self-directed learning. Advances in Health Sciences Education: Theory and Practice 11: 23544.

Correspondence to: Dr Annabel Shepherd, NHS Education for Scotland, 2 Central Quay, 89 Hydepark Street, annabel.Glasgow, G3 8BW, UK. Email: shepherd@nes.scot.nhs.uk

Accepted March 2010

158 A Shepherd and M Lough

What is a good GP? The development and evaluation of a MSF instrument for GP appraisal

159

160 A Shepherd and M Lough

What is a good GP? The development and evaluation of a MSF instrument for GP appraisal

161

162 A Shepherd and M Lough

What is a good GP? The development and evaluation of a MSF instrument for GP appraisal

163

164 A Shepherd and M Lough

Copyright of Education for Primary Care is the property of Radcliffe Publishing and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.