Cracking content – a guide to measuring the media, present and future

Michael Blowers
5, St. John’s Lane London EC1M 4BH Tel: +44 (0)20 7549 2829 www.mediaevaluation.eu

Michael Blowers

Media Evaluation Research 2006

Abstract:
The current methods of measuring public relations in common use range across a variety of disciplines, subject area and depth of research. On one level, traditional quantitative media research, measuring column inches and advertising value equivalent, still represent a sizeable portion of the lower cost / do-it-yourself end of the market. These quantitative measurement techniques have been around for many years and while there is pressure to replace them with a more sophisticated approach, they will likely continue to have a role. At the opposite end, there are PR departments within large organisations undertaking integrated research programmes (both quantitative and qualitative) on a continuous and project basis. Often these projects are undertaken with help from specialist agencies and feature analysis of media exposure with the option to integrate results from market research and other forms of market analysis. In between, there is a broad range of media analysis projects largely coming under the term referred to in the CIPR Toolkit on media evaluation (July 2003) as media content analysis. This involves the analysis of media data using a number of metrics, both quantitative and qualitative, involving the gathering and analysing the content of text (Neuman, 1997, 272-273). Specialist agencies dealing with media evaluation have at their core the analysis of message and favourability. Message analysis often comes in two forms. Most often a brand or organisation will have a series of messages it would like featured within the media coverage. These messages could be about some unique property of a product, or some general qualities with which the organisation would like to be associated. Additionally, there is an application for the tracking of other messages relevant to the organisation. These will be the responsive messages, or the things which the media are saying about the organisation. Sometimes negative, these messages provide feedback particularly when undertaken alongside a similar analysis of competitors. The result is the building of a picture of the organisations media profile and underlying sentiment and is often viewed as a tool to message refinement, feedback and planning (Macnamara, 2003). The other main qualitative measure applied is favourability. There is no uniform method of measuring favourability, although the most commonly applied uses the categories of positive, neutral and negative. Other ratings

2

Michael Blowers

Media Evaluation Research 2006

which exist include a +10 to -10 rating, and strongly favourable, slightly favourable, slightly unfavourable and strongly unfavourable. There are a number of other variables which are frequently tracked within the media exposure including the use of branding, subject or topic analysis, media pro-activity and the use of press releases, spokespeople and individual journalists and media titles. These are frequently cross referenced against favourability as a way of drilling back to how the good and bad publicity arose. This report will use primary and secondary research to illustrate which of these techniques and processes are most commonly adopted and provide some indication of the importance with which PR attaches to them. The primary research will be undertaken using a questionnaire to sample respondent’s attitudes towards different methods of measurement. A blog has also been created to convey concepts and ideas, ahead of their use in this report. A number of example media content reports have also been sourced which will form part of the secondary research. The secondary research will examine published research and the methods of measuring media content, indicating the shortcomings and potential benefits of various methods and systems. It will be necessary to discuss the comparative role of the different types of media, from measurement perspective, and to the greater PR community. It is also relevant to consider the likely changes that media measurement will go through in the coming years; which will lead on from a discussion about the recent changes which media measurement has already been through. Key to this topic will be the spotting for likely trends and the significant influences to the process. It is anticipated that this can be segmented into areas of changing demand and changing supply. Changes in demand reference the changes PR has had on media measurement. The change in supply refers to how the media will change in the coming years. For example, the continuing decline in the circulation of newspapers is well charted and in May 2005 Rupert Murdoch spoke about his increasing concern over the gathering pace of change in people’s news access habits. The implications of this shift are fundamental for PR. According to the Carnegie Corporation (2005) young people (aged 18-34) are over twice as likely to refer to the web for news compared to the traditional newspaper press. Anyone in PR now trying to target the young can not ignore the online media and as this characteristic feeds through to other age groups so the necessity for online communication will become paramount.

3

Michael Blowers Media Evaluation Research 2006 With the trend for increasing use of online media sources. the trends resulting in the present situation. Consideration will be given to the development of media content analysis. particularly if importance is placed on comparing results between different media. marketing or business intelligence”. Additionally. When considering the methods of measuring the media. Within this discussion there will be an examination of the types of audience statistics available for online sources. White and Blamphin (1994) found that evaluation ranked number one amongst a list of priorities for PR in a UK study of practitioners and academics. The designated sources of research compose a questionnaire and blog for primary research and desk research making up the secondary sources. pre-testing. and a view on where this might be taking PR measurement in the future. PR will be looking for a method of justification. and conclude with a number of recommendations applicable to both strategy and ultimately the tactics of media content analysis. media evaluation would include activities relevant to planning and targeting. The final phase is made up of post examination. with a view to framing the notion that audience is not an accurate and supportable method of measuring media output. The questionnaire was circulated to a number of PR representatives with the aim 4 . reference is being made towards the activity of media content analysis. In the same way PR uses the key metric of audience to measure their impact in the traditional media. it will obviously look to apply the same measure for online exposure. it would be reasonable to say that PR is taking measurement increasingly seriously. Introduction If judged by the amount of academic research and discussion filling the specialist press and more recently the Financial Times . through to interim sampling and message refinement. This reports role is to summarise the challenges facing the discipline. Media content analysis is a specific function making up part of the greater discipline referred to as media evaluation. Consideration will then be given to examining other methods of measuring online media output. This would seem common sense. with media content analysis usually composing a large share of the research. According to AMECi “media evaluation is a strategic and tactical tool for people working within PR. This report seeks to shed light on what current methods are being adopted by the PR community. message selection.

The concept of media content analysis was introduced as a method of studying the mass media by Harold Laswell in 1927. Media content analysis According to Neuendorf (2002. 5) describe content analysis as “a new research technique for making inference by systematically and objectively identify specific characteristics within text”. 1927 in Newbold. This is supported by Neuendorf in her benchmarked text. precision.1) “in the field of mass communications research. content analysis has been the fastest growing technique over the past 20 years or so”. the former which looks to identify what the media is saying about society.79). what is said on a given subject in a given place at a given time”. and generality. Smith and Ogilvie (1966. The Content Analysis Guidebook (2002). generalisability. Media content analysis became increasingly popular during the 1920s and 1930s for investigating the rapidly expanding communication content of movies. Other early definitions of media content analysis concentrated on the quantitative elements of the discipline. Other authors of media content analysis do not support the notion that it is a strictly quantitative research tool. Dunphy. She strongly argues for the use of scientific methods “including attention to objectivity-intersubjectivity. 3) the renowned author on media content Pamela Shoemaker (1996) argues for the inclusion of qualitative techniques through her argument on the relevance of the humanist and behaviourist tradition. The blog (http://mediaevaluation.com/) was to act as a sounding board for ideas and the chance to explore new ideas and concepts before committing them to this report. Its application to the public relations industry was apparent. 10). 5 . a priori design. not qualitative. reliability. In McNamara (2003. replicability. 2002. who initially used it to study propaganda (Lasswell. while the latter looks to identify future affects. how they used the results and their thoughts on if this might change in the future. Stone. 34) who said “content analysis is a technique which aims at describing with optimum objectivity. something supported by Lasswell (1952. and hypothesis testing space”(2002.Michael Blowers Media Evaluation Research 2006 of understanding what types of method they currently use for analysing media content.blogspot. et al. where she argues that media content analysis is a quantitative measure. validity.

the human contribution to content analysis is still paramount” (Neuendorf. while the inferential and predictive roles of content analysis facilitate (rather than being conclusive) an exploration into the likely effects of mass media representations on society and audiences. for example creating tables charts or graphs. Computers are an integral part of this type of research and are most commonly used for storing data such as coding annotations by researchers. 40). Descriptive 2. Inferential 3. 9) that the majority of the media content research expected by PR should be described as quantitative. Methods of evaluating the media There is broad agreement that when undertaking media content analysis the greatest success has resulted from following these procedures: 1. 2002. although most leading researchers in this area do not support this approach: “The notion of a completely automatic content analysis via computer is a chimera….Michael Blowers Media Evaluation Research 2006 It is clear that from the research undertaken by Macnamara (2003. Attempts have been made to create systems to automatically scan text and identify the coding of words and phrases. 4) highlights three qualities most closely associated with media content analysis and which will have the greatest relevance to the social sciences: 1. and finally for facilitating inference (Neuendorf. McNamara (2003. 52). Berelson (1952) in Neuendorf (2002. 6 . 2002. secondly as a method of testing hypothesis. Predictive The descriptive role provides an insight into the specific messages within the discourse represented in mass media. 52) suggests five main purposes of content analysis: • • • • • To describe the substance and characteristics of message content To describe the form characteristics of message content To make inferences to the producers of content To make inferences to the audiences of content And to predict the effects of content on the audiences Carney (1971) simplifies this by suggesting that content analysis can be firstly useful for its descriptive qualities. analysing data and reporting.

To support the resulting analysis it may be necessary to provide information on intercoder reliability in addition to the code book/coding list and details of the methodology used (Snyder-Duch. coding guidelines and instructions to coders. Generalisability refers to the extent to which research findings can be applied and taken as a measure of the target population. as a method of agreeing the key variables and messages to be tracked. combined with an appropriate methodology. et al. Macnamara (2003. 2001). The inclusion of a priori design will enhance to be objectivity/ intersubjectivity of the research as it will prevent the list of issues and messages being added to as the research is undertaken. coders should be prepared to submit ‘blind coding’ to help support the level of confidence in the resulting research. 5. 3. As a practical measure Macamara suggests that at least 10% of the media content is analysed by a different coder. 10) indicates that reliability requires that two or more coders be used.Michael Blowers Media Evaluation Research 2006 2. Supported by training. 4. Replicability is the ability and degree of difficulty for the coders to replicate the research. and the coding rules before the observation begins. Unreliable coding can fundamentally undermine any media content analysis project. 7 . with a view to maintaining consistency. Neuendorf (2002. 102-103) suggests examining a representative subset of the content with a view to concluding all decisions on variables. Replicability is determined by the full disclosure of information on methodology and procedures. something which is supported by Tinsley and Weiss (1975. their measurement. and would specifically involve the code book/coding list. It would follow that the generalisability is largely determined by a representative and sufficiently large sample. with a view to confirming or challenging the results. as supported by the research. 359) who said “obtaining ratings are not the idiosyncratic result of one rater’s subjective judgment”. 6. The validity of content analysis is maintained by thoroughly understanding the research objectives. As the body of research relevant to media content analysis increases a key feature will be the increasing adoption of this best practice. This can be achieved by the coder thoroughly immersing themselves in the ‘message pool’ before commencing the sample.

Size of the article or a length of broadcast segment. 3. high rating or influence. the order in an electronic media bulletins and use of photos and visuals. narrative. Positioning would track the use of headlines. 6 Undertaking qualitative message analysis relevant to the media would include analysis of text. different groups of readers (age. interpretive signs. Source: Macnamara. different sources within the media have a level of credibility which can alter the importance of the message. These can be condensed down to two main themes: narratology (examining the narrative or storytelling within the text). Media weighting to reflect high circulation. Sources would illustrate the balance of supportive and opposing sources quoted within the text (including comment on position and credibility) • • • • Qualitative content analysis is most often undertaken simultaneously with the quantitative analysis. 2. Perceptions of media credibility. as well as representing passing references and share of voice. Audience characteristics. 2002. 2003. semiotics which examines the signs within the text and how the reader might interpret and decode them (Newbold. other events within and outside the media which can magnify or alter the message. Positive and negative words and phrases can be analysed to identify the tone and text and in addition analysts can record notations during coding in relation to contextual factors. Context. race. Proponents of qualitative text analysis highlight the following qualitative factors as having the greatest relevance to PR: 1. first paragraph mentions and other prominent mentions. et al. gender. rhetorical factors. ethnicity. 8 . 84). education and socio-economic position) will have a different take on media content. semiotics and discourse.Michael Blowers Media Evaluation Research 2006 The application of quantitative research would provide results in five key areas: • Media prominence which would take account of the impact factors such as page number.

16) this contrasts with later post-structuralist influenced semiotics where the work of Klaus Brahn Jensen (1995) combined the early social semiotic theory of mass communication with the later work on active audience participation to interpret mediated meanings. firstly by identifying the signs in the text and then the dominant characteristics. for the audience may not see this latent dimension. More significant are the symbols within language.an image of a man holding a baby suggests fatherhood. 60-65) and others identify pronouns as key signifiers of meaning within text. The essential concept that underlines this approach is that words and images are signs that stand for or signify something else beyond their obvious meaning. The development of semiotics point towards a structuralist approach: seeking the meaning of signs according to a set system. The techniques adopted for qualitative analysis are less prescriptive than for quantitative analysis. the analysis may be longer than the text. 9 . According to Selby and Cowdery (1995. and often tells us what we already know in a language we don't understand”. second or third person. these can be positive and negative and give a strong indication of the writer's stance. According to Macnamara (2003. Other symbols are recognised internationally.Michael Blowers Media Evaluation Research 2006 One of the greatest challenges for those undertaking qualitative analysis is the maintenance of objectivity and repeatability.58) “these signs can be analysed as a result of selection and combination”. et al said: “So in studying media texts . family commitment and possibly gentleness and caring. Newbold. etc). for example ‘$’ signifying money and ‘©’ signifying copyright. we can use these ideas as they can provide a way of assessing the meaning in a text”. Together they form codes: -collectives of signs that produce certain meanings. Van Zoonen (1994) however attempts to explain how semiotic analysis can be practically applied to media text. The viewpoint of the narrator (taking the first person. Campbell and Pennebaker (2003. is questionable. Whether verbs are active or passive voice. and privileging them over the more obvious ‘manifest’ ones. The task is time-consuming. Macnamara (2003) highlighted eight areas where text elements can be studied for their qualitative content: • • • Adjectives used as descriptors.. Some of the key signs would include images and photographs . Newbold.et al warn: “The logic of deconstructing latent meanings..

sources quoted and impact on the meaning taken from the text. As long as objectives are clearly established for PR and corporate communications the return on investment would be best described as the achievement of these objectives. it aims to illustrate to management an areas ability to justify its contribution. It involves the creation of a priori schema. whatever they may be (Macnamara. emotional language and sarcasm. and many view financial return as unrealistic for PR for the following reasons: 10 . That is not to say that its implementation should not come without a number of conditions. 2003. Qualitative data has the benefit of yielding representative results of a high probability. Mayring (2000) suggests two ways of analysing qualitative text. Double meanings.Michael Blowers Media Evaluation Research 2006 • • • • • Tonal qualities. using inductive and deductive methodology. The implications of a measurement culture on PR As with many things. In this instance there is a role for it to play in assisting PR because there is the rightful precursor of financial return and it follows that this will be a measure of outcome. others include key performance indicators. such as aggressiveness. which is a prescriptive framework created before the sample is analysed in detail. By starting with predetermined categories there is an increase in the systematicity of the qualitative analysis. ROI is not the only term. Factors dictated by context: position and credibility of spokespersons. ‘Return on Investment’ (ROI) is one. Visual imagery within the text. benchmarking and balanced scorecards. Specifically the method requires a matching of a category to a passage of text. Mayring also recommends that intercoder reliability be assessed. although according to Macnamara there is the tendency with qualitative to suppress or ‘dumb down’ research findings. with only fully trained analysts being used. as with many of these terms. its precise definition is somewhat unclear and it is often misapplied. flippancy. there are a number of fashionable management buzzwords organisations like to use. Importantly this procedure brings some systematic steps to qualitative text analysis. However. Use of metaphors and similes. how these are positioned and used. 5). and not the matching of text to a category.

profit or share value and other financial objectives. and the legal department by their revenue generating capacity.Michael Blowers Media Evaluation Research 2006 1. 2004). 4. It is unreasonable to assume that the only form of communication between organisations and its publics will be via the communications/PR department (Hutton. Charles Fombrun (1996) suggested the idea of the ‘triple bottom line’ which. Judging PR in financial terms would be no more realistic than evaluating HR. R&D. Often campaign objectives are not directly linked to sales or share price (even though they might eventually contribute towards these). 2. as well as considering financial performance. positive media coverage 11 . For example Nike attracted worldwide criticism and falling sales because of its poor treatment of workers in developing countries. The benefits of having all three of these factors working beneficially in unison is well charted and a number of case studies prove that getting one wrong has implications for the other two. Within commercial organisations financial returns should be viewed as a team effort and it would be inaccurate for any single business unit to claim credit for financial results. 3. also factors in an organisation's environmental activities and its community performance. PR operates in all these areas and accordingly their contribution should be measured in more ways than just financial. This negative coverage about the treatment of communities had an overall impact upon the other two factors. 5. therefore making it difficult to financially assess the impact of any one form of communication. preferably. Often PR campaigns have a long-term set of objectives and are integrated with other forms of communication. Public sector organisations and other not-for-profit groups do not have sales. The ways that PR can provide a return on investment would be by proving successful in the following areas: • • • • • Fully informed and motivated employees Supportive and considerate relations with government and regulators Constructive and positive relations with key non-governmental organisations Well informed and ideally positive financial community Balanced or.

3) When considering the return on investment for PR it is necessary to measure outcomes and not simply outputs (see page 19). etc. to see that they are clearly established. attitudes and behaviour. complementing and contributing to overall organisational objectives. The methods of measuring return on investment for PR will be dependent upon the activities and objectives to be measured. understandable. It will also provide capabilities to consider trends. There is also the capacity to examine competitors and contemporaries motive as a way of understanding the impact of alternative strategies.Michael Blowers Media Evaluation Research 2006 • • • Supportive and loyal shareholders Well informed consumers Support and understanding of local communities where they operate (Macnamara. 12 . determining key message placement. PR needs to have its own selection of specific objectives. They should however be sufficiently discreet to make outcomes traceable aside from the objectives of other departments. make predictions and illustrate favourability. Whilst the limitations of media content analysis have already been considered. management wants and expects the representation of outcomes. making profits and gaining stakeholder support. target audience reached and comparative share of voice. In effect these involve having a good reputation. focus groups and/or interviews with representative groups and stakeholders. One key area which can contribute is media content analysis. While the measurement of outputs is usable. It is also crucial that the measures adopted are aligned with the overall objectives of the organisation. requiring research such as surveys. it may be necessary to complement the media analysis results with a detailed understanding of stakeholders and target audiences. when it comes to measuring return on investment it should be included as one of the methods alongside a selection. building brand equity. For example. Organisations need to consider very carefully their objectives. It is therefore relevant to highlight that return on investment for PR is not measured only in financial terms. achievable and measurable. Allied to this. be-it metrics on awareness. 2003.

In almost all the sample reports there is an explanation of the methodology and terminology featured. Few defined the factors or the process involved in the classifications. resulting from a lack or defendable research to support its application. Two providers use weighted opportunities to see (WOTS). Only one provider’s sample (Carma International) featured message qualities (eg: good customer service) separated into positive and negative. PR’s view of media measurement To support the secondary research a questionnaire was circulated to a number of PR professionals for their views on media content analysis. This is not an exclusive list and often the discipline is accused of approaching the issue in too subjective a way. however there is also the impression that the reports are aimed at a PR orientated/trained audience. Most (though they might not advertise it) are able to provide equivalent advertising value data and some highlight their ability to track messages and competitor activity over time to portray an impression of likely trends. although from the information available each base it on a different methodology. underlying motive and use of spokespeople. The reason for using a questionnaire was flexibility as a method of collecting detailed feedback relevant to the specific area under consideration.100 score. The factors which influence favourability would include branded use in the headline and body-copy. It was designed to be 13 . ‘neutral’ and ‘negative’ to classify exposure. There are a number of commonalities and in Appendix 1 a sample selection of reports have been examined with a view to illustrating the comparative approaches and methodologies etc. The majority came from in-house PR departments with the remainder being spread amongst agencies of various sizes. Carma International uses a 0 . where 50 is neutral. Fewer undertake the qualitative research components. and clarify that neutral coverage would be ascribed a slightly positive rating. as they have been described. Metrica feature a 4 stage classification of strongly positive and negative and slightly positive and negative. style of narrative and language.Michael Blowers Media Evaluation Research 2006 Characteristics of external analysis providers The majority of external media content researchers are able to provide analysis of the quantitative research elements as highlighted previously. instead of audience figures/ opportunities-to-see (MB Precis and Panarc). The majority of providers use ‘positive’. making anything beyond the summary of little comprehension to those not in PR.

When asked which of the media was of greatest interest almost every respondent indicated that press was the most important. and 16% also indicated that assignments tend to be evenly divided between internal and outside suppliers.Michael Blowers Media Evaluation Research 2006 concise and quick to complete. The remaining 14 . 16% indicated that research is conducted mostly using outside counsel or suppliers. Just over a half of respondents had over the past two years conducted media measurement with the assistance of an external service provider. Assignments evenly divided betw een doing them internally and suppliers 16% Unsure 5% Little or no measurement w ork is done. This however wouldn't appear to be exclusively the case and two-thirds had also undertaken media measurement themselves. 17% of respondents indicated that the increasing amount of research would exceed 10% of the current level. which aimed to establish a level of agreement with a series of statements. Interestingly 16% of respondents indicated that no measurement work is currently done. This was subsequently supported by the majority of respondents indicating that most of the media analysis work is done internally. 44% of respondents indicated that the amount of research undertaken would broadly stay the same. The picture is therefore of the majority of PR representatives undertaking research in-house while just over half also seek the assistance of external providers on an occasional basis. and adopted a series of measures including the Likert Scale (1932). This compared to 33% of respondents who indicated that the amount of research on the media would increase by 1-9%. 16% Most of the w ork is done by outside suppliers 16% Most of the w ork is done internally 47% Almost half of the audience indicated that the majority of research is undertaken internally. None of the respondents indicated that the amount of research would decrease.

which is further supported by only a third of respondents indicating that they track their media exposure against a given target audience. 11% did indicate that it would be something they would be interested in doing. The tracking of key messages elicited a positive response. and volume of media placements. for example by region or country. whether it was a product review or editorial reference. The survey interestingly highlighted that only a third of respondents link their target media with that generated. The questionnaire featured a list of the main media measurement techniques from which respondents were asked to indicate their preference and interest. Almost a third of respondents indicated that they track media exposure for the presence of branded references.Michael Blowers Media Evaluation Research 2006 mediums were very closely ranked with online and television coverage being marginally ahead of radio. making it the most popular variable to track. The aim of this. being verbal or visual. reach (also known as opportunities-to-see). One theme hard not to ignore is the apparent lack of targeting. Fractionally less popular was categorising media output by the type of news feature. 10% of the sample did indicate an interested in tracking the activities of their competitors in the media and in particular their key messages and share of voice. Although not as large in number it should be noted that 60% of respondents indicated that they currently use advertising value equivalents. While half of respondents do not track the journalists who write about them. The most used media metric were column inches/broadcast time. and nobody indicated that this will be something they would like to use in the future. It is notable that 17% of respondents would like to adopt this measure. A similar number also track their coverage for inclusion of graphics. press conference. was to compare and contrast these methods and highlight those which could become more popular. etc. A proportion of the sample was made up of public relations professionals from the public and non-profit sectors which would support the findings that only a third of respondents were interested in tracking product and services mentions and references to competitors. Half of the sample linked their media output with the methods of news generation. the main part of the questionnaire. A third of respondents indicated that they also track coverage by geographical area. A fifth said that it would be a measure they would like to use in the future. for example news release. Almost all the respondents categorised their output into different media groups. photos and use of colour. with almost twothirds indicating that they currently use this measure. Of those there was a 40:60 split between those using and not using a multiplier. It should also be noted that few were interested in using these measures if they did not already do so. media pitch. charts. 15 . etc.

classed as less important was advertising value equivalent. the cost of message placement and the number of media representatives attending events were largely downplayed by respondents with over twothirds indicating that they did not use these measures. In particular there was the view that key message analysis and 16 . The following heat table indicates which of the measures are on the whole more or less likely to become important: Measurement by audience/ opportunities to see Measurement by advertising value equivalent Media monitoring Key message content analysis Favourability of media content Competitor analysis Connecting media output with organizational outcomes Presenting a return on investment Creating tangible results for PR activity Academically proven methodology Results presentation techniques Results clarifications/accessibility Accurately reflecting on-line media exposure More important Stay the same Less important Although it is not indicated in this table as being so. A number of these factors were highlighted as likely to become more important. An important aspect of this research was to try and establish the views on whether these measures are likely to become more or less important in the coming years. A notable number within the sample also highlighted this as an area they would like to apply in the future. the measure which was almost. The importance of tracking any calls-to-action within the media coverage.Michael Blowers Media Evaluation Research 2006 Over half of the respondents highlighted the importance of spokesperson usage in the media and a similar number indicated support for the tracking of favourability within the coverage. As can be seen in the appendix (which includes a record of the actual responses) the sample was asked to indicate if they think that a measure will become more or less important in the future. but not quite.

In the final part of the questionnaire respondents were posed with a number of statements and asked to indicate how strongly they agreed or disagreed with them. In hindsight it would have been beneficial if the final section of the questionnaire had concentrated more on the application of coverage collection and media content analysis within the online environment. it was highlighted that there is a need to accurately reflect the importance of online media exposure. However the factor with the greatest expectation for likely future importance is the ability to connect media output with organisational outcomes. There was the view that media measurement could be used to educate colleagues and stakeholders of the role and value of public relations. There was less agreement that the results were understandable by non PR staff. It reinforced the view that message and favourability analysis are the most important roles for media content analysis. On the whole people said that the measurement methods for the media where easy to use. In addition it could be used to educate colleagues on the attitudes and opinions of stakeholders. There was also the view that increasingly PR needs to present a return on its costs and be able to express its results in a tangible manner. share price or other method of ranking. there is the media from where coverage will come. Finally.Michael Blowers Media Evaluation Research 2006 the measurement of favourability will become more important in future years. a topic to be dealt with in greater detail in the next section. that they were measuring what they wanted to. On the whole there was an acceptance that the movement of media to an online environment will make it more difficult to represent the value of coverage generated. and that the data produced usable results. Firstly. be-it via sales. 17 . An insight into the future applications for media research From the research undertaken it is clear that there are two particular areas where media content analysis will experience the greatest changes in the coming years. but there was an acceptance of the view that they would provide evidence for the success of a programme and would be usable for future strategic planning. The questionnaire proved a valuable insight into the priority attached to various methods of measurement. The challenges of measurement in the online environment will be dealt with in the next section. and secondly the method by which the information will be presented and passed on to the client/end user.

co. The metrics used to illustrate online exposure take a more qualitative approach in that they seek to limit the sample by confining the content to a 18 . There are however opportunities to create a trans-media comparison. Newspapers and national television broadcast news fare particularly poorly with this critical demographic group. in the absence of web page specific audience data. the only course of action is to consider online exposure aside from traditional media. Web log data for page visitor numbers are not publicly available making an estimate at best an educated guess. compared to 19% who look at a newspapers and 16% who watch a national TV broadcasts.Michael Blowers Media Evaluation Research 2006 The survey indicated that online exposure is currently viewed on a par with radio and TV coverage in terms of importance.com. Whereas with the traditional media it is possible to identify with some level of accuracy the opportunities-to-see for most of the mainstream publications and most slots on TV and radio. Site impression data is available for the larger websites from providers like Neilsen-Netratings and Media Metrix. and some way behind the press. compared to the traditional press.com.uk at least once a day. which will be discussed shortly. Their findings indicate that 44% of this group use news portals like Yahoo. radio and TV where an audited stream of audience data is currently available. it has become increasingly problematic to isolate with any kind of precision the opportunities-to-see for most online exposure. but this is where the commonality ends. MSN. For example the BBC website receives approximately 2 million visits a day. This is possibly surprising considering the research undertaken last year by the Carnegie Corporation on the news dietary habits of 18 to 34year-olds. condensed into a form where it can be presented to stakeholders outside PR. The introduction of a new set of metrics specifically to represent online exposure is unlikely to be considered a positive move when it comes to conveying media impact. Can someone with a mention on one of the sites 4 million pages claim all 2 million visitors saw the reference? This is obviously not supportable and. The survey supports this where respondents indicated that the most popular measure currently adopted is cross-media comparison. This action is regrettable as the use of media content data sees a number of relative metrics for different media placed alongside each other. There has been a fundamental change driven by technology and market forces and Carnegie’s research has found that this segment of the population intends to continue to increase their use of the Internet as a primary news source in the coming years. and BBC. What are the implications for media content analysis of the shift to online use feeds? There are obvious issues regarding the comparative metrics used to measure online coverage.

A typical set of categories for valuation of online exposure would be: UK Sites Opportunities-to-see High use site Medium use site Low use site 1.250 Is this style of categorisation the best-fit solution to an almost impossible problem. but would it then be representative to compare exposure on BBC1 TV equal to a reference on a blog? On a practical level KD Paine (2002) recommends the creation of categories for online coverage on the basis of an estimate for penetration/ viewership.Michael Blowers Media Evaluation Research 2006 specific sector or country. The online environment has created a wealth of consumer generated comment and it is possible for organisations to track in a meaningful way the value of this comment. There are obvious implications for privacy and confidentiality and so is unlikely to ever happen. Is there a case to start blogging.000 345.000 17. or is it pointless to even try? The ideal is to create a central repository of web log data collating all the individual page viewing as a method of allocating audience impact. It would be possible to simply rate volume of reference. put a friendly front door on the Web site or embark upon a “conversation” with consumers? With the absence of an audience impact score for online coverage it becomes difficult to rate the key variables across a selection of media. Is customer service and other business processes helping or hurting online buzz. Connection with marketing initiatives: level of engagement. what issues are coming around the corner. Instead the future is to work constructively with the methods available and to remain 19 . trends and specific issues are influencing a marketplace or brand. and where. The level of influence and control of word of mouth within the online environment. echo or backfire with consumers. What is being said. and what events. and are they consumers who are influential. resonance. Organisations like Nielsen BuzzMetrics and Visible Technologies aim to present the consumer generated marketing for individual sectors like pharmaceuticals or the automotive industry measuring the consumer sentiment using a number of general measures including: • • • • • • • How do consumers feel about a brand A measure of how many consumers are talking online. and how many other consumers are influenced by the conversation What specific issues are being discussed.150.

The pertinent aspect is that the information resulting from the different stages of associated research increase in ‘value’ as one ascends the pyramid. Macnamara (1992) pioneered the concept of media Inputs. Outputs and Outcomes through his Pyramid Model of PR Research. be-it attitudinal or behavioural. 20 . One such process is the BBC’s ‘News most popular now’ which demonstrates how with online data it is possible to make valued deductions on story outtakes. culminating in the achievement of the desired outcome. reproduced below. It is accepted that media measurement is adept at providing a measure of media inputs and outputs.Michael Blowers Media Evaluation Research 2006 open to the new ideas and opportunities which the technology will make available. The model works as a process guide with the wide base representing the planning process.

There is however some currency to it and it can be imagined that in future PR people will be saying their story was the second most read on the BBC website during a 24 hour period. In regards to the measurement of a wider sample of the web. It indicates which stories on the site are being most often read and emailed. there would be a considerable benefit if it were possible to include the access data for a wider segment of the web beyond the BBC’s site. There are obvious shortcoming – the system lacks any metrics and is only a rank of ‘relative’ popularity. This extract indicates a sample of what can be viewed: This is a classic example of online information ‘leap-frogging’ conventional media content analysis. 21 . In effect if it were possible to create a solution similar to the BBC’s ‘News most popular now’ but covering a wider sample of sites and with the addition of the ability to sort the results by specific country. for individual topic areas and regional zones. time-frame and using a measurable metric. going straight to media out-take (and one valuable step nearer to outcome).Michael Blowers Media Evaluation Research 2006 The BBC’s ‘News most popular now’ does one thing you can not do with the traditional media.

While it has not been central to this study it could become relevant to widen the remit by considering the impact of advertising.Michael Blowers Media Evaluation Research 2006 then this would represent a powerful and authoritative measure of online users out-take. there was a feeling that you were dealing with a finite quantity. If the public see communications as a single message then why don’t organisations bring its forms of communication together? Or is that what marketing communications aims to achieve? Organisations benefit from understanding ‘the message’.google. When dealing with the traditional media. the CIPR’s representative on evaluation was quoted in the Financial Times (20 June 2006) as saying “Consumers can't divorce PR and advertising. does raise issues on organisations structure. with advertising and PR often being separate. This is an interesting concept and although moving outside of the remit of this study. Specifically it also raises a series of questions. they consume communications holistically". It graphically presents the results and assigns links to specific stories to illustrate ‘spikes’ in the data. With the pervasive nature of online coverage and in particular blogging. including Google trends (http://www. A new skill will be understanding which of these are significant to stakeholders. With the explosion in content so there has to be an increasing risk of analysts not collecting the right sample and getting the results wrong. or run their own radio or TV station. This has to be seen as a threat to the industry which could become very real if one of the large online software organisations decides to further develop one of the existing semi-intelligent text analysis programmes. anyone now can publish electronically a newspaper. This could 22 . This need has been partly answered by a new breed of online analysts who specifically aim to understand and represent the web-players in a specific community or interest group. from what ever source.com/trends) which categorises the number of times various search terms have been used over set periods. Other options exist. locations and languages. Personal reflection My own view is that the fragmentation in the media is now at the point where the skills to capture the relevant coverage need to be as sophisticated as the resulting content analysis. Claire Spencer. but is the public not able to differential when they are being pitched to? That is not to say that PR and advertising messages can not be complimentary and co-ordinated to maximise impact.

lower level (and low cost if not free) online analysis package. and get it 80% right. The various processes which support quantitative and qualitative media measurement when effectively combined with rigorous methodology can create an accurate record of media outputs. The discussion also included some illustrations of the methods which can be adopted to measure media out-take (one-step nearer to outcome). Google. Primary research has pointed towards the growing importance of key message and favourability analysis. It will be particularly difficult to use comparable metrics to present media impact from traditional and new media largely because of the lack of effective audience data for the web. As has already been covered. Conclusions The task of this report has been to present a critique of the current provision and academic background supporting media content analysis. 23 . has already demonstrated through its Trends programme that it is thinking about this area. A number of research studies have indicated that PR will need to adjust to the growing importance of online media. there is the view that content analysis will always be able to be applied to understanding message usage and favourability. There is also a growing need to present an effective link between media output and organisational outcome.Michael Blowers Media Evaluation Research 2006 easily result in a widely adopted. a firm well adapt at finding coverage. This medium carries a whole selection of research challenges and opportunities. However if the end users can appreciate the complexity of the task then the industry will take one valuable step towards an effective solution. What ever happens. the metrics by which these are measured could pose challenges. All of sudden a PR results system if available which is able to track key messages and favourability.

How often do you report results on media content (check more than one. If you would be interested in finding out about the results we will be more than happy to send you a copy of the report. if relevant) 7 3 Monthly Quarterly 8 5 After all campaign Other (please indicate when) 24 . Responses numbers indicated in blue 1.most important 4 – least important) 1 4 Press Radio =2 =2 Television On-line 5. What types of media do you measure and how important are they to you? (Please number 1 to 4 in importance (1 .Michael Blowers Media Evaluation Research 2006 Appendix Media Measurement Survey This survey explores what media measurement methods are used by public relations professionals. Do you currently or have you in the past two years conducted measurement yourself? (highlight one) Yes 12 No 7 Unsure 0 4. For classification purposes. There are 13 questions which should not take more than 5 minutes to complete. please indicate where you work: 1 2 2 2. Independent practitioner Small agency (2-10 people) Large agency or agency (20+ people) 13 1 In-house department Medium-sized agency (11-20 people) Do you currently or have you in the past two years used outside companies for measurement of your public relations activities? (highlight one) Yes 11 No 8 Unsure 0 3.

Michael Blowers Media Evaluation Research 2006 6. local vs. editorial. color. photos. news release. national TV. Assignments tend to be evenly divided between doing them internally and using outside counsel and/or suppliers. Most of the work is done by outside counsel and/or suppliers. During the next two years. those you do use. Don’t use 4 11 13 4 6 0 6 7 11 11 4 12 6 10 8 10 12 12 12 11 6 Currently use 13 7 5 13 12 17 9 9 7 8 12 6 12 6 7 7 5 5 5 5 12 Would like to use 0 0 0 1 0 0 3 2 1 0 3 0 0 2 1 1 1 2 2 Item/Element Measured Column inches/broadcast time Advertising value equivalency without multiplier Advertising value equivalency with multiplier Reach (also called ‘opportunity to see’) Total number of placements or media impressions Media outlet category (daily/weekly paper.) Targeted geographic area (region. feature. press conference. do you anticipate that the volume of public relations measurement studies that your department or organization carries out annually will: 3 6 9 0 0 1 Increase by10% or more Increase by 1-9% Remain approximately the same.) Key message content analysis Targeted media (byline. etc. etc.etc. Decrease by 1 . country. Which of the following statements comes closest to describing how your department or organization currently handles public relations measurement? (PLEASE SELECT ONLY ONE RESPONSE) 3 9 4 3 1 Little or no measurement work is done. charts. Most of the work is done internally. specific broadcaster. unsolicited.) Source of story (media pitch. product review.) Use of branding (verbal and visual) Use by targeted journalist Client target audience or potential audience reached Client product/service mentions Competitor mentions Competitor service/product mentions Client/Company share of voice compared to competitors within story Competitor messages covered Spokespersons mentioned/quoted 25 . Other (please specify) __________________________________________________________________ 7. and those you would like to use. graphics. ADI markets.) Presentation (text only. etc. etc. etc.) Story form/type (hard news.9%) Decrease by 10% or more Not sure 8. Which of the following measurement methods are you currently using? Please indicate those that you don’t use. wire service.

etc. 6 7 7 14 14 13 13 11 8 10 2 4 5 5 2 2 1 2 0 0 1 If there are other items you measure or would like to measure that are not included here.Michael Blowers Media Evaluation Research 2006 Tone/stance toward client or company (also called slant) Headline tone/stance/slant Content favorability/unfavorability Call to action (phone/web/mail inquiries traceable to story Cost of message placement/distribution (news release distribution fees. etc.) Number of media representatives attending event/press conference Target audience response (via Web response. Looking into the future. brochure printing. please write them in the following spaces and provide a brief explanation of what they measure. Currently use Would like to use 1 Item Measured Donations to campaigns Tracking research 1 10.) 9. which of the following aspects do you think will become more or less important? Less important 2 5 0 0 0 0 0 0 1 3 3 2 2 Stay the same 11 8 11 6 9 8 2 6 6 8 5 7 3 More important 3 3 8 11 10 9 14 11 11 4 5 4 10 Not sure 3 2 0 1 0 2 3 2 1 5 5 6 3 Measurement by audience/opportunities to see Measurement by advertising value equivalent Media monitoring Key message content analysis Favourability of media content Competitor analysis Connecting media output with organizational outcomes Presenting a return on investment Creating tangible results for PR activity Academically proven methodology Results presentation techniques Results clarifications/accessibility Accurately reflecting on-line media exposure 26 . postcard return. phone inquiry.

scoring them accordingly Less important Blog monitoring Tracking product launches Stay the same More important 1 1 12. and 7 is strongly agree. Please indicate your agreement with the statements below. Strongly Neither Agree nor Disagree Strongly Agree Media measurement methods… Are easy to use Disagree 1 1 1 1 1 0 1 0 1 0 1 0 2 0 2 1 2 1 2 3 2 2 2 0 3 0 3 2 3 2 3 4 3 1 3 4 4 10 4 4 4 4 4 4 4 6 4 2 5 5 5 6 5 5 5 5 5 3 5 5 6 1 6 4 6 6 6 3 6 6 6 4 7 2 7 1 7 1 7 0 7 1 7 2 Measures what I want to measure Produces usable data Are understandable by clients/non-PR managers Answers my questions about program success Are usable in future strategic planning 27 .Michael Blowers Media Evaluation Research 2006 11. where 1 is strongly disagree. 4 is neither agree nor disagree. If you can think of any other relevant issues to media measurement missed out in the list above please indicate them in the table below.

and 7 is strongly agree. Strongly Disagree Neither Agree nor Disagree Strongly Agree I use media content analysis for strategic planning purposes 1 0 2 2 3 3 4 3 5 5 6 3 7 4 My current media content analysis measures what I want to measure 1 0 2 5 3 1 4 3 5 4 6 4 7 3 My current measurement methods produces usable data 1 0 2 1 3 1 4 6 5 5 6 3 7 3 I use media measurement to justify budget requests 1 0 2 2 3 3 4 4 5 6 6 0 7 2 The movement of media to on-line sources will make it difficult to effectively represent our output 1 0 2 2 3 2 4 6 5 3 6 1 7 4 Strongly Disagree Neither Agree nor Disagree Strongly Agree I use media measurement as a political defense against other departments 1 4 2 1 3 2 4 8 5 2 6 1 7 1 I use media measurement to educate executive management on the role/value of public relations 1 0 2 2 3 1 4 2 5 6 6 6 7 2 I use media measurement to educate colleagues on the roles/value of public relations 1 1 2 2 3 2 4 3 5 3 6 6 7 2 I use media measurement to educate colleagues on the attitudes/opinions of key stakeholders 1 0 2 3 3 2 4 2 5 4 6 6 7 2 Many thanks for your help and if you would like to add any comments or thoughts please continue below. where 1 is strongly disagree.Michael Blowers Media Evaluation Research 2006 13. Please indicate how well you agree with the statements below. 4 is neither agree nor disagree. Michael Blowers 28 .

pria.. Qualitative content analysis. (2003) Public relations principles and practice Thomson Lasswell.pria. K.Michael Blowers Media Evaluation Research 2006 Bibliography Berelson.com. The Social Semiotics of Mass Communication.com. Hoover Institute and Library on War. from http://qualitativeresearch. J. non-financial indicators? Macnamara J. New York: Hafner. & Pool. (Thousand Oaks. 1.. E. Lerner. Stanford University Press.(2003) The CIPR Toolkit: planning a research and evaluation for public relations success. (2004) Strategy Maps (converting intangible assets into tangible outcomes) (Harvard Business School Press) Kitchen P. (2003) Return on investment. A. from http://jerz.a dubious business Jensen. I. Revolution and Peace. Campbell. The Content AnalysisGuidebook. (1999) Effective Public Relations (Prentice Hall International) Fombrun C.edu/weblog/permalink. London: Sage Kaplan and Norton. (1999) Performance measurement: can PR/communications contribute to the new bottom line of intangible.(1952).au Mayring.au Macnamara J. D.. (2003) The secret life of pronouns: flexibility in writing style and physical health.au Macnamara J. Strategic Communications Management (2004) Calculating ROI . (1952) Content analysis in communication research. Merrill Brown. Views on reputation management Neuendorf. Online journal. (2002). Carma/PRIA. www. CA: Sage Publications) 29 . 2004. the brand and the bottom line Hutton P. S. D.pria. (2002) PR metrics-research for planning and the evaluation of PR and corporate communications. (2005) Corporate reputation. (2004). 1. Carma/PRIA.net/fqs/fqs-e/2-00inhalt-e. & Pennebaker. P.htm. Psychological Science14. (2). No.com.jsp?id=885. 2003. de Sola. W.setonhill. Forum Qualitative Sozialforschung/Forum Social Research. 60-65. Cutlip Center & Broom. R. Likely F. www. B. London Haywood R. (2003) Media content analysis-its uses and benefits and best practice methodology. (1995). B. K. Carnegie Reporter (Spring 2005) Abandoning the news Murrey & White. Chartered Institute of Public Relations. Retrieved July 19. www.(1996) Reputation: Realizing Value from the Corporate Image (Harvard Business School Press) Fairchild. The comparative study of symbol: an introduction. Retrieved December 18. H. Mass communications/PRIA. (2003).

(2001). (2001) The PR Handbook (Routledge) Van Zoonen. www.com www.com www.blogs.co. R.uk) Jon White and John Blamphin (1994) Priorities for Research into Public Relations Practice in the United Kingdrom.com www. Content analysis in communication: assessment and reporting of intercoder reliability. The media book. & Cowdery. C.brandweek. Boyd-Barrett.amec. K. M.com www. (London: Sage) Westaby M. (Jan'06) How PR works: media relations (Admap) White J.org.kdpaine. Washington. (2002). (1994)..co.. (White Plains. Feminist media studies. O. J.com/bw/news/spotlight/article_display. & Lombard.com i Association of Media Evaluation Companies. S. KD (2002). (London.mediaproofonline.. and Reese. City University Business School/Rapier Marketing) http://news. L.jsp?vnu_content_id=1002876797 www.cipr. NY: Longman) Spencer C. P. (Basingstoke: Macmillan) Shoemaker. How to study television.nielsenbuzzmetrics. (1995). C. Mediating the message: theories of influences on mass media content. London: Arnold (Hodder Headline).bbc.uk/1/shared/bsp/hi/live_stats/html/map. (2005) The evaluation needs of the PR industry Selby. J. DC.uk 30 .alexa. (Presentation to the International Communication Association. D. (2005) Measurement and Evaluation: moving the debate forward (www.nielsen-netratings. (2004) PR past and future – onward and upward (World Advertising Research) Synder-Duch. (1996). H. May) Theaker A. C. Paine. Bracken.stm www.Michael Blowers Media Evaluation Research 2006 Newbold. Measures of Success for Cyberspace Phillips D. & Van Den Bulck.

Sign up to vote on this title
UsefulNot useful