Cracking content a guide to measuring the media, present and future

Michael Blowers
5, St. Johns Lane London EC1M 4BH Tel: +44 (0)20 7549 2829 www.mediaevaluation.eu

Michael Blowers

Media Evaluation Research 2006

Abstract:
The current methods of measuring public relations in common use range across a variety of disciplines, subject area and depth of research. On one level, traditional quantitative media research, measuring column inches and advertising value equivalent, still represent a sizeable portion of the lower cost / do-it-yourself end of the market. These quantitative measurement techniques have been around for many years and while there is pressure to replace them with a more sophisticated approach, they will likely continue to have a role. At the opposite end, there are PR departments within large organisations undertaking integrated research programmes (both quantitative and qualitative) on a continuous and project basis. Often these projects are undertaken with help from specialist agencies and feature analysis of media exposure with the option to integrate results from market research and other forms of market analysis. In between, there is a broad range of media analysis projects largely coming under the term referred to in the CIPR Toolkit on media evaluation (July 2003) as media content analysis. This involves the analysis of media data using a number of metrics, both quantitative and qualitative, involving the gathering and analysing the content of text (Neuman, 1997, 272-273). Specialist agencies dealing with media evaluation have at their core the analysis of message and favourability. Message analysis often comes in two forms. Most often a brand or organisation will have a series of messages it would like featured within the media coverage. These messages could be about some unique property of a product, or some general qualities with which the organisation would like to be associated. Additionally, there is an application for the tracking of other messages relevant to the organisation. These will be the responsive messages, or the things which the media are saying about the organisation. Sometimes negative, these messages provide feedback particularly when undertaken alongside a similar analysis of competitors. The result is the building of a picture of the organisations media profile and underlying sentiment and is often viewed as a tool to message refinement, feedback and planning (Macnamara, 2003). The other main qualitative measure applied is favourability. There is no uniform method of measuring favourability, although the most commonly applied uses the categories of positive, neutral and negative. Other ratings

Michael Blowers

Media Evaluation Research 2006

which exist include a +10 to -10 rating, and strongly favourable, slightly favourable, slightly unfavourable and strongly unfavourable. There are a number of other variables which are frequently tracked within the media exposure including the use of branding, subject or topic analysis, media pro-activity and the use of press releases, spokespeople and individual journalists and media titles. These are frequently cross referenced against favourability as a way of drilling back to how the good and bad publicity arose. This report will use primary and secondary research to illustrate which of these techniques and processes are most commonly adopted and provide some indication of the importance with which PR attaches to them. The primary research will be undertaken using a questionnaire to sample respondents attitudes towards different methods of measurement. A blog has also been created to convey concepts and ideas, ahead of their use in this report. A number of example media content reports have also been sourced which will form part of the secondary research. The secondary research will examine published research and the methods of measuring media content, indicating the shortcomings and potential benefits of various methods and systems. It will be necessary to discuss the comparative role of the different types of media, from measurement perspective, and to the greater PR community. It is also relevant to consider the likely changes that media measurement will go through in the coming years; which will lead on from a discussion about the recent changes which media measurement has already been through. Key to this topic will be the spotting for likely trends and the significant influences to the process. It is anticipated that this can be segmented into areas of changing demand and changing supply. Changes in demand reference the changes PR has had on media measurement. The change in supply refers to how the media will change in the coming years. For example, the continuing decline in the circulation of newspapers is well charted and in May 2005 Rupert Murdoch spoke about his increasing concern over the gathering pace of change in peoples news access habits. The implications of this shift are fundamental for PR. According to the Carnegie Corporation (2005) young people (aged 18-34) are over twice as likely to refer to the web for news compared to the traditional newspaper press. Anyone in PR now trying to target the young can not ignore the online media and as this characteristic feeds through to other age groups so the necessity for online communication will become paramount.

Michael Blowers

Media Evaluation Research 2006

With the trend for increasing use of online media sources, PR will be looking for a method of justification. In the same way PR uses the key metric of audience to measure their impact in the traditional media, it will obviously look to apply the same measure for online exposure. This would seem common sense, particularly if importance is placed on comparing results between different media. Within this discussion there will be an examination of the types of audience statistics available for online sources, with a view to framing the notion that audience is not an accurate and supportable method of measuring media output. Consideration will then be given to examining other methods of measuring online media output. This reports role is to summarise the challenges facing the discipline, and conclude with a number of recommendations applicable to both strategy and ultimately the tactics of media content analysis.

Introduction
If judged by the amount of academic research and discussion filling the specialist press and more recently the Financial Times , it would be reasonable to say that PR is taking measurement increasingly seriously. White and Blamphin (1994) found that evaluation ranked number one amongst a list of priorities for PR in a UK study of practitioners and academics. When considering the methods of measuring the media, reference is being made towards the activity of media content analysis. This report seeks to shed light on what current methods are being adopted by the PR community. Consideration will be given to the development of media content analysis, the trends resulting in the present situation, and a view on where this might be taking PR measurement in the future. Media content analysis is a specific function making up part of the greater discipline referred to as media evaluation. Additionally, media evaluation would include activities relevant to planning and targeting, message selection, pre-testing, through to interim sampling and message refinement. The final phase is made up of post examination, with media content analysis usually composing a large share of the research. According to AMECi media evaluation is a strategic and tactical tool for people working within PR, marketing or business intelligence. The designated sources of research compose a questionnaire and blog for primary research and desk research making up the secondary sources. The questionnaire was circulated to a number of PR representatives with the aim

Michael Blowers

Media Evaluation Research 2006

of understanding what types of method they currently use for analysing media content, how they used the results and their thoughts on if this might change in the future. The blog (http://mediaevaluation.blogspot.com/) was to act as a sounding board for ideas and the chance to explore new ideas and concepts before committing them to this report.

Media content analysis

According to Neuendorf (2002,1) in the field of mass communications research, content analysis has been the fastest growing technique over the past 20 years or so. The concept of media content analysis was introduced as a method of studying the mass media by Harold Laswell in 1927, who initially used it to study propaganda (Lasswell, 1927 in Newbold, et al, 2002,79). Media content analysis became increasingly popular during the 1920s and 1930s for investigating the rapidly expanding communication content of movies. Its application to the public relations industry was apparent, something supported by Lasswell (1952, 34) who said content analysis is a technique which aims at describing with optimum objectivity, precision, and generality, what is said on a given subject in a given place at a given time. Other early definitions of media content analysis concentrated on the quantitative elements of the discipline. Stone, Dunphy, Smith and Ogilvie (1966, 5) describe content analysis as a new research technique for making inference by systematically and objectively identify specific characteristics within text. This is supported by Neuendorf in her benchmarked text, The Content Analysis Guidebook (2002), where she argues that media content analysis is a quantitative measure, not qualitative. She strongly argues for the use of scientific methods including attention to objectivity-intersubjectivity, a priori design, reliability, validity, generalisability, replicability, and hypothesis testing space(2002, 10). Other authors of media content analysis do not support the notion that it is a strictly quantitative research tool. In McNamara (2003, 3) the renowned author on media content Pamela Shoemaker (1996) argues for the inclusion of qualitative techniques through her argument on the relevance of the humanist and behaviourist tradition, the former which looks to identify what the media is saying about society, while the latter looks to identify future affects.

Michael Blowers

Media Evaluation Research 2006

It is clear that from the research undertaken by Macnamara (2003, 9) that the majority of the media content research expected by PR should be described as quantitative. Berelson (1952) in Neuendorf (2002, 52) suggests five main purposes of content analysis: To describe the substance and characteristics of message content To describe the form characteristics of message content To make inferences to the producers of content To make inferences to the audiences of content And to predict the effects of content on the audiences

Carney (1971) simplifies this by suggesting that content analysis can be firstly useful for its descriptive qualities, secondly as a method of testing hypothesis, and finally for facilitating inference (Neuendorf, 2002, 52). McNamara (2003, 4) highlights three qualities most closely associated with media content analysis and which will have the greatest relevance to the social sciences: 1. Descriptive 2. Inferential 3. Predictive The descriptive role provides an insight into the specific messages within the discourse represented in mass media, while the inferential and predictive roles of content analysis facilitate (rather than being conclusive) an exploration into the likely effects of mass media representations on society and audiences.

Methods of evaluating the media

There is broad agreement that when undertaking media content analysis the greatest success has resulted from following these procedures: 1. Computers are an integral part of this type of research and are most commonly used for storing data such as coding annotations by researchers, analysing data and reporting; for example creating tables charts or graphs. Attempts have been made to create systems to automatically scan text and identify the coding of words and phrases, although most leading researchers in this area do not support this approach: The notion of a completely automatic content analysis via computer is a chimera. the human contribution to content analysis is still paramount (Neuendorf, 2002, 40).

Michael Blowers

Media Evaluation Research 2006

2. The inclusion of a priori design will enhance to be objectivity/ intersubjectivity of the research as it will prevent the list of issues and messages being added to as the research is undertaken. Neuendorf (2002, 102-103) suggests examining a representative subset of the content with a view to concluding all decisions on variables, their measurement, and the coding rules before the observation begins. 3. Unreliable coding can fundamentally undermine any media content analysis project. Macnamara (2003, 10) indicates that reliability requires that two or more coders be used, something which is supported by Tinsley and Weiss (1975, 359) who said obtaining ratings are not the idiosyncratic result of one raters subjective judgment. Supported by training, coders should be prepared to submit blind coding to help support the level of confidence in the resulting research. As a practical measure Macamara suggests that at least 10% of the media content is analysed by a different coder, with a view to maintaining consistency. To support the resulting analysis it may be necessary to provide information on intercoder reliability in addition to the code book/coding list and details of the methodology used (Snyder-Duch, et al, 2001). 4. The validity of content analysis is maintained by thoroughly understanding the research objectives. This can be achieved by the coder thoroughly immersing themselves in the message pool before commencing the sample, as a method of agreeing the key variables and messages to be tracked. 5. Generalisability refers to the extent to which research findings can be applied and taken as a measure of the target population. It would follow that the generalisability is largely determined by a representative and sufficiently large sample, combined with an appropriate methodology. 6. Replicability is the ability and degree of difficulty for the coders to replicate the research, with a view to confirming or challenging the results. Replicability is determined by the full disclosure of information on methodology and procedures, and would specifically involve the code book/coding list, coding guidelines and instructions to coders. As the body of research relevant to media content analysis increases a key feature will be the increasing adoption of this best practice, as supported by the research.

Michael Blowers

Media Evaluation Research 2006

The application of quantitative research would provide results in five key areas: Media prominence which would take account of the impact factors such as page number, the order in an electronic media bulletins and use of photos and visuals. Media weighting to reflect high circulation, high rating or influence. Size of the article or a length of broadcast segment. Positioning would track the use of headlines, first paragraph mentions and other prominent mentions, as well as representing passing references and share of voice. Sources would illustrate the balance of supportive and opposing sources quoted within the text (including comment on position and credibility)

Qualitative content analysis is most often undertaken simultaneously with the quantitative analysis. Positive and negative words and phrases can be analysed to identify the tone and text and in addition analysts can record notations during coding in relation to contextual factors. Proponents of qualitative text analysis highlight the following qualitative factors as having the greatest relevance to PR: 1. Audience characteristics; different groups of readers (age, gender, race, ethnicity, education and socio-economic position) will have a different take on media content. 2. Context; other events within and outside the media which can magnify or alter the message. 3. Perceptions of media credibility; different sources within the media have a level of credibility which can alter the importance of the message. Source: Macnamara, 2003, 6 Undertaking qualitative message analysis relevant to the media would include analysis of text, rhetorical factors, interpretive signs, narrative, semiotics and discourse. These can be condensed down to two main themes: narratology (examining the narrative or storytelling within the text), semiotics which examines the signs within the text and how the reader might interpret and decode them (Newbold, et al, 2002, 84).

Michael Blowers

Media Evaluation Research 2006

One of the greatest challenges for those undertaking qualitative analysis is the maintenance of objectivity and repeatability. Newbold,et al warn: The logic of deconstructing latent meanings, and privileging them over the more obvious manifest ones, is questionable, for the audience may not see this latent dimension; the analysis may be longer than the text. The task is time-consuming, and often tells us what we already know in a language we don't understand. The essential concept that underlines this approach is that words and images are signs that stand for or signify something else beyond their obvious meaning. Together they form codes: -collectives of signs that produce certain meanings. The development of semiotics point towards a structuralist approach: seeking the meaning of signs according to a set system. According to Macnamara (2003, 16) this contrasts with later post-structuralist influenced semiotics where the work of Klaus Brahn Jensen (1995) combined the early social semiotic theory of mass communication with the later work on active audience participation to interpret mediated meanings. Newbold, et al said: So in studying media texts ... we can use these ideas as they can provide a way of assessing the meaning in a text. The techniques adopted for qualitative analysis are less prescriptive than for quantitative analysis. Van Zoonen (1994) however attempts to explain how semiotic analysis can be practically applied to media text, firstly by identifying the signs in the text and then the dominant characteristics. According to Selby and Cowdery (1995,58) these signs can be analysed as a result of selection and combination. Some of the key signs would include images and photographs - an image of a man holding a baby suggests fatherhood, family commitment and possibly gentleness and caring. Other symbols are recognised internationally, for example $ signifying money and signifying copyright. More significant are the symbols within language. Campbell and Pennebaker (2003, 60-65) and others identify pronouns as key signifiers of meaning within text. Macnamara (2003) highlighted eight areas where text elements can be studied for their qualitative content: Adjectives used as descriptors, these can be positive and negative and give a strong indication of the writer's stance. Whether verbs are active or passive voice. The viewpoint of the narrator (taking the first person, second or third person, etc).

Michael Blowers

Media Evaluation Research 2006

Tonal qualities, such as aggressiveness, flippancy, emotional language and sarcasm. Double meanings, how these are positioned and used. Visual imagery within the text. Use of metaphors and similes. Factors dictated by context: position and credibility of spokespersons, sources quoted and impact on the meaning taken from the text.

Mayring (2000) suggests two ways of analysing qualitative text, using inductive and deductive methodology. Importantly this procedure brings some systematic steps to qualitative text analysis. It involves the creation of a priori schema, which is a prescriptive framework created before the sample is analysed in detail. Specifically the method requires a matching of a category to a passage of text, and not the matching of text to a category. By starting with predetermined categories there is an increase in the systematicity of the qualitative analysis. Mayring also recommends that intercoder reliability be assessed, with only fully trained analysts being used. Qualitative data has the benefit of yielding representative results of a high probability, although according to Macnamara there is the tendency with qualitative to suppress or dumb down research findings.

The implications of a measurement culture on PR

As with many things, there are a number of fashionable management buzzwords organisations like to use. Return on Investment (ROI) is one; it aims to illustrate to management an areas ability to justify its contribution. ROI is not the only term; others include key performance indicators, benchmarking and balanced scorecards. However, as with many of these terms, its precise definition is somewhat unclear and it is often misapplied. In this instance there is a role for it to play in assisting PR because there is the rightful precursor of financial return and it follows that this will be a measure of outcome. As long as objectives are clearly established for PR and corporate communications the return on investment would be best described as the achievement of these objectives, whatever they may be (Macnamara, 2003, 5). That is not to say that its implementation should not come without a number of conditions, and many view financial return as unrealistic for PR for the following reasons:

10

Michael Blowers

Media Evaluation Research 2006

1. Public sector organisations and other not-for-profit groups do not have sales, profit or share value and other financial objectives. 2. Within commercial organisations financial returns should be viewed as a team effort and it would be inaccurate for any single business unit to claim credit for financial results. Judging PR in financial terms would be no more realistic than evaluating HR, R&D, and the legal department by their revenue generating capacity. 3. It is unreasonable to assume that the only form of communication between organisations and its publics will be via the communications/PR department (Hutton, 2004). 4. Often campaign objectives are not directly linked to sales or share price (even though they might eventually contribute towards these). 5. Often PR campaigns have a long-term set of objectives and are integrated with other forms of communication, therefore making it difficult to financially assess the impact of any one form of communication. Charles Fombrun (1996) suggested the idea of the triple bottom line which, as well as considering financial performance, also factors in an organisation's environmental activities and its community performance. The benefits of having all three of these factors working beneficially in unison is well charted and a number of case studies prove that getting one wrong has implications for the other two. For example Nike attracted worldwide criticism and falling sales because of its poor treatment of workers in developing countries. This negative coverage about the treatment of communities had an overall impact upon the other two factors. PR operates in all these areas and accordingly their contribution should be measured in more ways than just financial. The ways that PR can provide a return on investment would be by proving successful in the following areas: Fully informed and motivated employees Supportive and considerate relations with government and regulators Constructive and positive relations with key non-governmental organisations Well informed and ideally positive financial community Balanced or, preferably, positive media coverage

11

Michael Blowers

Media Evaluation Research 2006

Supportive and loyal shareholders Well informed consumers Support and understanding of local communities where they operate (Macnamara, 2003, 3)

When considering the return on investment for PR it is necessary to measure outcomes and not simply outputs (see page 19). It is also crucial that the measures adopted are aligned with the overall objectives of the organisation. While the measurement of outputs is usable, management wants and expects the representation of outcomes, be-it metrics on awareness, attitudes and behaviour. In effect these involve having a good reputation, building brand equity, making profits and gaining stakeholder support, etc. It is therefore relevant to highlight that return on investment for PR is not measured only in financial terms. Organisations need to consider very carefully their objectives; to see that they are clearly established, understandable, achievable and measurable. Allied to this, PR needs to have its own selection of specific objectives, complementing and contributing to overall organisational objectives. They should however be sufficiently discreet to make outcomes traceable aside from the objectives of other departments. The methods of measuring return on investment for PR will be dependent upon the activities and objectives to be measured. One key area which can contribute is media content analysis, determining key message placement, target audience reached and comparative share of voice. It will also provide capabilities to consider trends, make predictions and illustrate favourability. There is also the capacity to examine competitors and contemporaries motive as a way of understanding the impact of alternative strategies. Whilst the limitations of media content analysis have already been considered, when it comes to measuring return on investment it should be included as one of the methods alongside a selection. For example, it may be necessary to complement the media analysis results with a detailed understanding of stakeholders and target audiences, requiring research such as surveys, focus groups and/or interviews with representative groups and stakeholders.

12

Michael Blowers

Media Evaluation Research 2006

Characteristics of external analysis providers

The majority of external media content researchers are able to provide analysis of the quantitative research elements as highlighted previously. Fewer undertake the qualitative research components, as they have been described. There are a number of commonalities and in Appendix 1 a sample selection of reports have been examined with a view to illustrating the comparative approaches and methodologies etc. Most (though they might not advertise it) are able to provide equivalent advertising value data and some highlight their ability to track messages and competitor activity over time to portray an impression of likely trends. Only one providers sample (Carma International) featured message qualities (eg: good customer service) separated into positive and negative. The majority of providers use positive, neutral and negative to classify exposure. Few defined the factors or the process involved in the classifications. Carma International uses a 0 - 100 score, where 50 is neutral. Metrica feature a 4 stage classification of strongly positive and negative and slightly positive and negative, and clarify that neutral coverage would be ascribed a slightly positive rating. The factors which influence favourability would include branded use in the headline and body-copy, style of narrative and language, underlying motive and use of spokespeople. This is not an exclusive list and often the discipline is accused of approaching the issue in too subjective a way, resulting from a lack or defendable research to support its application. Two providers use weighted opportunities to see (WOTS), instead of audience figures/ opportunities-to-see (MB Precis and Panarc), although from the information available each base it on a different methodology. In almost all the sample reports there is an explanation of the methodology and terminology featured, however there is also the impression that the reports are aimed at a PR orientated/trained audience, making anything beyond the summary of little comprehension to those not in PR.

PRs view of media measurement

To support the secondary research a questionnaire was circulated to a number of PR professionals for their views on media content analysis. The majority came from in-house PR departments with the remainder being spread amongst agencies of various sizes. The reason for using a questionnaire was flexibility as a method of collecting detailed feedback relevant to the specific area under consideration. It was designed to be

13

Michael Blowers

Media Evaluation Research 2006

concise and quick to complete, and adopted a series of measures including the Likert Scale (1932), which aimed to establish a level of agreement with a series of statements. Just over a half of respondents had over the past two years conducted media measurement with the assistance of an external service provider. This however wouldn't appear to be exclusively the case and two-thirds had also undertaken media measurement themselves. This was subsequently supported by the majority of respondents indicating that most of the media analysis work is done internally. The picture is therefore of the majority of PR representatives undertaking research in-house while just over half also seek the assistance of external providers on an occasional basis. Interestingly 16% of respondents indicated that no measurement work is currently done. 16% indicated that research is conducted mostly using outside counsel or suppliers, and 16% also indicated that assignments tend to be evenly divided between internal and outside suppliers.

Assignments evenly divided betw een doing them internally and suppliers 16%

Unsure 5%

Little or no measurement w ork is done. 16%

Most of the w ork is done by outside suppliers 16% Most of the w ork is done internally 47%

Almost half of the audience indicated that the majority of research is undertaken internally. 44% of respondents indicated that the amount of research undertaken would broadly stay the same. This compared to 33% of respondents who indicated that the amount of research on the media would increase by 1-9%. 17% of respondents indicated that the increasing amount of research would exceed 10% of the current level. None of the respondents indicated that the amount of research would decrease. When asked which of the media was of greatest interest almost every respondent indicated that press was the most important. The remaining
14

Michael Blowers

Media Evaluation Research 2006

mediums were very closely ranked with online and television coverage being marginally ahead of radio. The questionnaire featured a list of the main media measurement techniques from which respondents were asked to indicate their preference and interest. The aim of this, the main part of the questionnaire, was to compare and contrast these methods and highlight those which could become more popular. The most used media metric were column inches/broadcast time, reach (also known as opportunities-to-see), and volume of media placements. Although not as large in number it should be noted that 60% of respondents indicated that they currently use advertising value equivalents. Of those there was a 40:60 split between those using and not using a multiplier. Almost all the respondents categorised their output into different media groups, making it the most popular variable to track. Half of the sample linked their media output with the methods of news generation; for example news release, media pitch, press conference, etc. It is notable that 17% of respondents would like to adopt this measure. Fractionally less popular was categorising media output by the type of news feature, whether it was a product review or editorial reference. A third of respondents indicated that they also track coverage by geographical area; for example by region or country, etc. A similar number also track their coverage for inclusion of graphics, charts, photos and use of colour. It should also be noted that few were interested in using these measures if they did not already do so. The tracking of key messages elicited a positive response, with almost twothirds indicating that they currently use this measure. A fifth said that it would be a measure they would like to use in the future. The survey interestingly highlighted that only a third of respondents link their target media with that generated, and nobody indicated that this will be something they would like to use in the future. Almost a third of respondents indicated that they track media exposure for the presence of branded references, being verbal or visual. While half of respondents do not track the journalists who write about them, 11% did indicate that it would be something they would be interested in doing. One theme hard not to ignore is the apparent lack of targeting, which is further supported by only a third of respondents indicating that they track their media exposure against a given target audience. A proportion of the sample was made up of public relations professionals from the public and non-profit sectors which would support the findings that only a third of respondents were interested in tracking product and services mentions and references to competitors. 10% of the sample did indicate an interested in tracking the activities of their competitors in the media and in particular their key messages and share of voice.

15

Michael Blowers

Media Evaluation Research 2006

Over half of the respondents highlighted the importance of spokesperson usage in the media and a similar number indicated support for the tracking of favourability within the coverage. A notable number within the sample also highlighted this as an area they would like to apply in the future. The importance of tracking any calls-to-action within the media coverage, the cost of message placement and the number of media representatives attending events were largely downplayed by respondents with over twothirds indicating that they did not use these measures. An important aspect of this research was to try and establish the views on whether these measures are likely to become more or less important in the coming years. As can be seen in the appendix (which includes a record of the actual responses) the sample was asked to indicate if they think that a measure will become more or less important in the future. The following heat table indicates which of the measures are on the whole more or less likely to become important:
Measurement by audience/ opportunities to see Measurement by advertising value equivalent Media monitoring Key message content analysis Favourability of media content Competitor analysis Connecting media output with organizational outcomes Presenting a return on investment Creating tangible results for PR activity Academically proven methodology Results presentation techniques Results clarifications/accessibility Accurately reflecting on-line media exposure

More important Stay the same Less important

Although it is not indicated in this table as being so, the measure which was almost, but not quite, classed as less important was advertising value equivalent. A number of these factors were highlighted as likely to become more important. In particular there was the view that key message analysis and

16

Michael Blowers

Media Evaluation Research 2006

the measurement of favourability will become more important in future years. However the factor with the greatest expectation for likely future importance is the ability to connect media output with organisational outcomes, be-it via sales, share price or other method of ranking. There was also the view that increasingly PR needs to present a return on its costs and be able to express its results in a tangible manner. Finally, it was highlighted that there is a need to accurately reflect the importance of online media exposure, a topic to be dealt with in greater detail in the next section. In the final part of the questionnaire respondents were posed with a number of statements and asked to indicate how strongly they agreed or disagreed with them. On the whole people said that the measurement methods for the media where easy to use, that they were measuring what they wanted to, and that the data produced usable results. There was less agreement that the results were understandable by non PR staff, but there was an acceptance of the view that they would provide evidence for the success of a programme and would be usable for future strategic planning. There was the view that media measurement could be used to educate colleagues and stakeholders of the role and value of public relations. In addition it could be used to educate colleagues on the attitudes and opinions of stakeholders. On the whole there was an acceptance that the movement of media to an online environment will make it more difficult to represent the value of coverage generated. The challenges of measurement in the online environment will be dealt with in the next section. The questionnaire proved a valuable insight into the priority attached to various methods of measurement. It reinforced the view that message and favourability analysis are the most important roles for media content analysis. In hindsight it would have been beneficial if the final section of the questionnaire had concentrated more on the application of coverage collection and media content analysis within the online environment.

An insight into the future applications for media research

From the research undertaken it is clear that there are two particular areas where media content analysis will experience the greatest changes in the coming years. Firstly, there is the media from where coverage will come, and secondly the method by which the information will be presented and passed on to the client/end user.

17

Michael Blowers

Media Evaluation Research 2006

The survey indicated that online exposure is currently viewed on a par with radio and TV coverage in terms of importance, and some way behind the press. This is possibly surprising considering the research undertaken last year by the Carnegie Corporation on the news dietary habits of 18 to 34year-olds. There has been a fundamental change driven by technology and market forces and Carnegies research has found that this segment of the population intends to continue to increase their use of the Internet as a primary news source in the coming years. Newspapers and national television broadcast news fare particularly poorly with this critical demographic group. Their findings indicate that 44% of this group use news portals like Yahoo.com, MSN.com, and BBC.co.uk at least once a day, compared to 19% who look at a newspapers and 16% who watch a national TV broadcasts. What are the implications for media content analysis of the shift to online use feeds? There are obvious issues regarding the comparative metrics used to measure online coverage, compared to the traditional press, radio and TV where an audited stream of audience data is currently available. Site impression data is available for the larger websites from providers like Neilsen-Netratings and Media Metrix, but this is where the commonality ends. Whereas with the traditional media it is possible to identify with some level of accuracy the opportunities-to-see for most of the mainstream publications and most slots on TV and radio, it has become increasingly problematic to isolate with any kind of precision the opportunities-to-see for most online exposure. Web log data for page visitor numbers are not publicly available making an estimate at best an educated guess. For example the BBC website receives approximately 2 million visits a day. Can someone with a mention on one of the sites 4 million pages claim all 2 million visitors saw the reference? This is obviously not supportable and, in the absence of web page specific audience data, the only course of action is to consider online exposure aside from traditional media. This action is regrettable as the use of media content data sees a number of relative metrics for different media placed alongside each other, condensed into a form where it can be presented to stakeholders outside PR. The survey supports this where respondents indicated that the most popular measure currently adopted is cross-media comparison. The introduction of a new set of metrics specifically to represent online exposure is unlikely to be considered a positive move when it comes to conveying media impact. There are however opportunities to create a trans-media comparison, which will be discussed shortly. The metrics used to illustrate online exposure take a more qualitative approach in that they seek to limit the sample by confining the content to a

18

Michael Blowers

Media Evaluation Research 2006

specific sector or country. The online environment has created a wealth of consumer generated comment and it is possible for organisations to track in a meaningful way the value of this comment. Organisations like Nielsen BuzzMetrics and Visible Technologies aim to present the consumer generated marketing for individual sectors like pharmaceuticals or the automotive industry measuring the consumer sentiment using a number of general measures including:

How do consumers feel about a brand A measure of how many consumers are talking online, and how many other consumers are influenced by the conversation What specific issues are being discussed; what issues are coming around the corner; and what events, trends and specific issues are influencing a marketplace or brand. What is being said, and where, and are they consumers who are influential. Connection with marketing initiatives: level of engagement, resonance, echo or backfire with consumers. Is customer service and other business processes helping or hurting online buzz. The level of influence and control of word of mouth within the online environment. Is there a case to start blogging, put a friendly front door on the Web site or embark upon a conversation with consumers?

With the absence of an audience impact score for online coverage it becomes difficult to rate the key variables across a selection of media. It would be possible to simply rate volume of reference, but would it then be representative to compare exposure on BBC1 TV equal to a reference on a blog? On a practical level KD Paine (2002) recommends the creation of categories for online coverage on the basis of an estimate for penetration/ viewership. A typical set of categories for valuation of online exposure would be:
UK Sites Opportunities-to-see

High use site Medium use site Low use site

1,150,000 345,000 17,250

Is this style of categorisation the best-fit solution to an almost impossible problem, or is it pointless to even try? The ideal is to create a central repository of web log data collating all the individual page viewing as a method of allocating audience impact. There are obvious implications for privacy and confidentiality and so is unlikely to ever happen. Instead the future is to work constructively with the methods available and to remain

19

Michael Blowers

Media Evaluation Research 2006

open to the new ideas and opportunities which the technology will make available. One such process is the BBCs News most popular now which demonstrates how with online data it is possible to make valued deductions on story outtakes. Macnamara (1992) pioneered the concept of media Inputs, Outputs and Outcomes through his Pyramid Model of PR Research, reproduced below.

It is accepted that media measurement is adept at providing a measure of media inputs and outputs. The model works as a process guide with the wide base representing the planning process, culminating in the achievement of the desired outcome, be-it attitudinal or behavioural. The pertinent aspect is that the information resulting from the different stages of associated research increase in value as one ascends the pyramid.

20

Michael Blowers

Media Evaluation Research 2006

The BBCs News most popular now does one thing you can not do with the traditional media. It indicates which stories on the site are being most often read and emailed, for individual topic areas and regional zones. This extract indicates a sample of what can be viewed:

This is a classic example of online information leap-frogging conventional media content analysis, going straight to media out-take (and one valuable step nearer to outcome). There are obvious shortcoming the system lacks any metrics and is only a rank of relative popularity. There is however some currency to it and it can be imagined that in future PR people will be saying their story was the second most read on the BBC website during a 24 hour period. In regards to the measurement of a wider sample of the web, there would be a considerable benefit if it were possible to include the access data for a wider segment of the web beyond the BBCs site. In effect if it were possible to create a solution similar to the BBCs News most popular now but covering a wider sample of sites and with the addition of the ability to sort the results by specific country, time-frame and using a measurable metric,

21

Michael Blowers

Media Evaluation Research 2006

then this would represent a powerful and authoritative measure of online users out-take. Other options exist, including Google trends (http://www.google.com/trends) which categorises the number of times various search terms have been used over set periods, locations and languages. It graphically presents the results and assigns links to specific stories to illustrate spikes in the data. While it has not been central to this study it could become relevant to widen the remit by considering the impact of advertising. Claire Spencer, the CIPRs representative on evaluation was quoted in the Financial Times (20 June 2006) as saying Consumers can't divorce PR and advertising, they consume communications holistically". This is an interesting concept and although moving outside of the remit of this study, does raise issues on organisations structure, with advertising and PR often being separate. Specifically it also raises a series of questions. If the public see communications as a single message then why dont organisations bring its forms of communication together? Or is that what marketing communications aims to achieve? Organisations benefit from understanding the message, from what ever source; but is the public not able to differential when they are being pitched to? That is not to say that PR and advertising messages can not be complimentary and co-ordinated to maximise impact.

Personal reflection
My own view is that the fragmentation in the media is now at the point where the skills to capture the relevant coverage need to be as sophisticated as the resulting content analysis. When dealing with the traditional media, there was a feeling that you were dealing with a finite quantity. With the pervasive nature of online coverage and in particular blogging, anyone now can publish electronically a newspaper, or run their own radio or TV station. A new skill will be understanding which of these are significant to stakeholders. This need has been partly answered by a new breed of online analysts who specifically aim to understand and represent the web-players in a specific community or interest group. With the explosion in content so there has to be an increasing risk of analysts not collecting the right sample and getting the results wrong. This has to be seen as a threat to the industry which could become very real if one of the large online software organisations decides to further develop one of the existing semi-intelligent text analysis programmes. This could

22

Michael Blowers

Media Evaluation Research 2006

easily result in a widely adopted, lower level (and low cost if not free) online analysis package. All of sudden a PR results system if available which is able to track key messages and favourability, and get it 80% right. Google, a firm well adapt at finding coverage, has already demonstrated through its Trends programme that it is thinking about this area.

Conclusions
The task of this report has been to present a critique of the current provision and academic background supporting media content analysis. The various processes which support quantitative and qualitative media measurement when effectively combined with rigorous methodology can create an accurate record of media outputs. Primary research has pointed towards the growing importance of key message and favourability analysis. There is also a growing need to present an effective link between media output and organisational outcome. The discussion also included some illustrations of the methods which can be adopted to measure media out-take (one-step nearer to outcome). A number of research studies have indicated that PR will need to adjust to the growing importance of online media. This medium carries a whole selection of research challenges and opportunities. It will be particularly difficult to use comparable metrics to present media impact from traditional and new media largely because of the lack of effective audience data for the web. What ever happens, there is the view that content analysis will always be able to be applied to understanding message usage and favourability. As has already been covered, the metrics by which these are measured could pose challenges. However if the end users can appreciate the complexity of the task then the industry will take one valuable step towards an effective solution.

23

Michael Blowers

Media Evaluation Research 2006

Appendix
Media Measurement Survey This survey explores what media measurement methods are used by public relations professionals. There are 13 questions which should not take more than 5 minutes to complete. If you would be interested in finding out about the results we will be more than happy to send you a copy of the report. Responses numbers indicated in blue 1. For classification purposes, please indicate where you work: 1 2 2 2. Independent practitioner Small agency (2-10 people) Large agency or agency (20+ people) 13 1

In-house department Medium-sized agency (11-20 people)

Do you currently or have you in the past two years used outside companies for measurement of your public relations activities? (highlight one) Yes 11 No 8 Unsure 0

3.

Do you currently or have you in the past two years conducted measurement yourself? (highlight one) Yes 12 No 7 Unsure 0

4. What types of media do you measure and how important are they to you? (Please number 1 to 4 in importance (1 - most important 4 least important) 1 4 Press Radio =2 =2 Television On-line

5. How often do you report results on media content (check more than one, if relevant) 7 3 Monthly Quarterly 8 5 After all campaign Other (please indicate when)

24

Michael Blowers

Media Evaluation Research 2006

6. Which of the following statements comes closest to describing how your department or organization currently handles public relations measurement? (PLEASE SELECT ONLY ONE RESPONSE) 3 9 4 3 1 Little or no measurement work is done. Most of the work is done internally. Most of the work is done by outside counsel and/or suppliers. Assignments tend to be evenly divided between doing them internally and using outside counsel and/or suppliers. Other (please specify) __________________________________________________________________

7.

During the next two years, do you anticipate that the volume of public relations measurement studies that your department or organization carries out annually will: 3 6 9 0 0 1 Increase by10% or more Increase by 1-9% Remain approximately the same. Decrease by 1 - 9%) Decrease by 10% or more Not sure

8.

Which of the following measurement methods are you currently using? Please indicate those that you dont use, those you do use, and those you would like to use. Dont use 4 11 13 4 6 0 6 7 11 11 4 12 6 10 8 10 12 12 12 11 6 Currently use 13 7 5 13 12 17 9 9 7 8 12 6 12 6 7 7 5 5 5 5 12 Would like to use 0 0 0 1 0 0 3 2 1 0 3 0 0 2 1 1 1 2 2

Item/Element Measured Column inches/broadcast time Advertising value equivalency without multiplier Advertising value equivalency with multiplier Reach (also called opportunity to see) Total number of placements or media impressions Media outlet category (daily/weekly paper, wire service, local vs. national TV,etc.) Source of story (media pitch, news release, press conference, unsolicited, etc.) Story form/type (hard news, feature, product review, editorial, etc.) Targeted geographic area (region, country, ADI markets, etc.) Presentation (text only, graphics, charts, photos, color, etc.) Key message content analysis Targeted media (byline, specific broadcaster, etc.) Use of branding (verbal and visual) Use by targeted journalist Client target audience or potential audience reached Client product/service mentions Competitor mentions Competitor service/product mentions Client/Company share of voice compared to competitors within story Competitor messages covered Spokespersons mentioned/quoted

25

Michael Blowers

Media Evaluation Research 2006

Tone/stance toward client or company (also called slant) Headline tone/stance/slant Content favorability/unfavorability Call to action (phone/web/mail inquiries traceable to story Cost of message placement/distribution (news release distribution fees, brochure printing, etc.) Number of media representatives attending event/press conference Target audience response (via Web response, phone inquiry, postcard return, etc.) 9.

6 7 7 14 14 13 13

11 8 10 2 4 5 5

2 2 1 2 0 0 1

If there are other items you measure or would like to measure that are not included here, please write them in the following spaces and provide a brief explanation of what they measure. Currently use Would like to use 1

Item Measured Donations to campaigns Tracking research

10. Looking into the future, which of the following aspects do you think will become more or less important? Less important 2 5 0 0 0 0 0 0 1 3 3 2 2 Stay the same 11 8 11 6 9 8 2 6 6 8 5 7 3 More important 3 3 8 11 10 9 14 11 11 4 5 4 10 Not sure 3 2 0 1 0 2 3 2 1 5 5 6 3

Measurement by audience/opportunities to see Measurement by advertising value equivalent Media monitoring Key message content analysis Favourability of media content Competitor analysis Connecting media output with organizational outcomes Presenting a return on investment Creating tangible results for PR activity Academically proven methodology Results presentation techniques Results clarifications/accessibility Accurately reflecting on-line media exposure

26

Michael Blowers

Media Evaluation Research 2006

11. If you can think of any other relevant issues to media measurement missed out in the list above please indicate them in the table below, scoring them accordingly Less important Blog monitoring Tracking product launches Stay the same More important 1 1

12. Please indicate your agreement with the statements below, where 1 is strongly disagree, 4 is neither agree nor disagree, and 7 is strongly agree.
Strongly Neither Agree nor Disagree Strongly Agree

Media measurement methods Are easy to use

Disagree

1 1 1 1 1 0 1 0 1 0 1 0

2 0 2 1 2 1 2 3 2 2 2 0

3 0 3 2 3 2 3 4 3 1 3 4

4 10 4 4 4 4 4 4 4 6 4 2

5 5 5 6 5 5 5 5 5 3 5 5

6 1 6 4 6 6 6 3 6 6 6 4

7 2 7 1 7 1 7 0 7 1 7 2

Measures what I want to measure

Produces usable data

Are understandable by clients/non-PR managers

Answers my questions about program success

Are usable in future strategic planning

27

Michael Blowers

Media Evaluation Research 2006

13. Please indicate how well you agree with the statements below, where 1 is strongly disagree, 4 is neither agree nor disagree, and 7 is strongly agree.
Strongly Disagree Neither Agree nor Disagree Strongly Agree

I use media content analysis for strategic planning purposes

1 0

2 2

3 3

4 3

5 5

6 3

7 4

My current media content analysis measures what I want to measure

1 0

2 5

3 1

4 3

5 4

6 4

7 3

My current measurement methods produces usable data

1 0

2 1

3 1

4 6

5 5

6 3

7 3

I use media measurement to justify budget requests

1 0

2 2

3 3

4 4

5 6

6 0

7 2

The movement of media to on-line sources will make it difficult to effectively represent our output

1 0

2 2

3 2

4 6

5 3

6 1

7 4

Strongly Disagree

Neither Agree nor Disagree

Strongly Agree

I use media measurement as a political defense against other departments

1 4

2 1

3 2

4 8

5 2

6 1

7 1

I use media measurement to educate executive management on the role/value of public relations

1 0

2 2

3 1

4 2

5 6

6 6

7 2

I use media measurement to educate colleagues on the roles/value of public relations

1 1

2 2

3 2

4 3

5 3

6 6

7 2

I use media measurement to educate colleagues on the attitudes/opinions of key stakeholders

1 0

2 3

3 2

4 2

5 4

6 6

7 2

Many thanks for your help and if you would like to add any comments or thoughts please continue below. Michael Blowers

28

Michael Blowers

Media Evaluation Research 2006

Bibliography
Berelson, B. (1952) Content analysis in communication research. New York: Hafner. Campbell, R. S., & Pennebaker, J. W. (2003) The secret life of pronouns: flexibility in writing style and physical health. Psychological Science14, No. 1, 60-65. Retrieved July 19, 2004, from http://jerz.setonhill.edu/weblog/permalink.jsp?id=885. Cutlip Center & Broom, (1999) Effective Public Relations (Prentice Hall International)

Fombrun C,(1996) Reputation: Realizing Value from the Corporate Image (Harvard Business School Press)
Fairchild. E.(2003) The CIPR Toolkit: planning a research and evaluation for public relations success. Chartered Institute of Public Relations, London Haywood R, (2005) Corporate reputation, the brand and the bottom line Hutton P. Strategic Communications Management (2004) Calculating ROI - a dubious business Jensen, K. B. (1995). The Social Semiotics of Mass Communication. London: Sage Kaplan and Norton, (2004) Strategy Maps (converting intangible assets into tangible outcomes) (Harvard Business School Press) Kitchen P, (2003) Public relations principles and practice Thomson Lasswell, H. D., Lerner, D., & Pool, I. de Sola.(1952). The comparative study of symbol: an introduction. Stanford University Press, Hoover Institute and Library on War, Revolution and Peace. Likely F, (1999) Performance measurement: can PR/communications contribute to the new bottom line of intangible, non-financial indicators? Macnamara J, (2002) PR metrics-research for planning and the evaluation of PR and corporate communications. Mass communications/PRIA. www.pria.com.au Macnamara J, (2003) Media content analysis-its uses and benefits and best practice methodology. Carma/PRIA. www.pria.com.au Macnamara J. (2003) Return on investment. Carma/PRIA. www.pria.com.au Mayring, P, (2003). Qualitative content analysis. Forum Qualitative Sozialforschung/Forum Social Research. Online journal, 1, (2). Retrieved December 18, 2003, from http://qualitativeresearch.net/fqs/fqs-e/2-00inhalt-e.htm. Merrill Brown, Carnegie Reporter (Spring 2005) Abandoning the news Murrey & White, (2004), Views on reputation management Neuendorf, K. A. (2002), The Content AnalysisGuidebook, (Thousand Oaks, CA: Sage Publications)

29

Michael Blowers

Media Evaluation Research 2006

Newbold, C., Boyd-Barrett, O. & Van Den Bulck, H. (2002). The media book. London: Arnold (Hodder Headline). Paine, KD (2002). Measures of Success for Cyberspace Phillips D. (2005) The evaluation needs of the PR industry Selby, K. & Cowdery, R. (1995). How to study television. (Basingstoke: Macmillan) Shoemaker, P. J. and Reese, S. D. (1996). Mediating the message: theories of influences on mass media content. (White Plains, NY: Longman) Spencer C, (2004) PR past and future onward and upward (World Advertising Research) Synder-Duch, J., Bracken, C. C, & Lombard, M. (2001). Content analysis in communication: assessment and reporting of intercoder reliability. (Presentation to the International Communication Association, Washington, DC, May) Theaker A, (2001) The PR Handbook (Routledge) Van Zoonen, L. (1994). Feminist media studies. (London: Sage) Westaby M., (Jan'06) How PR works: media relations (Admap) White J, (2005) Measurement and Evaluation: moving the debate forward (www.cipr.co.uk) Jon White and John Blamphin (1994) Priorities for Research into Public Relations Practice in the United Kingdrom, (London, City University Business School/Rapier Marketing)

http://news.bbc.co.uk/1/shared/bsp/hi/live_stats/html/map.stm www.brandweek.com/bw/news/spotlight/article_display.jsp?vnu_content_id=1002876797 www.kdpaine.blogs.com www.mediaproofonline.com www.nielsen-netratings.com www.nielsenbuzzmetrics.com www.alexa.com

Association of Media Evaluation Companies, www.amec.org.uk

30