A study to analyse the effectiveness of multimedia and text-only e-mail communications and discover whether consumers attitudes

towards e-mail reflect their behaviour

By Owen Simon Ball

Submitted in partial fulfilment of the requirements for the Degree: Bachelor of Arts in Marketing & Advertising

University of the Arts London London College of Communication: School of Creative Enterprise

March 2007

1. Acknowledgements
This Dissertation would not have come to fruition without the help, wittingly or unwittingly, of a great many people too numerous to mention. This section provides a platform to recognise and acknowledge those fortunate enough to spring to mind. My dissertation tutor Jon Kitto should be commended for the guidance, support, and wealth of ideas he provided in order to see this work completed. I would like to express my gratitude to the BA Marketing & Advertising staff, specifically Colin Watson for his introductory coffee on my first visit to the institution and his subsequent help with the questionnaire designs for this study. A huge thank-you Kenji Lim from Jail Clothing, without whom this study would not have been devised or executed. The staff at Apple, TextMate, Mellel & Campaign Monitor for developing exceptional software that made my life a little easier, and saved me an overwhelming amount of time and effort. Luke Branley, Aimee Gillett and Reynaldo Canlas Jr. for kindly taking the time to assist me with my research and distract me in equal measure. All the unnamed people who kindly responded to the Questionnaire, I am indebted to you. Martin Hall in the Reprographics department at EHDC for providing the facilities to print and bind this study. Finally, my family for providing help, advice, and motivation when it was required.

-1-

2. Abstract
Technological developments have given rise to a global network of computers and servers allowing its users to communicate with each other without geographic or time constraints. The most popular, and established, medium of communication in this global network is e-mail and it is this of communication that will be scrutinised in this study. In its early days e-mail was constrained to plain text messages. Advances in software and internet protocols have enabled the medium to carry a far richer subset of information including, but not limited to, colour information, images and varied typefaces. This new breed of multimedia e-mail, facilitated by HTML, has polarised opinion. Proponents laud the new design capabilities made possible whilst at the same time detractors lament the surfeit of rendering issues, incompatibility and broken links they argue plague the format. Placed in this context, the objectives of the research were two-fold. The first was to identify the format of e-mail that was most effective. The second was to discover whether recipients attitudes toward e-mail format reflected the behaviour observed in relation to the first objective. In order to conduct the study three methods of research were utilised. Desk research provided data for comparison and contrast, an experiment was conducted to obtain metrics for the effectiveness of each format of e-mail. Finally, a questionnaire was sent to the same sample used in the experiment to determine whether their attitudes toward e-mail correlated with the observed behaviour. The study found that e-mail open and click-through rates both provided significant evidence to show that multimedia e-mail was in fact more effective than text-only. These findings were backed up by statistical analysis that showed that the probability of the results being significant was extremely high. In addition to the findings from the experiment the questionnaire responses suggested that there is a strong correlation between the the attitudes of recipients and their behaviour. Respondents preferred multimedia over text-only by a ratio of 3:1, a ratio that when compared to click-through rates for each format was almost identical. This study ultimately produced two noteworthy conclusions. The first conclusion is that HTML e-mail is more effective than text-only e-mail, thereby rendering the research hypothesis invalid. The second conclusion, relating to the second objective, is that consumers attitudes show a strong correlation to their observed behaviour, both showing a significant preference for multimedia e-mail.

-2-

3. Table of Contents
1. Acknowledgements 2. Abstract 3. Table of Contents 4. List of Appendices 5. Abbreviations 6. Introduction 7. Literature Review 7.1 7.2 Introduction E-mail Marketing 7.2.1 7.2.2 7.3 7.3.1 7.3.2 7.3.3 7.4 ..............................................................p. 1 ..............................................................p. 2 ..............................................................p. 3 ..............................................................p. 8 ..............................................................p. 9 ..............................................................p. 10 ..............................................................p. 12 ..............................................................p. 12 ..............................................................p. 12 E-mail ..............................................................p. 12 E-mail Marketing ..............................................p. 13 ..............................................................p. 15 Evidence for Text-only .....................................p. 15 Evidence for HTML-only .................................p. 16 Evidence for both formats ................................p. 17 ..............................................................p. 18 ..............................................................p. 21 ..............................................................p. 21

E-mail by Format

Further Theory

8. Research Questions 8.1 8.2 8.3 Research Aims

Area for Further Research ........................................................p. 21 Research Hypothesis ..............................................................p. 22

-3-

9. Methodology 9.1 9.2 Overview 9.2.1 9.2.2 9.3 9.3.1

..............................................................p. 23 ..............................................................p. 23 Quantitative vs. Qualitative ..............................p. 23 Triangulation.....................................................p. 24 ..............................................................p. 25 ..............................................................p. 25 Sample ..................................................p. 25 Content ..................................................p. 26 Measurement.........................................p. 26 Experiment ........................................................p. 25 a. Process i. ii. iii.

Research Considerations ...........................................................p. 23

Research Methods

b. Limitations ..............................................................p. 26 9.3.2 Survey ..............................................................p. 27 a. Process i. ii. iii. ..............................................................p. 27 Sample ..................................................p. 28 Content ..................................................p. 28 Measurement.........................................p. 28

b. Limitations ..............................................................p. 29 10. Results & Findings 10.1 10.2 10.3 JailMail Multimedia 10.1.1 10.2.1 10.3.1 10.3.2 10.4 10.4.1 10.4.2 10.4.3 10.4.4 10.4.5 JailMail Text-only ..............................................................p. 30 ..............................................................p. 30 ..............................................................p. 32 Metrics for Multimedia Campaign ...................p. 30 Metrics for Text-only Campaign ......................p. 32 Comparisons .....................................................p. 34 Chi Squared ......................................................p. 35 ..............................................................p. 39 Survey Statistics................................................p. 39 Gender Analysis................................................p. 40 Age Analysis .....................................................p. 42 E-mail software analysis ...................................p. 43 Browser based e-mail analysis..........................p. 44
-4-

Experiment Analysis ..............................................................p. 34

JailMail Survey

10.4.6 10.4.7 10.4.8 10.4.9 10.4.10 10.4.11 10.4.12 10.4.13 10.5

Frequency respondents check e-mail analysis ..p. 45 E-mail format analysis ......................................p. 46 Reasons for choosing multimedia e-mail analysis ..............................................................p. 49 Reasons for choosing text-only e-mail analysis ..............................................................p. 50 Respondent's preferred hyper-link format analysis ..............................................................p. 51 Preferred frequency for e-mail communications analysis .............................................................p. 52 Behaviour towards unwanted commercial e-mail communication analysis....................................p. 53 The reasons respondents gave for not choosing to unsubscribe analysis .........................................p. 55 ..............................................................p. 56 Comparisons .....................................................p. 56 a. b. Gender Vs. E-Mail Format ...............................p. 56 Age Vs. E-Mail Format ....................................p. 58 ..............................................................p. 61 ..............................................................p. 61 Experiment ........................................................p. 61 a. b. Open Rate .........................................................p. 61 Click-through Rate............................................p. 62 Questionnaire ....................................................p. 63 ..............................................................p. 64 ..............................................................p. 65 ..............................................................p. 66 ..............................................................p. 73

Survey Analysis 10.5.1

11. Conclusions 11.1 Key Conclusions 11.1.1

11.1.2 11.2 11.3 11.4 Limitations

Qualifying Statement ..............................................................p. 63 Further Research

12. Appendices 13. Bibliography

-5-

List of Tables Table 1: Comparing unique click-throughs by format .....................................p. 18 Table 2: Effectiveness of Image & Text dominant web banner ads .................p. 19 Table 3: A Taxonomy of Research Methodologies ..........................................p. 24 Table 4: Unique E-mail Opens Chi Squared Figures .......................................p. 35 Table 5: Unique E-Mail Clicks Chi Squared Figures .......................................p. 37 Table 6: Total E-mail Opens Chi Squared Figures ...........................................p. 37 Table 7: Survey - Gender Chi Squared .............................................................p. 40 Table 8: Survey - Multimedia vs Text Chi Squared .........................................p. 46 Table 9: Survey - Revised Multimedia vs Text Chi Squared ...........................p. 48 Table 10: Gender vs Preferred E-Mail Format .................................................p. 56 Table 11: Age vs Preferred E-Mail Format ......................................................p. 58 Table 12: Further Research Possibilities ..........................................................p. 65 List of Figures Figure 1: Opens, Bounces & Unopened numbers for multimedia campaign ...p. 30 Figure 2: Total links clicked for multimedia campaign....................................p. 31 Figure 3: Opens, Bounces & Unopened numbers for text campaign ...............p. 32 Figure 4: Total links clicked for text campaign ................................................p. 33 Figure 5: Survey Statistics Figure 6: Q1: Respondent Gender Figure 7: Q2: Respondent Age ..............................................................p. 39 ..............................................................p. 40 ..............................................................p. 42

Figure 8: Q3: Respondent's e-mail software .....................................................p. 43 Figure 9: Q3a: Respondent's browser based e-mail choice ..............................p. 44 Figure 10: Q4: Frequency respondents check e-mail .......................................p. 45 Figure 11: Q5: Respondents preference of e-mail format ................................p. 46 Figure 12: Click-through compared to Preferred Format .................................p. 47 Figure 13: Q5a: Reasons for choosing multimedia e-mail ...............................p. 49 Figure 14: Q5b: Reasons for choosing text-only e-mail ...................................p. 50 Figure 15: Q6: Respondent's preferred hyper-link format ................................p. 51

-6-

Figure 16: Q7: Respondent's preferred frequency for e-mail communications ..............................................................p. 52 Figure 17: Q8: Behaviour towards unwanted commercial e-mail communication ..............................................................p. 53 Figure 18: Open & Unsubscribe Percentages comparison ...............................p. 54 Figure 19: Q8a: The reasons respondents gave for not choosing to unsubscribe ..............................................................p. 55 Figure 20: Gender vs Preferred E-Mail Format ................................................p. 57 Figure 21: Age vs Preferred E-Mail Format % by Age ....................................p. 59 Figure 22: Age vs Preferred E-Mail Format Frequency by Age ......................p. 59

-7-

4. List of Appendices
Appendix 1: JailMail Multimedia Format .......................................................p. 66 Appendix 2: JailMail Text-only Format ..........................................................p. 67 Appendix 3: JailMail Questionnaire ..............................................................p. 68 Appendix 4: Chi Squared Test Table ..............................................................p. 72

-8-

5. Abbreviations & Definitions
Click-through rate E-mail HTML IPT ISP Link(s) MIME Open-rate Spam The number of hyperlinks accessed from a campaign Electronic Mail Hyper Text Mark-up Language Interactive Prospect Targeting (company) Internet Service Provider Hyperlink (Method of navigating to content) Multipurpose Internet Mail Extensions The number of e-mails opened by recipients Unsolicited Commercial E-mail

-9-

6. Introduction
The Global Reach research agency (2004) estimated that by the year 2005 there would be over 1 billion users of the internet globally. Whilst it impossible to provide truly accurate numbers this still represents a massive potential audience for marketing messages, and because the internet transcends the limitations of time and space this audience can be reached almost instantaneously. E-mail is one of the most established methods of communication used online, according to IPT (2004), 99% of consumers use the internet for email which must place the medium as a one of few communication methods with such a large reach.

If we accept then that e-mail has vast potential in terms of numbers, the next question to be asked is what are the implications for marketers. According to the same IPT study, e-mail marketing ranked second as the medium consumers felt was most effective. It is important then, considering the numbers involved and consumers receptiveness to the format, that marketers assess the best way with which to communicate their messages. There are currently two clear options for e-mail messages. Traditional text formatted e-mail, or multimedia email facilitated by the use of HTML code. In some cases, it is very easy for a marketer to make the choice to use one or the other, however, how does a marketer know whether to spend the time and money required to develop an HTML based email campaign? This research, in an attempt to provide empirical evidence on this issue, uses an experiment, and questionnaire to discover what format of e-mail is most effective, and whether consumers attitudes toward e-mail marketing reflect their behaviour in response to it. The aim of the research then is to provide the marketer with quantifiable evidence that can be used to argue in favour of either approach.

The dissertation is broken down into five broad section, beginning with a review of current literature in the subject area. This is followed by the Research Questions and Research Methodology chapters that define the area of research and how the research will be
- 10 -

conducted. The following section illustrates the findings from the experiment and questionnaire. The dissertation is concluded by a chapter that draws together the key conclusions from the findings.

- 11 -

7. Literature Review
7. 1. Introduction
The relatively recent development of email, in particular multimedia based email facilitated by HTML, as a marketing tool ultimately restricts the pool of previous academic research in the field. This study concerns itself with the effectiveness of two broad categories of email communication, that of graphically rich multimedia email and that of text-only email. It is perhaps to be expected that the majority of information available regarding email and email marketing is based on the world wide web rather than in books and journals, and it is true to say that there is a large amount of subjective and anecdotal evidence that can be found supporting the hypothesis that either type of email is more effective than the other. This literature review will provide an objective view of the material available using, where appropriate web-based and print-based material. The literature review will be segmented into four logical sections. The first section will focus on email marketing, citing work from figures and companies well-regarded in the industry alongside more anecdotal evidence. The second section will be more empirical in nature addressing previous studies and research supporting the case for either format. The third section will discuss relevant theory relating primarily to the composition of communications. The final section will draw conclusions from the research presented.

7. 2.

E-mail Marketing
7. 2. 1. E-mail

The aim of this section is to define email marketing and give the reader an understanding of the importance of email and why it is an important element of a marketing mix. According to the Oxford American Dictionary e-mail stands for "messages distributed by electronic means
- 12 -

from one computer user to one or more recipients via a network." The internet, in its current form, was not a prerequisite for e-mail. Indeed, the first crude form of e-mail was sent by Ray Tomlinson of ARPANET in 1971 (Griffiths, R.T., 2002), however, in order for the subject area of this research, email marketing, to be utilised by marketers, the internet along with more advanced software and networking technology was required. With this backbone in place Hoffman (2000) argues that e-mail has become "the most important innovation since the development of the printing press" able to "transform not just the way individuals go about conducting their business with each other, but also the very essence of what it means to be a human being in society." This is certainly an extreme viewpoint, but there is some justification to Hoffman's views. The following figures make impressive reading and certainly suggest that e-mail has become an intrinsic part of many peoples lives. According to Messaging Online (2000) "e-mail is the fastest-growing communications technology in history. From only 2m e-mail accounts in 1985, this grew to 891.1m e-mail accounts at the beginning of 2001." This number is now even higher, according to The Radicati Group (2006) there are a staggering "668 million email users worldwide, with over 1.2 billion active email accounts." When this is coupled with "worldwide email traffic per day total[ing] about 135 billion messages" the result is a behemoth of a communication medium.

7. 2. 2.

E-mail Marketing

Purely in terms of metrics the previous paragraph illustrates just how valuable e-mail could be to an online marketing program, and it is true to say this is being followed up by marketers. An IPT email marketing survey (2004) found that 92% percent of professional respondents currently used email marketing with over half (51%) declaring that the budget would increase the following year. It is perhaps not surprising that these figures are so positive given the findings from the consumer part of the survey. According to IPT, 99% of consumers use the internet for email creating a vast potential audience for marketing messages. Even more encouraging for marketers is the medium consumers felt was the most effective for marketing communication. Email ranked second at 32%, a score only 7% behind television, but markedly higher than press, radio and other forms. Clearly these statistics have to be approached with caution, the answers consumers give, compared to their actual
- 13 -

behaviour can often be distinctly different. Still, the figures do show consumers are still receptive to email communication despite that fact that "the average email user gets between 54 and 93 e-mails per week, a number that increases in direct proportion to the length of time the person has been using email" (Brondmo, H. P., 2001).

Utilised as a marketing tool email has both advantages and disadvantages. The key advantages of email marketing are two-fold and intrinsically linked. The first is low costs and the second is the ability to utilise a digital work-flow. (Jackson & DeCormier, 1999; Peppers & Rodgers, 2000). Actual figures vary by campaign but Dave Chaffey (2006) estimates that "currently, e-mail costs range from $5 to $7 per thousand compared to $500 to $700 per thousand for direct mail." If e-mail costs just 1% per thousand to reach customers compared to direct mail the casual observer may question whether, in the words of the age-old idiom, "you get what you pay for." According to Hans Peter Brondmo (2001) "the best email marketing programs, in fact, routinely enjoy response rates above 30 percent, while even fairly simple promotional marketing campaigns routinely see a 4 to 6 percent response (compared to 1 to 2 percent response rate from traditional direct mail campaigns)." The figures are a positive re-enforcement for email, but Brondmo continues with a note of caution. He says that "with all the information consumers are exposed to on a daily basis, it's getting harder and harder to separate the good from the bad." This a viewpoint expanded by Windham (2000) who suggests that email marketing has one major disadvantage, the scourge that is spam, or unsolicited email. Not only does this taint the reputation of email as a form of communication, but it directly affects response rates to campaigns. In addition many counter spam measures introduced by ISPs and email providers have had a significant detrimental effect on email open rates and created rendering issues (DoubleClick, 2005).

- 14 -

7. 3.

E-mail by Format
7. 3. 1. Evidence for Text-only

The following paragraph will assess evidence that suggests text-only email messages are more effective than HTML. It is to perhaps to the credit of what Jeanne Jennings (2004) describes as "the anti-HTML group", a "small but vocal minority" that there is any debate on the issue of what format of email is most effective. Evidence in favour of text-only is underwhelming and, in addition, predominately based on figures from the first couple of years of the twentieth century. A report on consumers' preferred e-mail advertisement formats by eMarketer (2002) discovered that a staggering 62% of respondents preferred text email messages , with only 35% favouring HTML. These results supported findings from another eMarketer report carried out in 2001 assessing response rates to e-mail marketing campaigns. It found that text email enjoyed an 18.5% click-through rate compared to 15.6% for the HTML equivalent. Research undertaken in the years proceeding the data above tend to show a slight preference for HTML as we shall see in the following section, however there is some small consolation for text email. Jupiter Research (2005) attempted to identify what impact the content of an email communication has upon response behaviour. Out of a sample size of 1166 consumers, written copy with 40% of responses was identified as the second most important factor for respondents. This compares with 12% and 9% for one large image and multiple smaller images respectively. This result does have to be put in some context. Because there is no distinction between HTML and text email, it is entirely possible that readers preferred HTML email, but still felt the body copy was the most important factor for initiating response. Finally, A DoubleClick (2005) email trend report for Q1 2005 notes that "Despite performing the least well, text showed the greatest year-on-year increase in clickthrough rates up 19.6%, from 5.6% to 6.7%", and "the UK, in particular, saw its text clickthrough rates increase by 136.7%." The report suggests this may be influenced by a trend towards email access on mobile devices. This is a view shared by Dave Chaffey (2006) who notes that there are many email readers that cannot easily display HTML, such as Blackberry and PDA devices.
- 15 -

7. 3. 2.

Evidence for HTML-only

The previous section alluded to a trend where earlier research favoured text email and later HTML. The metrics suggest that this is indeed the case, however, an eMarketer report (2001) focused on consumer attitudes towards e-mail formats rather than email delivery statistics. The results are of particular interest because they show a huge disparity with much of the research at that time. The key finding was that 60% of respondents would rather receive HTML over text email. This result supports more recent research suggesting that the case for HTML email being the bastion of email marketing is stronger than that for text. This viewpoint is supported by anecdotal alongside empirical data both of which will be detailed in this section. To begin, a reader comments section on a text vs. HTML article by Jeanne Jennings (2004 ) for Clickz resulted in "less than 3 percent of responses [being] anti-html." This does not of course mean that 97 percent of respondents were pro HTML email, but it does suggest that there is not a large swath of antipathy towards HTML. In addition comments e-mailed to the author by marketers strongly suggest HTML is now the de-facto standard for email. Paul Maloy (2004) says "the newsletter list that I maintain favors HTML by about 12 to 1", a sentiment shared by Paula Skaper (2004) who adds that "more than 90 percent of readers choose the HTML version over text." And "until that metric changes significantly, there's simply no solid business case for dumping HTML."

Data from DoubleClick (2005) echoes the views given in the preceding paragraph. Discussing email format trends the report writes "HTML was, by far, the most popular email format in Q1 2005 with 64.3% of e-mails being sent in this format. This was followed by Multi-part ... at 20.9% and Text at 14.8%." The report continues "in Q1 2005, higher clickthrough rates were achieved by formatted content with both HTML (11.5%) and Multi-Part (11.0%) performing above the overall average of 10.0% for all types." However "Text continued to perform least well at 6.7%". These results show that e-marketers clearly feel HTML is the most effective method of communication, with the metrics backing up their argument. Indeed the Q2 report from DoubleClick (2005) provides similar statistics. One of the key findings in the report concluded that "although open rates declined from 36.0% in Q2 2004 to 27.5% in Q2 2005, click rates remained relatively stable, dipping slightly from 7.7%
- 16 -

to 7.2%". The report continues to say that "this relative stability in click rates, despite the decline in opens, mirrors the HTML click-to-open ratio (which increased from 26.5% to 32.0% from Q2 2004 to Q2 2005)." A higher click-to-open ratio suggests that the content of an email is effective in stimulating click-through once an email has been opened with HTML the format that can facilitate this.

7. 3. 3.

Evidence for both formats

In a survey of 50000 readers split evenly between HTML "lite", HTML and text based email, it was found that almost no difference in effectiveness could be measured. The completion rates were 7.2%, 7% and 7.5% respectively casting doubt on the supposition that either text or HTML are more effective than the other (Marketing Sherpa, 2003). The article continues to suggest that if a strong enough relationship between the business and consumer has been forged, email format becomes a non-issue. In support of these findings, a survey of 600 people carried out by Lynda Partner (2003) for MarketingProfs discovered that just over half preferred HTML email and just under half plain text.

Perhaps the most pertinent article is that of Tom O'Leary (2006) for GroupMail. O'Leary proposes that due to technological advances in email composition software and readers the best format to use is both. Using something called a multi-MIME message, it is possible to send both HTML and text in the same communication. If a reader cannot read the HTML correctly it can degrade gracefully down to the text version included, or alternatively the reader can be programmed to show only the text-based version when available (Chaffey, 2006). To support the point O'Leary provides figures from a GroupMetrics report on unique click-through from a list size of 10000.

- 17 -

Table 1: Comparing unique click-throughs by format HTML Only Text Only HTML and Text Source: GroupMetrics 1436 Unique clicks 1353 Unique clicks 1518 Unique clicks

Text is the worst performer, though not significantly below the performance of HTML. It is clear however that the combination approach yields the best results. These results also suggest that HTML would perhaps perform better without some of the technical and rendering difficulties that can be associated with it, and which cause the reader to default to text.

7. 4.

Further Theory

A study conducted by Diameter for DoubleClick Media (2001) aimed to analyse the effectiveness of each element of a web banner advertisement on ad recall. The study concluded that in some situations visual images are second to brand logo in eliciting recall. Whilst it did not discuss image versus text formatted advertisements, the findings suggest that companies sending HTML formatted email should pay close attention to the branding of the email above other image content. Two more studies, although assessing web banners as opposed to email, can be applied to email formatting. The first of the studies carried out by Electronic Telegraph/Ogilvy & Mather (1997) measured the effectiveness of each of the creative elements incorporated into banner advertising. The findings of the study suggested that image dominant adverts had a stronger brand recall than that of text dominant advertising. However, closer scrutiny reveals only a slight difference. Image and text dominated adverts had recall percentages of 60% and 56% respectively.

- 18 -

Utilising a similar comparison between image and text dominant web banner adverts, but focusing on click-through-rate, BannerTips discovered that text dominant banners performed better than their image dominant counterparts. In the context of email marketing these findings suggest that the goal of each email communication should be reflected in the formatting used, with the appropriate formatting utilised depending on whether the email was designed to be a "call to action" or informative. The following table summarises these findings.

Table 2: Effectiveness of Image & Text dominant web banner ads Effectiveness Attribute Image Dominant Text Dominant Source: Park, M.H., 2002 Brand-Reinforcement Increase Decrease Click-through Decrease Increase

It should be re-iterated that the findings in the studies preceding this paragraph provide data based on the observation of banner adverts, not email. It is not then surprising that research on the factors affecting click-through-rate in e-mail marketing (Chittenden, L. & Rettie, R., 2002) produced contradictory results. Comparing click-through rate to creative information they noted that "the number of images is less in the lower responding group, suggesting that the more colourful and attractive e-mails generate greater response." If these results are considered representative they provide a strong argument that not only are consumers receptive to HTML email, but using HTML to provide colour and images actually improves performance.

Eyetrack III (2004) undertook research studying the recall of information presented in text vs. multimedia format. Although this study did not measure how effective each format was at eliciting a physical response (click-through), it is important to remember not all email marketing efforts are for that purpose. According to the IPT Email Marketing Survey (2004),
- 19 -

75% of marketers use email for customer retention, 15% for acquisition, 9% for cross selling and 1% for branding. Retention may include information or news designed to keep the brand in the customers mind rather than be a specific call to action. That said, the research shares some similarities with email marketing and therefore it is possible to apply the results. The aim of the research was to assess whether "the presentation of editorial content in multimedia format [would] help readers understand and remember more [news] story information." Half the participants viewed a news story in text only, and half in multimedia after which they all completed the same recall quiz. Without delving too far into the data the study uncovered three findings of relevance.

• Users that received information in text form seemed to have a better recall of specific factual information, names, places etc. • Information about an unfamiliar process or procedure was more correctly recalled after viewing the multimedia. • There was no significant difference between men and women regardless of presentation format.

Source: Eyetrack III, 2004

In the context of e-mail marketing the first finding suggests that important information should be presented to customers in text regardless of the format used. The second suggests that new or unfamiliar products can be better represented through multimedia or a combination of both. The final finding suggests that there will be no discernible difference in the effectiveness of HTML or text email with regard to gender. This is a stance supported by a Pew Internet report (2005) looking at how men and women use the internet. The report concludes that although there are differences, men and women are more similar than different in their online lives. Further research would be needed to identify whether this trend follow through to email usage and effectiveness.

- 20 -

8. Research Questions
8. 1. Area for Further Research
The primary conclusion from the desk research was that although current research points to HTML being the more effective format there is sufficient evidence to suggest that this is not conclusive (see chapters 7.3.1 and 7.3.3). Indeed, if consider that the majority of evidence in favour for HTML is based on metrics, and these metrics are not a "100% accurate measure" (Patterson, 2007), then it would be ill-advised to infer that HTML is more effective. The GroupMail report (2006) outlined in chapter 7.3.3, whilst concluding that sending e-mail in both formats is preferable, also provide, numbers (Table 1) see that suggest there is virtually no distinction in click-through rate between the formats. The lack of a unanimous agreement on the topic provides scope for further research to be carried out. This study will attempt to ascertain whether there is indeed a difference in effectiveness between HTML and text email. In addition, rather than focus entirely on metrics this study will explore consumer attitudes toward the formatting of e-mail marketing communications and discover whether these attitudes match the metric data.

8. 2.

Research Aims

To determine if there is quantifiable difference in effectiveness obtained by utilising either multimedia (HTML) or text-only email communications.

To identify if a sample groups attitudes toward multimedia (HTML) and text e-mail matched their actions when presented with either type.

- 21 -

8. 3.

Research Hypothesis

"There is no difference in effectiveness between HTML and text-only email"

- 22 -

9. Methodology
9. 1. Overview
This dissertation tests the hypothesis "There is no difference in effectiveness between multimedia and text-only email." Two complimentary types of primary research were used, an experiment and a questionnaire. The goal of the experiment was to discover what format of e-mail, HTML or text, was the most effective. A questionnaire was then used to assess whether the measured results reflected their thought process or whether the two were in opposition. For this purpose it was important to send the experimental e-mail campaign first to prevent any bias corrupting the results after a questionnaire had been carried out. It is important to set down in words that although participants were unaware that their actions were being measured they remained completely anonymous in the experiment. Those participants who took part in the questionnaire also remained anonymous. Upon the conclusion of the primary research, both sets of data could be cross-analysed and patterns, trends and other conclusions could be drawn from the results. This chapter then provides the reader with an overview of the research utilised, and a justification for the choices taken for this study. This chapter will be broken down into sections, each covering one part of the process.

9. 2.

Research Considerations
9. 2. 1. Quantitative vs. Qualitative

A positivist or scientific approach to research asserts that reality is stable and may be observed objectively (Levin, 1988), this approach is in contrast to an interpretivist outlook which proposes that "only through the subjective interpretation of and intervention in reality can that reality be fully understood" (Davison, 1998). In an attempt to nullify the "paradigm
- 23 -

wars" between positivists and interpretivists, Benbasat (1984) and Pervan (1994) observe that the methodology chosen for research should be suited to the problem being investigated and the objectives of the research. The specific nature of the research and the ability to prove or disprove the hypothesis with purely quantitative data influenced the decision to follow a positivist rather than interpretivist approach. The two methods of primary research used to collect data for the study have been identified by Galliers (1991) to be positivist (see table 3), and these along with justification for there use are detailed in the following sections.

Table 3: A Taxonomy of Research Methodologies

Positivist Lab Experiment Field Experiment Surveys Case Studies Theorem Proof Forecasting Simulation X X

Interpretivist Argumentative Reviews Action Research Case Studies Descriptive Futures Research Role/Game playing

9. 2. 2.

Triangulation

Triangulation identifies the process of using multiple forms research in order to get two or more viewpoints on topic being studied, utilising this technique prevents anomalous results from one set of data biasing the overall findings. This study has used three sources of data for analysis. Desk research conducted prior to the primary research unearthed an abundance of metrics directly related to the subject topic. This research was then supplemented by quantifiable primary research in the form of an experiment and a survey.

- 24 -

9. 3.

Research Methods
9. 3. 1. Experiment

The objective of the experiment used for the study was to discover whether there is a measurable difference in effectiveness between multimedia HTML email and text-only email. In order for this to be achieved, an email communication formatted in both HTML and text was proposed. The sample was split in half with each half receiving one format of email. The effectiveness of each format of email would be assessed on two criteria. The first is open rate, the number of unique opens from the sample. The second criteria is click-through, how many people, after opening the email, ultimately went on to click one of the enclosed links.

a.

Process

In order for the experiment to be fully realised a set of variables first had to be defined. This section will deal with each of these variables in turn which, in turn, will describe the process undertaken to complete the experiment.

i.

Sample

To carry out a live experiment using e-mail is problematic because US and EU anti-spam legislation requires that all recipients must have opted-in to receive the communication. This issue was circumvented by conducting the research on behalf of a company called Jail Clothing. Jail Clothing has a house list of opted-in subscribers totalling 166 at the time of the experiment.

- 25 -

ii.

Content

The next issue was providing the content for both formats of email. It is difficult to obtain measurements of effectiveness if there is no incentive for the viewer to open or respond to the email. The partnership with Jail Clothing negated this problem. Jail send out regular email newsletters, and it is one of these that was modified for the experiment. Recipients would already recognise the sender ID and so the email was far less likely to be Blocked by an ISP or filtered into junk or spam email folders by recipients. The two e-mails (see appendix 1 and 2) were crafted to have identical content as far as possible in order for the formatting of each to be the only variable, and there-fore any difference in effectiveness between the two could be measured. Both e-mails were hand-coded in TextMate and based on a template provided by Campaign Monitor. Finally, because the text e-mail had to be sent in HTML format (see Limitations below) a conscious decision was made to format hyperlinks in HTML so that they were embedded in words. This way the links looked the same to the recipient regardless of which format was delivered.

iii.

Measurement

In order to measure the open and click through numbers for both e-mail campaigns Campaign Monitor, an on-line e-mail service provider was utilised. The browser based software, already in use by Jail for its e-mail communications, allows the user to measure, amongst other occurrences, the open rate and click-through metrics that will used as a measurement of effectiveness in this study. In addition Campaign Monitor generated and exported reports directly saving the user the need to tackle this manually.

b.

Limitations

There were three obvious limitation in the design of the experiment, one inherent to the research methodology and the other two specific to this study. The inherent limitation of an

- 26 -

experiment is that it does not find out why the results have happened, in other words, it is difficult to extrapolate meaning from quantified data. This limitation was bridged through the use of triangulation (discussed above). The second limitation was a purely down to the technology available. The text-only version of the e-mail had to be sent in HTML format. It is impossible to measure open rates for text e-mail because in order for the measurement to take place a tiny invisible image has to be requested from the host server, and this feature is available only in HTML (Patterson, 2007). The final limitation is a limitation of format. It was impossible for both versions of e-mail to be truly identical in everything except format due to constraints with text-only formatting. Compromises had to be made, predominately due to the lack of support for columns. The links down the left hand side of the multimedia email (appendix 1) had to be placed after the content of the e-mail for the text version (appendix 2).

9. 3. 2.

Survey

The objective of the survey questionnaire was to evaluate two things. The first, whether or not the recipients views on e-mail format reflected the actions they had taken when receiving the test campaign. And second, to identify if there were any other significant patterns or trends that could be extrapolated from the answers given. These could have included variables such as gender and e-mail provider.

a.

Process

The following section will outline the processes undertaken to enable the questionnaire to be carried out. In similar circumstances to the experiment there were a number of variables that needed to addressed and a number of compromises that needed to be made.

- 27 -

i.

Sample

In order for the sample used for the questionnaire to provide meaningful results it was imperative that the recipients were the same ones who had received the email campaign. With the permission of Jail Clothing the questionnaire was sent out to the subscriber list for Jail’s newsletter. Due to the “live” nature of the study and to stay compliant to US and EU antispam regulation it was necessary to send the survey only to those people who had opted in to receive information from Jail. The questionnaire was sent by e-mail invitation to 166 recipients.

ii.

Content

The final content of the questionnaire (appendix 3) was written to satisfy the needs of the study but, due to the link to Jail Clothing, the language used and nature of certain questions were written to appeal to Jail’s current subscribers. The content of the questionnaire covered basic demographic information, namely gender and age, alongside email usage and reasoning behind email usage questions. The questionnaire was composed and piloted using online software from Question Pro. The digital nature of the survey allowed changes to be implemented instantly in response to feedback from test recipients improving the quality and accuracy of the results obtained. The live questionnaire was subsequently delivered using the software's built in e-mail list manager using a recipient list imported from Campaign Monitor.

iii.

Measurement

Conducting the survey online enabled the questionnaire to be delivered to recipients directly, and responses recorded and collated in real-time. In addition, analogous to the Campaign Monitor tools, the software provided real-time customised reports negating the need to complete this procedure manually.

- 28 -

b.

Limitations

There were two limitations observed with the design and execution of the survey questionnaire. The first was a relatively minor issue concerning the wording in certain questions. Bartholomew (1963) suggested that uncommon words and jargon can be included in questionnaires provided that the recipients are familiar with them. Piloting suggested that the majority of terms were acceptable, however, "web-based e-mail" was deemed ambiguous and this was changed in the live questionnaire to "browser-based e-mail" with examples provided. In addition it was considered that "HTML"was too technical in nature and may not be understood. "HTML" was subsequently substituted with "multimedia" to signify the use of colour, formatting and images within an e-mail. This definition was provided to participants along side the question.

The major limitation of the survey was the sample used and the consequences arising directly from this decision. In order to discover a set of significant results it was necessary to use the same sample group for the experiment and survey. Due to the remote nature of the sample the only viable method of reaching them was to use e-mail. This in turn meant that the questionnaire became vulnerable to the same problems that faced the e-mail campaign. Indeed, the same metrics used to analyse the effectiveness of the e-mail experiment (openrate and click-through rate) would be significant factors in the response rate for the survey. Walonick (1993) states that "Low response is the curse of statistical analysis" and it can "dramatically lower confidence in the results." In order to aid response rates for the questionnaire two techniques were used. The first was to place an article in the e-mail campaign outlining Jails intention to conduct a survey. Participants who read the e-mail would there-fore be expecting the questionnaire. The second technique was to provide the recipient with an incentive to complete the survey. An agreement was made with Jail Clothing for prizes to be provided for this purpose.

- 29 -

10.
10. 1.

Results & Findings
JailMail Multimedia
Metrics for Multimedia Campaign

10. 1. 1.

Figure 1: Opens, Bounces & Unopened numbers for multimedia campaign

Sent to Delivered Total Opens Unique Opened Clicks Unsubscribed Bounced Total Unopened

83 recipients on 15 Feb 2007 at 5:00 PM 79 (95.18%) 47 36 (43.37%) 15 (41.67%) recipients clicked 25 unique links 0 (0%) 4 (4.82%) 47 (56.63%)

- 30 -

Figure 2: Total links clicked for multimedia campaign

Figure 2 show the total, not unique, clicks on hyperlinks from the multimedia campaign. 8 different links were accessed a total of 28 times.

- 31 -

10. 2.

JailMail Text-only
Metrics for Text-only Campaign

10. 2. 1.

Figure 3: Opens, Bounces & Unopened numbers for text campaign

Sent to Delivered Total Opens Unique Opened Clicks Unsubscribed Bounced Total Unopened

83 recipients on 15 Feb 2007 at 5:00 PM 75 (90.36%) 32 27 (32.53%) 5 (18.52%) recipients clicked 5 unique links 1 (1.2%) 8 (9.64%) 56 (67.47%)

- 32 -

Figure 4: Total links clicked for text campaign

Figure 4 show the total, not unique, clicks on hyperlinks from the text-only campaign. 3 different links were accessed a total of 5 times. Significantly less than the multimedia campaign.

- 33 -

10. 3.

Experiment Analysis
Comparisons

10. 3. 1.

There is no significant difference in the delivery rate between the two campaigns. This suggests that some technological issues such as ISP image blocking (DoubleClick, 2005) have not directly affected the results. The same conclusion can be drawn for the "unique opens" numbers which show that although there is a declining trend for open rates (DoubleClick, 2005), this has not influenced the results. The two most significant comparisons are the "unique opens" and the "click-throughs". Both criteria show the most disparate results between the two campaigns. Using just the data from the table these two results strongly suggest that the null-hypothesis should be abandoned in favour of declaring HTML more effective. However, further statistical analysis is required (see 10.3.2)

- 34 -

10. 3. 2.

Chi Squared

The Chi Squared analysis is used "to test whether the number of individuals in different categories fit a null hypothesis" (Deacon, J., n.d). The result, after being compared to a table of values for Chi Squared, indicates the probability that the observations recorded in an experiment are significant. If there is a statistically significant difference between the observed numbers and the estimated numbers, this test provides the researcher with supporting evidence to reject the null hypothesis. Conversely, a Chi squared value that is less than the tabulated value for Chi squared gives the researcher no reason to reject the null hypothesis. As a final note, according to Deacon (n.d) "When there are only two categories (e.g. male/female) or, more correctly, when there is only one degree of freedom, the c2 test should not, strictly, be used. There have been various attempts to correct this deficiency, but the simplest is to apply Yates correction to our data." It is enough for this study to express that this deficiency has been noted, and the Yates correction has been duly applied. Three Chi Squared analysis will be presented in this section with accompanying explanation. Each of the three tables represents one of the metrics extracted from the e-mail campaign that can be used to assess effectiveness.

Table 4: Unique E-mail Opens Chi Squared Figures

- 35 -

When the Chi Squared Value of 1.02 is compared to the analysis table (appendix 4) it can be seen that with 1 degree of freedom a value of 3.84 is required for a 95% probability that the result is significant. The recorded value is distinctly lower than 3.84 and, with a probability at less than 75% does not provide compelling evidence for the null hypothesis to be abandoned.

- 36 -

Table 5: Unique E-Mail Clicks Chi Squared Figures

In the case of Unique Clicks the Chi Squared value of 4.05 at 1 degree of freedom means that we can say with a 95% probability that this result is a significant departure from the null hypothesis. (Note: the values used represent the number of unique recipients who clicked on a minimum of one link, not the total number of links clicked.)

Table 6: Total E-mail Opens Chi Squared Figures

The final table shows a Chi squared analysis for the total number of times the e-mail was opened rather than just the unique open values. The Chi squared value of 2.48 along with 1

- 37 -

degree of freedom is lower than the value of 3.84 required for a 95% probability. However, it does fall in the 90-95% probability range suggesting it is still a significant finding.

- 38 -

10. 4.

JailMail Survey

Figure 5: Survey Statistics

10. 4. 1.

Survey Statistics analysis

The statistics show that from a sample of 166 e-mail recipients 31 (18.67%) clicked through to the questionnaire from the invitation. 21 (12.65%) respondents completed the questionnaire. This a small response rate that jeopardises the reliability of the observations recorded, however, if we consider the average click-through fell to 7.2% in 2005 (DoubleClick, 2005) then the click-through and response rate for this questionnaire is statistically much greater than the average. Only 3 recipients dropped out of the survey before completion suggesting that the majority of respondents felt comfortable with, and understood the questions.

- 39 -

Figure 6: Q1: Respondent Gender

10. 4. 2.

Gender Analysis

Table 7: Survey - Gender Chi Squared

- 40 -

The results show that there is a significantly higher proportion of male respondents to female, and that this is significant. The value of 3.05 gives a 90-95% probability of significance. There are a number of reasons this could be. The first possibility is for the JailMail list to be gender biased. Gender statistics for the whole sample group were unavailable to test this theory. The second possibility is that males were more likely to respond to the questionnaire there-by biasing the results. The final explanation is that the sample size and resultant response rate were too low for the gender sample to even out. Whatever the reason, this finding will bias findings drawn from the questionnaire.

- 41 -

Figure 7: Q2: Respondent Age

10. 4. 3.

Age Analysis

The key finding that two thirds of respondents are between the ages of 18 and 27 is significant because it is likely to bias findings drawn from the questionnaire. Jail Clothing has strong links with extreme sports, in particular snowboarding, and whilst a young age is not a prerequisite for pursuing these sports a Mintel (2005) report into extreme sports suggests that extreme sports "have much higher penetration levels amongst those under 25 ... due to the physical nature of the sports in question."

- 42 -

Figure 8: Q3: Respondent's e-mail software

10. 4. 4.

E-mail software analysis

It is clear from the results that accessing e-mail via a browser is the preferred method for the sample group. It is possible that this factor would influence e-mail effectiveness. Most e-mail clients provide superior rendering for HTML than their web-based counterparts and so this would certainly not have a negative influence upon click through. However, many e-mail clients have preview panes that automatically download tracking images thereby causing the tracking software to register this as an open (Patterson, 2007). As 86% of the total respondents (21) use browser-based e-mail, for the purpose of this research this factor can be discounted as an influence on effectiveness.

- 43 -

Figure 9: Q3a: Respondent's browser based e-mail choice

10. 4. 5.

Browser based e-mail analysis

Those users who selected browser-based e-mail in the previous question were asked to clarify their choice. Over 60% chose Hotmail which is significantly high. As a comparison, according to Emailcenter (2004) 40% of their test list email addresses were Hotmail or MSN. It is certainly possible for the choice of e-mail provider to affect an e-mail campaign, in particular a campaign utilising HTML code. A silverPOP (2005) report suggests that 40% of e-mails contain missing or blocked images and this would negatively impact a multimedia campaign. In the scope of this research there is not enough evidence to form a conclusion as to whether the choice of e-mail provider directly affects campaign metrics.

- 44 -

Figure 10: Q4: Frequency respondents check e-mail

10. 4. 6.

Frequency respondents check e-mail analysis

These results show that checking e-mail everyday has been selected by close to 100% of the sample. The result is close enough to be classified as ubiquitous and accordingly it be assumed that the presence of an e-mail will be detected within one day of its delivery. We can also confidently predict that this factor will not have had an influence on the effectiveness of the campaign.

- 45 -

Figure 11: Q5: Respondents preference of e-mail format

10. 4. 7.

E-mail format analysis

Table 8: Survey - Multimedia vs Text Chi Squared

- 46 -

The Chi Squared table above uses the numbers of respondents from the questionnaire who selected a preference for either multimedia (HTML) e-mail and text-only. The expected numbers are, in this case, derived from the null hypothesis, assuming no difference between the two formats of e-mail. The Chi squared value of 4.76 is very high signifying a 95-99% probability that this is a significant result. It is interesting that this result is so distinctive from the "total opens" numbers recorded in the experiment. If this result, on its own, is linked back to the second research aim (8.2), we can see that there is a disparity. The sample claim to prefer multimedia by a ratio of 3:1, however this is not born out in the total opens for each format. The ratio is however almost identical to the "click through" numbers observed in the experiment (Figure 12) which suggests a strong positive correlation between the two.

Figure 12: Click-through compared to Preferred Format

The previous section shows that there is a significant difference between recipients preferred format of e-mail and the null-hypothesis that states there will be no difference in effectiveness between the two. In order to fulfil the research aims (8.2) it is necessary to
- 47 -

repeat the Chi Squared analysis using estimated figures calculated from the numbers observed in the experiment. In this way we can observe whether there is a significant difference between the actions and attitudes of the sample group. Table 9 uses expected numbers derived from the ratio of unique opens observed in the experiment. Table 9: Survey - Revised Multimedia vs Text Chi Squared

We can see that there is still a difference between the observed numbers and the revised expected numbers though not as evident. A Chi squared value of 2.42 suggests a 75-90% probability that the result is significant.

- 48 -

Figure 13: Q5a: Reasons for choosing multimedia e-mail

10. 4. 8.

Reasons for choosing multimedia e-mail analysis

The responses to this follow up question are distributed relatively evenly across the answers. Taken in context with the recipients preferred choice of e-mail format these figures suggest that Chittenden & Rettie (2002) were right to propose that "the more colourful and attractive e-mails generate greater response." This factor could have been the driver behind the clickthrough effectiveness of the multimedia campaign being dramatically higher than that for the text-only message.

- 49 -

Figure 14: Q5b: Reasons for choosing text-only e-mail

10. 4. 9.

Reasons for choosing text-only e-mail analysis

The number of responses recorded for a preference for text-only e-mail is too low to extrapolate any significant findings from this follow up question. Of the answers received the predominant reason for choosing text-only e-mail was to not be distracted by the colour and images contained in a multimedia approach. This is perhaps due to the quantity of e-mails received each week, 54 - 93 on average (Brondmo, 2000), limiting the time some users fell they can spend reading each one. However, to reiterate the earlier point, there are not enough responses to make confident assertions and the above analysis is conjecture only.

- 50 -

Figure 15: Q6: Respondent's preferred hyper-link format

10. 4. 10.

Respondent's preferred hyper-link format analysis

The results show that respondents were split between preferring links to be shown in full, and embedded in a word or phrase. These results are slightly contradictory to both the observations taken in the experiment and the responses given thus far in the questionnaire. One would expect the results for "embedded in a word or phrase" and "embedded in an image" to be significantly greater than "link shown in full" which would then correlate with the preference shown by respondents to multimedia e-mail and the higher click-through numbers observed in the experiment. This is not the case suggesting one of two possible causes. Firstly that respondents did not fully understand the question signifying an inadequacy in the piloting of the questionnaire. Or, the second possibility is that there is a difference between the sample groups actions and their attitude towards this question. Further research would be needed to fully assess the influence the format of a hyperlink has on effectiveness as it falls just outside of the scope of this study.

- 51 -

Figure 16: Q7: Respondent's preferred frequency for e-mail communications

10. 4. 11.

Preferred frequency for e-mail communications analysis

The results show a clear preference in the sample group for monthly e-mail communications. Jail currently send information to customers on a monthly basis but only if there is anything new to say. The campaign sent preceding the experiment was delivered in October 2006 which left approximately 4 months between messages. It is possible this may have influenced the campaign effectiveness, however it is unlikely. Response rates tend to drop due to a high frequency of e-mail messages which can annoy recipients (Chaffey, 2006). This is definitely an area that, given the time and resources, would be beneficial to study more closely. E-mail is so easy to filter and delete that any actions leading to the disenfranchisement of customers could be costly.

- 52 -

Figure 17: Q8: Behaviour towards unwanted commercial e-mail communication

10. 4. 12.

Behaviour towards unwanted commercial e-mail communication analysis

This last question, and the follow-up (10.4.13) where applicable, primarily relate to the formatting of future JailMail campaigns. There is one important observation that can be made that has the potential to impact performance metrics. Over 50% of respondents did not select unsubscribe as the option they would take if they no longer wished to receive messages. This is significant because each would classify as a delivered message even though it were unlikely to be read, and thus would lower the open rate accordingly. If we plot the figures for the two campaigns (open % from delivered) next to the projected unsubscribe rate from the table above there is a significant relationship between the two, in particular with the multimedia campaign.

- 53 -

Figure 18: Open & Unsubscribe Percentages comparison

It is not in the remit of this study to explore this in depth although some indicators may be found in the next section, these findings should be observed as stimulation for research focusing solely on this factor.

- 54 -

Figure 19: Q8a: The reasons respondents gave for not choosing to unsubscribe

10. 4. 13.

The reasons respondents gave for not choosing to unsubscribe analysis

The final question in the survey was a follow up to the previous question for those people who chose one of the option that was not "unsubscribe." The even spread of answers suggests that there is not a unanimous reason why people choose not to unsubscribe given the choice. The answers given by the respondents in this section, could, combined with the data from the last section be used as a starting point for more focused research.

- 55 -

10. 5.

Survey Analysis
Comparisons

10. 5. 1.

The key question in the survey that relates to the hypothesis is the question that asks the respondent "which format e-mail communications they would prefer to receive." The questionnaire research method provided extra variables that can be used to analyse this question in more depth. This section will compare two sets of data at a time in an attempt to discover trends or draw conclusions that the previous analysis has been unable to provide.

a.

Gender Vs. E-Mail Format

Table 10: Gender vs Preferred E-Mail Format

- 56 -

The table above compares gender to preferred e-mail format. It was noted in section 10.4.2 that there is a gender bias in the sample. The table shows that 72.73% of respondents were male. The table also shows no significant difference between the genders with regards to email format preference. A higher percentage of females preferred multimedia, but the sample size is too small to infer anything of value from this. The male sample is larger and the 3:1 ratio in favour of multimedia strongly suggests that males prefer the format, however, this cannot be assumed for females. As noted in the Research Methods chapter (9.3.2) the inherent flaw in the research is the sample size used for the questionnaire, and in order to get an accurate finding for females further research is required with a significantly larger sample size. With this limitation in mind, it is still valuable to visualise the data, clearly showing a correlation between the two genders responses.

Figure 20: Gender vs Preferred E-Mail Format

- 57 -

b.

Age Vs. E-Mail Format

This section will assess what effect, if any, age has upon e-mail format preference.

Table 11: Age vs Preferred E-Mail Format

The most significant finding that can be drawn from the table is that 68.18% of the respondents were aged between 18 and 27. Delving deeper into the data reveals that out of this age group 80% prefer multimedia e-mail. These two results observed side by side suggest that age is a factor in determining e-mail format preference, and whilst conjecture, is likely to have been a significant factor in the results observed in the experiment.

- 58 -

Figure 21: Age vs Preferred E-Mail Format % by Age

Figure 22: Age vs Preferred E-Mail Format Frequency by Age

- 59 -

Figure 21 illustrates the percentages of each group that selected a preference for each format of e-mail. These results should be viewed with caution, and in relation to Figure 22 which shows the number of respondents who answered in each section. The three columns denoting 100% are misleading due to only having 1 or 2 respondents in each. The sample list used, and the small list are the likely causes of this anomaly. Figure 22 clearly shows the correlation between the age ranges and the format preferred. Jail Clothing's links to extreme sports is likely to be the cause of the age bias in this research, however, there is room for additional research in this area. It is possible that older age groups view the e-mail paradigm in a different way to the younger audiences, and this would likely be an influencing factor in email effectiveness.

- 60 -

11.
11. 1.

Conclusions
Key Conclusions
Experiment

11. 1. 1.

This section details the conclusions that can be drawn from the research experiment. They are presented in such a way to build a convincing argument with respect to the null hypothesis outlined in the Research Questions section (8.3) as the following;

"There is no difference in effectiveness between HTML and text-only email".

a.

Open Rate

The "unique opens" observed for each format of e-mail appear to clearly favour the HTML campaign, a result that, viewed alone, leads to the conclusion that the null hypothesis maybe invalid. Further, statistical analysis using the Chi Squared method revealed a probability of less than 70% that this finding was significant and so cast some doubt on the previous finding. There is a high probability that the findings were obtained through chance instead of indicating a trend, and therefore this study suggests that the data from "unique opens" is not conclusive enough to disprove the hypothesis, but does suggest there may be some grounds for this to occur.

The next conclusion that can be made is extrapolated from the "total e-mail opens" metric. Unsurprisingly the figures show a strong correlation to the "unique opens" figures indicating a significant trend in favour of HTML e-mail. The statistical analysis for these numbers
- 61 -

produced a 90-95% probability that the result is significant and so we can confidently assert that the "total opens" figures give a strong indication that HTML is more effective. If this assertion is assessed along side the conclusion outlined in the preceding paragraph then there is now plausible evidence in place to conclude that HTML is indeed more effective rendering the null hypothesis invalid.

There is an inherent problem with both conclusions drawn so far, they both rely on open rates. Whilst being useful for measuring trends in a campaign, Mathew Patterson (2007) from Campaign Monitor warns that "you should never take your open rate as a hard and fast number, because you can never know the true figure." Accordingly, while the conclusions drawn to date are valid, detached from subsequent findings they are not reliable enough to be used to address the null hypothesis.

b.

Click-through Rate

Click-through as a measure of effectiveness is significantly more accurate because a recipient has to open, digest, and respond to an e-mail for a measurement to take place. The figures used to analyse this factor were the number of unique recipients who clicked on a minimum of one link. This method therefore produced results that showed for how many unique recipients the e-mail campaigns had effected a response. The click-through numbers recorded are compelling in their own right. 15 unique clicks to 5, or a 3:1 ratio in favour of HTML formatted e-mail. This result is supported by the findings from a DoubleClick (2005) report that found that the click-to-open ratio for HTML had increased from 26.5% to 32%. In context "a higher click-to-open ratio suggests that the content of an email is effective in stimulating click-through once an email has been opened with HTML the format that can facilitate this." In support of this, Chittenden and Rettie (2002) found that "more colourful and attractive e-mails generate greater response." Finally, when the Chi Squared value, showing a >95% probability of significance, is factored in to the findings, this study concludes that the results provide overwhelming evidence to render the null hypothesis invalid. With the supporting evidence derived from the open-rate conclusions drawn above

- 62 -

the key conclusion from the experiment is that the null hypothesis has been confuted, HTML is more effective than text-only e-mail.

11. 1. 2.

Questionnaire

The second research question posed in this study related to the follow up questionnaire, and asked whether the recipient’s attitude towards e-mail reflected the behaviour observed in the experiment. The crucial question in the questionnaire asked the recipients to select whether they prefer to receive multimedia or text-only e-mail (See 7.4.7). The results were definitive and showed a strong preference for Multimedia e-mail. The Chi Squared analysis confirms this and so this study concludes that the recipient’s attitude toward e-mail format is in accord with their behaviour.

Finally, the results from the questionnaire showing gender and e-mail format preference were cross-tabulated to determine if there were any correlation. This study concludes that gender appears not to be an issue with respect e-mail format choice. Both genders favoured HTML e-mail by a significant margin.

11. 2.

Qualifying Statement

This study has concluded that HTML e-mail is more effective at eliciting a response, regardless of recipient, than text-only e-mail. The null hypothesis proposed in the Research Questions should be rejected and replaced with a hypothesis stating "HTML e-mail is more effective than text-only e-mail".

- 63 -

11. 3.

Limitations

Limitations anticipated prior to the research being conducted have been covered in the Research Methodology Chapter and will not all be repeated here. However, a number of limitations were identified during and post the research being conducted and they will be included in this section. The first limitation was acknowledged earlier in the study, and it is an important one. The sample size for questionnaire was very low and as a consequence this factor is likely to have caused gender bias and age bias in the results.

The second limitation is a result of the constitution of recipients in the JailMail list that was used for the sample. The market sector the company operates in attracts a certain demographic of customer and this is likely to have created bias in the resulting findings.

- 64 -

11. 4.

Further Research

There is a multitude of variables that could be used as the basis for further research, with many highlighted in the questionnaire findings. The following table outlines a number of areas where, given the time and resources, further research could have been used to obtain more conclusive findings. Table 12: Further Research Possibilities

- 65 -

12.

Appendices

Appendix 1: JailMail Multimedia Format

- 66 -

Appendix 2: JailMail Text-only Format

- 67 -

Appendix 3: JailMail Questionnaire
http://www.questionpro.com/akira/frame.do?mode=editsurvey 03/08/2007 07:27 PM

Questions marked with a * are required

Hi, You are invited to participate in our survey for JailMail about your email usage. Approximately 150 of you will be asked to complete this survey to help Jail improve their email marketing. It shouldn't take any longer than five minutes to complete the questionnaire. And, as a big thankyou to those who complete the survey, you will be entered into a draw to win some fabulous Jail Clothing goodies. Taking part in this survey is completely voluntary, however, if you feel uncomfortable answering any questions, you can withdraw from the survey at any point. Your opinions will help us to make JailMail the best it can be, so we would really appreciate it if you can finish the survey. Don't worry, the responses you give and your information will all be kept strictly confidential and will be coded to remain private. If you have questions at any time about the survey or the procedures, you may contact Owen@JailMail by email at the email address specified below. Thank you very much for your time and support. Please start with the survey now by clicking on the Continue button below. *

What is your gender? * Male Female

How old are you? * Under 18 18-22 23-27 28-32 33-37 38 and over

How do you receive email? (select all that apply) * Browser based (Hotmail, Yahoo etc.) Email Client (Outlook, Thunderbird, etc.)
http://www.questionpro.com/akira/frame.do?mode=editsurvey Page 1 of 4

- 68 -

http://www.questionpro.com/akira/frame.do?mode=editsurvey

03/08/2007 07:27 PM

Mobile Device (Blackberry etc.) Other (please specify)

You selected that you use a browser-based email service, which do you use? (select all that apply) * Hotmail Yahoo Gmail AOL Other

How often do you you check your email? * Everyday 2-3 times a week Weekly Bi-weekly Monthly Other

There are two main formats an email newsletter can take, multimedia or text. Multimedia is defined as "using html to display pictures, images and colours in an email". Text emails are just that, text only. Given the choice of one of the following options, which would you prefer to receive? * Multimedia email Text-only email

You selected a preference for multimedia (html) email. Would you be able to provide a reason for your choice? Select as many options as you need. * I like how multimedia (html) emails can be presented I like the use of colour in emails I like to see images and pictures in my emails I like changes in text fonts and styles Other

You selected a preference for text-only email, Can you provide us with some more information about your choice? Select as many option as you need. * My email client or reader does not render multmedia (html) correctly I just want the "meat" of the email without the distraction of colour or images

http://www.questionpro.com/akira/frame.do?mode=editsurvey

Page 2 of 4

- 69 -

http://www.questionpro.com/akira/frame.do?mode=editsurvey

03/08/2007 07:27 PM

I like to read my email off-line and don't like the broken image links I have slow internet access and multimedia (html) emails take longer to download Other

When you are reading an email, in which situation are you most likely to click on a hyper-link? * When the link is shown in full When the link is embedded in a word or phrase When the link is embedded in an image or picture Other (please specify)

How often do you prefer to receive email newsletters? * Daily Weekly Bi-Weekly Monthly Quarterly Other (please specify)

If you no longer wish to subscribe to an email newsletter which of the following are you most likely to do? * Unsubscribe Bounce the email Delete without reading Filter in Junk/Spam Other (please specify)

You didn't select unsubscribe as the option you would take if you no longer wanted to receive email communications. What is the reason for this? * It is difficult to find unsubscribe links in most emails I don't trust the unsubscribe links given, they can lead to more spam It takes less effort just to filter or block an email Other

Is there anything you would like to add about email marketing that has not been covered by the questionnaire? Please use the box below to inform us.

http://www.questionpro.com/akira/frame.do?mode=editsurvey

Page 3 of 4

- 70 -

http://www.questionpro.com/akira/frame.do?mode=editsurvey

03/08/2007 07:27 PM

Please contact owen@jailclothing.com if you have any questions regarding this survey.
Privacy | Security Surveys | Email Marketing | Web Polls

http://www.questionpro.com/akira/frame.do?mode=editsurvey

Page 4 of 4

- 71 -

Appendix 4: Chi Squared Test Table

Source: Adapted from (Deacon, J., n.d)

- 72 -

13.

Bibliography

Benbasat, I.G. (1984) An Analysis of Research Methodologies, in: McFarlan, F.W. (ed.) The Information Systems Research Challenge, Harvard Business School Press, Boston, MA, 47-85. Pervan, G.P. (1994b) A Case for More Case Study Research in Group Support Systems, TC8 AUS IFIP Conference, Bond University, Gold Coast, Qld, May 8-11, 485-496. Bartholomew, W., 1963, Questionnaires in Recreation; Their Preparation and Use. New York: National Recreation Association. Walonick, D., 1993, StatPac Gold IV: Survey & Marketing Research Edition. Minneapolis, MN: StatPac Inc. BannerTips, 2001, Comparison between image and text dominant banners and click rate, http://www.bannertips.com/ BT025_answers.shtml Bartholomew, W., 1963, Questionnaires in Recreation; Their Preparation and Use. New York: National Recreation Association. Benbasat, I.G. (1984) An Analysis of Research Methodologies, in: McFarlan, F.W. (ed.) The Information Systems Research Challenge, Harvard Business School Press, Boston, MA, 47-85. Brondmo, H.P. (2001) The Eng@ged Customer: The New Rules of Internet Direct Marketing. Piatkus Books, Chaffey, D., 2006, Total E-mail Marketing, Second Edition: Maximizing your results from integrated e-marketing (Emarketing Essentials), Butterworth-Heinemann Newton, MA, USA Chittenden, L., and Rettie, R., 2003, An evaluation of email marketing and factors affecting response, Journal of Targeting, Measurement and Analysis for Marketing, 11(3), 203 – 218.

- 73 -

Davison, R.M., 1998, An Action Research Perspective of Group Support Systems: How to Improve Meetings in Hong Kong, http://www.is.cityu.edu.hk/staff/isrobert/phd/phd.htm Deacon, J., n.d, The Really Easy Statistics Site, http://helios.bto.ed.ac.uk/bto/statistics/ table2.html#Chi%20squared%20test DoubleClick, 2001, Online brand Effectiveness Study, http://www.doubleclick.com/us/ knowledge_central/documents/RESEARCH/dc_online_branding_0107.pdf DoubleClick, 2005, DoubleClick EMEA Q1 2005 Email Trend Report, http:/ /emea.doubleclick.com/WEB_ADMIN/documents/ dc_Q1_05%20EMEA%20email%20trend%20report.pdf DoubleClick, 2005, DoubleClick EMEA Q2 2005 Email Trend Report, http:/ /emea.doubleclick.com/WEB_ADMIN/documents/ dc_Q2_05%20EMEA%20email%20trend%20report.pdf Electronic Telegraph/Ogilvy & Mather, 1997, How Internet Advertising Works, http:/ /www.intelliquest.com/resources/ whitepapers/interadvert.pdf Emailcenter, 2004, Can your customers read your email newsletters?, /www.emailcenteruk.com/maxenews/ html_email_client_compatability.pdf http:/

eMarketer, 2001, Response Rates of US E-Mail Marketing Campaigns, by Campaign Format, 2001 (as a % of respondents), http://www.emarketer.com eMarketer, 2001, US Internet User Attitudes toward Commercial E-Mail Formats, 2001 (as a % of respondents), http://www.emarketer.com eMarketer, 2002, Consumers’ Preferred E-Mail Advertisement Formats Worldwide, 2002 (as a % of respondents receiving commercial e-mail), http://www.emarketer.com Eyetrack III, 2004, Recall of Information Presented in Text vs. Multimedia Format, http:/ /www.poynterextra.org/eyetrack2004/multimediarecall.htm

- 74 -

Galliers, R.D. (1991) Choosing Information Systems Research Approaches, in: Galliers, R.D. (Ed) (1992) Information Systems Research: Issues, Methods and Practical Guidelines, Alfred Waller: Henley-on-Thames, 144-162. Global Reach, 2004, Evolution of Online Linguistic Populations, http://global-reach.biz/ globstats/evol.html Griffiths, R.T., 2002, Chapter Three: /www.let.leidenuniv.nl/history/ivh/chap3.htm History of Electronic Mail, http:/

Hoffman, D. I., 2000, The Revolution Will Not Be Televised: Introduction to the Special Issue on Marketing Science and the Internet, Marketing Science, Vol. 19, No. 1 IPT, 2004, IPT Email Marketing Survey 2004, http://www.ipt-ltd.co.uk/downloads/press/ 04_10_13_IPT_Research.pdf Jackson, A. and DeCormier, R., 1999, ‘E-mail survey response rates: Targeting increases response’, Journal of Marketing Intelligence and Planning, Vol. 17, No. 3, pp. 135–139. Jennings. J., 2004, The E-Mail Text Vs. HTML Debate: Readers Weigh In, http:/ /www.clickz.com/showPage.html?page=3448841 Jupiter Research, 2005, E-mail Marketing Content Best Practices: Identifying the Impact of Content on Response Behavior, http://www.jupiterresearch.com Levin, W.C., 1988, Sociological Ideas: Concepts and Applications, Wadsworth: California. Maloy, P., 2004, The E-Mail Text Vs. HTML Debate: Readers Weigh In, http:/ /www.clickz.com/showPage.html?page=3448841 Marketing Sherpa, 2003, Surprising Email Test Results: Text vs HTML Survey Versions, http://www.marketingsherpa.com/sample.cfm?contentID=2440# Messaging Online, 2000, Year End 2000 Mailbox Report, http://www.messagingonline.com

- 75 -

Mintel, 2005, Extreme Sports - UK, http://academic.mintel.com/sinatra/academic/ search_results/show&&type=RCItem&page=0&noaccess_page=0/display/id=114733/ display/id=185037&section?select_section=185040 O'Leary, T., 2006, HTML vs. Text: The Debate is Over, http://www.group-mail.com/asp/ common/articles.asp?id=191 Park, M.H., 2002, A Study of Effective Web Advertising Design to Maximize Click-Through and Brand Awareness, http://www.idemployee.id.tue.nl/g.w.m.rauterberg/conferences/ CD_doNotOpen/ADC/final_paper/130.pdf Partner, L., 2003, Why Readers Prefer Text or HTML, http://www.marketingprofs.com/3/ partner3.asp Patterson, M., 2007, All about email open rates, http://www.campaignmonitor.com/blog/ archives/2007/01/all_about_email_open_rates.html Peppers, D. and Rogers, M., 2000, ‘Email marketing maximized’, Peppers and Rodgers Group, Stamford, CA. Pervan, G.P. (1994b) A Case for More Case Study Research in Group Support Systems, TC8 AUS IFIP Conference, Bond University, Gold Coast, Qld, May 8-11, 485-496. Pew Internet, 2005, How Women and Men Use the Internet, http://www.pewinternet.org/ pdfs/PIP_Women_and_Men_online.pdf Radicati Group, 2006, The Radicati Group, Inc. Releases Q4 2005 Market Numbers Update, http://www.radicati.com/uploaded_files/news/Q4-2005_PressRelease.pdf silverPOP, 2005, SilverPOP's "2005 Broken Link Study", http://www.silverpop.com/ downloads/BrokenLink_Study.pdf Skaper, P., 2004, The E-Mail Text Vs. HTML Debate: Readers Weigh In, http:/ /www.clickz.com/showPage.html?page=3448841 Walonick, D., 1993, StatPac Gold IV: Survey & Marketing Research Edition. Minneapolis, MN: StatPac Inc.

- 76 -

Windham, L.,2000, ‘The soul of the new consumer’, Allworth Press, New York, NY.

- 77 -

Master your semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master your semester with Scribd & The New York Times

Cancel anytime.