You are on page 1of 17

Fire Safety Engineering – An Audience & Data Group Case Study

//

Conditional Content Testing
Team
Technology for Marketing & Advertising

Challenge 1) For TFM 2009 the team were keen to improve recipient engagement by conditionalising their emails so that they focused on an aspect of the show matching recipient’s areas of interest. 2) Produce a conditional content process that the Audience & Data Group can successfully roll out and support. Results 1) Test results showed that the TFM audience were more receptive to an email that covered all aspects of the event rather then an email that focused on only an area that they had previously expressed interest in. 2) Test results showed that the TFM audience put greater trust in the TFM brand name then they did that of a keynote speaker. 3) Problems arose prohibiting Online Execs from completing their Conclusion Conditional content: 1) Cannot at present be supported by email polices and procedures imposed upon ADG 2) Cannot at present by carried out by marketing teams without first being given training on how to correctly apply the HTML code used to drive the conditional content, and secondly being willing to allocate a considerable amount of time to its application and testing. 3) Can be valuable for gaining better insight into customer behaviour and hopefully in future once relevant segments have been For more information, get in identified be valuable for improved customer engagement. touch by email Ian.Strong@ubm.com

A UBMi Case Study:
Conditional Content

FSE Background What is the Newsletter Model and what are its benefits? 3-5 Who is eligible for migration on to the Newsletter model? 6 Challenge Solution 8

2

6 7-

Identifying crossover opportunities and measuring their effectiveness

Page 2

Conditional Content

Page 3

Background

Soft Subscribe Strategy Soft Subscribe Strategy

TFM&A is the UK’s only integrated marketing-solutions event for marketing, media and advertising professionals. As such in the run up to the 2009 event the Marketing team were keen to ensure that their own campaigns practice what they preach, namely clever and insightful segmentation and targeting of its customers through its marketing communications. In order to do this the team were keen to segment their customer base by areas of the show that people had previously expressed an interest in; examples included Online Advertising and CRM. These segments would then be sent an email that focused on an aspect of the show matching their areas of interest. The number of segments they wished to target was numerous (5 for one driver) and as such the team were concerned that they would not be able to secure the job slots to fulfil them or have the time to produce the numerous different creatives required for each. As a result the team were keen to see if they could pioneer the use of conditional content via the OTIS and MessageFocus system. A conditional content campaign is defined by the Audience & Data Group as being an email campaign in which multiple campaign creatives can be sent all driven by a unique code in the creative that is specific to a segment. This means Online Advertising people can be sent an email with an Online Advertising focus and CRM people can be sent an email with a CRM focus all through the same SMS job and email template in MessageFocus. At present the Audience & Data Group does not support conditional content as part of a standard email job. As a result if a marketer wishes to send an email to its marketers with a marketing slant and an email to its sales segment with a sales slant it would need to book in two jobs and produce two creative for them.

The aim therefore of the test is to find out if conditional content: A. can be supported by the ADG team B. can be carried out successfully by marketing teams C. has a significant impact upon campaign metrics

Page 3

Conditional Content

Success Criteria

The success of the conditional content test will be measured against four sets of criteria: 1) Does each segment receive a different creative 2) Can a conditional content job be completed under the agreed email fulfilment procedures? 3) Is the time and effort involved in fulfilment of a conditional content campaign shorter than that involved in fulfilling individual campaigns for marketers 4) Does the use of a different creative for each segment have a significant impact on campaign results?

The Challenges

The main challenges facing both the TFM and ADG team were to find a way to; 1) Produce mutually exclusive segments in OTIS. A record can only appear in one segment so as to ensure that it receives the correct creative. Use the cell codes created in OTIS to drive the conditional content in MessageFocus. Cell codes are the only aspect of a file that can be amended for conditional content and be referenced in MessageFocus.

2)

3) Test that the conditional content is correct for each segment. The current approval process allows users to review only one email creative per campaign. However for conditional content campaigns teams would need to review all the types of email they wish to send. 4) Review results and identify if conditional content had an impact upon campaign metrics. Campaign metrics in MessageFocus are reported at campaign or launch ID level only. As a result for conditional content campaigns results are consolidated together preventing us from being able to identify the performance of individual segments.

Page 4

Conditional Content

5) In addition the ADG team would also, dependent upon the results, be required to find a way of rolling this functionality out to the entire business.

The Solution

1) Mutually exclusive segments can be created by the TFM team in OTIS using exclusion queries to suppress records higher up in the dedupe hierarchy. (See below) As such segments and files need to be created in hierarchical order.

(A query with an exclusion file included)

2) Conditional content will be driven by the cell code that the user has appended to the data in OTIS (See below) and subsequently referenced within the HTML code they apply to their MessageFocus template.

Page 5

Contact Ian.strong@ubm.com for further details on how to append cell codes. Conditional Content

3) MessageFocus automatically dedupes by email address, as such a single user is unable to receive the different iterations of the same campaign in their inbox to approve. The only way of reviewing all iterations of a multi creative campaign is for TFM to book in a specific approval job in addition to their final email job. This will be done the day preceding the real email launch and the Online Execs will be able to amend the MessageFocus dedupe process enabling the approver to receive all iterations of the multi creative content and review. 4) The SSRS report “Activity Summary Report- Email” allows teams to break their email campaigns down to cell code level. This means that users who append cell codes to their OTIS files can now go into this report and compare the performance of their segments. Full details of how to use this report can be found on the Wiki under SSRS or by contacting Ian.strong@ubm.com.

In order to identify if multi creative content had an impact on campaign metrics or not a control group of 10% will be created for each segment TFM produce. The Click Open and Unsub Open results of the 10% sample will then be compared with the results of the rest of the segment (the remaining 90%) for statistical difference by: a) Calculating an estimated level of difference for each campaign metric e.g. Click Open Rate.

The estimated difference is the minimum difference that would need to exist between the two results for us to infer that differences can be explained by more then sampling error.

b) Calculating the actual difference between the segments i.e. Click rates for control group = 1.7%, Click Rates of rest of segment = 1.0% so actual difference = 0.7%.

A sampling error is the error caused by observing a sample instead of the whole population; e.g. it may transpire that because our non personalised email was sent to a random sample that all these randomly selected people were women. Women may typically not open up emails as often as men and so we need to factor in an error to account for that possibility.

Page 6

Contact Ian.strong@ubm.com for more details on calculating estimated differences. Conditional Content

Results

We carried out 3 conditional content tests with the TFM team over the course of their 2009 driver campaign and 1 conditional from name test. In each test we were able to successfully test and deliver a different creative to each cell code however during the campaign launch it was identified that: • • • • No Unsub link was being recognised in the HTML Check list ADG could not run a spam check in MessageFocus An error page was encountered when an approval was sent to teams The time taken for TFM to produce the different segments in OTIS and append the correct code in MessageFocus was substantial

The nature of segmentation and content within each creative differed across each test and accordingly we saw the results differ also. Conditional content Test 1 In this test the TFM team segmented their data by the following areas of interest: a. b. c. d. e. Digital Marketing Advertising CRM Data Marketing

90% of those records who fell into the above segments were sent an email in which the: 1) Opening Paragraph 2) Leading suppliers 3) Industry News aspects of the email were amended dependent upon the recipient to focus on their expressed area of interest. I.e. a person who had ticked area of interest as Digital Marketing received an email where the Opening paragraph, leading suppliers and industry news sections focused on all Digital Marketing related content only.

Page 7

Conditional Content

The remaining 10% of records were sent an email in which the Opening paragraph, leading suppliers and industry news sections focused on all aspects of the TFM event. I.e. so a person who had previously indicated they were only interested in advertising would receive an email that talked about all aspects of the event. The consolidated results of the segments are shown below:
T argeted Content 1.3% 98.7% 17.7% 9.5% 5.5% 0.5% Non T argeted Content 1.5% 98.5% 17.7% 9.7% 5.6% 0.4%

Bounced Rate Delivery Rate Open Rate Click Open Rate Unsub Open Rate Spam Open Rate

Driver 1 2009 Met rics 20.0% 15.0% 10.0% 5.0% 0.0%
Bounced Rate Open Rate Click Open Rate Unsub Open Rate Spam Open Rate

M ulti C rea tiv e C ontent

Non M ulti C rea tiv e C ontent

Click Open Rate – Test for statistical significance
Estimated Difference Actual Difference Significant 0.95 -0.20 NO

Unsub Open Rate – Test for statistical significance
Estimated Difference Actual Difference Significant 0.49 0.20 NO

Results showed that there was no statistically significant difference in the Click Open or Unsub Open results across the two strategies (See above) meaning amending the opening paragraph, leading Suppliers, Industry News

sections of an email creative to reflect a person’s area of interest was no more successful at encouraging recipients to act then a general email.

Page 8

Conditional Content

Conditional content Test 2 For driver 3 the TFM team were sending an email out to promote their keynote sessions. As such they segmented their data by the following areas of interest: a. Those interested in Digital marketing, Data & CRM b. Those interested in Digital Marketing only c. Those interested in CRM only 90% of Segment a. received an email creative that promoted both keynotes, 90% of Segment b received an email creative that promoted the Digital Marketing keynote only. 90% of Segment c. received an email creative that promoted the CRM keynote only. The remaining 10% of records were sent a non targeted email that promoted both keynote sessions. The consolidated results of the segments are shown below:

Driver 3 2009 Met rics 25.0% 20.0% 15.0% 10.0% 5.0% 0.0%
B ounced Rate Open Rate Click Open Rate Unsub Open Rate

Ta rgeted C ontent

Non ta rgeted C ontent

Bounced Rate Delivery Rate Open Rate Click Open Rate U nsub Open Rate

T argeted Content 1.7% 98.3% 21.2% 9.5% 2.2%

Non targeted Content 1.1% 98.9% 18.0% 11.6% 2.4%

Click Open Rate – Test for statistical significance
Estimated Difference Actual Difference Significant 1.23 -2.10 Yes

Unsub Open Rate – Test for statistical significance
Estimated Difference Actual Difference Significant 0.59 -0.20 NO

Page 9

Conditional Content

The difference in Unsub Open Rates across the targeted emails and non targeted emails was recorded at 0.2% which was not statistically significant. However the difference in Click Open Rates was recorded at 2.1% this exceeded the estimated difference of 1.23% meaning that the difference was statistically significant and that we can assume that the conditional content did have a direct impact on the number of clicks that the email attained. It transpired though that the non targeted content out performed the targeted content suggesting that people were more receptive to the email that told them about both

keynotes rather then the email that told them only about the one they had previously expressed an interest in. Conditional content Test 3 In driver 4 TFM were sending an email out to promote the seminar halls they had available at the Event and the seminars that were taking place in each hall. To replicate the focus of the halls the team segmented the data by the following areas of interest: a. b. c. d. e. Online Advertising CRM Data Web Digital

90% of records in each of the above segments received an email that presented the times and titles of seminars held in the hall that covered their area of interest. I.e. anyone who had expressed an interest in Online Advertising received an email that contained a table with times and titles for seminars held in the Online Advertising hall only. The remaining 10% of records in each segment received a generic timetable highlighting key talks from each section. The consolidated results of the segments are shown below:

Bounced Delivery Rate % Open Rate % Click Open Rate % Unsub Open Rate %

T argeted Content 2.0% 98.0% 15.7% 7.5% 2.3%

Non targeted Content 2.0% 98.0% 14.8% 13.0% 1.7%

Page 10

Newsletter Model

Click Open Rate – Test for statistical significance
Estimated Difference Actual Difference Significant 1.28 -5.50 Yes

Unsub Open Rate – Test for statistical significance

Top Tip

Estimated Difference Actual Difference Significant

0.51 0.60 YES

The significant difference in Click Open Rates of 5.5% that existed across the targeted and non targeted content was sufficient enough for us to deduce that the use of conditional content did have an impact on campaign metrics. However the results show that the email that covered key parts from each seminar hall was more successful at engaging its audience then the email that attempted to engage its audience based only on the areas they had previously expressed interest in. We also see that the use of conditional content had an impact on the Unsub Open rates too with Unsub Open Rates for the targeted Content being significantly higher then from the non targeted content. The decision to send seminar specific content based on areas of interest appears to have reduced Click and Increased Unsubscribes. Conditional From Name Test 1 –From Name of Driver 3 (6th January 2009) In driver 3 the TFM team decided they would also like to test Conditional “From Name”. In the email they were sending to the CRM segment they obtained permission to use the Keynote speakers name in the emails “From Name”. As such we sent 90% of the CRM segment an email that was addressed from Stuart Lauchlan and the remaining 10% of the segment received the same email addressed from TFM. The consolidated results are shown below:
Conditional F rom Name 1.4% 98.6% 19.2% 11.1% 2.4% Non Conditional F rom Name 1.2% 98.8% 18.8% 12.7% 0.6%

Bounced Delivered Opened Clicked Open U nsub Open

Side by side the results show differences across the campaign metrics with Open and Unsub Open Rates higher in the Conditional From Name campaign and Click Open Rates higher in the non conditional from name campaign.
Open Rate – Test for statistical significance

Top Tip

Estimated Difference Actual Difference Significant

2.78 0.40 NO

Click Open Rate – Test for statistical significance
Estimated Difference Actual Difference Significant 2.35 -1.60 NO

Unsub Open Rate – Test for statistical significance
Estimated Difference Actual Difference Significant 0.63 1.80 YES

The differences in the Open and Click Open Rates were not pronounced enough for us to deduce with any confidence that the use of the keynote speakers name had any impact upon campaign metrics. However the difference in Unsub Open Rates of 1.8% was statistically significant and as such we can say that using the keynote speakers name as the From Name did result in more unsubscribes than using the TFM brand name.

Conclusion

The overriding aims of the tests were to find out if conditional content: A. can be supported by the ADG team B. can be carried out successfully by marketing teams C. has a significant impact upon campaign metrics We attempted to answer this by using the following success criteria: 1) Does each segment receive a different creative? 2) Can a conditional content job be completed out under the agreed email fulfilment policies and procedures? 3) Is the time and effort involved in fulfilment of a multicreative campaign shorter than that involved in fulfilling individual campaigns for marketers? 4) Does use of a different creative for each segment have a significant impact on campaign results?

It was very pleasing to see that in each of the 3 conditional content tests the ADG and TFM team were able to successfully deliver a different email creative to each segment. However we have seen that this was only achievable by completing the job outside of the UBMI agreed email fulfilment policies and procedures. These checks are in place to ensure that every email UBMi sends out is data protection compliant and constructed in a way that will not negatively affect our listing with email service providers who monitor spam scores. Unfortunately for reasons that are still unknown, MessageFocus prevented the Online Execs from completing these on the conditional content campaigns. As such this means that ADG cannot roll out the use of conditional content on a wider premise until the bugs are fixed for fear of launching emails that may contravene the data protection act. The concept of conditional content was designed to fulfil in one email job what had previously only been possible in multiple jobs. In turn freeing up resource with the ADG team to complete more jobs for other teams. However because there is no approval process in MessageFocus for checking multiple creative a specific approval job was needed to be booked by teams the day preceding the launch to review the numerous iterations. As such conditional content is only worth doing if you have 3 or more segments in your data. Whether teams segment their data for use in a conditional content campaign or across a series of individual campaigns the process is the same. As such the use of conditional content did not shorten the time it takes for teams to create their data file nor would it ever. The real difference was supposed to be experienced in the time it takes to product the creative with amendments being shorter then the time taken to produce numerous different creatives. The skills required by the team to include conditional content were new and had to be learnt and this combined with the fact that numerous segments were being tested meant that the time it took TFM to produce a functional conditional content campaign was substantail. This process sped up as the test progressed but was still felt to be very time consuming and nowhere near as speedy as first hoped and anticipated. For TFM the decision to segment their data by area of interest and to use this to drive the conditional content in their emails yielded some valuable insight. In test 1 the minor email amendments made to the opening paragraph, suppliers and industry news had no significant impact upon campaign metrics and therefore recipient engagement.

However in test 2 and test 3 where more noticeable conditionalisation was applied the results suggested with some conviction that the TFM audience were more receptive to emails that cover all aspects of the event rather then emails that focus only on areas that recipients have previously expressed an interest in. This would appear to validate the previous marketing strategy carried out by the TFM team and would also indicate that area of interest, which was a multiple choice question and hence not exclusive, is not a strong enough segmentation set for conditional content to be tested on. So in reflection it is fair to say that conditional content: A. Cannot at present be supported by email polices and procedures imposed upon ADG B. Cannot at present by carried out by marketing teams without first being given training on how to correctly apply the HTML code used to drive the conditional content, and secondly being willing to allocate a considerable amount of time to its application and testing. C. Can be valuable for gaining better insight into customer behaviour and hopefully in future once relevant segments have been identified be valuable for improved customer engagement.

Conclusion

The ADG team are now in dialogue with MessageFocus to get a better understanding of the problems encountered in the tests above. Together we will address and attempt to achieve a solution that can enable conditional content to be carried out by teams in line with UBMis current email campaign fulfilment policies and practices. The ADG team will also begin to formulate a document/training session that can be used to educate users around the business in the correct creation of conditional content. The ADG team will explore more suitable segments for conditional content to be undertaken on.

Newsletter Model

If you decide to do all this using the new newsletter model you wont have to maintain the integrity of your lists any further as Top Tip the system will take care of any new additions/losers. Contact