26 ANSWERS

ON HOW WE ENSURE QUALITY IN OUR ONLINE PANEL

EPINION

2011

ESOMAR is the global organization for professional market research agencies. They have challenged the industry by formulating 26 questions all research buyers should ask when buying online surveys. Here are our answers. If you have further questions, dont be afraid to challenge us.

EPINION ONLINE PANEL FAQ

>3
YEARS IN OPERATION

01

What experience does your company have with providing online samples for market research?
The Epinion Sample Only technology has been in operation in Denmark for more than 3 years, and recently we have passed our rst milestone which was 2 million completed interviews on the system. We built our rst online panel back in 2001 and have been rening and developing our online methodologies and approaches ever since.

>2.0
MILLION INTERVIEWS WAS COMPLETED BY THIS SYSTEM

02

Please describe and explain the types of source(s) for the online sample that you provide (are these databases, actively managed panels, direct marketing lists, web intercept sampling, river sampling or other)?
Our strategy is to resemble as closely as possible the principles of equal sampleextraction probability known from traditional oline research. Our recruitment ensures that any online population over an extended period of time is equally exposed to invitations to join the panel (in no cases, we allow people to opt in to the panel without being invited). We do this by making sure that we have a large number of open recruitment channels at all times. Moreover, our system supports distributing recruit-

ment eorts into various channels/sites in order to admit desired panelist groups in short periods. In line with the principle of equal extraction probability, we have created a global quarantine system that ensures no potential panelist gets invited to our panel more than once per month. As a result, for each of our country panels, we measure and dene quality in three dimensions: Current size of active panelists Current monthly inow of new panelists Current active recruitment channels (we aim at 75 or more open channels per country)

03

What do you consider to be the primary advantage of your sample over other sample sources in the marketplace?
The competitive advantages of the Epinion recruitment strategy in terms of delivering superior quality sample originate from two distinct traits of the business. First, we keep very close attention to maintaining a large number of dierent recruitment channels in each country as this is the only way to resemble the random probability sampling techniques used in good quality oline interviewing. We use the term resembling oline sampling, as it is obviously not in any real sense

EPINION ONLINE PANEL FAQ

possible to create a known, well dened, universe from which to draw respondents. However, representativeness is all about giving internet users an (approximated) equal probability to be exposed for an invitation to join the panel and later on to answer surveys. The Epinion approach adheres strictly to this principle. Secondly, the unique Epinion sample builder makes it possible to stratify sample so precisely that it is rarely necessary to poststratify data (weigh data) according to demographics. This is so, because: Regardless of what combination of Sociodemographics are used to dene a sample (say, women 15-29 years of age), the system ensures that all the sub-segments in the underlying sample-matrix (region, education, income etc.) are representatively distributed based on the latest statistics from the relevant National Bureau of Census. The system automatically monitors whether the required number of interviews in each of the sub-segments in the samplematrix is reached. If not, additional invitations are distributed to the specic sub-segment in question. This repetitive process ensures incredibly representative sample. In combination, these advantages produce superior samples compared to traditional online sampling. EPINION ONLINE PANEL FAQ

04

If the sample source is a panel or database, is the panel or database used solely for market research? If not, please explain?
The panel is only used for market research nothing else. The sample source is a panel consisting of panelists that have actively opted in to participate in the Epinion panel.

05

How do you source groups that may be hard-to-reach on the internet?

We source these groups by contacting sites that are visited primarily by hard-to-reach target groups. For example, in many countries young low-educated males are a hard to reach target group. In these countries, we make a special eort in reaching out to local footballsites where we know that this group is overrepresented.

06

What are people told when they are recruited?

People are rst asked a series of questions mainly related to background demographics and then they are faced with the following question: Would you be interested in becoming a member of Epinion's shared opinions panel?

8.0
INTERVIEWS ON AVERAGE OF A PANELIST LIFE TIME

Yes, I would like to join No, thank you. The exact wording of the question is slightly dierent for each person, since we have long ago established dierent types of arguments appealing to dierent socio demographic groups. Young people should be recruited using a dierent approach than old people.

08

Please describe the opt-in process

In order to become a member of Epinions internet panel you have to be invited by Epinion. We do not accept voluntarily signups at our website or at other places. All respondents must complete a 4 step verication process to become members: When invited, rst of all: 1. You have to accept our Terms & Conditions, 2. Then you have to accept to participate in our establishment survey, 3. You need to provide us with full demographic background information in the establishment survey, 4. Provide us with your e-mail address and then one more time, reconrm that you indeed wish to become a member of the panel. Unless all of the above steps have been completed using Epinion recruitment system (no afliate recruitment is allowed), we do not consider the applicant to be a member of Epinions panel.

4
STEPS TO OPT-IN > Accept Terms and Conditions > Accept Survey Participation > Provide demographic background > Provide contact email address

07

If the sample comes from a panel, what is your annual panel turnover/attrition/retention rate and how is it calculated?
We do not have an exact number for panel retention. Actually, we dont see panel retention as a goal in and of itself. We do not believe in the panel retention strategy as many other online research companies do. It may be good for prots and is even used as a sign of quality to build up a narrow pool of people making good money out of answering questions, but data from professional panelists does not mirror the average consumer. Therefore, we only use prize draws for small amounts of money as monetary incentives. However, as we care for giving respondents good feedback opportunities, appealing invites and not too long surveys, we have many panelists that complete multiple surveys. The average lifetime number of completed surveys per panelist is approximately 8 surveys.

EPINION ONLINE PANEL FAQ

09

Do you have a conrmation of identity procedure? Do you have procedures to detect fraudulent respondents at the time of registration with the panel? If so, please describe?
Since we do not pay individual monetary rewards to respondents, we can rely on e-mail identication as our main identication procedure. All invitations are distributed / linked to a unique e-mail address. Without access to this e-mail address, you cannot participate. As part of the survey process we monitor fraudulent respondents using a series of methods described in greater detail in question 24. Also, we select and invite panel members, they dont select us. Respondents cannot nd ways to self-recruit, thus making it more difcult for fraudulent respondents to create multiple accounts.

countries (elsewise a very difcult thing to handle when making cross-country surveys). As a result, it is possible to make exact comparisons across dierent countries, for example: high educated vs. low educated or between occupational categories. In the early days of our panel build-up, we proled the panelists with lots of dierent variables: Car ownership, newspaper subscription etc. However, it turned out that out of perhaps 150 variables we only made good use of a few. And often we found that the variables were obsolete when we actually needed them. In the meantime, a panelist might have bought a new car, or changed his newspaper subscription etc. Therefore, we now take a more dynamic approach to proling, meaning that we in ad hoc surveys reach narrow target groups by using a sophisticated screening system. When wording screeners we take care to avoid self-selection bias. For instance by using a wording like: What newspaper do you read? (and then present a range of newspapers where one of them is the newspaper in question). Instead of asking: Do you read newspaper A. Yes or No? Also, if clients want to create a specic subpanel, we prole over a short period of time a large part of the panel with the requested prole.

10

What prole data is kept on panel members? For how many members is this data collected and how often is this data updated?
The Respondents are proled with sociodemographic criteria. The exact wordings of the prole questions are developed carefully and are in line with the acclaimed World Values Survey. This makes sure that all categories are compatible across dierent

EPINION ONLINE PANEL FAQ

211,000
CURRENT ACTIVE PANEL MEMBERS

11

What is the size and/or the capacity of the panel, based on active panel members on a given date? Can you provide an overview of active panelists by type of source?
At all times, one can see the current number of active panelists, the current inow of new panelists and the number of active recruitment channels in all of our country panels at www.respomondo.com/panels An active panelist is dened in a rather narrow way, meaning that is a panelist that has responded to a survey invitation within the last 6 invitations.

latest statistics from the relevant National Census Bureau. The system automatically monitors whether the required number of interviews in each of the sub-segments in the samplematrix is reached. If not, additional invitations are distributed to the specic sub-segment in question. This repetitive process ensures incredibly representative sample.

Vietnam Sweden Finland Norway Ukraine Denmark

41 24 15 21 5 105
12

13

Explain how people are invited to take part in a survey. What does a typical invitation look like?
People are invited to surveys by way of e-mail invitations. We work with acclaimed email newsletter-type companies and consistently work hard to maintain an industry leading mail reputation which is crucially important for mail deliverability among leading email providers (Hotmail, Yahoo! mail etc.). We operate with many dierent types of invitations depending on the type of survey. But under no circumstances can an invitation be sent out without active decline and panel opt out links as well as the accept link.

MEMBERS OF ONLINE PANEL

(IN THOUSAND)

Please describe your sampling process including your exclusion procedures if applicable. Can samples be deployed as batches/replicates, by time zones, geography, etc? If so, how is this controlled?
The sampling process is at the very heart of what makes Epinion very special in terms of online quality sampling. Our clients or we on behalf of the client can dene a sample (say, women 15-29 years of age) and then the system will ensure that all the sub-segments in the underlying sample-matrix (region, education, income etc.) are representatively distributed based on the

14

Please describe the nature of your incentive system(s). How does this vary by length of interview, respondent

EPINION ONLINE PANEL FAQ

characteristics, or other factors you may consider?

Over the cause of the years, we have done quite a bit of research on what incentives work the best. As a result, we have structured our incentive/motivational system around the followings: Monetary incentives: We almost always only use prize draws. We draw frequent winners but we award only small amounts of money at a time. Weve found this works better than infrequent, bigger, awards. Even more, our ndings suggest that micro payments to all participants do increase participation a little bit but rst and foremost, it creates more awed answers, more fraud, and above all a pool of professional panelists which in no meaningful way can be said to be representative for the average consumer. Feedback/Motivational incentives. After each survey, each respondent are prompted to give immediate feedback both by ranking the survey and by giving verbatim feedback. This provides the project manager with valuable feedbacks, but even more importantly, it gives respondents an opportunity to vent whatever feelings they have about the survey. This adds to our general know-and-appreciate-yourrespondent policy. In some instances, we oer our respondents access to selected results from the survey and other specically targeted incentives. EPINION ONLINE PANEL FAQ

15

How often are individual members contacted for online surveys within a given time period? Do you keep data on panelist participation history and are limits placed on the frequency that members are contacted and asked to participate in a survey?
Yes, we apply very stringent respondent quarantine practices. No respondent is invited to more than one survey every 12 days. They only receive one reminder per survey invitation We save full statistics on invitations sent, responses (accept, decline), mail-server statistics, participation records on all respondents as long as they are members of the panel.

16

Is there a privacy policy in place? If so, what does it state? Is the panel compliant with all regional, national and local laws with respect to privacy, data protection and children e.g. EU Safe Harbour, and COPPA in the US? What other research industry standards do you comply with e.g. ICC/ESOMAR International Code on Market and Social Research, CASRO guidelines etc.?
We adhere to national and local laws, and beyond this, we follow ESOMARs International Code on Market and Social Research.

30-45%
PARTICIPATE RATE

17

What data protection/security measures do you have in place?

Epinion is concerned with safeguarding the information gathered through our services. We employ a variety of measures designed to protect your information from unauthorized access. We work only with well-known and professional database suppliers and only authorized personnel are allowed to conduct database operations.

19

Do you conduct online surveys with children and young people? If so, please describe the process for obtaining permission
In line with the ESOMAR code of conduct, we do not commission surveys among children unless we have asked their parents for permission.

1-25%
RANGE OF DROP-OUT RATE

20 18
Do you apply a quality management system? Please describe it
As most of our survey processes is embedded in sophisticated IT-technology much of the quality control is embodied in the IT-development process itself. In this process, a variety of techniques has been put to use in manual as well as automated testing. In terms of work-procedures between dierent departments and individuals and with respect to the content of deliveries designed and produced by our consultants or other sta, our processes and deliveries are internally audited every 12 months.

10-15%
AVERAGE DROP-OUT RATE

Do you supplement your samples with samples from other providers? How do you select these partners? Is it your policy to notify a client in advance when using a third party provider? Do you de-duplicate the sample when using multiple sample providers?
Occasionally, we ask some of our partners to provide samples in countries where our own panels are too small or over-extended. We only use Partners from a preferred supplier list. That is, trusted partners with proven quality and service. As we do not split samples between our own panel and partner panels, we do not need to de-duplicate samples between two dierent panels.

EPINION ONLINE PANEL FAQ

21

Do you have a policy regarding multipanel membership? What eorts do you undertake to ensure that survey results are unbiased given that some individuals belong to multiple panels?
Our panel management system automatically controls that there are no duplicate emails. In addition, in each survey, by way of browsersession, IP-address and cookie-tagging, we make sure that no respondent can answer any given survey (including tracking surveys) more than once. Overall, probably because we rarely pay individual monetary rewards to respondents, we have never encountered any specic problems in terms of having respondents with multiple panel-memberships.

depend greatly on length of the survey, as well as the content of the survey. We continuously monitor drop-out rates, accept-rates etc. on each survey. If performance is below average, we introduce suggestions to improve the content of the survey and we very often penalize surveys with high drop-out rates. It is our philosophy to maintain a healthy relationship with the respondents to ensure repeat business. In order to maintain that relationship, we often reject long surveys (or insist that they broken down into several independent surveys), and evaluate the language used in the surveys using the WYMAT-principle (Would Your Mother Approve This?).

23 22
What are likely survey start rates, drop-out and participation rates in connection with a provided sample? How are these computed?
Participation rate is usually between 30-45%. This rate depends greatly on a series of parameters, like time of day the invitations are distributed, the day of the week, the target group selected, e-mail invitation text provided etc. Drop-out rates are between 1-25%, with an average around 10-15%. The drop-out rates

Do you maintain individual level data such as recent participation history, date of entry, source, etc., on your panelists? Are you able to supply your client with a per job analysis of such individual level data?
We maintain a full record of all individual participants. Since we source only respondents from our own in-house fully proled panel, we are able to retrieve a full log of all actions performed by our system in regards to a single respondent, as well as a full record of all a respondent actions in relation to our survey invitations.

EPINION ONLINE PANEL FAQ

10

24

Do you use data quality analysis and validation techniques to identify inattentive and fraudulent respondents? If yes, what techniques are used and at what point in the process are they applied?
We use a mix of dierent techniques to identify fraudulent users. We combine hard data from the participants computer/browser with behavioral data analysis in order to identify fraudulent respondents. Hard data consist of: Session cookies, browser cookies, IP-address validation, e-mail validation (double opt-in required, no e-mail equals no survey) + a wide range of dierent browser plug-ins. We utilize a wide range of information about the browser used as well as the network connection used in order to verify the participant. All these hard data are combined with our ability to monitor deviations in behavior when a participant completes a survey. We analyze features like: incoherent response patterns, average time spent deviations, probability of completing open-text answers etc. Together, all these information creates a credibility-index, which allows us to monitor suspicious behaviors closely.

But our most eective weapon against fraudulent behavior is to remove the monetary incentive to perform fraud. Epinion almost always oers price-draws as a reward for participation. No cash rewards, rebates or points are rewarded as payment for each completed survey. We do not require that participants t a specic prole in order to receive the reward in this case participate in the prize-draw. If you spend time answering our surveys, you participate in the prize-draws no matter if the participant ts the target group or not. We have completely removed the incentive to provide false answers in order to gain an advantage by earning money, points etc.

EPINION ONLINE PANEL FAQ

11

25

Do you measure respondent satisfaction?

Yes, after each survey is completed, all respondents are motivated to give feedback. A typical feedback page looks like this (but will be modied to suit dierent brands and languages):

with the survey or questions that are difcult to understand) that requires immediate action is automatically forwarded to the project manager by e-mail. The feedbacks collected from the respondents are then combined with hard data of completion rates, average interview time etc. into an aggregated easy-to-read feedback report format.

We also use the invitation e-mails to motivate the respondents to provide us with feedbacks, also to those respondents who have failed to complete the survey. Epinion measures respondent satisfaction on ve comparable parameters as well as collecting open verbatim feedbacks from the participants of each survey. These feedbacks go directly to the client project manager. The project manager can always access the respondent feedback report which is updated in real time manner as a respondents feedback comes in. Critical feedback from the participants (like technical problems

EPINION ONLINE PANEL FAQ

12

26

What information do you provide to debrief your client after the project has nished?
Along with the respondent feedback report shown above, the client can at all times access the current completion report showing exactly the progression of the survey in its lifecycle. Drop-out rates, participations rates, invitations rounds executed, success rates etc. are all supplied in an easy-to-read interface. Epinion also provides the project manager with a General Statistics report benchmarking her/his survey performance with Epinion averages internationally. The same report evaluates not only current performance, but also past performance of the project managers surveys, and provides recommendations on how to improve future survey projects. See slide from benchmark report below:

EPINION ONLINE PANEL FAQ

13

EPINION COPENHAGEN
RYESGADE 3F 2200 COPENHAGEN DENMARK T: +45 70 23 14 23 E: INFO@EPINION.DK W: WWW.EPINION.DK

EPINION AARHUS
SNDERGADE 1A 8000 AARHUS C DENMARK T: +45 87 30 95 00 E: INFO@EPINION.DK W: WWW.EPINION.DK

EPINION SAIGON
11TH FLOOR, DINH LE BUILDING 1 DINH LE STREET, DISTRICT 4, HCMC, VIETNAM T: +84 8 38 26 89 89 E: OFFICE@EPINION.VN W: WWW.EPINION.VN

EPINION ONLINE PANEL FAQ