Customer Metrics

Essen1al
of Customer Metrics Introduc*on to digital measurement for public sector web managers
GSA Web Manager University Alex Langshur, President TwiFer: @alangshur PublicInsite Inc. November 3/3, 2010
Converting visits to value
PublicInsite.com
Web Manager University: Essential of Customer Metrics PublicInsite Web Analytics Inc. 2010 1
Web Manager University: Essential of Customer Metrics PublicInsite Web Analytics Inc. 2010
Common ques1ons
What should I measure?
What should I track to help me understand the eec*veness of my ini*a*ves and op*mize the value across each channel?
How should I measure?

What are the best prac*ces that I should use as a guide? What are the technical issues I should be aware of?
How and what should I report?

What metrics will best grab and hold senior management a@en*on?
Seminar 1metable Day 1

Time
9:00 9:30 9:30 10:30 10:30 10:45 10:45 12:00 12:00 1:00 1:00 2:30 2:30 2:45 2:45 3:15 3:15 4:15 4:15 4:25 4:00 Introduc1ons Session 1: Analy1cs primer review of the fundamentals Morning break Session 2: Analy1cs data core concepts Lunch Session 3: A framework for measuring advanced measurement concepts AXernoon break Plenary case study: evalua1ng the impact of non-human visitors Session 3: con1nued Day 1 review & Q&A Session close
Session subject
Seminar 1metable Day 2

Time
9:00 9:10 9:10 10:30 10:30 10:45 10:45 12:00 12:00 1:00 1: 00 1:30 1:30 2:30 2:30 2:45 2:45 3:30 3:30 3:45 Review of previous days session Session 4: measuring oine and online campaigns Morning break Session 5: measuring social media Lunch Session 6: Integra1ng call centre with web Session 7: Mobile measurement AXernoon break Session 8: Communica1ng the value of analy1cs to internal audiences/crowdsourcing Course wrap-up and session close
Session subject
Speed introduc1ons
Introduce yourself to the person next to you
Someone you dont know
Why theyre here, what they hope to learn

Their role Explore some of the interests inside & outside work
How you are currently measuring performance

What is working for you, what isnt & why
You have 3 minutes each
About PublicInsite
A leading web analy*cs & search engine op*miza*on/marke*ng (SEO) rm
Experts in WebTrends, Google Analy*cs, Urchin, NetInsight etc. Google Analy*cs Cer*ed Partner, Yahoo Web Analy*cs Network Partner, Google Adwords Qualied & Yahoo Search Marke*ng Ambassadors
Specialized in public sector performance measurement

Build and implement measurement frameworks that are aligned to MAFs and focus on tracking outcomes Provide concise, clear and strategic roadmaps that empower managers to assess and improve the quality and eciency of their websites Over 200 client engagements with public sector departments and agencies since 2003
Oces located in Boston and O@awa

Key learning objec1ves

1. Have an good grasp the issues surrounding analy*cs, performance measurement and customer metrics
1. More than enough to be dangerous
2. 3. 4.
Clear sense of the immediate, prac*cal next steps you can take with your analy*cs data and measurement metrics Learn new ways to present and communicate analy*cs data Have fun
2nd 1er learning objec1ves

1. 2. 3. 4. Know whats going with analy*cs and performance measurement in the public sector sphere Review the tools and approaches for web site performance measurement Learn how to use key performance indicators to track progress and measure outcomes Understand the strategic value of performance measurement
1. 2. Its role in the decision making and con*nuous improvement process How to communicate results in a compelling fashion
9
Session 1 Analy1cs primer

1. Technical fundamentals & key concepts 2. Metrics and KPIs
10
When a user clicks on a link

1. Browser sends a requests for a page 2. Web server sends the page to the IP address 3. The page is built by many discrete elements (all calls to the server, or hits) 4. The result: one page view comprising many hits
11
12
How web trac data is collected: Log les

web server 1
Saskatchewan
Log File 1
California
web server 2
Log File 2 Log File 3 Log File 4
Florida
Data analysis and reporting software
Maine
web server 3
Quebec
web server 4
13
How web trac data is collected: Log les

How
Default standard methodology - web server ac*vity is logged Files are parsed by analy*cs sogware Captures all trac to the site (including all non-humans)
Drawbacks
Log les can be huge, par*cularly for popular content rich sites Parameter dense URLs are complex to analyse Caching (browser and proxy) these never make it to the logs! Combining les from mul*ple servers in many loca*ons
Benets
Virtually no implementa*on eort Tracks everything all le types, all visitor types, all pages
How web trac data is collected: Page tags

1 1. Browser sends requests for web page 2 Server responds to request, sends page along
with embedded page tag 2. Browser reads & executes page tag 3
1 1 2
web server
/images/tracking/pagetag.gif
script, sends image request (along with appended data) to server specified in script (collection point)
3
Data collection point
Data analysis and reporting software

15
How web trac data is collected: Page tags

How
Small bit of Javascript placed on each page and executes each *me a page loads in the browser of the visitor Image request along with a query string is sent to the web server, which parses the query and returns the image to complete the transac*on
Drawbacks
Reliance on Javascript and cookies (neither of which may work!) Tag implementa*on: every page and can be complex (data loss) Not all visits captured (e.g. SE spiders), nor all le types (e.g. PDF)
Benets
Track Flash les and much more suited for e-commerce Improved accuracy: tracking clicks, defeats caching issues Quicker data repor*ng: can be in near to real *me if desired
16
Why numbers dont match sessioniza1on.

Raw data Page Views Visits Aggregated by Visitors
17 Web Manager University: Essential of Customer Metrics PublicInsite Web Analytics Inc. 2010
Understanding sessioniza1on:
Session: the sequence of pages seen by one visitor during their visit Sessioniza*on: process used to iden*fy and group ac*ons associated with a visitor. Op*ons are:
least accurate IP Address, IP+Agent: (Basic) technique fraught with inaccuracies Session cookie / ID: (Be@er) cannot track visitors from one visit to the next (all visitors appear as new) Persistent cookie: (S*ll be@er) enables measurement of visitors (as repeat, new, or unique) User login: (Best) Same benets as persistent cookies, but not prac*cal for many sites most accurate
Sessionizing explained: IP address + user agent

What they are
The IP address of the visitor along with the user agent informa*on, i.e. browser type & version, OS type & version, language + other bits
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/ 1.5.0.6
Key issues
Proxy servers can turn thousands of visitors into 1 undercoun*ng! IP addresses can change mid-session (e.g. AOL) overcoun*ng!! Masking/anonymizing of IP addresses
Limita*ons
Many: numbers are suspect, but trends generally are OK Path analysis, unique visitors, *me based analysis virtually impossible
19
A log le entry with cookies

LOG ENTRY for June 1 2005: 2005-06-01 00:25:01 199.212.18.131 GET /pagename/8C3F7D55A9EE-46F2-8CB1-294250AFE77A/sky-clouds.jpg - 304 164 423 0 HTTP/1.1 www.dept.gc.ca Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+.NET+CLR+1.1.4322) http:// department.gc.ca/english/default.cfm COOKIE_ID=X01H37M39QR006YMLW59320001BX4
A unique, randomly generated number (the cookie) is placed in the log to enable tracking of the visitor should they return
10
Sessionizing explained: cookies

What they are
Text with a unique ID placed on the visitors computer Be@er than IP address+user agent (proxy servers no longer an issue)
Session cookies
Dies ager the session ends: if a visitor returns to the site ager ending a session, they are considered to be a new visitor
New visitors are over counted and session *meouts become an issue
S*ll lack ability to measure any metric over *me dimension
Persistent cookies
Survive for predetermined period on machine, enabling sessioniza*on to occur over dimension of TIME. Provides unique visitors, repeat visitors, recency, frequency, path analysis and steps to conversion
The problem with cookies

If cookies are removed or deleted:
Returning visitors will appear to be new visitors
This will also have the aect of ina*ng the unique visitor count
If cookies are rejected:

Sessioniza*on can be severely aected
Impact: the visit might not even be picked up (if using only cookies to sessionize) Must have a back-up sessioniza*on method (IP address & user agent)
In either case:
There is a degrada*on in the data, the amount of which depends on
Degree of reliance on cookies; If analy*cs sogware requires cookies for sessioniza*on
22
11
Use of cookies in public, non-prot and other sectors
Source: WAA Public Sector Benchmark Study, 2009

First principles: the concept of parameters

www.canada.gc.ca/ac*on?source=La*mes&medium=printdaily&campaign=funding_campaign
parameter
value
1.Source: iden*es where the trac is coming from

Parameter example: source= Variable example: BosGlobe, newsle@er0509, AdWords_PPC, etc.
2.Medium iden*es the medium used for the campaign

Parameter example: medium= Variable example: print, radio, e-mail, CPC (cost-per-click), Banner
3.Campaign name iden*es a specic promo*on or strategic campaign

Parameter example: campaign= Variable example: border safety, immuniza*on, swine u, infrastructure
12
Example: URL variable
25
Key Performance Indicators (KPIs)

Tell you how well the site is mee*ng / performing against desired goals
KPIs express or incorporate a business goal, and are therefore usually *ed to a target. Ogen a KPI represents how far a metric is above or below a pre-determined target. Rates, Ra*os, Averages, percentages
Translate complex measures into a simple number
Metrics are dened as:

Direct numerical measure that represents a piece of business data. This measure can be a count (e.g. number of) or a ra*o (the measure is *ed to another dimension, e.g. visitors per month.)
The measure is number of visitors and the dimension is *me (month)
26
13
Sample of common KPIs in public sector

1. Average visit dura*on
Crude proxy of engagement now used by Comscore
2. Page views per visit (view visit ra*o)

Trend & compare to visit dura*on: another proxy of engagement
3. Key page impressions

Visit trends (absolute and % change) to areas you want read
4. Bounce rate
Rise or fall of number of 1 page visits (heavily inuenced by site architecture)
5. Conversion ra*os (various)
27
Dont worry about accuracy or precision: focus on trends
28
14
Want more info on KPIs?
Standards Committee Webanalyticsassociation.org
29
Session 2 Analy1cs data: core concepts (the numbers!)

Its Monday, June 1st
Preliminary analysis quiz
Gezng good numbers

Data scrubbing and the impact of non-humans on site stats
A word about averages The role of Informa*on Architecture

The rela*onship between site structure and results
15
Typical analy1cs report

Metric Visitor sessions Unique visitors Repeat visitors Average visits per visitor Pages viewed Average page views per visit Average dura1on (min:sec) Jan 9,700 7,500 3,250 1.3 55,400 5.7 7:30 Feb 16,900 14,000 5,630 1.2 109,850 6.5 7:15 Mar 14,000 11,500 4,240 1.2 72,800 5.2 6:50 Apr 18,700 15,800 5,460 1.2 100,980 5.4 4:50 May 20,100 16,200 6,120 1.2 112,560 5.6 4:30
31
Exploring visitor sessions
32
16
Exploring visitor sessions
33
Be careful about the numbers

Visit counts must be considered approximate
Sogware use proprietary algorithms counts vary between tools Non-human visits inate numbers
Rise in hacks, automated surveillance, mirroring sites. How should you treat these? Have all search engines been removed?
Should a visitor who leg immediately be counted as a visitor?

Would they be counted as a client in a store? Has value been exchanged?
Try to integrate visit quality into analyses At the macro level, visit trends and scale are more important than the actual number
Actual numbers take precedence at the micro level
17
Impact of non-humans
Site mirroring
Public sector sites frequently focus of third party mirrors with huge page view impact
Suspect visitors
Drama*c rise in repeated 1:1 ra*o visits from Asia & eastern Europe
Cumula*ve impact (remember the Long Tail!) can be substan*al
New search engines / spiders

Sogware updates dont keep track with rapidly evolving market
Internal visitors
Can be huge source of high page view trac. Are they in or out?
Link checkers / code validators / Usability (Sec*on 508) bots

Crawlers that can create drama*c page view boost
Site watchers / monitoring services

Companies that crawl every day to see whats new (e.g. Cyveillance)
The non-human dimension
Isnt this being take care of?
The answer is
NO
18
Impact of non-humans on trac
37
Impact of non-humans on indicators

Email subscription rates
Key document take rates
19
A simple thing to do
Sort host trac by page views instead of visits
Simple step immediately brings to the fore hosts that have are likely non-human Glance at *me online reveals their impact on average visit dura*on
These 6 hosts = 15.5% of page views But less than 0.1% of visits
A word about suspicious trac (are they evil?)

Increasing in amount and frequency Characteris*cs include:
Mul*ple visits from closely spaced IP ranges over short *me period Visits from similar IP block with a page view per visit ra*o of 1
Ogen under 100 visits, 100 page views and
Ogen from eastern Europe (.ro, .hu, .bg), Russia (.ru) and Asia (.cn) Visit day ager day ager day
Numbers can begin to really add up
Bad bots list can be found at:

h@p://www.kloth.net/internet/badbots.php h@p://blamcast.net/ar*cles/block-bots-hotlinking-ban-ip-htaccess
and dont confuse these with RSS readers

20
Bot theres just one problem

Improved quality of analysis Lower traffic numbers
Be suspicious of averages
Averages imply a normal distribu*on curve
Mean and the median are =
The web is all about the Power Law (Long Tail)

Mean and median are
Over reliance on averages (especially if non-humans havent been removed) can lead to incorrect interpreta*on and possibly bad decision making Consider using frac*ons or % of total visits
For example: % of visitors who:
Spent 2 4 minutes on the site View 2 4 pages on the site
21
Power law (Zipf distribution) of web traffic statistics
30% of visits are less than 1 minute: what is their value? 50% visits 2 minutes or less
The long tail - where many valuable visitors lie
43
Informa1on Architecture (IA)

Informa*on Architecture denes how your site is structured
Goal is to create a produc*ve & useful user experience
Good IA reects personas and task accomplishment

Personas (or client journeys) have goals (outcomes) Goal oriented design can be tracked
Good IA -> reliable metrics -> insigh}ul KPIs

Analy*cs can track virtually anything, but its most important to track the things that ma@er most
22
Database sites and analy1cs

Selec*on of content management system ogen done by IT
Measurement or tracking requirements rarely considered in purchase process
IA, persona development and site design ogen outsourced

Few rms consider measurement implica*ons
Result? URLs like:

h@p://webanaly*csassocia*on.org/en/cms/?594 h@p://www.cdc.gov/node.do/id/0900f3ec8000e044 h@p://www.cancer.gov/Templates/drugdic*onary.aspx?CdrID=256422 h@p://www.globesports.com/servlet/story/RTGAM.20061028.wspt-leafs-habs28/GSStory/ GlobeSportsHockey/home h@p://www.canadian-health-network.ca/servlet/ContentServer? cid=1047656077028&pagename=CHN-RCS%2FPage%2FGTPageTemplate&c=Page&lang=En
Plan and design for desired outcomes - ideal

Drive to site initiatives
SEO, email, ads, earned media, etc.
Persona Task funnel
call to action action acknowledgment
Step 1 Step 2 Step 3
Influence on IA, content, measures
Web Manager University: Essential of Customer Metrics 46 PublicInsite Web Analytics Inc. 2010
23
The Public Sector reality
Web Manager University: Essential of Customer Metrics 47 PublicInsite Web Analytics Inc. 2010
Linking analy1cs with Informa1on Architecture
If you want to eat breakfast with the morning sun coming in the window, make sure you build your house with the breakfast nook facing east
48
24
Session 3 A framework for measuring

Understanding pre-click and post-click behavior
How upstream ac*vi*es aect downstream trac
Four dimensions of measurement

Visibility - Referrer analysis Alignment - Keyword analysis Intensity - Naviga*on analysis Conversions - outcome analysis
49
50
25
How important is branded search?

Observe linkage between level of branded search and:
n = 96.7 million visits
- Type of org - Size of org - Size of ad budget - Size of site (# pages)
* weighted average based on the total number of visits
51
How important is the home page?
Sites include public sector, non-profit, and education sectors, and range in visit volume from 10 million pg views/mth down to 100,000 pg views/ mth
26
The Search Results Page (SERP): Ground Zero in the competitive marketplace of ideas The journey of discovery starts here... ... And moves to here Paid ranking
Organic search results (unpaid)
53
While you were arguing about the home page
54
27
Pre-click & post click measurement
55
Benchmarks
Benchmarks = perspective
Understand potential audience size Performance relative to peers against key metrics Excellent free sources include:
Google Analytics Compete.com Quantcast.com Index.fireclick.com Your analytics vendor
56
28
Benchmarking: indices available in Google Analy1cs
Benchmark Your site
57
Compete.com
58
29
59
Web Analy1cs Associa1on Public Sector

30
Visibility
Visibility
Referrer analysis Keyword research Search Engine Optimization Marketing outreach
61
Referrer analysis
Referrer
The URL that sent the visitor to your site Ask yourself: What are the trends? What is the propor*on of one type of referrer to the other? Is the quality of trac dierent from one to the other? Bounce rate, *me on site, exit rates, etc. Do goals change by referrer?
Sources:
Search engines Email newsle@ers Direct access Oine campaign Referring Web sites:
Wikis Press releases Social media (blogs, Tweets, etc.) Other Web sites
62
31
WAA benchmarks for visitor sources used in PS

63
Referrer analysis
64
32
Referrer analysis in Google Analy1cs
Note impact of social media (well return to this later)
65
Keyword analysis
What keywords did people use in search engines that resulted in visits to your site? You can learn about:
Whats popular on your site? Visitor intent
Type and syntax of words Noun versus verbs Adjec*ves used
Trend the use of keywords. Identify seasonality you can use to advantage. Uncover unmet needs that your site may be attracting but are not currently fulfilling. High conversion micro-segments.
66
How good your content is at drawing trac How op*mized your site is to draw search engine trac Improve content alignment
33
Tens of thousands of keywords are they telling you anything, and if so what and how do you figure it out?
67
The now familiar Long Tail / Power Law curve
We rarely to look beyond here but there is important stuff out here
and little of additional substantive value past here
68
34
Our approach
Reverse engineer the Informa*on Architecture (IA) Tag and categorize by main IA branch
Most will be obvious Many will be ambiguous Some should be dumped
Consistency is key!
Categorizer must be very familiar with the organiza*on
Learn how to use Lookup and Pivot Tables
69
canada food guide Whmis canada's food guide Health canadian food guide Hepatitis flesh eating disease food guide smoking diseases ear candling diabetes papilloma virus adderall hpv primary health care nutrition healthy living laser eye surgery west nile virus quit smoking
36,231 24,840 18,873 16,932 14,326 12,975 11,983 9,870 6,392 5,306 5,294 4,986 4,593 4,501 4,204 3,918 3,743 3,295 2,845 2,582
hepatitiss smokin movies syphilis canada's food guide to healthy eating hepatitus canada food celebrex influenza body mass index herpes ketek crestor bdpp second hand smoke bmi canadas food guide sti smoking canadian food adderall xr
2,207 1,934 1,302 1,132 903 875 849 812 723 702 686 613 572 543 421 408 367 348 332 298
70
35
36,231 24,840 18,873 16,932 14,326 12,975 11,983 9,870 6,392 5,306 5,294 4,986 4,593 4,501 4,204 3,918 3,743 3,295 2,845 2,582
2,207 1,934 1,302 1,132 903 875 849 812 723 702 686 613 572 543 421 408 367 348 332 298
71
36,231 24,840 18,873 16,932 14,326 12,975 11,983 9,870 6,392 5,306 5,294 4,986 4,593 4,501 4,204 3,918 3,743 3,295 2,845 2,582
2,207 1,934 1,302 1,132 903 875 849 812 723 702 686 613 572 543 421 408 367 348 332 298
72
36
36,231 24,840 18,873 16,932 14,326 12,975 11,983 9,870 6,392 5,306 5,294 4,986 4,593 4,501 4,204 3,918 3,743 3,295 2,845 2,582
Hepatitiss smokin movies Syphilis canada's food guide to healthy eating Hepatitus canada food Celebrex Influenza body mass index Herpes Ketek crestor bdpp second hand smoke bmi canadas food guide sti smoking canadian food adderall xr
2,207 1,934 1,302 1,132 903 875 849 812 723 702 686 613 572 543 421 408 367 348 332 298
73
36,231 24,840 18,873 16,932 14,326 12,975 11,983 9,870 6,392 5,306 5,294 4,986 4,593 4,501 4,204 3,918 3,743 3,295 2,845 2,582
Hepatitiss smokin movies Syphilis canada's food guide to healthy eating Hepatitus canada food Celebrex Influenza body mass index Herpes Ketek Crestor Bdpp second hand smoke Bmi canadas food guide sti smoking canadian food adderall xr
2,207 1,934 1,302 1,132 903 875 849 812 723 702 686 613 572 543 421 408 367 348 332 298
74
37
36,231 24,840 18,873 16,932 14,326 12,975 11,983 9,870 6,392 5,306 5,294 4,986 4,593 4,501 4,204 3,918 3,743 3,295 2,845 2,582
2,207 1,934 1,302 1,132 903 875 849 812 723 702 686 613 572 543 421 408 367 348 332 298
75
36,231 24,840 18,873 16,932 14,326 12,975 11,983 9,870 6,392 5,306 5,294 4,986 4,593 4,501 4,204 3,918 3,743 3,295 2,845 2,582
2,207 1,934 1,302 1,132 903 875 849 812 723 702 686 613 572 543 421 408 367 348 332 298
76
38
36,231 24,840 18,873 16,932 14,326 12,975 11,983 9,870 6,392 5,306 5,294 4,986 4,593 4,501 4,204 3,918 3,743 3,295 2,845 2,582
2,207 1,934 1,302 1,132 903 875 849 812 723 702 686 613 572 543 421 408 367 348 332 298
77
36,231 24,840 18,873 16,932 14,326 12,975 11,983 9,870 6,392 5,306 5,294 4,986 4,593 4,501 4,204 3,918 3,743 3,295 2,845 2,582
2,207 1,934 1,302 1,132 903 875 849 812 723 702 686 613 572 543 421 408 367 348 332 298
78
39
Categorizing >1.5 million keywords yields
79
Impact of branded search in response to campaign
80
40
Alignment
What is our alignment to the audience and its needs (audience segmentation): Are we reaching the right audience? Are we tracking the right audience? Can we segment, and if so, how? Culture, location, sector, demographic
81
Easy segmenta1on approaches

Always do a quick check for the interna*onal visitors
Assess their impact as % of total and whether they are a key target audience.
If no, remove them. If yes, segment them.
Segment by loca*on
Dierences in behavior ogen *ed to geography (and DC impact is likely quite high)
Segment by new vs. Returning

Provides insight as to what drives engaged visitors, but use only if have cookies!
Segment by behavior
Visitors who look at more 5 pages/visit, or spend 3 mins/visit, or look at key pages
Segment by alia*on
Visitors with common alia*on reect group behaviors and needs that dis*nguish them from the rest (e.g. Post-secondary students, K-12, government, etc.)
41
Results from the WAA Benchmarking survey
Tapping the Potential of Web Analytics for Public Sector and Non-Profit Sites 2009 Web Analytics Association
Impact of interna1onal trac Finance site
42
Impact of interna1onal trac Health site
Impact of interna1onal trac Health site
43
Things to consider
Are interna*onal visitors among your key target audiences?
The decision yes or no should be one of commission not omission
If not, should you allow them to inuence your sta*s*cs?

Leaving them in can have nega*ve impact on key performance indicators
If yes, you should treat them as a separate audience

Segment their trac much as you would spider trac Make sure their naviga*on preferences are not skewing your data interpreta*on Measure their behaviour against key indicators
Intensity framework
Intensity
What is the change in key performance indicators (KPIs) for visits? Wide range of KPIs to choose from:
Page views/visit, average visitor duration Visits/unique visitor, recency, repeat visit rate User generate content volume Bounce / exit rate Related links click-throughs/visit Completed videos/visit
88
44
Trending core intensity indicators
89
Page level analysis of intensity

Entry (landing) pages where a visit starts
Home page top entry page in terms of access
fuhgedabou*t
But campaigns, events, news and other external factors can drive up specic pages
NOTE: For campaigns, we speak of LANDING PAGES Impact from SEO is very big too
Key ques1ons when looking at these indicators: Is anything way out of normal If so, is there an underlying cause? Campaign Home page feature External referrer? Is there an issue that requires a@en*on?
Bounce rate super important indicator

Bounce rate: Percent of people who arrive and leave without going elsewhere in the site. Unclear call to ac*on. Read ar*cle, was fullled, leg. May indicate misdirected visitor
Par*cularly the case with Interna*onal visitors Cannot do *me on page with single access pages!
90
45
Entry (landing) page report
91
Bounce rate analysis (parameter based CMS)
92
46
Exit pages
Some pages are des*na*ons. Exit is expected.
Contact us Downloads Sign-up complete Sent an email
Unexpected high number of exits may be a symptom:

Unclear call to ac*on. Confusion. Couldnt nd the next step. Something on the page caused the visitor to exit.
Lack of clear privacy or informa*on-use policy. Registra*on form was complicated. Signup process was too onerous. Visitor became *red or frustrated.
93
Exit page examples
94
47
Keep in mind
Use of government sites is:
Episodic (ling income taxes, get a license form, pay a ne) Periodic (visit museum website, travel advisories abroad) Con*nuous (weather site, government phone/contact book)
Therefore trying to increase usage may be problema*c

Dont fall into trap of growing visit or page view numbers!
What ma@ers
Awareness and ability to nd the site when the need arises Ease of use and degree to which expecta*ons are fullled Key stakeholders that are targeted are making use
95
Scanning & naviga1on

As visitors make their way through the site
You can understand what naviga*on elements they prefer What content they want, and dont want How they progress through and engage with the site
Enables you to answer

Whether behaviours are changing In which groups they are changing Possibly why they are changing
Evidence which you can use

To realign resources Re-balance content Push back on demands for home page real estate
48
97
One slide can say it all
98
49
Or say it another way
99
Conversion measurement
Conversions
How are value events performing?
Drive to site inititatives Email sign-ups RSS feeds and click-throughs Key content downloads (policies, podcasts, etc.) AJAX, Flash events (e.g. video starts, widget usage) Send to a friend, print page Social tags, bookmarking Satisfaction survey results
100
50
Ensuring meaningful analysis

Ignoring analy*cs requirements during campaign design results in untrackable and unreportable campaigns
Indis*nguishable clicks create analy1cal problems
Email campaigns launched without tagging links Look like direct access visits. in the email, each Impact of layout on clicks? with a separate code. PPC campaigns launched without tagging links Indis*nguisable from organic trac. on the ad. Cannot tell which category is most Mul*ple ads on YellowPages.com eec*ve trac driver. all link to the home page. Looks like direct access. Oine campaign without a Onsite search data may help. unique and memorable URL.
101
Examples of conversion events

Registers for seminar Downloads a form (or prints a form) Downloads a key document / publica*on (or 5 papers) Registers for a e-newsle@er Looks at more than 7 pages in a visit Spends more than 5 minutes on the site Prints a par*cular page Ra*o of keywords to targeted content page views Submits an applica*on Emails content to a friend Looks at the contact us page and leaves Provides user generated content (e.g. on a blog) Watches a video / listens to a podcast Clicks an outbound link (that you have specially tagged)
51
Conversion visualiza1on via funnel reports
103
Measuring drive to site ini1a1ves

Drive to site ac*vi*es include:
Pay Per Click (PPC) Google AdWords, Yahoo Overture, etc. Email marke*ng O line marke*ng (TV, newspapers, radio, brochures) Viral Marke*ng techniques (e.g. YouTube, videos)
All essen*ally varia*ons on a theme

See a link or piece of content in which will take me to the site
Need to instrument all dimensions of the interac*on

Email Embedded link Landing page Ac*on event
104
52
Measuring drive to site ini1a1ves

Measure click-throughs with unique URLs
105
When best to use ROI

Cost per visit
Rough site construc*on and opera*ng costs / number of visitors
Well suited to trac boost resul*ng from outreach campaigns (e.g. $100,000 ad campaign -> 10,000 new visits = $10/incremental visit)
Conversion & take rates for communica*on materials

One of the easiest ROI measures, good metrics on costs
Savings per transac*on

Es*mate cost of online channel vs. other channels, iden*fy savings and mul*ply by number of transac*ons
These are easy to track, easy to report and mostly tangible
106
53
Expecta1on indicator: the cri1cal one

Client sa*sfac*on is the cri*cal indicator
All encompassing metric that sums up the client feelings about the web site
Highly direc*onal informa*on on two key points:

Likelihood to return to the site
It met my expecta*ons the content was aligned to my needs
Likelihood to recommend the site

I didnt waste my *me, I found it valuable & Id recommend it
Can (and should) be tracked over *me

Core feedback tool of the con*nuous improvement cycle
Easy to collect, sta*s*cally valid and adds depth to data

Can inquire about visitor demographics High quality op*ons available with extensive suppor*ng benchmark data
107
Session 4 Measuring oine online

Core concepts
The trinity: the media plan, the measurement plan & the engagement strategy
Tac*cs and approaches

How to measure, what to measure
108
54
Breakdown of adver1sing in private sector...

1 2 2 2 2 3
Breakdown for spend by CMOs in private sector:

TV, print is about same Radio is much less Magazine is much more Internet is slightly more
Global 2007 spend total: $1,900 billion
Source: Zenith Op1media 2007 Ad forecast
A simple measurement concept: Inputs response

Correlate the oine marke*ng / adver*sing inputs with the observed online response Campaign
Oine input 1. Na1onal TV campaign 2. Local radio Post campaign evalua1on 1. POR 2. Analy1cs
Oine input 1. Events 2. Print media buys 3. Out of home
Baseline data 1. 3 6 months min 2. No events 3. No major site modica1ons
Online responses 1. Change in metrics and indicators 2. Growth in branded search 3. Target audience reach 4. Calls to ac1on
Net campaign liX
Time
55
Two requirements to measure oine online 1) An engagement strategy 2) A media/measurement plan

Not enough to drive visits to the site The media plan iden*es:
What do you want them to do? What outcomes are you accountable for? What ac*ons reect message uptake? Are their dierent target audiences?
If so, what are they and how do they see themselves on the site
The mix of media you will purchase Your media spend strategy When you will be in the market The target market (demo / geo)
The measurement plan overlays the media plan and iden*es

The unique variables that are assigned to each adver*sing component How each plan is phased
Either by schedule, geography or both
The goal of a measurement plan is to enable you to understand which media (was it the TV spot or the billboard?) that sent a visitor your the website. Ideally developed at same >me as media plan
The key indicators to track for rush results measurement

111
#1 A media plan (blocking chart)
Typical media plan strategies address *ming, channels, budget, frequency, target audience, etc.
56
#2 A measurement plan
Maps to the media plan and outlines how you will track each channel. Has three main elements
Unique iden*ers to be assigned to each media element
URL, 1-8XX number, keyword, email address, code
Maps out the geographic dimension Syncs up the *ming between channels to evaluate impact over *me
The media plan and measurement plan are co-dependant and must be developed at the same *me
Media plan developed in absence of measurement plan = SADNESS Media plan developed in conjunc*on with measurement plan = BLISS
113
Remember parameters?
www.canada.gc.ca/ac*on?source=La*mes&medium=printdaily&campaign=funding_campaign
parameter
value
1. Source: iden*es where the trac is coming from

Parameter example: source= Variable example: BosGlobe, newsle@er0509, AdWords_PPC, etc.
2. Medium iden*es the medium used for the campaign

Parameter example: medium= Variable example: print, radio, e-mail, CPC (cost-per-click), Banner
3. Campaign name iden*es a specic promo*on or strategic campaign

Parameter example: campaign= Variable example: border safety, immuniza*on, swine u, infrastructure
57
The measurement variables: The loca1on The geographic market you target with out-of-home adver*sing
Can use a URL that is geography specic that redirects via parameters
www.mmm.com/safeglass redirects to www.mmm.com/scg/index.htm?loca*on=on www.mmm.com/secureglass redirects to www.mmm.com/scg/index.htm?loca*on=bc
Or simply run your ads in widely dispersed markets and use geoloca*on tools to track visits
Geoloca*on while not perfect is pre@y darn good, and provides city level detail that you can use to correlate back to the adver*sing channel - but only if youve made sure to phase your spend in each market by media over *me
Example: loca1on mapping

Pre- campaign
During the campaign
116
58
The measurement variables: The keyword Use copy that promotes keywords!
You can choose keywords that are unusual and unlikely to be used by others.
Feature them prominently in the call to ac*on/tag Make sure your site is well SEOd for those words (buy AdWords if you arent ab-sa-looot-ely sure
Now you measure instances of branded search

Filter your keywords for instances of your branded terms. These visitors are likely only to have been inuenced to use the keyword as a result of an ad impression
117
RSS newsfeed from CPSC (opened in Outlook)

Note mul1ple feeds per day based on personal selec1on Teaser paragraph with call to ac1on (view ar1cle) hyperlinked in preview pane brings readers to site (measurable).
118
59
Session 5 Measuring social media

The wild world of measurement
Strategy drives measurement - a social media strategy Categories of measures Bragging about your tagging
Puzng it all together

A measurement framework
Tools you can use
119
The private sector is inves1ng in social media
We dont use (social media) to sell or market. We look at the comments to get a general sense of customer commitment.
- Tony Hsieh, CEO of Zappos.com
60
But many seem to be struggling with how to measure
121
The future is coming
122
61
123
What is it that were trying to measure anyway??

Inuence
Can we have an impact on the ac*ons of others?
Engagement
Are there ac*ons/reac*ons from the crowd in response to your eorts/issues/ programs/organiza*on/ etc.?
Support / Advocacy
Clear expressions of people ac*vely promo*ng your point of view/perspec*ve
Sen*ment
In sum, what is the opinion of the crowd and how is it expressed?
Return
On budget/investment/spend some tangible demonstra*on of value derived from the eort expended
62
Social media measurement categories

Ac*vity indicators
Metrics that are easy to quan*fy, measure and report such as search volume, fans, followers, views, Tweets, etc.
Embracement, inuence & engagement indicators

Things that are tough to measure, but ma@er: sen*ment, tone, and reach Things that tell us about the author, such as rank, authority, clout Things that are straigh}orward like trends, reTweets, user generated content,
Conversion/outcome indicators
Have I learned enough to want to nd out more?
Do I feel strongly enough to want what you are providing
On page events such as click-throughs, downloads, *me on site, pages viewed, sign-ups etc.
If your SM-based content has been tagged bonus! Much more trackable
First, you need a baseline
Type of social media channel Social networking Social media sharing Social knowledge Social news Social bookmarking
Example Facebook, MySpace YouTube, Flickr, Slideshare Wikipedia, TripAdvisor Digg, Twi@er, Reddit, Blogs StumbledUpon, Delicious
Categorize your current sources of trac to give you a sense of what is the nature of social media involvement
63
Two types of ac1vity metrics
Passive interest # fans/friends/followers # page views, visits, *me onsite # RSS subscribers # *me on site # faves # video views Ra*ngs (videos, images) # photos posted
Ac1ve interest # hashtags, # Tweets # Blog pos*ngs Trend & # in share this # Comments # of social bookmarks # Search volume # social media referrals
127
Passive: YouTube sta1s1cs for An1 H1N1 vaccine video
128
64
Ac1ve: Social bookmarks
How the crowd categorizes and perceives your content / role is reected in the organic taxonomies and number of related bookmarks
129
Embracement, authority & engagement indicators

Have your audiences embraced your social media initiative as evidenced by how vibrant, extensive, supportive and engaged they are across your social media platforms?
Assessable
Sentiment (+, -, neutral) Tone (debate, argument) Influencers (who) Reach (people, geographic distribution) Scope (across how many media) Authority and rank
Measureable
ReTweets (number , frequency & clicks) User generated content Volume, frequency, type, incidence of re-purposing Can be on or off site, e.g. comments section of newspaper Net promoter score
130
65
Sen1ment analysis review
Machine analysis 50,000 conversa1ons scored by machine 84% neutral or passive; 11% posi1ve; 5% nega1ve Human analysis 23% of conversa1ons were irrelevant Of the remaining, 30% were neutral or passive (54% dierence); and 63% were posi1ve (52% dierence) See:
Dont trust automated sen*ment scoring (h@p:// bit.ly/9j2bgV) Syncapse Guerilla experiment on sen*ment analysis (h@p://bit.ly/cFzAuS)
So how should you measure sen1ment??

Dont rely on a tool
They s*ll are poor approxima*ons at best
KD Paine recommends that in the absence of a tool, you need to measure sen*ment manually
Take a random sample vs. census (every 10th comment) Categorize the comments into topics and ascribe sen*ment to each
Making and observa*on Expressing support Oering / pushing an opinion Making a sugges*on Asking a ques*on Expressing cri*cism Adver*sing or self promo*on (usually includes a link)
Pay aFen1on to inuencers!
66
CDC Facebook user embraced but not onsite
Edelman Trustbarometer... someone like me this isnt the organiza1on talking

Conversion: understanding the onsite impact
DO THIS ONE THING:
TAG YOUR LINKS

(please)
134
67
SM phone home: tagging inbound trac

Need to dis*nguish which visitors are organic vs. generated from your social media ini*a*ve. Two steps:
1) build the landing page; 2) tag your inbound links
www.PublicInsite.com/NewLandingPage.htm is the page URL www.PublicInsite.com/NewLandingPage.htm? source_var=TwiFer&medium_var=BannerLink&campaign_var=NewSocialMediaCampaignID
Adding addi*onal informa*on to the URL allows us to iden*fy the visitor when they come to our site, to measure the social media campaign
?source_var=TwiFer&medium_var=BannerLink&campaign_var=NewSocialMediaCampaignID
135
SM phone home: tagging inbound trac

Twi@er posts can (and should) also be tagged for tracking purposes. Use URL shorteners such as TinyURL, or Bit.ly which enables you to compress a long URL (like the one in the previous example) into fewer characters: h@p://*ny.cc/QBCXi
When clicked this *ny URL will send visitors to the much longer URL Raw referral informa*on preserved TwiFer has a 140 character limit, so we need a URL shortener!!!
136
68
TwiFer example
Visitor sees your Twi@er post Visitor clicks your Tiny URL Tiny URL invisibly receives the URL informa*on Tiny URL translates the compressed URL Tiny URL seamlessly forwards the visitor to your site as if they had gone directly to your site The original full length URL of your new landing page displays in the visitors browser address bar The addi*onal informa*on about which social media link click brought the visitor to your site is registered, and associated with this visit
A baseline for social media visit

Referred traffic is 50% of all traffic. Of referred traffic, SM represents less than 1%
Social media
0% 10% 20% 30%
all others
40% 50% 60%
70%
80%
90%
100%
All other Social Media Referrers

69
Session 6: Integra1ng web and call centre data

Core concepts
The issue of linking call channel to other outreach channels How you can track sources of calls Measurement
Tac*cs and approaches

The most important call center metrics
139
Two op1ons for linking web and phone
http://www.youtube.com/watch?v=ck-U2He_Fk&feature=player_embedded
140
70
Web analy*cs tool
$
Sample call center metrics

Average Talk Time (ATT) Ager Call Work (ACW) Produc*vity or Average Handle Time (AHT = ATT + ACW) Customer service (rst call resolu*on - FCR) Schedule (adherence % - how closely CSRs stay on schedule) Quality (quality %) Average wait *me Abandon rates (abandoned calls/total incoming calls)
142
71
Session 7 Mobile measurement

Essen*al stats on mobile
Growth and penetra*on rates
The challenges of mobile measurement

What you need to know What are your op*ons
Benchmarks on use of government sites by mobile
143
144
72
Smartphone sales
145
How big is mobile??
Mobile applica*ons expected to generate $6.2B in sales in 2010 and grow to $30B by end of 2013.
Gartner news release, Jan 2010
eBay did US$1.5 billion(!!) via its mobile channel to date in 2010
Bob Page, VP Analy*cs, eMetrics Keynote 2010
146
73
Smartphones are driving the mobile web trac

Handset sales as percentage of sales is growing Sales of unlimited (or very large) data plans growing More Wi-Fi hotspots as coverage con*nues to expand
Jump o the carrier network
Carrier networks are gezng more robust and faster

iPhone alone places huge data strain on carrier networks Huge investments in upgrading capacity (at least in the US/Canada
147
No dierence in private and public sector
148
74
You wake up and check your email on your bedside iPad thats one app. During breakfast you browse Facebook, Twi@er, and The New York Times three more apps. On the way to the oce, you listen to a podcast on your smartphone. Another app. At work, you scroll through RSS feeds in a reader and have Skype and IM conversa*ons. More apps. At the end of the day, you come home, make dinner while listening to Pandora, play some games on Xbox Live, and watch a movie on Ne}lixs streaming service. Youve spent the day on the Internet but not on the Web. And you are not alone.
Chris Anderson & Michael Wol Wired Magazine, August 17 2010
149
10 reasons why mobile web is beFer than apps

1. Greater reach
Apps lock you into a device, sites are device independent
2. Mobile ecommerce works

Many sites have large mobile sales without recourse to an app
3. Growth of Android
Mobile web does not require apps for each pla}orm
4. Findability
A lot of search via mobile (for which you can op*mize) and be present for impulse buys
5. Promo*ng your app

Over 50 stores, a universe of 400K apps = very compe**ve
75
10 reasons why mobile web is beFer than apps

6. Mobile sites can emulate an apps strong points
Many of the key features can be designed into an HTML page
7. Links
The deep linking of the web is not currently possible on an app
8. No approval process!!!
The Apple approval black box can be mys*fying
9. HTML5
Promises to deliver rich, dynamic and measurable web experiences
10. Easier/be@er for SMEs

One site that rules all
152
76
So.. What are the op1ons?
153
Op1on 1: JavaScript based page tags

Lots of comfort and experience using this approach In-situ site management systems governing tag deployment
BUT Will not work on all browsers and all pla}orms

Creates issues for uniformity and compliance to standards
Might not be enough room in the image request to capture all desired informa*on
154
77
Op1on 2: Server side image requests

Much lighter load on the client Rela*vely straigh}orward to implement
BUT Will create more load for the web servers Might not always be compa*ble with exis*ng tag implementa*ons
Par*cularly if you are pushing the tags to deliver a lot of custom informa*on
Will count visits from crawlers and search engine spiders
155
Op1on 3: Specialized systems

PercentMobile Locali*cs Bango analy*cs AppClix Systmes interne/hybride Motally Tapmetrics (App tracking)
156
78
Op1on 4: The Peterson/Opinion lab method
157
Some benchmarks for mobile data
158
79
Change in mobile visits

160,000 0.80% 140,000 Total mobile visits 120,000 Average % of mobile visits Max % of mobile visits
Total mobile visits
0.70%
0.60%
mobile visits as % total visits
159 160
100,000
Min % of mobile visits
0.50%
80,000
0.40%
60,000
0.30%
40,000
0.20%
20,000
0.10%
2007 2008 2009
0.00%
Change in mobile page views

300,000 0.42%
250,000
Total mobile page views Max % of mobile page views Min % of mobile page views
0.35%
Total mobile page views
200,000
Average % of mobile page views
0.28%
150,000
0.21%
100,000
0.14%
50,000
0.07%
2007 2008 2009
0.00%
mobile page views as % total visits
80
8.00%
Percentage of Traffic by Hour of Day
Sitewide vs Mobile Percentage of Traffic by Hour of Day

Mobile Average Sitewide Average
7.00% 6.00% 5.00% 4.00% 3.00% 2.00% 1.00% 0.00%

m am pm pm pm m m -1 1p 11 161 162
1a
-1 1a
2a
3a
4a
5a
6a
7a
8a
9a
2p
3p
4p
5p
6p
7p
8p
10
-1
9p
12
10
h-
1-
2-
3-
4-
5-
6-
7-
8-
1-
2-
3-
4-
5-
6-
7-
9-
8-
11 -
10
9-
12
12
Mobile visitors want informa1on, not brands
Mobile visitors are nearly half (42%) less likely to do a brand based search than non- mobile visitors. This has deep implica1ons with respect to site SEO requirements.
10
12
am
81
Mobile vs. non-mobile users query lengths

Keyword Length (Sitewide)
35
Mobile Keyword Length
30
Number of Text Characters
25
20
15
10
0 Org "A" Org "B" Org "C" Org "D" Org "E" Org "F" Org "G" Org "H" Org "I" Org "J" Org "K" PublicInsite Benchmark
Mobile visitors are more likely to search using longer keywords, typically indica1ng more specic query requests (on average 28% more characters). They obviously will take the 1me to type a detailed and specic query. 163
Session 8 Communica1ng the value of analy1cs

Gezng buy-in
Becoming an evangelist
The con*nuous improvement cycle

Becoming a center of strategic knowledge
How to communicate results

Best prac*ce approaches
164
82
The premise of the cluetrain manifesto is that organiza*ons have been blind to the sea change the Internet represents, desperately clinging to methods that worked wonders in the broadcast era but that are radically unproduc*ve online.
This is par*cularly true of the public sector
165
The last mile is the rst hurdle

The medium is the message
Senior management have short a@en*on spans They forget their logins Few come from the numbers side of the shop Always another priority Reports and dashboards rarely generate phone calls Inboxes are already too full
166
83
Ge}ng out of the analy1cs gheFo

Refuse to produce shelfware Take the *me to use less *me
I would have wri@en a shorter le@er if I had more *me
Leverage every opportunity to engage

Find allies and champions, demand accountability for yourself and others
Never underes*mate the power of free donuts

People will sit through a half hour presenta*on, but wont take ve minutes to read a document
167
Our experience
Good ol fashioned paper has loooong shelf life
Two pages tops, full color, nice stock, easy to hand out
Distribute a publica*on schedule

Deep-dive on a dierent topic each month Ask others what theyd like to know about (make it relevant) Focus on impacts, outcomes and key stakeholders
Supplement the analysis

Oer access to online reports and tools Schedule regular brown bags
168
84
Engaging management Mind the ps & qs

Pique the readers interest
Use a format, structure, design that is user friendly
Parse the metrics

Highlight and focus on the most meaningful metrics (separate business & strategic from diagnos*c metrics)
Provide interpreta*on
Numbers and graphs dont speak for themselves. You must add your analysis and this must be in plain, easy to understand language
Present data exibly

Dont be wedded to what the analy*cs sogware provides. Use formats (charts, graphs, approaches) that are common to the business
Know when to Quit

Always ask: does this ma@er as much to them as to me?
Adapted from Evantage Consulting

169
Repor1ng results sample format
170
85
3 key take aways

Develop and track a maximum of 15 KPIs
See that each one aligns with the mission and goals of the organiza*on Try to develop KPIs that align by level and that resonate with clients
Make sure your data is clean!

Regardless whether using log les or page tags, you must carefully review, iden*fy and then remove non-human trac Track the amount of interna*onal trac: if 20% or more of total, you must make the decision on whether to factor it in or take out (of your KPIs)
Never provide data without interpreta*on

Test out your reports on clients. If they dont understand the signicance of the data, add more interpreta*on Push back on measuring success by visits and views says nothing about engagement, value or outcomes
Further informa1on
For more informa*on, case studies and updates visit www.publicinsite.com Or contact: Alex Langshur, President alangshur@publicinsite.com Twi@er: @alangshur; #measuregov tel: 781 874-0250
172
86

Customer Metrics

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Customer Metrics

Uploaded by

Copyright:

Available Formats

Essen1al

Converting visits to value

How should I measure?

How and what should I report?

Seminar 1metable Day 1

Seminar 1metable Day 2

Why theyre here, what they hope to learn

How you are currently measuring performance

You have 3 minutes each

Specialized in public sector performance measurement

Oces located in Boston and O@awa

Key learning objec1ves

2nd 1er learning objec1ves

Session 1 Analy1cs primer

When a user clicks on a link

How web trac data is collected: Log les

Log File 2 Log File 3 Log File 4

Data analysis and reporting software

How web trac data is collected: Log les

How web trac data is collected: Page tags

Data analysis and reporting software

How web trac data is collected: Page tags

Why numbers dont match sessioniza1on.

Sessionizing explained: IP address + user agent

A log le entry with cookies

Sessionizing explained: cookies

S*ll lack ability to measure any metric over *me dimension

The problem with cookies

If cookies are rejected:

Use of cookies in public, non-prot and other sectors

Source: WAA Public Sector Benchmark Study, 2009

First principles: the concept of parameters

1.Source: iden*es where the trac is coming from

2.Medium iden*es the medium used for the campaign

3.Campaign name iden*es a specic promo*on or strategic campaign

Example: URL variable

Key Performance Indicators (KPIs)

Translate complex measures into a simple number

Metrics are dened as:

Sample of common KPIs in public sector

2. Page views per visit (view visit ra*o)

3. Key page impressions

5. Conversion ra*os (various)

Dont worry about accuracy or precision: focus on trends

Want more info on KPIs?

Standards Committee Webanalyticsassociation.org

Session 2 Analy1cs data: core concepts (the numbers!)

Gezng good numbers

A word about averages The role of Informa*on Architecture

Typical analy1cs report

Exploring visitor sessions

Exploring visitor sessions

Be careful about the numbers

Should a visitor who leg immediately be counted as a visitor?

New search engines / spiders

Link checkers / code validators / Usability (Sec*on 508) bots

Site watchers / monitoring services

The non-human dimension

Isnt this being take care of?

Impact of non-humans on trac

Impact of non-humans on indicators

Key document take rates

A word about suspicious trac (are they evil?)

Bad bots list can be found at:

Sll lack ability to measure any metric over me dimension

3.Campaign name idenes a specic promoon or strategic campaign

If not, should you allow them to inuence your stascs?

All essenally variaons on a theme