Measuring Compliance Effectiveness

Guide for risk managers and evaluators
Measuring compliance
effectiveness
Evaluating effectiveness
NAT 73580-06.2012
OUR COMMITMENT TO YOU
We are committed to providing you with accurate, consistent
and clear information to help you understand your rights and
entitlements and meet your obligations.
If you follow our information in this publication and it turns out
to be incorrect, or it is misleading and you make a mistake as
a result, we must still apply the law correctly. If that means you
owe us money, we must ask you to pay it but we will not charge
you a penalty. Also, if you acted reasonably and in good faith we
will not charge you interest.
If you make an honest mistake in trying to follow our information
in this publication and you owe us money as a result, we will
not charge you a penalty. However, we will ask you to pay
the money, and we may also charge you interest. If correcting
the mistake means we owe you money, we will pay it to you.
We will also pay you any interest you are entitled to.
If you feel that this publication does not fully cover your
circumstances, or you are unsure how it applies to you,
you can seek further assistance from us.
We regularly revise our publications to take account of any
changes to the law, so make sure that you have the latest
information. If you are unsure, you can check for more recent
information on our website at www.ato.gov.au or contact us.
This publication was current at June 2012.
© AUSTRALIAN TAXATION OFFICE FOR THE PUBLISHED BY

COMMONWEALTH OF AUSTRALIA, 2012
Australian Taxation Office
You are free to copy, adapt, modify, transmit and distribute this material as Canberra
you wish (but not in any way that suggests the ATO or the Commonwealth June 2012
endorses you or any of your services or products).
JS 24302
CONTENTS
03
INTRODUCTION 2
Understanding compliance effectiveness 2
Evaluating compliance effectiveness 3
REPORTING ON EFFECTIVENESS 24
01
The executive summary 25
The detailed report 26
04
DEVELOPING YOUR SUITE OF INDICATORS 6
Indicators 7
Identifying potential indicators 7
Validating your indicators 9 APPENDIXES 27
Measuring your indicators 13 Appendix 1 28
Identifying your data requirements 16 Data collection methods and sources 28
Appendix 2 31
02
References 31
Appendix 3 32
Glossary 32
EVALUATING EFFECTIVENESS 18 MORE INFORMATION inside back cover

Gathering your data 19
Analysing your indicators 19
Determining your effectiveness 21
MEASURING COMPLIANCE EFFECTIVENESS 1

INTRODUCTION
Evaluating the extent to which our compliance strategies have
resulted in positive and sustainable changes in compliance For more detailed information about the different
behaviour and community confidence is critical to knowing measurement methods and analysis techniques and an
whether or not we have been effective in achieving the ATO intent extensive range of external reference material, refer to
as outlined in our Strategic Statement. Literature review – Measuring compliance effectiveness
2007 (NAT 71078).
We have developed a compliance effectiveness framework
and methodology which helps us measure the impact of our
compliance strategies on voluntary compliance and community
confidence. This guide is part of a suite of publications designed UNDERSTANDING COMPLIANCE
to help people understand the framework and apply the EFFECTIVENESS
methodology, as well as develop effective compliance Compliance effectiveness is the extent to which the actual
strategies. outcomes of our compliance strategies align with our desired
outcomes. Did our strategies have the effect or impact on
This booklet provides guidance to risk managers and evaluators
compliance behaviour and/or community confidence that we
in applying phases 3 and 4 of the compliance effectiveness
wanted them to?
methodology. It should be read in conjunction with Measuring
compliance effectiveness – Applying our methodology The compliance effectiveness methodology is the tool we use to
(NAT 72342). plan the right mix of strategies, measure our effectiveness and
identify opportunities for continuous improvement. We use it for
This booklet focuses on:
both planning and evaluation purposes. What we learn about
n developing a suite of indicators our effectiveness informs our future planning and helps re-shape
n evaluating effectiveness our compliance improvement strategies.
n reporting on effectiveness.
There are four iterative phases in our methodology (figure 1):
n Phase 1 is about understanding and articulating the risk
and making sure it aligns with the ATO intent.
n Phase 2 is about clearly expressing our desired outcomes
and what would be different if we are successful. It also
involves identifying the right mix of treatment strategies which
treat the drivers of compliance behaviour and not just the
behaviour.
n Phase 3 involves identifying potential indicators and validating
them to ensure they are viable and useful. The suite of
indicators should be capable of providing a defensible picture
of our effectiveness.
n Phase 4 deals with evaluating and reporting on the extent
to which we have been effective in changing compliance
behaviour, building community confidence, or both, over
the immediate, intermediate and long term.
Phases 1 and 2 focus on planning to be effective while phases
3 and 4 focus on evaluating the extent to which we have been
effective.
2 MEASURING COMPLIANCE EFFECTIVENESS

INTRODUCTION
EVALUATING COMPLIANCE When should an evaluation occur?

EFFECTIVENESS Identifying when an evaluation should occur is based
When you evaluate compliance effectiveness you assess the on when you expect to see an effect.
extent to which your compliance strategies have resulted in In deciding when to undertake an evaluation consider
positive and sustainable changes in compliance behaviour with the following questions:
one or more of the key compliance obligations (registration,
n When do you expect to start seeing the change?
lodgment, reporting and payment) and community confidence.
n When do you think the full effects will be seen?
The methodology uses indicators to provide insights into the n How often will you need to do an evaluation to be able
effectiveness of our strategies because it is not always possible, to understand whether the change has been sustained?
practical or cost effective to use more definitive measures.
A suite of indicators is used to identify whether or not the
success goals, and ultimately the desired outcomes, have
been met. This is done because no single indicator will
provide sufficient information to enable you to assess the
overall effectiveness of your compliance strategies in
achieving your desired outcomes.
When evaluating compliance effectiveness you need to
answer the following key questions:
n Is there a change in compliance behaviour and/or
community confidence?
n If there is a change, did we cause it (attribution)?
To understand this, you need to explore behaviour at:

n the broad or macro level to identify whether there are any
observable changes
n the strategy or micro level to determine whether those
changes can be attributed to our compliance strategies.

INTRODUCTION
FIGURE 1: Compliance effectiveness methodology
Phase 1 Phase 2
Articulate risk Define outcomes
Align with ATO intent Develop strategies
What outcomes
How does the ATO
are you seeking
intent translate into
to achieve by
the context you’re addressing
working in? the risk?
What are the What is the What strategies will How do you define
behaviours and compliance risk to you use to deliver success in terms of
drivers of the risk? achieving the intent? these outcomes? more specific goals?
Who’s involved Who are your

in the risk? target groups?
CHECKPOINTS CHECKPOINTS
You should ensure that: You should ensure that:
n the risk aligns to the ATO intent n the desired outcomes align to the ATO
n the risk adequately reflects intent and the risk you have identified
the behaviours and drivers n the overall desired outcomes
n the risk is refined, concise and not are adequately covered by your
open to interpretation. specific (success) goals
n the strategies target participants of
the risk and drivers of the behaviour
n the strategies address any unintended
consequences.

INTRODUCTION
Phase 3 Phase 4
Design indicators Validate indicators
Determine extent of effectiveness
What are How will the

the potential evaluation be
indicators? conducted?
Which indicators Which of these What are the What data will be
are feasible? indicators will paint indicators required and where
a defensible picture? telling us? will we get it?
What is the description of the What’s our

indicators and their purpose? baseline?
CHECKPOINTS CHECKPOINTS
You should ensure that: You should ensure that:
n each success goal can be measured n data exists, is available, or can 
and has a corresponding indicator be acquired at a reasonable cost
(or suite of indicators) n we can explain how effective our
n potential indicators are aligned with compliance strategies have been
the success goals, desired outcomes – that is, the extent to which actual
and the ATO intent outcomes align with desired
n indicators are viable and provide a outcomes.
balanced picture of performance
n both qualitative and quantitative
information is used.

DEVELOPING
01
YOUR SUITE
OF INDICATORS
The answers to the questions posed in phases

1 and 2 of the methodology should have been agreed
with the risk owner and endorsed by the relevant risk
management committee (or equivalent decision making
body) before developing your indicators.
01 DEVELOPING YOUR SUITE OF INDICATORS
Before you can determine whether your strategies have Bring together people who have the relevant business
had the desired effect you must: knowledge and those familiar with the business data.
n identify potential indicators Consider involving:
1
n validate those indicators n the risk manager
n determine a suite of indicators that will help you paint n intelligence experts
a defensible picture n strategy experts
n decide how you will measure your indicators. n risk analysts
n business and data analysts.
You need to undertake these activities soon after the risk
statement, desired outcomes and strategies have been
endorsed by your risk management committee (RMC) or Identifying the indicators
equivalent. Working through these steps will allow you to Identify appropriate indicators by looking at each success
put the processes and tools in place to ensure that the goal and thinking about the changes you would expect to
relevant data can be collected and analysed. see if the strategies were effective.
There is usually a considerable time lag between setting up Think about how those changes could be measured. Ask
and actually doing the evaluation because the strategies will yourself these questions:
take time to have an effect. n How would you know if there was an improvement in
If you are evaluating effectiveness retrospectively you will be voluntary compliance?

in a position to undertake these activities almost immediately n What would tell you if there was a change in community
after completing phases 1 and 2 of the methodology. confidence?
n How did you know there was a problem in the first place?
INDICATORS
An indicator is not a measure – it is a signpost that will help you Example of a potential indicator
understand whether your strategies have resulted in the change
you were seeking. It forms part of a suite of indicators that will Success goal
enable you to paint a defensible picture of effectiveness.
Sustained improvement in voluntary compliance
Indicators are not designed to measure the effectiveness or
with reporting obligations.
efficiency of an individual strategy. Rather they help you to
understand whether the strategies implemented have resulted Potential indicator
in improved compliance behaviour and community confidence.
Change in the effective tax rate.
When developing your suite of indicators it’s not essential to
develop different indicators for each success goal. You may, in
fact, find that one indicator can be used against several success At this stage, don’t be too concerned with how difficult it might
goals. This can be helpful in reducing the overall cost of tracking be to measure the indicator. Avoid choosing only indicators that
your progress. are easy to measure. You may find that even though an indicator
can’t be measured at the moment it may be useful at a later
date when more data becomes available.
IDENTIFYING POTENTIAL INDICATORS
You need to identify a list of potential indicators that will show Use a combination of indicators that focus on both the broad
whether you have achieved your success goals. and the detailed level to help you to identify whether there has
been an observable change in compliance behaviour and
community confidence. This can also help you to understand
Who should be involved? whether your compliance strategies are responsible for
Involve a range of subject matter experts to explore all
the change.
possibilities including people who understand the risk and the
desired outcomes. They should also be aware of the success When identifying your indicators consider whether they are
goals and strategies identified in phases 1 and 2. capable of showing:
n sustainability – identify indicators that can demonstrate your
progress and whether a positive change is maintained or
declines over time. Select indicators that can show change
over the immediate, intermediate and long terms.

n indirect or flow on effects – include indicators that can show Documenting the discussion
whether the strategies have had an effect on the wider Document the discussion about each indicator including:
population beyond the part of the population directly targeted n what the indicator is supposed to do
by your strategies.
n the expected result if the strategies are successful
n unintended consequences – consider indicators that will
n the external factors that may affect the usefulness of
show whether any unintended consequences have emerged
the indicator.
as a result of your strategies.
This will help to ensure that what the indicator actually
Identifying potential indicators often results in a list of 15 to 25
measures is in line with the original intent.
indicators. While this number would be impractical and costly to
track, the discussion between people from diverse backgrounds It also ensures that these considerations are available to
and capabilities is invaluable in identifying indicators that might others who may work with the indicators at a later date.
otherwise be overlooked.
TABLE 1: Aligning indicators to success goals
Characteristics of indicators
When developing your potential indicators make sure they are: Success goals Potential indicators
n able to be measured – indicators should be based on a
combination of quantitative and qualitative data. The community Change in the:
is less tolerant n number of reports received on
Quantitative data is information that can be counted. For
of non-compliant the Tax Evasion Referral Centre
example, 78% of individual taxpayers use a tax agent to
behaviour n responses to questions relating
lodge their income tax return.
Qualitative data is descriptive data that is difficult to measure to non-compliant behaviour in our
in numeric terms. For example, the taxpayer reported the community, business and tax
reason his claim was incorrect was because he did not practitioner perceptions surveys
understand his obligations. n tone of media comment relating
Quantitative data can tell you how much of something has to the non-compliant behaviour.
happened while qualitative data is often extremely useful in Taxpayers are more Change in the:
explaining why it happened. This may also help you learn compliant with their n percentage of the population that
more about your risk. lodgment obligations lodge their tax returns on time
n expressed in a neutral form – the purpose of an indicator
n percentage of taxpayers that
is to identify change. It should make no assumptions about
lodge their activity statements
the direction the change should take.
by the due date.
For example, express the indicator as a change in the ratio
of work related expenses to salary and wages rather than
as a reduction in the ratio of work related expenses to salary
and wages.
Aligning indicators to success goals

The purpose of evaluation is to understand the extent to which
the success goals and ultimately the desired outcomes have
been achieved. Consequently you must ensure that your
indicators are directly linked to what you are trying to achieve.
Table 1 provides examples of aligning indicators to success
goals. It also shows how the indicators can work together to
help understand the extent to which the success goals have
been achieved.
ALIGNMENT
When identifying potential indicators make sure there is a
clear link between the indicators and your success goals.

VALIDATING YOUR INDICATORS n Measurable

Once you have identified the potential suite of indicators you You need to look at how you are going to measure changes
need to explore each indicator to understand whether it will from one period to the next. Consider the following questions
provide the information that you are expecting. This process when deciding which measurement method to use:
is called validation. – Can you express the indicator in a numeric form?
Validate your indicators by applying a range of tests including – Will you compare the data against some form of
those in the SMART test: benchmark?
n Specific – Is a calculation such as a ratio involved? What are the
n Measurable
components of the calculation?
n Achievable
– Will you measure the whole population of interest or
restrict your analysis to a sample?
n Relevant
n Timed.
This test explores whether your indicators are specific, EXAMPLE

measurable, achievable, relevant and timed.
A change in average rental income.
You should also explore whether:
Calculated by dividing total dollar value of rental income
n your indicators can identify whether a change can be
by total number of taxpayers reporting rental income.
attributed to your strategies
n the data can be compared over time
n the results are able to be substantiated
n your indicators are capable of presenting an unbiased For more information on measurement methods
picture of the extent to which your success goals have and data collection, see pages 12–17.
been achieved.
n Achievable
The smart test Indicators should be both practical and reasonable.
n Specific You need to make sure that the data for each indicator is
Indicators should have a clear, unambiguous description readily available or can be obtained at a reasonable cost.
so that everybody has a common understanding of the
information the indicator is expected to deliver.
EXAMPLE
Express the indicators in a way that will help you to identify
whether compliance behaviour has changed as a result of A change in net tax payable to total income ratio.
your strategies. A simple way to do this is to describe the
indicator as ‘a change or a difference in the rate or ratio or This indicator is based on data that is readily available
number of …’ from our Data Warehouse.
EXAMPLE
A change in the percentage of activity statements lodged

by the due date by sole traders with a turnover of less
than $1 million.
Comparing the compliance behaviour before you

implemented your strategies to the behaviour afterwards
allows you to understand whether there is a change in
behaviour that correlates with your strategies.

n Relevant
Indicators should be able to measure the expected or EXAMPLE
desired changes. They should also have a direct link with
your success goals and desired outcomes. Change in the number of individual income tax returns
lodged compared to the number of people identified in
Don’t just choose an indicator simply because it’s easy to the Australian Bureau of Statistics’ (ABS) Australian labour
measure. Make sure it contributes to the evidence you’re force statistics.
gathering to help you understand the extent to which your
success goals have been achieved. These statistics are generally presented as data which
spans a calendar year rather than an income year. There
is a notable time lag between collection and presentation.
EXAMPLE When comparing this ABS data against ATO data, ensure
the correct months are represented in the data set for
Success goal
the income year rather than calendar year and that they
Individual taxpayers voluntarily comply with their income correspond to the year in which lodgment behaviour is
tax lodgment obligations. being assessed.
Indicator
A change in the percentage of individual returns lodged
by the due date.
Additional validation tests
When validating your indicators you should also make sure
An increase in this indicator will provide evidence of that they are:
improved voluntary compliance with lodgment obligations. n attributable – indicators need to measure something that
your strategies can reasonably be expected to influence.
You also need to understand what the indicator should show You need to think about whether a change in behaviour
in the immediate and longer term. For example, a successful is the result of your strategies or whether that change
strategy may show a significant improvement in the short was influenced by some other factors.
term that levels off in the future – an immediate effect with
a sustained change. Bear in mind that:
n Timed n the behavioural change might have occurred regardless
You need to understand whether your strategies affect of your compliance strategies
voluntary compliance and community confidence over time. n your strategies may only be partially responsible for the
Select indicators that will identify change and show your observed change because of some other influencing factor
progress over the immediate, intermediate and long term. n there may be no obvious change in behaviour because your
Indicators also need to be based on data that can be strategies have halted deterioration or maintained the
produced regularly enough to track progress and quickly status quo.
enough for it to be useful, with only a short time between Where other factors are likely to have an impact on the usability
the period the data covers and when it becomes available. of the indicator, record both the issue and how you expect it to
Consider whether: affect the results.
– there is a time lag before the data becomes available
– the data is available on a regular or infrequent basis Other factors that could cause a change in behaviour include:
n other ATO strategies
– the data is available for all of the relevant time period.
n court decisions on tax matters
There is often a time lag between the collection and availability n changes in tax rates
of external data. You need to carefully examine your data
n economic fluctuations
sources to understand how timing issues will affect the
n globalisation
usability of your indicator.
n natural disasters
n media coverage of tax related issues
n unemployment
n taxpayer literacy and numeracy levels.

For more information about external factors, refer EXAMPLE: Weak and strong indicator
to pages 46–48 of our Literature Review – Measuring Success goal
compliance effectiveness (NAT 71078).
The community displays a reduced tolerance for
participation in the cash economy.
n comparable – continuity is necessary in order to make
comparisons in trend analysis over time. Weak indicator
Data can be subject to continual revision. For example, new
Change in the frequency and tone of media comment
data may be received over time that may have an impact on
relating to the cash economy.
your original benchmark population.
While it is not always possible to avoid differences in the This indicator is weak because it is likely to be too broad
underlying data, recognition of those differences and the to identify any real behavioural change.
influence they have on the indicator analysis should be
documented. Measuring something more specific can often strengthen
the indicator.
n able to be substantiated – the results and conclusions
drawn during the analysis of an indicator should be capable Strong indicator
of being substantiated by an independent authority. In other
words, an independent reviewer with relevant qualifications Change in responses to specific questions in our surveys
should be able to come to the same conclusions. (including community, business and tax practitioner surveys).
– Unbiased – the information used to measure effectiveness
should be able to be impartially collected, analysed and
evaluated. Accepting or rejecting indicators
For example, when data is reported can result in a bias. The validation process enables you to make informed decisions
Taxpayers who are due a refund tend to lodge earlier than as to whether an indicator should be:
those who will end up having a tax debt. The timing of the n accepted – if you plan to accept an indicator that has
collection and analysis may produce different results. validation issues consider the impact these flaws will have on
the strength of the indicator. Also consider the impact of these
In the spirit of continuous improvement, it is essential to not only
flaws on its value in the overall suite of indicators
embrace our successes but to learn from our experiences. The
n rejected – if you plan to reject an indicator consider the
results must present a true picture of the extent to which we
have achieved our success goals. impact this will have on the usefulness of the overall suite
of indicators. If its absence would have a detrimental effect
then consider whether there are any alternatives to fill the gap
Understanding strength n retained for future use – generally, indicators that don’t meet
Indicators are not used in isolation to understand the extent all of the validation requirements should be discarded. But if
to which a success goal has been achieved. When considered the flaw is only temporary, such as where a time-based
together as part of a suite, they should provide a defensible indicator hasn’t been in existence long enough to be able
picture of the outcomes. to track progress, then it should be retained for consideration
Weak indicators often perform a useful function in the overall in any future evaluations.
suite and should not be discounted on their strength alone.
However, too many weak indicators may make it difficult to Painting a defensible picture
present a defensible picture. You can attempt to overcome After you have completed the validation process you need
this by exploring ways to strengthen or replace some of to determine whether your suite of indicators will provide
the indicators. a defensible picture.
Ask yourself whether, overall, the suite of indicators is likely
to provide:
n sufficient evidence to justify your conclusions regarding
the extent of your effectiveness
n a picture that is strong enough to withstand scrutiny
and challenge.

Consider whether your indicators will show that: observation, there is no way to understand whether this change
n there has been a change in behaviour of the risk population is caused by your strategies or whether it simply represents
n there has been a change in community confidence natural fluctuations in the data being observed.
n the change has been sustained Observing historical trends in the data before your strategies
n your strategies have contributed to the change are implemented will help you to identify whether a change
n the effect can be seen more broadly than those targeted has occurred.
by your strategies Figure 2 looks at on-time lodgment behaviour over a three-year
n there were any unintended consequences as a result of period. Note the general direction evident in the data.
your strategies.
If not you should go back to the beginning of the indicator FIGURE 2: Understanding historical behaviour
identification process and identify more potential indicators.
Remember to validate any new indicators. 100
Seeking endorsement
% of on time lodgments
Once you have finalised your suite of indicators you should 98

seek endorsement from the relevant decision making body.
Approval should generally be obtained from your business 2
Baseline
96
line’s RMC or equivalent.
Trend line
ENDORSEMENT 94
The validated suite of indicators should be endorsed by

your RMC (or equivalent) before continuing further.
92
2004-05
5
0
–0
–0
–0
–0
–0
–1
04
05
06
07
08
09
20
20
20
20
20
20
MEASURING YOUR INDICATORS
Once you have decided on your indicators you then need to Compliance behaviour
determine how you will measure them. What method will you
use to identify whether: Identifying trends
n a change in behaviour or community confidence has occurred A trend is the general direction in which something tends to
n the change can be attributed to your strategies. move.
Trend analysis involves the collection and presentation
100
Identifying a change of information in order to identify patterns.
Before you can understand whether there has been a change
in the behaviour or community confidence, you need to know Trend data is usually presented as a set of data points
measured over uniformly-spaced, successive time periods.
what the compliance behaviour or community confidence 98

looked like before your strategies were implemented. A trend will be evident where the data shows a general
3 tendency to move in the same direction over time. By drawing
Baseline a line96through the historical data to highlight the trend and
A baseline is a reference point against which other things can extending that line over future observation points, you get a
be compared or measured. It is used to identify whether there sense of where the subsequent observations should be if the
has been a change over time. trend94is maintained and there is no change.
A baseline measurement should reflect the starting point prior A trend line does not necessarily need to be straight.
to the strategies being implemented. It should then be used for Sometimes exponential growth or decline may be observed.
comparison purposes to determine whether there has been
a change over time. Figure922004-05
3 shows a difference between the trend line and the
5
new data point. This indicates that something has affected

6
0
–0
–0
–0
–0
–0
–1
04
Comparing against a single data point, however, is not always

05
06
07
08
09
the behaviour.
20
20
20
20
20
20
a reliable method of identifying a change. While you may see

Compliance behaviour
a difference between the original measurement and a later
100
98
98

7
2 96
96
94
94
92
2004-05 92
05
2004-05
06
07
08
09
10
–
06
07
08
09
10
11
05
–
–
04
05
06
07
08
09
–
–
20
05
06
07
08
09
10
04
20
20
20
20
20
20
20
20
20
20
20
20
20
Compliance behaviour Comparison
Analysing trends
FIGURE 3: Using a trend line to understand Several methods can be used to understand whether there
normal patterns are changes in trends. These include:
n pre and post studies
100
Pre and post studies assess changes as a result of an
100
intervention. Measures are taken both before and after the
intervention in an effort to understand whether there has been
98 a change.
98
For example, tracking behaviour over time, both before and
3 Baseline Difference after the compliance strategies are implemented, can help
96
you to understand whether there has been a change in
behaviour 96
8 that corresponds with the implementation of the
strategies. But it will not tell you whether the strategies
94 Trend line caused that change.
94
n longitudinal studies
A longitudinal study involves repeated observations of a
92
2004-05 specific set of participants over a period of time. It allows you
5
92
to measure key variables
2004-05 at different points in time. The same
6
0
–0
–0
–0
–0
–0
–1
04
05
06
07
08
09
data is collected from the same participants at regular

20
1
5
20
20
20
20
20
–0
–0
–0
–0
–1
–1
–0
–
05
06
07
08
09
10
11
04
Compliance behaviour intervals. Longitudinal studies are particularly useful for 20
20
20
20
20
20
20
20
understanding the long-term effects behaviour

Compliance of your strategies. Comparison st
To understand historical trends, the more observations you When choosing your sample for a longitudinal study you must
have, the easier it will be to identify patterns. A longer time consider the effect of churn on the sample size. Churn is the
frame can often help to identify less obvious patterns such number of people who move in and out of a particular group
as those
100 presented in seasonal data. at any given time. If your population has a high churn rate you
In the tax world, collecting reliable data over a longer period is are likely to lose a significant number of participants along the
problematic because it introduces a range of other variables way. Increasing your sample size to allow for natural attrition
98
that can affect the observations particularly when dealing with of the participants over time is one way to overcome this.
annual tax data. n measuring against100
standards
These variables include: A standard is a reference point against which compliance
96 behaviour9or community confidence can be measured. It sets
n changes in the way the data is collected
4 98
a required level of quality or performance that indicates quality
n differences in the timing of the intervals between data
or success.
collection
94
n changes in the law For example, measuring behaviour against a standard
96
n changes in the way we administer the law allows you to see whether behaviour is coming back into
n economic turbulence
line with acceptable standards (see figure 4).
92
n societal changes.
2004-05
94
5
0
–0
–0
–0
–0
–0
–1
04
05
06
07
08
09
While it is preferable to have enough observations to make

20
20
20
20
20
20
the claim of a trend valid andbehaviour

Compliance reliable, having less does Standard
not rule
out the indicator as a useful contributor to the overall suite of 92
2004-05
indicators. It can still add valuable insight and support for
6
1
5
–0
–0
–0
–0
–1
–1
–0
other indicators.
05
06
07
08
09
10
04
20
20
20
20
20
20
20
Compliance behaviour Compariso

100
98
5
96

94
98
98
3 100
96
96
8
98
94
94
96
92 4
2004-05
05
92
06
07
08
09
10
–
2004-05
–
–
04
05
06
07
08
09
20
06
07
08
09
10
11
05
20
20
20
20
20
94
–
–
05
06
07
08
09
10
11
04
20
20
20
20
20
20
20
20
Compliance behaviour Comparison st
Figure
92 5 shows that both the benchmark and the population of
FIGURE 4: Comparing against a standard 2004-05
interest were tracking together until the start of the treatment
05
06
07
08
09
10
–
period. The measure taken at the end of the treatment period
–
04
05
06
07
08
09
20
20
20
20
20
20
100 shows a distinct difference in the behaviour pattern of the
two groups. Compliance behaviour Standard
98
FIGURE 5: Comparing against a benchmark
100
Treatment 100
96 period
4 9
98
98
94
5
96
Treatment
96 period
92
2004-05
94
5
0
–0
–0
–0
–0
–0
–1
04
05
06
07
08
09
20
20
20
20
20
20
94
Compliance behaviour Standard
92
2004-05
Attribution
6
08
1
5
–0
–0
–0
–1
–1
–0
92
9 7–
05
06
08
09
10
04
Identifying a change in compliance behaviour or community
–0 00
2004-05
20
820
20
20
20
20
2
5
0
–0
confidence does not necessarily mean that your compliance

–0
–0
–0
–1
Compliance behaviour Compariso
04
05
06
07
08
09
20
20
20
20
20
20
strategies
100 caused that change.
Compliance behaviour Benchmark
Using a comparison technique enables you to separate the
effects of your strategies from other factors that could be Benchmarks are often based on external data, such as
causing98the change. This will help you to identify whether the ABS statistics, and are comparable across the broader
5 observed change is a result of something you have done or population base.
a reflection of changes in the environment that are outside
your control.
96
Several different types of comparisons can be used. Some EXAMPLE: External benchmark
are suitable for high level measurements (for example,
Australian resident population data from the ABS could be
benchmarks)
94 while others are more suitable for drilling down
a useful benchmark to understand changes in the number
to a sub-population level (for example, target and control
of resident individuals who are registered in the tax system.
group studies).
92
Benchmarks
2004-05 Benchmarks are particularly useful when examining behavioural
5
0
–0
A benchmark is a reference point for comparison purposes. It change at the broader level but may become less useful when
–0
–0
–0
–0
–1
04
05
06
07
08
09
20
allows you to isolate the effects of the strategies from the effects drilling down to a specific sub-population.
20
20
20
20
20
of external influences such as economic

Compliance behaviour fluctuations. Benchmark
Inferential statistics
For example, comparing the behaviour of a risk population to an Inferential statistics are a group of methodologies that allow you
appropriate benchmark is useful in understanding whether the to draw inferences about a population based on the behaviour
observed behaviour is in line with expectations. If your strategies of a sample.
have had an effect, you would expect to see a change in the
relationship between the benchmark and the compliance When an appropriate benchmark is not available you can use
behaviour of the risk population. inferential statistics to identify whether or not there is a causal
relationship between the behaviour being observed and your
strategies.
They use sampling methodologies to identify target and control
groups from within a specified population (see figure 6).

n Population – A group of taxpayers who share a common

set of characteristics. EXAMPLE: Randomised control trial
n Target group – A sub-group of taxpayers selected from the
Taxpayers are randomly selected from the population
population and used as the focal point for your strategies.
and separated into a target group and a control group.
n Control group – A sub-group of taxpayers selected from
the population that is isolated from your strategies for the The target group is used to test the effectiveness of
purposes of the study. an advisory letter on their compliance behaviour.
Figure 6 illustrates the relationship between the population, The control group is isolated from the treatment.
the target group and the control group. The difference in behaviour between the groups is then
used as an indicator of the effectiveness of the strategy.
FIGURE 6: Target and control groups within
a specified population Theoretically, random selection should result in the selection
of samples that are representative of the population. However,
POPULATION where a population has identifiable sub-groups that are likely
to behave differently, the accuracy of the study can be further
enhanced by randomly selecting cases from each sub-group.
This eliminates the possibility of a sub-group being over-
represented within the sample.
TARGET GROUP CONTROL GROUP You should seek expert assistance when setting up these types
of comparisons to ensure that you don’t build unintended flaws
into your indicator analysis.
Treated Not treated You should consider:
n appropriate sample sizes
Having too small a sample will limit the confidence that
Comparing target and control groups from the same population can be attached to the results.
allows you to screen out the effects of unknown variables. n proper selection techniques for group membership.
Theoretically both groups will be subject to the same
environmental factors except for the compliance strategies. Any When using a randomised control trial you need to select your
differences in behaviour can be attributed to your strategies. target and control groups before you implement your strategies
to ensure that the control group can be isolated from the
Methods for comparing target and control groups include: treatment.
n randomised control trials
n matched groups
Both the target and the control groups are randomly selected
from within the population that is to be targeted by the When it isn’t possible to use a randomised control trial you can
strategies. The control group is then isolated from treatment. use a matched group instead. This is a comparison group that
is similar to the treatment group. Groups are matched according
As the target group has been subjected to the treatment to similarities in their characteristics.
and the control group has not, any observed behavioural
differences between the two groups can be attributed to While the results from a matched study are adversely affected
the compliance strategies. by unknown differences between the groups, well-matched
groups can approach the rigor of a randomised control trial.
Seeking expert assistance

Assistance in understanding the broader environment or
identifying appropriate external benchmarks can be obtained
through internal specialist areas.

Another useful resource is the ABS/ATO Information There is often a cost associated with the collection of data.
Management team located in the Office of the Chief Knowledge Potential costs to the ATO can range from payments to an
Officer. This unit can provide assistance in identifying, obtaining external organisation to the opportunity cost of diverting internal
and interpreting information from the ABS. resources to the task of collecting the data. Potential costs to
the community include an additional burden on the time and
Assistance in setting up your comparison studies can be
resources of respondents.
obtained from the relevant business and data analysis
specialists within your business line. Before making a final decision on the data source consider
the following:
IDENTIFYING YOUR DATA REQUIREMENTS n Are there any existing data sources that will suit your needs?
Collecting new data is generally more expensive and time
Who’s responsible? consuming than collecting existing data. There may be
You should allocate ownership of the data gathering and existing data sources such as previous evaluation reports,
indicator analysis tasks. Making your expectations clear from other agency data or other internal and external data that
the outset will allow you to identify and deal with potential issues will suit your needs.
that may impact the timely availability of the information. There is a range of existing surveys and studies that may also
suit your purpose. But you must understand the purpose of
What data sources will you use? the original study and any limitations it might have in relation
You must be as specific as possible so that people will know to your particular indicator.
exactly what data is required. You may be able to influence the questions that are included
Consider the following questions when identifying your in a survey when it is being set up. This can reduce the costs
data sources: involved with running a separate study.
n Is the data from an internal or external source?
Our Corporate Research Centre can provide advice in relation
to surveys and other research.
Internal data is data obtained from sources such as the Data
Warehouse or our community perceptions surveys. External n Is the data in a form that is able to be analysed for your
data is data obtained from sources outside of the Tax Office, particular purpose?
such as the ABS. Information which is collected at source (primary data) for
n Are there any issues around ownership of the data that will the particular purpose of meeting the evaluation objective
affect your ability to acquire it? is more reliable than information derived from a pre-existing,
For example, data from state-owned property databases may data source. This is because the primary data has been
require special permission before you can access it. You must collected with the particular purpose in mind.
allow sufficient lead time for that permission to be obtained. Secondary data has usually been collected, summarised or
n Are there any issues with regard to the availability of the data interpreted for a different purpose and may not be entirely
that would affect its usability? relevant when used as a substitute for primary data.
For example, there is a significant lag time between the This may weaken the reliability of your indicator.
collection and subsequent availability of ABS census data
that restrict its usefulness as a benchmark. Before you decide to use a secondary data source you
n When and how often do you need to collect the data? must understand:
n the purpose for which the data was originally collected
You need to obtain data at regular intervals so that progress
n how the data was collected
can be tracked over time.
n the currency of the data
n Can the data be replicated? n whether the data is representative of your population
The data may need to be replicated for various reasons of interest.
including:
– the need to provide the same information at specific
intervals for time-based analysis For more information about the major methods
– quality assurance processes to independently verify and sources used for collecting data for evaluations,
your results. see Appendix 1.

Are there any data limitations?

You need to understand whether there are any limitations or
issues that will impact on the reliability of the underlying data.
These limitations include:
n changes over time in the method of data collection
or compilation
n unavailability of data in some years
n possible time lags in the availability of internal and
external data.
What comparisons will you use?

You need to be clear about which comparisons you intend to
use to make sense of the changes for each indicator. In making
a decision consider whether you:
n know how you’re going to measure your baseline
n will use a benchmark and whether it will be based on internal
or external information
n need to select a control and target group
n need to identify a similar group for comparison purposes.
Once you have decided which comparison technique to use,

you should identify what needs to be put in place to collect
the data.
Documenting your intentions

You must record information relating to the underlying data
sources, data limitations and comparisons associated with
each indicator. This provides a sound basis for quality assurance
processes and allows for independent verification or replication
at a later date.

EVALUATING
02
EFFECTIVENESS
BEFORE PROGRESSING FURTHER,

MAKE SURE YOU HAVE:
n validated your indicators
n formulated your data requirements and assigned
responsibility for collection and analysis of
your indicators
n a suite of indicators that can paint a defensible
picture of effectiveness
n received endorsement of your suite of indicators
from your RMC (or equivalent).
02 EVALUATING EFFECTIVENESS
Data collection and analysis must stay focused on the original Quality assurance
intent for which the indicators were developed. Otherwise you Your data collection techniques must meet appropriate quality
will end up ‘data rich and insight poor’. assurance standards. You need to address and document:
n any limitations of the data source (for example, is the survey
GATHERING YOUR DATA poorly designed, are there sampling errors and have these
Once sufficient time has elapsed for your strategies to have an been articulated?)
effect on voluntary compliance and/or community confidence n any assumptions made
you need to begin gathering the data to enable you to analyse n the approach taken and whether it is appropriate
each indicator. (for example, is the data coding approach correct?).
Who’s involved? Data storage

The person or area responsible for gathering data and analysing Once collected, data needs to be appropriately stored. Storage
each indicator should have been identified during the indicator requirements vary, depending on the type of data. The key
identification and validation processes. It is highly likely that elements to consider are:
different people will be responsible for each indicator due to n security of the data, particularly where it is classified as
the specialised knowledge required to understand and interpret in-confidence or above
the data. n ease of data retrieval.
You also need to store the detailed data specifications in a way

Data integrity that will allow the data to be reproduced at appropriate intervals
When collecting your data, ensure that there is a sound process
for cyclical evaluations, ongoing trend analysis and quality
in place to verify and validate it. You should consider:
assurance processes.
n relevance of the data – the data should actually measure
the characteristic it claims to measure. This is particularly
important if you’re using a data set that has been created ANALYSING YOUR INDICATORS
for another purpose. If using secondary data, you need to Once the data for each indicator has been collected you then
rigorously review your data specifications to ensure that the need to analyse it to understand where there is any evidence
data is appropriate for your purpose. of a change in behaviour and/or community confidence.
n data availability and collection – ideally, data should be The results for each indicator are brought together to determine
collected in a form that is tailored to your analysis whether, as a whole, they provide defensible evidence of the
requirements. This won’t always be possible particularly success of your strategies in meeting your success goals and
where secondary data is used. If using secondary data you ultimately your desired outcomes.
need to explore the underlying data properties to understand
any limitations they may pose on your analysis.
Who’s involved?
n consistency of data – you must look for data that will be
Indicator analysis should be done by business or data analysts
consistent over time. This allows for repeated observations to who understand both the business and the underlying data. In
check whether the effects of your strategies are sustainable. many cases the analysts will need to work closely with other
If this isn’t possible, look for ways to mitigate the limitations subject matter experts in order to deliver a comprehensive
of the data in the longer term. analysis. These experts may include risk and strategy experts,
n timing of data – where data is sourced from external and economists and revenue analysts.
databases or reflects external market trends, you must make
sure there are no issues with differing time frames when
making comparisons. If there are any differences you must Making sense of the data
account for the time lag in your analysis. To evaluate your effectiveness you need to firstly explore
whether there is a change in compliance behaviour and
n expertise of data collectors – data quality can be impacted
secondly, whether your strategies caused that change.
by the skills and expertise of the people involved in gathering
the information. You may need to seek the advice of experts
to ensure your data collection techniques are appropriate.

Is there a change? Comparison data can be generated through the use of

An easy way to identify a change is to use a line graph to benchmarks and inferential statistical techniques such
explore patterns and changes. as randomised control trials or matched groups.
To do this you need to: Is the change in line with your expectations?
n plot your compliance data on a line graph, preferably using Sometimes your strategies can have a different effect than
one that allows you to plot more than one data series. This originally anticipated. Where you see a change that is
will make it easier to compare your data with, for example, unexpected you must explore it further.
a benchmark
If potential unintended consequences have been anticipated
n identify the point in time when your strategies were
from the outset there may be indicators specifically designed
implemented
to see whether these consequences have emerged.
n look at the historic observations before your strategies were
implemented and identify any obvious trends Be careful not to view the emergence of an unintended
n draw a line that depicts that trend and extend this line into consequence as necessarily negative. Some may, in fact, be
the future positive and even if negative the insights they provide are
n examine any observations taken after your strategies were
valuable in re-shaping our compliance improvement strategies.
implemented and compare them with the previous trend. If
there is a difference you will be able to conclude that there Identifying sustained change
is a change. A sustained change is one that lasts for the longer term.
Identifying whether a behavioural change has been sustained
needs to be done with repeated measures over time.
For more information on baselines and trends, see
pages 12–13. Figure 7 shows a change in the relationship between the
compliance behaviour and the comparison statistic that
corresponds with the treatment period. The 2008–09
However, where the historical data shows considerable observation indicates that the new relationship is holding.
fluctuation it may be difficult in the short term to identify whether
a change has occurred. You may need to continue monitoring
over time to understand whether there is a difference. FIGURE 7: Identifying a sustainable change in behaviour
Did the strategies cause the change? 100

Once you have identified a change you need to see if that
change can be attributed to your strategies or whether it is a
reflection of changes in the environment. Comparing your data
98
to something else, such as a benchmark or control group, will
help you to isolate the effects of your strategies from the general
fluctuations caused by other changes in the environment. 7 Treatment
96 period
To do this you need to:
n look at the comparison method identified earlier
n plot your comparison data on the graph 94
n compare the pattern of movements in both the comparison
data and your compliance data to see if there are any
differences. 92
4-05
05
2004-05
6
If the data shows a difference in the pattern of movement after

–0
–0
–0
–0
–1
4–
2
5
–0
–0
–0
–0
–1
–1
–1
–0
05
06
07
08
09
00
05
06
07
08
09
10
11
04
the strategies were implemented, you can say with some

20
20
20
20
20
20
20
20
20
20
20
20
20
confidence that your strategies

Compliance behaviourhad an effect on compliance
Compliance behaviour Comparison statistic
behaviour or community confidence. If, however, your
compliance data displays the same pattern as the comparison
statistic, you can conclude that the observed changes in If the behavioural change is sustained, you would expect to see
behaviour were caused by environmental factors. the paths of the two groups continuing to track in parallel over
an extended period of time as seen in figure 8.
100
lodgments
20 98 MEASURING COMPLIANCE EFFECTIVENESS

% of on time lodg
7
100 96
98 94
7
96 92
2004-05 06
07
08
09
10
11
12
05
–
–
05
06
07
08
09
10
11
04
20
20
20
20
20
20
20
20
94
Compliance behaviour Comparison statistic
92
2004-05
06
07
08
09
10
11
12
05
There is no set period of time over which you should track

–
–
–
05
06
07
08
09
10
11
04
FIGURE 8: Sustained change in behaviour your data to identify whether a change is sustainable. However,
20
20
20
20
20
20
20
20
Compliance behaviour Comparison statistic you will need to track behaviour for several years to really
100 understand whether your strategies have had a lasting impact.
DETERMINING YOUR EFFECTIVENESS

98
While the analysis of each indicator provides some useful
insights, it won’t provide you with a defensible picture of the
Treatment extent to which your strategies have improved voluntary
100 96 period compliance and/or community confidence. You need to bring
8
together the analysis results for all of the indicators and examine
them as a whole to determine the extent of your effectiveness
98 94 and draw defensible conclusions.
Who should be involved?

96 92 Evaluating compliance effectiveness should be done by people
2004-05 who understand both the risk and the environment affecting that
6
2
5
–0
–0
–0
–0
–1
–1
–1
–0
risk. While the risk manager will generally take responsibility for
05
06
07
08
09
10
11
04
20
20
20
20
20
20
20
20
94
making sense of the story, it’s good practice to involve other
Compliance behaviour Comparison statistic subject matter experts to help make sense of the information
as a whole.
Tracking the behaviour for a short period after the intervention As for the indicator analysis, the people that should be involved
92
will not always give a reliable indication of whether the effect is
2004-05 in bringing together a defensible story should include the
6
2
5
sustainable over time. As figure 9 illustrates, while the behaviour

–0
–0
–0
–0
–1
–1
–1
–0
relevant risk and strategy experts as well as analysts who have

05
06
07
08
09
10
11
04
is tracking in parallel with the comparison statistic in the

20
20
20
20
20
20
20
20
an understanding of the environmental variables that can affect

2008–09 income year,
Compliance the slight decline inComparison
behaviour compliancestatistic your risk area.
behaviour continued. The declining behaviour was originally
masked100 by changes in the environment. Looking at the pattern
of observations over a longer period clearly shows that the Bringing the story together
effect of the strategies was short-lived and not sustained. When the suite of indicators was originally chosen, a number
of assumptions were made about what you’d expect to see if
9
98
your strategies were effective. Now that you’re ready to bring
FIGURE 9: Unsustained change in behaviour the results together, you should revisit these early assumptions
to see whether the indicator results are as expected.
100 96
If the behaviour is in line with your earlier expectations then
determining the extent of your effectiveness should be relatively
easy. If any indicators show a different result, however, you must
9
98 94
objectively investigate the result to understand how it impacts
on your conclusions about the extent of your effectiveness.
Treatment
96 92 period Where an indicator shows an unexpected result explore
2004-05 whether:
6
2
5
–0
–0
–0
–0
–1
–1
–1
n the result is significant in relation to the overall suite of

–0
05
06
07
08
09
10
11
04
20
20
20
20
20
20
20
20
94
indicators – if the indicator is weak its impact on the overall
Compliance behaviour Comparison statistic outcome may not be significant. If it carries a lot of weight
within the suite you need to understand how it affects the
outcome.
92
2004-05 n any other indicators show similar results – if the result is
supported by a range of other indicators, then it would be
6
2
5
–0
–0
–0
–0
–1
–1
–1
–0
05
06
07
08
09
10
11
04
reasonable to conclude that the strategies have not been

20
20
20
20
20
20
20
20
Compliance behaviour Comparison statistic effective.

n itis an unintended consequence – any unintended Table 2 looks at evaluating the extent to which you have
consequences must be explored in order to understand achieved a success goal. It shows the linkage between the
their impact on the risk and for planning future indicators and the success goal. A good way to evaluate a
compliance strategies. success goal is to turn the goal into a question. In this case,
n further analysis is necessary – you may need to do further the question would be: To what extent does the community
analysis to understand the reason for the result. Consider demonstrate confidence in our ability to deal with
issues such as a flaw in the data that was not previously non-compliance relating to [risk]?
identified or a defect in the original thinking. In this example, the overall conclusion is supported by
the evidence.
Drawing your conclusions
Evaluating compliance effectiveness is based on understanding
whether your compliance strategies have moved you towards
achieving your success goals and ultimately your desired
outcomes.
Look at the indicators aligned to each success goal and decide
whether, collectively, they show the extent to which you have
achieved each goal. If your evaluation shows that you have met
your success goals then you can, with a reasonable degree of
confidence, conclude that you have made progress towards
achieving your desired outcomes.
TABLE 2: Evaluating the extent to which a success goal has been achieved
Success goals Indicators Evaluation outcome

The community is n Trend in the number Conclusion
confident that the of reports made by
While survey results have remained stable over the period 2006–08,
ATO is able to deal the community to the
the increased levels of engagement by the community indicate an
with non-compliance Tax Evasion Referral
improvement in the community’s confidence in our ability to deal with
relating to Centre (TERC).
non-compliance relating to this risk.
participants in n Responses to our
the risk surveys including Summary of evidence
community, business n There has been a 75% increase in the number of reports made by
and tax practitioner the community to the TERC in the 2007–08 income year compared
perceptions surveys. to the 2005–06 income year.
n Levels of engagement
n Perceptions survey results since July 2005 suggest that the community
of representative bodies. is confident in our ability to deal with participants in the risk.
n Frequency and tone of
n There has been an increased willingness by representative associations
media comment. to work with us to design, develop and promote fact sheets to their
members and others which is indicative of an improvement in the levels
of engagement.
n Media commentary, specialist advice and letters to the editor in the past
12 months have been supportive of our approach. This contrasts with
negative commentary in relation to several high profile cases in the
previous period of review.

Making recommendations Remember, evaluating your effectiveness is about presenting

The lessons learned from evaluating effectiveness can help a defensible story. It is not about providing a definitive
re-shape our compliance improvement strategies. measurement. Knowing where to draw the line between
definitive and defensible is quite difficult because your level
Consider the following questions when making your of comfort in drawing conclusions about effectiveness is
recommendations: intrinsically linked to your personal risk stance.
n Was the right mix of compliance strategies used to address
the risk? If not, what mix would be better? When deciding on the level of information you need to support
n How could your strategies be improved?
your story of effectiveness, think about how comfortable you
would be if you were asked to justify your conclusions. Involving
n Does more risk assessment work need to be done
others with relevant business knowledge in the evaluation can
(for example, more research into a particular population’s
go a long way to increasing your comfort levels.
compliance behaviour)?
n Did any unintended consequences emerge? How might
any negative outcomes be mitigated? Quality assurance
n How can the application of the compliance effectiveness
Involving the right people throughout the process will help
methodology be improved (for example, better articulation you to increase the quality of your conclusions.
of success goals, data collection methods or analysis As a final step, however, you should have your claims
techniques)? independently verified to ensure that your conclusions are
soundly based.
Presenting a quality outcome
A quality evaluation of whether, and to what extent, your
strategies have achieved your success goals and ultimately your
desired outcomes can provide you with a wealth of knowledge
on which to base future risk treatment decisions.
To ensure that you can present a quality evaluation consider:
n the analysis methodology – choosing the right methodology
is particularly important when looking at attribution issues. You
need to select a methodology that can identify whether
changes are caused by your strategies or by factors outside
your control.
n any over-emphasis on one or more indicators – while it is
acceptable for some indicators to carry more weight in the
overall picture than others, you must take care not to rely too
much on particular indicators simply because they tell the
story that you want them to. Remember, the value you get
from evaluating compliance effectiveness is not about
success or failure but rather from the lessons learned and
how they feed into the continuous improvement process.
n personal bias – you need to resist the temptation to
manipulate your evaluation outcomes to paint only a positive
picture. Evaluating effectiveness is not meant to be a
statement of success or failure. It is meant to provide an
understanding of whether you are on the right track with your
strategies and to provide some guidance for future planning.
Smoothing over any negative results can mask valuable
lessons that could influence future risk treatment plans.
n defensible versus definitive – understanding what the
indicators are really telling you can be difficult due to the
subjectivity inherent in arriving at a conclusion.

REPORTING ON
03
EFFECTIVENESS
03 REPORTING ON EFFECTIVENESS
Evaluating compliance effectiveness presents an opportunity for THE EXECUTIVE SUMMARY

continuous improvement based on an improved understanding Your report should be presented in a way that makes the
of the risk. Presenting a report is the main means of information easy to understand. The best way to do this is
communicating the outcome of your evaluation. It is usually to present an executive summary supported by a more
prepared by the risk manager in consultation with other subject detailed report.
matter experts.
The executive summary should provide the reader with an
understanding of your conclusions along with a brief outline
When to report of the evidence underlying those conclusions. It should also
The regularity of the reporting cycle can be affected by a range provide recommendations for the future. Your executive
of issues including: summary should usually be no longer than two pages.
n the planning cycle – you need to identify when the report
will be needed for decision-making purposes. Understanding
What should you include in the summary?
the planning and reporting cycles will help you to determine
You should include responses to the following questions:
when the report is likely to be required.
n Have you been effective?
n the evaluation cycle – the priority, pattern and severity of
the risk may also require an evaluation to be undertaken Include a brief summary outlining your conclusions about
on a more regular basis. the extent to which you achieved the desired outcomes.
n Significant external factors such as volatility in the market You should also discuss whether there have been any ripple
place or the economic environment may also impact on the effects and whether the results have been sustained.
timing of an evaluation.
n How do you know?
n other business reporting cycles – compliance effectiveness
Provide an outline of the evidence that supports your
may need to be included in other reporting requirements at
conclusions. Include brief details of any data limitations,
the business line and corporate levels.
assumptions and inferences that are relevant to your
You should consult your RMC (or equivalent) to identify the conclusions. You should also include any relevant contextual
most appropriate time to produce and present your report. information, such as the effect that other factors may have
had on the behaviour or the results, which will help the reader
How to present your report make sense of the story.
You need to prepare a detailed report for reference purposes. Don’t at this point delve into the detail. Remember that the
You also need to prepare a summary document that presents executive summary is meant to provide a high level view of
the key points in a clear and concise form. the outcomes. A more detailed report can be made available
The actual format that you use to present your report may differ for further reference.
according to your business line’s reporting requirements and n What have you learned?
those of the forum to which it is being presented. The content, Include what you learned during the evaluation including:
however, will be the same regardless of the intended audience – improvements in your understanding of the risk
or the required format.
– what treatments did or didn’t work
– any unintended consequences.
n Where to from here?
Include any recommendations for future risk treatment
strategies. Consider:
– the mix of strategies. For example, a greater emphasis
on help and education rather than active compliance.
– future risk assessment work. For example, there may be
a need for more research to increase your understanding
of the risk population.
– the way the compliance effectiveness methodology is
applied in the next evaluation cycle. For example, there may
be a need to refine the desired outcomes, success goals or
indicators if they were unrealistic, unsuitable or vague.
– the evaluation process. For example, it may be necessary
to set up systems to capture more useful data.

THE DETAILED REPORT Presenting your report

You need to prepare a detailed report of the evidence that Before submitting your report to your RMC you should submit
supports your conclusions. It should include: both your detailed report and executive summary to your risk
n the analysis results for each indicator focusing on the specific owner for endorsement.
behavioural shifts that occurred and whether these changes Once that has been done you should submit your report
were evident in the wider population or restricted to the through the normal channels.
specific target group
n a discussion of the evidence that supports your conclusion
that the change in behaviour and/or community confidence
can be attributed to your compliance strategies
n whether the suite of indicators demonstrates the extent to
which you’ve achieved each of your success goals and
ultimately your desired outcomes.
Where you’ve identified a positive shift in compliance behaviour
or community confidence you should discuss whether:
n it is possible, at this point, to identify a sustained change.
It may be too soon to determine whether a behavioural
change can be sustained over the longer term.
n any unintended consequences have emerged. You should
include suggestions for mitigating any negative outcomes
in the future.
In your recommendations include any findings about:
n which compliance strategies worked and which didn’t
and any suggestions for improvements in the future
n whether the right mix of strategies was used to address
the risk
n whether the risk is still manifesting in the same way
or whether it has changed
n whether more risk assessment work is necessary
to understand the risk.

APPENDIXES
04
04 APPENDIXES
APPENDIX 1
DATA COLLECTION METHODS AND SOURCES
Method or source Description Advantages Disadvantages
ATO data Data from tax returns, activity Readily accessible Misunderstandings sometimes
warehouse statements and other forms occur between analysts and
Up-to-date data (refreshed
lodged by taxpayers and data extractors. Data
Provides on a weekly basis)
intermediaries. specifications must be
quantitative data
discussed in detail.
Corporate surveys Data collected from a sample n Offers anonymity n Might not get carefully
of the population. n Relatively inexpensive considered feedback
Provides both
qualitative and Market research surveys are to administer n Wording can bias clients’
quantitative data generally made up of a range n Easy to compare over time responses
of questions aimed at gauging provided questions remain n Not the full story – breadth
See e-library, the same or similar rather than depth
public opinion.
especially ATO n Can cover many people n Danger of ‘over-surveying’
research which ATO corporate surveys typically certain segments (e.g.
n Able to collect large amounts
includes links to cover a range of issues, tax practitioners)
of data
corporate research including awareness and
n Many sample questionnaires n Need a certain level of
and surveys understanding of tax
already exist expertise for reliable sampling,
obligations, perceptions of tax
n Can be tailored to specific questionnaire design, data
fairness and attitudes towards
issues at the risk, product collection and analysis. This
the ATO and tax compliance.
and market levels as well limits the opportunity for
These surveys include our as the corporate level. in-house surveys and puts
Business Perceptions Survey, the onus on commissioning
Community Perceptions experts
Survey, Professionalism n Sampling and non-sampling
Survey and a range of tax errors
practitioner surveys. n The sampling error is
Surveys are also commissioned, associated with inferring

or run in-house, on specific the whole population
topics such as influences on characteristics from a
taxpayer claiming behaviour, small sample
awareness and satisfaction n The non-sampling error
with specific products and includes non-response bias
understanding of specific and biases introduced by the
tax obligations. framing or order of questions.
Interviews Used when you want to fully n Obtains the full range and n Often time-consuming
understand someone’s depth of information n Difficult to analyse and
Provides qualitative
impressions or experiences, n Develops a relationship compare
data
or to learn more about their with the client n Potentially costly
answers to questionnaires. n Flexibility with the client n Interviewer may bias clients’
n Data is collected with a responses which may make
specific purpose in mind it difficult to legitimately
which makes the data more compare results from
consistent with the objective. different interviewers
n Generally can’t provide
statistically reliable samples.

04 APPENDIXES

Focus groups Used to explore a topic n An efficient way to obtain n Problematic to analyse
(for example, reactions to good range and depth of responses
Provides
an experience or suggestion, information in a short time n Need a good facilitator
qualitative data
understanding common n Data is collected with a for safety and closure
complaints) in depth through specific purpose in mind n Difficult to schedule a group
group discussion. which makes the data more of 6–8 people together at
consistent with the objective the same time
n Valuable for exploring new n Unrepresentative of wider
areas and developing ideas population
and hypotheses. Often used n Tendency of participants
as a preliminary data to treat the experiment as
gathering exercise to help a game rather than a
inform the design of a more real-life situation
comprehensive survey.
n Generally can’t provide
statistically reliable samples
Case studies Provide an in-depth examination n Provides a deeper n Usually time consuming to
of a single instance or event. understanding of the collect, organise and describe
Provides
behaviour of a particular kind n Represents depth of
qualitative data
of taxpayer or segment which information rather than
may lead to the generation breadth and may not be
of hypotheses and more necessarily representative
extensive research of the wider population
n Can be undertaken, to some n Costly to conduct when time
extent, without the need for and resource intensive
taxpayer co-operation n Generally can’t provide
because existing data can statistically reliable samples.
be used.
Academic Academic researchers publish n Researchers are n Information might be too
research tax-related research on a range highly-qualified and broad and general for our
of topics. specialised purposes
Provides both
n You can commission n Usually a time lag exists
qualitative and
quantitative data academic researchers between research and
to do specific work publication which means the
n The work may already results might not be applicable
have been done. to the period of interest.

04 APPENDIXES

Public opinion Can include market research n Gives a broad and current Can be difficult to analyse for
research surveys and scans of topical view of public opinion various reasons including:
issues receiving media attention. n A partial analysis can provide n Media scans need to be fairly
Provides
qualitative data some useful context to other comprehensive over the
findings. period of review and range
Library & of media
Information n Need to use consistent and
Services subscribe standard definitions for terms
to several such as positive and negative
databases which media perceptions
may be useful n Resource intensive and likely
for this kind to mean that most business
of research. areas wouldn’t conduct their
own ongoing media analysis.
Other government Most government websites Easy to access n Might use units of
agencies allow access to a range of measurement (for example,
published information which different definitions for
Material can
provides information on external particular population
be found on
factors or external benchmarks segments) that aren’t
the relevant
relating to tax compliance specific to our needs.
government
indicators.
websites. A good
place to start is These include:
the Australian, n Australian Bureau of Statistics
State, Territory (ABS) – provides a wide range
and Local of statistics
governments n Reserve Bank of Australia
website at (RBA) – has a range of
www.gov.au, statistics on financial
which has links indicators
to federal and n Productivity Commission
state government – undertakes research on
sites in Australia. economic, industry, business
and demographic issues
including a number of
business benchmarking
studies.
Professional Professional associations and May provide information on n Might use units of
associations industry bodies represent the external factors or external measurement (for example,
interests of their members and benchmarks relating to tax different definitions for
regulate their practice for the compliance indicators particular population
benefit of the community. Some segments) that aren’t
associations publish regular specific to our needs
updates and reports relating n Some reports are only
to their industry. available by paid subscription.

04 APPENDIXES
APPENDIX 2
REFERENCES
Evaluation references Other useful tools and resources
Australian National Audit Office 1998, Better practice
The Programme Managers Planning, Monitoring
principles for performance information, Australian National
and Evaluation Toolkit
Audit Office, Canberra
http://www.unfpa.org/monitoring/toolkit.htm
Babbie, ER, 2006, The Practice of Social Research,
Information is also available on these websites:
11th edn, Thomson/Wadsworth, California
Australasian Evaluation Society
Bamberger, M, Rugh, J, Mabry, L, 2006, RealWorld Evaluation: http://www.aes.asn.au/
working under budget, time, data, and political constraints,
Sage Publications, California American Evaluation Association
http://www.eval.org/
Creswell, JW, 2003, Research Design: Qualitative, Quantitative
and Mixed Method Approaches, Sage Publications, California
Department of Finance 1994, Doing Evaluation: A practical
guide, Department of Finance, Canberra
Kumar, R, 2005, Research Methodology: A Step By Step
Guide For Beginners, Sage Publications, London
Langbein, L, Felbinger, CL, 2006, Public Program Evaluation
– A Statistical Guide, ME Sharpe Inc, New York
National Performance Review 1997, Serving the American
public: Best Practices in Performance Measurement,
National Performance Review
http://www.orau.gov/pbm/links/npr2.html
Organisation of Economic Cooperation and Development 2006,
Tax Administration in OECD and selected non OECD countries:
Comparative information series, OECD, Paris
Owen, JM, 1993, Program Evaluation: Forms and Approaches,
Allen & Unwin, Australia
Patton, MQ, 2002, Qualitative Research & Evaluation Methods,
3rd edn, Sage Publications, California
Posavac, EJ, 2007, Program evaluation: methods and case
studies, Pearson Prentice Hall
Public Sector Management Office, Western Australia, 2007,
Preparing performance indicators – A practical guide
http://www.publicsector.wa.gov.au/
SiteCollectionDocuments/Non-Current - Preparing
Performance Indicators - A Practical Guide.pdf
Stufflebeam, DL, and Shinkfield, AJ, 1985, Systematic
Evaluation: A Self-Instructional Guide to Theory and
Practice, USA

04 APPENDIXES
APPENDIX 3
GLOSSARY
Comparison statistics Statistics or data that are used as a comparison. For example, a benchmark or control group
can be used as a comparison statistic.
Component A part of a larger whole.
Data point The individual point where a value is graphed, as a point on a line, bar or pie slice. Each data
point maps to an individual cell in a spreadsheet’s data range. For example, the percentage
of activity statements lodged on-time for the 2005–06 income year.
Data series A complete series of data, corresponding to the same type of data points. For example, the
percentage of activity statements lodged on-time over a period of seven income years.
Inferential statistics A group of methodologies that allow you to draw inferences about a population based on
the behaviour of a sample.
Primary data Data which is collected at source for the particular purpose of meeting the evaluation objective.
Risk management Oversees all compliance risk work in a business line and is responsible for prioritising compliance
committee risk work for that business line.
Sample A small part or quantity intended to show what the whole is like. For example, a subset of the
population of interest.
Secondary data Data that has been collected, summarised or interpreted for a different purpose.
Trend A general direction and tendency.
Trend line A line indicating the general course or tendency of something. For example, a set of points
on a graph.
Trend analysis Involves the collection and presentation of information in order to identify patterns.

MORE INFORMATION
For more information or to provide feedback on this
publication, email our Compliance Effectiveness Centre
of Expertise at ComplianceEffectivenessCoE@ato.gov.au
More information is also available in our publications:
n Literature review – Measuring compliance effectiveness
(NAT 71078)
n Measuring compliance effectiveness – Our methodology
(NAT 72341)
n Measuring compliance effectiveness – Applying our
methodology (NAT 72342)
n Developing effective compliance strategies (NAT 72788).
To view these publications visit

www.ato.gov.au/complianceeffectiveness
To order printed copies visit www.ato.gov.au/onlineordering

Measuring Compliance Effectiveness

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Measuring Compliance Effectiveness

Uploaded by

Copyright:

Available Formats

Guide for risk managers and evaluators

© AUSTRALIAN TAXATION OFFICE FOR THE PUBLISHED BY

EVALUATING EFFECTIVENESS 18 MORE INFORMATION inside back cover

MEASURING COMPLIANCE EFFECTIVENESS 1

2 MEASURING COMPLIANCE EFFECTIVENESS

EVALUATING COMPLIANCE When should an evaluation occur?

To understand this, you need to explore behaviour at:

MEASURING COMPLIANCE EFFECTIVENESS 3

FIGURE 1: Compliance effectiveness methodology

Who’s involved Who are your

4 MEASURING COMPLIANCE EFFECTIVENESS

What are How will the

What is the description of the What’s our

MEASURING COMPLIANCE EFFECTIVENESS 5

The answers to the questions posed in phases

If you are evaluating effectiveness retrospectively you will be voluntary compliance?

MEASURING COMPLIANCE EFFECTIVENESS 7

Aligning indicators to success goals

8 MEASURING COMPLIANCE EFFECTIVENESS

VALIDATING YOUR INDICATORS n Measurable

This test explores whether your indicators are specific, EXAMPLE

A change in the percentage of activity statements lodged

Comparing the compliance behaviour before you

MEASURING COMPLIANCE EFFECTIVENESS 9

10 MEASURING COMPLIANCE EFFECTIVENESS

MEASURING COMPLIANCE EFFECTIVENESS 11

Once you have finalised your suite of indicators you should 98

The validated suite of indicators should be endorsed by

what the compliance behaviour or community confidence 98

new data point. This indicates that something has affected

Comparing against a single data point, however, is not always

a reliable method of identifying a change. While you may see

12 MEASURING COMPLIANCE EFFECTIVENESS

01 DEVELOPING YOUR SUITE OF INDICATORS

data is collected from the same participants at regular

Compliance behaviour intervals. Longitudinal studies are particularly useful for 20

understanding the long-term effects behaviour

While it is preferable to have enough observations to make

the claim of a trend valid andbehaviour

Compliance behaviour Compariso

MEASURING COMPLIANCE EFFECTIVENESS 13

Identifying a change in compliance behaviour or community

confidence does not necessarily mean that your compliance

of external influences such as economic

14 MEASURING COMPLIANCE EFFECTIVENESS

n Population – A group of taxpayers who share a common

Seeking expert assistance

MEASURING COMPLIANCE EFFECTIVENESS 15

16 MEASURING COMPLIANCE EFFECTIVENESS

Are there any data limitations?

What comparisons will you use?

Once you have decided which comparison technique to use,

Documenting your intentions

MEASURING COMPLIANCE EFFECTIVENESS 17

BEFORE PROGRESSING FURTHER,

Who’s involved? Data storage

You also need to store the detailed data specifications in a way

MEASURING COMPLIANCE EFFECTIVENESS 19

Is there a change? Comparison data can be generated through the use of

Did the strategies cause the change? 100

If the data shows a difference in the pattern of movement after

the strategies were implemented, you can say with some

confidence that your strategies

MEASURING COMPLIANCE EFFECTIVENESS 1

2 MEASURING COMPLIANCE EFFECTIVENESS

MEASURING COMPLIANCE EFFECTIVENESS 3

4 MEASURING COMPLIANCE EFFECTIVENESS

MEASURING COMPLIANCE EFFECTIVENESS 5

MEASURING COMPLIANCE EFFECTIVENESS 7

8 MEASURING COMPLIANCE EFFECTIVENESS

MEASURING COMPLIANCE EFFECTIVENESS 9

10 MEASURING COMPLIANCE EFFECTIVENESS

MEASURING COMPLIANCE EFFECTIVENESS 11

12 MEASURING COMPLIANCE EFFECTIVENESS

MEASURING COMPLIANCE EFFECTIVENESS 13

14 MEASURING COMPLIANCE EFFECTIVENESS

MEASURING COMPLIANCE EFFECTIVENESS 15

16 MEASURING COMPLIANCE EFFECTIVENESS

MEASURING COMPLIANCE EFFECTIVENESS 17

MEASURING COMPLIANCE EFFECTIVENESS 19

20 98 MEASURING COMPLIANCE EFFECTIVENESS

MEASURING COMPLIANCE EFFECTIVENESS 21

22 MEASURING COMPLIANCE EFFECTIVENESS

MEASURING COMPLIANCE EFFECTIVENESS 23

MEASURING COMPLIANCE EFFECTIVENESS 25

26 MEASURING COMPLIANCE EFFECTIVENESS

28 MEASURING COMPLIANCE EFFECTIVENESS

MEASURING COMPLIANCE EFFECTIVENESS 29

30 MEASURING COMPLIANCE EFFECTIVENESS

MEASURING COMPLIANCE EFFECTIVENESS 31

32 MEASURING COMPLIANCE EFFECTIVENESS