1

THE RESEARCH INQUIRY

a) Title of the Study

“Empirical Analysis of the Reliability and Validity of Balanced Scorecard
Measures and Dimensions”
Author: Emilio Boulianne
Source: Advances in Management Accounting, Volume 15, Emerald
Group Publishing Limited, pp.127 – 142, doi: 10.1016/S1474-
7871(06)15006-6
Date of online publication: March 12, 2015
ISBN: 978-0-76231-352-5
eISBN: 978-1-84950-447-8
Number of pages: 16

b) Theoretical and conceptual framework

This study is anchored on the balanced scorecard (BSC) by Kaplan and
Norton (1992) originally developed as a performance measurement
framework that added strategic non-financial performance. Unlike earlier
performance measurement systems which focused exclusively through a
financial scope, the BSC measures performance across a number of four
(4) different key areas that impact the organizational performance:
financial, customer, internal business process and learning and growth
dimensions. The balanced scorecard is a strategic management tool used for
translating an organization's strategic objectives into a set of performance
indicators distributed among four quadrants. Some of these indicators are
maintained to measure an organization's advancement towards its vision and
other indicators are maintained to measure the long term drivers of success. 

c) Management accounting problem or concept

The balanced scorecard (BSC) is the most widely applied performance
management system today. However, little is known about the reliability and
validity of the measures and dimensions it proposes. Hence, this study aims
to help in the design and implementation of balanced scorecards in business
units by empirically examining the BSC dimensions and its suggested
 2

measures. So, the reader should look the BSC as a construct aiming to
assess business unit performance. This is the original aspect of this paper,
since previous studies on the BSC took for granted the suggested measures
and the four quadrants/dimensions.

d) Prevalent internal and external environment

The four dimensions of BSC are considered essential to almost all
organizations and this study was employed among targeted Canadian
business units comprising both the service and manufacturing industries.
Moreover, a list of measures associated with each of the dimensions are
identified which are common across the business units and considered
appropriate. The author selected the BSC measures based on Kaplan and
Norton (1996, 2001) and Kaplan and Atkinson (1998), where several
scorecards are presented. These measures take into account the availability
of both financial and nonfinancial data from business units. For the Financial
Dimension, return on assets and net profit margin reflect the financial
performance while working capital ratio reflects asset utilization. For the
Customer Dimension, marketing expenses to revenues reflects marketing
efforts to solicit new customers while revenue growth is a proxy for market
share. For the Internal Business Process Dimension, number of new products
introduced over the last 3 years, number of product offers, and R&D expenses
to revenues reflect innovation initiatives. Finally, for the Learning and Growth
Dimension, employee absenteeism rate and employee turnover rate reflect
employee satisfaction, training expenses to revenue reflects employees’
training efforts, and revenue per employee reflects employee productivity. So
as to examine the reliability of the common measures selected and the validity
of the BSC dimensions, the author gathered these measures among business
units. It is to be noted that only common measures count as evaluations of
performance across units while measures unique to individual business units
tend to be ignored.

e) Managerial accounting analysis, approach and model

This study reports on the reliability of the balanced scorecard measures
and the factorial validity of the BSC dimensions. Validity problems impact
 3

on the importance and credibility allocated by management to certain BSC

measures. This study's objective is to empirically examine the construct
validity of the BSC. Through a literature review and field study, a set of
measures associated with all four BSC dimensions is selected. Then, a
survey research in partnership with CGA-Canada, a Canadian professional
accounting association, was conducted among ninety (90) Canadian
business units which had one hundred (100) employees or more. The
business units were in pulp and paper, textile, transformation, construction,
industrial products, food products, retailer, wholesaler, leasing, and dealers.
Respondents were asked to provide financial and non-financial data to
calculate the return on asset, net profit margin, working capital ratio, revenue
growth, marketing expenses to revenues, number of new products, number of
product offers, R&D expenses to revenues, training expenses to revenues,
and revenue per employee measures. For the employee absenteeism rate
and employee turnover rate measures, respondents were asked to classify
their business units’ compared with peers’ using a 7-point scale. Touching on
the concepts of content validity, internal consistency reliability, and factorial
validity, results indicate that the BSC four dimensions with a set of common
measures represent a valid performance model.

f) Managerial implications of the study

The purpose of measuring performance is not only to know how the entity is
performing but also to enable it to perform better. The balanced scorecard is
the pioneer tool in focusing on a range of perspectives which included
financial and non-financial factors. The tool is built to focus on past and future
of the business with its lagging and leading measures. However, in an ever
evolving business environment, the BSC has failed to evolve with time and
the balanced scorecard depends heavily upon how it is used or interpreted.
Therefore the management commitment towards BSC is vital for its success.
Accordingly, this study responds to research calls and an initiative toward a
theory-building perspective on the importance of validating the BSC
framework and its associated measures as a performance model in order to
enhance consistency on BSC research. Lastly, there has been no other study
 4

that has empirically investigated the BSC from a construct validity

perspective, so the present study provides evidence in this area.

THE CRITIQUE

a) Point of view
The balanced scorecard system has been implemented by thousands of
companies and organizations worldwide (Olsson, Karlsson, & Sharma,
2000); proving its value in time as the scorecard format is quite valuable and
the four dimensions for strategy and performance measurement are
undeniably an improvement over traditional thought. The essence of the
balanced scorecard was (is) that it takes a multi-layered approach to business
performance measurement and management. That is, it evaluates and helps
actively manage the key strategic elements and cause and effect relationships
that combine to deliver the desired business results (Kaplan, 2012).
Unfortunately, Kaplan and Norton do not provide any empirical evidence that
these are the correct four perspectives to provide proper balance for the
strategic management and evaluation of a business (Flamholtz, 2003). But
this literature gap is supplemented by Boulianne, (2006) who has empirically
investigated the balanced scorecard from a construct validity perspective and
claims BSC as a credible performance model thereby providing evidence in
this area. However still, the balanced scorecard lacks empirical evidence
regarding effectiveness on both theoretical and practical grounds. From a
theoretical point of view, Norreklit (2003) has argued that the balanced
scorecards are not based on any proven economic or financial theory and
have no basis in the decision sciences. This suggests that the balanced
scorecard design and implementation process is entirely subjective, without a
solid actuarially or economic basis. Another criticism is related to the balanced
scorecard’s efficiency for business planning, in the idea that it is just a list of
metrics which does not offer a unified view or clear recommendations where it
creates a silo organizational objectives and metrics instead of connected
corporate goals. In practice, despite widespread use and practitioner-oriented
literature suggesting the BSC has beneficial values especially in enhancing
 5

organizational performance and strategy achievement, a large body of

academics is sceptical about the relationship between BSC and
organizational outcomes. In fact, a leading criticism asserts the widespread
practice of the BSC in itself does not demonstrate the BSC is beneficial to
organizations (Kraaijenbrink, 2012). On the other hand, Isoraite (2008) has
argued that the usefulness of the balanced scorecard relies on its conformity
to the overall business strategy. Companies may, thus, bias their scorecard to
the dimensions that closely support their strategic direction, ignoring aspects
that do not seem to have a direct, observable and measurable impact on the
company’s strategic goals. Related to this is an example wherein as creditors
and shareholders are mainly, if not exclusively, interested in financial
performance rather than operational performance, the management will be
somehow inclined to focus more on the organization’s financial perspective.
Thus, this may lead to an ‘’imbalanced’’ balanced scorecard.

b) Thesis statement
While there is no such thing as a perfect performance measurement tool due
to the rising complexities of organizations all the world over time, the balanced
scorecard framework has established itself as a useful concept in this field.
When carried out correctly, it can provide decision makers with the essential
guidance to direct companies in achieving their strategic goals. Balanced
scorecard’s limits, however, should also be understood, in order to avoid
reducing the company’s complex activity to a set of performance indicators
that may, or may not, prove relevant on the long run.

c) Critiquing
The study of Boulianne on the validity of balanced scorecards as a
performance model through empirical analysis is presented fairly well and
should be of interest to the readers as it contributes value to the limited
literature of performance evaluation and management accounting. It is
commendable that the author is the first to support its claim about the
cogency of the BSC as a tool in business performance metrics focusing on
the four quadrants. Throughout the journal article, the paper is concise and
easy to understand. The author’s collection of RRL is refined and appropriate.
 6

RRL scripts are coherent and transition smoothly by injecting them from start
to finish. These statements contributed to the overall understanding of the
subject and to the reasoning for establishing the problem statement. Clearly,
there is consistency of words used, connection of concepts/theories and
integration of ideas. Moreover, the methods used to gather the data for this
article were clearly explained. The statistics are appropriate for the study. The
instruments and development were explained, and the reliability and validity of
all possible tests were given. But it could have been better if the sample size
is increased and other BSC measures construct are included so that it will
enhance the robustness and reliability of BSC studies and offer a more solid
base in theory development. Meanwhile, the conclusion were based on the
findings and logically stated. It is also laudable for the author that he
acknowledged weak points and limitations in his study for improvement in
future researches. Overall, it is well-written, well-organized article and a very
interesting and significant contribution to the field of research.
 7

List of References

Boulianne, E. (2006). Empirical analysis of the reliability and validity of balanced

scorecard measures and dimensions. In Advances in Management
accounting (pp. 127-142). Emerald Group Publishing Limited.

Flamholtz, E. G. (2003). Putting balance and validity into the balanced

scorecard. Journal of Human Resource Costing & Accounting, 7(3), 15-
26.

Isoraite, M. (2008). The balanced scorecard method: From theory to

practice. Intelektine Ekonomika, (1).

Kaplan, R. S. (2012). The balanced scorecard: comments on balanced scorecard

commentaries. Journal of Accounting & Organizational Change, 8(4), 539-
545.

Kraaijenbrink, J. (2012). Five reasons to abandon the Balanced

Scorecard. Retrieved, 10(11), 2012.

Nørreklit, H. (2003). The balanced scorecard: what is the score? A rhetorical

analysis of the balanced scorecard. Accounting, organizations and
society, 28(6), 591-619.

Olsson, B., Karlsson, M., & Sharma, E. (2000). Towards a theory of implementing
the balance scorecard: A study in association with the Swedish
telecommunication firm Ericsson. Journal of Human Resource Costing &
Accounting, 5(1), 59-84.