See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/270685478

BIM cloud score: Benchmarking BIM performance

Article  in  Journal of Construction Engineering and Management · July 2014

DOI: 10.1061/(ASCE)CO.1943-7862.0000891

CITATIONS READS
38 1,977

3 authors:

Jing Du Rui Liu

Texas A&M University University of Florida
55 PUBLICATIONS   651 CITATIONS    51 PUBLICATIONS   763 CITATIONS   

SEE PROFILE SEE PROFILE

Raja R A Issa
University of Florida
245 PUBLICATIONS   3,628 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Building Information Modeling (BIM) and Multi-user Vritual Reality (VR) System View project

BIM for Facility Management View project

All content following this page was uploaded by Rui Liu on 12 February 2015.

The user has requested enhancement of the downloaded file.

 BIM Cloud Score: Benchmarking BIM Performance
Jing Du 1; Rui Liu 2; and Raja R. A. Issa, F.ASCE 3

Abstract: A variety of building information modeling (BIM) performance evaluation initiatives have been proposed to quantify BIM uti-
lization capacity of enterprises. These initiatives were designed for evaluating, instead of benchmarking, an organization’s performance in
BIM utilization. Unlike evaluation that mainly focuses on ascertaining the achievement of BIM utilization within an organization, bench-
marking is more interested in comparing one organization’s BIM performance to their industry peers. By identifying gaps in specific areas,
decisions to make improvements can be facilitated. This paper proposes a cloud-based BIM performance benchmarking application called
building information modeling cloud score (BIMCS) to automatically collect BIM performance data from a wide range of BIM users nation-
Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.

wide. It utilizes the software as a service (SaaS) model of cloud computing to make the collection, aggregation, and presentation of bench-
marking data autonomous and interactive. Based on the big data collected from the BIMCS database, an overall view of the industry status
quo of BIM utilization may be obtained, and ultimately, a protocol for BIM performance can be developed on the basis of a better knowledge
discovery process. Furthermore, BIMCS data will help individual companies compare and improve their performance in BIM utilization with
respect to their industry competitors. DOI: 10.1061/(ASCE)CO.1943-7862.0000891. © 2014 American Society of Civil Engineers.
Author keywords: Building information modeling (BIM); Cloud computing; Building information modeling cloud score (BIMCS);
Benchmarking; Information technologies.

Introduction The greatest difference between evaluation and benchmarking is

A messy building information modeling (BIM) model will
ultimately lead to a messy project, which is difficult to execute,
control, coordinate and will probably introduce additional cost
(Suermann 2009). To continuously increase an industry’s aware-
ness of BIM performance issues, a variety of BIM performance
evaluation frameworks have been proposed to score enterprises’
capacity of BIM development, technological maturity, and other
administrative and technical perspectives of BIM performance
(CIFE 2013; NIBS 2012; Succar 2009; Succar et al. 2013).
Despite the great success of current BIM performance evalu-
ation initiatives, they are susceptible to omissions and several lim-
itations. The greatest shortcoming is that current initiatives were
designed for evaluation instead of benchmarking. Evaluation is
the systematic process to determine the merit, value, and signifi-
cance of the design, implementation, and outcomes of the evaluated
subject (e.g., a program) (Rossi et al. 2004). The primary purpose
of evaluation is to ascertain the degree of achievement of an organi-
zation, program, or project in regard to the aim, objectives, and
results that have been achieved (Rossi et al. 2004). In contrast,
benchmarking is the process of comparing one’s business processes
and performance metrics to the industry bests (Costa et al. 2006).
1
Assistant Professor, Dept. of Construction Science, Univ. of Texas at
San Antonio, 501 W.César E. Chávez Blvd., San Antonio, TX 78207
(corresponding author). E-mail: jing.du@utsa.edu
2
Assistant Professor, Dept. of Construction Science, Univ. of Texas at
San Antonio, 501 W.César E. Chávez Blvd., San Antonio, TX 78207.
E-mail: rui.liu@utsa.edu
3
Univ. of Florida Research Foundation and Holland Professor, Rinker
School of Construction Management, Univ. of Florida, P.O. Box 115703,
Gainesville, FL 32611. E-mail: raymond-issa@ufl.edu
Note. This manuscript was submitted on January 31, 2014; approved on
May 1, 2014; published online on July 3, 2014. Discussion period open
until December 3, 2014; separate discussions must be submitted for indi-
vidual papers. This paper is part of the Journal of Construction Engineer-
ing and Management, © ASCE, ISSN 0733-9364/04014054(13)/$25.00.
the cross-organizational comparison. Through cross-organizational
comparison, an organization can learn how well the targets perform
and more importantly, the business processes that explain why the
leading organizations are successful. Then, lessons learned from
other organizations can be used to establish improvement targets
and promote changes in the organization (Barber 2004).
Cross-organizational benchmarking, compared to internal
evaluation, can significantly improve an organization’s perfor-
mance by explicitly identifying the gaps with their best peers
(Costa et al. 2006). Unfortunately, current BIM performance evalu-
ation initiatives have several limitations in their designs, making
benchmarking harder. First, most initiatives were originally
designed for internal evaluation of BIM utilization within a single
organization (NIBS 2007; Succar et al. 2013); the metrics used are
either subjective or overspecific, unsuitable for benchmarking
purpose. Second, although certain BIM evaluation initiatives were
claimed for benchmarking (CIFE 2013; Sebastian and van Berlo
2010), they use self-reported questionnaires as the source of data,
which is more questionable in the benchmarking process. Last,
many initiatives are commercialized (not free of charge) and thus
make it difficult to collect sufficient data for statistical conclusions.
Without a proper benchmarking function, these initiatives can
hardly renovate BIM performance, point the future directions,
and identify the gaps easily. Following the above notions, two
challenges need to be addressed:
1. Can a system be designed that allows automated BIM perfor-
mance benchmarking?
2. How can enough data be aggregated for benchmarking BIM
performance?
To overcome these challenges, this study proposes a free cloud-
based BIM performance benchmarking application called building
information modeling cloud score (BIMCS) to automatically
collect BIM performance data from a wide range of BIM users
nationwide. The overall BIM performance is quantified based on
two critical aspects: the BIM model (product) and BIM modeling
(process). By comparing their BIM performance scores against

© ASCE 04014054-1 J. Constr. Eng. Manage.

J. Constr. Eng. Manage. 2014.140.

 industry peers, users of BIMCS are able to assess the competitive
position of their BIM performance to identify gaps in their BIM
model and modeling processes and to decide improvement direc-
tions. Facilitated by the software as a service (SaaS) model of
cloud computing, the collection, aggregation, and presentation
of benchmarking data in BIMCS are autonomous and interactive.
Furthermore, based on the big data collected from the nationwide
benchmarking process, the BIMCS makes in-depth data mining
possible to discover critical BIM performance aspects, the group-
ings of BIM utilization areas, and market sectors. It provides an
overall view of the status of the industry’s BIM utilization efforts.
Ultimately, a protocol of BIM performance may be developed on
the basis of better knowledge discovery of the BIM utilization pro-
vided by BIMCS.
Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.
Literature Review
BIM Evaluation
Since 2007, there have been various attempts at evaluating BIM
performance in the construction industry. One of the pioneering
efforts was the national BIM standard’s interactive capability
maturity model (ICMM) (NIBS 2007). ICMM is an interactive
Microsoft Excel worksheet that helps organizations measure the
degree to which their BIM projects implemented a mature BIM
standard (NIBS 2007). Eleven metrics were used to evaluate infor-
mation management pertaining to BIM utilization, including data
richness, life-cycle views, roles or disciplines, business processes,
timeliness of response, delivery methods, graphical information,
spatial capability, information accuracy, change management, and
interoperability/industry foundation classes (IFC) support. ICMM
was designed for internal use only and not for comparing BIM proj-
ects across organizations (Suermann et al. 2008).
Since 2009, Succar (Succar 2010; Succar et al. 2012, 2013) has
developed a series of BIM evaluation frameworks to measure the
quality of individuals, teams, and organizations that implement
BIM. His first model, the BIM maturity matrix, aimed to evaluate
administrative capacity of an organization in utilizing BIM, rather
than assessing a specific BIM-assisted project (Succar 2010). Ten
key maturity areas (KMAs) were used to assign an organization
into one of the five maturity levels (Succar 2010). Shortly after this
work, Succar et al. (2012) developed a BIM competency index to
quantify the abilities of individuals (BIM developers) within
professional and academic settings. This framework highlights
individuals as a key metric of BIM performance measurement.
Then, competency is measured as an individual competency index
(ICI) consisting of five levels of increasing competence: (1) none,
(2) basic, (3) intermediate, (4) advanced, and (5) expert. These
levels are treated as significant predictors of BIM performance out-
comes (Succar et al. 2013). Recently, the above framework has
been integrated into a larger project called BIM excellence (BIMe)
(Succar et al. 2013). BIMe provides a 360-degree BIM perfor-
mance assessment tool to assess individuals, organizations, teams,
or projects according to multiple competency areas.
Unlike ICMM and Succar’s works, certain BIM performance
evaluation initiatives declared themselves as benchmarking tools
from the very beginning. One of such works is the virtual design
and construction (VDC) and BIM scorecard developed by
Stanford’s Center for Integrated Facility Engineering (CIFE
2013). This framework evaluates the maturity of virtual design
and BIM practices on specific projects on the basis of the results
of four survey forms pertaining to planning, adoption, technology
and performance (CIFE 2013). Users are able to benchmark BIM
© ASCE 04014054-2
J. Constr. Eng. Manage. 2014.140.
performance against industry standards. Another representative
framework, QuickScan, was introduced in the Netherlands in
2009 by the Netherlands Organization for Applied Scientific
Research (TNO) (Sebastian and van Berlo 2010). According to
TNO, QuickScan is a benchmarking tool to evaluate BIM perfor-
mance of organizations utilizing BIM. Four categories of criteria
are used which include (1) organization and management, (2) men-
tality and culture, (3) information structure and information flow,
and (4) tools and applications, with each category consisting of
weighted key performance indicators (KPIs). Similar to the CIFE,
VDC, and BIM scorecard, the QuickScan performance data was
collected via multiple choice questionnaires from BIM manage-
ment personnel (Sebastian and van Berlo 2010). These benchmark-
ing instruments are based on self-reported questionnaires as the
source of data and thus tend to be questionable.
Benchmarking
Benchmarking is a structured approach of measuring and compar-
ing an organization’s processes, activities, and performance against
that of other organizations in key business areas (Barber 2004).
Although in practice, benchmarking may refer to multiple types
such as process benchmarking, strategy benchmarking, and perfor-
mance benchmarking (Bogan and English 1994), this paper is
concerned with performance benchmarking, i.e., focused on the
comparison of key performance outcomes.
During the process of benchmarking, decision makers identify
the best peers in the same area and compare the performance out-
comes of those studied organizations (called targets) to their own
performance outcomes (Costa et al. 2006). In this process, several
benefits can be obtained. The greatest benefit is that it allows an
organization to assess its competitive position in the industry.
Performance outcomes are not only quantified, but also ranked
and shown as a percentile value among industry competitors
(Bendell et al. 1993). It is also possible to convert such percentile
values to the possibility of success (Lohuis et al. 1992). The bench-
marking process can also encourage innovation (Garvin 1993).
Benchmarking involves a community of similar organizations that
compare results and share practices for continuous improvement.
This circumstance establishes benchmarking clubs where people
share the same concerns and are willing to learn from the best
practices (Costa et al. 2006). Finally, benchmarking can also raise
the awareness of the decision makers in the key business processes
of an organization. It helps managers understand the ways that lead
to better performance (Camp and Camp Robert 1995). According
to Garvin (1993), it also encourages managers to act more proac-
tively in business processes rather than depending exclusively on
outcomes.
In recognition of the importance of benchmarking, various
benchmarking initiatives have been established in the construction
industry. Representative works include the following:
• Construction industry institute benchmarking and metrics
(CII BM&M) (Construction Industry Institute 2013);
• U.K.’s construction best practice programme (CBPP)
(Construction Excellence 2013);
• National benchmarking system for the Chilean construction
industry (NBS-Chile) (Ramírez et al. 2004);
• Benchmarking for lean construction (Marosszekey and Karim
1997); and
• Other initiatives developed in Brazil, Chile and Hong Kong
(Costa et al. 2006).
Although serving different regions with different foci and tar-
geted at different market sectors, these construction benchmarking
initiatives are all targeted at developing common definitions and
J. Constr. Eng. Manage.
 guidance for performance measurement, providing performance
norms for the industry, and disseminating the identified best prac-
tices via reports and benchmarking networks. Certain benchmark-
ing initiatives also attempt to establish support networks for
participants and domain experts to discuss the best practices in
the industry and determine how to implement them for performance
improvement (Construction Industry Institute 2013).
Current benchmarking initiatives face three challenges:
• One challenge is a commonly agreed list of benchmarking
metrics. For example, one of the CII BM&M’s objectives is to
reach an “agreement on common definitions for metrics of per-
formance and practice use (Costa et al. 2006).” The metrics used
in benchmarking must be specific enough for all organizations.
The metrics typically measured are quality, time, and cost.
Others may include cost per unit of measure, productivity
per unit of measure, cycle time of x per unit of measure, or de-
Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.
fects per unit of measure (Construction Industry Institute 2013).
However, by comparing four major benchmarking initiatives in-
cluding CII BM&M and CBPP, Costa et al. (2006) still found
significant divergence among different benchmarking initia-
tives, suggesting the lack of a benchmarking protocol in the con-
struction industry.
• The second challenge is the weighting method of benchmarking
metrics. Although the literature has suggested multiple objective
weighting methods (Datta et al. 2009; Greco et al. 2001; Zeleny
and Cochrane 1982), it was noticed that many of these bench-
marking initiatives still apply subjective weight methods and
extensively rely on opinions of domain experts. This is ques-
tionable when a benchmarking is designed for a wider range
of subjects.
• The last challenge is the method used to collect benchmarking
data. Although some benchmarking initiatives, such as the
CII BM&M (Construction Industry Institute 2013), have created
web applications to make the benchmarking process more
friendly, this process is still not fully open to the industry practi-
tioners (for example, CII requires membership for benchmark-
ing service) and is not autonomous (most benchmarking
initiatives require manual input of performance data). Cloud
computing can be used to address the above challenges.
Cloud Computing
According to NIST, cloud computing is a model for “enabling
ubiquitous, convenient, on-demand network access to a shared pool
of configurable computing resources (e.g., networks, servers,
storage, applications, and services) that can be rapidly provisioned
and released with minimal management effort or service provider
interaction (Mell and Grance 2011).” NIST (Mell and Grance 2011)
identified three service models that are currently employed by
cloud computing including (1) software as a service (SaaS),
(2) platform as a service (PaaS), and (3) infrastructure as a service
(IaaS). Users have different capacities of exercising control over the
components of the infrastructure. For example, IaaS users are al-
lowed to provision processing, storage, networks, and other funda-
mental computing resources, whereas users of SaaS may only use
the provider’s applications running on a cloud infrastructure. The
applications are accessible from various client devices through ei-
ther a thin client (e.g., a web browser or a program interface) or a fat
client (e.g., a tower personal computer). Depending on the acces-
sibility of a cloud service, cloud can be deployed as a private cloud,
community cloud, public cloud, or hybrid cloud (Subashini and
Kavitha 2011). The choice of a cloud deployment method depends
on the concerns of the users such as missions, security policies, and
compliance considerations (Marinescu 2013). Despite deployment
© ASCE
J. Constr. Eng. Manage. 2014.140.
differences, a cloud infrastructure may be owned, managed, and
operated by the organizations themselves, third-party service prov-
iders, or some combination of them.
Cloud computing has attracted increasing interest from the ar-
chitecture, engineering, and construction (AEC) industry. Applica-
tions can be grouped into three categories: (1) cloud computing in
construction management (Gong and Azambuja 2013; Kumar et al.
2010; Liang and Luo 2013; Senescu et al. 2014); (2) cloud com-
puting in engineering design and analysis (Chen et al. 2013; Gracia
and Bayo 2013; Karamouz et al. 2013; Latu et al. 2013; Liu et al.
2013); and (3) cloud computing in BIM. Particularly, BIM cloud
application focuses on utilizing cloud technologies to advance the
collaborativity, interoperability, and usability of BIM applications.
Chuang et al. (2011) developed a web application for BIM visu-
alization and operation by utilizing the SaaS aspect of cloud com-
puting. Users can operate 3D BIM models through the web without
the limitations of time or distance. Redmond et al. (2012) found
that a main problem of the industry foundation classes (IFCs)
was the difficulty of storing and carrying relevant data for all multi-
featured construction processes. They therefore proposed a
cloud-based platform for seamless data exchange among various
disciplines at the early stage of a project. The proposed platform
harnesses the capability of IFC extensible markup language (XML)
with simplified markup language (SML) subsets of XML for para-
metric modelling and interaction. Recently, Zhang and Issa (2012)
compared various BIM cloud computing systems including Revit
Server, Revit Cloud and STRATUS, and general purpose cloud
services such as Advance 2000 and Amazon. Based on the find-
ings, they proposed a framework for making decisions on adopting
cloud computing for BIM applications in specific contexts.
The Autodesk 360 suite of cloud-based BIM services (Field,
Glue, etc.) is an example of a commercially available package
(Autodesk 2014).
Methodology
To develop the BIMCS a three-step roadmap was followed as
shown in Fig. 1:
1. The first step focuses on developing a list of metrics that are
suitable for benchmarking BIM performance. As discussed
above, a prerequisite of benchmarking is an agreement on
the definitions and measures of the common set of metrics
(Costa et al. 2006). At first, an initial list of metrics was pro-
posed based on existing literature and modeling theories
(Sargent 2005; Succar 2009; Succar et al. 2013; Underwood
and Isikdag 2010). Then, a survey was distributed to domain
experts to validate the list. The pool of respondents included
architects, contractors, engineers, owners, and BIM software
developers from the AEC industry. They were asked to select
the metrics that they believed were irrelevant to the context and
to provide metrics that they believed to be critical for BIM
performance benchmarking. Another purpose of the survey
was to obtain the subjective weights of the proposed metrics
by asking domain experts to respond to five-point Likert scale
questions (Issa and Suermann 2009). At the time of writing
this paper, the work for step 1 is ongoing.
2. The second step aims to perform the cloud-based benchmark-
ing of BIM performance. Since multiple metrics are consid-
ered, the benchmarking starts with a multivariate evaluation
(Fornell 1982). For any BIM project, the performance on each
of the 20 metrics will first be scored individually according to
predetermined standards. Then based on the initial weights ob-
tained from step 1, the overall performance score of a BIM
04014054-3 J. Constr. Eng. Manage.
Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.
Fig. 1. Methodology for developing BIM cloud score
project is calculated as the weighted summation of the sub-
scores of 20 metrics. A cloud-based questionnaire is used
to collect the calculated BIM performance scores. Per the per-
mission of the end-users, BIMCS automatically calculates the
performance score and submits the information to a cloud ser-
ver maintained for the purposes of the study. The information
is then aggregated into a database called BIM performance da-
tabase. A probability density function (PDF) will be generated
on the basis of the aggregated data. End-users of BIMCS are
therefore able to compare their performance scores to other
industry peers. BIMCS can be implemented as an add-in
for the off-the-shelf BIM software or a pure web application.
Also of importance is the fact that BIM performance depends
on the various project types and their organizational scales. An
inquiry about the background information such as project
type and magnitude will be sent out to the end users before
the benchmarking process takes effect. The data collected
from step 2 can be later classified, grouped, and processed
for further analysis based on the associated background infor-
mation. Such background information can be used as the filters
of benchmarking results because users may only interested in
particular sectors.
3. The third step is designed to improve the system on the basis
of accumulated information. The quality of benchmarking de-
pends on an appropriate list of metrics and scientific weighting
of the metrics (Costa et al. 2006). The BIMCS developed in
step 1 is based on the perceptions of domain experts. Once
sufficient data is obtained in the BIM performance database,
a set of data mining analyses will be performed to generate a
more proper weighting system for the metrics. Possible meth-
ods include maximum entropy method, principal component
analysis, and multiple correlation method (Diakoulaki et al.
1995; Srdjevic et al. 2004; Sheng et al. 2007). A factor ana-
lysis (Var 1998) will also be applied to find out a new list of
metrics which are the linear combinations of the original list.
This process will be repeated through several iterations until
an objective and unbiased benchmarking system is obtained.
The importance of incorporating opinions of domain experts in
the final version of benchmarking system is also recognized. As a
result, subjective weight methods will also be applied such as Del-
phi sessions. These sessions will be based on the BIM performance
database and therefore informed discussions can be conducted. The
final version of the BIMCS will reflect the pattern of the collected
data and the perspectives of the domain experts.
© ASCE 04014054-4
J. Constr. Eng. Manage. 2014.140.
BIMCS: A BIM Performance Benchmarking
Apparatus
Overview
The BIMCS is a web-based application for BIM performance
benchmarking. The BIMCS utilizes the SaaS service model (Fig. 2).
Compared to PaaS and IaaS, SaaS allows the users of BIMCS to
have the full capacity of transporting the performance data, while
reducing the risk of manipulating the system to the minimal level.
The BIMCS is deployed as a community cloud (Fig. 2). The com-
munity cloud environment ensures the accessibility of an exclusive
community of users from different organizations that have shared
concerns such as mission, policy, and compliance considerations
(Marinescu 2013). Unlike private cloud and public cloud, it pro-
vides a secured cross-organizational environment that satisfies
the requirements of the BIMCS.
A BIMCS user may select to submit BIM-related information
such as generic background information and BIM model outputs
to a remote Hadoop distributed file system (HDFS) server (Borthakur
2007) called the benchmarking server. The benchmarking server ag-
gregates the submitted information and calculates statistics of inter-
est. The user can then see their own performance as a percentile or
probability of success compared with all other industry peers. A set of
filters may be set so that the user can query and focus only on the
benchmarking aspects/metrics that the user is interested in.
Benchmarking Metrics
To facilitate the benchmarking of BIM performance metrics, the
following requirements must be satisfied:
• Objective: Metrics should be objective. BIM users may have
divergent perceptions and interpretations for a given metric.
Objective metrics ensure that the measure of any metric is built
on the same denotation.
• Quantifiable: Following the objectiveness requirement, metrics
should also be able to be quantified so that they can be defini-
tively defined and defended.
• Inherent: Metrics should be inherent in the BIM database. In
other words, measures of metrics must be able to be read and
pulled directly from the BIM database. As a result, fast and
automatic performance benchmarking can be realized.
• Generic: Metrics should focus on the common aspects of differ-
ent BIM projects. Specific BIM projects may have very unique
J. Constr. Eng. Manage.
 Fig. 2. BIM cloud score as SaaS
Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.
features that can hardly be compared against other projects
and thus, benchmarking works on the common features that
are valid in general settings. For example, the development dura-
tion of a complex BIM project and a simple BIM project are not
comparable, but the number of objects developed per unit of time
reflects the productivity of development to some extent. There-
fore, the latter one is a better candidate for benchmarking metrics.
• Representative: Metrics should be representative of critical
aspects of the BIM model and modeling processes.
Based on the above requirements, 20 metrics were developed
that are categorized into six aspects: (1) productivity, (2) effective-
ness, (3) quality, (4) accuracy, (5) usefulness, and (6) economy
(Table 1). The first two aspects quantify the performance of
BIM modeling (production), and the rest captures the performance
of BIM models (product). Some metrics rely on increasing values
as good performance indicators (e.g., number of objected created
per week) while some rely on decreasing values as good perfor-
mance indicators (e.g., number of structural warnings/number of
nonstructural warnings); therefore, the metrics selected need to
be normalized before their use in BIMCS.
Cloud Benchmarking
Fig. 3 illustrates the architecture of the BIMCS. It applies the
client—server model of centralized computing, i.e., off-loading
computational tasks to a remote Hadoop server in the network
(Nieh et al. 2000). There are four major components in this design:
• Client: A fat client that has many resources and does not rely on
the server for essential functions, or a thin client that relies heav-
ily on network resources for computation and storage;
• Internet: A computer network that uses the standard internet
protocol suite (transmission control protocol/IP) to serve as
the communication channel;
• Web server: A web server that supports not only hypertext trans-
fer protocol (HTTP), but also server-side scripting using active
server pages (ASP), hypertext processer, or other scripting
languages; and
• Benchmarking server: A Hadoop server (HDFS) that assumes
the mass storage and benchmarking related computation
requirements.
The first step in implementing this architecture is to set up the
benchmarking application, i.e., configuring the client for executing
the necessary application files. Although the main benchmarking
computation is done on the remote HDFS server, some application
files need to be set up in the client to access local BIM databases
and to compute metric values. One solution is through a web
© ASCE 04014054-5
J. Constr. Eng. Manage. 2014.140.
browser application. Users access the benchmarking application
server through the uniform resource locator (URL) of SaaS, which
sends the necessary files for executing the benchmarking applica-
tion, such as java script application (JSP), to the users’ terminals.
Port 80 is used to send requests to the benchmarking server as a
pure endpoint for the HTTP. It is compatible with Microsoft Inter-
net Explorer and therefore easier for the client side. Other ports,
such as Port 20, can be used to send back application files.
On the back end, the installed application files query the BIM da-
tabase to compute metric values. IFC as an industry standard can be
queried by the BIM server through Java (Liu 2012). On the front end,
users have the ability to view and operate benchmarking applications.
The best way to implement this kind of interactive applications is
using the model-view-controller (MVC) pattern (Krasner and Pope
1988). MVC is a classic design pattern that depends on three major
components: (1) models for maintaining data, (2) views for display-
ing purposes, and (3) controllers for handling user initiated events
(Krasner and Pope 1988). The other way to set up the benchmarking
application is through add-ins to off-the-shelf BIM software. The
Revit database can be queried in Visual Studio though C#. The fol-
lowing are sample code lines for such an add-in in Revit:
using System;
using System.Collections.Generic;
using System.Text;
using Autodesk.Revit.DB;
using Autodesk.Revit.UI;
using Autodesk.Revit.ApplicationServices;
using System.Drawing;
using Autodesk.Revit;
using System.Collections.ObjectModel;
namespace BIMCloudScore
{
// <summary>
// This class contain the data about object changes (getting from Revit)
// </summary>
public class ChangesofObjects
{
private UIApplication m_revit; //Application of Revit
private WireFrame m_sketch; //Profile information of opening
private BoundingBox m_boundingBox; //BoundingBox of Opening
public UIApplication Revit
{:::}
// Other methods not shown
:::
}
After the application is set up in the client, metric scores will be
metered locally. Then, they are sent to the benchmarking server to
J. Constr. Eng. Manage.
 Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.

Table 1. Proposed BIMCS Benchmarking Metrics

Category Explanation Metrics Measurement Notes

© ASCE
Modeling (production)
Productivity How fast a BIM Number of objects created
model was developed per week
Number of absolute object
number changes per week
Model LOD per number of
coordination meeting
Project data changes
per week
Effectiveness How effectively a Variance of QTO
BIM model was
developed Number of steps per object
Average changes per object
Model (product)
Quality Degree to which a set Number of warnings
of inherent per object
characteristics of Criticality of warnings
BIM models fulfills
desired requirements Consistency of 3D model
and 2D references
Models’ analytical
reporting quality
Number of objects/duration (weeks)
Number of absolute changes of object number/
duration (weeks)
Model LOD/number of coordination meetings
Total number of project data entries/duration
(weeks)
ðQTO produced by the modal at each phase −
average QTOÞ2 =number ofphases
Total number of modeling steps/total number of
objects
Number of object changes/number of objects
Total number of warnings/total number of objects
Number of structural warnings/number of
nonstructural warnings
Number of errors in reference to 2D deliverables/
number of objects
Number of objects need to be modified or added
before reporting /number of objects
It indicates on average how many objects were created per week
It indicates the absolute object number changes per week, which may include
created or removed objects per week
It indicates on average how many meetings were needed to increase LOD a
level
It shows on average how many nongeometric project parameters (entries) were
entered per week
Consistent QTO prediction indicates a more effective development process
It shows on average how many steps it took to develop an object; too many
steps per object indicates an ineffective development (e.g., corrections/rework)
It shows on average how many times an object was changed during the
development; more changes indicate a less effective development
More warnings indicate a less quality model
Structural warnings (e.g., warnings pertaining to beams and columns) are more
critical; this ratio shows the criticality of warnings
It reflects how consistent the BIM generated 2D drawings are in reference to
the 2D drawings in use
Before utilizing the BIM model in any analytical study (such as structural
study), if more objects need to be modified or added, it reflects a less reliable
model

04014054-6
Accuracy Degree to which the QTO accuracy Summation of absolute difference between A lagging indicator showing the ability of a BIM model in predicting quantity
BIM models BIM-yield, QTO, and the actual quantity obtained
precisely reflects the by the end of the project

J. Constr. Eng. Manage. 2014.140.

physical real world Discrepancies between
conditions of a each discipline’s models
project Average number of generic
objects per assembly
Constructability
(Clash detection)
Usefulness How useful the How often the model gets
model information accessed
and geometry is over Ease of construction
the project lifecycle documentation creation
Reliability of model data for
end users during operations
and maintenance
Economy Degree to which the File size per SF
BIM model is (at certain LOD)
constructed in the Number of objects
most cost and time created per SF
efficient manner
Number of discrepancies between each
discipline’s model per number of objects
Number of generic objects per number of
assemblies
Number of detected clashes per number of objects
Number that the model was accessed per duration
(weeks)
Number of objects that need to be added or
modified before documentation creation per
number of objects
Number of discrepancies between BIM model and
actual physical building
File size (MB) per SF of the building
Number of objects per SF of the building
It captures the situation where information in one discipline’s model
contradicts information in another discipline’s model.
This refers to the quality of the assemblies that are modeled; bigger portion of
generic objects means very little data to extract
More detected clashes indicates a less accurate model
A more accessed model indicates that it was more useful in the project lifecycle
Before generating a construction document, if more objects need to be
modified or added, it reflects a less useful model
It indicates how useful a BIM model can be used during operations and
maintenance
Larger file size in relation to the square footage of the modeled building
indicates a less economic (or more complex) model
Larger number of objects in relation to the square footage of the modeled
building indicates a less economic (or more complex) model

Note: LOD = level of development; MB = megabyte; QTO = quantity takeoff; SF = square foot.

J. Constr. Eng. Manage.

Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.

Fig. 3. Architecture of BIM cloud score

calculate overall performance scores. Following the multi-criteria
evaluation process, the final score of a BIM project is given as
a weighted summation (Fornell 1982)
X
m
Xi ¼ ωj xij ; for i ¼ 1; 2; 3; : : : ; n
j¼1
where X i is the performance score of the ith BIM project in the
database, ωj is the weight of the jth metric (obtained by standard-
izing survey results), xij is the score of the ith BIM project on the
jth metric, m is the number of metrics (in the current version, it is
20), and n is the number of BIM projects contained in the database.
xi and the scores of separate metrics are sent to the benchmarking
server as strings encoded in UTF-8 format (universal character set
transformation format—8-bit) (Consortium 2011). UTF-8 is com-
patible with ASCII and is not as complicated as UTF-16 and UTF-
32. More importantly, it is increasingly being used as the default
character encoding in application programming interfaces which
typically serve as the basis of add-ins to off-the-shelf BIM software.
ð1Þ In a similar fashion, all users will send performance metrics in-
formation to the benchmarking server, as shown in Fig. 4. Upon
reception, performance information will be first formatted and
processed in a range of structured query language (SQL) tables.
As a standard domain-specific language (DSL), SQL is suitable
for the relational database queries of the BIMCS (ISO 2008). Then,
the benchmarking server will aggregate metrics information,
denoted as X
X ¼ fX 1 ; X 2 ; X 3 ; : : : ; X n g ð2Þ

X is a collection of the performance scores of n BIM projects.

It is used to fit the probability distribution of all submitted data.
Statistical tests such as the chi-squared test are then applied to find
out the fittest PDFs (Moore 1976). A set of statistics may also be
Benchmarking
Benchmarking
results results calculated to demonstrate the pattern of the result including μ, σ,
Aggregate skewness, kurtosis, and critical percentiles that are of interest.
The users may opt into a process of benchmarking specific BIM
Benchmarking performance metrics, other than the overall performance. The set of
Server
filters shown in Fig. 5 can be set to classify the BIM projects in the
performance database so that the user can tailor their query and
focus only on their areas or aspects of interest.
Submissions and requests Given the large amount of data to be processed in BIMCS (typ-
ically gigabytes to terabytes in size), the Apache HDFS is used to
deploy the benchmarking server (Borthakur 2007). HDFS is a
standards-based open-source framework built on Google’s MapRe-
duce and Google file system (GFS). It is designed to leverage the
power of massive parallel computation to deal with Big Data,
Fig. 4. Benchmarking BIM performance
i.e., data sets that are too large and complex for traditional data

© ASCE 04014054-7 J. Constr. Eng. Manage.

J. Constr. Eng. Manage. 2014.140.

 Market Sector Annual Volume
Residential >=1 billion
Commercial
100 MM~999 MM
Industrial
50 MM ~ 99 MM
Heavy Civil
5 MM ~ 49 MM
Health Care
Other <5MM
Fig. 5. Filters of BIM cloud score
Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.
processing applications (Welch et al. 2008). Using the master-slave
structure, lots of inexpensive commodity servers are deployed to
yield scalable data processing power (Welch et al. 2008). HDFS
provides reliable interfaces to the BIMCS application and builds
high reliability and scalability of distributed systems.
Posterior Reformation
BIMCS is dynamic in nature. It evolves as information accumu-
lates. Fig. 6 illustrates how the system is reformed when sufficient
performance data is obtained in the BIM performance database. In
particular, three aspects of the system are to be improved to reflect
the pattern of performance data: (1) weights of metrics, (2) metrics,
and (3) classification.
Reform Weights of Metrics
In the survey (step 1), respondents were asked to weight the im-
portance of BIM benchmarking metrics, which are used to calculate
the BIM performance score in the earlier phase of the benchmark-
ing process. When there is a lack of existing data, this method is
easier to implement. However, the data should speak for itself
(Shannon 1948). As data starts to accumulate in the BIM perfor-
mance database, the importance of metrics can be calculated by
investigating the pattern of the data. Metric importance has
Fig. 6. Reforming BIM cloud score based on the aggregated performance data
© ASCE 04014054-8
J. Constr. Eng. Manage. 2014.140.
Project Delivery Method
Design Bid Build
CM at Risk
CM Agency
Design Build
IPD
multifold meanings in statistics: (1) metric variability (Zeleny and
Cochrane 1982) reflects how much information is contributed by a
metric. If the value of a metric is more volatile, the metric contains
more useful information (Shannon 1948) and is more important;
(2) metric independence (Datta et al. 2009) reflects the pure infor-
mation explained by a metric. If a metric is more independent of
other metrics, it contains less repeated information and thus is more
important; and (3) metric distinguishing capability (Greco et al.
2001) reflects the capability of a metric in distinguishing the differ-
ence between samples. If removing a metric will significantly
change the classification of the outcomes, the metric is more im-
portant. Following the above notion, a variety of objective weight-
ing methods can be applied to reform the weights of metrics based
on solicited performance data, including the coefficient of variation
method (Zeleny and Cochrane 1982), maximum entropy method
(Srdjevic et al. 2004), intercriteria correlation method (Diakoulaki
et al. 1995), rough set method (Greco et al. 2001), and principal
component analysis method (Sheng et al. 2007). For example, fol-
lowing Shannon’s information theory (Shannon 1948), entropy is a
measure of the system’s disorder (or uncertainty) that can be used to
quantify the expected value of information contained in a message.
If a benchmarking metric contains more information (i.e., its en-
tropy is bigger), it should be assigned a larger importance weight.
J. Constr. Eng. Manage.
 To calculate the entropy of a metric, the results of metrics should
first be standardized
xij
aij ¼ Pn ; for j ¼ 1; 2; 3; : : : ; m
i¼1 xij
where xij is the performance score of the ith BIM project on the jth
metric, m is the number of metrics (in the current version, it is 20),
and n is the number of BIM projects contained in the database.
Then, Shannon’s entropy of the jth metric is given by
Xn

1
Hj ¼ −k aij ln aij k ¼ ð4Þ
i¼1
ln n
The importance weight of the jth metric is given by
Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.
ð1 − H j Þ
wj ¼ P ð5Þ
ðm − m j¼1 H j Þ
Hagerty and Land (2007) recognized the importance of the val-
ues judgment of domain experts in multicriteria evaluation. Such
judgment should be made in an educated and structured manner.
With the collected performance data in the BIM performance data-
base, a range of methods can be employed to elicit educated opin-
ions from domain experts. These methods include the Delphi
method (Linstone and Turoff 1975) and analytical hierarchy
processing (Handfield et al. 2002). These methods are designed
to reach a consensus among a group by utilizing repetitive and in-
formed information and statistical tests. The results from these
methods will be integrated with those from the objective weighting
methods.
Reform Metrics
The initial list of benchmarking metrics may contain overlapping
information. There are likely more fundamental metrics that are
harder to identify at the early phase of benchmarking, which are
called latent variables (Bartholomew et al. 2011). Unlike observ-
able variables, latent variables are those that cannot be directly ob-
served but are rather inferred from other variables that are directly
measured (Bartholomew et al. 2011). Finding latent variables is
critical to the success of BIMCS because certain aspects of BIM
performance may be impossible to measure with a single metric
and require a linear combination of several metrics (Mulaik 1987).
Factor analysis is used to find out a new list of metrics that are
independent of each other (Var 1998). Suppose there are n BIM
project data points and m metrics. All the data in the BIM perfor-
mance database can then be denoted as an m × n matrix, Xm×n .
Then, the mathematical model of factors is
ðX − μÞm×n ¼ Lm×n Zm×n þ εm×n
where μm×n = mean matrix of X m×n , Lm×n = loading matrix, Zm×n =
factor matrix, and εm×n = matrix for independent error terms. It is
also known that
EðZm×n Þ ¼ 0 and CovðZm×n Þ ¼ I
Therefore, Lm×n can be found by solving
CovðX − μÞm×n ¼ Lm×n LTm×n þ ψ
Zm×n contains a list of new metrics that are linear combinations
of the original metrics. The new metrics are independent of each
other, and several of them assume a maximum possible variation of
the data.
© ASCE
J. Constr. Eng. Manage. 2014.140.
Reform Classification
The classification of BIM projects, such as the one shown in Fig. 5,
can be improved based on the cluster analysis of accumulated
ð3Þ information in the database (Anderberg 1973). Cluster analysis
groups outcomes in such a way that outcomes in the same group
are more similar to each other than in other groups. In the BIMCS,
it means that BIM projects with the same group membership are
more similar in terms of metric scores and overall performance.
Compared to the original classification of BIM projects as shown
in Fig. 5, the one that emerges from this data may lead to an un-
biased classification. It may provide a better guideline for users to
query the benchmarking results and focus only on their areas of
interest.
A challenge of applying cluster analysis in the BIM perfor-
mance database is of high dimensionality. Beyer et al. (1999) have
proven that distance-based cluster methods fail as dimensionality
increases. This is because when dimensionality reaches a certain
point, as few as 10–15 dimensions, the distance to the nearest data
point approaches the distance to the farthest data point
didtmax − didtmin
limd→∞ ¼ ¼0 ð9Þ
didtmin
As a result, high dimensional cluster analysis methods are used
in BIMCS to generate a new classification of BIM projects. Avail-
able methods include PROCLUS (Aggarwal et al. 1999), CLIQUE
(Agrawal et al. 1998), LAC (Domeniconi et al. 2004), COSA
(Friedman and Meulman 2004), and hybrid methods such as
DOC (Procopiuc et al. 2002) and DiSH (Achtert et al. 2007).
Because of the limitation of the scope of this study, the details
of these methods will not be discussed.
Consistency Test
Kendall’s rank correlation analysis is used in BIMCS to check
whether results from multiple weighting methods are consistent
(Kendall 1938). Kendall’s rank correlation analysis builds on a
set of statistics called Kendall’s tau (τ ) to measure the similarity
between two ranking results. The τ is obtained from
nc − nd
τ B ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð10Þ
ðn0 − n1 Þðn0 − n2 Þ
where nc = number of concordant
P pairs, nd = number of discordant
pairs, n0 ¼ nðn − 1Þ=2, n1 ¼ ti ðti − 1Þ=2 (ti is the number of
tied valuesPin the ith group of ties for the first ranking result),
and n2 ¼ uj ðuj − 1Þ=2 (uj is the number of tied values in
the jth group of ties for the second ranking result).
If Kendall’s rank correlation analysis cannot be satisfied, it sug-
gests significant discrepancy between different weighting methods.
Then, compound weighting methods are to be used to integrate
ð6Þ
the results, including the minimum bias synthesis method, the
multiplicative synthesis method, the Spearman synthesis method,
and the sum of squared deviation (SSD) based synthesis method
(Yang 2006).
ð7Þ
Implementation Example
The following example shows how the BIMCS is implemented.
ð8Þ The BIMCS can be installed as an add-in to Revit (Autodesk
2014). After installing it, a link is created under the external tools
tab of Revit (Fig. 7). There are three main functions: (1) start/
terminate monitoring, (2) start/terminate uploading information,
and (3) view benchmarking results. The first function controls
the start or end of the monitoring actions. The second function
04014054-9 J. Constr. Eng. Manage.
Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.
Fig. 7. Add-in of Revit (hypothetical case)
allows users to upload their BIM performance information to the
server, and the third function displays the benchmarking result.
After starting the monitoring function, the BIMCS will screen
and monitor the BIM database continuously and meter the scores of
each performance metric on the back end. The user’s BIM model-
ing activities will not be affected. If the start/terminate uploading
information is turned on, performance information will be uploaded
to the benchmarking server automatically on a regular basis. The
uploaded information is classified, processed, and aggregated in the
remote server.
Then, the user can view the results using the add-in. By clicking
view benchmarking result, a window will pop out and show the
results as a probability distribution curve and tabular results (Fig. 8).
The weights of metrics are also shown in the right hand side. Users
may revise the weights according to their own needs. The final re-
sult of BIMCS is shown as a percentile value (from 0 to 100%). The
percentile value shows the amount of projects that the assessed
BIM project outperforms. For example, if a BIM project’s result
is 95%, it means 95% of the assessed projects are worse than it,
or it is in the top 5%.
Filters are also provided with the results. Users may check the
benchmarking results for specific categories such as productivity
and quality. Users may also view the results for specific market
sectors such as commercial or residential.
A detailed analysis report may also be generated (Fig. 9). It pro-
vides a new window for summary presentation of the benchmark-
ing results. In the detailed analysis section, performance based on
each of the 20 metrics is reviewed and analyzed.
When sufficient data has been collected, BIMCS can be updated
automatically. One possible update is the weights of metrics. For
example, the statistical analysis on the aggregated benchmarking
data may find the variance of number of objects developed per
month to be bigger than that of other metrics. Eqs. (3)–(5) will
be used to recalculate the importance weights of the 20 metrics.
© ASCE 04014054-10
J. Constr. Eng. Manage. 2014.140.
The updated weights are sent to the BIMCS add-in automatically
with a notice to the end-users, giving the end-users the option of
adopting the new weights. This procedure can be done on a regular
basis to reflect the latest trend of BIM performance. Another sig-
nificant update of BIMCS is the list of metrics. For example, it may
be found by a factor analysis that the scores of how often the model
gets accessed are highly correlated with the scores of ease of con-
struction documentation creation, which indicates the fact that
these two metrics capture similar performance aspects and should
be combined into one. If a new list of metrics is suggested, it will be
sent to the user add-in for validation, and if new metrics are ac-
cepted, a new version of BIMCS will be launched for installation.
Discussion
The authors have introduced the architecture and implementation
method of a cloud-based application called BIMCS for benchmark-
ing BIM performance. Two unique features make the success of
this system possible:
1. BIMCS provides an open environment for BIM users, where
they can identify their competitiveness with respect to all their
industry competitors. It provides sufficient accessibility to the
practitioners and is free of charge. This feature is attractive to
the practitioners and companies who wish to continuously im-
prove their performance in BIM utilization by identifying their
gaps with the industry’s best practitioners.
2. BIMCS is able to correct/adjust itself based on the Big Data
collected nationwide. A variety of data mining techniques will
be applied to dynamically improve the benchmarking metrics,
metric importance weights, and classification method. Each
newer version of the BIMCS reflects the direct pattern of
the national BIM performance data and provides an overall
view of the status quo of the user’s BIM utilization.
J. Constr. Eng. Manage.
Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.

Fig. 8. Showing benchmarking results in Revit (hypothetical case)

Because the modeling process of BIM, such as the productivity
of object development, is of concern, it is a nontrivial task to
monitor and meter performance metrics in the BIMCS. One chal-
lenge is that time stamps should be implanted onto BIM databases
to document the dynamic change of performance metrics. For
example, to monitor the rework of a BIM model or the access fre-
quency to some BIM model, the exact time stamps of such changes
should be well documented. Therefore, an add-in called BIM
diagnostics will be developed in the future. The add-in has two
functions:
1. It creates a log file for each BIM model to record the changes,
model outcomes, and project indicators that were retrieved
from the BIM model at different times.
2. It provides BIM model forensic analysis for comparing the
difference between two BIM models or the same BIM model
at different times.
BIM diagnostics add-in should be able to seamlessly commu-
nicate to the BIM benchmarking database. The next task will be
focused on the development of BIM diagnostics. Although the
proposed application has the potential to be utilized by BIM

Fig. 9. Sample report of BIM cloud score

© ASCE 04014054-11 J. Constr. Eng. Manage.

J. Constr. Eng. Manage. 2014.140.

 coordinators (mainly employed by general contractors), it currently
targets BIM authors (mainly employed by subcontractors and de-
signers) as the main users. The proposed metrics aim to capture the
technical aspects of the development process and final products of
BIM. They should be used by BIM authors to examine and com-
pare the performance of BIM development efforts. In fact, these
metrics were developed and validated based on a survey, where
60.9% of the respondents were architects, BIM developers, or BIM
managers, and 41.3% of them were from architecture companies
compared to 13% from general contractors. Therefore, the present
version of BIMCS mainly meets the need of BIM authors.
Conclusion
In order to focus the industry’s attention on the performance of
Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.
BIM utilization, a variety of BIM performance evaluation frame-
works have been proposed. Because of the designs of these frame-
works, they are mostly used for evaluating an organization’s
performance in BIM utilization, i.e., to assess the level of BIM
utilization within an organization. In contrast, benchmarking, com-
pared to internal evaluation, was found to be more helpful for BIM
utilizers to make decisions for improvement plans. Through cross-
organizational comparison, an organization can locate its competi-
tive position among all industry peers. Then, lessons learned from
other organizations can be used to establish improvement targets
and to promote changes in the organization.
This paper introduces a cloud-based BIM performance bench-
marking application called BIMCS to automatically collect BIM
performance data from a wide range of BIM users nationwide.
Twenty validated benchmarking metrics are used to quantify BIM
utilization performance in terms of the BIM model (the product)
and BIM modeling (the production). BIMCS is a community cloud
environment that utilizes SaaS to make the collection, aggregation,
and presentation of benchmarking data autonomous and interac-
tive. Based on the Big Data collected from the BIMCS database,
an overall view of the industry status quo of BIM utilization may be
obtained, and ultimately, a protocol for BIM performance may be
developed on the basis of this improved knowledge discovery
process.
References
Achtert, E., Böhm, C., Kriegel, H.-P., Kröger, P., Müller-Gorman, I., and
Zimek, A. (2007). “Detection and visualization of subspace cluster
hierarchies.” Advances in databases: Concepts, systems and applica-
tions, Springer, Berlin, Germany, 152–163.
Aggarwal, C. C., Wolf, J. L., Yu, P. S., Procopiuc, C., and Park, J. S. (1999).
“Fast algorithms for projected clustering.” Proc., Association for Com-
puting Machinery Special Interest Group on Management of Data
(ACM SIGMOD) Record, Association for Computing Machinery,
New York, 61–72.
Agrawal, R., Gehrke, J., Gunopulos, D., and Raghavan, P. (1998).
“Automatic subspace clustering of high dimensional data for data min-
ing applications.” Proc., Association for Computing Machinery Special
Interest Group on Management of Data (ACM SIGMOD) Record,
Asoociation for Computing Machinery, New York.
Anderberg, M. R. (1973). “Cluster analysis for applications.” Defense tech-
nical information center (DTIC) document, Academic Press, New York.
Autodesk. (2014). “Revit overview.” 〈http://www.autodesk.com/products/
autodesk-revit-family/overview〉 (Jan. 15, 2014).
Barber, E. (2004). “Benchmarking the management of projects: A review of
current thinking.” Int. J. Project Manage., 22(4), 301–307.
Bartholomew, D. J., Knott, M., and Moustaki, I. (2011). Latent variable
models and factor analysis: A unified approach, Wiley, West Susses,
U.K.
© ASCE
J. Constr. Eng. Manage. 2014.140.
Bendell, T., Boulter, L., and Kelly, J. (1993). Benchmarking for competitive
advantage, Financial Times/Pitman, London.
Beyer, K., Goldstein, J., Ramakrishnan, R., and Shaft, U. (1999). “When is
‘nearest neighbor’ meaningful?” Database Theory—Int. Conf. on
Database Theory (ICDT)’99, Springer, Jerusalem, Israel, 217–235.
Bogan, C. E., and English, M. J. (1994). Benchmarking for best practices:
Winning through innovative adaptation, McGraw-Hill, New York.
Borthakur, D. (2007). “The hadoop distributed file system: Architecture
and design.” 〈http://www.lhelper.org/dev/hadoop-0.12.2/docs/hdfs
_design.pdf〉 (Jan. 15, 2014).
Camp, R. C., and Camp Robert, C. (1995). Business process benchmark-
ing: Finding and implementing best practices, American Society for
Quality Control (ASQC) Quality, Milwaukee, WI.
Center for Integrated Facility Engineering (CIFE). (2013). “VDC and BIM
scorecard.” 〈https://vdcscorecard.stanford.edu/home〉 (Jan. 15, 2014).
Chen, Z., Chen, J., Shen, F., and Lee, Y. (2013). “Collaborative mobile-
cloud computing for civil infrastructure condition inspection.”
J. Comput. Civ. Eng., 10.1061/(ASCE)CP.1943-5487.0000377,
04014066.
Chuang, T.-H., Lee, B.-C., and Wu, I.-C. (2011). “Applying cloud comput-
ing technology to BIM visualization and manipulation.” Proc., 28th Int.
Symp. on Automation and Robotics in Construction, International
Association for Automation and Robotics in Construction (IAARC),
Bratislava, Slovakia, 144–149.
Construction Excellence. (2013). “Construction best practice programme.”
〈http://www.constructingexcellence.org.uk〉 (Jan. 15, 2014).
Construction Industry Institute. (2013). “CII benchmarking and
metrics.” 〈https://www.construction-institute.org/nextgen/index.cfm〉
(Jan. 15, 2014).
Costa, D., Formoso, C., Kagioglou, M., Alarcón, L., and Caldas, C. (2006).
“Benchmarking initiatives in the construction industry: Lessons learned
and improvement opportunities.” J. Manage. Eng., 10.1061/(ASCE)
0742-597X(2006)22:4(158), 158–167.
Datta, S., Nandi, G., Bandyopadhyay, A., and Pal, P. K. (2009). “Applica-
tion of PCA-based hybrid Taguchi method for correlated multicriteria
optimization of submerged arc weld: A case study.” Int. J. Adv. Manuf.
Technol., 45(3–4), 276–286.
Diakoulaki, D., Mavrotas, G., and Papayannakis, L. (1995). “Determining
objective weights in multiple criteria problems: The critic method.”
Comput. Oper. Res., 22(7), 763–770.
Domeniconi, C., Papadopoulos, D., Gunopulos, D., and Ma, S. (2004).
“Subspace clustering of high dimensional data.” Proc., 2004 SIAM
(Society for Industrial and Applied Mathematics) Int. Conf. on Data
Mining, SIAM, Philadelphia, PA.
Fornell, C. (1982). A second generation of multivariate analysis. 2.
Measurement and evaluation, Praeger Publishers, Westport, CT.
Friedman, J. H., and Meulman, J. J. (2004). “Clustering objects on subsets
of attributes.” J. R. Stat. Soc. Series B (Stat. Method.), 66(4), 815–849.
Garvin, D. A. (1993). “Building a learning organization.” Harv. Bus. Rev.,
71, 78–91.
Gong, J., and Azambuja, M. (2013). “Visualizing construction supply
chains with google cloud computing tools.” Proc., ICSDEC 2012:
Developing the Frontier of Sustainable Design, Engineering, and
Construction, ASCE, Reston, VA, 671–678.
Gracia, J., and Bayo, E. (2013). “Integrated 3D web application for struc-
tural analysis software as a service.” J. Comput. Civ. Eng., 10.1061/
(ASCE)CP.1943-5487.0000217, 159–166.
Greco, S., Matarazzo, B., and Slowinski, R. (2001). “Rough sets theory
for multicriteria decision analysis.” Eur. J. Oper. Res., 129(1), 1–47.
Hagerty, M. R., and Land, K. C. (2007). “Constructing summary indices of
quality of life a model for the effect of heterogeneous importance
weights.” Socio. Meth. Res., 35(4), 445–496.
Handfield, R., Walton, S. V., Sroufe, R., and Melnyk, S. A. (2002).
“Applying environmental criteria to supplier assessment: A study in
the application of the analytical hierarchy process.” Eur. J. Oper.
Res., 141(1), 70–87.
International Organization for Standardization (ISO). (2008). “Information
technology—Database languages—SQL—Part 1: Framework (SQL/
framework).” ISO/International Electrotechnical Commission (IEC)
04014054-12 J. Constr. Eng. Manage.
 9075-1:2008, Nederlands Normalisatie Institute, ISO, Geneva,
Switzerland.
Issa, R. R., and Suermann, P. (2009). “Evaluating industry perceptions of
building information modeling (BIM) impact on construction.” J. Inf.
Technol. Constr., 14(2009), 574–594.
Karamouz, M., Zahmatkesh, Z., and Saad, T. (2013). “Cloud computing in
urban flood disaster management.” Proc., World Environmental and
Water Resources Congress 2013, ASCE, Reston, VA.
Kendall, M. G. (1938). “A new measure of rank correlation.” Biometrika,
30(1–2), 81–93.
Krasner, G. E., and Pope, S. T. (1988). “A description of the model-view-
controller user interface paradigm in the smalltalk-80 system.” J. Object
Oriented Program., 1(3), 26–49.
Kumar, B., Cheng, J. C., and McGibbney, L. (2010). “Cloud computing
and its implications for construction IT.” Proc., Int. Conf. in Computing
in Civil and Building Engineering, W. Tizani, ed., International Society
for Computing in Civil and Building Engineering, Nottingham, U.K.
Downloaded from ascelibrary.org by Rui Liu on 02/12/15. Copyright ASCE. For personal use only; all rights reserved.
Latu, K., Swain, N., Christensen, S., Jones, N., Nelson, E., and Williams,
G. (2013). “Essential GIS technologies for hydrologic simulation appli-
cations in cloud computing.” Proc., World Environmental and Water
Resources Congress 2013, ASCE, Reston, VA.
Liang, F., and Luo, Y. (2013). “A framework of the civil engineering CAD
experimental platform based on cloud computing.” Proc., Int. Conf. on
Construction and Real Estate Management 2013: Construction and
Operation in the Context of Sustainability, ASCE, Reston, VA.
Linstone, H. A., and Turoff, M. (1975). The delphi method, Addison-
Wesley Reading, Boston, MA.
Liu, J., Wang, H., Ge, Y., and Huang, J. (2013). “Application of multi-
source information fusion technology in the construction of a secure
and emergent transportation platform.” Proc., Int. Conf. on Transpor-
tation Information and Safety 2013: Improving Multimodal Transpor-
tation Systems—Information, Safety, and Integration, ASCE, Reston,
VA, 237–243.
Liu, R. (2012). “BIM-based life cycle information management: Integrating
knowledge of facility management into design.” Ph.D. thesis, Univ. of
Florida, Gainesville, FL.
Lohuis, M. M., Dekkers, J., and Smith, C. (1992). “Probability of success
and predicted returns of sires in progeny test programs.” J. Dairy Sci.,
75(6), 1660–1671.
Marinescu, D. C. (2013). Cloud computing: Theory and practice, Elsevier,
Philadelphia, PA.
Marosszekey, M., and Karim, K. (1997). “Benchmarking—A tool for lean
construction.” Proc., 5th Int. Conf. of the International Group for Lean
Construction, International Group for Lean Construction (IGLC), Gold
Coast, Australia, 157–167.
Mell, P., and Grance, T. (2011). “The NIST definition of cloud computing.”
National Institute of Standards and Technology (NIST) special
publication, Vol. 800, Gaithersburg, MD, 7.
Moore, D. S. (1978). “Chi-square tests.” Studies in statistics, R. V. Hogg,
ed., Vol. 19, Mathematical Association of America, Washington, DC.
Mulaik, S. A. (1987). “A brief history of the philosophical foundations
of exploratory factor analysis.” Multivariate Behav. Res., 22(3),
267–305.
National Institute of Building Sciences (NIBS). (2007). “National building
information modeling standard—Version 1.0.” NIBS Rep., Washington,
DC.
National Institute of Building Sciences (NIBS). (2012). “National building
information modeling standard—Version 2.0—Chapter 5.2 minimum
BIM.” NIBS Rep., Washington, DC.
Nieh, J., Yang, S. J., and Novik, N. (2000). “A comparison of thin-client
computing architectures.” Technical Rep. CUCS-022-00, Dept. of
Computer Science, Columbia Univ., New York.
Procopiuc, C. M., Jones, M., Agarwal, P. K., and Murali, T. (2002).
“A Monte Carlo algorithm for fast projective clustering.” Proc., 2002
Association for Computing Machinery Special Interest Group on
Management of Data (ACM SIGMOD) Int. Conf. on Management of
© ASCE 04014054-13
View publication stats J. Constr. Eng. Manage. 2014.140.
Data, Association for Computing Machinery (ACM), New York,
418–427.
Ramírez, R. R., Alarcón, L. F. C., and Knights, P. (2004). “Benchmarking
system for evaluating management practices in the construction
industry.” J. Manage. Eng., 10.1061/(ASCE)0742-597X(2004)20:
3(110), 110–117.
Redmond, A., Hore, A., Alshawi, M., and West, R. (2012). “Exploring how
information exchanges can be enhanced through cloud BIM.” Autom.
Constr., 24(2012), 175–183.
Rossi, P. H., Lipsey, M. W., and Freeman, H. E. (2004). Evaluation:
A systematic approach, Sage, Thousand Oaks, CA.
Sargent, R. (2005). “Verification and validation of simulation models.”
Winter Simulation Conf., WSC, 130–143.
Sebastian, R., and van Berlo, L. (2010). “Tool for benchmarking BIM
performance of design, engineering and construction firms in the
Netherlands.” Archit. Eng. Des. Manage., 6(4), 254–263.
Senescu, R., Haymaker, J., Meza, S., and Fischer, M. (2014). “Design pro-
cess communication methodology: Improving the effectiveness and
efficiency of collaboration, sharing, and understanding.” J. Archit.
Eng., 10.1061/(ASCE)AE.1943-5568.0000122, 05013001.
Shannon, C. (1948). “A mathematical theory of communication.” Bell Syst.
Tech. J., 27(3), 379–423.
Sheng, Z., Sun, S., Wang, J., and Chu, W. (2007). “Comprehensive evalu-
ation of river water environmental quality based on the principal
component analysis.” Environ. Sci. Manage., 32(12), 172–175.
Srdjevic, B., Medeiros, Y., and Faria, A. (2004). “An objective multi-
criteria evaluation of water management scenarios.” Water Resour.
Manage., 18(1), 35–54.
Subashini, S., and Kavitha, V. (2011). “A survey on security issues in ser-
vice delivery models of cloud computing.” J. Network Comput. Appl.,
34(1), 1–11.
Succar, B. (2009). “Building information modelling framework: A research
and delivery foundation for industry stakeholders.” Autom. Constr.,
18(3), 357–375.
Succar, B. (2010). “Building information modelling maturity matrix.”
Handbook of research on building information modelling and construc-
tion informatics: Concepts and technologies, J. Underwood and
U. Isikdag, eds., IGI, Hershey, PA, 65–103.
Succar, B., Sher, W., and Williams, A. (2012). “Measuring BIM perfor-
mance: Five metrics.” Archit. Eng. Des. Manage., 8(2), 120–142.
Succar, B., Sher, W., and Williams, A. (2013). “An integrated approach to
BIM competency assessment, acquisition and application.” Autom.
Constr., 35(2013), 174–189.
Suermann, P., Issa, R., and McCuen, T. (2008). “Validation of the
U.S. national building information modeling standard interactive
capability maturity model.” Proc., 12th Int. Conf. on Computing in Civil
and Building Engineering, International Society for Computing in Civil
and Building Engineering, Nottingham, U.K.
Suermann, P. C. (2009). “Evaluating the impact of building information
modeling (BIM) on construction.” Ph.D. dissertation, Univ. of Florida,
Gainesville, FL.
Underwood, J., and Isikdag, U. (2010). Handbook of research on building
information modeling and construction informatics: Concepts and
technologies, Information Science Reference.
Unicode Consortium. (2011). “Unicode 6.0.0.” 〈http://www.unicode.org/
versions/Unicode6.0.0/〉 (Jan. 15, 2014).
Var, I. (1998). “Multivariate data analysis.” Vectors, 8(2), 125–136.
Welch, B., et al. (2008). “Scalable performance of the panasas parallel file
system.” Proc., File and Storage Technologies, USENIX (Advanced
Computing Systems Association), Berkeley, CA, 1–17.
Yang, Y. (2006). “Weighting methods in multivariate evaluation.” Stat.
Decis., 2006(13), 17–19.
Zeleny, M., and Cochrane, J. L. (1982). Multiple criteria decision making,
McGraw-Hill, New York.
Zhang, L., and Issa, R. (2012). “Comparison of BIM cloud computing
frameworks.” Proc., Computing in Civil Engineering (2012), ASCE,
Reston, VA, 389–396.
J. Constr. Eng. Manage.