Crowdsourcing Construction Activity Analysis

from Jobsite Video Streams

Kaijian Liu, S.M.ASCE 1; and Mani Golparvar-Fard, A.M.ASCE 2
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.

Abstract: The advent of affordable jobsite cameras is reshaping the way on-site construction activities are monitored. To facilitate the
analysis of large collections of videos, research has focused on addressing the problem of manual workface assessment by recognizing
worker and equipment activities using computer-vision algorithms. Despite the explosion of these methods, the ability to automatically
recognize and understand worker and equipment activities from videos is still rather limited. The current algorithms require large-scale
annotated workface assessment video data to learn models that can deal with the high degree of intraclass variability among activity cat-
egories. To address current limitations, this study proposes crowdsourcing the task of workface assessment from jobsite video streams.
By introducing an intuitive web-based platform for massive marketplaces such as Amazon Mechanical Turk (AMT) and several automated
methods, the intelligence of the crowd is engaged for interpreting jobsite videos. The goal is to overcome the limitations of the current
practices of workface assessment and also provide significantly large empirical data sets together with their ground truth that can serve
as the basis for developing video-based activity recognition methods. Six extensive experiments have shown that engaging nonexperts
on AMT to annotate construction activities in jobsite videos can provide complete and detailed workface assessment results with 85% ac-
curacy. It has been demonstrated that crowdsourcing has the potential to minimize time needed for workface assessment, provides ground
truth for algorithmic developments, and most importantly allows on-site professionals to focus their time on the more important task of
root-cause analysis and performance improvements. DOI: 10.1061/(ASCE)CO.1943-7862.0001010. © 2015 American Society of Civil
Engineers.
Author keywords: Activity analysis; Construction productivity; Video-based monitoring; Workface assessment; Crowdsourcing;
Information technologies.

Introduction on-site observations that is needed to guarantee statistically signifi-

On-site operations are among the most important factors that
influence the performance of a construction project (Gouett et al.
2011). Timely and accurate productivity information on labor and
equipment involved in on-site operations can bring an immediate
awareness of specific issues to construction management. It also
empowers them to take prompt corrective actions, thus avoiding
costly delays. To streamline the cyclical procedure of measuring
and improving the direct-work rates, the time proportion of activ-
ities devoted to actual construction, the Construction Industry
Institute (CII) recently proposed new procedures for conducting
activity analysis (CII 2010). Activity analysis offers a plausible
solution for monitoring on-site operations and supports root-
cause analysis on the issues that adversely affect their produc-
tivity. Nevertheless, the current procedure for implementing
activity analysis has inefficiencies that prevents a wide-spread
adoption. The limitations are (1) the large scale of the manual
1 (e.g., Peddi et al. 2009; Teizer and Vela 2009; Rezazadeh Azar et al.
Graduate Student, Dept. of Civil and Environmental Engineering,
Univ. of Illinois at Urbana-Champaign, Newmark Civil Engineering
Laboratory, 205 N. Mathews Ave., Urbana, IL 61801. E-mail: kliu15@
illinois.edu
2
Assistant Professor and National Center for Supercomputing Applica-
tions (NCSA) Faculty Fellow, Dept. of Civil and Environmental Engineer-
ing and Dept. of Computer Science, Univ. of Illinois at Urbana-Champaign,
Newmark Civil Engineering Laboratory, 205 N. Mathews Ave., Urbana, IL
61801 (corresponding author). E-mail: mgolpar@illinois.edu
Note. This manuscript was submitted on October 31, 2014; approved on
March 26, 2015; published online on May 29, 2015. Discussion period open
until October 29, 2015; separate discussions must be submitted for individual
papers. This paper is part of the Journal of Construction Engineering and
Management, © ASCE, ISSN 0733-9364/04015035(19)/$25.00.
cant workface data; and (2) the necessary visual judgments of
the observers that may produce erroneous data because of the
over-productiveness phenomenon caused by construction workers
under direct observations, instantaneous reaction of the observers
to benchmarking activity categories, the necessary distance limits
to construction workers, and finally, observers’ bias and fatigue
(Khosrowpour et al. 2014b). Current labor-intensive processes take
away time from determining the root causes of issues that affect
productivity and how productivity improvements can be planned
and implemented (CII 2010).
To address limitations of manual workface assessment, a large
body of research has focused on methods that lead to automation.
These methods range from application of ultra wide band (Cheng
et al. 2011; Teizer et al. 2007; Giretti et al. 2009), radio frequency
identification (RFID) tags (Costin et al. 2012; Zhai et al. 2009), and
global positioning system (GPS) sensors (Pradhananga and Teizer
2013; Hildreth et al. 2005) to vision methods using video streams
2012). The majority of these methods build on nonvisual sensors
and track the location of the workers and equipment. However,
without interpreting the activities and purely based on location in-
formation, deriving workface data is challenging (Khosrowpour
et al. 2014a). For example, for drywall activities, distinguishing
between direct-work or tool-time purely based on location is diffi-
cult because during these activities, the location of a worker may
not necessarily change.
In contrast to location tracking sensors, the growing number
of cameras on jobsites and the rich information available in site
videos provide a unique opportunity for automated interpretation
of productivity data. Nevertheless, computer-vision methods are

© ASCE 04015035-1 J. Constr. Eng. Manage.

J. Constr. Eng. Manage., 2015, 141(11): 04015035

 not advanced enough to enable detailed assessments from these
videos. This is because the methods for detection and tracking
equipment and workers especially when workers interact with tools
(e.g., Brilakis et al. 2011; Park and Brilakis 2012; Memarzadeh
et al. 2013) and the methods for interpreting activities from long
sequences of videos (e.g., Gong et al. 2011; Golparvar-Fard et al.
2013) are not mature. Beyond the CII-defined activities, the tax-
onomy of construction activities are also not fully developed to
enable visual activity recognition at the operational level. Finally,
training and testing models used in computer-vision methods for
activity analysis requires large amount of empirical data which
is not yet available to the research community. In the absence of
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.
efficient video interpretation methods, tedious manual reviewing
will still be required to extract productivity information from re-
corded videos, and this takes away time from the more important
task of conducting root-cause analysis.
In this paper, a new workface assessment framework is intro-
duced which provides an easy way to collect and interpret accurate
labor and equipment activity information from jobsite videos. The
idea is simple: The task of workface assessment is crowdsourced
from jobsite video streams. By introducing an intuitive web-based
platform on Amazon Mechanical Turk (AMT), the intelligence of
the crowd is engaged for interpreting jobsite videos. The goal is to
overcome the limitations of the current practices of activity analysis
and also provide significantly large empirical data sets together
with their ground truth that can serve as the basis for developing
automated video-based activity recognition methods. Through ex-
tensive validation on various parameters of the platform, it is shown
that engaging nonexperts on AMT to annotate construction activ-
ities on jobsite videos can achieve accurate workface assessment
results. In the following section, the related works are reviewed,
methods are introduced to develop this proposed tool, and exper-
imental results are discussed.
Related Work
Time-lapse photography and videotaping have proven for many
years to be very useful means for recording workface activities
(Golparvar-Fard et al. 2009). Since the earlier work of Oglesby et al.
(1989) until now, many researchers have proposed procedures,
guidelines, and also manual and semiautomated methods for inter-
pretation of jobsite video data. Videos have the advantage of being
understandable by any visually-able person, provide detailed and
dependable information, and allow detailed reviews by the analysts
and on-site management away from the work sites. In the next sec-
tion, some of the most relevant works on the topics of video-based
workface assessment are first reviewed. Then, a review on the con-
cept of crowdsourcing is provided, followed by research on video
annotation tools and existing databases for activity recognition.
Computer-Vision Methods for Video-Based
Construction Activity Analysis
Over the past few years, many computer-vision methods have
emerged for inferring the activities of workers and equipment from
jobsite videos. A reliable method for video-based activity analysis
requires two interdependent components: (1) methods for detecting
and tracking resources; and (2) procedures for activity recognition.
The majority of the previous works have addressed these compo-
nents as two separate tasks. Brilakis et al. (2011) and Park et al.
(2011) applied scale invariant feature transforms to track construc-
tion resources in both 2D and 3D scenarios. Teizer and Vela (2009),
Gong and Caldas (2009), Rezazadeh Azar and McCabe (2012), and
Chi and Caldas (2011) explored construction resources detection
© ASCE 04015035-2
J. Constr. Eng. Manage., 2015, 141(11): 04015035
and tracking by using different visual feature representations
and learning algorithms. Park and Brilakis (2012) and Memarzadeh
et al. (2013) explored a more structured approach by using machine
learning techniques and shape and color templates for detection and
tracking. Kim and Caldas (2013) also explored joint modeling of
worker actions and construction objects as the basis of construction
worker activity recognition.
A few studies have also been focused on end-to-end activity
analysis. For example, Gong and Caldas (2009) presented a con-
crete bucket model trained by boosted cascade simple features
to analyze its cyclic operations. Yang et al. (2014) explored a
finite-state machine model to classify tower-crane’s activities into
concrete pouring and nonconcrete material movements. Azar and
McCabe (2012) introduced a logical framework using computer
vision-based techniques to study construction equipment working
cycles. Instead of assuming strong priors on the relationships be-
tween activities and locations as in the previous works, Gong et al.
(2011), Golparvar-Fard et al. (2013), and Escorcia et al. (2012) pro-
posed bag-of-words models with different discriminative and gen-
erative classifiers to recognize atomic activities of construction
workers and equipment. By recognizing atomic activities, activities
observed are classified in self-contained videos where in a single
resource starts an activity and ends the same activity within the
video. Khosrowpour et al. (2014b) is also one of the earliest at-
tempts to recognize the full sequence of activities for multiple
workers from red-green-blue-depth (RGB-D) data. RGB-D data
collected from depth sensors such as Microsoft Kinect bypass
the challenges in detection and tracking and provide sequences
of worker-body skeleton which can be the input to such activity
recognition methods. Jaselskis et al. (2015) also proposed an ap-
proach of monitoring construction projects in the field by off-site
personal through live video streams. Despite the explosion of these
methods, the ability to automatically recognize and understand
worker and equipment activities is still limited. Challenges relate
to the large existing variability in execution of construction oper-
ations, the lack of formal taxonomies for construction activities
in terms of expected worker/equipment roles, and sequence of ac-
tivities. The complexity of the visual stimuli in activity recognition
in terms of camera motion, occlusions, background clutter, and
viewpoint changes are other existing challenges. Finally and most
importantly, there is lack of data sets together with ground truth
for more exhaustive research on automated activity recognition
methods.
Crowdsourcing
To overcome the current limitations in the standard task of activity
recognition, the computer-vision community has recently initiated
several projects to investigate the potential of crowdsourcing.
Crowdsourcing refers to collaborative participation of a crowd
of people to help solve a specific problem and typically involves
a rewarding mechanism, for example, paying for participation
(Howe 2008). In recent years, Internet has enabled crowdsourcing
in a broadened and a more dynamic manner. Crowdsourcing from
anyone, anywhere, as needed is now common for image and video
processing, information gathering, and data verification to creative
tasks such as coding, analytics, and production development
(Wightman 2010; Yuen et al. 2011; Shingles and Trichel 2014).
The wide range of business in crowdsourcing has also promoted
the development of specialized platforms:
• simple, microtasks-oriented crowdsourcing: Amazon Mechani-
cal Turk and Elance;
• complicated, experience-oriented crowdsourcing: 10EQS and
oDesk;
J. Constr. Eng. Manage.
 • open-ended, creative crowdsourcing: IdeaConnection and
Innocentive; and
• funding, consumption, and contribution crowdsourcing: Indie-
gogo, Kickstarter, and Wikipedia (Shingles and Trichel 2014).
Among all platforms, the Amazon Mechanical Turk, introduced
by Amazon in 2005, has gained popularity within the computer-
vision community. Traditionally, computer-vision researchers had
to manually create substantially large amounts of annotations
(i.e., labeled training data). This was always considered as a simple
but labor-intensive and costly process. However, with the assistance
of AMT marketplace, researchers, known as requester, only need
to post entire annotation tasks in the form of microtasks or human
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.
intelligence tasks (HITs) and compensate online users, known as
worker or annotator, with predefined payment for the rendered re-
sults. Through crowdsourcing, easy human annotation tasks which
are extremely difficult or even impossible for computers to perform
can be accomplished in a timely manner. This is by leveraging the
AMT’s massive labor market which contains hundreds of thou-
sands of annotators who can complete 60% of HITs in 16 h and
complete 80% of HITs around 64 h (Ipeirotis 2010). The task can
also be done at a low price of $5 per hour on the AMT. A reason-
able quality can also be achieved. This is because the AMT keeps
track of the annotators performance and uses that as a screening
condition to guarantee appropriate level of quality in results. Also,
many techniques are developed to assist requesters with getting
high-quality annotations.
Sorokin and Forsyth (2008) is the first within the computer-
vision community to present a data annotation framework to
quickly obtain inexpensive project-specific annotations through
the AMT crowdsourcing platform. The proposed framework revo-
lutionized large-scale static image annotation (Vondrick et al.
2013). The subsequent efforts to seek the value of massive data sets
of labeled images promoted design of efficient visual annotation
tools and databases. Russell et al. (2008) introduced LabelMe as
a web-based annotation tool that supports dense polygon labeling
on static image. Deng et al. (2009) presented a crowdsourcing im-
age annotation platform based on ImageNet that is an image data-
base of over 11 million images. Everingham et al. (2010) described
a high-quality image collection strategy for the PASCAL visual
object classes (VOC) challenge.
Crowdsourcing Video Annotations
Despite the success of crowdsourcing annotations for still imagery,
the dynamic nature of videos makes their annotation more chal-
lenging (Vondrick et al. 2013). Video annotation requires cost-
aware and efficient methods instead of frame-by-frame labeling
(Vondrick et al. 2013; Wah 2006). Ealier works such as ViPER-
GT tool (Doermann and Mihalcik 2000) gathered groundtruth
video data without any intelligent method for assisting the anno-
tation task (Di Salvo et al. 2013). Obviously, the significant num-
ber of frames in a video requires smarter ways for propagating
annotations from a subset of keyframes; otherwise, a method will
not be scalable. More recently, Yuen et al. (2009) introduced
LabelMe Video which employs linear interpolation with constant
3D-velocity assumption to propagate nonkeyframe spatial annota-
tions in a video. Ali et al. (2011) presented FlowBoost, a tool that
can annotate videos from a sparse set of keyframe annotations.
Kavasidis et al. (2012) proposed GTTool to support automatic con-
tour extraction, object detection, and tracking to assist annotations
across video frame sequences. Vondrick et al. (2013) presented
a video annotation tool, VATIC, to address the nonlinear motions
in video. Beyond spatial aspects of video annotation, recently, re-
searchers have also focused on extracting temporal information
© ASCE 04015035-3
J. Constr. Eng. Manage., 2015, 141(11): 04015035
for video annotation. Gadgil et al. (2014) presented a web-based
video annotation system to analyze real-time surveillance videos
by annotating each video events’ time interval for forensic analy-
sis. Heilbron and Niebles (2014) introduced an automatic video
retrieval system to annotate the time interval of videos that contain
the interested activities. Kim et al. (2014) presented ToolScape for
crowdsourcing temporal annotation in videos.
Despite popularity and the benefits of crowdsourcing computer-
vision tasks, directly applying it to the task of video-based con-
struction workface assessment can be challenging. Daily site videos
exhibit different number of workers and equipment for various op-
erations. Workers continuously interact with different tools, and
both workers and equipment exhibit changing body postures even
when the same activity is being performed. These issues beyond
typical challenges in the task of activity recognition could nega-
tively affect the optimal length of annotation tasks or the number
of necessary keyframes for annotation. Also, because of the com-
plexity of construction operations and the lack of formal categori-
zation beyond CII defined activities, crowdsourcing video-based
workface assessment necessitates a taxonomy and customized data
frameworks to describe construction activities. Finally, the reliabil-
ity of crowdsourcing for video-based construction workface assess-
ment has not been validated. Particularly recruiting nonexpert
annotators from a crowdsourcing marketplace such as AMT may
negatively affect the quality of the assessment results. Beyond tech-
nical challenges, detailed experiments are necessary to examine the
capability of nonexperts against expert control groups and also de-
vise strategies for improving the accuracy of crowdsourcing tasks.
Method
To address the challenges of applying crowdsourcing to video-
based construction workface assessment task, a goal was set on
creating a new web-based platform. The nonexpert crowd and spe-
cifically Amazon Mechanical Turk is heavily relied on to conduct
workface assessment. Customized user interfaces have been de-
signed to enable construction productivity data retrieval, visualiza-
tion, and crossvalidation. Exhaustive experiments are designed and
conducted to fine tune parameters of the tool including annotation
method, annotation frequency, and video length for crowdsourcing
video-based construction workface assessment as individual tasks
on the AMT. A compositional structure taxonomy has also been
created for construction activities to decode complex construction
operations. The performance of the experts and nonexpert annota-
tors for detecting, tracking, and recognizing activities are also ex-
haustively analyzed. Applying crossvalidation methods to improve
workface assessment accuracy is also investigated. Particularly,
several experiments are conducted to seek optimal fold number for
the crossvalidation process. Fig. 1 shows an overview of the work-
flows involved in leveraging the proposed crowdsourcing platform
for workface assessment. A companion video of this manuscript
also shows various functionalities. In the following section, these
modules and experiments are presented in more detail.
Collecting Jobsite Videos
Because of the lack of precedent experience on how to design and
validate a crowdsourcing platform specific to video-based con-
struction workface assessment, it is necessary to use real-world
jobsite videos. This allows careful examination of the platform’s
performance throughout the entire design and validation process.
Videos chosen for collection focus on concrete placement opera-
tions which contain a range of visually complex activities and
are common on almost all projects. This provides various validation
J. Constr. Eng. Manage.
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.

Fig. 1. Workflow in the crowdsourcing workface assessment tool (images by authors)

scenarios for examining the application of the workface assessment
tool. The collection of videos follows two principles: (1) to guar-
antee the videos cover various jobsite and video-recording condi-
tions; and (2) the videos exhibit different level of difficulty for
activity annotations. In this research, activity difficulty is defined
as physical and reasoning efforts that are required to complete
video-based workface assessment using the workface assessment
tool.
To quantify the difficulty level in video annotations, the follow-
ing criteria is introduced:
• construction activity conditions: the size of the construction
crew, and the frequency of changes in sequence of activities
conducted by individual craft workers;
• visibility conditions: different occlusion conditions, illumina-
tion condition, and background clutter; and
• recording conditions: camera viewpoint, distance, and camera-
motion conditions.
These criteria help determine the amount of the physical effort
and reasoning that is needed to perform the assessment tasks.
As shown in Table 1, the collected videos exhibit a large range
of changes based on these criteria. This allows the capability of
conducting construction workface assessment to be thoroughly
investigated under different conditions.
Real-world videos of concrete placement operations from three
different construction sites were collected. For validating the plat-
form, a total of eight job site videos (45 min) were chosen. Based
on the criteria defined in Table 1, these videos were classified into
categories as easy, normal, and hard. Fig. 2 shows example snap-
shots from these videos and their levels of difficulty. The collected
concrete placement videos cover almost all types of direct work,
such as place concrete, erect rorm, position rebar, etc., and
substantial amounts of nondirect work, such as preparatory work,
material handling, waiting, etc.
Activity Analysis User Interface
The workface assessment tool has two main interfaces for task
management and workface assessment. Task management interface
assists requesters—site managers and engineers, or researchers—to
manage workface assessment tasks including task publication and
result retrieval. The workface assessment interface provides anno-
tators access to complete video-based workface assessment tasks.
As shown in Fig. 1, first, the requesters use task management inter-
face to break an entire video of construction operations into several
human intelligent tasks (HITs) and publish them online or offline.
Online mode allows the AMT annotators to complete HITs,
whereas offline mode makes HITs only accessible to a predefined
group of users which can, for example, include expert annotators
invited by the requesters. Before publishing the videos for annota-
tion, for privacy purposes, the videos can be processed using
image analysis methods to automatically blur human faces when
needed. Once logged in, the annotators can then accept published
HITs to generate workface assessment results using the Workface
Assessment Interface. When all HITs belonging to the same video
are completed, the requesters can then retrieve, visualize, and cross-
validate assessment results. They can also generate formal assess-
ment reports through the task management interface.
To access the task management interface, a requester should
provide a set of valid username and password, as shown in Fig. 3.
This verification step can address copyright and privacy issues of

Table 1. Various Criteria Used to Select the Candidate Videos for Experiments
Activity conditions Visibility conditions Recording conditions
Changes in Background Camera Camera
Videos Crew size (person) activities Occlusion Illumination clutter Viewpoint distance (m) motion
Easy videos 2 Low Rare Daylight Low Level 3–15 None
4–5 Low Rare Daylight Normal Level 5–15 None
3–5 Low Rare Cloudy Low Level 15–25 Rare
Normal videos 4–7 Medium Medium Daylight Medium Level 3–25 Rare
6–9 High Rare Daylight Medium Level 5–25 Rare
8 Low Severe Sunny Varied Level 5–25 Severe
Hard videos 10–12 Medium Severe Daylight Sever Tilted 15–45 Rare

© ASCE 04015035-4 J. Constr. Eng. Manage.

J. Constr. Eng. Manage., 2015, 141(11): 04015035

Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.

Fig. 2. Videos exhibit different levels of difficulty for video annotation purposes (images by authors)

any uploaded video to protect authentic requesters from sharing
videos under their accounts to unwanted parties. Once passed the
verification step, the requester can use the following functions as-
sociated with the task management interface:
• Video links: This function assists the requesters in managing the
progress of the annotators and control the quality of their work
on the workface assessment HITs. To do so, the HITs are
presented by unique names and hyperlinks and are also classi-
fied in two categories of published and completed.
• Video upload: The tool is prototyped to simultaneously support
crowdsourcing workface assessment and collection of large-
scale ground-truth data set for both academia and industry.
Thus, allowing requesters to upload their own videos is integral
to the design of task management interface. To achieve this,
video upload is designed, shown in Fig. 4, as a function for the
requesters to upload their videos and also automatically break
them into several HITs based on a desired length. This function
also associates most frequently used labels to the HITs and then
publishes them online or offline. Most frequently used labels are
compositional structure taxonomies that describe the activities
of the workers or equipment in a video.
• Crossvalidation and accuracy: The online mode for the tool
is prototyped to leverage the knowledge of the nonexpert anno-
tators from the AMT marketplace. To avoid using inaccurate
results produced by these annotators, quality assurance and con-
trol should be applied to both preassessment and postassessment
steps. To do so, crossvalidation and accuracy is introduced as
functions that report the accuracy of the completed assessment

Fig. 3. Task management interface: (a) log in; (b) video links

© ASCE 04015035-5 J. Constr. Eng. Manage.

J. Constr. Eng. Manage., 2015, 141(11): 04015035

Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.
Fig. 4. Task management interface: (a) video upload; (b) crossvalidation and accuracy
against groundtruth at the postassessment step. They also allow
the requester to crossvalidate different completed assessments
for the same video to achieve a more accurate assessment result.
Fig. 4 presents the interface for these functions.
• Video visualization: To retrieve activity analysis information
at any desired level of granularity, video visualization presents
the workface assessment results in forms of annotated videos,
crew-balance charts, or pie charts. The annotated videos, shown
in Fig. 5, contain worker/equipment trajectories and activity
type information which are both annotated on the videos. The
experts can monitor specific operations and conduct the root-
cause analysis for hidden issues that may not easily be re-
vealed by other visualization forms. The crew-balance charts
represent the time series of worker activities. Unlike annotated
videos, the time series provide a depiction of the construction
operations in a concise manner. Finally, the pie charts charac-
terize the percentage of time spent on various activity cate-
gories, based on the CII 2010 taxonomy. This method is also
capable of accurately examining the hourly average percentages
for the overall jobsite which can provide significant benefit to
activity analysis.
This worface assessment interface is where the annotators work
on the assigned HITs. It is the most important component of
the tool as it is where the actual workface assessment happens.
To achieve simplicity and functionality in the design of the inter-
face, structured rules are followed. The interface consists of three
main components: (1) video player, (2) label drop-down list, and
(3) assisting functions, shown in Fig. 6. The player–taking the
most space–shows video content that requires assessment. The
drop-down list contains labels that are presented hierarchically
to describe activities of each worker or equipment. Assisting
© ASCE 04015035-6
J. Constr. Eng. Manage., 2015, 141(11): 04015035
functions are located above or under the video player and label
drop-down list. The main functions include
• Workface assessment function: Using this function, in each as-
signed HIT, the annotators can annotate the labels necessary for
the activities of the workers and equipment and their body pos-
ture. This involves (1) associating a role to each new construc-
tion worker/equipment, (2) drawing bounding boxes to localize
the worker/equipment in 2D video frames, and (3) selecting
labels to describe their activity types, body posture, and tools.
As the video proceeds, this procedure continues with updating
the position of the bounding boxes and their associated labels.
To create new roles for the workers, the annotators can use
+NewResource button which brings out a list of existing roles.
After selecting a role, the cursor will be activated to allow
the annotator to draw a bounding box around the construction
worker or equipment and pinpoint the location in 2D. Next, the
annotators select labels from the drop-down list to describe
the observed activities. The Play and Rewind buttons together
with the video progress control bar allows the role/activity/
posture labels to be updated for the observed workers and equip-
ment in a video. Upon completion, the annotator can save the
annotation results by pressing on the SaveWork button.
• Assisting function: This function is designed to help annotators
in generating accurate workface assessment results. This func-
tion consists of Introductions, +NewLabel, and Options. Intro-
ductions are provided to help the annotators understand how the
assigned tasks can be performed. For efficiency and concise-
ness, the drop-down list only contains most frequently used
labels which allows the annotators to quickly find the labels of
interest. The platform does not require a comprehensive list of
labels to begin with. Rather, the annotators can use +NewLabel
J. Constr. Eng. Manage.
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.

Fig. 5. Video visualization in the task management interface: (a) annotated video; (b) crew-balance chart; (c) detailed and CII-type pie chart; the
annotations in the upper left corner of each box shows the role-activity-tool-body posture of each worker (images by authors)

to insert the necessary new labels (or customize existing ones) to
complete their assessments. The Options enables annotators to
adjust video player and fine tune the monitoring settings. These
settings include different video speeds, hide/show bounding box
and labels on the videos, and enable/disable resizing of the
bounding boxes. To keep the user interface constrained and
simple, all assisting functions are displayed in different win-
dows which can be triggered by their corresponding functional
buttons.
Compositional Structure Taxonomy for Construction
Activities
Activity analysis requires a detailed description of construction
worker and equipment activities. A sequence of different activities
allows the analysis on the root causes for low productivity rate and
also planning and implementing productivity improvement. How-
ever, construction activities are complex to describe. They exhibit a
large amount of interclass and intraclass variability among different
activities that can be associated with the roles of the workers and
equipment. Because of the dynamic nature of construction opera-
tions, the temporal sequence of these activities changes frequently
as well. Without a systematic description, it will be difficult to pro-
vide accurate activity analysis information. To address the current
limitation, the CII (2010) proposed a new taxonomy that classifies
all activities into seven categories of direct work, preparatory work,
tools and equipment, material handling, waiting, travel, and per-
sonal. Although this taxonomy is generally applicable to all con-
struction operations, it does not provide detailed description
of tasks that would be necessary for the development of visual
activity–recognition algorithms. Any vision-based method requires
distinct visual features on worker and equipment activities, and that
could only be achieved if different labels are used to describe each
group of similar visual features.
In this research, a compositional structure taxonomy is intro-
duced to decode complex construction activities in the following
format: worker role is conducting CII activity category in form
of a visual activity category using tool, body posture and is visible,
occluded, or outside of the video frame. As a starting point, worker
type contains 19 different roles for construction worker such as
concrete finisher, carpenter, electrician, bricklayer, etc. The second
layer is CII activity categories which describes worker activities in
form of direct and nondirect work (preparatory work, tools and
equipment, material handling, waiting, travel and personal activ-
ity). The third layer is the visual activity category introduced to
provide detailed information on activities, tools, and body posture
on direct work activities. Tools vary based on different types of
activities; nevertheless, they can be an important visual indicator
for training vision-based algorithms. Because of the large number
of tools involved in any given activity, the interface is designed
such that it provides illustrative images of each tool to help anno-
tators to identify them easily. Posture can also indicate an activity

© ASCE 04015035-7 J. Constr. Eng. Manage.

J. Constr. Eng. Manage., 2015, 141(11): 04015035

Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.
Fig. 6. Workface assessment interface (images by authors)
has changed, and thus, it can be beneficial for the training of the
vision-based algorithms. This visual activity category layer with
detailed representation on activity-tool-body posture can enable
extraction of proper visual features and devising appropriate
computer-vision methods. The method also describes nondirect
works with worker body postures to enable a better synthesis of
nondirect work activities. Finally, jobsite videos contain severe
occlusions and background clutters can create noise in a represen-
tative activity data set. Thus, visibility information of each anno-
tation is associated—occluded and outside of video. Because of
space limitations, Fig. 7 presents only a part of the proposed
compositional structure taxonomy of construction worker activities
related to concrete placement operations. A more detailed represen-
tation is available at http://activityanalysis.cee.illinois.edu/.
Using the proposed taxonomy, construction professionals can
analyze both CII and task level activities for root-cause productivity
analysis purposes. To better plan productivity improvements, the
interactions between task level activities and construction worker’s
posture and tool is introduced. Not only are these interactions
meaningful for development of robust vision-based algorithms,
but they also enable construction professionals to analyze the re-
lationship between tool utilization and direct work rate that could
lead to productivity improvements through better tool utilization.
The designed platform is also very flexible. For example, for
quick assessments, the requester can choose role-CII activity struc-
ture to annotate construction videos based on the CII taxonomy of
activities. For detailed workface assessment and collecting data sets
for the development of computer-vision methods, the requester
can choose to leverage the worker-activity-posture-tool-visibility
compositional structure taxonomy of construction activities. To
facilitate annotation process and minimize time required to find a
specific activity category from a long list, the platform automati-
cally inserts the most frequently used compositional structure tax-
onomy during the task publication stage. To cope with the needs
© ASCE 04015035-8
J. Constr. Eng. Manage., 2015, 141(11): 04015035
for different types of construction operations, it also enables the
annotators to add missing or new taxonomies on the fly.
Extrapolating Annotations from Keyframes
The dynamic nature of videos makes frame-by-frame annotations
necessary but labor-intensive and costly. Crowdsourcing can
reduce human efforts, time, and cost for workface assessment;
nevertheless, video annotation still needs strategies to propagate
assessment results from a sparse set of keyframes. Keyframes are
frames in a video sequence that benchmark the start and end of a
construction activity (or a change in role). In the platform, these
changes need to be captured manually by the annotators. The non-
keyframes are the following frames that contain the same construc-
tion activities as the previous keyframe, although the position of the
workers or equipment performing the activity may have changed.
In this section, the extrapolation methods that are developed to
support propagation of the annotations from the keyframes to
nonkeyframes are described. Inspired by Vondrick et al. (2013),
linear and detection-based extrapolation methods are implemented.
T is defined as the total number of frames where T ¼ time ×
ðframe per secondÞ and B [Eq. (1)] is defined as the 2D pixel co-
ordinates of each annotation bounding box
B ¼ ½xmin ; xmax ; ymin ; ymax  ð1Þ
where xmin , ymax denote the coordinates of upper-left corner and
xmax , ymin denote the coordinates of the lower-right corner of the
bounding box. Bt ð0 ≤ t ≤ TÞ is defined as bounding box coordi-
nates at time t. Fig. 8 shows examples of applying extrapolation
methods to generate nonkeyframe annotations (Bt ) from known
keyframe annotations (B0 and BT ).
Linear extrapolation assumes that workers and equipment
that have constant velocity in 3D will also keep their velocity
J. Constr. Eng. Manage.
 Visual Activity Category
Worker CII Activity
Body
Type Category Atomic Activities Tool
Posture
Bucket
Concrete placement Scoop
Shovel
Bull float
Concrete spreader
Concrete tamper
Spreading, leveling Hand float
and smoothing Hand screed
concrete Hand trowel
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.

Power screed
Power trowel

Bending, Sitting, Standing

Direct work Vibrator
Concrete finisher

Covering or protecting
concrete
Grout pump
Curing
Power sprayer
Concrete edger
Molding expansion
Concrete groover
joint & edge
Straightedge
Broom
Surfacing
Concrete brush
Concrete saw
Cutting concrete
Line tool
Preparatory work
Material handling
Tool and equipment
Waiting
Travel
Personal
Erecting/dismantal Level
scaffold Plumb rule
Erecting/stripping Hammer
formwork Power saw

Bending, Sitting, Standing

Erecting/dismantling Air reviter
temporary structure Claw hammer
Carpenter

Direct work Chisel

Hammer
Installing door & Sander
finishes Square
Tape measure
Volume pneumatic nail gun
Assisting concrete
Bucket
pouring
Non-direct work**
Rebar hickey
Positioning rebar
Rod bending machine
Hand tying tool
Tying rebar Plier
Ironworker

B.S.S*

Direct work Power tying tool

Power saw
Metal shear
Cutting rebar
Hacksaw
Bar cutter
Non-direct work**
B.S.S*: Bending, Sitting, Standing
Non-direct work activity categories**: preparatory work,material handling, tool and equipment,
waiting, travel, personal

Fig. 7. Compositional structure taxonomy of construction worker activities

© ASCE 04015035-9 J. Constr. Eng. Manage.

J. Constr. Eng. Manage., 2015, 141(11): 04015035

 Fig. 8. Extrapolation nonkeyframes’ annotation from keyframes (images by authors)
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.
unchanged in 2D. Of course, similar to Yuen et al. (2009) in
LabelMe Video, homography-preserving shape interpolation method
can be applied to rectify linear extrapolation. However, the anecdotal
observations indicate that most construction videos exhibit limited
perspective effects and depth variation so that the 2D velocity fol-
lows the constant 3D velocity, and thus, linear extrapolations are
applied. Assuming 2D constant velocity in both x and y direction,
if a point in x direction is at B0 ðxmin Þ and then at BT ðxmin Þ, the
Bt ðxmin Þ should follow
BT ðxmin Þ − B0 ðxmin Þ
Bt ðxmin Þ ¼ ðT − tÞ ×
T−0
Applying all coordinates in keyframes’ bounding boxes to
Eq. (2), the Bt can be calculated as
BT − B 0
Bt ¼ ðT − tÞ ×
T tracking finds the best candidate from all possible detections for
To localize the position of the workers and equipment in
nonkeyframes, the detection-based extrapolation method treats
keyframe annotations as positive samples for training machine
learning classifiers in computer-vision algorithms. To build a
proper classifier for detecting workers and equipment, visual fea-
tures should be properly formed. Shape-based feature descriptors,
such as histogram of oriented gradients (HOG) with or without
histogram of color (HOC) have gained popularity in worker and
equipment detection (e.g., Park and Brilakis 2012; Memarzadeh
et al. 2013). Thus, visual feature descriptors xi consisting of
HOG and HOC features are built, as shown in the following
equation:


HOG
xi ¼
HSV
where HOG is computed based on Dalal and Triggs (2005), and
HOC is a nine-dimensional feature containing three means and six
covariance computed from hue, saturation, and value color chan-
nels. xi is applied to keyframe annotations to construct positive
samples and to automatically extract patches from keyframes’
background to construct negative samples. To learn a specific vis-
ual classifier that is able to assign positive samples with high
scores, the same procedure is followed in Memarzadeh et al.
(2013), and a binary support vector machine (SVM) is introduced
per type of resource because SVM is a commonly used discrimi-
native classifier in machine learning literature and performs well
with carefully created groundtruth data. Each binary SVM classifier
is trained by feeding all training samples ðxi þ 1Þ=ðxi ; −1Þ to op-
timize the following objective function:
© ASCE
J. Constr. Eng. Manage., 2015, 141(11): 04015035
1 Xl
min arg max wT w þ C ξi
w;b;ξ w;b;ξ 2 i¼1
subject to yi ½wT ϕðxi Þ þ b ≤ 1 − ξ i ξ i ≥ 0; i ¼ 1; : : : ; l
ð5Þ
where ϕðxi Þ is a kernel function that maps xi to a higher-
dimensional space, and C is the penalty parameter. The trained
visual classifier with weight vector w (normal vector to the hyper-
plane that separates positive and negative detection features) will be
ð2Þ used to detect construction workers and equipment for nonkey-
frames’ annotation propagation.
Because of the presence of frequent occlusions and background
clutter on construction sites, it is very difficult to propagate non-
keyframe annotations at a 100% level accuracy. Therefore, the
constrained tracking of Vondrick et al. (2013) is applied to reduce
ð3Þ the error in detection of the workers and equipment. Constrained
each frame to constitute a path with minimum cost. The path with
minimum score is defined as B0∶T ¼ B0 ; B1 ; : : : ; BT−1 ; BT , where
B0 and BT are manually generated keyframes’ annotations, and
B1 ; : : : ; BT−1 are automatically generated from the trained SVM
visual classifier. The optimization problem is then defined as
X
T−1
arg min U t ðBt Þ þ PðBt ; Bt−1 Þ ð6Þ
b1∶T−1 t¼1
where the unary cost U t ðBt Þ is defined by Eq. (7), and pairwise cost
PðBt ; Bt−1 Þ is defined by Eq. (8)
2
min½−w · ϕðBt Þ; α1  þ α2 kBt − Blin
t k ð7Þ
ð4Þ
PðBt ; Bt−1 Þ ¼ α3 kBt − Bt−1 k2 ð8Þ
The unary cost U t ðBt Þ calculates the cost of the potential detec-
tion in each frame by the score of the visual classifier and l-2 norm
of its bounding box difference between the SVM detection and the
linear extrapolation. The SVM associates the most possible predic-
tion with the highest score to minimize the most likely cost for the
detection. In this research, −w · ϕðBt Þ is used as the score of the
visual classifier. Because of the presence of occlusions, some video
frames t may not contain a groundtruth detection. These frames
cause false negatives with small scores to be the potential Bt . In
such situations, the annotations for nonkeyframes will rely on
the linear extrapolation method and replaces classifier score −w ·
ϕðBt Þ with a very small (zero number) α1 .
The pairwise cost calculates the smoothness of the detection
path for each worker/equipment. In this research, the position of
04015035-10 J. Constr. Eng. Manage.
 the bounding box does not change if the camera motion is minimal.
Thus, a true path should have the minimum pairwise cost among
all possible candidates. This pairwise cost has been adopted as
a gauge to test and select the best candidates for the path in each
frame.
Annotating Multiple Workers and Equipment
Workface assessment videos always contain a few crew members
and possibly an equipment. Thus, efficient annotation method is
imperative to reduce time and guarantee quality for dense annota-
tions. Three annotation methods are introduced that are categorized
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.
as one-by-one, all-at-once, and role-at-once.
To explain each method, each task is defined on annotating a
video with T frames and N construction workers/equipment. Using
the all-at-once method, the annotators annotate or update the labels
of all N workers and equipment in frame t (0 ≤ t ≤ T) simultane-
ously. This requires the annotators to watch the entire video prior to
conducting the annotations. Using the one-by-one method, anno-
tators annotate or update one worker or equipment for all T frames
and then rewind the video to start for the next worker or equipment
until all N workers and equipment are annotated. This method re-
quires the annotators to watch the entirety of the video for N times.
Finally, the role-at-once method assumes there are M different
roles. It then requires the annotators to annotate or update the labels
of all workers and equipment with the same type of role for all T
frames and rewind the video to start the next group of workers/
equipment until all M role types are annotated. This method re-
quires watching the video for M times.
Previous works show that annotators on AMT tend to select the
all-at-once approach as their primary annotation method. Com-
pared with other methods, the all-at-once method may seem to save
time by only requiring the annotator to watch the video once. How-
ever, this choice is not optimal for all conditions. For example,
when annotating all workers at once, the annotators may lose track
of each specific construction worker. Reviewing and correcting
such labels would require additional time. In this research, focusing
only on one worker/equipment may increase the familiarity of the
annotator and ultimately save time during the annotation updating
process. One should also note that time is not the only indicator
that needs attention. The trade-off between time and accuracy could
also affect the final annotation results. To quantify the time, accu-
racy, and their trade-off relationship in real application scenarios,
experiments are conducted to examine the most efficient annota-
tion approach. The results will be discussed in the “Experimental
Results” section.

SRt ðN R Þ − SIt ðN I Þ;
Costt ðN R ; N I Þ ¼
2 × ½SRt ðN R Þ − SIt ðN I Þ;
where Costt ðN R ; N I Þ is the cost between N R -th annotation
from reference and N I -th annotation from input at frame t
(0 ≤ t ≤ T); SRt ðN R Þ − SIt ðN I Þ is the area difference (bounding
box overlap) between N R -th bounding box and N R -th bound-
ing box; and RoleNt R and RoleNt I are the different annotation
labels in comparison. The calculated costs are used to consti-
tute the cost matrix for reference and input annotation results,
which is shown in Fig. 9. Annotations from reference and
© ASCE
J. Constr. Eng. Manage., 2015, 141(11): 04015035
Quality Assurance and Quality Control Methods
for AMT
The AMT is a marketplace of hundreds of thousands of annotators
for solving microtask (HITs) quickly and effortlessly. However,
because of poor or malicious judgment of the annotators, quick
assessments may lead to erroneous results. To lower the risk of
obtaining low quality results, AMT annotators are classified into
the following: (1) skilled annotators who posses the ability to pro-
vide accurate workface assessment results; (2) ethical annotators
who are honest but may be incapable of providing results with
high accuracy because of poor judgments; and (3) unethical anno-
tators who only try to finish as many tasks as possible in a random
fashion just be able to earn money. To reject the unethical annota-
tors and improve the accuracy of skilled and ethical annotators, pre-
assessment and postassessment quality control steps are designed
as follows:
• Preassessment: This step is used to select skilled and ethical
annotators and reject unethical annotators from the AMT mar-
ketplace by testing their performance. In the preassessment
procedure, a short testing video is added—for which the ground
truth has been previously generated—to the start of each HIT.
At the very first time, the annotators are directed to this testing
video. The platform compares these annotations with the ground-
truth and reports the annotator’s accuracy for the given HIT. If the
accuracy is above the requester’s predefined threshold, the anno-
tator can continue to work on the actual HITs. Otherwise, the
annotator is prohibited to work on any other HITs.
• Postassessment: One should note that most if not all annotators
are aware of the preassessment screening tests on the AMT mar-
ketplace (Ipeirotis et al. 2010; Snow et al. 2008; Le et al. 2010).
This provides an opportunity for the unethical annotators to
provide a perfect testing performance to get the pass and then
generate random assessments for the actual HITs. Thus, besides
quality assurance, quality control is needed to examine the ac-
curacy of the actual assessments and to correct potential errors.
This is done through a repeated-labeling approach. This requires
a video to be annotated for multiple times by several annotators.
Sheng et al. (2008) also leverages repeated-labeling to deal
with noisy data for assessment quality improvement. To do so,
a matching schema is defined that uses cost matrix to find
corresponding annotations across repeated/multiple assessment
results and then apply majority voting strategy to generate the
final assessment. Matching schema randomly selects an an-
notation result as the reference and feeds reference and the re-
maining assessments to cost equation [Eq. (9)] to generate cost
matrices for matching corresponding annotations
if RoleNt R ≡ RoleNt I ; t ¼ 0; : : : ; T
ð9Þ
otherwise
input are grouped based on the minimum cost value in the
cost matrix. Once all groups have been matched, majority vot-
ing is performed to annotations of same construction worker/
equipment at each frame. For example, groups are first elim-
inated which have an annotation number less than half of
the repeat time. Then, average bounding boxes’ coordinates
AVGðBallt Þ of all repeated annotations for the same construc-
tion worker/equipment are calculated. Annotation(s) whose
04015035-11 J. Constr. Eng. Manage.

1 2
1 Costt (1,1)
Input Annotations
2 Costt (1,2)
at frame t
…
NI Costt (1, N I )
N I+1 Costt (1, N I+1 )
…
Fig. 9. Cost matrix at frame t
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.
sum of Bt − AVGðBall t Þ is greater than the defined threshold
at frame t are eliminated. Finally, the average of bounding
boxes’ coordinates was recalculated and the majority label
for each level of compositional structure taxonomy from the
rest of annotations was selected to generate the final annota-
tion for each construction worker/equipment—the annotation
group—at frame t.
Although repeated-labeling improves the quality of workface as-
sessment, the unnecessary repeated labelings will cost extra money
and time. To save money and time, it is necessary to find an optimal
repeat time. Thus, experiments are conducted using crossvalidation
method to examine how the accuracy will be changed based on dif-
ferent repeat times which are discussed in the following section.
Designing Microtasks from Construction Videos
A video of a construction operation is typically several hours long.
This makes assessing it much more difficult than completing a typ-
ical short-length microtask on the AMT. To make crowdsourcing
feasible, an entire video is broken down into several shorter HITs.
For effectiveness, the following is considered:
• The length of a HIT: The longer a HIT is, the less time the an-
notator will need to complete the task of annotation for the en-
tirety of the video. This is because the annotator will spend less
time on understanding the video content and would not require to
watch a video for multiple times. However, longer HITs can lead
to the tiredness and the boredom of the annotators which can in
turn lower the accuracy and increase the annotation time. To study
their trade-off relationship, experiments are conducted and experi-
ment results are presented in the “Experimental Results” section.
• Annotation frequency: Although dense labeling seems desirable
for improving workface assessment accuracy, over labeling may
lead to cost and time overrun. Manually annotating a sparse set
of keyframes and applying extrapolation to generate annotations
for nonkeyframes can provide the same level of accuracy with
less time and cost.
• Method of stitching several HITs together for deriving final as-
sessment results: The platform breaks an entire operation video
into multiple microtask HITs. To stitch the results of these HITs
and derive the final workface assessment result, one-second
overlap is placed between each HITs. Then, the same method
used to match repeated annotation results is applied, as shown
in Eq. (9), to all the overlapping annotations of the HITs. They
are then stitched together to derive the final workface assess-
ment results.
Results and Discussion
Experiment Setup and Performance Measures
To identify the key parameters of the platform and validate the ef-
fectiveness of crowdsourcing the workface assessment task, a pool
© ASCE
J. Constr. Eng. Manage., 2015, 141(11): 04015035
Reference Annotations at frame t
… NR N R+1 …
Costt (2, 1) Costt (N R , 1) Costt (N R+1 , 1)
Costt (2, 2) Costt (N R , 2) Costt (N R+1 , 2)
Costt (2, N I ) Costt (N R , N I ) Costt (N R+1 , N I )
Costt (2, N I+1 ) Costt (N R , N I+1 ) Costt (N R+1 , N I+1 )
of 10þ nonexpert annotators—typically found on AMT—were as-
sembled together with a control group of experts [5þ professionals,
who in this experiment are (1) field engineers with experience on
productivity assessments; and (2) students that took a course in pro-
ductivity and have experience on productivity assessment]. All ex-
pert participants in this experiment have done at least productivity
assessment for more than a one-day operation. Easy, normal, and
hard videos of concrete placement operations introduced earlier in
the paper were leveraged. With these videos, three separate experi-
ments were conducted with the annotators from the controlled
group of construction experts to investigate the impact of the differ-
ent annotation methods, video lengths, and annotation frequencies
on the accuracy of the workface assessment results.
To validate the hypothesis that crowdsourcing video-based con-
struction workface assessment through the AMT marketplace is a
reliable approach, three additional experiments were conducted on
the impact of the platform on accuracy by (1) comparing the
performance of nonexpert annotators with the controlled group
of construction experts; (2) testing the linear extrapolation and
detection-based extrapolation methods and presenting the perfor-
mance of each extrapolation method against ground truth data gen-
erated by the expert annotators; and (3) testing the performance
of the postassessment quality control procedure and exploring
the best repeated labeling times for desirable level of accuracy
by experimenting crossvalidation with different randomly selected
folds from both expert and nonexpert annotation results.
To compare experiments and choose the parameters that achieve
optimal performance, two validation criteria were chosen: (1) the
annotation time spent to complete each experiment; and (2) the ac-
curacy of the workface assessment results. Because the platform
applies compositional structure taxonomy of construction activ-
ities, three separate discussions on accuracy are presented:
1. Completeness accuracy examines annotators capability to
completely annotate all workers in a HIT video without miss-
ing one.
2. Bounding box accuracy investigates the accuracy of worker
localization. In this case, 50% overlap is used between experi-
mental assessment and groundtruth as the acceptable threshold
for accuracy.
3. Tool accuracy examines the annotator’s capability to correctly
label construction tools or the nondirect work categories.
In the following section, each experiment is introduced and
the experimental results are reported.
Experimental Results
Annotation Method
The experiments for choosing the best annotation method include
leveraging one-by-one, all-at-once, and role-at-once methods to
annotate easy, Normal, and Hard videos. The time durations (in
seconds) required for each annotation method across the expert
04015035-12 J. Constr. Eng. Manage.
 Table 3. Average Accuracy of Each Annotation Method
Bounding
Methods Completeness box Role Activity Posture Tool
AM01 0.95 0.98 0.83 0.80 0.97 0.84
AM02 0.95 0.98 0.88 0.71 0.97 0.83
AM03 0.97 0.98 0.99 0.80 0.96 0.82

different video lengths of 10, 30, and 60 s. The experimental

results, as shown in Fig. 11 and Table 4, indicate that increasing
video length can reduce the annotation time. It is observed that vid-
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.

Fig. 10. Time spent using each annotation method for easy (0–15 min),
normal (15–30 min), and hard (30–45 min) videos
annotator group are shown in Fig. 10. The average and total time
for annotations are provided in Table 2.
From these results, it is observed that when there is a small num-
ber of construction workers in a video, the one-by-one annotation
method shows the best performance. As the number of workers
increase, the all-at-one method performs best. The one-by-one an-
notation method requires a repeating video for each worker. Thus, it
will take a longer time to annotate hard videos. It is also observed
that when the activities of each construction worker change fre-
quently, the one-by-one labeling method performs best. This is be-
cause this method will save time by allowing annotators to focus on
one construction worker at a time and minimizes the chance of
mistakes and the need for unnecessary revisions. In contrary, the
high frequency of changes in activities of the workers overwhelms
annotators who use the all-at-once or role-at-once methods. These
methods will require extra time to be spent on revising activity cat-
egories because the annotators may then require to rewind the video
frequently and make all annotations consistent with one another.
When the worker activity categories do not frequently change, the
all-at-once methods shows the best performance.
The results in Table 3 show that the accuracy in assessment re-
sults for categorizing activities does not change significantly across
different annotation methods. However, among all, the role-at-once
method shows the highest accuracy in labeling the worker roles.
This is because this method requires the annotators to reason about
the role of the workers, and thus, it consequently increases the
accuracy of labeling roles.
Impact of the Video Length
To examine the relationship between video length and the speed
of workface assessment, an experiment was conducted using three
eos with 60-s length require the least annotation time. Particularly
the saving in the annotation time by posting 60-s videos for easy
and normal videos is significant. For hard videos, presenting videos
of 60-s length does not significantly save annotation time compared
to posting videos with other time durations. This is because com-
bining six 10-s video into one larger 60-s video allows the anno-
tators to become familiar with the video content. This, in turn,
reduces the time needed for interpreting the content multiple times.
It also reduces the need for redrawing the bounding boxes.
As shown in Table 5, it is observed that choosing videos with
different length does not make a significant difference in the accu-
racy of the assessment results. However, compared with other video
lengths, the 60-s video length exhibits lower accuracy for labeling
tools (around 10% lower). This suggests that longer video length
could possibly cause tiredness and boredom for the annotators, and
this makes the accuracy lower when compared with the annotation
of the videos that have a shorter length.
Frequency of Choosing Keyframes for Annotation
Annotating a sparse set of frames as opposed to annotating frames
one-by-one can minimize the annotation time, but it may also neg-
atively impact the accuracy of the assessment results. To explore
the trade-off relationship between time and accuracy, three different
fixed annotation frequencies of three times, five times, and nine
times per minute were experimented with. In other words, the an-
notators were asked to only use these prefixed times per minute for
their annotation purposes. The annotation times of each frequency
are shown in Fig. 12, and the average and the total time duration
of the annotations are presented in Table 6. The accuracy of each
frequency is also provided in Table 7. From these experiments, the
hypothesis that a sparse annotation can save time is validated. The
sparser the annotations are, the less time annotation will require.
The results particularly show that this method significantly shortens
the annotation time for easy and normal videos that exhibit high
frequency of changes in activity categories. Nevertheless, the gains
are not as obvious for hard videos which exhibit a low frequency of
change in these categories.
Although choosing a smaller number of keyframes saves time
required for annotation, it also lowers the accuracy in assessment
results. As shown in Tables 6 and 7, the three times per minute

Table 2. Average and Total Annotation Time for Each Annotation Method
Videos with different levels of difficulty in the annotation task
Time Methods Easy_01 Easy_02 Easy_03 Normal_01 Normal_02 Normal_03 Hard_01 Hard_02 Hard_03
Average AM01 550 592 534 1,491 1,317 1,359 1,307 1,084 1,254
AM02 363 836 607 1,404 1,276 1,879 871 823 804
AM03 478 922 788 1,736 1,507 1,647 1,174 908 1,576
Total AM01 — 8,380 — — 20,841 — — 18,232 —
AM02 — 9,034 — — 22,801 — — 12,495 —
AM03 — 10,943 — — 24,459 — — 18,298 —
Note: AM01 = one-by-one; AM02 = all-at-once; AM03 = role-at-once; time is measured in seconds.

© ASCE 04015035-13 J. Constr. Eng. Manage.

J. Constr. Eng. Manage., 2015, 141(11): 04015035

 Table 5. Average Accuracy in Workface Assessment Results of Each
Video Length
Lengths
(s) Completeness Bounding box Role Activity Posture Tool
10 0.95 0.98 0.83 0.80 0.97 0.84
30 0.90 0.95 0.88 0.81 0.97 0.83
60 0.90 0.96 0.90 0.82 0.97 0.74
Note: The bold font shows the maximum average accuracy in each
category.
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.

Fig. 11. Annotation time for easy (0–15 min), normal (15–30 min),
and hard (30–45 min) videos

annotation frequency has the lowest average accuracy. This, how-

ever, does not indicate that requesters who prefer higher accuracies
in their workface assessment should not choose to guide the anno-
tators to use lower annotation frequencies. Rather, the experimental
results indicate that the requesters can conduct a cost-benefit analy-
sis for choosing accuracy versus the time needed for completing the
workface assessment tasks. For example, results show that for 60-s
videos, the three times per minute frequency is 66% faster than
other annotation frequencies and results only in 7% reduction in Fig. 12. Annotation time of each video annotation frequency: easy
the accuracy of the workface assessments. (0–15 min), normal (15–30 min), and hard (30–45 min) videos

Expert versus Nonexpert Annotator

To validate the reliability of crowdsourcing on the AMT market-
place, an experiment was conducted to compare the annotation time
and accuracy of a large pool of nonexperts (10þ) against a con-
trolled group of construction experts (5þ). Fig. 13 shows the differ-
ence in the annotation time between the nonexpert and expert
annotators. Fig. 13 shows that on average, an expert is 22% faster
than a nonexpert annotator.
The accuracy of annotations between the nonexpert and expert
control groups was compared and linear regression was used to in-
terpolate all observation points. As shown in Figs. 14 and 15, the
results testify that the experts, in general, perform slightly better
than the nonexpert groups on the AMT. Fig. 16 shows the differ-
ence in accuracy in the categories of the percentage of complete-
ness of work, the bounding box, body posture, role, activity, and
tool. Although the nonexpert annotators produce higher accuracy in
selecting roles, as shown at the top of Fig. 15, the difference is less
than 2% and is not necessarily more advantageous. Table 8 shows
the comparison in the accuracy of the workface assessment results
between the nonexpert and expert groups. This difference is only
about 3% better for the expert group. This encouraging result
testifies that for video-based workface assessment, nonexpert an-
notators on the AMT have the potential to perform as well as the
expert groups.
Crossvalidation
The tool applies repeated labeling as a postassessment quality con-
trol step. In this case, the requester can improve the accuracy in
assessments and also minimize the risk in collecting inaccurate
data. The idea is to crossvalidate the repeated labeling results across
multiple AMT annotators to generate final assessment results with
potentially higher accuracies. However, unnecessary repetitions
lead to additional cost and can potentially increase the assessment
time (unless the annotators work simultaneously). Either way, it is
necessary to explore how many rounds of labeling are needed to
guarantee satisfactory workface assessment results. Experimental
results are presented in Figs. 16(a–c).
To conduct a systematic experiment, the worst annotation result
was chosen from each video category as the original annotation

Table 4. Average and Total Time Spent on Annotating Videos with Different Length
Videos with different levels of difficulty in the annotation task
Time Lengths (s) Easy_01 Easy_02 Easy_03 Normal_01 Normal_02 Normal_03 Hard_01 Hard_02 Hard_03
Average 10 550 592 534 1,492 1,317 1,359 1,307 1,084 1,254
30 488 655 527 1,292 1,230 1,835 662 529 850
60 279 377 354 806 635 1,062 513 436 614
Total 10 — 8,380 — — 20,841 — — 18,232 —
30 — 8,349 — — 21,784 — — 10,227 —
60 — 5,050 — — 12,520 — — 7,812 —
Note: Time is measured in seconds.

© ASCE 04015035-14 J. Constr. Eng. Manage.

J. Constr. Eng. Manage., 2015, 141(11): 04015035

 Table 6. Average/Total Annotation Time Spent with Each Annotation Frequency
Videos with different levels of difficulty in the annotation task
Frequencies
Time (times/min) Easy_01 Easy_02 Easy_03 Normal_01 Normal_02 Normal_03 Hard_01 Hard_02 Hard_03
Average 9 184 410 313 588 450 608 334 260 401
5 488 655 161 337 336 460 253 182 292
3 83 125 118 265 189 237 243 198 273
Total 9 — 4,586 — — 8,229 — — 4,975 —
5 — 2,572 — — 5,668 — — 3,630 —
3 — 1,636 — — 3,451 — — 3,575 —
Note: Time is measured in seconds.
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.

(onefold), and then annotations from each video category were video conditions and annotation frequencies, two extrapolation
randomly selected to constitute threefold, fourfold, fivefold, six- methods were experimented on easy, normal, and hard videos
fold, sevenfold, and eightfold crossvalidations. For easy, normal, with different annotation frequencies. Annotation frequency is
and hard videos, it is observed that the increase in accuracy tends indicated by average clicks per frame per construction worker,
to be steady after a threefold crossvalidation. Figs. 16(a–c) show
results for easy, normal, and hard videos respectively. Fig. 16(c)
also shows that the activity accuracy of hard videos drop from a
threefold to an eightfold crossvalidation. The accuracy drop after
a sevenfold crossvalidation in Fig. 16(a) may result from the unnec-
essary repeated labelings which generate more erroneous data and
thus cause the incorrect labels to stand out. The activity accuracy
drop in Fig. 16(c) may also result from the unnecessary repeated
labeling or result from the farther camera distance which can chal-
lenge the accuracy of observations. Based on the average perfor-
mance of each fold crossvalidation accuracy, it was concluded
that threefold crossvalidation can provide the optimal performance
and that increasing the fold number increases the risk of producing
erroneous data.

Linear and Detection-Based Extrapolation Method

To assist AMT annotators in generating nonkeyframe annotations,
the tool enables linear and detection-based extrapolation methods.
The extrapolation method treats user-assisted keyframe annotations
as input and generates nonkeyframe annotations automatically.
To examine the performance of each method under different
Fig. 14. Percent of completeness, bounding box, and posture accuracy
difference between expert and nonexpert annotators
Table 7. Average Accuracy of Assessment Results for Different
Annotation Frequencies
Frequencies Bounding
(times/min) Completeness box Role Activity Posture Tool
9 0.94 0.90 0.99 0.78 0.93 0.83
5 0.87 0.85 0.95 0.74 0.94 0.86
3 0.83 0.77 0.86 0.66 0.95 0.85
Note: The bold font shows the maximum average accuracy in each
category.

Fig. 13. Annotration time difference between expert and nonexpert Fig. 15. Percent role, activity, and tool accuracy differences between
annotators expert and nonexpert annotators

© ASCE 04015035-15 J. Constr. Eng. Manage.

J. Constr. Eng. Manage., 2015, 141(11): 04015035

Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.

Fig. 16. Crossvalidation results for (a) easy; (b) normal; and (c) hard videos

Table 8. Difference in Accuracies between the Expert and Nonexpert Discussion on the Proposed Method and Research
Annotators Challenges
Accuracy of an expert when The results validate the hypothesis that crowdsourcing construction
Category compared with a nonexpert
activity analysis from jobsite videos on the AMT, a marketplace
Completeness þ0.02 with nonexpert annotators, is a reliable approach for conducting
Bounding box þ0.01 activity analysis. In addition, the platform facilitates collection of
Role −0.01 large data sets with their ground truth that could be used for the
Activity þ0.03 development of computer-vision algorithms for automatic activity
Posture 0.00
recognition. Particularly, it is shown that expert annotators are, on
Tool þ0.03
average, 22% faster than nonexpert annotators in terms of their
annotation time. However, the accuracy of annotation among the
nonexperts is within 3% of the accuracy of the expert groups. To
which includes 0.001, 0.005, 0.01, 0.05, and 0.1. Figs. 17(a–c)
fine tune the platform, the impact of different annotation methods,
present the error rates of each method against the ground-truth
different HIT video lengths, and the frequency of requiring anno-
annotations. tations were experimented on and discussed. Based on these exper-
Fig. 17 illustrates that the increase in annotation frequency can imental results, the following conclusions can be made:
lead to the decrease of error rate for both extrapolation methods. 1. The one-by-one annotation method works best with videos
It is observed that increasing annotation frequency after 0.01 clicks that have a small number of construction workers and high
per frame per construction worker could only marginally reduce frequency of changes in activities, whereas the all-at-once an-
error rate. This experiment indicates that dense annotation cannot notation method works best with videos that have a high
necessarily guarantee higher accuracy in annotation performance. number of construction workers and low frequency of changes
As indicated by the error rate for both extrapolation methods in work activities.
in Figs. 17(a and b), linear extrapolation performs as well as the 2. Increasing a HIT video length can reduce the annotation time.
detection-based extrapolation method, and the error rate differ- For example, the 60-s long videos save 47 and 37% annotation
ence is within 5% on average. However, it is also observed that time compared to the 10-s and 30-s long videos. It was also
linear extrapolation method performs much better than detection- observed that the accuracy of workface assessment results
based extrapolation method in Fig. 17(c). This difference is likely slightly improves with an increase in the HIT video length.
caused by the difficulties in extracting effective visual features for 3. Manual annotation of a sparse set of video keyframe is reliable
hard videos and detecting workers in cluttered construction site for achieving complete frame-by-frame annotations. At the
conditions. most extreme case, the three times per minute annotation

Fig. 17. The error rates of the linear and detection-based extrapolation methods for annotating nonkeyframes for (a) easy; (b) normal; and
(c) hard videos

© ASCE 04015035-16 J. Constr. Eng. Manage.

J. Constr. Eng. Manage., 2015, 141(11): 04015035

 frequency reduces the average annotation time by 57% while
dropping the accuracy of workface assessment only by 7%.
The prominent decrease in annotation time by three times
per minute could also result in cost savings because AMT
charges the requester based on the time the annotator devotes
to each HIT.
4. A threefold crossvalidation provides the best accuracy-cost
trade-off for workface assessment. Increasing the fold number
(beyond threefold) does not raise accuracy significantly. Also,
quality assurance and control steps are important to guarantee
the reliability of the assessment results. The repeated label-
ing can also improve the accuracy of workface assessment.
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.
The optimal performance can be achieved with a threefold
(i.e., hiring three AMT annotators per HIT).
5. Increasing the sparsity of annotations by increasing the num-
ber of keyframes may not necessarily increase the accuracy
of 2D localization because the increase in accuracy tends to
plateau to þ0.05 clicks per frame per construction worker.
Finally, increasing the number of keyframes may not necessa-
rily lead to the significant increase in accuracy of 2D localiza-
tion. The 2D localization accuracy tends to increase from 0.01
to 0.1 clicks per frame per construction worker; however, the
increase is within 18%. Also, it was observed from the experi-
ments that increasing keyframe number after 0.05 clicks per
frame per construction worker barely leads to an increase in
accuracy.
6. Linear extrapolation method can perform as well as detection-
based extrapolation method for easy and normal videos. How-
ever, because of the difficulties in extracting visual features
and detecting construction workers in severe construction site
conditions from hard videos, the detection-based extrapolation
method fails to compete with the linear extrapolation method.
Conclusion and Future Work
This paper presents a novel method that supports crowdsourc-
ing construction activity analysis from jobsite video streams.
The proposed method leverages human intelligence recruited from
a massive crowdsourcing marketplace, AMT, together with auto-
mated vision-based detection/tracking algorithms to derive timely
and reliable construction activity analysis in different challenging
conditions such as severe occlusion, background clutter, and cam-
era motions. The experimental result with average accuracy of 85%
in workface assessment tasks shows the promise of the proposed
method. The comparisons conducted between nonexperts and con-
struction validate the hypothesis that crowdsourcing video-based
construction activity analysis through AMT nonexperts could
achieve similar (or even the same) accuracy as conducting activity
analysis by construction experts.
To improve the platform, future work should focus on (1) the
design of a more robust detection/tracking algorithm that can work
well with sparse human input to effectively generate accurate non-
keyframe annotations; and (2) the design of a quality control
method that does not require repeated labeling, to reduce reques-
ters’ cost and avoid erroneous data at the voting stage. As part of
the study, a new compositional structure taxonomy for construction
activities is also created that models the interactions between body
posture, activities, and tools. This representation can improve
detection/tracking by enhancing the propagation of manual annota-
tions to nonkeyframes. Also, studies that focus on using the hidden
Markov model to automatically infer construction activities from
long sequences of jobsite videos could be beneficial to detection/
tracking and quality control steps. Learning a set of transition
© ASCE 04015035-17
J. Constr. Eng. Manage., 2015, 141(11): 04015035
and emission probabilities between each pair of construction activ-
ities from a crowdsourcing platform can improve inference on the
categories of subsequent activity types for each frame and in turn,
improves the quality control process.
To facilitate frequent implementation of crowdsourced activity
analysis, short-term future work involves devising workflows to
assist foremen to videotape their operations on a daily basis. This
can be done by placing consumer-level cameras on tripods away
from the operation of the crew such that for the most part, the crew
stays within the line-of-sight of the cameras. For the long-term,
automatically placing a relocatable network of cameras around
the site and leveraging action cameras mounted on the hardhat or
the safety vest of the workers can address the issues of conducting
data collection throughout the site and also the line-of-sight and
visibility for each camera.
Meanwhile, as part of future work, the following will be con-
sidered: (1) plotting the workface assessment results over the
course a day to get a better understanding on how soon crafts
are getting on their tools in the morning, where there are excessive
breaks, and where and when crews quit an operation too early; and
(2) integrating observations from different viewpoints. Finally, to
comprehensively validate this new method for construction video-
based analysis, a set of detailed crowdsourcing market investiga-
tions and experiments should be conducted, not only to test the
technical parameters, but also to build a process model to test
the cost associated with crowdsourcing, the time span between pub-
lishing and retrieval tasks, and potential risks of affecting worker
privacy by outsourcing construction video annotations containing
construction workers to the crowd. This platform is publicly acces-
sible at http://activityanalysis.cee.illinois.edu. A video is also pro-
vided as a companion for better illustration of the functionalities
of this platform.
Acknowledgments
The authors would like to thank Zachry Construction Corpora-
tion and Holder Construction Group for their support with data
collection. The authors thank Professor Carl Haas for his very
constructive feedbacks during the development of the workface
assessment platform. The technical support of Deepak Neralla
with the development of web-based tool is appreciated. The authors
also thank the support of real-time and automated monitoring and
control (RAAMAC) lab’s members, the graduate and undergradu-
ate civil engineering students, and other AMT nonexpert annota-
tors. This work was financially supported by the University of
Illinois Department of Civil and Environmental Engineering’s
Innovation Grant. The views and opinions expressed in this paper
are those of the authors and do not represent the views of the in-
dividuals or entities mentioned above.
Supplemental Data
A video demonstration of the RAAMAC Crowdsourcing Workface
Assessment Tool is available online in the ASCE Library (www
.ascelibrary.org).
References
Ali, K., Hasler, D., and Fleuret, F. (2011). “Flowboost—Appearance learn-
ing from sparsely annotated video.” 2011 IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), IEEE, Colorado Springs,
CO, 1433–1440.
J. Constr. Eng. Manage.
 Azar, E. R., and McCabe, B. (2012). “Vision-based recognition of dirt
loading cycles in construction sites.” Proc., Construction Research
Congress, ASCE, Reston, VA, 1042–1051.
Brilakis, I., Park, M.-W., and Jog, G. (2011). “Automated vision tracking of
project related entities.” Adv. Eng. Inf., 25(4), 713–724.
Cheng, T., Venugopal, M., Teizer, J., and Vela, P. (2011). “Performance
evaluation of ultra wideband technology for construction resource
location tracking in harsh environments.” Autom. Constr., 20(8),
1173–1184.
Chi, S., and Caldas, C. H. (2011). “Automated object identification using
optical video cameras on construction sites.” Comput.-Aided Civ. Infra-
struct. Eng., 26(5), 368–380.
CII (Construction Industry Institute). (2010). “Guide to activity analysis.”
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.
Rep. No. IR252-2a, Univ. of Texas, Austin, TX.
Costin, A., Pradhananga, N., and Teizer, J. (2012). “Leveraging passive rfid
technology for construction resource field mobility and status monitor-
ing in a high-rise renovation project.” Autom. Constr., 24, 1–15.
Dalal, N., and Triggs, B. (2005). “Histograms of oriented gradients for
human detection.” IEEE Computer Society Conf. on Computer Vision
and Pattern Recognition, 2005 (CVPR 2005), Vol. 1, IEEE, San Diego,
886–893.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009).
“Imagenet: A large-scale hierarchical image database.” IEEE Conf. on
Computer Vision and Pattern Recognition, 2009 (CVPR 2009), IEEE,
Miami, FL, 248–255.
Di Salvo, R., Giordano, D., and Kavasidis, I. (2013). “A crowdsourcing
approach to support video annotation.” Proc., Int. Workshop on Video
and Image Ground Truth in Computer Vision Applications, ACM,
New York, 1–6.
Doermann, D., and Mihalcik, D. (2000). “Tools and techniques for video
performance evaluation.” Int. Conf. on Pattern Recognition, IEEE
Computer Society, Hilton Head, SC, 4167–4167.
Escorcia, V., Davila, M. A., Niebles, J. C., and Golparvar-Fard, M. (2012).
“Automated vision-based recognition of construction worker actions
for building interior construction operations using RGBD cameras.”
Construction Research Congress 2012, ASCE, Reston, VA, 879–888.
Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman,
A. (2010). “The pascal visual object classes (voc) challenge.” Int. J.
Comput. Vision, 88(2), 303–338.
Gadgil, N. J., Tahboub, K., Kirsh, D., and Delp, E. J. (2014). “A web-based
video annotation system for crowdsourcing surveillance videos.” IS&T/
SPIE Electronic Imaging, International Society for Optics and Photon-
ics, 90270A–90270A.
Giretti, A., Carbonari, A., Naticchia, B., and DeGrassi, M. (2009). “Design
and first development of an automated real-time safety management
system for construction sites.” J. Civ. Eng. Manage., 15(4), 325–336.
Golparvar-Fard, M., Heydarian, A., and Niebles, J. C. (2013). “Vision-
based action recognition of earthmoving equipment using spatio-
temporal features and support vector machine classifiers.” Adv. Eng.
Inf., 27(4), 652–663.
Golparvar-Fard, M., Peña-Mora, F., Arboleda, C. A., and Lee, S. (2009).
“Visualization of construction progress monitoring with 4D simulation
model overlaid on time-lapsed photographs.” J. Comput. Civ. Eng.,
10.1061/(ASCE)0887-3801(2009)23:6(391), 391–404.
Gong, J., and Caldas, C. H. (2009). “An intelligent video computing
method for automated productivity analysis of cyclic construction
operations.” Proc., 2009 ASCE Int. Workshop on Computing in Civil
Engineering, ASCE, Reston, VA, 64–73.
Gong, J., Caldas, C. H., and Gordon, C. (2011). “Learning and classifying
actions of construction workers and equipment using bag-of-video-
feature-words and bayesian network models.” Adv. Eng. Inf., 25(4),
771–782.
Gouett, M. C., Haas, C. T., Goodrum, P. M., and Caldas, C. H. (2011).
“Activity analysis for direct-work rate improvement in construction.”
J. Constr. Eng. Manage., 10.1061/(ASCE)CO.1943-7862.0000375,
1117–1124.
Heilbron, F. C., and Niebles, J. C. (2014). “Collecting and annotating
human activities in web videos.” Proc., Int. Conf. on Multimedia
Retrieval, ACM, New York, 377.
© ASCE 04015035-18
J. Constr. Eng. Manage., 2015, 141(11): 04015035
Hildreth, J., Vorster, M., and Martinez, J. (2005). “Reduction of short-
interval gps data for construction operations analysis.” J. Constr. Eng.
Manage., 10.1061/(ASCE)0733-9364(2005)131:8(920), 920–927.
Howe, J. (2008). Crowdsourcing: Why the power of the crowd is driving
the future of business, 1st Ed., Crown Publishing Group, New York.
Ipeirotis, P. G. (2010). “Analyzing the amazon mechanical turk market-
place.” Assoc. Comput. Mach. Mag. Stud., 17(2), 16–21.
Ipeirotis, P. G., Provost, F., and Wang, J. (2010). “Quality management on
amazon mechanical turk.” Proc., Association for Computing Machinery
(ACM) Special Interest Group on Knowledge Discovery and Data Min-
ing (SIGKDD) Workshop on Human Computation, ACM, New York,
64–67.
Jaselskis, E., Sankar, A., Yousif, A., Clark, B., and Chinta, V. (2015).
“Using telepresence for real-time monitoring of construction opera-
tions.” J. Manage. Eng., 10.1061/(ASCE)ME.1943-5479.0000336,
A4014011.
Kavasidis, I., Palazzo, S., Di Salvo, R., Giordano, D., and Spampinato, C.
(2012). “A semi-automatic tool for detection and tracking ground truth
generation in videos.” Proc., 1st Int. Workshop on Visual Interfaces for
Ground Truth Collection in Computer Vision Applications, ACM,
New York, 6.
Khosrowpour, A., Fedorov, I., Holynski, A., Niebles, J. C., and Golparvar-
Fard, M. (2014a). “Automated worker activity analysis in indoor
environments for direct-work rate improvement from long sequences
of rgb-d images.” Construction Research Congress 2014 Construction
in a Global Network, ASCE, Reston, VA, 729–738.
Khosrowpour, A., Niebles, J. C., and Golparvar-Fard, M. (2014b). “Vision-
based workface assessment using depth images for activity analysis of
interior construction operations.” Autom. Constr., 48, 74–87.
Kim, J., Nguyen, P. T., Weir, S., Guo, P. J., Miller, R. C., and Gajos, K. Z.
(2014). “Crowdsourcing step-by-step information extraction to
enhance existing how-to videos.” Proc., 32nd Annual ACM Conf. on
Human Factors in Computing Systems (CHI ‘14), ACM, New York,
4017–4026.
Kim, J. Y., and Caldas, C. H. (2013). “Vision-based action recognition in
the internal construction site using interactions between worker actions
and construction objects.” International Association for Automation and
Robotics in Construction (IAARC), Chennai, India, 661–668.
Le, J., Edmonds, A., Hester, V., and Biewald, L. (2010). “Ensuring
quality in crowdsourced search relevance evaluation: The effects of
training question distribution.” Proc., SIGIR 2010 Workshop on
Crowd Sourcing for Search Evaluation, Microsoft, Geneva, 21–26.
Memarzadeh, M., Golparvar-Fard, M., and Niebles, J. C. (2013). “Auto-
mated 2D detection of construction equipment and workers from site
video streams using histograms of oriented gradients and colors.”
Autom. Constr., 32, 24–37.
Oglesby, C. H., Parker, H. W., and Howell, G. A. (1989). Productivity
improvement in construction, McGraw-Hill, New York.
Park, M.-W., and Brilakis, I. (2012). “Construction worker detection
in video frames for initializing vision trackers.” Autom. Constr., 28,
15–25.
Park, M.-W., Koch, C., and Brilakis, I. (2011). “Three-dimensional tracking
of construction resources using an on-site camera system.” J. Comput.
Civ. Eng., 10.1061/(ASCE)CP.1943-5487.0000168, 541–549.
Peddi, A., Huan, L., Bai, Y., and Kim, S. (2009). “Development of human
pose analyzing algorithms for the determination of construction produc-
tivity in real-time.” Construction Research Congress, ASCE, Seattle,
11–20.
Pradhananga, N., and Teizer, J. (2013). “Automatic spatio-temporal analy-
sis of construction site equipment operations using gps data.” Autom.
Constr., 29, 107–122.
Rezazadeh Azar, E., Dickinson, S., and McCabe, B. (2012). “Server-
customer interaction tracker: Computer vision-based system to estimate
dirt-loading cycles.” J. Constr. Eng. Manage., 10.1061/(ASCE)CO
.1943-7862.0000652, 785–794.
Rezazadeh Azar, E., and McCabe, B. (2012). “Automated visual recogni-
tion of dump trucks in construction videos.” J. Comput. Civ. Eng.,
10.1061/(ASCE)CP.1943-5487.0000179, 769–781.
J. Constr. Eng. Manage.
 Russell, B. C., Torralba, A., Murphy, K. P., and Freeman, W. T. (2008).
“Labelme: A database and web-based tool for image annotation.”
Int. J. Comput. Vision, 77(1–3), 157–173.
Sheng, V. S., Provost, F., and Ipeirotis, P. G. (2008). “Get another label?
improving data quality and data mining using multiple, noisy labelers.”
Proc., 14th Association for Computing Machinery (ACM) Special
Interest Group on Knowledge Discovery and Data Mining (SIGKDD)
Int. Conf. on Knowledge Discovery and Data Mining, ACM, New York,
614–622.
Shingles, M., and Trichel, J. (2014). “Industrialized crowdsourcing.” 〈http://
dupress.com/articles/2014-tech-trends-crowdsourcing/〉 (Feb. 1, 2014).
Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. Y. (2008). “Cheap and
fast—but is it good?: Evaluating non-expert annotations for natural lan-
Downloaded from ascelibrary.org by University of California, San Diego on 12/19/15. Copyright ASCE. For personal use only; all rights reserved.
guage tasks.” Proc., Conf. on Empirical Methods in Natural Language
Processing, Association for Computational Linguistics, Stroudsburg,
PA, 254–263.
Sorokin, A., and Forsyth, D. (2008). “Utility data annotation with amazon
mechanical turk.” IEEE Computer Society Conf. on Computer Vision
and Pattern Recognition Workshops, IEEE, Anchorage, AK, 1–8.
Teizer, J., Lao, D., and Sofer, M. (2007). “Rapid automated monitoring of
construction site activities using ultra-wideband.” Proc., 24th Int. Symp.
on Automation and Robotics in Construction, International Association
for Automation and Robotics in Construction (IAARC), Chennai, India,
19–21.
© ASCE 04015035-19
J. Constr. Eng. Manage., 2015, 141(11): 04015035
Teizer, J., and Vela, P. A. (2009). “Personnel tracking on construction sites
using video cameras.” Adv. Eng. Inf., 23(4), 452–462.
Vondrick, C., Patterson, D., and Ramanan, D. (2013). “Efficiently scaling
up crowdsourced video annotation.” Int. J. Comput. Vision, 101(1),
184–204.
Wah, C. (2006). “Crowdsourcing and its applications in computer vision.”
Univ. of California, San Diego.
Wightman, D. (2010). “Crowdsourcing human-based computation.” Proc.,
6th Nordic Conf. on Human-Computer Interaction: Extending
Boundaries, ACM, New York, 551–560.
Yang, J., Vela, P., Teizer, J., and Shi, Z. (2014). “Vision-based tower crane
tracking for understanding construction activity.” J. Comput. Civ. Eng.,
10.1061/(ASCE)CP.1943-5487.0000242, 103–112.
Yuen, J., Russell, B., Liu, C., and Torralba, A. (2009). “Labelme video:
Building a video database with human annotations.” 2009 IEEE 12th
Int. Conf. on Computer Vision, IEEE, Miami, FL, 1451–1458.
Yuen, M.-C., King, I., and Leung, K.-S. (2011). “A survey of crowd-
sourcing systems.” 2011 IEEE 3rd Int. Conf. on Social Computing
(socialcom), IEEE, Boston, 766–773.
Zhai, D., Goodrum, P. M., Haas, C. T., and Caldas, C. H. (2009). “Relation-
ship between automation and integration of construction information
systems and labor productivity.” J. Constr. Eng. Manage., 10.1061/
(ASCE)CO.1943-7862.0000024, 746–753.
J. Constr. Eng. Manage.