Crowdsourcing for a Living

Manuel Valle Torre

Delft University of Technology
Delft, Netherlands
Web Science & Engineering

m.valletorre@student.tudelft.nl

ABSTRACT
Technology and innovation are creating new solutions for
all kinds of tasks, sometimes helping humans in their performance and others completely substituting the need for
their participation, but at the same time new doors are
opening, transforming the job market and economy as
we know it. Crowdsourcing is growing quickly and it is
starting to get attention as a job substitute. In this work
we analyze the literature regarding motivation and rewards of crowdsourcing in an attempt to determine if the
current situation allows this substitution to happen.

1.

INTRODUCTION

In 2013, 75% of the almost 1 million new jobs added to

the US economy were part time according to CNBC [4],
and from 2008 to 2014 the number of adults between 18
and 29 years old who felt as middle class dropped from
54% to 40% [20]. Similar to the Industrial Revolution,
technology is reinventing the job market: medium skill
tasks such as bank tellers or secretaries are slowly being
replaced by innovative devices, just like production lines
did with artisans. While this does not mean that there
will be less jobs, it means that there is a need to adapt
and transform how we work; from a wide variety of options like learning software programming or web development, some people are turning, maybe without even
knowing the concept, to Crowdsourcing [17]. A simple
example is a regular taxi driver becoming an Uber chauffeur.
Crowdsourcing (CS) has proven difficult to be described,
but efforts towards an integrated definition state that it
is a type of participative online activity in which an
individual, an institution, a non-profit organization, or
company proposes to a group of individuals of varying
knowledge, heterogeneity, and number, via a flexible open
call, the voluntary undertaking of a task [5]. This means
that for crowdsourcing to exist theres the need for people to perform a task with the help or use of the web,
which sounds like a fairly easy notion; most people with
access to the Internet are generating content every day
and participating in a community. There is an ample
variety of websites that can be considered in this group,
including Wikipedia 1 , Threadless 2 , iStockPhoto 3 , and
1
https://www.wikipedia.org/ Free encyclopedia built
collaboratively
2
https://www.threadless.com/ T-Shirt design community
3
http://www.istockphoto.com/ Online picture provider

many more; as well as platforms specifically dedicated to

crowdsourcing, such as Amazon Mechanical Turk 4 .

2.

AMAZON MECHANICAL TURK

Amazons Mechanical Turk or MTurk is a platform dedicated to the performance of crowdsourcing; where requesters create Human Intelligence Tasks (HITs) that the
crowdworkers execute. As mentoned before, most of the
times these HITs consist of simple annotations, labelling
of an image, audio transcriptions and many others; which
usually take less than 10 minutes and are rewarded by
payments of a few cents. MTurk allows a wide variety of
options for the execution of tasks, like restricting workers
to perform only once, or splitting a HIT in many equal
tasks for different workers so the results can be aggregated for an "average" [2].

Figure 1: Amazon Mechanical Turk for workers

2.1

Research in MTurk

Most research about CS is performed in MTurk since it

is the biggest dedicated platform today, where scientists
create special tasks or polls and then analyze performance
and compare the results obtained with different variations of said HITs. The concern is that this presents a
pitfall, since MTurk was not designed for research, and
theres no way to randomize the test subjects. Furthermore, certain tasks attract certain kind of workers, for
4
https://www.mturk.com/mturk/welcome One of the
main marketplaces for crowdsourcing

example creative or longer tasks may not be of interest for

workers without a good English level, biasing the results
obtained [25]. But even with this adversities, research on
MTurk has been proven as a valid way to analyze current
situations in crowdsourcing, sometimes even choosing demographics by asking workers to fill a questionnaire and
depending on the answers they are contacted by the researchers or not [11] [13].

3.

PROBLEM

The process of consolidating the concept of crowdsourcing is still in infancy and there are plenty of different
views regarding its many features, such as motivation,
design or execution; in this case we will focus more towards the reward perceived by the typically self-selected
volunteers or part-time workers contributing to the CS
community. For example, some definitions state that
this type of contribution should be done for free or significantly less than the cost of a regular employee [7],
while some state that it needs to fulfill at least one [28]
of Maslows individual needs: either the aforementioned
financial aspect, development individual skills, to simply
have fun, share knowledge, or form part of a community
[21]. On the other hand, some researchers consider that
CS has to be rewarded in an economic way [18], as well
as demographic studies point out that around 20% of the
crowdworkers on MTurk have a household income of less
than $15,000 U. S. dollars [13] (compared to the average $55,000 in the United States [31]) and rely on the
financial reward to make ends meet.
There has been work trying to pinpoint what the motivation for crowdsourcing should be and why it is important; whether it is the money, skill development [3],
task autonomy or work variety [15]. On the contrary, research towards the reward itself is scarce and while some
of them just say that money is the prime reward, for
example stating specific amounts such as remunerating
tasks with $0.01 to $0.10 [7]; others say that the feeling
of belonging, recognition or participation are enough.
Most approaches focus on how to motivate workers to get
the best quality from CS, but fall short when defining the
compensation. As mentioned by Estells and Gonzlez
[5], the aspect of reward is surprisingly not mentioned
enough in relevant literature. One big trend is to note
aspects such as that the integration of consumers in the
process, which is intrinsically the reward [29]. The problem is that there is not that much research directed towards crowdsourcing rewards as a regular job, like a new
alternative to outsourcing tasks, such as Human Resource
screening [10]. This leads to the question for this work:
Can crowdsourcing substitute a traditional job income?
In this review the goal is to find the literature available
and set base to determine if the current state of crowdsourcing, technologically and socially, would be a viable
option for a person to make a living. The fact that crowdsourcing has not been defined concretely makes it difficult
to take in account or disregard both good and bad examples of how it can substitute traditional jobs. Along
with theory and experiment anaysis, we revise different
types of CS and present examples of successful ways that

people use it as a job.

4.

RELEVANT LITERATURE

From the literature available, there are several trends and

topics that can help formulate a conclusion and future
work, since there are many examples like the previously
explained, but it is difficult to find concrete research supporting or refusing the capability of crowdsourcing as an
alternative for a traditional job.

4.1

Motivation in Crowdsourcing

The first thing that comes to mind for what motivates

someone to perform a task is money, but it has been
proven before that in CS it is not a rule that more money
means more or better work, sometimes it even has the
adverse effect [13] [15]. Money is considered an extrinsic
motivator and in some cases it is not the main driver of
the person performing a task, like stated by Rogstadius
et al. [25], where they compared the speed and quality of
task execution in a 3x2 experiment where the worker is
paid either 0, 3 or 10 USD cents for a task performed for
a non-profit or a for-profit organization. The HIT consisted on filling a questionnaire regarding the workers
demographics and only after completing this, the worker
had to identify blood cells infected with malaria, being
either for the Global Health Council or Rimek International pharmaceutical company. As expected, the accuracy of the task did not show any difference when there
was a different remuneration, but the notable finding was
that the precision of execution incremented for the nonprofit, on average and on every level of complexity; with
this they demonstrate how the motivation of helping a
health organization instead of a company has a bigger
effect on performance than money.
In another study based on the effect of financial motivation by Harris in 2011 [10], he analyzed the impact of
incentives in CS by having 4 different task models: normal pay against positive, negative or combined incentives
based on comparing the performance to a golden standard. The HIT consisted on Human Resource screening
by assessing the relevance of a resume, from job-searching
platforms like Monster or Jobing 5 to a certain position.
The results proved that positive and combined incentives
had a beneficial effect on CS tasks, but also proved that
CS is a viable option for mid-skill work like HR screening. For the focus of this work, it is very important
to see the proof that CS could substitute outsourcing,
but theres the missed opportunity of studying from the
workers point of view, because this is exactly a job that
could be full-time for CS participants. Similarily, Pilz
and Gewlad[22] concluded that CS is a viable option for
small and medium enterprises as a source of inexpensive
labor.
As stated before, the research in motivation for participating in online activities similar to crowdsourcing is extensive, ranging from doing tasks just for fun[27] [15],
to share experience or obtain knowledge [8] or even the
exposure to being creative in massive projects [19]. A
5
www.monster.com and www.jobing.com two of the
biggest employment websites

very interesting study by Horton and Chilton [12] states

that workers really analyze cost-benefit when selecting
and performing tasks, so harder tasks for the same pay
are left undone and it is more likely that they perform
easier tasks for the same pay of harder tasks, even if it
takes the same time. The bottom line on this aspect is
that while money is important it is not the only motivator for participation for online participation activities.
The deficiency on all of this literature is that there is no
experiment focused explicitly on workers intending to use
CS as the primary source of income.

4.2

Studies on Rewards

As mentioned before, the problem with crowdsourcing is

that the term involves so many different types that its
difficult to define a valid reward for all of them. One
example is the case of the Fiat Mio, where the company launched a website and invited people to participate on the design and features of the vehicle, obtaining
over 11,000 ideas for the worlds first crowdsourced car;
the 17,000 participants were not required to have any
previous knowledge or design experience, they were just
"potential buyers" [23]. In this case, the only reward was
the integration of the consumer in the creation process,
the feeling of being part of the design and giving input
of the characteristics they really want, which also invites
the user to buy the car in an inmersive way [29].
Another case of different crowdsourcing rewards is iStock,
a micropayment business model website that offers millions of photos, illustrations, videoclips, music and other
kinds of art; content is valued in credits and is completely generated by one of the 140 thousand artists in
the community [14]. When a customer acquires a product, the creator receives the payment minus a percentage
for the website, and eventhough the community tries to
help each other, that feeling of belonging is not the main
motivation or reward. The rewards that iStockers most
value are the development practical and creative skills,
and money, but not only directly from selling their material but also from being able to leverage a non-iStock
employment [3].
In the work by Estelles and Gonzalez [5], they research
over 60 documents around the topic of crowdsourcing,
in efforts to create an integrated definition, and in the
reward section, there is no mention of it being used as a
main source of income. Like mentioned before, it is either
about the impact of rewards in quality or speed, or how
different motivators act together in the performance of
workers. Bottom line is that there is no defined set of
rules or ways to reward crowdsourcing, and if theres no
stability, it is very difficult for someone to consider it as
a primary source of income. On a demographic study by
Ross et al. [26] they established that only around 8%
of MTurk workers can always cover their needs with the
income received.

5.

DISRUPTIVE CROWDSOURCING

Even though theres no solid scientific research on the

field, it is important to mention a type of CS that, as of
today can be the best alternative for a traditional job.
This category is can be considered crowdsourcing since

it is powered by online platforms and Internet networks

but consist on doing real world activities, such as Uber6 ,
Reflik7 , Airbnb8 or Alfred9 .
Uber consists of a mobile app where users request a ride
like a taxi service, but they can see where the car is and
how long will it take to arrive, as well as the best route
and details on the driver; the most important features
for the user include security, simplicity and credit card
payment directly on the app, not to the driver. For the
driver, working with Uber means that they can have their
own schedule and work as much as they want for an average of around $12 an hour, as long as they and their
car meet Ubers requirements10 , such as having a license
and insurance.
Alfred is a personal butler service, scheduled and ondemand, where for 99$ a month, the user requests all
kind of chores and tasks to be done, such as shopping,
dry-clean runs, house activities, etc. In this platform,
the workers also decide their own schedule and can work
the amount of hours that they want, receiving a fixed
wage of 18 USD an hour and even getting benefits if they
work more the equal or more hours than a full-time job
[30]. This is the first service of its kind, where the workers
are officially considered employees instead of independent
contractors like in Uber [16].
Reflik is a referral driven recruiting platform to bring together workers and employers in a social manner, making
around $9 an hour for internal workers [24]. Airbnb is a
platform to list, find and rent lodging; where people post,
many times their primary, rooms or homes. An example
is someone that pays $2,000 per month, but can rent their
apartment for $150 a night, so putting it on Airbnb for
7 days in a month, they reduced their rent to less than
half.
Uber, Alfred, Airbnb and many other platforms that
manage to mix online and real-world activities in a CS
fashion are the main representatives on how it is possible
to arrange a wage well over the poverty line of $11,770
for a single person household [6] and even challenging a
traditional job. This means that being an Uber driver
or Alfred butler during the day, while renting the guest
house for Airbnb some nights may be the future for some
people [17].

6.

CONCLUSION

Disruptive crowdsourcing is a very clear example of the

potential in this type of work, and even if it not sticks
perfectly to the definition since it is not performed entirely online, it leads a way for the rest of the crowdwork.
Like any disruptive element, it has found hard resistance
from the people, especially those who perform the traditional alternatives; in Mexico City 5 Uber cars were
destroyed by taxi drivers and their families, who say that
6

www.uber.com
www.reflik.com
8
www.airbnb.com
9
www.helloalfred.com
10
https://www.uber.com/driver-jobs
7

the online service has reduced their income up to 50% [9].

On a less violent but equally assertive manner, Airbnb is
considered illegal in cities where a person can not rent
living space without an explicit government authorization, but the rules were never made for services like this
so it is complicated to determine and sometimes renters
are prosecuted and fined [1].
The intrinsic motivation like fun or leisure may be the
main impulse for workers to do "brainless" or simple tasks
quick and with good quality; on the other hand, learning,
feeling part of the community or simply having an impact in something the worker is passionate about is more
important in projects such as Open Source Software or
disease symptom recognition. For these mentioned kinds
of tasks, money is not the most relevant stimulus, but
the wave of using CS to make a living is just starting to
take strength around the world [17].
This work is an intent to put together the information
and studies available on crowdsourcing motivation and
rewards, to give the reader the possibilty to have an
understanding and generate a position in the research
question: Can crowdsourcing substitute a traditional job
income? The short answer is no, current technology restricts CS in massive platforms such as MTurk, so for
now it is not possible to count on it as a unique income
for a household. But what matters is the potential and
real capability of a crowdsourcing driven workforce.

7.

[10]

[11]

[12]

[13]
[14]
[15]

FUTURE WORK

Better worker segmentation and task recommendation

seems to be the most logical way to start the transformation of the crowdsourcing market towards a stable economic model, but there is a lack of literature to back or
disprove these implementations. For now, the tools provided by MTurk and other crowdsourcing platforms are
deficient for its growth, and workers are the main affected
group.

8.

[9]

[16]

[17]
[18]

REFERENCES

[1] Airbnb. Airbnb what legal and regulatory issues

should i consider before hosting on airbnb?, 2015.
[2] Amazon. Amazon Mechanical Turk overview, 2016.
[3] D. C. Brabham. Moving the crowd at istockphoto:
The composition of the crowd and motivations for
participation in a crowdsourcing application. First
monday, 13(6), 2008.
[4] CNBC. Obamacare, tepid us growth fuel part-time
hiring, 2013.
[5] E. Estells-Arolas and
F. Gonzlez-Ladrn-De-Guevara. Towards an
integrated crowdsourcing definition. J. Inf. Sci.,
38(2):189200, Apr. 2012.
[6] A. S. for Planning and Evaluation. ASPE 2015
poverty guidelines, 2015.
[7] G. G. V. Frank Kleemann and K. Rieder.
Un(der)paid innovators: The commercial
utilization of consumer work through
crowdsourcing. In Science, Technology &
Innovation Studies 4, pages 526, 2008.
[8] R. A. Ghosh, R. Glott, B. Krieger, and G. Robles.

[19]

[20]
[21]

[22]

[23]
[24]
[25]

Free/libre and open source software: Survey and

study, 2002.
L. Gomez Flores. La Jornada destrozan autos de
uber cerca del aeropuerto, 2015.
C. Harris. Youre hired! an examination of
crowdsourcing incentive models in human resource
tasks. Proceedings of the Workshop on
Crowdsourcing for Search and Data Mining
(CSDM) at the Fourth ACM International
Conference on Web Search and Data Mining
(WSDM), pages 1518, 2011.
J. Heer and M. Bostock. Crowdsourcing graphical
perception: Using mechanical turk to assess
visualization design. In Proceedings of the SIGCHI
Conference on Human Factors in Computing
Systems, CHI 10, pages 203212, New York, NY,
USA, 2010. ACM.
J. J. Horton and L. B. Chilton. The labor
economics of paid crowdsourcing. In Proceedings of
the 11th ACM Conference on Electronic Commerce,
EC 10, pages 209218, New York, NY, USA, 2010.
ACM.
P. G. Ipeirotis. Analyzing the amazon mechanical
turk marketplace. XRDS, 17(2):1621, Dec. 2010.
iStock. iStock about us, 2016.
N. Kaufmann, T. Schulze, and D. Veit. More than
fun and money. worker motivation in
crowdsourcing: a study on Mechanical Turk. In
Proceedings of the Seventeenth Americas
Conference on Information Systems, pages 111,
2011.
M. Kosoff. Business Insider 2 harvard students were
sick of their dirty apartments, so they built a
company that will do your chores for you, 2015.
S. Kunst. The Daily Beast is crowdsourced labor
the future of middle class employment?, 2014.
G. La Vecchia and A. Cisternino. Collaborative
workforce, business process crowdsourcing as an
alternative of bpo. In Proceedings of the 10th
International Conference on Current Trends in
Web Engineering, ICWE10, pages 425430, Berlin,
Heidelberg, 2010. Springer-Verlag.
K. Lakhani and R. G. Wolf. Why hackers do what
they do: Understanding motivation and effort in
free/open source software projects. 2003.
T. Luhby. CNN - Economy americas disappearing
middle class, 2014.
A. H. Maslow. A theory of human motivation. In
Originally Published in Psychological Review, pages
370396, 1943.
D. Pilz and H. Gewald. Does money matter?
motivational factors for participation in paid- and
non-profit-crowdsourcing communities. 2013.
A. Pinheiro. Idea Connection fiat mio, the worlds
first crowdourced car, 2011.
Reflik. Reflik about us, 2016.
J. Rogstadius, V. Kostakos, A. Kittur, B. Smus,
J. Laredo, and M. Vukovic. An assessment of
intrinsic and extrinsic motivation on task
performance in crowdsourcing markets. In L. A.
Adamic, R. A. Baeza-Yates, and S. Counts, editors,

ICWSM. The AAAI Press, 2011.

[26] J. Ross, L. Irani, M. S. Silberman, A. Zaldivar, and
B. Tomlinson. Who are the crowdworkers?:
Shifting demographics in mechanical turk. In CHI
10 Extended Abstracts on Human Factors in
Computing Systems, CHI EA 10, pages 28632872,
New York, NY, USA, 2010. ACM.
[27] R. M. Ryan and E. L. Deci. Self-determination
theory and the facilitation of intrinsic motivation,
social development, and well-being. American
psychologist, 55(1):68, 2000.
[28] O. Stewart, J. M. Huerta, and M. Sader. Designing
crowdsourcing community for the enterprise. In
Proceedings of the ACM SIGKDD Workshop on
Human Computation, HCOMP 09, pages 5053,
New York, NY, USA, 2009. ACM.
[29] A. Tran, S. ul Hasan, and J. Park. Crowd
participation pattern in the phases of a product
development process that utilizes crowdsourcing.
Industrial Engineering & Management Systems,
11(3):266275, 2012.
[30] C. Woodward. Xconomy home-assistant startup
alfred launches, payinggasp!real wages, 2014.
[31] WorldBank. Gdp per capita (current us$), 2015.