COMMUNITY INNOVATION - CROWD SOURCING IN SOFTWARE TESTINGA WHITEPAPER
PASS Technologies AG, Dufourstrasse 91, CH-8008 Zürich
if a Software-as-a-Service (SaaS) application does notrender in a particular browser or a critical software toolfails to deliver its intended functionality, a bug fix or a patch is promised and the vicious cycle starts all over again. Either way, the customer withstands the worst of inadequate testing, especially when faced with theescalating costs of software maintenance, performance,etc. For the software development company,ramifications include distress around brand image, perceived quality, relationship and potential future projects, trust, etc.
Enter crowd sourced software testing
Welcome to the new world of crowd sourced testing,an emerging trend in software engineering that exploitsthe benefits, effectiveness, and efficiency of crowdsourcing and the cloud platform towardssoftware quality assurance and control. With this newform of software testing, the product is put to testunder diverse platforms, which makes it morerepresentative, reliable, cost-effective, fast, and aboveall, bug-free.Crowd sourced testing, conceived around a Testing-as-a-Service (TaaS) framework, helps companies reachout to a community to solve problems and remaininnovative. When it comes to testing softwareapplications, crowdsourcing helps companies reduceexpenses, reduce time to market and increase resourcesfor testing, manage a wide range of testing projects,test competence needs, exigency to resolve higher defects rates, and use 3
party’s test environment tosubside the project requirements.It differs from traditional testing methods in that thetesting is carried out by a number of different testersfrom across the globe, and not by locally hiredconsultants and professionals. In other words, crowdsourced testing is a form of outsourced software testing,a time-consuming activity, to testers around the world,thus enabling small startups to use ad-hoc quality-assurance teams, even though they themselves couldnot afford traditional quality assurance testing teams.
Why does it work?
To understand why crowd sourced testing works, it isimportant to understand the set of biases that infestmost testers and test managers around the world. This phenomenon is called, “The Curse of Knowledge,” a phrase used in a 1989 paper in
The Journal of Political Economy
. It means that for a particular subject expert,it is nearly impossible to imagine and look beyond theknowledge the tester has acquired i.e. the set of concepts, beliefs and scenarios that the tester knows or predicts. As a result, it is particularly challenging tothink outside the box and conceive the various ways atypical end user would use particular software.
This phenomenon has been empirically proven throughan infamous experiment conducted by a StanfordUniversity graduate student of psychology, Elizabeth Newton. She illustrated the phenomenon through asimple game, people were assigned to one of two roles,namely tappers and listeners. Each tapper was to selecta well-known song, such as “Happy Birthday,” and tapthe rhythm on a table. The listeners were to guess thesong from the taps. However, before the listenersguessed the song, tappers were asked to predict the probability that listeners would guess correctly. They predicted 50%. Over the course of the experiment, 120songs were tapped out, but listeners guessed only threeof the songs correctly - a success rate of merely 2.5%The explanation is as follows: when tappers tap, it isimpossible for them to avoid hearing the tune playingalong to their taps. Meanwhile, all the listeners couldhear is a kind of bizarre Morse code. The problem isthat once we know something, we find it impossible toimagine the other party not knowing it.Extrapolating this experiment to software testing, mosttesters conduct a battery of tests that they feel isrepresentative and that captures the set of end-user scenarios for how the software would be used. Thereality is far from this. Any expert tester would assetthat it is impossible to capture the complete set of scenarios that an end user may throw at a softwaresystem. As a result, critical paths of the code under certain scenarios go untested, which leads to softwaremalfunctioning, production system crashes, customer escalations, long hours of meetings, debugging, etc.Crowd sourced testing circumvents all these headaches by bringing a comprehensive set of code coveragemechanisms and end user scenarios during the designand development stages of software engineering,during which the cost of modification is meager. Thisresults in identifying critical use cases early on and providing for those contingencies, which reducessoftware maintenance costs later on during and after productive deployment. Besides progressive codecoverage, the quality and depth of software testingamong various vital software modules is achieved,which ultimately results in a higher code quality,among other benefits.
At the heart of crowd sourced testing is the communitythat tests a given software product. The communityencompasses people from diverse backgrounds,cultures, geographies, languages, all with a diverseapproach to software usage. The community,represented by a diverse and extended user space, testsany given software by putting it to use under realisticscenarios, which a tester in the core test team may not