Welcome to Scribd, the world's digital library. Read, publish, and share books and documents. See more
Standard view
Full view
of .
Save to My Library
Look up keyword
Like this
0 of .
Results for:
No results containing your search query
P. 1
Community innovation - crowd sourcing in software testing

Community innovation - crowd sourcing in software testing

Ratings: (0)|Views: 175|Likes:
Published by Crowdsourcing.org

More info:

Published by: Crowdsourcing.org on May 14, 2013
Copyright:Attribution Non-commercial


Read on Scribd mobile: iPhone, iPad and Android.
See more
See less


PASS Technologies AG, Dufourstrasse 91, CH-8008 Zürich
In today's rapidly growing and truly global softwareindustry, success is now more than ever dependent onthe quality of the software. A good quality product or service leads to satisfied and loyal customers - themain ambition of every company. A business thatcompromises on quality loses loyal customers in thelong run. The challenge that most IT managers andexecutives have is common: how to effectively testsoftware applications on time and within budget while balancing resources? Under the right circumstances,“crowdsourcing” offers a rapid and cost-effectiveapproach to software testing.
Traditionally, software testing was a neglectedafterthought. It was a tedious chore for developers oncetheir real work was done and was based on a set of formal verification mechanisms that were costly toapply. Rigorous testing is particularly significant for mission critical applications because a formal approach,though a proven method for reducing defects insoftware, requires highly specialized talent that isunavailable in most organizations. Furthermore,formalized testing cannot be scaled to cover the scopeof the complex software deployed most in the modernIT landscape.
The changing landscape
An increasing number of today’s tech-savvyconsumers, often termed as “digital natives”, a armedwith gadgets who are not only connected to the Internetat all times – many of them live much of their livesthere.This generation wants to decide which channel to usefor doing any kind of business interaction and expectthe provider to adapt accordingly in terms of servicequality, security and ease of use. They expect quick adaptability to their changing needs, and have littlesense of brand loyalty due to the plethora of availableoptions at hand. All these changes are naturally pushing the software testing industry to find animproved strategy focusing on the customer experience.Several trends are changing the perception in favor or crowd sourced testing. The emergence and massiveadoption of the Agile software developmentmethodology that advocates shorter “sprints”(iterations) of development brings software testerscloser to the creative development process.The advent of web applications, whereby software isdeployed on the Internet and bugs become apparentvery quickly. Set with a budgetary limit (testers are paid a certain rate per valid bugs found), ‘crowdsourced’ testing is a highly compelling innovation thatcan be used to complement existing test cycles, used in pre-release deployments, and even post-deploymenttesting.
Many hands make software work 
The stakes for Microsoft, which was outlining itsOffice 2010 product strategy, were extremely high.According to Microsoft's earnings statements,Microsoft Office productivity suite generates morerevenue than any other business division, says GreggKeizer, who covers Microsoft and general technologynews for Computerworld.
Months before Microsoft released Office 2010 productivity suite, 9 million people downloaded the beta version to test the software and to providefeedback. Through this program, Microsoft collected 2million valuable comments and insights from thosetesters.
Denise Carlevato, a Microsoft usability engineer for 10years, and her colleagues from Microsoft's VirtualResearch Lab observed how people used new features.Their objective was to make Microsoft Office fit theway millions of people used their product and to helpthem work better. It was a massive, controlledcrowdsourcing project.According to Carlevato,
“That's just what you have to do to cater to asbroad an audience as possible”. This is not especially easy; to retain the loyalty of millions is aherculean task. “While the team was building it and giving insights and information, it was still all like a big jigsaw puzzle. Toward the end when thisall came together I had the opportunity to go out and see how the web apps were actually going tolive with real people.”
The scenario
Developing a new software product is always exciting,especially to watch ideas take form and truly become areality. Sometimes a fresh perspective or an innovativeuse case is all it takes to turn a product from good togreat. However, when it comes to testing, we often findourselves in unchartered waters wondering if the product will actually work in the diverse customer landscapes. It is virtually impossible to test the vastnumber of devices and configurations of software thatweb-based software can run on today. Truly robusttesting is time consuming, and ensuring that every possible permutation and combination of features,localizations, and platforms works, as intended isnearly impossible.Often times, comprehensive testing is a challenge and buggy code is delivered to the customer. For example,
PASS Technologies AG, Dufourstrasse 91, CH-8008 Zürich
if a Software-as-a-Service (SaaS) application does notrender in a particular browser or a critical software toolfails to deliver its intended functionality, a bug fix or a patch is promised and the vicious cycle starts all over again. Either way, the customer withstands the worst of inadequate testing, especially when faced with theescalating costs of software maintenance, performance,etc. For the software development company,ramifications include distress around brand image, perceived quality, relationship and potential future projects, trust, etc.
Enter crowd sourced software testing
Welcome to the new world of crowd sourced testing,an emerging trend in software engineering that exploitsthe benefits, effectiveness, and efficiency of crowdsourcing and the cloud platform towardssoftware quality assurance and control. With this newform of software testing, the product is put to testunder diverse platforms, which makes it morerepresentative, reliable, cost-effective, fast, and aboveall, bug-free.Crowd sourced testing, conceived around a Testing-as-a-Service (TaaS) framework, helps companies reachout to a community to solve problems and remaininnovative. When it comes to testing softwareapplications, crowdsourcing helps companies reduceexpenses, reduce time to market and increase resourcesfor testing, manage a wide range of testing projects,test competence needs, exigency to resolve higher defects rates, and use 3
party’s test environment tosubside the project requirements.It differs from traditional testing methods in that thetesting is carried out by a number of different testersfrom across the globe, and not by locally hiredconsultants and professionals. In other words, crowdsourced testing is a form of outsourced software testing,a time-consuming activity, to testers around the world,thus enabling small startups to use ad-hoc quality-assurance teams, even though they themselves couldnot afford traditional quality assurance testing teams.
Why does it work?
To understand why crowd sourced testing works, it isimportant to understand the set of biases that infestmost testers and test managers around the world. This phenomenon is called, “The Curse of Knowledge,” a phrase used in a 1989 paper in
The Journal of Political  Economy
. It means that for a particular subject expert,it is nearly impossible to imagine and look beyond theknowledge the tester has acquired i.e. the set of concepts, beliefs and scenarios that the tester knows or  predicts. As a result, it is particularly challenging tothink outside the box and conceive the various ways atypical end user would use particular software.
Psychological underpinnings
This phenomenon has been empirically proven throughan infamous experiment conducted by a StanfordUniversity graduate student of psychology, Elizabeth Newton. She illustrated the phenomenon through asimple game, people were assigned to one of two roles,namely tappers and listeners. Each tapper was to selecta well-known song, such as “Happy Birthday,” and tapthe rhythm on a table. The listeners were to guess thesong from the taps. However, before the listenersguessed the song, tappers were asked to predict the probability that listeners would guess correctly. They predicted 50%. Over the course of the experiment, 120songs were tapped out, but listeners guessed only threeof the songs correctly - a success rate of merely 2.5%The explanation is as follows: when tappers tap, it isimpossible for them to avoid hearing the tune playingalong to their taps. Meanwhile, all the listeners couldhear is a kind of bizarre Morse code. The problem isthat once we know something, we find it impossible toimagine the other party not knowing it.Extrapolating this experiment to software testing, mosttesters conduct a battery of tests that they feel isrepresentative and that captures the set of end-user scenarios for how the software would be used. Thereality is far from this. Any expert tester would assetthat it is impossible to capture the complete set of scenarios that an end user may throw at a softwaresystem. As a result, critical paths of the code under certain scenarios go untested, which leads to softwaremalfunctioning, production system crashes, customer escalations, long hours of meetings, debugging, etc.Crowd sourced testing circumvents all these headaches by bringing a comprehensive set of code coveragemechanisms and end user scenarios during the designand development stages of software engineering,during which the cost of modification is meager. Thisresults in identifying critical use cases early on and providing for those contingencies, which reducessoftware maintenance costs later on during and after  productive deployment. Besides progressive codecoverage, the quality and depth of software testingamong various vital software modules is achieved,which ultimately results in a higher code quality,among other benefits.
The framework 
At the heart of crowd sourced testing is the communitythat tests a given software product. The communityencompasses people from diverse backgrounds,cultures, geographies, languages, all with a diverseapproach to software usage. The community,represented by a diverse and extended user space, testsany given software by putting it to use under realisticscenarios, which a tester in the core test team may not
PASS Technologies AG, Dufourstrasse 91, CH-8008 Zürich
 be able to envision, given a tester’s constraints, such aslimited bounds of operation, knowledge, scenarios.Thus, it is easy to observe the broad set of usage patterns that put the software under intense scrutiny.Crowdsourcing software testing draws its benefits fromdelegating the task of testing a web or software project,while in development, on to a number of Internet users,to ensure that the software contains no defects.The method of crowd sourced testing is particularlyuseful when the software is user-centric, whensoftware’s success and adoption is determined by itsuser feedback. It is frequently implemented withgaming or mobile applications, when experts who may be difficult to find in one place are required for specifictesting, or when the company lacks the resources or time to carry out internal testing.The spectrum of issues that such test efforts coulduncover within a short lead-time is particularlynoteworthy. Such testing efforts yield productiveresults with reasonable costs. Often times, the productcompany pays only for those valid reported bugs.Hence, the Return on Investment (ROI) is highcompared to the traditional means of software testing.
How does it work?
Most crowd sourced testing companies provide the platform for the testing cycles. Clients specify the typeof tests that they wish to have performed and the typesof devices that the software product must be tested on.Testers complete a profile, indicating the skills theyhave, the devices to which they have access to, and thecountries where they reside. Once a tester hascompleted his profile, he/she can check the projectdashboard for a listing of projects and releases that areavailable for testing. The dashboard may also includesample test scenarios, additional tools and scripts,instructions for testers about what is expected fromthem, etc. Usually, the testers are required to submit aQA plan, which outlines both high level test cases anddetailed test scenarios. The plan may also includewhether or not the test can be automated and expectedresults.A qualified Project Manager, who is typically a provencommunity leader or a person from the client/the platform company, reviews such plans, and approvesor amends such plans to cater to the client’s specifictesting requirements.Each project includes an explanation and access to aforum where bugs and issues are discussed andadditional questions can be asked. Testers document bug reports and are rated based on the quality of their reports. The amount the testers earn increases as their rating increases.The community combines aspects of collaboration andcompetition, as members work to finding solutions tothe stated problem. Forums facilitate networking anddiscussion of bugs or relevant issues; rating systemsallow for recognition of a job well done, which helps participants gain credibility and improved career.The crowd source testing team is usually in addition tothe organization's testing team, and not a replacement.
Checks & Balances
Security is a crucial element to crowd source testing.More often than not, confidential customer informationis exposed to testers during application testing. Any breach of this data can lead to serious damage, both tothe brand and the business. Test data managementensures the availability and security of test data byobfuscating sensitive information for large-scaletesting engagements. Masking such information or creating ‘test-only’ data helps maintain privacy andsecurity while using crowd sourced testing services.In almost all cases, the testers are required to sign a Non-Disclosure Agreement (NDA) when they join thecommunity. The NDA forbids them from talking aboutcustomers, their products or specific defects, bothoffline and online on Facebook, Twitter, personal blogsor anywhere outside the confines of the private testing platform. Beyond that, the customers can upload acustomized NDA, which testers must sign beforeviewing the customer’s project. For projects thatrequire a high level of security, a pre-screened list of white hat engineers, that have a long professionalrelationship with the platform company are selected.Furthermore, standardized communication patternshelp users secure their data and gain confidence in their testing vendors, which results in a seamless transition.By combining an internal, permanent team of testerswith a crowd of experienced software testers workingfrom around the globe, superior quality in testing isdelivered. By constantly filtering the network of testersto accept only experienced software testing professionals, applicants without formal training andsignificant professional experience are eliminated. Thisensures the quality and the validity of the bugs reported.Last but not the least, tests are dispatched to individualtesters based on their experience, available material,and languages mastered. The testers and test projectexposure are continually monitored to ensure bothquality and integrity, not only of the test results, butalso of the associated environment.
What to look out for?
Crowd sourced testing is best when the product under development is consumer-centric rather thanenterprise-centric, such as gaming or web driven

You're Reading a Free Preview

/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->