Le Services SI 622: Evaluation of Systems and ne i er USABILITY TESTING REPORT ath ,C rg AND FINAL RECOMMENDATIONS a aG rim Ga Team Newsies:

u, F Caroline Moran, Catherine Le lu an Danlu Fu, Garima Garg D April 2010

lin o ar ,C

Mo e

an r

Why Usability Testing?

To understand real users
How they interact with the system e eL How they feel about the systemn i

lin o ar ,C

Mo e

an r

a ,C Experts know too much rg a aG rim to fix problem after launch Too expensive Ga u, F nlu Da Adapted From lecture slides

er th

Usability Testing

METHODS Ga ,
D nl a Fu u

aG rim

,C rg a

he at

Le ne ri

lin o ar ,C

Mo e

an r

Participant Selection
Target Participants
Age range: 20 - 30 years old Novice users

Recruiting

5 participants aG Our personalim r networks, our colleagues and graduate a students G other schools at u, F Incentives: $25 gift cards lu

,C rg a

he at

Le ne ri

lin o ar ,C

Mo e

an r

an D

Process
lin o When, Where, Who and What ar ,C April 6 and April 7, 2010 Le Ann Arbor.com office conferencee in room er1 tester 1 moderator, 1 note-taker and ath Laptops with a trial version of Camtasia Studio ,C rg a aG rim Ga u, F nlu Da

Mo e

an r

Process
During the testing
Introduction Pre-questionnaire 5.5 Tasks Debriefing

nl a

Fu u

Ga ,

aG rim

,C rg a

he at

Le ne ri

lin o ar ,C

Mo e

an r

Process
Tasks

Task 1: Register a new account Le Task 2: Find a specific article and e in share it on Facebook er area Task 3: Find a crime report in your th

lin o ar ,C

Mo e

an r

G achoose a concert Task 4: Find and rim promotions Task 5: Find some Ga u, F nlu Da

3a: [if user does not use the Neighborhoods section] g, the Neighborhoods section Find the crime report in ar

a C

Results and Analysis

lin Jakob Nielsens (1993) rating scale aro usability for C problems to rank the severity e,of each key eL finding: rin he at ,C rg a aG rim Ga u, F nlu Da

Mo e

an r

Usability Testing

KEY FINDINGS Ga ,
D nl a Fu u

aG rim

,C rg a

he at

Le ne ri

lin o ar ,C

Mo e

an r

Finding 1

lin o Neighborhoods Section does not display ar ,C correctly in all browsers Le ne i er ath ,C rg a aG rim Ga u, F IE 8.0 browser nlu Firefox 3.6.2 browser on a D the same computer

Mo e

an r

Finding 2

lin o Users have difficulty navigating within the ar ,C Events Calendar Le 3 out of 5 users failed the task to e an event in find r ethere is an actual Most users do not notice that ath calendar ,C rg a Most users did not navigate using the links at the top aG of the Events Calendar section rim an article about an event through Users often found Ga the Entertainment section u, F nlu Da

Mo e

an r

Finding 2
Le ne ri lin o ar ,C

Mo e

an r

nl a

Fu u

Ga ,

aG rim

,C rg a

he at

Finding 3

in lare o In the Neighborhoods Section, users ar ,C unclear as to which geographiceregion of Ann L e choice Arbor is the most appropriate rin All users who navigated to the Neighborhoods tab he at paused and expressed , C confusion about the region rg labels a aG rim Ga u, F nlu Da

Mo e

an r

Finding 3

lin the It was equally intuitive for users to aro seek C neighborhood crime information ,under the Le News sub-section of the drop-down menu. ne i er Arbor-wide crime Unfortunately, it shows an Ann ath report rather than the , C neighborhood-specific crime rg report. a aG rim Ga u, F nlu Da

Mo e

an r

Finding 4

lin cityo Hard to differentiate and find Ann Arbor ar ,C wide news versus neighborhood-specific Le news ne i er While searching for neighborhood-specific crime news, th aArbor-wide crime report some users found the Ann ,C rg whereas others found a more neighborhood-specific a crime report. a G m There is noari to access the neighborhood-specific way ,G crime report from the Crime/News section and there is Futo access the Ann Arbor-wide crime report nolu way an each Neighborhoods section. D from

Mo e

an r

Finding 5

lin o Users have difficulty finding the Events ar ,C Calendar Le Users often went to the Entertainment section first to ne i er look for local events such ash t concerts aon the right side of the The link is actually located ,C rg drop-down menu, but not very visible. a aG rim Ga u, F nlu Da

Mo e

an r

Usability Testing

RECOMMENDATIONS Ga ,
D nl a Fu u

aG rim

,C rg a

he at

Le ne ri

lin o ar ,C

Mo e

an r

Recommendation 1

lin Run browser compatibility testing aro ,C Further testing should be done to ensure that all Le consistent across features and are compatible withe n and i all common browsers and platforms. er ath ,C rg a aG rim Ga u, F nlu Da

Mo e

an r

Recommendation 2

lin o Increase visibility of the map of AnnrArbor a ,C area for the Neighborhoods section Le A map of Ann Arbor overlaid with the corresponding ne ri eonce the Neighborhoods geographic labels is displayed athfeedback should be section is reached, but more ,C provided at the onset. rg a Either a zip code query or map in the drop-down menu aG rim would alleviate the confusion users face when trying to Ga determine which geographic label link is the most u, F relevant for them to click. nlu Da

Mo e

an r

Recommendation 3

lin Include an Events Calendar link in aro the ,C Entertainment section Le There seems to be a strong andine intuitive association of Events (such as concerts) with r Entertainment he the at section ,C rg One such example being Boston's a aG alternative news source, the Weekly rim Dig (http://www.weeklydig.com) Ga u, F nlu Da

Mo e

an r

Recommendation 4

linthe o To focus attention, organize events rby a ,C week's most highly featured recommendations Le In LA Weeklys calendar, only a handful of events are ne i er considered staff picks each week. ath ,C rg a aG rim Ga u, F nlu Da

Mo e

an r

Recommendation 5

in la glance Add more cues to describe events aro at ,C Icons to the left describe the event type, but no legend Le is given. ne i er such as "Concert, Describe the event type in tags, ath who are not able to Folk, or the Ark", for those ,C g determine the eventrtype from titles. a aG rim Ga u, F nlu Da

Mo e

an r

Recommendation 6

lin Increase consistency of access to aro articles ,C across site Le It is equally intuitive to choose one route or another, ne i er or the Neighborhoods such as choosing the Crime h ttab tab when looking to findCa reports about neighborhood g, crime. ar If articles are cross-linked in multiple related subaG rim sections, initial mistakes in information-seeking can be Ga there will be a safety net of multiple minimized since u, F modes and routes to access the same information. nlu Da

Mo e

an r

Usability Testing

DISCUSSION Ga ,
D nl a Fu u

aG rim

,C rg a

he at

Le ne ri

lin o ar ,C

Mo e

an r

New Users

lin o 5 participants who have never visited ar AnnArbor.com ,C Le use these In real cases new users probably wont ne very first visit specific functions in our taskseon their ri During their navigation through news articles, they ath ,C might notice some non-news related sections such as g rwhich might further indicate what a the Events Calendar, aG else they can do on AnnArbor.com. rim Further Research Ga u, Recruiting some regular users who never used F functions other than reading the news nlu Da

Mo e

an r

o ar ,C All of the participants came to Ann Arbor Le within the past year ne i er not familiar with the 2 participants said that theyh t are area distribution of Ann Ca because they are Arbor newcomers here g, ar aG Further Research rim Ga local young participants so as to Recruit several u, the problem of area location for F evaluate nlu Neighborhoods section a D

Unfamiliarity with the city of oran M Ann Arbor ine l

Task Wording

lin o Many participants clicked Entertainment ar , instead of the Events CalendaretoCfind concert eL events rin It might be wise to test othere th kinds of events which are a not related to any tab on top menu so as to eliminate ,C the impact of otherarg options as much as possible. aG rim Further Research Ga u, tasks such as finding a workshop can be used Example F to lreplace the concert task nu Da

Mo e

an r

Usability Testing

SUMMARY Ga ,
D nl a Fu u

aG rim

,C rg a

he at

Le ne ri

lin o ar ,C

Mo e

an r

 Findings:

lin o Participants have difficulty finding their neighborhood ar ,C within Ann Arbor. Le They also have difficulty with understanding whether ne i er they should go to the Entertainment or the Events athinformation. Calendar section for concert ,C rg a Navigation within the Events Calendar section didnt aG work as users expected. rim news is only in the main News Ann Arbor-wide Ga category, while related neighborhood-specific news is u, F only available in Neighborhood sections. nlu Da

Mo e

an r

 Recommendations:

lin Unambiguous geographic identification in ro a neighborhood sections. ,C Levisible to users The Events Calendar should be more ne such as i and integrated with other sections er Entertainment. ath ,C Better and clearer navigation within Events Calendar rg a should also be supported so as to provide flexible G aevents. paths to desired rim Increased consistency across the general News sections Ga and Neighborhood sections should be considered for u, F future improvements. nlu Da

Mo e

an r

Conclusion

FINAL RECOMMENDATIONS Ga ,
D nl a Fu u

aG rim

,C rg a

he at

Le ne ri

lin o ar ,C

Mo e

an r

Recommendation 1

Create a more concrete image of what AnnArbor.com is in lin order to enforce the legitimacy of what they are o ar producing. Manage user expectations about AnnArbor.coms function ,C Le and role ne i Mix of old journalistic standards + rnew media community e ath building ,C rg Articles and features should be: a
Clearly labeled as written by staff aG Thoroughly fact-checked, proofread and held to journalistic rim a standards , G

Mo e

an r

StrongerF u staff role in:

nl Increasing dialogue with community members Da Evaluating the most important events and news

Mo itself, Include basic information pages about the website ine group them, and make them easy to find. ol ar Such pages would include (and are not limited to): ,C Le About Us/Staff page ne i Site Map er FAQs ath ,C Help documentation rg a Acceptable Use Policy G a Legend/Key to icons used on the site rim Group these pages and link to them in an easy-to-find Ga u, location F nlu Linka them where relevant (e.g. link to the Acceptable to D Use Policy in the account creation section)

Recommendation 2

an r

Mo More intuitive and effective use of verbal and visual labeling. ine ol Wording ar ,C Section names can be unclear or very broad Le Word order: only the Deals & Moreine r subcategories are he alphabetized in the drop-down tmenu a ,C rg a aG rim Ga u, F nlu Da

Recommendation 3

an r

Mo More intuitive and effective use of verbal and visual labeling. ine ol Icons (and/or Tags) ar ,C Look into how much theyre used and how useful they are Le ne Images should be more descriptiveriand obvious he Some encompass many subcategories at Some are not obvious at first glance ,C rg a aG rim GaCommunity, Business, Arts & Entertainment, Food & Drink, and Music Event icons for: Recreation, u, F nlu Da

Recommendation 3

an r

Mo More intuitive and effective use of verbal and visual labeling. ine ol Color Usage ar ,C Use colors to differentiate between staffe L members and user contributors on their author pages ne i er easy to tell apart Use very different colors so theyre ath ,C rg a aG rim Ga u, F nlu Da

Recommendation 3

an r

Recommendation 4
Increase User Account Control

lin o ar Provide users with greater control over account functions ,C Le and content contribution ne subscriptions i Allow users to manage their newsletter er ath Allow users to edit/delete comments ,C rg Allow users to delete userpics instead of only replacing a them aG rim Ga u, F nlu Da From Facebook
Edit Delete

Mo e

an r

Recommendation 5
lin o ar Why is it the right rail? Why not a left rail? ,C Le F Reading Pattern: ne i er ath ,C rg a aG rim Ga u, F nlu Da Screenshot from http://www.useit.com/alertbox/reading_pattern.html
Locations of Navigational Tools

Mo e

an r

Recommendation 5
Locations of Navigational Tools

Visibility of the Events Calendar

Le Have the actual calendar on the home page ne the drop-down menu i Include a link in the Entertainment section of er ath ,C rg a aG rim Ga u, F nlu Da

lin o ar ,C

Mo e

an r

Recommendations 6 & 7 o
6. Clarify and Enforce Validity of Ratings & Selection lin o ar Have separate Staff and User ratings C

M e

an r

g, 7. Advertisement Placement ar aG Find out where users prefer to have ads located on a rim webpage Ga Users were split 50/50 on where they liked the ads to be u, F Find out where ads are most effective on the webpage nlu Da

Make it easier to tell where ratings come frome in what numbers L and Staff ratings: provide a rubric to explain what merits specific ratings ne i er wherein more ratings = User ratings: provide the wisdom of h t crowds, a higher level of trust (e.g. Amazon.com ratings) C

Recommendations 8 & 9 o
M e

an r

8. Streamline user account creation, verification and lin o log-in process ar

,C When verifying new account, fill in the Le username and/or password for new users when they iclick on the verification ne er link ath Have more prompts for fields on the log-in page ,C rg 9. Create a dedicated archives section a G a month in which they were posted Group articles by the rim for users to browse older articles This will make ita G easier u, Do it now while it is still relatively manageable (<1 year of F content) nlu Da

nl a

Fu u

Ga ,

Le Thank you! ne i er Q & tA ah ,C rg a aG rim

lin o ar ,C

Mo e

an r