This action might not be possible to undo. Are you sure you want to continue?
Bryan Dosono & Evan Abdalla HCDE 417: Usability Research
TABLE OF CONTENTS
Executive Summary ......................... 2 Background ........................................ 2 Purpose ............................................... 2 Research Questions ....................... 2 Participant Recruitment .............. 3 Method ................................................. 3 Technique .......................................... 3 Roles of the Research Team ....... 3 Project Timeline .............................. 3 Pilot ....................................................... 4 Test Procedures .............................. 4 Data Collection ................................. 4 Results ................................................. 5 Recommendations ........................... 5 Findings and Discussion .............. 5 Design Suggestions ........................ 6 Limitations and Strengths ........... 7 Conclusion .......................................... 7 Appendix ............................................. 8 F1: Website Interface .................... 8 F2: Participant Demographics .. 9 F3: Participant Feedback ............. 10 F4: Dendrogram .............................. 11
GO PURPLE. BE GOLD.
Card sorting is a method that sheds insight to how users think the information and navigation should be presented within the product. A pilot user requirements study that took the form of an open group card sort evaluated the site structure and navigation of the UW HUB website. Many students who use the site regularly have expressed their frustration in accessing critical site information and functionality (e.g. making facility reservations). Results show that among 24 recruited participants, 16 “disagreed” and 4 “strongly disagreed” with the current organization of the site. A dendrogram of all eight testing sessions was generated in WebSort to present the hierarchical similarities between the various cards used. An average of 15 cards were used throughout the eight sessions, and the four most common clusters of pages were: “about,” “services,” “entertainment,” and “contact.” The research team recommends (1) removing redundant menu items and specify those pages further, (2) including new pages for commonly expected features, and (3) moving existing menu items into a different cluster of pages. The development team hopes to see its research-‐based recommendations influence future iterative design changes throughout the lifecycle of the UW HUB website.
The University of Washington Husky Union Building is a quintessential space on campus that brings together students from various groups to partake in activities that enhance student life on campus. Utilized by a population of over 40,000 students, staff, and faculty, the UW HUB completed its renovation and reopened to the public in 2012. However, a renovated HUB did not come with a renovated website. We as the research team visit the UW HUB website frequently for a variety of reasons, but its current navigation layout makes it difficult to access the information we need. We cannot quickly find commonly used features like making room reservations, and other information pertaining to critical UW HUB services remain hidden in multilevel menus.
We are interested in investigating how can the site map of the UW HUB website be redesigned to allow for users to quickly find the information they need. We want to know what pages are identified as the highest priority for regular users, and which pages should be organized and grouped together to provide a more streamlined experience for regular users. We hope to apply the usability techniques we learned from our HCDE 417 Usability Research course to solve these questions and even address other user requirements activities in the future.
We recruited participants through an online directory of Registered Student Organization (RSO) student leaders from various academic departments, multicultural clubs, and other special interest groups. Receiving input from active student users that represent diverse campus communities provided a more holistic perspective as to how the general student body used the UW HUB website. The user profile that we targeted composed of regular users who have visited the site at least once per quarter. Feedback from active users was assumed to be more relevant and rich in content than students who never visited the site and do not fully understand its functionality.
A card sort activity gave insight into how users think the information and navigation should be within the site. By way of an open group card sort, participants were given instructions to free-‐list all items relevant to the product by naming every page item they can think of that should be associated with the HUB website. Users were then tasked to sort cards into groups that they felt were appropriate and asked to describe each group. Unlike other generic usability tests that typically involve one user evaluating a product, groups of three students in this exercise allowed for collaboration and discourse between participants. Eight open card sort activities were conducted with groups of three students that fit the desired user profile of enrolled UW students that have visited the HUB website at least twice within the past month. Upon the completion of all card sorts, we collected feedback from 24 participants.
The moderator guided the exercise and discussion of the card sorting activity, addressed any difficulties the users had while not influencing their decisions, and kept track of time to ensure everything went according to plan. The recorder took notes on user input and feedback, photographed participants’ card sorting clusters, and used an audio device to capture interesting quotes that participants may have.
Identified research question: October 24, 2012 Conducted pilot test: November 1, 2012 Created test plan and kit: November 15, 2012 Recruited and confirmed participants: November 16, 2012 Conducted user requirements activities: November 21, 2012 Transcribed and analyzed data: November 28, 2012 Presented findings and recommendations: December 6, 2012
Prior to conducting our first card sort activity, we ran a pilot to ensure the viability of the test design. We contacted three students listed on the public RSO directory from various campus organizations via email to participate in our pilot. Although two of the recruited participants have completed online card sorting activities in the past, they both expressed that they preferred our pen and paper card sort activity because of its hands-‐on interactivity. We discovered that although each participant had their own mental mapping of how the UW HUB website should be structured, the final organization represents a collaborative compromise of the best design elements from each participants’ ideas. The participants understood our instructions and their role in our usability study, and as expected, their final group card sort design did not match the current layout of the site.
Prior to each card sort activity, we had each participant fill out a consent form and pre-‐test questionnaire. The starting state for each card sorting activity began with an empty table and unwritten sticky notes. After the orientation script was read aloud by the moderator, participants populated the sticky notes with names of relevant pages of the UW HUB website. The participants then grouped pages together by relevance. Participants were allowed to use a laptop we provided to reference the current layout of the site. After the card sort was completed, the recorder photographed the final cluster of sticky notes that represented the participants’ suggested site map of the UW HUB website, noted the length of the activity, and tracked the amount of cards used as performance measures for successful task completion. We concluded the activity by distributing a post-‐test questionnaire to the participants and compensated them accordingly for their time.
Once all eight testing sessions have been completed, we transcribed audio from the card sorting activities and noted interesting feedback. We manually entered the card sorting clusters from each session on WebSort (http://websort.net), an online card sort tool that is widely used by information architects, web designers, and content strategists. Through WebSort, we created dendrogram visualizations that evaluated how pages were most commonly grouped throughout all sessions. This data, cross-‐ referenced with participant feedback, was used to make informed design improvements to the UW HUB website’s final information architecture. The post-‐ test questionnaire collected participant preferences by way of Likert scales, which indicated their degree of agreement or disagreement with various claims about the UW HUB website and the card sorting exercise itself. The range of the rating scale in the post-‐test questionnaire was 1-‐5. Because we focused on collecting data from the card sorting activity and not evaluating the mannerisms of actual user participants, observations of non-‐performance and non-‐preference data (e.g. body language, eye tracking, etc.) were not relevant to the card sorting activity and were not included in the transcription of the usability tests.
According to data generated from pre-‐test questionnaires in Figures 2, our efforts in recruiting students from various ethnicity, gender, ability, and class standing have resulted in a diverse participant pool. The post-‐test questionnaires revealed the general dissent of the navigation architecture of the UW HUB website. According to Figure 3, 16 “disagreed” and 4 “strongly disagreed” with the current organization of the site. A dendrogram of all eight testing sessions generated in Figure 4 via WebSort presents the hierarchical similarities between the various cards used. An average of 15 cards were used throughout the eight sessions, and the four most common clusters of pages were: “about,” “services,” “entertainment,” and “contact.”
Findings and Discussion
The page definitions of the current site organization caused much confusion among our users. One of the participants stated, “As a international student, one of the most important aspects of the HUB website for me is clarity in its language. It’s hard for students with my background of English as a second language to find the information we need when it’s mislabeled.” Six out of eight groups reported inconsistencies with how menu items were named. Another user stated, “These redundant menu items are confusing. I wish this site was more descriptive with its page titles.” Pages such as “Entertainment” had redundant sub menu items named “Entertainment,” which prompted us to change the naming convention used for that sub menu item from “Entertainment” to a further specified page (e.g. “Games Area”). The current organization of the UW HUB website was also lacking critical features that students expected to see on its homepage. Integration with social media was also a reoccurring comment in four different card sort sessions. One of the participants asserted, “I would like to see social media pages included so that I can connect to the HUB beyond its website.” Moreover, there was at least one participant in all eight sessions that mentioned the need for an easily identifiable facilities reservation page. “I make facility reservations on behalf of my RSO every week and it’s frustrating that I can’t do that on the home page,” boasted a frequent student user of the site.
Several pages were moved out of their current cluster and placed into a new one altogether. For instance, seven out of the eight testing sessions demanded the “Contact” page to be a main menu instead of a sub menu under “Discover the HUB.” Participants took this suggestion further and added a “Map” page under this new “Contact” page because the content of directions provided to get to the HUB was perceived as relevant contact information. “Seeing a visual map of where the HUB is located on campus is way more helpful to me than a long text of directions,” said a user. In fact, five groups preferred to see an actual map of where the HUB was located on campus than reading turn-‐by-‐turn text directions. A couple groups removed the “Directory” page as a main menu item and placed it under the “Contact” page, saying that a directory of HUB staffed personnel was not as important of a priority than the actual contact information of the HUB.
1. (Boxed) Remove redundant menu items and specify those pages further. 2. (Starred) Include new pages for commonly expected features. 3. (Arrowed) Move existing menu items into a different cluster of pages.
Limitations and Strengths
The primary weakness of the card sort that we discovered, primarily after the usability testing and during the transcription phase of the project, was the nature of the subjective data we had to interpret. The subjective issues we had with understanding definitions of synonymous pages could have been combatted by asking them to write a short description of any cards that were not explicit. Overall, the card sort helped us to more easily identify what type of pages users thought were related, which pages were prioritized over others, and what content needed to be changed altogether. Additionally card sorts can be overlaid and combined into one larger site map, which helped to combine the collective opinions of many people. Using an open card sort was useful because it provided our participants the ability to create their own content and site architecture. The result of this open card sort method was the creation of new content, such as social media buttons, which we had never even considered to add. Considering the overall discontent of the current site organization, an open card sort was clearly the best option to solve our research problem.
We conducted eight rounds of card sorting, which total 24 participants who fit the targeted user profile of active UW HUB website visitors. The research team performed a correlation analysis by inputting pen and paper card sort data into computer software (WebSort) to generate a dendrogram of commonly clustered pages across the eight sessions of card sorting. The interpretation of the pre-‐test and post-‐test questionnaires distributed on the testing site illustrated both the diversity of the participant sample and noticeable trends in their preferences. We communicated high level findings from the card sorting activity and summarized key results based off of interesting feedback from UW HUB website users to recommend an improved navigation layout intended for the actual designer of the UW HUB website. The development team hopes to see its research-‐based recommendations influence future iterative design changes throughout the lifecycle of the UW HUB website.
Figure 1: UW HUB Website Interface
Screen capture taken on October 24, 2012.
Participant responses to the Pre-‐Test Questionnaire in Figure 5 have been distributed in a pie chart to illustrate diversity within the participant pool.
Participant responses to the Post-‐Test Questionnaire in Figure 7 have been visualized in a bar graph to illustrate trends in user preferences. “The content of the current UW HUB website is well organized.”
“I prefer my proposed site structure to the current site structure of the UW HUB website.
Generated a hierarchical tree of commonly grouped pages.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.