Professional Documents
Culture Documents
net/publication/279179170
CITATIONS READS
5 1,864
5 authors, including:
Some of the authors of this publication are also working on these related projects:
All content following this page was uploaded by Stefano Federici on 09 November 2015.
tended to rate their satisfaction with higher (better) scores than more than 5,000 individual SUS responses, Sauro and Lewis
users with shorter terms of experience. (2012) found the overall mean score of the SUS to be 68 with a
In summary, researchers and practitioners assess user satis- standard deviation of 12.5.The Sauro and Lewis CGS assigned
faction by means of questionnaires, but there are only a few grades as a function of SUS scores ranging from F (absolutely
empirical studies that have systematically analyzed the vari- unsatisfactory) to A+ (absolutely satisfactory), as follows:
ation of the outcomes of satisfaction scales when filled out
• Grade F (0–51.7)
by users with different amounts of experience in the use of a
• Grade D (51.8–62.6)
product (Kortum & Johnson, 2013; Lindgaard & Dudek, 2003;
• Grade C– (62.7–64.9)
McLellan et al., 2012; Sauro, 2011).
• Grade C (65.0–71.0)
• Grade C+ (71.1–72.5)
1.2. The System Usability Scale • Grade B– (72.6–74.0)
Several standardized tools are available in the literature to • Grade B (74.1–77.1)
measure satisfaction (for a review, see Borsci et al., 2013). • Grade B+ (77.2–78.8)
An increasing trend favors the use of short scales due to their • Grade A– (78.9–80.7)
speed and ease of administration, either as online surveys for • Grade A (80.8–84.0)
customers or after a usability test. One of the most popular is • Grade A+ (84.1–100)
the System Usability Scale (SUS; Lewis, 2006; Sauro & Lewis, Although they should be interpreted with caution, the grades
2011; Zviran, Glezer, & Avni, 2006), which has been cited in from the CGS provide an initial basis for determining if a mean
more than 600 publications (Sauro, 2011) and is considered SUS score is below average, average, or above average (Sauro
an industry standard. Its popularity among HCI experts is due & Lewis, 2012).
to several factors, such as its desirable psychometric proper-
ties (high reliability and demonstrated validity), relatively short
length (10 items), and low cost (free; Bangor, Kortum, & Miller, 1.3. Ultrashort Scales: The UMUX and UMUX-LITE
2008; McLellan et al., 2012). Although the SUS is a quick scale, practitioners sometimes
The 10 items of the SUS were designed to form a need to use reliable scales that are even shorter than the SUS
unidimensional measure of perceived usability (Brooke, 1996). to minimize time, cost, and user effort. “This need is most
The standard version of the questionnaire has a mix of positive pressing when standardized usability measurement is one part
and negative tone items, with the odd-numbered items having of a larger post-study or online questionnaire” (Lewis, 2014,
a positive tone and the even-numbered items having a negative p. 676). As a consequence, quite recently, two new scales have
tone. Respondents rate the magnitude of their agreement with been proposed as shorter proxies of the SUS: the Usability
each item using a 5-point scale from 1 (strongly disagree) to Metric for User Experience (UMUX; see Table 1), a four-item
5 (strongly agree). To compute the overall SUS score, (a) each tool developed and validated by Finstad (2010, 2013), and the
item is converted to a 0–4 scale for which higher numbers indi- UMUX-LITE composed of only the two positive-tone questions
cated a greater amount of perceived usability, (b) the converted from the UMUX (Lewis et al., 2013, this issue). The scale for
scores are summed, and (c) the sum is multiplied by 2.5. This the UMUX items has 7 points, from 1 (strongly disagree) to 7
process produces scores that can range from 0 to 100. (strongly agree).
As Lewis (2014) recently stated, there are still lessons
that have to be learned about SUS, in particular about its
dimensionality. Despite the SUS having been designed to be TABLE 1
unidimensional, several researchers recently showed that the Items of the UMUX and UMUX-LITE
items of SUS might load in two dimensions: usability and
Item No. — Scale Item Content
learnability (Bangor et al., 2008; Borsci, Federici, & Lauriola,
2009; Lewis & Sauro, 2009; Lewis, Utesch, & Maher, 2013, this Item 1 – UMUX [This system’s] capabilities meet
issue; Sauro & Lewis, 2012). Since 2009, however, there have my requirements.
been reports of large-sample SUS data sets for which two-factor Item 1 – UMUX-LITE
structures did not have the expected item-factor alignment Item 2 – UMUX Using [this system] is a
(Items 4 and 10 with Learnable, all others with Usable), indi- frustrating experience.
cating a need for further research to clarify its dimensional Item 3 – UMUX [This system] is easy to use.
structure and the variables that might affect it (Lewis, 2014). Item 2– UMUX-LITE
In recent years, the growing availability of SUS data from a Item 4 – UMUX I have to spend too much time
large number of studies (Bangor et al., 2008; Kortum & Bangor, correcting things with [this
2012) has led to the production of norms for the interpretation system].
of mean SUS scores, for example, the Curved Grading Scale
(CGS; Sauro & Lewis, 2012). Using data from 446 studies and Note. UMUX = Usability Metric for User Experience.
486 S. BORSCI ET AL.
Some findings have shown the UMUX to be bidimensional 1.4. Research Goals
as a function of the item tone, positive versus negative (Lewis, The use of short scales as part of UX evaluation proto-
2013; Lewis et al., 2013), despite the intention to develop a cols could sensibly reduce the costs of assessment, as well as
unidimensional scale. The UMUX’s statistical structure might users’ time and effort to complete the questionnaires. Currently,
be an artifact of the mixed positive/negative tone of the items few studies have investigated the relationship among the SUS,
and in practice might not matter much. In light of this, both the UMUX, and UMUX-LITE, and none have analyzed their relia-
UMUX and its reduced version, the UMUX-LITE, are usually bilities as a function of different amounts of interaction with a
interpreted as unidimensional measures. product.
By design (using a method similar to but not exactly the same The primary goal of this article was to analyze the variation
as the SUS), the overall UMUX and UMUX-LITE scores can of SUS, UMUX, and UMUX-LITE outcomes when completed
range from 0 to 100. Their scoring procedures are as follows: concurrently by users with different levels of experience in the
use of a website. To reach this goal, we pursued three main
• UMUX: The odd items are scored as [score − 1] and
objectives. First, we aimed to explore the variation of UMUX
even items as [7 − score]. The sum of the item scores
and UMUX-LITE outcomes when administered to users with
is then divided by 24 and multiplied by 100 (Finstad,
two different levels of product experience. Second, we aimed
2010).
to observe whether, at different levels of product experience,
• UMUX-LITE: The two items are scored as [score − 1],
the correlations among the SUS, UMUX, and UMUX-LITE
and the sum of these is divided by 12 and multiplied by
were stable, with particular interest in the generalizability of
100 (Lewis et al., 2013). For correspondence with SUS
Equation 1. Finally, we checked whether the levels of respon-
scores, this sum is entered into a regression equation
dents’ product experience affected the dimensional structure of
to produce the final UMUX-LITE score. The follow-
the SUS. It may be that the Learnable scale might not emerge
ing equation combines the initial computation plus the
until respondents have sufficient experience with the product
regression to show how to compute the recommended
they are rating. To achieve these aims, we performed two stud-
UMUX-LITE score from the ratings of its two items.
ies with the three standardized usability metrics to measure
the self-report satisfaction of end-users with different levels of
UMUX − LITE = .65(([Item 1 score] experience with an e-learning web platform known as CLab
(http://www.cognitivelab.it).
+ [Item 2 score] − 2)100/12) + 22.9.
(1)
2. METHODOLOGY
Prior research (Finstad, 2010, 2013; Lewis et al., 2013) has Students enrolled in the bachelor’s degree of psychology
shown that the SUS, UMUX, and UMUX-LITE are reliable program at the University of Perugia are strongly encouraged to
(Cronbach’s alpha between .80 and .95) and correlate signif- use the CLab target platform as an e-learning tool. Commonly,
icantly (p < .001). In the research reported to date, UMUX students access it at least once a week for several reasons:
scores have not only correlated with the SUS but also had a sim- for instance, to look for information about courses and exam
ilar magnitude. However, for the UMUX-LITE, it is necessary timetables, to sign in for mandatory attendance classes, to
to use the preceding formula (Equation 1) to adjust its scores to download course materials, to book a test/exam, or to post a
achieve correspondence with the SUS (Lewis et al., 2013). For question and discuss issues with the professor.
the rest of this article, reported UMUX-LITE values are those For each study, a sample of volunteer students was asked to
computed using Equation 1. Thus, the literature on the UMUX assess the interface after different times of usage (based on their
and UMUX-LITE (Finstad, 2010; Lewis et al., 2013) suggests date of subscription to the platform) by filling out the SUS,
that these two new short scales can be used as surrogates for the UMUX, and UMUX-LITE questionnaires, presented in a ran-
SUS. dom order. The Italian version of the SUS used in Borsci et al.
Currently, three studies (Kortum & Johnson, 2013; McLellan (2009) was administered. In addition, translations and retransla-
et al., 2012; Sauro, 2011) have investigated the relation- tions were made by an independent group of linguistic experts
ship between SUS scores and amount of product experience. to produce Italian versions of the UMUX and UMUX-LITE.
The results of these studies have consistently indicated that Participants of the two studies were invited to fill out the
more experienced users had higher satisfaction outcomes (SUS scales 2 months (Study 1) or 6 months (Study 2) after they
scores). Notably, researchers have not yet studied this effect on first accessed CLab. In these studies, participants received the
the outcomes of quick scales such as the UMUX and UMUX- same instruction before the presentation of the questionnaires:
LITE. The comparative analyses of these instruments were “Please, rate your satisfaction in the use of the platform on the
performed mainly to validate the questionnaires, without con- basis of your current experience of use” [Per favore, in base
sidering the effect of the different levels of experience in the alla tua attuale esperienza d’uso, valuta la tua soddisfazione
use of a website (Finstad, 2010; Lewis et al., 2013). nell’utilizzo della piattaforma].
USER SATISFACTION IN ERA OF USER EXPERIENCE 487
The two studies were organized to measure different times Therefore, we expected students in Study 2 (cumulative
of participants’ exposure to CLab, thus measuring different UX condition) to rate the CLab with all the three scales as
moments of UX acquisition, as follows: more satisfactory compared to users in the first study due
to their greater product exposure (6 months). Concurrently,
• Study 1, carried out 2 months after the students first
we expected participants with greater levels of product
accessed CLab. The participants’ number of access
frequency of use to rate the CLab with all the scales as
times and interaction with the platform (time exposure)
more satisfactory than participants with lower levels of
ranged from eight (once a week) to 56 (once a day).
use.
• Study 2, carried out 6 months after the students first
accessed CLab. The participants’ number of access
times and interaction with the platform (time exposure)
2.2. Data Analysis
ranged from 24 (once a week) to 168 (once a day).
For each study, principal components analyses were per-
The two studies were reviewed and approved by the formed to assess the SUS’ dimensionality—focusing on
Institutional Review Board of the Department of Philosophy, whether the item alignment of the resulting two-component
Social and Human Sciences and Education, University of structure was consistent with the emergence of Learnable and
Perugia. All participants provided their written informed con- Usable components. Only if this expected pattern did not
sent to participate in this study. No minors/children were emerge did we plan to follow up with a multidimensional latent
enrolled in this study. The study presented no potential risks. class item response theory model (LC IRT) to more deeply
test the dimensional structure of the scale (Bacci, Bartolucci,
& Gnaldi, 2014; Bartolucci, 2007; Bartolucci, Bacci, & Gnaldi,
2.1. Hypotheses
2014). The primary purpose of this additional analysis would
Concerning the effect of usage levels on the SUS’ dimen-
be to confirm if a unidimensional structure was a better fit to the
sional structure, we expected the following:
data than the expected bidimensional structure—an assessment
that is not possible with standard principal components analy-
H1: The SUS dimensionality would be affected by the level of
sis because it is impossible to rotate a unidimensional solution
experience acquired by the participants with the product
(Cliff, 1987).
before the administration of the scale.
Descriptive statistics (mean, standard deviation) and Pearson
correlation analyses among the scales were performed to com-
The second hypothesis concerns the correlations among the
pare the outcomes of the three scales and observe their rela-
tools (SUS, UMUX, UMUX-LITE). Recently (Finstad, 2010;
tionships. Moreover, one-way analyses of variance (ANOVAs)
Lewis et al., 2013), researchers have reported strong correla-
were carried out to assess the effect of experience on user satis-
tions among the SUS, UMUX, and UMUX-LITE. There is
faction as measured by the SUS, UMUX, and UMUX-LITE for
as yet, however, no data on the extent to which the correla-
each study, and a final comprehensive ANOVA was conducted
tions might be affected by the users’ levels of acquired product
to enable comparison of results between the two studies. The
experience. Thus, we expected the following:
MultiLCIRT package of R software by Bartolucci et al. (2014)
was used to estimate the multidimensional LC IRT models. All
H2: Significant correlations among the overall scores of the ®
other analyses were performed using IBM SPSS 22.
SUS, UMUX, and UMUX-LITE, for all studies indepen-
dent of the administration conditions, that is, different
amounts of interaction time with the target website. 3. THE STUDIES
TABLE 2
Principal Components Analysis of the System Usability Scale in Study 1 Showing One- and Two-Component Solutions
Eigenvalues Extraction Unidimensional Bidimensional
TABLE 3
Unidimensional and Bidimensional LC IRT Models for System Usability Scale Data: Number of Latent Classes (C), Estimated
Maximum Log-Likelihood (), Number of Parameters (#par), Bayesian Information Criterion (BIC) Index
Unidimensional Model Bidimensional Model
TABLE 5
Means and Standard Deviations of Overall Scores of the SUS, UMUX, and UMUX-LITE for Study 1 With 95% (and Associated
CGS Grades)
Scale M SD 95% CI Lower 95% CI Upper
SUS 70.88 (C) 6.703 69.9 (C) 71.8 (C+)
UMUX 84.66 (A+) 12.838 82.8 (A) 86.5 (A+)
UMUX-LITE 73.83 (B − −) 9.994 72.4 (C+) 75.3 (B)
Note. SUS = System Usability Scale; UMUX = Usability Metric for User Experience; CI = confidence interval.
TABLE 6
Principal Components Analysis of the System Usability Scale in Study 2
Items Usability Learnability
8. I found this website very cumbersome/awkward to use. .910 .121
1. I would like to use this website frequently. .869 .105
5. I found the various functions in this website were well integrated. .800 .122
3. I thought this website was easy to use. .769 .226
7. I would imagine that most people would learn to use this website .754 .258
very quickly.
2. I found this website unnecessarily complex. .739 .273
6. I thought there was too much inconsistency in this website. .708 .183
9. I felt very confident using this website. .683 .106
10. I needed to learn a lot of things before I could get going with .133 .841
this website.
4. I think that I would need assistance to be able to use this website. .206 .753
Note. Loadings greater than .4 in bold.
TABLE 7
Unidimensional and Bidimensional Latent Class Item Response Theory Models for System Usability Scale Data: Number of
Latent Classes (C), Estimated Maximum Log-Likelihood (), Number of Parameters (#par), Bayesian Information Criterion
(BIC) Index
Unidimensional Model Bidimensional Model
TABLE 8
Correlations Among SUS, UMUX and UMUX-LITE in Study 2 (With 95% CI)
UMUX CI UMUX-LITE CI
TABLE 9
Means and Standard Deviations of Overall Scores of the SUS, UMUX, and UMUX-LITE for Study 2 With 95% CIs (and
Associated Curved Grading Scale Grades)
Scale M SD 95% CI Lower 95% CI Upper
SUS 75.24 (B) 13.037 72.6 (B) 77.0 (B+)
UMUX 87.69 (A+) 10.291 85.6 (A+) 89.8 (A+)
UMUX-LITE 76.45 (B) 9.943 74.4 (B) 78.5 (B+)
Note. SUS = System Usability Scale; UMUX = Usability Metric for User Experience; CI = confidence interval.
TABLE 10
Main Effects and Interactions for Combined Analysis of Variance
Scale Effect Outcome
SUS Main effect of duration F(1, 263) = 10.7, p = .001
Main effect of frequency of use F(2, 263) = 30.9, p < .0001
Duration × Frequency interaction F(2, 263) = 15.8, p < .0001
UMUX Main effect of duration F(1, 263) = 17.4, p < .0001
Main effect of frequency of use F(2, 263) = 22.2, p < .0001
Duration × Frequency interaction F(2, 263) = 3.4, p = .035
UMUX-LITE Main effect of duration F(1, 263) = 4.7, p = .03
Main effect of frequency of use F(2, 263) = 16.8, p < .0001
Duration × Frequency interaction F(2, 263) = 4.7, p = .01
Note. SUS = System Usability Scale; UMUX = Usability Metric for User Experience.
TABLE 11
Summary of Study Outcomes by Hypothesis
Hypothesis Result Meaning
Hypothesis 1 Supported SUS dimensionality was affected by the different levels of product experience. When
administered to users with less product experience, the SUS had a unidimensional
structure, whereas it had a bidimensional structure for respondents with more product
experience.
Hypothesis 2 Supported All the three scales were strongly correlated, independent of the administration
conditions.
Hypothesis 3 Supported Participants with more product experience were more satisfied than those with less
product experience regardless of whether that experience was gained over a duration of
exposure or by frequency of use.
Note. SUS = System Usability Scale.
unidimensional scale, so we recommend against par- detailed investigation of their relationships and psychometric
titioning it into Usable and Learnable components in properties.
that context. Moreover, practitioners should anticipate
the satisfaction scores of newer users will be signif-
ACKNOWLEDGMENTS
icantly lower than the scores of more experienced
people. We thank Dr. James R. Lewis, senior human factors engineer
• When the SUS is administered to more experienced at IBM Software Group and guest editor of this special issue,
users, the scale appears to have bidimensional prop- for his generous feedback during the preparation of this paper.
erties, making it suitable to compute both an overall
SUS score and its Learnable and Usable components.
The overall level of satisfaction will be higher than that REFERENCES
Bacci, S., Bartolucci, F., & Gnaldi, M. (2014). A class of multidimen-
among less experienced users. sional latent class IRT models for ordinal polytomous item responses.
• Due to their high correlation with the SUS, in par- Communications in Statistics – Theory and Methods, 43, 787–800.
ticular, the UMUX and UMUX-LITE overall scores doi:10.1080/03610926.2013.827718
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation
showed similar behaviors.
of the System Usability Scale. International Journal of Human-Computer
• If using one of the ultrashort questionnaires as a proxy Interaction, 24, 574–594. doi:10.1080/10447310802205776
for the SUS, the UMUX-LITE (with its adjustment Bartolucci, F. (2007). A class of multidimensional IRT models for test-
formula) appears to provide results that are closer in ing unidimensionality and clustering items. Psychometrika, 72, 141–157.
doi:10.1007/s11336-005-1376-9
magnitude to the SUS than the UMUX, making it the Bartolucci, F., Bacci, S., & Gnaldi, M. (2014). MultiLCIRT: An R package
more desirable proxy. for multidimensional latent class item response models. Computational
Statistics & Data Analysis, 71, 971–985. doi:10.1016/j.csda.2013.05.018
The UMUX and UMUX-LITE are both reliable and valid Borsci, S., Federici, S., & Lauriola, M. (2009). On the dimensionality of the
proxies of the SUS. Nevertheless, Lewis et al. (2013) suggested System Usability Scale (SUS): A test of alternative measurement models.
using them in addition to the SUS rather than instead of the Cognitive Processing, 10, 193–197. doi:10.1007/s10339-009-0268-9
Borsci, S., Kuljis, J., Barnett, J., & Pecchia, L. (2014). Beyond the user
SUS for critical usability work, due to their recent development
preferences: Aligning the prototype design to the users’ expectations.
and still limited employment. In particular, on the basis of our Human Factors and Ergonomics in Manufacturing & Service Industries.
results, we recommend that researchers avoid using only the doi:10.1002/hfm.20611
UMUX for their analysis of user satisfaction because, at least in Borsci, S., Kurosu, M., Federici, S., & Mele, M. L. (2013). Computer systems
experiences of users with and without disabilities: An evaluation guide for
the current study, this scale seemed too optimistic. In the forma- professionals. Boca Raton, FL: CRC Press. doi:10.1201/b15619-1
tive phase of design or in agile development, the UMUX-LITE Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan,
could be adopted as a preliminary and quick tool to test users’ B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability
reactions to a prototype. Then, in advanced design phases or in evaluation in industry (pp. 189–194). London, UK: Taylor & Francis.
Cliff, N. (1987). Analyzing multivariate data. San Diego, CA: Harcourt Brace
summative evaluation phases, we recommend using a combina- Jovanovich.
tion of the SUS and UMUX-LITE (or UMUX) to assess user Dix, A., Finlay, J., Abowd, G. D., & Beale, R. (2003). Human–computer
satisfaction with usability (note that because the UMUX-LITE interaction. Harlow, UK: Pearson Education.
was derived from the UMUX, when you collect the UMUX Finstad, K. (2010). The usability metric for user experience. Interacting with
Computers, 22, 323–327. doi:10.1016/j.intcom.2010.04.004
you also collect the data needed to compute the UMUX-LITE). Finstad, K. (2013). Response to commentaries on “The Usability Metric
Over time this could lead to a database of concurrently collected for User Experience.” Interacting with Computers, 25, 327–330.
SUS, UMUX, and UMUX-LITE scores that would allow more doi:10.1093/iwc/iwt005
USER SATISFACTION IN ERA OF USER EXPERIENCE 495
Goodman, L. A. (1974). Exploratory latent structure analysis using Sauro, J., & Lewis, J. R. (2011). When designing usability question-
both identifiable and unidentifiable models. Biometrika, 61, 215–231. naires, does it hurt to be positive? In Proceedings of CHI 2011
doi:10.1093/biomet/61.2.215 (pp. 2215–2223). Vancouver, Canada: ACM. doi:10.1145/1978942.
Hassenzahl, M. (2005). The thing and I: Understanding the relationship between 1979266
user and product. In M. Blythe, K. Overbeeke, A. Monk, & P. Wright Sauro, J., & Lewis, J. R. (2012). Quantifying the user experience:
(Eds.), Funology: From usability to enjoyment (Vol. 3, pp. 31–42). Berlin, Practical statistics for user research. Burlington, MA: Morgan
Germany: Springer. doi:10.1007/1-4020-2967-5_4 Kaufmann.
Hassenzahl, M., & Tractinsky, N. (2006). User experience—A research agenda. Schwarz, G. (1978). Estimating the dimension of a model. The Annals of
Behaviour & Information Technology, 25, 91–97. doi:10.1080/01449290 Statistics, 6, 461–464. doi:10.2307/2958889
500330331 Tractinsky, N. (1997). Aesthetics and apparent usability: Empirically assess-
Hassenzahl, M., Wiklund-Engblom, A., Bengs, A., Hägglund, S., & ing cultural and methodological issues. In Proceedings of CHI 1997
Diefenbach, S. (2015). Experience-Oriented and Product-Oriented (pp. 115–122). Atlanta, GA: Association for Computing Machinery.
Evaluation: Psychological Need Fulfillment, Positive Affect, and Product doi:10.1145/258549.258626
Perception. International Journal of Human-Computer Interaction, 31, van der Linden, W., & Hambleton, R. K. (1997). Handbook of modern item
530–544. response theory. New York, NY: Springer.
ISO 9241-11:1998 Ergonomic requirements for office work with visual display Zviran, M., Glezer, C., & Avni, I. (2006). User satisfaction from commercial
terminals – Part 11: Guidance on usability. web sites: The effect of design and use. Information & Management, 43,
ISO 9241-210:2010 Ergonomics of human-system interaction – Part 210: 157–178. doi:10.1016/j.im.2005.04.002
Human-centred design for interactive systems.
Kortum, P. T., & Bangor, A. (2012). Usability ratings for everyday prod-
ucts measured with the System Usability Scale. International Journal
of Human–Computer Interaction, 29, 67–76. doi:10.1080/10447318.2012.
681221 ABOUT THE AUTHORS
Kortum, P. T., & Johnson, M. (2013). The relationship between levels of user Simone Borsci is a Research Fellow in Human Factors
experience with a product and perceived system usability. Proceedings of at Imperial College of London NHIR-Diagnostic Evidence
the Human Factors and Ergonomics Society Annual Meeting, 57, 197–201.
doi:10.1177/1541931213571044 Cooperative group. He has over 10 years of experience as psy-
Lallemand, C., Gronier, G., & Koenig, V. (2015). User experience: A con- chologist and HCI expert in both industry and academia. He
cept without consensus? Exploring practitioners’ perspectives through has worked as the UX lead of the Italian Government’s working
an international survey. Computers in Human Behavior, 43, 35–48. group on usability, and as a researcher at University of Perugia,
doi:10.1016/j.chb.2014.10.048
Lazarsfeld, P. F., & Henry, N. W. (1968). Latent structure analysis. Boston, MA: Brunel University, and Nottingham University.
Houghton, Mifflin. Stefano Federici is currently Associate Professor of General
Lee, S., & Koubek, R. J. (2012). Users’ perceptions of usability and aesthetics Psychology at the University of Perugia. He is the coordi-
as criteria of pre- and post-use preferences. European Journal of Industrial
nator of a research team of CognitiveLab at University of
Engineering, 6, 87–117. doi:10.1504/EJIE.2012.044812
Lewis, J. R. (2006). Usability testing. In G. Salvendy (Ed.), Handbook of human Perugia (www.cognitivelab.it). His research is focused on assis-
factors and ergonomics (pp. 1275–1316). New York, NY: Wiley & Sons. tive technology assessment processes, disability, and cognitive
Lewis, J. R. (2013). Critical review of “The Usability Metric for and human interaction factors.
User Experience.” Interacting with Computers, 25, 320–324.
doi:10.1093/iwc/iwt013
Michela Gnaldi is currently an Assistant Professor of
Lewis, J. R. (2014). Usability: Lessons learned . . . and yet to be learned. Social Statistics at the Department of Political Sciences of the
International Journal of Human-Computer Interaction, 30, 663–684. University of Perugia. Her main research interest concerns mea-
doi:10.1080/10447318.2014.930311 surement in education. On this topic, she participated in several
Lewis, J. R., & Sauro, J. (2009). The factor structure of the System Usability
Scale. In M. Kurosu (Ed.), Human centered design (Vol. 5619, pp. 94–103). research projects of national interest in Italy and in the UK,
Berlin, Germany: Springer. doi:10.1007/978-3-642-02806-9_12 where she has been working as a statistician and researcher at
Lewis, J. R., Utesch, B. S., & Maher, D. E. (2013). UMUX-LITE: When there’s the National Foundation for Educational Research.
no time for the SUS. In Proceedings of CHI 2013 (pp. 2099–2102). Paris, Silvia Bacci has been an Assistant Professor of Statistics at
France: ACM. doi:10.1145/2470654.2481287
Lindgaard, G., & Dudek, C. (2003). What is this evasive beast we the University of Perugia. Her research interests concern latent
call user satisfaction? Interacting with Computers, 15, 429–452. variable models, with a special focus on models for categorical
doi:10.1016/S0953-5438(02)00063-2 and longitudinal/multilevel data, latent class models, and item
McLellan, S., Muddimer, A., & Peres, S. C. (2012). The effect of experience on
response theory models. Now she participates in a FIRB project
System Usability Scale ratings. Journal of Usability Studies, 7, 56–67.
Nering, M. L., & Ostini, R. (2010). Handbook of polytomous item response funded by the Italian government.
theory models. New York, NY: Taylor & Francis. Francesco Bartolucci is Full Professor of Statistics at the
Petrie, H., & Bevan, N. (2009). The evaluation of accessibility, usability, and Department of Economics of University of Perugia. He is the
user experience. In C. Stephanidis (Ed.), The universal access handbook
(pp. 299–314). Boca Raton, FL: CRC Press.
Principal Investigator of the research project “Mixture and
Sauro, J. (2011). Does prior experience affect perceptions of usability? latent variable models for causal inference and analysis of
Retrieved from http://www.measuringusability.com/blog/prior-exposure. socio-economic data” (FIRB 2012 - “Futuro in ricerca” – Italian
php Government).