Reading
Report 4 – “Usability Testing: What Have We Overlooked?”
Citation:
Lindgaard, G., & Chattratichart, J. (2007).
Usability Testing: What Have We Overlooked? CHI 2007, (pp. 1415-1424). Retrieved
from http://www.itu.dk/~hulda/Usability/artikler/p1415-lindgaard.pdf
Article
can be found here: http://www.itu.dk/~hulda/Usability/artikler/p1415-lindgaard.pdf
Key
words:
Usability
testing, Metrics, UEM (Usability Evaluation Method) , participant recruitment,
number of users, number of tasks, recruitment of users
This
article is about (main discussion):
The
article explains research into usability testing to see if “The Magic Number 5”
for users is accurate. The research is to find:
- That there is a correlation between number of users and the proportion of problems found.
- That there is a correlation between number of user tasks and the proportion of problems found.
Upon completing their statistical analysis they found:
- The first hypothesis that there is a correlation between the number of users and the proportion of problems found was not supported.
- The second hypothesis that there is a correlation between the number of user tasks and the proportion of problems found was supported.
- There has been too great of a focus in ID on the number of users when the focus should really be on the number of user tasks and also the quality of participants.
- The Magic Number of 5 does not hold. In this study, “the percentage of problems found by the nine teams ranged from 7% to 43% - nowhere near the predicted 85%. The argument [the 5 users will find 85% of the problems] is therefore not upheld by the present data” (p. 1422)
- Giving users a persona to imagine during the tasks is helpful because “[t]he persona might have helped their test users place themselves in the real users’ shoes and hence carry out the required tasks the way real users would do. As a result, Team S [the only team with a persona] performed better than expected” (p. 1423)
- Rather than the number of users, the focus should instead be on “careful participant recruitment […]. It [also] pays off to give many sets of user tasks to a small number of users in a usability test rather than giving many users the same limited set of user tasks in a usability test” (p. 1423).
Less resources
and research should be spent on studying the number of users to use for
usability testing. Instead the focus of further research should be on “the role of
user tasks on improving usability testing approaches as well as into the
importance of recruitment of test users” (p. 1424).
References:
16, (1981-2004)
Relating
this to our project:
- When creating our usability tests, we should focus on increasing the number of user tasks
- The quality of the user tasks and scenarios must also be a consideration
- It will be more fruitful to give a higher quality usability test to a small number of users
- Care and attention must be taken to ensure that we have a good variety of users. The users should have varying levels of computer savvy and from different age groups. We should not just focus on using one group of users because they may all find different errors or problems. For example, a more tech savvy user may uncover issues with the usability than a less skilled user. They may approach the slide collection in a much different way
- It may be beneficial to have the users take on a persona while completing the user tasks
