The Impact of the Combination between Task Designs and Think-Aloud Approaches on Website Evaluation

The growth of the Internet and new technologies has created new dynamic websites that are growing rapidly in use and are having a significant impact on many businesses. These dynamic websites increasingly being developed in the midst of the Internet revolution and ever-improving information technologies. For example, e-learning websites are now essential for all universities that have a physical workplace. They have websites, and particularly those for e-learning, that have become an integral part of their businesses. The Internet revolution has even led to a large number of universities functioning solely online, without the need of a physical workplace. To keep pace with this development; some companies and organizations such as Intel® Education and the BBC seek to build free online learning websites that are oriented to world-class education at all educational levels. This development in lifelong learning has made the learners’ intention to continue using e-learning an increasingly critical issue. Consequently, quality is considered crucial to education in general, and to e-learning in particular. Web design is a key factor in determining the success of e-learning websites, and the users should take priority in the designers’ eyes because usability problems in educational websites can have serious ramifications over and above failing to meet the users’ needs.

Usability is one of the most significant aspects affecting the user experience. The techniques for measuring the quality of users’ experience have been classified under the heading of ‘ergonomics’ and ‘ease-of-use’, but more recently under the heading of ‘usability’ [Oztekin & Uysal, 2010].  ISO defines usability as “the extent to which the product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” [ISO 9241-11, 1998].  During the past decade a variety of Usability Evaluation Methods (UEMs) have been developed, and a number of research studies have compared the effectiveness of different UEMs [Jeffries et al., 1991; Alrobai et al., 2012; AlRoobaea et al., 2013; AlRoobaea et al., 2013; AlRoobaea et al., 2013; AlRoobaea et al., 2013; Ali et al., 2013; Ahmed et al., 2013].  UEMs can be categorized as analytic or empirical; analytic UEMs include techniques such as Heuristic Evaluation (HE), whereas empirical UEMs include a wide range of methods and procedures that are often referred to simply as ‘usability (or user) testing’ [Nielsen & Molich, 1990; Lewis & Polson, 1992].
 
Briefly, usability testing is labour-intensive, as it requires real users of the product to be examined whilst performing a set of tasks.  These users need to be observed carefully by evaluators in order to record the usability problems encountered by them.  Overall, the most important aim of usability testing is the compilation of a list of the usability problems encountered, which will then indicate the necessary changes to make in order to improve the product [Lawand & Hvannberg, 2002]. 

The recent literature has raised a number of questions and issues in the usability testing field that deserve further research from various viewpoints. For example, many researchers such as Fang and Holsapple (2011), and AlShamari and Mayhew (2010) have investigated the impact of task types on user testing results. However, the impact of task formulation and combining it with different ‘think-aloud’ approaches is a missing area in usability testing. This paper aims to investigate the efficacy of using three different combinations of task formulation and ‘think-aloud’ approaches in usability testing. Three out of nine combinations were chosen and tested, which were ‘specific task with constructive interaction think-aloud’, ‘guess task with concurrent think-aloud’, and finally, ‘free task with retrospective think-aloud’. These were measured against the number of problems found, the users’ performance and the overall user experience, as well as other usability measures. This paper is organized thus: Section 2 presents a brief literature review relating to this study; Section 3 describes the methodology and techniques adopted; Section 4 contains the data analysis; Section 5 presents a discussion and highlights interesting findings; and Section 6 presents the conclusion and suggestions for future work.

Literature Review

The concept of user experience has become central to interaction design; it defines how a product behaves and is used in the real world [Rubin & Chisnell, 2008]. Garrett (2010) describes user experience as what happens to the users when they interact with the system in question.  Experience varies from person to person and from product to product because of something calling ‘usability’.  Usability is a one of the core concepts to have emerged from the human-computer interaction (HCI) field. The literature reveals that usability is not a single ‘one-dimensional’ property of a user interface; there are many usability attributes that should be taken into account and measured during experimental sessions.  For example, whether or not it is easy to learn and to remember, efficient and subjectively pleasing to use, and whether or not there are errors.  There are a variety of methods for measuring the usability of a system or product, and they can be divided into three categories: usability testing (below), usability inspection (e.g. Heuristic Evaluation), and inquiry methods (e.g. interviewing users).  As mentioned before, the first is empirical and the other two are analytical. 

Usability testing (or user testing) is the method addressed in this paper. This is considered to be the most important evaluation method for ensuring system quality, and particularly for websites, although it is claimed to be costly [Oztekin & Uysal, 2010].  It requires participants to perform a set of tasks, usually under laboratory conditions.  These tasks are performed without including any information or clues as to how to complete the tasks or scenarios, and with no help provided to the user during the test session.  The performance of these tasks are monitored through ‘observation’ and then assessed by the evaluator/researcher in order to record the usability problems encountered by the users.  All the observed data, such as error numbers, time spent, success rate and user satisfaction, are recorded for analysis [Nielsen, 1994].  Dumas and Redish (1991) point out that a fruitful usability testing session requires careful planning and attention to detail.  Thus, there is a general procedure for conducting user testing: 1) Planning a usability test; 2) Selecting a representative sample and recruiting participants; 3) Preparing the test materials and actual test environment; 4) Conducting the usability test; 5) Debriefing the participants; 6) Analysing the data of the usability test; and 7) Reporting the results and making recommendations to improve the design and effectiveness of the system or product. The aim of applying usability testing is to procure a list of usability problems, which are defined as flaws in the design of a system that make the attainment of a particular goal (through the use of the system) ineffective and/or inefficient, and thus lower the users’ level of satisfaction in using it [Lewis & Polson, 1992].  These problems can be classified into different groups, in which a numeric scale from 0 to 4 is often used to measure the severity of each problem.  0 means “not a usability problem at all”, 1 means “cosmetic problem that should be fixed if enough time is available”, 2 means “minor problem with low priority”, 3 means “major problem with high priority”, and 4 means “catastrophic problem that is imperative to fix”.  Usability testing involves the collection of measurable data that can be compared with those of other websites; in particular, data on time spent, task success rate, error numbers, satisfaction rate and other metrics [Tan et al., 2009].

There are various factors affecting usability testing and its results, as Figure 1 shows.  These factors include the number of users (‘sample size’), the usability measures employed, the users’ characteristics (‘user profiling’), the use of the ‘think-aloud’ protocol (in its varying forms), the formulation of the tasks, and environment testing.  Firstly, usability testing requires real users to perform realistic tasks; however, there is no agreement amongst researchers about the most appropriate number of users.  Nielsen (2000) claims that five users are enough to reveal 85% of all the usability problems in a user interface, and that time and resources are wasted if more than five are involved.  Rubin and Chisnell (2008) confirm that the first of the three to five users can detect the majority of the usability problems, and that each additional user is unlikely to uncover new ones.  Several researchers have suggested that 5 to 12 users are sufficient for the purposes of website usability testing; however, others have suggested that 20 users, for the purpose of benchmarking, are needed for an experiment [Nielsen, 2006; Rubin & Chisnell, 2008].  Lindgarrd and Chattatichart (2007) confirm, in the findings of their paper which discusses user numbers and tasks, that “the result casts doubt on the role of the number of users in a usability test but confirms the important role of task coverage”.

The second factor is ‘usability measures’. This is the most important factor because any failure in measuring the level of usability means failure in the whole evaluation. There are three main ISO standards for measuring usability, and these are efficiency, effectiveness and user satisfaction.  Efficiency can be calculated through measuring input rate, task completion time and other factors.  Effectiveness can be worked out through measuring binary task completion, accuracy and the number of errors.  Satisfaction can be measured by asking about the user’s feelings towards the system or through conducting standardized questionnaires with the users after the test [Sauro & Kindlund, 2005; Hornbaek, 2006].  However, there are other usability measures such as success rate, which is defined by Nielsen (2001) as the percentage of tasks that users complete correctly and successfully. He divides task completion into three categories, which are ‘completed successfully without error’, ‘completed partially with some errors but with recovery from those errors’, and ‘failure to complete’ [Nielsen, 2001]. Overall, usability measures can produce different types of result, but which system to measure, what to measure and how to measure it are critical questions in determining the most appropriate usability measures.

The third factor is ‘user characteristics’; it is important to consider the user profile from all angles. Users differ in age, nationality, background, gender and, crucially, computing skills.  Each one of these factors may play a significant role in usability testing.    Molich et al., (2004) states, “usability testing effectiveness is dependent on the tasks given, the methodology used and the users’ characteristics”.

Regarding the fourth factor, there are various techniques that supplement usability testing, the ‘think-aloud’ protocol being the most widely used.  It is employed during the test when the users are asked to think out loud whilst performing their tasks and their thoughts, feelings, and opinions are recorded.  This technique can effectively help evaluators to capture how users interact with an interface and what is happening on the screen [Rubin & Chisnell, 2008].  It has been claimed that one-third of ‘severe’ usability problems can be discovered through this technique [Ebling & John, 2000].  However, the environmental setting of the usability test can sometimes influence the effectiveness of the ‘think-aloud’ protocol, and it does not always help when the users are not in their natural surroundings; this means that users may not feel as relaxed and may feel unable to talk or express their thoughts and ideas freely in a restricted and unfamiliar laboratory environment [Haak & Jong, 2004].  Furthermore, Rubin and Chisnell (2008) suggested that if the tasks are designed to assess the efficiency of a system (i.e. measuring time spent on tasks) then ‘think-aloud’ protocol should be avoided as it may negatively impact on the performance of the users.  The ‘think-aloud’ protocol has been generally used in achieving three goals; firstly, to find evidence for models and theories of cognitive processes; secondly, to discover and understand general patterns of behaviour in the interaction with documents or applications in order to create a scientific basis for designing; and thirdly, to test specific new documents or applications in order to troubleshoot and revise them [Krahmer & Ummelen, 2004]. 

There are three types of ‘think-aloud’ protocol, which are ‘concurrent’, ‘retrospective’ and ‘constructive interaction’.  The concurrent ‘think-aloud’ type is the most common; this involves participants verbalising their thoughts whilst performing tasks in order to evaluate a product.  ‘Retrospective think-aloud’ is less frequently used; in this method participants perform their tasks silently, and comment on their work afterwards by means of recording of their performance.  Constructive interaction is more commonly known as ‘co-discovery learning’, where two participants work together in performing their tasks, verbalising their thoughts through interaction (with each other?) [Van den Haak et al., 2004].  On the one hand it has been argued that ‘think-aloud’ protocol should be avoided in certain circumstances, as mentioned above, but on the other hand, Tullis and Albert (2008) assess the degree to which it can actually influence users’ performance, concluding that this technique can in fact enhance performance because it helps users to focus more on the tasks.  However, some researchers when employing the concurrent type have concerns about reactivity, i.e. the possibility that the act of speaking concurrently may influence user performance through distracting their attention and concentration; the effort of fully verbalizing the steps taken in the task may change the ways that users attend to the task components.  For this reason the retrospective ‘think-aloud’ type has been proposed in order to avoid the problems of ‘concurrent think-aloud; it is assumed to be the most fruitful in terms of problems reported per participant [Van den Haak et al., 2004].  Furthermore, ‘co-discovery learning’ (constructive interaction) has been claimed to be the most suitable method for evaluating collaborative systems and also to be the most appropriate method for usability testing with children [Nielsen, 1994].
 
The fifth factor in usability testing is the design of the tasks.  Many researchers are aware that task design is an important factor in the design of adequate Web usability tests.  The tasks designed for Web usability testing should focus on the main functions of the system.  The tasks should cover the following aspects: 1) Product page; 2) Category page; 3) Display of records; 4) Searching features; 5) Interactivity and participation features; and 6) Sorting and refining features. The tasks can be selected from four different perspectives.  These are: 1) Tasks that are expected to detect usability problems; 2) Tasks that are based on the developer’s experience; 3) Tasks that are designed for specific criteria; and 4) Tasks that are normally performed on the system.  Researchers also recommend that the tasks should be short and clear, in the users’ language, and based on the system’s goals.  Alshamari and Mayhew (2008) found that task design can play a vital role in the usability testing results, where it can be shown that changing the design of the task can produce differences in the results. They propose and examine three different types of task formulations, which are structured tasks, uncertain tasks and problem-solving tasks. They find that structured tasks are able to identify superficial and minor usability problems whereas the problem-solving tasks are able to identify usability disasters and major problems. Furthermore, Fang and Holsapple (2011), having designed two sets of tasks, which were ‘simple tasks’ and ‘complex tasks’, find that simple tasks impact differently on the website navigation structure, user performance, and perceptions of usability than complex tasks.

The sixth factor is the test environment.  Generally, user testing takes place in a controlled laboratory.  Tullis et al., (2002) discover several cases where the product works fine in the laboratory but not in the real world.  They find that the conditions under which the product’s use is tested are different to the conditions for actual use.  Dix et al., (2004) list four aspects when seeking to understand why a laboratory experiment sometimes fails: 1) The users’ motivation can be greatly diminished or even destroyed by the atmosphere of a controlled laboratory; 2) A laboratory does not take into account the social context (of supporting and motivating the users if they need it); 3) A laboratory setting does not consider the time context where, in reality, users may leave their work and resume it later; and 4) A laboratory does not take into account the user’s work context (users may feel disinclined to invest time and effort in something that they see as someone else’s job).  However, conducting experiments in a laboratory can increase their validity, can facilitate system comparisons, and can offer a controlled area where all interactions with the system can be closely recorded and monitored [Lazar et al., 2010; Nielsen, 2009].

In conclusion, further research should be carried out to investigate the impact of task formulation on usability testing, as recommended by Fang and Holsapple (2011) and Alshamari and Mayhew (2008). Furthermore, there is no research that investigates the impact on the results of user testing of a combination of different types of task formulation and different types of think-aloud technique. Many researchers use traditional user testing which consist of simple tasks and observation [Dale et al., 2012; Manzari and Trinidad, 2013]. This research will also compare the proposed combined methods here with the results of the traditional user testing in the paper written by AlRoobaea et al., (2013) on the same websites. This will help us discover which method can provide a better performance in the identification of usability problems. Consequently, this research will consider all these gaps and it will present useful information in the HCI field. 

Research Methodology

The experimental approach was selected to achieve the research objectives outlined above. The educational domain was chosen, and three educational websites were selected for different reasons; firstly, they are all completely free educational websites; secondly, they are interactive and are supported by three well-known companies with a wide range of users; and thirdly, they meet the research objectives and requirements through having various functions, processes and features. These websites are BBC KS3bitesize, Skoool, and Academic Earth. Three experiments were employed on each. Each experiment employed one design of task and one type of ‘think-aloud’ protocol. This section presents an explanation of the test procedures in these experiments, including the quantitative and qualitative data.

Testing procedures

The user testing was conducting by giving a quick introduction to the task designs; think-aloud approaches and the purpose of the study to each user. The next step was to explain the testing environment and how to use the equipment, followed by a quick demonstration of how to ‘think aloud for each test’ while performing the given tasks. Prior to the tests the users had to read and sign the consent letter and fill in a user sheet that included his/her demographic data, such as computer skills, nationality and gender. All the above steps took approximately ten minutes for each test session. The timing of each test started when the user was given the task scenario sheet and asked to read and then perform the task. Once they had finished the session they were asked to rate their satisfaction levels. This was followed by a brief discussion.

The Proposed Tasks

The literature suggests that further work on task design is needed [Lindgaard & Chattratichart, 2007; Alshamari & Mayhew, 2008].  This is because giving users only one task design may not achieve the testing goals; however, offering users different designs of task may reveal more usability problems.  Three designs were proposed for the tasks to be employed in this study.  These three types were:

1.    Specific Task: this design of task will guide the users to finding specific information relating to a number of specific questions.  This task involves constructive interaction ‘think-aloud’ type (co-discovery learning), where two users work together in performing their tasks.  This is designed to reveal potential usability problems. For example, the following task is to measure the usability of a website’s registration process.
–    Work with your friend and go to the website and register.
–    Work with your friend and fill in the fields, following the instructions given.

2.    Guess Task: in this task design the users should guess aloud, saying what information they expect to find behind the functions targeted in the test.  This design of task relies on the fact that users are usually uncertain as to whether they will find the information that they are looking for whilst surfing a website.  The task involves the concurrent ‘think-aloud’ type, which is the most commonly used.  The following is an example of this task, which is to measure the usability of a website’s post-a-question facility.
–    Guess how you can post your question.

3.    Free Task: in this task users are allowed to explore the target website freely, based on their own interests.  This task involves the retrospective ‘think-aloud’ type, which means that users perform their tasks silently, and afterwards comment on their work by means of a recording of their performance.  The following is an example of this task; it is to measure the usability of a website game.
–    Go to the website and enjoy your favourite game in the English section.

Each of these tasks consists of four sub-tasks which represent the key functional aspects of the selected website; i.e. they represent the main features of the website.  Although they have been formed differently, the task goals are the same.

Building the Questionnaire

After the users finish their tasks they will be asked to rate their level of satisfaction with the website and task formalization on a scale of 1 to 7, where 1 indicates ‘highly unsatisfactory’ and 7 indicates ‘highly satisfactory’.  This scale has been suggested to measure the levels of satisfaction felt by users truthfully following a test [Nielsen & Loranger, 2006].  Also, they will be asked to write down their comments and thoughts in a feedback questionnaire, explaining any reactions that they observed during the test.

User Characteristics

Selecting and recruiting participants must be done carefully; the selection must reflect real users of the targeted websites because engaging inappropriate users will lead to incorrect results, thereby invalidating the experiments.  Appropriate users will give confident results; the selection of these users will also encourage them to take part in the experiment [Dumas & Redish, 1999].  In this experiment the majority of users of the targeted websites were students.  They were divided into three groups consisting of 20 students in each; each group was mixed in terms of sex, age, education and computer skills.  These conditions should deliver reliable and accurate results; they may also play a significant role in the accuracy of the usability testing results.  Of the 60 students, 40 were pre-university, and 20 were a mix of undergraduates and postgraduates.  The users performing the specific tasks were called SC-Group, the users performing the guess tasks were called GC-Group, and the users performing the free tasks were called FR-Group.

The Number of Users Participating in the Experiment

As mentioned in the literature review, 20 users are required for testing each design of task and ‘think-aloud’ type.  This means that 60 students are needed for each website, and, therefore, 180 users were required to conduct this experiment [Nielsen & Loranger, 2006].  They were all introduced to the concept of the ‘think-aloud’ protocol and its three types, and how to perform the different task types. 

Test Environment

The test environment was controlled; it was conducted in a quiet, child-friendly room.  Also, we attempted to find out what our users regularly used, i.e. a familiar type of machine, browser and Internet connection speed, and set it up for them before the test.
 

Quantitative data is an important part of usability data analysis.  It provides a number of useful results, derived from various aspects that include success rate, error number, user satisfaction and time spent.  The following sections explore and discuss the quantitative data collected in these experiments and reveal how each group performed with regard to usability measures.

The Skoool groups ranked second in terms of success rate.  They scored 62%, which means that 62% of the tasks, on average, were completed successfully by the users.  FR-Group again had the highest success rate, followed by SC-Group and GC-group respectively.  It can be seen clearly that GC-Group were able to score more partially successful tasks than successful ones, as they attempted different ways and visited different pages to accomplish each task.  Also, the concurrent ‘think-aloud’ procedure may have played a role in producing more failed tasks.  This may be because this procedure distracted their attention, lowered their level of concentration and ultimately influenced their task performance. The Academic Earth groups had the highest success rate of the three website groups.  These groups scoured 63%, which means that 63% of the tasks, on average, were completed successfully by the users.  SC-Group had the highest number of failed tasks of the three groups.  This may be because they had doubts and were uncertain of their ability to guess and accomplish a task successfully.  Overall, the FR-Group success rate was somewhat higher than those of SC-Group and GC-Group.

Despite these observations none of the three websites scored below 50%, which is claimed by Nielsen to be the average for most websites, arguing that users generally fail to perform a task successfully the first time [Nielsen, 2001].  In terms of the group types, GC-Group achieved below 50% in BBC KS3bitesize, and came close to this score with the other websites.  This implies that the website’s success rate can be described as acceptable and is in line with this study.

The main findings of this experiment are as follows: