Introduction
‘Usability’ is a means of measuring how well people can use man-made objects such as computer-interfaces or web pages for their intended purpose. Researchers and practitioners have engaged in usability to portray the usefulness of websites and general information systems (Fang and Holsapple, 2007). Following the above definition, user-friendly websites need to have attributes such as user satisfaction, efficiency and effectiveness. A valid perception of the usability of a system entails a deep appreciation of the system and its usage (Hertzum, 2010). There are several usability evaluation methods used to measure the usability of websites; for example, ‘testing’ (e.g. think-aloud protocol); ‘inspection’ (e.g. heuristic evaluation/‘expert evaluation’) and ‘inquiry’ (e.g. questionnaires).
Social networking sites have quickly become one of the most popular means of online communication. According to Boyd and Ellison (2008), a social network can be described as a social structure that allows individuals or organizations to build a public profile within an enclosed system, articulate a list of other users with whom they share a connection, and view and transverse their list of connections and those of others. All this is accessible via the web with a web browser (Golbeck, 2007), which excludes networks where users would need to download particular software in order to take part. Most academic research on social networking systems has focused on the concerns of privacy and identity, although users are at liberty to share as much (or as little) data as they desire (Hart et al., 2008). The majority of users are aware of the visibility of their profiles and rely on their ability to control the information they share by managing their data using the privacy settings (Dwyer et al., 2007). In this research, usability is gauged in terms of how well users understand and can operate social networking sites; this performance is measured by usability evaluation methods: user testing (which includes questionnaires and ‘think- aloud’), heuristic evaluation, success rate, navigation speed and task correctness. Usability is also gauged by directly asking users to reflect on and rate how functional the navigation structure is.
Some of the attributes of social networking websites that need to be measured are the ease with which users can: 1) create and modify their personal profile; 2) change privacy and security settings; 3)interact, i.e. post text, single chat and multiple chat; 4) navigate, i.e. the quality of the websites’ navigation systems; 5) interpret error messages; 6) search for other users;7) join a group; 8) create a group; 9) connect and/or revoke unwanted friends/connections; 10) modify/remove posts; and 11) access appropriate and innovative help when needed.
The aim of this research is to investigate the usability of social networking systems, using expert evaluation and user testing, in order to provide recommendations for developers in the improvement of the social networking site-LinkedIn based on the difficulties(usability problems) experienced by both ‘users’ and ‘experts’, and also to measure the performance of both novices and experienced users.
This paper is set out as follows: Section One provides a brief introduction to the topic. Section Two provides a literature review on the Internet, social networking systems, usability and usability evaluation methods. Section Three outlines the research methodology. It consists of the methods used to evaluate the social networking site LinkedIn, the evaluation procedures and the participants involved. Section Four presents the results and analysis of the user testing and expert evaluation. Section Five is a discussion and recommendations for improving the social networking site LinkedIn, and Section Six presents the conclusion and recommendations for future work.
Literature Review
The Internet and the Web
The Internet is one of the most successful examples of the benefits and dedication to research and growth brought by information communications technology; it has transformed the computer and communications world (Leiner et al., 2009). As new technologies come out and large firms become ‘organizational foundations of industrial society’, the Internet is the technological foundation for the organizational form of a network (Castells, 2003).In the early days, websites were quite simple – read-only information sites where interaction amounted to navigation to other parts of the site or to other sites. Things have evolved in recent times. A lot of improvements and changes have been made to websites to make them more usable. According to Cleary (2000), statistics show that the number of people who access the World Wide Web for entertainment, business, research and education is growing daily. This suggests that the Internet is an available and effective way of doing many different things – shopping, publishing information and much more, including social networking. Despite the Internet’s usefulness and popularity, some websites still need to become more user-friendly.
Social Networking Systems (SNSs)
Social networks that enable people to make contact with others based on common interests have become very popular. With interfaces that allow people to follow the lives of friends, acquaintances and families, the amount of people on social networking systems has grown rapidly since the turn of the century (Hubermanet al., 2009). Social networking systems are popular platforms for interaction, communication and information sharing on the Internet (Wilson et al., 2009). The definition of a ‘social network’ can be said to be the concept of all users who share a social relationship, while in reality people relate with few of those listed as part of their network (Huberman et al.,2009). The visibility of a user’s profile, which is his/her description, varies from site to site and is uploaded at the user’s discretion. In general, social networking sites vary to a great extent in their features and potential. Some have photo sharing such as Flicker, or video sharing such as YouTube. Some have instant messaging technology and also provide storage and support, as well as limited mobile interactions as is the case with Facebook and MySpace; others are networks of trusted business connections (OMurchu et al., 2004) for business opportunities and for people to share contacts, such as Ecademy and LinkedIn.
Recent research on social computing reveals that users of social networking systems frequently use the public display of their connections to show off their identity and status even though there may be no shared interests (Donath and Boyd, 2004). The first dedicated online networking site that allowed users to create profiles and connect to a network of friends was SixDegrees.com. It eventually shut down after four years of operation having failed to become a successful business (Donath and Boyd, 2004). After that, a new generation of SNSs began when Ryze.com was launched in 2001 to help people in the building of their business networks. From 2003 onwards, many new SNSs were launched. LinkedIn, founded in May2003, focuses on professional users creating networks of co-workers and other business associates (OMurchu et al., 2004). This business social network is the main focus of this research.
LinkedIn as a Social Networking Site
Online communities are accessible over the web with a web browser and are certainly having a significant positive impact on the way people interact by providing a sense of community to individuals who would never otherwise form a social network. The impact can also be negative, an example being the instant global exposure of negative or illicit activities, or even the introduction of a new security threat, as social networking sites have become targets for viruses (Wentz and Lazar, 2009). People are making use of social networking sites for private and professional use, communications, new business developments and contacts by making use of an efficient and simple way to build their social networking systems online (OMurchu et al., 2004). They can be notified more quickly through online social networking, enabling them to become more engaged and involved with one another. With the development of this new means of social and business communication, more people have been motivated to join up with a precise aim in mind.
LinkedIn is a business networking site that focuses on professional users creating and building a network of colleagues and other business connections (Keenan and Shiri, 2009).It has a clean and professional design, and is perhaps the site with the least (if any) potential for social interactions (OMurchu et al., 2004). This business network allows business professionals, owners and entrepreneurs to connect with each other and search for contacts, either by expertise or location. Members of the network can look for jobs, seeking out experts in a particular area, or make contact with other professionals through a string of trusted connections. Some other key benefits of LinkedIn are:
1. It keeps users informed about their contacts and industry.
2. It helps users to find the people and knowledge they need to achieve their goals.
3. It enables members to control their professional identities.
4. It helps members find the answers that they seek.
5. It helps members get the most from the professional network.
6. It helps in the discovery of inside connections when looking for a job or new business opportunities.
7. Past and present colleagues and classmates can be found quickly.
8. LinkedIn simply makes staying in touch easy (OMurchu et al., 2004).
User profiles are a means of providing an identity for users online. Business profiles allow professionals to interact with one another through business-orientated information, testimonials and reputations. The main purpose of a social networking site is its clear representation of relationships, as LinkedIn is aiming at professional business users. Business networking sites such as LinkedIn will perhaps ‘prove to be more profitable’ (OMurchu et al.,2004) because they offer many valuable features for the construction and maintenance of users’ business networks and list of contacts, making it an easier and more effective way to make and maintain valuable business contacts. The function and goal of this networking site(LinkedIn) influence the way in which the site is designed and which information collected through the user profiles will be shown to which particular users.
LinkedIn encourages interaction between connections through professional networking. As a business orientated social networking site, LinkedIn focuses on encouraging professional relationships, this having the three main functions of:
1. Reconnecting with colleagues and associates.
2. Exploring the hidden job market through connections.
3. Making contact with industry experts to learn about a specific topic.
As a social site, LinkedIn focuses on business networking and professional relationships. All services on the website function exclusively to encourage professional relationships. LinkedIn represents a growing group of social sites that are very much focused on professional and business associates (Keenan and Shiri, 2009). Similar to earlier manifestation of social websites, role-focused social sites accommodate a particular clientele.
Usability
The reviewed literature shows that usability is not a single ‘one-dimensional’ property of a user interface. There are many usability attributes that should be taken into account and measured. Shackel and Richardson (1991) propose four-dimensional attributes that influence the acceptance of a product, which are ‘effectiveness’, ‘learn ability’, ‘flexibility’ and ‘attitude’. Nielsen introduces some different usability attributes based on a ‘system acceptability model’ (Nielsen, 1993). ‘Utility’ and ‘usability’ are classified as sub-categories under ‘usefulness’. The former is used to describe the extent to which the product provides the right kind of functionality to help users perform relevant tasks, while the latter analyses the question of how well users can use that functionality (Preece et al. 2006). The five major attributes of usability that Nielsen highlighted are as follows:
1) Easy to learn: a system should be easy to learn for the first time.
2) Efficient to use: the relationship between accuracy and time spent in performing a task.
3) Easy to remember: a user should be able to use a system after a long period without spending time learning it again.
4) Few errors: the system should prevent users from making errors; this also addresses the question of how easy it is to recover from errors.
5) Subjectively pleasing: this addresses the user’s feelings about the system.
According to Nielsen (2003), The word ‘usability’ also refers to methods of improving ease of use during the design process, and Andreas Holzinger (2005) describes it as the acceptability of a system and its ease of use for a specific set of users performing specific tasks in a precise setting. Thus both definitions suggest that usability can be described as a feature that weighs up the effort required by a user to interact with interfaces. Their components, such as utility, navigation, interaction, appearance, efficiency and satisfaction, actually determine the rate at which users can surf their way through the website and leave feeling satisfied. It is acknowledged that an information system ought to provide the essential functions so that users can get their tasks completed. However, functionality alone is insufficient. The determinants of making a system successful are both functionality (the level at which the system offers the functions needed by users to carry out their tasks) and usability (ease of use). Usability is a combination of features below (Holzinger, 2005):
1. Learn ability: the user is able to start work on the system fast.
2. Efficiency: enabling the user to reach a high level of output after learning how to use the system.
3. Memorability: permitting the user to return to use the system after an ‘idle’ time without having to learn everything again from the start.
4. Satisfaction: making the system pleasant to use in order to ensure user satisfaction.
5. Low error rate: the system should have a low error rate so that when users make mistakes using the system they can correct them.
Benefits and Relevance of Usability
Users tend to go away if they get lost on a website, if the website is hard to use or if elements on the site are not clear enough. If an organization’s homepage fails to describe clearly what it does or offers, users will leave. Time wasted by users trying to find their way around the website is considered as money and effort wasted. If users are unable to find a product on a site, they cannot purchase it either. Thus, visual design should be built in to aid users. The most relevant elements should be made to stand out the most. If users are required to remember where things are on the site, it becomes difficult to use and they are prone to making errors. Some examples of the benefits of usability are as follows (Uldall-Espersen, 2005):
1. Users will be satisfied and not frustrated with the website.
2. Users will be able to find information on websites.
3. Users will enjoy interacting with the website.
4. Users will achieve their goals effectively and efficiently.
5. User errors and costs will be reduced.
Some examples of the relevance of usability are as follows (Uldall-Espersen, 2005):
1. Users’ expectations increase along with their experience.
2. It allows quantitative data to be collected on the number of errors and time spent.
3. Users become loyal to a website when satisfied.
4. Users gain more confidence with the systems and so feel less insecure.
Poor usability due to a badly designed interface leads to wasted time and effort, an unnecessary increase in internet traffic (because of the problems encountered), user frustration and the discouragement of exploration. Thus usability has always been accepted as a major contributor to the perceived success of a system. For web-based systems, usability is absolutely critical. Jakob Nielsen puts this very succinctly in the following two quotations:
1) “Usability rules the web. Simply stated, if the customer can’t find a product, then he or she will not buy it.”
2) “The web is the ultimate customer-empowering environment. He or she who clicks the mouse gets to decide everything. It is so easy to go elsewhere; all the competitors in the world are but a mouse click away” (Nielsen, 1999).
Usability Evaluation Methods (UEM)
Usability evaluation is an important activity to ensure the quality of the user experience (Schmettow, 2009). Many usability evaluation methods can be used to assess transactional web applications but problems come up when deciding which of the evaluation methods fetch more information (Otaiza et al., 2010). Usability evaluation methods (UEMs) are a set of techniques that are used to measure usability attributes. They can be divided into three categories: inspection, testing and inquiry.
This section highlights the following usability evaluation methods which will be used in this research experiment: heuristic evaluation (inspection); user testing (which involves think-aloud protocol) with field observation; and questionnaires (inquiry methods):
Heuristic Evaluation
Heuristic evaluation is one type of inspection method. It was developed by Nielsen and Molich (1990a), guided by a set of general usability principles or ‘heuristics’ – see Table 1, below. It can be defined as a process that requires a specific number of experts to use the heuristics in order to find usability problems in an interface in a short time and with little effort. Magoulas et al. (1990) stated that “heuristic evaluation is a widely accepted method for diagnosing potential usability problems and is popular in both academia and industry”. It is superior to all other methods in that evaluation can be carried out in less time and by a smaller number of expert evaluators; it does not require any special tasks or equipment. Also, it can be used early in the development process, and may continue to be used throughout the development of the site (Nielsen and Molich1990a). However, it is a subjective assessment relying on the evaluators’ experience, and can produce a large number of false positives which are not usability problems (Holzinger, 2005; Nielsen and Loranger, 2006; Chattratichart and Lindgaard, 2008).
Table 1. Nielsen’s Heuristics
There is no specific procedure for performing heuristic evaluation. However, Nielsen (Nielsen, 1994) suggested a model procedure with four steps. Firstly, pre-evaluation coordination session (a.k.a training session) is very important. Before the expert evaluators evaluate the targeted website, they should take few minutes browsing the site to familiarize themselves with it. Also, they should take note of the actual time taken for familiarization. If the domain is not familiar to the evaluators, the training session provides a good opportunity to present the domain. Also, it is recommended that in the training session, the evaluators evaluate a website using the heuristics in order to make sure that the principles are appropriate (Chen and Macredie, 2005). Secondly, actual evaluation which each evaluator is expected to take around one to one and half hour to list all usability problems. However, the actual time taken for evaluation should always be noted. Next, debriefing session would be conducted primarily in a brainstorming mode and would focus on discussion of possible redesigns to address the major usability problems and general problematic aspects of the design. A debriefing is also a good opportunity for discussing the positive aspects of the design, since heuristic evaluation does not otherwise address this important issue. Finally, results of the evaluations are collected into actual evaluation tables, and then combined into a single table after removing any redundant data. After the problems are combined, the evaluators should agree on the severity of each individual problem (Nielsen, 1994).
Applying heuristics produces a list of usability problems (Nielsen, 1994). These problems are classified in different groups in which a numerical scale from 0 to 4 is used to measure the severity of each problem. 0 means “not a usability problem at all”, 1 means “cosmetic problem that should be fixed if enough time is available”, 2 means “minor problem with low priority”, 3 means “major problem with high priority”, and 4 means “catastrophic problem that it is imperative to fix”. These categories are explained to the evaluators in their instructions.
User Testing
From research on usability evaluation methods and design practice, user testing is believed to be one of the best techniques for acquiring insights into usability problems (Vermeerenet al., 2003). This involves asking users of the site to carry out tasks and observing their actions, where they have difficulties and where they are successful. These tasks should cover the main functionality of the system and simulate expected usage patterns. The collected data, such as the time taken to complete the task, number of pages accessed on the website and whether the participants completed the tasks correctly, can be extracted from the observations made.
The usability of web-based systems has an immense impact on users and these users may not revisit the website if they encounter problems in using the system. As a result, the usability of web-based systems is essential in finding out how successful they are. The user testing method includes examples such as ‘think-aloud’ protocol, questionnaires and field observation. Think-aloud protocol involves end users constantly verbalising their thoughts while making use of the system. This enables specialists to gain a detailed picture of users’ conduct, which can be analysed to expose the usability issues. Despite these positive aspects of the think-aloud protocol method, there are some problems, such as its validity and reliability, that have been ignored because it is such a useful method for convincing developers that problems exist (Ramey et al., 2006). These issues must be considered when testing the value of this method. In conducting the analysis of a site, it is important to establish the areas that are of interest.
Field Observation
Field observation is the simplest of all methods and involves visiting one or more users in their places of work (Holzinger, 2005). Notes are taken as discreetly as possible to avoid interfering with their work as disturbance and any form of noise can lead to inaccurate results. If possible, the observer should be almost invisible to ensure normal working conditions. Videos are used occasionally to make the observation process less interfering because they can be replayed to go over the scenes that are captured. Observation focuses on key usability disasters that tend to be very obvious the first time they are observed and thus do not require a second test session.
Questionnaire
Many aspects of usability can best be discovered by questioning the users. This is especially true when studying user-satisfaction or uneasiness, which are difficult to measure objectively. Questionnaires are an indirect method (Holzinger, 2005) since this technique does not study the actual user interface but only gathers the opinions of users about the interface. Questionnaires are useful for studying how end users use the system and their preferred features. Indirect usability tests can be supplemented by direct usability ones – such as think-aloud. User statements cannot always be taken at face value and the data about their actual behaviour should be given more precedence than their claims of what they think they do.
User satisfaction, user preferences and possible uneasiness can be identified easily by using questionnaires, which can also be used to gather statistics. The disadvantages are that indirect methods result in low validity; that this method requires a sufficient number of responses to be significant (30 users being the lower limit for a study), and that it recognizes fewer problems than the other methods. The questionnaire can be in the form of questions or a checklist to aid in the evaluation of the social networking site for usability. Indirect usability testing methods such as questionnaires or interviews must be supplemented with direct usability testing such as think-aloud protocol or observation (Holzinger, 2005).
Research Methodology
The objective of this study is to evaluate the usability of the social networking site LinkedIn, to discuss and analyze the results, and to make recommendation(s) for social networking site developers in order to improve the website based on the difficulties discovered by users and experts. LinkedIn was chosen to be examined closely because of its growing recognition on the web as a business social network. The number of websites was limited to one in order to allow for a thorough assessment, discussion and evaluation of the social network’s major features.
To achieve this objective two methods were used: ‘expert evaluation’ and ‘user testing’ (including think-aloud protocol, questionnaires and observation). This is because research done on websites has shown that using two methods uncovers more issues (Holzinger, 2005).
Evaluation Methods Used and Justification for Using them
The methods used in this research are expert evaluation and user testing (think-aloud and the use of questionnaires)
Expert Evaluation (Heuristics Evaluation)
This method is used to discover usability problems on the LinkedIn website from the point of view of the expert evaluators. It looks at the complete system or website from many angles and reveals possible difficulties such as inconsistency, support for different ways of working, visibility of information and language use. It also enables elements such as error messages to be thoroughly investigated. A poor interface design ruins the experience of a user (Keenan and Shiri, 2009). The researchers made improvements in the ten Nielson’s heuristics by creating a checklist specifically orientated to the LinkedIn website, dividing each heuristic into sub-heuristics to provide more clarity and, also, to facilitate the evaluation process for the evaluators.
User Testing
Testing with end users is the most basic method for testing usability and is also essential (Holzinger, 2005). This is because it uncovers direct information about how people use systems and their specific problems with an interface (or website in this context) are noted. Interface design, the structural and visual design of a website, presides over how a user interacts with that website (Keenan and Shiri, 2009). This approach permits “usability engineers to determine accurately where novices encounter trouble and the frequency, number of errors, and duration of the usability problem at that step” (MacDorman et al., 2011).Also, Nielsen (1992) recommended conducting usability testing together with heuristics evaluation because each one is complementary to the other. The aim of the user testing, with both novice and experienced users, is to identify usability problems associated with the user-interface of LinkedIn and to compare novice and experienced users’ performances.
Think-aloud is a known method employed in usability testing. During the testing, where the user is carrying out the task as part of the procedure, the evaluators ask the user to voice out loud whatever goes through his/her mind, his/her emotions and opinions as he/she interacts with the system. This method was chosen for both the experienced and novice users (and particularly for the novice users as they are new to social networking) to enable the evaluators to understand and evaluate the social network site (i.e. LinkedIn).
Also an observation method was chosen to observe the behaviours of users as they performed the given tasks; this ensured that the observers (researchers) took note of different and interesting activities observed. The experiment was recorded using a computer program (see Section 3.3.2) which could then be taken home to analyze. The software was used to measure the success rate of completed tasks and the time taken to perform each task.
Questionnaire
In this study there are two questionnaires, which are pre-test (pre-evaluation) and post-test (post -evaluation). The pre-test was used in both the expert evaluation and user testing to collect demographic data of users and expert evaluators. The second, post-test, questionnaire was used in the expert evaluation for collecting the experts’ opinions and in the user testing for two purposes: 1) to collect users’ opinions and 2) to measure users’ levels of frustration and/or satisfaction when using LinkedIn. The data collected from both users and experts gives us a clearer understanding on the usability of LinkedIn for end users.
The Recruitment of Participants
The expert evaluators and users (both experienced and novice) were asked by email if they would be willing to take part in this experiment. The email explained the aims of the research, the procedures that would be used, and the type of people who might participate in the experiment (expert evaluators and experienced/novice users). Also the age of the experienced/novice users was requested to be between 21 and 40 because this age falls within the age range of people who often make use of social networking systems. The aim of having both experienced and novice users was to measure the difference in performance between the two groups, this being achieved by carefully and closely observing users’ behaviours during the user testing and noting down relevant observations.
Those who agreed were contacted to arrange a convenient time for the test. There were 3 expert evaluators who had been using LinkedIn for many years, and 33 users, ten of whom had joined LinkedIn for the specific purpose of this evaluation, making them first-time users of the social network (novices). The other 23 users had already had prior experience with social networking sites (experienced).A pre-test (demographic) questionnaire was completed by both evaluators and users, the results of which are shown in Tables 2,3and4 below:
Procedures for Conducting the Experiments
Procedure for the ‘Expert Evaluation’ (Heuristics Evaluation)
After developing the checklist specifically designed for LinkedIn, it was sent to the selected expert evaluators who conducted the evaluation by judging and rating each heuristic individually by assigning it a value from 0 to 4 as mentioned in the literature review section. The results were then sent back and a debriefing session was conducted, using Skype, to discuss the severity ratings and other issues related to improving the website.
Procedure for the Actual ‘User Testing’
The experiment was carried out in a quiet room. The tools used in carrying out the evaluation in this study included a laptop with Camtasia studio 7 software, which was installed on the laptop and used for the usability testing (think-aloud). The users came at a time of their convenience and were cordially welcomed before starting their test. The procedure and what was expected from them was introduced, together with the concept of think-aloud protocol and the tasks that would be performed on the LinkedIn website as well as the procedure for doing the post-test questionnaires (level of satisfaction and other comments about the site).The usability testing was administered in such a way that only one user at a time could perform the experiment, in order to enable the careful monitoring of their actions and behaviour. The tasks consisted of the following:
– Creating an account.
– Adding new information about yourself (personal info, experience, education etc.) to your profile.
– Uploading a picture to your profile.
– Adding a new connection and sending him/her a message.
– Changing your privacy settings so that everyone can view your profile.
– Changing the notifications option for messages to include all the opportunities available.
– Joining your school/workplace/industry group.
– Asking for a recommendation from your connection (at school or in the workplace).
Pilot Studies
Both ‘user testing’ and ‘expert evaluation’ instruments were piloted to make sure that the language, logistics and questions would achieve the result for which they were designed.
Heuristics Evaluation Pilot Study
To confirm the suitability of the general setting of the heuristics evaluation, a usability expert was requested to evaluate the setting. The final draft of the checklist was given to one usability specialist for his comments, and some further improvements were then made, including word corrections.
User Testing Pilot Study
The pilot study was conducted on the post-test questionnaires and tasks scenario with two users (an experienced user and a novice) and one independent evaluator. The evaluator went through the tasks with both users to get their initial opinions about the social networking site and feedback through the questionnaire. Both users carried out the experiment using the ‘think-aloud’ method and informal interviews. Observations were noted down during the ‘think-aloud’ process. This pilot study helped to improve the quality of the document (questionnaires and tasks scenario) by improving the wording of the document. Furthermore, corrections were made to the task description by including the LinkedIn (URL) address.
In the next section, the results of the actual experiment performed and their analysis will be discussed: how easy or difficult the participants and expert evaluators found the different tasks, and their feedback and opinions about the social networking site.
Results and Analysis
This section presents the results obtained from employing the two methods (expert evaluation and user testing) used in this experiment. It starts by detailing the results of expert evaluation, which include the number of problems that were discovered using the usability checklist on the LinkedIn website, and expert evaluators’ feedback. It then summarizes the difficulties faced by each user during the user testing and the usability measures calculated, which includes efficiency (success rate and time) and satisfaction scores.
Heuristics Evaluation Results
Checklist Result and Questionnaire
The researchers developed a usability checklist for the LinkedIn website and the results obtained from the expert evaluations are outlined below:
Table 5 shows the detailed quantitative results after conducting the heuristics evaluation on LinkedIn by each heuristic. In this experiment the heuristics evaluation discovered a total of 57 problems, of which 35 were cosmetic problems, this being 61% of all usability problems and 20 minor problems were also discovered (35% of all usability problems). Finally, 2 major problems were discovered, making 4% of all usability problems. The ‘visibility of system status’ heuristic identified more problems, with 4 cosmetic, 7 minor and 2 major ones. This means that the website was found to be a little difficult to navigate. Moreover, users are not being kept informed about their journey around the website through appropriate feedback within a reasonable time. For instance, one expert evaluator commented, “It is because of how thoroughly you have browsed the website; you should always have a link to take you back easily. Similarly, whenever a user faces any difficulties they should be able to post their feedback easily and receive a response quickly”. Another expert evaluator maintained that the user was overwhelmed with too much information and an overcrowded interface, making navigation difficult. The next heuristic was ‘error prevention’, with seven cosmetic and two minor problems. This indicates that the LinkedIn website was not carefully designed to prevent problems occurring in the first place, but rather, to show error messages. The third and fourth heuristics are ‘Match between system and the real world’ and ‘Help and documentation’, with 8 cosmetic problems found in the former, 4 cosmetic and 4 minor problems in the latter. This means that LinkedIn is also a little difficult to use because it ‘does not speak the user’s language’ using words, phrases and concepts familiar to them so that the information appears in a natural and logical order. Also, LinkedIn does not provide enough help and documentation as and when the user needs it. For example, one expert evaluator commented that when he wanted help, “I had to struggle to search for information in the help centre and it was difficult to spot how to send feedback and request help”. Another expert evaluator commented that “it was a bit difficult to go to the help section and then get back to the previous work because the website does not hold the current tasks while you get help, and hence you feel that you need to start again from scratch. The other heuristics faired less well in identifying usability problems. However, some expert evaluators highlighted the following problems regarding these heuristics:
1) If a connection sent an invitation by mistake, it cannot be revoked.
2) Some icons are not visually and conceptually distinct.
3) There is no visual feedback in menus or dialog boxes about which choices (options) are selectable.
4) How to undo actions is not clearly defined.
In conclusion, the expert evaluators agreed that LinkedIn could be quite difficult for novice users because it was created for professional users, but it could become much easier and more understandable for novices over time.
Table 5. Heuristics Evaluation Results
User Testing Results
Task Analysis
For the period of the evaluation process it is not necessary to examine all the tasks the participants carried out in great detail. For some of the tasks which were carried out the most successfully, an examination in outline will suffice. For the usability problems and other tasks that were difficult, a more expanded description would be more appropriate. Both the expanded and concise tasks must be carefully analyzed so as to make sure that they are a true illustration of the social networking site. In order to learn the possible reasons for the outcome from the participants’ task performance, each of the tasks are reviewed and analyzed in this section.
– Creating an Account / Adding New Information to One’s Profile
Creating and editing an account was an easy task for all the participants. From the observations and data collected via ‘think-aloud’ protocol, users performed the tasks with ease as the task location was easily visible. From Figure 1, below, the account set-up details are clearly situated on the right hand-side of the page, which makes it simple for users to create accounts.
Figure 1. LinkedIn Homepage
– Uploading a Profile Picture
Uploading a profile picture is reported to be very easy on LinkedIn. This task appears to be straight-forward as is the link is clearly visible on every user’s profile. Users simply click on the ‘Add photo’ and ‘Upload photo’ option next to the space for the profile image. The site is very particular about the image size that can be uploaded, the maximum size being 4MB. If participants try to upload an image that is too large, an error message reading ‘File size too large’ is given as feedback. This pointer makes it easy for users to become aware of the problem and solve it.
– Adding New Connections and Sending a Message
Adding connections (or requesting for connections) and sending messages were part of the simple tasks users carried out successfully (see Table 6). The link for performing the task is situated amongst other links on every user’s profile, making it easy to locate and thus perform the task.
– Changing Privacy Settings
Privacy on LinkedIn operates in an unusual way. LinkedIn has created a unique alternative for users to make contact with potential associates or other target professionals. Using a web of relationships, LinkedIn allows a user’s ‘connection’ (LinkedIn’s term for ‘friend’ or accepted user) to bring in mutual ‘connections’. Users separated by one or two degrees of separation can be introduced by a mutual connection. This enables users to network with potential colleagues through mutual connections, a characteristic exclusive to LinkedIn.
Changing privacy settings for existing and potential connections to be able to view a user’s profile was a difficult task for both experienced and novice users. This was due to the fact that users needed to use the drop-down menu and click on the ‘Settings’ link positioned in the top right-hand corner of the page (see Figure 2 below). The novice users were expecting to find this feature, together with other links, on their profile pages; some of the experienced users had forgotten where the link was situated despite their experience with the social network. From the pilot study, and as observed during the experiment, the location of the link to the task was not easily located by either the experienced or novice users. According to an experienced user, this is because the location of the link is ‘hidden’, and therefore difficult to find.
Figure 2. Location of the Link for Changing Notifications and Privacy Settings on LinkedIn
– Changing the Notification Option Task
Changing the notification options to include all opportunities available (as seen on other social networking sites), so that an alert will be given when a message was particularly difficult, even for experienced users. As stated in the task above, the location, which is the same as for the ‘Changing privacy settings’ task (see Figure 2), is in the top right-hand corner of the page. More details are given in the discussion of the results.
– Joining a School/Workplace/Industry Group
Joining a group of any category was not a problem for most users, except for three novice users during the ‘think-aloud’ testing. The ‘Groups’ link was clearly visible, together with other links, at the top of the page on every user’s profile, but did not specify as to whether users could join any group. However, users could click on the link and connect with groups of any kind.
Figure 3. Location to the Link for Joining Different Groups on LinkedIn
- – Requesting a Recommendation from a Connection
Requesting a recommendation from a connection was not a difficult task to perform for the users. This task is clearly situated on every user’s profile, making the link easy to locate in order to perform the task successfully.
LinkedIn can be described as being designed to be used by users easily and effectively’ (Shackel, 1991).From the recorded data, the tasks‘ changing the notification option’ and ‘changing privacy settings’ were the most difficult to perform. Users experienced the most success (and least usability problems) with creating accounts, editing their profiles, uploading a picture, adding connections and joining a school/workplace/industry group (95% average success). Effectiveness was measured here in terms of human performance. Asking for recommendation, changing privacy settings and notifications (specifically) presented problems for participants.
Overall Success and Difficulty
Table 6 below shows the number of users that were successful in each task carried out on the social networking site. Although most of the tasks were completed successfully within or around the expected completion time, a few of them were found to be difficult. Nielsen (2001b) defines task completion rate (i.e. success rate) as “the percentage of tasks that users complete correctly”. He uses the following formula for measuring success rate:
Success rate = 
Across the tasks, the most difficult were ‘Change your privacy settings so that everyone can view your profile’ and ‘Change the notifications option for messages to include all opportunities available’ with success rates of 77% and 70%, respectively. The next most difficult task was ‘Join your school/workplace/industry group’ with a success rate 94%. As can be observed from the experiment, more novice users had usability problems, resulting in more time spent in completing the tasks. Nielsen (2001b) pointed out that the success rate for the majority of websites is below 50%. In this study, the success rate was above 50%.This might be due to the fact that only particular aspects of the LinkedIn website were tested.
Table 6. Summary of Users’ Overall Usage of, and Success with the Tasks
Efficiency (Success Rate and Time Taken)
Most experienced users took the least amount of time to complete most of the tasks productively on the social networking site, this being between five to eight minutes (M = 6.5 minutes), compared to some of the novice users who spent between seven to eleven minutes (M = 9.5 minutes) as indicated in Table 7. As observed, some of the novice users first needed to understand the content and structure of the website before they could steer through it. As a result, the analysis of user’s efficiency in terms of navigation indicates that the experienced users were able to achieve an 80% success rate whereas the novice users achieved only 65%.
Table 7. Success Rate and Time Taken for All the Tasks
Users’ Satisfaction
From the results of the test carried out by the users as shown in Table 8 below, LinkedIn was rated as having a high level of user satisfaction. In analyzing the answers to the nine questions, twenty two users rated themselves as having the highest overall satisfaction (66%)(all experienced users), whereas three users (9%),comprising both experienced users and novices, rated themselves the lowest in overall satisfaction. Eight novice users (25%) were unbiased (neutral) about the social networking site. Thus we can say, based on the results, that most users were content with the social networking site despite some of the difficulties they experienced while carrying out the tasks, especially by the novice users.
Table 8. Users’ Satisfaction Regarding Features of the LinkedIn Website
Discussion and Recommendations
This section highlights the feedback of both users and experts on the difficulties that were encountered during both experiments, and a set of recommendations is provided.
The results gathered in the experiment (of user testing) show that the novice users had slightly more difficulty in performing some of the given tasks than the experienced users. The expert evaluators agreed that novice users would experience some difficulties, but that these could be overcome with time. For example, in the user testing the three main tasks of ‘Changing privacy settings’, ‘Changing notifications’ and ‘Joining school/workplace/industry group’, proved difficult for most of the users who were novices. For the first task, ‘Changing privacy settings’, twenty three users, most of whom were experienced users, were able to find the link to ‘Settings’; the ten users who were not successful were mostly novice users (8 novices, 2 experienced). The placement for the link ‘Settings’ was difficult to locate, it not being visible on the profile (see Figure 2) and thus more time was spent in completing the tasks that required this link. In the second task ‘Changing the notifications option’; thirteen users (8 novices and 5 experienced users) had difficulties. In the third task the location of the link for ‘Joining school/work/industry groups’ was not clear to users; when a user hovers the cursor over the ‘Groups’ link at the top of their profile page, it reveals other options such as ‘Groups you may like’, ‘Groups Directory’ etc. (see Figure 3) but does not state clearly whether or not they can join a specific group. Possibly, if there was an option which said ‘Join a Group’ then users would have less of a struggle in completing the task.
As observed during the user testing and from the feedback of both the expert evaluators and users, the main problem areas are the layout of the website and the organization and structure of the buttons and contents. Users found the organization and layout of the social networking site to be not very well structured, causing them some difficulty in locating a few of the links. For example, the terms used on the social network were found to be somewhat confusing: LinkedIn’s term for ‘friend’ is described as a ‘connection’ which is understandably puzzling; this term being used because the social networking site is aimed at professional networking. The structure of buttons and contents which presented difficulties to most novice users could be due not only to their unfamiliarity with the social networking site but also to their ‘novice’ status regarding ICT in general. Possibly, frequent use of the social network will reduce such usability issues, as was observed with experienced users during the ‘user’ testing.
The age group and gender had little or no effect on the outcome of the testing for two reasons:1) the users’ age group falls between 21 and 40, which is highlighted by previous studies as the group that use the social networking the most, and 2) the ratio of male: female was unbalanced (26:7). However, the experience factor contributed considerably more to the results of the experiment than the other factors. Looking, for example, at the results of ‘Good feedback on errors’, three novice users considered the feature to be poor, this being possibly because they had no knowledge of the social networking site before the experiment and had difficulty in understanding the error messages.
To improve the usability of interactive systems such as LinkedIn, user satisfaction is one of the main features of usability and should be kept in mind when designing social networking sites, not only for first-time users but also to retain existing users of the website.
Some suggestions are listed below on how to improve the social networking site LinkedIn:
– Use consistent and familiar terminology. This is related to participants’ confusion over the term ‘connection’ for ‘friend’ which was unfamiliar to them.
– Contents of the website should be described in clear terms. The link ‘Groups’ or ‘Groups You May Like’ for joining a group appeared vague to some of the participants. A clear label for the link saying ‘Join a Group’ would indicate to them precisely what to do.
– Improve link placement. The ‘Settings’ link position for changing notifications and privacy settings should be easily located, making this task easy to perform. The link should be positioned within easy view of the user on the profile page along with the other links. Also, the creation of a link for going back to previously viewed pages or previously viewed menus is a must.
– The instructions and information should appear in the same style on each page of the website, and without any irrelevant information.
– The interface should be clear enough to make users aware that they are able to deselect some options using the ‘undo’ button.
– The website should allow users to reverse an action, such as revoking the invite message.
– The FAQ and help should be available and clearly labelled and positioned.
– The users should be able to enquire about anything and receive feedback either instantly or via email.
– It would be good to confirm an action a second time with a warning pop-up window before it is finally completed.
– The navigation should be maintained using cookie-crumb.
– The menu choices should be logical, distinctive and mutually exclusive.
– Each page should have a title, and it should match the link it refers to.
– Adequate feedback should be provided to the users. Too often users were unsuccessful in completing actions simply because they were not sure whether or not the system had performed their task.
Conclusion
The results of both experiments indicate that LinkedIn is slightly more difficult for novice users, and especially first time users, than for experienced users. This result shows the importance of assessing first time usage because negative first impressions may dissuade users from returning to the SNS site. About 18% of SNS users recently surveyed revealed that lack of usability was a significant reason for not returning to the site (Brandtzaeg and Heim, 2008). With this in mind, it is important that SNSs follow usability guidelines so that they retain users.
That being said, despite the slight difficulties and usability problems encountered while carrying out the different tasks in this study, it seems that the LinkedIn website is highly usable. The evidence for this is that 61% of the problems discovered during the expert evaluation were cosmetic in nature, which means that they could be fixed if enough time was available. The difficulties that were discovered by the novice users were 35% minor and 4% major. No catastrophic usability problems were encountered. Therefore, there is a need to improve the areas which participants pointed out as being difficult in order to reduce usability problems and attain optimum user satisfaction for the novice users. Hence, following the set of recommendations summarised above on how to improve the overall use of social network sites should solve many of the usability problems.
Further studies can be conducted by considering the following issues in order to overcome the drawbacks that occurred in this study:
– Employing a wider age range of participants (i.e. upper age more than 40 and lower age less than 20).
– Employing a balanced gender ratio.
– Using remote testing or employing a set of specific generated heuristics for social networking sites.
– Conducting the same study on different social networking sites.
References
Alsumait, A. & Al-Osaimi, A. (2009). “Usability Heuristics Evaluation for Child E-Learning Applications,” In Proceedings of the 11th International Conference on Information Integration and Web-based Applications & Services, pages 425-430.ACM.
Publisher – Google Scholar
Benbunan-Fich, R. (2001). “Using Protocol Analysis to Evaluate the Usability of a Commercial Website,” Information & Management, 39(2):151-163.
Publisher – Google Scholar
Benson, L., Elliott, D., Grant, M., Holschuh, D., Kim, B., Kim, H. Lauber, E., Loh, S. & Reeves, T. C. (2002). “Usability and Instructional Design Heuristics for E-Learning Evaluation,” In World Conference on Educational Multimedia, Hypermedia and Telecommunications, pages 1615-1621.
Publisher – Google Scholar
Bevan, N. & Spinhof, L. (2007). “Are Guidelines and Standards for Web Usability Comprehensive? Human Computer Interaction,” Interaction Design and Usability, pages 407-419.
Publisher – Google Scholar – British Library Direct
Blackmon, M. H., Polson, P. G., Kitajima, M. & Lewis, C. (2002). “Cognitive Walkthrough for the Web,” In Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, pages 463-470. ACM.
Publisher – Google Scholar – British Library Direct
Boyd, D. M. & Ellison, N. B. (2008). “Social Network Sites: Definition, History, and Scholarship,” Journal of Computer-Mediated Communication, 13(1):210-230.
Publisher – Google Scholar
Castells, M. (2003). The Internet Galaxy: Reflections on the Internet, Business, and Society, Oxford University Press, USA.
Publisher – Google Scholar
Chattratichart, J. & Lindgaard, G. (2008). “A Comparative Evaluation of Heuristic-Based Usability Inspection Methods,” InCHI’08 Extended Abstracts on Human Factors in Computing Systems, pages 2213-2220. ACM.
Publisher – Google Scholar
Chen, S. Y. & Macredie, R. D. (2005). “The Assessment of Usability of Electronic Shopping: A Heuristic Evaluation,”International Journal of Information Management, 25(6):516-532, 2005.
Publisher – Google Scholar
Cleary, Y. (2000). ‘An Examination of the Impact of Subjective Cultural Issues on the Usability of a Localized Website: The Louvre Museum Website,’ In Museums and the Web, pages 153-162.
Donath, J. & Boyd, D. (2004). “Public Displays of Connection,” BT Technology Journal, 22(4):71-82.
Publisher – Google Scholar – British Library Direct
Dwyer, C., Hiltz, S. R. & Passerini, K. (2007). “Trust and Privacy Concern within Social Networking Sites: A Comparison of Facebook and MySpace,” In Proceedings of AMCIS. Citeseer.
Publisher – Google Scholar
Fang, X. & Holsapple, C. W. (2007). “An Empirical Study of Website Navigation Structures’ Impacts on Website Usability,” Decision Support Systems, 43(2):476-491.
Publisher – Google Scholar
Federoff, M. A. (2002). “Heuristics and Usability Guidelines for the Creation and Evaluation of Fun in Video Games,”PhD thesis, Citeseer.
Publisher – Google Scholar
Golbeck, J. (2007). “The Dynamics of Web-Based Social Networks: Membership, Relationships, and Change,” First Monday, 12(11):1-17.
Publisher – Google Scholar
Hart, J., Ridley, C., Taher, F., Sas, C. & Dix, A. (2008). “Exploring the Facebook Experience: A New Approach to Usability,” In Proceedings of the 5th Nordic Conference on Human-computer Interaction: Building Bridges, Pages 471-474. ACM. 43
Publisher – Google Scholar
Henninger, S. (2001). “An Organizational Learning Method for Applying Usability Guidelines and Patterns,” Engineering for Human-Computer Interaction, pages 141-155.
Publisher – Google Scholar – British Library Direct
Hertzum, M. (2010). “Images of Usability,” International Journal of Human-Computer Interaction, 26(6):567-600.
Publisher – Google Scholar
Holzinger, A. (2005). “Usability Engineering Methods for Software Developers,” Communications of the ACM, 48(1):71-74.
Publisher – Google Scholar
Hom, J. (2002). “Usability Evaluation Methods,” Accessed on 9/06/05, Available at: [http://www.pages.drexel.edu/~zwz22/UsabilityHome.html]
Publisher
Huberman, B., Romero, D. & Wu, F. (2009). “Social Networks that Matter: Twitter under the Microscope,” First Monday, 14(1):8.
Publisher – Google Scholar
Hvannberg, E. T., Law, E. L. C. & Larusdottir, M. K. (2007). “Heuristic Evaluation: Comparing Ways of Finding and Reporting Usability Problems,” Interacting with Computers, 19(2):225- 240.
Publisher – Google Scholar
Ivory, M. Y. & Hearst, M. A. (2001). The State of the Art in Automating Usability Evaluation of User Interfaces, ACM Computing, vol. 33 (4), pp. 470-516.
Publisher – Google Scholar
Kaki, M. (2004). “Proportional Search Interface Usability Measures,” In Proceedings of the Third Nordic Conference on Human-computer Interaction, Pages 365-372. ACM.
Publisher – Google Scholar
Keenan, A. & Shiri, A. (2009). “Sociability and Social Interaction on Social Networking Websites,” Library Review, 58(6):438-450.
Publisher – Google Scholar
Leiner, B. M., Cerf, V. G., Clark, D. D., Kahn, R. E., Kleinrock, L., Lynch, D. C., Postel, J., Roberts, L. G. & Wolff, S. (2009). “A Brief History of the Internet,” ACM SIGCOMM Computer Communication Review, 39(5):22-31.
Publisher – Google Scholar
Ling, C. & Salvendy, G. (2005). “Extension of Heuristic Evaluation Method: A Review and Reappraisal,”
Publisher – Google Scholar
MacDorman, K. F., Whalen, T. J., Ho, C.- C. & Patel, H. (2011). “An Improved Usability Measure Based on Novice and Expert Performance,” International Journal of Human Computer Interaction, 27(3):280-302.
Publisher – Google Scholar
Mack, R. & Nielsen, J. (1993). “Usability Inspection Methods,” ACM SIGCHI Bulletin, 25(1):28-33.
Publisher – Google Scholar – British Library Direct
Magoulas, G. D., Chen, S. Y. & Papanikolaou, K. A. (1990). “Integrating Layered and Heuristic Evaluation for Adaptive Learning Environments,” UM2001, page 5.
Publisher – Google Scholar
Mendoza, V. & Novick, D. G. (2005). “Usability over Time,” In Proceedings of the 23rd Annual International Conference on Design of Communication: Documenting & Designing for Pervasive Information, Pages 151-158. ACM.
Publisher – Google Scholar
Molich, R. & Nielsen, J. (1990a). “Improving a Human-Computer Dialogue,” In Communications of the ACM 33, 3 (March), pages 338-348. ACM.
Publisher – Google Scholar
Muir, A., Shield, L. & Kukulska-Hulme, A. (2003). “The Pyramid of Usability: A Framework for Quality Course Websites,” In Proceedings of EDEN 12th Annual Conference of the European Distance Education Network, The Quality Dialogue: Integrating Quality Cultures in Flexible, Distance and eLearning, Rhodes, Greece, pages 15-18, 2003.
Publisher – Google Scholar
Nielsen, J. (1992). “Finding Usability Problems through Heuristic Evaluation,” In Proceedings ACM CHI’92 Conference (Monterey, CA, May 3-7), pages 373-380. ACM.
Publisher – Google Scholar – British Library Direct
Nielsen, J. (1993). Usability Engineering, Morgan Kaufmann.
Publisher – Google Scholar
Nielsen, J. (1994). “Heuristic Evaluation,” Usability Inspection Methods, pages 25-62.
Publisher – Google Scholar
Nielsen, J. (1994). “Usability Inspection Methods,” In Conference Companion on Human Factors in Computing Systems, pages 413-414. ACM.
Publisher – Google Scholar
Nielsen, J. (2001). ‘Success Rate: The Simplest Usability Metric,’ Book Success Rate: The Simplest Usability Metric, Series Success Rate: The Simplest Usability Metric.
Google Scholar
Nielsen, J. (2003). Jakob Nielsens Alertbox, August 25, (2003). “Usability 101: Introduction to Usability,” Useit.com.
Publisher – Google Scholar
Nielsen, J. & Loranger, H. (2006). “Prioritizing Web Usability,” New Riders Publishing Thousand Oaks, CA, USA.
Publisher – Google Scholar
Nielsen, J. & Molich, R. (1990b). “Heuristic Evaluation of User Interfaces,” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Empowering people, 249–256.ACM.
Publisher – Google Scholar
OMurchu, I., Breslin, J. G. & Decker, S. (2004). “Online Social and Business Networking Communities,” In Proceedings of ECAI 2004 Workshop on Application of Semantic Web Technologies to Web Communities. Citeseer.
Publisher – Google Scholar
Otaiza, R., Rusu, C. & Roncagliolo, S. (2010). ‘Evaluating the Usability of Transactional Websites,’ In Advances in Computer-Human Interactions, 2010. ACHI’10. Third International Conference on, pages 32-37. IEEE.
Pinelle, D., Wong, N. & Stach, T. (2008). “Heuristic Evaluation for Games: Usability Principles for Video Game Design,” In Proceeding of the Twenty-sixth Annual SIGCHI Conference on Human Factors in Computing Systems, pages 1453-1462. ACM,.
Publisher – Google Scholar – British Library Direct
Ramey, J., Boren, T., Cuddihy, E., Dumas, J., Guan, Z., van den Haak, M. & De Jong, M. (2006). “Does Think-Aloud Work?: How do We Know?,” In CHI’06 Extended Abstracts on Human Factors in Computing Systems, pages 45-48. ACM.
Publisher – Google Scholar
Schmettow, M. (2009). “Controlling the Usability Evaluation Process under Varying Defect Visibility,” In Proceedings of the 2009 British Computer Society Conference on Human Computer Interaction, pages 188-197. British Computer Society.
Publisher – Google Scholar
Shackel, B. (1991). Usability-Context, Framework, Definition, Design and Evaluation, Human Factors for Informatics Usability, pages 21-37.
Publisher – Google Scholar
Shackel, B. & Richardson, S. J. (1991). Human Factors for Informatics Usability, Cambridge Univ Press.
Publisher – Google Scholar
Sharp, H., Rogers, Y. & Preece, J. (2006). Interaction Design: Beyond Human-Computer Interaction, John Wiley.
Publisher – Google Scholar
Squires, D. & Preece, J. (1996). “Usability and Learning: Evaluating the Potential of Educational Software,” Computers & Education, 27(1):15-22.
Publisher – Google Scholar – British Library Direct
Uldall-Espersen, T. (2005). “Benefits of Usability Work-does it Pay off?,” In International COST294 Workshop on User Interface Quality Models, page 7. Citeseer.
Publisher – Google Scholar
Vermeeren, A. P. O. S., van Kesteren, I. E. H. & Bekker, M. M. (2003). “Managing the Evaluator Effect in User Testing,” In Proceedings of the IFIP 9th International Conference on Human Computer Interaction-Interact, volume 3, pages 647-654.
Publisher – Google Scholar
Wentz, B. & Lazar, J. (2009). “Email Accessibility and Social Networking,” Online Communities and Social Computing, pages 134-140.
Publisher – Google Scholar
Wilson, C., Boe, B., Sala, A., Puttaswamy, K. P. N. & Zhao, B. Y. (2009). “User Interactions in Social Networks and their Implications,” In Proceedings of the 4th ACM European conference on Computer systems, pages 205-218. ACM.
Publisher – Google Scholar
Wixon, D. (2003). “Evaluating Usability Methods: Why the Current Literature Fails the Practitioner,” Interactions, 10(4):28-34.
Publisher – Google Scholar – British Library Direct
