Web Portal Design: Employment of a Range of Assessment Methods

14 Web Portal Design: Employment of a Range of Assessment Methods Andrina Granic´, Ivica Mitrovic´ and Nikola Marangunic´

Abstract This chapter reports on the experience regarding usability evaluation of web portals. The study is placed in Croatian web sphere where the most visited portals are the broad-reach web ones. Consequently, such a research may be of interest to the communities with comparable user population and market characteristics. The evaluation methodology advocates a number of usability test methods along with specialists’ inspection. The results of the first study indicated that the chosen research instruments, measures and methods for usability testing were consistent. Conversely, the results of the second study, which employs the guideline-based inspection, did not agree with those obtained through the end-user testing. Although showing significant potential, the methodology needs to be improved. Keywords Usability evaluation



Web portals



Guideline inspection



User testing



Discount approach

1. Introduction Usability evaluation plays a fundamental role in a human-centered design process, because it enables and facilitates design according to usability engineering principles. It is related to ease of use and ease of learning. As usability is defined as a relationship between task, user and system purpose, there is no simple definition or meaningful single measure of usability. Most assessment methods are grouped into two usability test methods, namely, user-based methods which involve end users or usability inspection methods engaging usability experts. Recent research has tended to bring together these two basic approaches, cf. [7]. When considering usability of a web portal, a site that functions as a point of access to information on the www, it should be noted that current usability evaluation research is mostly concerned with focused, domain-specific portals. This is the result of a global trend of portal specialization which has made it difficult to find studies related to assessments of broad-reach web portals. This chapter reports on our experience with the design of a ‘‘discount evaluation approach’’ to web portal usability assessment. The main motivation for undertaking this research initiative came from reports stating that the most visited Croatian web sites are broad-reach web portals. In order to evaluate how easy to use and efficient those portals are, we conducted an experiment employing a range of assessment methods, both empirical and analytic. A major strength of such an approach is the chance to supplement results from both the guideline evaluation and the empirical end user-based one, enhanced by users’ feedback on their comfort while working with the web portal. The complementing usability test methods employed in the first study proved to be consistent. Conversely, results of the guideline-based evaluation from the second study were not in agreement with the ones obtained from the usability testing, raising some Andrina Granic´  Split, Split, Croatia

Faculty of Science, University of Split, Split, Croatia Ivica Mitrovic´  Arts Academy, University of Nikola Marangunic´  Faculty of Science, University of Split, Split, Croatia

G.A. Papadopoulos et al. (eds.), Information Systems Development, DOI 10.1007/b137171_14, Ó Springer ScienceþBusiness Media, LLC 2009 131

132

Andrina Granic´ et al.

concerns which will be addressed in our future work. The results support the assertion that we should not rely on isolated evaluations. Instead, usability assessment methods should be combined, giving rise to different kinds of usability improvement suggestions. The chapter is organized as follows. Section 2 briefly introduces some background to the research, addressing web portals and web usability. Section 3 presents the experimental study for web portal usability testing, while an approach which advocates guideline inspection is offered in Section 4. Section 5 concludes the chapter and in addition provides some directions for future research.

2. Background to the Research A web portal can be defined as a personalized, single point of access to information, resources and services covering a wide range of topics [21]. As an Internet-based application, a portal typically uses more advanced technologies that go beyond the simple interface used for the solely information-based, standard web page. Through the blend of information, services and collaboration among users, a portal’s primary objective is to create a working environment that users can easily navigate through. Broad-reach web portals, also called ‘‘general’’ or ‘‘generic’’ portals, as gateways to the web, represent starting points in user browsing and may serve as users’ anchor sites. They offer a collection of services such as search engines, online shopping, e-mail, news, forums, maps, event guides, employment, travel and other kinds of information. Nowadays portals have evolved from monolithic systems to loosely coupled compounds that glue together different application [12]. Furthermore, information presented in each page addresses a very large user group with highly diverse needs and interests, and portal design has to reflect this. Market research findings related to the Croatian web sphere, and which was undertaken in the last few years, report that broad-reach portals are the most visited web sites, cf. [5]. This is the basic distinction between Croatia and the countries with high levels of Internet literacy, where more specialized web portals are seen as gateways to varieties of web information related to specific contexts [15]. In the context of the global trend of web portal specialization, recent research related to usability evaluation is mostly connected with focused portals, often called ‘‘vertical’’ or ‘‘domain specific", such as enterprise portals [1], travel portals [4], news portals [19], library web portals [2], tourist portals [10], healthcare portals [16] and similar. Apparently, while there are a number of studies related to the evaluation of specialized portals, some of which also taking into account the particular cultural context, e.g. [16, 19] there is very little research dealing with broad-reach portal assessment. Taking into consideration outcomes of prior research related to web portals and usually employed usability assessment methods, the aim of the study is to design a methodology for usability evaluation of web portals. The study is placed in the Croatian web sphere where broad-reach portals are much more popular and accepted than specific ones, thus implying that they would be familiar to end users and designers. For that reason such a research may be helpful to other countries/communities with similar context.

3. Study 1: Usability Testing We conducted a controlled experiment which advocates scenario-guided user evaluations involving a number of usability testing methods, cf. [14], used to collect both quantitative data and qualitative ‘‘remarks’’ (refer to [6] too). The study involved 30 participants with basic computer literacy skills. According to their practical experience in web design, they were classified into two different groups composed of 15 participants: (i) the ‘‘practitioner’’ group was composed of three independent subgroups of randomly chosen participants including computer science experts, marketing experts familiar with Internet issues and students of web design, and (ii) the ‘‘non-practitioner’’ group was composed of three independent subgroups of randomly chosen young, middle aged and elderly participants. End user testing was based on criteria expressed in terms of two types of measures [9]: (i) objective performance measurement of effectiveness and efficiency, and (ii) subjective users’ assessment. The System Usability Scale (SUS), a simple, standard, 10-item attitude questionnaire with a five-point Likert scale [3],

Web Portal Design: Employment of a Range of Assessment Methods

133

Figure 14.1. User interface screenshots of evaluated portals (clockwise, starting from top left: Index portal, Net portal, Vip portal and T-Portal).

was used for the subjective evaluation. As additional subjective feedback, answers to the semi-structured interview were collected. We included four broad-reach web portals in our experiment, the most visited and also the firstestablished ones, see Fig. 14.1: Index portal (www.index.hr), Net portal (www.net.hr), Vip portal (www.vip.hr) and T-Portal (www.tportal.hr). 3.1. Research Design and Methodology In order to understand the effect of web portal design in a sample work situation, we described a work scenario, a sequence of typical tasks and user actions. To test assigned tasks and time interval, clarity and unambiguity of measuring instruments for subjective assessment and adequacy of hardware and software support, pilot testing was performed. We chose several typical tasks whose structure and location on the portals had not changed over time. The tasks covered different topics, offering diverse groups of participants a similar opportunity for finding task-related information. For each portal selected, the tasks undertaken were the same and the probability of their completion was similar. The evaluation procedure was carried out individually with each test user, using a personal computer with Internet access in addition to software and hardware support for tracing and recording users’ actions and navigation. Within each evaluation session all the portals were assessed, with the order of their

Andrina Granic´ et al.

134

evaluation randomly selected. The allocated session’s average time for each participant was 45 min. An evaluation procedure consisted of the following steps:

 task-based end user testing,  usability satisfaction questionnaire and  semi-structured interview. Task-based end user testing involved a scenario-guided user assessment with tasks selected to show the basic portal functionality. It enabled us to determine user efficiency (time on task) and effectiveness (percent of task completed) while working with the web portal. A user’s objective accomplishment measure, labeled as fulfillment, was calculated as the average time spent on all allocated tasks weighted with successfulness of task completion. For each user, the time limit for all assigned tasks was 15 min per portal. A usability satisfaction questionnaire enabled the assessment of the users’ subjective satisfaction with diverse types of interaction. We used the SUS questionnaire, as it is argued that this yields the most reliable results across sample sizes [20]. Its questions address different aspects of the user’s reaction to the portal as a whole, providing an indication of the level of statement agreement on a five-point Likert scale. The feedback was augmented with the users’ answers in a semi-structured interview. In this interview we asked the participants to rate and comment on the portal’s visual attractiveness as well. In the following section we present experimental results and findings.

3.2. Results and Discussion of Findings Descriptive statistics of the objective accomplishment measure fulfillment, including arithmetic means, standard deviations and significance levels (Kolmogorov-Smirnov coefficient) for normality of distribution are shown in Table 14.1. We noted that the results for the distribution of measure fulfillment on the T-Portal differs significantly from normal distribution (K-S = 0.008). Accordingly, Friedman’s test as a non-parametrical procedure was performed. A statistically significant value of chi square (w2 = 49.4, df = 3, p < 0.01) indicates the existence of differences in the objective accomplishment measure among portals (see Table 14.1). Table 14.1. Objective performance achievement (the lower M score indicates a better result). Fulfillment

M

SD

K-S

M Rank

df

w2

p

Index portal Net portal Vip portal T-Portal portal

59.77 108.40 62.13 171.64

38.726 46.300 17.211 168.143

0.292 0.720 0.656 0.008

1.57 2.93 1.87 3.63

3

49.4