Multi-Criteria Evaluation of the Web-based E-learning System

Computers & Education 50 (2008) 894–905 www.elsevier.com/locate/compedu

Multi-criteria evaluation of the web-based e-learning system: A methodology based on learner satisfaction and its applications Daniel Y. Shee *, Yi-Shun Wang Department of Information Management, National Changhua University of Education, Changhua 500, Taiwan

Received 30 May 2006; received in revised form 19 August 2006; accepted 11 September 2006

Abstract

The web-b web-base ased d e-lear e-learnin ningg syste system m (W (WELS ELS)) has emerg emerged ed as a new means means of skill skill traini training ng and knowl knowled edge ge acquis acquisiti ition, on, encour encour-aging both academia and industry to invest resources in the adoption of this system. Traditionally, most pre- and post-adoption tion tasks tasks relat related ed to evalua evaluati tion on are carrie carried d out from from the the viewpo viewpoint intss of techno technolog logy. y. Since Since users users have have been been widely widely recogn recognize ized d as being a key group of stakeholders in influencing the adoption of information systems, their attitudes toward this system are pivotal. Therefore, based on the theory of multi-criteria decision making and the research products of user satisfaction from the fields of human–computer interaction and information systems, this study proposed a multi-criteria methodology from the perspective of learner satisfaction to support those evaluation-based activities taking place at the pre- and post-adoption phases of the WELS life cycle. In addition, by following this methodology, this study empirically investigated learners’ percepti ceptions ons of the relat relative ive import importanc ancee of decisi decision on crite criteria ria.. This This inves investig tigati ation on carrie carried d out a surve surveyy of colleg collegee stude students nts,, and the data data thus thus obtain obtained ed was then then analyz analyzed ed by analyt analytic ic hierar hierarch chyy proces processs in order order to derive derive an integr integrate ated d prefer preferenc encee struct structur uree of learn learners ers as a ground for evaluation. We found that learners regarded the learner interface as being the most important dimension of decision criteria. Future applications of these results are recommended and the implications are discussed. Ó

2006 Elsevier Ltd. All rights reserved.

Keywords: Web-based e-learning system; Multi-criteria methodology; Learner satisfaction

1. Introduction

The capability and flexibility of the web-based e-learning system (WELS) have been demonstrated in both training training and education, education, resulting in its adoption adoption by the academia as well as the industry. industry. Since the commercial application package (or commercial off-the-shelf ) strategy of system development is so widespread ( Whitten, Bentley, & Dittman, 2004), 2004), the proliferation of WELS applications has created confusion for the potential adopters when they have to make a decision regarding the selection from candidate products or solutions. At the same time, those organizations which have already adopted a system are faced with issues that arise in the post-adoption phase. For example, what are the improvements or enhancements that must be carried *

Corresponding author. Tel.: +886 4 7232105x7614; fax: +886 4 7211162. E-mail addresses: [email protected] (D.Y. Shee), [email protected] (Y.-S. Wang).

0360-1315/$ - see front matter Ó 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.compedu.2006.09.005

D.Y. Shee, Y.-S. Wang / Computers & Education 50 (2008) 894–905

895

out? What is the priority maintenance item that deserves the limited resources of the organization? A prior evaluation is required to answer these questions, and a sound methodology is the key to effective evaluation. Conventional approaches for evaluating an information system (IS) have leaned towards the standpoints of technical personnel (Kao, 1998; Karat, 1988; Smith & Williams, 1999). In contrast, the WELS places particular stress on certain areas, such as, the content and the ways in which it is presented, demonstrating that it is a highly user-oriented system. Since users are widely recognized as the key stakeholders in any IS or IS service (Jiang, Klein, Roan, & Lin, 2001), their attitudes toward the system are pivotal and should be valued. This is evidenced by the fact that user satisfaction is often seen as a key antecedent to predict the success of a particular IS (DeLone & McLean, 2003; Melone, 1990; Raymond, 1987), or to anticipate a user’s behavior of reuse (Gelderman, 1998; Lin & Wang, 2006; Lin, Wu, & Tsai, 2005). Hence, this study will apply the construct of user satisfaction to evaluate WELS. In the context of WELS, however, there is a special group of users, the learners, who hold a unique view regarding satisfaction ( Wang, 2003). Actually, they are the e-learners. This means that traditional measures for assessing user satisfaction with IS and for assessing learner satisfaction in the context of classroom teaching are not suitable for web-based e-learning. The purpose of this study is twofold. First, it will propose a step-based, multi-criteria methodology from the perspective of e-learner satisfaction in order to support important evaluation-related tasks (e.g., selection from candidate products or solutions, and maintenance), which will be carried out at the pre- or post-adoption phase of the WELS life cycle. This includes defining the constituent steps and recommending tools and techniques which can be used in each step. Second, by following the proposed methodology, this study will investigate how e-learners perceive the relative importance of the decision criteria and the dimensions of these criteria in order to construct a preference structure, which is key in this decision-making process. The remainder of this paper is organized as follows: Section 2 discusses WELS, the development of user satisfaction scales, and the theory of multi-criteria decision making. Section 3 proposes and explains the methodology. Section 4 describes how the empirical investigation is carried out to derive the preference structure, and presents the results of the investigation. Section 5 provides a case study as an illustrative example. Finally, we present the discussions in Section 6 and draw our conclusions in Section 7. 2. Theoretical background

2.1. Web-based e-learning system

E-learning refers to the use of electronic devices for learning, including the delivery of content via electronic media such as Internet/Intranet/Extranet, audio or video tape, satellite broadcast, interactive TV, CD-ROM, and so on (Kaplan-Leiserson, 2000). This type of learning moves the traditional instruction paradigm to a learning paradigm (Jo¨nsson, 2005), thereby relinquishing much control over planning and selection to the learners. In addition, it is capable of bringing the following advantages to learners: cost-effectiveness, timely content, and access flexibility (Hong, Lai, & Holton, 2003; Lorenzetti, 2005; Rosenberg, 2001). E-learning applications may appear with different forms of designation such as web-based learning, virtual classrooms, and digital collaboration (Kaplan-Leiserson, 2000; Khalifa & Kwok, 1999). This study is focused on web-based e-learning which is conducted using the Internet (or Intranet/Extranet) and web technologies. This type of e-learning places a greater emphasis on the enabling or facilitating role technology plays in data search and transmission, interactivity, and personalization (Piccoli, Ahmad, & Ives, 2001). Regarding the design and construction of a WELS, the trend is toward incorporating two different technologies: webpagebased computer-assisted instruction (Fletcher-Flinn & Gravatt, 1995), which is basically the tutorial, drill and practice, and learning networks (Hiltz & Turoff, 2002), including extensive learner–learner and instructor–learner communication and interaction, into an integrated environment. Learners in this environment can thus access remote resources, as well as interact with instructors and other learners to satisfy their requirements. 2.2. From user satisfaction to e-learner satisfaction

Satisfaction is ‘‘the pleasure or contentment that one person feels when she/he does something or gets something that she/he wanted or needed to do or get’’ ( Collins Cobuild English Dictionary, 1999). When used in

896


research, satisfaction is usually conceptualized as the aggregate of a person’s feelings or attitudes toward the many factors that affect a certain situation (Bailey & Pearson, 1983). In the field of human–computer interaction, user satisfaction is usually visualized as the expression of affections gained from an interaction ( Mahmood, Burn, Gemoets, & Jacquez, 2000 ). This means that user satisfaction is the ‘‘subjective sum of interactive experiences’’ influenced by many affective components in the interaction ( Lindgaard & Dudek, 2003). In the field of IS, the concept of user satisfaction is usually used to represent the degree to which users believe the IS they are using conforms to their requirements ( Cyert & March, 1963). In the past, many scholars have attempted to measure user satisfaction. The results of their efforts revealed that user satisfaction is a complex construct and its substance varies with the nature of the experience or case. In the field of human–computer interaction, user satisfaction is traditionally measured in terms of visual appeal, productivity, and usability (Hassenzahl, Beau, & Burmester, 2001; Lindgaard & Dudek, 2003 ). Since the early 1980s, many scholars in the field of IS began to conduct systematic studies to develop a comprehensive and reasonable set of factors to measure user satisfaction. For example, Bailey and Pearson (1983) developed an instrument with 39 items to measure the level of perceived user satisfaction with IS. Ives, Olson, and Baroudi (1983) proposed the scale of User Information Satisfaction which consists of three parts: electronic data processing staff and services, information product, and knowledge or involvement. Doll and Torkzadeh (1988) developed a questionnaire with 18 items, which can be classified into five dimensions: system content, system accuracy, report format, ease of use, and system timeliness, to assess the End-user Computing Satisfaction. New scales developed in the past decade are largely based on the aforementioned products. The prevalence of web-based e-learning applications stimulates the development of e-learner satisfaction scales by directly adapting from teaching quality scales in the field of educational psychology (e.g., Cashin & Downey, 1992; Cohen, 1981; Marsh, 1991), or from user satisfaction scales in the field of human–computer interaction or IS. However, the application of the achievement from any single field is deemed insufficient because it can omit some critical aspects of learner satisfaction with WELS. Based on the scales of student’s evaluation of teaching effectiveness and user satisfaction, Wang (2003) conducted an exploratory study directed at e-learners. The results of his work showed that a total of 17 items applicable to measuring e-Learner Satisfaction could be classified into the following dimensions: content, personalization, learning community and learner interface. 2.3. Multi-criteria decision making

Multi-criteria decision making (MCDM), which deals mainly with problems about evaluation or selection (Keeney & Raiffa, 1976; Teng, 2002), is a rapidly developing area in operational research and management science. The complete MCDM process involves the following basic elements: criterion set, preference structure, alternative set, and performance values (Yu, 1985). While the final decision will be made based on the performance of alternatives, a well-defined criterion set and preference structure are key influential factors and should be prepared in advance. In order to obtain the criterion set and preference structure, a hierarchical analysis must be carried out. Such an analysis helps decision makers to preliminarily derive an objective hierarchy structure to demonstrate the relationship between the goal and the decision criteria ( MacCrimmon, 1969). The goal of the hierarchy may be ‘‘a perceived better direction of a decision organization’’ ( Teng, 2002). On the other hand, the criteria represent the ‘‘standards for judging’’ ( Hwang & Masud, 1979), which should be complete, operational, decomposable, non-redundant, and minimal in size ( Keeney & Raiffa, 1976; Teng, 2002). Based on this hierarchy structure, decision makers can set about deriving the relative importance of the criteria and then assessing alternatives against each criterion. By integrating the assessments of alternatives with the relative importance of criteria, an organization can select one alternative which best meets its requirements to accomplish its goal. 3. The methodology

Fig. 1a shows the system life cycle under a commercial application package implementation strategy. The major characteristic of this strategy, compared with an in-house development one, is that the organization communicates its system requirements in a form of either request for proposal or request for quotation to

897


a

System change request (improvements or enhancements)

1. SCOPE DEFINITION

2. Problem statement

Start

THE USER COMMUNITY

4. ADOPTION (INSTALLATION & Operational CUSTOMIZATION) system

A product or

(3 or 5).2. HIERARCHY STRUCTURE DEVELOPMENT Hierarchical analysis from an e-learner satisfaction perspective Empirical investigation Posterior examination

Business requirement

3. DECISION ANALYSIS

2.1. LOGICAL DESIGN WELS VENDORS

Logical design

Alternative products/solutions

solution

b (3 or 5).1. PROBLEM & GOAL DEFINITION

Request for proposal or request for quotation statement

Route for in-house development

Working solution

5. OPERATION & MAINTENANCE

REQUIREMENT ANALYSIS

(3 or 5).3. DERIVING THE PREFERENCE STRUCTURE

5.4. SYSTEM EVALUATION Rating-based method

3.4. Survey of e-learners AHP (pool first and pool last methods)

GATHERING THE ALTERNATIVES

5.5. IMPROVEMENTS & ENHANCEMENTS IDENTIFICATION Predefined threshold Weighted distance from perfection

3.5. ALTERNATIVES EVALUATION

3.6. ALTERNATIVE SELECTION

Rating-based method Ranking-based method

Fig. 1. The methodology. (a) System life cycle under the commercial application package implementation strategy (except the shadowed area) (adapted from Whitten et al., 2004) and (b) multi-criteria methodology for evaluating WELS from the perspective of e-learner satisfaction.

candidate WELS vendors. Afterwards, those vendors submit their products or solutions as alternatives for evaluation (Whitten et al., 2004). The methodology proposed in this study aims at supporting the tasks that take place in the phase of decision analysis or the phase of operation and maintenance. As shown in Fig. 1b, this methodology is logically divided into steps. The explanations, including the tasks and the tools or techniques applicable in each step, are as follows. The start-up step defines the problem and the goal . The problems may be those, as mentioned in the first section, which are experienced by an organization in the phase of decision analysis or operation and maintenance when the commercial application package implementation strategy is used. To solve them, evaluation is necessary. Therefore, the goal is defined as the evaluation of WELS alternatives. After defining the problems and the goal, the next step involves the development of the hierarchy structure. In this step, a hierarchical analysis based on e-learner satisfaction is to be carried out. Literature review, systematic analysis, empirical investigation, brainstorming, and interpretive structural modeling are the feasible methods ( MacCrimmon, 1969). The completion of this step will answer the following questions: What are the criteria that can be applied to this context? How are these criteria classified into dimensions? How can these criteria and dimensions be arranged into a hierarchy. The hierarchy structure used in this study for evaluating WELS alternatives was adapted from Wang’s (2003) empirical work, because what we refer to as learner satisfaction in the WELS context is conceptually close to his e-Learner Satisfaction. However, a considerable amount of Wang’s measurement items, which would violate the principle of minimal size in the criterion set and complicate the MCDM process, inspired us to further examine those items. This examination was carried out through discussions with three professors of MIS and five experienced WELS learners in order to rate the relevance of each item in terms of WELS evaluation and to check if there were any conceptually or connotatively redundant items. As a result, a total of four items were eliminated, with the remainder being transformed into the form of decision criteria. As shown in Fig. 2, we ended up with four dimensions, comprising a total of 13 criteria in the hierarchy structure.

898


Goal

Dimensions

Criteria C01 Ease of use C02 User-friendliness

s e v i t a n r e t l A S L E W f o n o i t a u l a v E e h T

D1: Learner Interface

C03 Ease of understanding C04 Operational stability C05 Ease of discussion with other learners D2: Learning Community

C06 Ease of discussion with teachers C07 Ease of accessing shared data C08 Ease of exchanging learning with the others C09 Up-to-date content

D3: System Content

C10 Sufficient content C11 Useful content C12 Capability of controlling learning progress

D4: Personalization

C13 Capability of recording learning performance

Fig. 2. The hierarchy structure for evaluating WELS.

Based on the hierarchy structure, the third step, deriving the preference structure, explores learners’ perceptions of the relative importance of the criteria and the dimensions of these criteria. This may help answer what it is that users regard highly in terms of learner satisfaction in the context of WELS. Among several procedures that have been proposed, the analytic hierarchy process (AHP), developed by Saaty (1980), is recommended because of its superiority in judgment and measuring scales over others ( Forman, 1989; Lane & Verdini, 1989). Satty also proposed a consistency ratio to examine the rationality of the judgment of the decision-maker. A value of consistency ratio less than 0.1 is deemed as being sufficiently consistent. After the preference structure is obtained, the center of MCDM activities shifts to the evaluation of WELS alternatives. For the phase of decision analysis, organizations can gather the alternatives and then evaluate the alternatives against the criteria. In this evaluation, two methods, rating-based and ranking-based, are recommended. The rating-based method involves assessing a particular alternative by rating it under each criterion or dimension. The overall performance of this alternative can be acquired by summating its weighted performance under each criterion or dimension, and the decision organization can thus select from alternatives according to the overall performance of each alternative. The simplest form of this method, as shown in the right-half of Fig. 3, is the simple additive weight method. On the other hand, as shown in the left-half of Fig. 3, the ranking-based method involves ranking the alternatives by their key attributes. Traditional ranking procedures used in the field of social science include: method of rank order, method of paired comparisons, method of constant stimuli, and method of successive categories (Yang, Wen, Wu, & Li, 2001). In the present paper, we recommend the method of paired comparisons because it can be integrated with the preference structure (weights of criteria and dimensions) to facilitate overall assessment of the alternatives. Moreover, the use of the method of paired comparisons means that another AHP must be carried out in this step. This makes the evaluation a straightforward task for organizations, because they have already had quite a few experiences with AHP from the previous step ( deriving the preference structure). Under this method, the alternatives will be pairwise-compared with respect to each criterion or dimension to derive normalized relative priorities of each alternative. The overall priority of each alternative and their rankings can then be used as the basis for selection. Finally, in the phase of operation and maintenance, the existing system will be assessed against criteria or dimensions. Under these circumstances, only the rating-based method is applicable. Under a particular criterion or dimension, a performance value below the pre-defined threshold indicates that this is an area which needs improvement or enhancement. If there are many such areas, and if organizational resources are so limited that maintenance efforts are only allowed to be devoted to one area, then the area with the greatest

899


Dimensions (Criteria) Pairwise comparisons

Weights

Alternative A 1

Alternative A 2

c1 x

A i1

w1

c2 x

A i 2

w2

c3 x

A i 3

w3

… … …

…

d i1= 3

∑ Aij wj j =1

…

p

A i = ∑ d ik = k =1

c n-1

Alternative A m

cn

A i(n-1) A in

w n-1 wn

n

∑ A ij w j j =1

d ip= n

∑ Aij wj =n−1

Ranking-based evaluation

Rating-based evaluation

Note: c j denotes criterion j and w j denotes weight of criterion j, j = 1, 2, …, n. A i denotes WELS i, i = 1, 2,…, m. A ij denotes the performance score of WELS i under criterion j. d ik denotes the performance scoreof WELS i under dimension k , k = 1, 2, …, p.

Fig. 3. The principle of rating-based and ranking-based evaluations.

weighted distance from perfection, defined by weighted distance from perfection = weight o f a particular criterion or a particular dimension · (the perfect score performance score), will be the priority. 1 À

4. Deriving the preference structure: an empirical study by AHP

In order to derive the preference structure, a survey was carried out. Data were gathered from students enrolled in courses taught by means of WELS at a large university in northern Taiwan. Six classes were selected and an AHP questionnaire was distributed to each student (see Appendix for an example of the AHP questionnaire). A total of 276 valid samples were returned. Among the 276 respondents, 86 were freshmen (31.2%), 69 were sophomores (25.0%), 46 were juniors (16.7%), 72 were seniors (26.1%), and three were classified as others (1.1%); as for the colleges to which they were affiliated, 23 were from the College of Science (8.3%), 80 were from the College of Social Sciences (29.0%), 114 were from the College of Commerce (41.3%), 6 were from the College of Law (2.2%), 24 were from the College of Liberal Arts or Foreign Languages (8.7%), 13 were from the College of Communication (4.7%), and 16 were from others (5.8%); the frequency distribution of their experience with computer usage was 4 with less then one year (1.4%), 115 with one to five years (41.7%), 139 with five to ten years (50.4%), and 18 with more than ten years (6.5%); more than half (149 out of 276, 54.0%) indicated that they had experience in using WELS. In this study, the integration of the preferences of each respondent was carried out by pool first and pool last methods (Buckley, 1985), respectively. The pool last method also provides the coefficient of variation to represent the level of variation in the respondents’ perceptions, since this method will produce a preference for each respondent. It must be noted that the weights of each dimension were calculated on the basis of pairwise comparisons between dimensions, and the local weights of each criterion were calculated on the basis of pairwise comparisons between criteria within the same dimension. A criterion’s overall weight can then be obtained by multiplying its local weight by the weight of the dimension to which it belongs. Moreover, not every set of responses with respect to dimensions or goal from each respondent passed the consistency test (a value of consistency ratio less than 0.1). Consequently, for both pool first and pool last methods, we provide results for the entire sample and for that sample but after removing those responses which did not pass the test, respectively, to show the difference between before and after the adjustment. The results are shown in Tables 1 and 2. It is found that the preference structures produced by these two methods are identical, showing 1

Under the rating-based method, for example, if a 1-to-10 (lowest to highest) scale is used, the perfect score is 10.

900


Table 1 Weights of dimensions and criteria (pool first) Dimension D1 D2 D3 D4

Learner interface Learning community System content Personalization

Before adjustment

After adjustment

Weight

Consistency ratio

Weight

Consistency ratio

0.297 0.195 0.272 0.236

0.004

0.319 0.197 0.262 0.222

0.000

Criterion

Local weight

Overall weight

Consistency ratio

Local weight

Overall weight

Consistency ratio

C01 C02 C03 C04 C05 C06 C07 C08 C09 C10 C11 C12 C13

0.230 0.212 0.261 0.297 0.228 0.269 0.257 0.246 0.217 0.353 0.430 0.576 0.424

0.068 0.063 0.078 0.088 0.044 0.052 0.050 0.048 0.059 0.096 0.117 0.136 0.100

0.001

0.248 0.213 0.260 0.279 0.230 0.254 0.280 0.236 0.221 0.361 0.418 0.576 0.424

0.079 0.068 0.083 0.089 0.045 0.050 0.055 0.046 0.058 0.095 0.110 0.128 0.094

0.001

Ease of use User-friendliness Ease of understanding Operational stability Ease of discussion with other learners Ease of discussion with teachers Ease of accessing shared data Ease of exchanging learning with the others Up-to-date content Sufficient content Useful content Capability of controlling learning progress Capability of recording learning performance

0.002

0.004

0.000

0.001

0.001

0.000

The before adjustment column lists results for the entire sample; the after adjustment column lists results for the sample after removing those responses which do not pass the consistency test.

little difference between the results before and after the adjustment. Most of the consistency ratio values after adjustment are, especially in Table 2, lower than those before the adjustment, as are the values of coefficient of variation in Table 2. Further discussions regarding the results are provided in Section 6. 5. Illustration: a case study

In order to prove the applicability of the proposed methodology, three real WELS products, all involving English learning, were employed as illustrative alternatives. A pseudonym was used to protect their anonymity. Ten experienced WELS learners were invited to be the evaluators to demonstrate how a decision can be arrived at for WELS alternatives based on the obtained preference structure and both rating-based and ranking-based methods. The evaluators were first asked to assess the WELS alternatives by using the rating-based method. Under each criterion, these alternatives were rated on a 1-to-10 scale (lowest to highest) which expressed the judgment of the evaluators to the extent to which the WELS alternatives met a particular criterion. The evaluators were then asked to compare these alternatives with respect to each criterion by using the ranking-based method. There were three alternatives with thirteen criteria, resulting in a total of 39 pairwise comparisons under this method. Finally, each WELS alternative’s overall performance or priority can then be acquired by summating its weighted performance scores or weighted priority scores under each criterion. The results are shown in Table 3. In Table 3, we find that both methods produce the same pattern of priority of alternatives except for the one under C07. Let’s assume that the evaluators must decide among the three WELS alternatives: A, B , and C . In this case, WELS B appears to be preferable. If only a few criteria or dimensions are emphasized, for example, when considering D1 (learner interface), WELS A performs better. In addition to supporting the decision of the selection, these results can also be applied to the single-system evaluation in order to find out what improvements or enhancements are required. For example, regarding WELS B , if a threshold value of 7 is set for each dimension under the rating-based method, further efforts of system maintenance should be concentrated on the learning community (D2) and personalization (D4). However, if maintenance is subject to limited organizational resources, the personalization (D4) is the priority because it has a greater weighted distance from perfection.

901


Table 2 Weights of dimensions and criteria (pool last) Dimension

D1 Learner interface D2 Learning community D3 System content D4 Personalization

Before adjustment

After adjustment

Weight

Coefficient of variation

Consistency ratio

Weight


Consistency ratio

0.287 0.204

0.467 0.527

0.164

0.307 0.202

0.414 0.523

0.040

0.268 0.241

0.481 0.544

0.264 0.227

0.463 0.552

Criterion

Local weight

Overall weight


Consistency ratio

Local weight

Overall weight


Consistency ratio

C01 Ease of use C02 User-friendliness C03 Ease of understanding C04 Operational stability C05 Ease of discussion with other learners C06 Ease of discussion with teachers C07 Ease of accessing shared data C08 Ease of exchanging learning with the others C09 Up-to-date content C10 Sufficient content C11 Useful content C12 Capability of controlling learning progress C13 Capability of recording learning performance

0.235 0.218 0.255

0.067 0.063 0.073

0.519 0.491 0.399

0.158

0.246 0.214 0.259

0.076 0.066 0.080

0.385 0.384 0.356

0.045

0.292 0.230

0.084 0.047

0.479 0.457

0.281 0.228

0.086 0.046

0.459 0.386

0.277

0.057

0.507

0.267

0.054

0.485

0.253

0.052

0.427

0.271

0.055

0.341

0.240

0.049

0.442

0.234

0.047

0.382

0.233 0.350 0.417 0.571

0.062 0.094 0.112 0.138

0.699 0.316 0.326 0.409

0.237 0.356 0.407 0.571

0.063 0.094 0.107 0.130

0.643 0.262 0.303 0.409

0.429

0.103

0.544

0.429

0.097

0.544

0.185

0.144

0.000

0.036

0.041

0.000

The before adjustment column lists results for the entire sample; the after adjustment column lists results for the sample after removing those responses which do not pass the consistency test.

6. Discussions

Based on the adjusted AHP results, it is found that WELS learners regard the learner interface as being the most important dimension. Since many IS-related studies have pointed out that the user interface is an area where a high level of interaction takes place ( Dam, 2001; Kumar, Smith, & Bannerjee, 2004), a well-designed, user-friendly learner interface therefore becomes one of the critical factors in determining whether learners will enjoy using the WELS. This should remind those responsible for maintaining WELS to ensure that the present learner interface conforms to learners’ requirements. In addition to a user-friendly interface, learners also place great value on system content. As a result, emphasizing the non-technical aspect of the system content is critical. This also points out that in addition to technical engineers, a sound WELS needs a high level of participation from other non-technical experts, such as teachers, teaching material editors, and pedagogy professionals in the construction phase as well as in the subsequent operation and maintenance. Regarding the criteria of each dimension, we discover that respondents place the greatest emphasis on the stability of the learner interface. As to the learning community, the key issue for the learners is to be able to easily access shared data. When it comes to system content, learners care most about whether they find it useful. As to personalization, the results reflect that the learners’ most important requirement is being able to control their learning progress. When we look at all the criteria with respect to the overall goal of the hierarchy structure, those criteria which belong to personalization and system content are given more weight. This

902


Table 3 Evaluation of three WELS alternatives

C01 Ease of use

Rating-based method (performance score)

Ranking-based method (priority score)

A = 7.0 B = 7.8 POA: B A C A = 7.1 B = 7.6 POA: B A C A = 8.1 B = 7.1 POA: A B C A = 8.2 B = 7.6 POA: A B C A = 7.64 B = 7.52 POA: A B C A = 6.9 B = 6.1 POA: A B C A = 6.7 B = 7.3 POA: B A C A = 6.1 B = 6.1 POA: A = B C A = 7.1 B = 6.2 POA: A B C A = 6.67 B = 6.43 POA: A B C A = 7.2 B = 7.8 POA: B A C A = 6.9 B = 7.8 POA: B A C A = 6.8 B = 7.8 POA: B A C A = 6.93 B = 7.80 POA: B A C A = 5.9 B = 6.8 POA: B A C A = 6.1 B = 6.2 POA: B A C A = 5.98 B = 6.55 POA: B A C A = 6.90 B = 7.17 POA: B A C

A = 0.392 POA: B A = 0.395 POA: B A = 0.433 POA: A A = 0.499 POA: A A = 0.433 POA: A A = 0.480 POA: A A = 0.382 POA: B A = 0.424 POA: A A = 0.447 POA: A A = 0.432 POA: A A = 0.347 POA: B A = 0.353 POA: B A = 0.351 POA: B A = 0.351 POA: B A = 0.386 POA: B A = 0.367 POA: B A = 0.378 POA: B A = 0.399 POA: B

!

C02 User-friendliness

!

C03 Ease of understanding

!

C04 Operational stability

!

D1 Learner interface C05 Ease of discussion with other learners C06 Ease of discussion with teachers C07 Ease of accessing shared data

C = 5.8

!

!

C = 6.2

!

!

C = 6.0

!

!

C = 3.4

!

!

!

!

!

!

!

C = 5.27 C = 4.6 C = 4.7 C = 4.7

!

C08 Ease of exchanging learning with the others

!

D2 Learning community

!

C09 Up-to-date content

!

C10 Sufficient content C11 Useful content D3 System content C12 Capability of controlling learning progress C13 Capability of recording learning performance

!

!

!

!

!

!

!

!

C = 5.8 C = 5.5 C = 5.56 C = 4.7 C = 4.4

!

!

B

!

!

B

!

!

A

!

!

B

!

!

!

B

!

!

A

!

!

A

!

!

A

!

!

A

!

!

A A

C = 0.112 C = 0.160 C = 0.178 C = 0.179 C = 0.157 C = 0.167 C = 0.170

!

A

A

C = 0.183

!

!

!

C = 5.08

B

C = 0.195

!

!

!

C = 4.57

B

C = 0.159

!

B

!

C = 5.3

A

B = 0.449 C B = 0.410 C B = 0.384 C B = 0.389 C B = 0.407 C B = 0.342 C B = 0.439 C B = 0.419 C B = 0.386 C B = 0.399 C B = 0.478 C B = 0.449 C B = 0.483 C B = 0.470 C B = 0.440 C B = 0.454 C B = 0.446 C B = 0.430 C

!

!

!

C = 4.70

!

!

!

Overall

!

!

!

D4 Personalization

C = 4.8

A

C = 0.175 C = 0.198 C = 0.165 C = 0.179 C = 0.174 C = 0.179

!

C = 0.176

!

C = 0.170

!

‘‘POA’’ is abbreviated from priority of alternatives; ’’ ’’ denotes superior to, i.e. A B indicates that WELS A is superior to WELS B . All results obtained by ranking-based method pass the consistency test (a value of consistency ratio less than 0.1). The figures in each row of criteria (C01-C13) are original performance or priority scores; the figures in each row of dimensions (D1-D4), which are calculated on the basis of the local weights of criteria, and in the row of Overall , which are calculated on the basis of the overall weights of criteria, are weighted performance or priority scores. All the weights used in this evaluation are from the column of after adjustment in Table 1. !

!

may be because these dimensions possess fewer criteria. However, it is worth noting that we discover that the learning community is regarded by the learners as having the least relative importance. We also find that the criterion about which learners care the least (ease of discussion with other learners) belongs to this dimension. The reason for this result might be that the course design did not incorporate interacting with members of the learning community as a key component, or may be the functions related to the learning community of the WELS we investigated were not fully perceived by the learners. However, since interactions with the learning community can enhance learning achievement, this study recommends the establishment of certain mechanisms within WELS to promote a higher level of interactive learning behaviors. In this study, we have proposed a multi-criteria methodology for evaluating WELS from the perspective of learner satisfaction. Contrary to many MCDM-related studies, which simply implemented ‘‘expert opinions’’ to establish their hierarchy structure and preference structure, this study has adopted an empirically derived e-learner satisfaction scale as the foundation of building a hierarchy structure. This way, compared with the

903


traditional expert-based approach, contributes significantly to the identification of several sets of decision criteria which have a higher level of reliability and validity in terms of e-learner satisfaction. As for the preference structure, a large-sample survey of WELS learners was conducted to explore the relative importance of the criteria. This user-based empirical effort promises a more valid result, and alleviates the concerns regarding the potential risks of overly relying on experts in MCDM practice. In addition, the results produced by the pool first and pool last methods reveal an identical pattern of learners’ preferences. This convergence indicates the stability of the derived preference structure across the methods. Finally, this methodology accommodates the decision makers with a scheme, including rating-based and ranking-based methods, for the evaluation of alternatives. Our proposed methodology can be visualized as a ‘‘procedural synthesis’’, with its main advantage being its flexibility. Those who are responsible for making the decisions can either follow the entire process or they can customize part of it. For example, they can either use the AHP results exactly as obtained in this study, or carry out a distinct AHP within their organization so as to derive their own preference structure. When evaluating and selecting, they can choose using the rating-based method, the ranking-based method, or both. Such flexibility also signifies that future research efforts are possible to modify or consolidate the procedures in order to acquire variants, so that decision makers can be supplied with a variety of tools to comprehensively evaluate their WELS alternatives, and to make the right decision. 7. Conclusions

The use of the e-learner-satisfaction perspective and a large-sample, learner-based AHP, contribute to adapting the conventional MCDM paradigm to problems that are highly user-oriented. Our methodology supplies management in both education and industry with not only a less complex but also a more appropriate and flexible way to effectively analyze their currently deployed WELS. It can also support their selections of an appropriate WELS product, solution, or module by assessing the alternatives available when their organizations wish to adopt such a technological innovation. At the same time, it allows the technical personnel of WELS vendors (e.g., the analysts or designers) to gain a better understanding of learners’ preferences toward system features before a WELS is implemented, and it also can pinpoint any necessary improvements or enhancements. This allows a higher level of e-learner satisfaction to be achieved, and in the process increases the level of system acceptance and continued use. Acknowledgement

This study was supported by the National Science Council of Taiwan under contract number NSC 94-2416H-018-019. Appendix. An example of question items in AHP questionnaire

When assessing the WELS, the following dimensions (or criteria) will be used. Considering the dimensions (or criteria) listed on the right-hand and the left-hand sides, please indicate the relative importance between them. Dimension

Dimension

(or criterion)

A is more important than B

B is more important than A

A Learner interface

(or criterion) B

9

8

7

6

5

4

3

2

1

2

3

4

5

6

7

8

9 Learning community

The descriptions of each scale: 1: equal importance; 3: weak importance; 5: essential importance; 7: very strong importance; 9: absolute importance; 2, 4, 6, and 8: intermediate values.

904


References Bailey, J. E., & Pearson, S. W. (1983). Development of a tool for measuring and analyzing computer user satisfaction. Management Science, 29(5), 530–545. Buckley, J. J. (1985). Fuzzy hierarchical analysis. Fuzzy Sets and Systems, 17 (3), 233–247. Cashin, W. E., & Downey, R. G. (1992). Using global student rating items for summative evaluation. Journal of Educational Psychology, 84(4), 563–572. Cohen, P. A. (1981). Student ratings of instruction and student achievement. Review of Educational Research, 51(3), 281–309. Collins Cobuild English Dictionary (1999). London, UK: HarperCollins Publishers. Cyert, R. M., & March, J. G. (1963). A behavior theory of the firm. Englewood Cliffs, NJ: Prentice-Hall. Dam, A. V. (2001). User interfaces: disappearing, dissolving, and evolving. Communications of the ACM, 44(3), 50–52. DeLone, W. H., & McLean, E. R. (2003). The DeLone and McLean model of information systems success: a ten-year update. Journal of Management Information Systems, 19(4), 9–30. Doll, W. J., & Torkzadeh, G. (1988). The measure of end-user computing satisfaction. MIS Quarterly, 12(2), 259–274. Fletcher-Flinn, C. M., & Gravatt, B. (1995). The efficacy of computer assisted instruction (CAI): a meta-analysis. Journal of Educational Computing Research, 12(3), 219–241. Forman, E. H. (1989). AHP is intended for more than expected value calculation. Decision Science, 21(3), 670–672. Gelderman, M. (1998). The relation between user satisfaction, usage of information systems and performance. Information and Management, 34(1), 11–18. Hassenzahl, M., Beau, A., & Burmester, M. (2001). Engineering joy. IEEE Software, 18(1), 70–76. Hiltz, S. R., & Turoff, M. (2002). What makes learning networks effective? Communications of the ACM, 45(4), 56–59. Hong, K. S., Lai, K. W., & Holton, D. (2003). Students’ satisfaction and perceived learning with web-based course. Educational Technology and Society, 6 (1), 116–124. Hwang, C. L., & Masud, A. S. M. (1979). Multiple objective decision making: methods and applications. New York: Spring-Verlag. Ives, B., Olson, M. H., & Baroudi, J. J. (1983). The measurement of user information satisfaction. Communications of the ACM, 26 (10), 785–793. Jiang, J. J., Klein, G., Roan, J., & Lin, J. T. M. (2001). IS service performance: self-perceptions and user perceptions. Information & Management, 38(8), 499–506. Jo¨nsson, B.-A. (2005). A case study of successful e-learning: a web-based distance course in medical physics held for school teachers of the upper secondary level. Medical Engineering & Physics, 27 (7), 571–581. Kao, C. (1998). Performance of several nonlinear programming software packages on microcomputers. Computers & Operations Research, 25(10), 807–816. Kaplan-Leiserson, E. (2000). e-Learning glossary, Available from http://www.learningcircuits.org/glossary.html. Karat, J. (1988). Software evaluation methodologies. In M. Helander (Ed.), Handbook of human–computer interaction (pp. 891–903). Amsterdam, Netherland: Elsevier Science. Keeney, R., & Raiffa, H. (1976). Decision with multiple objectives: preference and value tradeoffs. New York: Wiley and Sons. Khalifa, M., & Kwok, R. C.-W. (1999). Remote learning technologies: effectiveness of hypertext and GSS. Decision Support Systems, 26 (3), 195–207. Kumar, R. L., Smith, M. A., & Bannerjee, S. (2004). User interface features influencing overall ease of use and personalization. Information & Management, 41(3), 289–302. Lane, E. F., & Verdini, W. A. (1989). A consistency test for AHP decision makers. Decision Science, 20(3), 575–590. Lindgaard, G., & Dudek, C. (2003). What is this evasive beast we call user satisfaction? Interacting with Computers, 15(3), 429–452. Lin, H.-H., & Wang, Y.-S. (2006). An examination of the determinants of customer loyalty in mobile commerce contexts. Information & Management, 43(3), 271–282. Lin, C. S., Wu, S., & Tsai, R. J. (2005). Integrating perceived playfulness into expectation-confirmation model for web portal context. Information & Management, 42(5), 683–693. Lorenzetti, J. P. (2005). How e-learning is changing higher education: a new look. Distance Education Report, 22(July), 4–7. MacCrimmon, K. R. (1969). Improving the system design and evaluation process by the use of trade-off information: an application to northeast corridor transportation planning. RM-5877-Dot, Santa Monica, CA: The Rand Corporation. Mahmood, M. A., Burn, J. M., Gemoets, L. A., & Jacquez, C. (2000). Variables affecting information technology end-user satisfaction: a meta-analysis of the empirical literature. International Journal of Human–Computer Studies, 52(4), 751–771. Marsh, H. W. (1991). Multidimensional students’ evaluations of teaching effectiveness: a test of alternative higher-order structures. Journal of Educational Psychology, 83(2), 285–296. Melone, N. P. (1990). A theoretical assessment of the user-satisfaction construct in information systems research. Management Science, 36 (1), 76–91. Piccoli, G., Ahmad, R., & Ives, B. (2001). Web-based virtual learning environments: a research framework and a preliminary assessment of effectiveness in basic IT skills training. MIS Quarterly, 25(4), 401–426. Raymond, L. (1987). Validating and applying user satisfaction as a measure of MIS success in small organization. Information and Management, 12(4), 173–179. Rosenberg, M. J. (2001). E-learning: Strategies for delivery knowledge in the digital age. New York: McGraw-Hill. Saaty, T. L. (1980). The analytic hierarchy process. New York: McGraw-Hill. Smith, C. U., & Williams, L. G. (1999). A performance model interchange format. Journal of Systems and Software, 49(1), 63–80.


905

Teng, J.-Y. (2002). Project evaluation: method and applications. Keelung, Taiwan: National Taiwan Ocean University. Wang, Y.-S. (2003). Assessment of learner satisfaction with asynchronous electronic learning systems. Information & Management, 41(1), 75–86. Whitten, J. L., Bentley, L. D., & Dittman, K. C. (2004). System analysis & design methods. New York: McGraw-Hill. Yang, G. S., Wen, C. Y., Wu, C. S., & Li, Y. Y. (2001). Social and behavioral science research methods. Taipei, Taiwan: Bookcake Publishing. Yu, P.-L. (1985). Behavior and decision. Taipei, Taiwan: Sinica.

Multi-Criteria Evaluation of the Web-based E-learning System

Recommend Documents