Information & Management 33 (1998) 251±262
Research
An empirical assessment of a multimedia executive support system Albert H. Huanga,*, John C. Windsorb a
Department of Business Administration, Division of Business and Economics, University of Hawaii ± Hilo, Hilo, Hawaii 96720, USA b Department of Business Computer Information Systems, College of Business Administration, University of North Texas, Denton, Texas 76203, USA Received 10 December 1997; accepted 1 January 1998
Abstract An experiment was conducted to assess the use of `Multimedia' in ESS (executive support system) and its potential impact on the effectiveness of information analysis. Data was collected from 40 managers and/or professional employees in three business organizations. The primary task was to analyze information, stored in an ESS prototype, in order to identify some possible threats and opportunities. The results of the study suggest that multimedia may not be an appropriate presentation format for analytical tasks. Managers and professional employees who used a multimedia ESS identi®ed fewer threats and opportunities than those who used a text-based system. Contrary to the results of several previous studies, the use of multimedia did not improve information retention. During the post-experiment interview, subjects indicated that the use of sound annotations had adversely affected their ability to analyze information. They also suggested that some multimedia components, such as animated charts or animated text blocks, were unnecessary. # 1998 Elsevier Science B.V. All rights reserved Keywords: Multimedia; Management information systems; Executive support systems; Executive information systems; User interface; Presentation format; Animation; Sound annotation; Information analysis; Experimental design
1. Introduction Many of the decisions that executives make require information displayed in a wide variety of presentation formats [30]. Ideally, different types of information presentation should be used to facilitate different decision-making tasks. However, most existing empirical studies on information presentation focus only on visual formats, such as text, tables, and graphics. Little attention is paid to non-traditional presentation formats, such as multimedia. *Corresponding author. Tel.: (808) 974-7554; e-mail:
[email protected] 0378-7206/98/$19.00 # 1998 Elsevier Science B.V. All rights reserved PII : S-0378-7206(98)00032-9
Multimedia presentation is one of the fastest-growing sectors of the computer industry. Applications have appeared in many areas, such as training, education, business presentation, merchandising, and communications. In the corporate world, multimedia aspects receive a great deal of attention, especially in the area of job training and employee education [11, 25]. It is believed that multimedia will soon become an indispensable tool for enhancing corporate productivity [14]. However, the development of multimedia applications is progressing much faster than the supporting research. Although many existing studies have investigated the potential bene®ts of multimedia applica-
252
A.H. Huang, J.C. Windsor / Information & Management 33 (1998) 251±262
tions in training and educational environments [1, 40], most of these efforts focused on learning effectiveness, such as retention of information and learning time. Few studies assess the impact of multimedia in a business environment. A knowledge of the effect of multimedia on information analysis and its outcomes is essential. The goal of this research is to determine whether multimedia is a useful addition to ESS for decision makers in performing information analysis. Its effectiveness is measured by the number of threats and opportunities identi®ed for a business event. Multimedia is de®ned as the combination of text, graphics, animated charts, animated text blocks, and sound annotations. Unlike visual presentation formats, such as graphics and tables, multimedia is unique to computer information systems (IS). Paper-based IS cannot simultaneously and interactively display text, graphic, animation, and sound. Only computerized systems can do this automatically. Unlike most existing studies of multimedia conducted in educational settings, e.g. [36], this study was carried out in a business environment using business professionals as subjects. The experimental task was a familiar one performed by most business managers and executives. Most existing studies of information presentation formats use decision outcomes as the dependent variable; this study uses effectiveness. 2. Prior research Multimedia is a format of information presentation. Information presentation formats refer to the way in which information is conveyed to users. Presenting it in graphic, table, or other formats to satisfy both decision style and task requirement is of critical interest to designers of decision support systems (DSS). In many studies, researchers altered information presentation formats and observed their impact on outcomes. They assumed that presentation formats were critical in providing information for making decisions [28, 54] and have investigated the effects of tables, bar charts, line charts, computer plots, and three-dimensional graphics [3, 12, 22]. Other studies indicated that the use of graphics was advantageous [9, 29, 40, 44, 53].
Watson and Driver [49] suggested several possible explanations for the superiority of graphical information presentation. First, graphical information is more interesting. Humans are naturally more attracted to graphics than tables. Second, they are better able to illustrate the relationships between variables. Decision makers can clearly see the relationships between multiple factors. Third, graphics simplify the interpretation and communication of information and save decision makers time. Fourth, a graph can incorporate multiple variables. It offers decision makers a more comprehensive view of a problem. Finally, graphics can demonstrate trends and patterns. They help decision makers see hidden facts and stimulate creative or analytical thinking. It should be noted that some studies suggest that the use of graphical presentation modes is not always bene®cial. For example, Lucas and Nielsen [27] found no solid support for hypotheses relating to the impact of additional information or graphical presentation of information. In another study, Lucas [26] found limited support for the use of graphics presentation for decision-making tasks. Watson and Driver suggested that three-dimensional graphics did not result in greater recall of information than did a tabular presentation for an assigned recall task. To explain such inconsistency among various studies of information presentation formats, Benbasat et al. [5] pointed out: ``The dramatic claims made about the bene®ts of colors and graphics in the professional literature appear to be somewhat exaggerated and are not well substantiated by empirical evidence; research ®ndings are inconclusive and often in con¯ict with each other; and as a result, there does not appear to be a well-grounded empirical or theoretical base in the information systems and administrative sciences ®eld to guide practitioners who are designing presentation formats.'' Many people believe that the effectiveness of a speci®c presentation format depends on task characteristics, such as complexity [19]. For instance, Jarvenpaa and Dickson [21] suggest that graphics are more effective than tables when the task requires a large volume of data. Graphics are also more effective for tasks that require decision makers to identify patterns [15].
A.H. Huang, J.C. Windsor / Information & Management 33 (1998) 251±262
Another important variable for research of presentation formats is color. Its use becomes more important as the cost of using color graphics decreases. With inexpensive `Video Graphic Array' (VGA) or Super VGA color monitors, colors are inexpensive. Gremillion and Jenkins [16] studied the effect of color-coded transparencies on information retention. They used color to emphasize distinctions and relationships among information items and found a signi®cant improvement in recall performance with color. Previous studies have also shown that it can be used to group information in sets to speed up the location of speci®c items [7, 8, 10] and to improve comprehension of information [24]. Benbasat et al. [4] studied the effect of color on decision-making effectiveness in a series of laboratory experiments. They concluded that, to some extent, color improves decision-making quality. Also, color was more bene®cial for graphics than for tables. The impact was more signi®cant on the performance of ®eld-dependent than ®eld-independent individuals, and when time was constrained [6]. Due to the advancement in computer technology, multimedia has become an important alternate presentation format. It is usually considered to be an application that combines a rich mix of text, graphics, pictures, sound, and animation [51]. Traditionally, the goal of multimedia systems is to present information so that it is bene®cial to learning and long-term retention [17]. Thus, research studies of multimedia also focus on learning effectiveness and information retention. For example, a United States Department of Defense (DOD) study found that multimedia training was 40% more effective than traditional training methods; it also had a 30% greater retention rate and a 30% shorter learning curve. The results of this study also indicated that multimedia was generally more effective than conventional training in terms of subjects' knowledge, performance, and retention [41]. The effect of multimedia was also found to be not industry speci®c. General Electric Aerospace successfully used multimedia-training tools to enhance retention, promote better use of information, and foster the development of more capable employees [42]. A sporting-goods retailer used a multimedia system to train in-store sales representatives and cashiers, and dropped the training time from an average of 8 h to between 2-to-4 h [52]. A major telecommunication company reported that the use of multimedia as a
253
training tool reduced the training time of a training program by 80% and increased the retention level by 40% [35]. A study that investigated the effectiveness of multimedia learning in classrooms reported that students using multimedia to learn class material did better in the posttest. The superiority of multimedia lies in that the multimedia applications provide a variety of input and output modalities to the users, therefore, motivation and stimulation are greater, and thus, learning is enhanced [17]. One study investigated the ability of multimedia presentations to persuade [34]. The study varied presentation formats and measured the attitude change of message recipients. The two presentation formats were a printed brochure and a multimedia computer kiosk. Apparently, the multimedia kiosk was more persuasive than the printed brochure: the multimedia presentation had a positive effect on attitude change; the vividness factor of the presentations was also important. The combination of textual and graphical presentation is common for most computer systems. However, the addition of high-quality sound is a signi®cant improvement. Traditional computer systems can only generate simple sound effects such as the typewriter bell, but a high-speed sound card and related software makes the presentation of high-quality music and speech possible. Yet, one must be careful when using sound and speech with text due to the limitation of human information processing capacity. There are two important issues to be considered when using sound. First, it is public. If the volume is high, others near-by will hear it, causing lost of privacy and creating sound pollution. Although users can use headphones, it may not be practical in an of®ce environment. Wearing headphones prohibits users from hearing other information, such as important conversations in the of®ce. The same is true for speech input. Second, auditory information must be presented serially, and this places a heavier burden on a user's short-term memory. A study of telephone voice response system showed that listeners have dif®culty in remembering overlapping spoken menus [23]. Hapeshi and Jones suggest that auditory computer systems should use visual stimuli to provide a context to aid the intelligibility of speech. Sayeed and Brightman [38] studied the effect of computerized information systems on managerial pro-
254
A.H. Huang, J.C. Windsor / Information & Management 33 (1998) 251±262
blem ®nding. They found that managers who used computerized information systems found potential problems and opportunities in a more timely fashion and used information from more sources. However, the use of computer information systems did not help managers produce more accurate problems and opportunities. 3. Theoretical background Two theories derived from existing studies form the theoretical foundation of this study. These suggest possible relationships between the use of multimedia and the effectiveness of information analysis. Theory I combines the ®ndings of Hayth-Roth and Walker [18], etc. It suggests that a potential positive relationship exists between the use of multimedia and the effectiveness of information analysis. Theory II is the extension of the cognitive ®t theory [47], which implies a negative relationship between the use of multimedia and the effectiveness of information analysis, if multimedia is deemed as a less analytical presentation format. 3.1. Theory I The ability to consolidate information from various information sources and to detect logical implication of integrated information is the key to the human learning process. In the business environment, this is essential for managers and executives; they must detect potential threats and opportunities based on information from many sources. Recognition of logical connections or implications of temporally and spatially separated facts require them to be placed together and related according to certain rules. Therefore, business managers and executives need to be able to relate facts presented to them at different times, different places, with different media, in different formats to detect new ideas and problems. Hayth-Roth and Walker studied the relationship between memory and ability to derive logical inferences. They showed that recognizing and con®guring facts to support hypothesized inferences is very dif®cult unless the facts have been committed to memory. Simply becoming familiar with relevant information is inadequate. They believe people are not capable of
locating logical con®gurations of external facts that support hypothesized inferences. Automated memory mechanisms acquire information so that the con®gural information is directly accessible. Their ®ndings have important implications for research on information presentation formats. Early MIS literature suggested that the purpose of information systems was to support decision making [31]. The common purpose of many studies has been to look for ways to improve information presentation formats so that users can make better decisions quicker. However, due to the lack of understanding, many studies arbitrarily manipulate presentation formats without a sound theoretical foundation to justify the research design. Such studies generally fail to provide an explanation due to a lack of supporting theory [20]. Another problem is the dif®culty of measuring decision quality [45]. Because there are no practical ways to measure the quality of real business decisions, researchers often focus on surrogate measures, such as user satisfaction or system usage [13, 32]. Since there is no strong evidence that user satisfaction or system usage is directly related to decision quality, a strong link between a particular information presentation format and better decisions have not been established. Considering Hayth-Roth and Walker's ®ndings, decision makers may make better logical inferencing if they can commit more information to memory. Thus, researchers could focus on enhancing memory and retention of information, and leave the decisionmaking processes and decision outcomes to the decision makers themselves. Several studies suggest that the use of multimedia improves information retention, e.g. [2]. Thus, the use of multimedia in ESS should enable decision makers to remember more information so as to have better logical inferencing. Therefore, there is potentially a positive relationship between the use of multimedia and the effectiveness of information analysis. 3.2. Theory II Theory II is based on cognitive ®t theory, which argues that the problem representation and any tool or aid employed should support the strategies and methods of processes required to perform that task in order to be effective [46]. Here, the problem-solving space
A.H. Huang, J.C. Windsor / Information & Management 33 (1998) 251±262
has three components: the problem representation; the problem-solving task; and the mental representation. The problem solving effectiveness is the outcome of the relationship between the problem representation and problem-solving task. The problem solver acts on information in the problem representation and the problem-solving task to produce the mental representation in the user's memory. The mental representation then produces the problem solution. When the types of information in the problem-solving representation and the problem-solving task are the same, the problem solver can use processes that match the information type to solve the problem. There is a ®t among all three components: the problem-solving process will be more ef®cient and accurate because the problem solver does not have to transform the information in the problem representation into a mental representation for the task. For instance, if the task requires the acquisition of spatial information, a spatial problemrepresentation would be likely to simplify the problem-solving process. Intuitively, multimedia presentation formats are less analytical than numerical tables. In other words, there is a mismatch between a multimedia presentation format and an analytical task. Traditional analytical presentation formats, such as aggregated tables, are more appropriate for such a task. Therefore, there is potentially a negative relationship between the use of multimedia and the effectiveness of information analysis
4. Research method The two primary research questions of this study are: Q1: Does the use of visual multimedia affect the subjects' ability to analyze information? Q2: Does the use of audio-visual multimedia affect the subjects' ability to analyze information? These two translate to three sets of statistical hypotheses: H1: There is no difference between the numbers of threats and opportunities identi®ed by subjects who use the visual multimedia ESS and subjects who use the text ESS.
255
H2: There is no difference between the numbers of threats and opportunities identi®ed by subjects who use the audio-visual multimedia ESS and subjects who use the text ESS. H3: There is no difference between the numbers of threats and opportunities identi®ed by subjects who use the audio-visual multimedia ESS and subjects who use the visual multimedia ESS. In order to validate Theory I, retention of critical business information is also measured using a multiple-choice questionnaire. Three additional hypotheses to be tested are: H4: There is no difference between the retention performances of subjects who use the visual multimedia ESS and subjects who use the text ESS. H5: There is no difference between the retention performances of subjects who use the audio-visual multimedia ESS and subjects who use the text ESS. H6: There is no difference between the retention performances of subjects who use the audio-visual multimedia ESS and subjects who use the visual multimedia ESS. 4.1. Research variables The independent variable of this study is the type of presentation format. The surrogates are three custom built ESS prototypes called: A; B; and C. These store the same set of information regarding a business proposal. Prototype-A is text-based: it presents information using only conventional text and table formats. Prototype B and C both are multimedia systems. Prototype-B presents information using visual multimedia, including text, graphics, animated charts, and animated text blocks. Prototype-C presents information using all formats used in Prototype-B plus sound annotations. Two different versions of multimedia ESS are used for a reason. In most studies of multimedia, the relative effectiveness of auditory and visual presentation has been ignored. It is unclear whether the effect of multimedia is a result of visual presentation, auditory presentation, or their interaction. By having two versions of multimedia systems, we hope to provide some empirical evidence to distinguish the relative effectiveness of visual and auditory presentations.
256
A.H. Huang, J.C. Windsor / Information & Management 33 (1998) 251±262
The dependent variable of this study is the effectiveness of the information analysis. The surrogate for this is the number of threats and opportunities identi®ed. Helping in the identi®cation of threats and opportunities is generally regarded as one of the primary purposes of ESS. 4.2. Experimental design The experiment used a three-group randomized design. Group A was the comparison group. Subjects in Group A used the text version of the ESS prototype. Groups B and C were the experimental groups. Subjects in Groups B and C used the visual multimedia and audio-visual multimedia ESS prototypes, respectively. All three groups performed the same task ± analyzing the information regarding a business proposal and identifying potential threats and opportunities based on that information. 4.3. Instrument design and validation The three ESS prototypes used in the experiment contain information about a business proposal regarding an expansion plan of a restaurant chain. This scenario is selected for the experiment, because most people are more familiar with the restaurant business than with other businesses. All three ESS prototypes use the same hypermedia structure to store the information. Each ESS contains three segments: External Information System; Internal Information System; and E-mail system. Fig. 1 shows the common main menu of the three systems. The External Information System contains information on the business environment, including vital statistics, crime statistics, and industry trends. The Internal Information System contains information about the locations of the existing restaurants, customer pro®les, current policies, and information about the expansion plan. The E-mail system contains several important messages regarding the proposal. Users must use the mouse to click on buttons, hot words, or pictures to access the information, in what is commonly known as the drill-down process, a method of obtaining information in many information systems. In both the pilot and the actual experiments, all subjects indicated that they were familiar with the use of the mouse and the drill-down process.
Fig. 1. A common submenu screen of the ESS.
The Multimedia ToolBook 3.0 authoring software was used to construct the ESS prototypes. This software uses a three-layer structure for its applications. The ®rst layer is a Book, which contains all necessary elements to make a complete presentation of information. The second layer is a Page, which is usually one screen of information; it may contain several screens if other screens are embedded as viewer windows within it. The third layer is an Object, which can be a button, a line, a polygon, a text ®eld, a picture, a bitmap graphic, a menu, a video clip, a sound clip, an animation clip, etc; these are the fundamental elements of a ToolBook application. The text version of the ESS prototype primarily uses text ®elds and tables to present information. Text ®elds contain non-numerical information such as descriptions of events, mission statements, and E-mail messages, while tables display vital statistics, business statistics, ®nancial and operational data, and other numerical data. Fig. 2 is a sample screen of this version. The visual and audio-visual ESS use more presentation formats to present information. They contain several types of bar charts and pie charts. They also use graphics to illustrate buildings, people, houses, automobiles, and maps. These graphics are used to enhance the presentation of both numerical and nonnumerical data. All graphics in the ESS prototypes are directly related to the accompanied textual information. There are no irrelevant or unnecessary graphics in the system.
A.H. Huang, J.C. Windsor / Information & Management 33 (1998) 251±262
257
Fig. 2. A sample screen of the text-based ESS.
Fig. 3. A sample screen of the visual and audio-visual ESS.
Both the visual and audio-visual ESS prototypes use animation to assist the presentation of textual and graphical information. Animated text blocks and charts enable users to receive information in smaller chunks incrementally. For example, an animated bar chart shows one bar at a time, instead of showing all bars at once. An animated text block shows one sentence at a time, instead of showing an entire paragraph. This technique is intended to help reduce information load and make sure that users receive the complete set of information. It also allows users to control the speed at which the information is displayed. Every time the user clicks on the mouse button, the system presents one information item. Fig. 3 is a sample screen of the visual and the audio-visual ESS. The audio-visual ESS has sound annotations of the information contained in the screens. The audio-visual version of the ESS prototype includes sound annotations consisting of pre-recorded human voices equivalent to the numerical information. The human voice used in the system was ®rm and clear; users can easily hear and understand it. The technical speci®cation of the recording is mono, 8 bit/s, and 11 kHz. This provides satisfactory sound quality while maintaining good response time and using relatively little hard disk storage. The sound clips are played automatically. Users can also click on the `Click #' buttons (see Fig. 3) to replay the sound. Steps were taken to make sure that the three systems contain the same information. First, the three
systems were built in a speci®c order. The textbased ESS was built ®rst, based on a proposal of a new restaurant. The visual ESS was built next, based on the completed text-based system. Thus, the information contained in the visual system could not be more than the information contained in the text-based system. The only addition to the visual system was graphical representations of the same information contained in the text system. The audio-visual ESS was built last, based on the completed visual system. The only addition to the audio-visual system was sound annotations of the same information contained in the other two systems. The three completed systems were reviewed by an expert panel and revised several times until the panel agreed that the three systems contain the same information. A close-ended questionnaire that consisted of 19 multiple-choice questions was used to measure subjects' retention of critical business information. Each question asked subjects to select an item based on the information they learned from the system. In order to make sure that the questionnaire actually measured retention, it was pre-tested on 30 college students. The split-half reliability test was used, because it required only one administration of the questionnaire. The resulting reliability index of the retention questionnaire was 0.78. This value indicated that 78 percent of the variation in the results were the true variance of retention. The other 22 percent were random error.
258
A.H. Huang, J.C. Windsor / Information & Management 33 (1998) 251±262
4.4. Subject Since the study dealt with the effectiveness of multimedia on information analysis in a business environment, the quali®cation of subjects was important. Desirable characteristics of subjects were business and managerial experiences. Thus, desirable subjects for this study are managers or professional employees who have adequate information scanning, analysis, and/or decision-making experience. The three business organizations participated in the study were Lockheed-Martin Tactical Aircraft Systems (LMTAS), American Airlines (AA), and Baylor Hospital (BH). LMTAS, in Fort Worth ± Texas, is a subsidiary of the Lockheed-Martin (LM). Lockheed is a company who has been using advanced executive information systems since the 1980s [48, 50]. The site of the data collection currently employs more than 12 000 employees and provides an ideal source of subjects. AA is one of the largest commercial airlines in the world. The data collection site was an AA facility in Irvin ± Texas, which also has a large number of suitable subjects. The BH is a major regional hospital in the Dallas Fort Worth metropolitan area. The site of data collection is a BH Home Care facility in Grapevine, Texas. Multiple sites were chosen to increase the generalizability of the results by reducing the potential effects of organization-speci®c issues. All subjects from the organizations were treated equally during the group assignment process. Factors such as culture, type of business, technology, and other organizational characteristics are irrelevant here. Participation was voluntary. Subjects neither received cash rewards nor were they coerced by company management. The primary incentive was an opportunity to see a multimedia implementation of business systems. This incentive is generally more effective than a cash payment. Many subjects participated because they wanted to contribute to scienti®c research. Since the experimental task involves auditory information and color graphics, visually challenged (color-blind) and hearing-impaired individuals did not participate. A total of 40 subjects participated. Regardless of their organizations, all 40 subjects were randomly
assigned to treatment groups. The experiment ended when the sample size of each treatment group was more than ten. The ®nal treatment group sizes were 15, 15, and 10 for the text group, visual multimedia group, and audio-visual multimedia group, respectively. It is possible to assume a normal approximation when there are more than ten subjects in each group. 4.5. Procedure The investigator randomly assigned each subject to one of the three groups, irrespective of subjects' organizations, genders, backgrounds, etc. Subjects reviewed the assigned system and then ®lled out the two questionnaires. Identi®ed threats and opportunities were measured via an open-ended questionnaire. This asked subjects to list potential opportunities and threats they identi®ed by using the system, giving a brief reason and providing supporting information to justify the comment. Retention of information was measured by the multiple-choice questionnaire. At the end of the experiment, the investigator interviewed each subject to collect information about their perception of the pros and cons of the system they used. This retention questionnaire was easy to analyze. The scores were simply the count of correct answers. The open-ended questionnaire was more dif®cult to evaluate. In order to avoid grader bias, two reviewers independently counted the numbers of valid threats and opportunities identi®ed by each subject and then compared the results. The validity of the threats and opportunities was based on the supporting information and reasoning. For each potential threat or opportunity identi®ed, as long as the supporting information was correct and the reasoning logical, the subject was assigned one point. 5. Results The results are summarized in Table 1 and Fig. 4. The results indicate that subjects who used the textbased ESS performed signi®cantly better than subjects who used the audio-visual multimedia ESS. This contradicts to the results of several previous studies. Also, there was no signi®cant improvement in reten-
A.H. Huang, J.C. Windsor / Information & Management 33 (1998) 251±262
259
Table 1 Summarized results
Average number of threats and opportunities identified by subjects Standard deviation of threats and opportunities identified Average retention score of subjects Standard deviation of retention score
Text group
Visual multimedia group
Audio-visual multimedia group
9.6 3.9 13 2.4
8.5 3.7 13.3 3.1
7.3 4.0 12.4 2.9
Fig. 4. Statistical signi®cance of the differences between groups.
tion of critical business information due to the use of visual or audio-visual multimedia. The recall performance scores of the three different groups were almost identical. 6. Discussion of findings Many previous studies of multimedia suggest that the use of multimedia improves learning effectiveness or recall performance [39, 43]. However, the results of this study showed that the average recall performance scores of the three groups were almost identical. There was no apparent improvement in the retention of information due to the use of multimedia. There may be several reasons for the difference between our ®ndings and previous ones. First, the instrument is different. Most previous studies compare multimedia systems with paper-based reports or human instructions. Here, all three groups receive information from computer-based systems. Second, the design of the multimedia system is different from those in other studies. For example, there are at least four different ways to combine speech and image in a multimedia system. Third, the task of previous studies was to learn as much information as possible. In contrast, the task of
this study was to analyze the information and identify threats and opportunities. The primary dependent variable was the effectiveness of information analysis, measured by the number of identi®ed threats and opportunities. The analysis shows that the text group performed better than the visual multimedia group. The visual multimedia group performed better than the audio-visual multimedia group. The difference between the text group and the audio-visual multimedia group is statistically signi®cant. Thus, the use of multimedia does not help decision makers in identifying threats and opportunities. Also the addition of graphics, animated charts, and sound may hinder their ability to analyze information. This con®rmed Theory II. According to the cognitive ®t theory, an analytical presentation format is best for solving analytical problems. The text version ESS has the most analytical presentation format. It contains aggregated tables of important ®gures. In contrast, the others contain animation and graphics, which are more holistic. The text group might be presumed to have performed well because there was a match between the type of task and the presentation format. Previous research has shown that irrelevant background speech can impair retention of information [37]. According to the post-test interviews, subjects
260
A.H. Huang, J.C. Windsor / Information & Management 33 (1998) 251±262
believed that speech was a disturbing factor when they tried to concentrate. Almost without exception, subjects wished there was a way to turn off the sound when they spotted critical information items, even though the speech actually contained meaningful information. Most subjects would like to have sound annotations for introductory information, but not for complicated or numerical information. They also mentioned that sound annotations should be played only once, and not be played when information is received a second time. Subjects mentioned two reasons for this. First, they read much faster than they can hear. Second, if they are familiar with the information, they can analyze it immediately. For most subjects, analyzing visual information is much easier than analyzing auditory information. Visual information is available for inspection over a longer period. A user has more control over the pace of the learning process and a greater opportunity to commit the material to memory. There are also fewer burdens on the short-term memory of decision makers [33]. In contrast, auditory information is temporary and sequential. Reviewing a portion of a long speech many times is inef®cient. As a vice-president level subject pointed out, time-series information is useful for indicating trends and patterns. Individual numbers without reference points are meaningless and therefore visualizing trends and patterns from auditory information is dif®cult. 7. Limitations This study has several limitations. First, the ESS prototypes were custom-built systems specially designed for the experiment. Other ESS with different designs may have different results. Second, this study was conducted in a laboratory that was quiet and free from outside interruptions. The setting was different from an actual business environment where decisionmakers were under constant pressure and subject to interruptions from many sources. These differences may reduce the external validity of the results. Third, the sample size was small due to the dif®culty of obtaining quali®ed subjects to participate. Due to these limitations, the results of this study should not be unconditionally applied to other multimedia systems.
8. Conclusion The initial hypothesis was that multimedia could affect decision makers' ability to identify threats and opportunities of a business situation. An experiment conducted in three organizations supplied the data to test this hypothesis. The experimental task was to identify threats and opportunities of a business proposal presented in three different ESS systems. Participants included 40 business managers or professionals with decision-making experiences or managerial responsibilities. The ®ndings did not suggest any causality between multimedia and the ability to identify threats and opportunities. They simply support the cognitive-®t theory and indicate that multimedia is not an appropriate presentation format for an analytical task. The use of multimedia does not necessarily improve learning and retention of information. Organizations should be more cautious in the use of multimedia technology. Although multimedia is one of the most advanced presentation formats available to users, it may not be bene®cial for all applications. Inappropriate use of this technology does not just waste valuable resources, rather it creates negative impacts on user performance. Organizations should carefully evaluate the pros and cons of the multimedia technology before committing signi®cant resources to it.
References [1] R.W. Amos, A Comparative Analysis of Hypermedia-Aided Instruction Versus Traditional Instruction for Statistical Process Control, Ph.D. diss., The University of Alabama in Huntsville, 1992. [2] C. Bayard-White, Multimedia Notes, London, Chrysalis Interactive Service Ltd., 1990. [3] J. Benbasat, R. Schroeder, An experimental investigation of some MIS variables, MIS Quarterly 1(1), 1977, pp. 37±49. [4] I. Benbasat, A. Dexter, P. Todd, The in¯uence of color and graphical information presentation in a managerial decision simulation, Human-Computer Interaction 2, 1986, pp. 65±92. [5] I. Benbasat, A. Dexter, P. Todd, An experimental program investigating color-enhanced and graphical information presentation: An integration of ®ndings, Communications of the ACM 29(11), 1986, pp. 1094±1105.
A.H. Huang, J.C. Windsor / Information & Management 33 (1998) 251±262 [6] I. Benbasat, A. Dexter, An experimental evaluation of graphical and color-enhanced information presentation, Management Science 31(11), 1985, pp. 1348±1364. [7] M. Cahill, R.C. Carter, Color code size for searching displays of different density, Human Factors 18(3), 1976, pp. 273±280. [8] R.C. Carter, Search time with color display: Analysis of distribution functions, Human Factors (1982) 203±212. [9] N. Chervany, G. Dickson, K.A. Kozar, An Experimental Gaming Framework for Investigating the In¯uence of Management Information Systems on Decision Effectiveness, Working Paper, Management Information Research Center, University of Minnesota, 1972. [10] R.E. Christ, The effect of extended practice on the evaluation of visual display codes, Human Factors 25(1), 1983, pp. 71±84. [11] H. Clancy, The new celluloid heroes, Computer Reseller News 22, 1994, pp. S8±S11. [12] G. DeSanctis, S. Jarvenpaa, An investigation of the table versus graphs controversy in a learning environment, Proceedings of the Sixth International Conference on Information Systems, Chicago 1985, by Society of Information Management, 1979, pp. 134±144. [13] W. Doll, J. Torkzadeh, The measurement of end-user computing satisfaction, MIS Quarterly, (1988) 259±273. [14] W. Eckerson, Managers look to multimedia future, Network World 9(35), 1992, pp. 25±31. [15] J.H. Gerlach, F. Kuo, Understanding human-computer interaction for information systems design, MIS Quarterly 15(4), 1991, pp. 526±549. [16] L.L. Gremillion, A.M. Jenkins, The effects of color enhanced information presentation, Proceedings of the Second International Conference on Information Systems, Cambridge. Mass., December 1981, by the Society of Information Management, 1981, 121±134. [17] K. Hapeshi, D. Jones, Interactive multimedia for instruction: A cognitive analysis of the role of audition and vision, International Journal of Human-Computer Interaction 4(1), 1992, pp. 79±99. [18] B. Hayth-Roth, C. Walker, Con®gural effects in human memory: The superiority of memory over external information sources as a basis for inference veri®cation, Cognitive Science 3, 1979, pp. 119±140. [19] I.H.M. Hwang, B.J.P. Wu, The effectiveness of computer graphics for decision support: A meta-analytical integration of research ®ndings, Database, (1990) 11±20. [20] S.L. Jarvenpaa, Methodological issues in experimental IS research: Experiences and recommendations, MIS Quarterly, (1985) 141±156. [21] S.L. Jarvenpaa, G.W. Dickson, Graphics and managerial decision making: Research-based guidelines, Communications of ACM 31(6), 1988, pp. 764±774. [22] P.G.W. Keen, M.S. Scott Morton, Decision Support Systems: An Organizational Approach. Massachusetts, AddisonWesley, 1978. [23] A.L. Kidd, Problems of man-machine dialog design, Proceedings of the Sixth International Conference on Computer Communication, (1982) 531±536.
261
[24] R.J. Lamberski, F.M. Dwyer, The instructional effect of coding color and black and white on information acquisition and retrieval, ECTJ 31(1), 1983, pp. 9±21. [25] P. Langley, C. Porter, The multimedia way to teach HR, Personnel Management 26(9), 1994, pp. 38±41. [26] H.C. Lucas, Jr., An experimental investigation of the use of computer-based graphics in decision making, Management Science 27(7), 1981, pp. 757±768. [27] H.C. Lucas, Jr., N.R. Nielsen, The impact of information presentation on learning and performance, Management Science 26(10), 1980, pp. 982±993. [28] R.D. Mason, Ian I. Mitroff, A program for research on management information systems, Management Science 19(5), 1973, pp. 475±487. [29] I.M. Miller, Computer graphics for decision making, Harvard Business Review 47(6), 1969, pp. 121±132. [30] H. Mintzberg, The Nature of Managerial Work, Englewood Cliffs, Prentice-Hall, 1973. [31] R.L. Nolan, J. Wetherbe, Toward a comprehensive framework for MIS research, MIS Quarterly, (1980) 1±19. [32] J.H. Nord, G.D. Nord, Executive information systems: A study and comparative analysis, Information and Management 29(2), 1995, pp. 95±106. [33] D.A. Norman, Memory and Attention: An Introduction to Human Information Processing, New York, Wiley, 1968. [34] L.L. Ottinger, Understanding the Effectiveness of Multimedia Technology as a Persuasive Tool: An Experimental Investigation, Ph.D. diss., Texas A and M University, 1993. [35] E. Oz, L. White, Multimedia for better training, Journal of Systems Management, (1993) 34±43. [36] S. Reisman, A comparative study of multimedia personal computing and traditional instruction in business school curriculum, Information Resource Management Journal 6(4), 1993, pp. 15±21. [37] P. Salame, A.D. Baddeley, Disruption of short-term memory by irrelevant speech: Implications for the structure of working memory, Journal of Verbal Learning and Verbal Behavior 21, 1982, pp. 150±164. [38] L. Sayeed, H.J. Brightman, Can information technology improve managerial problem ®nding?, Information and Management 27(6), 1994, pp. 377±390. [39] N. Schwartz, R.W. Kulhavy, Map structure and the comprehension of prose, Education and Psychology Research 7, 1987, pp. 113±128. [40] K. Shostack, C. Eddy, Management by computer graphics, Harvard Business Review 49(6), 1971, pp. 52±63. [41] R.E. Shulman, Multimedia a high-tech solution to the industry's training malaise, Supermarket Business, 47(4) (1992) 23±24, 77. [42] J.C. Sipior, J. Townsend, A case study of general electric's multimedia training systems, Information Resources Management Journal 6(4), 1993, pp. 23±31. [43] Z. Stoneman, G.H. Brody, Immediate and long-term recognition and generalization of advertised products as function of age and presentation mode, Developmental Psychology 19, 1983, pp. 56±61.
262
A.H. Huang, J.C. Windsor / Information & Management 33 (1998) 251±262
[44] H. Takeuchi, A.H. Schmidt, New promise of computer graphics, Harvard Business Review 58(1), 1980, pp. 122±131. [45] P. Todd, I. Benbasat, Process tracing method in decision support systems research: Exploring the black box, MIS Quarterly, (1987) 493±512. [46] N. Umanath, I. Vessey, Multiattribute data presentation and human judgment: A cognitive ®t perspective, Decision Sciences 25(5/6), 1994, pp. 795±824. [47] I. Vessey, Cognitive ®t: A theory-based analysis for the graph versus graphics literature, Decision Sciences 22(2), 1991, pp. 219±241. [48] H. Watson, How to ®t an EIS into a competitive context, Information Strategy: The Executive's Journal 8(2), 1992, pp. 5±10. [49] C.J. Watson, R.W. Driver, The in¯uence of computer graphics on the recall of information, MIS Quarterly, (1983) 45±53. [50] H. Watson, M. Frolick, Determining information requirements for an EIS, MIS Quarterly 17(3), 1993, pp. 255±269. [51] L. Wigins, M. Shiffer, Planning with hypermedia: Combining text, graphics sounds, and video, Journal of American Planning Association 56(2), 1990, pp. 226±235. [52] C. Wilder, Multimedia training: For good sports, Computerworld 4, 1992, pp. 35. [53] R.W. Zmud, An empirical investigation of the dimensionally of the concept of information, Decision Science 9(2), 1978, pp. 187±195. [54] R.W. Zmud, Individual differences and MIS success: A review of empirical literature, Management Science 25(10), 1979, pp. 966±979.
Albert H. Huang is currently an Assistant Professor of MIS at the University of Hawaii, Hilo. He received his Ph.D. degreee in Business Computer Information Systems from the University of North Texas in 1996. He is also a Certified Novell Engineer and Administrator. His current research interests include human-computer interaction, telecommunications management, electronic commerce, and distance training. In addition to Information & Management, Dr. Huang has published in several other journals, including the Journal of Applied Management Studies, Information Systems Management, Journal of Educational Technology Systems, and Journal of Education for Business. He has also published many papers in several national and international conferences. John C. Windsor serves as Chair of the Business Computer Information Systems department at the University of North Texas. He received his Ph.D. in Decision Sciences from Georgia State University. He has published over six books and many articles in journals such as Journal of Systems Management, IIE Transactions, and Computers & Security. His research interests include software and data engineering, systems security, software and data quality, and the organizational impact of information technology.
