OPACs: A research review

LISR 19,111~133 (1997)

OPACs: A Research Review Andrew Large and Jamshid Beheshti Graduate School of Library and IGformation Studies, h4cGill University

A large number of studies have investigated the Online Public Access Catalog (OPAC). This article identifies and analyzes studies that have been published since 1990, excluding comparative OPAC reviews, system descriptions and opinion pieces. It discusses the problems facing researchers as a result of the many variables at play in OPAC research-&sers, library settings, search strategies, and systems-as well as the difficulty in defining the crucial concept of “relevance.” Data collection methodologies are examined: experiment, interviews and questionnaires, observation, think aloud, and transaction logs. Research results are considered in terms of known-item searches and subject searches. Research recommendations are grouped under three headings: bibliographic record enhancement, search capabilities, and interface design.

Although Online Public Access Catalogs (OPACs) made their appearance in the mid- 197Os, it was only at the beginning of the next decade that libraries in significant numbers began to switch from card to automated catalogs (Su, 1994). From the outset the OPAC in general proved popular with users, eliminating the timeconsuming and tiresome need to search through tightly-packed drawers of filing cards. As with other aspects of library automation, growth was spectacular and, by 1985, there were at least 48 OPAC system vendors in the United States (Matthews, 1985). The so-cabled first genemt~on of OPACs incfuded basicaIIy the same bibliographic information and access points as the superseded card catalog. Many OPACs required character-by-character matching between the user’s search input and the bibliographic record, thereby reducing the likelihood of a good match. Users seemed more inclined to conduct subject (in contrast to known-item) searches on an OPAC than in the case of the card catalog, but were not greatly assisted in this endeavor by the need to confine their searches typically to Library of Congress Subject Headings (LCSH), the arcane nature of which eluded a majority. Direct nil corresponderrce ro: Andrew Large, Graduate School of Library University, 3459 McTavish, Montreal, Quebec H3A 1Y I, Canada.

& Information

Studies,

McGill

Library & inform~fion Science Research, Voiume 19, Number 2, pages II l-133 Copyright Q 1997 Ablex Publishing Corporation All rights of reproduction in any form reserved. ISSN: 0740-8188

111

112

Large & Beheshti

These early OPAC designs were replaced with more user-friendly systems-the so-called second generation OPACs that still are to be found in most libraries. Such OPACs include more refined features such as keyword searching on titles and other fields within the bibliographic record, Boolean matching, browsing facilities and improved online help provision. Despite such improvements, and notwithstanding the popularity of OPACs among users, it soon became clear that searching often proved problematic, and results not necessarily satisfactory. Several reasons can be posited for this state of affairs: Library OPAC users themselves are not a uniform breed, but come in many guises. Some may bring a considerable expertise to information retrieval tasks in general, and in particular to the use of the individual OPAC system at hand, while others may be novices in both respects. Users also bring to an OPAC search different levels of subject knowledge in the domain of their search, and more or less accurate information concerning the sought items. As Micheline Beaulieu and Christine Borgman (1996, p. 492) state, OPACs “are utilized by an extremely heterogeneous user population-probably the broadest of any type of information retrieval system.” l The OPAC systems themselves vary considerably in search capabilities, interface design, response times, database size, and the bibliographic content of individual records. l A typical OPAC encompasses a wide range of subject domains rather than being restricted to a limited subject field, and must be capable of handling effectively searches in any one of these fields. 4 OPACs at the same time are primarily confined to one form of documentmonographs; they contain within their covers a considerable amount of information but this is encapsulated in OPAC records merely in summary fashion, typically by title, subject heading, or classification number. l

The OPAC may therefore have to match a very specific info~ation request against a highly generalized description of the book’s content. While these problems are widely acknowledged, their solution has proved intractable. Third-generation OPACs are slowly appearing that include, for example, enhanced search and match techniques or ranking of retrieved output according to query relevance, but in most cases such OPACs are still at the experimental stage. These enhanced features are a consequence of the intense research effort that has been expended on OPAC investigation since the earliest days and which has contributed to an extensive corpus of information on OPACs and their users, At the same time it must be conceded that this information has raised at least as many questions as it has answered. There is no unanimity on the solution to the OPACs’ problems, pa~icu~arly in the area of subject searching. Researchers suggest improved interfaces, better online help, or even better prelimina~ user ins~ction; others argue the need for more sophisticated retrieval engines or natural language processing of user requests. Still others say that the bibliographic record itself holds the key: it must contain more subject information before subject requests can be accurately matched, no matter what the interface or retrieval mechanism. Finally,

OPACs

113

there are those researchers who pessimistically conclude that OPACs cannot be sufficiently improved by any of these methods to give users the bibliographic info~ation they seek, without missing relevant titles or delivering large numbers of irrelevant titles to the hapless searcher. OPAC STUDIES

An extensive corpus of writings on OPACs has accumulated over the past 35 years, even pre-dating their actual implemen~tion in libraries: the first article discussing the possibility of an automated catalog, written by Don Swanson, was published in 1964 (Su, 1994). By 1985 over 600 papers were available and the predicted levelling off in production (Efthimiadis, 1990) has not been realized: since 1985, a further 1,300 or so references to OPACs have been indexed in Library Literature. Indeed, so great is this literary ou~o~ing that by now a number of useful review articles have appeared, including those by Sharon Seymour (1991) Ray Larson (1991 b), Shabahat Husain and Ann O’Brien (1992) and O’Brien (1994). Any new review should avoid their duplication. This article will therefore be confined within specific boundaries. First, it focuses on more recent work, and recency has been set somewhat arbitrarily at 1990. This has the virtue of excluding the time period covered by Seymour’s article (199 1) in Library & ~~~r~at~on Science ~e~e~r&~ which took 1989 as its cut-off date (and was further restricted to research methodologies (but not results) from 16 selected OPAC studies). This is not to suggest, of course, that all pre- 1990 research is passe, but OPAC enhancements in recent years encourage an emphasis on more recently undertaken research. Second, this review is limited to work on OPACs per se and omits studies in the wider field of info~ation retrieval. OPACs are but one category of info~ation retrieval system and therefore research on the design and use of these systems is potentially relevant to the OPAC researcher. Nevertheless, as Prudence Dalrymple and Douglas Zweizig (1992) comment on information retrieval (IR) research in general (in contrast to OPAC research), “the greatest concentration of research effort has been directed toward the search inte~edia~, rather than the end user, and toward info~ation retrieval systems originally designed to be searched by intermediaries rather than...directly by users” (p. 168). Third, all the literature reviewed here is explicitly reporting research about OPACs and their users: system descriptions, prognostications, opinion pieces and the like, however interesting, have deliberately been left aside. So have comparative studies such as those by Maja Zumer and Lei Zeng (1994) who compare and evaluate the functional capabilities and interface characteristics of 17 OPACs using six different software packages, or Joan Cherry and Joseph Cox (1996), who evaluate bibliographic displays in 10 Web interfaces to OPACs in academic libraries. To be found below, then, is a critical survey of research methodologies and results concerning OPACs as reported in the Enghshlanguage literature published between 1990 and mid- 1996. RESEARCH

PROBLEMS

OPAC research has one major objective: to gather information that will enable more effective OPAC systems to be designed and deployed in libraries. As Lynn Con-

114

Large & Beheshti

naway, John Budd, and Thomas Kochtanek (1995, p. 142) put it, “there is an underlying assumption that by identifying the needs and behaviors of catalog users, user-centered catalogs can be developed.” The research seeks to assess the extent to which a user can search for bibliographic information at the OPAC, and retrieve those references held in the catalog which are deemed relevant by the user for the task in hand, while excluding all the irrelevant references held in the catalog. To the extent that the OPAC falls short of this goal, the research normally concludes by suggesting ways to correct this sho~coming by means of various enhancements to the system (or less commonly to the user through pre-search instl~ction or duringsearch guidance by experienced librarians). This research objective is far from straightforward to achieve. Jean TagueSutcliffe (1996, p. 1) suggests that the process of information retrieval consists of six major components: the set of records or documents (document set), the indexing or access method for the document set (access method), the information need of the user (user need), the verbalization of this need in a sequence of search statements or menu selections (the search strategy), the sequence of items presented as a result of the search strategy (the retrieved set or sequence) and the degree to which the retrieved set satisfies the user’s need (relevance judgement). While each component poses a new challenge for researchers (as discussed below) the last one is a particularly ubiquitous problem in IR research of any kind (see, e.g., the articles by David Blair, David Ellis, and Stephen Harter in the 1996 “Special Topic” issue of the J~u~~a~ cf the ~~~e~~c~~ Society for I~~~~~~~~~t~~rl Science). It is now widely agreed that the relevance or otherwise of a retrieved set must be judged subjectively by the individual searcher rather than objectively in some way by the researcher. OPAC researchers are therefore faced with the problem, when measuring the precision of a search, of determining whether references retrieved at the OPAC by searchers are considered relevant by those searchers. As Micheline Hancock-Beaulieu, Margaret Fieldhouse, and Thien Do (1995) comment, users themselves often find it difficult to measure the relevance to themselves of documents they have traced at the OPAC. Furthermore, the researchers must also determine how many potentially relevant references stored in the OPAC were missed by the searcher, in order to measure search recall. Joseph Janes (1994) finds some evidence to suggest that experienced tibrarians are able to judge which references a user would find relevant, and experienced library school students slightly less so, but this conclusion must remain controversial. Linda Schamber ( 1994) states that the more judges know about a given subject area, the more their judgments tend to agree; the more judges know about the subject area, the fewer records in a retrieved set do they judge relevant, and judges tend to agree more on judgments of non-relevance than on relevance. In her research, Joy Tillotson ( 1995) relied upon library school students’ judgments of relevance, while F.W. Lancaster, Tschera Cornell, Nancy Bishop, and Sherry McGowan ( 1991) compared records retrieved from the OPAC by expert searchers with the recommended readings of subject experts (as listed at the end of articles in specialized encyclopedias).

OPACs

115

Previous OPAC research reviewers (Lewis, 1989; Seymour, 1991) have been critical of the methodologies employed, citing in particular lack of rigor and errors in statistical analysis. The real problems, however, are more f~damental. OPAC users try to match search terms they have chosen against terms contained in the OPAC records. A search can be judged successful if the user is able to establish one or more such matches. But this raises more questions than it supplies answers. First, will the retrieved references, in fact, prove relevant to the user? OPAC research normally assumes that any retrieved references are good for the user. But can this assumption be made? The user may walk away from the OPAC with a list of references which even at this stage seem irrelevant or of potentially low relevance. A greater problem is whether relevance can be judged at all until the actual document has been found and examined by the user. Yet, few OPAC studies do follow up samplings of users once they have completed their OPAC session. HancockBeaulieu (1990), one researcher who has examined this problem, concludes that (p. 334) “the main subject searching activity is undertaken at the shelves and the bibliographic tool serves to direct the searcher to the relevant areas of the collection.” O’Brien (1990, p. 270) recognizes this problem: “the information-seeking behavior of library end users extends way beyond the OPAC... Overall judgement of the relevance of documents may change between retrieving citations, browsing the shelves and finding (or not finding) the documents.” It would be possible to go further by arguing that relevance can only properly be measured after the user not only has chosen the documents at the shelves, but actually employed them for the work in hand. For researchers, of course, such measures are difftcult to make; it is far easier to judge search success while still at the OPAC. Second, many researchers have assumed that failure of the user to make a match at the OPAC denotes a failed search, without taking into account that the OPAC may simply not list any relevant references. Third, the search may have matched with relevant references but still missed the “best” references. Lancaster et al. (1991, p. 379) argue that “a subject search in the catalog of a library cannot be considered fully successful unless the user is able to locate the material that is, in some sense, the ‘best.“’ Any research involving human subjects is likely to encounter immense difficulties in controlling the numerous variables at play. These typically include such factors as age; gender; knowledge of and experience with information retrieval in general, with OPACs, and with the specific OPAC under investigation; domain knowledge; commitment, time available for the search, cognitive skills in evaluating information; and so on (Borgman, 1989; Hsieh-Yee, 1993; Qui, 1993). The search itself provides another set of variables. It can be to find known items, a subject search (and in practice this distinction, crucial for evaluation of search success, can prove difficult for the researcher to establish when analyzing search records), or both. Subject searches may be intended to locate a few relevant items in the library, or an exhaustive search to find everything on the topic held in the library. The search may be confined to one concept or invoIve the combination of several concepts, thereby increasing its complexity. The search may be on a topic likely to be dealt with by monographs (the primary document type found in most OPACs) and retrievable by searching on keywords in titles, series names, and subject descriptions, or on subject headings (the typical subject search keys available in an OPAC). The search might also be for topics more likely to be covered by

116

Large & Beheshti

journal or encyclopedia articles, or dealt with in monographs but at a level of detail that precludes their indexing in a typical OPAC. Much research has assumed that the session under observation is self-contained, neither being preceded by nor preceding related sessions in which other search terms and/or strategies are employed. Amanda Spink and Leslie Beatty (1995) have pointed out, however, that users do not necessarily limit their interaction to a single search session but rather may conduct several linked sessions driven by an evolving and changing information problem. Carol Hert (1996) found that users’ goals did not change or at least only changed minimally during an OPAC interaction (she suggests that users do not necessarily have enough information about the retrieved records judged relevant to change their goals), but nevertheless she concedes that a search session is part of a larger information-gathering process; the observed session was preceded by and will be followed by related sessions in which other strategies can be employed. Yet another variable is provided by the OPAC itself. There is no such thing as “an OPAC.” OPACs are produced by rival companies for a variety of different library environments and exhibit a variety of interfaces, retrieval engines, help facilities, response times, and so forth. This can make it difficult to compare the findings of a study on one OPAC with that on a second OPAC (or even the same OPAC at a later time if system enhancements have been implemented). Rhonda Hunter (199 1) explains the overall failure rate of 54.2% in her study with a failure rate ofjust 40.5% in an earlier study (Peters, 1989) by the fact that her OPAC lacked keyword searching capabilities. The size of the OPAC database can also influence research findings. Large databases pose different search problems than small databases. For example, enhancements of bibliographic records to include more subject information may improve recall, but equally it may have a negative impact on precision that is much more marked in a large than a small database. Finally, when considering OPAC research, account should be taken of the kind of library in which the OPAC is located. This is likely to affect the type of documents listed in the OPAC, the types of searches typically conducted and especially the user characteristics. Almost all OPAC research has been conducted in university libraries with under~aduate and graduate students or faculty; little has been done in school libraries and almost nothing in public libraries. These reservations should be borne in mind when reviewing OPAC research in general, and particularly when comparing results. To this list of variables should be added one more: the different ways in which researchers collect their data. DATA COLLECTION A number of methodologies have been developed to collect data on OPAC users and usage. These methodologies are not mutually exclusive, and many studies employ more than one technique to collect data. Experiment Research variables, as discussed above, can best be controlled in an artificial environment. By modifying one variable while trying to hold all others constant,

OPACs

117

researchers can hope to measure the impact of the chosen variable. In practice, controlling for all the possible variables at play in OPAC research (assuming they all can even be identi~ed) is usually impossible. The number of variables introduced just by the human subjects would require an unacceptably large sample to have the slightest chance of achieving this ideal, as Bryce Allen (1991), for example, acknowledges. The experimental approach has led some researchers to opt for a test database rather than a real OPAC database. For example, Martin Dillon and Patrick Wenzel (1990) added abstracts and tables of contents to standard bibliographic records in their test database in order to establish whether this additional subject information influences recall and precision ratios. Christine Borgman, Sandra Hirsh, Virginia Walter, and Andrea Gallagher (1995) also used experimental systems to study children’s IR behavior. Such test databases are likely to be much smaller than those found in real OPACS, and database size is itself a factor in searching. The relative smallness of the test database also often makes it unsuitable for use by real library patrons going about their normal OPAC searching. The database is likely to be confined to a limited range of subject areas and/or time periods. The researchers may therefore have to construct artificial search queries to match against the test database (as did DilIon and Wenzel), and find volunteer or paid searchers to participate in the experiments. Allen ( 199l), and Maaika D. Kiestra, Mia Stokmans, and Jan Kamphuis (1994) hired students for this role. The searchers will then no longer be seeking to meet their own information needs but rather those established by the researchers. Will they behave “normally” in such circumstances, or will the experimental setting introduce abnormal behavior, either by encouraging patrons to try harder than usual to do better in the “test,” or by reducing the level of effort through lack of se1f-motivation? As Stephen Wiberley, Robert Daugherty, and James Danowski (1995, p. 248 ) comment, “studies in a normal working environment are important for understanding persistence because experimental settings may change motivation and significantly alter results:” they fear that subjects might exhibit abnormal persistence even in carefully designed experiments. The use of guinea pig researchers raises further problems, especially in measuring the relevance of retrieved references. In some experiments this problem has been tackled by using members of the research team as searchers (Tillotson, 1995), but this is unlikely to improve the authenticity of the search process, especially if the chosen research searchers are very well informed on OPACs and searching techniques. In such cases the rigor of the experimental setting must be set against the artificiality of the database, the searchers, and the queries. Gunnar Knutson (199 1) managed to combine an experimental modification of a small number of catalog records with retention of the complete OPAC database for searching by regular patrons. In order to investigate the relationship between the catalog record and collection use, he added extra subject headings and contents notes to a sample of records and then monitored the circulation statistics for this sample and a control sample. In this case the users were unaware of the experiment’s existence. Karen Drabenstott and Marjorie Weller (1996a) developed a small experimental catalog based on subsets of the collections in two libraries to investigate spelling errors in query words, but users then conducted actual searches without any intervention from the researchers.

118

tatge & Beheshti

Another experimental technique is mathematical modelling and simulation, primarily utilized in testing IR (rather than OPAC) systems. Larson (1992) applied this methodology to an experimental OPAC to test a probabiIistic retrieval model. He used a vector model where the documents that were similar to a query were considered relevant to the information needs of the user who submitted that query. The advantages of working with genuine searchers seeking to match real queries against a real database outweigh for many researchers any benefits accruing from a controlled experimental environ~nent. Nevertheless, the experimental route to data collection has been quite widely used, as evidenced by the studies cited above. Jennifer Mendelsohn (1994) employed what might be termed a quasi-experimental methodology. She divided her user sample into two groups: one group was offered help at the terminal by reference librarians; the second group was not. The users then undertook their own searches on the actual OPAC but the control group (no help) enabled the researcher to draw concfusions about the importance of intermediary assistance. Similarly, Tillotson (1995) was able to assess the relationship between catalog size and the effectiveness of keyword searching by comparing searching on two similar OPACs, one with 700,000 records and the other with seven

million records.

If OPAC users are the focus of research attention (as is often the case) then a relatively straightforward method of gaining information about a search is to interview users after a search (or in some cases also i~ediately before the search) at the OPAC (see, e.g., I-Iert, 1996; Beheshti, Large, & Biaiek, t 9%). Connell f I 995) believes that structured interviews perform well at eliciting structure and process. The focus group, as used in the Toronto Public Library study (White, Deane, & Livingston, 1996) is a variant on the standard interview. A common alternative to an interview is to invite users to complete a questionnaire. This methodoiogy enables basic biographical data to be collected as welt as search-related informatjon such as search objectives, search strategies, and search evaluation by the user. Interviews and questionnaires, however, are both open to telling criticism. Care must be employed with sampling to ensure that a cross-section of the target population is chosen as well as that the sampte is large enough to support reliable statistical analysis of collected data. The questions, whether oral or written, must be carefully formulated to avoid ambiguity on the one hand, or Ieading the respondent on the other. Interviewing is a more reliable way than questionnaire completion, especially if unsupervised, of ensuring a response from the user, but imposes practical limitations on the sample size. The greatest problem, however, rests with the reliability of the data. Respondents may not always find it easy to recall the precise steps in a complicated OPAC search. These problems are shared by focus groups. To representative composition, leading questions and accuracy of recall can be added the danger that one or more dominant members will hijack the discussion and provide a misleading impression of the views held by other members of the group.

OPACs

119

Typically, paper-based questionnaires are presented to users to complete either on the spot or subsequently (in the latter case reducing the likelihood of a high response rate). An alternative is to include the questio~aire online at the OPAC itself. Terry Ferl and Larry Millsap (1996) installed questionnaires on OPAC terminals that were displayed when a session-closing command was issued by the user or after about eight minutes of inactivity (timeout). Hancock-Beaulieu et al. (1995) created online versions of both a pre- and post-search questionnaire. The pre-search questionnaire had to be completed in order to begin the search, and hence no problems were encountered with poor response rates. The same could not be said, however, for the post-search questionnaire. Despite online pleas for users to take the time to answer several questions after their searches, a “disappointing” response rate of only 39% was achieved. This rate, however, is no worse than for many paper-based questionnaires. Pat Ensor (1992), for example, reports a response rate of only 35% to a questionnaire (though it did have 27 questions, da~tingly long for many would-be respondents). Online questionnaires have obvious virtues in studies like that by Pamela Snelson (1993) who was investigating remote access to a university OPAC where conventional questionnaires or interviews could not have been easily employed. It should be noted, however, that the sample choosing to answer an online questio~aire is self-selecting and therefore cannot be assumed representative of the entire user population.

Observation Obse~ation of users at the OPAC is employed by a few researchers but seems to be relatively unpopular as a data-gathering instrument. It is difficult unobt~sively to watch a user at an OPAC terminal so carefully as to be able to observe the search steps undertaken. Observation also raises issues of user privacy. Is it ethical to involve library patrons in a research project without informing them of its purpose and formally seeking their consent to be involved? Library users do not expect to be su~eptitiously watched at the catalog or followed around the library either by staff or researchers. Informing users that they will be observed risks inducing un-typical behavior on the users’ part. Nevertheless, observation can supplement other data-gathering techniques. Wiberley et al. (1995), for example, observed user sessions so as to identify the start and finish of sessions that were captured online in a transaction log, thus overcoming a colon problem with transaction logs of knowing where one session ends and a new session begins. Mendelsohn (1994) also employed observation. She was investigating the need and wish of OPAC users for in-person help, and users were therefore approached individually by reference librarians and offered help. If they agreed, the OPAC-user interaction was observed (a post-search questionnaire was also included). Mendelsohn does not comment on the possibility of the user’s search being modi~ed as a result of obse~ation. M. Kiestra, M. Stokmans, and .I. Kamphuis ( 1994) videotaped their experimental search sessions as well as employing an observer to monitor subject performance on an additional terminal, and asking their subjects to think aloud while performing the recorded task.

120

Large & Beheshti

Think Aloud Think aloud techniques by users while searching an OPAC represent a compromise between interview and observation. The resulting monolog can be taped for later analysis. In such a case the user clearly must be informed ahead of time that the search will be monitored, risking an adjustment of the user’s typical behavior. Compared with interviews or questionnaires, the searcher is more likely to recount accurately the search process, although it may prove difficult for a respondent to remember the need to think aloud, especially for the frequent and experienced OPAC user. Intermediate steps in the process may be carried out without conscious thought and therefore without verbalization (Connell, 1995). Like observation, this technique is demanding on researcher time and is unlikely to be used with large samples. Paul Solomon (1993) included think aloud along with observation and questio~aires in his study of OPAC use in an elementary school library. Connell (1995) used think aloud in conjunction with transaction log and structured interviews to determine the type of knowledge used by experienced librarians when searching for subjects in OPACs.

Transaction Logs The drawbacks of interviews, questionnaires, observation, and think aloud center upon the reliability of the user responses or even the normality of the searching behavior when users know they are the subjects of a research investigation. One data collection technique can avoid such problems. Users’ keyboard actions as well as the system’s responses can be captured at the OPAC as a log of each user’s transactions. Such transaction logs are an invaluable source of data about OPAC usage though, as Drabenstott and Vizine-Goetz (1994) point out, standards for these logs are still being developed and therefore content and format vary from system to system. The data are straightforward to capture, involving no effort on the researcher’s part, though subsequent data analysis can be slow and tedious. A great deal of data can be collected, thus allowing large samples to be analyzed. For example, Hunter (1991) and Wallace (1993) both analyzed around 4,000 transactions, while Larson (199 1a) processed over five million search transactions. As Wallace (1993, p. 240) says, “visually analyzing a transaction log quickly enables researchers to put themselves in the user’s place, and they begin thinking as the searcher must have thought at the time of the search.” There is no possibility of researcher bias in such data collection, and the user will not be influenced by the presence of researchers hovering around the OPAC. As Connaway et al. (1995) argue, transaction logs provide a record of what the searchers did rather than say they did. There is little wonder, then, that transaction logging has become a very popular research methodology in OPAC studies. Nevertheless, ~ansa~tion logs do have drawbacks. First, a search cannot be linked easily with a particular searcher, and therefore it is impossible to investigate the relationship between user characteristics and searching behavior. It is also extremely difficult to know the user’s ultimate objective for the search or whether, from the user’s perspective, it was achieved. Both the searcher’s motives and the searcher’s reaction to retrieved records remain unknown to even the most perspicacious of researchers. As Larson (1991a, p. 198) says, “transaction monitoring alone can

OPACs

121

provide highly detailed information about how users actually interact with an online system, but it cannot reveal their intentions or whether they are satisfied with the results.” It is not even possible necessarily to establish something as basic as whether a search using keywords in the title field is looking for a known item by title or is a subject search (Wildemuth & O’Neil, 1995). Hunter (1991) had to repeat some logged searches as it was impossible to tell from the log whether a hit had or had not been scored. It may also be difficult in practice to identify where one search ends and another begins. To overcome this particular probIem, Wiberley et al. (1995) used research assistants to observe the sessions and log the begi~ing and ending time of a search session using clocks synchronized with the OPAC (although they admit that exactness in noting when sessions began and ended was sometimes limited by distance from the terminal and traffic between observers and terminals). For such reasons, transaction logs typically are used in conjunction with other methods. Hancock-Beaulieu, Stephen Robertson, and Cohn Neilson (I 991) concluded that a basic screen logging facility is an inadequate tool for evaluating user performance at an OPAC because it can only partly inform on user search behavior. It must be used alongside other methods (Connaway et al., 1995, with questionnaires and observation; Hancock-Beaulieu, Fieldhouse, & Do, 1995, with questionnaires; and Connell, 1995, with think aloud protocoIs and structured interviews). RESEARCH RESULTS Most OPAC studies concentrate on subject searching--searches conducted on a topic where the user wants to locate in the catalog all items that deal with the topic. This is contrasted with known-item searches where the user wants to find out from the catalog whether a specific item about which some biblio~aphic info~ation is known (e.g., author, editor, or title) is listed. Many studies have been undertaken to establish the distribution of OPAC searches between known-item and subject. The results vary considerably, as Wildemuth and O’Neil(l995) report. Transaction log analyses, for example, suggest that anything from 24% to 78% of OPAC searches are for authors or titles. User surveys point to a slightly narrower range: between 33% and 67%. Such wide discrepancies in estimates point to a problem in recognizing types of search. One difficulty is caused by the ambiguity of keyword title searches, Is a search on keywords in the title field intended to locate by its title a known item, or is it a subject search to f”lndall items in the catalog dealing with a specific subject as identified by the presence of specific words in the title field? Such ambigui~ is especially prevalent when data are gleaned from ~ansaction logs and therefore where the client is unavailable to shed tight on motives. HancockBeaulieu (1990) found that, in Parts 1 and 2 of her study, 36% and 37% respectively of cases initiated as known-item searches developed into subject searches, but the reverse (known-item searches disguised as subject searches) hardly ever occurred, She suspects that title subject searches in past catalog studies could have been mistaken for specific ~o~-item searches. Known-Item Searches There is general agreement that known-item searches pose many fewer retrieval problems than subject searches. Wildemuth and O’Neil (I 995) found that 70% of

122

Large & Beheshti

users in a university library found the items they were seeking. Success (i.e., location of the sought item, or non-location in the case of that: item not being listed in the catalog) in known-item searches is largely dete~ined by the type, extent and accuracy of the bibliographic information known to the user. Empirical data concerning the kind of bibliographic information brought to the OPAC, however, are almost non-existent. In an interview of users in a university library, Wildemuth and O’Neil(l995) found that the most common search element was the title (94%), followed by the publication date, especially forjournals (70%) and the author (50%). Title searching is prone to failure when an exact match is required, as users cannot be relied upon to remember the precise wording of titles. Hunter ( 199 1) found that almost 14% of title searches failed because of the inclusion of an initial article. The addition of keyword searching to many OPACs has greatly alleviated this exactmatch problem. Author searches are plagued by the need to invert first and surnames in most OPACs. Hunter found that 15% of author searches failed because the first name was entered first. Su~arizing results from a number of known-item studies, Wildemuth and O’Neil (1995) report that the most frequent causes for failure, accounting for around one third of the total, were typographical errors and misspellings. The average failure rate for known-item searches was 25%. Subject Searches Different OPAC systems provide different techniques for undertaking subject searching, but generally it is possible to search for subjects as represented in the bibliographic records by subject headings, typically LCSH, and keywords in the title, subject heading, and other fields. Some OPACs may provide additional info~ation such as book tables of contents or even back-of-book indexes, though this is still more common in experimental systems. OPACs also differ in the means they provide to combine search terms to express complex subjects and to match these terms with the subject terms in the records, but typically Boolean matching, right-handed truncation are available-perhaps also with some way of browsing subject indexes. If there is general agreement that OPACs are challenged most in subject searching, this consensus does not extend to the solution of the subject-searching problem. Unlike a known-item search, whose objective is to establish whether a specific record is stored on the OPAC, a subject search is much more open-ended. The client must first conceptualize the need for info~ation on a subject, then express that conceptualization in specific search terms, and match these terms against the terms used in the OPAC records to express subject aboutness. The client must also know how to reconfigure a search if no records, too many records, or irrelevant records are retrieved at the first attempt, and when to terminate the search because everything that can be found has been located. These difficulties may be responsible for users beginning to adopt title keyword searching instead of searching subject indexes. Larson (I99 la) reports a rate of decline of about 2% per year for subject searching. Connell (1995) studies IO experienced librarians’ subject searching behavior. The results show that subject searching requires a considerable knowledge base about syntactic and semantic structures, Boolean operators, controlled vocabulary, and auxiliary tools such as LCSH.

OPACs

123

When analyzing results it must always be remembered that not all zero hits represent failures and not all hits represent successes. A research student seeking a fresh topic to work on may be delighted to discover a virgin field. The possibility of the researcher misinterpreting search results is especially high if a data collection methodology like a transaction log is used. Keyword versus subject searching interests many researchers. Tillotson (1995) found that if users had entered the same terms as keywords instead of subjects they would have had more satisfactory results, but found no evidence that they were happier with the results when they did use keyword searching. She thinks that people will choose both keyword and controlled vocabulary searching, and therefore both should be available. Larson (1991) accepts that title keywords offer a limited form of natural language access but that they also place a burden on the user to take account of all synonyms for any concept.

RESEARCH RECOMMENDATIONS Researchers have suggested a variety of ways in which an OPAC might be redesigned to provide better results for the client seeking to identify records on the OPAC. These can be categorized under three headings: Improvement Enhancement o Improvement l

l

of the bibliographic records; of the search capabilities; and of the interface.

bibliographic Record Enhancement The amount of subject information available in a typical OPAC record is extremely limited. Can users retrieve records in a subject search based on nothing more than a title, a few subject headings, and perhaps several classification numbers? The task is made the more difficult, as O’Brien (I 990) reminds us, by the artificial nature of subject headings and classi~catio~ nu~nbers that are not necessarily easy for users to interpret. A study of public library users with low or limited literacy skills found that most had no knowledge of the Dewey Decimal Classitication (DDC) or even what it was (White, Deane, & Livingston, 1996). According to Knutson (1991, p. 63, “insufficient subject access in standard cataloguing has become almost a truism in the critical literature.” In his study of social science titles, he found that only 2.6 LCSHs, on average, were allocated per record. He suggested that one reason why many books in large academic libraries, seldom if ever, circulate is due to inadequate subject access: “part of the challenge facing research libraries is to find ways to modify current subject cataloguing practices so that materials are more accessible to users” (p. 66). Nevertheless, he is compelled to admit that his own results on the relationship between more controlled subject headings and higher circulation were inconclusive. A number of researchers have suggested the enhancement of the bibliographic record in one way or another. Shirley Cousins (1992), for example, found that the addition of table of contents and back-of-book indexes combined with automatic truncation, provides an increase in recall over bare MARC records, This increase

124

Large & Beheshti

in recall was “slightly better” than when she enhanced the records instead with PRECK terms, which, in turn, were more effective than LCSH. She makes the interesting point that such e~an~~rne~ts not only aid users in retrieving records but also help them to make relevance judgments at the catalog rather than having to find the documents themselves, an observation supported with some empirical evidence (Byrne & Micco, 1988). Cousins ( 1992, p. 292) concludes that “in the long term improving the subject content of OPAC records seems the only viable approach to achieving significant improvements in subject access, since there is a limit to the extent that the current minimal subject informatjon can be expfoited.‘” Karen M. Drabenstott, Ann Demeyer, Jeffrey Gerckens and Daryl T. Poe (1990) suggest records can be successfully enriched using text drawn from the DDC classification scheme. The subject of the document can be matched with the schedule caption or the index entry from the relevant part of the DDC classification schedule (and this can be done largely automatically). If researchers agree that subject enhancement of bibliographic records can increase recall, they are equally in concord that its effect both on absolute recall and on precision often causes new problems (Dillon & Wenzell, 1990). The sheer number of records in a typical OPAC database is likely to generate recall in the hundreds or even thousands once improved subject access is provided. In the absence of some kind ofranking or weighting of retrieved records, this is likely to overwhelm many users. Lancaster et al. (1991) reject the suggestion that the problems of subject access could largely be solved by storing in a form suitable for searching the text of contents pages and/or indexes: “even ifthis were economically feasible, it would make little practical difference to the retrieval capabilities of a large catalog because the resulting precision would be completely intolerable” (p. 387). Search Capabilities The variations between the searching capabilities of the different OPACs investigated by researchers mean that generalization of~ndings must be treated cautious@. Users are often not sophisticated in their utilization of those searching options that are available, and OPAC research in common with research in other information retrieval environments points to an under use of search facilities (Hunter, 1991). Thus, Connaway et al. (1995, p. 150) report that “the vast majority of the searches conducted were conservative; that is, they made use of the simplest and most s~aightforward search modes and searchable Gelds available.” Hancock-~eaulieu f 1990) argues for more sophisticated matching such as term weighting, reverse frequency ranking, and automatic stemming. If added, however, these features must be transparent to users. Drabenstott and Weiler (1996b) have used their earlier experiences to create an experimental online catalog, ASTUTE (A Search Tree Underlying the Experiments to test the concept of search trees, Users of online catalogs usually have several choices for accessing the subject index; ASTUTE is designed to assume the burden of determining which subject-searching approach is likely to produce useful information. The experimental results suggest that search trees may be better tools for selecting more appropriate paths to subject searching than users’ choices. Draben-

OPACs

125

stott (1996) concludes that the time has come to implement a new subject-access design in online catalogs. She recommends the inclusion of subject trees that would remove the burden of dete~ining which subject-searching approach is likely to produce use&d information for user queries from the user to the system. Users encounter many problems, as in other IR contexts, in choosing suitable search terms in which to represent their subject interest. Hunter (199 1) found that some people entered very broad terms and then spent the entire search session paging through hundreds of titles without even looking at a biblio~aphic display of any title. Some entered long phrases or sentences but did not use Boolean operators to link terms, According to Hirsh and Borgman (1995), children encounter difficulties in formulating Boolean queries, understanding many terms used as subject headings, selecting appropriate subject terms, especially for complex topics, and generating alternative terms if the first attempt was unsuccessful. Borgman et al. (1995, p. 665) are blunt in stating that “with few exceptions . . . the current generation of online catalogs are (sic) designed for adults and do (sic) not meet the special needs and capabilities of children.” One suspects that older users do not necessarily fare much better. Yee (199 1) summarized the problems encountered by novice users in general as finding appropriate subject terms, large numbers of hits and failure to reduce the retrieved sets, zero hits and failure to increase the retrieved sets, failure to ~ders~nd cataloging rules, spelling and typographical errors. In addition, lack of unders~nding of indexes, files and basic database structure leads to the use of articles, stop words, entering an author’s first name before the surname and hyphenation problems. Users, whether children (Hirsh & Borgman, 1995), low literacy public library patrons (White, Deane, & Livingston, 1996) or university staff and students (Connaway et al., 1995) regularly enco~ter ~po~aphical problem~isspellings and keyboarding errors are common. Spelling correction would be a major OPAC enhancement (Drabenstott & Weller, 1996a). Interface Design A wide variety of OPAC design features can be considered under the rubric of the interface, but researchers have tended to concentrate critical attention on a few specific aspects. Overall, as Charles Hildreth (1995) states, there has been little research on most components of the OPAC interface. There is general agreement that the menu sequence in which search options are offered will influence user selection (Ballard, 1994; ~~cock-Beaulieu, 1990). Users are apt to choose an option higher rather than lower in a list. Wallace (1993) concludes that there is a very short window of opportunity when searchers are amenable to instruction, and that successful screen designs should therefore focus on presenting first the quick-searching needs of the majority of users. If true, of course, this would have the effect of reducing even further the use of more sophisticated (and perhaps more effective) retrieval facilities. Kiestra et al. (1994) found that too many options simply caused confusion, at least for less experienced OPAC users. Wiberley et al. (1995) found a relationship between user persistence at the OPAC and features such as the arrangement of postings in a help&l order, ease of movement among screens, and simplicity of commands. Allen ( 1994) found

126

Large &

Beheshti

that the display of subject headings as the first element in a bibliographic record improved subject searching on some tasks. Dorothy McGarry and Elaine Svenonius f 1991) looked at how displays of subject terms might better be presented. In particular they explain how browsable displays might be compressed so that subject heading displays could be confined just to a few screens. Allen (1993) agrees that techniques can be employed to reduce the amount of screen scanning necessary to find subject headings, but he also found that it achieved no improvement in search performance. Brian Williams, Stacy Sawyer, and Sarah Hut~hinson (1995, p. 145) favor the graphical interface with its windows, icons and point-and-click devices as being the easiest to use. Hildreth (1995, p. 2) however, urges caution: Looking for documents or other publications in an online catalog is not just a me&han~sti~ infor~~at~on seeking activity. It is a dynamic, decision making activity which requires that careful consideration be given not only to the information to be provided, but also to the manner in which that information is presented in displays and to the set ofdecision making facilities available to assist the user in carrying out primary tasks and subtasks. He reminds us that not all UPAC users are the same, and that one OPRC interface will be less usable for some than others, Mendelsohn (1994) believes that improvements to the interface, including online help, cannot replace the human expert. She found that in more than half the cases where users requested human help it was for procedural assistance, suggesting that the OPAC does not fully support self-service. White et al. ( 1986) found that patrons with Iow or limited literacy in a public library were unfamiliar with the layout of the computer keyboard (e.g., the return key or the erase keys), many only read the center of the screen and ignored information on the left side, top or bottom, were unaware of the significance of highlighted lines, thought there was too much print on the screen, became disoriented and confused when moving back and forth from screen to screen, and did not understand jargon such as ‘“enter a line number,” ‘&u-t over,” “truncated,” “display,” “select,” “format,” or “keyword.” Would they have found a graphical interface any easier? Seniors encountered difficulties with screen glare and often misread commands. Most OPAC studies, of course, have not left the confines of the university library with its specialized user groups. Recent advances in information technofogy have focused the attention of a few researchers on browsing as an alternative search technique for novice users. Shan-Ju Chang and Ronald Rice (1993) have reviewed over 100 publications to arrive at a conceptual framework and a taxonomy for browsing. Gary Marchionini (1995, p. 100) states that browsing is a “natural and effective approach to many types of info~ation-seeking problems.” One reason for this surge interest is the tindings of earlier research which show that library patrons browse the actual shelves even after consulting the online catalog (Markey, 1984). Hancock-Beaulieu ( 1990) found that between 30% and 45% of all online searches, regardless of type of search, are concluded with browsing the library shelves.

of

OPACs

127

Querying in an online catalog imposes a large cognitive load, compelling users to devise search strategies, conceptualize appropriate search terms, cope with Boolean logic, and manage large or zero number of hits. As more features are added to online systems to increase their retrieval effectiveness, the cognitive load on the user rises proportionally (Hancock-Beaulieu et al., 1995). Browsing, however, requires a “smaller cognitive load than analytical search strategies do” (Marchionini, 1995, p. 103). It is easier to recognize information presented in a famiIiar metaphor, such as browsable lists, than to recall it from memory (Borgman et al., 1995). In many situations, novice cfients may use a combination of search techniques, including browsing (Belkin, Marchetti, 2%Cool, 1993). Browsing is primarily visual and needs direct manipulation ofobjects on the screen because “it involves cognitive, perceptual, and motor systems concurrently and continually” (Marchionini, 1995, p. 104). Beheshti, Large, and Bialek (1996) have designed a browsable GUI OPAC, Public Access Catalogue Extension (PACE), which presents MARC records as simulated images of books. PACE is designed to augment or extend the capabilities of existing online systems. Simulation data are derived from the physical description of books, including number of pages and height embedded in MARC records. This information is used to generate an image of a book, the dimensions ofwhich are proportional to the actual size. PACE was tested in a college library against a text-based OPAC in a reaf operational environment. The results show that a simple browsable retrieval interface performed as well as a second-generation, full-featured OPAC in terms of retrieval speed and search success. The overwhelming majority of students, however, found PACE very easy to use and preferred its browsing capability through the familiar metaphor of books and library shelves to a traditional OPAC. CONCLUSION Over two decades have passed since the first OPACs were introduced in libraries. While first generation OPACs were unsophisticated and had few features, over the years the quality and characteristics of second generation OPACs were vastly improved. Yet, the obstacles facing users prompted one researcher to state that the second generation OPACs were powerful and efficient but also dumb, passive systems which require resourceful, active, intelligent human searchers to produce acceptable results (Hildreth, 1989). Beaulieu and Borgman (1996, p. 49 1) could still state: “Despite the advances in technofogy, research continues to show that these systems are ineffective and hard to use.” Borgman (1996, p. 501), in an article on users’ problems with online catalogs, concludes. * “Most of the improvements are in surface features rather than in the core functionality.” She suggests that to improve existing systems, knowledge of end-users’ searching behavior should be integrated into OPAC retrieval engines and interfaces. Although these second-generation OPACs are still used today, technological advances have resulted in improved systems. The client/server model, Open System Interconnection (OSI), TCPIIP and 239.50 standards and the World Wide Web have now opened new doors for third-generation OPACs. Many libraries have seized upon this opportunity to provide access to their OPACs through the Web-to-Catalog

Interfaces or “webbed” interfaces. Although many public and private organizations are designing new software, the implementation of 239.50 standards has ensured remarkable ~~fo~i~ among various OPAC interfaces. As more resources are becoming available on the Web, the Library of Congress, QCLC, and other organizations are augmenting MARC records to reflect the recent developments in information delivery, Tag 856 in MARC (Electronic location and access) contains the Uniform Resource Locator (URL) of the item, linking the record to any type of digital unit. The unit may be a full-text document, stilt image, video, program, or a server. Users can “click” on the records (or Tags) to connect automatically to particular servers and the actual items will then be displayed on their client stations. These Web-to-Catalog interfaces are currently designed for end-users; however, in the near future their interfaces will accommodate more sophisticated search interfaces. The proposed Query Types in the 239.50- 1995 standards are powerful search tools aIlowing users to create requests such as: ~~~~~ni~~d na%ms and] UWCEF cy) IL0 nor) (HakaIa, 1996). Other proposed features include index browsing and Ranked List Query (RLQ). The recent developments in OPAC design should offer significant advantages to users. Improved interfaces, ranked lists of retrieved output, and enhanced browsing features now appearing in third generation QPACs are all in line with recommendations from many research studies (it would be an ~~ter~st~ngresearch project in itself to establish the extent to which commercial OPAC designers, in fact, have been influenced by research findings rather than working on their own intuitive and commercial hunches). Despite such advances, however, the fundamental problems of OPAC use, and indeed of IR systems in general, still remain. How should the performance of an IR system be measured so that effective comparisons can be drawn between rival retrieval techniques? Distrust of the relevance judgments that underlie recall and precision measures is growing, yet no-one has suggested a viable alternative quantitative measure. An inability to measure performance effectiveness is a major barrier to research intended to identify more effective systems. Much work of a qualitative nature has been completed, in part at least because of the problems in quantitative measurement. Users have been endlessly studied (although most of these users have been located in higher education environments and therefore their representativeness of the general population can be doubted). Yet, we still do not know exactly why some individuals are better searches than others: which specific bundle of characteristics make a successful searcher. Determining users’ behavior represents an ihusive research problem. After hundreds of studies, there are still more questions than answers; is an individual user conducting one search session pursuing a single goal, or is the goai dynamic and changing constantly‘? Does the user divide up the search goal among several search sessions? What user’s characteristics might have an effect on determining goals and search strategies? Finally, no OPAC as yet exists that can take the natural language query of a novice user and convert into a search strategy that will find exactly what the user is seeking. Most of the cognitive work involved in searching continues to rest with the searcher rather than the system. The existence of these major research areas seems guaranteed to ensure that OPAC researchers will not face extinction in the foreseeable future.

OPACs

129

REFERENCES Allen, Bryce. (1991). Topic knowledge and online catalog search fo~ulation. Library Quarters, 61, 188-213. Allen, Bryce. (1993). Improved browsable displays: An experimental test. Information Technology & Libraries, 12,203-208.

Allen, Bryce. (1994). Cognitive abilities and information system usability. Infirmation Processing & Management, 30, 177-I 9 1. Ballard, Terry. (1994). Compamtive searching styles of patrons and staff at a university library. Library Resources & Technical Services, 3%,293-305. Beaulieu, Micheline, & Borgman, Christine L. (1996). A new era for OPAC research: Introduction to special topic issue on current research in online public access systems. Journal of the American Society for Information Science, 47, 491-492. Beheshti, Jamshid, Large, Valerie, & Bialek, Mary. (1996). Public access catalog extension (PACE): A browsable graphical interface. Information Technology and Libraries, I5,23 l-240.

Belkin, Nicholas J., Marchetti, P.G., & Cool, Colleen. (1993). BRAQUE: design of an interface to support user interaction in information retrieval. Information Processing & management,

29,325-344.

Blair, David C. (1996). STAIRS redux: noughts on the STAIRS evaluation, ten years after. Journal of the American Societyfor Information Science, 47,4-22. Borgman, Christine L. (1989). All users of information systems are not created equal: An exploration into individual differences. Information Processing & Management,

25,237-252.

Borgman, Christine L. (1996). Why are online catalogs still hard to use? Jo#rna~ of the American Society for Info~ation

Science, 47,493-503.

Borgman, Christine L., Hirsh, Sandra G., Walter, Virginia A., & Gallagher, Andrea L. (1995). Children’s searching behavior in browsing and keyword searching online catalogs: the Science Library Catalog Project. Journal of the American Society for Information Science, 46,6&Z&684.

Byrne, Alex, & Micco, Mary. (1988). Improving OPAC subject access: The ADFA experiment. CoElege & Research Libraries, 49,432-441. Chang, Shan Ju, & Rice, Ronald E. (1993). Browsing: A multidimensional fi-amework. In M.E. Williams (Ed.), Annual review of information science and technology, Vol. 28 (pp. 23 l-276). White Plains: Knowledge Industries. Cherry, Joan M., & Cox, Joseph P. (1996). World Wide Web displays of bibliographic records: An evaluation. . Connaway, Lynn Silipigni, Budd, John M., & Kochtanek, Thomas R. (1995). An investigation of the use of an online catalog: User characteristics and transaction log analysis. Library Resources & Technical Services, 39, 142-152. Connell, Tschera Harkness. (1995). Subject searching in online catalogs. Joumat of the American Society for Info~ation

Science, 46, 506-5 18.

Cousins, Shirley Anne. (1992). Enhancing subject access to OPACs: Controlled vocabulary vs natural language. Journal of Documentation, 48,291-309. Dalrymple, Prudence Ward, & Zweizig, Douglas L. (1992). Users’ experience of information retrieval systems: An exploration of the relationship between

130

Large % Beheshti

search experience and affective measures. Library & Information Science Research, 14, 167- 18 I. Dillon, Ma~in, & Wenzell, Patrick. (1990). Retrieval effectiveness of enhanced bibliographjc records. Library Hi-Tech, 8(3), 43-46. Drabenstott, Karen M. (1996). Enhancing a new design for subject access to online catalogs. Library Iii Tech, 14, 87-109. Drabenstott, Karen Markey, Demeyer, Ann, Gerckens, Jeffrey, & Poe, Daryl T. (1990). Analysis of a bibljograph~c database enhanced with a library classification. Library Resources & Technical Services, 34, 179198. Drabenstott, Karen M., & Vizine-Goetz, Diane. (1994). Using subject headings for online retrieval: Theory, practice and potential. San Diego: Academic Press. Drabenstott, Karen M., & Weller, Marjorie S. (1996a). Handling spelling errors in online catalog searches. Library Resources & Technical Services, 40,113-l 32. Drabenstott, Karen M., & Weller, Marjorie S. (1996b). Failure analysis of subject searches in a test of a new design for subject access to online catalogs. JournaZ of the American Society for Information Science, 47, 5 19-537. Efthimiadis, Efthimios N. (1990). The growth of the OPAC literature. Journal of the American Society for Information Science, 41, 342-347. Ellis, David. (1996). The dilemma of measurement in information retrieval research. Journal of the American Society for Information Science, 47,23-36. Ensor, Pat. (1992). User characteristics of keyword searching in an OPAC. College & Research Libraries, 53,72-80. Ferl, Terry Ellen, & Millsap, Larry. (1996). The knuckle-cracker’s dilemma: A transaction log study of OPAC subject searching. Information Technology & Libraries, 1.5, 8 l-91. Hakala, Juha. (1996). 239.50-1995: Info~ation retrieval protocol: An in~oduction to the standard and its usage