A scenario approach for assessing new business concepts

Manuscript: A scenario approach for assessing new business concepts1 Authors: Kalle Piirainen*, [email protected] Bio: Kalle Piirainen, M. Sc. (Tech.), works as a Research Assistant in the Laboratory of Innovation Management in the Department of Industrial Management at Lappeenranta University of Technology. His research interests include decision support and foresight in the management of technology.

Samuli Kortelainen, [email protected] Bio: Samuli Kortelainen, M. Sc. (Tech.), works as a PhD student at Lappeenranta University of Technology. His research areas include management of technology, innovation process, and group support systems. He works as an assistant in Technology Management and Decision Supports Systems courses at Lappeenranta University of Technology.

Kalle Elfvengren, [email protected] Bio: Bio: Kalle Elfvengren, Dr. Sc. (Tech.), works as a researcher in the Department of Industrial Management at Lappeenranta University of Technology, Finland. He teaches courses on Decision Support Systems and methods on Technology Management at the University. His areas of interest include group support systems, decision analysis and creative group work methods, and how these tools can be utilized in the early phases of the innovation process.

Markku Tuominen, [email protected] Bio: Markku Tuominen is a Professor and the Dean of the Faculty of Technology 1

An earlier version of this paper has been published in the Proceedings of the XVIII International Society for

Professional Innovation Management Annual Conference, 17-20th June 2007, Warsaw Poland.

Management at Lappeenranta University of Technology, Finland. Prof. Tuominen received the D.Sc. (Tech.) degree from Helsinki University of Technology, Finland, in 1980. Since 2001 Prof. Tuominen has been a member of the science and technology board of the Finnish Academy. His current research interests include computer aided strategic analysis, technology management, and decision support in engineering management. He has published widely in international journals on these research areas.

Address: Lappeenranta University of Technology, Faculty of Technology Management, Department of Industrial Management, Laboratory of Innovation Management, P.O. Box 20, FIN-53850 Lappeenranta, Finland

*Corresponding Author. Email: [email protected], Phone: +358 5 621 2671, mobile: +358 40 5838348, fax: +358 5 621 2667

Manuscript: A scenario approach for assessing new business concepts Category: Research paper Abstract: Purpose: The purpose of this paper is to examine challenges in the front end of innovation and to propose a scenario-based approach to alleviate some of these problems, particularly as regards uncertainty in opportunity recognition. Research design/methodology/approach: The paper answers the main research question through a literature review and a case study. The paper employs the design-oriented approach to propose an artifact which solves the underlined problem, and validate the artifact through the case study. Findings: According to the literature review, scenarios should offer a viable method for opportunity recognition. The case study supports the theoretical proposition, and suggests that scenarios can be used to alleviate the effect of uncertainty in the front end of innovation. Research limitations/Implications: The empirical results are limited to the level of proof-ofconcept. The scenario process as such was rated positively, which corresponds to the theory and previous similar experiments, but the benefits of the scenario approach have yet to be verified. Practical implications are a novel method for finding and assessing new business concepts. Contribution: The main contribution of this study is the formed process artifact to alleviate the challenges in the front end of innovation. The scenario approach can be used to probe the near future for business development purposes. Keywords: management of technology; technological forecasting; innovation process; new product development; scenarios; scenario planning; group support systems;

Manuscript: A scenario approach for assessing new business concepts

1 Introduction

Long lead-times and high costs characterize research & development projects in many industries, and most project costs are irrecoverable if for instance the path of technology changes during the project (Herstatt & Verworn, 2001; Herstatt et al. 2004). Innovation management literature regards the so-called front end of innovation, or the first phases of the innovation process, as very important for gaining success later on in the process, but also the most risk prone one (e.g. Herstatt et al. 2004; Koen et al. 2001). In this game of high stakes, it is impossible to forecast the outcome reliably, so it is reasonable to look for some ways of managing the uncertainty in the process. In the rapidly changing business environment, scenario planning in its different forms has recently gained attention as a versatile method for addressing the uncertainty. Various authors have written about the “haywire future”(e.g. Fatt. 2002), increasing uncertainty and new requirements for business management. Compared to more traditional and rigid methods of foresight, e.g. linear forecasting, modeling or trend analysis, the use of scenarios offers some important gains, such as, a built-in contingency perspective through multiple scenarios, increased general awareness of future options and increased decision making ability (e.g. Chermack, 2004; Coyle, 1997; Schwartz, 1996). In most sources, scenario planning is associated with organizational strategy and policy decisions, but there are also suggestions for widening the domain of scenario techniques for example to the field of product concept development (Kokkonen et al. 2005), road mapping (Naumanen, 2006) and other areas of innovation management. Benefits of scenario planning include more open-minded views to the future, learning to think ‘outside the box’during the process, and reduced uncertainty through multiple plausible scenarios, see e.g. (Ralston & Wilson, 2006; Chermack, 2004; van der Heijden et al. 2002). Based on the proposed benefits, it would seem plausible that a scenario-based approach to identifying new business ideas and concepts could present a rewarding addition for managing uncertainty in

the innovation process. This study builds on previous studies on scenario techniques and the innovation process, which have sought for means to improve the manageability of these processes. The purpose of this study is to investigate the scenario process in the front end of innovation (FEI) process (Herstatt & Verworn, 2001). The objective is to adapt techniques from the scenario process to avert some of the challenges commonly found in the front end of the innovation process. When considering the framework of innovation space, scenario techniques can be seen to be associated with strategic innovation, but this study aims at moving toward the product and service end of the spectrum. The study adopts the knowledge-based view of capabilities and competences as the driver of the innovation process, after Tidd et al. (2001). Additional attention is paid to the concept of the so-called fourth generation research and development (R&D) practices (Miller & Morris, 1999), which aim to leverage the convergence of different industries to create inimitable innovations. The main research question of this paper: ‘How can scenario-based techniques be used to alleviate problems resulting from uncertainty?’ The main question can be separated to two main parts; the implicit first part of the question being ‘what are the effects of uncertainty in the front end of innovation’. The answer to the first part lays the foundation for the answer to the second, main, part, which will be the main contribution of the study. The paper answers the question by describing an artifact, which is a computer-supported scenario process. Rowley (2002) and Yin (1994) among others have noted that case studies are well suited to answering questions beginning with “How?”. Thus, the case strategy is relevant for this paper: the first part of the research question is answered through a literature review on the effects of uncertainty on FEI, and the second part of the question is answered through the description of the artifact which demonstrates how the uncertainty in FEI can be alleviated. The validation of the artifact is based on accumulated research; the basic validation has been done through trial sessions and laboratory experiments

with graduate students (Piirainen et al. 2006; 2007). The purpose of this paper is to broaden the validation of the process through a real case of using the intuitive decision oriented scenario process, henceforth IDEAS-process. The paper offers a rich qualitative description (Eisenhardt & Graebner, 2007) of the session and employs basic triangulation techniques (Jack & Raturi, 2006) to enhance the validity of the conclusions. The study begins by discussing the front end of innovation, specifically idea generation and identification of business opportunities. The aim is to frame the challenges presented in these phases, to be able to identify requirements for support techniques. The second stage is an examination of the theory of scenario planning and methods to form a concept for assessing business opportunities with the support of scenario techniques. The third stage is to take the generic IDEAS-process and adapt it to opportunity recognition, and to put it to use in the case. Lastly, the paper closes with concluding remarks and directions for further research.

2 Process of innovation and opportunity identification 2.1 Innovation process and concepts

Innovations can be classified in relation to the target of the innovation; one proposition is a division into product, service, process and behavioral/organizational innovations (Tidd et al. 2001). Another major factor in classifications is the novelty of an innovation at a micro (endogenous) and macro (exogenous/external) level in relation to the innovating firm, as well as novelty in relation to the end market (e.g. Garcia and Calantone 2002). Put simply, incremental product variations are trivial compared to the development of new organizational structures and processes, which may not need only new skills and everyday adaptation, but also developing new perspectives and identities. Decision making, identifying and rating ideas in the outer segments of the organization’s knowledge domain becomes more unstructured and complex as existing ways of thinking may not support the new mental models required for reflecting on the new business opportunities.

The mainstream of innovation research has revolved around the process of innovation (Cooper, 1990). Figure 1 illustrates the process of new product development (NPD) and the primary tasks in each phase. Differing from the original stage-gate process, the idea generation and screening before the decision of starting the formal product development process can be divided into two parts forming “the fuzzy front end”(Herstatt and Verworn, 2001). The process requires product ideas, market knowledge and technological knowledge as input, which are then forged to product or business concepts. “Business”refers to a more general construct compared to the product concept, including new business models, business ideas, as well as product and service bundles. On the output side, the front end activities should result in a well-defined product concept and specification, backed up with a preliminary project plan for the rest of the development (Herstatt & Verworn, 2001; Koen et al. 2001; Khurana & Rosenthal, 1997). Fig. 1 about here

One interesting question is the position of the more or less formalized NPD process in an environment of swift technological advance and high uncertainty. In the presence of high uncertainty, strict process models do not perform according to expectations in terms of market success (Eisenhardt & Martin, 2000). In an oligopolistic market, longer gating processes including developing alternative projects and choice representing the “learn-before-doing approach” seem to be superior. When the market uncertainty rules and industry structures become more or less undefined, shorter processes and simple heuristics may ultimately lead to better outcomes. The concept of capabilities is connected to the innovation process through the organizational structures and mental models required for the innovation. Dynamic capabilities (Teece et al., 1997) are defined as a capability to use and develop new competencies for sustained competitive advantage over rival firms. In other terms it is a learned pattern of

routines through which the organization generates and modifies its operational routines in pursuit of improved effectiveness (Teece et al. 1997; Zollo & Winter, 1999; Kogut & Zander 1992). Competencies (Prahalad & Hamel, 1990) are distinct, hard-to-imitate skills, processes, structures or pools of knowledge, which are manifested in an ability to create advantage through for example rapid product changes, and which give an opportunity to invest in new markets flexibly (Javidan, 1998). Figure 2 illustrates the layered nature of innovation types in relation to the innovator. As many writers have operationalized the concept of capabilities as organizational structures, conventions or routines are pictured as the base of the figure. The formal processes are built on top of these structures. A routine, in this model is a more informal and intuitive structure, the formal processes are formal statements that are either direct descriptions of the routines. These two levels form the backend structure normally invisible to outsiders. Capabilities and their paths form the underlying logic of higher level operations. The relevance of these considerations of the relation between capabilities and innovation are better understood in the realm of 4th generation R&D (Miller & Morris’1999). 4th Generation R&D structures can be considered double-loop learning, where the aim is to find out the latent customer need, which guides the acquisition and building of capabilities to cater to these needs, and these capabilities in turn manifest themselves in the operations, as illustrated in Figure 2. Fig. 2 about here

2.2 Challenges in the front end activities

The challenges in the front end of innovation, including concept assessment, have been specifically discussed in NPD and front-end literature, but relating elements can be identified in R&D project selection and general innovation management literature. Quick examination

of the literature on the subject (Griffin & Page, 1996; Khurana & Rosenthal, 1997; Lehtoranta, 2005; Cooper et al. 2002a; 2002b; Zhang & Doll, 2001; Calantone et al. 2006; Hart et al. 2003) suggests that the challenges in the front end can be further divided to three categories, presented in Figure 2. The first level is the product/service, which represents the challenges closely associated with the offering itself and its qualities. The second level includes the technological challenges, which are broader challenges relating to the offering and the underlying technological aspects. The third level consists of the organizational aspects of executing the project and the capabilities held by the organization. In this study, the marketing function and process management are given less emphasis due to our concentration on the front-end activities. In the cited studies, the most common success factors include identifying the market need, proficient R&D/innovation process management, proficient marketing, and superior user-perceived qualities of the products (Cooper & Kleinschmidt, 1987; Cooper, 1999; Cooper et al. 2002a). In the present context the most interesting challenges are identifying the technological opportunities, the market need and the relation of the factors to the organization’s capabilities, a view true to the technological push perspective. If the dynamic capabilities are the basis of sustained technological know-how and capability, and they are path-dependent and form a path of possibilities by which technological progress is built (Cohen and Levinthal, 1990). Figure 3 is a derivation of Figure 2, and it presents the hierarchy of challenges in the assessment of business opportunities. At the technological level, the discrepancy between the views in the organization and the potential customer come into play. The customer has their distinct needs regarding technology, which may or may not be in possession by the organization. Similarly, there is the actual customer need, as the customer conceives it, and the image the organization has of the actual need. The discrepancy between the need and identifying it may come from deficient market analysis, misconception of the customer’s statement, or the customer’s inability to

formulate the present needs in an explicit manner. Lastly, there is the offering and the customers’perceptions of it, which may differ from the intended use or value proposition put forward by the developer. Fig. 3 about here

3 Scenarios and uncertainty in the innovation process

The evident uncertainty in the front-end activities poses a problem. Uncertainty about the future poses problems in selecting the most promising ideas and business opportunities for further development. Studies on forecasting future innovation suggest that managers tend to be more vulnerable to cognitive biases in the face of an unstructured decision, especially if the decision carries high financial risk, and while processes and structural methods alleviate the problems, they require strict adherence and supervision (Åsterbo & Koehler, 2007). Furthermore, even with strict and well-defined processes, the problem of market discontinuity still plagues industry (Paap & Katz, 2004). Many times incumbent companies are well aware of the drivers or even the technologies that will ultimately shake the market foundations, but for some reason fail to either take them seriously or link the technology and its potential to the competitive landscape and its changes (Paap and Katz, 2004). The fuzziness of the front end of innovation has been identified to be a result of uncertainty about several aspects of the idea and its future (Khurana & Rosenthal, 1997). The uncertainties in the front end include technological uncertainty, unclear product strategy, and uncertain information about customers and competition; all factors that can be linked to path dependency and resource constraints (Zhang and Doll, 2001). Path dependency paves the way for using scenarios in the innovation process. Scenario planning has an advantage over linear forecasting and foresight methods in complex, poorly structured situations of high uncertainty. Innovation management is said to be an activity with high uncertainty in a possible discontinuity, and the path of possibilities is hard to predict accurately, which would

seem to point to the generally attributed strengths of the scenario method. 3.1 Scenarios Definition Scenarios have been given a variety of definitions in the literature (Chermack 2004; Coyle 2004; Kahn & Wiener, 1967; Schwartz 1996; Schoemaker 1991; 1995). From this literature we define scenarios as a set of separate, logical paths of development, which lead from the present to a defined state in the future. Furthermore, scenarios are not descriptions of a certain situation some time in the future, nor are they a simple extrapolation of past and present trends. We now refer to a single scenario is referred to as a scenario, and multiple scenarios developed as a set are referred to as scenarios. The other dimension in scenarios is the relationship of entities in a scenario set (Schwartz 1996; Blanning & Reinig, 2005). We stipulate that these drivers create movement in the operational field, which can be reduced to a chain of related events. These chains of events are in turn labeled as scenarios, leading from the present status quo to the defined end state during the time span of the respective scenarios. It is not assumed that a driver has one defined state, but multiple possible states. Thus, a driver can influence multiple events, which may or may not be inconsistent in a given set of scenarios, but of course, according to the definition of a scenario, not in a single scenario.

3.2 Process and methods

Despite obvious differences in approaches, there are common elements across the field of scenario planning. The IDEAS-process used as a reference in this study is a development of this generic scenario process by Piirainen et al. (2006). The name IDEAS stands for the main tasks in the process: I for objectIve definition, D for Drivers of change, E for identification of plausible future Events, A for Assessment of the scenario components and finally S for formulating the Scenarios. The process is intuitive and utilizes a group support system (GSS) to facilitate the group process and synthesis of expert opinion. The possibilities and challenges of using a GSS in this sort of activity have been

discussed in the literature (Blanning & Reinig, 2005; Piirainen et al. 2006), and are not the focal area of this study. Generally, group support systems are a collection of applications aimed to facilitate group work and communication similar to groupware (Turban et al. 2005; Jessup & Valacich, 1999). In the hierarchy of decision support systems (DSS), the GSS is placed in the branch of communication driven DSSs (Power, 2002). GSS implementations generally feature tools for idea generation, prioritization, commenting and discussion, packaged into a software suite (Turban et al. 2005). The aim of using the GSS in this study is to make the process more efficient by harvesting the benefits modern support methods can offer. The suggested benefits of utilizing GSS include improved process structuring and increased efficiency through parallel input and group memory, greater commitment and possibility for democratic collaboration through anonymity, increased quality of results through analytic tools, a possibility to engage a large group through the system, better handling of the results through automatic recording of the session input, and as a result, more efficient sessions and higher quality results compared to unsupported face-to-face sessions. (Turban et al. 2005; Garavelli, 2002; Huang et al. 2002; Kwok & Khalifa, 1998) Previous experience in the laboratory suggests that the GSS is a viable option for facilitating the scenario process, and as the possibility to use such methods exists, this study adopts the support framework (Piirainen et al. 2007). The main reasons for utilizing the GSS, besides the general benefits discussed above, are the possibility to engage participants from different organizations through anonymous communication and to achieve an efficient gathering of information about the needs, the technological path etc. from a diverse group of experts in a manageable time. Figure 4 illustrates the IDEAS-process, which forms the basis for building the process for identifying new business opportunities. Fig. 4 about here

The GSS process has been previously used for charting external uncertainties in an organizational strategy context (Piirainen et al. 2007). In the first phase the outline, scope and method of the scenario process have to be outlined before the actual workshops. The actual scenario session then begins in the second phase with a discussion and identification of the drivers. After the drivers have been identified, they are prioritized through voting, and the most important drivers are circulated to the participants for the third phase. The next phase is identifying the events the drivers may trigger during the period of the scenarios. The events may be brainstormed and integrated into one set, or depending on the required scope, the participants may be led by instituting categories for the events to draw attention to the preferred direction. Depending on the resulting number of events, they may be prioritized again or left as is and moved to ranking. The chosen events are ranked by alternative analysis according to their probability of occurrence and their impact on the organization in question. The fourth step is grouping the events to sets, which form the backbone of the scenarios. The grouping may be done graphically from a scatter plot, as Blanning and Reinig (2005) originally proposed, or for example by cluster analysis. In the end of the session, the resulting event sets are examined in the group and can be further arranged in a chronological order or formed to a concept map. When the workshop ends, the researcher should have the scenarios in the form of event sets which are to be formed to the final scenarios.

3.3 Scenario approach to new business concepts

In a general business strategy context, the drivers are the key players and uncertainties which shape the environment. In the context of innovation, there are inherent uncertainties stemming from the customer, competition and technology, which can perhaps be viewed as a sort of meta-drivers. Generally, the capabilities, competence and technological knowledge possessed by the firm should be well defined. One of the most important factors of success is the ability to meet customer demand, which suggests that the process should include as an input the

knowledge of what is and will be possible technically, and what will be demanded by the customers. Using the resource-based view analysis, the drivers depict the present resource and environmental constraints, as well as the market drivers which shape the demand. The opportunities identified from these factors will give a map to develop the competence to meet the demand with technological expertise. As stated, scenarios comprise traceable chains of events triggered by drivers. In the situation of assessing new business concepts, the drivers are of technological, consumer need, and business nature. The events are business opportunities or are translatable to such. The opportunities may be new or newly defined customer needs, new technological opportunities opened by the unique path of the organization, or new business/revenue models based on technology or demand side factors. Of course, one consideration or possibility is whether there would be a need to develop multiple separate scenario sets from different viewpoints to be compared with each other. The basic IDEAS framework has been adjusted. The modifications primarily concern the objective setting, as in this case the objective is specifically to identify feasible business concepts instead of scenario paths, but using an approach leaning on scenarios as the basic principle. When the wanted objective is set, the process will proceed to identifying the drivers that will result in business opportunities. The third main event is the identification of the business opportunities based on the drivers. When the events have been identified, they will be evaluated against each other in terms of feasibility, potential and impact with a set of criteria, which may be at least partially derived from NPD success factors. Evaluation and discussion will conclude the session and will give a signal to proceed to forming the scenarios for the paths of business opportunities. As shown, the difference is in the context of the process more than anywhere else; normally scenarios are chains of events and now they are chains of business opportunities. Thus the scenarios allow a twofold use, either as a roadmap for capability development, or a

roadmap for reflecting on the feasibility of a given business idea or opportunity against plausible future development, to assess its long term impact and the possibilities it will create for further business.

4 Case description

The test session was held in the Laboratory of Innovation Management and Group Decision Making of the Department of Industrial Management at Lappeenranta University of Technology. The Laboratory is used for teaching and research in the field of group decision support processes and systems. It has been utilized for defining the critical success factors of companies, selecting strategies and executing “SWOT”analyses, generating concepts for new products, defining new concepts, selecting projects, carrying out customer need assessment, promoting the selection process of R&D measures, and specifying the requirements for systems that are to be purchased. The laboratory is a PC-equipped local area network-based meeting room designed especially for decision making, and various commonly used decision support software have been installed. The extensive collection of decision support tools gives a solid foundation to a decision room. The software bundle includes tools for system dynamics, the analytic hierarchy process, causal mapping and decision trees.

4.1 Methodology description

The subject outline for the workshop on creating technology scenarios and identifying future business opportunities was to seek new opportunities in the intersection of a manufacturing industry and a complementing sector. For this setting, the participants were selected so that the group included participants from each industry, as well as academics and general experts in the field. The process followed the outline depicted in Figure 4. The problem for the group was to identify and assess new business concepts in the intersection of two manufacturing industries.

The study started with a presentation of the aims of the research project and scenarios in general, followed by the outline and objectives of the study (held in a workshop setting). After the presentation, the first actual phase was the identification of drivers, which was done with the GroupSystems categorizer tool, with preset categories for the PESTEL framework. After idea generation, the drivers were discussed by category, removing duplicates and editing ambiguous wording. After the discussion, the drivers were prioritized by voting on one category at a time. When the drivers had been sorted, the work proceeded to identify the events, or in this case, business opportunities. The identification was completed in two phases to interface the idea generation better with the drivers. The basic questions to aid the event recognition were “What kind of business opportunities can the identified trends open in ten years’time?” “What events will these opportunities create?” After idea generation, the events were discussed and clarified in the group. The phase of evaluation is paramount to the final scenarios, as the evaluations form the basis for grouping the events to scenarios. Differing from previous setups, the events were evaluated in three dimensions using the GroupSystems Alternative Analysis -tool. The first dimension was the probability of occurrence for the event, but the impact was split to two dimensions, which differs from the original set-up (Piirainen et al. 2006; 2007). The second dimension was the impact each event will have in the business and earning logic of the industry implementing the product, and the third dimension was the impact an event will have on the earning logic of the supplying industry and technology. This dual dimension was selected because of the setting explained above, first to separate the consideration of impact and usability clearly to one perspective at a time and to serve the interest groups equally in terms of the results. The major practical differences to the previously reported study workshops were that an auxiliary presentation was on a smart board alongside the common video screen, where

each phase was briefly outlined. The second difference was that brainstorming of events was initially completed with pen and paper with a print of the prioritized drivers and then switched to brainstorming the events or future business opportunities with the GSS. The reason for the two-part event identification was participant input from previous sessions, which stated that the events were not necessarily well connected with the drivers. (Piirainen et al. 2007) The study workshop concluded after the evaluation and presentation of the intermediate results. The events were presented in a scatter plot, the grouping was discussed preliminarily, and the voting results were examined. The final grouping of the scenario sets was left to be done with cluster analysis, but the possible themes were discussed freely and the scenario writers took notes of the comments. The final task in the workshop was to fill in a questionnaire to evaluate the workshop and the methods.

4.2 Results

The session participants filled in a questionnaire after the session as a standard measure to get feedback about the workshop arrangements and to validate the artifact. The main instruments in the questionnaire were satisfaction with the process, tools and the results, as seen in Table 1. Meeting satisfaction is an important instrument in the validation of the artifact (Briggs et al. 2003, Reinig, 2003). In the absence of meeting satisfaction, it is unlikely that the users will adopt the artifact, regardless of any productivity gains that might be realized (Reinig, 2003). According to Briggs et al. (2003), there are at least two aspects of a meeting, which a participant could feel satisfaction with: the meeting outcomes and the process by which the outcomes are attained. In addition to the questionnaire, the authors recorded observations during the session and also documented some reflections afterwards. The aim was to enrich the case description and to confirm or dispute the questionnaire data in the spirit on triangulation (Jack & Raturi, 2006). The observations and reflections were divided to four identifiable levels affecting the success of the session, including meeting satisfaction as specified above. The first level was

the technical implementation of the process, including the physical environment and the technical aspects of the GSS system. The second was the process level, the outline of the workshop, proper task definition, support in completing the tasks, and facilitation. The third level was the participant experience, satisfaction with the previous two facets in the workshop and the level of satisfaction with the process. The fourth level was the participant experience and satisfaction with the results of the session. Together these levels contributed to the overall satisfaction and willingness to adopt the artifact. The validation was based on the questionnaire filled in after the session and direct/participative observation during the session and the reflections. Table 1 presents the questionnaire items. All the items were evaluated with a Likerttype 10-step scale and each category held at least one negatively phrased item to prevent positive response bias in the group. Excluding the researchers, the number of respondents was n=7, which is very low and defeats the use of many statistical tests, or affects their reliability significantly. However, a risk was taken to examine the results further, and to achieve comparable results with the results of previous similar sessions, the correlations between the items were calculated. The calculations were computed with a significance level p=0.05 where applicable, and the calculated correlations were 2-tailed Spearman’s Rho. It needs to be noted that the small sample will degrade the validity and reliability of the correlation coefficients, and should be considered only as suggestive evidence. Table 1 about here

Examining the basic statistics, the participants were generally somewhat familiar with the GSS environment and were more skeptical about the effect of the support methods compared to previous studies, and even slightly agreed that an unsupported session would give better results. On average, the basic premises seemed to be in order, as the session goals seemed to be clear to the participants, the goals were apparently reached and the most

important factors were also included in the scenarios. What can be seen as a negative point was that the group only slightly disagreed that the time was too short, even though they mostly agreed that there was enough time for the evaluation of the events. Again, on average, the results were seen as trustworthy, but the participants only slightly agreed that it was because of the process, but then again agreed strongly that the work methods brought trustworthiness to the results. Judging by the answers, the GSS seems to have had a positive impact on the results, as the GSS seems to have fitted effortlessly to the process as a support tool and have helped in systemizing the process and creating trustworthy results. As mentioned above, the sample was too small to support valid variance or regression analysis, but Spearman’s correlations were used to examine dependencies between the items. The correlation table of 30 times 30 cells is omitted from the report, but the results can be summarized briefly as follows. Firstly, trust in the results and experience in working with a GSS are associated, which could suggest some bias in the answers. The most important factors in reaching the goals of a session would be equal treatment of each participant’s contribution, democratic treatment of participants, commitment to the process, and systematic working. Concerning the practical aspects, the more systematic the process was seen to be, and the more time the participants felt they had at their disposal to complete the phases, the better the trustworthiness of the results. These factors lead to a conclusion that the GSS had a positive impact through systemizing the process and anonymity, which helped in democratizing the process and supported the evaluation of the events. Observations of the case session and the reflections have been condensed to Table 2. The observations have been gathered by the authors, two of whom participated in the session. One author was present at the sessions as a participant and one as the technical facilitator. The table also incorporates open-ended answers from the participants as entered in the postsession survey. To better distinguish between fact and reflection, the direct participant

feedback, through open-ended items in the questionnaire, is presented in the boldface style, direct observation in normal style and reflective observations in italicized style. Table 2 about here

In the open-ended answers, the input concentrated on the process level, although some of the comments were hard to isolate. It is probably safe to assume that most of the input covered the participant experience, as well as the process and technical levels. The openended answers were in line with the questionnaire results, satisfaction with the process and arrangements was high. On the other hand, the results may also indicate a slight positive response bias, as negative comments were absent and critique or suggestions for improvement scarce. Concerning the recorded critique, it would seem that the relatively strict timetable was one of the main causes of critique. The introduction and session time interventions are of great importance to the process and the content, as also recorded in the participant feedback. Especially the introduction has surprising power to steer the group and to introduce ideas. The steering function can be seen as an opportunity to create an innovative and forward looking attitude in the group, but the power also brings responsibility to the facilitator not to exploit the leverage to get convenient results from the group. An additional twist in the case was the inclusion of representatives of three different industries and multiple organizations. In fact, most of the participants were from different organizations or at least departments, except for the authors who were present. Based on the feedback and observations, the system functioned well under the duress, the participants seemed to be well motivated, open toward the group and constructive, and all the participants contributed significantly to the session. Comparing the questionnaire, open-ended answers, and observations, the effect of the GSS dominates in the questionnaire, but the rest of the data points out that the technical level

of the session is just one aspect. Overall, the results correlate reasonably between the data sources and researchers. Based on the feedback, it can be summarized that the GSS enabled efficient working and offered usable tools, but the real value comes from the process used together with the GSS.

4.3. Discussion

To take an overall look at the results, the content aspect of the artifact remains somewhat a black box, despite the overall favorable results. The picture that can be formed from the answers seems to be similar to previous sessions. The overall outlook is positive in general, with some concerns, such as the time available. The interviewed participants of one session were not able to distinguish clearly whether the reason for the high ratings was the environment, session facilitation, work methods, or the process, and stated that the positive response to the session was not due to any particular aspect, but the session as a whole (Piirainen et al. 2006). One aspect that tends to be forgotten in formal evaluation is the impact of finishing touches in the workshop, it may well be that the possible positive response bias noted above may be a side effect of the workshop arrangements. Concerning the process, the results are somewhat mixed; on one hand the results were seen as fairly trustworthy and relevant, the goals of the workshop were met, and the process seemed to be systematical and logical, according to the survey. On the other hand, the group tended to slightly disagree that the trust in the results would be down to the process, but rather support methods. Of course this brings forward the questions of how the survey items were understood by the group and what is the line between ‘work method’ and ‘process’. Nevertheless, it would seem that the process and technical level together make up the participant experience, as one is nothing without the other. A tentative conclusion would be that using the presented artifact would be a reasonable tool for assessing future business opportunities. The results also suggest that the

method of constructing scenarios in a process supported with a GSS seems to be a viable option and would lead to reasonable results. The IDEAS process has been benchmarked against a well known scenario method called the field anomaly relaxation (FAR) in a recent study by Kokkonen et al. (2008). The authors conclude that both scenario methods have their own strengths and weaknesses, as where the FAR method provides certain analytical rigor and traceability of the causal chain from the drivers to the final scenarios, the IDEAS method enables fast and convenient gathering of data and efficient drafting of scenarios. At the content level, the IDEAS method is more driven by the data collected from the participants, whereas FAR tends to rely more on the scenario practitioners’analytical abilities. As for other competition, the literature on technology management and R&D project selection has proposed multiple comparable and complementary tools. Most often the concept assessment is tied to a formal product development process inside the organization (Cooper, et al. 2002a, 2002b; Ettlie & Elsenbach 2007). Herps et al. (2003) give an example of a practical framework for product development decisions in their paper, where the decision making is a part of the stage-gate process and employs relatively simple decision heuristics in each step. Cooper (1999) has surveyed the methods for project selection and has found that most real world decisions are based on relatively simple methods, possibly used in combinations. Most companies do not seem to use methodologically advanced techniques for technological foresight or forecasting. Additionally, the sensitivity of linear forecasting to the selection of model and basic assumptions has been discussed, as well as the problems with cash flow and probability estimates in real options (e.g. Piirainen et al. 2007). Either way, the additional benefit of the scenario approach is that it allows, even demands, exploring multiple plausible paths of development, which also forces the decision maker to consider the options and different concepts in different plausible market and technological conditions. This is not to say that scenarios are necessarily superior to the mentioned methods, but they can be used in conjunction with other methods, to explore the possible future, to form contingency plans

and to test assumptions used in other concept assessment and selection techniques. If the aspect of support systems is added to the consideration, the GSS as such is a viable tool to facilitate group work in using scoring models and similar techniques, but the achievable gains in participant experience, process efficiency and to some amount content, are achievable by correct implementation of GSS tools with most problem contexts alike. Using the GSS as a vehicle for facilitating the group work in this case does not improve or worsen the relative advantage offered by scenarios over other concept assessment methods.

4.3 Validity and limitations

When considering the validity of the results, it is evident that the sample size may cast a shadow over the retrieved results and conclusions. Furthermore, some available statistical tools had to be dismissed because of the limited data, so also the explanatory power of the survey is limited. This paper has strived to overcome the limitation by using data source and researcher triangulation through observations by multiple researchers as a complement to a questionnaire. One important limitation is that this paper has evaluated an artifact, the IDEAS-process, which also means that the results concern the process, but the results of content aspect of the artifact remains a black box. The participants gave a favorable rating to the results, but the open-ended items or the observations are not enough to evaluate the content and results of the session. Regarding design research, particularly constructive research, it has been proposed that if an artifact passes a weak market test, that is a real manager adopts the proposed practice, the artifact can be deemed valid (Lukka & Kasanen, 1995). The participants of the session included industrial managers and experts not associated with the researchers, which speaks for the proposition that the session approaches a real setup, but if the market test demands that the construct is adopted to continual use, the result is inconclusive. As it seemed that the participants were pleased with the sessions and its results, and the results are in line with previous findings from similar sessions and with the theory proposition, it can be

suggested that the IDEAS process, and the scenario method in general can be used in identifying and assessing future business concepts. Moreover, the results generally correspond well with the presented theoretical framework and previous results presented in Piirainen et al. (2006). This correspondence suggests that the process is repeatable to some extent, and the results concerning the artifact would be similar in different contexts. On the whole, the accumulation of research referred to in the introduction indicates that the process construct works as intended even different problem and organizational contexts, which speaks for analytic generalization (Yin, 1994; Eisenhardt & Graebner, 2007). With respect to limitations, the context of the testing was technological scenarios in the intersection of a traditional manufacturing industry and a complementary industry, and the participants were researchers, industry experts and managers from large incumbent industrial companies. An obvious limitation is that the result, if generalized, would apply mostly to a population with an engineering background and experience in large public or industrial enterprises, and a situation where there are participants from different organizations working for common scenarios, i.e. the characteristic of convergent innovation is present. A further limitation is that the results can be applied automatically only to facilitated face-to-face GSSsupported scenario workshops. Although the results might be similar in a different setting, the results do not scale up. The same is true of scenario methods, as the base of the workshop is a generic scenario process, but there is no guarantee that the results would apply to differently composed scenarios.

4.4 Significance and contribution

The concept of using scenarios in NPD may be novel, but not without precedent. Roadmaps of different flavors have been a recommended practice for some considerable time, and scenarios have recently gained attention in the context of innovation and technology management (Naumanen, 2006; Drew, 2006). The question is then, what sets this study apart

from the aforementioned? The immediate contribution of this paper is to present a practical framework method for the technology-oriented scenario process and evaluation for the said artifact. The results can be interpreted as a type of proof-of-concept. On this basis, the main significance of these results is that they shed some light on the real world applicability of scenarios as a tool in the innovation process. However, as discussed, the results are somewhat tentative and do not necessarily predict the performance of the construct outside the test setting accurately. What is needed is to consider the actual effect of scenarios on NPD in a company and to develop the practice of technology-oriented scenarios further.

5 Conclusion

The purpose of this paper was to examine the effect of uncertainty in the front end of innovation and to test whether scenarios could be used to moderate uncertainty. The literature review revealed that the effect of uncertainty in the front end is a well recognized problem and its effects range from NPD projects not meeting their goals to complete failure of new products. Much of the popular research has focused on making the lead time and process of NPD shorter and more predictable, thus also reducing uncertainty in the process. However, scenarios have additional benefits in recognizing the effect of path dependency and different possible paths of development, complementing the existing research on best practices in NPD. The study presented an artifact of a scenario process adapted for assessing new business concepts through technology-oriented scenarios. The paper concentrated on adapting the IDEAS-process to be used in the identification and assessment of business concepts in the front end of innovation. The end product of the study is a process and practice for technologically-oriented scenario formulation. The validity of the construct and testing was discussed. The drawn conclusions are tentative, but the artifact received a positive response from the participants. This result may

indicate that the conception of utilizing scenarios in assessing business concepts indeed has real world credibility. However, the results do not tell the net effect of this practice on profitability of NPD and whether a real company should decide to adopt the proposed process. It should be kept in mind that the paper studied a response to an artificial session, and there is no guarantee that the results of the scenarios would be used in management decision making. If the scenario approach is accepted, it may have multiple benefits. Scenarios in general and as a process are thought to have an impact on the mental models of the participants in addition to the scenarios. The method provides an opportunity to bring together technical, research and management staff to ponder about the future, which may help in securing top management support and integrating the NPD effort in the strategy, whether different wings of the organization bring their respective perspective to the process. The positive welcome of the process opens up avenues for further research. The feasibility of the concept of using scenarios as an approach for assessing new business concepts was rated positively by the participants in the test sessions, so the road is open to include scenarios in further research and as a practical arsenal of technology management.

References and Notes

Blanning, R. W. & Reinig, B. A. (2005) A Framework for Conducting Political Event Analysis Using Group Support Systems, Decision Support Systems, Vol. 38, pp. 511-527. Briggs, R. O. de Vreede G.-J. & Reinig, B. A. (2003) A Theory and Measurement of Meeting Satisfaction, in the Proceedings of 36th Hawaii International Conference on System Sciences, Calantone, R. J., Chan, K. & Cui A. S. (2006) Decomposing Product Innovativeness and Its Effects on New Product Success, The Journal of Product Innovation Management, Vol. 23, Issue 5, pp. 408-421. Chermack, T. J. (2004) Improving decision-making with scenario planning. Futures, Vol. 36, pp. 295-309.

Cohen, W. M. & Levinthal, D. A. (1990) Absorptive Capacity: A new Perspective on Learning and Innovation, Administrative Science Quarterly, Vol. 35, pp. 128-152 Cooper, R. G. (1990) Stage gate systems: A new tool for managing new products, Business Horizons, Vol. 33, Issue 3, pp. 44-54. Cooper, R.G. (1993) Winning at New Products: Accelerating the Process from Idea to Launch, 2nd ed., Addison-Wesley, 325 p. Cooper R.G. (1999) The invisible success factors in product innovation, Journal of Product Innovation Management, Vol. 16, Issue 2, pp. 115-133. Cooper, R. G. and Kleinschmidt, E. J. (1987) New Products: What Separates Winners from Losers? Journal of Product Innovation Management, Vol. 4, pp. 169-184. Cooper, R. G., Edgett, S. J., Kleinschmidt, E. J. (1999) New Product Portfolio Management: Practices and Performance, Journal of Product Innovation Management, Vol. 16, pp. 333351. Cooper, R.G., Edgett & Kleinschmidt, E. (2002a) Optimizing the Stage-Gate Process: What Best Companies Are Doing –Part I, Research Technology Management, Vol. 45, Issue 5. Cooper, R.G., Edgett & Kleinschmidt, E. (2002b) Optimizing the Stage-Gate Process: What Best Companies Are Doing –Part II, Research Technology Management, Vol. 45, Issue 5. Coyle, G. (1997) The nature and value of futures studies or do futures have a future?, Futures, Vol. 29, No. 1, pp. 73-93. Coyle, G. (2004) Practical Strategy: Structured Tools and Techniques, Glasgow: Pearson Education Ltd., 308 p. Drew, S. A. (2006) Building technology foresight: using scenarios to embrace innovation, European Journal of Innovation Management, Vol. 9, Issue 3, pp. 241-257. Eisenhardt, K. M. & Graebner, M. E. (2007) Theory Building from Cases: Opportunities and Challenges, Academy of Management, Journal, Vol. 50, No. 1, pp. 25-32. Eisenhardt, K. M. & Martin, J. A. (2000) Dynamic Capabilities: What are they? Strategic

Management Journal, Vol. 21, pp. 1105-1121. Ettlie, J. E. & Elsenbach, J. M. (2007) Modified Stage-Gate® Regimes in New Product Development, The Journal of Product Innovation Management, Vol. 24, pp. 20-33. Garavelli, A. C. et al., (2002) Managing knowledge transfer by knowledge technologies, Technovation, Vol. 22, pp. 269-279 Fatt, J. P. T. (2002) Wanted: Business Leaders in a Haywire Future, Management Research News, Vil. 25, No. 6/7, pp. 84-91. Garcia, R. & Calantone, R. (2002) A critical look at technological innovation typology and innovativeness terminology: a literature review, The Journal of Product Innovation Management, Vol. 19, pp. 110-132. Griffin, A. & Page, A. L. (1996) PDMA Success Measurement Project: Recommended Measures for Product Development and Failure, The Journal of Innovation Management, Vol. 13, pp. 478-496. Hart, S., Hultink, J., Tzokas, N. & Commandeur, H. R. (2003) Industrial Companies’ Evaluation Criteria in New Product Development Gates, The Journal of Product Innovation Management, Vol. 20, pp. 22-36. Herstatt, C., Verworn, B. (2001) The “Fuzzy Front End” of Innovation. Working Paper No. 4, Department of Technology and Innovation Management, Technical University of Hamburg, 24 p. Herstatt, C., Verworn, B., Nagahira, A. (2004) Reducing Project Related Uncertainty in the ‘Fuzzy Front End’of Innovation: A Comparison of German and Japanese Product Innovation Projects, International Journal of Product Development, Vol. 1, Issue 1. pp. 43-65 Huang, W. W. Wei, K-K. Watson, R. T. Tan, B. C. Y. (2002) Supporting virtual teambuilding with a GSS: an empirical investigation, Decision Support Systems, Vol. 34, pp. 359367 Jack, E. P. & Raturi, A. S. (2006) Lessons learned from methodological triangulation in

management research, Management Research News, Vol. 29, No. 6, pp. 345-357. Javidan, M (1998) Core Competence: What Does it Mean in Practice?, Long Range Planning, Vol. 31, Issue 1, pp. 60-71. Jessup L. & Valacich, J. (1999) Information Systems Foundations, Prentice Hall College Div., 300 p. Herps, J. M. J., van Mal, H. H. & Halman, J. I. M. (2003) The process of Selecting Technology Development Projects: A Practical Framework, Management Research News, Vol. 26, No.8, pp. 1-15. Kahn, H. & Wiener, A. l. (1967) The Year 2000: A Framework for Speculation on the Next Thirty-Three Years, London: Collier-Macmillan Limited, 460 p. Khurana, A. & Rosenthal, S. R. (1997) Integrating the Fuzzy Front End of New Product Development, Sloan Management Review, Winter/1997, pp. 103-120. Koen, P., Ajamian, G., Burkhart, R. Clamen, A., Davison, J., D’Amore, R., Elkins, C., Herald, K., Incorvia, M., Johnson, A., Karol, R., Seibert, R., Slavejkov, A. & Wagner, K. (2001) Providing Clarity and a Common Language to the “Fuzzy Front End”, ResearchTechnology Management, Vol. 44, No. 2, pp. 46-55. Kogut, B. & Zander, U. (1992) Knowledge of the Firm, Combinative Capabilities, and the Replication of Technology, Organization Science, Vol. 3, Issue 5, pp. 383-397. Kokkonen, V. Kuuva, M. Leppimäki, S. Lähteinen, V. Meristö, T. Piira, S. and Sääskilahti, M. (2005) Visioiva tuotekonseptointi: Työkalu tutkimus- ja kehitystoiminnan ohjaamiseen (Vision

Driven

Product

Concept

Development:

A

Tool

for

Managing

R&D),

Teknologiateollisuuden julkaisuja 4/2005, Helsinki, FIN: Teknologiateollisuus ry. 341 p. (in Finnish) Kokkonen, K., Piirainen, K. & Kässi, T. (2008) E-business opportunities in the Finnish forest sector –a multi-method scenario study in the Proceedings of XVIII International Conference of International Society for Professional Innovation Management, Tours, France.

Kwok R. C. W. & Khalifa, M. (1998) Effect of GSS on Knowledge Acquisition, Information & Management, Vol. 34, pp. 307-315 Lehtoranta, O. (2005) Determinants of Innovation and the Economic Growth of Innovators: tracing the evolution of innovative firms over time, VTT Working Papers 25, 34p. Lukka, K. & Kasanen, E. (1995) The Problem of Generalizability: anecdotes and evidence in accounting research, Accounting, Auditing & Accountability Journal, Vol. 8, Issue 4, pp. 7190. Miller, W. L. and Morris, L. (1999) Fourth Generation R&D: Managing Knowledge, Technology and Innovation, New York: John Wiley, 347 p. Naumanen, M. (2006) Roadmap – Kartta menestykseen (Roadmap – The Map for Success), Helsinki, FIN: MET Kustannus Oy, 91 p. (bilingual in Finnish and English) Paap, J. & Katz, R. (2004) Anticipating Disruptive Innovation: Predicting the “unpredictable”, Research & Technology Management, September-October/2004, pp. 13-22. Piirainen, K., Kortelainen, S., Elfvengren, K. & Tuominen, M. (2006) A Framework for Utilizing Group Support Systems in Scenario Process, in the Proceedings of Frontiers of eBusiness Research Conference, Tampere, Finland. Piirainen, K., Tuominen, M., Elfvengren, K., Kortelainen, S. & Niemistö, V.-P. (2007) Developing Support for Scenario Process: A Scenario Study on Lappeenranta University of Technology from 2006 to 2016, Lappeenranta University of Technology, Department of Industrial Engineering, Research Report 182, 46 p. Power, D. J. (2002) Decision Support Systems, Concepts and Resources for Managers, Westport CT: Quorum Books, 252 p. Prahalad, C. K. & Hamel, G. (1990) The Core Competence of the Corporation, Harvard Business Review, May/June 1990, pp. 79-91. Ralston, B. & Wilson, I. (2006) The Scenario-Planning Handbook: A Practitioner’s Guide to Developing and Using Scenarios to Direct Strategy in Today’s Uncertain Times,

Crawfordsville IN, Thomson/South-Western, 256 p. Reinig, B. A. (2003) Toward an Understanding of Satisfaction with the Process and Outcomes of Teamwork, Journal of Management Information Systems, Vol. 19, No. 4, pp. 6583. Rowley, J. (2002) Using Case Studies in Research, Management Research News, Vol. 25, Issue 1, pp. 16-27. Schoemaker, P. J. H. (1991) When and How to Use Scenario Planning: A Heuristic Approach with Illustration, Journal of Forecasting, Vol. 10, Issue 6, pp. 549-564. Schoemaker, P. J. H. (1995) Scenario Planning: A Tool for Strategic Thinking, Sloan Management Review, Vol. 36, Issue 2, pp. 25-40. Schwartz, P. (1996) The Art of the Long View: Planning for the Future in an Uncertain World, New York, NY: Doubleday Dell Publishing Inc., 272 p. Teece, D. Pisano, G. & Shuen, A. (1997) Dynamic Capabilities and Strategic Management, Strategic Management Journal, Vol. 18, Issue 7, pp. 509-533. Tidd, J., Bessant, J. and Pavitt, K. (2001) Managing Innovation: Integrating Technological, Market and Organizational Change, 2nd Ed., Chichester, UK: John Wiley & Sons Ltd. 381 p. Turban, E., Aronson, J. & Liang, T.-P. (2005) Decision Support Systems and Intelligent Systems, 7th Ed. A Simon & Schuster Company, 848 p van der Heijden, K. Bradfield, R. George, B. Cairns G. and Wright, G. (2002) The Sixth Sense –Accelerating Organizational Learning with Scenarios, Chichester: John Wiley & Sons, 307 p. Yin, R. K. (1994) Case Study Research: Design and Methods, 2nd Edition, Thousand Oaks, CA: Sage Publishing, 153 p. Zhang, Z. & Doll, W. J. (2001) The fuzzy front end and success of new product development, European Journal of Innovation Management, Vol. 4, Issue 2, pp. 95-112. Zollo, M. & Winter, S. (1999) From Organizational Routines to Dynamic Capabilities, A

Working Paper of the Reginald H. Jones Center, The Wharton School of University of Pennsylvania, WP99-07, 38 p. Åsterbo, T. & Koehler, D. J. (2007) Calibration Accuracy of a Judgmental Process that Predicts the Commercial Success of New Product Ideas, Journal of Behavioral Decision Making, Vol. 20, Issue 4, pp. 381-403.

Figure 1. The process of innovation and the fuzzy front end (Herstatt & Verworn, 2001) Phase I - Market knowledge - Technological knowledge - Technological and market capabilities - Existing projects

Inputs

- Idea generation (market, technology and cost orientation) - Idea assessment (attractiveness/risk) - Alignment with present portfolio

Phase II - Product concept definition - Market analyses - Product planning - Product architecture and specifications

“The Fuzzy Front End”

Phase III - Platform concepts - Product development - Product concepts - Design reviews - Service concepts - Industrial design - Product specifications - Project outlines

Outputs

Phase IV - Prototype testing - Market tests - Finalization of design - Preparation for production

Phase V - Market launch - After market activities

Figure 2. Innovations by target and the level of organizational impact Organizational impact

Product innovations

Products

Service innovations

Services Visible to outside Invisible

Process innovations

Organizational / behavioral innovations

Formal processes

Competencies Capabilities Organizational structures and routines Individual beliefs, values, traits

Figure 3. Challenges in innovation in terms of organizational levels (Cooper, 1993; Teece et al. 1997; Zhang & Doll, 2001)

Offering (intended)

Perceived Customer need

Technological paths Available technologies based on capabilities

Offering (perceived)

Actual Customer need

Technological paths Needed technological paths based on competition

Capability paths The available paths based on current capabilities and competence

Figure 4. The IDEAS-process (Piirainen et al. 2006; 2007) Pre-Phase - Objective definition - Preparations (pre-meeting)

Phase I - Identification of the drivers of change

Phase II - Identification of probable events - Based on the drivers

Phase III - Evaluation of the events - Grouping the events to scenario sets

Phase IV - Review of the results - Iteration if needed

Post-Phase - Forming of the final scenarios - Implementation to use

Conf.

St.Dev.

Md

Questionnaire items

Avg.

Table 1. Test session ratings and questionnaire items

1.

Have you used or tried GSS tools previously?

1.1

I have worked with a GSS previously (1 never - 10 regularly)

5.71

8.00

3.09

2.29

1.2

How do you think the environment affected the results (1 extremely negatively, 10 extremely positively)

7.86

8.00

0.90

0.67

1.3

I haven't used a GSS environment but I have otherwise participated in similar sessions (1 never –10 regularly)

5.00

6.00

2.83

2.10

1.4

I believe that unsupported idea generation creates better results

5.29

5.00

1.60

1.19

2.

The brainstorming process (1 completely disagree - completely agree 10)

2.1

The goals of the session were clear

8.57

9.00

1.40

1.04

2.2

The goals were reached

8.14

8.00

1.21

0.90

2.3

Do you feel that the process provides useful results

8.50

9.00

1.22

0.91

2.4

Do you feel that the process included the most important factors

8.00

8.00

1.00

0.74

2.5

Do you consider the results as realistic and relevant to your company

7.86

8.00

1.35

1.00

2.6

Are the results trust-inspiring to you

8.00

8.00

0.82

0.60

2.7

The session was confusing and relevant steps were skipped

2.43

3.00

0.79

0.58

2.8

The results are not realistic or relevant to our company

2.14

2.00

1.07

0.79

2.9

The results are trustworthy because of the process

4.71

5.00

2.29

1.70

2.1 0

The results are trustworthy because of the used work methods

7.29

8.00

1.38

1.02

3.

Work methods (1 completely disagree - completely agree 10)

3.1

The process helped in getting and outline ideas

7.71

8.00

1.25

0.93

3.2

The ideas were clear and understood

7.43

8.00

1.13

0.84

3.3

Everyone’s input had equal treatment

7.86

9.00

2.41

1.79

3.4

Evaluation was a useful and relevant phase

8.43

9.00

1.51

1.12

3.5

The process was logical and proceeded fluently

8.29

8.00

1.50

1.11

3.6

The evaluation did not clarify the ideas

2.57

2.00

1.27

0.94

3.7

The available time was too short

5.36

5.00

2.43

1.80

3.8

I had time to concentrate on the evaluation and the results were reliable

6.86

8.00

2.48

1.84

4.

GSS-environment in the process (1 completely disagree - completely agree 10)

4.1

GSS fitted naturally with the scenario process

8.86

9.00

1.21

0.90

4.2

GSS systematized the process

8.86

9.00

1.21

0.90

4.3

GSS did not have added value in this task

2.36

2.50

0.75

0.55

4.4

GSS helped in observing different perspectives

7.57

8.00

2.64

1.95

4.5

Using the GSS made the working more difficult

2.29

2.00

1.11

0.82

4.6

GSS helped in getting committed to the process

7.29

8.00

1.60

1.19

4.7

GSS helped in creating trustworthy results

7.43

7.00

1.27

0.94

4.8

The GSS was a confusing experience and made working more difficult

2.00

2.00

1.00

0.74

Table 2. Observations on the case session Strengths

Weaknesses

Opportunities

Challenges, suggestions for improvement

4. Result level

- The intended tasks were completed; the group was able to generate the drivers and events as planned - The input was usable in building scenarios

- Verbal interaction during the session enables some amount of domineering if there are charismatic participants - Aims to avoid “social risk” may have impact on the participants’assessments of the process - The representation of the two industries under scope was not totally even; there were more representatives of the complementing sector

- Interventions allow steering the group

- The introduction and intervention during the session potentially affect both the workflow and content - Longer session time would allow more thorough discussion on the items - Participant selection has major impact in the content and results

3. Participant experience level

- Well organized and facilitated workshop - The group seemed to be motivated and active throughout the session - Open and warm atmosphere among the participants

- The timetable was strict considering the complexity of the task

- Small things in the arrangements, such as refreshments, well organized lunch breaks and such have a positive impact on the group

- The finding a time slot for participants from diverse organizations proved to be difficult - Positive overall experience may skew the evaluation of the artifact toward more favorable

2. Process level

- Strong agenda - Well planned session, clear objectives - Everyone’s opinions were taken into account - Effective idea generation, no individual domineering (- The group seemed to be motivated and active)

- Organizing the input and removing duplicates is challenging in the given time - Strict timetable, the natural workflow had to be interrupted to change phases in the process

- Use of up-front homework - Use on pre- and post-session assignment to facilitate reflection on the issues - Splitting the process to multiple (2) sessions

- More participants - Clearer phrasing of questions for the phases - Longer session time (- Verbal interaction during the session enables some amount of domineering if there are charismatic participants) - The introduction and intervention during the session potentially affect both the workflow and content

1. Technical level

- Good environment (-Everyone’s opinions were taken into account) (- Effective idea generation, no individual domineering) - Effective management of timetable and group process - The system required next to none orientation before the group was able to use it

-The answers and comments of the two industries’ representatives can not be separated for industry-level analysis

- Remote access system, to eliminate traveling to the workshop - The system enables engaging people from diverse backgrounds

- Some lag in the system - Technical performance affects the user experience, but does not seem to affect the content significantly