OPPORTUNITIES AND NEEDS FOR LOGGED USAGE DATA ANALYTICS OF COMPLEX INDUSTRIAL SYSTEMS Research in Progress Väätäjä, Heli, Tampere University of Technology, Tampere, Finland,
[email protected] Heimonen, Tomi, Tiitinen, Katariina, Hakulinen, Jaakko, Turunen, Markku, University of Tampere, Tampere, Finland, {
[email protected]} Harri Nieminen, Fastems, Tampere, Finland,
[email protected] Hannu Paunonen, Jouni Ruotsalainen, Jaakko Oksanen, Valmet Automation, Tampere, Finland,
[email protected] Iiro Lindborg, Rolls-Royce Marine, Rauma, Finland,
[email protected]
Abstract Industrial information systems record and store data about the status and use of the complex underlying production systems and processes. These data can be analyzed to improve existing, and innovate new products, processes, and services. This work focuses on a relatively unexplored area of industrial data analytics – understanding of end-user behaviors and their implications to the design, implementation, training and servicing of industrial systems. We report the initial findings from a requirements gathering workshop conducted with industry participants to identify the expected opportunities and goals with logged usage data and related needs to support the aims. Our key contributions include a characterization of the types of data that need to be collected and visualized, how these data can be used to understand product usage, description of the business purposes the information can be used for, and experience goals to guide the development of a novel usage data analytics tool. Interesting future research direction could include the privacy issues related to using logged usage data when limited number of users are logged. Keywords: Software instrumentation, user behavior analysis, visual analytics, data analytics, requirements elicitation, experience goals.
1
Introduction
Suppliers of complex industrial systems in manufacturing and process industry are increasingly interested in using logged data from systems after their deployment on the market. This interest is driven by the introduction of connected sensor platforms ("Internet of Things"), and the search for novel business opportunities and models especially in the area of new and continuous service development (Hänel et al., 2014). The life-cycle of systems in this domain can span from 20 to 40 years, and the use of logged usage data could support continuous system development throughout its life-cycle as suggested by research in other application domains (Olsson et al., 2014). However, this opportunity is still largely unexplored by suppliers and their customers alike in case of complex industrial systems. How and for what could logged usage data be used, what is needed to support gaining of insights, and what is of interest to be logged and analyzed? In the context of this paper, logged usage data means data logged from system use based on end-user interactions. This includes the system features and functionalities being used along with the associated
Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey, 2016
Väätäjä et al. /Data analytics of industrial systems
metadata (e.g., time, data input, and automation state). Little previous work exists specifically in the domain of exploratory user interaction analysis of complex industrial systems such as on manufacturing systems. In our project with three industrial case companies from metals and engineering industry, we are developing a framework and tools to support logged usage data analytics and visualization. In this paper, we report our results from a first workshop with representatives from the three companies to identify the opportunities and needs related to using logged usage data in metals and engineering industry. In addition, we present the experience goals defined for a usage data analytics tool for analysts using the tool. We also outline our ongoing and future research and development.
2
Related Work
The use of interactive systems can be logged as a part of usability and user experience (UX) research with the aim of identifying potential problems and issues affecting the experience. Such logging can address multiple needs within the product development organization. UX measurement can include use of device functions, access of features by different user groups, or identification of changes in user behavior (Ketola et al., 2009). However, relying on behavioral data alone is not enough to understand the reasons for behavior (Kim et al., 2008). Product metrics that capture user experience qualities such as happiness, retention and engagement have been found useful in supporting data-driven and usercentered decision-making in product development at Google (Rodden et al., 2010). Therefore, both data-driven, such as instrumentation based logging of usage, and subjective measurements are needed in product development. Our aim here is to support gaining insight based on logged usage data of user interactions and related events. A common approach to acquiring usage data is to instrument software applications to log user interaction events. A straightforward but laborious way is to add logging instructions directly into source code, but this approach may introduce new complexity into the system (Bateman et al., 2009). As an alternative, many instrumentation frameworks aim to reduce the burden on the application developer by semi-automatically logging relevant interactions (e.g., Atterer et al., 2007; Bateman et al., 2009; Google Analytics). The complexity and level of abstraction of the instrumentation varies by its purpose, but when implementing instrumentation and analysis techniques, one should consider the levels of abstraction of the captured events, how higher level interactions are composed of lower level events, and how to capture contextual information (Hilbert et al., 2000). Once the log data is collected, several steps are required to proceed from the data to insights, including steps such as processing the data to transform it into a suitable format, data verification and modeling, and reporting the analysis procedures and results (Kandel et al., 2012). Although basic descriptive statistics can provide a rough idea of system usage, typically some form of data preprocessing needs to be performed prior to detailed analysis, especially when dealing with low level events (Renaud et al., 2004). Preprocessed event sequences can then be mined and interpreted (Fern et al., 2010). Beyond computing summary statistics, some of the key goals in such analyses are detecting or comparing patterns, characterizing system usage, which can be supported with visualization and integrated analytical support in the evaluation environment (Hilbert et al., 2000). One of the main challenges of data processing and analysis is to facilitate analysts' tasks. Experts can be working on challenging problems to which no direct answer is available, and they participate in inventing, innovating or discovering (Shneiderman et al. 2006). Designed tools should support human performance, error-free performing of tasks, creative exploration, hypothesis building, and history keeping as well as collaboration and dissemination to others (ibid.). The exploration and visualizations is described to provide insight (Card et al. 1999) or enabling and discovering insight (Thomas et al. 2006). Spontaneous insight is related to solving difficult problems and is free from applying para-
Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey, 2016
2
Väätäjä et al. /Data analytics of industrial systems
digms and schematic structures, such as mental models (Chang et al. 2009). Knowledge-building insight is described as a form of learning by using rules for problem-solving and reasoning (ibid.). In the domain of big data analytics, the analysis tasks are exploratory and happen on-demand, the results are aimed at audiences with little background in data science, and with the need to produce reproducible and reliable results (Fisher et al., 2012). In effect these tasks fall within a continuum that spans activities focusing on producing known, low impact insights (e.g., static reports) to activities that aim to uncover high impact, previously unknown insights (e.g., real-time alerts, predictions, and recognition of patterns) (Liu et al., 2013). One of the aims of producing such tools can be thought of as increasing the analytics literacy of the development team (Medler et al., 2011). Interactive data analysis should help improve the productivity of technically proficient users but also be accessible to users with limited programming skills (Heer et al., 2012).
3
Methods
In the first phase of our research we wanted to identify the views of the three supplier companies on opportunities, needs and requirements as we were aiming to develop solutions for logged usage data analytics for them. The workshop was divided into two main parts as follows. The first part of the workshop focused on identifying the opportunities and goals related to logged usage data within the companies: • what opportunities can be found for using logged usage data within company operations, • who could benefit from the use of the data, • what is the goal of using the data, • what type of data is needed to reach the goals, and • what type of data and their combinations could be useful. The second part of the workshop aimed to gather requirements from the participating three supplier companies for data analytics and visualization that could be common for all these companies. First, one of the researchers introduced the different possibilities to analyze and visualize logged usage data based on the prior research literature. After this, the following questions were addressed in a joint discussion: • what would be relevant to analyze and visualize from the data, and • what are the motivations/goals in R&D to use logged usage data in development activities. Participants. Five representatives from three supplier companies (one from company A, three from company B, and one from company C), and four researchers involved in the research project participated the workshop. The company representatives were invited by email to company contact persons. The contact persons at the companies were given an opportunity to invite colleagues to participate the workshop. All company representatives were from the R&D department of the supplier companies. The industry domains and types of complex systems of the case companies differ slightly. Company A manufactures propulsion systems and is interested in analyzing the usage profiles by different users based on the data captured from the propulsion control unit. Companies B and C produce process automation and manufacturing systems, respectively, that share many similarities considering the volume and complexity of generated data. The main interest in this study lies in understanding the usage of system features by end-users in various situations. In all of the cases one of the key challenges is that current systems do not explicitly record all user interactions, use of different features etc. The goal of prior logging has been primarily
Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey, 2016
3
Väätäjä et al. /Data analytics of industrial systems
to support debugging or identify problem situations, for example, not to understand user behaviors. Therefore, logged usage data has not been collected and used in supplier’s R&D or other operations systematically when starting this research. Solutions to support data analytics and visualization of logged usage data have therefore not existed. Procedure & Materials. Prior to the starting of the workshop, A1-sized posters with the previously presented questions as titles were attached to the wall of the workspace. As the activity started all workshop participants gathered around the posters. The themes were discussed jointly one theme at a time. One researcher was the main responsible for facilitating the discussion and writing short notes on the posters on the emerging themes in the discussion. In addition, another researcher wrote more detailed notes with a computer on the discussion. In the end of the day the posters were photographed to digitize them. Analysis. The notes written on the posters and electronic notes written with a computer during the workshop were merged in digitized form. The analysis was done first according to the original workshop questions by grouping and theming the findings. If needed, themes were merged from different groups or if additional themes had been recorded, these were grouped and themes created for them. The findings are presented in this paper according to the main themes that emerged.
4
Results
We report the identified central themes from the workshop reflecting the company cases. Our aim is to use the insights gained for building a framework to support the development of logged usage data analytics, as well as related solutions and tools for metals and engineering industry domain.
4.1
Current access to logged usage data
One of the themes that emerged during the workshop discussion was the supplier’s access to logged usage data. Among the three participating supplier companies, differences exist in regard to their current access to logged usage data. In the studied domain, the logged usage data is usually customer’s data that needs to be granted access to by the customer. Company A measures several sensor measurements from the complex system that uses their technology. This data is readily available for use, but has not previously been used to identify usage related patterns and user profiles. Company B has access to customer data through service agreements, which enables following up the process-related data and using it for service-related activities based on agreement. However, using it for other purposes, such as R&D related activities, needs a separate agreement with the customer. Company C has currently no access to customer's usage data except in the following cases. First, access is given in case of fault situations so that teleservice can with the permission of the customer company access the usage logs remotely. Second, in case of software bugs that need to be fixed, developers can access the usage logs to identify the possible issues to solve. Otherwise the logged usage data is not used within company C. It was also mentioned that customer companies can be protective towards sharing the logged usage data, as it can reveal company-related information on the production and related processes and orders to externals. To be able gain access to customer data, customers need to see value in sharing the data with the supplier company. This calls for being able to show to the customer the value of the given access, e.g., how the usage information can be used to enhance the customer’s production and business. Furthermore, it is essential to build trust between the customer and supplier company for sharing the data. This can be created by informing and agreeing on how and for what data is used for and who can access the data for what type of purposes. Also trust for security of the data needs to be built by secure and efficient solutions for data transfer, handling, and storage. The data transfer from customer to sup-
Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey, 2016
4
Väätäjä et al. /Data analytics of industrial systems
plier therefore needs planning, including the capacity of the customer companies ICT and supplier's solution at customer end to handle possible data transfers for large amounts of data.
4.2
Benefits for Suppliers, Customers, and End-users
Several expected benefits were discussed in the workshop. Benefits were related to the supplier company and its operations, customer company, and to the end-users within the customer company. First, within the supplier company, the company participants expected benefits for R&D and service business development related activities. In case of R&D, the information provided by analysis and visualization of usage logs could benefit system developers, user interface (UI) designers, as well as UX designers. This is discussed further later in this paper. Service business and its development were elaborated to include development of novel reactive and proactive services for maintenance, and support developing teleservice activities and related services. Furthermore, life cycle related services, including system update services that are in connection with technology renewal were raised in discussion as one promising area. In addition, development and novel ways of offering training services were seen promising. Based on identifying usage patterns and feature use, customized training could be offered to the customer. For example, if it is noticed that there is no use of the features in the customer company that could be useful for them, or the system properties are used inefficiently, the supplier could offer training services to support the customers as well as end-users in system use. Finally, marketing within the supplier could use the usage data to communicate the value of the system and its features and functionalities. Second, the customer could benefit from the visualizations of logged data by enabling tracking of production and the efficiency of system use. Improving the customer’s business by supporting the efficiency and effectiveness of customer's production functions is one of the key things that can bring competitive advantage and create increased value for the customer. This can be supported by increased understanding of customer goals, system use, and actions - why is customer or users in a customer company acting in specific ways – based on the logged usage data analytics. Finally, the end-users of the solutions could benefit from the use of logged usage data. One of the company representatives commented that the traditional aim in their industry is to minimize the role of the user through automation. However, a new approach is emerging that better utilizes the human capacity and focuses on experiential aspects. For example, motivational feedback utilizing the logged usage data could provide benefits to end-users and to the customer company. Utilizing logged usage data in, e.g., feedback to users could enhance learning, understanding of the system, and guide system use. Gamification and providing different types of challenges for own development could be created by using logged usage data. This could lead to increased wellbeing and motivation at work and therefore to other benefits for the customer company.
4.3
Empowering Technology Renewal and Continuous Development
The discussion in the workshop on the use of logged usage data in R&D centered around two main topics: technology renewal, and continuous development and deployment of the solutions. Company representatives expected that by understanding how users use the system, solutions that fit better to the workflow could be developed. Furthermore, a more satisfying user experience could be created by understanding on how the system is used, alternative ways to use can be supported and created, and development of novel innovative features is enabled. Company representatives expected that the logged usage data could specifically provide information on which features and functionalities are used and how. This provides an understanding of what is important and relevant for the users. Today, contextual knowledge is first gained in customer companies by field studies to understand how the system is used. With the help of logged usage data analytics and visualization, the focus of user studies in the field studies was expected to shift directly to studying
Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey, 2016
5
Väätäjä et al. /Data analytics of industrial systems
why the system is used as it is used. At the same time understanding gained with the help of logged usage data gives a good background for the user studies carried out in the field. The implementation of the field studies was expected to become more efficient by focusing directly on the reasons behind the identified behavioral patterns instead of first studying how the system is used. As mentioned, companies expected to identify feature usage insights, such as which features are used frequently or used very infrequently, or not at all. If some features or functionalities are not used, maintaining and updating them can be unnecessary and resources could be used for other development. In user interface design, much used features could be brought easily available to the users in the UI, or important features that are "hidden" brought up to the user. It would also be possible to find differences between optimal usage patterns and real usage. In addition, logged usage data was expected to provide information on how the system is operated in different situations, and end-users and customers could be profiled based on how they operate the system. Continuous development and deployment enables that the provided solution can be changed within a very short change cycle and in longer term the basic system can be made better. Specifically identification of fault situations and subsequent user actions were considered potentially informative with respect to developing the alarms and mechanisms for recovering from fault situations further.
4.4
Needs related to types of data and information
One of the goals of the workshop was to identify the data types that could be useful for the companies. Table 1 presents the relevant data types to be addressed in usage data analytics and visualization. Identified data types are used in creating a framework and tools for usage data analytics within our project. When focusing on user actions and interaction with the system, the following interactions were identified as examples of what can be interesting to focus on in continuous development as they support a more in-depth understanding of user’s interactions related to the situation, task and goals at hand: • User actions that focus on the system and/or UI. • User reactions to different situations and how the system is used in these situations. • Mouse movements on the UI (or other interaction type, e.g., touch, finger movement), eye movements on the UI and its elements. • Information that is shown to the user, sound played, or types of notifications to the user. • Sequences of events: What is done, with what, what feedback is given to the user, what happened previously and what happens next? • Connecting the user behavior and actions to user's goal when something happened (e.g., using the work or task list that the user should be executing as reference). • In process industry, what follows (user behavior, actions) when user observes (perceives) something, and gaining understanding why something was observed, or not observed. • Usage of features readily available in UI are used – what user should have noticed as possibilities and available for use. • Information needs based on different user roles. The identified examples of interesting data types and information needs are used within the project to develop a framework for logged usage data analytics. Findings provide information on what data needs to be collected for and used in analytics, and the types of issues that need to be addressed in the solutions. To summarize the findings, companies expressed that they want to gain an understanding on 1) usage combinations, such as the customer’s production type and in what mode they use the system,
Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey, 2016
6
Väätäjä et al. /Data analytics of industrial systems
2) patterns of use, and types of user groups and profiles that can be found, 3) summarizations of the system use based on logged data, 4) identifying problem or fault situations (individual and possible patterns), and 5) changes in use (such as features) over weeks or months such as how the taking of system into use and learning to use is progressing to identify issues needing support or training, and whether problems or faults appear over time. Needed data types
Purpose needed for
Logs of different events
To identify relevant events related to interaction or prior or after an event.
Continuous sensor data measurements
To provide information on system status.
User-related data
To identify different users and enabling profiling, in future personalized feedback, motivation and guidance etc.
Context-related data
To identify and gain information on aspects that are relevant for understanding the logged usage data, e.g., related to task, temporal, physical, technology and information, and social context.
Models of optimal sequences (e.g., user manual operating instructions in digitized sequence form)
To be able to compare the logged sequences of use and process to optimal sequence models to identify what actually happens and how system is used.
R&D intervention events and related actions, such as when changes are done to the system software
To identify effects of the interventions and changes in usage or user behaviour based on logged usage data.
Other intervention events (e.g., support, teleservice, maintenance)
To identify effects of the interventions in usage or user behaviour based on logged usage data.
User created diaries
To provide user created data related to logged events – to support analysis and insights gained by analysts. Creating diaries could be supported by the system.
Table 1. Identified needed data types useful for the case companies.
When going from these findings towards a framework for logged usage data analytics and visualization, and its practical implementation, identified issues to be solved include a specification for data interchange formats, identifying precisely which data is needed to be logged to reach the analytical goals (e.g., system/UI events, timestamps, etc.), how the data need to be processed, which available sensor measurements are needed to supplement the logged usage data, and how to visualize the analyzed data. These implementation issues will be addressed in our future research.
5
Experience goals for usage data analytics tool development
Based on our findings as well as prior literature on data analytics and visualization we defined experience goals (Varsaluoma et al. 2015, Väätäjä et al. 2015) for users, i.e., analysts, for a data analytics tool in the context of our study. These experience goals are used to guide our iterative experience design and evaluation (Shneiderman et al. 2006) of the data analytics and visualization tool. The defined experience goals are the following: 1. The feeling of gaining insights. The user gets novel and meaningful insights on user behaviour, user profiles, or activities, as well as on system use and its functioning by exploring and analysing the data, and by the visualizations created based on the data. The insights gained enable the user to use them creatively as inspiration in continuous development of systems and services, and in innovation of novel solutions for system and service development.
Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey, 2016
7
Väätäjä et al. /Data analytics of industrial systems
2. The feeling of being effective. The user understands a difficult problem or challenge to be solved or identifies the opportunity space by exploring the data and by provided visualizations. This gives the user a feeling of competence by deeper understanding of the problem domain, and by identifying and solving the issues at hand effectively in a professional manner. It also supports identifying the further questions – that can be answered for example by field visits to the customer or observing the users in context. 3. The feeling of being in control. The user feels being in control of the exploration and analysis. The tool supports the user to be aware of the analysis process and its progress as well as the data and analyses behind the visualizations. 4. The feeling of relatedness by collaboration with others. The user feels being related to others in his/her organization by being able to collaborate with them on the data and by sharing the insights gained to be used in development and innovation activities within the company.
6
Conclusions and future work
Using the logged usage data based on end-user interactions provides suppliers of complex systems opportunities to understand end-user behaviors and use of the systems. Results from a workshop with representatives from three case companies indicate, that the knowledge and insight gained is aimed to improve supplier’s systems, as well as to develop new products and services. Beyond suppliers, benefits for customers and end-users of the supplier’s solutions can be achieved by increased value and enhanced user experience. We reported data types and interaction events and patterns that are in the case companies’ interests as well as the specific interests of R&D for understanding product usage based on the data, including efficient gaining of understanding how a system is used to support further field studies. We concluded with four experience goals for users of usage data analytics and visualization tools. Our future work includes, e.g., developing and implementing the analytics framework, and a longitudinal user study with iterative development of a visual analytics tool prototype based on user feedback. In addition, privacy issues are of interest in future research, as in industrial environments with a limited number of users there is a possibility to identify individual users from the logged data.
Acknowledgements Authors gratefully acknowledge the support by TEKES (The Finnish Innovation Agency) for UXUS (User experience and usability in complex systems) research programme.
References Atterer, R., and Schmidt, A. (2007). Tracking the interaction of users with AJAX applications for usability testing. In Proc. CHI 2007, ACM Press, 1347-1350. Bateman, S., Gutwin, C., Osgood, N., and McCalla, G. (2009). Interactive usability instrumentation. In EICS 2009, ACM Press, 45-54. Card, S.K., Mackinlay, J.D., and Shneiderman B., eds. (1999). Readings in Information Visualization: Using Vision to Think, Morgan Kaufmann. Chang, R., Ziemkiewicz, C., Green, T. M., and Ribarsky, W. (2009). Defining insight for visual analytics. Computer Graphics and Applications, IEEE, 29(2), 14-17. Fern, X., Komireddy, C., Grigoreanu, V., and Burnett, M. (2010). Mining problem-solving strategies from HCI data. ACM Transactions on Computer-Human Interaction, 17, 1, Article 3. Fisher, D., DeLine, R., Czerwinski, M., and Drucker, S. (2012). Interactions with big data analytics. Interactions, 19:3, 50-59. Heer, J., and Kandel, S. (2012). Interactive analysis of big data. XRDS, 19, 1, 50-54.
Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey, 2016
8
Väätäjä et al. /Data analytics of industrial systems
Hilbert, D. M., and Redmiles, D. F. (2000). Extracting usability information from user interface events. ACM Computing Surveys, 32, 4, 384-421. Hänel, T., and Felden, C. (2014). The Role Of Operational Business Intelligence In Customer Centric Service Provision. In Proc. AMCIS 2014. Jung, T., and Wickrama, K. (2008). An introduction to latent class growth analysis and growth mixture modelling, Social and Personality Psychology Compass, 2, 1, 302-317. Kandel, S., Paepcke, A., Hellerstein, J.M., and Heer, J. (2012). Enterprise data analysis and visualization: An interview study. IEEE Trans. on Visualization and Computer Graphics, 12, 18, 2917-2926. Ketola, P., and Roto, V. (2009). On User Experience Measurement Needs - Case Nokia. International Journal on Technology and Human Interaction (IJTHI), 5, 3, 78-89. Kim, J. H., Gunn, D. V., Schuh, E., Phillips, B., Pagulayan, R. J., and Wixon, D. (2008). Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems. In Proc. CHI 2008, ACM Press, 443-452. Liu, Q., Vorvoreanu, M., Madhavan, K. P. C., and McKenna, A. F. (2013). Designing discovery experience for big data interaction: a case of web-based knowledge mining and interactive visualization platform. In Proc. DUXU 2013, Springer-Verlag, 543-552. Medler, B., John, M., and Lane, J. (2011). Data cracker: developing a visual game analytic tool for analyzing online gameplay. In Proc. CHI 2011, ACM, 2365-2374. Olsson, H. H., and Bosch, J.( 2014). Post-Deployment Data Collection in Software-Intensive Embedded Products. Continuous Software Engineering, Springer International Publishing, 143-154. Reise, S. P., Ventura, J., Nuechterlein, K. H., Kim, K. H. (2005). An Illustration of Multilevel Factor Analysis. Journal of Personality Assessment, 84, 2, 126-136. Renaud, K., and Gray, P. (2004). Making sense of low-level usage data to understand user activities. In Proc. SAICSIT 2004, 115-124. Rodden, K., Hutchinson, H., and Fu, X. (2010). Measuring the user experience on a large scale: usercentered metrics for web applications. In Proc. CHI 2010, ACM, 2395-2398. Shneiderman, B., and Plaisant, C. (2006). Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies. In Proc. 2006 AVI workshop on Beyond time and errors: novel evaluation methods for information visualization, ACM, 1-7. Thomas, J. and Cook, K. (2005). Illuminating the Path: The Research and Development Agenda for Visual Analytics, Nat’l Visualization and Analytics Center. Varsaluoma, J., Väätäjä, H., Kaasinen, E., Karvonen, H., and Lu, Y. (2015). The fuzzy front end of experience design: Eliciting and communicating experience goals. In Proc. of the 27th Australian Computer-Human Interaction Conference (OZCHI 2015), ACM, 8 pages. Väätäjä, H., Savioja, P., Roto, V., Olsson T., Varsaluoma, J. (2015). User experience goals as a guiding light in design and development – Early findings. In INTERACT 2015 Adjunct Proceedings. Univ. of Bamberg Press, 521-527.
Twenty-Fourth European Conference on Information Systems (ECIS), İstanbul,Turkey, 2016
9
