A Semiotic Systems Approach to User Experience Design

A Semiotic Systems Approach to User Experience Design Karen Cham, Director, Digital Media Kingston Kingston University London Abstract The International Organisation for Standardisation defines ‘user experience’ as ‘a person's perceptions and responses that result from the use or anticipated use of a product, system or service’ 1. UXD is a subset of the broader fields of experiential marketing and customer and/or brand experience design and is best described as ‘an approach to the design of computer-related products, services and environments’2 As recent advances in digital media computing technologies have moved into mobile, ubiquitous, social and tangible applications, such products, services and environments include, but are not limited to, web sites, mobile phone apps, digital television, interactive artworks, computer games, software and intelligent environments. User experience design for such products and services is complex and as yet is a new and evolving field. Some visual communication designers do not account for UX at all, many technical developers don’t have the skills to implement design concepts and yet others build user interfaces based purely on business requirements. All of this poor design practice has a negative impact on communication; a bad interface may do its job but a bad UX will mean a product or service isn’t understood. For an holistic consideration of the users’ experience, good UXD should aim to optimise the integration of functionality and aesthetics in digital interaction to reinforce and promote the communication goals. Digital interaction is specifically used here to refer to ‘a machine system which reacts in the moment, by virtue of automated reasoning based on data from its sensory apparatus’3. Visual communications that are digitally interactive in this way are participatory, navigable in a nonlinear manner and/or open to user generated content and require different design methodologies. The author has previously proposed that designing for digital media is best approached by an integration of post structuralism and complex systems theory 4. This paper argues that the designer must successfully integrate visual communication design, information architecture and usability by purposefully designing for semiotic autopoesis 5; a fundamental dialectic between structure and function must be designed into the system and its use. This proposal requires good UX designers to design diachronic grammatical structures that can adapt and evolve whilst consistently providing a coherent synchronic experience under multitudinous variables. Good user experience design for digital media visual communication systems requires designers to paint with language and leave it wet.

 

1  

Digital media, the User, Interaction, Experiences and Design As a result of the sheer velocity of technological change, practice leads theory in the field of digital media design. Recent developments include mobile, ubiquitous, social and tangible applications, products, services and spaces that includes but is not limited to, web sites, mobile phone apps, digital television, interactive artworks, computer games, software and ‘smart’ products and environments. As digital media design is for interactive products and services, ‘user experience design’ is pertinent to all these products and more because it is concerned with ‘experiences created and shaped through technology….and how to deliberately design those1 The term ‘digital interaction’ is specifically used here to refer to ‘a machine system which reacts in the moment, by virtue of automated reasoning based on data from its sensory apparatus’3. Most digital interaction is experienced by an integration of peripheral devices such as a keyboard, console, screen etc with a graphical user interface (GUI). As such, ‘user interaction design’ (UID) is at the root of designing any user experience. UXD is a complex, new and evolving field. It is recognized that “there is a need for more researchers starting to speculate and experiment with models and theories for user experience.”2 A quick Google search of the term ‘user experience design’ demonstrates a wealth of conflicting perspectives of varying degrees of usefulness, represented by multitudinous permutations of the Venn diagram. The only common agreement seems to be that UXD is a ‘convergent’ subject; that is, a subject consisting of a convergence of other subjects. It is correct to understand UXD as convergent in that it is most relevant to products that are themselves a convergence of ‘television, telecommunications and computing’3, and thus makers must work with computer science, product design, graphics and media production techniques and methodologies to create products and services. UID itself is another convergent subject and is commonly used interchangeably with the term UXD and/or ‘interaction design’. ‘"Interaction Design" refers to the shaping of interactive products and services with a specific focus on their use. Broadly speaking, there are two main senses of the concept, coming out of different intellectual traditions but increasingly converging in practice and research.’4 Lowgren firstly defines the tradition of interaction design as it evolves out of product design. This perspective is manifest in the approach of the RCAs seminal Design Interactions MA, Donald Normans book ‘The Design of Everyday Things’ and Durrell Bishops legendary marble answer machine (1992).

                                                                                                                1

http://www.interaction-design.org/encyclopedia/user_experience_and_experience_design.html Kari Kuutti. 2010. Where are the Ionians of user experience research?. In Proc. NordiCHI '10. ACM, NY, USA, 715-718 3 REF 4 Lowgren 2

 

2  

Fig 1 Durrel Bishops Marble Telephone Answering Machine (2002) The development in parallel to which Lowgren refers is really better described as ‘user interaction design’. A term used in the study, planning, and design of the interaction between users and computers as it evolved out of ‘Human Computer Interaction’ (HCI), itself first used in "The Psychology of Human-Computer Interaction" by Card, Moran, and Newell (1986). 5 Whilst HCI is often understood as focusing upon technical function and basic performative ‘usability’, as pertinent to early computer systems, the term HCI itself should be understood to embody the connotation that ‘working with an interactive computer system has many possibilities, that those possibilities take place in a dialog between the user and the computer and that they have a dynamic cognitive impact’6. HCI is also naturally multidisciplinary itself drawing upon computer science, cognitive psychology and ergonomics amongst other fields. Significantly, its emergence around 1980 is concurrent to that of personal computing, when new and diverse non specialist user groups started using computer systems. Durrell Bishops marble telephone is a ten year old innovation in tangible computing that demonstrates an innovative physical interface for accessing recorded data. However, it fails even in terms of basic usability providing a very ‘heavy’ interface with a lot of interaction for little result, it is far too complex to perform such a simple task. Any user centred design method applied to the task of collecting messages would not involve a phone, two trays, a shelf with marbles on and Durrels volcano shaped desk top box. It could easily be argued that this design is a decorative flourish,                                                                                                                 5 6

 

Lawrence Erlbaum Associates; New edition edition (February 1, 1986) http://cogscent.com/human-computer-interaction.html

3  

that flouts basic good design rules such as ‘form ever follows function’7. Designers working with new computational materials should not defy established good practice in the pursuit of the novel. Even the ultra novel iPhone series is ergonomic for the hand, has icons (just) finger sized and a screen, microphone and speakers that hint at its functions. Concurrently, Mads Soegaard online encyclopedia of interaction design http://www.interaction-design.org/encyclopedia/interaction_design.html has a dreadful ‘innovative’ interface itself where a giant left hand tab skids up and down the page as you scroll and whose menu flicks across and blocks the mainframe on rollover. The attendant sub menu is so intuitive it requires a ‘scribbled’ instruction and an arrow stating “Explore With Your Mouse”. This menu will not retract unless you purposefully rollover it again. Unfortunately for the user this undermines everything they says about good UX. It may apply to Phillips Wake Up Alarm but not to a web GUI – here the ‘medium is the message’8 and this would fail an Undergraduate module on interface design but it is fronting what purports to be the ‘Interaction-Design.org Foundation’. In what is considered by many to be the definitive textbook on UID, “Interaction Design, Beyond HCI”9, Yvonne Rogers, Professor of Interaction Design and Director of UCLIC at UCL usefully defines the role of UID in UXD. She also includes many academic disciplines in her definition of UID in addition to those associated with HCI such as design, informatics, engineering and sociology. Her definition has evolved as she has worked on ‘augmenting and extending everyday learning and work activities with novel technologies…..designing enhanced and engaging user experiences through using a diversity of technologies, including mobile, wireless, handheld and pervasive computing’10. Here, Rogers, Preece et al query three models that are ‘well known in interaction design’ to conceptualize UX in terms of emotion, pleasure, and experience: Norman's (2004) emotional design model beyond the notion of visceral/behavioural/reflective levels of cognition. it relies very heavily on his previous seminal text ‘The Design of Everyday Things’ and ‘does not suggest a method for systematically integrating Norman’s insightful model of cognition and affect into the practice of user experience design’.11 Indeed Norman himself presupposes that UX for ‘pleasurable’ tasks does not need to be as well designed as that needed for ‘serious’ activities as a dubious binary application of his useful hierarchical model of visceral, behavioural and reflective interaction. Most of us know from our own use of interactive products and services that this is far too simplistic. They also look at Jordan's (2000) pleasure model for product design, offering four hierarchical cognitive levels of pleasure in relation to using products ‘physio/socio/psycho/ideo-pleasure ideological’ pleasure and McCarthy and Wright's                                                                                                                 7

Louis Sullivan coined the phrase ‘form ever follows function’, in 1896, in his article ‘The Tall Office Building Artistically Considered’. It is pertinent to architecture and product design 8 Understanding Media: The Extensions of Man, Marshal McLuhan 1964 9 Sharp, Helen, Rogers, Yvonne and Preece, Jennifer J. (2007): Interaction Design: Beyond HumanComputer Interaction. John Wiley and Sons 10 http://mcs.open.ac.uk/yr258/ 11 http://www.uxmatters.com/mt/archives/2005/11/personas-goals-and-emotional-design.php

 

4  

(2004) ‘technology as experience framework’ based upon Dewys (1934) ‘Art As Experience’ recognizing the role of aesthetics in experience to define sensual/emotional/compositional/spatio-temporal modes of interaction. For the purposes of this paper, the term ‘UID’ is used to specifically refer to interaction by means of and through any form of GUI and is different to ‘interaction design’ which is used here to also include non digital interactive products such as door handles, bicycles and tin openers etc. It is proposed that UID it a core design method for UXD which as we have seen is informed by many other disciplines. It is however, interesting to note just how few of the Googled Venn diagrams put the user at the centre. What is most important to UXD is not what disciplines are converging and how they converge, but that they converge only by means of and through the users experience of their interaction with digitally interactive media systems. UXD is above all else, user centred design for digitally interactive products and services. The disciplinary perspectives brought to bear within this practice have varying degrees of convergence and success dependent upon the task at hand. The International Organisation for Standardisation defines ‘user experience’ as ‘a person's perceptions and responses that result from the use or anticipated use of a product, system or service’ 1. This anchors UXD in the users perceptions of their interaction and has lead to arguments that ‘you cannot design the UX… because you cannot design the user; you can only design for UX’12 Whilst this is an interesting exercise in semantics itself, designing users perceptions of a product, system or service is a core visual communication design activity, the multi-disciplinary field that encompasses communication design methods for graphic design, illustration, drawing and painting, multimedia, photography and moving image. This point is noted by Fredheim in his argument but then skipped over. As ever, it appears those in many fields are completely unaware of the rich and complex disciplinary practices in visual art, design and media. Whilst it is impossible to provide a comprehensive overview of all philosophical approaches that are relevant to interactivity, indeed the usefulness of the exercise would be questionable, Lowgren includes ‘Cognitive science: Interaction as information processing; Heidegger: Interaction as tool use and Merleau-Ponty: Interaction as perception’ alongside 1000 words on ‘art and media approaches’. Although he acknowledges the importance of the Bauhaus perspective on understanding media and materials, he only looks at Arnheim & Gestalt in addition. In practice, artists and designers working with visual media have been integrating semiotics, deconstruction & post structuralism13 as design methodologies for decades, to the extent they are considered part of ‘design basics’14. Any acknowledgement of this practice and its relevance to interaction design and UX seems to be profoundly lacking, yet from the poststructuralist perspective, every visual artifact is understood as a design construct, where meaning is produced                                                                                                                 12 13

 

http://uxdesign.smashingmagazine.com/2011/03/15/why-user-experience-cannot-be-designed/ http://www.semioticsolutions.com/our_story.htm 14 Basics Design 04: Image, Gavin Ambrose; Paul Harris AVA Publishing

5  

‘through the interplay of differences’ within a ‘systems of distinct signs’15 that is completed by the ‘reader’. This is clearly of core relevance to user centred experience design through any GUI. Helpfully, UXD is also often defined as a subset of the broader fields of experiential marketing and customer and/or brand experience design as ‘an approach to the design of computer-related products, services and environments’2 Perhaps if we drew any type of Venn diagram at all it should be firstly centred on the user and secondly a convergence of HCI, product design and media production ? However, it is also important to recognize where arts and media approaches themselves fail to address UXD. Whilst we have seen where adventures in product interfaces without a core understanding of usability can lead, many good interface designers fail to address UX. A digital interface can allow multiple users to participate in a simultaneous and instantaneous reproduction and dissemination of their divergent interpretations of an artifact as part of a networked participatory process, yet in the field it is common to find great graphic designers, who describe themselves as ‘web designers’, providing no system design. No site map, no architecture, no core SEO, no user journeys, no task analysis, no design of consequential aesthetic progress through a system – just a pile of separate interfaces that it is a developers task to thread together. This is not web design. Whilst traditionally presented as a subjective and therefore relatively useless term for serious argument, the term aesthetics actually comes from the Greek ‘aisthetikos’ and was coined by the philosopher Alexander Gottlieb Baumgarten in 1735 to mean the science of how things are known via the senses. [4] Aesthetic values can be driven by the senses, emotions, intellect, will, desires, culture, preferences, subconscious behaviour, conscious decision, training, instinct, sociological institutions, or some complex combination of these, depending on exactly which theory one employs. [5] An aesthetic value can be driven by a complex and convoluted cultural code, and it is probably true to say that all aesthetic judgments are to 9 some extent culturallyconditioned, i.e. linked to judgments of economic, political or moral value. In this way aesthetic judgements are also almost always established or upheld by some form of consensus, best understood as based on a consensus about desirable or preferred qualities. So UXD is also the aesthetics of interactive media use. UXD; The Aesthetics of Interactive Media in Use ‘It's very important for people to understand that technology has always been a spur to creativity. When the paint brush was invented, it allowed artists to paint with more creativity. The electric guitar helped rock-and-roll. You need to bring in people with skills required for digital. Fifty or 60 years ago, the television became a powerful medium. We needed people who understood television, who could write TV commercials, and who could write commercials, therefore, that captured people's attention and imagination. They weren't just print ads on wheels – words on wheels as we call it.                                                                                                                 15

 

Royle, Nicholas (2004) Jacques Derrida, pp. 62–63

6  

We had to develop an understanding of a different kind.’ John Hegarty, Cofounder and Worldwide Creative Director at Bartle Bogle Hegarty (BBH)16 If, for good user experience design, interaction needs to be an integrated aesthetic feature of the system, rich knowledge on aesthetics and communication are from the arts; sequential graphic communication; the juxtapositioning of image elements and the relationship between those elements eg film grammar, television advertising, Visual communications that are digitally interactive in this way are participatory, navigable in a nonlinear manner and/or open to user generated content and require different design methodologies The problem with old disciplines converging on new practices is that there are gaps. Digital media has evolved embedded in other subjects as we have seen, and is often described as interdisciplinary, trans disciplinary and even post disciplinary. However, what is important for UXD, perhaps the core design issue for digital media, is the effective theorizing of interactive visual communication design. The communicative act is core and ancient; it has always been interactive and of course, already mediated, from red ochre to silver halides to hypertext. We have many rigorous communication models based on interaction that could inform good UXD not least semiotics.

In Semiotics, the study of the science of signs, we find an analytical approach that provides us with an understanding of visual communication on the basis of • • •

Semantics: the relationship of signs to what they stand for Syntactics (or syntax): the formal or structural relations between signs Pragmatics: the relation of signs to interpreters

It is immediately clear that semantics should define good interface design, which it often does in practice, but that relational syntax should be defining the architectures that underpin and link those interfaces and that pragmatics is the reality of usability. Whilst this provides a strong basis from which to proceed with proposing a theoretical continuum for design and digital media, in  digital  interaction  we  can  commonly   observe  a  tangible  feedback  loop  between  the  pragmatics  and  semantics  which  is   not  accounted  for  in  traditional  semiotic  theory.  Furthermore,  in  complex   networked  games  for  example,  where  adaptations  of  the  system  are  encouraged   this  feedback  loop  is  observable  between  the  pragmatics,  the  semantics  and  the   syntax.  In  digital  practice,  these  dialogic  processes  are  not  theoretical,  as  they   may  be  observed  diachronically  in  relation  to  analogue  forms,  but  manifest   synchronically  through  the  medium.     Thus,  in  semiotics  we  have  a  solid  account  of  interpretation  but  not  necessarily   of  interaction  as  pertinent  to  digital  media.  A digital interface may, for example, allow multiple ‘authors’ and multiple ‘readers’ to participate in a simultaneous and instantaneous reproduction and dissemination of their divergent interpretations of an artefact as part of a networked participatory process. This is new and as yet                                                                                                                 16

 

http://www.thehindubusinessline.com/features/brandline/article1693994.ece

7  

unaccounted for in poststructuralist theory, which accounts for our interpretations and perceptions of artefacts to change synchronically and diachronically but not for the artefacts themselves to manifest those changes. This interaction with media on the part of the ‘reader’ requires us to look for additional models of communication. Although widely criticised as simplistic in the social sciences17, Shannon and Weavers ‘Mathematical Model of Communication’ (1949) provides a context for the technical processes in digital media. Developed as part of their work as engineers working for Bell Telephone Labs in the United States, this model gave rise to the mathematical study of information science and perhaps more importantly, it is a significant model for electronic signal processing, the core technical system behind both analogue and digital media transmission. As digital media is a convergence of the aesthetic, social and technical it is legitimate to look at integrating models from across those fields. At the very least Shannon & Weaver provide us with a model for semiotic feedback as a result of the technically interactive architecture of a digital communication process. Based on Systems Theory, this model also accounts for feedback as part of the communication process, indeed most importantly, feedback is enshrined as a self regulating mechanism of that system. For our purposes here that feedback is most pertinently user input.

    Fig  2  Shannon  &Weavers  Mathenatical  Model  of  Communication  (1949)   http://communicationtheory.org/shannon-­‐and-­‐weaver-­‐model-­‐of-­‐ communication/       Systems theory is an holistic approach to analysis that views whole systems based upon the links and interactions between the component parts and their relationship to each other within their environment. It is a way of thinking rather than a specific set of rules and has been a common across disciplines as divergent as art & design, thermodynamics, biology, sociology, physics, economics and law since the late 1960s. It has recently given rise to ‘complex systems theory’, whereby a system demonstrates specific capacities of ‘complexity’ such as ‘self organization’ and ‘emergence’. The study of complex systems is very interdisciplinary and thus encompasses more than one theoretical framework, so there is no single unified                                                                                                                 17

 

http://www.aber.ac.uk/media/Documents/short/trans.html

8  

Theory of Complexity, but several different theories have arisen concurrently. The dynamic capacity of digitally interactive systems in use places digital interactivity well within the realm of complex systems science; digital interaction through a GUI is a graphic model of a complex system which is generated by interaction. Compare the traditional ‘top down’ sending and receiving of a message in mass media to the evolving media eco system of digital news.

Fig 3. ‘Top Down’ news coverage and the ‘emerging media eco-system’ (Bowman & Willis) 2004 ? It is in design for interactive media arts, where algorithms meet images, and the user can interact, adapt and amend the artefact, that self-organisation, emergence, interdependence, feedback, the space of possibilities, co-evolution and the creation of new order are embraced on a day to day basis by artists, designers and users alike. A digitally interactive environment such as the world wide web, clearly demonstrates all the key aspects of a complex system. Indeed, it has already been described as a ‘complexity machine’.18 The author has previously proposed a convergence of poststructuralist and complexity theory as a means of understanding the ‘material’ qualities of digital media cultural systems19. It is a proposal built upon the basis of a theoretical continuum from Saussurian linguistics and the concern with synchronic systems and performance; it predicates post structuralism and the role of the ‘reader’ in ‘completing’ the emergent meaning of ‘open’ texts and integrates systems thinking and complex systems theory with systems art and digital media. For an holistic consideration of the users’ experience then, good UXD should aim to optimise the integration of functionality and aesthetics in digital interaction to reinforce and promote the communication goals. This requires designing systems that can be adapted, amended and allowed to evolve. Perhaps the core question should be how to extend our knowledge of experiential marketing into UXD ?

                                                                                                                18  (Qvortup 2006).   19  (Cham 2007).    

9  

Fig 4: The Changing Media Landscape; Owned, Bought & Earned Media, LBi London

As you can see form the diagram above the landscape that a brand must now cover has expanded from simply owned and bought media into earned media. How can we ensure that user behaviours are ‘on message ? Branding of course, has long been established as a type of user experience ; ‘moments of engagement between people and brands, and the associations these moments create’. 20 Indeed, in the nascent field of ‘neuro economics’, even an EEG machine can detect ones exposure to ‘putative branding moments within TV commercials’21. Concurrently, as both the design and the interpretation of a media message now often take place via a GUI, other new disciplines such as ‘behavioural economics’ are looking at how a product’s UX design can even encourage user behaviours that are detrimental to users’ best interests; a design style known ominously as ‘dark patterns’22. Dark Patterns are not bad design, deriving from ignorance, laziness and a lack of attention to detail etc, Dark Patterns are designed systems that have been ‘crafted with great attention to detail, and a solid understanding of human psychology, to trick users into do things they wouldn’t otherwise have done’23. We can see from dark patterns that it is really not that difficult to design for intended user behaviours through GUI based systems. Simultaneoulsy, brand identity is being rapidly integrated into UX and expanded across analogue and digital platforms in a mixed reality user experience; so its no longer a case of real experiences feeding back into our perception of a brand identity but of a co-evolving ecosystem of associations in a ‘cybernetic’ version of ‘touch                                                                                                                 20  (Ardill 2010).   21  (Braeutigam 2004)   22  (Brignull 2010)   23  (Brignull 2010)    

10  

point orchestration’24 Cybernetics is the interdisciplinary study of the structure of regulatory systems. Perhaps we also need to recognise UXD as second order cybernetics; the construction of models of self regulating systems ? ‘When designing digitally interactive artifacts we design parameters or co ordinates to define the space within which a performative autopoeitic process will take place. We can never begin to predict precisely what those processes might become through interaction, emergence and self organization, but we can and do establish and then author parameters that guide and delineate the space of possibilities’25. Conclusions We have seen that User Experience Design is a convergent subject with User Centered Design for digitally interactive products and services at its core. We have demonstrated the difference in User Experience Design & User Interaction Design and acknowledged the role played by UID in UXD. We have proposed that UXD is about the aesthetics of interactive media use and although it could be defined by an integration of HCI, product design and media production, we have acknowledged a gap in methodology when designing digitally interactive communication experiences that could be filed by an integration of post structuralism and complexity theory. This paper argues that any UX designer must successfully integrate visual communication design, information architecture and usability into the design of the system, product or service, as a fundamental dialectic between structure and function designed into the system and its use. Furthermore, for many complex projects they must design for semiotic autopoesis5, engineering emergent behaviours in line with brand values for example, or legal, moral and ethical parameters. This proposal requires good UX designers to design diachronic grammatical structures that can adapt and evolve whilst consistently providing a coherent synchronic experience under multitudinous variables. Good user experience design for digital media visual communication systems requires designers to paint with language and leave it wet

                                                                                                                24  (Abbing & Gessel 2010)   25  (Cham 2007)    

11