A Configurable Multimodal Framework

Imperial Journal of Interdisciplinary Research (IJIR) Vol-3, Issue-3, 2017 ISSN: 2454-1362, http://www.onlinejournal.in

A Configurable Multimodal Framework Alipta Ballav1 & Dr. Mainak Ghosh2 1

2

Independent Researcher Indian Institute of Technology, Kharagpur.

Abstract: The Internet has begun delivering technologies that are inaccessible. Users with disabilities are posed with significant challenges in accessing a webpage content. For example, due to the absence of coding norms blind and low-vision users do not get the equivalent non-visual elements. Similarly, users with motor impairments cannot access a webpage solely using a keyboard. The internet holds enormous possibilities for users with disabilities. It allows them to overcome barriers and challenges of the physical world. This paper proposes a configurable multimodal framework for users with disabilities enabling them to configure the input and output modality as per their choice. Keywords—Disabled users, Configurable Multimodal interface, Multimodal interaction, Web accessibility.

1. Introduction Advancing innovation is making websites accessible compliance a testing proposition for developers. Regardless of legislation mandating it, accessibility is moderately overlooked and misjudged as a practice. With the constant evolution of technology navigating websites are getting complex and posing extreme challenges to users with disabilities. This is a threat to people with disabilities and making them fall into the category of secondclass citizens of an information society [1]. Fundamentally, understanding user needs aid design team to build a maintainable, usable and sustainable system. Identifying the range of users groups who might be accessing data through the web is additionally a key angle. It is unrealistic to address each conceivable user’s issues reliably. Yet, it is sensible to assess user inputs and test during the discovery stage. A person with certain disabilities expects an online framework to work the same way it does for others. However, for a disabled user, the web offers tremendous accessibility challenge. The desire to access the web and perform tasks like shopping, internet banking, and paying bills is a complicated proposition for them. A report reveals that there are almost more than 1 billion disabled people in the world today. It is estimated that persons with disabilities make up roughly 15% of the world’s population [2]. An estimated 80% of all people with

Imperial Journal of Interdisciplinary Research (IJIR)

disabilities in the world live in developing countries. A visually impaired user depends on a screen reader to access the web content. Screen readers are audio interfaces [7]. A screen reader provides users only the textual content of the interfaces. This is challenging for a user interacting with a webpage. The data substitution of any photo-realistic component depends on the degree that designer of the page has included supportive descriptors known as alternate text [8]. A user with cognitive disabilities has attention deficit disorder. Users falling in the category have difficulty focusing on a particular task. Similarly, a user with motor disability have difficulty accessing the mouse, the user has to completely rely on the keyboard to navigate the web. A user experience designer and accessibility expert jointly work with the end users to plan design decisions to build an accessible system. Web users with specific inabilities need to rely upon what is furnished leaving them with no choice to arrange the framework in view of what is advantageous and desirable for them. A disabled user is always presented with whatsoever the web frameworks provides. This lead to a state where accessing information becomes challenging for users with certain disability type.This paper proposes a configurable multimodal framework where users can choose the input & output modes of his/her choice. Our proposed framework can solve the challenge of the inaccessible web. We take the example of a visually impaired user who needs to browse a website to access information, the user conveys the framework about the disability type, and in view of that information the framework, recommends the user a suggested input with a proposed output. The user has the adaptability to alter both the input and output modality in case the suggested option provided by the framework does not match the user need.

2. Evolution of Web Accessibility A disability frequently present challenges in the extreme, but they may at times provide singular opportunities [10] .World Wide Web Consortium (W3C) officially began in 1996 with a group of likeminded people with an effort to make the web accessible to people with disabilities. However, accessibility discovery took place way back in 1808 Page 620

Imperial Journal of Interdisciplinary Research (IJIR) Vol-3, Issue-3, 2017 ISSN: 2454-1362, http://www.onlinejournal.in when an Italian inventor Pellegrino Turri invented the mechanical typing machine for his blind lover. The “Talking Book” made its appearance in 1934, intended to help meet the information needs of people who were blind [10]. In 1936 Bell Labs invents the first electronic speech synthesizer called “Voder” [11]. In 1973 the United States Rehabilitation Act of 1973 made it illegal for organizations to discriminate against people with disabilities, calling for greater equality in the workplace [11]. In the year 1998 Section 508 is added to the Rehabilitation Act, requiring that government agencies make their websites more accessible to people with disabilities. Finally, in the year 1999, World Wide Web Consortium (W3C) released the first version of Web Content Accessibility guidelines (WCAG). There were 14 guidelines in total each describing a general principle of accessible design [12]. WCAG 2.0 was released on Dec 2008 consisting of 12 guidelines organized under four principles. Websites must be perceivable, operable understandable and robust. Prior to the launch of new accessibility standards, the web started getting complex and started opening up new possibilities. Though dynamic web came in 1995 with the introduction of JavaScript but for the first 10 years of JavaScript's life, professional programmers denigrated JavaScript because its target audience consisted of "amateurs"[13, 14]. Over time from a merely read-only static content based pages, the web evolved as a dynamic platform enabling user customization and personalized experiences. With the introduction of JavaScript and later JS frameworks the web became interactive and produced exceptional results for normal users. However, the birth of dynamic web posed extreme challenges for users with disabilities in accessing content. The usage of AJAX (Asynchronous Javascript and XML) to dynamically change content without a page refresh resulted in accessibility problems for users of Assistive Technology. Screen readers were able to capture static webpage content and present it to the user, but the dynamic content was unnoticed. To make JavaScript and dynamic content accessible W3C released a set of guidelines in the year 2014 known as WAI-ARIA (Web Accessibility Initiative – Accessible Rich Internet Applications). WAI-ARIA is a W3C protocol for enhancing and supporting accessibility of scripted and dynamic content. WAI ARIA adds semantic sugar to the HTML code, link each item by creating accessible regions. The problem of inaccessible dynamic content was solved to a large extent.

3. The Rise of Multimodal Interaction Devices, for example, screen readers and braille gadgets place limitations to the users in skimming the web, by outputting information in a linear fashion

Imperial Journal of Interdisciplinary Research (IJIR)

[16]. Blind users can just shape a one-dimensional mental model of a site page when accessing through a screen reader, developing an awareness of the spatial layout of objects on a web page always present a challenge [16]. Haptic technology made an effort to bridge the gap. The haptic perceptual system uses a combination of tactile and kinesthetic information about the environment [15].Traditionally, this sense has been called touch [15]. With a haptic feedback, blind users may be able to develop a mental model of how a page “looks” [16]. There has been a constant effort to improve the assistive technology tools and involve other interaction technique like haptic feedback to communication information [15]. People communicate using a combination of different modalities such as speech, gesture, facial expressions and so on. An effort by accessibility experts started taking a shape in the form of enabling multimodal interaction for disabled users. The concept of multimodal interaction involves the use of multiple tools for human-computer interactions. While using several of our senses in day-to-day life, we are still limited by at most one or two in our digital life [3]. The concept was first introduced by Richard A. Bolt (Architecture Machine Group, Massachusetts Institute of Technology) in 1980 [4]. His research was called “Put that there”, where he demonstrated a system which processed speech and touchpad pointing performing in a parallel way. Sharon Oviatt defined multimodal systems as a combination of natural input modalities in a coordinated manner like pen touch, speech hand gestures, eye gaze, and head and body movements [5]. Matthew Turk and George Robertson mentioned that multimodal research focuses on human perceptual channels taking advantage of natural human interaction techniques [6]. It enables users to interact with the computer without using keyboard or mouse and use natural techniques like gestures, head movement & blinks. Multimodal interaction uses recognition-based technologies and it recognizes human language & behavior which are formed naturally [9]. Multimodal interfaces can improve accessibility for diverse users groups and usage contexts. It can advance the application performance, improve stability, robustness, and efficiency of communication. Multimodal interaction research focuses on including more natural human communication channels into HCI [6]. Due to the limited bandwidth of some of the available channels accessibility research is probing for substitute ways of communication. Accordingly, numerous things from both the areas of accessibility & multimodal research can be summed up with the goal to get more conceptual & broader perspective on them [6]. This could lead to the utilization of a solution from one Page 621

Imperial Journal of Interdisciplinary Research (IJIR) Vol-3, Issue-3, 2017 ISSN: 2454-1362, http://www.onlinejournal.in space to another area. More than two or more user input modes are used like speech, touch, gesture, head & body movements in designing a multimodal system. All these inputs work in a coordinated manner to yield a multimodal output. This methodology leads to exploit different data modalities and its expanded ease of use; the shortcomings of one methodology are balanced by the qualities of another. For example, through speech, a visually impaired person can communicate to the system. Another exceptionally regular case of multimodal collaboration is seen while driving and utilizing a cell telephone. Diverting oneself from the street ahead, and dialing a number on the cellular telephone could be extremely risky. The driver rather utilizes voice input to dial a number or speak out an entire number. The framework likewise gives soundrelated input so the driver does not need to take a look at the screen.

4. Current Work While accessing the web users with specific inabilities need to rely on upon what is furnished to them leaving with no choice to arrange the framework in view of what is desirable. Exploiting each discrete component, this paper proposes a multimodal framework for users with disabilities. A user has the ability to modify the parameters and make it relevant for his/her use. We propose to develop a framework that can be adjusted to the user’s needs, taking into account certain inputs, and giving the user the flexibility to pick both the input and output modality. For example, a visually impaired user while browsing can outline the framework what sort of incapacity the user have and in view of that information the framework recommends the user a suggested input with a proposed output. The user can further go ahead and change both the input & output parameters in case the suggested options does not match their requirement. In our example, for a visually impaired user, the framework recommended Input is Gesture and Speech, samples of gestural interfaces are those that can detect moving hands, head gestures, and guiding fingers [3]. Gestures can be paired with the speech to allow the user to point at something on a screen and say, “move that there” [3]. The computer would know what you wanted to move, and then where to move it based on your pointing gesture. This sort of interface is extremely normal. For example, while interacting with a waiter in a restaurant and pointing out an item on the menu and asking, "What is this?" a user and waiter both know what they are talking about [3].

Imperial Journal of Interdisciplinary Research (IJIR)

Figure 1. A Conceptual Framework of the configurable Multimodal Framework.

5. Future Work According to report published by World Health Organization (WHO) there are estimated 285 million people who are visually impaired, out of these 39 million are blind and 246 have low vision. The report further says that 82% of people living with blindness are aged 50 and above [17]. A visually impaired user has to rely on braille keyboard to provide input to the web and further depends on screen readers like JAWS, NVDA, and Windows-eye to access web content. These screen readers fail to deliver a full proof solutions due to its system limitations and also because of poorly designed websites. WEBAIM conducted a survey in May 2012 on user preference of screen readers. As per the survey result, the most problematic areas with screen readers is reading a CAPTCHA, accessing flash content, reading complex forms & data tables and accessing dynamic content. From the Design perspective, there are problems like lack of keyboard support, improper headings, lack of skip links and improper alternate text description. Some low vision users even face text legibility issue due to improper contrast ratio. Text that does not have a contrast with its background are difficult to read for low vision users. A contrast ratio of 3:1 is recommended by [ISO9241-3] and [ANSI-HFES-100-1988] for standard text and vision [18]. Keeping the above challenges in mind our framework suggests configurability based on individual disability type. Imagine a visually impaired user who wants to access the web logs into the framework. The framework inquires the user about his/her disability type and suggests the input & the output modality. In the case of a visually Page 622

Imperial Journal of Interdisciplinary Research (IJIR) Vol-3, Issue-3, 2017 ISSN: 2454-1362, http://www.onlinejournal.in impaired user, the framework would suggest Speech to Speech modality. Similarly, for the case of a cognitively disabled user, there is hardly any standards. The American Disabilities act proposes three board guidelines which include keeping content simple and make it available in smaller chunks which are easily digestible, keeping navigation in one location without scattering, and make use of graphics & illustrations to convey information. A cognitively disabled user face the challenges of the inaccessible web in the form of information overload, difficulty in comprehending a large chunk of textual content, understanding mathematical expressions. To address these accessibility barriers our proposed framework would extract content from the website & present it in a simpler manner, confine the navigational items into a single location and make use of relevant images & graphics to convey information. In the initial phase, we plan to pick up a use case where a visually impaired user is attempting to fill a web form. In order to keep it simple, we will have just three input fields in the form. The user has the option to either set the modality manually or ask the framework to do it for them. The key input required by the framework would be the type of disability the user has so that the framework correctly predicts the modality. We plan to develop a prototype supporting three different combination of input & output modalities like Speech to Text, Speech to Speech and Text to Speech. We will test this conceptual model with end users having disabilities and revise it based on the obtained feedback.

6. Conclusion This paper proposes a route forward by leveraging the benefits of multimodal interaction. It offers the web user a configurable multimodal system allowing them to choose the input and output modality they would like to use in light of the individual’s disability or inability type.

7. References [1] L. Jonathan and P. Jaeger, "Reducing barriers to online access for people with disabilities.," Issues in Science and Technology, vol. 17, pp. 68-82, 2011. [2] N. E. Groce, "Disability and the Millennium Development Goals," United Nations Publication, 2011. [Online]. Available: http://www.un.org/disabilities/documents/review_of_disabi lity_and_the_mdgs.pdf.

http://johnnyholland.org/2008/11/the-future-of-interactionis-it-multimodal/. [5] S. Oviatt, "Multimodal Interfaces," The humancomputer interaction handbook: Fundamentals, evolving technologies and emerging applications, vol. 14, pp. 286 301, 2002. [6] Z. Obrenovic, J. Abascal and D. Starcevic, "Universal accessibility as a multimodal design issue," Communications of the ACM - ACM at sixty: a look back in time , vol. 50, no. 5, pp. 83-88, 2007. [7] WebAIM, "Designing for Screen Reader Compatibility," WebAIM, 19 November 2014. [Online]. Available: http://webaim.org/techniques/screenreader/. [8] S. Harper and Y. Yesilada, "Web Accessibility: A Foundation for Research," Springer Publishing Company, 2008. [9] M. Turk, "Multimodal interaction: A review," Pattern Recognition Letters, vol. 36, p. 189–195, 2014. [10] G. Hogan, The Inclusive Corporation: A Disability Handbook for Business Professionals, Swallow Press, May. [11] J. Hernandez, S. Mahajan and S. Victor, "Accessible Technology in the 21st Century - A stanford sophomore college project," 2005. [Online]. Available: http://cs.stanford.edu/people/eroberts/courses/soco/projects /2005-06/accessibility/firstwave.html. [12] "Web Content Accessibility Guidelines 1.0.," [Online]. Available: https://www.w3.org/TR/WCAG10/. [13] M. Hristova, "Dynamic Web Pages," [Online]. Available: https://www.ischool.utexas.edu/~hristova/ia/. [14] D. Buytaert, "A history of JavaScript across the stack," 22 February 2016. [Online]. Available: http://buytaert.net/a-history-of-javascript-across-the-stack. [15] M. Hersh and M. A. Johnson, Assistive Technology for Visually Impaired and Blind People, Springer, Science & Business Media, 2010. [16] D. McIsaac, "Re-envisioning the Keyboard as a Spatial User Interface," 2016 [17] WHO, "Visual impairment and blindness," World health Organisation, August 2014. [Online]. Available: http://www.who.int/mediacentre/factsheets/fs282/en/. [Accessed 1 March 2017]. [18] W3C, "Understanding WCAG 2.0," W3C, 2016. [Online]. Available: https://www.w3.org/TR/UNDERSTANDINGWCAG20/visual-audio-contrast-contrast.html. [Accessed 1 March 2017].

[3] D. Hruska, "Multimodal Interfaces as an Alternative to Unimodal Speech Inputs," [Online]. Available: http://courses.washington.edu/mobileux/multimodalalternative.html. [4] U. Seth, "The future of interaction. Is it multimodal?," 11 November 2008. [Online]. Available:

Imperial Journal of Interdisciplinary Research (IJIR)

Page 623