Contactless bio-behavioral technologies for virtual reality

Annual Review of Cybertherapy and Telemedicine 2013 B.K. Wiederhold and G. Riva (Eds.) IOS Press, 2013 © 2013 Interactive Media Institute and IOS Press. All rights reserved. doi:10.3233/978-1-61499-282-0-149

149

Contactless Bio-behavioral Technologies for Virtual Reality Pietro CIPRESSO a,1, Silvia SERINO a, Andrea GAGGIOLI a, Giovanni ALBANI b, Giuseppe RIVA a,c a Applied Technology for Neuro-Psychology Lab, IRCCS Istituto Auxologico Italiano, Via Pellizza da Volpedo 41, 20149 Milano, Italy b Department of Neurosciences, Istituto Auxologico Italiano, Piancavallo-Verbania, Italy c Psychology Department, Catholic University of Milan, Largo Gemelli, 1, 20123 Milan, Italy

Abstract. The use of biosensors in human experimental research has become one of the most reliable methods to objectively quantify participants' behavioral, psychological and physiological measures. Modern bio-behavioral technologies can be integrated in Virtual Reality platforms, for a contactless assessment of affective states. This manuscript proposes a paradigm to deeper reach accuracy in assessment and diagnostics of rehabilitation processes. Keywords. Contactless, Bio-behavioral, Virtual Reality, New Technologies, Cybertherapy, Assessment, Neurorehabilitation

Introduction Experimental psychology and virtual reality paradigms use methods and techniques involving experimental trials with humans and animals. In particular, the use of biosensors in human experimental research has become one of the most reliable methods to objectively quantify participants' behavioral, psychological and physiological measures during virtual reality immersion. The most crucial problem to consider with biosensors is the obtrusiveness. The more we want to examine objective behaviors, the more we interfere with the experiment directly conditioning the participants. In many kinds of experimental research, from emotions to affective states, to cognitive assessment, this is a critical aspect that can interfere with possible outcomes and conclusions. The problem is relevant and well known among researchers in all the related fields, and big steps have been taken in many directions. Firstly, biosensors have been developed to be more precise, and also less obtrusive. Also, there has been a big development of biosensor integration, above all making them wearable, to reduce the discomfort of patches and cables.

1

Corresponding Author.

150

P. Cipresso et al. / Contactless Bio-Behavioral Technologies for Virtual Reality

However the problem of these technologies remain exactly the same, i.e. experiments involving human beings are highly conditioned by sensors and direct observation [1]. The observer effect is amplified in bio-behavioral science when the measures regard affective states, emotions, psychiatric disease, stress, and all the aspects of psychological sphere. Several studies on Virtual Reality make use of methods and techniques involving biosensors for a precise measurement of psychophysiological states. In particular the use of biosensors in human experimental research has became one of the most reliable methods to objectively quantify participants' behavioral, psychological and physiological measures. The most crucial problem to consider with biosensors is their obtrusiveness, above all if this is already high by using a head mounted display. This manuscript aims to deeper understand which contactless technologies can be integrated in Virtual Reality platforms and to provide the Contactless Bio-behavioral research methods to be effectively used for cybertherapy.

1. Bio-Behavioral Research: Scientific Standards Bio-behavioral states are detected through cardiorespiratory activity, behavioral states, arousal-pleasantness states and attentional-contextual situations, as following explained. Cardiorespiratory Activity: Cardiovascular and respiratory activity is monitored to evaluate both voluntary and autonomic effect of respiration on heart rate, analyzing RR and NN interval extracted from electrocardiogram (ECG) and respiration (RSP) signals, and their interaction. According to the guidelines of Task force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology, typical Heart Rate Variability (HRV) temporal and spectral indexes can be extracted to evaluate the autonomic nervous system response [2]. Behavioral Patterns: Activity-related behaviors suggest a clear aspect of continuous regulatory actions, observable in movement qualities, contours, expressions, and also perceived in vocal tonality. Gesture and posture are considered as parts of a wider semiotic system that underlies human communication. In this perspective, nonverbal behaviors could be interpreted to indicate the ‘‘unsaid’’ elements representing our internal states [3]. On the other hand, speech analysis is used to recognize affective and cognitive states from the user's voice. Arousal And Pleasantness Patterns: Arousal-Pleasantness models have been extensively used in psychophysiological research as an objective way to measure affective states. Physiological arousal can be measured through skin temperature, galvanic skin response, respiration rate and pupil dilation [4]. Pleasantness can be measured monitoring the activation of zygomatic major and corrugator supercilii facial muscles [4]. In general the analysis of facial expression through a classification in Action units can be useful to detect specific emotion and consequently pleasantness. Eventually pupil dilation can be used to quantify the pleasantness intensity. Contextual And Attentional Patterns: Voluntary, but also involuntary eye movement responses reflect internal processing and how such processing is accomplished, yielding insight into the content

P. Cipresso et al. / Contactless Bio-Behavioral Technologies for Virtual Reality

151

of information processing. Cognitive Psychology is composed of plenty of subdisciplines concerned with different aspects of thinking, such as attention, memory, perception, and coordination. There are many specific paradigms, such as visual searching, prosaccade and antisaccade tasks. On the other hand, the context to which the attention is shifted is also to be considered, and also the social context dealing with the interaction among subjects.

2. Contactless technologies The above indicated scientific standards for bio-behavioral research can also be assessed by the means of contactless technology already available to researchers. Following the available contactless technologies and the relative correlates that can be extracted to assess bio-behavioral states (see also figure 1 and 2).

Figure 1. Contactless technologies and the extracted measures for the bio-behavioral states.

Eye-Tracker: A technology to record eye movement data consisting of momentto-moment measures of the eyes' displacements along the vertical and horizontal axes within the spatial working area of a scene. Stereo Camera: A low cost, single view depth imaging camera that collects spatiotemporal information of the subject’s movements (an example of this is the Microsoft Kinect device).

152

P. Cipresso et al. / Contactless Bio-Behavioral Technologies for Virtual Reality

Motion Capture System from Cameras: A digital video solution to map a body in a virtual digital tridimensional space, to record any activity in a defined environment. Non-intrusive ECG through a Chair: Capacitive sensors through the back of a chair can be employed for measuring the complete electrocardiogram of a human heart without electric contact with the skin. Surrounding microphones: A system based on low-cost microphones to be used for speech analysis based on voice detection and noise removing. Accelerometers: Small devices to measure proper acceleration. Can be inserted in an object and track its movement in an environment. Can also transmit data via Bluetooth or record data locally. Digital Infrared Thermal Imaging Camera: Is a non-invasive camera to record images and videos (thermograms) and involves no exposure to radiation. Thermograms were initially used to detect body temperature, but a recent study used these cameras to detect respiration, cardiovascular patterns and muscle activations also for the detection of face expressions.

3. Integration into a Virtual Reality Platform Most of Contactless devices make available the Software Development Kit (SDK) that makes it possible to integrate third-party software into a Virtual Reality Platform [5]. Moreover, recently, the framework of contactless technologies has been widely extended, fostering from the classic clinical assessment to a more active use as input platform. In particular, the paradigm aim at implementing the integration of Contactless Technologies to provide the following advantages: 1) increased accuracy in assessment and diagnostics of ongoing rehabilitation processes; 2) ability to correlate specific mental states with specific activities executed into the environments, through the use of Kinect and eye-tracker for the synchronization of the ocular path within the action performed by the user; 3) ability to study the variables related to attention, perception, and cognition in the framework of simulations representing realistic situations and daily contexts, increasing the ecological validity of gathered data.

4. Conclusion Behavioral and medical sciences, and in particular psychiatry, can benefit enormously from operating within this methodological paradigm while using Virtual Reality. At the moment, paradigms in cognitive sciences worked on the obtrusiveness dimension of biosensors, however another perspective, that has not been considered, is the proximity level of biosensors. This paradigm fosters to reach an ambitious achievement: to switch from unobtrusiveness to a totally transparent technology, where a participant doesn't wear any sensors, being these out of the body's subjects, detecting bio-behavioral states at distance. In a typical experimental situation, participants and in particular patients affected by schizophrenia, autism, psychosis, bipolar disorders, or other mental disorders, reject the idea to wear biosensors or also to be touched, with important consequences on the outcome of these researches, potentially conditioned by the personal discomfort of the participants.

P. Cipresso et al. / Contactless Bio-Behavioral Technologies for Virtual Reality

153

Figure 2. Contactless technologies and the extracted features.

Acknowledgments This work was supported by the Italian funded project "VRehab. Virtual Reality in the Assessment and TeleRehabilitation of Parkinson's Disease and Post-Stroke Disabilities" - RF-2009-1472190.

References [1] P. Cipresso, Welcome to the Contactless Era, Contactless Bio-Behavioral Research Methods 1 (2012). [2] M. Malik et al., Heart rate variability, Circulation, 93 (1996), 1043-1065. [3] D. Giakoumis et al, Using Activity-Related Behavioural Features towards More Effective Automatic Stress Detection, PloS ONE 7.9 (2012): e43571. [4] M. Mauri et al., Psychophysiological signals associated with affective states, Engineering in Medicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE. IEEE, 2010. [5] G. Riva et al, NeuroVR 2--a free virtual reality platform for the assessment and treatment in behavioral health care, Studies in health technology and informatics 163 (2011): 493.