HSI 2008
Krakow, Poland, May 25-27, 2008
Data Mining for Gaze Tracking System Breanna Heidenburg, Michael Lenisa, Daniel Wentzel, and Aleksander Malinowski, Senior member, IEEE
Abstract — An eye tracking system has been developed to control a computer cursor by tracking the user’s gaze. This tracking is achieved via a small form factor camera mounted near the user's eye, which communicates image data to a computer system. The computer system processes the image and determines the direction of the user's gaze. This position data is used to move a cursor on a display. By using this as an input method, the system allows full hands-free use of the computer. Additionally, an augmented reality system has been created using a head mounted micro display. This creates a heads-up display for the user. Keywords — human interface device, hands free operation, video-based gaze tracking, video-based eye-tracking.
position tracking a wearable head mounted display is used. The two most significant contributions of this research are the use of blob detection instead of circular shape in the image of pupil, and use of higher dimensional polynomials to calibrate and map the detected gaze position to the position on the computer screen. The remainder of the paper is organized as follows: the system block diagram and components are presented in section II. The image processing algorithms and novel data mining technology used to extract eye movement is presented in section III and is the main contribution of this manuscript to the advancement in this field. Experimental results of a system manufactured at Bradley University are presented in section IV.
I. INTRODUCTION
II. SYSTEM BLOCK DIAGRAM
tracking has been used for many decades as a tool to study human cognitive process including reading, driving, watching commercials, and others [1]. With advent of personal computers potential integration of such systems has been considered only relatively recently in 1991 by Jacob [2]. Successful attempts to employ gaze tracking as user interface were made to allow users with movement disabilities to type by looking at virtual keyboard [3], or for mouse pointer control [4]. This kind of system has also been used by users without any disabilities to enhance performance [5, 6]. The continuing absence of consumer-grade eye-tracking human computer interface is result of high price of eye tracking technology and intrusiveness of such systems even though technologies allowing doing so existed for many years [7]. Only very recently relatively low cost systems have been investigated, for example by Li et al in 2006 [8-10]. In a gaze tracking system, images of the eye are taken by a camera and sent to an image processing system. This image data is a picture of the user’s eye from a specific vantage point. The vantage point can be close to the user’s eye, as in a head mounted device, or further away, as in a device on a table near the user. A head mounted camera is used to monitor eye movements. The image is processed to determine the location of the user’s iris in relation to the rest of the eye. This location is passed through a specialized set of algorithms and gaze direction is determined. This data is used to position a cursor on a computer display. The cursor follows where the user is looking on the screen. In order to integrate the system as wearable computer and to simplify the problem of head
The proposed system can be broken down into THREE major hardware components: The eye tracking camera, the computer system, and the display. These components are shown in Figure 1, and are described in further detail below.
E
YE
This project is sponsored by Northrop Grumman. http://www.northropgrumman.com/ Contact person: Aleksander Malinowski, ECE Department, Bradley University, Peoria, Illinois, USA, +1-309-677-2776,
[email protected]
Fig. 1. System block diagram for the gaze tracking. A. Eye tracking camera The eye tracking camera will be located near the user’s eye, and will periodically sample images so as to digitize the user’s eye as an input to the computer system. As was discussed above, in this design the camera will be on a head mounted system. Other alternatives included using a camera that was placed farther away from the user, such as on a table. This creates the problem that the camera position is not stable in relation to the user’s eye, and thus calculating the position of their gaze in relation to the eye becomes difficult. In the head mounted system, the camera will be placed slightly below the user’s eye, looking back up at the eye. This positioning reduces the impairment on the user’s vision, by placing the camera in such a position that their other eye will be able to view what is blocked. As noted above, the stability seen between the camera position and the user’s eye ensures that the image processing application always receives data that can be processed correctly. B. Image processing application The image processing application will process the
image data from the sensor, and determine where the user is looking on the display. By means of having the image being very stable due to the camera positioning, the image processing application will always accept an image of the user’s eye. The image processing application processes the image and outputs cursor coordinates and is described in details in section III. C. Computer System The computer system’s operating system controls a cursor based on coordinates from image processing system. The operating system will accept cursor movement commands from the image processing application, and move the cursor accordingly. This cursor will interact with applications which are displayed back to the user. Currently, the operating system of choice is Windows XP, as drivers for most consumer level cameras are very robust in Windows XP (as compared to alternatives, such as Linux). Consumer grade webcam manufacturers, such as Logitech, often write very specialized drivers for main market operating systems like Windows XP. This often leaves lesser used operating systems with drivers that are barely functional. After preliminary research it was discovered that much of the hardware level control was unavailable in Linux versions of the device drivers and thus Linux development path was abandoned. D. Display A head mounted micro-display, which places the user interface directly in front of the user’s eye is used in the developed gaze tracking system. By positioning the display properly, the user is able to both view the display and have images taken of their eye. The display is transparent, and has variable opacity. These features facilitate the creation of a Heads-Up-Display (HUD) which can be overlaid onto the user’s normal vision. As this is still an emerging technology, the cost of such devices is not trivial. Entry level HMDs still start at about thousand dollars. However, with future high volume production future devices should be less pricy and thus allow for lower cost interface. III. IMAGE PROCESSING AND DATA MINING FOR GAZE TRACKING The image processing application involves a number of steps in order to determine cursor coordinates from an image of a user’s eye. These steps are shown in Figure 2, and are discussed in more detail below.
Fig. 2. Data flow in the eye image processing. Image filtering and preprocessing is applied to the
incoming image. Images coming from the camera may have inadequate light levels, or may contain characteristics which would make shape detection difficult. The filtering prepares the image for shape detection. Examples of filters being used are: • Smoothing • Color conversion • Threshold Filtering • Adjusting Exposure Time • Adjusting White Balance After preprocessing the system determines the position of the user’s eye. There are many possible ways to approach this portion of the software. Other similar projects for gaze tracking have mostly been based on shape detection, and have had good results. In our case shape detection has proven to have marginal results in this application. Strict shape detection has proven to have marginal results in this application. Strict shape detection, such as detecting a circle (the user’s pupil) in an image proves too unreliable when the user moves their eye. At certain points in the user’s gaze, the pupil loses its fully circular shape and takes on a more elliptical shape. This adversely affects the circular shape detection algorithm. To combat this problem a method of blob detection has been used. Instead of looking for strict shapes, the image processing algorithm searches for blobs (areas of similar color) matching that of a pupil. This allows for near perfect recognition of the pupil, and allows for identifying false positives. Once the position of the user’s pupil is found within the captured image, it is passed to a mapping algorithm. A multiple variable linear regression algorithm is used to map pupil locations to cursor locations on a screen. Coefficients for this algorithm are predetermined by means of a calibration algorithm. By accepting an eye position, the algorithm determines a cursor location. Significant effort was made to determine the order of the polynomial used for calibration as it is described in the section IV. After determining a cursor position, it may be found that the output may be noisy, causing the cursor to jump around on the screen. While it would be easy to attribute this to poor accuracy in the image detection algorithm, there are other physical factors associated with the human eye that could contribute to the cursor jitters. Movements known as saccadic eye movements are eye movements wherein the human eye moves within 1 degree of where the eye is looking. These saccadic eye movements, along with digitization of the pupil position can accumulate to noise in the cursor position. In order to combat this, the cursor position is filtered before being sent to the operating system. In addition to placing a cursor on a screen, the system needs to be able to initiate an action, in a similar fashion to a mouse click. Prior art has suggested various methods for detecting when the user is initiating an action. Methods include pushing a button, using a microphone for vocal queues, using facial gestures such as winks, and using eye gestures such as dwell time. After weighing such options, the design team has taken the dwell time approach. By monitoring the cursor position, actions are initiated when
the position has been constant within a certain tolerance level for a given amount of time. These tolerance values and delay times will be determined by monitoring experimental data. IV. EXPERIMENTAL RESULTS
display. To combat this, the design team developed an interface that used the corners of the screen in a very limited fashion. Of course, the data used in the figure above was from a very accurate set of calibration results. In system trials at Bradley University it was found that the calibration algorithm worked very well on some individuals, while the recognition and mapping of other user’s eyes was less than ideal. As was previously stated, a key area of interest for the design team was the responsiveness of such a system. The goal was to create a human machine interface which reacted quicker to human input than the conventional trackpad or mouse. An application was developed for recording a user’s reaction time. In this application the users were requested to move the cursor to a location on the screen, as well as click on the location. These trials were conducted with the use of a standard laptop track pad, a standard wired mouse, and the newly developed eye tracking system. A chart displaying the results of the experiment are shown here:
Fig. 3. The experimental Gaze Tracking System developed at Bradley University in action The experimental system developed at Bradley University [11] that is shown in Figure 3 has proven to have very good results. The two major sets of results that were measured in the trials at Bradley University were related to the accuracy of the cursor positioning system and the response time of the system. In regards to the accuracy of the cursor positioning system, the design team found that due to the mapping of a three dimensional shape – such as an eye – to a two dimensional surface – such as a computer screen – certain areas of the screen were more error prone than others. Trials were performed on multiple mapping algorithms of various orders. In these trials We found that a 4th order parametric mapping was the most effective at minimizing errors. Error in the mapping algorithm was analyzed and various three dimensional plots were generated. A sample of the 4th degree error plot is shown here:
Fig. 5. Comparison of mean time required for a user to position a cursor on an object on the screen using different means of computer input. Figure 5 depicts the mean of the times required for a user to position a cursor on an object on the screen by various means of computer input. For the eye tracking method, extremely long durations – longer than 2 seconds – have been removed from the data set. These long durations represent locations on the screen where the user’s eye either became difficult to recognize or the calibration algorithm produced error larger than the size of the object displayed on the screen (resulting in the user not being able to place the cursor on the object without first compensating for a poor calibration.) The results prove that eye tracking methods can in fact accelerate human computer input. However, due to the accuracy, there are still developments that need to be made in this field before such a technology can become a full replacement for the current means of human-computer interaction. V. CONCLUSION
Fig. 4. Distribution of error of gaze detection algorithm with output calibrated using 4th order polynomial. As it is shown in Figure 4, error in the mapping algorithms used is relegated to the less used corners of the
The gaze tracking system described herein has shown beneficial merit in terms of field applications. The field of research pertaining to accurate gaze tracking has only become a feasible research in the past few years due to pricing of acceptable cameras. Also, most of the research
that has been done in this field has been solely devoted to tracking the user’s eye, but has neglected to give any return back to the user via a display. Our image processing approach allows for successful gaze tracking using a camera that observes the eye from a position below an eye and does not need to placed directly in front of it and thus does not require additional expensive optical settings. ACKNOWLEDGMENT Authors would like to thank Northrop Grumman for sponsoring the project by purchasing a durable and advanced technology head mounted display. REFERENCES [1]
Duchowski, A. 2002. A breadth-first survey of eye-tracking applications. Behavior Research Methods, Instruments and Computers, vol. 34, No. 4, pp. 455-470. [2] Jacob, R. 1991. The use of eye movements in human-computer interaction techniques: what you look at is what you get. ACM Transactions on Information Systems, vol. 9, No. 2, pp. 152-169. [3] Majaranta, P., Raiha, K., “Twenty years of eye typing: systems and design issues,” Proceedings of the symposium on Eye tracking research and applications, 2002, pp. 15-22. [4] Hornof, A. J., Cavender, A., Hoselton, R., “Eyedraw: A system for drawing pictures with eye movements,” ACM SIGACCESS Conference on Computers and Accessibility, Atlanta, Georgia, 2004, pp. 86-93. [5] Silbert, L., Jacob, R., “Evaluation of eye gaze interaction,” Proceedings of the SIGCHI conference on Human factors in computing systems, 2000, pp. 281-288. [6] Tanriverdi, V., Jacob, R., “Interacting with eye movements in virtual environments,” Proceedings of the SIGCHI conference on Human factors in computing systems, 2000, pp. 265-272. [7] Young, L., Sheena, D., “Survey of eye movement recording methods,” Behavior Research Methods and Instrumentation 7, 1975, pp. 397-429. [8] Babcock J., Pelz, J., “Building a lightweight eyetracking headgear,” Eye Tracking Research and Applications Symposium, 2004, pp. 109-114. [9] Pelz, J., Canosa, R., Babcock J., Kucharczyk, D., Silver, A., Konno, D., “Portable eyetracking: A study of natural eye movements” Proceedings of the SPIE, Human Vision and Electronic Imaging, 2000, pp. 566-582. [10] Li, D., Babcock J., and Parkhurst, J.D.. “openEyes: a low-cost head-mounted eye-tracking solution.” AMC Eye Tracking Research and Applications Symposium, 2006. [11] Heidenburg, B., Lenisa, M., Wentzel, D., Malinowski, A., February 2008, Gaze Tracking System, Project Web Site, Available: http://cegt201.bradley.edu/projects/proj2008/iralar/
