A Quality Control Mechanism for Networked Virtual Reality System with Video Capability Kazutoshi Fujikawa Osaka City University Osaka 558-8585, Japan
[email protected] Seiwoong Oh Tongmyong University of Information Technology Pusan 608-711, Korea
[email protected] Shinji Shimojo Osaka University Osaka 567-0047, Japan
[email protected]
Abstract Introduction of motion video including live video into networked virtual reality systems makes virtual spaces more attractive. To handle live video in networked virtual reality systems based on VRML, the scalability of networked virtual reality systems becomes very important on the Internet where the performance of the network and the end systems varies dynamically. In this paper, we propose a new quality control mechanism suitable for networked virtual reality systems with live video capability. Our approach is to introduce the notion of the importance of presence (IoP) which represents the importance of objects in virtual spaces. According to IoP, the degree of the deterioration of object presentation will be determined in case of the starvation of system resources.
1. Introduction Virtual Reality Modeling Language (VRML)[1] offers a capability that users on even inexpensive workstations or PCs can build their own computer generated threedimensional virtual spaces easily and publish them in the WWW fashion. Therefore, a lot of users on the Internet can navigate the same virtual spaces by exchanging the VRML description. In addition, Living Worlds[2] provides the framework where multiple users can share virtual spaces
Tomohiro Taira Sony Corporation Tokyo 141-0032, Japan
[email protected] Daisuke Kado Nara Institute of Science and Technology Nara 630-0101, Japan
[email protected] Hideo Miyahara Osaka University Osaka 560-8531, Japan
[email protected]
described in VRML. It defines the specification for exchanging the shared information. Moreover, several virtual reality systems take multi-user aspects into consideration[4, 5, 6]. In these systems, participants exchange avatar’s motion and behavior as well as the descriptions of virtual spaces. We call such kind of application as a networked virtual reality (NVR) system. In shared virtual spaces, introduction of motion video including live video makes virtual spaces more attractive. For example, using a live video such as an attendee’s facial image in a virtual meeting room enhances interactivity of NVR systems[7]. Motion video as well as avatar’s motion and behavior are considered as live information. In case of handling live information, NVR systems have to be scalable due to the nature of the Internet, that is, the performance of the network and the end system such as bandwidth and CPU’s load varies dynamically. Even if the network and the end system could preserve a certain amount of resources, the required resources for NVR systems change dynamically according to the user’s navigation. VRML provides the notion of the scalability called level of detail (LoD). Using LoD, a user can define the different degrees of the quality for each object in a virtual space, which consumes different degree of end system and network resources. However, most VRML systems change LoD of objects according not to the currently available system resources but to the distance between the user and the object. However, few works[3, 4, 5, 6] consider the scalability of
3D object
User’s eye vector
&H
l
the object, the angle between the object and the direction of the user’s movement, and the actual size of the object displayed on the computer screen, respectively. Then IoP value of the object can be calculated as a function of l, , and S ; f (l; S; ). Assume that there are N objects in a virtual space, and let IoPi be the IoP value of ith object (i = 1 : : : N ) as follows:
Xfflil;kS;iS; k;i k
IoPi = N
k=1
User
Figure 1. Positional relation between a user and an object.
the system in terms of the performance of the network and the end system, although most systems provide the scalability for large scale virtual environment. In this paper, we propose a new quality control mechanism for NVR systems with live video capability. The proposed NVR system can tolerate changes of both network and end system resources by reducing the quality of less important objects dynamically. Our approach is to introduce the notion of the importance of presence (IoP) which represents the degree of the importance of an object in virtual spaces. According to the IoP, the degree of the deterioration of objects will be determined in case of the starvation of system resources. We have been implementing a prototype system on SGI workstations.
2. Quality control for networked virtual reality Considering that NVR systems handle live information on the Internet, the scalability should be provided. By using LoD, a user can define the different degrees of the quality for each 3D object, but LoD considers only the distance between a user and objects as a criterion for choosing the proper quality. Therefore, we introduce a new notion of the scalability called importance of presence (IoP) for objects in a virtual space. IoP takes account of the user’s perception as a criterion for choosing the proper quality. When the resources of the network or the end system are going to starve, our NVR system deteriorates the quality of objects in a virtual space according to their IoP.
(
)
(
(1) )
Here, li , Si , and i are l, S , and of the ith object respectively. We determine f (li ; Si ; i ) as follows:
S f (li ; Si ; i ) = i (cos i + ) (2) li Here, 1 and 0, in order to be f (li ; Si ; i ) 0. represents how important the object behind user’s back. For example, if a user wants to ignore the object behind his back (f = 0), = 1 because cos i = ?1. is used to give a weight to a moving object such as a video object or an avatar because a user is likely to perceive a moving object more than a stationary object.
2.2. Object Scaling Algorithm If the network or the end system can not provide guaranteed service, their resources might be going to starve. Even if these systems can provide the resource reservation, the required resources for NVR systems may exceed the reserved one according to the user’s navigation. To adapt currently available resources, our NVR system deteriorates the quality of each object according to its IoP. Now, we explain the algorithm of the object scaling. In our NVR system based on VRML framework, we assume that:
The NVR system can detect the amount of currently available resources immediately. Each object has several grades of the quality. The object knows how much resources each grade of the quality consumes.
At first, our NVR system tries to display all objects visible to a user at the highest quality. When the resource starvation occurs, the quality of the object is deteriorated as follows:
2.1. Importance of presence
Now, we explain how to calculate the IoP of objects. Figure 1 shows the spatial relation between a user and an object. Let l, , and S be the distance between the user and
The NVR system allocate available resources according to the IoP of each object. Each object selects a grade of the quality appropriate for the
feedback
Server
Camera object information
IoP calculator
VRML parser
3D object
IoP Stored video
Quality controller live video System monitor
video
VRML file
3D object
Object streaming subsystem
Network
user status
Object streaming subsystem live video System monitor
video
3D object
Quality controller live video
User information sender
user status Rendering system
video 3D object
Scene graph builder
scene graph
Figure 3. Virtual museum. Client
Figure 2. Overview of our networked virtual reality system.
If the object can select the highest grade and the resources for allocation are still left, the remaining resources will be re-allocated to another object from the object which has the highest IoP.
This algorithm is applied to both the transmission and the presentation of virtual spaces, that is, both the network resources and the end system ones.
3. Implementation Figure 2 shows the overview of our NVR system with the proposed quality control mechanism. Our NVR system is implemented on SGI workstations. In our NVR system, the server consists of a VRML parser, a IoP calculator, a quality controller, and an object streaming subsystem (sender). The client consists of a rendering system, a scene graph builder, a user information sender, a quality controller, and an object streaming subsystem (receiver). A system monitor exists in both the server and the client. Our NVR system can handle a live video sending from a camera at a server as well as a video stored in a server. The video formats that the our NVR system can handle are motion-JPEG, intra-H.261, and raw RGB. Due to the limitation of the rendering system, every video format is transformed into raw RGB in the scene graph builder.
The IoP calculator acquires the information about user’s motion from user information sender and the information about the system resources from both system monitors, and calculates the IoP of each object. Then, the quality controller changes the quality of each object according to the calculated IoP and decides which object should be sent.
4. Experimental results Now, we describe the experimental results of our NVR system equipped with the proposed quality control mechanism. To prove that the proposed quality control mechanism can adapt the quality of the objects in a virtual space to the available resources of the network and the end system, we carried out an experiment. We construct a virtual museum with two video objects for this experiment(see Figure 3). In this figure, the image of a man dancing at right-center and the facial image are video objects. Both video images are RGB format. The video object of a man dancing is close to the user and another video object is far from the user. In this experiment, single server sends two pieces of motion video as well as the VRML description of the virtual museum to single client. The user moves through the virtual museum toward two video objects. To simplify the experiment, we calculate IoP only for two video objects, therefore we set = 1 for video objects and = 0 for the others. In this experiment, we set = 1. The other objects are sent from the server to the client according to the distance against the user. The result is shown in Figure 4 and 5. Video 1 represents the video object of the facial image and video 2 does
video 1 video 2
1 0.9 0.8
IoP
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
20
40 60 80 elapsed time(sec)
100
120
Figure 4. IoP values of two video objects.
500
throughput(kbytes/sec)
video 1 video 2 400
and the fine interactivity in moving through a large virtual space. To keep the quality of a virtual space even though network and local resources are going to starve, our NVR system introduces the Importance of presence (IoP) of objects and adapts to the starvation by reducing the degree of the quality of objects according to IoP. Experimental results show our NVR system equipped with the proposed quality control mechanism is efficient in respect of (1) adaptation for the available resources of the network and the end systems and (2) smooth navigation. Although our experiment was done on single-user environment, the proposed quality control mechanism can be applicable to the shared virtual environment such as a virtual meeting room. To provide higher interactivity, our NVR system provides the object streaming which can treat a large model interactively. Besides, our NVR system adopts the prediction of user’s movement for smooth navigation. Now, we are extending the prototype system for the multi-user environment.
Acknowledgments 300
200
100
0 0
20
40 60 80 elapsed time(sec)
100
120
Figure 5. Throughput of two video objects.
the video object of a man dancing in Figure 3. Figure 4 represents the variation of IoP value of two video objects. Until about 75 sec of the elapsed time, since the IoP of video 2 is always higher than the one of video 1, the quality of video 2 is always higher than the one of video 1. As the user passed by the object of video 2 about 75 sec of the elapsed time and the object went out of his view, the IoP of the object became zero suddenly. Figure 5 shows the throughput of each video object. After 75 sec of the elapsed time, the server stopped sending of video 2 and upgraded the quality of video 1 according to IoP. As the total throughput does not exceed about 500 kbytes/sec all the time in Figure 5, we can recognize that our NVR system can avoid the resource starvation.
5. Conclusion In this paper, a quality control mechanism is presented for the seamless integration of a live video into a virtual space
We thank Prof. Arikawa of Hiroshima City University and members of H2O project. This work was supported in part by Research for the Future Program of JSPS under the Project “Integrated Network Architecture for Advanced Multimedia Application Systems” (JSPS-RFTF97R16301) and “Researches on Advanced Multimedia Contents Processing” (JSPS-RFTF97P00501).
References [1] VRML 2.0. http://vrml.sgi.com/moving-worlds/spec/, August 1996. [2] Living worlds. http://www.livingworlds.com/, July 1997. [3] T. Funkhouser. RING: a client-server system for multi-user virtual environments. Proceedings of ACM SIGGRAPH Special Issue on 1995 Symposium on Interactive 3D Graphics, pages 85–92, 1995. Monterey, CA, USA. [4] O. Hagsand. Interactive multiuser VEs in the DIVE system. IEEE MultiMedia, 3(1):30–39, Spring 1996. [5] R. Lea, Y. Honda, K. Matsuda, O. Hagsand, and M. Stenius. Issues in the design of a scalable shared virtual environment for the internet. Proceedings of HICSS’97, (1), January 1997. Hawaii, USA. [6] M. Macedonia, M. Zyda, D. Pratt, P. Barham, and S. Zeswitz. NPSNET: a network software architecture for large scale virtual environments. Presence, 3(4):265–287, Fall 1994. [7] S. Oh, D. Kado, T. Taira, K. Fujikawa, T. Matsuura, S. Shimojo, M. Arikawa, and H. Miyahara. QoS mapping for networked virtual reality system. Proceedings of SPIE Conference on Performance and Control of Network Systems, pages 18–26, November 1997. Dallas, USA.
