Panoramic imaging system for camera phones

2.2-5

in Consumer Electronics (ICCE), 2010 Digest of Technical Papers International Conference on, 9-13 2010, pp. 45 –46

Panoramic Imaging System for Camera Phones Kari Pulli1, Marius Tico1, Yingen Xiong1 , Xianglin Wang1, and Chia-Kai Liang1,2 1 Nokia Research Center 2National Taiwan University Abstract-- We introduce a mobile system for high resolution panoramic image creation. In our system the user can rotate the camera arbitrarily and see the preview panorama in real-time. The system automatically captures high resolution images and generates a high-quality result with unlimited viewing angle. We employ a coarse-to-fine method for high-quality registration, and a seam-finding method to remove the ghosting effect due to moving objects. The proposed system has been tested on several types of camera phones, and the tests reveal that the system can efficiently provide a high-quality panoramic image in spite of the low computational power and memory available in such devices.

I. INTRODUCTION Panorama can provide a stunning wide-angle representation of the scene beyond a normal photograph. However, it is painstaking to capture a good panorama. Typically, user has to fix the camera on a tripod to capture the images. Those images are then stitched into a high resolution panorama via computationally demanding PC software, such as Adobe Photoshop© . This two-step procedure makes it difficult to infer the appearance and quality of the final result during acquisition. This disadvantage can be overcome by mobile implementations [1, 2]. However, designing a high-quality mobile panoramic system is challenging due to low computational resources available in such devices. Therefore, previous methods usually impose strong assumptions to the input data. For example, most systems can only capture images along a onedimensional path, and require the user to manually align the images to simplify or bypass the registration step. Also, they all use fast methods to blend the captured images. However, they cannot handle moving objects and result in serious ghosting effects. II. THE PROPOSED SYSTEM We propose a complete solution for mobile panoramic imaging which can handle unlimited angle of view (both horizontally and vertically) and moving objects, and generate results comparable to the PC software. The system flow is shown in Fig. 1. Our system allows the user to move the camera arbitrarily by tracking camera motion in real-time, and automatically determining when to take high-resolution images. The highresolution images are then registered on the spherical manifold. After warping, all images are stitched together using a highquality blending method that removes all artifacts. The user can examine the resulting panorama on the phone with interactive panning and zooming operations. A. Image capturing Camera motion is estimated by tracking consecutive low-

Fig. 1. System flow of the proposed solution. The final 360° panorama here is generated from 18 input images.

Fig. 2. Illustration of the capture interface. As the user moves the camera, the motion is tracked in real-time and shown as a red box. The next image is automatically captured when the motion is large enough. The preview panorama is immediately updated.

resolution viewfinder frames captured at 30 frames per second, using the algorithm proposed in our previous work [3]. During capture, a low-resolution panorama generated from previously captured images is displayed, along with a moving box indicating the position of the current frame. So the user can easily decide the size and shape of panorama (see Fig. 2). When the camera motion with respect to the previous image exceeds a threshold, the high-resolution images for the current position are automatically captured. B. Image registration and warping Image registration of high resolution captured image frames is essential for ensuring an accurate and seamless representation of the panoramic scene. We designed a hybrid registration approach that follows a coarse-to-fine strategy. The algorithm starts by matching two images at coarse resolution in order to estimate large displacements, followed by a progressive refinement of the registration parameter by matching salient image features (e.g., corners) at finer resolution levels. Both image and feature matching operations are carried out using similarity metrics that are invariant to illumination changes. In addition, feature-based matching is performed in conjunction with RANSAC to achieve the necessary robustness to outliers, such as moving objects. We enable an unlimited angle of view by mapping the panorama onto a spherical manifold. In order to reduce the number of image warping operations, the proposed registration method acts directly in the final spherical manifold coordinates. More-

978-1-4244-4316-1/10/$25.00 ©2010 IEEE Authorized licensed use limited to: Nokia Information Services. Downloaded on July 02,2010 at 04:56:22 UTC from IEEE Xplore. Restrictions apply.

(a)

4 Inputs (2400× 700)

(b)

5 Inputs (2600× 670)

(c)

(d)

(e)

Alpha blending

After seam finding

Poisson blending

Fig. 3. High-resolution panorama results of the proposed system. The last row also shows the inputs of a high-dynamic-range scene. TABLE I TEST DEVICE SPECIFICATION AND PERFORMANCE Processing time Nokia N95 8GB ARM 11 4 inputs (150º), alpha CPU 332Mhz 4 inputs (150º), Poisson Memory 64 MB 5 inputs (180º), alpha Input image size 1024× 768 5 inputs (180º), Poisson

18 sec. 36 sec. 23 sec. 44 sec.

over, in contrast to the spherical registration proposed in [6], our approach achieves a higher efficiency by carrying out also the corresponding feature selection and matching directly in the spherical coordinates. Noting that spherical manifold warping changes the relative coordinates between features but has a small effect on local neighborhood around each feature, we perform feature matching directly in the spherical manifold domain without any intermediate image warping operations. The image warping and spherical mapping are then performed in a single step and only once for each input image, based on the estimated registration parameters. C. Image blending Properly blending the registered images is crucial to the quality of the resulting panorama. While a complex blending method can generate a plausible result, a fast method can provide a preview to the user immediately after the acquisition. Our system provides two methods for both requirements. In the first method called alpha blending, a smooth mask peaked at the image center is assigned for each source image. Then the masks are warped together with the corresponding images using GPU acceleration. The final pixel value is the average of overlapped pixels, weighted by the mask values. To remove artifacts due to moving objects, parallax, and registration errors, we find the optimal seams in the overlapping areas of the source images using graph cut [4], and let each pixel in the result only come from one input image; thus ghosting and blurring is reduced. To further remove the remaining color differences along the seams, we apply a Poisson blending [5]. It performs optimization to find a least-distortion panorama from the combined gradient field. In particular, we solve the Poisson equation in the coarse multi-spline domain [7], which reduces the memory and runtime by orders of magnitude.

Fig. 4. Effects of different blending methods. (a) Input images. (b) The final panorama using Poisson blending. (c) Alpha blending with artifacts. (d) Labeling for (e) Poisson blending.

III. RESULTS AND DISCUSSIONS Several panorama results captured by our system are shown in Figs. 1, 3, and 4 (b). We can see that the proposed system can generate artifact-free panoramas of the scene with crowded dynamic objects, arbitrary viewing angles, and even a widely varying dynamic range of inputs. Fig. 4 illustrates the effects of blending. The fast alpha blending has ghosting artifacts, indicated by the yellow arrows in Fig. 4 (c). After the optimal seams are obtained, ghosting is removed (Fig. 4 (d)). After Poisson blending, all artifacts are removed (Fig. 4 (e)). The proposed panoramic imaging system has been implemented on several mobile devices (e.g., Nokia E71 and N95) and tested on various scenes with different conditions. The results shown here are obtained on a Nokia N95 (see Table I). The processing time is linear in the number of input images. For high-quality results, image registration and Poisson blending each take 40% of the processing time. IV. CONCLUSION In this paper we have proposed a complete system for automatic panoramic imaging. Our system allows the user to move the camera freely and generates the artifact-free panorama with unlimited viewing angles. The proposed system is efficient enough to implement on mobile consumer electronics, yet generates results comparable to complex PC software. REFERENCES [1] [2] [3] [4] [5] [6] [7]

S.J. Ha, et al., "Panorama mosaic optimization for mobile camera systems," IEEE Trans. Consumer Electronics, vol. 53, no. 4, 2007. K. Park, "Photographing for on-line mobile panorama," in Int’l Conf. Consumer Electronics, Jan. 2008. A. Adams, N. Gelfand, and K. Pulli, "Viewfinder alignment," Computer Graphics Forum, vol. 27, no. 2, pp. 597– 606, 2008. A. Agarwala, et al., "Interactive digital photomontage," ACM Trans. Graph., vol. 23, no. 3, pp. 294– 302, 2004. P. Pérez, M. Gangnet, and A. Blake, "Poisson image editing," ACM Trans. Graph., vol. 22, no. 3, pp. 313– 318, 2003. R. Szeliski and H.-Y. Shum, "Creating full view panoramic image mosaics and environment maps," in Proc. SIGGRAPH’97, 1997. R. Szeliski, M. Uyttendaele, and D. Steedly "Fast Poisson blending using multi-splines," Microsoft Technical Report, 2008.

Authorized licensed use limited to: Nokia Information Services. Downloaded on July 02,2010 at 04:56:22 UTC from IEEE Xplore. Restrictions apply.