Abstract
This paper addresses the problem of remapping the disparity range of stereoscopic images and video. Such operations are highly important for a variety of issues arising from the production, live broadcast, and consumption of 3D content. Our work is motivated by the observation that the displayed depth and the resulting 3D viewing experience are dictated by a complex combination of perceptual, technological, and artistic constraints. We first discuss the most important perceptual aspects of stereo vision and their implications for stereoscopic content creation. We then formalize these insights into a set of basic disparity mapping operators. These operators enable us to control and retarget the depth of a stereoscopic scene in a nonlinear and locally adaptive fashion. To implement our operators, we propose a new strategy based on stereoscopic warping of the input video streams. From a sparse set of stereo correspondences, our algorithm computes disparity and image-based saliency estimates, and uses them to compute a deformation of the input views so as to meet the target disparities. Our approach represents a practical solution for actual stereo production and display that does not require camera calibration, accurate dense depth maps, occlusion handling, or inpainting. We demonstrate the performance and versatility of our method using examples from live action post-production, 3D display size adaptation, and live broadcast. An additional user study and ground truth comparison further provide evidence for the quality and practical relevance of the presented work.
Supplemental Material
Available for Download
In this zip file we have two version of our video. stereoscopic_warping_anaglyph.mov - This version is shown in Red-Cyan Anaglyph for viewing on general displays. stereoscopic_warping_topdown.mov - This version is shown in right over left format for viewing on 3D displays with an appropriate player. We recommend Stereoscopic Player from www.3dtv.at to view stereo content.
- 3dtv.at, 2010. Stereoscopic player, Jan. http://www.3dtv.at/.Google Scholar
- Agrawal, A., and Raskar, R. 2007. Gradient domain manipulation techniques in vision and graphics. In ICCV Courses.Google Scholar
- Akeley, K., Watt, S. J., Girshick, A. R., and Banks, M. S. 2004. A stereo display prototype with multiple focal distances. ACM Trans. Graph. 23, 3, 804--813. Google ScholarDigital Library
- Baker, S., and Matthews, I. 2004. Lucas-Kanade 20 years on: A unifying framework. IJCV 56, 3, 221--255. Google ScholarDigital Library
- Banks, M. S., Gepshtein, S., and Landy, M. S. 2004. Why is spatial stereoresolution so low? Journal of Neuroscience 24, 2077--2089.Google ScholarCross Ref
- Bleyer, M., Gelautz, M., Rother, C., and Rhemann, C. 2009. A stereo approach that handles the matting problem via image warping. In CVPR, 501--508.Google Scholar
- Burt, P., and Juelsz, B. 1980. A disparity gradient limit for binocular fusion. Science 208, 4444 (5), 615--617.Google Scholar
- Carroll, R., Agrawala, M., and Agarwala, A. 2009. Optimizing content-preserving projections for wide-angle images. ACM Trans. Graph. 28, 3. Google ScholarDigital Library
- Criminisi, A., Blake, A., Rother, C., Shotton, J., and Torr, P. H. 2007. Efficient dense stereo with occlusions for new view-synthesis by four-state dynamic programming. Int. J. Comput. Vision 71, 1, 89--110. Google ScholarDigital Library
- Cutting, J. E., and Vishton, P. M. 1995. Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth. In Handbook of perception and cognition, Perception of space and motion, W. Epstein and S. Rogers, Eds., vol. 5. Academic Press, San Diego, CA.Google Scholar
- David, H. A. 1963. The Method of Paired Comparisons. Charles Griffin & Company.Google Scholar
- Feldmann, I., Schreer, O., and Kauff, P. 2003. Nonlinear depth scaling for immersive video applications. WIAMIS.Google Scholar
- Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The lumigraph. In SIGGRAPH, 43--54. Google ScholarDigital Library
- Guo, C., Ma, Q., and Zhang, L. 2008. Spatio-temporal saliency detection using phase spectrum of quaternion Fourier transform. CVPR.Google Scholar
- Guttmann, M., Wolf, L., and Cohen-Or, D. 2009. Semiautomatic stereo extraction from video footage. In ICCV.Google Scholar
- Hoffman, D. M., Girshick, A. R., Akeley, K., and Banks, M. S. 2008. Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. Journal of Vision 8, 3 (3), 1--30.Google ScholarCross Ref
- Howard, I. P., and Rogers, B. J. 2002. Seeing in Depth. Oxford University Press, New York, USA.Google Scholar
- Kim, M.-B., Lee, S., Choi, C., Um, G.-M., Hur, N.-H., and Kim, J.-W. 2008. Depth scaling of multiview images for auto-multiscopic 3D monitors. In 3DTV08.Google Scholar
- Krähenbühl, P., Lang, M., Hornung, A., and Gross, M. 2009. A system for retargeting of streaming video. ACM Trans. Graph. 28, 5. Google ScholarDigital Library
- Lambooij, M., IJsselsteijn, W., Fortuin, M., and Heynderickx, I. 2009. Visual discomfort and visual fatigue of stereoscopic displays: A review. Journal of Imaging Science and Technology 53, 3, 030201.Google ScholarCross Ref
- Levoy, M., and Hanrahan, P. 1996. Light field rendering. In SIGGRAPH, 31--42. Google ScholarDigital Library
- Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3D video stabilization. ACM Trans. Graph. 28, 3. Google ScholarDigital Library
- Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 2, 91--110. Google ScholarDigital Library
- Mahajan, D., Huang, F.-C., Matusik, W., Ramamoorthi, R., and Belhumeur, P. N. 2009. Moving gradients: a path-based method for plausible image interpolation. ACM Trans. Graph. 28, 3. Google ScholarDigital Library
- Matusik, W., and Pfister, H. 2004. 3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Trans. Graph. 23, 3, 814--824. Google ScholarDigital Library
- Mendiburu, B. 2009. 3D Movie Making: Stereoscopic Digital Cinema from Script to Screen. Focal Press.Google Scholar
- Mobile 3DTV, 2010. Stereo video data-sets, Jan. http://sp.cs.tut.fi/mobile3dtv/stereo-video/.Google Scholar
- Neuman, R., 2009. Personal Communication with Robert Neuman, Chief Stereographer, Disney Animation Studios.Google Scholar
- Paris, S., and Durand, F. 2006. A fast approximation of the bilateral filter using a signal processing approach. In ECCV (4), 568--580. Google ScholarDigital Library
- Pritch, Y., Ben-Ezra, M., and Peleg, S. 2000. Automatic disparity control in stereo panoramas (omnistereo). In OMNIVIS. Google ScholarDigital Library
- Reinhard, E., Ward, G., Pattanaik, S., and Debevec, P. 2005. High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting. Morgan Kaufmann. Google ScholarDigital Library
- Sattler, T., Leibe, B., and Kobbelt, L. 2009. SCRAMSAC: Improving RANSAC's efficiency with a spatial consistency filter. In ICCV.Google Scholar
- Seitz, S., and Dyer, C. 1996. View morphing. In SIGGRAPH 96, 21--30. Google ScholarDigital Library
- Shade, J., Gortler, S. J., Li-wei, H., and Szeliski, R. 1998. Layered depth images. In SIGGRAPH, 231--242. Google ScholarDigital Library
- Shamir, A., and Sorkine, O. 2009. Visual media retargeting. In SIGGRAPH ASIA Courses. Google ScholarDigital Library
- Siegel, M., and Nagata, S. 2000. Just enough reality: Comfortable 3-D viewing via microstereopsis. IEEE Transactions on Circuits and Systems for Video Technology 10, 3 (4), 387--396. Google ScholarDigital Library
- Smolic, A., Mller, K., Dix, K., Merkle, P., Kauff, P., and Wiegand, T. 2008. Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems. In ICIP, IEEE, 2448--2451.Google Scholar
- Stelmach, L. B., Tam, W. J., Meegan, D. V., and Vincent, A. 2000. Stereo image quality: effects of mixed spatio-temporal resolution. IEEE Transactions on Circuits and Systems for Video Technology 10, 2, 188--193. Google ScholarDigital Library
- Sun, G., and Holliman, N. 2009. Evaluating methods for controlling depth perception in stereoscopic cinematography. Stereoscopic Displays and Virtual Reality Systems XX, Proceedings of SPIE 7237 (1).Google Scholar
- the Foundry, 2010. Ocular, Nuke, Jan. http://www.thefoundry.co.uk/.Google Scholar
- van den Hengel, A., Dick, A. R., Thormählen, T., Ward, B., and Torr, P. H. S. 2007. Videotrace: rapid interactive scene modelling from video. ACM Trans. Graph. 26, 3, 86. Google ScholarDigital Library
- Wang, C., and Sawchuk, A. A. 2008. Disparity manipulation for stereo images and video. SPIE, vol. 6803.Google Scholar
- Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4, 600--612. Google ScholarDigital Library
- Wang, Y.-S., Fu, H., Sorkine, O., Lee, T.-Y., and Seidel, H.-P. 2009. Motion-aware temporal coherence for video resizing. ACM Trans. Graph. 28, 5. Google ScholarDigital Library
- Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., and Bischof, H. 2009. Anisotropic Huber-L1 optical flow. In British Machine Vision Conference (BMVC).Google Scholar
- Weyrich, T., Deng, J., Barnes, C., Rusinkiewicz, S., and Finkelstein, A. 2007. Digital bas-relief from 3D scenes. ACM Trans. Graph. 26, 3, 32. Google ScholarDigital Library
- Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S. A. J., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23, 3, 600--608. Google ScholarDigital Library
Index Terms
- Nonlinear disparity mapping for stereoscopic 3D
Recommendations
Nonlinear disparity mapping for stereoscopic 3D
SIGGRAPH '10: ACM SIGGRAPH 2010 papersThis paper addresses the problem of remapping the disparity range of stereoscopic images and video. Such operations are highly important for a variety of issues arising from the production, live broadcast, and consumption of 3D content. Our work is ...
Content-aware disparity adjustment for different stereo displays
In this paper, we present an effective disparity mapping method for binocular stereoscopic image. It is inspired by the observation that its displayed depth would change, when a stereoscopic image is displayed on different size screens. The phenomenon ...
Depth information from binocular disparity and familiar size is combined when reaching towards virtual objects
VRST '16: Proceedings of the 22nd ACM Conference on Virtual Reality Software and TechnologyReaching movements towards stereoscopically presented virtual objects have been reported to be imprecise. This might be a problem for touch interaction with virtual environments. Estimating the distance to an object in personal space relies on binocular ...
Comments