Yichang Shih
Abstract
We introduce "time hallucination": synthesizing a plausible image at a different time of day from an input image. This challenging task often requires dramatically altering the color appearance of the picture. In this paper, we introduce the first data-driven approach to automatically creating a plausible-looking photo that appears as though it were taken at a different time of day. The time of day is specified by a semantic time label, such as "night".
Our approach relies on a database of time-lapse videos of various scenes. These videos provide rich information about the variations in color appearance of a scene throughout the day. Our method transfers the color appearance from videos with a similar scene as the input photo. We propose a locally affine model learned from the video for the transfer, allowing our model to synthesize new color data while retaining image details. We show that this model can hallucinate a wide range of different times of day. The model generates a large sparse linear system, which can be solved by off-the-shelf solvers. We validate our methods by synthesizing transforming photos of various outdoor scenes to four times of interest: daytime, the golden hour, the blue hour, and nighttime.
Supplemental Material
Available for Download
Supplemental material.
- Barnes, C., Shechtman, E., Goldman, D., and Finkelstein, A. 2010. The generalized patchmatch correspondence algorithm. European Conference on Computer Vision, 29--43. Google Scholar
Digital Library
- Bousseau, A., Paris, S., and Durand, F. 2009. User-assisted intrinsic images. In ACM Transactions on Graphics (SIGGRAPH), vol. 28, 130. Google ScholarDigital Library
- Bychkovsky, V., Paris, S., Chan, E., and Durand, F. 2011. Learning photographic global tonal adjustment with a database of input/output image pairs. In IEEE Conference on Computer Vision and Pattern Recognition, 97--104. Google ScholarDigital Library
- Caputo, R. 2005. In Potography field guide, National Geographics, 104--115.Google Scholar
- Dalal, N., and Triggs, B. 2005. Histograms of oriented gradients for human detection. In IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, 886--893. Google ScholarDigital Library
- Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In Annual Conference on Computer Graphics and Interactive Techniques, 341--346. Google ScholarDigital Library
- Eisemann, E., and Durand, F. 2004. Flash photography enhancement via intrinsic relighting. In ACM Transactions on Graphics (SIGGRAPH), vol. 23, 673--678. Google ScholarDigital Library
- Freeman, W., Pasztor, E., and Carmichael, O. 2000. Learning low-level vision. International Journal of Computer Vision 40, 1, 25--47. Google ScholarDigital Library
- Freeman, W., Jones, T., and Pasztor, E. 2002. Example-based super-resolution. IEEE Computer Graphics and Applications 22, 2, 56--65. Google ScholarDigital Library
- HaCohen, Y., Shechtman, E., Goldman, D., and Lischinski, D. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM Transactions on Graphics (SIGGRAPH) 30, 4, 70. Google ScholarDigital Library
- Hays, J., and Efros, A. 2007. Scene completion using millions of photographs. In ACM Transactions on Graphics (SIGGRAPH), vol. 26, 4. Google ScholarDigital Library
- He, K., Sun, J., and Tang, X. 2010. Guided image filtering. European Conference on Computer Vision, 1--14. Google ScholarDigital Library
- Hertzmann, A., Jacobs, C., Oliver, N., Curless, B., and Salesin, D. 2001. Image analogies. In Conference on Computer Graphics and Interactive Techniques, 327--340. Google ScholarDigital Library
- Irony, R., Cohen-Or, D., and Lischinski, D. 2005. Colorization by example. In Eurographics Conference on Rendering Techniques, 201--210. Google ScholarDigital Library
- Jacobs, N., Roman, N., and Pless, R. 2007. Consistent temporal variations in many outdoor scenes. In IEEE Conference on Computer Vision and Pattern Recognition, 1--6.Google Scholar
- Kopf, J., Neubert, B., Chen, B., Cohen, M., Cohen-Or, D., Deussen, O., Uyttendaele, M., and Lischinski, D. 2008. Deep photo: Model-based photograph enhancement and viewing. In ACM Transactions on Graphics (SIGGRAPH), vol. 27, 116. Google ScholarDigital Library
- Laffont, P.-Y., Bousseau, A., Paris, S., Durand, F., Drettakis, G., et al. 2012. Coherent intrinsic images from photo collections. ACM Transactions on Graphics 31, 6. Google ScholarDigital Library
- Lalonde, J., Efros, A., and Narasimhan, S. 2009. Webcam clip art: Appearance and illuminant transfer from time-lapse sequences. In ACM Transactions on Graphics (SIGGRAPH), vol. 28, 131. Google ScholarDigital Library
- Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. In ACM Transactions on Graphics (SIGGRAPH), vol. 23, 689--694. Google ScholarDigital Library
- Levin, A., Lischinski, D., and Weiss, Y. 2006. A closed form solution to natural image matting. In IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, 61--68. Google ScholarDigital Library
- Liu, C., Yuen, J., Torralba, A., Sivic, J., and Freeman, W. 2008. Sift flow: Dense correspondence across different scenes. European Conference on Computer Vision, 28--42. Google ScholarDigital Library
- Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., and Toyama, K. 2004. Digital photography with flash and no-flash image pairs. In ACM Transactions on Graphics (SIGGRAPH), vol. 23, 664--672. Google ScholarDigital Library
- Pitie, F., Kokaram, A., and Dahyot, R. 2005. N-dimensional probability density function transfer and its application to color transfer. In IEEE International Conference on Computer Vision, vol. 2, 1434--1439. Google ScholarDigital Library
- Pouli, T., and Reinhard, E. 2011. Progressive color transfer for images of arbitrary dynamic range. Computers and Graphics 35, 1, 67--80. Google ScholarDigital Library
- Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Computer Graphics and Applications 21, 5, 34--41. Google ScholarDigital Library
- Rowell, G. 2012. In Mountain Light, Sierra Club Books. Xiao, J., Hays, J., Ehinger, K., Oliva, A., and Torralba, A. 2010. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE conference on Computer Vision and Pattern Recognition, 3485--3492.Google Scholar
- Yedidia, J. S., Freeman, W. T., Weiss, Y., et al. 2000. Generalized belief propagation. In Advances in Neural Information Processing Systems, vol. 13, 689--695.Google ScholarDigital Library
Index Terms
- Data-driven hallucination of different times of day from a single outdoor photo
Recommendations
Stereo Day-for-Night: Retargeting Disparity for Scotopic Vision
Special Issue SAP 2014Several approaches attempt to reproduce the appearance of a scotopic low-light night scene on a photopic display (“day-for-night”) by introducing color desaturation, loss of acuity, and the Purkinje shift toward blue colors. We argue that faithful ...
Casual Stereoscopic Photo Authoring
Stereoscopic 3D displays become more and more popular these years. However, authoring high-quality stereoscopic 3D content remains challenging. In this paper, we present a method for easy stereoscopic photo authoring with a regular (monocular) camera. ...
Comments