Yi-Hsuan Tsai
Abstract
Skies are common backgrounds in photos but are often less interesting due to the time of photographing. Professional photographers correct this by using sophisticated tools with painstaking efforts that are beyond the command of ordinary users. In this work, we propose an automatic background replacement algorithm that can generate realistic, artifact-free images with a diverse styles of skies. The key idea of our algorithm is to utilize visual semantics to guide the entire process including sky segmentation, search and replacement. First we train a deep convolutional neural network for semantic scene parsing, which is used as visual prior to segment sky regions in a coarse-to-fine manner. Second, in order to find proper skies for replacement, we propose a data-driven sky search scheme based on semantic layout of the input image. Finally, to re-compose the stylized sky with the original foreground naturally, an appearance transfer method is developed to match statistics locally and semantically. We show that the proposed algorithm can automatically generate a set of visually pleasing results. In addition, we demonstrate the effectiveness of the proposed algorithm with extensive user studies.
Supplemental Material
Available for Download
Supplemental files.
- Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., and Nayar, S. K. 2008. Face swapping: Automatically replacing faces in photographs. ACM Trans. Graph. (proc. SIGGRAPH) 27, 3. Google Scholar
Digital Library
- Boykov, Y., and Kolmogorov, V. 2004. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. PAMI, 1124--1137. Google ScholarDigital Library
- Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L. 2015. Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR.Google Scholar
- Dale, K., Johnson, M. K., Sunkavalli, K., Matusik, W., and Pfister, H. 2009. Image restoration using online photo collections. In ICCV.Google Scholar
- Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., and Sen, P. 2012. Image melding: Combining inconsistent images using patch-based synthesis. ACM Trans. Graph. (proc. SIGGRAPH) 31, 4. Google ScholarDigital Library
- Galasso, F., Nagaraja, N., Cardenas, T., Brox, T., and Schiele, B. 2013. A unified video segmentation benchmark: Annotation, metrics and analysis. In ICCV. Google ScholarDigital Library
- HaCohen, Y., Shechtman, E., Goldman, D. B., and Lischinski, D. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM Trans. Graph. (proc. SIGGRAPH) 30, 4. Google ScholarDigital Library
- Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Trans. Graph. (proc. SIGGRAPH) 26, 3. Google ScholarDigital Library
- He, K., Sun, J., and Tang, X. 2013. Guided image filtering. PAMI 35, 6, 1397--1409. Google ScholarDigital Library
- Hoiem, D., Efros, A. A., and Hebert, M. 2007. Recovering surface layout from an image. IJCV 75, 1. Google ScholarDigital Library
- Johnson, M. K., Dale, K., Avidan, S., Pfister, H., Freeman, W. T., and Matusik, W. 2011. Cg2real: Improving the realism of computer generated images using a large collection of photographs. IEEE Trans. Vis. Comp. Graph. 17, 9. Google ScholarDigital Library
- Kaufman, L., Lischinski, D., and Werman, M. 2012. Content-aware automatic photo enhancement. Comp. Graph. Forum 31, 8. Google ScholarDigital Library
- Kumar, M. P., Torr, P., and Zisserman, A. 2005. Obj cut. In CVPR. Google ScholarDigital Library
- Laffont, P.-Y., Ren, Z., Tao, X., Qian, C., and Hays, J. 2014. Transient attributes for high-level understanding and editing of outdoor scenes. ACM Trans. Graph. (proc. SIGGRAPH) 33, 4. Google ScholarDigital Library
- Lalonde, J.-F., and Efros, A. A. 2007. Using color compatibility for assessing image realism. In ICCV.Google Scholar
- Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Trans. Graph. (proc. SIGGRAPH) 26, 3. Google ScholarDigital Library
- Lalonde, J.-F., Narasimhan, S. G., and Efros, A. A. 2010. What do the sun and the sky tell us about the camera? IJCV 88, 1. Google ScholarDigital Library
- Lalonde, J.-F., Efros, A. A., and Narasimhan, S. G. 2011. Estimating the natural illumination conditions from a single outdoor image. IJCV 98, 2, 123--145. Google ScholarDigital Library
- Lazebnik, S., Schmid, C., and Ponce, J. 2006. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In CVPR. Google ScholarDigital Library
- Lee, J.-Y., Sunkavalli, K., Lin, Z., Shen, X., and Kweon, I. S. 2016. Automatic content-aware color and tone stylization. In CVPR.Google Scholar
- Liu, Y., and Yu, Y. 2012. Interactive image segmentation based on level sets of probabilities. IEEE Transactions on Visualization and Computer Graphics 18, 2, 202--213. Google ScholarDigital Library
- Liu, C., Yuen, J., and Torralba, A. 2011. Nonparametric scene parsing via label transfer. PAMI 33, 12, 2368--2382. Google ScholarDigital Library
- Liu, Y., Cohen, M., Uyttendaele, M., and Rusinkiewicz, S. 2014. Autostyle: Automatic style transfer from image collections to users image. Comp. Graph. Forum 33, 4.Google ScholarCross Ref
- Long, J., Shelhamer, E., and Darrell, T. 2015. Fully convolutional networks for semantic segmentation. In CVPR.Google Scholar
- Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. (proc. SIGGRAPH) 22, 3. Google ScholarDigital Library
- Pitié, F., and Kokaram, A. 2007. The linear mongekantorovitch linear colour mapping for example-based colour transfer. In CVMP.Google Scholar
- Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comp. Graph. Appl. 21, 5, 34--41. Google ScholarDigital Library
- Rother, C., Kolmogorov, V., and Blake, A. 2004. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. (proc. SIGGRAPH) 23, 3. Google ScholarDigital Library
- Shahrian, E., Rajan, D., Price, B., and Cohen, S. 2013. Improving image matting using comprehensive sampling sets. In CVPR. Google ScholarDigital Library
- Shih, Y., Paris, S., Durand, F., and Freeman, W. T. 2013. Data-driven hallucination of different times of day from a single outdoorphoto. ACM Trans. Graph. (proc. SIGGRAPH Asia) 32, 6. Google ScholarDigital Library
- Sunkavalli, K., Johnson, M. K., Matusik, W., and Pfister, H. 2010. Multi-scale image harmonization. ACM Trans. Graph. (proc. SIGGRAPH) 29, 4. Google ScholarDigital Library
- Tai, Y.-W., Jia, J., and Tang, C.-K. 2005. Local color transfer via probabilistic segmentation by expectation-maximization. In CVPR. Google ScholarDigital Library
- Tao, L., Yuan, L., and Sun, J. 2009. Skyfinder: Attribute-based sky image search. ACM Trans. Graph. (proc. SIGGRAPH) 28, 3. Google ScholarDigital Library
- Tao, M. W., Johnson, M. K., and Paris, S. 2013. Error-tolerant image compositing. IJCV 103, 2, 178--189. Google ScholarDigital Library
- Tighe, J., and Lazebnik, S. 2013. Superparsing: Scalable non-parametric image parsing with superpixels. IJCV 101, 2, 329--349. Google ScholarDigital Library
- Torralba, A., Oliva, A., Castelhano, M., and Hen-derso, J. M. 2006. Contextual guidance of attention in natural scenes: The role of global features on object search. Psychological Review 113, 10, 766--786.Google ScholarCross Ref
- Tsai, Y.-H., Hamsici, O., and Yang, M.-H. 2015. Adaptive region pooling for object detection. In CVPR.Google Scholar
- Wang, X., Yang, M., Zhu, S., and Lin, Y. 2013. Regionlets for generic object detection. In ICCV. Google ScholarDigital Library
- Wu, F., Dong, W., Knog, Y., Mei, X., Paul, J.-C., and Zhang, X. 2013. Content-based coulour transfer. Comp. Graph. Forum 32, 1.Google ScholarCross Ref
- Xue, S., Agarwala, A., Dorsey, J., and Rushmeier, H. 2012. Understanding and improving the realism of image composites. ACM Trans. Graph. (proc. SIGGRAPH) 31, 4. Google ScholarDigital Library
- Yan, Z., Zhang, H., Wang, B., Paris, S., and Yu, Y. 2016. Automatic photo adjustment using deep neural networks. ACM Trans. Graph. 35, 2. Google ScholarDigital Library
- Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., and Torr, P. 2015. Conditional random fields as recurrent neural networks. In ICCV. Google ScholarDigital Library
- Zhu, J.-Y., Krähenbühl, P., Shechtman, E., and Efros, A. A. 2015. Learning a discriminative model for the perception of realism in composite images. In ICCV. Google ScholarDigital Library
Index Terms
- Sky is not the limit: semantic-aware sky replacement
Recommendations
A physically-based night sky model
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniquesThis paper presents a physically-based model of the night sky for realistic image synthesis. We model both the direct appearance of the night sky and the illumination coming from the Moon, the stars, the zodiacal light, and the atmosphere. To accurately ...
Total relighting: learning to relight portraits for background replacement
We propose a novel system for portrait relighting and background replacement, which maintains high-frequency boundary details and accurately synthesizes the subject's appearance as lit by novel illumination, thereby producing realistic composite images ...
Performance relighting and reflectance transformation with time-multiplexed illumination
SIGGRAPH '05: ACM SIGGRAPH 2005 PapersWe present a technique for capturing an actor's live-action performance in such a way that the lighting and reflectance of the actor can be designed and modified in postproduction. Our approach is to illuminate the subject with a sequence of time-...
Comments