Abstract
Cloud image processing is often proposed as a solution to the limited computing power and battery life of mobile devices: it allows complex algorithms to run on powerful servers with virtually unlimited energy supply. Unfortunately, this overlooks the time and energy cost of uploading the input and downloading the output images. When transfer overhead is accounted for, processing images on a remote server becomes less attractive and many applications do not benefit from cloud offloading. We aim to change this in the case of image enhancements that preserve the overall content of an image. Our key insight is that, in this case, the server can compute and transmit a description of the transformation from input to output, which we call a transform recipe. At equivalent quality, our recipes are much more compact than JPEG images: this reduces the client's download. Furthermore, recipes can be computed from highly compressed inputs which significantly reduces the data uploaded to the server. The client reconstructs a high-fidelity approximation of the output by applying the recipe to its local high-quality input. We demonstrate our results on 168 images and 10 image processing applications, showing that our recipes form a compact representation for a diverse set of image filters. With an equivalent transmission budget, they provide higher-quality results than JPEG-compressed input/output images, with a gain of the order of 10 dB in many cases. We demonstrate the utility of recipes on a mobile phone by profiling the energy consumption and latency for both local and cloud computation: a transform recipe-based pipeline runs 2--4x faster and uses 2--7x less energy than local or naive cloud computation.
Supplemental Material
Available for Download
Supplemental files.
- Aubry, M., Paris, S., Hasinoff, S. W., Kautz, J., and Durand, F. 2014. Fast local laplacian filters: Theory and applications. ACM Trans. Graph. 33, 5 (Sept.), 167:1--167:14. Google ScholarDigital Library
- Barr, K. C., and Asanović, K. 2006. Energy-aware lossless data compression. ACM Trans. Comput. Syst. 24, 3 (Aug.), 250--291. Google ScholarDigital Library
- Berthouzoz, F., Li, W., Dontcheva, M., and Agrawala, M. 2011. A framework for content-adaptive photo manipulation macros: Application to face, landscape, and global manipulations. ACM Transactions on Graphics 30, 5. Google ScholarDigital Library
- Burt, P. J., and Adelson, E. H. 1983. The laplacian pyramid as a compact image code. IEEE Transactions on Communications 31, 4, 532--540.Google ScholarCross Ref
- Bychkovsky, V., Paris, S., Chan, E., and Durand, F. 2011. Learning photographic global tonal adjustment with a database of input / output image pairs. In IEEE Conference on Computer Vision and Pattern Recognition. Google ScholarDigital Library
- Chen, Q., Li, D., and Tang, C.-K. 2012. Knn matting. In IEEE Conference on Computer Vision and Pattern Recognition, 869--876. Google ScholarDigital Library
- Deutsch, P., 1996. Deflate compressed data format specification version 1.3.Google Scholar
- Farbman, Z., Fattal, R., Lischinski, D., and Szeliski, R. 2008. Edge-preserving decompositions for multi-scale tone and detail manipulation. In ACM Transaction on Graphics (SIGGRAPH), ACM, New York, NY, USA, SIGGRAPH '08, 67:1--67:10. Google ScholarDigital Library
- Farbman, Z., Fattal, R., and Lischinski, D. 2011. Convolution pyramids. ACM Transactions on Graphics (Proc. of SIGGRAPH Asia) 30, 6. Google ScholarDigital Library
- Freeman, W. T., and Torralba, A. 2002. Shape recipes: Scene representations that refer to the image. In Vision Sciences Society Annual Meeting, MIT Press, 25--47.Google Scholar
- Hamilton, E. 1992. Jpeg file interchange format. C-Cube Microsystems.Google Scholar
- Heeger, D. J., and Bergen, J. R. 1995. Pyramid-based texture analysis/synthesis. In Proceedings of the 22Nd Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH '95, 229--238. Google ScholarDigital Library
- Huang, J., Qian, F., Gerber, A., Mao, Z. M., Sen, S., and Spatscheck, O., 2012. A close examination of performance and power characteristics of 4g lte networks.Google Scholar
- Huffman, D. 1952. A method for the construction of minimum-redundancy codes. Proceedings of the IRE 40, 9 (Sept), 1098--1101.Google ScholarCross Ref
- Jeong, W.-K., Johnson, M. K., Yu, I., Kautz, J., Pfister, H., and Paris, S. 2011. Display-aware image editing. In International Conference on Computational Photography.Google Scholar
- Kaufman, L., Lischinski, D., and Werman, M. 2012. Content-aware automatic photo enhancement. Computer Graphics Forum 31, 8, 2528--2540. Google ScholarDigital Library
- Kim, J.-H., Jang, W.-D., Sim, J.-Y., and Kim, C.-S. 2013. Optimized contrast enhancement for real-time image and video dehazing. J. Vis. Comun. Image Represent. 24, 3 (Apr.), 410--425. Google ScholarDigital Library
- Kumar, K., Liu, J., Lu, Y.-H., and Bhargava, B. 2013. A survey of computation offloading for mobile systems. Mob. Netw. Appl. 18, 1 (Feb.), 129--140. Google ScholarDigital Library
- Laffont, P.-Y., Ren, Z., Tao, X., Qian, C., and Hays, J. 2014. Transient attributes for high-level understanding and editing of outdoor scenes. ACM Transaction on Graphics (SIGGRAPH) 33, 4 (July), 149:1--149:11. Google ScholarDigital Library
- Lee, K., Chu, D., Cuervo, E., Kopf, J., Grizan, S., Wolman, A., and Flinn, J. 2014. Outatime: Using speculation to enable low-latency continuous interaction for cloud gaming. Tech. Rep. MSR-TR-2014-115.Google Scholar
- Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. ACM Transaction on Graphics (SIGGRAPH) 23, 3 (Aug.), 689--694. Google ScholarDigital Library
- Levin, A., Lischinski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (Feb), 228--242. Google ScholarDigital Library
- Levoy, M. 1995. Polygon-assisted jpeg and mpeg compression of synthetic images. In Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH '95, 21--28. Google ScholarDigital Library
- LiKamWa, R., Priyantha, B., Philipose, M., Zhong, L., and Bahl, P. 2013. Energy characterization and optimization of image sensing toward continuous mobile vision. In Proc. of International Conference on Mobile Systems, Applications, and Services, ACM, 69--82. Google ScholarDigital Library
- Mantiuk, R., and Seidel, H.-P. 2008. Modeling a generic tone-mapping operator. Computer Graphics Forum (Proc. of Eurographics) 27, 2.Google ScholarCross Ref
- Paris, S., Hasinoff, S. W., and Kautz, J. 2011. Local laplacian filters: Edge-aware image processing with a laplacian pyramid. In ACM Transaction on Graphics (SIGGRAPH), ACM, New York, NY, USA, SIGGRAPH '11, 68:1--68:12. Google ScholarDigital Library
- Rabbani, M., and Jones, P. W. 1991. Digital Image Compression Techniques, 1st ed. Society of Photo-Optical Instrumentation Engineers (SPIE), Bellingham, WA, USA. Google ScholarDigital Library
- Ragan-Kelley, J., Adams, A., Paris, S., Levoy, M., Amarasinghe, S., and Durand, F. 2012. Decoupling algorithms from schedules for easy optimization of image processing pipelines. ACM Transactions on Graphics 31, 4 (July), 32:1--32:12. Google ScholarDigital Library
- Ragan-Kelley, J., Barnes, C., Adams, A., Paris, S., Durand, F., and Amarasinghe, S. 2013. Halide: A language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. In Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation, ACM, New York, NY, USA, PLDI, 519--530. Google ScholarDigital Library
- Rhemann, C., Rother, C., Wang, J., Gelautz, M., Kohli, P., and Rott, P. 2009. A perceptually motivated online benchmark for image matting. In IEEE Conference on Computer Vision and Pattern Recognition, 1826--1833.Google Scholar
- Shih, Y., Paris, S., Durand, F., and Freeman, W. T. 2013. Data-driven hallucination of different times of day from a single outdoor photo. ACM Transaction on Graphics (SIGGRAPH) 32, 6 (Nov.), 200:1--200:11. Google ScholarDigital Library
- Shih, Y., Paris, S., Barnes, C., Freeman, W. T., and Durand, F. 2014. Style transfer for headshot portraits. ACM Transaction on Graphics (SIGGRAPH) 33, 4 (July), 148:1--148:14. Google ScholarDigital Library
- Skodras, A., Christopoulos, C., and Ebrahimi, T. 2001. The jpeg 2000 still image compression standard. IEEE Signal Processing Magazine 18, 5, 36--58.Google ScholarCross Ref
- Tibshirani, R. 1994. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B 58, 267--288.Google Scholar
- Torralba, A., and Freeman, W. 2003. Properties and applications of shape recipes. In IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, II-383-90 vol.2.Google Scholar
- Wallace, G. 1992. The jpeg still picture compression standard. IEEE Transactions on Consumer Electronics 38, 1 (Feb), xviii--xxxiv. Google ScholarDigital Library
- Welch, T. 1984. A technique for high-performance data compression. Computer 17, 6 (June), 8--19. Google ScholarDigital Library
- Witten, I. H., Neal, R. M., and Cleary, J. G. 1987. Arithmetic coding for data compression. Communications of the ACM 30, 6, 520--540. Google ScholarDigital Library
- Xu, L., Lu, C., Xu, Y., and Jia, J. 2011. Image smoothing via l0 gradient minimization. ACM Transaction on Graphics (SIGGRAPH) 30, 6 (Dec.), 174:1--174:12. Google ScholarDigital Library
- Ziv, J., and Lempel, A. 1977. A universal algorithm for sequential data compression. IEEE Transactions on Information Theory 23, 3 (May), 337--343. Google ScholarDigital Library
Index Terms
- Transform recipes for efficient cloud photo enhancement
Comments