Abstract
Compared to DSLR cameras, smartphone cameras have smaller sensors, which limits their spatial resolution; smaller apertures, which limits their light gathering ability; and smaller pixels, which reduces their signal-to-noise ratio. The use of color filter arrays (CFAs) requires demosaicing, which further degrades resolution. In this paper, we supplant the use of traditional demosaicing in single-frame and burst photography pipelines with a multiframe super-resolution algorithm that creates a complete RGB image directly from a burst of CFA raw images. We harness natural hand tremor, typical in handheld photography, to acquire a burst of raw frames with small offsets. These frames are then aligned and merged to form a single image with red, green, and blue values at every pixel site. This approach, which includes no explicit demosaicing step, serves to both increase image resolution and boost signal to noise ratio. Our algorithm is robust to challenging scene conditions: local motion, occlusion, or scene changes. It runs at 100 milliseconds per 12-megapixel RAW input burst frame on mass-produced mobile phones. Specifically, the algorithm is the basis of the Super-Res Zoom feature, as well as the default merge method in Night Sight mode (whether zooming or not) on Google's flagship phone.
Supplemental Material
Available for Download
Supplemental material
- Adobe. 2012. Digital Negative (DNG) Specification. https://www.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf.Google Scholar
- Simon Baker and Takeo Kanade. 2002. Limits on super-resolution and how to break them. IEEE Trans. PAMI 24, 9 (2002), 1167--1183. Google ScholarDigital Library
- Bryce E Bayer. 1976. Color imaging array. US Patent 3,971,065.Google Scholar
- Stefanos P Belekos, Nikolaos P Galatsanos, and Aggelos K Katsaggelos. 2010. Maximum a posteriori video super-resolution using a new multichannel image prior. IEEE Trans. Image Processing 19, 6 (2010), 1451--1464. Google ScholarDigital Library
- Moshe Ben-Ezra, Assaf Zomet, and Shree K Nayar. 2005. Video super-resolution using controlled subpixel detector shifts. IEEE Trans. PAMI 27, 6 (2005), 977--987. Google ScholarDigital Library
- Josef Bigün, Goesta H. Granlund, and Johan Wiklund. 1991. Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Trans. PAMI 8 (1991), 775--790. Google ScholarDigital Library
- James F Blinn. 1982. A generalization of algebraic surface drawing. ACM TOG 1, 3 (1982), 235--256. Google ScholarDigital Library
- Edward Chang, Shiufun Cheung, and Davis Y Pan. 1999. Color filter array recovery using a threshold-based variable number of gradients. In Sensors, Cameras, and Applications for Digital Photography, Vol. 3650. 36--44.Google ScholarCross Ref
- CIPA. 2018. CIPA Report. www.cipa.jp/stats/report_e.html. {Online; accessed 29-Nov-2018}.Google Scholar
- Sabrina Dammertz and Alexander Keller. 2008. Image synthesis by rank-1 lattices. In Monte Carlo and Quasi-Monte Carlo Methods 2006. 217--236.Google Scholar
- Michael Drobot. 2014. Hybrid reconstruction anti-aliasing. In ACM SIGGRAPH Courses.Google Scholar
- Joan Duran and Antoni Buades. 2014. Self-similarity and spectral correlation adaptive algorithm for color demosaicking. IEEE Trans. Image Processing 23, 9 (2014), 4031--4040.Google ScholarCross Ref
- Michael Elad and Arie Feuer. 1997. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Processing 6, 12 (1997), 1646--1658. Google ScholarDigital Library
- Sina Farsiu, Michael Elad, and Peyman Milanfar. 2006. Multiframe demosaicing and super-resolution of color images. IEEE Trans. Image Processing 15, 1 (2006), 141--159. Google ScholarDigital Library
- David J Fleet and Allan D Jepson. 1990. Computation of component image velocity from local phase information. IJCV 5, 1 (1990), 77--104. Google ScholarDigital Library
- Flickr. 2017. Top Devices of 2017 on Flickr. https://blog.flickr.net/en/2017/12/07/top-devices-of-2017/. {Online; accessed 11-Jan-2019}.Google Scholar
- Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian. 2008. Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data. IEEE Trans. Image Processing 17, 10 (2008), 1737--1754. Google ScholarDigital Library
- Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand. 2016. Deep joint demosaicking and denoising. ACM TOG 35, 6 (2016), 191. Google ScholarDigital Library
- Clément Godard, Kevin Matzen, and Matt Uyttendaele. 2018. Deep burst denoising. In Proc. ECCV, Vol. 11219. 560--577.Google Scholar
- Tomomasa Gotoh and Masatoshi Okutomi. 2004. Direct super-resolution and registration using raw CFA images. In Proc. CVPR, Vol. 2. Google ScholarDigital Library
- Chris Harris and Mike Stephens. 1988. A combined corner and edge detector.. In Alvey Vision Conference, Vol. 15. 10--5244.Google ScholarCross Ref
- Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM TOG 35, 6 (2016), 192. Google ScholarDigital Library
- Sung Hee Park and Marc Levoy. 2014. Gyro-based multi-image deconvolution for removing handshake blur. In Proc. ICCV. 3366--3373.Google ScholarDigital Library
- Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pajkk, Dikpal Reddy, Orazio Gallo, Jing Liu, Wolfgang Heidrich, et al. 2014. FlexISP: A flexible camera image processing framework. ACM TOG 33, 6 (2014), 231. Google ScholarDigital Library
- Carl W Helstrom. 1969. Detection and resolution of incoherent objects by a background-limited optical system. JOSA 59, 2 (1969), 164--175.Google ScholarCross Ref
- Robert Herzog, Elmar Eisemann, Karol Myszkowski, and H-P Seidel. 2010. Spatio-temporal upsampling on the GPU. In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games. 91--98. Google ScholarDigital Library
- Keigo Hirakawa and Thomas W Parks. 2005. Adaptive homogeneity-directed demosaicing algorithm. IEEE Trans. Image Processing 14, 3 (2005), 360--369. Google ScholarDigital Library
- Keigo Hirakawa and Thomas W Parks. 2006. Joint demosaicing and denoising. IEEE Trans. Image Processing 15, 8 (2006), 2146--2157. Google ScholarDigital Library
- Terence D Hunt. 2004. Image Super-Resolution Using Adaptive 2-D Gaussian Basis Function Interpolation. Technical Report. Air Force Inst of Tech Wright-Patterson AFB OH School of Engineering.Google Scholar
- Michal Irani and Shmuel Peleg. 1991. Improving resolution by image registration. CVGIP: Graphical models and image processing 53, 3 (1991), 231--239. Google ScholarDigital Library
- Jorge Jimenez, Diego Gutierrez, Jason Yang, Alexander Reshetov, Pete Demoreuille, Tobias Berghoff, Cedric Perthuis, Henry Yu, Morgan McGuire, Timothy Lottes, Hugh Malan, Emil Persson, Dmitry Andreev, and Tiago Sousa. 2011. Filtering Approaches for Real-Time Anti-Aliasing. In ACM SIGGRAPH Courses. Google ScholarDigital Library
- Takeo Kanade and Masatoshi Okutomi. 1991. A stereo matching algorithm with an adaptive window: Theory and experiment. In Proc. IEEE ICRA. IEEE, 1088--1095.Google ScholarCross Ref
- Brian Karis. 2014. High-Quality Temporal Supersampling. In ACM SIGGRAPH Courses.Google Scholar
- Darwin T Kuan, Alexander A Sawchuk, Timothy C Strand, and Pierre Chavel. 1985. Adaptive noise smoothing filter for images with signal-dependent noise. IEEE Trans. PAMI 2 (1985), 165--177. Google ScholarDigital Library
- Yeon Ju Lee and Jungho Yoon. 2010. Nonlinear image upsampling method based on radial basis function interpolation. IEEE Trans. Image Processing 19, 10 (2010), 2682--2692. Google ScholarDigital Library
- Brian Leung, Gwanggil Jeon, and Eric Dubois. 2011. Least-squares luma-chroma demultiplexing algorithm for Bayer demosaicking. IEEE Trans. Image Processing 20, 7 (2011), 1885--1894. Google ScholarDigital Library
- Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, and Jonathan Ragan-Kelley. 2018. Differentiable programming for image processing and deep learning in Halide. ACM Trans. Graph. (Proc. SIGGRAPH) 37, 4 (2018), 139:1--139:13. Google ScholarDigital Library
- Xin Li, Bahadir Gunturk, and Lei Zhang. 2008. Image demosaicing: A systematic survey. In Visual Communications and Image Processing 2008, Vol. 6822. 68221J.Google Scholar
- Ce Liu and Deqing Sun. 2011. A Bayesian approach to adaptive video super resolution. In Proc. CVPR. IEEE, 209--216. Google ScholarDigital Library
- Xiao-Ming Lu, Hari Krovi, Ranjith Nair, Saikat Guha, and Jeffrey H Shapiro. 2018. Quantum-optimal detection of one-versus-two incoherent optical sources with arbitrary separation. arXiv preprint arXiv:1802.02300 (2018).Google Scholar
- Bruce D Lucas and Takeo Kanade. 1981. An iterative image registration technique with an application to stereo vision. (1981).Google Scholar
- Rastislav Lukac and Konstantinos N Plataniotis. 2004. Normalized color-ratio modeling for CFA interpolation. IEEE Trans. Consumer Electronics 50, 2 (2004), 737--745. Google ScholarDigital Library
- Hugh Malan. 2012. Real-Time Global Illumination and Reflections in Dust 514. In ACM SIGGRAPH Courses.Google Scholar
- John Marshall and E Geoffrey Walsh. 1956. Physiological tremor. Journal of neurology, neurosurgery, and psychiatry 19, 4 (1956), 260.Google ScholarCross Ref
- Ben Mildenhall, Jonathan T Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll. 2018. Burst denoising with kernel prediction networks. In Proc. CVPR. 2502--2510.Google ScholarCross Ref
- Yusuke Monno, Daisuke Kiku, Masayuki Tanaka, and Masatoshi Okutomi. 2015. Adaptive residual interpolation for color image demosaicking. In Proc. IEEE ICIP. 3861--3865.Google ScholarDigital Library
- Matthias Müller, Barbara Solenthaler, Richard Keiser, and Markus Gross. 2005. Particle-based fluid-fluid interaction. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation. 237--244. Google ScholarDigital Library
- NIH. 2018. Tremor Fact. www.ninds.nih.gov/Disorders/Patient-Caregiver-Education/Fact-Sheets/Tremor-Fact-Sheet. {Online; accessed 29-Nov-2018}.Google Scholar
- Harry Nyquist. 1928. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers 47, 2 (1928), 617--644.Google ScholarCross Ref
- Athanasios Papoulis. 1977. Generalized sampling expansion. IEEE Trans. Circuits and Systems 24, 11 (1977), 652--654.Google ScholarCross Ref
- Cameron N Riviere, R Scott Rader, and Nitish V Thakor. 1998. Adaptive cancelling of physiological tremor for improved precision in microsurgery. IEEE Trans. Biomedical Engineering 45, 7 (1998), 839--846.Google ScholarCross Ref
- Dirk Robinson and Peyman Milanfar. 2004. Fundamental performance limits in image registration. IEEE Trans. Image Processing 13, 9 (2004), 1185--1199. Google ScholarDigital Library
- Dirk Robinson and Peyman Milanfar. 2006. Statistical performance analysis of super-resolution. IEEE Trans. Image Processing 15, 6 (2006), 1413--1428. Google ScholarDigital Library
- Yaniv Romano, John Isidoro, and Peyman Milanfar. 2017. RAISR: Rapid and accurate image super resolution. IEEE Trans. Computational Imaging 3, 1 (2017), 110--125.Google ScholarCross Ref
- Mehdi S. M. Sajjadi, Raviteja Vemulapalli, and Matthew Brown. 2018. Frame-recurrent video super-resolution. In Proc. CVPR. 6626--6634.Google ScholarCross Ref
- Marco Salvi. 2016. An excursion in temporal super sampling. GDC2016 From the Lab Bench: Real-Time Rendering Advances from NVIDIA Research (2016).Google Scholar
- E. A. Schäfer. 1886. On the rhythm of muscular response to volitional impulses in man. The Journal of Physiology 7, 2 (1886), 111--117.Google ScholarCross Ref
- Morteza Shahram and Peyman Milanfar. 2006. Statistical and information-theoretic analysis of resolution in imaging. IEEE Trans. Information Theory 52, 8 (2006), 3411--3437. Google ScholarDigital Library
- Masao Shimiziu and Masatoshi Okutomi. 2005. Sub-pixel estimation error cancellation on area-based matching. IJCV 63 (2005), 207--224. Issue 3. Google ScholarDigital Library
- Tiago Sousa. 2013. Graphics Gems CryENGINE3. In ACM SIGGRAPH Courses.Google Scholar
- Molly M Sturman, David E Vaillancourt, and Daniel M Corcos. 2005. Effects of aging on the regularity of physiological tremor. Journal of Neurophysiology 93, 6 (2005), 3064--3074.Google ScholarCross Ref
- H Takeda, S Farsiu, and P Milanfar. 2006. Robust kernel regression for restoration and reconstruction of images from sparse noisy data. Proc. IEEE ICIP (2006), 1257--1260.Google ScholarCross Ref
- H Takeda, S Farsiu, and P Milanfar. 2007. Kernel regression for image processing and reconstruction. IEEE Trans. Image Processing 16, 2 (2007), 349. Google ScholarDigital Library
- Hanlin Tan, Xiangrong Zeng, Shiming Lai, Yu Liu, and Maojun Zhang. 2017. Joint demosaicing and denoising of noisy bayer images with ADMM. In Proc. IEEE ICIP. 2951--2955.Google ScholarCross Ref
- R.Y. Tsai and T.S. Huang. 1984. Multiframe image restoration and registration. Advance Computer Visual and Image Processing 1 (1984), 317--339.Google Scholar
- Patrick Vandewalle, Karim Krichane, David Alleysson, and Sabine Süsstrunk. 2007. Joint demosaicing and super-resolution imaging from a set of unregistered aliased images. In Digital Photography III, Vol. 6502. International Society for Optics and Photonics, 65020A.Google ScholarCross Ref
- Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T Barron, Yael Pritch, and Marc Levoy. 2018. Synthetic depth-of-field with a single-camera mobile phone. ACM TOG 37, 4 (2018), 64. Google ScholarDigital Library
- Hermann Weyl. 1910. Über die Gibbs'sche Erscheinung und verwandte Konvergenzphänomene. Rendiconti del Circolo Matematico di Palermo (1884--1940) 30, 1 (01 Dec 1910), 377--407.Google Scholar
- Bartlomiej Wronski and Peyman Milanfar. 2018. See Better and Further with Super Res Zoom on the Pixel 3. https://ai.googleblog.com/2018/10/see-better-and-further-with-super-res.html.Google Scholar
- Xiaolin Wu and Lei Zhang. 2006. Temporal color video demosaicking via motion estimation and data fusion. IEEE Trans. Circuits and Systems for Video Technology 16 (2006). Issue 2. Google ScholarDigital Library
- J Yen. 1956. On nonuniform sampling of bandwidth-limited signals. IRE Trans. Circuit Theory 3, 4 (1956), 251--257.Google ScholarCross Ref
- Jihun Yu and Greg Turk. 2013. Reconstructing surfaces of particle-based fluids using anisotropic kernels. ACM TOG 32, 1 (2013), 5. Google ScholarDigital Library
- Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. 2011. Color demosaicking bylocal directional interpolation and nonlocal adaptive thresholding. Journal of Electronic imaging 20, 2 (2011), 023016.Google ScholarCross Ref
Index Terms
- Handheld multi-frame super-resolution
Recommendations
High dynamic range and super-resolution from raw image bursts
Photographs captured by smartphones and mid-range cameras have limited spatial resolution and dynamic range, with noisy response in underexposed regions and color artefacts in saturated areas. This paper introduces the first approach (to the best of our ...
Solving a two-colour problem by applying probabilistic approach to a full-colour multi-frame image super-resolution
Reconstruction based algorithms play an important role in the multi-frame super-resolution problem. A group of images of the same scene are fused together to produce an image with higher spatial resolution, or with more visible details in the high ...
PIRM2018 Challenge on Spectral Image Super-Resolution: Methods and Results
Computer Vision – ECCV 2018 WorkshopsAbstractIn this paper, we describe the Perceptual Image Restoration and Manipulation (PIRM) workshop challenge on spectral image super-resolution, motivate its structure and conclude on results obtained by the participants. The challenge is one of the ...
Comments