RAW Image Reconstruction Using a Self-contained sRGB–JPEG Image with Small Memory Overhead

The vast majority of images used in computer vision and image processing applications are 8-bit standard RGB (sRGB) images, typically saved using the JPEG compression standard. Virtually all imaging application workflows support sRGB and JPEG images. There are many drawbacks, however, when working with sRGB images. For example, it is well known that sRGB images are processed by a number of nonlinear operations that make it difficult to relate the sRGB values back to scene radiance (Chakrabarti et al. 2014; Debevec and Malik 2008; Grossberg and Nayar 2003; Kim et al. 2012; Mann et al. 1995; Mitsunaga and Nayar 1999).

Most cameras now allow images to be saved in a RAW image format that is an uncompressed, minimally processed image format representing the response from the camera sensor. RAW has many advantages over sRGB, including linear response to scene radiance, wider color gamut, and higher dynamic range (generally 12–14 bits). Not surprisingly, RAW is desirable for many computer vision applications, such as photometric stereo, image restoration (e.g., deblurring), white balance, and more. RAW is also preferred by photographers as it allows flexibility for post-processing manipulation. One of the major drawbacks to using RAW images is that RAW files take up significantly more space than their sRGB counterpart. In addition, the vast majority of existing image-based applications are designed to work with 8-bit sRGB images. Images saved in RAW format must undergo some intermediate process to convert them into sRGB to be compatible with many existing applications.

Given the utility of RAW image data, there have been a number of approaches to map sRGB images back to their RAW values. Work by Yuan and Sun (2011) demonstrated an effective hybrid-image method that stored a lower-resolution version of the original RAW image (e.g., \(\frac{1}{2}\) or \(\frac{1}{4}\) resolution) and applied smart upsampling that leveraged the sRGB image. One impetus for Yuan and Sun (2011)’s work is that many cameras now support a small-RAW format that saves the RAW image in either half- or quarter-size resolutions. However, it is important to note that these smaller RAW images still require approximately 1.5–6 MB to store. Other closely related work (Xiong et al. 2012; Chakrabarti et al. 2014; Kim et al. 2012; Lin et al. 2011, 2012) used a calibration procedure to parameterize a model of the onboard camera processing pipeline in order to reverse sRGB values back to their RAW values. While this calibrated data is generally smaller than the 1.5–6 MB needed using the small-RAW strategy, these methods still have a drawback in that the additional data needs to be saved separately. In addition, the calibration procedure used by these methods needs to be performed for several different camera settings. Moreover, the model parameters are estimated in order to minimize reconstruction errors of a wide range of colors, because it is assumed that the original RAW is not given when converting back from sRGB. In our problem scenario, we have the original RAW image and corresponding sRGB image as input and can therefore estimate model parameters that better fit the given image pair.

The goal of this paper is to compute the necessary data to allow an sRGB–JPEG input to reconstruct its corresponding RAW image. In addition, we want to do this with a small memory overhead that can be embedded within the JPEG image such that it is 100% compatible with the existing JPEG standard. Specifically, we restrict our data to only 128 KB:64 KB for the camera-pipeline parameters and 64 KB for saturation correction. We show that with only 128 KB of data we can reconstruct the original RAW image with an overall error of less than 0.5%. Furthermore, this data can be embedded in two 64 KB JPEG comment fields allowed by the JPEG file format. Figure 1 shows an example of our proposed method’s ability.

We note that a shorter conference version of this paper appeared in Nguyen and Brown (2016). Our initial conference paper did not address the problem of saturation. This manuscript addresses this issue and provides additional analysis on the trade-off between using more data (i.e., a larger memory footprint) and the overall image reconstruction quality.

Work related to RAW image reconstruction can be categorized into two areas: (1) radiometric and camera color calibration and (2) methods for image-upsampling.

Radiometric and color calibration are methods that aim to compute the necessary mappings to invert the nonlinear transformations applied on board cameras in order to have pixel values that are linear with respect to scene radiance. Conventional radiometric calibration algorithms used multiple images taken with controlled exposures in order to compute the response function of a camera’s output intensity values with respect to the amount of light falling on the sensor. These methods targeted greyscale images (Lin and Zhang 2005) or computed an individual response function per color channel (Debevec and Malik 2008; Grossberg and Nayar 2003; Mann et al. 1995; Mitsunaga and Nayar 1999). The main difference among these methods is the model used to represent the response function—for example, exponentiation (Mann et al. 1995), polynomial (Mitsunaga and Nayar 1999), non-parametric (Debevec and Malik 2008), and PCA-based (Grossberg and Nayar 2003).

These early methods discarded RGB values that were too saturated, treating them as outliers to the radiometric model. Work in Chakrabarti et al. (2009) and Kim et al. (2012) found that these color outliers were due to limitations of existing radiometric models that treated each channel independently. To overcome this, Chakrabarti et al. (2009) proposed a method that used combinations of cross-channel linear transforms with per-channel multivariate polynomials to model the camera color-mapping process. Kim et al. (2012) proposed an in-camera imaging model that introduced an additional 3D gamut mapping step for handling the out-of-gamut colors. Later, Chakrabarti et al. (2014) extended this idea and suggested using uncertainty modeling for handling the quantization of the sRGB colors.

These recent radiometric calibration methods significantly improve the ability to reverse sRGB images back to their RAW values; however, they do have several limitations with respect to the problem addressed in this paper. The first limitation is the need to calibrate the color models for a given camera. As discussed by Kim et al. (2012), this involves computing multiple parameterized models for different camera settings. For example, each picture style (e.g., landscape, portrait, vivid, standard, neutral) requires its own set of parameters. As a result, a single camera would have numerous color mappings that would need to be saved. Such camera calibration is burdensome. Another drawback is the parameterized models need to be saved offline as an additional file and are still too large to be suitable to be embedded into a JPEG image. Finally, this type of color calibration is done in a generic manner, attempting to minimize reconstruction errors over a wide range of input images. This is because these methods assume that the original RAW image is unknown, and only an sRGB image is available as input. In our application, we have access to the original RAW and sRGB image pair and therefore can compute model parameters specific to the input pair.

Image upsampling is a method that attempts to increase the resolution, or quality, of an image. Representative work includes interpolation-based methods (Hou and Andrews 1978; Thévenaz et al. 2000), edge-based methods (Dai et al. 2007; Fattal 2007; Sun et al. 2008), and example-based methods (Freeman et al. 2000; Glasner et al. 2009). These methods leverage a dictionary of image patches learned from high-quality images that are used to guide the upsampling process. Work by Yuan and Sun (2011) presented work that is similar to the problem addressed in this paper. In particular, Yuan and Sun (2011) proposed a hybrid-image method that stored a low-resolution version of the RAW image together with the original high-resolution sRGB–JPEG image. The low-resolution RAW image was upsampled to have the same resolution as the sRGB image by using the sRGB image to guide the upsampling process. The RAW images used in this work were one half or one quarter the size of the original RAW image. While these small RAW images are smaller than the original RAW image, they still require 1.5–6 MB to store. Also, like the work of Kim et al. (2012) and Chakrabarti et al. (2014), this approach requires additional data to be stored separately from the JPEG image in order to perform upsampling.

The work in this paper is distinguished from prior work in two ways. First, the aim here is to embed the necessary information for RAW reconstruction inside the existing sRGB–JPEG image. This is more practical than requiring the user to maintain a companion file containing the necessary data for reconstruction (e.g., small RAW file or camera calibration data). Second, unlike the radiometric calibration approaches, our approach does not require a calibration procedure involving multiple images. Instead, we need only to estimate a mapping between the given sRGB and RAW image pair provided by the camera. To this end, our goal is to efficiently estimate this necessary data such that it can fit inside text-based comments that can be embedded in the JPEG file.

The work by Kim et al. (2012) and Chakrabarti et al. (2014) has shown that the nonlinear processes that convert RAW to sRGB can be described by a number of parameters that model various steps in the onboard camera color-processing pipeline. In particular, there is a white-balance matrix \(T_w\) and a color-correction matrix \(T_s\) that are first applied. The color-correction matrix is used to convert the camera-specific RAW color space into a device-independent color space. This is followed by a tone map operator f applied to all color channels and a gamut-mapping g that maps a 3D color value to a new 3D color value. The work in Kim et al. (2012) and Chakrabarti et al. (2014) parameterized these models by using many sRGB–JPEG pairs of color charts under different illuminations captured by the camera with specific settings. This type of parameterization is necessary when the input is an arbitrary sRGB image. We follow a similar procedure, but our problem is simplified as we need only to compute these parameters for a single pair of RAW image E and sRGB–JPEG image I. In addition, our approach needs to keep the results to within a small memory overhead and embed this as a text comment in the JPEG image.

Figure 2 illustrates the overall procedure of our method. Our input is the RAW image captured from the camera and the corresponding sRGB–JPEG. We assume the RAW image has been demosaicked. We use the DCRAW utility (Coffin 1997) to perform this task; however, other methods could also be used—for example, Gharbi et al. (2016). The data storage budget for the camera model and saturation table is pre-allocated as shown in Table 1. The total budget for both the camera model parameters and saturation table is less than 128 KB because this data will later be converted to a text format that avoids a 0x00 bit sequence (described in Sect. 3.4). The following sections describe how this data is computed and embedded in the sRGB–JPEG file.

In this section, we compared our RAW reconstruction strategy with alternative techniques, including our method using k-means clustering, our method using the proposed octree method, our method with and without saturation correction, and the upsampling method by Yuan and Sun (2011). Yuan and Sun (2011) do not provide code and we have re-implemented their approach. We note that we have modified their approach to use the recent state-of-the-art method proposed by Ferstl et al. (2013) to perform the image upsampling. Images used in this paper were taken from various online data sets that contain RAW-sRGB pairs, including images from Kim et al. (2012), Cheng et al. (2014), Nguyen et al. (2014), and Dang-Nguyen et al. (2015).

We start with examples that do not require saturation correction. Figure 8 shows the results for images from various cameras and scenes. A jet map is used to show the error between the original and reconstructed RAW image. The root mean square error (RMSE) is also shown to quantitatively evaluate the results. For Yuan and Sun’s method (2011), RAW images at resolutions of half of the original size are used for upsampling. For a fairer comparison, we also used a small RAW image of resolution \(100 \times 90\), which can be placed inside 64 KB of data. Table 3 shows the results in terms of RMSE. For each camera, 30 images were examined and the average RMSE was reported. The proposed octree partitioning approach provides RAW reconstructions with the lowest RMSE. Note that the errors shown also include quantization errors in the 8-bit sRGB image, which is around 0.001, or 0.1% in average.

We also test our approach for images where saturation correction is needed. We compared our RAW reconstruction with or without saturation correction, and the upsampling method by Yuan and Sun (2011). Figure 9 shows the results. As can be seen, with an additional 64 KB of RAW values our method can significantly reduce the error in the saturated regions.

We note that our proposed method does not attempt to remove any compression artifacts that arise due to the lossy nature of JPEG. We assume that the input image is saved as a high-quality JPEG; however, for the sake of completeness, we examine the effect of different JPEG qualities on the reconstructed RAW results. Most digital cameras support three image quality types, fine, normal, and basic, which correspond to the compression ratios (1 / 4, 1 / 8, and 1 / 16). Figure 10 shows two examples: a correctly exposed image captured by a Nikon D5200 and an overexposed image captured by a Sony \(\alpha \)57. It is clear that the JPEG quality does affect the reconstructed RAW-RGB images; however, the increased error is still within acceptable levels.

We conclude our results section by demonstrating the usefulness of the reconstructed RAW image data with three applications: white balance, image deblurring, and photometric stereo. It is well known that having access to the original RAW image is advantageous for these applications.

We have described a method to encode the necessary data with the sRGB image for reconstructing a high-quality RAW image. Our approach produces a fully self-contained, 100% compliant JPEG file that can be used as-is, not affecting any existing image workflows. This method can reconstruct the original RAW to within 0.5% error with only 64 KB of overhead for camera-pipeline parameters and an additional 64 KB for saturation correction. Although we focused only on the JPEG compression format as it is the most popular format for storing sRGB images, we believe that the proposed encoding technique can be easily extended to other formats that can support the inclusion of metadata (e.g., PNG).

One drawback of our method is that we have optimized our framework to minimize error for backward mapping from sRGB to RAW; however, for many photography tasks (e.g., our white-balance example), the forward mapping from RAW back to sRGB is needed. A future topic would be to modify our method to consider the bidirectional reconstruction error. Also, while our method provides better results in the saturated region than prior work, there are still notable errors in these regions, especially about boundaries. Improving the results in these areas is a topic suitable for future work.