Image-Based Relighting of Moving Objects with Specular Reflection

In the fields of Augmented Reality (AR) and Virtual Reality (VR), inserting an object into a scene (real or virtual) requires proper matching of their lighting conditions. If not, the resulting image may look unnatural. In particular, it is important to describe the position and shape of specular reflection accurately if the object has specular reflection. In this paper, we propose an approach to relighting a moving object based on the separation of specular and diffuse reflection. To relight an object, two or more images taken under the condition that the position of the object is fixed but the lighting condition is different, we call

synchronized images

, are required. However, it is impossible to obtain such images in case of a moving object. Therefore, we propose a method that computationally obtains the synchronized images using the consecutive fields of a controlled video sequence containing a moving object. For example, if the virtual (

n

+ 1) –

th

field is interpolated from

n

–

th

field and (

n

+ 2) –

th

field using the motion compensation technique, both the virtual (

n

+ 1) –

th

field and the real (

n

+ 1) –

th

field have the condition that the position of the object is fixed. If the virtual and real image have different lighting condition, the method applied to static object is applicable to moving object as it is. After the specular and diffuse reflection are separted, the relit image is synthesized using the linear interpolation and morphing technique. The experimental results of applying the proposed method to real and synthetic images are given. We verify the effectiveness of the proposed method by comparing the resulting image with a ground-truth image.