Top

Published in:

2018 | OriginalPaper | Chapter

3D Scene Flow from 4D Light Field Gradients

Authors : Sizhuo Ma, Brandon M. Smith, Mohit Gupta

Published in: Computer Vision – ECCV 2018

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

This paper presents novel techniques for recovering 3D dense scene flow, based on differential analysis of 4D light fields. The key enabling result is a per-ray linear equation, called the ray flow equation, that relates 3D scene flow to 4D light field gradients. The ray flow equation is invariant to 3D scene structure and applicable to a general class of scenes, but is underconstrained (3 unknowns per equation). Thus, additional constraints must be imposed to recover motion. We develop two families of scene flow algorithms by leveraging the structural similarity between ray flow and optical flow equations: local ‘Lucas-Kanade’ ray flow and global ‘Horn-Schunck’ ray flow, inspired by corresponding optical flow methods. We also develop a combined local-global method by utilizing the correspondence structure in the light fields. We demonstrate high precision 3D scene flow recovery for a wide range of scenarios, including rotation and non-rigid motion. We analyze the theoretical and practical performance limits of the proposed techniques via the light field structure tensor, a \(3 \times 3\) matrix that encodes the local structure of light fields. We envision that the proposed analysis and algorithms will lead to design of future light-field cameras that are optimized for motion sensing, in addition to depth sensing.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Stereo Relative Pose from Line and Point Feature Triplets

next chapter Direct Sparse Odometry with Rolling Shutter

Available only for authorised users

For a rotating object, in general, the motion of small scene patches can be modeled as translation, albeit with a change in the surface normal. For small rotations (small changes in surface normal), the brightness of a patch can be assumed to be approximately constant [31].

This is true under the assumption that the light sources are distant such that \(\mathbf {N}\cdot \mathbf {L}\), the dot-product of surface normal and lighting direction, does not change [31].

Structure tensors have been researched and defined differently in the light field community (e.g., [23]). Here it is defined by the gradients w.r.t. the 3D motion and is thus a \(3\times 3\) matrix.

Although the structure tensor theoretically has rank 2, the ratio \(\frac{\lambda _1}{\lambda _2}\) of the largest and second largest eigenvalues can be large. This is because the eigenvalue corresponding to Z motion depends on the range of (u, v) coordinates, which is limited by the size of the light field window. Therefore, a sufficiently large window size is required for motion recovery.

Adelson, E.H., Wang, J.Y.A.: Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 14(2), 99–106 (1992)CrossRef

Alexander, E., Guo, Q., Koppal, S., Gortler, S., Zickler, T.: Focal flow: measuring distance and velocity with defocus and differential motion. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 667–682. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_41CrossRef

Black, M.J., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996)CrossRef

Bok, Y., Jeon, H.G., Kweon, I.S.: Geometric calibration of micro-lens-based light field cameras using line features. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 39(2), 287–300 (2017)CrossRef

Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_3CrossRef

Bruhn, A., Weickert, J., Schnörr, C.: Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods. Int. J. Comput. Vis. (IJCV) 61(3), 211–231 (2005)CrossRef

Chandraker, M.: On shape and material recovery from motion. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 202–217. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10584-0_14CrossRef

Chandraker, M.: What camera motion reveals about shape with unknown BRDF. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2171–2178. IEEE, Washington (2014)

Chandraker, M.: The information available to a moving observer on shape with unknown, isotropic brdfs. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 38(7), 1283–1297 (2016)CrossRef

10.

Dansereau, D.G., Mahon, I., Pizarro, O., Williams, S.B.: Plenoptic flow: closed-form visual odometry for light field cameras. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4455–4462. IEEE, Washington (2011)

11.

Dansereau, D.G., Schuster, G., Ford, J., Wetzstein, G.: A wide-field-of-view monocentric light field camera. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Washington (2017)

12.

Gottfried, J.-M., Fehr, J., Garbe, C.S.: Computing range flow from multi-modal Kinect data. In: Bebis, G., et al. (eds.) ISVC 2011. LNCS, vol. 6938, pp. 758–767. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24028-7_70CrossRef

13.

Heber, S., Pock, T.: Scene flow estimation from light fields via the preconditioned primal-dual algorithm. In: Jiang, X., Hornegger, J., Koch, R. (eds.) GCPR 2014. LNCS, vol. 8753, pp. 3–14. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11752-2_1CrossRef

14.

Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1–3), 185–203 (1981)CrossRef

15.

Hung, C.H., Xu, L., Jia, J.: Consistent binocular depth and scene flow with chained temporal profiles. Int. J. Comput. Vis. (IJCV) 102(1–3), 271–292 (2013)CrossRef

16.

Jaimez, M., Souiai, M., Gonzalez-Jimenez, J., Cremers, D.: A primal-dual framework for real-time dense RGB-D scene flow. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 98–104. IEEE, Washington (2015)

17.

Johannsen, O., Sulc, A., Goldluecke, B.: On linear structure from motion for light field cameras. In: IEEE International Conference on Computer Vision (ICCV), pp. 720–728. IEEE, Washington (2015)

18.

Levoy, M., Hanrahan, P.: Light field rendering. In: SIGGRAPH Conference on Computer Graphics and Interactive Techniques, pp. 31–42. ACM, New York (1996)

19.

Li, Z., Xu, Z., Ramamoorthi, R., Chandraker, M.: Robust energy minimization for BRDF-invariant shape from light fields. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1. IEEE, Washington (2017)

20.

Lucas, B.D., Kanade, T., et al.: An iterative image registration technique with an application to stereo vision. In: International Joint Conference on Artificial Intelligence, pp. 674–679. Morgan Kaufmann, San Francisco (1981)

21.

Navarro, J., Garamendi, J.: Variational scene flow and occlusion detection from a light field sequence. In: International Conference on Systems. Signals and Image Processing (IWSSIP), pp. 1–4. IEEE, Washington (2016)

22.

Neumann, J., Fermuller, C., Aloimonos, Y.: Polydioptric camera design and 3D motion estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, p. II-294. IEEE, Washington (2003)

23.

Neumann, J., Fermüller, C., Aloimonos, Y.: A hierarchy of cameras for 3D photography. Comput. Vis. Image Underst. 96(3), 274–293 (2004)CrossRef

24.

Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P.: Light field photography with a hand-held plenoptic camera. Comput. Sci. Tech. Rep. CSTR 2(11), 1–11 (2005)

25.

Odobez, J.M., Bouthemy, P.: Robust multiresolution estimation of parametric motion models. J. Vis. Commun. Image Represent. 6(4), 348–365 (1995)CrossRef

26.

Shi, J., Tomasi, C.: Good features to track. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 593–600. IEEE, Washington (1994)

27.

Srinivasan, P.P., Tao, M.W., Ng, R., Ramamoorthi, R.: Oriented light-field windows for scene flow. In: IEEE International Conference on Computer Vision (ICCV), pp. 3496–3504. IEEE, Washington (2015)

28.

Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2432–2439. IEEE, Washington (2010)

29.

Sun, D., Sudderth, E.B., Pfister, H.: Layered RGBD scene flow estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 548–556. IEEE, Washington (2015)

30.

Tao, M.W., Hadap, S., Malik, J., Ramamoorthi, R.: Depth from combining defocus and correspondence using light-field cameras. In: IEEE International Conference on Computer Vision (ICCV), pp. 673–680. IEEE, Washington (2013)

31.

Vedula, S., Baker, S., Rander, P., Collins, R., Kanade, T.: Three-dimensional scene flow. In: IEEE International Conference on Computer Vision (ICCV), vol. 2, pp. 722–729. IEEE, Washington (1999)

32.

Wang, T.C., Chandraker, M., Efros, A.A., Ramamoorthi, R.: SVBRDF-invariant shape and reflectance estimation from light-field cameras. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5451–5459. IEEE, Washington (2016)

33.

Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 36(3), 606–619 (2014)CrossRef

34.

Wedel, A., Rabe, C., Vaudrey, T., Brox, T., Franke, U., Cremers, D.: Efficient dense scene flow from sparse or dense stereo data. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5302, pp. 739–751. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88682-2_56CrossRef

35.

Zhang, Y., Li, Z., Yang, W., Yu, P., Lin, H., Yu, J.: The light field 3D scanner. In: IEEE International Conference on Computational Photography (ICCP), pp. 1–9. IEEE, Washington (2017)

Title: 3D Scene Flow from 4D Light Field Gradients
Authors: Sizhuo Ma
Brandon M. Smith
Mohit Gupta
Publisher: Springer International Publishing
Book: Computer Vision – ECCV 2018
Print ISBN: 978-3-030-01236-6

Electronic ISBN: 978-3-030-01237-3

Copyright Year: 2018
DOI: https://doi.org/10.1007/978-3-030-01237-3_41

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner