Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
International Journal of Computer Vision
Published in: International Journal of Computer Vision 1/2021

17-08-2020

Solving Rolling Shutter 3D Vision Problems using Analogies with Non-rigidity

Authors: Yizhen Lao, Omar Ait-Aider, Adrien Bartoli

Published in: International Journal of Computer Vision | Issue 1/2021

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

We propose an original approach to absolute pose and structure-from-motion (SfM) which handles rolling shutter (RS) effects. Unlike most existing methods which either augment global shutter projection with velocity parameters or impose continuous time and motion through pose interpolation, we use local differential constraints. These are established by drawing analogies with non-rigid 3D vision techniques, namely shape-from-template and non-rigid SfM (NRSfM). The proposed idea is to interpret the images of a rigid surface acquired by a moving RS camera as those of a virtually deformed surface taken by a GS camera. These virtually deformed surfaces are first recovered by relaxing the RS constraint using SfT or NRSfM. Then we upgrade the virtually deformed surface to the actual rigid structure and compute the camera pose and ego-motion by reintroducing the RS constraint. This uses a new 3D-3D registration procedure that minimizes a cost function based on the Euclidean 3D point distance. This is more stable and physically meaningful than the reprojection error or the algebraic distance used in previous work. Experimental results obtained with synthetic and real data show that the proposed methods outperform existing ones in terms of accuracy and stability, even in the known critical configurations.
previous article A Camera Model for Line-Scan Cameras with Telecentric Lenses
next article Temporally Coherent General Dynamic Scene Reconstruction

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now
Footnotes
4
estimateWorldCameraPose function in MATLAB.
 
Literature
Agudo, A., & Moreno-Noguer, F. (2015). Simultaneous pose and non-rigid shape with particle dynamics. In CVPR.
Agudo, A., Moreno-Noguer, F., Calvo, B., & Montiel, J. M. M. (2016). Sequential non-rigid structure from motion using physical priors. PAMI, 38(5), 979–994.CrossRef
Ait-Aider, O., Andreff, N., Lavest, J. M., & Martinet, P. (2006). Simultaneous object pose and velocity computation using a single view from a rolling shutter camera. In ECCV.
Ait-Aider, O., Bartoli, A., & Andreff, N. (2007). Kinematics from lines in a single rolling shutter image. In CVPR.
Ait-Aider, O., & Berry, F. (2009). Structure and kinematics triangulation with a rolling shutter stereo rig. In ICCV.
Akhter, I., Sheikh, Y., Khan, S., & Kanade, T. (2009). Nonrigid structure from motion in trajectory space. In NIPS.
Albl, C., Kukelova, Z., Larsson, V., & Pajdla, T. (2019). Rolling shutter camera absolute pose. PAMI, 42(6), 1439–1452.CrossRef
Albl, C., Kukelova, Z., & Pajdla, T. (2015). R6p-rolling shutter absolute camera pose. In CVPR.
Albl, C., Kukelova, Z., & Pajdla, T. (2016a). Rolling shutter absolute pose problem with known vertical direction. In CVPR.
Albl, C., Sugimoto, A., & Pajdla, T. (2016b). Degeneracies in rolling shutter SfM. In ECCV.
Bartoli, A., Gérard, Y., Chadebecq, F., Collins, T., & Pizarro, D. (2015). Shape-from-template. PAMI, 37(10), 2099–2118.CrossRef
Chhatkuli, A., Pizarro, D., Bartoli, A., & Collins, T. (2017). A stable analytical framework for isometric shape-from-template by surface integration. PAMI, 39(5), 833–850.CrossRef
Collins, T., & Bartoli, A. (2015). [POSTER] Realtime shape-from-template: System and applications. In ISMAR.
Dai, Y., Li, H., & Kneip, L. (2016). Rolling shutter camera relative pose: Generalized epipolar geometry. In CVPR.
Dierckx, P. (1993). Curve and surface fitting with splines. New York: Oxford University Press.MATH
Dryden, I. L., & Mardia, K. V. (2016). Statistical shape analysis: with applications in R (Vol. 995). John Wiley & Sons.
Duchamp, G., Ait-Aider, O., Royer, E., & Lavest, J. M. (2015). A rolling shutter compliant method for localisation and reconstruction. In VISAPP.
El Gamal, A., & Eltoukhy, H. (2005). CMOS image sensors. IEEE Circuits and Devices Magazine, 21(3), 6–20.CrossRef
Famouri, M., Bartoli, A., & Azimifar, Z. (2018). Fast shape-from-template using local features. Machine Vision and Applications, 29(1), 73–93.CrossRef
Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381–395.MathSciNetCrossRef
Forssén, P. E., & Ringaby, E. (2010). Rectifying rolling shutter video from hand-held devices. In CVPR.
Gallardo, M. (2018). Contributions to monocular deformable 3D reconstruction: Curvilinear objects and multiple visual cues. Ph.D. thesis, University Clermont Auvergne
Gao, X. S., Hou, X. R., Tang, J., & Cheng, H. F. (2003). Complete solution classification for the perspective-three-point problem. PAMI, 25(8), 930–943.CrossRef
Gotardo, P. F., & Martinez, A. M. (2011). Kernel non-rigid structure from motion. In ICCV.
Haouchine, N., Dequidt, J., Berger, M. O., & Cotin, S. (2014). Single view augmentation of 3D elastic objects. In ISMAR.
Haralick, R. M., Lee, D., Ottenburg, K., & Nolle, M. (1991). Analysis and solutions of the three point perspective pose estimation problem. In CVPR.
Hartley, R., & Zisserman, A. (2003). Multiple view geometry in computer vision. New York: Cambridge University Press.MATH
Hedborg, J., Forssen, P. E., Felsberg, M., & Ringaby, E. (2012). Rolling shutter bundle adjustment. In CVPR.
Hedborg, J., Ringaby, E., Forssén, P. E., & Felsberg, M. (2011). Structure and motion estimation from rolling shutter video. In ICCV workshops.
Horn, B. K., Hilden, H. M., & Negahdaripour, S. (1988). Closed-form solution of absolute orientation using orthonormal matrices. JOSA A, 5(7), 1127–1135.MathSciNetCrossRef
Hu, Y., Zhang, D., Ye, J., Li, X., & He, X. (2013). Fast and accurate matrix completion via truncated nuclear norm regularization. PAMI, 35(9), 2117–2130.CrossRef
Im, S., Ha, H., Choe, G., Jeon, H. G., Joo, K., & Kweon, I. S. (2018). Accurate 3D reconstruction from small motion clip for rolling shutter cameras. PAMI, 41(4), 775–787.CrossRef
Ito, E., & Okatani, T. (2017). Self-calibration-based approach to critical motion sequences of rolling-shutter structure from motion. In CVPR.
Kim, J. H., Cadena, C., & Reid, I. (2016). Direct semi-dense slam for rolling shutter cameras. In ICRA.
Kumar, S., Ghorakavi, R. S., Dai, Y., & Li, H. (2019). Dense depth estimation of a complex dynamic scene without explicit 3D motion estimation. ArXiv preprint arXiv:​1902.​03791.
Lao, Y., & Ait-Aider, O. (2018). A robust method for strong rolling shutter effects correction using lines with automatic feature selection. In CVPR.
Lao, Y., Ait-Aider, O., & Bartoli, A. (2018). Rolling shutter pose and ego-motion estimation using shape-from-template. In ECCV.
Lee, J. (1997). Riemannian manifolds: An introduction to curvature. New York: Springer.CrossRef
Leng, D., & Sun, W. (2009). Finding all the solutions of PnP problem. In International workshop on imaging systems and techniques.
Lovegrove, S., Patron-Perez, A., & Sibley, G. (2013). Spline fusion: A continuous-time representation for visual-inertial fusion with application to rolling shutter cameras. In BMVC.
Magerand, L., & Bartoli, A. (2010). A generic rolling shutter camera model and its application to dynamic pose estimation. In International symposium on 3D data processing, visualization and transmission.
Magerand, L., Bartoli, A., Ait-Aider, O., & Pizarro, D. (2012). Global optimization of object pose and motion from a single rolling shutter image with automatic 2D–3D matching. In ECCV.
Magnenat, S., Ngo, D. T., Zünd, F., Ryffel, M., Noris, G., Roethlin, G., et al. (2015). Live texturing of augmented reality characters from colored drawings. TVCG, 21(11), 1201–1210.
Malti, A., Bartoli, A., & Hartley, R. (2015). A linear least-squares solution to elastic shape-from-template. In CVPR.
Malti, A., & Herzet, C. (2017). Elastic shape-from-template with spatially sparse deforming forces. In CVPR.
Oth, L., Furgale, P., Kneip, L., & Siegwart, R. (2013). Rolling shutter camera calibration. In CVPR.
Ovrén, H., & Forssén, P. (2018). Spline error weighting for robust visual-inertial fusion. In CVPR.
Ovrén, H., & Forssén, P. (2019). Trajectory representation and landmark projection for continuous-time structure from motion. IJRR, 38(6), 686–701.
Ovrén, H., Forssén, P., & Törnqvist, D. (2013). Why would I want a gyroscope on my RGB-D sensor? In IEEE workshop on robot vision.
Parashar, S., Pizarro, D., & Bartoli, A. (2018). Isometric non-rigid shape-from-motion with riemannian geometry solved in linear time. PAMI, 40(10), 2442–2454.CrossRef
Patron-Perez, A., Lovegrove, S., & Sibley, G. (2015). A spline-based trajectory representation for sensor fusion and rolling shutter cameras. IJCV, 113(3), 208–219.CrossRef
Pizarro, D., & Bartoli, A. (2012). Feature-based deformable surface detection with self-occlusion reasoning. IJCV, 97(1), 54–70.CrossRef
Purkait, P., Zach, C., & Leonardis, A. (2017). Rolling shutter correction in Manhattan world. In ICCV (pp. 882–890).
Quan, L., & Lan, Z. (1999). Linear \(n\)-point camera pose determination. PAMI, 21(8), 774–780.CrossRef
Rengarajan, V., Balaji, Y., & Rajagopalan, A. (2017). Unrolling the shutter: CNN to correct motion distortions. In CVPR.
Rengarajan, V., Rajagopalan, A. N., & Aravind, R. (2016). From bows to arrows: Rolling shutter rectification of urban scenes. In CVPR.
Rueckert, D., Sonoda, L. I., Hayes, C., Hill, D. L. G., Leach, M. O., & Hawkes, D. J. (1999). Nonrigid registration using free-form deformations: Application to breast MR images. TMI, 18(8), 712–721.
Russell, C., Yu, R., & Agapito, L. (2014). Video pop-up: Monocular 3D reconstruction of dynamic scenes. In ECCV.
Salzmann, M., & Fua, P. (2011). Linear local models for monocular reconstruction of deformable surfaces. PAMI, 33(5), 931–944.CrossRef
Saurer, O., Koser, K., Bouguet, J. Y., & Pollefeys, M. (2013). Rolling shutter stereo. In ICCV.
Saurer, O., Pollefeys, M., & Hee Lee, G. (2016). Sparse to dense 3D reconstruction from rolling shutter images. In CVPR.
Saurer, O., Pollefeys, M., & Lee, G. H. (2015). A minimal solution to the rolling shutter pose estimation problem. In IROS.
Taylor, J., Jepson, A. D., & Kutulakos, K. N. (2010). Non-rigid structure from locally-rigid motion. In CVPR.
Varol, A., Salzmann, M., Tola, E., & Fua, P. (2009). Template-free monocular reconstruction of deformable surfaces. In ICCV.
Wu, C. (2013). Towards linear-time incremental structure from motion. In International Conference on 3D Vision-3DV 2013 (pp.127–134). IEEE.
Zhuang, B., Cheong, L. F., & Lee, G. H. (2017). Rolling-shutter-aware differential SfM and image rectification. In ICCV.
Zhuang, B., Tran, Q., Ji, P., Cheong, L., & Chandraker, M. (2019). Learning structure-and-motion-aware rolling shutter correction. In CVPR.
Metadata
Title
Solving Rolling Shutter 3D Vision Problems using Analogies with Non-rigidity
Authors
Yizhen Lao
Omar Ait-Aider
Adrien Bartoli
Publication date
17-08-2020
Publisher
Springer US
Published in
International Journal of Computer Vision / Issue 1/2021
Print ISSN: 0920-5691
Electronic ISSN: 1573-1405
DOI
https://doi.org/10.1007/s11263-020-01368-1

Other articles of this Issue 1/2021

International Journal of Computer Vision 1/2021 Go to the issue

OriginalPaper

DeepVS2.0: A Saliency-Structured Deep Learning Method for Predicting Dynamic Visual Attention

OriginalPaper

View Transfer on Human Skeleton Pose: Automatically Disentangle the View-Variant and View-Invariant Information for Pose Representation Learning

OriginalPaper

Scene Text Detection and Recognition: The Deep Learning Era

OriginalPaper

Pixel-Wise Crowd Understanding via Synthetic Data

OriginalPaper

Improving Image Description with Auxiliary Modality for Visual Localization in Challenging Conditions

OriginalPaper

Temporally Coherent General Dynamic Scene Reconstruction

Premium Partner

    Image Credits