Top

Published in:

2025 | OriginalPaper | Chapter

TrackNeRF: Bundle Adjusting NeRF from Sparse and Noisy Views via Feature Tracks

Authors : Jinjie Mai, Wenxuan Zhu, Sara Rojas, Jesus Zarzar, Abdullah Hamdi, Guocheng Qian, Bing Li, Silvio Giancola, Bernard Ghanem

Published in: Computer Vision – ECCV 2024

Publisher: Springer Nature Switzerland

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Neural radiance fields (NeRFs) generally require many images with accurate poses for accurate novel view synthesis, which does not reflect realistic setups where views can be sparse and poses can be noisy. Previous solutions for learning NeRFs with sparse views and noisy poses only consider local geometry consistency with pairs of views. Closely following bundle adjustment in Structure-from-Motion (SfM), we introduce TrackNeRF for more globally consistent geometry reconstruction and more accurate pose optimization. TrackNeRF introduces feature tracks, i.e.connected pixel trajectories across all visible views that correspond to the same 3D points. By enforcing reprojection consistency among feature tracks, TrackNeRF encourages holistic 3D consistency explicitly. Through extensive experiments, TrackNeRF sets a new benchmark in noisy and sparse view reconstruction. In particular, TrackNeRF shows significant improvements over the state-of-the-art BARF and SPARF by \({\sim }8\) and \({\sim }1\) in terms of PSNR on DTU under various sparse and noisy view setups. The code is available at https://tracknerf.github.io/.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation

Agarwal, S., et al.: Building Rome in a day. Commun. ACM 54(10), 105–112 (2011)CrossRef

Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-Nerf: a multiscale representation for anti-aliasing neural radiance fields. In: CVPR, pp. 5855–5864 (2021)

Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: MIP-Nerf 360: unbounded anti-aliased neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5470–5479 (2022)

Charatan, D., Li, S.L., Tagliasacchi, A., Sitzmann, V.: pixelSplat: 3D gaussian splats from image pairs for scalable generalizable 3d reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19457–19467 (2024)

Chen, R., Chen, Y., Jiao, N., Jia, K.: Fantasia3D: disentangling geometry and appearance for high-quality text-to-3D content creation. arXiv preprint arXiv:2303.13873 (2023)

Chen, S., Li, J., Zhang, Y., Zou, B.: Improving neural radiance fields with depth-aware optimization for novel view synthesis. arXiv preprint arXiv:2304.05218 (2023)

Chen, S., Zhang, Y., Xu, Y., Zou, B.: Structure-aware nerf without posed camera via epipolar constraint. arXiv preprint arXiv:2210.00183 (2022)

Chen, Y., Lee, G.H.: DBARF: deep bundle-adjusting generalizable neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24–34 (2023)

Chen, Y., et al.: Local-to-global registration for bundle-adjusting neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8264–8273 (2023)

10.

Chen, Y., Xu, H., Wu, Q., Zheng, C., Cham, T.J., Cai, J.: Explicit correspondence matching for generalizable neural radiance fields. arXiv preprint arXiv:2304.12294 (2023)

11.

Chen, Y., et al.: MVSplat: efficient 3D Gaussian splatting from sparse multi-view images. arXiv preprint arXiv:2403.14627 (2024)

12.

Cheng, K., et al.: UC-Nerf: neural radiance field for under-calibrated multi-view cameras in autonomous driving. arXiv preprint arXiv:2311.16945 (2023)

13.

Chng, S.F., Ramasinghe, S., Sherrah, J., Lucey, S.: Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13693, pp. 264–280. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19827-4_16CrossRef

14.

Deng, K., Liu, A., Zhu, J.Y., Ramanan, D.: Depth-supervised NeRF: fewer views and faster training for free. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

15.

DeTone, D., Malisiewicz, T., Rabinovich, A.: Superpoint: self-supervised interest point detection and description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 224–236 (2018)

16.

Du, Y., Smith, C., Tewari, A., Sitzmann, V.: Learning to render novel views from wide-baseline stereo pairs. In: CVPR (2023)

17.

Dusmanu, M., Schönberger, J.L., Pollefeys, M.: Multi-view optimization of local feature geometry. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 670–686. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_39CrossRef

18.

Faugeras, O.D.: What can be seen in three dimensions with an uncalibrated stereo rig? In: Sandini, G. (ed.) ECCV 1992. LNCS, vol. 588, pp. 563–578. Springer, Heidelberg (1992). https://doi.org/10.1007/3-540-55426-2_61CrossRef

19.

Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., Kanazawa, A.: Plenoxels: radiance fields without neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5501–5510 (2022)

20.

Fu, H., Yu, X., Li, L., Zhang, L.: CBARF: cascaded bundle-adjusting neural radiance fields from imperfect camera poses. arXiv preprint arXiv:2310.09776 (2023)

21.

Fu, Y., Liu, S., Kulkarni, A., Kautz, J., Efros, A.A., Wang, X.: COLMAP-free 3D Gaussian splatting (2023)

22.

Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 270–279 (2017)

23.

Guangcong, Chen, Z., Loy, C.C., Liu, Z.: SparseNerf: distilling depth ranking for few-shot novel view synthesis. In: IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

24.

Hamdi, A., Ghanem, B., Nießner, M.: Sparf: large-scale learning of 3D sparse radiance fields from few input images. arxiv (2022)

25.

Hamdi, A., Giancola, S., Ghanem, B.: Voint Cloud: multi-view point cloud representation for 3D understanding. In: The Eleventh International Conference on Learning Representations (2023). https://openreview.net/forum?id=IpGgfpMucHj

26.

Hamdi, A., et al.: GES: generalized exponential splatting for efficient radiance field rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19812–19822 (2024)

27.

Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer series in statistics, Springer (2009). https://books.google.com.sa/books?id=eBSgoAEACAAJ

28.

Heise, P., Klose, S., Jensen, B., Knoll, A.: PM-Huber: PatchMatch with Huber regularization for stereo matching. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2360–2367 (2013)

29.

Hong, S., Jung, J., Shin, H., Yang, J., Kim, S., Luo, C.: Unifying correspondence, pose and nerf for pose-free novel view synthesis from stereo pairs. arXiv preprint arXiv:2312.07246 (2023)

30.

Hu, S., et al.: ConsistentNerf: enhancing neural radiance fields with 3D consistency for sparse view synthesis. arXiv preprint arXiv:2305.11031 (2023)

31.

Huang, P.H., Matzen, K., Kopf, J., Ahuja, N., Huang, J.B.: DeepMVS: learning multi-view stereopsis. In: CVPR, pp. 2821–2830 (2018)

32.

Jain, A., Tancik, M., Abbeel, P.: Putting nerf on a diet: semantically consistent few-shot view synthesis. In: CVPR, pp. 5885–5894 (2021)

33.

Jensen, R., Dahl, A., Vogiatzis, G., Tola, E., Aanæs, H.: Large scale multi-view stereopsis evaluation. In: CVPR, pp. 406–413 (2014)

34.

Jeong, Y., Ahn, S., Choy, C., Anandkumar, A., Cho, M., Park, J.: Self-calibrating neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5846–5854 (2021)

35.

Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3D Gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42(4) (2023). https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

36.

Kim, I., Choi, M., Kim, H.J.: Up-Nerf: unconstrained pose prior-free neural radiance field. In: Advances in Neural Information Processing Systems, vol. 36 (2024)

37.

Kim, M., Seo, S., Han, B.: InfoNerf: ray entropy minimization for few-shot neural volume rendering. In: CVPR, pp. 12912–12921 (2022)

38.

Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 25 (2012)

39.

Lao, Y., Xu, X., Cai, Z., Liu, X., Zhao, H.: CorresNerf: image correspondence priors for neural radiance fields. arXiv preprint arXiv:2312.06642 (2023)

40.

Li, J., et al.: DNGaussian: optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20775–20785 (2024)

41.

Li, M., Wang, P., Zhao, L., Liao, B., Liu, P.: USB-Nerf: unrolling shutter bundle adjusted neural radiance fields. arXiv preprint arXiv:2310.02687 (2023)

42.

Lin, C.H., et al.: Magic3D: high-resolution text-to-3d content creation. In: CVPR (2023)

43.

Lin, C.H., Ma, W.C., Torralba, A., Lucey, S.: BARF: bundle-adjusting neural radiance fields. In: IEEE International Conference on Computer Vision (ICCV) (2021)

44.

Lindenberger, P., Sarlin, P.E., Larsson, V., Pollefeys, M.: Pixel-perfect structure-from-motion with featuremetric refinement. In: ICCV (2021)

45.

Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-1-to-3: zero-shot one image to 3D object. arXiv preprint arXiv:2303.11328 (2023)

46.

Liu, T., Ye, X., Zhao, W., Pan, Z., Shi, M., Cao, Z.: When epipolar constraint meets non-local operators in multi-view stereo. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 18088–18097 (2023)

47.

Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., Sheikh, Y.: Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.07751 (2019)

48.

Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60, 91–110 (2004)CrossRef

49.

Mai, J., Hamdi, A., Giancola, S., Zhao, C., Ghanem, B.: EgoLoc: revisiting 3D object localization from egocentric videos with visual queries. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 45–57 (2023)

50.

Melas-Kyriazi, L., Rupprecht, C., Laina, I., Vedaldi, A.: Realfusion: 360\(^\circ \) reconstruction of any object from a single image. In: CVPR (2023)

51.

Mildenhall, B., et al.: Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph. (TOG) 38(4), 1–14 (2019)CrossRef

52.

Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24CrossRef

53.

Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. 41(4), 102:1–102:15 (2022). https://doi.org/10.1145/3528223.3530127

54.

Niemeyer, M., Barron, J.T., Mildenhall, B., Sajjadi, M.S., Geiger, A., Radwan, N.: RegNerf: regularizing neural radiance fields for view synthesis from sparse inputs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5480–5490 (2022)

55.

Park, K., Henzler, P., Mildenhall, B., Barron, J.T., Martin-Brualla, R.: Camp: camera preconditioning for neural radiance fields. ACM Trans, Graph (2023)

56.

Philip, J., Deschaintre, V.: Floaters no more: radiance field gradient scaling for improved near-camera training. In: Ritschel, T., Weidlich, A. (eds.) Eurographics Symposium on Rendering. The Eurographics Association (2023). https://doi.org/10.2312/sr.20231122

57.

Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: DreamFusion: text-to-3D using 2D diffusion. In: ICLR (2022)

58.

Qian, G., et al.: Magic123: one image to high-quality 3D object generation using both 2D and 3D diffusion priors. In: ICLR (2024)

59.

Rosinol, A., Leonard, J.J., Carlone, L.: Nerf-slam: real-time dense monocular slam with neural radiance fields. In: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3437–3444. IEEE (2023)

60.

Sargent, K., et al.: ZeroNVS: zero-shot 360-degree view synthesis from a single real image. In: CVPR, 2024 (2023)

61.

Sattler, T., et al.: Benchmarking 6dof outdoor visual localization in changing conditions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8601–8610 (2018)

62.

Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

63.

Schönberger, J.L., Zheng, E., Frahm, J.-M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 501–518. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_31CrossRef

64.

Seo, H., Kim, H., Kim, G., Chun, S.Y.: Ditto-nerf: diffusion-based iterative text to omni-directional 3d model. arXiv preprint arXiv:2304.02827 (2023)

65.

Sun, Y., et al.: iComMa: inverting 3D gaussians splatting for camera pose estimation via comparing and matching. arXiv preprint arXiv:2312.09031 (2023)

66.

Szymanowicz, S., Rupprecht, C., Vedaldi, A.: Splatter image: ultra-fast single-view 3D reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10208–10217 (2024)

67.

Tagliasacchi, A., Mildenhall, B.: Volume rendering digest (for nerf). arXiv preprint arXiv:2209.02417 (2022)

68.

Tristram, F., Gasperini, S., Tombari, F., Navab, N., Busam, B.: Re-nerfing: enforcing geometric constraints on neural radiance fields through novel views synthesis. In: arXiv preprint, under review (2023)

69.

Truong, P., Danelljan, M., Timofte, R., Van Gool, L.: PDC-Net+: enhanced probabilistic dense correspondence network. IEEE Trans. Pattern Anal. Mach. Intell. (2023)

70.

Truong, P., Rakotosaona, M.J., Manhardt, F., Tombari, F.: Sparf: neural radiance fields from sparse and noisy poses. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR (2023)

71.

Verbin, D., Hedman, P., Mildenhall, B., Zickler, T., Barron, J.T., Srinivasan, P.P.: Ref-NeRF: structured view-dependent appearance for neural radiance fields. In: CVPR, pp. 5481–5490. IEEE (2022)

72.

Wang, P., Zhao, L., Ma, R., Liu, P.: Bad-nerf: bundle adjusted deblur neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4170–4179 (2023)

73.

Wang, S., Leroy, V., Cabon, Y., Chidlovskii, B., Revaud, J.: DUSt3r: geometric 3D vision made easy (2023)

74.

Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRef

75.

Wang, Z., Wu, S., Xie, W., Chen, M., Prisacariu, V.A.: NeRF\(--\): neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 (2021)

76.

Wu, R., et al.: ReconFusion: 3D reconstruction with diffusion priors. arXiv (2023)

77.

Xia, Y., Tang, H., Timofte, R., Van Gool, L.: SiNeRF: sinusoidal neural radiance fields for joint pose estimation and scene reconstruction. arXiv preprint arXiv:2210.04553 (2022)

78.

Xiong, H., Muttukuru, S., Upadhyay, R., Chari, P., Kadambi, A.: SparseGS: real-time 360 \(^\circ \) sparse view synthesis using gaussian splatting. arXiv preprint arXiv:2312.00206 (2023)

79.

Yang, J., Pavone, M., Wang, Y.: FreeNeRF: improving few-shot neural rendering with free frequency regularization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR) (2023)

80.

Yao, Y., Luo, Z., Li, S., Fang, T., Quan, L.: MVSNet: depth inference for unstructured multi-view stereo. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 767–783 (2018)

81.

Yao, Y., Luo, Z., Li, S., Shen, T., Fang, T., Quan, L.: Recurrent MVSNet for high-resolution multi-view stereo depth inference. In: CVPR, pp. 5525–5534 (2019)

82.

Yin, Z., Shi, J.: GeoNet: unsupervised learning of dense depth, optical flow and camera pose. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1983–1992 (2018)

83.

Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelNeRF: neural radiance fields from one or few images. In: CVPR, pp. 4578–4587 (2021)

84.

Yu, Z., Gao, S.: Fast-MVSNet: sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement. In: CVPR, pp. 1949–1958 (2020)

85.

Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)

86.

Zhao, W., Liu, S., Guo, H., Wang, W., Liu, Y.J.: ParticleSfM: exploiting dense point trajectories for localizing moving cameras in the wild. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13692, pp. 523–542. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19824-3_31CrossRef

87.

Zhou, Y., Barnes, C., Lu, J., Yang, J., Li, H.: On the continuity of rotation representations in neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5745–5753 (2019)

88.

Zhu, H., He, T., Chen, Z.: CMC: few-shot novel view synthesis via cross-view multiplane consistency. arXiv preprint arXiv:2402.16407 (2024)

89.

Zhu, Z., Fan, Z., Jiang, Y., Wang, Z.: FSGS: real-time few-shot view synthesis using gaussian splatting. arXiv preprint arXiv:2312.00451 (2023)

90.

Zhu, Z., et al.: NICE-SLAM: neural implicit scalable encoding for slam. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12786–12796 (2022)

Title: TrackNeRF: Bundle Adjusting NeRF from Sparse and Noisy Views via Feature Tracks
Authors: Jinjie Mai
Wenxuan Zhu
Sara Rojas
Jesus Zarzar
Abdullah Hamdi
Guocheng Qian
Bing Li
Silvio Giancola
Bernard Ghanem
Publisher: Springer Nature Switzerland
Book: Computer Vision – ECCV 2024
Print ISBN: 978-3-031-73253-9

Electronic ISBN: 978-3-031-73254-6

Copyright Year: 2025
DOI: https://doi.org/10.1007/978-3-031-73254-6_27

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"