Skip to main content
Top

2024 | OriginalPaper | Chapter

X-NR: Towards An Extended Reality-Driven Human Evaluation Framework for Neural-Rendering

Authors : Lorenzo Stacchio, Emanuele Balloni, Lucrezia Gorgoglione, Marina Paolanti, Emanuele Frontoni, Roberto Pierdicca

Published in: Extended Reality

Publisher: Springer Nature Switzerland

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

The joint usage of Extended Reality (XR) and Artificial Intelligence (AI) has enabled different Metaverse-related use cases. Such paradigms were recently adopted for immersive content creation, particularly considering Neural Rendering (NR) techniques to project scenes from the real world in the 3D realm. These methods are particularly beneficial in the field of Cultural Heritage (CH), where digitizing and visualizing cultural assets in 3D is crucial. However, current evaluation protocols lack a robust integration of human judgments through a Human-In-The-Loop (HITL) approach to humanly evaluate the quality of the generated 3D models, which could also support model optimization. To bridge this gap, we here introduce X-NR, a novel XR framework designed to evaluate and compare 3D reconstruction methodologies, including NR in the context of CH. We contextualize and validate such a framework through case studies on cultural heritage sites in the Marche region (Italy), employing various data-capturing and 3D reconstruction methodologies. The study concludes with a validation of the framework by CH domain experts, underscoring its potential advantages over traditional 3D editing software.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
2.
go back to reference Balloni, E., Gorgoglione, L., Paolanti, M., Mancini, A., Pierdicca, R.: Few shot photogrametry: a comparison between nerf and mvs-sfm for the documentation of cultural heritage. Int. Arch. Photogramm. Remote. Sens. Spat. Inf. Sci. 48, 155–162 (2023)CrossRef Balloni, E., Gorgoglione, L., Paolanti, M., Mancini, A., Pierdicca, R.: Few shot photogrametry: a comparison between nerf and mvs-sfm for the documentation of cultural heritage. Int. Arch. Photogramm. Remote. Sens. Spat. Inf. Sci. 48, 155–162 (2023)CrossRef
3.
go back to reference Brooke, J., et al.: Sus-a quick and dirty usability scale. Usability Evaluation Industry 189(194), 4–7 (1996) Brooke, J., et al.: Sus-a quick and dirty usability scale. Usability Evaluation Industry 189(194), 4–7 (1996)
4.
go back to reference Chen, M., Jin, Y., Goodall, T., Yu, X., Bovik, A.C.: Study of 3d virtual reality picture quality. IEEE J. Selected Topics Signal Process. 14(1), 89–102 (2019)CrossRef Chen, M., Jin, Y., Goodall, T., Yu, X., Bovik, A.C.: Study of 3d virtual reality picture quality. IEEE J. Selected Topics Signal Process. 14(1), 89–102 (2019)CrossRef
5.
go back to reference Croce, V., Caroti, G., De Luca, L., Piemonte, A., Véron, P.: Neural radiance fields (nerf): review and potential applications to digital cultural heritage. Int. Arch. Photogramm. Remote. Sens. Spat. Inf. Sci. 48, 453–460 (2023)CrossRef Croce, V., Caroti, G., De Luca, L., Piemonte, A., Véron, P.: Neural radiance fields (nerf): review and potential applications to digital cultural heritage. Int. Arch. Photogramm. Remote. Sens. Spat. Inf. Sci. 48, 453–460 (2023)CrossRef
6.
go back to reference Deng, C., et al.: Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20637–20647 (2023) Deng, C., et al.: Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20637–20647 (2023)
8.
go back to reference Fangi, G., Wahbeh, W., Malinverni, E.S., Di Stefano, F., Pierdicca, R.: Documentation of syrian lost heritage: From 3d reconstruction to open information system (2022) Fangi, G., Wahbeh, W., Malinverni, E.S., Di Stefano, F., Pierdicca, R.: Documentation of syrian lost heritage: From 3d reconstruction to open information system (2022)
9.
go back to reference Gao, K., Gao, Y., He, H., Lu, D., Xu, L., Li, J.: Nerf: Neural radiance field in 3d vision, a comprehensive review. arXiv preprint arXiv:2210.00379 (2022) Gao, K., Gao, Y., He, H., Lu, D., Xu, L., Li, J.: Nerf: Neural radiance field in 3d vision, a comprehensive review. arXiv preprint arXiv:​2210.​00379 (2022)
10.
go back to reference Guédon, A., Lepetit, V.: Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.12775 (2023) Guédon, A., Lepetit, V.: Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:​2311.​12775 (2023)
11.
go back to reference Hart, S.G., Staveland, L.E.: Development of nasa-tlx (task load index): results of empirical and theoretical research. In: Advances in psychology, vol. 52, pp. 139–183. Elsevier (1988) Hart, S.G., Staveland, L.E.: Development of nasa-tlx (task load index): results of empirical and theoretical research. In: Advances in psychology, vol. 52, pp. 139–183. Elsevier (1988)
12.
go back to reference Hily, A., Dupont, L., Arbelaez-Garces, G., Camargo, M., Dinet, J.: Evaluation and validation process of extended reality applications developed in an industrial context: a systematic review. SN Comput. Sci. 4(5), 637 (2023)CrossRef Hily, A., Dupont, L., Arbelaez-Garces, G., Camargo, M., Dinet, J.: Evaluation and validation process of extended reality applications developed in an industrial context: a systematic review. SN Comput. Sci. 4(5), 637 (2023)CrossRef
13.
go back to reference Hirzle, T., Müller, F., Draxler, F., Schmitz, M., Knierim, P., Hornbæk, K.: When xr and ai meet-a scoping review on extended reality and artificial intelligence. In: Proceedings of the 2023 CHI Conference on Human Factors in ComputingmSystems, pp. 1–45 (2023) Hirzle, T., Müller, F., Draxler, F., Schmitz, M., Knierim, P., Hornbæk, K.: When xr and ai meet-a scoping review on extended reality and artificial intelligence. In: Proceedings of the 2023 CHI Conference on Human Factors in ComputingmSystems, pp. 1–45 (2023)
14.
go back to reference Holzwarth, V., Gisler, J., Hirt, C., Kunz, A.: Comparing the accuracy and precision of steamvr tracking 2.0 and oculus quest 2 in a room scale setup. In: Proceedings of the 2021 5th International Conference on Virtual and Augmented Reality Simulations, pp. 42–46 (2021) Holzwarth, V., Gisler, J., Hirt, C., Kunz, A.: Comparing the accuracy and precision of steamvr tracking 2.0 and oculus quest 2 in a room scale setup. In: Proceedings of the 2021 5th International Conference on Virtual and Augmented Reality Simulations, pp. 42–46 (2021)
15.
go back to reference Huynh-The, T., Pham, Q.V., Pham, X.Q., Nguyen, T.T., Han, Z., Kim, D.S.: Artificial intelligence for the metaverse: a survey. Eng. Appl. Artif. Intell. 117, 105581 (2023)CrossRef Huynh-The, T., Pham, Q.V., Pham, X.Q., Nguyen, T.T., Han, Z., Kim, D.S.: Artificial intelligence for the metaverse: a survey. Eng. Appl. Artif. Intell. 117, 105581 (2023)CrossRef
16.
go back to reference Jin, Y., Chen, M., Goodall, T., Patney, A., Bovik, A.C.: Subjective and objective quality assessment of 2d and 3d foveated video compression in virtual reality. IEEE Trans. Image Process. 30, 5905–5919 (2021)CrossRef Jin, Y., Chen, M., Goodall, T., Patney, A., Bovik, A.C.: Subjective and objective quality assessment of 2d and 3d foveated video compression in virtual reality. IEEE Trans. Image Process. 30, 5905–5919 (2021)CrossRef
17.
go back to reference Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42(4) (2023) Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42(4) (2023)
18.
go back to reference Li, B., Xu, X., Tang, S., Yu, L., Wang, Z.: Human perception-guided meta-training for few-shot nerf. In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 13206–13210. IEEE (2024) Li, B., Xu, X., Tang, S., Yu, L., Wang, Z.: Human perception-guided meta-training for few-shot nerf. In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 13206–13210. IEEE (2024)
19.
go back to reference Li, K., Schmidt, S., Rolff, T., Bacher, R., Leemans, W., Steinicke, F.: Magic nerf lens: interactive fusion of neural radiance fields for virtual facility inspection. Front. Virtual Reality 5, 1377245 (2024)CrossRef Li, K., Schmidt, S., Rolff, T., Bacher, R., Leemans, W., Steinicke, F.: Magic nerf lens: interactive fusion of neural radiance fields for virtual facility inspection. Front. Virtual Reality 5, 1377245 (2024)CrossRef
20.
go back to reference Li, Z., et al.: Neuralangelo: high-fidelity neural surface reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8456–8465 (2023) Li, Z., et al.: Neuralangelo: high-fidelity neural surface reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8456–8465 (2023)
21.
go back to reference Li, Z., Zhang, Y., Wu, C., Zhu, J., Zhang, L.: Ho-gaussian: Hybrid optimization of 3d gaussian splatting for urban scenes. arXiv preprint arXiv:2403.20032 (2024) Li, Z., Zhang, Y., Wu, C., Zhu, J., Zhang, L.: Ho-gaussian: Hybrid optimization of 3d gaussian splatting for urban scenes. arXiv preprint arXiv:​2403.​20032 (2024)
22.
go back to reference Liang, H., Wu, T., Hanji, P., Banterle, F., Gao, H., Mantiuk, R., Oztireli, C.: Perceptual quality assessment of nerf and neural view synthesis methods for front-facing views. arXiv preprint arXiv:2303.15206 (2023) Liang, H., Wu, T., Hanji, P., Banterle, F., Gao, H., Mantiuk, R., Oztireli, C.: Perceptual quality assessment of nerf and neural view synthesis methods for front-facing views. arXiv preprint arXiv:​2303.​15206 (2023)
24.
go back to reference Martin, P., Rodrigues, A., Ascenso, J., Queluz, M.P.: Nerf-qa: neural radiance fields quality assessment database. In: 2023 15th International Conference on Quality of Multimedia Experience (QoMEX), pp. 107–110. IEEE (2023) Martin, P., Rodrigues, A., Ascenso, J., Queluz, M.P.: Nerf-qa: neural radiance fields quality assessment database. In: 2023 15th International Conference on Quality of Multimedia Experience (QoMEX), pp. 107–110. IEEE (2023)
25.
go back to reference Mazzacca, G., et al.: Nerf for heritage 3d reconstruction. Inter. Archiv. Photogrammetry, Remote Sensing Spatial Inform. Sci. 48(M-2-2023), 1051–1058 (2023) Mazzacca, G., et al.: Nerf for heritage 3d reconstruction. Inter. Archiv. Photogrammetry, Remote Sensing Spatial Inform. Sci. 48(M-2-2023), 1051–1058 (2023)
26.
go back to reference Meng, X., Chen, W., Yang, B.: Neat: learning neural implicit surfaces with arbitrary topologies from multi-view images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 248–258 (2023) Meng, X., Chen, W., Yang, B.: Neat: learning neural implicit surfaces with arbitrary topologies from multi-view images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 248–258 (2023)
27.
28.
go back to reference Nagta, A., Sharma, B., Sharma, A., et al.: Oculus: a new dimension to virtual reality. In: 2022 International Conference on Automation, Computing and Renewable Systems (ICACRS), pp. 1169–1172. IEEE (2022) Nagta, A., Sharma, B., Sharma, A., et al.: Oculus: a new dimension to virtual reality. In: 2022 International Conference on Automation, Computing and Renewable Systems (ICACRS), pp. 1169–1172. IEEE (2022)
29.
go back to reference Nehmé, Y., Dupont, F., Farrugia, J.P., Le Callet, P., Lavoué, G.: Visual quality of 3d meshes with diffuse colors in virtual reality: subjective and objective evaluation. IEEE Trans. Visual Comput. Graph. 27(3), 2202–2219 (2020)CrossRef Nehmé, Y., Dupont, F., Farrugia, J.P., Le Callet, P., Lavoué, G.: Visual quality of 3d meshes with diffuse colors in virtual reality: subjective and objective evaluation. IEEE Trans. Visual Comput. Graph. 27(3), 2202–2219 (2020)CrossRef
30.
go back to reference Onuoha, C., Flaherty, J.A., Luo, S., Huong, T.T., Thang, T.C.: An evaluation of quality metrics for neural radiance field. In: 2023 IEEE 15th International Conference on Computational Intelligence and Communication Networks (CICN), pp. 619–623. IEEE (2023) Onuoha, C., Flaherty, J.A., Luo, S., Huong, T.T., Thang, T.C.: An evaluation of quality metrics for neural radiance field. In: 2023 IEEE 15th International Conference on Computational Intelligence and Communication Networks (CICN), pp. 619–623. IEEE (2023)
31.
go back to reference Pepe, M., Alfio, V.S., Costantino, D.: Assessment of 3d model for photogrammetric purposes using ai tools based on nerf algorithm. Heritage 6(8), 5719–5731 (2023)CrossRef Pepe, M., Alfio, V.S., Costantino, D.: Assessment of 3d model for photogrammetric purposes using ai tools based on nerf algorithm. Heritage 6(8), 5719–5731 (2023)CrossRef
32.
go back to reference Perez-Ortiz, M., Mikhailiuk, A., Zerman, E., Hulusic, V., Valenzise, G., Mantiuk, R.K.: From pairwise comparisons and rating to a unified quality scale. IEEE Trans. Image Process. 29, 1139–1151 (2019)MathSciNetCrossRef Perez-Ortiz, M., Mikhailiuk, A., Zerman, E., Hulusic, V., Valenzise, G., Mantiuk, R.K.: From pairwise comparisons and rating to a unified quality scale. IEEE Trans. Image Process. 29, 1139–1151 (2019)MathSciNetCrossRef
33.
go back to reference Picardi, A., Caruso, G.: User-centered evaluation framework to support the interaction design for augmented reality applications. Multimodal Technol. Interact. 8(5), 41 (2024)CrossRef Picardi, A., Caruso, G.: User-centered evaluation framework to support the interaction design for augmented reality applications. Multimodal Technol. Interact. 8(5), 41 (2024)CrossRef
34.
go back to reference Qu, Q., Liang, H., Chen, X., Chung, Y.Y., Shen, Y.: Nerf-nqa: no-reference quality assessment for scenes generated by nerf and neural view synthesis methods. IEEE Trans. Visualizat. Comput. Graph. (2024) Qu, Q., Liang, H., Chen, X., Chung, Y.Y., Shen, Y.: Nerf-nqa: no-reference quality assessment for scenes generated by nerf and neural view synthesis methods. IEEE Trans. Visualizat. Comput. Graph. (2024)
35.
go back to reference Qureshi, A.H., Alaloul, W.S., Murtiyoso, A., Saad, S., Manzoor, B.: Comparison of photogrammetry tools considering rebar progress recognition. Intern. Archiv. Photogrammetry, Remote Sensing Spatial Inform. Sci. 43, 141–146 (2022)CrossRef Qureshi, A.H., Alaloul, W.S., Murtiyoso, A., Saad, S., Manzoor, B.: Comparison of photogrammetry tools considering rebar progress recognition. Intern. Archiv. Photogrammetry, Remote Sensing Spatial Inform. Sci. 43, 141–146 (2022)CrossRef
36.
go back to reference Scorolli, C., Grasso, E.N., Stacchio, L., Armandi, V., Matteucci, G., Marfia, G.: Would you rather come to a tango concert in theater or in vr? aesthetic emotions & social presence in musical experiences, either live, 2d or 3d. Computers in Human Behavior, p. 107910 (2023) Scorolli, C., Grasso, E.N., Stacchio, L., Armandi, V., Matteucci, G., Marfia, G.: Would you rather come to a tango concert in theater or in vr? aesthetic emotions & social presence in musical experiences, either live, 2d or 3d. Computers in Human Behavior, p. 107910 (2023)
37.
go back to reference Series, B.: Methodology for the subjective assessment of the quality of television pictures. Recommendation ITU-R BT 500(13) (2012) Series, B.: Methodology for the subjective assessment of the quality of television pictures. Recommendation ITU-R BT 500(13) (2012)
38.
go back to reference Speicher, M., Cucerca, S., Krüger, A.: Vrshop: a mobile interactive virtual reality shopping environment combining the benefits of on-and offline shopping. Proc. ACM Interact. Mobile, Wearable Ubiquitous Technol. 1(3), 1–31 (2017)CrossRef Speicher, M., Cucerca, S., Krüger, A.: Vrshop: a mobile interactive virtual reality shopping environment combining the benefits of on-and offline shopping. Proc. ACM Interact. Mobile, Wearable Ubiquitous Technol. 1(3), 1–31 (2017)CrossRef
39.
go back to reference Stacchio, L., Scorolli, C., Marfia, G.: Evaluating human aesthetic and emotional aspects of 3d generated content through extended reality (2022) Stacchio, L., Scorolli, C., Marfia, G.: Evaluating human aesthetic and emotional aspects of 3d generated content through extended reality (2022)
40.
go back to reference Tancik, M., et al.: Nerfstudio: a modular framework for neural radiance field development. In: ACM SIGGRAPH 2023 Conference Proceedings, pp. 1–12 (2023) Tancik, M., et al.: Nerfstudio: a modular framework for neural radiance field development. In: ACM SIGGRAPH 2023 Conference Proceedings, pp. 1–12 (2023)
41.
go back to reference Teruggi, S., Grilli, E., Fassi, F., Remondino, F.: 3d surveying, semantic enrichment and virtual access of large cultural heritage. ISPRS Annals Photogrammetry, Remote Sensing Spatial Inform. Sci. 8, 155–162 (2021)CrossRef Teruggi, S., Grilli, E., Fassi, F., Remondino, F.: 3d surveying, semantic enrichment and virtual access of large cultural heritage. ISPRS Annals Photogrammetry, Remote Sensing Spatial Inform. Sci. 8, 155–162 (2021)CrossRef
42.
go back to reference Tewari, A., et al.: State of the art on neural rendering. In: Computer Graphics Forum, vol. 39, pp. 701–727. Wiley Online Library (2020) Tewari, A., et al.: State of the art on neural rendering. In: Computer Graphics Forum, vol. 39, pp. 701–727. Wiley Online Library (2020)
44.
go back to reference Venkatesh, V., Bala, H.: Technology acceptance model 3 and a research agenda on interventions. Decis. Sci. 39(2), 273–315 (2008)CrossRef Venkatesh, V., Bala, H.: Technology acceptance model 3 and a research agenda on interventions. Decis. Sci. 39(2), 273–315 (2008)CrossRef
45.
go back to reference Wang, Y., et al.: A survey on metaverse: Fundamentals, security, and privacy. IEEE Commun. Surv. Tutorials (2022) Wang, Y., et al.: A survey on metaverse: Fundamentals, security, and privacy. IEEE Commun. Surv. Tutorials (2022)
46.
go back to reference Yu, Z., et al.: Sdfstudio: A unified framework for surface reconstruction (2022) Yu, Z., et al.: Sdfstudio: A unified framework for surface reconstruction (2022)
47.
go back to reference Zerman, E., Hulusic, V., Valenzise, G., Mantiuk, R.K., Dufaux, F.: The relation between mos and pairwise comparisons and the importance of cross-content comparisons. Electronic Imaging 30, 1–6 (2018)CrossRef Zerman, E., Hulusic, V., Valenzise, G., Mantiuk, R.K., Dufaux, F.: The relation between mos and pairwise comparisons and the importance of cross-content comparisons. Electronic Imaging 30, 1–6 (2018)CrossRef
Metadata
Title
X-NR: Towards An Extended Reality-Driven Human Evaluation Framework for Neural-Rendering
Authors
Lorenzo Stacchio
Emanuele Balloni
Lucrezia Gorgoglione
Marina Paolanti
Emanuele Frontoni
Roberto Pierdicca
Copyright Year
2024
DOI
https://doi.org/10.1007/978-3-031-71707-9_25

Premium Partner