Savino Dambra
ABSTRACT
Streaming Virtual Reality (VR), even under the mere form of 360° videos, is much more complex than for regular videos because to lower the required rates, the transmission decisions must take the user's head position into account. The way the user exploits her/his freedom is therefore crucial for the network load. In turn, the way the user moves depends on the video content itself. VR is however a whole new medium, for which the film-making language does not exist yet, its "grammar" only being invented. We present a strongly inter-disciplinary approach to improve the streaming of 360° videos: designing high-level content manipulations (film editing) to limit and even control the user's motion in order to consume less bandwidth while maintaining the user's experience. We build an MPEG DASH-SRD player for Android and the Samsung Gear VR, featuring FoV-based quality decision and a replacement strategy to allow the tiles' buffers to build up while keeping their state up-to-date with the current FoV as much as bandwidth allows. The editing strategies we design have been integrated within the player, and the streaming module has been extended to benefit from the editing. Two sets of user experiments enabled to show that editing indeed impacts head velocity (reduction of up to 30%), consumed bandwidth (reduction of up to 25%) and subjective assessment. User's attention driving tools from other communities can hence be designed in order to improve streaming. We believe this innovative work opens up the path to a whole new field of possibilities in defining degrees of freedom to be wielded for VR streaming optimization.
- 360fly. 2016. 360fly 4K User Guide.Google Scholar
- P. Rondao Alface, M. Aerts, D. Tytgat, S. Lievens, C. Stevens, N. Verzijp, and J.-F. Macq. 2017. 16K Cinematic VR Streaming. In ACM Multimedia Conf. Google ScholarDigital Library
- Android application. 2017. Within: Storytelling for Virtual Reality. https://with.in/. (2017).Google Scholar
- B. Begole. 2016. Why The Internet Pipes Will Burst When Virtual Reality Takes Off. (Feb. 2016). Forbes.Google Scholar
- D. Bordwell and K. Thompson. 2006. Film Art: An Introduction. McGraw-Hill.Google Scholar
- J. Brillhart. 2016. VR and cinema, https://medium.com/@brillhart. (May. 2016). Google I/O conf.Google Scholar
- C. Brown, G. Bhutra, M. Suhail, Q. Xu, and E. D. Ragan. 2017. Coordinating attention and cooperation in multi-user virtual reality narratives. In IEEE VR.Google Scholar
- Cisco. 2017. Cisco Visual Networking Index: Forecast and Methodology, 2016-2021. (June 2017). White paper.Google Scholar
- X. Corbillon, A. Devlic, G. Simon, and J. Chakareski. 2017. Optimal Set of 360-Degree Videos for Viewport-Adaptive Streaming. In ACM Multimedia Conf. Google ScholarDigital Library
- X. Corbillon, G. Simon, A. Devlic, and J. Chakareski. 2017. Viewport-adaptive navigable 360-degree video delivery. In IEEE ICC.Google Scholar
- International Data Corporation. 2018. Demand for Augmented Reality/Virtual Reality Headsets Expected to Rebound in 2018. (Mar. 2018). Industry report.Google Scholar
- A. Coutrot and N. Guyader. 2017. Learning a time-dependent master saliency map from eye-tracking data in videos. CoRR abs/1702.00714 (2017). arXiv:1702.00714Google Scholar
- J.-E. Cutting, K.-L. Brunick, J.-E. DeLong, C. Iricinschi, and A. Candan. 2011. Quicker, faster, darker: Changes in Hollywood film over 75 years. i-Perception 2, 6 (2011), 569--576.Google Scholar
- S. Dambra, G. Samela, L. Sassatelli, R. Pighetti, R. Aparicio-Pardo, and A.-M. Pinna-Déry. 2018. TOUCAN-VR. Software ( 2018). https://github.com/UCA4SVR/TOUCAN-VRGoogle Scholar
- S. Dambra, G. Samela, L. Sassatelli, R. Pighetti, R. Aparicio-Pardo, and A.-M. Pinna-Déry. 2018. TOUCAN-VR-data. Software ( 2018). https://github.com/UCA4SVR/TOUCAN_VR_dataGoogle Scholar
- S. Dambra, G. Samela, L. Sassatelli, R. Pighetti, R. Aparicio-Pardo, and A.-M. Pinna-Déry. 2018. TOUCAN-VR-parametrizer. Software (2018). https://github.com/UCA4SVR/TOUCAN_VR_parametrizerGoogle Scholar
- S. Dambra, G. Samela, L. Sassatelli, R. Pighetti, R. Aparicio-Pardo, and A.-M. Pinna-Déry. 2018. TOUCAN-VR-preprocessing. Software (2018). https://github.com/UCA4SVR/TOUCAN-preprocessingGoogle Scholar
- V. A. de Jesus Oliveira, L. Brayda, L. Nedel, and A. Maciel. 2017. Designing a Vibrotactile Head-Mounted Display for Spatial Awareness in 3D Spaces. IEEE Trans. on Visualization and Computer Graphics 23, 4 (Apr. 2017), 1409--1417. Google ScholarDigital Library
- T. El-Ganainy and M. Hefeeda. 2016. Streaming Virtual Reality Content. CoRR abs/1612.08350 (2016). arXiv:1612.08350Google Scholar
- Facebook. 2017. Enhancing high-resolution 360 streaming with view prediction. (Apr. 2017). Facebook Developers Conference.Google Scholar
- Facebook. 2017. The Evolution of Dynamic Streaming. (Apr. 2017). Facebook Developers Conference.Google Scholar
- Facebook. 2017. VR201: Lessons from the Frontlines. (Apr. 2017). Facebook Developers Conference.Google Scholar
- C.-H. Fan, J. Lee, W.-C. Lo, C.-Y. Huang, K.-T. Chen, and C.-H. Hsu. 2017. Fixation Prediction for 360 Video Streaming in Head-Mounted Virtual Reality. In ACM NOSSDAV. Google ScholarDigital Library
- FFmpeg. 2017. FFmpeg. https://www.ffmpeg.org/. (2017).Google Scholar
- V. R. Gaddam, M. Riegler, R. Eg, C. Griwodz, and P. Halvorsen. 2016. Tiling in Interactive Panoramic Video: Approaches and Evaluation. IEEE Trans. on Multimedia 18, 9 (Sep. 2016), 1819--1831. Google ScholarDigital Library
- Google. 2017. Exoplayer. http://google.github.io/ExoPlayer/. (2017).Google Scholar
- GPAC. 2017. MP4Box. https://gpac.wp.imt.fr/mp4box/. (2017).Google Scholar
- GPAC. 2017. Osmo4. https://gpac.wp.imt.fr/player/. (2017).Google Scholar
- M. Graf, C. Timmerer, and C. Mueller. 2017. Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP: Design, Implementation, and Evaluation. In ACM MMSyS. Google ScholarDigital Library
- Q. Huynh-Thu, M. Barkowsky, and P. Le Callet. 2011. The Importance of Visual Attention in Improving the 3D-TV Viewing Experience: Overview and New Perspectives. IEEE Trans. on Broadcasting 57, 2 (Jun. 2011), 421--431.Google ScholarCross Ref
- Hollywood lexicon. 2000. 30 Degree Rule. http://www.hollywoodlexicon.com/thirtydegree.html. (2000).Google Scholar
- D. Liman. 2016. Invisible. http://www.imdb.com/title/tt6178894/. (2016). VR series, Samsung.Google Scholar
- O. Niamut, E. Thomas, L. D'Acunto, C. Concolato, F. Denoual, and S.-Y. Lim. 2016. MPEG DASH SRD: Spatial Relationship Description. In ACM MMSyS. Google ScholarDigital Library
- G.W. Oehlert. 2010. A First Course in Design and Analysis of Experiments. Macmillian.Google Scholar
- J. Orlosky, K. Kiyokawa, and H. Takemura. 2017. Virtual and Augmented Reality on the 5G Highway. J. of Information Processing 25 (2017), 133--141.Google ScholarCross Ref
- S. Petrangeli, V. Swaminathan, M. Hosseini, and F. De Turck. 2017. An HTTP/2-Based Adaptive Streaming Framework for 360 Virtual Reality Videos. In ACM Multimedia Conf. Google ScholarDigital Library
- S. Petrangeli, V. Swaminathan, M. Hosseini, and F. De Turck. 2017. Improving Virtual Reality Streaming Using HTTP/2. In ACM MMSyS. Google ScholarDigital Library
- J.-T. Petty. 2016. Gone. http://www.imdb.com/title/tt4876244/. (2016). VR series, Samsung.Google Scholar
- J. De Praeter, P. Duchi, G. Van Wallendael, J.-F. Macq, and P. Lambert. 2016. Efficient Encoding of Interactive Personalized Views Extracted from Immersive Video Content. In ACM Int. Workshop on Multimedia Alternate Realities. Google ScholarDigital Library
- M. Reilhac. 2016. Presence Design and Spatial Writing in Virtual Reality. (Jun. 2016). Lecture at FilmCamp Norway.Google Scholar
- S. Saarinen, V. Makela, P. Kallioniemi, J. Hakulinen, and M. Turune. 2017. Guidelines for Designing Interactive Omnidirectional Video Applications. In IFIP TNTERACT. Google ScholarDigital Library
- Samsung. 2017. Samsung Gear VR Framework. (2017).Google Scholar
- Preston Sturges. 1991. Preston Sturges by Preston Sturges: His Life in His Words. Touchstone.Google Scholar
- A. Taghavi Nasrabadi, A. Mahzari, J. D. Beshay, and R. Prakash. 2017. Adaptive 360-degree video streaming using layered video coding. In IEEE VR.Google Scholar
- C. Milk (Within). 2016. The birth of virtual reality as an art form. (Jun. 2016). TED talk.Google Scholar
- M. Xiao, C. Zhou, Y. Liu, and S. Chen. 2017. OpTile: Toward Optimal Tiling in 360-degree Video Streaming. In ACM Multimedia Conf. Google ScholarDigital Library
- C. Zhou, Z. Li, and Y. Liu. 2017. A Measurement Study of Oculus 360 Degree Video Streaming. In ACM NOSSDAV.Google Scholar
Index Terms
- Film editing: new levers to improve VR streaming
Recommendations
Heuristics for continuity editing of cinematic computer graphics scenes
Sandbox '09: Proceedings of the 2009 ACM SIGGRAPH Symposium on Video GamesWe present a set of heuristics for editing footage of 3D computer graphics cinematic sequences into a coherent movie clip which obeys the conventions of continuity editing. Our approach mimics the decision processes of an editor assembling a clip out of ...
Film clips retrieval using image queries
AbstractThe emergence of entertainment industry motivates the explosive growth of automatically film trailer. Manually finding desired clips from these large amounts of films is time-consuming and tedious, which makes finding the moments of user major or ...
AutoTag: automated metadata tagging for film post-production
AbstractFilm post-production can be time- and money-inefficient. The reason is that a lot of the work involves a person or group of people, called metadata taggers, going through each individual piece of media and marking it up with relevant tags, such as ...
Comments