Yue Wang
Justin M. Solomon
Skip Abstract Section
Abstract
Point clouds provide a flexible geometric representation suitable for countless applications in computer graphics; they also comprise the raw output of most 3D data acquisition devices. While hand-designed features on point clouds have long been proposed in graphics and vision, however, the recent overwhelming success of convolutional neural networks (CNNs) for image analysis suggests the value of adapting insight from CNN to the point cloud world. Point clouds inherently lack topological information, so designing a model to recover topology can enrich the representation power of point clouds. To this end, we propose a new neural network module dubbed EdgeConv suitable for CNN-based high-level tasks on point clouds, including classification and segmentation. EdgeConv acts on graphs dynamically computed in each layer of the network. It is differentiable and can be plugged into existing architectures. Compared to existing modules operating in extrinsic space or treating each point independently, EdgeConv has several appealing properties: It incorporates local neighborhood information; it can be stacked applied to learn global shape properties; and in multi-layer systems affinity in feature space captures semantic characteristics over potentially long distances in the original embedding. We show the performance of our model on standard benchmarks, including ModelNet40, ShapeNetPart, and S3DIS.
- Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 2016. 3D semantic parsing of large-scale indoor spaces. In Proceedings of the CVPR.Google Scholar
Cross Ref
- Matan Atzmon, Haggai Maron, and Yaron Lipman. 2018. Point convolutional neural networks by extension operators. ACM Trans. Graph. 37, 4, Article 71 (July 2018), 12 pages. DOI:https://doi.org/10.1145/3197517.3201301Google ScholarDigital Library
- Mathieu Aubry, Ulrich Schlickewei, and Daniel Cremers. 2011. The wave kernel signature: A quantum mechanical approach to shape analysis. In Proceedings of the ICCV Workshops.Google ScholarCross Ref
- Serge Belongie, Jitendra Malik, and Jan Puzicha. 2001. Shape context: A new descriptor for shape matching and object recognition. In Proceedings of the NIPS.Google Scholar
- Silvia Biasotti, Andrea Cerri, A. Bronstein, and M. Bronstein. 2016. Recent trends, applications, and perspectives in 3D shape similarity assessment. Comput. Graph. Forum 35, 6 (2016), 87--119.Google ScholarDigital Library
- Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael Bronstein. 2016. Learning shape correspondence with anisotropic convolutional neural networks. In Proceedings of the NIPS.Google Scholar
- Andrew Brock, Theodore Lim, James Millar Ritchie, and Nicholas J. Weston. 2016. Generative and discriminative voxel modeling with convolutional neural networks. In Proceedings of the NIPS.Google Scholar
- Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric deep learning: Going beyond euclidean data. IEEE Signal Process. Mag. 34, 4 (2017), 18--42.Google ScholarCross Ref
- Michael M. Bronstein and Iasonas Kokkinos. 2010. Scale-invariant heat kernel signatures for non-rigid shape recognition. In Proceedings of the CVPR.Google Scholar
- Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2013. Spectral networks and locally connected networks on graphs. arXiv:1312.6203 (2013).Google Scholar
- Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su et al. 2015. Shapenet: An information-rich 3D model repository. arXiv:1512.03012 (2015).Google Scholar
- Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the NIPS.Google Scholar
- Francis Engelmann, Theodora Kontogianni, Alexander Hermans, and Bastian Leibe. 2017. Exploring spatial context for 3D semantic segmentation of point clouds. In Proceedings of the CVPR.Google ScholarCross Ref
- Danielle Ezuz, Justin Solomon, Vladimir G. Kim, and Mirela Ben-Chen. 2017. GWCNN: A metric alignment layer for deep shape analysis. Comput. Graph. Forum 36, 5 (2017), 49--57.Google ScholarDigital Library
- Haoqiang Fan, Hao Su, and Leonidas J. Guibas. 2017. A point set generation network for 3D object reconstruction from a single image. In Proceedings of the CVPR.Google Scholar
- Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. 2018. SplineCNN: Fast geometric deep learning with continuous B-spline kernels. In Proceedings of the CVPR.Google ScholarCross Ref
- Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. arXiv:1704.01212 (2017).Google Scholar
- Aleksey Golovinskiy, Vladimir G. Kim, and Thomas Funkhouser. 2009. Shape-based recognition of 3D point clouds in urban environments. In Proceedings of the ICCV.Google ScholarCross Ref
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the NIPS.Google ScholarDigital Library
- Paul Guerrero, Yanir Kleiman, Maks Ovsjanikov, and Niloy J. Mitra. 2018. PCPNet: Learning local shape properties from raw point clouds. Comput. Graph. Forum 37, 2 (2018), 75--85. DOI:https://doi.org/10.1111/cgf.13343Google ScholarCross Ref
- Yulan Guo, Mohammed Bennamoun, Ferdous Sohel, Min Lu, and Jianwei Wan. 2014. 3D object recognition in cluttered scenes with local surface features: A survey. Trans. PAMI 36, 11 (2014), 2270--2287.Google ScholarCross Ref
- Oshri Halimi, Or Litany, Emanuele Rodolà, Alex Bronstein, and Ron Kimmel. 2018. Self-supervised learning of dense shape correspondence. arXiv:1812.02415 (2018).Google Scholar
- M. Henaff, J. Bruna, and Y. LeCun. 2015. Deep convolutional networks on graph-structured data. arXiv:1506.05163 (2015).Google Scholar
- Andrew E. Johnson and Martial Hebert. 1999. Using spin images for efficient object recognition in cluttered 3D scenes. Trans. PAMI 21, 5 (1999), 433--449.Google ScholarDigital Library
- Diederik P. Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv:1312.6114 (2013).Google Scholar
- Thomas N. Kipf and Max Welling. 2017. Semi-Supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR).Google Scholar
- Roman Klokov and Victor Lempitsky. 2017. Escape from cells: Deep Kd-networks for the recognition of 3D point cloud models. (2017).Google Scholar
- Ilya Kostrikov, Zhongshi Jiang, Daniele Panozzo, Denis Zorin, and Joan Bruna. 2017. Surface networks. In Proceedings of the CVPR.Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of the NIPS.Google ScholarDigital Library
- Yann LeCun, Bernhard Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne Hubbard, and Lawrence D. Jackel. 1989. Backpropagation applied to handwritten ZIP code recognition. Neural Comput. 1, 4 (1989), 541--551.Google ScholarDigital Library
- Ron Levie, Federico Monti, Xavier Bresson, and Michael M. Bronstein. 2017. CayleyNets: Graph convolutional neural networks with complex rational spectral filters. arXiv:1705.07664 (2017).Google Scholar
- Chun-Liang Li, Manzil Zaheer, Yang Zhang, Barnabas Poczos, and Ruslan Salakhutdinov. 2018b. Point cloud GAN. arXiv:1810.05795 (2018).Google Scholar
- Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. 2018a. PointCNN: Convolution On X-transformed points. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 820--830. Retrieved from http://papers.nips.cc/paper/7362-pointcnn-convolution-on-x-transformed-points.pdf.Google Scholar
- Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2016. Gated graph sequence neural networks. In Proceedings of the ICLR.Google Scholar
- Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. 2018. Deep continuous fusion for multi-sensor 3D object detection. In Proceedings of the ECCV.Google ScholarCross Ref
- Haibin Ling and David W. Jacobs. 2007. Shape classification using the inner-distance. Trans. PAMI 29, 2 (2007), 286--299.Google ScholarDigital Library
- Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Makadia. 2017a. Deformable shape completion with graph convolutional autoencoders. arXiv:1712.00268 (2017).Google Scholar
- Or Litany, Tal Remez, Emanuele Rodolà, Alex M. Bronstein, and Michael M. Bronstein. 2017b. Deep functional maps: Structured prediction for dense shape correspondence. In Proceedings of the ICCV.Google Scholar
- I. Loshchilov and F. Hutter. 2017. SGDR: Stochastic gradient descent with warm restarts. In Proceedings of the ICLR.Google Scholar
- Min Lu, Yulan Guo, Jun Zhang, Yanxin Ma, and Yinjie Lei. 2014. Recognizing objects in 3D point clouds with multi-scale local features. Sensors 14, 12 (2014), 24156--24173.Google ScholarCross Ref
- Siddharth Manay, Daniel Cremers, Byung-Woo Hong, Anthony J. Yezzi, and Stefano Soatto. 2006. Integral invariants for shape matching. Trans. PAMI 28, 10 (2006), 1602--1618.Google ScholarDigital Library
- Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope, Nadav Dym, Ersin Yumer, Vladimir G Kim, and Yaron Lipman. 2017. Convolutional neural networks on surfaces via seamless toric covers. In Proceedings of the SIGGRAPH.Google ScholarDigital Library
- Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. 2015. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the 3dRR.Google ScholarDigital Library
- Daniel Maturana and Sebastian Scherer. 2015. Voxnet: A 3D convolutional neural network for real-time object recognition. In Proceedings of the IROS.Google ScholarCross Ref
- Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and Michael M. Bronstein. 2017a. Geometric deep learning on graphs and manifolds using mixture model CNNs. In Proceedings of the CVPR.Google Scholar
- F. Monti, M. M. Bronstein, and X. Bresson. 2017b. Geometric matrix completion with recurrent multi-graph neural networks. In Proceedings of the NIPS.Google Scholar
- Federico Monti, Karl Otness, and Michael M. Bronstein. 2018. MotifNet: A motif-based graph convolutional network for directed graphs. arXiv:1802.01572 (2018).Google Scholar
- Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. 2012. Functional maps: A flexible representation of maps between shapes. Trans. Graph. 31, 4 (2012), 30.Google ScholarDigital Library
- Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J. Guibas. 2017a. Frustum PointNets for 3D object detection from RGB-D data. arXiv:1711.08488 (2017).Google Scholar
- Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017b. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the CVPR.Google Scholar
- Charles R. Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J. Guibas. 2016. Volumetric and multi-view CNNs for object classification on 3D data. In Proceedings of the CVPR.Google Scholar
- Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. 2017c. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In Proceedings of the NIPS.Google Scholar
- Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black. 2018. Generating 3D faces using convolutional mesh autoencoders. arXiv:1807.10267 (2018).Google Scholar
- Raif M. Rustamov. 2007. Laplace-beltrami eigenfunctions for deformation invariant shape representation. In Proceedings of the SGP.Google Scholar
- Radu Bogdan Rusu, Nico Blodow, and Michael Beetz. 2009. Fast point feature histograms (FPFH) for 3D registration. In Proceedings of the ICRA.Google ScholarCross Ref
- Radu Bogdan Rusu, Nico Blodow, Zoltan Csaba Marton, and Michael Beetz. 2008a. Aligning point cloud views using persistent feature histograms. In Proceedings of the IROS.Google ScholarCross Ref
- Radu Bogdan Rusu, Zoltan Csaba Marton, Nico Blodow, Mihai Dolha, and Michael Beetz. 2008b. Towards 3D point cloud-based object maps for household environments. Robot. Auton. Syst. J. 56, 11 (Nov. 2008), 927--941.Google ScholarDigital Library
- Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009. The graph neural network model. IEEE Tran. Neural Networks 20, 1 (2009), 61--80.Google ScholarDigital Library
- Syed Afaq Ali Shah, Mohammed Bennamoun, Farid Boussaid, and Amar A. El-Sallam. 2013. 3D-Div: A novel local surface descriptor for feature matching and pairwise range image registration. In Proceedings of the ICIP.Google Scholar
- Yiru Shen, Chen Feng, Yaoqing Yang, and Dong Tian. 2017. Neighbors do help: Deeply exploiting local structures of point clouds. arXiv:1712.06760 (2017).Google Scholar
- David I. Shuman, Sunil K. Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. 2013. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE Signal Process. Mag. 30, 3 (2013), 83--98.Google ScholarCross Ref
- Martin Simonovsky and Nikos Komodakis. 2017. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In Proceedings of the CVPR.Google ScholarCross Ref
- Ayan Sinha, Jing Bai, and Karthik Ramani. 2016. Deep learning 3D shape surfaces using geometry images. In Proceedings of the ECCV.Google ScholarCross Ref
- Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, and Jan Kautz. 2018. SPLATNet: Sparse lattice networks for point cloud processing. In Proceedings of the CVPR. 2530--2539.Google ScholarCross Ref
- Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015. Multi-view convolutional neural networks for 3D shape recognition. In Proceedings of the CVPR.Google ScholarDigital Library
- Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. 2009. A concise and provably informative multi-scale signature based on heat diffusion. Comput. Graph. Forum 28, 5 (2009), 1383--1392.Google ScholarCross Ref
- Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. 2017. Octree generating networks: Efficient convolutional architectures for high-resolution 3D outputs. In Proceedings of the ICCV.Google ScholarCross Ref
- Federico Tombari, Samuele Salti, and Luigi Di Stefano. 2011. A combined texture-shape descriptor for enhanced 3D feature matching. In Proceedings of the ICIP.Google ScholarCross Ref
- Oliver Van Kaick, Hao Zhang, Ghassan Hamarneh, and Daniel Cohen-Or. 2011. A survey on shape correspondence. Comput. Graph. Forum 30, 6 (2011), 1681--1707.Google ScholarCross Ref
- Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2017. Graph attention networks. arXiv:1710.10903.Google Scholar
- Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. 2018b. Deep parametric continuous convolutional neural networks. In Proceedings of the CVPR.Google ScholarCross Ref
- Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018a. Non-local neural networks. In Proceedings of the CVPR.Google ScholarCross Ref
- Lingyu Wei, Qixing Huang, Duygu Ceylan, Etienne Vouga, and Hao Li. 2016. Dense human body correspondences using convolutional networks. In Proceedings of the CVPR.Google ScholarCross Ref
- Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D shapenets: A deep representation for volumetric shapes. In Proceedings of the CVPR.Google Scholar
- Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. 2018. Feature denoising for improving adversarial robustness. arXiv:1812.03411.Google Scholar
- Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. 2018. FoldingNet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the CVPR.Google ScholarCross Ref
- Li Yi, Vladimir G. Kim, Duygu Ceylan, I. Shen, Mengyan Yan, Hao Su, A. R. Cewu Lu, Qixing Huang, Alla Sheffer, Leonidas Guibas et al. 2016. A scalable active framework for region annotation in 3D shape collections. Trans. Graph. 35, 6 (2016), 210.Google ScholarDigital Library
- Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. 2017. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In Proceedings of the ICRA.Google ScholarCross Ref
Index Terms
- Dynamic Graph CNN for Learning on Point Clouds
Recommendations
KeypointNet: Ranking Point Cloud for Convolution Neural Network
Image and GraphicsAbstractIn recent years, convolutional neural networks on point clouds have greatly improved the performance of point cloud classification and segmentation. However, the irregularity and disorder of point clouds make the convolution operation ill-suited ...
Learning Key Features Transformer Network for Point Cloud Processing
Pattern Recognition and Computer VisionAbstractDue to the unordered and irregular nature of point cloud data, it is challenging for neural networks to learn from it. Attention mechanisms have shown promising results in point cloud processing. It is also inherently permutation-invariant when ...
Representation Learning for Point Clouds with Variational Autoencoders
Computer Vision – ECCV 2022 WorkshopsAbstractDeep generative networks provide a way to generalize complex multi-dimensional data such as 3D point clouds. In this work, we present a novel method that operates on depth images and with the use of geometric images is able to learn the ...
Comments