skip to main content
research-article

Deep extraction of manga structural lines

Authors Info & Claims
Published:20 July 2017Publication History
Skip Abstract Section

Abstract

Extraction of structural lines from pattern-rich manga is a crucial step for migrating legacy manga to digital domain. Unfortunately, it is very challenging to distinguish structural lines from arbitrary, highly-structured, and black-and-white screen patterns. In this paper, we present a novel data-driven approach to identify structural lines out of pattern-rich manga, with no assumption on the patterns. The method is based on convolutional neural networks. To suit our purpose, we propose a deep network model to handle the large variety of screen patterns and raise output accuracy. We also develop an efficient and effective way to generate a rich set of training data pairs. Our method suppresses arbitrary screen patterns no matter whether these patterns are regular, irregular, tone-varying, or even pictorial, and regardless of their scales. It outputs clear and smooth structural lines even if these lines are contaminated by and immersed in complex patterns. We have evaluated our method on a large number of mangas of various drawing styles. Our method substantially outperforms state-of-the-art methods in terms of visual quality. We also demonstrate its potential in various manga applications, including manga colorization, manga retargeting, and 2.5D manga generation.

Skip Supplemental Material Section

Supplemental Material

papers-0399.mp4

mp4

334.4 MB

References

  1. Pablo Arbelaez, Michael Maire, Charless C. Flowlkes, and Jitendra Malik. 2011. Contour Detection and Hierarchical Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33, 5 (2011), 898--916. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. ACM Transactions on Graphics (Proc. SIGGRAPH) 28, 3 (Aug. 2009).Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. John Canny. 1986. A Computational Approach to Edge Detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 6 (1986), 679--698. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. François Chollet. 2015. Keras. https://github.com/fchollet/keras. (2015).Google ScholarGoogle Scholar
  5. James Hays, Marius Leordeanu, Alexei A Efros, and Yanxi Liu. 2006. Discovering texture regularity as a higher-order correspondence problem. In Computer Vision-ECCV 2006. Springer, 522--535.Google ScholarGoogle Scholar
  6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016a. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27--30, 2016. 770--778. Google ScholarGoogle ScholarCross RefCross Ref
  7. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b. Identity Mappings in Deep Residual Networks. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV. 630--645. Google ScholarGoogle ScholarCross RefCross Ref
  8. Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6--11 July 2015. 448--456.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H. Adelson. 2014. Crisp Boundary Detection Using Pointwise Mutual Information. In ECCV. Google ScholarGoogle ScholarCross RefCross Ref
  10. Kota Ito, Yusuke Matsui, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2015. Separation of Manga Line Drawings and Screentones. In Eurographics 2015 - Short Papers, Zurich, Switzerland, May 4--8, 2015. 73--76.Google ScholarGoogle Scholar
  11. Henry Kang, Seungyong Lee, and Charles K. Chui. 2007. Coherent line drawing. In Proceedings of the 5th International Symposium on Non-Photorealistic Animation and Rendering 2007, San Diego, California, USA, August 4--5, 2007. 43--50. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014).Google ScholarGoogle Scholar
  13. Iasonas Kokkinos. 2015. Pushing the boundaries of boundary detection using deep learning. arXiv preprint arXiv:1511.07386 (2015).Google ScholarGoogle Scholar
  14. Johannes Kopf and Dani Lischinski. 2012. Digital reconstruction of halftoned color comics. ACM Transactions on Graphics (TOG) 31, 6 (2012), 140.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3--6, 2012, Lake Tahoe, Nevada, United States. 1106--1114.Google ScholarGoogle Scholar
  16. Xueting Liu, Xiangyu Mao, Xuan Yang, Linling Zhang, and Tien-Tsin Wong. 2013. Stereoscopizing Cel Animations. ACM Transactions on Graphics (SIGGRAPH Asia 2013 issue) 32, 6 (November 2013), 223:1--223:10.Google ScholarGoogle Scholar
  17. Yanxi Liu, Tamara Belkina, James Hays, and Roberto Lublinerman. 2008. Image Defencing. In Proceedings of CVPR 2008.Google ScholarGoogle Scholar
  18. Yanxi Liu, Robert T Collins, and Yanghai Tsin. 2004a. A computational model for periodic pattern perception based on frieze and wallpaper groups. Pattern Analysis and Machine Intelligence, IEEE Transactions on 26, 3 (2004), 354--371.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Yanxi Liu, Wen-Chieh Lin, and James Hays. 2004b. Near-regular texture analysis and manipulation. In ACM Transactions on Graphics (TOG), Vol. 23. ACM, 368--376. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Xiangyu Mao, Xueting Liu, Tien-Tsin Wong, and Xuemiao Xu. 2015. Region-based structure line detection for cartoons. Computational Visual Media 1, 1 (2015), 69--78. Google ScholarGoogle ScholarCross RefCross Ref
  21. Yusuke Matsui, Kota Ito, Yuji Aramaki, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2015. Sketch-based Manga Retrieval using Manga109 Dataset. CoRR abs/1510.04389 (2015).Google ScholarGoogle Scholar
  22. Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. 2015. Learning Deconvolution Network for Semantic Segmentation. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7--13, 2015. 1520--1528. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Yingge Qu, Tien-Tsin Wong, and Pheng-Ann Heng. 2006. Manga colorization. ACM Transactions on Graphics (TOG) 25, 3 (2006), 1214--1220. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015 - 18th International Conference Munich, Germany, October 5 - 9, 2015, Proceedings, Part III. 234--241. Google ScholarGoogle ScholarCross RefCross Ref
  25. Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang Zhang. 2015. Deep-Contour: A deep convolutional feature learned by positive-sharing loss for contour detection. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7--12, 2015. 3982--3991.Google ScholarGoogle ScholarCross RefCross Ref
  26. Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016. Learning to simplify: fully convolutional networks for rough sketch cleanup. ACM Trans. Graph. 35, 4 (2016), 121. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014).Google ScholarGoogle Scholar
  28. Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. 2014. Striving for Simplicity: The All Convolutional Net. CoRR abs/1412.6806 (2014).Google ScholarGoogle Scholar
  29. Daniel Sýkora, Jan Buriánek, and Jiří Žára. 2004. Unsupervised colorization of black-and-white cartoons. In International Symposium on Non-Photorealistic Animation and Rendering. 121--127. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Carlo Tomasi and Roberto Manduchi. 1998. Bilateral Filtering for Gray and Color Images. In ICCV. 839--846. Google ScholarGoogle ScholarCross RefCross Ref
  31. Luminita A. Vese and Stanley Osher. 2003. Modeling Textures with Total Variation Minimization and Oscillating Patterns in Image Processing. J. Sci. Comput. 19, 1--3 (2003), 553--572.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Saining Xie and Zhuowen Tu. 2015. Holistically-Nested Edge Detection. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7--13, 2015. 1395--1403. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Li Xu, Cewu Lu, Yi Xu, and Jiaya Jia. 2011. Image smoothing via L0 gradient minimization. ACM Trans. Graph. 30, 6 (2011), 174. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Li Xu, Qiong Yan, Yang Xia, and Jiaya Jia. 2012. Structure extraction from texture via relative total variation. ACM Trans. Graph. 31, 6 (2012), 139. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Chih-Yuan Yao, Shih-Hsuan Hung, Guo-Wei Li, I-Yu Chen, Reza Adhitya, and Yu-Chi Lai. 2017. Manga Vectorization and Manipulation with Procedural Simple Screentone. IEEE Trans. Vis. Comput. Graph. 23, 2 (2017), 1070--1084. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Song-Hai Zhang, Tao Chen, Yi-Fei Zhang, Shi-Min Hu, and Ralph R Martin. 2009. Vectorizing cartoon animations. IEEE Transactions on Visualization and Computer Graphics 15, 4 (2009), 618--629.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. 2016. Loss Functions for Neural Networks for Image Processing. CoRR abs/1511.08861 (2016).Google ScholarGoogle Scholar

Index Terms

  1. Deep extraction of manga structural lines

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Graphics
      ACM Transactions on Graphics  Volume 36, Issue 4
      August 2017
      2155 pages
      ISSN:0730-0301
      EISSN:1557-7368
      DOI:10.1145/3072959
      Issue’s Table of Contents

      Copyright © 2017 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 20 July 2017
      Published in tog Volume 36, Issue 4

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader