Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
The Journal of Supercomputing
Published in: The Journal of Supercomputing 6/2021

16-11-2020

Mining user–user communities for a weighted bipartite network using spark GraphFrames and Flink Gelly

Authors: T. Ramalingeswara Rao, Soumya Kanti Ghosh, Adrijit Goswami

Published in: The Journal of Supercomputing | Issue 6/2021

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Large-scale graph processing is one of the recently developed significant research areas relevant to big data analytics. Distributed graph analytics is useful to see the intuitive insights of node interactions from large-scale network data. Distributed graph computing is an upcoming area in graph data mining that explores crucial node relationships for a given graph dataset. In this paper, we propose a new method to discover top-k user–user communities for a weighted bipartite network by defining a weighted similarity measure. We extend the structural similarity metric, namely Otsuka–Ochiai coefficient, by adding weights of nodes and quantifies the similarity between distinct items of a user–item network. We propose a new method to mine top-k user–user communities based on the similarity of items using a weighted similarity measure. Further, two algorithms, namely TUCSGF, TUCFlink, are presented to mine top-k user–user communities in a distributed approach based on the strength of the item-to-item similarities. Moreover, we execute the TUCSGF algorithm using Apache Spark by utilizing the advantage of Spark GraphFrames to mine top-k user–user communities. Also, we implement the TUCFlink algorithm to mine top-k communities using Apache Flink by utilizing the functionalities of Flink Gelly. Further, we explore two real-world network applications online learning network, chain of hospitals network with various graph methods that are to be applied for both the applications. Furthermore, we systematically perform various experiments concerning execution time, memory consumption, and CPU usage of both TUCSGF, TUCFlink on three distinct datasets. The performance of TUCFLINK is far better than TUCSGF concerning computing time. Applying distributed graph analytics for various complex networks using distributed graph processing tools GraphX, GraphFrames and Gelly provides more intuitive insights about distinct types of node interactions in graph data mining.
previous article OKCM: improving parallel task scheduling in high-performance computing systems using online learning
next article Performance optimization in delay tolerant networks using backtracking algorithm for fully credits distribution to contrast selfish nodes

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now
Literature
1.
2.
Alzahrani T, Horadam KJ, Boztas S (2014) Community detection in bipartite networks using random walks. In: Contucci P, Menezes R, Omicini A, Poncela-Casasnovas J (eds) Complex networks V. Studies in Computational Intelligence, vol 549. Springer, Cham, pp 157–165. https://​doi.​org/​10.​1007/​978-3-319-05401-8_​15. ISBN: 978-3-319-05401-8.
3.
Avery C (2011) Giraph: large-scale graph processing infrastructure on hadoop. Proc Hadoop Summit Santa Clara 11(3):5–9
4.
Baeza-Yates R, Ribeiro-Neto B et al (1999) Modern information retrieval, vol 463. ACM Press, New York
5.
Banadaki SVM, Lattanzi S, Feldman JE, Epasto A, Leonardi S, Lynch H, Sharma V (2015) Efficient similarity ranking for bipartite graphs. US Patent App. 14/278,811
6.
Barber MJ (2007) Modularity and community detection in bipartite networks. Phys Rev E 76(6):066102MathSciNet
7.
Beckett SJ (2016) Improved community detection in weighted bipartite networks. R Soc Open Sci 3(1):140,536MathSciNet
8.
Bhih A, Johnson P, Randles M (2020) An optimisation tool for robust community detection algorithms using content and topology information. J Supercomput 76(1):226–254
9.
Boccaletti S, Latora V, Moreno Y, Chavez M, Hwang DU (2006) Complex networks: structure and dynamics. Phys Rep 424(4–5):175–308MathSciNetMATH
10.
Bu Y, Howe B, Balazinska M, Ernst MD (2010) Haloop: efficient iterative data processing on large clusters. Proc VLDB Endow 3(1–2):285–296
11.
Carbone P, Katsifodimos A, Ewen S, Markl V, Haridi S et al (2015) Apache flink: stream and batch processing in a single engine. Bull IEEE Comput Soc Tech Committee Data Eng 36(4):28–38
12.
Carrington PJ, Scott J, Wasserman S (2005) Models and methods in social network analysis, vol 28. Cambridge University Press, Cambridge
13.
14.
Chen R, Shi J, Chen Y, Zang B, Guan H, Chen H (2019) Powerlyra: differentiated graph computation and partitioning on skewed graphs. ACM Trans Parallel Comput (TOPC) 5(3):13
15.
16.
17.
18.
Cui Y, Wang X (2016) Detecting one-mode communities in bipartite networks by bipartite clustering triangular. Phys A 457:307–315
19.
Dave A, Jindal A, Li LE, Xin R, Gonzalez J, Zaharia M (2016) Graphframes: an integrated API for mixing graph and relational queries. In: Proceedings of the Fourth International Workshop on Graph Data Management Experiences and Systems, pp 1–8
20.
Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51(1):107–113
21.
Deshpande M, Karypis G (2004) Item-based top-n recommendation algorithms. ACM Trans Inf Syst (TOIS) 22(1):143–177
22.
Du N, Wang B, Wu B, Wang Y (2008) Overlapping community detection in bipartite networks. In: 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology. IEEE, vol 1, pp 176–179
23.
Ekanayake J, Li H, Zhang B, Gunarathne T, Bae SH, Qiu J, Fox G (2010) Twister: a runtime for iterative mapreduce. In: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, pp 810–818
24.
Fani H, Jiang E, Bagheri E, Al-Obeidat F, Du W, Kargar M (2020) User community detection via embedding of social network structure and temporal content. Inf Process Manag 57(2):102,056
25.
26.
Fortunato S, Barthelemy M (2007) Resolution limit in community detection. Proc Natl Acad Sci 104(1):36–41
27.
28.
Girvan M, Newman ME (2002) Community structure in social and biological networks. Proc Natl Acad Sci 99(12):7821–7826MathSciNetMATH
29.
Gonzalez JE, Low Y, Gu H, Bickson D, Guestrin C (2012) Powergraph: distributed graph-parallel computation on natural graphs. In: OSDI, vol 12, p 2
30.
Grujić J (2008) Movies recommendation networks as bipartite graphs. In: International Conference on Computational Science, Springer, pp 576–583
31.
GSA (2019) Introducing gelly: graph processing with apache flink. Apache Flink. Accessed 20 Aug 2019
32.
Guimerà R, Sales-Pardo M, Amaral LAN (2007) Module identification in bipartite and directed networks. Phys Rev E 76(3):036,102
33.
Heidari S, Simmhan Y, Calheiros RN, Buyya R (2018) Scalable graph processing frameworks: a taxonomy and open challenges. ACM Comput Surv (CSUR) 51(3):60
34.
Jackson DA, Somers KM, Harvey HH (1989) Similarity coefficients: measures of co-occurrence and association or simply measures of occurrence. Am Nat 133(3):436–453
35.
Jeh G, Widom J (2002) Simrank: a measure of structural-context similarity. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp 538–543
36.
Kalavri V, Vlassov V, Haridi S (2018) High-level programming abstractions for distributed graph processing. IEEE Trans Knowl Data Eng 1:1–1
37.
Kang U, Tsourakakis CE, Faloutsos C (2009) Pegasus: a peta-scale graph mining system implementation and observations. In: Ninth IEEE International Conference on Data Mining, 2009. ICDM’09. IEEE, pp 229–238
38.
Kang U, Tsourakakis CE, Faloutsos C (2011) Pegasus: mining peta-scale graphs. Knowl Inf Syst 27(2):303–325
39.
Kholod I, Shorov A, Titkov E, Gorlatch S (2019) A formally based parallelization of data mining algorithms for multi-core systems. J Supercomput 75(12):7909–7920
40.
Kuzilek J, Hlosta M, Zdrahal Z (2017) Open university learning analytics dataset. Sci Data 4(170):171
41.
Lambiotte R, Ausloos M (2005) Uncovering collective listening habits and music genres in bipartite networks. Phys Rev E 72(6):066107
42.
Lambiotte R, Ausloos M (2006) On the genre-fication of music: a percolation approach. Eur Phys J B Condens Matter Compl Syst 50(1–2):183–188
43.
Lehmann S, Schwartz M, Hansen LK (2008) Biclique communities. Phys Rev E 78(1):016,108MathSciNet
44.
Leicht EA, Newman ME (2008) Community structure in directed networks. Phys Rev Lett 100(11):118,703
45.
Li K, Pang Y (2014) A unified community detection algorithm in complex network. Neurocomputing 130:36–43
46.
Linden G, Smith B, York J (2003) Amazon.com recommendations: item-to-item collaborative filtering. IEEE Internet Comput 7(1):76–80
47.
Liu N, Ds Li, Ym Zhang, Xl Li (2020) Large-scale graph processing systems: a survey. Front Inf Technol Electron Eng 21:384–404
48.
Low Y, Bickson D, Gonzalez J, Guestrin C, Kyrola A, Hellerstein JM (2012) Distributed graphlab: a framework for machine learning and data mining in the cloud. Proc VLDB Endow 5(8):716–727
49.
Ma T, Wang Y, Tang M, Cao J, Tian Y, Al-Dhelaan A, Al-Rodhaan M (2016) LED: a fast overlapping communities detection algorithm based on structural clustering. Neurocomputing 207:488–500
50.
Malewicz G, Austern MH, Bik AJ, Dehnert JC, Horn I, Leiser N, Czajkowski G (2010) Pregel: a system for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data. ACM, pp 135–146
51.
Mitrović M, Tadić B (2010) Bloggers behavior and emergent communities in blog space. Eur Phys J B 73(2):293–301MATH
52.
Mitrović M, Paltoglou G, Tadić B (2010) Networks and emotion-driven user communities at popular blogs. Eur Phys J B 77(4):597–609
53.
Murata T (2009) Detecting communities from bipartite networks based on bipartite modularities. In: 2009 International Conference on Computational Science and Engineering. IEEE, vol 4, pp 50–57
54.
55.
Papagelis M, Plexousakis D (2005) Qualitative analysis of user-based and item-based prediction algorithms for recommendation agents. Eng Appl Artif Intell 18(7):781–789
56.
Park C, Park HM, Kang U (2020) Flexgraph: flexible partitioning and storage for scalable graph mining. PLoS ONE 15(1):e0227,032
57.
Rashid AM, Karypis G, Riedl J (2005) Influence in ratings-based recommender systems: an algorithm-independent approach. In: Proceedings of the 2005 SIAM International Conference on Data Mining. SIAM, pp 556–560
58.
Resnick P, Varian HR (1997) Recommender systems. Commun ACM 40(3):56–59
59.
Sarwar B, Karypis G, Konstan J, Riedl J (2001) Item-based collaborative filtering recommendation algorithms. In: Proceedings of the 10th International Conference on World Wide Web, pp 285–295
60.
Taguchi H, Murata T, Liu X (2020) Bimlpa: community detection in bipartite networks by multi-label propagation. In: International Conference on Network Science. Springer, pp 17–31
61.
Valiant LG (1990) A bridging model for parallel computation. Commun ACM 33(8):103–111
62.
Wang G, Xie W, Demers AJ, Gehrke J (2013) Asynchronous large-scale graph processing made easy. CIDR 13:3–6
63.
Wang R, Ma X, Jiang C, Ye Y, Zhang Y (2020) Heterogeneous information network-based music recommendation system in mobile networks. Comput Commun 150:429–437
64.
Xin RS, Gonzalez JE, Franklin MJ, Stoica I (2013) Graphx: a resilient distributed graph system on spark. In: First International Workshop on Graph Data Management Experiences and Systems. ACM, p 2
65.
Xin RS, Crankshaw D, Dave A, Gonzalez JE, Franklin MJ, Stoica I (2014) Graphx: unifying data-parallel and graph-parallel analytics. arXiv preprint arXiv:​14022394
66.
Yan B, Gregory S (2012) Detecting community structure in networks using edge prediction methods. J Stat Mech Theory Exp 09:P09,008
67.
Yen TC, Larremore DB (2020) Community detection in bipartite networks with stochastic blockmodels. arXiv preprint arXiv:200111818
68.
Zaharia M, Xin RS, Wendell P, Das T, Armbrust M, Dave A, Meng X, Rosen J, Venkataraman S, Franklin MJ et al (2016) Apache spark: a unified engine for big data processing. Commun ACM 59(11):56–65
69.
Zhou S, Kannan R, Prasanna VK, Seetharaman G, Wu Q (2019) Hitgraph: high-throughput graph processing framework on FPGA. IEEE Trans Parallel Distrib Syst 30:2249–2264
Metadata
Title
Mining user–user communities for a weighted bipartite network using spark GraphFrames and Flink Gelly
Authors
T. Ramalingeswara Rao
Soumya Kanti Ghosh
Adrijit Goswami
Publication date
16-11-2020
Publisher
Springer US
Published in
The Journal of Supercomputing / Issue 6/2021
Print ISSN: 0920-8542
Electronic ISSN: 1573-0484
DOI
https://doi.org/10.1007/s11227-020-03488-4

Other articles of this Issue 6/2021

The Journal of Supercomputing 6/2021 Go to the issue

OriginalPaper

CRState: checkpoint/restart of OpenCL program for in-kernel applications

OriginalPaper

Feature clustering and feature discretization assisting gene selection for molecular classification using fuzzy c-means and expectation–maximization algorithm

OriginalPaper

Design and implementation of multiplication algorithm in quantum-dot cellular automata with energy dissipation analysis

OriginalPaper

An application of MOGW optimization for feature selection in text classification

OriginalPaper

An evolutionary game approach to IoT task offloading in fog-cloud computing

OriginalPaper

Toward a general framework for jointly processor-workload empirical modeling

Premium Partner

    Image Credits