nach oben

Erschienen in:

2021 | OriginalPaper | Buchkapitel

FLBench: A Benchmark Suite for Federated Learning

verfasst von : Yuan Liang, Yange Guo, Yanxia Gong, Chunjie Luo, Jianfeng Zhan, Yunyou Huang

Erschienen in: Intelligent Computing and Block Chain

Verlag: Springer Singapore

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Federated learning is a new machine learning paradigm. The goal is to build a machine learning model from the data sets distributed on multiple devices–so-called an isolated data island–while keeping their data secure and private. Most existing federated learning benchmarks work manually splits commonly-used public datasets into partitions to simulate real-world isolated data island scenarios. Still, this simulation fails to capture real-world isolated data island’s intrinsic characteristics. This paper presents a federated learning (FL) benchmark suite named FLBench. FLBench contains three domains: medical, financial, and AIoT. By configuring various domains, FLBench is qualified to evaluate federated learning systems and algorithms’ essential aspects, like communication, scenario transformation, privacy-preserving, data distribution heterogeneity, and cooperation strategy. Hence, it becomes a promising platform for developing novel federated learning algorithms. Currently, FLBench is open-sourced and in fast-evolution. We package it as an automated deployment tool. The benchmark suite is available from https://www.benchcouncil.org/flbench.html.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel Failure Prediction for Large-Scale Clusters Logs via Mining Frequent Patterns

Nächstes Kapitel Fake News Detection Using Knowledge Vector

Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., Shmatikov, V.: How to backdoor federated learning. In: International Conference on Artificial Intelligence and Statistics, pp. 2938–2948. PMLR (2020)

Bhagoji, A.N., Chakraborty, S., Mittal, P., Calo, S.: Analyzing federated learning through an adversarial lens. In: International Conference on Machine Learning, pp. 634–643 (2019)

Chen, M., Yang, Z., Saad, W., Yin, C., Poor, H.V., Cui, S.: A joint learning and communications framework for federated learning over wireless networks. arXiv: Networking and Internet Architecture (2019)

Coleman, C., et al.: Dawnbench: an end-to-end deep learning benchmark and competition. Training 100(101), 102 (2017)

Gao, W., et al.: Aibench scenario: Scenario-distilling AI benchmarking (2020)

Zheng, C., Zhan, J. (eds.): AIBench: towards scalable and comprehensive datacenter AI benchmarking. Bench 2018. LNCS, vol. 11459, pp. 3–9. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32813-9_1

Hao, T., et al.: Edge AIBench: towards comprehensive end-to-end edge computing benchmarking. In: Zheng, C., Zhan, J. (eds.) Bench 2018. LNCS, vol. 11459, pp. 23–30. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32813-9_3CrossRef

He, C., et al.: Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007.13518 (2020)

Hsu, T.M.H., Qi, H., Brown, M.: Federated visual classification with real-world data distribution. arXiv preprint arXiv:2003.08082 (2020)

10.

Hu, S., Li, Y., Liu, X., Li, Q., Wu, Z., He, B.: The oarf benchmark suite: Characterization and implications for federated learning systems. arXiv preprint arXiv:2006.07856 (2020)

11.

Konen, J., McMahan, H., Yu, F., Richtrik, P., Suresh, A., Bacon, D.: Federated learning: Strategies for improving communication efficiency (2016)

12.

Janapa Reddi, V., et al.: Mlperf inference benchmark, pp. 446–459 (2020). https://doi.org/10.1109/ISCA45697.2020.00045

13.

Jiang, Z., et al.: HPC AI500: a benchmark suite for HPC AI systems. In: Zheng, C., Zhan, J. (eds.) Bench 2018. LNCS, vol. 11459, pp. 10–22. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32813-9_2CrossRef

14.

Johnson, Johnson, A.E., et al.: Mimic-iii, a freely accessible critical care database. Sci. Data 3(1), 1–9 (2016)CrossRef

15.

Kairouz, P., et al.: Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 (2019)

16.

Konen, J., Mcmahan, H.B., Ramage, D., Richtrik, P.: Federated optimization: Distributed machine learning for on-device intelligence (2016)

17.

Li, Q., Wen, Z., He, B.: Practical federated gradient boosting decision trees (2020)

18.

Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: Challenges, methods, and future directions. arXiv (2019)

19.

Li, T., Sanjabi, M., Beirami, A., Smith, V.: Fair resource allocation in federated learning. arXiv preprint arXiv:1905.10497 (2019)

20.

Li, X., Huang, K., Yang, W., Wang, S., Zhang, Z.: On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189 (2019)

21.

Luo, C., et al.: AIoT Bench: towards comprehensive benchmarking mobile and embedded device intelligence. In: Zheng, C., Zhan, J. (eds.) Bench 2018. LNCS, vol. 11459, pp. 31–35. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32813-9_4CrossRef

22.

Luo, J., et al.: Real-world image datasets for federated learning. arXiv preprint arXiv:1910.11089 (2019)

23.

Mattson, P., et al.: Mlperf training benchmark (10 2019)

24.

McMahan, H., Moore, E., Ramage, D., Agera y Arcas, B.: Federated learning of deep networks using model averaging (2016)

25.

Mohri, M.: Gary Sivek. Agnostic federated learning, A.T.S. (2019)

26.

Mohri, M., Sivek, G., Suresh, A.T.: Agnostic federated learning. arXiv preprint arXiv:1902.00146 (2019)

27.

Peng, X., Huang, Z., Zhu, Y., Saenko, K.: Federated adversarial domain adaptation. arXiv preprint arXiv:1911.02054 (2019)

28.

Petersen, R.C., et al.: Alzheimer’s disease neuroimaging initiative (ADNI): clinical characterization. Neurology 74(3), 201–209 (2010)CrossRef

29.

Rothchild, D., et al.: Fetchsgd: Communication-efficient federated learning with sketching. In: International Conference on Machine Learning, pp. 8253–8265. PMLR (2020)

30.

Song, T., Tong, Y., Wei, S.: Profit allocation for federated learning, pp. 2577–2586 (12 2019). https://doi.org/10.1109/BigData47090.2019.9006327

31.

Tang, F., et al.: AIBench training: balanced industry-standard AI training benchmarking. In: 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE Computer Society (2021)

32.

Thapa, C., Chamikara, M.A.P., Camtepe, S.: Splitfed: When federated learning meets split learning. arXiv preprint arXiv:2004.12088 (2020)

33.

Toyoda, K., Zhang, A.N.: Mechanism design for an incentive-aware blockchain-enabled federated learning platform. In: International Conference on Dig Data (2019)

34.

Triastcyn, A., Faltings, B.: Federated learning with Bayesian differential privacy. arXiv: Learning (2019)

35.

Wang, G., Dang, C.X., Zhou, Z.: Measure contribution of participants in federated learning. In: International Conference on Dig Data (2019)

36.

Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., Khazaeni, Y.: Federated learning with matched averaging. arXiv preprint arXiv:2002.06440 (2020)

37.

Wang, Y., Tong, Y., Shi, D.: Federated latent Dirichlet allocation: a local differential privacy based framework. In: The 34th AAAI Conference on Artificial Intelligence, 7 February - 12 February 2020. AAAI 2020, New York City, NY, USA (2020)

38.

Xie, C., Huang, K., Chen, P.Y., Li, B.: Dba: Distributed backdoor attacks against federated learning. In: International Conference on Learning Representations (2020). https://openreview.net/forum?id=rkgyS0VFvr

39.

Yang, Q., Liu, Y., Chen, T., Tong, Y.: Federated machine learning: concept and applications. ACM TIST 10(2), 12:1–12:19 (2019)

40.

Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K., Hoang, T.N., Khazaeni, Y.: Bayesian nonparametric federated learning of neural networks. In: International Conference on Machine Learning (2019)

Titel: FLBench: A Benchmark Suite for Federated Learning
verfasst von: Yuan Liang
Yange Guo
Yanxia Gong
Chunjie Luo
Jianfeng Zhan
Yunyou Huang
Verlag: Springer Singapore
Buch: Intelligent Computing and Block Chain
Print ISBN: 978-981-16-1159-9

Electronic ISBN: 978-981-16-1160-5

Copyright-Jahr: 2021
DOI: https://doi.org/10.1007/978-981-16-1160-5_14

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner