Skip to main content

2020 | OriginalPaper | Buchkapitel

Optimization in Big Data Analysis Based on Kolmogorov-Shannon Coding Methods

verfasst von : Georgy K. Kamenev, Ivan G. Kamenev, Daria A. Andrianova

Erschienen in: Advances in Optimization and Applications

Verlag: Springer International Publishing

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

The article describes the optimization problem solving for multidimensional and bulks Big Data (data with more than 10 characteristics and \(10^8\) observations or higher), as well as machine-generated data of unlimited volume. It is difficult to analyze and visualize data of such volume and complexity using traditional methods. In contrast (and in addition) to machine learning methods widely used in Big Data analysis, it is proposed to use stochastic methods of data sets’ coding and approximation using Kolmogorov-Shannon metric nets, which are optimal for the entropy of the code. While adapting these methods, new methods are proposed for metrics construction for characteristics with nominal and ordinal scales.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Lotov, A.V., Bushenkov, V.A., Kamenev, G.K.: Feasible Goals Method: Mathematical Foundations and Environmental Applications. The Edwin Mellen Press, Lewiston (1999) Lotov, A.V., Bushenkov, V.A., Kamenev, G.K.: Feasible Goals Method: Mathematical Foundations and Environmental Applications. The Edwin Mellen Press, Lewiston (1999)
2.
Zurück zum Zitat Lotov, A.V., Bushenkov, V.A., Kamenev, G.K.: Interactive Decision Maps. Approximation and Visualization of Pareto Frontier. Applied Optimization, vol. 89. Kluwer Academic Publishers, Boston/Dordrecht/New York/London (2004) Lotov, A.V., Bushenkov, V.A., Kamenev, G.K.: Interactive Decision Maps. Approximation and Visualization of Pareto Frontier. Applied Optimization, vol. 89. Kluwer Academic Publishers, Boston/Dordrecht/New York/London (2004)
3.
Zurück zum Zitat Shannon, C.: The mathematical theory of communication. Bell Syst. Tech. J. 27–28, 379–423, 623–656 (1948) Shannon, C.: The mathematical theory of communication. Bell Syst. Tech. J. 27–28, 379–423, 623–656 (1948)
4.
Zurück zum Zitat Kolmogorov, A.N., Tikhomirov, V.M.: Epsilon-entropy and Epsilon-capacity of sets in functional spaces. Adv. Math. Sci. 25(2), 3–86 (1959)MATH Kolmogorov, A.N., Tikhomirov, V.M.: Epsilon-entropy and Epsilon-capacity of sets in functional spaces. Adv. Math. Sci. 25(2), 3–86 (1959)MATH
5.
Zurück zum Zitat Kamenev, G.K.: Approximation of completely bounded sets by the deep holes method. Comput. Math. Math. Phys. 41(11), 1667–1675 (2001)MathSciNetMATH Kamenev, G.K.: Approximation of completely bounded sets by the deep holes method. Comput. Math. Math. Phys. 41(11), 1667–1675 (2001)MathSciNetMATH
8.
Zurück zum Zitat Davenport, T.H., Barth, P., Bean, R.: How big data is different. MIT Sloan Manag. Rev. 54(1), 1–5 (2012) Davenport, T.H., Barth, P., Bean, R.: How big data is different. MIT Sloan Manag. Rev. 54(1), 1–5 (2012)
13.
Zurück zum Zitat Kamenev, G.K., Kamenev, I.G.: Metric analysis of multidimensional sociological samples [Metricheskij analiz mnogomernyh sociologicheskih vyborok] In: Pospelov, I.G., et al. (eds). Proceedings of the conference “Modeling of Coevolution of Nature and Society: Problems and Experience. To the 100-th Anniversary from the Birthday of Academician N.N. Moiseev", pp. 198–209. FRC CSC of RAS, Moscow (2017) Kamenev, G.K., Kamenev, I.G.: Metric analysis of multidimensional sociological samples [Metricheskij analiz mnogomernyh sociologicheskih vyborok] In: Pospelov, I.G., et al. (eds). Proceedings of the conference “Modeling of Coevolution of Nature and Society: Problems and Experience. To the 100-th Anniversary from the Birthday of Academician N.N. Moiseev", pp. 198–209. FRC CSC of RAS, Moscow (2017)
14.
Zurück zum Zitat Kamenev, I.G., Andrianova, D.A.: Mathematical and statistical methods of pre-processing and exploratory analysis of Big social Data on the example of payments stream analysis. In: Proceedings of the 62nd all-Russian Scientific Conference of MIPT, Applied Mathematics and Computer Science, pp. 40–42. MIPT, Moscow (2019) Kamenev, I.G., Andrianova, D.A.: Mathematical and statistical methods of pre-processing and exploratory analysis of Big social Data on the example of payments stream analysis. In: Proceedings of the 62nd all-Russian Scientific Conference of MIPT, Applied Mathematics and Computer Science, pp. 40–42. MIPT, Moscow (2019)
15.
Zurück zum Zitat Andrianova, D.A., Kamenev, I.G.: The method of metric data analysis in big data in transport streams research. In: Proceedings of the IX Moscow International Conference of Operations Research, ORM 2018, vol. 2, pp. 506–510 (2018) Andrianova, D.A., Kamenev, I.G.: The method of metric data analysis in big data in transport streams research. In: Proceedings of the IX Moscow International Conference of Operations Research, ORM 2018, vol. 2, pp. 506–510 (2018)
17.
Zurück zum Zitat Berezkin, V.E., Kamenev, G.K., Lotov, A.V.: Program for visualization of multidimensional Pareto-frontier in non-convex multi-criteria optimization problems (PFV-II) (2019) Berezkin, V.E., Kamenev, G.K., Lotov, A.V.: Program for visualization of multidimensional Pareto-frontier in non-convex multi-criteria optimization problems (PFV-II) (2019)
Metadaten
Titel
Optimization in Big Data Analysis Based on Kolmogorov-Shannon Coding Methods
verfasst von
Georgy K. Kamenev
Ivan G. Kamenev
Daria A. Andrianova
Copyright-Jahr
2020
DOI
https://doi.org/10.1007/978-3-030-65739-0_13