Top

previous chapter Finite Games with Perturbed Payoffs

next chapter Construction of a Technologically Feasible Cutt...

Hint

Swipe to navigate through the chapters of this book

2020 | OriginalPaper | Chapter

Optimization in Big Data Analysis Based on Kolmogorov-Shannon Coding Methods

Read first chapter

Authors: Georgy K. Kamenev, Ivan G. Kamenev, Daria A. Andrianova

Publisher: Springer International Publishing

Published in: Advances in Optimization and Applications

» Get access to the full-text

Abstract

The article describes the optimization problem solving for multidimensional and bulks Big Data (data with more than 10 characteristics and \(10^8\) observations or higher), as well as machine-generated data of unlimited volume. It is difficult to analyze and visualize data of such volume and complexity using traditional methods. In contrast (and in addition) to machine learning methods widely used in Big Data analysis, it is proposed to use stochastic methods of data sets’ coding and approximation using Kolmogorov-Shannon metric nets, which are optimal for the entropy of the code. While adapting these methods, new methods are proposed for metrics construction for characteristics with nominal and ordinal scales.

Please log in to get access to this content

To get access to this content you need the following product:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 69.000 Bücher
über 500 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Umwelt
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Testen Sie jetzt 30 Tage kostenlos.

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 50.000 Bücher
über 380 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Umwelt
Maschinenbau + Werkstoffe

Testen Sie jetzt 30 Tage kostenlos.

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 58.000 Bücher
über 300 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Testen Sie jetzt 30 Tage kostenlos.

inform now

Lotov, A.V., Bushenkov, V.A., Kamenev, G.K.: Feasible Goals Method: Mathematical Foundations and Environmental Applications. The Edwin Mellen Press, Lewiston (1999)

Lotov, A.V., Bushenkov, V.A., Kamenev, G.K.: Interactive Decision Maps. Approximation and Visualization of Pareto Frontier. Applied Optimization, vol. 89. Kluwer Academic Publishers, Boston/Dordrecht/New York/London (2004)

Shannon, C.: The mathematical theory of communication. Bell Syst. Tech. J. 27–28, 379–423, 623–656 (1948)

Kolmogorov, A.N., Tikhomirov, V.M.: Epsilon-entropy and Epsilon-capacity of sets in functional spaces. Adv. Math. Sci. 25(2), 3–86 (1959) MATH

Kamenev, G.K.: Approximation of completely bounded sets by the deep holes method. Comput. Math. Math. Phys. 41(11), 1667–1675 (2001) MathSciNetMATH

Leetaru, K. How Do We Define Big Data And Just What Counts As A Big Data Analysis? https://www.forbes.com/sites/kalevleetaru/2019/01/09/how-do-we-define-big-data-and-just-what-counts-as-a-big-data-analysis/#13d886081b66

Ward, J.S., Barker, A.: Undefined by data: a survey of big data definitions. arXiv preprint arXiv:1309.5821 (2013)

Davenport, T.H., Barth, P., Bean, R.: How big data is different. MIT Sloan Manag. Rev. 54(1), 1–5 (2012)

Kamenev, G.K., Kamenev, I.G.: Multidimensional statistical sets and their metric analysis. Proc. Inst. Syst. Anal. Russ. Acad. Sci. CC FRC CSC RAS 68(2), 30–33 (2018). https://doi.org/10.14357/20790279180207

10.

Bolgov, M.V., Buber, A.L., Lotov, A.V.: Support for making strategic decisions on the water supply of the lower Volga River based on the Pareto frontier visualization. Sci. Tech. Inf. Process. 45(5), 297–306 (2018). https://doi.org/10.3103/S0147688218050027 CrossRef

11.

Kamenev, G.K., Sarancha, D.A., Polyanovsky, V.O.: Investigation of the class of one-dimensional unimodal mappings obtained in the modeling of the lemming population. Biophysics 63(4), 596–610 (2018). https://doi.org/10.1134/S0006350918040097 CrossRef

12.

Kamenev, G.K., Kamenev, I.G.: Discrete-dynamic modeling of governance for human capital. Matem. Mod. 32(6), 81–96 (2020). https://doi.org/10.20948/mm-2020-06-06

13.

Kamenev, G.K., Kamenev, I.G.: Metric analysis of multidimensional sociological samples [Metricheskij analiz mnogomernyh sociologicheskih vyborok] In: Pospelov, I.G., et al. (eds). Proceedings of the conference “Modeling of Coevolution of Nature and Society: Problems and Experience. To the 100-th Anniversary from the Birthday of Academician N.N. Moiseev", pp. 198–209. FRC CSC of RAS, Moscow (2017)

14.

Kamenev, I.G., Andrianova, D.A.: Mathematical and statistical methods of pre-processing and exploratory analysis of Big social Data on the example of payments stream analysis. In: Proceedings of the 62nd all-Russian Scientific Conference of MIPT, Applied Mathematics and Computer Science, pp. 40–42. MIPT, Moscow (2019)

15.

Andrianova, D.A., Kamenev, I.G.: The method of metric data analysis in big data in transport streams research. In: Proceedings of the IX Moscow International Conference of Operations Research, ORM 2018, vol. 2, pp. 506–510 (2018)

16.

Python 3.8.3 documentation: random - generate pseudo-random numbers. https://docs.python.org/3/library/random.html

17.

Berezkin, V.E., Kamenev, G.K., Lotov, A.V.: Program for visualization of multidimensional Pareto-frontier in non-convex multi-criteria optimization problems (PFV-II) (2019)

Title

Optimization in Big Data Analysis Based on Kolmogorov-Shannon Coding Methods

Book

Advances in Optimization and Applications

Print ISBN: 978-3-030-65738-3

Electronic ISBN: 978-3-030-65739-0

https://doi.org/10.1007/978-3-030-65739-0

DOI

https://doi.org/10.1007/978-3-030-65739-0_13

Authors:

Georgy K. Kamenev
Ivan G. Kamenev
Daria A. Andrianova

Publisher

Springer International Publishing

Sequence number

Premium Partner

Image Credits