Top

Journal of Network and Systems Management

Published in:

03-02-2017

A Modular Traffic Sampling Architecture: Bringing Versatility and Efficiency to Massive Traffic Analysis

Authors: João Marco C. Silva, Paulo Carvalho, Solange Rito Lima

Published in: Journal of Network and Systems Management | Issue 3/2017

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The massive traffic volumes and heterogeneity of services in today’s networks urge for flexible, yet simple measurement solutions to assist network management tasks, without impairing network performance. To turn treatable tasks requiring traffic analysis, sampling the traffic has become mandatory, triggering substantial research in the area. Despite that, there is still a lack of an encompassing solution able to support the flexible deployment of sampling techniques in production networks, adequate to diverse traffic scenarios and measurement activities. In this context, this article proposes a modular traffic sampling architecture able to foster the flexible design and deployment of efficient measurement strategies. The architecture is composed of three layers—management plane, control plane and data plane—covering key components to achieve versatile and lightweight measurements in diverse traffic scenarios and measurement activities. Each component of the architecture is described considering the different strategies, technologies and protocols that compose the several stages of a measurement process. Following the proposed architecture, a sampling framework prototype has been developed, providing a fair environment to assess and compare sampling techniques under distinct measurement scenarios, evaluating their performance in balancing computational burden and accuracy. The results have demonstrated the relevance and applicability of the proposed architecture, revealing that a modular and configurable approach to sampling is a step forward for improving sampling scope and efficiency.

previous article Decreasing the Management Burden in Multi-tier Systems Through Partial Correlation-Based Monitoring

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

The framework is available for download at http://1drv.ms/1IggkCa as a Raspbian image ready to be deployed.

Note that the evaluation of flow classification methodologies and tools is beyond the scope of this work, which resorts to a port-based classification technique for distinguishing flows.

Zseby, T., Molina, M., Duffield, N.: Sampling and Filtering Techniques for IP Packet Selection RFC 5475. Technical report, IETF. http://datatracker.ietf.org/doc/rfc5475/ (2009)

Silva, J.M.C., Carvalho, P., Rito Lima, S.: Analysing traffic flows through sampling: a comparative study. In: 20th IEEE Symposium on Computers and Communication (ISCC), Cyprus (2015)

Jadwab, J., Phall, P., Pinna, B.: Traffic estimation for the largest sources on a network using packet sampling with limited storage. Technical report, Hewllet-Packard Laboratories, Bristol (1992)

Claffy, K.C., Polyzos, G.C., Braun, H.W.: Application of sampling methodologies to network traffic characterization, SIGCOMM. Comput. Commun. Rev. 23(4), 194–203 (1993). doi:10.1145/167954.166256 CrossRef

Cozzani, I., Giordano, S.: Traffic sampling methods for end-to-end QoS evaluation in large heterogeneous networks. Comput. Netw. ISDN Syst. 30(16–18), 1697–1706. http://www.sciencedirect.com/science/article/pii/S0169755298001986 (1998)

Amer, P., Cassel, L.: Management of sampled real-time network measurements. In: Proceedings of 14th Conference on Local Computer Networks. IEEE Comput. Soc. Press, pp. 62–68. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=65244 (1989)

Tammaro, D., Valenti, S., Rossi, D., Pescapé, A.: Exploiting packet-sampling measurements for traffic characterization and classification. Int. J. Netw. Manag. 22(6), 451–476 (2012). doi:10.1002/nem.1802 CrossRef

Duffield, N.: Fair sampling across network flow measurements. ACM SIGMETRICS Perform. Eval. Rev. 40(1), 367 (2012). doi:http://dl.acm.org/citation.cfm?id=2318857.2254800

Hernandez, E.A., Chidester, M.C., George, A.D.: Adaptive sampling for network management. J. Netw. Syst. Manag. 9(4), 409–434 (2001). doi:10.1023/A:1012980307500 CrossRef

10.

Silva, J.M.C., Carvalho, P., Rito Lima, S.: A multiadaptive sampling technique for cost-effective network measurements. Comput. Netw. 57(17), 3357–3369 (2013). doi:10.1016/j.comnet.2013.07.023

11.

Duffield, N.G., Grossglauser, M.: Trajectory sampling for direct traffic observation. ACM SIGCOMM Comput. Commun. Rev. 30(4), 271–282 (2000). doi:10.1145/347057.347555 CrossRef

12.

Estan, C., Varghese, G.: New directions in traffic measurement and accounting. SIGCOMM Comput. Commun. Rev. 32(4), 323–336 (2002). doi:10.1145/964725.633056 CrossRef

13.

Singh, R., Kumar, H., Singla, R.K.: Analyzing statistical effect of sampling on network traffic dataset. In: Satapathy, S.C., Avadhani, P.S., Udgata, S.K., Lakshminarayana, S. (eds.). ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India. Springer International Publishing, pp. 401–408. http://link.springer.com/chapter/10.1007/978-3-319-03107-1_43 (2014)

14.

Yang, L., Michailidis, G.: Sampled based estimation of network traffic flow characteristics. In: IEEE INFOCOM 2007---26th IEEE International Conference on Computer Communications, (IEEE) pp. 1775–1783. doi:10.1109/INFCOM.2007.207. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4215789 (2007)

15.

Carela-Español, V., Barlet-Ros, P., Cabellos-Aparicio, A., Solé-Pareta, J.: Analysis of the impact of sampling on NetFlow traffic classification. Comput. Netw. 55(5), 1083–1089 (2011). doi:10.1016/j.comnet.2010.11.002 CrossRef

16.

Lin, R., Li, O., Li, Q., Dai, K.: Exploiting adaptive packet-sampling measurements for multimedia traffic classification. J. Commun. 9(12) (2014). http://www.jocm.us/uploadfile/2014/1231/20141231030404520

17.

Kandula, S., Mahajan, R.: Sampling biases in network path measurements and what to do about it. In: Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement Conference IMC ’09 (ACM, New York, NY, USA) , pp. 156–169. doi:10.1145/1644893.1644912 (2009)

18.

Lee, M., Duffield, N., Kompella, R.: Two samples are enough: opportunistic flow-level latency estimation using NetFlow. In: 2010 Proceedings IEEE INFOCOM, pp. 1–9. doi:10.1109/INFCOM.2010.5462044. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5462044 (2010)

19.

Mahmood, A.N., Hu, J., Tari, Z., Leckie, C.: Critical infrastructure protection: resource efficient sampling to improve detection of less frequent patterns in network traffic. J. Netw. Comput. Appl. 33(4), 491–502 (2010). http://www.sciencedirect.com/science/article/B6WKB-4YBMFB6-1/2/9b91d8daa2364e0d025aed6088160da7

20.

Zhang, J., Luo, X., Perdisci, R., Gu, G., Lee, W., Feamster, N.: Boosting the scalability of botnet detection using adaptive traffic sampling. In: Proceedings of the 6th ACM Symposium on Information, Computer and Communications Security, Ser. (ACM, New York, NY, USA), ASIACCS ’11, pp. 124–134. doi:10.1145/1966913.1966930 (2011)

21.

Huang, Y., Pullen, J.: Countering denial-of-service attacks using congestion triggered packet sampling and filtering. In: Proceedings Tenth International Conference on Computer Communications and Networks (Cat. No.01EX495), (IEEE), pp. 490–494 (2001). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=956309

22.

Brauckhoff, D., Tellenbach, B., Wagner, A., May, M., Lakhina, A.: Impact of packet sampling on anomaly detection metrics. In: Proceedings of the 6th ACM SIGCOMM Conference on Internet Measurement, Ser. (ACM, New York, NY, USA) IMC ’06, pp. 159–164. doi:10.1145/1177080.1177101 (2006)

23.

Paredes-Oliva, I., Barlet-Ros, P., Solé-Pareta, J.: Portscan detection with sampled Netflow. In: Traffic Monitoring and Analysis (Springer), pp. 26–33. http://link.springer.com/chapter/10.1007/978-3-642-01645-5_4 (2009)

24.

Mai, J., Chuah, C.N., Sridharan, A., Ye, T., Zang, H.: Is sampled data sufficient for anomaly detection? In: Proceedings of the 6th ACM SIGCOMM on Internet measurement—IMC’06, Ser. (ACM Press, New York, NY, USA) p. 165 (2006). http://portal.acm.org/citation.cfm?doid=1177080.1177102

25.

Jae-Hyun, J., Cheol-Woong, A., Dongjoon, L., Sung-Ho, K.: DDoS attack detection using flow entropy and packet sampling on huge networks. In: ICN 2014 : The Thirteenth International Conference on Networks (IARIA), pp. 183–190 (2014)

26.

Zseby, T.: Deployment of sampling methods for SLA validation with non-intrusive measurements. In: Proceedings of Passive and Active Measurements Conference (Fort Collins) (2002)

27.

Zseby, T.: Comparison of sampling methods for non-intrusive SLA validation. In: Proceedings of the Second Workshop on End-to-End Monitoring Techniques and Services (E2EMon) (2004)

28.

Serral-Gracia, R., Cabellos-Aparicio, A., Domingo-Pascual, J.: Packet loss estimation using distributed adaptive sampling. In: Network Operations and Management Symposium Workshops, 2008. NOMS Workshops 2008. IEEE (IEEE), pp. 124–131 (2008). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4509938

29.

Sommers, J., Barford, P., Duffield, N., Ron, A.: Improving accuracy in end-to-end packet loss measurement. In: Proceedings of the 2005 conference on Applications, Technologies, Architectures, and Protocols for Computer Communications—SIGCOMM ’05, (ACM Press, New York, New York, USA), vol. 35, p. 157 (2005). http://dl.acm.org/citation.cfm?id=1080091.1080111

30.

Dogman, A., Saatchi, R., Al-Khayatt, S.: An adaptive statistical sampling technique for computer network traffic. In: 7th International Symposium on Communication Systems Networks and Digital Signal Processing (CSNDSP, 2010), pp. 479–483 (2010)

31.

Gu, Y., Breslau, L., Duffield, N., Sen, S.: On passive one-way loss measurements using sampled flow statistics. In: IEEE INFOCOM 2009—The 28th Conference on Computer Communications (IEEE), pp. 2946–2950 (2009). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5062264

32.

Androulidakis, G., Chatzigiannakis, V., Papavassiliou, S.: Network anomaly detection and classification via opportunistic sampling. IEEE Netw. 23(1), 6–12 (2009). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4804318

33.

Choi, B.Y., Bhattacharyya, S.: Observations on cisco sampled netflow. ACM SIGMETRICS Perform. Eval. Rev. 33(3), p. 18 (2005). http://portal.acm.org/citation.cfm?doid=1111572.1111579

34.

Zseby, T., Hirsch, T., Claise, B.: Packet sampling for flow accounting: challenges and limitations. In: Claypool, M., Uhlig, S. (eds.) Passive and Active Network Measurement, Ser. Lecture Notes in Computer Science, vol. 4979, (Springer Berlin / Heidelberg), pp. 61–71 (2008). doi:10.1007/978-3-540-79232-1_7

35.

Pescape, A., Rossi, D., Tammaro, D., Valenti, S.: On the impact of sampling on traffic monitoring and analysis. In: 2010 22nd International Teletraffic Congress (lTC 22) (IEEE), pp. 1–8. (2010). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5608718

36.

Chabchoub, Y., Fricker, C., Guillemin, F., Robert, P.: Deterministic versus probabilistic packet sampling in the internet. In: Mason, L., Drwiega, T., Yan, J. (eds.) Managing Traffic Performance in Converged Networks, Lecture Notes in Computer Science, vol. 4516, Springer, Berlin, Heidelberg, pp. 678–689 (2007). http://link.springer.com/chapter/10.1007/978-3-540-72990-7_60

37.

Castro, V., Carvalho, P., Lima, S.R., In: A cooperative network monitoring overlay. Smart Spaces and Next Generation Wired/Wireless Networking, Springer, pp. 475–486 (2011). http://link.springer.com/chapter/10.1007/978-3-642-22875-9_43

38.

Schad, J., Dittrich, J., Quiané-Ruiz, J.A.: Runtime measurements in the cloud: observing, analyzing, and reducing variance. Proc. VLDB Endow. 3(1–2), 460–471 (2010). doi:10.14778/1920841.1920902 CrossRef

39.

Pras, A., Schoenwaelder, J.: On the difference between information models and data models—RFC 3444. Technical Report, IETF (2003). https://datatracker.ietf.org/doc/rfc3444/

40.

Claise, B., Trammell, B.: Specification of the IP Flow Information eXport (IPFIX) Protocol for the Exchange of Flow Information. RFC 7011 (2013). http://datatracker.ietf.org/doc/draft-ietf-ipfix-protocol-rfc5101bis/

41.

Claise, B., Trammel, B.: Information Model for IP Flow Information Export (IPFIX)—RFC 7012. Technical Report IETF (2013). https://datatracker.ietf.org/doc/rfc7012/

42.

Dietz, T., Claise, B., Aitken, P., Dressler, F., Carle, G.: Information Model for Packet Sampling Exports. Technical Report, IETF RFC 5477 (2009). https://datatracker.ietf.org/doc/rfc5477/

43.

IP Flow Information Export (IPFIX).: Entities (2015).http://www.iana.org/assignments/ipfix/ipfix.xhtml

44.

Dietz, T., Claise, B., Quittek. J.: Definitions of Managed Objects for Packet Sampling. RFC 6727 (2012). http://datatracker.ietf.org/doc/rfc6727/

45.

Case, J., Mundy, R., Partain, D., Stewart, B.: Introduction and Applicability Statements for Internet-Standard Management Framework—RFC 3410. Technical Report, IETF (2002). https://datatracker.ietf.org/doc/rfc3410/

46.

Aitken, P., Claise, B., McDowall, C., Schoenwaelder, J.: Exporting MIB Variables using the IPFIX Protocol draft-ietf-ipfix-mib-variable-export-09. Technical Report, IETF (2015). https://datatracker.ietf.org/doc/draft-ietf-ipfix-mib-variable-export/

47.

McCloghrie, K., Seligson, J., Reichmeyer, F., Smith, A., Sahita, R.: Structure of policy provisioning information (SPPI)—RFC 3159. Technical Report, IETF (2001). https://datatracker.ietf.org/doc/rfc3159/

48.

Uslar, M., Specht, M., Rohjans, S., Trefke, J., González, J.M.: The Common Information Model CIM: IEC 61968/61970 and 62325—A Practical Introduction to the CIM, vol. 66. Springer, New York (2012)

49.

Silva, J.M.C., Carvalho, P., Rito Lima, S.: Enhancing traffic sampling scope and efficiency. In: 2013 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) (IEEE), pp. 71–72 (2013). http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=6562848

50.

Hofstede, R., Celeda, P., Trammell, B., Drago, I., Sadre, R., Sperotto, A., Pras, A.: Flow monitoring explained: from packet capture to data analysis With NetFlow and IPFIX. IEEE Commun. Surv. Tutor. 16(4), 2037–206 (2014). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6814316

51.

Claise, B., Johnson, A., Quittek. J.: Packet Sampling (PSAMP) Protocol Specifications. RFC 5476 (2009). http://datatracker.ietf.org/doc/rfc5476/

52.

Orebaugh, A., Ramirez, G., Beale, J.: Wireshark and Ethereal Network Protocol Analyzer Toolkit. Syngress, Rockland (2006)

53.

Jacobson, V., McCanne, S.: Lawrence Berkeley Laboratory, Berkeley, CA (2009)

54.

Alcock, S., Lorier, P., Nelson, R.: ACM SIGCOMM Comput. Commun. Rev. 42(2), 42 (2012). http://dl.acm.org/citation.cfm?doid=2185376.2185382

55.

Silva, J.M.C., Carvalho, P., Lima, S.R.: Computational weight of network traffic sampling techniques. In: 2014 IEEE Symposium on Computers and Communications (ISCC) (IEEE, Madeira, Portugal), pp. 1–6 (2014). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6912467

56.

Shannon, C., Aben, E., Claffy, K., Andersen, D., Brownlee, N.: The CAIDA UCSD Anonymized Internet Traces 2008—equinix-chicago.dirA.20080430-170200.UTC.anon. Downloaded from http://www.caida.org/data/passive/passive_2008_dataset.xml (2008)

57.

Shannon, C., Aben, E., Claffy, K., Andersen, D., Brownlee, N.: The CAIDA UCSD Anonymized Internet Traces 2014—-equinix-chicago.dirA.20140619-131100.UTC.anon. Downloaded from http://www.caida.org/data/passive/passive_2014_dataset.xml (2014)

58.

Krishnan, R., Yong, L., Ghanwani, A., So, N., Khasnabish, B.: Mechanisms for Optimizing Link Aggregation Group (LAG) and Equal-Cost Multipath (ECMP) Component Link Utilization in Networks—RFC 7424. Technical Report, IETF (2015). https://datatracker.ietf.org/doc/rfc7424/

59.

Silverman, B.W.: Density Estimation for Statistics and Data Analysis, vol. 26. CRC Press, Boca Raton (1986)CrossRefMATH

Title: A Modular Traffic Sampling Architecture: Bringing Versatility and Efficiency to Massive Traffic Analysis
Authors: João Marco C. Silva
Paulo Carvalho
Solange Rito Lima
Publication date: 03-02-2017
Publisher: Springer US
Published in: Journal of Network and Systems Management / Issue 3/2017
Print ISSN: 1064-7570
Electronic ISSN: 1573-7705
DOI: https://doi.org/10.1007/s10922-017-9404-5

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 3/2017

Automatic Hidden Bypasses in Software-Defined Networks

CRVR: Connectivity Repairing in Wireless Sensor Networks with Void Regions

On the Interplay of Network Structure and Routing Strategies on Network Design Methods for Mitigation of Intentional Attacks in Scale-Free Networks

Decreasing the Management Burden in Multi-tier Systems Through Partial Correlation-Based Monitoring

Model-Based Probabilistic Reasoning for Self-Diagnosis of Telecommunication Networks: Application to a GPON-FTTH Access Network

Survivable Network Capacity Allocation and Topology Design Using Multi-period Network Augmentation

Premium Partner