Report of the Third Workshop on the Usage of NetFlow/IPFIX in Network Management

Paul Aitken and Benoit Claise (Cisco) started their presentation discussing current issues related to NetFlow/IPFIX adoption. As new applications for NetFlow/IPFIX emerge, as well as new functions are standardized, the complexity of the protocol increases. They listed three possible problems for the deployment of NetFlow/IPFIX:

Regarding the future of NetFlow/IPFIX, Aitken and Claise listed some new flow-based applications that are already emerging both in IPFIX research and at the IETF:

These new applications would require further development in the IETF Working Group to redefine the IPFIX applicability (RFC 5472) and to define the mediation protocol. They expect that some of those functions will be supported by equipment in the market soon (e.g. by Cisco Flexible NetFlow).

One of the new flow-based applications that Aitken and Claise cited was further discussed in two other presentations in the workshop. A first requirement for application visibility is the ability to recognize the traffic (flows) created by a protocol. Pavel Piskac (Masaryk University/INVEA-TECH) is studying the use of time characteristics in NetFlow data for improving protocol detection. Although the need for protocol detection is widely accepted and several methods for that have already been proposed, time characteristics are not commonly employed. The main reason for that is the lack of support for precise time exporting in metering equipment (e.g. with microsecond resolution).

Piskac implemented detection methods to classify flow data, and used the SSHv2 protocol as a case study. In general, he was not able to classify interactive SSH traffic, because interactive SSH sessions generate traces similar to HTTP and IMAP, for example. However, using only time characteristics, dictionary attacks against SSH servers could be identified. He expects to improve his method when more precise time characteristics become available.

Nikolay Melnikov (Jacobs University Bremen) presented his progresses on identifying users based on their network activity (represented by NetFlow records). For that purpose, he evaluated if it is possible to define and to identify signatures for some of the most popular applications. Some applications generate typical traffic, even when users are not interacting with them. For example, the latest version of Google Chrome contacts a Google server immediately after being opened in order to download an updated list of phishing and malware web sites. From that behavior, Google Chrome could be identified in network traces. Assuming that other applications also have a typical behavior, and that users in a network use a distinctive set of applications, the identification of users would be possible.

Melnikov evaluated 13 applications, grouped into 4 categories: web browsers, instant messaging clients, mail clients and media players. Whenever possible, Linux and Windows versions were considered. For each application, he defined a flow-based signature using a special query language called Flowy [4]. In his first experimental evaluation, he was able to identify 10 voluntary users using 5 applications, with only 2 application mismatches. Melnikov is planning to continue his research creating signatures for more applications, doing experiments in uncontrolled environments (real traffic), and extending his analysis to also consider the date/time in which each application is used.

A consequence of IPFIX application visibility is that flow-based application monitoring becomes possible. Shingo Kashima (NTT) demonstrated his solution for IPTV monitoring. Multicast is finally in use in some provider networks, especially because of IPTV. Current monitoring tools are not appropriate for large-scale multicast monitoring, either because they are not focused on real traffic (as in the case of multicast ping and traceroute), or because they are not scalable (as in the case of full traffic capture). According to Kashima, a solution for IPTV monitoring has to fulfill the following 4 requirements: (1) It must be able to detect service deterioration (like loss and delay variation); (2) it must identify users and channels affected by a failure; (3) it must identify the location of a failure; and (4) it must be ready for immediate deployment. Kashima used IPFIX/PSAMP for IPTV traffic monitoring. His solution (called QCast) uses Property Match Filtering and Systematic Time-Based Sampling, both from PSAMP, to collect the input packets, and IPFIX enterprise-specific information elements to carry data from RTP headers. The traffic received by IPFIX collectors is processed and alerts are triggered to operators. Kashima demonstrated his software during his presentation. The tool, however, is not available for public download.

Jochen Kögel (University of Stuttgart) is researching methods to extract network characteristics (like one-way delay, round trip time and packet loss) from the correlation among flows representing the same communication, but collected in different metering points. He identified several accuracy problems in the NetFlow exporters. Kögel collected data for his experiments in a real network connecting two locations through five routers—three of them under control of his institution. Kögel noted problems both in the precision of flow data and in the consistency of data collected in different metering points. Loss and duplication of information (NetFlow records), clock inaccuracy and imprecision in the packet/byte counters were the main issues. For example, in one of his experiments, he showed that the counters of different routers were precisely recording the number of packets, but not the number of bytes. Some deeper analyzes revealed that one router was rounding the byte counter.

From those experiments, he concluded that each router behaves in a peculiar manner, and that corrections are needed for better results. When routers with more accurate measurement capabilities are available, the characteristics of the network can be estimated. In general, a profile describing the artificial effects introduced by the exporter is necessary. Although such a profile could be provided by the manufacturer, a calibration step would be more realistic. The inclusion of profile information into IPFIX records was suggested as a measure to correct bias on the monitoring system.

Two presentations covered the use of NetFlow/IPFIX in network security. Anna Sperotto (University of Twente) opened this session presenting her experience in creating a labeled data set for flow-based intrusion detection. Although several flow-based intrusion detection approaches have already been proposed in the literature, most of them were validated using proprietary traces, which makes it difficult to compare different approaches. Sperotto created the first public data set of NetFlow records with ground truth, from traces collected at a honeypot at the University of Twente. The log files of the server hosting the honeypot were used to trigger alerts, which were later correlated to flow traces. Besides showing details of this data set, Sperotto also described the steps for creating it, which involved several non-automatic activities to label the flows. The data set has been anonymized using Nfdump, and is publicly available at [2].

Pavel Celeda (Masaryk University) is researching methods for malware detection based on the analysis of the network behavior. According to Celeda, as traffic acquisition and storage becomes easier, security analysis becomes more important and challenging. Neflow-based approaches for malware detection are promising because they are scalable and suitable for early detection of malwares and vulnerabilities (zero day exploits). The Chuck Norris Botnet, discovered at Masaryk University on 2 December 2009, was used as an example. The Chuck Norris Botnet is a Linux malware programmed to propagate itself in ADSL modems and routers with weak passwords. The infection occurs through brute force password discovery (dictionary attack) on the TELNET server of ADSL equipment. As soon as the attacker discovers the password, a malware is downloaded from a command host. The Botnet was used for DDoS attacks and DNS spoofing, aiming the infection of hosts using the infected ADLS devices. The Botnet was estimated to have infected around 33,000 distinct hosts from October 2009 to February 2010—most of them in South America. The activity of this Botnet was interrupted on February 2010.

In the last section, three tools that can help in NetFlow/IPFIX research were presented. Brian Trammell (ETH Zurich) presented his ripfix: an environment for fast prototyping and debugging of IPFIX. The tool is optimized for development, and therefore ideal for situations where running code is needed to speed up the solution of issues and the development of new ideas (like in the IETF IPFIX Working Group). The tool, developed in Ruby, supports most of what is specified in RFC 5101, RFC 5103 and RFC 5655, besides some other functions specified on draft standards. In the current version, TCP is used as transport, but SCTP support will be included in the future. Several other functionalities specified in current IETF drafts, like anonymization of IPFIX records and structured data, are planned to be included in the tool soon. The tool is open source and can be downloaded from [8].

Stefan Burschka (Swisscom/CSIT) presented the Tranalyzer and Traviz tools. Tranalyzer is an open source tool designed to help with network analysis. Traviz is a graphical interface for Tranalyzer. Besides extracting basic NetFlow information from raw network data, Tranalyzer provides several methods for both flow and packet analysis, like TCP state machine reconstruction, malicious packet detection, and advanced statistics extraction. Burschka presented some scenarios where Tranalyzer could be employed as, for example, the detection of routing anomalies, and the detection of applications by automatic reverse engineering their state machines. Tranalyzer also has simple interfaces with several other tools, like Matlab, GnuPlot and Excel. Tranalyzer is available for downloading at [6].

Lucjan Janowski (AGH Poland) closed this session presenting his trace2netFlow tool. Although IPFIX has been in development for several years, there are few tools for converting package traces into IPFIX records. The trace2netFlow aims to fill part of this gap, providing easy and flexible exporting capabilities. Janowski presented in details how the tool can be used to convert binary network data into IPFIX records, with configurable output fields and sampling techniques. The result data set can be directly exported into MATLAB, for example. The Trace2netFlow is available for downloading at [1].

The Workshop on the Usage of NetFlow/IPFIX in Network Management has been considered a valuable place to discuss new ideas related to NetFlow/IPFIX. The third edition was remarkable for the interaction between people from the NMRG and the IETF IPFIX Working Group. The minutes and all presentations are available at the website of the 78th IETF [7]. A next edition of the Workshop is planned for the second half of 2011.