Ling Huang
Minos Garofalakis
Joseph Hellerstein
ABSTRACT
Recent research has proposed efficient protocols for distributed triggers, which can be used in monitoring infrastructures to maintain system-wide invariants and detect abnormal events with minimal communication overhead. To date, however, this work has been limited to simple thresholds on distributed aggregate functions like sums and counts. In this paper, we present our initial results that show how to use these simple threshold triggers to enable sophisticated anomaly detection in near-real time, with modest communication overheads. We design a distributed protocol to detect "unusual traffic patterns" buried in an Origin-Destination network flow matrix that: a) uses a Principal Components Analysis decomposition technique to detect anomalies via a threshold function on residual signals [10]; and b) efficiently tracks this threshold function in near-real time using a simple distributed protocol. In addition, we speculate that such simple thresholding can be a powerful tool for a variety of monitoring tasks beyond the one presented here, and we propose an agenda to explore additional sophisticated applications.
- Babcock, B. and Olston, C. Distributed Top-K Monitoring. In ACM SIGMOD, (2003). Google Scholar
Digital Library
- Clark, D., Partridge, C., Ramming, J. C., and Wroclawski, J. T. A knowledge plane for the internet. In ACM SIGCOMM (2003). Google ScholarDigital Library
- Hanson, E. N., Bodagala, S., and Chadaga., U. Trigger condition testing and view maintenance using optimized discrimination network. IEEE TKDE, 14(2) (2002). Google ScholarDigital Library
- Huang, L., Garofalakis, M., Joseph, A. and Taft, N. Communication-efficient tracking of distributed triggers. Tech. rep., February 2006.Google Scholar
- Huebsch, R., Hellerstein, J., Lanham, N., Loo, B.-T., Shenker, S. and Stoica, I. Querying the internet with pier. In VLDB (2003). Google ScholarDigital Library
- Jain, A., Hellerstein, J. M., Ratnasamy, S., and Wetherall, D. A wakeup call for internet monitoring systems: The case for distributed triggers. In HotNets (2004).Google Scholar
- Jain, A., Chang, E. Y., and Wang, Y.-F. Adaptive stream resource management using kalman filters. In ACM SIGMOD (2004). Google ScholarDigital Library
- Jackson, J. E. and Mudholkar, G. S. Control procedures for residuals associated with principal component analysis. In Technometrics, pages 341--349, 1979.Google ScholarCross Ref
- Keralapura, R., Cormode, G. and Ramamirtham, J. Communication-efficient distributed monitoring of thresholded counts. In ACM SIGMOD (2006). Google ScholarDigital Library
- Lakhina, A., Crovella, M. and Diot, C. Diagnosing network-wide traffic anomalies. In ACM SIGCOMM, (2004). Google ScholarDigital Library
- Lakhina, A., Papagiannaki, K., Crovella, M., Diot, C., Kolaczyk, E. D. and Taft, N. Structural analysis of network traffic flows. In ACM SIGMETRICS, (2004). Google ScholarDigital Library
- Olston, C., Jiang, J., and Widom, J. Adaptive filters for continuous queries over distributed data streams. In ACM SIGMOD (2003). Google ScholarDigital Library
- Padmanabhan, V. N., Ramabhadran, S., and Padhye, J. Netprofiler: Profiling wide-area networks using peer cooperation. In IPTPS (2005). Google ScholarDigital Library
- Spring, N., Wetherall, D., and Anderson, T. Scriptroute: A facility for distributed internet measurement. In USITS (2003).Google Scholar
- Widom, J., and S. Ceri. Active Database Systems: Triggers and Rules for Advanced Database Processing. Morgan Kaufmann, 1996. Google ScholarDigital Library
- Xie, Y., Kim, H.-A., O'hallaron, D. R., Reiter, M. K., and Zhang, H. Seurat: A pointillist approach to anomaly detection. In RAID (2004).Google Scholar
- Yegneswaran, V., Barford, P., and Jha, S. Global intrusion detection in the domino overlay system. In NDSS (2004).Google Scholar
- Zhang, Y., Ge, Z.-H., Greenberg, A., and Roughan, M. Network anomography. In IMC, (2005). Google ScholarDigital Library
Index Terms
- Toward sophisticated detection with distributed triggers
Recommendations
Toward Explainable Deep Anomaly Detection
KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data MiningAnomaly explanation, also known as anomaly localization, is as important as, if not more than, anomaly detection in many real-world applications. However, it is challenging to build explainable detection models due to the lack of anomaly-supervisory ...
Attribute Normalization in Network Intrusion Detection
ISPAN '09: Proceedings of the 2009 10th International Symposium on Pervasive Systems, Algorithms, and NetworksAnomaly intrusion detection is an important issue in computer network security. As a step of data preprocessing, attribute normalization is essential to detection performance. However, many anomaly detection methods do not normalize attributes before ...
Communication-Efficient Tracking of Distributed Cumulative Triggers
ICDCS '07: Proceedings of the 27th International Conference on Distributed Computing SystemsIn recent work, we proposed D-Trigger, a framework for tracking a global condition over a large network that allows us to detect anomalies while only collecting a very limited amount of data from distributed monitors. In this paper, we expand our ...
Comments