John McHugh
Abstract
In 1998 and again in 1999, the Lincoln Laboratory of MIT conducted a comparative evaluation of intrusion detection systems (IDSs) developed under DARPA funding. While this evaluation represents a significant and monumental undertaking, there are a number of issues associated with its design and execution that remain unsettled. Some methodologies used in the evaluation are questionable and may have biased its results. One problem is that the evaluators have published relatively little concerning some of the more critical aspects of their work, such as validation of their test data. The appropriateness of the evaluation techniques used needs further investigation. The purpose of this article is to attempt to identify the shortcomings of the Lincoln Lab effort in the hope that future efforts of this kind will be placed on a sounder footing. Some of the problems that the article points out might well be resolved if the evaluators were to publish a detailed description of their procedures and the rationale that led to their adoption, but other problems would clearly remain./par>
- ALESSANDRI, D. 2000. Using rule-based activity descriptions to evaluate intrusion-detection systems. In RAID2000, H. Debar, L. Me, and S. F. Wu, Eds. Springer-Verlag, New York, NY, 183-196.]] Google Scholar
- ALLEN, J., CHRISTIE, A., FITHEN, W., MCHUGH, J., PICKEL, J., AND STONER, E. 2000. State of the practice of intrusion detection technologies. CMU/SEI-99-TR-028,CMU/SEI. Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA.]]Google Scholar
- AXELSSON, S. 1999. The base-rate fallacy and its implications for the difficulty of intrusion detection. In Proceedings of the 6th ACM Conference on Computer and Communications Security. 1-7.]] Google Scholar
- AXELSSON, S. 2000. Intrusion-detection systems: A taxonomy and survey. 99-15 (March).]]Google Scholar
- AXELSSON, S. 2000. A preliminary attempt to apply detection and estimation theory to intrusion detection. 00--4 (March).]]Google Scholar
- BELLOVIN, S. M. 1993. Packets found on an internet. SIGCOMM Comput. Commun. Rev. 23, 3 (July), 26-31.]] Google Scholar
- CERT COORDINATION CENTER. 2000. Cert security improvement modules. http://www.cert. org/security-improvement]]Google Scholar
- DAS, K. 2000. Attack development for intrusion detection. Master's Thesis. Massachusetts Institute of Technology, Cambridge, MA.]]Google Scholar
- DEBAR, H., DACIER, M., WESPI, A., AND LAMPART, S. 1998. An experimentation workbench for intrusion detection systems. Res. Rep. RZ 2998 (#93044) (Sept.). Research Division, IBM, New York, NY.]]Google Scholar
- DHARANIPRAGADA,S.AND ROUKOS, S. 1998. A fast vocabulary independent algorithm for spotting words in speech. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (May). 233-236.]]Google Scholar
- DURST, R., CHAMPION, T., WITTEN, B., MILLER, E., AND SPAGNUOLO, L. 1999. Testing and evaluating computer intrusion detection systems. Commun. ACM 42, 7, 53-61.]] Google Scholar
- EGAN, J. P. 1975. Signal Detection Theory and ROC Analysis. Academic Press, Inc., Orlando, FL.]]Google Scholar
- EINBINDER, H. 1964. The Myth of the Britannica. Grove Press, New York, NY.]]Google Scholar
- GRAF, I., LIPPMANN, R., CUNNINGHAM, R., FRIED, D., KENDALL, K., WEBSTER, S., AND ZISSMAN,M. 1998. Results of DARPA 1998 offline intrusion detection evaluation. http://ideval.ll.mit. edu/results-html-dir/]]Google Scholar
- HEBERLEIN,L.T.,DIAS,G.V.,LEVITT,K.N.,MUKHERJEE, B., WOOD, J., AND WOLBER, D. 1990. A network security monitor. In Proceedings of the IEEE Symposium on Research in Security and Privacy (Oakland, CA). IEEE Computer Society Press, Los Alamitos, CA, 296-30304. http://olympus.cs.ucdavis.edu/papers.html.]]Google Scholar
- HUFF, D. 1954. How to Lie with Statistics. W. W. Norton & Co., Inc., New York, NY.]] Google Scholar
- JAMES,D.A.AND YOUNG, S. J. 1994. A fast lattice-based approach to vocabulary independent wordspotting. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing 2. 337-380.]]Google Scholar
- JEANRENAUD, P., SIU, M., ROHLICEK,J.R.,METEER, M., AND GISH, H. 1994. Spotting events in continuous speech. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing 2. 381-384.]]Google Scholar
- JUNKAWITSCH,J.AND H~GE, H. 1998. Keyword verification considering the correlation of succeeding feature vectors. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (May). 221-224.]]Google Scholar
- KENDALL, K. 1999. A database of computer attacks for the evaluation of intrusion detection systems. Master's Thesis. Massachusetts Institute of Technology, Cambridge, MA.]]Google Scholar
- KLATT, D. H. 1977. Review of the ARPA speech understanding project. J. Acoust. Soc. Amer. 62, 1345-1366.]]Google Scholar
- KORBA, J. 2000. Windows NT attacks for the evaluation of intrusion detection systems. Master's Thesis. Massachusetts Institute of Technology, Cambridge, MA.]]Google Scholar
- LIPPMANN, R., HAINES,J.W.,FRIED,D.J.,KORBA, J., AND DAS, K. 2000. The 1999 DARPA off-line intrusion detection evaluation. In RAID2000, H. Debar, L. Me, and S. F. Wu, Eds. Springer-Verlag, New York, NY, 162-182.]] Google Scholar
- LIPPMANN,R.P.,CHANG,E.I.,AND JANKOWSKI, C. R. 1994. Wordspotter training using figure-of-merit back propagation. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing 2. 385-388.]]Google Scholar
- LIPPMANN,R.P.,CUNNINGHAM,R.K.,FRIED,D.J.,GARFINKEL,S.L.,GORTON,A.S.,GRAF, I., KENDALL,K.R.,MCCLUNG,D.J.,WEBER,D.J.,WEBSTER,S.E.,WYSCHOGROD, D., AND ZISSMAN, M. A. 1988. MIT Lincoln Laboratory offline component of DARPA 1998 intrusion detection evaluation. http://ideval.ll.mit.edu/intro-html-dir/.]]Google Scholar
- LIPPMANN,R.P.,FRIED, D., GRAF, I., HAINES, J., KENDALL, K., MCCLUNG, D., WEBBER, D., WEBSTER, S., WYSCHOGRAD, D., CUNNINGHAN, R., AND ZISSMAN, M. 2000. Evaluating intrusion detection systems: The 1998 DARPA off-line intrusion detection evaluation. In Proceedings of the on DARPA Information Survivability Conference and Exposition (DISCEX '00, Hilton Head, South Carolina, Jan. 25-27). IEEE Computer Society Press, Los Alamitos, CA, 12-26.]]Google Scholar
- MAXION,R.A.AND TAN, K. M. C. 2000. Benchmarking anomaly-based detection systems. In Proceedings of International Conference on Dependable Systems and Networks (June). 623-630.]] Google Scholar
- PAXSON, V. 1999. Bro: A system for detecting network intruders in real-time. Comput. Netw. J. 23-24 (Dec.), 2435-2463.]] Google Scholar
- POMERANZ, H. 1999. Solaris security: Step by step. http:www.sans.org]]Google Scholar
- PTACEK,T.H.AND NEWSHAM, T. N. 1998. Insertion, evasion, and denial of service: Eluding network intrusion detection. http://www.secinf.net/info/ids/idspaper/idspaper.html]]Google Scholar
- PUKETZA, N., CHUNG, M., OLSSON,R.A.,AND MUKHERJEE, B. 1997. A software platform for testing intrusion detection systems. IEEE Software 14, 5 (Sept.), 43-51. http://seclab.cs. ucdavis.edu/papers.html]] Google Scholar
- PUKETZA,N.J.,ZHANG, K., CHUNG, M., MUKHERJEE, B., AND OLSSON, R. A. 1996. A methodology for testing intrusion detection systems. IEEE Trans. Softw. Eng. 22, 10, 719-729. http://seclab.cs.ucdavis.edu/papers.html]] Google Scholar
- SIFT. 1984. Peer review of a formal verification/design proof methodology. RTI/2094/13-01F.]]Google Scholar
- SIFT. 1985. Peer review of a formal verification/design proof methodology. Conference Publication CP-2377.]]Google Scholar
- SWETS, J. A. 1988. Measuring the accuracy of diagnostic systems. Science 24, 48, 1285-1293.]]Google Scholar
- SWETS,J.A.AND PICKETT, R. M. 1982. Evaluation of Diagnostic Systems. Academic Press, Inc., New York, NY.]]Google Scholar
- TRIPWIRE,INC. 2000. Tripwire 2.2.1 for UNIX, Users Manual. WWW.TripwireSecurity.com.]]Google Scholar
- WEBER, D. 1998. A taxonomy of computer intrusions. Master's Thesis. Massachusetts Institute of Technology, Cambridge, MA.]]Google Scholar
- WHITING-O'KEEFE,Q.E.,HENKE, C., AND SIMBORG, D. W. 1984. Choosing the correct unit of analysis in medical care experiments. Med. Care 22, 12 (Dec.), 1101-1114.]]Google Scholar
Index Terms
- Testing Intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory
Recommendations
Syntax vs. semantics: competing approaches to dynamic network intrusion detection
Malicious network traffic, including widespread worm activity, is a growing threat to internet-connected networks and hosts. In this paper, we consider both syntax and semantics based approaches for dynamic network intrusion detection. The semantics-...
Rule generalisation in intrusion detection systems using SNORT
Intrusion Detection Systems (IDSs) provide an important layer of security for computer systems and networks. An IDS's responsibility is to detect suspicious or unacceptable system and network activity and to alert a systems administrator to this ...
A Methodology for Testing Intrusion Detection Systems
Intrusion Detection Systems (IDSs) attempt to identify unauthorized use, misuse, and abuse of computer systems. In response to the growth in the use and development of IDSs, we have developed a methodology for testing IDSs. The methodology consists of ...
Reviews
S. V. Nagaraj
This is a critique of the 1998 and 1999 DARPA intrusion detection system evaluations performed by MIT’s Lincoln laboratory. It discusses issues related to the design and execution of the evaluation. The methodologies used in the evaluation and also the appropriateness of the evaluation techniques themselves are questioned. The author mentions that the evaluators have published relatively little concerning aspects such as validation of test data. The author contends that if the evaluators were to publish a detailed description of their procedures and the rationale that led to their adoption, some of the problems would be resolved, but some would still remain. As the author acknowledges, the paper does not make a direct technical contribution or provide solutions to the problems raised. I feel that at least some leads could have been provided here. The author feels that the public records of such an evaluation should contain sufficient information to permit replication of the results and to understand the rationale behind most of the decisions made by the investigators. This is undoubtably true, but we can only speculate as to why it was not the case. Some suggestions are given for performing future evaluations that might overcome the deficiencies of the Lincoln lab work. The author leaves the reader with the following question: would the criticisms made in the paper have had a significant effect on the outcome of the evaluation__?__
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments