By Members of the deter and EMIST Projects CYBER DEFENSE TECHNOLOGY NETWORKINGANDEALUATION s the Internet has become pervasive and our critical infra- structures have become inextricably tied to information systems, the risk for economic, social, and physical dis- ruption due to the insecurities of information systems has increased immeasurably. Over the past 10 years there has been increased investment in research on cyber security technologies by U.S. government agencies(including NSE, DARPA, the armed forces)and industry. However, a large-scale deployment of security technology sufficient to protect the vital infrastructure is lack ing. One important reason for this deficiency is the lack of an experi- mental infrastructure and rigorous scientific methodologies for developing and testing next-generation cyber security technology. To date, new security technologies have been tested and validated only in small-to medium-scale private research facilities, which are not repre- sentative of large operational networks or of the portion of the Inter net that could be involved in an attack. To make rapid advances in defending against attacks, the state of the art in evaluation of network security mechanisms must be improved This will require the development of large-scale security testbeds [3] combined with new frameworks and standards for testing and benchmarking that make these testbeds truly useful. Current defi ciencies and impediments to evaluating network security mechanisms include lack of scientific rigor [6]; lack of relevant and representative network data 5]; inadequate models of defense mechanisms; and inadequate models of the network and both the background and attack traffic data [1]. The latter is challenging because of the com- plexity of interactions among traffic, topology, and protocols [1, 2 To address these shortcomings, we will create an experimental infrastructure network to support the development and demonstra- tion of next-generation information security technologies for cyber defense. The Cyber Defense Technology Exper tal Research net- work(DETER network) will provide the necessary infrastructure-
A s the Internet has become pervasive and our critical infrastructures have become inextricably tied to information systems, the risk for economic, social, and physical disruption due to the insecurities of information systems has increased immeasurably. Over the past 10 years there has been increased investment in research on cyber security technologies by U.S. government agencies (including NSF, DARPA, the armed forces) and industry. However, a large-scale deployment of security technology sufficient to protect the vital infrastructure is lacking. One important reason for this deficiency is the lack of an experimental infrastructure and rigorous scientific methodologies for developing and testing next-generation cyber security technology. To date, new security technologies have been tested and validated only in small- to medium-scale private research facilities, which are not representative of large operational networks or of the portion of the Internet that could be involved in an attack. To make rapid advances in defending against attacks, the state of the art in evaluation of network security mechanisms must be improved. This will require the development of large-scale security testbeds [3] combined with new frameworks and standards for testing and benchmarking that make these testbeds truly useful. Current deficiencies and impediments to evaluating network security mechanisms include lack of scientific rigor [6]; lack of relevant and representative network data [5]; inadequate models of defense mechanisms; and inadequate models of the network and both the background and attack traffic data [1]. The latter is challenging because of the complexity of interactions among traffic, topology, and protocols [1, 2]. To address these shortcomings, we will create an experimental infrastructure network to support the development and demonstration of next-generation information security technologies for cyber defense. The Cyber Defense Technology Experimental Research network (DETER network) will provide the necessary infrastructure— 58 September 2003/Vol. 46, No. 9 COMMUNICATIONS OF THE ACM By Members of the DETER and EMIST Projects CYBER DEFENSE TECHNOLOGY NETWORKING AND EVALUATION
Creating an experimental infrastructure for developing next-generation information security technologies networks, tools, and supporting processes--to support national-scale experimentation on emerging security research and advanced devel opment technologies. In parallel, the Evaluation Methods for Ir Security Technology(MIst) project will develop scientifical orous testing frameworks and methodologies for representative classes of network attacks and defense mechanisms. As part of this research approaches to determining domains of effective use for simulation emulation, hardware, and hybrids of the three are being examined The goal of this joint effort is to create, operate, and support a researcher-and vendor-neutral experimental infrastructure open to a wide community of users. It is intended to be more than a passive research instrument. It is envisioned to serve as a center for inter- change and collaboration among security researchers, and as a shared laboratory in which researchers, developers, and operators from gov- ernment,industry, and academia can experiment with potential cyber security technologies under realistic conditions, with the aim of accel erating research, development, and deployment of effective defenses for U.S.-based computer networks Information Security Challenges o develop a testbed framework for evaluating security mech- nisms,the project focuses on a select subset of the overall problem space. Several different types of attacks and defenses will be studied with two goals: to elevate the understanding of the particular attack or defense by thoroughly evaluating it via different testing scenarios; and to further the under standing of the degree to which these evaluations can be unified into a single framework that spans the diversity of the problem space ams involved in the joint effort: U.C. Berkeley, U C Davis, Universiry of Southern Cali- ional Computer Science Institute (ICSD), Purdue, SPARTA Inc, and SRI International. The project cludes an industrial advisory board consisting of equipment vendors, carriers, and ISPs including AOL, Cisco, Alcatel, Hewlett-Packard, IBM, Intel, Juniper, and Los Nert COMMUNICATIONS OF THE ACM March 2004/Vol 47, No 3 59
COMMUNICATIONS OF THE ACM March 2004/Vol. 47, No. 3 59 networks, tools, and supporting processes—to support national-scale experimentation on emerging security research and advanced development technologies. In parallel, the Evaluation Methods for Internet Security Technology (EMIST) project will develop scientifically rigorous testing frameworks and methodologies for representative classes of network attacks and defense mechanisms. As part of this research, approaches to determining domains of effective use for simulation, emulation, hardware, and hybrids of the three are being examined. The goal of this joint effort1 is to create, operate, and support a researcher- and vendor-neutral experimental infrastructure open to a wide community of users. It is intended to be more than a passive research instrument. It is envisioned to serve as a center for interchange and collaboration among security researchers, and as a shared laboratory in which researchers, developers, and operators from government, industry, and academia can experiment with potential cyber security technologies under realistic conditions, with the aim of accelerating research, development, and deployment of effective defenses for U.S.-based computer networks. Information Security Challenges T o develop a testbed framework for evaluating security mechanisms, the project focuses on a select subset of the overall problem space. Several different types of attacks and defenses will be studied with two goals: to elevate the understanding of the particular attack or defense by thoroughly evaluating it via different testing scenarios; and to further the understanding of the degree to which these evaluations can be unified into a single framework that spans the diversity of the problem space. Creating an experimental infrastructure for developing next-generation information security technologies. 1There are nine teams involved in the joint effort: U.C. Berkeley, U.C. Davis, University of Southern California-Information Sciences Institute (USC-ISI), Pennsylvania State University, NAI Laboratories, International Computer Science Institute (ICSI), Purdue, SPARTA Inc., and SRI International. The project also includes an industrial advisory board consisting of equipment vendors, carriers, and ISPs including AOL, Cisco, Alcatel, Hewlett-Packard, IBM, Intel, Juniper, and Los Nettos
Three different classes of attacks are focus areas for Testbed Architecture and Requirements our research: denial-of-service, worms, and attacks on he preliminary requirements for the DETER the Internets routing infrastructure, as well as attacks Testbed are drawn from four sources: a that are coordinated combinations of these three DARPA-funded study of security types. Together they span a broad range of general requirements B3], input from network secu- types of attacks. In addition, the project will closely rity researchers, general considerations on monitor new Internet security breaches in order to network research testbeds through a NSF analyze how new attack scenarios can be incorporated workshop [4], and experience with a variety of earlier into the developing testing methodology. The focus experimental and test networks. High-level require of this effort will be on attacks targeting network ments are briefly described here infrastructure, server end-systems, and critical end- The general objectives for the testbed design require user applications. Such attacks are difficult to accu- that it must be fully isolated from the Internet and all rately simulate experiments must be soundly confined within the because of the major challenges in accurately simulat- DETER network. Furthermore, it is expected the net- ing Internet phenomena in general [2, 4 work will be subjected to destructive traffic and that experiments may temporarily damage the network. Security Testing Methodologies Therefore, there must also be mechanisms for rapid esting frameworks will be adapted for dif- reconstitution of the testing environment ferent kinds of testbeds, including simula- The scale of the testbed will be approximately 1,000 rssuchasNs(seEwww.isiedu/Pcs,eachwithmultiplenetworkinterfacecardsanda nsnam/ns), emulation facilities such as significant number of commercial routers and pro- Emulab [8], and both small and large hard- grammable switches. Within this environment, the ware testbeds. The frameworks will include network must provide sufficient topological complex attack scenarios, attack simulators, generators for ity to emulate a scaled down but functionally accurate topology and background traffic, data sets derived representation of the hierarchical structure of the real from live traffic, and tools to monitor and summarize Internet, and to approximate the mixing of benign traf- est results. These frameworks will allow researchers fic and attack traffic that occurs. Initially, the network to experiment with a variety of parameters represent- will be formed using a homogeneous network of exist- the network environment including attack behav- ing technology. Carefully chosen hardware het rogene rs,deployed defense technology, and the ity--c commercia router boxes-w rill be added as the configuration of the defense mechanisms under test. effort progresses. Finally, conducting experiments with It will be critical to make progress on the very difficult large-scale denial-of-service attacks and defense tech problems, particularly nologies to protect the Internet infrastructure will require high-bandwidth componentry. How to construct realistic topologies, including In addition to the preceding infrastructure require bandwidth and inter-AS policies ments,there are various requirements for software to How to generate realistic cross-traffic across these facilitate experimentation. The utility of DETER will topologies, depend on the power, convenience, and flexibility of its How to quantify how accurate the models need to software for setting up and managing experiments be, and including registration, definition, generation control How to select the best metrics for evaluating vari- monitoring, check-pointing, and archiving. An impor- ous defense mechanisms tant aspect of the management software will be the quirement for sophisticated network monitoring and Conducting these tests will require incorporating traffic analysis tools for both experimenters and defense mechanisms into a testbed (either as models DETER network operators. Experimenters will also or as operational code), and applying and evaluating require traffic generation software to generate attack the frameworks and methodologies. Conducting traffic and typical day-to-day (legitimate use)traffic. these tests will also help to ensure the testbed frame- Preliminary Architecture. DETER will be built as work allows other researchers to easily integrate and three permanent hardware clusters, located at ISI in test network defense mechanisms of their own design. Los Angeles, ISI-East in Virginia, and UC-Berkeley. To Furthermore, the documentation of the tests will provide the earliest possible service to experimenters, serve as a tutorial for users of the testbed framework as initial development during the first six months focuses they confirm their results or evaluate their own mech- on building software and configurations for cyber secu- anisms and techniques rity experimentation on Planet Lab and/or Emulab [8 60 March 2004/Vol 47. No. 3 COMMUNICATONS OF THE ACM
60 March 2004/Vol. 47, No. 3 COMMUNICATIONS OF THE ACM Three different classes of attacks are focus areas for our research: denial-of-service, worms, and attacks on the Internet’s routing infrastructure, as well as attacks that are coordinated combinations of these three types. Together they span a broad range of general types of attacks. In addition, the project will closely monitor new Internet security breaches in order to analyze how new attack scenarios can be incorporated into the developing testing methodology. The focus of this effort will be on attacks targeting network infrastructure, server end-systems, and critical enduser applications. Such attacks are difficult to accurately simulate using existing testing frameworks because of the major challenges in accurately simulating Internet phenomena in general [2, 4]. Security Testing Methodologies T esting frameworks will be adapted for different kinds of testbeds, including simulators such as NS (see www.isi.edu/ nsnam/ns), emulation facilities such as Emulab [8], and both small and large hardware testbeds. The frameworks will include attack scenarios, attack simulators, generators for topology and background traffic, data sets derived from live traffic, and tools to monitor and summarize test results. These frameworks will allow researchers to experiment with a variety of parameters representing the network environment including attack behaviors, deployed defense technology, and the configuration of the defense mechanisms under test. It will be critical to make progress on the very difficult problems, particularly: • How to construct realistic topologies, including bandwidth and inter-AS policies, • How to generate realistic cross-traffic across these topologies, • How to quantify how accurate the models need to be, and • How to select the best metrics for evaluating various defense mechanisms. Conducting these tests will require incorporating defense mechanisms into a testbed (either as models or as operational code), and applying and evaluating the frameworks and methodologies. Conducting these tests will also help to ensure the testbed framework allows other researchers to easily integrate and test network defense mechanisms of their own design. Furthermore, the documentation of the tests will serve as a tutorial for users of the testbed framework as they confirm their results or evaluate their own mechanisms and techniques. Testbed Architecture and Requirements T he preliminary requirements for the DETER Testbed are drawn from four sources: a DARPA-funded study of security testbed requirements [3], input from network security researchers, general considerations on network research testbeds through a NSF workshop [4], and experience with a variety of earlier experimental and test networks. High-level requirements are briefly described here. The general objectives for the testbed design require that it must be fully isolated from the Internet and all experiments must be soundly confined within the DETER network. Furthermore, it is expected the network will be subjected to destructive traffic and that experiments may temporarily damage the network. Therefore, there must also be mechanisms for rapid reconstitution of the testing environment. The scale of the testbed will be approximately 1,000 PCs, each with multiple network interface cards, and a significant number of commercial routers and programmable switches. Within this environment, the network must provide sufficient topological complexity to emulate a scaled down but functionally accurate representation of the hierarchical structure of the real Internet, and to approximate the mixing of benign traffic and attack traffic that occurs. Initially, the network will be formed using a homogeneous network of existing technology. Carefully chosen hardware heterogeneity—commercial router boxes—will be added as the effort progresses. Finally, conducting experiments with large-scale denial-of-service attacks and defense technologies to protect the Internet infrastructure will require high-bandwidth componentry. In addition to the preceding infrastructure requirements, there are various requirements for software to facilitate experimentation. The utility of DETER will depend on the power, convenience, and flexibility of its software for setting up and managing experiments including registration, definition, generation control, monitoring, check-pointing, and archiving. An important aspect of the management software will be the requirement for sophisticated network monitoring and traffic analysis tools for both experimenters and DETER network operators. Experimenters will also require traffic generation software to generate attack traffic and typical day-to-day (legitimate use) traffic. Preliminary Architecture. DETER will be built as three permanent hardware clusters, located at ISI in Los Angeles, ISI-East in Virginia, and UC-Berkeley. To provide the earliest possible service to experimenters, initial development during the first six months focuses on building software and configurations for cyber security experimentation on PlanetLab and/or Emulab [8]
The architecture will also deploy aspects of the tor hindering wide-scale adoption of next-generation X-bone(seewww.isi.edu/xbone)toallowtopologiessolutionsResultsobtainedusingtheDetertestbed with revisitation, where, for example, a 10-node ring will contribute to the development of innovative new can be used to emulate a 100-node ring by visiting the technologies that increase commercial availability and same node multiple times. During the early stages of viability of new production networks and services, pro- the testbed, this will enable the simulation of topologies viding true cyber protection. C that are larger than can be supported with one-to-one mapping of physical resources. Meanwhile, a phased REFeRENCeS development effort, moving from carefully controlled 1. Floyd, S. and Kohler, E. Internet research needs better models.Hometr-I emulation environments to a mix of emulation and real 2. Floyd. S and Paxson. V. Difficulties in simulating the Internet. IEEEIACM network hardware will occur msactions on Networking 9. 4(Aug 2001), 392-403 3. Hardaker, W. et al. Justification quirements for a National DDas Defense Technology Emanation Facility. Network Associates Laboratories Conclusion he development of testing methodologies 4 Kurose, J. Ed Report of NSF Workshop on Netuork Reeanbh Tatbeds (Now nplemented by an experimental infra- 5. McH esting intrusion detection systems: A critique of the 1998 an structure will support the realis tent evaluation of mechanisms purported to Laboratory. ACM Transactions on Information and System Serurity 3 4(Nor, stic and con Sis- 1999 DARPA intrusion detection system valuations as performed by Lind mitigate large-scale attacks. This is an 6. Pawlikowski, K Jeong, H, and Lee, J On credibility of simulation studies extremely challenging undertaking-no of telecommunication networks. IEEE Communications Magazine (an. existing testbed or framework can be claimed to be 7. Peterson, L. Anderson, T. Culler, D . and Roscoe, T. A blueprint for intro effective. The research described here requires signifi- ducing disruptive technology into the Internet. In Proceedings of the Ist ACM cant advances in the modeling of network attacks and Workshop on Hot Topics in Neuorbs(Hot/)(Oct. 2002), 4-140 the interactions between attacks and their environ- ments,including deployed defense technology, back- tems Design and Implementation(OSD/02),(Dec 2002) ground traffic, topology, protocols, and applications. It rill also require advances in the understanding of met- MEMBERS OF THE DETER AND EMI ST NETWORK PROJECT ics for evaluating defense mechanisms. C BRODLEY, S FAHMY, S. FLOYD, W. HARDAKER, Our results will provide new scientific knowledge to A. JoSEPH, G.KESIDIS, KLEVITT,B LINDELL, P.LIU enable the development of solutions to cyber security D. MILLER, R. MUNDY, C NEUMAN, R. OSTRENGA problems of national importance. This will be accom- V. PAXSON, P PORRAS,C. ROSENBERG,J D. TYGAR, plished through experimentation and validation of S SASTRY, DSTERNE, AND SF.Wu cyber defense technologies using scientific methods. This project is funded joinly by the U.S.Department of Homeland Securiry(DHS)and The lack of open, objective, and repeatable validation the National Sance Foundation (NSh under grant AN-0335241 of cyber defense technologies has been a significant f 0204ACM0002-078204/0500$5.00 Coming Next Month in Communications Etiquette for Human-Computer relations a growing community of computer scientists, researchers, sociologists, psychologists, educators, human interaction with computers and other forms of advanced automation. And they are finding and industry practitioners are taking the"etiquette perspective"in designing, building, and analy among its many advantages, the etiquette approach facilitates user acceptance of systems and products and, more importantly, improves the accuracy and speed with which the users develop trust in such products It's a practice in its infancy and next month's special section introduces the concept of etiquette and provides a variety of perspectives on its use and importance, including a number of examples f current research and applications. Aloo in April Software Project Risks and their Impact on Outcomes.Cross-Cultural Issues in Global Software Outsourcing.Who Should Work for Whom?. Managing Academic E-Journals Economics of Wireless Networks. Managing Knowledge in Distributed Projects. Information Cascades in IT Adoption COMMUNICATIONS OF THE ACM March 2004/Vol 47. No.3 61
COMMUNICATIONS OF THE ACM March 2004/Vol. 47, No. 3 61 The architecture will also deploy aspects of the X-bone (see www.isi.edu/xbone) to allow topologies with revisitation, where, for example, a 10-node ring can be used to emulate a 100-node ring by visiting the same node multiple times. During the early stages of the testbed, this will enable the simulation of topologies that are larger than can be supported with one-to-one mapping of physical resources. Meanwhile, a phased development effort, moving from carefully controlled emulation environments to a mix of emulation and real network hardware will occur. Conclusion T he development of testing methodologies complemented by an experimental infrastructure will support the realistic and consistent evaluation of mechanisms purported to mitigate large-scale attacks. This is an extremely challenging undertaking—no existing testbed or framework can be claimed to be effective. The research described here requires significant advances in the modeling of network attacks and the interactions between attacks and their environments, including deployed defense technology, background traffic, topology, protocols, and applications. It will also require advances in the understanding of metrics for evaluating defense mechanisms. Our results will provide new scientific knowledge to enable the development of solutions to cyber security problems of national importance. This will be accomplished through experimentation and validation of cyber defense technologies using scientific methods. The lack of open, objective, and repeatable validation of cyber defense technologies has been a significant factor hindering wide-scale adoption of next-generation solutions. Results obtained using the DETER testbed will contribute to the development of innovative new technologies that increase commercial availability and viability of new production networks and services, providing true cyber protection. References 1. Floyd, S. and Kohler, E. Internet research needs better models. Hotnets-I (Oct. 2002). 2. Floyd, S. and Paxson, V. Difficulties in simulating the Internet. IEEE/ACM Transactions on Networking 9, 4 (Aug. 2001), 392–403. 3. Hardaker, W. et al. Justification and Requirements for a National DDoS Defense Technology Evaluation Facility. Network Associates Laboratories Report 02-052, July 26, 2002. 4. Kurose, J., Ed. Report of NSF Workshop on Network Research Testbeds (Nov. 2002); gaia.cs.umass.edu/testbed_workshop. 5. McHugh, J. Testing intrusion detection systems: A critique of the 1998 and 1999 DARPA intrusion detection system valuations as performed by Lincoln Laboratory. ACM Transactions on Information and System Security 3, 4 (Nov. 2000), 262–294. 6. Pawlikowski, K., Jeong, H., and Lee, J. On credibility of simulation studies of telecommunication networks. IEEE Communications Magazine (Jan. 2001). 7. Peterson, L., Anderson, T., Culler, D., and Roscoe, T. A blueprint for introducing disruptive technology into the Internet. In Proceedings of the 1st ACM Workshop on Hot Topics in Networks (HotNets-I) (Oct. 2002), 4–140. 8. White, B. et al. An integrated experimental environment for distributed systems and networks. In Proceedings of the Fifth Symposium on Operating Systems Design and Implementation (OSDI02), (Dec. 2002). Members of the DETER and EMIST network project include R. Bajcsy, T. Benzel, M. Bishop, B. Braden, C. Brodley, S. Fahmy, S. Floyd, W. Hardaker, A. Joseph, G. Kesidis, K. Levitt, B. Lindell, P. Liu, D. Miller, R. Mundy, C. Neuman, R. Ostrenga, V. Paxson, P. Porras, C. Rosenberg, J.D. Tygar, S. Sastry, D. Sterne, and S.F. Wu. This project is funded jointly by the U.S. Department of Homeland Security (DHS) and the National Science Foundation (NSF) under grant ANI-0335241. © 2004 ACM 0002-0782/04/0300 $5.00 c Coming Next Month in Communications Etiquette for Human-Computer Relations A growing community of computer scientists, researchers, sociologists, psychologists, educators, and industry practitioners are taking the “etiquette perspective” in designing, building, and analyzing human interaction with computers and other forms of advanced automation. And they are finding, among its many advantages, the etiquette approach facilitates user acceptance of systems and products and, more importantly, improves the accuracy and speed with which the users develop trust in such products. It’s a practice in its infancy and next month’s special section introduces the concept of etiquette and provides a variety of perspectives on its use and importance, including a number of examples of current research and applications. Also in April Software Project Risks and their Impact on Outcomes • Cross-Cultural Issues in Global Software Outsourcing • Who Should Work for Whom? • Managing Academic E-Journals • Economics of Wireless Networks • Managing Knowledge in Distributed Projects • Information Cascades in IT Adoption