New Directions in Traffic Measurement and Accounting Focusing on the Elephants, Ignoring the mice Cristian estan and george varghese University of california. San diego SIGCOMM 2002
SIGCOMM 2002 New Directions in Traffic Measurement and Accounting Focusing on the Elephants, Ignoring the Mice Cristian Estan and George Varghese University of California, San Diego
Talk outline Problem definition Sample and hold Multistage filters Validation. measurements Conclusions SIGCOMM 2002
SIGCOMM 2002 Talk outline • Problem definition • Sample and hold • Multistage filters • Validation, measurements • Conclusions
Traffic analysis today图 Router Workstation Concise analysis Large raw data collati results Measurement and vsis module ware Sampled packets Offline analysis fast link SIGCOMM 2002
SIGCOMM 2002 Traffic analysis today Router Fast link Measurement module Sampled packets Workstation Large raw data Collection and analysis software Concise analysis results Offline analysis
Our research agenda Ea Router Concise results Measurement Real-time analysis module Is it doable? Is it better? fast link SIGCOMM 2002
SIGCOMM 2002 Our research agenda Router Real-time analysis •Is it doable? •Is it better? Fast link Measurement module Concise analysis results
What is traffic analysis used for? Network planning: need to know traffic between pairs of networks(traffic matrix) Accounting: usage based billing Detecting Dos attacks: flood attacks Application characterization: breaking up the traffic based on port numbers SIGCOMM 2002
SIGCOMM 2002 What is traffic analysis used for? • Network planning: need to know traffic between pairs of networks (traffic matrix) • Accounting: usage based billing • Detecting DoS attacks: flood attacks • Application characterization: breaking up the traffic based on port numbers • …
Common abstractions Packets are grouped together into streams based on header fields Traffic matrix- by source and destination as >DoS attacks- by destination IP address Measuring large streams(this paper Estimating the number of active streams(poster SIGCOMM 2002
SIGCOMM 2002 Common abstractions • Packets are grouped together into streams based on header fields ➢Traffic matrix – by source and destination AS ➢DoS attacks – by destination IP address • Measuring large streams (this paper) • Estimating the number of active streams (poster) • …
Why is measuring streams hard? Cheap memories (dram)are too slow to count all packets Fast memories(SRAM)are too small to keep counters for all streams Opportunity: elephants matter, mice dont Problem: usually we dont know in advance which streams are large SIGCOMM 2002
SIGCOMM 2002 Why is measuring streams hard? • Cheap memories (DRAM) are too slow to count all packets • Fast memories (SRAM) are too small to keep counters for all streams • Opportunity: elephants matter, mice don’t • Problem: usually we don’t know in advance which streams are large
Problem definition Given a fixed definition for streams measure large streams accurately >Large=above 1% of link capacity over a 1 minute interval assumptions Mice don t matter Accuracy of results important SIGCOMM 2002
SIGCOMM 2002 Problem definition • Given a fixed definition for streams, measure large streams accurately ➢Large = above 1% of link capacity over a 1 minute interval • Assumptions ➢Mice don’t matter ➢Accuracy of results important
Talk outline Problem definition Sample and hold Multistage filters Validation. measurements Conclusions SIGCOMM 2002
SIGCOMM 2002 Talk outline • Problem definition • Sample and hold • Multistage filters • Validation, measurements • Conclusions
How does sample and hold work? stream memor Sample Insert stream1 1 SIGCOMM 2002
SIGCOMM 2002 How does sample and hold work? stream memory stream1 1 Sample Insert