1624 IEEE COMMUNICATIONS SURVEYS TUTORIALS.VOL.16.NO.3.THIRD QUARTER 2014 the controller would have to make a decision for each arriving packet.When controlling individual flows,the HIGH-LEVEL NETWORK decision made for the first packet of the flow can be ap- SERVICE(S)/APPLICATION(S) plied to all subsequent packets of that flow.The overhead may be further reduced by grouping flows together,such NORTHBOUND COMMUNICATION as all traffic between two hosts,and performing control decisions on the aggregated flows. SERVICE CONTROLLER INTERFACE Reactive vs.Proactive Policies NETWORK CONTROLLER Under a reactive control model,such as the one proposed by Ethane [20].forwarding elements must consult a OTHER SERVICE TOPOLOGY controller each time a decision must be made,such as ESSENTIAL MANAGER MANAGER when a packet from a new flow reaches a switch.In FUNCTIONS the case of flow-based control granularity,there will be NTROLLER SWITCH INTERFACE a small performance delay as the first packet of each new flow is forwarded to the controller for decision SOUTHBOUND COMMUNICATION (e.g.,forward or drop),after which future packets within (E.G.OPENFLOW) that flow will travel at line rate within the forwarding CONTROLLER SWITCH INTERFACE hardware.While the delay incurred by the first-packet may be negligible in many cases,it may be a concern if PACKET FORWARDING DEVICE(S) the controller is geographically remote (though this can be mitigated by physically distributing the controller [45]) or if most flows are short-lived,such as single-packet Fig.5.A controller with a northbound and southbound interface. flows.There are also some scalability issues in larger networks,as the controller must be able to handle a larger volume of new flow requests. controller.For security,OpenFlow 1.3.0 provides optional Alternatively,proactive control approaches push policy support for encrypted Transport Layer Security (TLS)com- rules from the controller to the switches.A good example munication and a certificate exchange between the switches of proactive control is DIFANE [35].which partitions and the controller(s):however,the exact implementation and rules over a hierarchy of switches,such that the controller certificate format is not currently specified.Also outside the rarely needs to be consulted about new flows and traffic is scope of the current specification are fine-grained security kept within the data-plane.In their experiments,DIFANE options regarding scenarios with multiple controllers,as there reduces first-packet delay from a 10ms average round-trip is no method specified to only grant partial access permissions time(RTT)with a centralized NOX controller to a 0.4ms to an authorized controller.We examine OpenFlow controller average RTT for new single-packet flows.It was also implementation options in greater detail in Section IV. shown to increase the new flow throughput,as the tested version of NOX achieved a peak of 50,000 single-packet E.Northbound Communication:Controller-Service flows per second while the DIFANE solution achieved External management systems or network services may 800,000 single-packet flows per second.Interestingly,it wish to extract information about the underlying network or was observed that the OpenFlow switch's local controller control an aspect of network behavior or policy.Additionally, implementation becomes a bottleneck before the central controllers may find it necessary to communicate with each NOX controller.This was attributed to the fact that com- other for a variety of reasons.For example,an internal control mercial OpenFlow switch implementations were limited application may need to reserve resources across multiple to sending 60-330 new flows requests per second at the domains of control or a "primary"controller may need to time of their publication (2010). share policy information with a backup,etc. As shown in Figure 5,a controller that acts as a network Unlike controller-switch communication,there is no cur- operating system must implement at least two interfaces:a rently accepted standard for northbound interactions and they "southbound"interface that allows switches to communicate are more likely to be implemented on an ad hoc basis for with the controller and a "northbound"interface that presents particular applications.We discuss this further in Section VI. an API to network control and high-level applications/services. F.Standardization Efforts D.Southbound Communication:Controller-Switch Recently,several standardization organizations have been An important aspect of SDNs is the link between the turning the spotlights towards SDN.For example,as previ- data-plane and the control-plane.As forwarding elements are ously mentioned,the IETF's Forwarding and Control Element controlled by an open interface,it is important that this link Separation (ForCES)Working Group [1]has been working remains available and secure. on standardizing mechanisms,interfaces,and protocols aim- The OpenFlow protocol can be viewed as one possible im-ing at the centralization of network control and abstraction plementation of controller-switch interactions,as it defines the of network infrastructure.The Open Network Foundation communication between the switching hardware and a network (ONF)[3]has been trying to standardize the OpenFlow1624 IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 16, NO. 3, THIRD QUARTER 2014 the controller would have to make a decision for each arriving packet. When controlling individual flows, the decision made for the first packet of the flow can be ap￾plied to all subsequent packets of that flow. The overhead may be further reduced by grouping flows together, such as all traffic between two hosts, and performing control decisions on the aggregated flows. • Reactive vs. Proactive Policies Under a reactive control model, such as the one proposed by Ethane [20], forwarding elements must consult a controller each time a decision must be made, such as when a packet from a new flow reaches a switch. In the case of flow-based control granularity, there will be a small performance delay as the first packet of each new flow is forwarded to the controller for decision (e.g., forward or drop), after which future packets within that flow will travel at line rate within the forwarding hardware. While the delay incurred by the first-packet may be negligible in many cases, it may be a concern if the controller is geographically remote (though this can be mitigated by physically distributing the controller [45]) or if most flows are short-lived, such as single-packet flows. There are also some scalability issues in larger networks, as the controller must be able to handle a larger volume of new flow requests. Alternatively, proactive control approaches push policy rules from the controller to the switches. A good example of proactive control is DIFANE [35], which partitions rules over a hierarchy of switches, such that the controller rarely needs to be consulted about new flows and traffic is kept within the data-plane. In their experiments, DIFANE reduces first-packet delay from a 10ms average round-trip time (RTT) with a centralized NOX controller to a 0.4ms average RTT for new single-packet flows. It was also shown to increase the new flow throughput, as the tested version of NOX achieved a peak of 50,000 single-packet flows per second while the DIFANE solution achieved 800,000 single-packet flows per second. Interestingly, it was observed that the OpenFlow switch’s local controller implementation becomes a bottleneck before the central NOX controller. This was attributed to the fact that com￾mercial OpenFlow switch implementations were limited to sending 60-330 new flows requests per second at the time of their publication (2010). As shown in Figure 5, a controller that acts as a network operating system must implement at least two interfaces: a “southbound” interface that allows switches to communicate with the controller and a “northbound” interface that presents an API to network control and high-level applications/services. D. Southbound Communication: Controller-Switch An important aspect of SDNs is the link between the data-plane and the control-plane. As forwarding elements are controlled by an open interface, it is important that this link remains available and secure. The OpenFlow protocol can be viewed as one possible im￾plementation of controller-switch interactions, as it defines the communication between the switching hardware and a network Fig. 5. A controller with a northbound and southbound interface. controller. For security, OpenFlow 1.3.0 provides optional support for encrypted Transport Layer Security (TLS) com￾munication and a certificate exchange between the switches and the controller(s); however, the exact implementation and certificate format is not currently specified. Also outside the scope of the current specification are fine-grained security options regarding scenarios with multiple controllers, as there is no method specified to only grant partial access permissions to an authorized controller. We examine OpenFlow controller implementation options in greater detail in Section IV. E. Northbound Communication: Controller-Service External management systems or network services may wish to extract information about the underlying network or control an aspect of network behavior or policy. Additionally, controllers may find it necessary to communicate with each other for a variety of reasons. For example, an internal control application may need to reserve resources across multiple domains of control or a “primary” controller may need to share policy information with a backup, etc. Unlike controller-switch communication, there is no cur￾rently accepted standard for northbound interactions and they are more likely to be implemented on an ad hoc basis for particular applications. We discuss this further in Section VI. F. Standardization Efforts Recently, several standardization organizations have been turning the spotlights towards SDN. For example, as previ￾ously mentioned, the IETF’s Forwarding and Control Element Separation (ForCES) Working Group [1] has been working on standardizing mechanisms, interfaces, and protocols aim￾ing at the centralization of network control and abstraction of network infrastructure. The Open Network Foundation (ONF) [3] has been trying to standardize the OpenFlow