Democratizing content publication with Coral Michael J. Freedman. Eric Freudenthal. David mazieres New York Universit http://www.scs.cs.nyu.edu/coral/ Abstract stream bandwidth utilization and improve the latency for CoralCDN is a peer-to-peer content distribution network This paper describes CoralCDN, a decentralized, self- performance and meets huge demand, all for the price of orgaNizing, peer-to-peer web-content distribution net- work. CoralCDN leverages the aggregate bandwidth of a cheap broadband Internet connection. Volunteer sites volunteers running the software to absorb and dissipate that run CoralCDN automatically replicate content as most of the traffic for web sites using the system. In so do- a side effect of users accessing it. Publishing through ing, CoralCDN replicates content in proportion to the con- CoralCDN is as simple as making a small change to the tent' s popularity, regardless of the publishers resources- hostname in an object's URL, a peer-to-peer DNS layer in effect democratizing content publication To use CoralCDN, a content publisher-or some cache nodes, which in turn cooperate to minimize load on one posting a link to a high-traffic portal-simply ap- to avoid creating hot spots that might dissuade volunteers pends".nyu. net: 8090"to the hostname in a uRL server One of the systems key goals is Through dNS redirection, oblivious clients with unmod- and hurt performance. It achieves this through Coral, ified web browsers are transparently redirected to nearby a latency-optimized hierarchical indexing infrastructure Coral web caches. These caches cooperate to transfer data based on a novel abstraction called a distributed sloppy from nearby peers whenever possible, minimizing both the load on the origin web server and the end-to-end la 1 Introduction tency experienced by browsers CoralCdN is built on top of a novel key/value indexing The availability of content on the Internet is to a large de- infrastructure called Coral. Two properties make Coral gree a function of the cost shouldered by the publisher. a ideal for CDNs. First, Coral allows nodes to locate nearby well-funded web site can reach huge numbers of people cached copies of web objects without querying more dis- through some combination of load-balanced servers, fast tant nodes. Second, Coral prevents hot spots in the in- network connections, and commercial content distribu- frastructure, even under degenerate loads. For instance tion networks(CDNs). Publishers who cannot afford such if every node repeatedly stores the same key, the rate of amenities are limited in the size of audience and type of requests to the most heavily-loaded machine is still only content they can serve. Moreover, their sites risk sudden logarithmic in the total number of nodes overload following publicity, a phenomenon nicknamed Coral exploits over lay routing techniques recently pop the "Slashdot"effect, after a popular web site that period- ularized by a number of peer-to-peer distributed hash ta- ically links to under-provisioned servers, driving unsus- bles(DHTs). However, Coral differs from DHTs in sev- tainable levels of traffic to them. Thus, even struggling eral ways. First, Corals locality and hot-spot prevention content providers are often forced to expend significant properties are not possible for DHTS. Second, Corals resources on content distribution architecture is based on clusters of well-connected Fortunately, at least with static content, there is an easy chines. Clusters are exposed in the interface to higher- way for popular data to reach many more people than level software, and in fact form a crucial part of the DNS publishers can afford to serve themselves-volunteers can redirection mechanism. Finally, to achieve its goals, Coral mirror the data on their own servers and networks. In- provides weaker consistency than traditional DHTs. For deed, the Internet has a long history of organizations with that reason, we call its indexing abstraction a distributed good network connectivity mirroring data they consider to sloppy hash table, or DSHT be of value. More recently, peer-to-peer file sharing has CoralCDN makes a number of contributions. It enables demonstrated the willingness of even individual broad- people to publish content that they previously could not or band users to dedicate upstream bandwidth to redistribute would not because of distribution costs. It is the first com- content the users themselves enjoy. Additionally, orga- pletely decentralized and self-organizing web-content dis- nizations that mirror popular content reduce their down- tribution network. Coral, the indexing infrastructure,pro-Democratizing content publication with Coral Michael J. Freedman, Eric Freudenthal, David Mazieres ` New York University http://www.scs.cs.nyu.edu/coral/ Abstract CoralCDN is a peer-to-peer content distribution network that allows a user to run a web site that offers high performance and meets huge demand, all for the price of a cheap broadband Internet connection. Volunteer sites that run CoralCDN automatically replicate content as a side effect of users accessing it. Publishing through CoralCDN is as simple as making a small change to the hostname in an object’s URL; a peer-to-peer DNS layer transparently redirects browsers to nearby participating cache nodes, which in turn cooperate to minimize load on the origin web server. One of the system’s key goals is to avoid creating hot spots that might dissuade volunteers and hurt performance. It achieves this through Coral, a latency-optimized hierarchical indexing infrastructure based on a novel abstraction called a distributed sloppy hash table, or DSHT. 1 Introduction The availability of content on the Internet is to a large de￾gree a function of the cost shouldered by the publisher. A well-funded web site can reach huge numbers of people through some combination of load-balanced servers, fast network connections, and commercial content distribu￾tion networks (CDNs). Publishers who cannot afford such amenities are limited in the size of audience and type of content they can serve. Moreover, their sites risk sudden overload following publicity, a phenomenon nicknamed the “Slashdot” effect, after a popular web site that period￾ically links to under-provisioned servers, driving unsus￾tainable levels of traffic to them. Thus, even struggling content providers are often forced to expend significant resources on content distribution. Fortunately, at least with static content, there is an easy way for popular data to reach many more people than publishers can afford to serve themselves—volunteers can mirror the data on their own servers and networks. In￾deed, the Internet has a long history of organizations with good network connectivity mirroring data they consider to be of value. More recently, peer-to-peer file sharing has demonstrated the willingness of even individual broad￾band users to dedicate upstream bandwidth to redistribute content the users themselves enjoy. Additionally, orga￾nizations that mirror popular content reduce their down￾stream bandwidth utilization and improve the latency for local users accessing the mirror. This paper describes CoralCDN, a decentralized, self￾organizing, peer-to-peer web-content distribution net￾work. CoralCDN leverages the aggregate bandwidth of volunteers running the software to absorb and dissipate most of the traffic for web sites using the system. In so do￾ing, CoralCDN replicates content in proportion to the con￾tent’s popularity, regardless of the publisher’sresources— in effect democratizing content publication. To use CoralCDN, a content publisher—or some￾one posting a link to a high-traffic portal—simply ap￾pends “.nyud.net:8090” to the hostname in a URL. Through DNS redirection, oblivious clients with unmod￾ified web browsers are transparently redirected to nearby Coral web caches. These caches cooperate to transfer data from nearby peers whenever possible, minimizing both the load on the origin web server and the end-to-end la￾tency experienced by browsers. CoralCDN is built on top of a novel key/value indexing infrastructure called Coral. Two properties make Coral ideal for CDNs. First, Coral allows nodes to locate nearby cached copies of web objects without querying more dis￾tant nodes. Second, Coral prevents hot spots in the in￾frastructure, even under degenerate loads. For instance, if every node repeatedly stores the same key, the rate of requests to the most heavily-loaded machine is still only logarithmic in the total number of nodes. Coral exploits overlay routing techniques recently pop￾ularized by a number of peer-to-peer distributed hash ta￾bles (DHTs). However, Coral differs from DHTs in sev￾eral ways. First, Coral’s locality and hot-spot prevention properties are not possible for DHTs. Second, Coral’s architecture is based on clusters of well-connected ma￾chines. Clusters are exposed in the interface to higher￾level software, and in fact form a crucial part of the DNS redirection mechanism. Finally, to achieve its goals, Coral provides weaker consistency than traditional DHTs. For that reason, we call its indexing abstraction a distributed sloppy hash table, or DSHT. CoralCDN makes a number of contributions. It enables people to publish content that they previously could not or would not because of distribution costs. It is the first com￾pletely decentralized and self-organizing web-content dis￾tribution network. Coral, the indexing infrastructure, pro- 1