《Structure and Interpretation of Computer Programs》web.pdf_大学文库

MASSACHVSETTS INSTITVTE OF TECHNOLOGY Depart ment of Electrical Engineering and Computer Science 6. 001-Structure and Interpret at ion of Computer Programs Spring Semester, 2005 Issued: Tuesday, March 15, 2005 Solut ions due on online tutor: Friday, April 1, 2005 by 6: 00 PM Crawling and Indexing the World wide Web This project explores some issues t hat arise in constructing a"spider"or a"web agent"that craw ls over document s in t he World wide Web. For purposes of this project, the Web is an extremely large collect ion of do cuments. each document cont ains some text and also links to ot her documents. in the form of urls In this project, we'll be working wit h programs that can start wit h an initial document and follow t he references to ot her document s to do useful things. For example, we could construct an index of all the words occurring in do cuments, and make thi ilable to people looking for informat ion on the web(as do many of the search engines on the web, such as Google or Yahoo Just in case you arent fluent wit h the det ails of Http, Urls, Uris Html, Xml, Xsl, Htt NG, DOM, and the rest of the alphabet soup that makes up the technical det ails of the Web, heres a simplified version of what goes on behind the scenes 1. The Web consists of a very large number of things called document s, ident ified by names called URLs(Uniform Resource Locators). For example, the oCw home page has the URL Urlhttp://ocw.mit.edu/.ThefirstportionofaUrl(Http://)revealsthenameofa protocol (in this case hypertext transmission protocol or Http) That can be used to fetch the do cument, and the rest of the URL cont ains informat ion needed by the protocol to specify which do cument is intended.(A protocol is a particular set of rules for how to communicate By using the Http protocol a program(most commonly a browser but any program can do this"web agent s"and spiders are examples of such programs that aren't browsers canretrieveadocumentwhoseUrlstartswithhttp.Thedocumentisreturnedtothe program, along wit h informat ion about how it is encoded, for example, ASCIi or Unico de text, HTML, images in gif or JPG or MPEG or PNG or some other format, an Excel or et, etC. 3. Document s enco ded in HTML(Hy per Text Markup Language)form can cont ain a mixture of text, images format ting informat ion, and links to ot her document s. Thus, when a browser (or ot her program) gets an HTML document, it can extract the links from it, yielding URL: for ot her document s in the Web. If these are in HTML format t hen t hey too can be retrieved and will yield yet more links and so on 4. A spi der is a program that st arts wit h an initial set of URLs, retrieves the corresponding documents, adds the links from these documents to the set of URLs and keeps on going Every time it retrieves a do cument, it does some(hopefully useful) work in addit ion to jus finding the embedded link

!""# $% %& '""( %%) %*& + (& '""( % ) *& & '""( ,* )"" $ +% - .% % %%% + % % /%0 /1, 0 + 1% 2 % + 3 3 3,! %% +% -& + 3, % .* %! + % % . % 4% + %& + 56 %! +% -& 17 , 14 1+ % + % 1+ 1 + % + % % +%! .& 1 % . + 1% %& 4 +% 2, 4 + 1, 8% * + %+ % + 1,& %+ % +9! :% % * 7 ; 1+ + % $& 56 %& 56 %& & 2% 1+ % ,+ + %%) ! + 3, %%% 2* , +% %& > ,* % 56 % 85 6% %9! .& + + +% + 56 ! + >% 56 89 2% + 8 +% % +*. %%% & $9 + , % + + & + % + 56 % ,* + %* 1++ % ! 8 % % % +1 !9 '! ?* % + $ & 8% * ,1% , * +%# /1, %0 %% .% %+ % + 7 ,1%%9 2 1+% 56 %% 1+ ! + % + & 1+ , +1 % & .& 5 .& & % :$ $ $ % + & . % %%+& ! @! % 8*. 4 9 . .& %& & 4% + %! +%& 1+ ,1% 8 + 9 % & . + 4% & * 56 % + % + 3,! +% & + +* , 2 1 * * 4%& % ! A! % + %% 1+ % 56 %& 2% + % %& % + 4% +% % + % 56 % 4% ! 2* 2% & % % 8+* %9 14 -% > + , 4%! OCW URL http://ocw.mit.edu/

Hlooject sing cemeste 3e 2oo5--P3o ject 3 5. xne particularly interest ing kind of spider constructs an 2VSCEof the document s it has seen dhis index is similar to the index at the end of a book: it has certain key words and phrases, and for each entry it lists all of the URhs that cont ain that word or phrase. dhere are many kinds of indexes, and the art /science of deciding what words or phrases to index and how to extract them is at the cutting edge of research(it's part of the discipline called VTENCHLV HFFACLCC We'll talk most ly about Y(GIFF ASCEVD which means that every word in the document( except, perhaps, the very common words like“and,”“th and“an”)is In this project, we'll be interested in three tasks related to searching the World Wide Web. t irst e will develop a way to think about the web"of links as a directed graph. wecond, we will build procedures to help in traversing or searching through graphs such as the Web. dhird, we will consider ways to build an index for some set of web pages to support fast retrieval of URhs that contain a given word. Directed graphs dhe essence of the Web, for the purpose of underst anding the process of searching, is captured by a formal abstraction called a SHeIG IECMS. a graph(like the one in t igure O, consists of VI SC1 and (IC1. In this figure, the nodes are labelled U through Z. eodes are connected to other no des via (SiCL In a directed graph, each edge has a direct on so that the existence of an I YEII /D (SIC from one node to anot her node(e. g. nodeX to node o) does not imply that there is an edge in the reverse direct ion(e.g. from node o to node X). e otice that there can be mult iple out going edges from a no de as well as mult iple iVel NAvDedges to a node, e.g. there are edges from both o and Z to W. dhe set of nodes reachable via a single out going edge from a given node is referred to as t he node,s ep SiV. tor example, the children of node W are nodes U and X. hast ly, a graph aid to cont ain a cy cle if you st art from some node and manage to ret urn to that same node after averring one or more edges. wo for example, the nodes w, X and o form a cycle, as does the node n by it self a second example of a directed graph is shown in t igure 2. dhis particular directed graph happens to be a tree: each node is pointed to by only one ot her node and thus there is no sharing of no des, and there are no cycles(or loops) In order to traverse a directed graph, let' s assume that we have two selectors for getting informat ion from the graph s(find-node-children IICM VISo ret urns a list of the nodes in IiMp that can be reached in one step by out bound edges from visC tor example, in t igure 2 the children of node B are I, b, g, and i -things that can be reached in one hop by an out going edge s(find-node-content s Iiap VISO ret urns t he contents of the node. tor example, when we represent the web as a graph, we will want the node content s to be an alphabet ized list of all of the words occurring in the document at VI SC eote, we have not said anyt hing yet about the act ual represent at ion of a graph, a no de, or an edge We are simply st at ing an abstract definition of a dat a structure

6.001, Spring Semester, 2005--Pro ject 3 1.1 The Web as a graph The Web it self can be thought of as a directed graph in which the nodes are HTML document s and t he edges are hyperlinks to other HTML document s. For example, in Figure 2 the no de labeled B would be a URL, and a directed edge exists between two nodes B and E if the document represented by node B cont ains a link to the do cument represented by node e(as it does in this case) As ment ioned earlier, a web spider (or web crawler) is a program that traverses the web. A web spider might support procedures such as 1 Forj scURLcgrjks web URD ret urns a list of the URLs that are out bound links from URL 1 FoEj SCURLC. Imx. web URD) ret urns an alphabet ized list of all of the words occurring in the document at URL Note, we have not said any thing yet about the act ual representat ion of the web. We are simply st at ing an abstract definition of a dat a struct ure In a real web craw ler, oFj scURLCgFjkS would involve retrieving the document over the network using its URL, parsing the HTML informat ion ret urned by the web server, and extract ing the link information from , and similar tags. Simila a real web craw ler, ForjScURLC Ix. web URD would retrieve the document, discard all of the mark-up commands(such as , , , etc), alphabet ize(and remove duplicates from) the lt ing list of word For this pro ject our programs will not actually go out and retrieve document s over the web. Inst we will represent a collection of web do cuments as a graph as discussed earlier. When you load t he code for t his project, you will have available a global variable,. 6ncwnb, w hich holds the graph represent at ion for a set of do cuments for use in t his project Not e: it is import ant to separate our particular represent ation of informat ion on the web from the idea of the web as a loose collect ion of documents. We are choosing to use a graph to capture a simple version of the web - this is simply to provide us wit h a concrete representat ion of the web, so that we can examine issues related to exploring the web. In pract ice, we would never build an ent ire graph represent ation of the web, we would simply take advant age of the abstraction of conceptualize the struct ure of the web, especially since it is a dynamic thing that const antly ch Our implement ation of opj scURlcgrjkS and opj ScURLC. mx. will simply use the graph pro cedures to get web links(children)and web page contents define(find-URL-links web url (find-node-children web url)) (define (find-URL-text web url) (find-node-contents web url)) In ot her words, we are convert ing operat ions that would normally apply to the web itself into operat ions t hat work on the internal represent ation of t he web as a graph

A + 3, % , ++ % + 1++ + % % + % +*4% + %! .& ' + , ? 1 , 56 & .%% ,1 1 % ? + % ,* ? % 4 + % ,* 8% % +% %9! % & 1, % 8 1, 19 % + 2%% + 1,! 1, % + % % %+ %) % % + 56 % + , 4% ! % +,E % + 1% + ! & 1 +2 % *+ * , + % + 1,! 3 %* % ,% > %! 1, 1& 1 22 2 + 2 + 14 % % 56 & % + ,* + 1, %2& . + 4 !"& # !" % %! *& 1, 1& 1 2 + & % + 4= % 8%+ % $ %"& "& "& !9& +,E 8 2 % 9 + .& + % % 1%! +% - % 1 * 2 % 2 + 1,! %& 1 1 % 1, % % % %%% ! 3+ * + +% -& * 1 +2 2, , 2,& &$& 1++ +% + + % % % % +% -! % % % + 1, + + 1, % % %! 3 +% % + % 2% + 1, D +% % %* 2 % 1+ % + 1,& % + 1 . %%% . + 1,! & 1 1 2 , + % + 1,& 1 1 %* 4 2 + ,% E + % + 1,& %* % % * + + %* +%! 1 %* % + + % 1, 4% 8+9 1, %) + 1%& 1 2 % + 1 * * + 1, % % + 14 + % + 1, % +!

6.001, Spring Semester, 2005--Pro ject 3 Find the children of the specified node in the graph (define(find-node-children graph node) Graph, Node -> listInull (let ((element (find-graph-element graph node))) (if (not (null? (gr aph -element->children element) ))) ind the contents of the specified node in the graph (define(find-node-contents graph node) Graph, N ->contents Inull grap (if (not (null? element)) (graph -element->cont ents In our represent ation above, we use node names(Node symbol) to reference a graph-element in a graph; the children of a no de are represented as a list of other node names. An alternat ive to this approach would be to make the node it self a full abstract dat a type, so that a node object would have ident ity, and the children of a node could be, for example, a list of the act ual children node object s. The tradeoff would be more work in building the graph(e.g. to link toget her actual node objects as nodes and edges are added to a graph), but subst ant ial savings when nodes are requested from the graph(i.e. by avoiding a linear search of the graph-elements for the mat ching node name). With such an alternative abstract ion, when requesting a child no de one can achieve const ant time access(in the size of the graph), as opposed to linear time access as in the current imp lementat ior 2. Searching a Graph How can we search a graph? The basic idea is that we need to st art at some node and traverse the graph in some fashion look ing for some goal. The search might succeed (meaning that some goal is found), or it might fail (meaning that some goal was not found). This very basic and abstract search behav ior can be capt ured in the following procedure search: Node, (Node->Boolean),(Graph, Node - List) Boolean (define ( h initial-state goal' initial-state is the start state of the search hat d reached the goal successors computes from the current state all succes sor states Deterge combines new states with the set of states still to explore ine (s inner still-to-do) (if (null? still-to-do) (let ((current (car still-to-do)) (if *sear ch-debugk

G / & % &, ( & ' / & , ( & ' % ,2& 1 % % 8' ! %$ 9 # # F + + % % % + %! 2 +% + 1 , 4 + % ,% *& % + 1 +2 *& + + ,& .& % + + , -%! + H 1 , 14 , + + 8!! 4 + , -% % % % +9& , %,% %2% 1+ % I% + + 8!! ,* 2 %+ + +=% + + 9! 3+ %+ 2 ,%& 1+ I% + +2 % %% 8 + %E + +9& % % %% % + ! 1 1 %+ +J + ,% % + 1 % % 2% + + % %+ 4 % ! + %+ + % 8 + % % 9& + 8 + % 1% 9! +% 2* ,% ,% %+ ,+2 , + 1 ) $ &0 & % & % & % & & % & & 0 ( ( . ( - 11

6.001, Spring Semester, 2005--Pro ject 3 (write-line (list ' now-at current))) (if (goal? current) (search-inner (merge (successors graph current)(cdr still-to-do))))))) (search-inner (list initial-state))) Note the use of the *search-debug*fag. If we set this global variable to #t, we will see the order in which t he procedure is traversing the graph 2.1 Looking at search What does t his search procedure do? let's look at it a bit more closely. Search takes several arguments. The first is the init ial st ate of the search. For our purposes, this will be a node or in other words, the name of some node in our graph. The second is a goal? procedure, which is applied to a node to determine if we have reached our goal. This procedure will presumably examine some aspect of the node(for example, maybe it want s to see if a particular word is cont ained in t he content s of that node) to decide if the search has reached its termination point. The third procedure for finding the successor s of the node, which in this case basically means finding the hildren of a node in the graph on which we are searching. The fourth is a procedure for combining the children of a node wit h any ot her no des that we still have to search. And the final argument is the graph over which we are searching Looking at the code, you can see t hat we start wit h a list of nodes to search. If the first one meets our goal? criterion, we stop. If not, we get the children of the current no de, and combine them in ome fashion with the ot her nodes in our collect ion to search. This then becomes our new list of odes to consider, and we continue 2.2 Search Strategies There are two common approaches for searching directed graphs, called depth-first search and readth-first search. In a dept h- first search we start at a node, pick one of t he outgoing links from it, explore that link(and all of that links out going links and so on) before ret urning to explore the next link out of our original node. For the graph in Figure 2, that would mean we would examine t he nodes (if we go left-to-right as well as dept h-first in the order: a, b, c, d, e, f, g, h,i,j, k, L and finally m(unless we found our goal earlier, of course). The name "dept h-first "comes from the fact that we go down the graph(in the above drawing) before we go across In a breadt h-first search, we v isit a node and then all of its"siblings "first, before exploring any hildren. For Figure 2, we'd visit the nodes in the order a, b, i, m, c, d, e, h,,k, 45, We can abstract the not ions of dept h-first, breadt h-first, and ot her kinds of searches using the idea of a search strategy. A search strategy will basically come down to what choice we make for how to order the nodes to be explored 2. 3 A Depth-First strategy Here's an init ial at tempt at a depth-first search strategy. It doesnt quite work on all cases, but it's a good place to st art

K ' ( - + % + ( $#( ;! 1 % +% , 2, )& 1 1 % + 1++ + % 2% + +! 3+ % +% %+ J 3& 7% 4 , %*! * 4% %2 %! + >% % + % + %+! %%& +% 1 , ' + 1%& + % +! + % % # + & 1++ % 1 +2 + ! +% 1 %,* . % % + 8 .& *, 1% % 1 % + % + 9 + %+ +% + % ! + + % > + + & 1++ +% % ,%* % > + + + + 1++ 1 %+! + + % , + + 1+ * + % + 1 % +2 %+! + > % + + 2 1++ 1 %+! 4 + & * % + 1 % 1+ % % %+! + >% % # + & 1 %! & 1 + + + & , + % %+ 1+ + + % %+! +% + ,% 1 % % %& 1 ! + 1 +% %+ +%& ! +=>% %+ 1 % & 4 + 4% & . + 4 8 + 47% 4%& % 9 , . + . 4 ! + + '& + 1 1 1 . + % 8 1 ==+ % 1 % +=>%9 + ) & & & & & & & & & & & & >* 8%% 1 & %9! + /+=>%0 % + + 1 1 + + 8 + ,2 19 , 1 %%! ,+=>% %+& 1 2% + % /%,%0 >%& , . * /+!0 '& 17 2% + % + & & & & & & & & & & & & ! 3 ,% + % +=>%& ,+=>%& + 4% %+% % + ! %+ %* 1 ,%* 1 1+ + 1 4 +1 + % , .! ! 7% +=>% %+ %*! %7 I 14 %%& , 7% %!

6.001, Spring Semester, 2005--Pro ject 3 (define (DFS-simple start goal? graph) earch start goal? find-node-children (lambda (new old)(append new old) graph)) And here is an example of using it (lambda (node)(eq? node 1)) est-graph) In this case, we are searching a particular graph test-graph, st arting from a no de wit h name a We are looking for a node named 1(hence our second argument ) We use our graph abstraction to extract the children of a node(i.e. find-node-children). The key element is how we choose to order the set of nodes to be explored. Note the fourt h argument. We can see that this will basically t ake the list of nodes to be explored, and add the new children to the end of the list. This should give us a dept h first search (you should think carefully about why This simple met hod does not work in general, but it does work for the graph in Figure 2.(See arm Up Exercise 2 for some thoughts on why this algorit hm does not work on all graphs. An index abstraction We will also be interested in constructing an index of web pages. To do this, we will first construct a general purpose index abstract ion, and then use it for our purpose of web indexing An Index enables us to associate values wit h keys, and to retrieve those values later on given the key. Here we will assume that a key is a Scheme sy mbol (i.e. Key symbol), and that a value is also a symbol (i.e. Val sy mbol). Our index will be a mutable data struct ure A concrete implement ation for an index is as follows. An Index will be a tagged dat a object holds a list of Index-Entries. Each Index-Entry asso ciates a key wit h a list of values for K ex Pair> ex-Entry Pair , null>> Thus our index implement at ion is shown(partially) below. You will be asked in the exercises to complete the implement at ion. The index implement ation makes use of the Scheme procedure assv; you will find it helpful to consult the Scheme manual as to what t his procedure does (define (make-index) void -> Index (define (index? index) anytype -> boolean (and (pair? index)(eq? 'index (car index))))

L 2/3 ( ( + % . % 2/3 ' )( ' +% %& 1 %+ + # & % 1+ ! 3 4 8+ % 9! 3 % + ,% . + + 8!! 9! + 4* % +1 1 +% + % % , .! + + ! 3 % + +% 1 ,%* 4 + % % , .& + 1 + + + %! +% %+ 2 % + >% %+ 8* %+ +4 * , 1+*9! +% % + % 14 & , % 14 + + '! 8 3 5 .% ' % ++% 1+* +% + % 14 +%!9 3 1 % , % % . 1, %! +%& 1 1 >% % % . ,%& + % % 1, .! , ,% % %% 2% 1+ 4*%& 2 +% 2% 2 + 4*! 1 1 %% + 4* % + %*, 8!! -% ! %$ 9& + 2 % % %*, 8!! . ! %$ 9! . 1 , , %! % % 1%! , 1 , , - + +% % , ! + , % %%% -% 1+ % 2% + -%& !! 4 ! 5%46 %4"&& 4" ! 5%7" 5% %8 & && +% . % %+1 8*9 ,1! 1 , %4 + .%% + ! + . 4% % + + /F * 1 > + % + + % 1+ +% %! . & 4 ' ( "" & ( )( '