Writing web crawlers is one of the most interesting parts of my job. The reason I love it so much is because it is both simple and complex. Simple because the basic idea is as easy as following these four steps:
- Send a GET request to the URL
- Extract the content from the response.
- Extract any urls that can be crawled next
- Repeat
The steps above hold true when we’re crawling on a single thread and sending a single request at a time. The problem here is that although this works efficiently with smaller websites, it would take ages to crawl bigger ones. Not to mention it would be a waste of computing power, network resources, and I/O operations. Computers nowadays can perform multiple network requests at the same time; processing data and performing I/O operations in parallel.
To illustrate this, think of the above steps as a ‘relay race’ with one race track (the computer), and one team of 4 runners using just one lane. The first runner finishes sending the ‘GET’ request, the next receives the response then extracts the content, and so on. But, doesn’t this seems like we’re wasting the race track? Why not have other teams join in the race simultaneously?
This example might not fit in perfectly with what happens in a real-life multi-threaded application, I believe the basic principles still hold true. We need more ‘teams’ making better use of the race track (the computer).
In computer programs this is done through threads, which if we go back to our race track example, one can think of as ‘track lanes’. Similarly to a track lane which is allocated to a given team in a race, a thread is basically a set of computer resources (memory, I/O, etc) allocated to execute a task without affecting the execution of other tasks.
Most programming languages support multi-threading (i.e. executing tasks in parallel/concurrently), however, this support comes at the cost of code complexity. Moreover, in most programming languages, writing multi-threaded applications is often cumbersome, requires lots of testing and debugging, a good knowledge of multi-threading issues and how to deal with them. ‘Go’ on the other hand was built with concurrency in mind. It differs from other languages in the fact that it has built-in concurrency features that makes it ideal for applications. I highly recommend you look through the Go language documentation to learn more.
Without going further into the technical detail, I’ll show the code to help get you started when writing your own Go crawler.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 |
package main //import packages import ( "fmt" //GO’s base package "io/ioutil" //reading/writing data from input/output streams "net/http" //for sending HTTP requests "net/url" //for URL formatting "regexp" //regular expressions "runtime" //GO runtime (used to set the number of threads to be used) "strings" //string manipulation and testing ) //how many threads to use within the application const NCPU = 8 //URL filter function definition type filterFunc func(string, Crawler) bool //Our crawler structure definition type Crawler struct { //the base URL of the website being crawled host string //a channel on which the crawler will receive new (unfiltered) URLs to crawl //the crawler will pass everything received from this channel //through the chain of filters we have //and only allowed URLs will be passed to the filteredUrls channel urls chan string //a channel on which the crawler will receive filtered URLs. filteredUrls chan string //a channel //a slice that contains the filters we want to apply on the URLs. filters []filterFunc //a regular expression pointer to the RegExp that will be used to extract the //URLs from each request. re *regexp.Regexp //an integer to track how many URLs have been crawled count int } //starts the crawler //the method starts two GO functions //the first one waits for new URLs as they //get extracted. //the second waits for filtered URLs as they //pass through all the registered filters func (crawler *Crawler) start() { //wait for new URLs to be extracted and passed to the URLs channel. go func() { for n := range crawler.urls { //filter the url go crawler.filter(n) } }() //wait for filtered URLs to arrive through the filteredUrls channel go func() { for s := range crawler.filteredUrls { //print the newly received filtered URL fmt.Println(s) //increment the crawl count crawler.count++ //print crawl count fmt.Println(crawler.count) //start a new GO routine to crawl the filtered URL go crawler.crawl(s) } }() } //given a URL, the method will send an HTTP GET request //extract the response body //extract the URLs from the body func (crawler *Crawler) crawl(url string) { //send http request resp, err := http.Get(url) if err != nil { fmt.Println("An error has occured") fmt.Println(err) } else { defer resp.Body.Close() body, err := ioutil.ReadAll(resp.Body) if err != nil { fmt.Println("Read error has occured") } else { strBody := string(body) crawler.extractUrls(url, strBody) } } } //adds a new URL filter to the crawler func (crawler *Crawler) addFilter(filter filterFunc) Crawler { crawler.filters = append(crawler.filters, filter) return crawler } //stops the crawler by closing both the URLs channel //and the filtered URLs channel func (crawler *Crawler) stop() { close(crawler.urls) close(crawler.filteredUrls) } //given a URL, the method will apply all the filters //on that URL, if and only if, it passes through all //the filters, it will then be passed to the filteredUrls channel func (crawler *Crawler) filter(url string) { temp := false for _, fn := range crawler.filters { temp = fn(url, crawler) if temp != true { return } } crawler.filteredUrls <- url } //given the crawled URL, and its body, the method //will extract the URLs from the body //and generate absolute URLs to be crawled by the //crawler //the extracted URLs will be passed to the URLs channel func (crawler *Crawler) extractUrls(Url, body string) { newUrls := crawler.re.FindAllStringSubmatch(body, -1) u := "" baseUrl, _ := url.Parse(Url) if newUrls != nil { for _, z := range newUrls { u = z[1] ur, err := url.Parse(z[1]) if err == nil { if ur.IsAbs() == true { crawler.urls <- u } else if ur.IsAbs() == false { crawler.urls <- baseUrl.ResolveReference(ur).String() } else if strings.HasPrefix(u, "//") { crawler.urls <- "http:" + u } else if strings.HasPrefix(u, "/") { crawler.urls <- crawler.host + u } else { crawler.urls <- Url + u } } } } } func main() { //set how many processes (threads to use) runtime.GOMAXPROCS(NCPU) //create a new instance of the crawler structure c := Crawler{ "http://www.thesaurus.com/", make(chan string), make(chan string), make([]filterFunc, 0), regexp.MustCompile("(?s)<a[ t]+.*?href="(http.*?)".*?>.*?</a>"), 0, } //add our only filter which makes sure that we are only //crawling internal URLs. c.addFilter(func(Url string, crawler Crawler) bool { return strings.Contains(Url, crawler.host) }).start() c.urls <- c.host var input string fmt.Scanln(&input) } |
Happy crawling! Let us know how you get on.
