1/13/2020 1/8 Course Search Engine: Crawling the catalog You must work alone on this assignment. Over the course of the next two assignments, you will build parts of a course search engine. We have...

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 8.8px Menlo; color: #1d6f3f} span.s1 {font-variant-ligatures: no-common-ligatures}

Crawling webpage. needs to know beautifulsoup for python

1/13/2020 1/8 Course Search Engine: Crawling the catalog You must work alone on this assignment. Over the course of the next two assignments, you will build parts of a course search engine. We have put together a simple web application that will use your code to do the work of finding courses that match the user’s criteria. In this part of the assignment, you will build a web crawler that crawls a shadow copy of the college catalog to construct a simple index. The purpose of this assignment is to give you more Python programming ex- perience and to have you work with HTML documents extracted from the web. You will use the index in PA #2. Getting started Before describing the specifics of your task, we will briefly explain how to work with URLs and grab pages from the web. Working with URLs URL stands for uniform resource locator. A URL, for example: has the following format: protocol://site address/path/filename The protocol field (http, in our example) specifies which protocol should be used to interact with the re- source. Common protocols include http, https, and ftp. We will be working with http and https, the hyper- text transport protocols. The site address specifies the host computer name (www), the domain name (classes.cs.uchicago), and the top level domain (edu). The path specifies part of the hierarchical loca- Loading [Contrib]/a11y/accessibility-menu.js Due: January 16th at 6pm http://www.classes.cs.uchicago.edu/archive/2005/winter/10122/pa/index.html We have seeded your repository with a directory for this assignment. To pick it up, change to your cs10122-win-20-username directory (where the string username should be replaced with your user- name), run git pull to make sure that your local copy of the repository is in sync with the server, and then run git pull upstream master to pick up the distribution. https://www.classes.cs.uchicago.edu/archive/2020/winter/30122-1/_images/screenshot.png 1/13/2020 2/8 ber, which we will not be using. Within a web page, a link can refer to an absolute URL (that is, it starts with http:// or https://) or a relative URL, which can be converted into an absolute URL. A relative URL refers to a page with a location that is relative to the current page (much like relative path names in Linux). For example, here is a URL taken from the index page for assignments mentioned above: pa1/index.html The absolute URL for this page is: URLs can use a fragment (denoted by a #) to refer to a specific location (known as an anchor) on a page. For our purposes, we only need the part of the URL that comes before the #. We have provided some useful functions for working with URLs: util.is_absolute_url(url): Takes a URL and returns true if it is an absolute URL and false otherwise. util.convert_if_relative_url(url1, url2): Takes the URL for a page (url1) and a URL found on that page (url2). If url2 is an absolute URL, then this function will just return it. If url2 is a relative URL, the function will return an equivalent absolute URL. The function will return None if it is unable to convert the URL. The following code, for example: would yield: as expected. util.remove_fragment(url): Takes a URL and removes the # and any text following it in the URL. For example, given the relative URL index.html#minorprogramincomputerscience, the function would return index.html. If the URL does not contain a fragment, the function just returns the original URL. Grabbing a web page We will be using requests, a Python package, for making http connections and retrieving documents. We have supplied the following wrapper functions: util.get_request(url): Takes a URL and returns a request object. This function will return None if the request fails. util.read_request(request): Takes a request object and returns a string containing the HTML document retrieved by the request. This function may generate a warning of the form: WARNING:root:Some characters could not be decoded, and were replaced with REPLACELoading [Contrib]/a11y/accessibility-menu.js url1 = "http://www.classes.cs.uchicago.edu/archive/2005/winter/10122/pa/ index.h url2 = "pa1/index.html" util.convert_if_relative_url(url1, url2) http://www.classes.cs.uchicago.edu/archive/2005/winter/10122/pa/pa1/index.html tion (archive/2005/winter/10122) of the desired file (index.html) in the file system on the host com- puter. The path to the archive directory is supplied by the configuration of the web server program run- ning on the host machine. The full path in the department’s file system for the example above is: /stage/classes/archive/2005/winter/10122/pa/index.html. URLs can also include a port num- http://www.classes.cs.uchicago.edu/archive/2005/winter/10122/pa/pa1/index.html 1/13/2020 3/8 You may ignore this warning. util.get_request_url(request): Takes a request object and returns the associated URL. Note that the returned URL may be different than the URL provided to the original call to get_request. This seeming anomaly occurs when the original URL redirects to another URL. Note that your crawler should check for failures (that is, checking if the output of any of these functions is None before proceeding further). Note also that even though the recent lab used a different library to download web pages for this assign- ment, you should follow the above functions. Catalog pages For this assignment, we are interested in three types of HTML tags: a, div, and p. We will use the first to find links to other pages, the second to find sections of the page pertinent to a particular course, and the third to extract actual course titles and descriptions from within the div tags. For example, here are three links from a page in the course catalog:
The University of Chicago
AZ Index

The first specifies a link using an absolute URL. The second specifies a link using a relative URL. And the third specifies a link using a relative URL with a fragment. Course descriptions are enclosed in a div tag of the form