Title: 8
Course: Start programming with Python in 10 steps

Web is the limit

A lot of data for linguistic analysis can be found in language corpora. Existing corpora can be really large and diverse, but you will often realise that they do not contain the kinds of texts that you need. In this situation, you will want to collect texts from the Internet yourself. Having come this far with programming, it's out of the question that you copy/paste texts manually. Python can get this data for you much faster and better, you just need to know how to make it talk to web pages. Well, you need to know a few things about the internet too, things like what the internet is for a programmer and how web pages look behind the scenes. With this knowledge, you will be ready to go through many pages and extract what you need.

👉    7 things to know at this step

1. Each web page that you see on the Internet or want to download in your data collection is a file (just like your notebooks, your input files, your Python scripts), this time a special kind of a text file whose name ends in .html.  which you have probably seen in web addresses.
2. HTML files are typically located on servers configured so that they can be publicly accessed. The exact location of a HTML file is described in its URL (Uniform Resource Locator). URLs are like paths on your systems, just that they apply over a much bigger space -- the whole internet.
3. Most of the time, we access and display HTML files using apps that are made for that (e.g. Firefox, Chrome, Safari), but you can download a HTML page from some server on your computer using a Python program and even using the command line.
4. HTML files contain a lot of text, but also some tags (HTML) used to mark the structure of the content and insert images and links to other HTML files. The HTML tags will typically be irrelevant to your tasks, so you will use regular expressions to remove them and keep just the text.
5. Sometimes, links to other pages are also interesting because you want to download them too. In this case too, you use regular expressions, this time to extract the URLs of other pages and pass them to your Python program for further downloads. Going from one page to another in this way is called web crawling.
6. Like with regular expressions, Python has a specialised library that you need to import in order to process HTML files.
7. Typically, you will download a HTML file to your computer, process its content and then save also locally the output of your processing.

🧑‍🎓    Learn actively

📹   Watch these Khan Academy videos to learn about the internet. They will help you get some background for working with web pages. They will resolve many puzzling terms such as protocols, IP address, HTTP, URL, DNS, HTML. These notions are relatively easy to understand and absolutely necessary for you to know, not just for programming but for everyday life.

📝   Read about the Python urllib library and how you can use it to open HTML files. If you don't have access to a server yet, only the first part (Fetching URLs) might be truly relevant to you, but if you do have access to a server, make sure to keep reading.

📝   Get the HTML basics from the w3schools, at least Introduction, Basic, Elements and Attributes.

❓  Download and clean a web page with Python

To try out how this works, write a program that accesses a web page and downloads its contents locally on your computer. Make two versions:

a)  a HTML file, saved as "alice.html"

b)  a plain text file, saved as "alice.txt"

To get you started, use the book Alice's Adventures in Wonderland by Lewis Carroll, available via Project Gutenberg: http://www.gutenberg.org/files/11/11-h/11-h.htm. Note that Project Gutenberg is not available in all countries; if you cannot access it, you can use any other web page.

Hint 1: To access the web page, use urllib.request.

Hint 2: Store the HTML page in a string variable. Note what happens when you use read() and read().decode('utf-8').

Hint 3: To extract plain text, extract the content between the tags <p> </p>; exclude the lines containing other tags inside of the tags <p> </p> (use re.findall), substitute the special HTML Unicode strings (such as &rsquo;, &lsquo;, &ldquo;, &rdquo;, &mdash;) with whitespaces (use re.sub).

Remember: