Title: 6
Course: Start programming with Python in 10 steps

Start matching with regular expressions

Now that we have made a full tour of the basic programming skills, we dig deeper into working with language and text data. Probably the most common task in working with language data is to find and count instances of a particular phrase or expression in a really large text or a dataset. To do this, we need to be able to formulate flexible search queries that take into account regular variability in language such as verb tenses, case endings, smaller variation in word order, etc. This is what regular expressions are for. This step is all about understanding what they can do for you and how to use them in the most effective ways.

👉    7 things to know at this step

1. Regular expressions are sentences (or phrases) of a really simple formal language whose purpose is to define the set of all and only those strings that belong to a regular language, something like the grammar of natural language, just much, much simpler.
2. The grammar of regular expressions is a general mathematical system, which can be implemented for different purposes in computational technology. This is why you find its elements in many shell programs, many other languages for querying data sets and even in the "Find" function of your text editor.
3. We use regular expressions to describe really small and regular parts of the grammar of natural language, approximating the rules of natural language, which, in reality, are always more complex.
4. Concrete implementations of regular expressions might show slight variations in how some symbols are interpreted, which can have a big impact on what your searches return. This is why, in addition to the general grammar, you need to know some details of the concrete implementation at hand.
5. Python does not understand regular expressions by default. If you want to use them, you need to import a library into your program.
6. How you call programs from a library is partially defined by general rules of Python and partially by specific rules contained in the library. The library-specific rules are described in the documentation of the library - do read it!
7. To import a library, the library has to be installed. A lot of libraries are already installed in your Python (including the one for regular expressions), but if they are not, you can easily install them with Anaconda (not covered in this guide).

🧑‍🎓    Learn actively

▶️   First, get a short introduction to what regular expressions are with this video by Martin Volk:

📹   Regular Expressions by Martin Volk

▶️   For learning the general rules of regular expressions, go through the Regex One lessons:  they are simple, short, interactive and really useful!

📝   Have a regex cheat sheet somewhere handy, e.g. this one.

▶️   To really appreciate the magic of the command line, try a few tricks with "grep". You can get some ideas here:

  - a short help page by Ubuntu
  - GREP command explained on Joseph Delgadillo's YouTube channel (note what is text for him!):  GREP command explained

📝   To see how regular expressions work in Python, read about the re library in the official documentation. The first part will be largely overlapping with Regex One. The most important parts for you come later, check especially the functions.

      ❗  re is a Python library, that is, a collection of programs that you can call from your Python script to perform specific tasks. You can call these programs like you would call built-in functions, as long as you import the library into your script first. To do that, you need to write import re at the top of your file. There are too many Python libraries to count out there, but all of them can be imported in the same way.

      ❗  Note carefully the syntax of the functions: you need to type things in your program exactly as shown in the documentation.

▶️   Try to understand and repeat the steps shown in this Real Python video.

❗  The interface in this video looks like you are in the command line, but you are actually in the Python interactive text interface, which you can recognise by the sign ">>>" before the cursor. This interface is very similar to the code cells in your notebook: you type in the commands and then press "Enter" instead of the "Run" button. To start this interface, you just type "python" on the command line (without calling any script).  Since you already work with notebooks, you don't need to learn more about the interactive interface. This info was just for you to know what is going on in this and many other videos that explain basic Python. Having read this, you can now go back to your Jupyter, open a new notebook and type the code cells as before. Alternatively, you can write a script and run it from the command line as you learned in the previous step.

📝   Regular expressions are awesome, but there are still some things that you can't do with them. Watch this other video by Martin Volk to keep the main limits in mind:

📹   Limits of Regular Expressions by Martin Volk

❗   Finally, when searching for files on your computer or simply using your favorite search engine, you might have heard of (or even already be using) wildcard characters, such as *, which are used as placeholders for a character or word that you don't know. It's very important to know that the * character does not have the same meaning in regular expressions as it does in search requests:

📹   Regular Expressions vs Wildcards by Martin Volk

📝   For more examples and explanations, go through Martin Volk's lecture slides.

❓  Test your skills on an almost real problem

Taking the file that you downloaded in Step 4 as input, find and count all the instances of first-declension nouns in the data.

- How many first-declension instances are there?
- What is the proportion of first-declension instances compared to the whole data set?

Hint 1: First declension endings are "-a" and "-ae"

Hint 2: Import the re library, use an if-statement, use variable updating

Remember: