Linux Fu: Literate Regular Expressions

Regular expressions — the things you feed to programs like grep — are a bit like riding a bike. It seems impossible until you learn to do it, and then it’s easy. Part of their bad reputation is because they use a very concise and abbreviated syntax that alarms people. To help people who don’t use regular expressions every day, I created a tool that lets you write them in something a little closer to plain English. Actually, I’ve written several versions of this over the years, but this incarnation that targets grep is the latest. Unlike some previous versions, this time I did it all using Bash.

Those who don’t know regular expressions might freak out when they see something like:

[0-9]{5}(-[0-9]{4})?

How long does it take to figure out what that does? What if you could write that in a more literate way? For example:

digit repeat 5 \

start_group \

   - digit repeat 4 \

end_group optional

Not as fast to type, sure. But you can probably deduce what it does: it reads US Zipcodes.

I’ve found that some of the most popular tools I’ve created over the years are ones that I don’t need myself. I’m sure you’ve had that experience, too. You know how to operate a computer, but you create a menu system for people who don’t and they love it. That’s how it is with this tool. You might not need it, but there’s a good chance you know someone who does. Along the way, the code uses some interesting features of Bash, so even if you don’t want to be verbose with your regular expressions, you might pick up a trick or two.

Tower of Babel

One of the problems is that there isn’t a single form of regular expressions. Every tool has a slightly different flavor with different rules and extensions. For the purposes of this, I’m targeting egrep, although much of it will work in other systems, too. Once you have the idea, it would be easy to extend this for different flavors of regular expressions.

Even grep has some uncommon regular expression elements, so I’m only going to work with a subset of patterns, but they are the ones you tend to use the most. It’s easy to add more exotic ones or even macros that contain multiple regular expression patterns if you decide you want to extend the program.

Tool Chest

There are a few things that are important in our quest for literate regular expressions. The idea is to have a small program that converts our literate text into a regular expression. We can naturally combine this with grep or any tool that needs a regular expression:

egrep $(regx start space zero_or_more digit repeat 5)

The $(...) construct runs the command within and whatever it writes out is placed on the command line. So, for example:

for I in $( mount | cut -d ' ' -f 3 )
do
   echo $I
   if [ -f "$I/mountinfo.txt" ]
   then
     cat "$I/mountinfo.txt"
   fi
done

This contrived example selects every mount point from the mount command and tries to locate and display the mountinfo.txt file.

So the key is to build a regx script that can convert our verbose syntax into regular expressions and then use $() to insert the patterns into the command line.

Another odd Bash tool used a bit in these scripts is the regular expression parameter expansion. For example, if $1="Hackanight" then ${1/night/day} will give you Hackaday.

Quoting

Another tool isn’t really necessary for the regx command, but I wanted to build something you can use instead of employing the $() notation with grep. The problem is you have a script getting arguments and then passing them to another program. When you have spaces, potentially, you have a problem.

If script A has $1="Hack A Day" you can assume the command line used quotes or backslashes to keep that together as one string. But passing it to another program could strip the quotes resulting in the other program seeing three different arguments. In this case, you could pass "$1” and that would be fine. But it isn’t always that simple.

To make litgrep work, you need to know about the Bash shell expansion that quotes a value so the shell can read it again:

VAR="${1@Q}"

In our previous example, VAR would now equal ‘Hack a Day’ (including the single quotes).

Why?

Why is this important? Because litgrep will pick off command line arguments and send them to regx. If you have a space in the middle of an argument, it needs to pass as a whole to regx.

Here’s an example:

litgrep Hack space a space Day space optional -- *.txt

The regx Script

The regx script itself is pretty simple. There are two functions to escape characters because so many special characters are present in a regular expression. The reesc function escapes backslash characters along with other metacharacters. Inside a class (that is, square brackets) there isn’t much quoting. You generally have to arrange the expression correctly. For example, to build a character class that has a dash, it needs to come first or last. I didn’t attempt to rearrange your class, but you could do that in the placeholder reescclass function. You could also use it for some other regular expression variants that have more escaping options.

There are three broad groups of patterns. The majority take no arguments like any_char (.) or end ($). The script uses shift to move these out of the way after processing.

The other groups take one or two arguments such as repeat or range. Those commands do extra shifts to dispose of their arguments Once you have the definitions, the script is almost anti-climatic.

The litgrep Script

The litgrep program is a bit more difficult to follow because it has to ensure that spaces are handled correctly.  The script pulls arguments out until it reads — as an argument and the rest of the command line goes to grep. That is, you can include grep arguments and file names after the –.  If you omit the –, then grep will read from standard input, the same as if you put the — with no file arguments after it.

The ${1@Q} syntax, as described above, makes sure the arguments are quoted properly. Then using eval when setting RELIST puts it back together in the right format to send to egrep.

Motivation

I have had versions of this tool floating around for years. My original version was in C++ and there’s been at least one version for Python inspired by the C version.

A tool like this is certainly handy if you don’t know regular expressions. But, honestly, you should really learn regular expressions. If you want a quick start, there’s a Linux Fu post for that. Or, take your chances and let a program infer your regular expression from a data set.