Simpleton Digest

Trials and tribulations of a professional computer nerd

Monthly Archives: January 2012

Building a better @macro

This is a follow up to a previous post about creating a sort of macro in bash. If you’re interested please take a look at that post for a better introduction to what I’m doing.
After I wrote my last post where I describe how to create a macro system in bash there was one piece that I wasn’t very happy with. That piece is the use of the DEBUG trap. It worked ok, but it’s wasteful since it runs before every simple command. Further, it is wide open to colliding with someone else’s use of it.

Happily, I have found a better way. It also gives me an excuse to go into more detail as to how exactly the @macro alias is constructed. Let me introduce to you the new and improved @macro:

alias @macro='MCMD="${BASH_COMMAND#*#\'\''}" eval '\''eval "$(eval "$MCMD")" #'\'
# or, if you just want to paste it in without using an alias
MCMD="${BASH_COMMAND#*#\'}" eval 'eval "$(eval "$MCMD")" #'

Ok, so obviously there is a decent amount going on here so let’s start from building blocks and work our way up. Before we get started let’s go over what our goal is.

What we want is a @macro command that executes the command that follows it, then captures the output of that command, and then executes that output in the context of the current function. This can be used, for example, to generate argument parsing code or (with some extra code) an @assert macro.

$ # Basic @macro usage
$ echo -e 'if [ "$VAR" == hello ]\nthen\n    echo world\nfi'
if [ "$VAR" == hello ]
    echo world
$ VAR=no
$ @macro echo -e 'if [ "$VAR" == hello ]\nthen\n    echo world\nfi'
$ VAR=hello
$ @macro echo -e 'if [ "$VAR" == hello ]\nthen\n    echo world\nfi'

Ok, on to building @macro.


$ # This code prints itself
$ MCMD="${BASH_COMMAND}" eval 'echo "$MCMD"'
MCMD="${BASH_COMMAND}" eval 'echo "$MCMD"'

This is the core trick of the alias and replaces the use of the DEBUG trap. If you read the part in the previous post about the DEBUG trap, you’ll recognize the $BASH_COMMAND variable. In that post I said that it was set to the text of the command before each call to the DEBUG trap. Well, it turns out it’s set before the execution of every command, DEBUG trap or no (e.g. run ‘echo “this command = $BASH_COMMAND”‘ to see what I’m talking about). By assigning it a variable (just for that line) we capture BASH_COMMAND at the outermost scope of the command, which will contain the entire command.

What about the eval? The reason we need an eval with it’s argument in single quotes is to prevent the evaluation $MCMD until after it’s been set. Normally, variable expansion happens before the command is actually executed:

$ VAR=bad
$ VAR=good echo "$VAR"

However, if you stick your expression in a single quoted eval argument, then the shell expansion won’t happen until the eval is actually executed, which is after the variables are set.

$ VAR=bad
$ VAR=good eval 'echo "$VAR"'

Magic Comment

In the previous @macro snippet which printed itself when executed there was a slight bug that you may have noticed if you played around with it.

$ MCMD="${BASH_COMMAND}" eval 'echo "$MCMD"' 1 2
MCMD="${BASH_COMMAND}" eval 'echo "$MCMD"' 1 2 1 2
$ # Uh oh, arguments are printed multiple times

Looking at the code it’s easy to see why this is happening. First the arguments are being echo’d because they are part of $BASH_COMMAND, but then they are printed again because they are being passed as additional arguments to echo. To solve this we use the same trick developed in the original post and append a comment (‘#’) at the end of the eval’d command.

$ MCMD="${BASH_COMMAND}" eval 'echo "$MCMD" #' 1 2
MCMD="${BASH_COMMAND}" eval 'echo "$MCMD" #' 1 2
$ # Hurray!

One thing to mention about this is that a comment in an eval works differently than a normal comment in one important way: It only applies to the current command.

$ echo hello # blah; echo world
$ eval 'echo hello #' blah; echo world

This means we can safely add a comment to our @macro alias while still allowing macros to be on the same line as other commands.

Stripping the cruft

The next problem we’re faced with is that, in fact, we don’t actually care about most of the BASH_COMMAND. All we really want is everything after @macro, which is going to be the macro function and it’s arguments. In the last post I used the length of the expanded macro definition to strip the front from BASH_COMMAND. This turned out to have a fatal flaw (the specifics of which I will leave as an exercise to the reader but which involves adding variable definitions to the front of a @macro call).

The new method I came up with has the dual advantages of being both safer and simpler.

$ MCMD="${BASH_COMMAND#*#\'}" eval 'echo "$MCMD" #' 1 2
 1 2

That’s it. This uses the ${VAR#PATTERN} syntax which, when evaluated, expands to the value of VAR with the smallest prefix that matches PATTERN removed. Let’s see some examples.

$ VAR="abcdcdef"
$ echo "${VAR#abc}"
$ echo "${VAR#*e}"
$ echo "${VAR#*cd}"

By using a pattern of *#’ (the ‘ must be escaped as \’ in the actual pattern) we strip off everything up to the comment symbol followed by the single quote, which marks the end of @macro alias. We also could have used ${VAR/*#\’} or ${VAR##*#\’} instead, both of which will match the longest prefix instead of the shortest, but that would open us up to bugs in situations like “@macro echo ‘#'” where the #’ pattern occurs in the arguments.

The problem with word splitting (or how I learned to stop worrying and love eval)

We’re very close now. The only remaining piece is how to get from having a string that holds our macro command ($MCMD) and evaluating it’s output. First off, because $MCMD is a string and not an array we have to deal with bash’s annoying word splitting rules.

If you’ve worked with bash at all you know that when you type a command into bash it will split your command into a sequence of “words” based on things like whitespace.

$ echo a     b
a b

You may also know that if you have a variable reference in a command, and it’s not in double quotes, it will also be word split.

$ A='a    b'
$ echo $A
a b
$ echo "$A"
a    b

Ok, so far so good. Here is the fun part though: the rules used to word split in these two situations are completely different. Specifically word splitting of variables is much more limited than when evaluating arguments of a command. This means that all kinds of fanciness that is allowed in command arguments completely breaks if you store the arguments in a variable first.

$ echo 'a    b' c
a    b c
$ # echo treats it as 2 arguments
$ A="'a    b' c"
$ echo $A
'a b' c
$ # echo treats it as 3 separate arguments, blindly separating by whitespace

Not only that, but if $A contained any variable references they wouldn’t be expanded into their values. When taken as a whole, the different splitting rules actually make a certain amount of sense, but in this context they can be quite frustrating.

As it turns out, the only way to evaluate a string as if it was a normal command is to use eval. For our macros, this applies to the contents of $MCMD, but it also applies to the code echo’d back by the macro command. This means we need to add two more eval statements, one for each.

$ MCMD="${BASH_COMMAND#*#\'}" eval 'eval "$(eval "$MCMD")" #' echo 'echo hello'

First we call eval “$MCMD” which evaluates the macro command, then we use $() to take it’s output and pass that output as the argument to the second eval.

Wrapping up

All that’s left is wrapping it up in an alias and trying it out. In this example we create a fairly useless ‘add’ macro command which takes two numbers and then prints ‘$[ NUM1 + NUM2 ]’.

$ alias @macro='MCMD="${BASH_COMMAND#*#\'\''}" eval '\''eval "$(eval "$MCMD")" #'\'
$ add () { echo "echo \$[ $1 + $2 ]"; }
$ @macro add 1 2

As before, all of this can be found in my project in a slightly more robust/featureful form. The (possibly historical) version of that corresponds to what is talked about in this post can be found here.