Writing scripts with copy and paste

Sure everyone who has tried to use expect-lite knows it is easy to use if you are just using > and < in the script. But what if you want to do something a bit more tricky, say capture a value from the screen, and make a comparison.

It is a bit contrived, but say for example, you want to check the number of packets received on the eth0 interface is below 100,000 (100 thousand). In expect-lite, it is quite easy to do the comparison using a dynamic variable, a variable which is assigned based on the output of a command at run time. Sometimes it can be tricky to get the regex right to capture the info you want, but I will keep regex simple, and use copy and paste to solve this puzzle.

Setting up the environment

Using el_shell.elt (in the examples directory) makes it easy to try out expect-lite lines using copy and paste between two windows (an editor and terminal window). The first thing you want to do is setup two windows like this:

two window setup

I prefer overlapping windows, but it is up to your preference. The key is that you start an editor (I use nedit) and a terminal (or xterm) and can see them both at the same time. In this exercise, we will write lines of expect-lite code in the editor, copy them, then paste them into the terminal which is running el_shell.elt. Having overlapping windows (rather than full screen) makes it easier to see what is going on.

Now that you have your editor started, go ahead and start el_shell.elt in the terminal by typing:
./el_shell.elt

el_shell.elt is pretty simple, it starts expect-lite, then does an *INTERACT to drop you into the debugger. It may look like you are just at the bash prompt. To confirm you are in the debugger, press ESC then h (in cygwin, press back-quote then h). You should see the following output:
IDE: Help
  Key          Action
  ----        ------
  <esc>s      Step
  <esc>k      sKip next step
  <esc>c      Continue
  <esc>v      show Vars
  <esc>0to9   Show N next lines of script
  <esc>-1to-9 Show N previous lines of script
  ctrl+D      Quit & Exit expect-lite
  <esc>h      this Help

[cvmiller@fedora-arm:~]$

Using el_shell.elt

Now that you have the environment setup, let's capture the interface RX counter. The ifconfig command will display interface counters (and other useful info). We use a simple regex, any digit,  \d, with the modifier of plus, for one or more (see Demystify Regex with 7 simple terms). This will match a contiguous run of numbers. In the editor type the following lines:
>ifconfig eth0
+$eth_rx=(\d+)

Select the 2 lines, copy, then switch to the terminal window, and paste the lines, the terminal output should look something like this:
[cvmiller@fedora-arm:~]$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.1.1.41  netmask 255.255.255.0  broadcast 10.1.1.255
        inet6 2001:470:1d:583:246:b3ff:fe05:cbe5  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::246:b3ff:fe05:cbe5  prefixlen 64  scopeid 0x20<link>
        ether 00:46:b3:05:cb:e5  txqueuelen 1000  (Ethernet)
        RX packets 114320  bytes 24060958 (22.9 MiB)
        RX errors 2  dropped 80  overruns 0  frame 1
        TX packets 20022  bytes 4036177 (3.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[cvmiller@fedora-arm:~]$
Assigned Var:eth_rx=0

You can see that the value assigned to $eth_rx is 0 (zero). Not quite what we were looking for. Using dynamic variable assignment (aka capture), only the part inside the parenthesis will be assigned. Let's switch back to the editor and refine the dynamic assignment statement a bit more:
>ifconfig eth0
+$eth_rx=packets (\d+)

Copy and paste (into the terminal) again. Now we see at the bottom of the terminal (removing the ifconfig output for brevity):
[cvmiller@fedora-arm:~]$
Assigned Var:eth_rx=114320

That is looking better. Let's make the script a bit more versatile by using a static variable for the interface, which could later be overridden with a constant (see The Power of Constants). And suppose we want to ensure that we always assign the RX packets, we could tweak the dynamic assignment statement a bit more in the editor:
$iface=eth0
>ifconfig $iface
+$eth_rx=RX packets (\d+)

Copy and paste (into the terminal) again. With confidence, we see at the bottom of the terminal:
[cvmiller@fedora-arm:~]$
Assigned Var:eth_rx=114320

Now we have a RX packet value we can compare, so let's update the script in the editor a bit more, and another copy and paste into the terminal:
$rx_limit=100000
$iface=eth0
>ifconfig $iface
+$eth_rx=RX packets (\d+)
?if $eth_rx > $rx_limit? [
    ;red RX packets over limit
    *FAIL
] 

The result now looks like: 
Assigned Var:eth_rx=114320

If: expr { "114320" > "100000" } |then   [|result=TRUE

 RX packets over limit

expect-lite directive: *FAIL

Even though the scriptlet failed, el_shell.elt is still running. This is because while in the debugger, one has immunity from failing. After all, it would not be helpful if the debugger quit just because you pasted in some code that failed.

Copy and Paste, a simple technique to create complex solutions

As you can see copying and pasting into the debugger allows you to run arbitrary expect-lite code, and see the results instantly. The debugger is not limited to simple commands, you can paste while loops, directives (like *DEBUG), include files, or other complex code and have it execute right away.

The mantra of expect-lite is keeping it simple. What could be simpler than copy and pasting scriptlets while you automate complex solutions.

PS. to simplify things, I have artificially kept the RX packet counter constant, in real life, it will probably increment
PPS. In Cygwin, you will have use ipconfig rather than ifconfig
PPPS. In Linux/Unix/Mac you need not actually copy and paste, since X has an automatic clipboard, by just selecting the text in the editor, moving the mouse to the terminal and pressing the middle button on the mouse (which will paste the selected text). This makes the overlapping window setup very fast.
PPPPS. the debugger does have a restriction on copy and paste, the first line must not be indented (lines below the first line can be indented).
PPPPPS. el_shell.elt was added in the examples directory in release 4.3.3

Consuming Output, eating tables for lunch

There are times when a table is presented, and you the script writer is interested in only part of the data. The traditional method of parsing tables is to figure out the rows and columns, and assign them to variables, and check the variables for the right values. But this is a lot of work, and not in keeping with expect-lite's credo, keeping it easy.

Take the following example, which can be done on any linux machine (in windows use just the route print command):
$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.1.1.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth0
0.0.0.0         10.1.1.1        0.0.0.0         UG    100    0        0 eth0
Without turning this into a networking lesson, you can see there are rows and columns. You will note on the last row, second column is '10.1.1.1' which is the the default gateway for this machine.

Suppose you are interested in checking that the default gateway is correct, you could script the following:
>route -n
<10.1.1.1
And you would be done. You don't have to worry about which row or column the value of 10.1.1.1 is in, since expect-lite will search the entire 'route -n' output.

But suppose you wanted to capture the default gateway into a dynamic variable, and there was no guarantee that it would always be 10.1.1.1? Now things are a little more tricky, but not too much.
>route -n
<\n0.0.0.0
+$default_gateway=(\d+\.\d+\.\d+\.\d+)

Only took 3 lines, still didn't have to know about rows and columns. The second line, '<\n0.0.0.0' is using a very powerful feature of expect-lite, one I call consuming output.

When a command is sent in expect-lite, and text is returned (like the output of route -n), it is held as a big blob (a buffer) of text. With each '<' or '<<' command, it eats or consumes the blob of text when it finds a match. So in the above 3 line example, line #2 consumes all the text up to and including the 0.0.0.0 on the last line (of the route -n output). This means the first 4 lines of the table are gone, consumed. The very next IP address in the remaining text blob is '10.1.1.1', which is what is assigned to the dynamic variable $default_gateway.

If you want to see text blob consumption at work, run the following:
*DEBUG
>route -n
<0.0.0.0
<0.0.0.0
<0.0.0.0

The output will look a bit chatty, but you will see the text blob (of route -n output) being gobbled by each '0.0.0.0'. I won't copy and paste the whole output here, but to give you and idea, it will look similar to this (abbreviated):
>route -n
...
find<<0.0.0.0>>
  in<<route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.1.1.0        0.0.0.0>>
The first 4 and a half lines (including the 'route -n' line) are consumed

find<<0.0.0.0>>
  in<<         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0>>

The next half and a line are consumed

find<<0.0.0.0>>
  in<<         255.255.0.0     U     1000   0        0 eth0
0.0.0.0>>
The last chunk is consumed, leaving the next thing in the text blob, the default gateway. Each time text is found in the text blob (0.0.0.0 in this example), the top of the text blob is consumed, and no longer available to be found (or expected).

Using consumption can be useful in using the not-expect feature as well. As it says in the documentation, how long does one wait for something to not appear? With expect-lite it is about 100ms, or about 1/10th of a second. But what if the unexpected item comes after 1/10th of a second? You can use consumption.

Suppose you never want to see the default gateway of 192.168.1.1. with the above example and using the not-expect feature, you could script the following:
>route -n
<\n0.0.0.0
-<192.168.1.1
It will only not-expect 192.168.1.1 after it consumes the text blob down to the last line of output.

Another problem with tables is finding the right value, when multiple instances are in the table, such as 0.0.0.0 in route -n output. I use consumption to bookend the value I want. For example, if I want to check that the second line (the one starting with 169.254.0.0) gateway was correct, I could use consumption to find something before and after the desired value, such as:
>route -n
<169.254.0.0
<0.0.0.0
<255.255.0.0

Without having to know which row or column these items are on, it is possible to expect 0.0.0.0 is the gateway for 169.254.0.0. Bookending and consuming output ensures that the correct 0.0.0.0 will be matched.

So go, and fear not ye tables with rows and columns, for expect-lite is here to slay and consume them, making automation quick and easy.

PS. you will remember from "Demystifying Regex" that \n is a new line.
PPS. one could create a "simpler" regex to capture the default gateway, but then you would have to know about character classes, such as:
+$default_gateway=([0-9.]+)
PPPS. Think of a shelf with books, old school but still can be found in the library. The things holding up the books at the end of the shelf are called bookends.

Demystify Regex with 7 simple terms

Regular Expressions aren't for everyone. Regex is a powerful cryptic conflagration of characters which mean something, if only you could figure out what.

But what if there were a handful of regex terms which did 90% of what you needed. Then you could harness the immense power of regex, without having to learn a whole new language. After all, expect-lite is about making it easy.

Regex Whats

Regex is made up of two parts, what to look for, and does that "thing" repeat. The whats are characters which describe a number, a letter or a non-printing character (such as tab). The four terms you will want to know start with a back-slash and are followed by a single letter:
\d  is a number
\w  is a letter
\n  is a new line (think of it as a carriage return)
\t  is a tab

Regex Repeats

Repeats are useful for finding a string of numbers, for example 123456.  Two terms for the repeats are:
*  repeats 0 or more times
+  repeats 1 or more times

Regex matching with expect-lite

With these six terms you can create very useful regex expressions. The following example shows the output of the route command:
$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.1.1.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth0
0.0.0.0         10.1.1.1        0.0.0.0         UG    100    0        0 eth0
You could use the following to match the interface:
>route -n
<eth\d

You could also match the first IP address:
>route -n
<\d+.\d+.\d+.\d+

Using the plus, +, means regex will match any digit repeating 1 or more times. An IP addresses can have 1 to 3 digits per octet (the number between the dots). The repeat makes it easy to match a variable number of digits.

Or even more useful would be to use a dynamic variable to grab the default gateway:
>route -n
<\n0.0.0.0
+$default_gw=(\d+.\d+.\d+.\d+)
 
By using the new line, the expect line will only match the 0.0.0.0 at the beginning of the line. I'll write more later about how to leverage expect-lite's capture buffer, but in this example, the <\n0.0.0.0 positions expect-lite to capture into a dynamic variable the very next thing that matches the regex \d+.\d+.\d+.\d+ which in this example the value of $default_gw would be 10.1.1.1

The regex OR

The seventh term of regex that is good to know is the OR command which is the vertical line or pipe, |
The pipe allows you to make a statement, such as mach this OR that. In expect-lite it would look like:
>echo "this"
<this|that

The above example is a bit contrived, but it is common to find the output of a command which might be true OR false, or enabled OR disabled, or UP or DOWN. This may be less useful for a simple match, but very useful in capturing a dynamic variable. The following command shows the interface state on a linux machine:
$ ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:30:65:96:b5:4a brd ff:ff:ff:ff:ff:ff

To capture that state of the interface (UP or DOWN) in expect-lite, it would as simple as:
>ip link show eth0
+$intf_state=(UP|DOWN)

By using a simple, yet powerful regex, it is easy to capture the states of the interface in the example above.

The power of 7

I have only scratched the surface of Regex here, but it should cover 90+ percent of what you might need.  The 7 simple regex terms here; the whats, the repeats, and OR, it is possible to match just about anything you need in expect-lite.



PS. The above is not entirely correct, as the dot, is also a regex expression, but the above examples will work without having to know this eighth term.

So you want to be root - calling sudo

Over the years, I have taken several different approaches to using sudo in my scripts.  When scripting expect-lite, the script either expects a response or it doesn't. sudo sometimes asks for a password, and other times it does not. Unfortunately, there is no built-in function to support sometimes.

But the need to use sudo remains. It is very useful when writing scripts which need to change file ownership, permissions, insert kernel modules or running tcpdump.

My latest technique of using sudo takes advantage of the sudo option '-S' which frees the script to not care if sudo requires a password.
# sudo password
$pass=secret

# run the ID command with sudo privileges
>echo "$pass" | sudo -S pwd
>sudo id
<uid=0

The sudo option '-S' means get the password from standard in (stdin). By using the echo command, it is easy to put the password on stdin and call sudo, as you can see in the above script. You will also note, that the second time I call sudo, I do not use the echo-technique, since I am expecting sudo to remember the previous authentication.

A note about sudo, the password which is passed to sudo, is in turn passed to the program (pwd in the above script). Not all programs take the password graciously, and in fact will give an error rather than execute as expected (tcpdump is one of these). Therefore, I find doing a seemingly non-related command like pwd is helpful in gaining authentication, and then executing the command I really want to run with sudo privileges.

There is a downside to the echo-technique. The user's password will be printed on the screen. However this can be somewhat mitigated by running the script as a test user (which has sudo privileges) rather than your user id. 

Go ahead, be root, it is easy with sudo in expect-lite.

A tale of two scripts

One of the cool things about expect-lite is that it has a very simple interpreter, which ignores everything it doesn't understand, rather than giving a syntax error. You can paste just about any text into an expect-lite script, such as release notes, network diagrams, or sample output from commands, and have it ignored. This feature can be exploited.

In this post, I will embed a bash script inside an expect-lite script. I'll use the bash script to set up the environment, and then have the script call itself, this time using expect-lite as the interpreter. Since expect-lite ignores anything it doesn't know, it will ignore the embedded bash script.

Here's the script, bash_env.elt (with 755 permissions)
#!/usr/bin/env bash 
# Little bootstrap bash script to adjust env then run expect-lite

export EL_CONNECT_USER=user
export EL_CONNECT_PASS=secret
export EL_REMOTE_HOST=10.1.1.15
export EL_CONNECT_METHOD=telnet

# re-run the script as an expect-lite script
/usr/bin/env expect-lite c=$0 $*

# end the bash script, and return expect-lite pass/fail
exit $?

#start the expect-lite script
*EXP_INFO
@3
$myvar=hello world
$user=admin

>echo "$myvar"
<world
>

Run the script:
./bash_env.elt

The script starts out like any bash script, and then sets (and exports) some environment variables which control how expect-lite connects to a remote host. Then the bash starts expect-lite feeding it the script name ($0) and any parameters that might have been typed on the command line ($*). Finally, the bash script exits (exit $?) preventing bash from reading the rest of the script.

The final part of the script is expect-lite, which will be executed after expect-lite telnets to 10.1.1.15, and logs in with credentials user and secret.

Of course, you don't want to hard code these parameters for every script. After all, you may use telnet as your connect method today, and change to ssh later.

Embedding other scripts inside expect-lite scripts might just solve your challenging problem.

The Power of Constants

Expect-lite has an old feature, constants, that when I put it in, I only used it occasionally. Constants are immutable, and override variables in the script. This is handy if you want to change the behaviour of the script without going in and editing the script. For example, assume you have the following script called loop.elt:
$max=10
$i=0
[ $i < $max
    >echo "hello world"
    <world
    +$i
]

Running this script would loop send "hello world" 10 times and check for the word "world" each time. But suppose you want to stress things out by running this 10,000 times. Using a constant on the command line when you start the script will do this quickly and easily.

loop.elt max=10000

Note: when using a constant on the command line, don't use the dollar sign ($), it tends to confuse the shell (such as bash).

Of course, you can get into trouble by trying to set a constant for $i, don't do this!

loop.elt i=9

You will end up with an infinite loop, because $i has been defined as a constant, and will always be 9, it can never increment, and never be larger than $max (10). If you wait long enough, you will discover another feature of expect-lite, infinite loop protection. After 5000 iterations (default value), infinite loop protection will kick in and stop the script and print:

ERROR: Infinite Loop Detected!

When creating loops, another trick you may want to use is to begin your script with another variable called $start:
$start=0
$max=10
$i=$start
...

This addition allows you to define $start as a constant, now allowing control via constants of when the loop starts and when it stops (via $max). If for example, you wanted to run the script over the range 3-7, start the script as follows:

loop.elt start=3 max=7

In this post, I have shown how it is possible to override counting variables with constants, but constants are not limited to integers. You can override any variable in your script such as usernames and passwords, or even expected values! Anywhere a variable can be used in expect-lite, it can be overridden by a constant.

Recently, I have begun to use this feature quite a bit. So much so, that I lose track of what variables are in the script, which could be used as constants. So as of version 4.3.0, I added a help feature which lists the variables in the script in "constant format" (without the dollar sign), making it easy to copy and paste into the command line. For example, typing loop.elt -h will print:
Displaying assigned variables, which can be overridden with
a Constant on command line e.g var=value
        i=$start
        max=10
        start=0

Constants make it a simple matter to override variables, making your scripts much more flexible, and powerful.

Expect More

Hi All,

This is a blog dedicated to making expect-lite even easier. Sure, anyone can use the simplicity of '>' and '<' in expect-lite, but there is much more.

I will be writing about tips and tricks to help people get the most out of expect-lite, all the while keeping it simple. After all, expect-lite is about automation for the rest of us.

Thanks for checking in, and let me know if you have ideas or topics you want to discuss.

Craig...
[author and maintainer of expect-lite]