Regular Expressions aren't for everyone. Regex is a powerful cryptic conflagration of characters which mean something, if only you could figure out what.
But what if there were a handful of regex terms which did 90% of what you needed. Then you could harness the immense power of regex, without having to learn a whole new language. After all, expect-lite is about making it easy.
Regex Whats
Regex is made up of two parts, what to look for, and does that "thing" repeat. The whats are characters which describe a number, a letter or a non-printing character (such as tab). The four terms you will want to know start with a back-slash and are followed by a single letter:
\d is a number
\w is a letter
\n is a new line (think of it as a carriage return)
\t is a tab
Regex Repeats
Repeats are useful for finding a string of numbers, for example 123456. Two terms for the repeats are:
* repeats 0 or more times
+ repeats 1 or more times
Regex matching with expect-lite
With these six terms you can create very useful regex expressions. The following example shows the output of the route command:
$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth0
0.0.0.0 10.1.1.1 0.0.0.0 UG 100 0 0 eth0
You could use the following to match the interface:
>route -n
<eth\d
You could also match the first IP address:
>route -n
<\d+.\d+.\d+.\d+
Using the plus, +, means regex will match any digit repeating 1 or more times. An IP addresses can have 1 to 3 digits per octet (the number between the dots). The repeat makes it easy to match a variable number of digits.
Or even more useful would be to use a dynamic variable to grab the default gateway:
>route -n
<\n0.0.0.0
+$default_gw=(\d+.\d+.\d+.\d+)
By using the new line, the expect line will only match the 0.0.0.0 at the beginning of the line. I'll write more later about how to leverage expect-lite's capture buffer, but in this example, the <\n0.0.0.0 positions expect-lite to capture into a dynamic variable the very next thing that matches the regex \d+.\d+.\d+.\d+ which in this example the value of $default_gw would be 10.1.1.1
The regex OR
The seventh term of regex that is good to know is the OR command which is the vertical line or pipe, |
The pipe allows you to make a statement, such as mach this OR that. In expect-lite it would look like:
>echo "this"
<this|that
The above example is a bit contrived, but it is common to find the output of a command which might be true OR false, or enabled OR disabled, or UP or DOWN. This may be less useful for a simple match, but very useful in capturing a dynamic variable. The following command shows the interface state on a linux machine:
$ ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:30:65:96:b5:4a brd ff:ff:ff:ff:ff:ff
To capture that state of the interface (UP or DOWN) in expect-lite, it would as simple as:
>ip link show eth0
+$intf_state=(UP|DOWN)
By using a simple, yet powerful regex, it is easy to capture the states of the interface in the example above.
The power of 7
I have only scratched the surface of Regex here, but it should cover 90+ percent of what you might need. The 7 simple regex terms here; the whats, the repeats, and OR, it is possible to match just about anything you need in expect-lite.
PS. The above is not entirely correct, as the dot, is also a regex expression, but the above examples will work without having to know this eighth term.
Friday, September 14, 2012
Sunday, August 19, 2012
So you want to be root - calling sudo
Over the years, I have taken several different approaches to using sudo in my scripts. When scripting expect-lite, the script either expects a response or it doesn't. sudo sometimes asks for a password, and other times it does not. Unfortunately, there is no built-in function to support sometimes.
But the need to use sudo remains. It is very useful when writing scripts which need to change file ownership, permissions, insert kernel modules or running tcpdump.
My latest technique of using sudo takes advantage of the sudo option '-S' which frees the script to not care if sudo requires a password.
# sudo password
$pass=secret
# run the ID command with sudo privileges
>echo "$pass" | sudo -S pwd
>sudo id
<uid=0
The sudo option '-S' means get the password from standard in (stdin). By using the echo command, it is easy to put the password on stdin and call sudo, as you can see in the above script. You will also note, that the second time I call sudo, I do not use the echo-technique, since I am expecting sudo to remember the previous authentication.
A note about sudo, the password which is passed to sudo, is in turn passed to the program (pwd in the above script). Not all programs take the password graciously, and in fact will give an error rather than execute as expected (tcpdump is one of these). Therefore, I find doing a seemingly non-related command like pwd is helpful in gaining authentication, and then executing the command I really want to run with sudo privileges.
There is a downside to the echo-technique. The user's password will be printed on the screen. However this can be somewhat mitigated by running the script as a test user (which has sudo privileges) rather than your user id.
Go ahead, be root, it is easy with sudo in expect-lite.
But the need to use sudo remains. It is very useful when writing scripts which need to change file ownership, permissions, insert kernel modules or running tcpdump.
My latest technique of using sudo takes advantage of the sudo option '-S' which frees the script to not care if sudo requires a password.
# sudo password
$pass=secret
# run the ID command with sudo privileges
>echo "$pass" | sudo -S pwd
>sudo id
<uid=0
The sudo option '-S' means get the password from standard in (stdin). By using the echo command, it is easy to put the password on stdin and call sudo, as you can see in the above script. You will also note, that the second time I call sudo, I do not use the echo-technique, since I am expecting sudo to remember the previous authentication.
A note about sudo, the password which is passed to sudo, is in turn passed to the program (pwd in the above script). Not all programs take the password graciously, and in fact will give an error rather than execute as expected (tcpdump is one of these). Therefore, I find doing a seemingly non-related command like pwd is helpful in gaining authentication, and then executing the command I really want to run with sudo privileges.
There is a downside to the echo-technique. The user's password will be printed on the screen. However this can be somewhat mitigated by running the script as a test user (which has sudo privileges) rather than your user id.
Go ahead, be root, it is easy with sudo in expect-lite.
Saturday, August 4, 2012
A tale of two scripts
One of the cool things about expect-lite is that it has a very simple interpreter, which ignores everything it doesn't understand, rather than giving a syntax error. You can paste just about any text into an expect-lite script, such as release notes, network diagrams, or sample output from commands, and have it ignored. This feature can be exploited.
In this post, I will embed a bash script inside an expect-lite script. I'll use the bash script to set up the environment, and then have the script call itself, this time using expect-lite as the interpreter. Since expect-lite ignores anything it doesn't know, it will ignore the embedded bash script.
Here's the script, bash_env.elt (with 755 permissions)
#!/usr/bin/env bash
# Little bootstrap bash script to adjust env then run expect-lite
export EL_CONNECT_USER=user
export EL_CONNECT_PASS=secret
export EL_REMOTE_HOST=10.1.1.15
export EL_CONNECT_METHOD=telnet
# re-run the script as an expect-lite script
/usr/bin/env expect-lite c=$0 $*
# end the bash script, and return expect-lite pass/fail
exit $?
#start the expect-lite script
*EXP_INFO
@3
$myvar=hello world
$user=admin
>echo "$myvar"
<world
>
Run the script:
./bash_env.elt
The script starts out like any bash script, and then sets (and exports) some environment variables which control how expect-lite connects to a remote host. Then the bash starts expect-lite feeding it the script name ($0) and any parameters that might have been typed on the command line ($*). Finally, the bash script exits (exit $?) preventing bash from reading the rest of the script.
The final part of the script is expect-lite, which will be executed after expect-lite telnets to 10.1.1.15, and logs in with credentials user and secret.
Of course, you don't want to hard code these parameters for every script. After all, you may use telnet as your connect method today, and change to ssh later.
Embedding other scripts inside expect-lite scripts might just solve your challenging problem.
In this post, I will embed a bash script inside an expect-lite script. I'll use the bash script to set up the environment, and then have the script call itself, this time using expect-lite as the interpreter. Since expect-lite ignores anything it doesn't know, it will ignore the embedded bash script.
Here's the script, bash_env.elt (with 755 permissions)
#!/usr/bin/env bash
# Little bootstrap bash script to adjust env then run expect-lite
export EL_CONNECT_USER=user
export EL_CONNECT_PASS=secret
export EL_REMOTE_HOST=10.1.1.15
export EL_CONNECT_METHOD=telnet
# re-run the script as an expect-lite script
/usr/bin/env expect-lite c=$0 $*
# end the bash script, and return expect-lite pass/fail
exit $?
#start the expect-lite script
*EXP_INFO
@3
$myvar=hello world
$user=admin
>echo "$myvar"
<world
>
Run the script:
./bash_env.elt
The script starts out like any bash script, and then sets (and exports) some environment variables which control how expect-lite connects to a remote host. Then the bash starts expect-lite feeding it the script name ($0) and any parameters that might have been typed on the command line ($*). Finally, the bash script exits (exit $?) preventing bash from reading the rest of the script.
The final part of the script is expect-lite, which will be executed after expect-lite telnets to 10.1.1.15, and logs in with credentials user and secret.
Of course, you don't want to hard code these parameters for every script. After all, you may use telnet as your connect method today, and change to ssh later.
Embedding other scripts inside expect-lite scripts might just solve your challenging problem.
Wednesday, July 25, 2012
The Power of Constants
Expect-lite has an old feature, constants, that when I put it in, I only used it occasionally. Constants are immutable, and override variables in the script. This is handy if you want to change the behaviour of the script without going in and editing the script. For example, assume you have the following script called loop.elt:
$max=10
$i=0
[ $i < $max
>echo "hello world"
<world
+$i
]
Running this script would loop send "hello world" 10 times and check for the word "world" each time. But suppose you want to stress things out by running this 10,000 times. Using a constant on the command line when you start the script will do this quickly and easily.
Note: when using a constant on the command line, don't use the dollar sign ($), it tends to confuse the shell (such as bash).
Of course, you can get into trouble by trying to set a constant for $i, don't do this!
You will end up with an infinite loop, because $i has been defined as a constant, and will always be 9, it can never increment, and never be larger than $max (10). If you wait long enough, you will discover another feature of expect-lite, infinite loop protection. After 5000 iterations (default value), infinite loop protection will kick in and stop the script and print:
When creating loops, another trick you may want to use is to begin your script with another variable called $start:
$start=0
$max=10
$i=$start
...
This addition allows you to define $start as a constant, now allowing control via constants of when the loop starts and when it stops (via $max). If for example, you wanted to run the script over the range 3-7, start the script as follows:
In this post, I have shown how it is possible to override counting variables with constants, but constants are not limited to integers. You can override any variable in your script such as usernames and passwords, or even expected values! Anywhere a variable can be used in expect-lite, it can be overridden by a constant.
Recently, I have begun to use this feature quite a bit. So much so, that I lose track of what variables are in the script, which could be used as constants. So as of version 4.3.0, I added a help feature which lists the variables in the script in "constant format" (without the dollar sign), making it easy to copy and paste into the command line. For example, typing loop.elt -h will print:
Displaying assigned variables, which can be overridden with
a Constant on command line e.g var=value
i=$start
max=10
start=0
Constants make it a simple matter to override variables, making your scripts much more flexible, and powerful.
$max=10
$i=0
[ $i < $max
>echo "hello world"
<world
+$i
]
Running this script would loop send "hello world" 10 times and check for the word "world" each time. But suppose you want to stress things out by running this 10,000 times. Using a constant on the command line when you start the script will do this quickly and easily.
loop.elt max=10000
Note: when using a constant on the command line, don't use the dollar sign ($), it tends to confuse the shell (such as bash).
Of course, you can get into trouble by trying to set a constant for $i, don't do this!
loop.elt i=9
You will end up with an infinite loop, because $i has been defined as a constant, and will always be 9, it can never increment, and never be larger than $max (10). If you wait long enough, you will discover another feature of expect-lite, infinite loop protection. After 5000 iterations (default value), infinite loop protection will kick in and stop the script and print:
ERROR: Infinite Loop Detected!
When creating loops, another trick you may want to use is to begin your script with another variable called $start:
$start=0
$max=10
$i=$start
...
This addition allows you to define $start as a constant, now allowing control via constants of when the loop starts and when it stops (via $max). If for example, you wanted to run the script over the range 3-7, start the script as follows:
loop.elt start=3 max=7
In this post, I have shown how it is possible to override counting variables with constants, but constants are not limited to integers. You can override any variable in your script such as usernames and passwords, or even expected values! Anywhere a variable can be used in expect-lite, it can be overridden by a constant.
Recently, I have begun to use this feature quite a bit. So much so, that I lose track of what variables are in the script, which could be used as constants. So as of version 4.3.0, I added a help feature which lists the variables in the script in "constant format" (without the dollar sign), making it easy to copy and paste into the command line. For example, typing loop.elt -h will print:
Displaying assigned variables, which can be overridden with
a Constant on command line e.g var=value
i=$start
max=10
start=0
Constants make it a simple matter to override variables, making your scripts much more flexible, and powerful.
Monday, July 23, 2012
Expect More
Hi All,
This is a blog dedicated to making expect-lite even easier. Sure, anyone can use the simplicity of '>' and '<' in expect-lite, but there is much more.
I will be writing about tips and tricks to help people get the most out of expect-lite, all the while keeping it simple. After all, expect-lite is about automation for the rest of us.
Thanks for checking in, and let me know if you have ideas or topics you want to discuss.
Craig...
[author and maintainer of expect-lite]
This is a blog dedicated to making expect-lite even easier. Sure, anyone can use the simplicity of '>' and '<' in expect-lite, but there is much more.
I will be writing about tips and tricks to help people get the most out of expect-lite, all the while keeping it simple. After all, expect-lite is about automation for the rest of us.
Thanks for checking in, and let me know if you have ideas or topics you want to discuss.
Craig...
[author and maintainer of expect-lite]
Subscribe to:
Posts (Atom)