Introduction to Cluster

Computing:
Shell scripting
Amit Amritkar
 How to edit a file
• Multiple options for command line editors (no GUI
needed) like emacs, nano, vim etc.

• most systems provide vim, and emacs

• vim is easier to learn, emacs is more powerful

• Learn one of the them

• We will use vim and nano for this course

$ vim myfirstfile
Learning Curves of Editors
 Vim editor - Modes
• 3 Modes:

1. Command mode (default) : all

keystrokes are interpreted as
commands

2. Insert mode: most keystrokes are

inserted as text (leaving out those with

modifier keys)

3. Visual mode: helps to visually select

some text, may be seen as a submode
of the the command mode.

• To switch from the insert or visual mode

to the command mode, type <Esc>.
 Vim editor - Movement
- h …move left

- l …move right

- k …move up

- j …move down
 Vim editor - Editing (command mode)
- i … insert after the cursor.
- d …delete the characters from the cursor position up the position given by the next
command (for example d$ deletes all character from the current cursor position up to
the last column of the line).
- c …change the character from the cursor position up to the position indicated by the
next command.
- x …delete the character under the cursor.
- X …delete the character before the cursor (Backspace).
- y …copy the characters from the current cursor position up to the position indicated
by the next command.
- p …paste previous deleted or yanked (copied) text after the current cursor position.
- P …paste previous deleted or yanked (copied) text before the current cursor position.
- r …replace the current character with the newly typed one.
- s …substitute the text from the current cursor position up to the position given by the
next command with the newly typed one.
- . …repeat the last insertion or editing command (x, d, p…).
- Doubling d, c or y operates on the whole line, for example yy copies the whole line.
 Vim editor - More
• /string or ?string Search for the string (forward or backward)

• Don’t be afraid to try the various commands, you can undo almost
anything using u in the command mode, and redo using Ctrl-r.

• Save your work by using :w in the command mode (:w

filename.txt to create a new file)

• Use :q in the command mode to quit

- :q! Ignore changes and quit
- :wq save changes and quit

• vimdiff command to compare two files.

 Exercise
• Create and save a text file on the cluster with the
following text,

• “Hello”

• File name should be “hello.txt”

• Modify the file to say,

• “Hello World!”

• Rename the file to “hello1.txt”

 File names
• File extensions are not necessary. A text file does
not have to end with .txt. Use command file to
check the file type.

• Avoid those special characters in file names 


&;|*?'"`[]()$<>{}#/\!~

• Make your file names meaningful. Use "-", ".", or "_"

as separator instead of white space.

• Use \ to quote a special character if you have to.

For example, if there is a file called in&out.txt,
access it using in\&out.txt.
File/Directory Permissions
• Three user types are associated with file permissions:
r (read), w (write), x (execute)

u (owner), g (group except the owner), o (other except the group and owner)

• User types and file permissions can be seen when we use ls -l

Modify File/Directory Permissions

• Control access to files & directories by setting permissions

• Use the chmod command to change permissions chmod filename

• Options include 3 permission levels (r, w, e) for 3 groups (u, g, o), while
keeping/overwriting the existing file permissions.
chmod u+r file Adds read permission for owner.
chmod ug+w file Adds write permission for owner and group.
chmod o-x file Removes execute permission for other.
chmod (a)+r file Adds read permission for everyone.

• Apply the recursive option -R for directories

chmod -R u+rx directory makes a directory readable
 Set File Permissions
• Sometimes you need to set the file permissions explicitly in
regardless of the existing ones.

chmod u=rwx,g=rx,o= myscript.sh sets the file myscript.sh
with permission -rwxr-x---.

• File permissions can be set using octal notation more concisely.

• “-rwxr-xr-x” = 755
• “-rw-rw-r--” = 664
• “-r-x--------” = 500

• Example:
# chmod 755 dictionary.txt
# chmod -R 755 ../intro2linux/
 Shell Scripts
• Shell script is a file containing a series of commands (including shell scripts!)

• shell reads this file and carries out the commands as if they were entered on
the shell prompt 


• Writing a shell script

✦ Shell scripts are ordinary text files, 


use a text editor to write a series of commands into file

✦ Start with #! /bin/sh to specify which shell to use.

✦ Use # for comments (won't be executed). Recommended to write

comments for your codes (author, time, usage, implementation, etc.)

✦ Add execute permission to the script file

 Shell Script Example
• Use your preferred text editor to create the following script and
store it as script.sh
#! /bin/bash


cd $HOME

tar -xf intro2linux.tgz

tar -cvzf example.tar.gz intro2linux

mkdir dustbin

mv example.tar.gz ./dustbin

cd dustbin

tar -xvf example.tar.gz
mv intro2linux newdir
ls newdir > contents.txt

cd $HOME
exit 0

• then make the file executable changing the permissions. chmod

u+x script.sh

• execute the file with ./script.sh or bash script.sh

 Bash Variables

#!/bin/bash
# Naked variables

echo

# When is a variable "naked", i.e., lacking the '$' in front?

# When it is being assigned, rather than referenced.

# Assignment
a=879
echo "The value of \"a\" is $a."

# Assignment using 'let'

let a=16+5
echo "The value of \"a\" is now $a."

exit 0
Bash Variables … are untyped
• Unlike many other programming languages, Bash does not segregate its variables
by "type." 

• Essentially, Bash variables are character strings, but, depending on context, Bash

permits arithmetic operations and comparisons on variables. The determining
factor is whether the value of a variable contains only digits.

#!/bin/bash
# int-or-string.sh

a=2334 # Integer.
let "a += 1"
echo "a = $a " # a = 2335
echo # Integer, still.

b=${a/23/BB} # Substitute "BB" for "23".

# This transforms $b into a string.
echo "b = $b" # b = BB35
declare -i b # Declaring it an integer doesn't help.
echo "b = $b" # b = BB35
 Positional Parameters
• Make your scripts more functional with positional parameters.

• Max number of parameters is system dependent, usually it is a giant number that

you won't reach. No need to specify how many parameters you will input for the
script.
#!/bin/sh

# pars

echo "There are $# parameters." 

echo "The parameters are $@"

echo "The script name is $0"

echo "The first parameter is $1" 

echo "The second parameter is $2" 

exit 0
$ ./pars apple orange

There are 2 parameters.


The parameters are apple orange 

The script name is ./pars

The first parameter is apple 

The second parameter is orange

• Use ${10} to call the 10th parameter! (recall parameter expansion)

 Exit Status
• Commands (including the scripts and shell functions we write) issue an exit
status to the system when they are terminated.

• An integer in the range of 0 to 255, indicates the execution status of a

command (success or failure). By convention, a value of zero indicates
success and any other value indicates failure. The shell provides a
parameter ($?) that we can use to examine the exit status.

[pzhang5@neches ~]$ ls -d /usr/bin

/usr/bin
[pzhang5@neches ~]$ echo $?
0
[pzhang5@neches ~]$ ls -d /bin/usr
ls: cannot access /bin/usr: No such file or directory
[pzhang5@neches ~]$ echo $?
2
 Exit Status Example 1
#!/bin/sh

# rem

rm junk

echo "The return code from rm was $?"
exit 0

$ touch junk

$ ./rem

The return code from rm was 0

$ ./rem

rm: junk: No such file or directory
The return code from rm was 1
 Flow Control - if…
• The if command makes a decision based on
the exit status of a command.

• Syntax: if commands; then

commands
[elif commands;
then
commands...]
[else
commands]
fi

(where commands is a list of commands)

 true/false command
• The shell provides two extremely simple built-in commands that do
nothing except terminate with either a zero or one exit status.

• The true command always executes successfully and the false

command always executes unsuccessfully:
$ true
$ echo $?
0
$ false
$ echo $?
1

• We can use these commands to see how the if statement works.

The if statement evaluates the success or failure of commands:
$ if true; then echo "It's true."; fi
It's true.
$ if false; then echo "It's true."; fi
 The test command
• The test command is used most often with the if command to perform true/
false decisions.

• The command is unusual in that it has two different syntactic forms:

# Syntax I
test expression
# Syntax II
[ expression ]
SPACE
• If the given expression is true, test exits with a status of zero; otherwise it exits with a
status of 1.

if [ -f ~/.bash_profile ]; then
echo "You have a .bash_profile. Things are fine."
else
echo "Yikes! You have no .bash_profile!"
fi
 Flow Control - for loops
Syntax
for index in [argument list]
do
command(s) ...
done
for i in 1 2 3 4 5 6 7 8 9 10
do
echo -n This is iteration $i
echo -n " and the time is "
date +%T
sleep 1
done
command substitution brace expansion

# loop1.sh # loop1.bash
for i in $(seq 1 10) for i in {1..10}
do do
... ...
done done
 Flow Control - for loops
• Another syntax in C/C++ Style:

#! /bin/bash
# loop2.sh

for ((i=1; i<=10; i++))

do

echo -n This is iteration ”$i”
echo -n " and the time is "
date +%T
done

• There are other loop statements, including while

and until.
 Termination of a for loop
• Use break command to exit a loop.
#! /bin/bash #! /bin/bash
# flow-control-loop1.sh # flow-control-loop2.sh

for i in {1..10} for i in {1..10}

do
 do

echo -n This is iteration ”$i” echo -n This is iteration ”$i”
if [ $i -eq 5 ]; then
break test $i -eq 5 && break
fi
done done
Accessing arrays with for loop @
#!/bin/sh
animals=("a cat" "ate my" "yellow fish")

# one way to access the array animals

for i in "${animals[@]}"
do
echo $i
done

# another way to access the array animals

for ((i=0; i<=2; i++))
do
echo ${animals[$i]}
done
 Functions
• Wherever there is repetitive code, when a task repeats with only
slight variations in procedure, then consider using a function.

• Syntax:

• word “function” is optional

• function-name is the name you use to call the function

• commands comprise the list of commands the function executes

when you call it
 Function Example
#! /bin/bash

# This is a function with no argument

function myscript () {
cd $HOME

tar -xf intro2linux.tar.gz

tar -cvzf example.tar.gz intro2linux

mkdir dustbin

mv example.tar.gz ./dustbin

cd dustbin

tar -xvf example.tar.gz; mv intro2linux newdir
ls newdir > contents.txt

cd $HOME
}

# call the function myscript

myscript
Variable Scope in functions
• Global variables are variables that can be accessed from anywhere in the script regardless of the
scope. In Bash all variables by default are defined as global, even if declared inside the function.

• Local variables can be declared within the function body with the local keyword and can be used
only inside that function. You can have local variables with the same name in different functions.

#!/bin/bash

var1='A'
var2='B'

my_function () {
local var1='C'
var2='D'
echo "Inside function: var1: $var1, var2: $var2"
}

echo "Before executing function: var1: $var1, var2: $var2"

my_function

echo "After executing function: var1: $var1, var2: $var2"

 Some Gotchas
• Never use test as the name of a variable or a shell script
file.

• When using = as an assignment operator, do not put

SPACEs around it.

• When using = as a comparison operator, you must put

SPACEs around it. Note “=“ is the same as “-eq”.

• When using if [ ] put SPACEs around the brackets.

 Final Example
#! /bin/sh
# list names of all files containing given words
if [ $# -eq 0 ]; then
echo "findtext word1 word2 word3 ..."
echo "lists names of files containing all given words"
exit 1
fi

for fyle in *; do
bad=0
for word in $@; do
grep $word $fyle > /dev/null 2> /dev/null
if [ $? -ne 0 ]; then
bad=1
break
fi
done
if [ $bad -eq 0 ]; then echo $fyle; fi
done

exit 0
Shell Scripting Programming Flow
More Unix Commands (useful in shell scripts)

• basename extract file name from path name

• cmp -s to compare files (silently) like diff

• expr (evaluate an expression)

• sleep to suspend execution for given time

• shift -n Shift positional parameters to the left by

n (recall usage of positional parameters in a shell
script)
 Further Reading
• The Linux Command Line: A Complete Introduction
Paperback by Shotts

• Practical Guide to Linux Commands, Editors, and Shell

Programming by Sobell

• Learning the bash Shell: Unix Shell Programming (In a

Nutshell (O'Reilly))

• Free Ebooks

Advanced Bash-Scripting Guide 

http://tldp.org/LDP/abs/html/

Bash Guide for Beginners 


http://tldp.org/LDP/Bash-Beginners-Guide/html/