Intermediate

Linux
Fang (Cherry) Liu, PhD
Scien:c Compu:ng Consultant
PACE, Gatech
fang.liu@oit.gatech.edu

Brief review
Common commands in Linux
cd, ls, le, cat, pwd, mkdir, mv, cp, nd, grep, tail,
head, di, scp, chmod, sort, uniq

I/O and pipe

>, >>, |

Man page, info, --help

Nano, Vi(m), Emacs editor

fang.liu@oit.gatech.edu

Outline
Environment variables and PACE modules
Regular expressions
Advance commands
grep, tar, cut, awk, tr, sed, ps

Bash scrip:ng
Basic building blocks, e.g. for loop, if statements
Example: queue usage analysis

Build packages
fang.liu@oit.gatech.edu

Environment Variables (EVs)

A set of dynamic named values that can aect the way
running processes behave on a computer.
Get list of EVs using command env

fang.liu@oit.gatech.edu

Environment Variables (EVs) (Cont.)

Commonly used EVs:
PATH : a list of directory paths separated with :, e.g.
export PATH={pathA}:{pathB}: this list is checked to
see whether it contains a path that leads to the
command.
HOME : users home directory in the local le system.
LD_LIBRARY_PATH : run:me linker to load code from
these loca:ons, e.g. export
LD_LIBRARY_PATH={pathA}:{pathB}
SHELL : tells what command interpreter is used.
fang.liu@oit.gatech.edu

Environment Variables (EVs) (Cont.)

How to change EVs
On command line: export NAME={content}
In ~/.bashrc: add export statements there, which
ensure the EVs will be set properly every :me
user logs in.
In ~/.pacemodules, a set of module load
statements can be put there to ensure the default
environment sefng.

fang.liu@oit.gatech.edu

PACE Modules
~/.pacemodules setup the default
environment for user
Module load/unload <packages>
Module purge allows the user to clean up the
default environment

fang.liu@oit.gatech.edu

Regular Expressions
Meaning

Example

Match zero, one or more of the previous

Th* matches Thhhh or T

Match zero or of the previous

Th? Matches Th or T

Match one or more of the previous

Th+ matches Tha or Thh

Used to escape a special character

When\? matches When?

Wildcard character, matches any single

character

do.* matches dog, door, dot

()

Group characters

See example for |

[]

Matches a range of characters

[cbf]ar matches car,bar,far

[0-9]+ matches any posi:ve int

Matche previous Or next character/group

(Mon)|(Tues)day matches
Monday or Tuesday

{}

Specied number of occurrences

Beginning of a string

End of a string

fang.liu@oit.gatech.edu

^hvp matches any url [^0-9]

matchs any character not 0-9
Ing$ matches interes:ng

Advance Commands (grep)

Grep with regex (regular expression)

Beginning of the line (^) : grep ^T grep.txt will return all lines
star:ng with T in the le.

fang.liu@oit.gatech.edu

Grep (Cont.)
End of the line ($) : grep 6$ grep.txt will return all lines ending with
6 in the le.
Empty lines (^$): grep c ^$ grep.txt will count all empty lines in the
le
Escaping the special character (\) : grep 127\.0\.0\.1 dFile will return
the line matches 127.0.0.1

fang.liu@oit.gatech.edu

10

Grep (Cont.)
Character class [0-9], [a-z] and [A-Z], grep will match
one out of list characters in the bracket

fang.liu@oit.gatech.edu

11

Advance Commands (tar)

The command tar (tape archive) creates and extracts
archives of le and directories.
.tar is uncompressed archive
.tgz or .tar.gz (zip) or .tbz (bzip2) are compressed archive

Create an archive
tar czf home.tgz home/

Extract an archive
tar tzf home.tgz # look inside the archive without
extrac:ng (-tjf home.tar.bz2)
tar xzf home.tgz
tar xjf home.tar.bz2
fang.liu@oit.gatech.edu

12

Advance Commands (cut)

Cut can extract por:on of text from a le by selec:ng
columns
Select the second column of characters : cut c2 cut.txt
Select the rst 3 characters on each line: cut c1-3 cut.txt

fang.liu@oit.gatech.edu

13

Cut (Cont.)
Using eld delimiter

select the rst eld: cut d; f1 cut.txt

select mul:ple elds: cut d: f2,3 cut.txt

fang.liu@oit.gatech.edu

14

Advance Commands (awk)

AWK is mostly used for pavern scanning and
processing. It searches les to see if they contain
lines that match the specied paverns and then
perform associated ac:ons.
Syntax: awk /search pavern1 {Ac:ons}
/search pavern2/ {Ac:ons} le
Let us create awk.txt to demonstrate usage

fang.liu@oit.gatech.edu

15

Awk (Cont.)
Print every lines : awk {print} awk.txt
Print all columns : awk {print $0} awk.txt
Print all columns but not 3rd column :
awk {$3=;print $0}

fang.liu@oit.gatech.edu

16

Awk (Cont.)
Print the rst column : awk {print $1} awk.txt
Print the third and forth column :
awk {print $3, $4 } awk.txt
awk {print $3 $4} awk.txt

fang.liu@oit.gatech.edu

17

Awk (Cont.)
Print the lines match paver a : awk /a/
awk.txt
Print the lines with more than 18 characters :
awk length($0) > 18 awk.txt

fang.liu@oit.gatech.edu

18

Advance Commands (tr)

tr (translate) is a UNIX u:lity for transla:ng, dele:ng,
or squeezing repeated characters, and it reads from
STDIN and write to STDOUT.
Syntax:
tr [op:on] SET1 [SET2]

Convert lower case to upper case

tr [:lower:] [:upper:]

fang.liu@oit.gatech.edu

19

Tr (Cont.)
Convert braces in a le with parenthesis
tr '{}' '()' < inpuile > outpuile

Convert white space to tab

echo "This is for tes:ng" | tr [:space:] '\t

Squeeze repe::on of characters using s

fang.liu@oit.gatech.edu

20

Advance Commands (sed)

sed is a stream editor used for modifying the
les in Unix (or Linux)
Syntax:

sed ADDRESSs/REGEXP/REPLACEMENT/FLAGS lename

sed PATTERNs/REGEXP/REPLACEMENT/FLAGS lename
s is subs:tute command
/ is a delimiter (one of @ % ; : can be used)
REGEXP is regular expression to match
REPLACEMENT is a value to replace
FLAGS (g replace all; n replace nth instance; i case
insensi:ve match; w le write out result to le)
fang.liu@oit.gatech.edu

21

Sed (Cont.)
Replacing or subs:tu:ng the rst old string to
new in each line of sed.txt : sed s/old/new/
test.txt

fang.liu@oit.gatech.edu

22

Sed (Cont.)
Replacing all occurrence of the pavern in a line : sed 's/old/
new/g sed.txt
Replacing the occurrence on the third line :sed 5s/old/
new/ sed.txt

fang.liu@oit.gatech.edu

23

Sed (Cont.)
Replace old string to new string for all
occurrence, and print out the line if there is
subs:tu:on happened, output the changed lines
to a new le.

fang.liu@oit.gatech.edu

24

Single quote vs double quote

Double Quotes (weak quote)
Use when you want to enclose variables or use
shell expansion inside a string.
All characters within are interpreted as regular
characters except for $ or ` which will be
expanded on the shell : echo My home is
$HOME and echo I am at `pwd`
Using backslash \ to ignore the next character : :
echo My home is \$HOME
fang.liu@oit.gatech.edu

25

Bash Script Building Blocks

The rst line in the script should be always as
#!/bin/bash (bash shell)

Passing arguments to the bash script

echo $1 $2 $3 ' -> echo $1 $2 $3
use $@ to print out all arguments at once

Execu:ng shell commands

echo `hostname`

If statement : if [ condi:on ]; then statements;

else statements;
For loop: for i in `ls`; do echo item: $i; done
fang.liu@oit.gatech.edu

26

Example: Queue usage analysis

The script will process the output from
command `showq w class=queueName |
grep Running`
The data le is shown as follows: [ JobID
UserID JobStatus #core Remaining Starfme]

fang.liu@oit.gatech.edu

27

Example: Queue usage analysis (Cont.)

To nd the number of running jobs per user at
given :me

fang.liu@oit.gatech.edu

28

Example: Queue usage analysis (Cont.)

Make it more exible to accept an input

fang.liu@oit.gatech.edu

29

Example: Queue usage analysis (Cont.)

Add error check to ensure gefng input le

fang.liu@oit.gatech.edu

30

Build a package under users space

Example package: poit is a free
implementa:on of the force-matching
algorithm to generate eec:ve poten:als
from ab-ini:o reference data
Google it online to get source downloaded at
hvp://poit.sourceforge.net/wiki/doku.php?
id=start
Unzip the tar le : tar xjf poit-0.7.1.tar.bz2
fang.liu@oit.gatech.edu

31

Build a package under users space

(cont.)
First to check if there is any Readme le
Reading Makele to nd some dependencies
This package needs MKL and ACML

Check if system already has those packages

module avail mkl
module avail acml

Decide what compiler to use

Change line SYSTEM = $(shell uname -m)-icc to SYSTEM =

$(shell uname -m)-gcc to use 64bit GNU compiler

Load depended modules

module load gcc/4.9.0

module load openmpi/1.8
module load mkl/11.2
module load acml/5.2.0-int64-fma4
fang.liu@oit.gatech.edu

32

Build a package under users space

(cont.)
Iden:fy the environment variables needed in
Makele : MKLDIR/ACML4DIR/ACML5DIR
Check the environment variables :

env |cut d = f1| grep i mkl => MKLROOT

env |cut d = f1| grep I acml => ACMLROOT/
ACMLROOT_MP

Modify Makele

MKLDIR=${MKLROOT}
ACML4DIR=${ACMLROOT}
ACML5DIR=${ACMLROOT}

make poit_mpi_evo_apot_terso_stress_fweight

fang.liu@oit.gatech.edu

33

Build a python package - phonopy

Step 1: check the website for installa:on instruc:ons at

hvp://phonopy.sourceforge.net/install.html
Step 2: download the version you want at
hvp://sourceforge.net/projects/phonopy/les/phonopy/phonopy-1.9/
aer downloading, le phonopy-1.9.7.tar.gz will be in the current directory
Unzip the tarball : tar xzfv phonopy-1.9.7.tar.gz,
a new directory will appear as phonopy-1.9.7
Load the compiler and system python: module load intel/15.0; module load
python/2.7
Check the installa:on op:ons:
python setup.py help (list all possible commands)
Python setup.py help install (list all op:ons when installing phonopy), pay
aven:on to prex, home, user op:ons which allow the package to be built
under users space
Install the package: python setup.py insall user, the installed package will be in
~/.local
Add PYTHONPATH to ~/.bashrc fang.liu@oit.gatech.edu
le

34

Use pip to install python package

pip (Python Package Index) is a package
management system used to install and
manage soware packages wriven in Python
Go to hvps://pypi.python.org/pypi/tabulate
for detail instruc:on for installing
tabulate0.7.5
Module load intel/15.0
Module load python/2.7
pip install tabulate --user
fang.liu@oit.gatech.edu

35

Other Useful commands

Unix2dos/dos2unix : convert les format
between Windows and Unix.
Always use data directory for package build
since it has large storage space and fast access
speed.
Create one directory called Packages to store all
source les
Create another directory called INSTALL to store
all soware installa:ons
fang.liu@oit.gatech.edu

36