Professional Documents
Culture Documents
Ebook Mastering Linux Shell Scripting Second Edition Mokhtar Ebrahim Andrew Mallett 2 Online PDF All Chapter
Ebook Mastering Linux Shell Scripting Second Edition Mokhtar Ebrahim Andrew Mallett 2 Online PDF All Chapter
https://ebookmeta.com/product/mastering-linux-shell-scripting-
second-edition-mokhtar-ebrahim-andrew-mallett/
https://ebookmeta.com/product/linux-shell-scripting-bootcamp-1st-
edition-james-kent-lewis/
https://ebookmeta.com/product/linux-shell-scripting-cookbook-2nd-
edition-shantanu-tushar-sarath-lakshman/
https://ebookmeta.com/product/linux-shell-scripting-cookbook-
quick-answers-to-common-problems-shantanu-tushar/
Linux introduce to beginners guide UNIX operating
system Linux shell scripting and command line Linux
System 1st Edition Erickson Karnel & Coding Hood
https://ebookmeta.com/product/linux-introduce-to-beginners-guide-
unix-operating-system-linux-shell-scripting-and-command-line-
linux-system-1st-edition-erickson-karnel-coding-hood/
https://ebookmeta.com/product/salt-open-automating-your-
enterprise-and-your-network-1st-edition-andrew-mallett/
https://ebookmeta.com/product/salt-open-automating-your-
enterprise-and-your-network-1st-edition-andrew-mallett-2/
https://ebookmeta.com/product/pro-bash-learn-to-script-and-
program-the-gnu-linux-shell-3rd-edition-jayant-varma/
https://ebookmeta.com/product/linux-fundamentals-second-edition-
richard-blum/
Mastering Linux Shell Scripting
Second Edition
Mokhtar Ebrahim
Andrew Mallett
BIRMINGHAM - MUMBAI
Mastering Linux Shell
Scripting Second Edition
Copyright © 2018 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any
form or by any means, without the prior written permission of the publisher, except in the case of brief
quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information
presented. However, the information contained in this book is sold without warranty, either express or implied.
Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages
caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products
mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the
accuracy of this information.
ISBN 978-1-78899-055-4
www.packtpub.com
mapt.io
Mapt is an online digital library that gives you full access to over
5,000 books and videos, as well as industry leading tools to help you
plan your personal development and advance your career. For more
information, please visit our website.
Why subscribe?
Spend less time learning and more time coding with practical
eBooks and Videos from over 4,000 industry professionals
Improve your learning with Skill Plans built especially for you
developers and tech professionals, just like you, to help them share
their insight with the global tech community. You can make a
general application, apply for a specific hot topic that we are
recruiting an author for, or submit your own idea.
Table of Contents
Title Page
Packt Upsell
Why subscribe?
PacktPub.com
Contributors
Conventions used
Get in touch
Reviews
Technical requirements
Command PATH
Configuring vim
Configuring nano
Configuring gedit
Hello Dolly!
Declaring variables
User-defined variables
Environment variables
Variable scope
Command substitution
Summary
Questions
Further reading
Technical requirements
Script comments
Try to be standard
Enhancing learning with simple scripts
Further reading
3. Conditions Attached
Technical requirements
Simple decision paths using command-line lists
Testing strings
Testing integers
Checking strings
Checking files and directories
Checking numbers
Combining tests
Summary
Questions
Further reading
4. Creating Code Snippets
Technical requirements
Abbreviations
Questions
Further reading
5. Alternative Syntax
Technical requirement
Recapping the test command
Testing files
Adding logic
Square brackets as not seen before
Setting defaults
When in doubt – quote!
Advanced tests using [[
White space
Summary
Questions
Further reading
6. Iterating with Loops
Technical requirement
for loops
Advanced for loops
The IFS
Summary
Questions
Further reading
7. Creating Building Blocks with Functions
Technical requirements
Introducing functions
Passing parameters to functions
Passing arrays
Variable scope
Returning values from functions
Recursive functions
Limiting substitution
Editing the file
Other sed commands
The delete command
Summary
Questions
Further reading
9. Automating Apache Virtual Hosts
Technical requirements
Apache name-based Virtual Hosts
Creating the virtual host template
First steps
Isolating lines
sed script files
Automating virtual host creation
Prompting for data during site creation
Summary
Questions
Further reading
10. AWK Fundamentals
Technical requirements
The history behind AWK
Displaying and filtering content from files
AWK variables
User-defined variables
Conditional statements
The if command
while loops
for loops
Formatting output
Further filtering to display users by UID
AWK control files
Built-in functions
Summary
Questions
Further reading
Anchor characters
The dot character
The character class
Ranges of characters
Using grep
Summary
Questions
Further reading
12. Summarizing Logs with AWK
Technical requirements
Resources hits
Identify image hotlinking
Summary
Questions
Further reading
Summary
Questions
Further reading
What is Python?
Supplying arguments
Counting arguments
Significant whitespace
String manipulation
Summary
Questions
Further reading
Assessments
Chapter 1
Chapter 2
Chapter 3
Chapter 4
Chapter 5
Chapter 6
Chapter 7
Chapter 8
Chapter 9
Chapter 10
Chapter 11
Chapter 12
Chapter 13
Chapter 14
Following this, you will learn how to define a variable and the
visibility of a variable. After this, you will learn how to store
command execution output into a variable, which is called command
substitution. Also, you will learn how to debug your code using bash
options and Visual Studio Code. You will learn how to make your
bash script interactive to the user by accepting input from the user
using the read command. Then, you will learn how to read options
and its values if the user passed them to the script. Following this,
you will learn how to write conditional statements such as if
statements and how to use case statements. After this, you will
learn how to create code snippets using vim and Visual Studio Code.
For repetitive tasks, you will see how to write for loops, how to
iterate over simple values, and how to iterate over directory content.
Also, you will learn how to write nested loops. Along with this, you
will write while and until loops. Then, we will move on to functions,
the reusable chunks of code. You will learn how to write functions
and how to use them. After this, you will be introduced to one of the
best tools in Linux, which is Stream Editor. As we are still talking
about text processing, we will introduce AWK, one of the best text
processing tools in Linux that you will ever see.
After this, you will learn how to empower your text processing skills
by writing better regular expressions. Finally, you will be introduced
to Python as an alternative to bash scripting.
Who this book is for
This book targets system administrators and developers who would
like to write a better shell script to automate their work. Some
programming experience is preferable. If you don't have any
background in shell scripting, no problem, the book will discuss
everything from the beginning.
What this book covers
, The What and Why of Scripting with Bash, will introduce
Chapter 1
Linux shells, how to write your first shell script, how to prepare your
editor, how to debug your shell script, and some basic bash
programming, such as declaring variables, variable scope, and
command substitution.
the user using read command, how to pass options to your script,
how to control the visibility of the entered text, and how to limit the
number of entered characters.
Chapter 6, Iterating with Loops, will teach you how to use for loops,
while loops, and until loops to iterate over simple values and complex
values.
engines, and how to use them with sed and AWK to empower your
script.
Chapter 12, Summarizing Logs with AWK, will show how to process the
httpd.conf Apache log file using AWK and extract useful well-formatted
data.
, A Better lastlog with AWK, will show you how to use AWK
Chapter 13
You should know some Linux basics such as the basic commands
such as ls, cd, and which.
Download the example code
files
You can download the example code files for this book from your
account at www.packtpub.com. If you purchased this book elsewhere, you
can visit www.packtpub.com/support and register to have the files emailed
directly to you.
Once the file is downloaded, please make sure that you unzip or
extract the folder using the latest version of:
The code bundle for the book is also hosted on GitHub at https://githu
b.com/PacktPublishing/Mastering-Linux-Shell-Scripting-Second-Edition. In case
We also have other code bundles from our rich catalog of books and
videos available at https://github.com/PacktPublishing/. Check them out!
Download the color images
We also provide a PDF file that has color images of the
screenshots/diagrams used in this book. You can download it from ht
tps://www.packtpub.com/sites/default/files/downloads/MasteringLinuxShellScriptingSec
ondEdition_ColorImages.pdf .
Conventions used
There are a number of text conventions used throughout this book.
Piracy: If you come across any illegal copies of our works in any
form on the Internet, we would be grateful if you would provide us
with the location address or website name. Please contact us at
copyright@packtpub.com with a link to the material.
Like in any other scripting language, variables are the basic blocks of
coding. You will learn how to declare variables such as integers,
strings, and arrays. Furthermore, you will learn how to export these
variables and extend their scope outside the running process.
Finally, you will see how to visually debug your code using Visual
Studio Code.
You can use VS Code as an editor instead of vim and nano; it's up to
you.
Install bashdb, which is a required package for the bash debug plugin.
If you are using a Red Hat-based distribution, you can install it like
this:
$ sudo yum install bashdb
Install the plugin for VS Code, called bash debug, from https://marketpl
ace.visualstudio.com/items?itemName=rogalmic.bash-debug. This plugin will be
https://github.com/PacktPublishing/Mastering-Linux-Shell-Scripting-Second-Edition/tr
ee/master/Chapter01
Types of Linux shells
As you know, Linux consists of some major parts, such as the kernel,
the shell, and the GUI interface (Gnome, KDE, and so on).
The shell translates your commands and sends them to the system.
Most Linux distributions are shipped with many shells.
Every shell has its own features, and some of them are very popular
among developers today. These are some of the popular ones:
Now we know the types of shells and we know that we are going to
use bash, so what is bash scripting?
What is bash scripting?
The basic idea of bash scripting is to execute multiple commands to
automate a specific job.
As you might know, you can run multiple commands from the shell
by separating them with semi colons (;):
ls ; pwd
You can say that the shell is the glue that binds these commands
together.
The bash command hierarchy
When working on the bash shell and when you are sitting
comfortably at your prompt eagerly waiting to type a command, you
will most likely feel that it is a simple matter of typing and hitting the
Enter key. You should know better than to think this, as things are
never quite as simple as we imagine.
Command type
For example, if we type and enter ls to list files, it is reasonable to
think that we were running the command. It is possible, but we
often will be running an alias. Aliases exist in memory as a shortcut
to commands or commands with options; these aliases are used
before we even check for the file. Bash's built-in type command can
come to our aid here. The type command will display the type of
command for a given word entered at the command line. The types
of command are listed as follows:
Alias
Function
Shell built-in
Keyword
File
We can extend this further to display all the matches for the given
command:
$ type -a ls
ls is aliased to 'ls --color=auto'
ls is /bin/ls
If we need to just type in the output, we can use the -t option. This
is useful when we need to test the command type from within a
script and only need the type to be returned. This excludes any
superfluous information, and thus makes it easier for us humans to
read. Consider the following command and output:
$ type -t ls
alias
The output is clear and simple, and is just what a computer or script
requires.
The built-in type can also be used to identify shell keywords such as
if, and case. The following command shows type being used against
You can also see that the function definition is printed when we
stumble across a function when using type.
Command PATH
Linux will check for executables in the PATH environment only when
the full or relative path to the program is supplied. In general, the
current directory is not searched unless it is in the PATH. It is possible
to include our current directory within the PATH by adding the
directory to the PATH variable. This is shown in the following
command example:
$ export PATH=$PATH:.
This appends the current directory to the value of the PATH variable;
each item in the PATH is separated using a colon. Now your PATH has
been updated to include the current working directory and, each
time you change directories, the scripts can be executed easily. In
general, organizing scripts into a structured directory hierarchy is
probably a great idea. Consider creating a subdirectory called bin
within your home directory and add the scripts into that folder.
Adding $HOME/bin to your PATH variable will enable you to find the scripts
by name and without the file path.
Also, we will use Visual Studio Code as a modern GUI editor to edit
and debug our scripts.
searched for
set autoindent: We indent our code often; this allows us to return
to the last indent level rather than the start of a new line on
each line break
set tabstop=4: Sets a tab to be four spaces
When these options are set, the $HOME/.vimrc file should look similar to
this:
set showmode
set nohlsearch
set autoindent
set tabstop=4
set expandtab
syntax on
Configuring nano
The nano text editor is increasing in importance and it is the default
editor in many systems. Personally, I don't like the navigation or the
lack of navigation features that it has. It can be customized in the
same way as vim. This time, we will edit the $HOME/.nanorc file. Your
edited file should look something like the following:
set autoindent
set tabsize 4
include /usr/share/nano/sh.nanorc
The history of our knowledge of the Cestodes dates back to ancient times, as the
presence and effects of tape-worms early attracted the attention of physicians.
Trematodes are first distinctly referred to in the sixteenth century, while Turbellaria
first figure in Trembley's memoir on Hydra (1744).[3] The whole subject of the
increase in our knowledge of parasitic Platyhelminthes is dealt with in the standard
work, The Parasites of Man, by Leuckart,[4] and a complete list of references in
zoological literature to Cestodes and Trematodes is to be found in Bronn's
Thierreich.[5] O. F. Müller[6] and Ehrenberg founded our knowledge of the
Turbellaria, but for a long time the group remained in a most neglected condition. In
this country Montagu, G. Johnston, and in Ireland, William Thompson, discovered
several marine species, one of which, Planocera folium (from Berwick), has not
again been met with on British shores. Dalyell[7] conducted classical researches on
the habits of Planarians, and Faraday[8] made interesting experiments on their
power of regenerating lost parts. The credit of assigning the correct interpretation
to most of the various organs of fresh-water Planarians belongs to von Baer[9] and
Dugès,[10] while Mertens[11] effected a similar service for the marine forms, or
Polyclads. The minute Rhabdocoels were first successfully investigated and
classified by Oscar Schmidt.[12] The great work on this group is, however, the
monograph by von Graff.[13] A similarly comprehensive and indispensable treatise
by Lang, on the Polycladida,[14] contains references to all previous publications on
the group, among which the papers by Quatrefages, Johannes Müller, Keferstein,
Minot, and Hallez stand out conspicuously. Moseley's work[15] on the Land
Planarians of Ceylon is undoubtedly the most revolutionary paper referring to this
group, and the best contribution towards elucidating the structure of the Tricladida
at a time when the subject was very obscure. A monograph on Land Planarians is
being prepared by von Graff.
The Turbellaria are divided into: (1) Polycladida, marine forms with multiple
intestinal branches; (2) Tricladida, marine, fresh-water, and terrestrial Planarians
with three main intestinal branches; (3) the Rhabdocoelida, as varied in habit as
the Triclads, but possessing a straight and simple or slightly lobed, intestine. A
detailed description of an example of the Polyclads, and then a comparative
account of each division, will now be given.
Turbellaria. I. Polycladida.
Fig. 2.—Leptoplana tremellaris. Three-quarters view from the ventral surface. The
pharynx (ph) is widely protruded through the month (mo) as in the act of
attacking prey. br, Brain with nerves, close to which are the four groups of eyes;
mg, stomach; mgc, "marginal groove"; pe, penis; sc, sucker; ut, uterus; vd, vasa
deferentia; ♀ , female genital aperture surrounded by the shell-gland; ♂ , male
aperture. (Semi-diagrammatic, and × 6.)
At low water Leptoplana may be found buried in mud or on the under surface of
stones, in pools where darkness and dampness may be ensured till the return of
the tide. It is, however, by no means easy to detect and remove it from the
encrusting Polyzoa, Ascidians, or Sponges with which it is usually associated. The
flat, soft, unsegmented body is so closely appressed to the substratum that its
presence is usually only betrayed by its movement, an even gliding motion of the
mobile body, which suggested the apt name "la pellicule animée" to Dicquemare.
The creeping surface is called ventral, the upper one dorsal, and as the broader
end of the body always goes first, it is anterior as opposed to the more pointed
posterior extremity. With a lens the characters shown in Figs. 1 and 2 may be
observed. The eyes are seen as black dots near the anterior end, and are placed
at the sides of a clear oval space, the brain. Along the transparent margin of the
body, the ends of the intestinal branches may be seen. These ramify from a lobed
stomach or main-gut, and should the specimen be mature, the "uterus" loaded with
eggs forms a dark margin round the latter (Figs. 1 and 2, ut). The ventral surface is
whitish, and through it the "pharynx," a frilled protrusible structure, may be dimly
observed. The "mouth,"[16] through which the pharynx at the time of feeding is
thrust out (Fig. 2, mo), is almost in the centre of the ventral surface. Behind this, a
white, V-shaped mark (vd) indicates the ducts of the male reproductive organs, and
still further back is the irregular opaque mark of the "shell-gland," by which the egg-
shells are formed (Fig. 2, ♀).
Fig. 3.—Leptoplana tremellaris in the act of swimming. A, Seen from the right side
during the downward stroke (the resemblance to a skate is striking); B, from
above, showing the upward stroke and longitudinal undulations of the swimming
lobes; C, side view during the upward stroke; D, transverse sections of the body
during the strokes. × 5.
We have few direct observations on the nature of the food of Leptoplana, or the
exact mode by which it is obtained. Dalyell,[18] who observed this species very
carefully, noticed that it was nocturnal and fed upon a Nereis, becoming greatly
distended and of a green colour after the meal, but pale after a long fast.
Keferstein[19] noticed a specimen in the act of devouring a Lumbriconereis longer
than itself, and also found the radulae of Chiton and Taenioglossate Molluscs in the
intestine. That such an apparently weak and defenceless animal does overpower
large and healthy Annelids and Mollusca, has not hitherto been definitely proved.
Weak or diseased examples may be chiefly selected. The flexible Leptoplana
adheres firmly to its prey, and the rapid action of the salivary glands of its mobile
pharynx quickly softens and disintegrates the internal parts of the victim. The food
passes into the stomach (Fig. 2, mg), and is there digested. It is then transferred to
the lateral branches of the intestine, and, after all the nutritious matters have been
absorbed, the faeces are ejected with a sudden contraction of the whole body
through the pharynx into the water.
Leptoplana probably does not live more than a year. In the spring or summer,
batches of eggs are laid and fixed to algae or stones by one individual, after having
been fertilised by another. Young Leptoplana hatch out in two to three weeks, and
lead a pelagic existence till they are three or four millimetres in length. In late
summer, numbers of such immature examples may be found among sea-weeds
and Corallina in tide pools. In the succeeding spring they develop first the male and
then the female reproductive organs.
Parenchyma.—The spaces between the main organs of the body are filled by a
tissue containing various kinds of cells, salivary glands, shell-glands, and prostate
glands. Besides these, however, we find a vacuolated, nucleated, thick-walled
network, and to this the word parenchyma is properly applied. Besides its
connective function, the parenchyma confers that elasticity on the body which
Leptoplana possesses in such a high degree. Pigment cells are found in the
parenchyma in many Polyclads.
Classification of Polycladida.
ACOTYLEA.
Family. Genus. British
Representatives.
Planocera (Fig. 8,
Planocera folium
A).
Grube. Berwick-
Imogine.
on-Tweed.
Planoceridae. Conoceros.
Stylochoplana
With dorsal tentacles. Stylochus.
maculata Quatref.
Mouth sub-central. Stylochoplana
Among brown
(Fig. 8, B).
weeds in
Diplonchus.
Laminarian zone.
Planctoplana.
Leptoplana
tremellaris O. F.
Müll.
Discocelis.
L. fallax Quatref.
Leptoplanidae. Cryptocelis.
Plymouth.
Without dorsal Leptoplana.
L. droebachensis
tentacles. Penis Trigonoporus.
Oe. Plymouth
directed backwards. ?Polypostia (see
Sound.
p. 27).
L. atomata O. F.
Müll. Doubtful
species.
Cestoplana (Fig. 8,
Cestoplanidae. C).
No tentacles. Body In Mediterranean
elongated. Penis and on French
directed forwards. side of the
Channel.
Enantiidae.
No sucker. No
tentacles. Main-gut Enantia.
very short. External Adriatic Sea.
apertures as in
Euryleptidae.
COTYLEA.
Anonymidae. Anonymus (Fig. 8,
Mouth central. No D).
tentacles. With two Naples (two
rows of penes. specimens).
Pseudoceridae. Thysanozoon (Fig.
Marginal tentacles 8, E).
folded. Mouth in Pseudoceros.
anterior half. Yungia.
Prostheceraeus
vittatus Mont. On
west coast.
P. argus Quatref.
Guernsey.
Cycloporus
Euryleptidae. papillosus Lang.
Tentacles usually Prostheceraeus. On Ascidians in
present and pointed, Cycloporus. 2-30 fms.
or represented by Eurylepta. Eurylepta cornuta
two groups of eyes. Oligocladus. O.F. Müll. On
Mouth close to Stylostomum. sponges and
anterior end. Aceros. shells, 2-10 fms.
Pharynx cylindrical. Oligocladus
sanguinolentus
Quatref.
O. auritus Clap.
Doubtful.
Stylostomum
variabile Lang.
Prosthiostomatidae.
Tentacles absent.
Body elongated.
Prosthiostomum
Pharynx long,
(Fig. 8, F).
cylindrical. Penis
with accessory
muscular vesicles.
Appearance and Size of Polyclad Turbellaria.—Polyclads are
almost unique amongst animals in possessing a broad and thin,
delicate body that glides like a living pellicle over stones and weeds,
moulding itself on to any inequalities of the surface over which it is
travelling, yet so fragile that a touch of the finger will rend its tissues
and often cause its speedy dissolution. The dorsal surface in a few
forms is raised into fine processes (Planocera villosa), or into hollow
papillae (Thysanozoon brocchii), and in very rare cases may be
armed with spines (Acanthozoon armatum,[26] Enantia spinifera); in
others, again, nettle-cells (nematocysts) are found (Stylochoplana
tarda, Anonymus virilis). Some Polyclads, especially the pelagic
forms, are almost transparent; in others, the colour may be an
intense orange or velvety black, and is then due to peculiar deposits
in the epidermal cells. Between these two extremes the colour is
dependent upon the blending of two sources, the pigment of the
body itself and the tint of the food. Thus a starved Leptoplana is
almost or quite white, a specimen fed on vascular tissue reddish.
Many forms are coloured in such a way as to make their detection
exceedingly difficult, but this is probably not merely due, as Dalyell
supposed, to the substratum furnishing them with food and thus
colouring them sympathetically, but is probably a result of natural
selection.