Special Tools and Utilities 6.1 THE STREAM EDITOR -sED ed used to be the standard text editor for UNIX. It is cryptic to use but is compact and programmable. sed stands for stream editor and is the only incarnation of ed that is commonly used today. sed allows editing of files non-interactively. Although the name sed implies editing. It is not a true editor; it does not change anything in the original file. Rather sed scans the input file, line by line, and applies a list of instructions (called a sed script) to each line in the input file. The script, which is usually a separate file, can be included in the command line if it is a one line command. The concept is shown in Figure 6.1 input file sed script sed options script file list usually ina ‘no automatic output separate file inline script -f:in-file script Figure 6.1 The sed Command 51.1 syntax The general syntax of sed command is: _ $ sed <'address-actionlist'> |UNIX Prog, 134 where, the options such as -e, -f and -n available with sed commana, options Hee a single line or number of lines on which action has to be a ete specifies the action or list of actions to be taken ; holds zero or more filenames from where the lines are searches ad filelist processed one by one, and sent to the monitor. When a filename ee specified, the input data is taken from the keyboard. 6.1.2 Options Options used with sed command are: -¢ + This option is used to give multiple instructions in a single command line, -£ + The -f option is for script file, when instructions are taken from a separate fl called the script file. The script file contains the sed instructions to be executca n+ By default sed displays or prints all the lines selected. This default display ca be suppressed by using the -n option. ‘The working mechanism of sed command is as follows: Each line or record is read from the input file which is held in the memory az called the pattern space. All commands are applied on this space. Since sed works on om line at a time, large files can be altered without invoking an editor, or worry about space requirements, The processing using sed command does not affect the originel contents of the input file. 6.1.3 Addressing By default, sed considers ever y line in the input file. However sed must be restritted ly the required lines and pr - Pattern which occurs in the line, The pattern could be * quble Pattern containing specific word or phrase, or it can be 2 | example 1, 3 specific: at |. addressing, Com, ; 1 mvcessing. This type of addressing is known *%. # ified ma @) between starting and ending range *P%special Tools and Utilities ios Single Line Set of Lines Range Nested of Lines Addresses Figure 6.2 Line Address in sed 6.1.4 Multiple Instructions Multiple instructions are given by one of the following methods: @ By using the -e option with every command. % By using quotes and giving each command in a separate line. By using a script file which holds all the command, one per line. 6.2 SCRIPTS The sed utility is called like any other utility. When there is only one command, it may be entered from the keyboard. Most of the time, however, instructions are placed in a file nown as sed script (program). Each instruction in a sed script contains an address and a command. 6.2.1 Script Formats When the script fits in a few lines, its instructions can be included in the command shown in Figure 6.3. (a) Note that in this case, the script must be enclosed in ‘$sed -e ‘address command input_file | (a) Inline Script $ sed -f script.sed input_file | (b) Script File Figure 6.3 Examples of sed Scripts For longer scripts, or for scripts that are going to be executed repeatedly over time, a “*varate script file is preferred. This file is created with a text editor and saved. sed to indicate that it is a sed script. Figure 6.3 (b) is an example of executing a sed script.UNIX Prosramin 136 6.2.2 Instruction Format instruction consists of an address and a command (Figure 6.4) Each in: ts of an addi d id (Fi, Complement (Optional) address | !| address Figure 6.4 instruction Format matches the address, 6.2.3 Comments is ignored by sed. Comment line beginning with @ comment token, which is the pound sign @) . If the comment requires more than one line, each line must start with the # sign. 6.3 sed Commanps sed consists of different types of sommands, for different types of actions on a single line or all selected lines. The most common sed commands are described as b Line 7 La | ee Output Branch Ce] Lai] Transform Files | Hold Space Figure 6.5 sea Commands elow:special Tools and Utilities 137 634 4 Command The q command picks up all the lines upto and including the line addressed from the input file, and then quits. It also displays these sel can be read as print the first 2 lines and wn in the example 6.1. lected lines by default. For example ms quit. Multiple commands can be given as sho’ 6.3.2 Modify commands Modify commands are used to insert, append, change, or delete one or more lines. Modify Insert Append | Change Delete Delete First Figure 6.6 sed Modify Commands 3.3. d Command The d command is used to delete a line or a range of lines as shown in the example 6.1. Example 6.1:q&d $ cat > phonelist Meenakshi : 26462646 Murugan: 22342234 Harish: 25252525 $ sed 2q phonelist Chitra : 23462346 Jyothi : 25342534 $ sed -e ‘Id’ -e '2q' phonelist Jyothi : 25342534 ctrl 2 $ sed "14d" phonelist Harish : 25252525 °2q' pri ines and quits. : its. Si The ce Teeanie eeres the Fest tine and prints the fee lines and quits. Since the frst line has been deleted it prints only the second 4 rest of the lines from The 3° example here deletes the first 4 lines and prints Phonelist,UME ty ee WA Von reipg 6.3.4 aCommand end one or more revordshines to an ey The a command allows the user to append one or more recordshines to ap sting file as in example 6.2, Example G.2ia _ $ sed ‘$ a\ Sumitha : 23602360 > 'phone.|st > phonelist The records to be added must he entered on ‘The command a rust te t will be appended after to be appended, except the last given with a $ sign as $a, When a is given without a §, th every line of the input file, The command Jing and the Jin line must be terminated with a backward slash ( 6.3.5 i Command Text can be inserted into a file at a speci ¢ line number, Vi i beginning of the input file. Using just i, inserts the ne w text before every line in the input file. Example 6.3 : i $ cat > list 001: Sneha : 78: BCA 002: Asha: 60: BSc 011: Trisha : 60: BSc Ctrl Z $ sed ‘3i\ 003: Mallika :75 : BSc ‘list > newlist $ cat newlist 001: Sneha : 78 : BCA 002: Asha: 60 : BSc 003 : Mallika : 75 : BSc O11: Trisha: 60 : BSc 6.3.6 ¢Command One or more lines/records can be changed or replaced in the input file by using the © command,special Tools and Utilities 139 Example 6.4: c $ sed “4c\ 004: Trisha : 60 : BSc ‘newlist 001: Sneha : 78 : BCA 002: Asha: 60 : BSc 003 : Mallika : 75 : BS 004: Trisha : 60 : BSc 6.3.7. Line Number Command (=) The line numbers can be printed generally displayed on a separate line t the = command. The line numbers are text es shown in example 6.5. Example 6.5: $ sed ’=" bsc.dat 1 002: Asha : 60: BSc 2 003 : Mallika : 75: BSc 3 011: Trisha : 60: BSc Though grep and sed have similar utilities, sed is much more powerful than grep. sep cannot do everything that sed can do. However grep is much easier to use. 6.3.8 p Command ‘The selected lines can be printed u: option, the default printing is suppre: example 6.6, the p command. When combined with -n d and only the specified lines are printed as in To use the -f option with the p command, the sed instructions must be kept in a Script file as sown in the example 6.6. The example stores the instructions in a script file called sep.fl, Example 6.6 : p Ssed-n -e ‘1,34’ -e ‘4p’ phonelist | Murugan : 22342234 $sed-ni, 3a 4P phonelist Murugan : 22342234 _UNIX Prosran, 140 ling [~¢ cat > script.sed # This isa comment line 1, 3d ap $ sed -n -f script.sed phonelist Murugan } 22342234 __ Since the first example contains multiple commands, -e option is used, It deletes the first 3 lines (range of lines 1,3) and prints the first 4 lines which containg only the 4" line. : ; The same command can be given without using -e by given each address in a different line. The 3" example shows creation and execution of a script file. 6.3.9 w Command ‘The output of a sed command can be written into a separate file using the write command w. The example selects all the BSc students and writes these records into a separate file called BSc.dat. Example 6.7 : w $ sed -n ‘/BSc/w bsc.dat’ newlist $ cat bsc.dat 002: Asha: 60 : BSc 003 : Mallika : 75 : BSc 004: Trisha: 60 : BSc $ sed -n ‘/BSc/!w others.dat’ newlist $ cat others.dat 001: Sneha : 78 : BCA 7 The use of the bang character ! with w (tw) selects all records which do not contait the pattern and writes these records into a file called others.dat, 6.3.10 r Command Example 6.8 : r $ sed '$r others.dat’ bsc.dat 002: Asha: 60 : BSctilitie special Tools aed a 141 003 : Mallika : 75 : BSc 004: Trisha :60 ; BSc 001: Sneha : 78 : BCA others.dat is appended to bsc.dat 6.4 USING SEARCH PATTERN To print all the lines containing a given pattern, the search pattern must be enclosed within two slashes (). In example 6.9, the first example picks up the line containing the name Meenakshi. In the second example it selects two lines contains the pattern ‘hi’ In the last example of 6,9, the sed command picks up all lines starting from the line that begins with Meenakshi, upto and including the line ending with the number 448) and displays them. Example 6.9 ; /pattern/ $ sed -n /Meenakshi/p phonelist Meenakshi : 26462646 § sed -n /hi/p phonelist Jyothi : 25342534 Meenakshi : 26462646 $ cat >> phonelist Pratibha : 23492349 Rama : 23562356 $.sed -n /49/p phonelist Pratibha : 23492349 $ sed -n ‘/Meenakshi/, /4$/p’ phonelist Meenakshi : 26462646 Murugan : 22342234 Pattern] /| Replacement string | /| Flag(s) Address Figure 6.7 The Substitute CommandUNIX Programming 142 and : aie stitution command, When the old pattern, followed by the for substitution in slashes, the s command substitutes the old. patter, Le ‘ample 5.16. By default the substitution is done on, 6.4.1 s_ stands replacement pat with the new pal on the first occurrenct ‘All occurrences of the searc! tern is ttern as shown in ©: ‘eof the search pattern. hh pattern can be substituted by using the global option g as in the example 6.10. Example 6.10 : $ $ sed -n s/Meenakshi/Parvathi/p phonelist Parvathi : 26462646 ] sed on s/Meenakshi/Parvathi/p phonelist > phone.Ist $ cat saying A friend in need is a friend indeed. A stitch in time saves nine. $ sed -n s/friend/beggar/p saying A beggar in need is a friend indeed. $ sed -n s/friend/beggar/gp saying A beggar in need is a beggar indeed. $ sed -n ‘s/:/ | /gp’ phone.|st > newphone ‘The s command is used to substitute old pattern with new pattern. In the 2"4 example Meenakshi is replaced with Parvathi and this line is written into a new file called phone. st. In the fourth example using the global option, all occurrences of friend are replaced by beggar. The 5‘ example replaces : with | and writes these replaced lines in a new file called newphone. 6.4.2 Sed Command Examples Consider the below text file as an input. $ cat file.txt unix is great os. unix is open source. unix is free os. learn the operating system. unixlinux which one would you choose? 1. Replacing or substituting string Sed command is mostly used to replace the text in a file. The below simple sed command replaces the word “unix” with “linux” in the file.i id Utilities cial Tools ant ee 143 $sed_‘s/unix, linux is great os. unix is open sow ; unix is free os, Jearn operating system, Jinuslinux which one would you choose? Here the “s” specifies the substitution operatio; : mn. The"? are delimitors, Tho “unix’ i the search pattern and the “linux” is the replacement string 2 By default, the sed command replaces the first occurrence of the pattern in each line and it won't replace the second, third...occurrence in the line. Replacing the nth occurrence of a pattern in a line. Use the /1, /2 ete flags to replace the first, second occurrence of a pattern in a line. The command below replaces the second occurrence of the word “unix” with “linux” ina line. $ sed ‘s/unix/linux/2’ file.txt unix is great os. linux is open source. unix is free os. learn the operating system. unixlinux which one would you choose? 3. Replacing all the occurrence of the pattern in a line. The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line. $ sed ‘s/unix/linux/g’ file.txt linux is great os. linux is open source. linux is free os. learn the operating system. linuxlinux which one would you choose? 4. Replacing from nth occurrence to all occurrences in a line. Use the combination of /1, /2 ete and /g to replace all the patterns from the nth occurrence of a pattern in a line. The following sed command replaces the third, fourth, fifth... “unix” word with “linux” word in a line, $ sed ‘s/unix/linux/39’_file.txt unix is great os. unix is open source. linux is free 0s. learn the operating system. linuxlinux which one would you choose? Duplicating the replaced line with /p flag The /p print flag prints the replaced line twice on the are “ ie dose ad have the search pattern and is not replaced, then the /p prints that line only once.UNIX Prony 144 *s/unix/linux/P" file.txt $ sed : pen source. unix is free os. is great os. unix is o| ii coat os. unix is open source. unix is free os. linux is gr learn the operating system. linuslinux which one would ne would you choose? you choose? linuxlinux which 0 6. Printing only the replaced lines Use the -n option along with the /p print flag to display only the replaced line. the -n option suppresses the duplicate rows generated by the /p flag and prin; replaced lines only one time. $ sed -n ‘s/unix/linux/p’ file.txt linux is great os. unix is open source. unix is free os. linuxlinux which one would you choose? If you use -n alone without /p, then the sed does not print anything. 7. Deleting lines You can delete the lines in a file by specifying the line number or a range c numbers. $ sed ‘2d’ file.txt $sed ‘5,$d' file.txt 8. Sed as grep command You can make sed command to work as similar to grep command. $ grep ‘unix’ file.txt $ sed _-n ‘/unix/p’_file.txt Here the sed command looks f a h for the pattern “unix” i ine of a file anéP those lines that has the pattern. i hi 9. Add a line after a match. The sed s command and can add a new line after a pattern match is found . sed tells it to add a new line after a match is found. $ sed ‘unix/ a “Add a new line” * file.txt unix is great os. unix is open source, unix is free os. “Add a new line” learn the operating system, unixlinux which one would you choose?He special Toots andl Ui i “add a new lino” Change a line The sed command can be used to re; command to sed tells it to change the 10. Place an e ; ntire line with a new line. The “c” ine, $ sed ‘/unix/ ¢ “Change line" * file.txt “Change line" learn the operating system, “Change line” 6.5 AWK PROGRAMMING LANGUAGE awk is a general purpose programming language that is designed for processing text-based data, either in files or data streams, and was created at Bell Labs in the 1970s. The name awk is derived from the family names of its authors - Alfred Aho, Peter Weinberger, and Brian Kernighan; however, it is not commonly pronounced as a string of separate letters but rather to sound the same as the name of the bird, auk (which acts as an emblem of the language). awk, when written in all lowercase letters, refers to the UNIX program that runs other Programs written in the awk programming language. Figure 6.8 The great awk A version of the awk language is a standard feature of nearly every modern Unis like operating system available today. Besides the Bourne shell, awk is the only other scripting language available in a standard UNIX environment. 6.6 OPERATIONAL MECHANISM OF awk awk is a language for processing files of text. A file is treated as a sequence of Tecords, and by default each line is a record. Each line is broken up into a sequence of felds, so we ean think of the first word in a line as the first field, the second word as the Second field, and so on. An awk program is of a sequence of pattern-action statements. wk reads the input one line at a time. A line is scanned for each pattern in the program, and for each pattern that matches, the associated action is executed. awk is an example of a programming language that extens vely oo string data RP®, associative arrays (that is, arrays indexed by Key strings), and regular expressions * Power, terseness, and limitations of awk programs and sed scripts inapied Laney Wal Write Perl. Because of their dense notation, all these languages are often used for MRE one-line programs. bh146 Unix Poth 6.7 EXECUTION OF AN AWK SCRIPT Phe awk utility ia called lke any othor ut . The gene ral form nat Ot an command is : " [fm [options] [seit | [es] J where, Options — aro ~ Fand total.dat 10 20 30 40 50 60 * 70 80 90 ctri-d $ cat total.awk #Begin Process BEGIN {print "Print Totals”} #Body Process {total = $1°+ $2 + $3} {print $1 "+" $2 “+” $3 "=" total} #€nd Process END {print “End of totals”} | Output $awk -f total.awk total.dat Print Totals 10 + 20 + 30 = 60 40 + 50 + 60 = 150 70 + 80 + 90 + 240 End of totals | | | $2 is the once In the above example $1 is the name of the first number in a line, number and $3 is the last number. The body instruction is executed 3 tim' record.UNIX Programming 138 id 6.3.4 aComman : The a command allows the user to append one or more records/lines to an existing file as in example 6.2. Example 6.2: a $ sed ‘$ a\ Sumitha : 23602360 > ‘ phone.|st > phonelist The records to be added must be entered on separate lines. The command @ must be given with a § sign as $a. When a is given without a every line of the input file. The command line and the li Hine must be terminated with a backward slash (\). 6.3.5 i Command $ sed ‘3iy 003 : Mallika; 75 00 BSc list > newlist $ cat newlist 001 : Sneha 178 : BCA 002: Asha: 9 BSc 003 : Mallika : 75 ; BSc O11: Trisha: 69 Bsc 6.3.6 ¢ Command One or moy © command,