The Stream Editor : sed Sed is designed to be especially useful in these cases:

To edit files too large for comfortable interactive editing; To perform multiple `global' editing functions efficiently in one pass through the input.

Takes input from a file, but its output is sent to std.out The original file is not altered Output from sed can be redirected to a new file if you want to save the changes or piped to another program (Vi , ed , ex editors create a temporary copy of the file you want to edit in /tmp. All changes are made to this copy. When you save these changes (:w) , the temporary file is copied to the original)

The Stream Editor : sed

-Takes a line from input , applies an editing command to it ,and prints the result. - Repeats this process till the whole file has passed through sed 1)Copies an input line to the pattern space (is a buffer capable of holding one or more text lines for processing) 2) Applies all the instructions in the script, one by one, to all pattern space lines that match the specified addresses in the instruction. echoamyenjoyshikingandbenenjoysskiing|sedes/skiing/hiking/g;s/hiking/biking/g specifiers : 3) Line Copies the contents of the pattern space to the output file unless directed not to by the n option Single Lines specifies only one line in the i/p file (line no. or $) 1)Sedreadinthelineamyenjoyshikingandbenenjoysskiingandexecutedthe first

substitutecommand. resulting line match in thezero pattern space: Set ofThe Lines - a RE that can or more lines, not necessarily consecutive (/^A/ command) amyenjoyshikingandbenenjoyshiking
Ranges an address range that defines a set of consecutive lines (lineno,lineno ; /RE/,lineno) { /never/ command } )

2) Then the second substitute command is executed on the line in the pattern space, and the result isAddresses : Nested an address contained within another address (20,30 amyenjoysbikingandbenenjoysbiking 3) And the result is written to standard out.

sedediting-commandfilename
Line specifier editing instruction

If no line specifier then command is applied to all lines.

3 Third line 5,8 lines 5 to 8 10,$ lines 10 to last line $-9,$ sed cannot process this /huge/ all lines containing huge /^[ \t]*$/ all blank lines 1,/gossip/ from line 1 to first mention of gossip <specifier>! Command applied to lines that do not match specifier

Note the slashes

Editing Instructions (immediately follows line specifier)

s (substitute)

p (print)

d (delete)

q (quit)

sed10qtextquitafterreadingline10 sed/^[\t]*$/dtextdeleteallblanklines sed/value/!dportfoliodeleteeverylineinportfolionotcontainingthestringvalue sed n9,11pemplist -n suppresses duplicates sed n/obr[iy][ae]n/p /lennon/pemplist eitherobriensorlennons
(note and separate lines)

sed -n1,2p 7,9p $pemplist sed n e1,2pe7,9pe$pemplist

Many sections can be selected in one command

s/oldstring/newstring/
seds/Michael/Michele/text sed1,/Jessup/s/Michael/Michele/text seds/UNIX/UNIX/adcopy seds/UNIX/&/adcopy seds/\(incredible\) and \(amazing\)/ \2 or \1/paper
tagged item #1 tagged item #2

3 main options to sed -n -e -f

simple strings, can contain special chars, tag part of the pattern, all of grep expressions ,&

* Replace all occurrences of foregoon and Foregone with foregone and Foregone

Sed..substitute
[yma@yma ~]$ cat sedtext this line has o'brien and this has a lennon this has neither and this has both o'bryan and lennon [yma@yma ~]$ sed 's/"o'brien"/oooooo/g' sedtext > bash: unexpected EOF while looking for matching `"' bash: syntax error: unexpected end of file [yma@yma ~]$ sed 's/"o\'brien"/oooooo/g' sedtext > bash: unexpected EOF while looking for matching `"' bash: syntax error: unexpected end of file yma@yma]~$seds/o'brien/oooooo/gsedtext this line has oooooo and this has a lennon this has neither and this has both o'bryan and lennon

seds/Michael/Michele/ s/Joanie/John/ s/Giant/Cub/chapter1

Multiple commands to sed One cmd per line

sed -f sedpat sedtext sed -f namech -f form memo

Allows sed to take editing commands from a file

sed -es/Boston/Wasco/g-f form temp.chap

Allows mixing of editing commands editing line and files

ls l | sed n/^..w/p
lists files which have write permissions for the group

Sed ..examples
[root@localhost ~]# cat therav Once upon a midnight dreary,while I pondered weak and weary. Over many a quaint and curious volume of forgotten lore While i nodded,nearly napping,suddenly there came a tapping display line numbers [root@localhost ~]# sed "=" therav 1 Once upon a midnight dreary,while I pondered weak and weary. 2 Over many a quaint and curious volume of forgotten lore 3 While i nodded,nearly napping,suddenly there came a tapping [root@localhost ~]#

Sed ..examples
Print lines beginning with O [root@localhost ~]# sed -n /^O/ therav sed: -e expression #1, char 4: missing command [root@localhost ~]# [root@localhost ~]# sed /^O/p therav Once upon a midnight dreary,while I pondered weak and weary. Once upon a midnight dreary,while I pondered weak and weary. Over many a quaint and curious volume of forgotten lore Over many a quaint and curious volume of forgotten lore While i nodded,nearly napping,suddenly there came a tapping [root@localhost ~]# [root@localhost ~]# sed -n /^O/p therav Once upon a midnight dreary,while I pondered weak and weary.

Over many a quaint and curious volume of forgotten lore

[root@localhost ~]#

Sed ..examples
Using substitute command [root@localhost ~]# echo 123abc456|sed 's/[0-9]//'

23abc456
If global substitution not specified, sed commands operate on only the first occurrence of the pattern

[root@localhost ~]# echo 123abc456|sed 's/[0-9]//g'

abc [root@localhost ~]#

Sed ..examples
adds spaces at beginning of line and -- to the end Script stored in a file called addpart.sed [root@localhost ~]# cat >addpart.sed s/^/ / s/$/--/ [root@localhost ~]# Invoking the script in sed [root@localhost ~]# sed -f addpart.sed therav Once upon a midnight dreary,while I pondered weak and weary.-Over many a quaint and curious volume of forgotten lore-While i nodded,nearly napping,suddenly there came a tapping-[root@localhost ~]#

Note: the lack of quotes every command on a separate line

Sed ..examples
Using the Back Reference (Whole pattern substitution) Once upon a midnight dreary,while I pondered weak and weary. Over many a quaint and curious volume of forgotten lore While i nodded,nearly napping,suddenly there came a tapping [root@localhost ~]# sed 's/on/&once/' therav Once upononce a midnight dreary,while I pondered weak and weary. Over many a quaint and curious volume of forgotten lore While i nodded,nearly napping,suddenly there came a tapping [root@localhost ~]# sed 's/o*e/&once/g' therav Onceonce upon a midnight dreonceary,whileonce I pondeoncereonced weonceak and weonceary. Oveoncer many a quaint and curious volumeonce of forgotteoncen loreonce Whileonce i noddeonced,neoncearly napping,suddeoncenly theoncereonce cameonce a tapping [root@localhost ~]#

10

sed ...contd
Reading a file ( r ) the r command lets you read in a file at a specified location. $sed/<FORM>/rtemplate.htmlform_entry.html
inserts the forms details from an external file template.html after the <FORM> tag

$sed '/Over/r addpart.sed' therav Inserting and changing text ( i , a , c) for appending text use a command and then enter as many lines as you want. you have to precede the [ENTER] key in each line except the last with a \.

$sed$a\ > place the following line at the end \ > The end ; > www_lib.pl > $$

What happens without the $

$sedi\ fil1

inserts a blank line before every line

Once upon a midnight dreary, while I pondered , weak and weary Over many a quaint and curious volume of forgotten lore While I nodded, nearly napping, suddenly there came a tapping

$sed2c\ Over many an obscure and meaningless problem of calculus bore fil1 11

sed..examples
Add a title to the file therav [root@localhost ~]# cat > insfil.sed #script to add a title to file 1i\ The Raven\ by Edgar Poe\ [root@localhost ~]# sed -f insfil.sed therav sed:fileinsfil.sedline5:unknowncommand:`E [root@localhost ~]# vi insfil.sed

Added the \ before the newline and after by

12

sed...examples
[root@localhost ~]# cat insfil.sed

#script to add a title to file 1i\ The Raven\ by\ Edgar Poe\

The Raven by Edgar Poe Once upon a midnight dreary,while I pondered weak and weary. Over many a quaint and curious volume of forgotten lore While i nodded,nearly napping,suddenly there came a tapping [root@localhost ~]#

[root@localhost ~]# [root@localhost ~]# sed -f insfil.sed therav

root@localhost ~]# sed 'i\ > > ' therav

Once upon a midnight dreary,while I pondered weak and weary. Over many a quaint and curious volume of forgotten lore While i nodded,nearly napping,suddenly there came a tapping

What does this do?

What does this do?

13

The Awful Truth about sed ---sed has a serious learning curve!!
Write a command in sed that adds == at the beginning of a line, !! at the end of the line; and a blank line between every line

sed...examples

[root@localhost ~]# abcdef Matches abcdef [root@localhost ~]# sed '$a\

Once upon a midnight dreary,while I pondered weak and weary. Over many a quaint and curious volume of forgotten lore > Tis a visitor\ a*b' While i nodded,nearly napping,suddenly there came a tapping Matches zero or more `a's followed by a single `b'. For example, `b' or `aaaaab > at the door ' therav >newfil Tis a visitor [root@localhost ~]# cat newfil at the door a\+b\+' Matches one or more `a's followed by one or more `b's: `ab' is the shortest possible match. .*' .\+' These two both match all the characters in a string; however, the first matches every string (including the empty string), while the second matches only strings containing at least one character.

^main.*(.*)' This matches a string starting with `main', followed by an opening and closing parenthesis. The `n', `(' and `)' need not be adjacent.
`^#' This matches a string beginning with `#'. `\\$' This matches a string ending with a single backslash. The regexp contains two backslashes for escaping.

`\$'

Instead, this matches a string consisting of a single dollar sign, because it is escaped.

14

Sed..expressions
[yma@yma

~]$ cat ab

ab abc aaab abab ababc aabababddc ab.abc a?b b abc\ ab$ $ / \ [yma@yma ~]$ sed -n '/\/$/p' ab / [yma@yma ~]$ sed -n '/\\$/p' ab abc\ \ [yma@yma ~]$ sed -n '/\$/p' ab ab$ $ $[yma@yma~]

sed...examples
[yma@pc18 ~]$ cat mydat this is a test file for removing the rest of the words in a file but keeping only the first word
[yma@pc18 ~]$ sed 's/\(on\)/\1eeee/' mydat
this is a test file for removing the rest of the words in a file but keeping only the first word

this is a test file for removing the rest of the words in a file but keeping oneeeely the first word

[yma@pc18 ~]$ sed 's/\(the\)\(of\)/\2 and \1/' mydat

Sowhydidntseddoanything?
16

sed...examples
[yma@pc18 ~]$ sed 's/\(the\) rest \(of\)/\2 and \1/' mydat this is a test file for removing

of and the
the words int a file but keeping only

the first word

Write a sed command To keep only the first word in a line [yma@pc18 ~]$ sed 's/\([a-z|A-Z]*\).*/\1/' mydat [yma@pc18 ~]$ sed 's/^\([[:alnum:]]*\).*/\1/g' mydat How does the following differ? [yma@pc18 ~]$sed -n 's/\([[:alnum:]]*\) .*/\1/p' mydat
17
this for the words the [yma@pc18 ~]$

sed...contd
Substitution : no specification --> done for first occurrence g --> done globally

[yma@pc18 ~]$ cat mydat this is a test file for removing the rest of the words in a file but keeping only the first word

Write a sed command To keep the first word on the line but delete the second

[yma@pc18 ~]$ sed 's/\([a-z]*\) \([a-z]*\)/\1 /' mydat

this a test file for the rest of the words a file but keeping the word

If you want to specify which pattern to match use /1 /2 ... (Do Not confuse this with \1 \2 ...)
18

Sed numbered patterns

[yma@yma ~]$ sed 's/[a-z]*/&!/' mydat this! is a test file sed 's/[a-z]*//2' ther [yma@pc18 ~]$ sed 's/[a-z]*//2' mydat for! removing the is a bird that flies birds fly ! the rest of T raven is a bird this a test file T is a blackbird the! for the rest of words! int this has an the ! a file but words line a file but [yma@localhost unixslides]$ sed 's/[[:alnum:]]*//2' ther ! keeping only keeping the is a bird that flies the word birds fly the! first word The is a bird This a blackbird [yma@yma ~]$ sed 's/[a-z]*/&!/g' mydat
this has an line [yma@localhost unixslides]$

[yma@yma ~]$sed 's/[a-z]*$/&!/' mydat

[yma@yma ~]$ sed 's/[a-z]*/&!/2' mydat

sed...examples
To add a character (!) after the 10th char on a line [yma@pc18 ~]$ vi mypass
nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin torrent:x:496:490:BitTorrent Seed/Tracker:/var/spool/bittorrent:/sbin/nologin backuppc:x:495:489::/var/lib/BackupPC:/usr/bin/nologin haldaemon:x:68:68:HAL daemon:/:/sbin/nologin gdm:x:42:42::/var/gdm:/sbin/nologin student:x:500:500:student:/home/student:/bin/bash yma:x:501:501::/home/yma:/bin/bash std1:x:502:502::/home/std1:/bin/bash

[yma@pc18 ~]$ sed 's/./&!/10' mypass nfsnobody:!x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin torrent:x:!496:490:BitTorrent Seed/Tracker:/var/spool/bittorrent:/sbin/nologin backuppc:x!:495:489::/var/lib/BackupPC:/usr/bin/nologin haldaemon:!x:68:68:HAL daemon:/:/sbin/nologin gdm:x:42:4!2::/var/gdm:/sbin/nologin student:x:!500:500:student:/home/student:/bin/bash yma:x:501:!501::/home/yma:/bin/bash std1:x:502!:502::/home/std1:/bin/bash [yma@pc18 ~]$
20

sed...contd
Sending output to a file :
[yma@pc18 ~]$ sed -n 's/./&!/10w mypass.out' mypass [yma@pc18 ~]$ cat mypass.out
nfsnobody:!x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin torrent:x:!496:490:BitTorrent Seed/Tracker:/var/spool/bittorrent:/sbin/nologin backuppc:x!:495:489::/var/lib/BackupPC:/usr/bin/nologin haldaemon:!x:68:68:HAL daemon:/:/sbin/nologin gdm:x:42:4!2::/var/gdm:/sbin/nologin student:x:!500:500:student:/home/student:/bin/bash yma:x:501:!501::/home/yma:/bin/bash std1:x:502!:502::/home/std1:/bin/bash

Sending arguments to a sed command

[yma@pc18 ~]$ vi sedscrp.sed sed 's/removing/replaces/g' mydat [yma@pc18 ~]$ ./sedscrp.sed
this is a test file for replaces the rest of the words in a file but keeping only the first word

Modify the script to accept an argument from the command line this is a test file for removing the rest of the words in a file but keeping only the first word

[yma@pc18 ~]$ vi sedscrp.sed sed 's/$1/replaces/g' mydat [yma@pc18 ~]$ ./sedscrp.sed removing SoWhydidntanythinghappen?

21

sed...examples
[yma@pc18 ~]$ vi sedscrp.sed sed 's/'$1'/replaces/g' mydat [yma@pc18 ~]$ ./sedscrp.sed removing
this is a test file for replaces the rest of the words in a file but keeping only the first word
[vi sedc1]

Remove a comment (#) in file until keyword stop is found

sed1,/stop/ s/#.*//'

Can pass other arguments too

[yma@pc18 ~]$ sedscrp.sed removing mydat

[yma@pc18 ~]$ vi sedscrp.sed sed's/'$1'/replaces/g'$2
22

Tr : character transliteration
Tr replaces characters in one list with corresponding characters in another list Tr is a pure filter (use redirection to access a file)
(file) Philosophy : I compute, therefore $tr eiou ~#$% < philosophy I c$mp%t~ , th~r~f$r~ I am case sensitive

I am

$tr eiou EU < philosophy

I cUmpUtE, thErEfUrE I am

$tr[A-Z][a-z] $tr\012#<philosophy
I compute ,therefore I am # $

ranges accepted
\012 is ASCII representation of newline
23

Tr..contd
Options
-d (causes list of characters to be deleted) -c (causes substitution to affect every char but those in the first list) -s (compresses strings of repeated o/p chars into a single char) $tr -d eoiu <philosophy $treiou<*philosophy to protect from shell $tr ceiou<*philosophy $tr cseiou<*philosophy I cmpt thrfr I am I c*mp*t* , th*r*f*r* I am ***o**u*e****e*e.. *o*u*e*e*e.

$tr -cs A-Za-z\012<file1 >file2

[Produce one word per line]

1st string consists of all alphabetic characters -c says substitution affects all chars accept those alphabetic characters All affected characters to be replaced by newline -s says all multiple newlines to be replaced by single newline

24