SED Introduction
sed(Stream Editor) is a very useful utility in unix which provides solutions to many problems related to editing/handling data and log files.
Lets quickly look into some of the awesome uses of sed before pitching into knowing how sed works.
Example 1: How to remove null characters in a file ?
Sometimes when we try to copy some text in textpad, we receive the below error message because of null characters.
“Cannot cut, copy, or drag and drop text containing null (code = 0) characters”
sed provides an easy way to replace all the null characters with space or remove all the null characters.
sed -i 's/\x0/ /g' FileContainingNull.txt ## Replace null characters with space. sed -i 's/\x0//g' FileContainingNull.txt ## Remove all null characters
(\x0) is hexadecimal zero. We can use decimal zero (\d0) or octal zero (\o0) also.
Explanation:
-i – Sed option for inplace editing.
‘s/\x0/ /g’ – sed substitution command substitutes hexadecimal zero(null characters) with a space. g – is the global flag, specified to replace all null characters in the pattern space.
Example 2: How to replace directory or file paths containing forward or backward slash ?
Here we are replacing all instances of “/home/user/” to “/tmp/” in a file.
sed -i 's|/home/user/|/tmp/|g' FileContainingPath.txt
Explanation:
-i : Sed option specifying inplace editing.
‘s|/home/user/|/tmp/|g’ : Sed substitution command replacing /home/user/ to /tmp/. g – global flag, specifying replacement of all instances in the pattern space. | is the delimiter separating the search string and the replacement string.
We can also use the below command.
sed -i 's/\/home\/user\//\/tmp\//g' FileContainingPath.txt ## But this looks wierd.
Explanation:
As per sed man page, in a substitution command, the ‘/’ characters may be uniformly replaced by any other single character within any given ‘s’ command. The ‘/’ character (or whatever other character is used in its stead) can appear in the REGEXP or REPLACEMENT only if it is preceded by a ‘\’ character.
Example 3: How to remove blank lines in a file ?
bash-3.2$ cat test.txt 1. leaf is green 2. sky is blue 3. snow is white 4. moon is white 5. flower is yellow bash-3.2$ sed -i '/^$/d' test.txt bash-3.2$ cat test.txt 1. leaf is green 2. sky is blue 3. snow is white 4. moon is white 5. flower is yellow bash-3.2$
Explanation:
-i – is the sed option for inplace editing.
/^$/d – ^$ is a regular expression to identify blank line. d is a command to delete the pattern space.
Example 4: How to remove the first and last line in a file ?
bash-3.2$ cat test.txt 1. leaf is green 2. sky is blue 3. snow is white 4. moon is white 5. flower is yellow bash-3.2$ sed '1d;$d' test.txt 2. sky is blue 3. snow is white 4. moon is white bash-3.2$
Explanation:
‘1d’ – deletes the pattern space if it is line 1.
‘$d’ – deletes the pattern space if it is last line.
Example 5: How to print all the lines following a specific pattern ?
bash-3.2$ cat test.log 2:00AM Successful transaction1 2:30AM Successful transaction2 2:45AM Successful transaction3 3:00AM DBError Starts Error trace 1 Error trace 2 3:15AM DBError Ends 3:30AM Successful transaction4 3:45AM Successful transaction5 4:00AM DBError Starts Error trace 3 Error trace 4 4:15AM DBError Ends 4:30AM Successful transaction6 bash-3.2$ sed -n '/DBError/,$p' test.log 3:00AM DBError Starts Error trace 1 Error trace 2 3:15AM DBError Ends 3:30AM Successful transaction4 3:45AM Successful transaction5 4:00AM DBError Starts Error trace 3 Error trace 4 4:15AM DBError Ends 4:30AM Successful transaction6 bash-3.2$
Explanation:
‘-n’ – is the sed option for silent mode.
‘/DBError/,$’ – Specifies the first line which contains the pattern ‘DBError’ till the last line ($).
‘p’ – Prints the pattern space.
Example 6: How to print all the lines between two words/pattern/regular expression.?
bash-3.2$ cat test.log 2:00AM Successful transaction1 2:30AM Successful transaction2 2:45AM Successful transaction3 3:00AM DBError Starts Error trace 1 Error trace 2 3:15AM DBError Ends 3:30AM Successful transaction4 3:45AM Successful transaction5 4:00AM DBError Starts Error trace 3 Error trace 4 4:15AM DBError Ends 4:30AM Successful transaction6 bash-3.2$ sed -n '/DBError Starts/,/DBError Ends/p' test.log 3:00AM DBError Starts Error trace 1 Error trace 2 3:15AM DBError Ends 4:00AM DBError Starts Error trace 3 Error trace 4 4:15AM DBError Ends bash-3.2$
Explanation:
‘-n’ – is the sed option for silent mode.
‘/DBError Starts/,/DBError Ends/’ – Specifies the lines which contains the pattern ‘DBError Starts’ till the line which contains the pattern ‘DBError Ends’.
‘p’ – Prints the pattern space.
Example 7: You have a huge log file. How to print or extract lines between particular line numbers?
bash-3.2$ sed -n '100000,200000p' huge.log > extracted.log
Explanation: The above sed command prints lines from 100,000 to 200,000 into extracted.log
To learn how sed works, please navigate to next page.