Complex sed command examples
The input text fle that we are going to use for the below examples is:
bash-3.2$ cat test.txt 1. leaf is green 2. sky is blue 3. snow is white 4. moon is white. 5. flower is yellow.
Provided below a brief description (from man page) of the commands used in the examples.
(Lowercase)p Print the current pattern space.
(Uppercase)P Print up to the first embedded newline of the current pattern space.
d Delete pattern space; start new cycle.
D Delete upto the first embedded new line in the pattern space and restart cycle with the resultant pattern space, without reading a new line of input.
q Immediately quit the sed script without processing any more input.
s/regexp/replacement/flag Attempt to match regexp against the pattern space. If successful, replace that portion matched with replacement.
The replacement may contain the special character & to refer to that pattern space which matched, and the special escapes \1 through \9 to refer to the corresponding
matching sub-expressions in the regexp.Flag can be zero or more. For example: g - replaces all the instances. number - replaces numberth instance. If no flag is provided it replaces the first instance.
= Print the current line number.
$ Match the last line.
n Replace the pattern space with the next line of input.
N Append the pattern space with the next line of input.
`: LABEL'
[No addresses allowed.]
Specify the location of LABEL for branch commands. In all other
respects, a no-op.
`b LABEL'
Unconditionally branch to LABEL. The LABEL may be omitted, in
which case the next cycle is started.
`t LABEL'
Branch to LABEL only if there has been a successful `s'ubstitution
since the last input line was read or conditional branch was taken.
The LABEL may be omitted, in which case the next cycle is started.
Example 1. How to remove all the new line characters in a file and replace it with a space?
bash-3.2$ sed ':a;N;$!ba;s/\n/ /g' test.txt 1. leaf is green 2. sky is blue 3. snow is white 4. moon is white. 5. flower is yellow. bash-3.2$
Explanation: Here the command ‘:a;N;$!ba’ appends all the lines of a file into the pattern space as shown below. Once all the lines are appended into pattern space we
use a simple substitution command to replace all the \n to space (s/\n/ /g). Flag ‘g’ specifies global replacement.
bash-3.2$ sed ':a;N;$!ba' test.txt 1. leaf is green 2. sky is blue 3. snow is white 4. moon is white. 5. flower is yellow. bash-3.2$
Explanation: Here first we created label a (:a). Then appended the second line to the first line in pattern space (N). Then branch to label ‘:a’ (ba) if it is not the last line of the file ($!). As $! is an address condition, we need not separate $! and ba with a semicolon. As sed is invoked with default option, sed autoprints the pattern space at the end after appending all lines. Once all the lines are appended into pattern space we can do global replacements using substitution command.
One common error people does here is to use sed command like this: sed ‘s/\n/ /g’ test.txt when trying to remove all \n characters. This will not work because, sed reads one line from the input stream, remove trailing new line and places it in the pattern space and then perform the substitution operation. Then print the output and add back the trailing new line if it was removed and proceed with next line. Hence in the overall output \n would not have been removed.
Example 2: Another way to append all lines into the pattern space to remove new line characters (Without using label and branching).
bash-3.2$ sed -n '1h;2,$H;${g;p}' test.txt
1. leaf is green
2. sky is blue
3. snow is white
4. moon is white.
5. flower is yellow.
bash-3.2$
Explanation:
Basically here are appending the whole file to hold space and then move the hold space to pattern space and print it.
1h: Replace the hold space with the pattern space for line 1.
2,$H: From line 2 to last line ($), append the lines to hold space.
${g;p}: Only for last line ($), replace the pattern space with contents of hold space (g) and then print the consolidated pattern space (p).
Once we have the appended input we can use the substitution command to replace the new line (\n) characters.
bash-3.2$ sed -n '1h;2,$H;${g;s/\n/ /g;p}' test.txt
1. leaf is green 2. sky is blue 3. snow is white 4. moon is white. 5. flower is yellow.
bash-3.2$
Example 3: How to reverse the lines of a file using sed (This is from ‘sed’ man page).
bash-3.2$ sed -n '1!G;h;$p' test.txt 5. flower is yellow 4. moon is white 3. snow is white 2. sky is blue 1. leaf is green bash-3.2$
Explanation: For line 1, G will not be executed because of the address condition (1!). Line 1 will be added into hold space (h).
From line 2, till the end of file, the line in hold space will be appended after the line in pattern space (G) and then stored into hold space (h).For example, in cycle2, line 1 from hold space will be added to line 2 in pattern space. Then pattern space will be stored in hold space. For cycle 3, line2,1 will be added to line 3 of pattern space and will be stored into hold space and so on. After adding the hold space to the last line, the pattern space will be printed using the command $p where $ specifies the last line. Thus reversing the order.
Example 4: Very nice piece of code to print the last n lines using sed(This is from ‘sed’ man page). The method is called sliding window method which will be very useful for a lot of different operations using sed. For this particular example lets add more lines to the sample test.txt file as shown below.
bash-3.2$ cat test.txt 1. leaf is green 2. sky is blue 3. snow is white 4. moon is white 5. flower is yellow 6. This is line 6 7. This is line 7 8. This is line 8 9. This is line 9 10. This is line 10 11. This is line 11 12. This is line 12 13. This is line 13 14. This is line 14 15. This is line 15 16. This is line 16 17. This is line 17 18. This is line 18 19. This is line 19 bash-3.2$
Command to print last n lines using slide window method.
sed ‘1h;2,<>{H;g};$q;1,<>d;N;D’ test.txt
For example, if we want to pring the last 4 lines below is the code.
bash-3.2$ sed '1h;2,4{H;g};$q;1,3d;N;D' test.txt
16. This is line 16
17. This is line 17
18. This is line 18
19. This is line 19
Explanation:
1h – For 1st line – Replace the hold space with pattern space.
2,4{H;g} – For lines 2 to 4, append pattern space to hold space (H). Then replace the pattern space with hold space (g).
 - Basically this command load the pattern space with the 1st four lines.
1,3d – Will remove the unwanted repetition of lines leaving only the 1st four lines in the pattern space.
$q – Exit on the last line of the file.
N;D – Append the 5th line to the pattern space (N) and remove the 1st line (D), then append the 6th line (N) and remove the 2nd line (D) and so on. This loop makes sure only the last four lines are availabe in the pattern space till it reaches the end of the file. Once the loop completes the pattern space is left with only the last four lines.
Example 5: Search for a particular pattern or word and if found, add the matched line to the previous line and print only the appended line.
The sample file that will be used for the below examples is provided below.
bash-3.2$ cat test_sed.txt some text some text OK (200 msec) ---------------------Hostname1-------------------Database1 some text some text OK (60 msec) ---------------------Hostname2-------------------Database2 some text some text some text OK (50 msec) ---------------------Hostname3-------------------Database3 some text some text some text Connection failed ---------------------Hostname4-------------------Database4 End
Now search for the pattern ‘—–‘ and if found, append the matched line to the end of previous line and print only the newly appended line.
bash-3.2$ sed -n ':a;$!N;s/\n\(-----\)/\1/;ta;/-----/P;D' test_sed.txt OK (200 msec)---------------------Hostname1-------------------Database1 OK (60 msec)---------------------Hostname2-------------------Database2 OK (50 msec)---------------------Hostname3-------------------Database3 Connection failed---------------------Hostname4-------------------Database4
Explanation:
:a – Create a label a.
$!N – Append the second line to line 1 in pattern space. (N). The condition ($!) specifies perform append (N) if it is not last line of file. So the pattern space will have two lines.
s/\n\(—–\)/\1/ – Search for the line that starts with the pattern ‘—–‘ and if found, remove new line and replace the pattern ‘—–‘ as it is (\1). This appends the matched line to the end of previous line.
ta – If there is a successful substitution, branch to label a. Now again the next line will be appended to the pattern space. If \n—– pattern is not found in the pattern space then ‘P;D’ will be executed.
/—–/P;D – Print the pattern space only till the first \n (P) if the pattern —– is found. Then delete that line (D). Again the next cycle continues till the end.
Example 6: Search for a particular pattern or word and if found, append the previous line to the end of matched line and print only the appended line.
Search for the pattern ‘—–‘ and if found, append the previous line to the end of matched line.
bash-3.2$ sed -n ':a;$!N;s/\(^.*\)\n\(-----.*\)/\2---------\1/;ta;/-----/P;D' test_sed.txt ---------------------Hostname1-------------------Database1---------OK (200 msec) ---------------------Hostname2-------------------Database2---------OK (60 msec) ---------------------Hostname3-------------------Database3---------OK (50 msec) ---------------------Hostname4-------------------Database4---------Connection failed
Explanation: This example is similar to the above example.
Only change is in the substitution command. In it we swapped the matched line and previous line. Reset all are similar to Example 5.