This article is part of the article series "Sed One-Liners Explained."
<- previous article next article ->
sed -- the superman of unix stream editing

Inspired by the success of my "Awk One-Liners Explained" article (30,000 views in first three days), I decided to explain the famous sed one-liners as well.

These one-liners, just like the Awk one-liners, are compiled by Eric Pement.

You may download them here: sed one-liners (link to .txt file).

Most people are only familiar with one particular command of sed, namely the "s" (substitute) comand. s/comand/command/. That is unsatisfactory. Sed has at least 20 different commands for you. You can even write tetris in it (not to mention that it's Turing complete).

My sed learning process was actually identical to Awk learning process. First I went through Bruce Barnett's sed tutorial; then I created a sed cheat sheet; and finally went through sed one-liners. I could not figure one of the one-liners in the file, so I ended up asking for help in comp.unix.shell.

Eric's sed one-liners are divided into several sections:

Update: Spanish translation of part one is available!

I'll divide this article in 3 parts. In the first part I will cover "File spacing", "Numbering" and "Text conversion and substitution". In the second part I will cover "Selective printing of certain lines" and in the third "Selective deletion of certain lines" and "Special applications".

Before I start explaining, I want to share the key idea that changed the way I think about sed. It was the four spaces of sed -- input stream, output stream, pattern space, hold buffer. Sed operates on input stream and produces an output stream. The lines from input stream are placed into the pattern space where they are modified. The hold buffer can be used for temporary storage. These four spaces changed the way I think about sed.

Awesome news: I have written an e-book based on this article series. Check it out:

Grab a copy of sed cheat sheet, print it and let's dive into one-liners!

1. File spacing.

1. Double-space a file.

sed G

This sed one-liner uses the 'G' command. If you grabbed the cheat-sheet you'd see that the command 'G' appends to the pattern space. It appends a newline followed by the contents of hold buffer. In this example the hold buffer is empty all the time (only three commands 'h', 'H' and 'x' modify hold buffer), so we end up simply appending a newline to the pattern space. Once all the commands have been processed (in this case just the 'G' command), sed puts the contents of pattern space to output stream followed by a newline. There we have it, two newlines -- one added by the 'G' command and the other by output stream. File has been double spaced.

2. Double-space a file which already has blank lines in it. Do it so that the output contains no more than one blank line between two lines of text.

sed '/^$/d;G'

Sed allows to restrict commands only to certain lines. This one-liner operates only on lines that match a regular expression /^$/. Which are those? Those are the empty lines. Note that before doing the regular expression match sed pushes the input line to pattern space. When doing it, sed strips the trailing newline character. The empty lines contain just the newline character, so after they have been put into pattern space, this only character has been removed and pattern space stays empty. Regular expression /^$/ matches an empty pattern space and sed applies 'd' command on it, which deletes the current pattern space, reads in the next line, puts it into the pattern space and aborts the current command, and starts the execution from the beginning. The lines which do not match emptiness get a newline character appended by the 'G' command, just like in one-liner #1.

In general sed allows to restrict operations to certain lines (5th, 27th, etc.), to a range of lines (lines 10-20), to lines matching a pattern (lines containing the word "catonmat"), and to lines between two patterns (lines between "catonmat" and "coders").

3. Triple-space a file.

sed 'G;G'

Several sed commands can be combined by separating them by ';'. Such commands get executed one after another. This one-liner does twice what the one-liner #1 does -- appends two newlines (via two 'G' commands) to output.

4. Undo double-spacing.

sed 'n;d'

This one-liner assumes that even-numbered lines are always blank. It uses two new commands - 'n' and 'd'. The 'n' command prints out the current pattern space (unless the '-n' flag has been specified), empties the current pattern space and reads in the next line of input. We assumed that even-numbered lines are always blank. This means that 'n' prints the first, third, fifth, ..., etc. line and reads in the following line. The line following the printed line is always an empty line. Now the 'd' command gets executed. The 'd' command deletes the current pattern space, reads in the next line, puts the new line into the pattern space and aborts the current command, and starts the execution from the first sed command. Now the the 'n' commands gets executed again, then 'd', then 'n', etc.

To make it shorter - 'n' prints out the current line, and 'd' deletes the empty line, thus undoing the double-spacing.

5. Insert a blank line above every line that matches "regex".

sed '/regex/{x;p;x;}'

This one liner uses the restriction operation together with two new commands - 'x' and 'p'. The 'x' command exchanges the hold buffer with the pattern buffer. The 'p' command duplicates input -- prints out the entire pattern space. This one-liner works the following way: a line is read in pattern space, then the 'x' command exchanges it with the empty hold buffer. Next the 'p' command prints out emptiness followed by a newline, so we get an empty line printed before the actual line. Then 'x' exchanges the hold buffer (which now contains the line) with pattern space again. There are no more commands so sed prints out the pattern space. We have printed a newline followed by the line, or saying it in different words, inserted a blank line above every line.

Also notice the { ... }. This is command grouping. It says, execute all the commands in "..." on the line that matches the restriction operation.

6. Insert a blank line below every line that matches "regex".

sed '/regex/G'

This one liner combines restriction operation with the 'G' command, described in one-liner #1. For every line that matches /regex/, sed appends a newline to pattern space. All the other lines that do not match /regex/ just get printed out without modification.

7. Insert a blank line above and below every line that matches "regex".

sed '/regex/{x;p;x;G;}'

This one-liner combines one-liners #5, #6 and #1. Lines matching /regex/ get a newline appended before them and printed (x;p;x from #5). Then they are followed by another newline from the 'G' command (one-liner #6 or #1).

2. Numbering.

8. Number each line of a file (named filename). Left align the number.

sed = filename | sed 'N;s/\n/\t/'

One-liners get trickier and trickier. This one-liner is actually two separate one-liners. The first sed one-liner uses a new command called '='. This command operates directly on the output stream and prints the current line number. There is no way to capture the current line number to pattern space. That's why the second one-liner gets called. The output of first one-liner gets piped to the input of second. The second one-liner uses another new command 'N'. The 'N' command appends a newline and the next line to current pattern space. Then the famous 's///' command gets executed which replaces the newline character just appended with a tab. After these operations the line gets printed out.

To make it clear what '=' does, take a look at this example file:

line one
line two
line three

Running the first one-liner 'sed = filename', produces output:

1
line one
2
line two
3
line three

Now, the 'N' command of the second one-liner joins these lines with a newline character:

1\nline one
2\nline two
3\nline three

The 's/\n/\t/' replaces the newline chars with tabs, so we end up with:

1     line one
2     line two
3     line three

The example is a little inaccurate as line joining with a newline char happens line after line, not on all lines at once.

9. Number each line of a file (named filename). Right align the number.

sed = filename | sed 'N; s/^/     /; s/ *\(.\{6,\}\)\n/\1  /'

This one-liner is also actually two one-liners. The first one liner numbers the lines, just like #8. The second one-liner uses the 'N' command to join the line containing the line number with the actual line. Then it uses two substitute commands to right align the number. The first 's' command 's/^/ /' appends 5 white-spaces to the beginning of line. The second 's' command 's/ *\(.\{6,\}\)\n/\1 /' captures at least six symbols up to a newline and replaces the capture and newline with the back-reference '\1' and two more whitespace to separate line number from the contents of line.

I think it's hard to understand the last part of this sed expression by just reading. Let's look at an example. For clearness I replaced the '\n' newline char with a '@' and whitespace with '-'.

$ echo "-----12@contents" | sed 's/-*\(.\{6,\}\)@/\1--/'
----12--contents

The regular expression '-*\(.\{6,\}\)@' (or just '-*(.{6,})@') tells sed to match some '-' characters followed by at least 6 other characters, followed by a '@' symbol. Sed captures them (remembers them) in \1.

In this example sed matches the first '-' (the '-*' part of regex), then the following six characters "----12" and '@' (the '*\(.\{6,\}\)@' part of regex). Now it replaces the matched part of the string "-----12@" with the contents of captured group which is "----12" plus two extra whitespace. The final result is that "-----12@" gets replaced with "----12--".

10. Number each non-empty line of a file (called filename).

sed '/./=' filename | sed '/./N; s/\n/ /'

This one-liner is again two one-liners. The output of the first one-liner gets piped to the input of second. The first one-liner filters out lines with at least one character in them. The regular expression '/./' says: match lines with at least one char in them. When the empty lines (containing just a newline) get sent to the pattern space, the newline character gets removed, so the empty lines do not get matched. The second one-liner does the same one-liner #8 did, except that only numbered lines get joined and printed out. Command '/./N' makes sure that empty lines are left as-is.

11. Count the number of lines in a file (emulates "wc -l").

sed -n '$='

This one-liner uses a command line switch "-n" to modify sed's behavior. The "-n" switch tells sed not to send the line to output after it has been processed in the pattern space. The only way to make sed output anything with the "-n" switch being on is to use a command that modifies the output stream directly (these commands are '=', 'a', 'c', 'i', 'I', 'p', 'P', 'r' and 'w'). In this one-liner what seems to be the command "$=" is actually a restriction pattern "$" together with the "=" command. The restriction pattern "$" applies the "=" command to the last line only. The "=" command outputs the current line number to standard output. As it is applied to the last line only, this one-liner outputs the number of lines in the file.

3. Text Conversion and Substitution.

12. Convert DOS/Windows newlines (CRLF) to Unix newlines (LF).

sed 's/.$//'

This one-one liner assumes that all lines end with CR+LF (carriage return + line feed) and we are in a Unix environment. Once the line gets read into pattern space, the newline gets thrown away, so we are left with lines ending in CR. The 's/.$//' command erases the last character by matching the last character of the line (regex '.$') and substituting it with nothing. Now when the pattern space gets output, it gets appended the newline and we are left with lines ending with LF.

The assumption about being in a Unix environment is necessary because the newline that gets appended when the pattern space gets copied to output stream is the newline of that environment.

13. Another way to convert DOS/Windows newlines (CRLF) to Unix newlines (LF).

sed 's/^M$//'

This one-liner again assumes that we are in a Unix environment. It erases the carriage return control character ^M. You can usually enter the ^M control char literally by first pressing Ctrl-V (it's control key + v key) and then Ctrl-M.

14. Yet another way to convert DOS/Windows newlines to Unix newlines.

sed 's/\x0D$//'

This one-liner assumes that we are on a Unix machine. It also assumes that we use a version of sed that supports hex escape codes, such as GNU sed. The hex value for CR is 0x0D (13 decimal). This one-liner erases this character.

15-17. Convert Unix newlines (LF) to DOS/Windows newlines (CRLF).

sed "s/$/`echo -e \\\r`/"

This one-liner also assumes that we are in a Unix environment. It calls shell for help. The 'echo -e \\\r' command inserts a literal carriage return character in the sed expression. The sed "s/$/char/" command appends a character to the end of current pattern space.

18. Another way to convert Unix newlines (LF) to DOS/Windows newlines (CRLF).

sed 's/$/\r/'

This one-liner assumes that we use GNU sed. GNU sed is smarter than other seds and can take escape characters in the replace part of s/// command.

19. Convert Unix newlines (LF) to DOS/Windows newlines (CRLF) from DOS/Windows.

sed "s/$//"

This one-liner works from DOS/Windows. It's basically a no-op one-liner. It replaces nothing with nothing and then sends out the line to output stream where it gets CRLF appended.

20. Another way to convert Unix newlines (LF) to DOS/Windows newlines (CRLF) from DOS/Windows.

sed -n p

This is also a no-op one-liner, just like #19. The shortest one-liner which does the same is:

sed ''

21. Convert DOS/Windows newlines (LF) to Unix format (CRLF) from DOS/Windows.

sed "s/\r//"

Eric says that this one-liner works only with UnxUtils sed v4.0.7 or higher. I don't know anything about this version of sed, so let's just trust him. This one-liner strips carriage return (CR) chars from lines. Then when they get output, CRLF gets appended by magic.

Eric mentions that the only way to convert LF to CRLF on a DOS machine is to use tr:

tr -d \r <infile >outfile

22. Delete leading whitespace (tabs and spaces) from each line.

sed 's/^[ \t]*//'

Pretty simple, it matches zero-or-more spaces and tabs at the beginning of the line and replaces them with nothing, i.e. erases them.

23. Delete trailing whitespace (tabs and spaces) from each line.

sed 's/[ \t]*$//'

This one-liner is very similar to #22. It does the same substitution, just matching zero-or-more spaces and tabs at the end of the line, and then erases them.

24. Delete both leading and trailing whitespace from each line.

sed 's/^[ \t]*//;s/[ \t]*$//'

This one liner combines #22 and #23. First it does what #22 does, erase the leading whitespace, and then it does the same as #23, erase trailing whitespace.

25. Insert five blank spaces at the beginning of each line.

sed 's/^/     /'

It does it by matching the null-string at the beginning of line (^) and replaces it with five spaces "     ".

26. Align lines right on a 79-column width.

sed -e :a -e 's/^.\{1,78\}$/ &/;ta'

This one-liner uses a new command line option and two new commands. The new command line option is '-e'. It allows to write a sed program in several parts. For example, a sed program with two substitution rules could be written as "sed -e 's/one/two/' -e 's/three/four'" instead of "sed 's/one/two/;s/three/four'". It makes it more readable. In this one-liner the first "-e" creates a label called "a". The ':' command followed by a name crates a named label. The second "-e" uses a new command "t". The "t" command branches to a named label if the last substitute command modified pattern space. This branching technique can be used to create loops in sed. In this one-liner the substitute command left-pads the string (right aligns it) a single whitespace at a time, until the total length of the string exceeds 78 chars. The "&" in substitution command means the matched string.

Translating it in modern language, it would look like this:

while (str.length() <= 78) {
 str = " " + str
}

27. Center all text in the middle of 79-column width.

sed  -e :a -e 's/^.\{1,77\}$/ & /;ta'

This one-liner is very similar to #26, but instead of left padding the line one whitespace character at a time it pads it on both sides until it has reached length of at least 77 chars. Then another two whitespaces get added at the last iteration and it has grown to 79 chars.

Another way to do the same is

sed  -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/'

This one-liner left pads the string one whitespace char at a time until it has reached length of 78 characters. Then the additional "s/\( *\)\1/\1/" command gets executed which divides the leading whitespace "in half". This effectively centers the string. Unlike the previous one-liner this one-liner does not add trailing whitespace. It just adds enough leading whitespace to center the string.

28. Substitute (find and replace) the first occurrence of "foo" with "bar" on each line.

sed 's/foo/bar/'

This is the simplest sed one-liner possible. It uses the substitute command and applies it once on each line. It substitutes string "foo" with "bar".

29. Substitute (find and replace) the fourth occurrence of "foo" with "bar" on each line.

sed 's/foo/bar/4'

This one-liner uses a flag for the substitute command. With no flags the first occurrence of pattern is changed. With a numeric flag like "/1", "/2", etc. only that occurrence is substituted. This one-liner uses numeric flag "/4" which makes it change fourth occurrence on each line.

30. Substitute (find and replace) all occurrence of "foo" with "bar" on each line.

sed 's/foo/bar/g'

This one-liner uses another flag. The "/g" flag which stands for global. With global flag set, substitute command does as many substitutions as possible, i.e., all.

31. Substitute (find and replace) the first occurrence of a repeated occurrence of "foo" with "bar".

sed 's/\(.*\)foo\(.*foo\)/\1bar\2/'

Let's understand this one-liner with an example:

$ echo "this is foo and another foo quux" | sed 's/\(.*\)foo\(.*foo\)/\1bar\2/'
this is bar and another foo quux

As you can see, this one liner replaced the first "foo" with "bar".

It did it by using two capturing groups. The first capturing group caught everything before the first "foo". In this example it was text "this is ". The second group caught everything after the first "foo", including the second "foo". In this example " and another foo". The matched text was then replaced with contents of first group "this is " followed by "bar" and contents of second group " and another foo". Since " quux" was not part of the match it was left unchanged. Joining these parts the resulting string is "this is bar and another foo quux", which is exactly what we got from running the one-liner.

32. Substitute (find and replace) only the last occurrence of "foo" with "bar".

sed 's/\(.*\)foo/\1bar/'

This one-liner uses a capturing group that captures everything up to "foo". It replaces the captured group and "foo" with captured group itself (the \1 back-reference) and "bar". It results in the last occurrence of "foo" getting replaced with "bar".

33. Substitute all occurrences of "foo" with "bar" on all lines that contain "baz".

sed '/baz/s/foo/bar/g'

This one-liner uses a regular expression to restrict the substitution to lines matching "baz". The lines that do not match "baz" get simply printed out, but those that do match "baz" get the substitution applied.

34. Substitute all occurrences of "foo" with "bar" on all lines that DO NOT contain "baz".

sed '/baz/!s/foo/bar/g'

Sed commands can be inverted and applied on lines that DO NOT match a certain pattern. The exclamation "!" before a sed commands does it. In this one-liner the substitution command is applied to the lines that DO NOT match "baz".

35. Change text "scarlet", "ruby" or "puce" to "red".

sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g'

This one-liner just uses three consecutive substitution commands. The first replaces "scarlet" with "red", the second replaced "ruby" with "red" and the last one replaces "puce" with "red".

If you are using GNU sed, then you can do it simpler:

gsed 's/scarlet\|ruby\|puce/red/g'

GNU sed provides more advanced regular expressions which support alternation. This one-liner uses alternation and the substitute command reads "replace 'scarlet' OR 'ruby' OR 'puce' with 'red'".

36. Reverse order of lines (emulate "tac" Unix command).

sed '1!G;h;$!d'

This one-liner acts as the "tac" Unix utility. It's tricky to explain. The easiest way to explain it is by using an example.

Let's use a file with just 3 lines:

$ cat file
foo
bar
baz

Running this one-liner on this file produces the file in reverse order:

$ sed '1!G;h;$!d' file
baz
bar
foo

The first one-liner's command "1!G" gets applied to all the lines which are not the first line. The second command "h" gets applied to all lines. The third command "$!d" gets applied to all lines except the last one.

Let's go through the execution line by line.

Line 1: Only the "h" command gets applied for the first line "foo". It copies this line to hold buffer. Hold buffer now contains "foo". Nothing gets output as the "d" command gets applied.
Line 2: The "G" command gets applied. It appends the contents of hold buffer to pattern space. The pattern space now contains. "bar\nfoo". The "h" command gets applied, it copies "bar\nfoo" to hold buffer. It now contains "bar\nfoo". Nothing gets output.
Line 3: The "G" command gets applied. It appends hold buffer to the third line. The pattern space now contains "baz\nbar\nfoo". As this was the last line, "d" does not get applied and the contents of pattern space gets printed. It's "baz\nbar\nfoo". File got reversed.

If we had had more lines, they would have simply get appended to hold buffer in reverse order.

Here is another way to do the same:

sed -n '1!G;h;$p'

It silences the output with "-n" switch and forces the output with "p" command only at the last line.

These two one-liners actually use a lot of memory because they keep the whole file in hold buffer in reverse order before printing it out. Avoid these one-liners for large files.

37. Reverse a line (emulates "rev" Unix command).

sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'

This is a very complicated one-liner. I had trouble understanding it the first time I saw it and ended up asking on comp.unix.shell for help.

Let's re-format this sed one-liner:

 sed '
   /\n/ !G
   s/\(.\)\(.*\n\)/&\2\1/
   //D
   s/.//
 ' 

The first line "/\n/ !G" appends a newline to the end of the pattern space if there was none.

The second line "s/\(.\)\(.*\n\)/&\2\1/" is a simple s/// expression which groups the first character as \1 and all the others as \2. Then it replaces the whole matched string with "&\2\1", where "&" is the whole matched text ("\1\2"). For example, if the input string is "1234" then after the s/// expression, it becomes "1234\n234\n1".

The third line is "//D". This statement is the key in this one-liner. An empty pattern // matches the last existing regex, so it's exactly the same as: /\(.\)\(.*\n\)/D. The "D" command deletes from the start of the input till the first newline and then resumes editing with first command in script. It creates a loop. As long as /\(.\)\(.*\n\)/ is satisfied, sed will resume all previous operations. After several loops, the text in the pattern space becomes "\n4321". Then /\(.\)\(.*\n\)/ fails and sed goes to the next command.

The fourth line "s/.//" removes the first character in the pattern space which is the newline char. The contents in pattern space becomes "4321" -- reverse of "1234".

There you have it, a line has been reversed.

38. Join pairs of lines side-by-side (emulates "paste" Unix command).

sed '$!N;s/\n/ /'

This one-liner joins two consecutive lines with the "N" command. They get joined with a "\n" character between them. The substitute command replaces this newline with a space, thus joining every pair of lines with a whitespace.

39. Append a line to the next if it ends with a backslash "\".

sed -e :a -e '/\\$/N; s/\\\n//; ta'

The first expression ':a' creates a named label "a". The second expression looks to see if the current line ends with a backslash "\". If it does, it joins it with the line following it using the "N" command. Then the slash and the newline between joined lines get erased with "s/\\\n//" command. If the substitution was successful we branch to the beginning of expression and do the same again, in hope that we might have another backslash. If the substitution was not successful, the line did not end with a backslash and we print it out.

Here is an example of running this one-liner:

$ cat filename
line one \
line two
line three
$ sed -e :a -e '/\\$/N; s/\\\n//; ta' filename
line one line two
line three

Lines one and two got joined because the first line ended with backslash.

40. Append a line to the previous if it starts with an equal sign "=".

sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'

This one-liner also starts with creating a named label "a". Then it tests to see if it is not the last line and appends the next line to the current one with "N" command. If the just appended line starts with a "=", one-liner branches the label "a" to see if there are more lines starting with "=". During this process a substitution gets executed which throws away the newline character which came from joining with "N" and the "=". If the substitution fails, one-liner prints out the pattern space up to the newline character with the "P" command, and deletes the contents of pattern space up to the newline character with "D" command, and repeats the process.

Here is an example of running it:

$ cat filename
line one
=line two
=line three
line four
$ sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D' filename
line one line two line three
line four

Lines one, two and three got joined, because lines two and three started with '='. Line four got printed as-is.

41. Digit group (commify) a numeric string.

sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta'

This one-liner turns a string of digits, such as "1234567" to "1,234,567". This is called commifying or digit grouping.

First the one-liner creates a named label "a". Then it captures two groups of digits. The first group is all the digits up to last three digits. The last three digits gets captures in the 2nd group. Then the two matching groups get separated by a comma. Then the same rules get applied to the line again and again until all the numbers have been grouped in groups of three.

Substitution command "\1,\2" separates contents of group one with a comma from the contents of group two.

Here is an example to understand the grouping happening here better. Suppose you have a numeric string "1234567". The first group captures all the numbers until the last three "1234". The second group captures last three numbers "567". They get joined by a comma. Now the string is "1234,567". The same stuff is applied to the string again. Number "1" gets captured in the first group and the numbers "234" in the second. The number string is "1,234,567". Trying to apply the same rules again fail because there is just one digit at the beginning of string, so the string gets printed out and sed moves on to the next line.

If you have GNU sed, you can use a simpler one-liner:

gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta'

This one-liner starts with creating a named label "a" and then loops over the string the same way as the previous one-liner did. The only difference is how groups of three digits get matched. GNU sed has some additional patterns. There are two patterns that make this one-liner work. The first is "\B", which matches anywhere except at a word boundary. It's needed so we did not go beyond word boundary. Look at this example:

$ echo "12345 1234 123" | sed 's/[0-9]\{3\}\>/,&/g'
12,345 1,234 ,123

It's clearly wrong. The last 123 got a comma added. Adding the "\B" makes sure we match the numbers only at word boundary:

$ echo "12345 1234 123" | sed 's/\B[0-9]\{3\}\>/,&/g'
12,345 1,234 123

The second is "\>". It matches the null string at the end of a word. It's necessary because we need to to match the right-most three digits. If we did not have it, the expression would match after the first digit.

42. Add commas to numbers with decimal points and minus signs.

gsed -r ':a;s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g;ta'

This one-liner works in GNU sed only. It turns on extended regular expression support with the "-r" switch. Then it loops over a line matching three groups and separates the first two from the third with a comma.

The first group makes sure we ignore a leading non-digit character, such as + or -. If there is no leading non-digit character, then it just anchors at the beginning of the string which always matches.

The second group matches a bunch of numbers. The third group makes sure the second group does not match too many. It matches 3 consecutive numbers at the end of the string.

Once the groups have been captured, the "\1\2,\3" substitution is done and the expression is looped again, until the whole string has been commified.

43. Add a blank line after every five lines.

sed 'n;n;n;n;G;'

The "n" command is called four times in this one-liner. Each time it's called it prints out the current pattern space, empties it and reads in the next line of input. After calling it four times, the fifth line is read into the pattern space and then the "G" command gets called. The "G" command appends a newline to the fifth line. Then the next round of four "n" commands is done. Next time the first "n" command is called it prints out the newlined fifth line, thus inserting a blank line after every 5 lines.

The same can be achieved with GNU sed's step extension:

gsed '0~5G'

GNU sed's step extensions can be generalized as "first~step". It matches every "step"'th line starting with line "first". In this one-liner it matches every 5th line starting with line 0.

Sed One-Liners Explained E-Book

I have written an e-book called "Sed One-Liners Explained". I improved the explanations of the one-liners in this article series, added new one-liners and added three new chapters - an introduction to sed, a summary of sed addresses and ranges, and debugging sed scripts with sed-sed. Please take a look:

Have fun!

Have fun with sed, the superman of unix stream editing. The second part of this article will be on "Selective printing of certain lines." I hope to see you on my blog again for the 2nd part of the article! :)

This article is part of the article series "Sed One-Liners Explained."
<- previous article next article ->

Comments

Paolo Bonzini Permalink
October 08, 2008, 11:05

Regarding the comment that the shortest one-liner which does a null-filter is

sed ''

actually if you count the number of characters to type, the shortest is

sed b

Also, regarding oneliner 37, it could be a little easier to understand if the //D command is changed to /^[^\n]/D. Alternatively you can use

/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;s/^\n//p;D

October 08, 2008, 13:48

This is real nice.

How good is sed for processing logs? I might use this to handle my debug logs.

October 08, 2008, 15:43

Wow, Peteris what a list! Very useful, thanks!

Rudi Permalink
October 08, 2008, 17:49

nice blog .. i like it .

October 08, 2008, 18:49

Paolo, thanks for the comment about shortest sed null-filter and one liner #37.

Anirudh, it's not good for processing logs, because you need complicated calculations, such as date/time calculations, various counters, keep lists of referrer links, etc. Not something sed can do easily.

Eric, you're welcome! :)

Rudi, thanks :)

RenaudD Permalink
October 08, 2008, 23:57

I am a sed addict and your nice site won't help me to cure. I love the

sed -n '$='

:).
Maybe this will interest you :
Sometimes you have to print blocks of lines that contains a special string. For instance if filename contains :

foo
baz
bar

bar
foo

goo
baz

and you would like to get only the blocks that contain 'baz' that is :

foo
baz
bar

goo
baz

I do sthg close to it that way :

sed -n 'H;/^$/x;/baz.*\n/p' filename

I said 'close to it' because unfortunately this command prints a bit too many empty lines before the selected blocks but generally I don't mind. Anyway if you find a way to straighten it I would be delighted to know it.

Note : you have to make sure the last line of filename is an empty line otherwise the last block won't be processed :(

Thanks again for the site !

PS sorry I am not very talented for HTML, I hope all this will print correctly

October 09, 2008, 23:32

Thanks once again. I was thinking of learning sed since a long time. Your post now gives me a motivation and a very good way to start it.

One question: PERL doesn't allow multi-line comments. How to emulate it using sed.
Ex:
If I give the following at some part of my code:
/*
line 1 of code to be commented
line 2
.
.
.
*/
actual code
.
.
/*
another commented section
*/

How to perl-comment these lines using sed?

anonymous Permalink
October 10, 2008, 16:02

To ankush:

Note that perl DOES allow multiline comments, using POD syntax.

with sed, you can do this:

sed '\|/\*|,\|\*/|s/^/#/'

that assumes your comments always span more than one line (ie, the opening /* and the closing */ are on different lines). Otherwise, you can do this:

sed '\|/\*|{:a;s/^/#/;\|\*/|!{n;ba;}}'

(good luck reading that!)

Note that both solutions are VERY fragile, and will break under a number of conditions.

Roman Permalink
October 10, 2008, 23:53

Peteris:

In #9, there is a regex pattern: " *\(.\{6,\}\)\n". However, after being understandablificated, it becomes "\(.\{6,\}\)@/" (the space-star at the beginning is lost). It is wrong, since it doesn't remove extraneous space at the beginning of the line, and hence doesn't right-align.

Also, it looks like 4 dashes in the explanation were accidentally merged into an über-dash. 8=]

October 13, 2008, 06:03

Peter,

Thanks so much for this and your previous article on AWK. Thanks especially for explaining how you achieved your understanding of these two useful tools. I was afraid it involved digging into the sources or some such daunting task. Although I am not a geek, I find these two tools very useful in everyday business tasks.

Thanks for your clear explanations of the "magic" of Sed and AWK.

Goofy Barney

Raj Permalink
November 20, 2008, 10:52

HI,

I want to remove all occurence of quotes (") but not if it occures at first and last of a lines.

How can I achieve this?

Regards,
Raj

December 21, 2008, 04:10

Raj, you can do it this way:

sed '1b;$b;s/"//g'

(Remove all occurence of quotes (") but not if it occures at first and last of a lines).

manish Permalink
January 05, 2009, 12:04

Hi All, i have a query....i want to search a refrence number in a given file (say ref num ABC123) and then copy the next 20 line after the reference number and exit....please help me how to do the same....thnx in advance

February 13, 2009, 12:34

In #31, if the pattern (foo) comes more than two times in the line then it replaces the second last one. As an example

echo "this is first foo, this is second foo another foo quux" | sed 's/\(.*\)foo\(.*foo\)/\1bar\2/'

then output will be:
this is first foo, this is second bar another foo quux

Which doesn't fulfill our requirements.

Regards,
Yogesh

Jeremy Permalink
February 20, 2009, 20:28

Rather then pipe sed to sed:
sed command1 | sed command2
you can simply use the -e option:
sed command1 -e command2

February 21, 2009, 00:03

Jeremy,

Generally yes, but not if the first command modifies output buffer directly. The '=' command for example modifies output stream directly and the following -e command is not able to modify the same line anymore because it has already been output.

Vijay Manohar Permalink
March 20, 2009, 06:14

Sir, I immensely benefited out your sed 1 liners descriptions.

Thanks and regards,
Vijay Manohar

Nicasio Permalink
April 14, 2009, 08:00

Hey,
I have a big file (bigger than 10MG) and this have one line only.
I want separate in different lines always shows the literal "A01".
Help.

June 24, 2009, 07:07

Hi Peter,
just to say Thanks for the one-liners!! I am a beginner in scripting, [coming from Windows world, trying to migrate to linux/OpenSource] finding it difficult to grasp the "quickies" using scripts; but your site helped me a lot to have quick solutions to my problems.. and learn from your description. [esp. using sed/awk] And I liked the google interview you have put as well..

-Manoj

Erik Permalink
July 29, 2009, 17:15

This is one that was driving me crazy - couldn't find it anywhere on the net:

DELETE ALL LINES BETWEEN TWO MARKER PATTERNS (but not the markers themselves):

e.g. in a hosts file

== /etc/hosts ==

192.168.1.2 host1
192.168.1.3 host2

### START ###
#delete
192.168.1.4 host3 #delete
192.168.1.5 host4 #delete
#delete
### END ###

192.168.1.6 host5

=============================

The money:

sed -i -e '1,/^### START ###/!{ /^### END ###/,/^### START ###/!d; }' /etc/hosts

I have no idea how it works, but it does. Now your markers are preserved for you to insert other stuff with your funky automation scripts.

Erik Permalink
July 29, 2009, 17:19

by the way, it doesn't work if the marker is on the first line. ho hum!

Paul Permalink
July 29, 2009, 17:56

More understandable in awk. Use a BEGIN case to initialise marker patterns e.g. reB = "^### START ###" and similarly reE. Then fast-forward over the section you want to delete.

awk '{ print; if ($0 ~ reB) while (getline > 0) if ($0 ~ reE) { print; break; }'

July 31, 2009, 03:50

Paul, thanks for your wonderful comments! I will use them to improve my articles!

January 22, 2010, 19:45

Hi,
I'm trying to extract stacktraces from log files, looking for the pattern "Exception". If a line contains "Exception", a stack trace will follow in multiple lines. If the next line begins with a timestamp entry in the format "[1/13/10 23:17:00:444 CST]", the stack trace has ended in the previous line. Additionally, I want to grab the nearest line containing a timestamp entry above the line containing the pattern "Exception". Sometimes it's on the same line and other times it's a few lines above it. How can I do write a sed command for this? Example:

[1/13/10 23:01:16:623 CST] 00000059 SystemOut     O 2010-01-13 23:01:16,623 [ORB.thread.pool : 0] INFO  
 - Exiting isAlertUpdateTimerRunning()
SystemOut     O 2010-01-13 23:01:16,623 [ORB.thread.pool : 0] INFO  com.dd.dddd.ddddddddr.ddddddr.dddddddddTriggerManager
org.hibernate.SessionException: Session is closed!
        at org.hibernate.impl.AbstractSessionImpl.errorIfClosed(AbstractSessionImpl.java:72)
        at org.hibernate.impl.SessionImpl.reconnect(SessionImpl.java:407)
        at com.ibm.ws.Transaction.JTA.RegisteredSyncs.distributeAfter(RegisteredSyncs.java:424)
        at com.ibm.ws.Transaction.JTA.TransactionImpl.distributeAfter(TransactionImpl.java:3885)
        at com.ibm.ws.Transaction.JTA.TransactionImpl.postCompletion(TransactionImpl.java:3864)
        at com.ibm.ws.Transaction.JTA.TransactionImpl.commitXAResources(TransactionImpl.java:2521)
        at com.ibm.ws.Transaction.JTA.TransactionImpl.stage1CommitProcessing(TransactionImpl.java:1647)
        at com.ibm.ws.Transaction.JTA.TransactionImpl.processCommit(TransactionImpl.java:1607)
        at com.ibm.ws.Transaction.JTA.TransactionImpl.commit(TransactionImpl.java:1542)
        at com.ibm.ws.Transaction.JTA.TranManagerImpl.commit(TranManagerImpl.java:240)
        at com.ibm.ws.Transaction.JTA.TranManagerSet.commit(TranManagerSet.java:164)
        at com.ibm.ejs.csi.TranStrategy.commit(TranStrategy.java:756)
        at com.ibm.ejs.csi.TranStrategy.postInvoke(TranStrategy.java:181)
        at com.ibm.ejs.csi.TransactionControlImpl.postInvoke(TransactionControlImpl.java:581)
        at com.ibm.ejs.container.EJSContainer.postInvoke(EJSContainer.java:3910)
        at com.ibm.ejs.container.EJSContainer.postInvoke(EJSContainer.java:3732)
[1/13/10 23:01:16:725 CST] 0000006e SystemOut     O 2010-01-13 23:01:16,725 [MessageListenerThreadPool : 12] INFO  com.ddconnector.cccccc.ccccccSession - Th
read[MessageListenerThreadPool : 12,5,main]

Here, I want to grab the first line, followed by the stack trace line starting "org.hibernate.SessionException" all the way until the last line of the stack trace.
Thanks.

Robby Permalink
September 28, 2010, 00:06

This comment is a little older, but I found it as a challenge none-the-less and decided to take a crack at it & post my results in case somebody later had the same question. As a bonus (& to make it more useful for myself), it prints the line number above each exception. Here is my result:

sed -e :b -e '/Exception/!d;=;n;:a' -e '/^\[/bb' -e 'n;ba'

Brief explanation in steps:
1. :b set a label to go back to the beginning. i had to use a label, b/c i wanted to be able to go back to the beginning w/o reloading the pattern space.
2. /Exception/!d start over w/ the next line unless we find the word Exception.
3. = print out the line number
4. n print out the current line and load the next line
5. :a set another label for looping over the body of the exception
6. /^\[/bb go back to the beginning if we find a line starting with a bracket. i didn't see a need to expand the regex to match the full timestamp.
7. n;ba print out the current line and loop back to label a

Les Permalink
March 09, 2010, 21:49

I really benefited from and enjoyed this great compilation and explanation. Thank you.

Eszter Permalink
August 12, 2010, 09:30

Hi!
Nice intro, thanks a lot!

But I still got a question:
How can I replace every "-"-s at the beginning and the end of a line with the same number of "N"-s?

for example:
in:
>taxon_1
----tgattagcat---
>taxon_2
tgattag---tga-tag-

out:
>taxon_1
NNNNtgattagcatNNN
>taxon_2
tgattag---tga-tagN

Robby Permalink
September 27, 2010, 23:05

Great article!

@Eszter
Nice challenge. There could be a shorter way to do this, but I used the branch technique with labels described above and came up with:

sed -e :a -e 's/^\(-*\)-/\1N/;ta' -e :b -e 's/-\(-*\)$/N\1/;tb'

The first two expressions set a label then loop substituting the dashes inside out. Then I basically mirrored the process for the backside.

$ cat taxon
----tgattagcat---
tgattag---tga-tag-
---tafasf--fasdf--

$ cat taxon | sed -e :a -e 's/^\(-*\)-/\1N/;ta' -e :b -e 's/-\(-*\)$/N\1/;tb'
NNNNtgattagcatNNN
tgattag---tga-tagN
NNNtafasf--fasdfNN
December 14, 2010, 05:00

This really was an excellent way to learn sed commands. Thank you.

I like the braces vs -e. Any problems with that, std wise.

This:
-- sed -e :a -e 's/^.\{1,78\}$/ &/;ta'
verses:
-- sed '{:a; s/^.\{1,78\}$/ &/; ta}'

If it's something I'm keeping I put the commands on separate lines and use # comments at the end of the line for each command, as necessary. The opengroup doc says, "Ignore the '#' and the remainder of the line", so I think modern sed is okay. I'm using gnu sed, and no errors, but s2p didn't like them.

Over at grymoire.com sed tutorial, his examples showed a space between the address/range and the command, and for some reason, visually that helped.

This:
-- sed '/start/,/stop/ s/#.*//'
verses:
-- sed '/start/,/stop/s/#.*//'

MrDuke Permalink
September 21, 2011, 10:46

I want to remove lines which having number of commas char greater than N.
counld anyone help me :( ?

September 21, 2011, 20:25

Here is a trick to do it. Suppose N=5, then this one-liner will delete all lines with 5 commas:

sed -n '/.*,.*,.*,.*,.*,/d
Shoilen Permalink
October 21, 2011, 11:35

I want to get the line count of a file, excluding the first line. How can I achieve it using sed? File content:

EMP_ID | DEPT_NAME
6965828|Physics One
7510197|Physics One
4789042|Physics One
7991528|Physics One

I tried
#sed -(n-1) '$=' Myfile
the result is 5 however I should be 4.

pandeeswaran Permalink
February 05, 2012, 14:50

This is the way you want i guess:
pandeeswaran@ubuntu:~/training$ cat Myfile
EMP_ID | DEPT_NAME
6965828|Physics One
7510197|Physics One
4789042|Physics One
7991528|Physics One
pandeeswaran@ubuntu:~/training$ sed -e '1d' Myfile|sed -n '$='
4

Shoilen Permalink
October 21, 2011, 11:40

Please read as

sed -n '$=' Myfile

the result is 5 however it should be 4.

January 03, 2012, 15:54

Thats really nice of you to explaing these one liners...Thanks

Duane Meyers Permalink
February 01, 2012, 15:15

I'm having no luck after many hours of either missing it or just not finding it in doing the google thing. I would greatly appreciate if someone could assist me in printing a line above the line that matches a regular expression and the line above also has the regular expression. I need to do this to find duplicate records in the database (library). File has been split based on 1st field so any lines that are exactly the same matching on the 1st field will be together. Anything not matching exactly will be by itself, thus not a duplicate and there is a blank line between each single and double lines. If the line has the regular expression SERSOL-EBOOK, print the line above it. Since there is blank line between non duplicate entries those single lines are ignored.

1992 Justin Beaver baby baby|SERSOL-EBOOK

1997 Justin Beaver baby baby baby|SERSOL-EBK
1997 Justin Beaver baby baby baby|SERSOL-EBK

pandeeswaran Permalink
February 05, 2012, 14:21

sed -n '$=' emulates wc -l only for non empty files.
[code]
pandeeswaran@ubuntu:~/training$ wc -l mail.sh
0 mail.sh
pandeeswaran@ubuntu:~/training$ sed -n '$=' mail.sh
pandeeswaran@ubuntu:~/training$
[/code]

Gajendran V.R. Permalink
March 18, 2012, 08:12

i want to grep the word "nfs" in /etc/fstab then comment the line using #. can any one ping the one liner please :)

May 14, 2012, 13:25

Here you go (in awk):

awk '/nfs/ { print "#" $0; next } { print $0 }'

Or in sed:

sed '/nfs/s/^/#/'
Gaurav Permalink
May 11, 2012, 14:06

Hi,

I just want to replace 50 variable in just one line for example i want to replace 1,2,3.....upto 50 by aa,ab,ac.... upto bx
kindly suggest...

May 14, 2012, 13:22

You can do it easily with a Perl one-liner:

perl -pe '@a = aa..bx; s/(\d+)/$a[$1-1]/ge'

Example:

$ perl -pe '@a = aa..bx; s/(\d+)/$a[$1-1]/ge'
1
aa
2
ab
3
ac
26
az
27
ba
50
bx
^C
Gaurav Permalink
May 14, 2012, 16:52

thnx very much peteris ..... can u please explain it at the point "/$a[$1-1]/" ..... and one more thing can we do the same thing by sed ?

May 16, 2012, 13:40

It's hard to do it in sed.

Here is what $a[$1-1] does. First (\d+) captures the number in the group $1, then $a[$1-1] just accesses the $1-1 element in the @a array.

For example, if the number is 2, then (\d+) captures 2 in $1, and $a[$1-1] is $a[1], which is string "bb", because array @a is ("aa", "bb", "cc", ..., "bx").

Gaurav Permalink
May 18, 2012, 09:23

thnx peteris for the explanation..... one more thing i would like to ask suppose i have 2 files in the first file i have these variables like aa, ab, ac and so on upto bx and in the second file i am having lines like
this is first line
this is second line.
now i want to combine these files and wanna get output like this
aa=this is first line
ab=this is second line
..... and so on upto
bx=this is 50th line.

many thanx in advance .....

May 18, 2012, 13:01

Very easy mate, use the paste unix command. Like this:

paste -d= file1 file2

Here is my example. I have this in file1:

a
b
c
d
e
f

And I have this in file2:

1
2
3
4
5
6

Now when I run paste -d= file1 file2, I get:

a=1
b=2
c=3
d=4
e=5
f=6
Gaurav Permalink
May 18, 2012, 15:12

thnx peteris :)

Gaurav Permalink
May 25, 2012, 15:07

hi, Peteris wanna ask u one thing can i create a GUI with help of shell scripting suppose i want to create a GUI of currency converter... can u please help me .....

romainc Permalink
May 29, 2012, 18:29

Oneliner 31 is not quite right, it doesn't replace the first occurence of foo but rather the previous one to last.
This is because * is greedy (not stopping on the first possible match).

$ echo "this is foo and foo another foo quux" | sed 's/\(.*\)foo\(.*foo\)/\1bar\2/'
this is foo and bar another foo quux

One way to do actually replace the first match :

$ echo "this is foo and foo  another foo quux" | sed 's/foo\(.*foo\)/bar\1/'
this is bar and foo  another foo quux

A non greedy operator does not seems to exist in GNU sed (but it exists in software with extended regex engine like vi, perl, ... )

Gaurav Permalink
June 02, 2012, 08:27

one liner 41 will not work correct on number like 12345.6789
kindly fix this bug....

Human Permalink
June 22, 2012, 13:13

Got an interesting requirement.
You have listed sed commands for printing paragraphs containing AAA or BBB or CCC & AAA and BBB and CCC.

I got an interesting requirement. How do I print a paragraph which contains (AAA or BBB) and CCC?

June 24, 2012, 23:09

Here is how:

sed -n '/AAA/{/CCC/p};/BBB/{/CCC/p}'
Human Permalink
June 25, 2012, 08:30

Thankyou. It worked like a charm.
I have been trying more on the lines of

sed -e '/AAA|BBB/!d;/CCC/!d'

but couldnt get it work. My reasoning says the above command should work (I assume I could use a regex as AAA|BBB in my first search). Am I missing something here?

June 27, 2012, 19:27

Your reasoning is correct, and your program is actually correct.

The only problem is with the type of regular expressions sed uses. It uses so called "basic" regexes where | is an ordinary character.

You've to escape it to make it work like an alternation operator:

sed -e '/AAA\|BBB/!d;/CCC/!d'

Try this and it will work.

Also check out the -r argument to sed. It enables extended regular expressions, and your program will work:

sed -re '/AAA|BBB/!d;/CCC/!d'
Human Permalink
July 25, 2012, 09:09

Thankyou Peter. Both your suggestions worked.

mark Permalink
July 13, 2012, 15:13

Hi
I am doing something like example 35:
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g'

The above example replaces 3 words, but I want to replace 214 words in my source file.
Sadly, I get the error:
"unterminated substitute pattern"

But, strangely, if I divide this sed command into smaller blocks (of roughly 40 words), then it works for some reason.
I am really hoping there is a fix since I have to do many of these and need to process at least 214 find/replace tasks at once in the 1 file without errors.
Any help very much appreciated.

Nirmal Arri Permalink
October 08, 2012, 14:58

Hi Peteris,
I highly appreciate this article, and all the effort you have put in. Here is my question.

I need to replace a line which starts with a word "update" and ends with "where", I have the following that works fine it there is only one "where" in the line, but doesn't work if there are more than one "where".

sed -e 's/^Update.*where//ig' and I pipe to replace that string with something I want it to be.

Any help would be highly appreciate..

Thanks

Nirmal

October 10, 2012, 01:47

Use sed pattern matching and nesting:

sed -e '/^Update/{/where$/{s/.../.../ig}}'

priyoma basu Permalink
November 30, 2012, 11:10

Hello, please help me with this question -
Write a sed command that will go through a file and eliminate any .5 at the end of a record. If .5 is anywhere else in the record, leave it alone. Have the sed command only display those records which are modified. (sample input & output as follows)


Input:

1:2:3:4
1:3:5:7.5
1.5:2.5:3.5:7.5
1.5:2:3:4
1:2.5:3:4

Output:

1:3:5:7
1.5:2.5:3.5:7

MALLAREDDY N Permalink
October 27, 2014, 06:08

cat q91.txt
1:2:3:4
1:3:5:7.5
1.5:2.5:3.5:7.5
1.5:2:3:4
1:2.5:3:4
sed -n 's/\.5$//gp' q91.txt
Output:
1:3:5:7
1.5:2.5:3.5:7

Description:

with -n don't print lines, and then with /p - print matched lines only

JESS Permalink
February 09, 2013, 00:06

Hi, Can you please help me with the following question.

I'm using sed to add HELLO at the end of the each DATE* word.

$ echo $TAB_COLS
ID,TITLE,DATE_PROG1,DATE_PROG2

I get the following.
$ echo $TAB_COLS | sed -e 's/\(,DATE.*\),*/\1 \"HELLO\"/g'
ID,TITLE,DATE_PROG1,DATE_PROG2 "HELLO"

But, I expact the output as copied below.

ID,TITLE,DATE_PROG1 "HELLO",DATE_PROG2 "HELLO"

How can I achive this? Does this have to do anything with sed grediness?

Thanks for your help.

February 14, 2013, 00:39

Yes, that's because .* is greedy.

This \(,DATE.*\),* matches ,DATE_PROG1,DATE_PROG2, and replaces it with ,DATE_PROG1,DATE_PROG2 "HELLO".

There is no way to make .* non-greedy in sed. You've to use [...]* trick:

$ echo $TAB_COLS | sed -e 's/\(DATE[^,]*\)/\1 \"HELLO\"/g'
ID,TITLE,DATE_PROG1 "HELLO",DATE_PROG2 "HELLO"
Indu Permalink
July 03, 2013, 07:00

Hi Peteris,
Could you please explain the command?
echo "1234"|sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'

I didnt understand how these many iterations are applied in just one input line without any t/b loop.

Appreciate your help!

Thanks,
Indu

Bård Permalink
December 19, 2013, 11:40

The loop is achieved by the //D.
We can write the expression a bit clearer (IMHO) like this:

echo "1234" |sed -r '/\n/! G; s|(.)(.*\n)|&\2\1|; /(.)(.*\n)/D; s|\n||'

I like to use the | for separating the different parts of a substitution command (to avoid the picket fence). The -r options (which might not be portable across sed versions, I think) removes the need for escaping grouping parentheses. I also added a space between commands and replaced the //D with /(.)(.*\n)/D to make things even more clear. The // just repeats the last used regular expression. And finally replaced the . in the last replace expression with \n just to make it very clear that the character we expect to find and replace is indeed a newline.
And now to the point:
The key here is the /(.)(.*\n)/D expression. As long as we are able to find any character followed by a newline, the D command will be run and the D command does the following (quoted from the gnu.org sed manual)

If pattern space contains no newline, start a normal new cycle as if the d command was issued. Otherwise, delete text in the pattern space up to the first newline, and restart cycle with the resultant pattern space, without reading a new line of input.

Every time (.)(.*\n) matches we indeed do have a newline in the pattern space, so we "delete text in the pattern space up to the first newline, and restart cycle with the resultant pattern space, without reading a new line of input". Which makes this a loop. This loop will go on until we do not get a match with (.)(.*\n), which will happen eventually (when the content of the pattern space starts with a newline). We then move on to removing the leading newline with s|\n|| and we're done.

To look at the resulting pattern space after each iteration we can use the l command in sed, which (again from the gnu.org sed manual):

Print the pattern space in an unambiguous form: non-printable characters (and the \ character) are printed in C-style escaped form; long lines are split, with a trailing \ character to indicate the split; the end of each line is marked with a $.

Like so:

echo "1234" |sed -r '/\n/! G; s|(.)(.*\n)|&\2\1|; l; /(.)(.*\n)/D; s|\n||'

We print the content of the pattern space after each substitution in the loop, so we see what the /(.)(.*\n)/D receives as input.

1234\n234\n1$
234\n34\n21$
34\n4\n321$
4\n\n4321$
\n4321$
4321

The first 5 lines here are the output from the 5 invocations of the l in the loop and the last line is what is final reversed string printed then the whole sed expression is finished.
Now we see pretty clearly what is going on in the loop and how and when the loop is terminated.

And I think we all can agree on that the person who wrote this one-liner initially is a very intelligent and knows his sed commands. :-)

December 07, 2013, 01:06

NOVEL, SHORT STORY. TRY TO SPEND A LITTLE TIME READING TO HELP RELEASE STRESS AND INCREASE KNOWLEDGE ABOUT HIS LIFE.

December 07, 2013, 01:42

IsowMail video, is an ultimate Video Sharing script

peggyd Permalink
December 24, 2013, 05:30

College essays are important because they let you reveal your personality. Most of the college students use writing services to complete their essays. College essay writing services are able to do any kind of essays with high quality contents.

Vivek Permalink
January 02, 2014, 13:31

Hi,

I have data set as follows

D|abd|def|~#
D|abc|sbdjf|~#
.
.
.
D|sfjh|sdbhf|~#

I wanted to append \n at the end of last record. (~#\n)
Any advise is of great help for me.

Regards
Patil

peggyd Permalink
January 18, 2014, 04:57

Great article. These types of articles are very helpful for the students. Students are now exploring the advantages of internet today. They are using essay writing companies online for doing their assignments. They are helping students by providing good essays.

vinay Permalink
May 14, 2014, 12:43

Hi Peteris,

This is a really really nice blog.

If possible, can you help me with the following scenario:

I want to append a character say a # to the previous line if the current line starts with say JU.
Example :
this is the previous line1
JU this is the current line2
this is previous line2
this is the current line2

The output should be:
#this is the previous line1
JU this is the current line2
this is previous line2
this is the current line2

Can you please help.

Thanks in advance.

Piyush Patel Permalink
July 05, 2014, 06:20

Dear Sir,

I am not fluent in sed but it is very easy to use. Now I have Text file with 7 different pages/sections and there is blankline after each page/section. Now Suppose if I want to get only second page/section. How can I use sed to get the specific section?

ramesh Permalink
August 10, 2014, 15:26

Loved the information here Lombardcomputerrepair.com for computer related repairs

sachin yadav Permalink
August 14, 2014, 13:09

father of computer charles babbage

he is the man who create real thing for us so never forget this man

October 28, 2014, 14:33

It's nice to come on your website. This is really helpful as well as interesting information here. Thanks for sharing this content through your portal. You are truly awesome.

Max Permalink
November 14, 2014, 16:11

I highly appreciate this article, and all the effort you have put in. Here is my question.

I need to replace a line which starts with a word "update" and ends with "where", I have the following that works fine it there is only one "where" in the line, but doesn't work if there are more than one "where".
www.irctclogin.co.in
www.showmejob.com
www.justswiftcodes.com

Leave a new comment

(why do I need your e-mail?)

(Your twitter name, if you have one. (I'm @pkrumins, btw.))

Type the word "halflife3": (just to make sure you're a human)

Please preview the comment before submitting to make sure it's OK.

Advertisements