This article is part of the article series "Awk One-Liners Explained."
<- previous article next article ->

Hello everyone! I have awesome news - I have written my first e-book ever! It's called "Awk One-Liners Explained" and it's based on the "Famous Awk One-Liners Explained" article series that I wrote here on my blog a few years ago and that has been read over 800,000 times!

I went through all the one-liners in the article series, improved the explanations, fixed the mistakes, added an introduction chapter to Awk one-liners, and two new chapters on the most commonly used Awk special variables and on idiomatic Awk.

Table of Contents

The e-book is 58 pages long and contains exactly 70 well-explained Awk one-liners. It's divided into the following chapters:

  • Preface.
  • 1. Introduction to Awk One-Liners.
  • 2. Line Spacing.
  • 3. Numbering and Calculations.
  • 4. Text Conversion and Substitution.
  • 5. Selective Printing and Deleting of Certain Lines.
  • 6. String and Array Creation.
  • Appendix A: Awk Special Variables.
  • Appendix B: Idiomatic Awk.
  • Index.

What is awk?

Awk is this awesome, little program that's present on nearly ever Unix machine. It's designed to carry out various text processing tasks easily, such as numbering lines, replacing certain words, deleting and printing certain lines.

Let's take a look at several examples.

Example 1: Print the second column from a file

awk '{ print $2 }'

That's all there is to it. Awk automatically splits each line into columns and puts each column in variables $1, $2, $3, etc. This one-liner prints just the 2nd column, which is in variable $2.

You can also specify the symbol or word that you wish to split on with the -F command line switch. This switch is explained in more details in the e-book and in the last example below.

Example 2: Number lines in a file

awk '{ print NR ": " $0 }' file

The whole line itself goes into variable $0. This one-liner prints it but prepends the NR special variable and a colon ": " before it. The special variable NR always contains the current line number.

There are many other special variables and they're all explained in the e-book and summarized in the appendix.

Example 3: Count the number of words in a file

awk '{ total = total + NF } END { print total }'

Here another special variable is used. It's the NF that stands for number of fields, or number of columns, or number of words in the current line. This one-liner then just sums the total number of words up and prints them before quitting in the END block.

Example 4: Print only lines shorter than 64 characters

awk 'length < 64'

This one-liner uses the length function to determine the length of the current line. If the current line is less than 64 characters in length, then length < 64 evaluates to true that instructs awk to print the line.

Finally, let's take a look at an example that compares an Awk program with an equivalent C program. Suppose you want to print the list of all users on your system. With Awk it's as simple as this one-liner:

awk -F: '{ print $1 }' /etc/passwd

This one-liner says, "Take each line from /etc/passwd, split it on the colon and print the first field of each line." Very straightforward and easy to write once you know Awk!

Suppose you didn't know Awk. Then you'd have to write it in some other language, such as C. Compare the example above with the example in C language:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define MAX_LINE_LEN 1024

int main () {
    char line [MAX_LINE_LEN];
    FILE *in = fopen ("/etc/passwd", "r");
    if (!in) exit (EXIT_FAILURE);
    while (fgets(line, MAX_LINE_LEN, in) != NULL) {
        char *sep = strchr(line , ':');
        if (!sep) exit (EXIT_FAILURE);
        *sep = '\0';
        printf("%s\n", line);
    }
    fclose(in);
    return EXIT_SUCCESS ;
}

This is much longer and you have to compile the program first, only then you can run it. If you make any mistakes, you have to recompile it again. That's why one-liners are called one-liners! They are short, easy to write and they do one thing really well. I am pretty sure you're starting to see how mastering Awk and one-liners can make you much more efficient when working in the shell and with text files in general.

Once you read the e-book and work through the examples, you'll be able to solve the most common text processing tasks, such as joining lines in a file, numbering lines, replacing certain words and printing certain lines.

Book preview

I prepared a book preview that contains the first 11 pages of the book. It includes the table of contents, preface, introduction and the first page of the second chapter.

Buy it now!

The price of the e-book is just $5.95 and you can buy it through PayPal.

PayPal - The safer, easier way to pay online!

After you have made the payment, my automated e-book processing system will send the PDF e-book to you in a few minutes!

Testimonials

Iain Dooley, CEO and founder of Working Software LTD:

It never ceases to amaze me that, even though I spend 50% - 70% of my day using a *nix command line and have done so for the past 6 years, there are still countless thousands of useful tools and tips to learn, each with their corresponding little productivity boosts. The trouble, of course, is finding the time to organise and prioritise them, deciding which are the most important and useful to learn. "Awk One Liners Explained" is a fantastic resource of curated examples that helped me rapidly pick up a few cool tricks that have already provided many times the value I initially paid for the book. Any professional who spends time working with *nix systems can benefit from this book.

Tweet about my book!

I am really excited about my book and I would appreciate your help spreading the word via Twitter! Here is a quick link for tweeting:

My other e-books!

I am so passionate about programming and writing about programming that I have now written my second e-book called "Sed One-Liners Explained". It's written in the same style as this e-book and it explains sed, the Superman of Unix stream editing. Sed One-Liners Explained contains 100 well-explained one-liners and it's 98 pages long. Take a look!

And I am not stopping here - I am going to release several other books. My next e-book is called "Perl One-Liners Explained" and it's based on my "Famous Perl One-Liners Explained" article series.

Enjoy!

Enjoy the book and let me know what you think about it!

Sincerely,
Peteris Krumins (@pkrumins on Twitter)

This article is part of the article series "Awk One-Liners Explained."
<- previous article next article ->

Comments

June 20, 2011, 15:12

I'd rather buy it if it wasn't only in PDF format... (.mobi || .prc prefered, or I can help you to make a conversion if needed)
Is there a chance?

June 20, 2011, 16:35

Thanks for the question. At the moment I only have the PDF but , yes, I'll have those formats available later. I'll announce it here on my blog when I have them ready!

June 20, 2011, 19:28

Thanks for Your answer Peteris, then my next question is: if I order the PDF, will I get the other (Kindle (prc||mobi)) version for free?

June 20, 2011, 19:52

If you buy the book right now, I'll send you the other version for free. Otherwise, it will be available for purchase on Kindle store!

But note that it will take me some time before I convert the book into those formats!

June 20, 2011, 21:33

Fair enough! Order placed. May I recommend to try out Calibre [for conversion] which does a great job especially from HTML (or markdown) input? Or if you have a LaTeX source, there's pandoc which happily translates to several output format AFAIK.

June 20, 2011, 21:41

Payment received, and my book system just sent you the book as attachment. If you don't see it in your inbox, please check your spam folder. (If still nothing, mail me!!!).

I have LaTeX source, I am gonna check out pandoc, or just prepare the document dimensions for Kindle (from their guidelines).

June 27, 2011, 08:44

Some error reporting (but still, thanks for the great tutorial):

- @2.2: awk 'BEGIN { ORS="\n\n" }; 1' the ; is not necessary AFAIK.
- @3.4: same applies in awk 'NF { $0=++ a " : " $0 }; { print }'
- @3.7: ; before END
- @3.9-3.11,3.14: ; before END
- @4,17: "This is the first time we see the -F argument passed to Awk." Actually we had seen in @1.1.

June 28, 2011, 15:56

Thanks for finding and reporting the errors!

There is nothing wrong with having ;, but yes, you're right, and they can be left out! One char shorter. :)

@4.17 - good catch!

Internet Troll Permalink
September 05, 2012, 15:48

It is good practise to program awk in C style. I would not want to work with anyone who believes code has to be as short as possible, sacrifying the readability.

June 21, 2011, 12:40

I just bought the book. It looks very interesting. I would really like a .mobi version too.

June 24, 2011, 13:09

Thanks for buying the book! I'll send you a .mobi for free when it's out!

Andrew Manson Permalink
June 20, 2011, 15:18

Is this ebook available on the Amazon kindle store?

June 20, 2011, 15:24

It's not yet available on the Amazon kindle store. I only have PDF for sale right now. I have planned on publishing a Kindle version later. I'll announce it on my blog when it's ready!

June 20, 2011, 15:29

Kindle will display .pdf well enough, though one usually rotates the screen for good justice.
Thanks for your efforts, Peteris!

June 20, 2011, 15:50

You're welcome! Thanks for buying a copy. :)

Talking about Kindle, I'd like it to be really fit for Kindle, so I'll take time to prepare a version just for Kindle.

Tim Hopper Permalink
September 23, 2011, 17:24

Any progress on the Kindle version?

September 23, 2011, 18:25

Yes, I wrote a parser that converts my LaTeX documents to Kindle. Now I need to format them nicely.

May 23, 2012, 02:12

Any update on when other formats might be available? Would also be nice to offer the whole set as one item instead of three separate orders.

May 24, 2012, 16:34

I've been so busy with Browserling lately that I haven't finished formatting them for Kindle. I'll announce it on my blog as soon as I'm done.

Yes, I'll be offering all 3 e-books packaged together.

If you follow me on Twitter, or have subscribed to my blog, you'll easily know when I do all this.

September 29, 2013, 19:33

Evangers Grain Free Super Premium Chicken Dinner Canned Dog Food 13oz (12 in a case) By:

Sarah Brisbane Permalink
June 20, 2011, 16:29

Wow! This is just what I need. I'm definitely going to buy a copy. But I've got to ask, will you have a version for Nook or the Sony Reader? I'm buying either of the two and of course I want to have a copy of your e-book uploaded into it. Thanks! Oh, congratulations!

June 20, 2011, 16:37

Thank you! Yes, I'll have a version for Nook and Sony Reader but that will take some time to make. I don't have an estimate but I'll announce it on my blog when it's ready!

June 20, 2011, 17:23

Nice! I like pdf and html, but will read epub with a FireFox addon (www.epubread.com).

Take a look at asciidoc. Documents, (e)books, etc., are written in text and than translated to html, pdf, epub, docbook, etc. I use it for documentation and notes, including math notes (w/the help of asciimathml js). Write it once in text without a lot of messy markup, but the final output can be any format. Good example at the link below --- although it supports most ebook readers, it looks like it doesn't support kindle.

http://www.methods.co.nz/asciidoc/publishing-ebooks-with-asciidoc.html

June 21, 2011, 15:57

I wrote it in LaTeX. Looks like it will be trickier to convert it to epub, but still shouldn't be too difficult. LaTeX is LateX afterall.

Ken Jackson Permalink
March 03, 2012, 17:49

I googled "latex to epub" and there seem to be lots of solutions.
I'd like to buy this book, and I will if it's an EPUB, but I'm not going to buy a PDF book.

Peter Keane Permalink
June 20, 2011, 17:52

Looks like it'll be great. I went ahead and paid for it on PayPal but have not yet received a link. Do you know how quickly that happens (it was a bit over an hour ago).

June 20, 2011, 18:37

Hi Peter,

I just looked at my book processing system and I see that the book was generated for you:

[2011-55-20 16:55:23] Processing new payment for awk_book from pkeane@[masked].
Preparing Awk Book...
Spawning Latex...
Spawning makeindex...
Spawning Latex 2nd time...
Spawning Latex 3nd time...
Sending the Awk book to Peter Keane (pkeane@[masked]).

And that it was sent for you successfully (your mail server accepted it):

Jun 20 11:55:30 catonmat sm-mta[789]: p5KGtThY000786: to=<pkeane@[masked]>, delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=641910, relay=inbound.mail.utexas.edu. [146.6.25.62], dsn=2.0.0, stat=Sent (ok:  Message 314653467 accepted)

It looks like your mail server didn't like that email for some reason and didn't deliver it further to you. Or if it did, can you check the spam folder? One other person who bought the book told me he found the book in his spam folder!

In any case, I just generated the book for you sent you an email with the book attached personally. Please confirm that you have received it.

And huge thanks for purchasing my book!

Update:

I just received an email from Peter, and indeed, the email (in fact both of them) were in the spam folder. Here is his response:

Yes -- there it is in my spam folder! (2 messages, in fact).  Sorry
for the trouble & thanks for the book.

So for everyone who purchases the book - if you don't receive it in 10 mins, please check the spam folder, it must be there!

Raslan Abuzant Permalink
June 21, 2011, 03:49

Maybe you can use some help with DKIM and more email white-listing techniques?

Congratulations pal on the new release, best of luck.

June 21, 2011, 15:57

Thanks and this sounds exactly what I need. Let's talk!

Rakesh Permalink
June 20, 2011, 22:34

Thank you! Could you please elaborate a bit on your hack "Prepared exclusively for foo (email address)"

June 20, 2011, 22:53

Sure. The ebook is written in LaTeX and I use the background package with the following configuration options to add the message to the book:

\usepackage{background}
\SetBgContents{Prepared exclusively for Peteris Krumins (peter@catonmat.net)}
\SetBgAngle{0}
\SetBgScale{1}
\SetBgOpacity{1}
\SetBgColor{preparedfor}
\SetBgPosition{current page.south}
\SetBgVshift{0.5cm}

The preparedfor color is defined by the help of xcolor package:

\usepackage{xcolor}
\definecolor{preparedfor}{HTML}{787878}
June 21, 2011, 03:40

I could not get your web page to load with Internet Explorer. Also what's up with the "congradulations you are the 100,000th visitor" at the bottom of the page? Must be some ad. It is annoying, because I hear the "congradulations" when I first load up your page in Chrome.

Besides all this, good luck with your book. Are you going to publish a hard copy version?

June 21, 2011, 15:58

Sorry about that nasty ad. My ad company promised that such ads won't appear on my site. Looks like they broke their promise.

I have plans to publish the hard copy version, but much later. I'll announce it on my blog.

June 21, 2011, 04:31

You showed AWK vs C.
Below is AWK vs Simple C++:

awk -F: '{ print $1 }' /etc/passwd
scc -ni: 'F(0)' < /etc/passwd 
June 21, 2011, 18:45

Interesting. I didn't know about Simple C++.

roger Permalink
June 21, 2011, 14:23

anyone got a torrent of this?

Thijs Permalink
May 17, 2013, 02:20

LOL

June 21, 2011, 15:05

The pdf email ended up in my Gmail spam folder. FYI :]

June 21, 2011, 15:59

Not cool. I am investigating whitelist techniques right now. I need gmail to know that my mailserver is really the mailserver for catonmat.net. I have the reverse DNS record for my mail server but that's not yet enough.

June 22, 2011, 00:45

We've been talking on IRC before, Peteris, so I won't tell anything new to you now, just congrats again for releasing your first book. :) But I want to write something for your visitors.

You must be aware that creating book is not such an easy thing. Those who are "creating" them in Word or LibreOffice Writer (for some publishers DOC/RTF is a common medium for interaction with authors) may be sometimes fooled. And on top of it, (La)TeX is not the easiest tool (and language) ever created, but it's still a really good and useful tool, that helps you to focus on writing, at least when you know it at acceptable level. I'm using LaTeX like ~8 years already (obviously not everyday though) and I am not proficient in it, because there is always something to learn. Moreover, I still haven't wrote and released any real book yet!

Peteris learned LaTeX (from 0!) enough to write a book within a month, which is quite a nice achievements. Surely it's not a mistake-free one, can be improved, etc. But there are always these early steps that must be done before proper walking (and sometimes unfortunately even running, which is rather bad, because creating books is the art and hastiness usually ruins the effect).

So don't be harsh on him even if you find some mistakes there. Errare humanum est, but per aspera ad astra!

June 23, 2011, 22:16

Thank you sir!

lin Permalink
June 23, 2011, 15:46

can you provide private system let users who are interesting in sed or perl book in the future. if release can inform us to buy.

June 24, 2011, 13:13

Sorry, I don't really understand what you mean? Can you explain it one more time?

Richard Lyons Permalink
June 27, 2011, 10:38

I think he means:

can you provide {
	private_system 
		"""let users who are interesting in sed or perl book in the future"""
		if release:
			inform (us, to buy)
}
June 28, 2011, 16:02

Ah, that explains it!

mcai8sh4 Permalink
June 24, 2011, 14:41

Peteris, Just thought I'd drop you a note to thank you for spending the time to produce this book. I find it's well written and explains things very clearly. It's also fun to work though. The only thing I struggled with was creating a suitable file to test the one liners on - I was in Vim staring at a blank screen for about 5 minutes. If I get writers block trying to create a 'somefile', then I can appreciate how hard it must be to write a book!

Great work, looking forward to any other works you publish.

June 24, 2011, 15:37

Thanks for the comment and I am glad that you liked my book! I never thought about including a file to test one-liners on, and it's a great suggestion. I'll make sure to include the files in my next revision of this book and my future books.

Until then, I can share the files that I used to test one-liners on.

The first one contains just 10 lines. I call this file n. I use it to test line spacing, matching lines, numbering lines, text conversion and other tasks.

l 1
l 2
l 3
l 4
l 5
l 6
l 7
l 8
l 9
l 10

Then I have the n2 file that is double-spaced version of n. I use it to test one liners such as 'number only non-blank lines in files' and 'delete all blank lines from a file'.

l 1

l 2

l 3

l 4

l 5

l 6

l 7

l 8

l 9

l 10

Then I have this crazy file called crazy that I used to test most everything else:

line one hello wold
line      two       foor    bar baz
line three esr rms lbt
line four french english spanish
another line

line five c cpp perl
line six energy power work

kakakdfjfgjfhhgfieiuowerwoieroiuqwe

line seven integral matrix set

repeated line
repeated line
repeated line muahaha
repeated line

line eight pen stool lamp
line nine you me him


line ten coffee tee pop last

That's about it. :)

mcai8sh4 Permalink
June 24, 2011, 19:24

Haha, thanks! You've made learning AWK an easy, but profitable job. Take care.

September 12, 2013, 19:22

Jerky Naturals Treats – 6 oz – Tender Turkey (ZU22558) – By:

pradeep.aradhya Permalink
June 27, 2011, 09:46

Hi Peteris i have a question
I have to delete a file by comparing today's date with the date in file if it is older than 7 days then delete it. Is it possible

June 28, 2011, 16:02

Awk is not the language for this. You want to use the find utility and combine it with rm.

Here is how to do it:

find /path -ctime -7 -exec rm {} \;

Note: If you use rm, always substitute -exec with -ls to make find will remove the right files!

July 20, 2011, 07:41

nice work kit!

Jaypal Singh Permalink
October 23, 2011, 08:14

Peteris, let me thank you for making my fear of programming go away. Your 2 e-books (sed and awk) are priceless. Can't wait for Perl e-book.

Keep up the good work! and yes shame on google ... now I know why they suck! May be Apple is the company for you! :)

God Bless!

October 23, 2011, 15:15

That's awesome! I'm glad I made your fear go away! :)

Bhupender Permalink
October 28, 2011, 07:50

Hi Peteris,
Could you please tell some short command to delete lines from a file between two specific patterns/strings.
I am using the command : cat File | sed '/ABC/,/DEF/ d'
By using above command I am able to delete all the lines between ABC and DEF. But at the same time the first line containing ABC and last lilne containg DEF also get deleted, which i dont want.
Kindly help.

October 28, 2011, 08:27

Here is how:

awk -vp=1 '/ABC/,/DEF/ { p=0; if (/ABC/ || /DEF/) p=1 } p'

The -vp=1 sets variable p (p for print) to 1 before the awk script starts.

If the range /ABC/,/DEF/ is matched, then set p to 0 (don't print). However if it's the /ABC/ or /DEF/, then set p to 1 (print).

Then evaluate p. If it's true, awk prints the current line.

Bhupender Permalink
October 28, 2011, 11:29

Hi Peteris,
The sequence of commands you mentioned are extremely smart(and are working fine). I can also mange to get the same output but may be with the help of 10-15 lines. And you did in just one line. Great!!
Thanks You very much.

November 07, 2011, 17:29

You're very welcome!

pandeeswaran Permalink
February 04, 2012, 13:59

Great work peteris!!

February 04, 2012, 15:02

Thanks!

Lee Crampton Permalink
March 12, 2012, 23:39

Can I just buy all 3 ebooks in one go? Can't be arsed to place 3 separate orders from a mobile. Cheers.

March 15, 2012, 18:52

Sure! I sent you an email about how to do it.

Shambhu Permalink
May 24, 2012, 09:08

Dear Peteris,
1]when i buy the set of this 2/3 book ...is there any discount.
2]can we get the hard & soft copy of this book
3]how long it will deliver

May 24, 2012, 16:31

Hi Shambhu!

Thanks for the questions.

1) I don't have a discount yet, but I'm thinking of offering all 3 e-books together soon with a nice discount. Actually I can manually send you all 3 e-books and give you a discount. Please email me peter@catonmat.net.

2) Hard & soft copy of this book is not available yet. I only offer an e-book.

3) It's delivered instantly to you through email as you buy it.

Juraj Lauko Permalink
June 08, 2012, 00:12

Hallo Peteris,

I happily found your webpage and I am really amused by reading all the above comments and your CV. I see your work almost as a humanitarian job by helping others:) I also belong to those who need your books and I would like to buy all three. What is the price and how can I pay please?

Thank you and keep have fun doing science.

Kind regards

Juraj

June 11, 2012, 22:38

Hi Juraj, I just emailed you.

Alyn R. Tiedtke Permalink
August 02, 2012, 08:10

Just bought this e-book, will make a nice companion to my sed & awk Pocket Reference.

many thanks.

August 13, 2012, 20:19

Hey Alyn, thanks for getting a copy of my book!

Narasimha Permalink
August 12, 2012, 11:18

Hi,

I have written an one line awk, where in the use of RS="</Final>" does not return the </Final> xml tag in the output.

Thanks in advance,
Narasimha

Thijs Permalink
May 17, 2013, 03:15

Just bought your book. To be honest, partially for the one-liners but mostly for your awesome thumbnail photograph.

Jotne Permalink
August 16, 2013, 07:00

Example 3
awk '{ total = total + NF } END { print total }'
I would have used
awk '{ total += NF } END { print total }'

Rama Permalink
February 26, 2014, 23:44

Hi there Peter!
I just bought your Awk and Sed book and I think they are fantastic. I went through the entire Awk book and learnt a lot in a very short time. There is however a "specific" question which I was looking an answer for and did not find any. I was hoping you could give me some insight into the same.

I have a text file which has a single column of numbers which are not serially ordered (e.g, 1,4,6,9,11,27 and so on...) A few hundred of them (n=768). These are actually the line numbers of a very BIG file which has 50,000 lines of text.

What I would like to do is for either Awk or Sed to read the numbers in the text file and extract the full contents of each of the numbered line from the BIG file.

Is there a direct way to do this in either Awk or Sed? Or do I have to rely on a bash script to first read the text file, create and array and then load the array into a awk one liner one at a time and extract that particular line from the Big file?

A lot of the examples were either serial numbers i.e, lines 8 through 12 or something like that. None of the examples were random lines like the issue I am facing.

I would hugely grateful for any pointers here.
Regards
Rama

Rama Permalink
February 27, 2014, 07:27

Hi Peteris,
Apologies for referring to you as "Peter" instead of Peteris.
Regards
Rama

April 15, 2014, 09:29

"Suppose you didn't know Awk. Then you'd have to write it in some other language, such as C. "

That's crazy!

cut -d: -f1 </etc/passwd

Leave a new comment

(why do I need your e-mail?)

(Your twitter name, if you have one. (I'm @pkrumins, btw.))

Type the first letter of your name: (just to make sure you're a human)

Please preview the comment before submitting to make sure it's OK.

Advertisements