Follow me on Twitter for my latest adventures!
Hello everyone! I have awesome news - I have written my first e-book ever! It's called "Awk One-Liners Explained" and it's based on the "Famous Awk One-Liners Explained" article series that I wrote here on my blog a few years ago and that has been read over 800,000 times!
I went through all the one-liners in the article series, improved the explanations, fixed the mistakes, added an introduction chapter to Awk one-liners, and two new chapters on the most commonly used Awk special variables and on idiomatic Awk.
Table of Contents
The e-book is 58 pages long and contains exactly 70 well-explained Awk one-liners. It's divided into the following chapters:
- Preface.
- 1. Introduction to Awk One-Liners.
- 2. Line Spacing.
- 3. Numbering and Calculations.
- 4. Text Conversion and Substitution.
- 5. Selective Printing and Deleting of Certain Lines.
- 6. String and Array Creation.
- Appendix A: Awk Special Variables.
- Appendix B: Idiomatic Awk.
- Index.
What is awk?
Awk is this awesome, little program that's present on nearly ever Unix machine. It's designed to carry out various text processing tasks easily, such as numbering lines, replacing certain words, deleting and printing certain lines.
Let's take a look at several examples.
Example 1: Print the second column from a file
awk '{ print $2 }'
That's all there is to it. Awk automatically splits each line into columns and puts each column in variables $1, $2, $3, etc. This one-liner prints just the 2nd column, which is in variable $2.
You can also specify the symbol or word that you wish to split on with the -F command line switch. This switch is explained in more details in the e-book and in the last example below.
Example 2: Number lines in a file
awk '{ print NR ": " $0 }' file
The whole line itself goes into variable $0. This one-liner prints it but prepends the NR special variable and a colon ": " before it. The special variable NR always contains the current line number.
There are many other special variables and they're all explained in the e-book and summarized in the appendix.
Example 3: Count the number of words in a file
awk '{ total = total + NF } END { print total }'
Here another special variable is used. It's the NF that stands for number of fields, or number of columns, or number of words in the current line. This one-liner then just sums the total number of words up and prints them before quitting in the END block.
Example 4: Print only lines shorter than 64 characters
awk 'length < 64'
This one-liner uses the length function to determine the length of the current line. If the current line is less than 64 characters in length, then length < 64 evaluates to true that instructs awk to print the line.
Finally, let's take a look at an example that compares an Awk program with an equivalent C program. Suppose you want to print the list of all users on your system. With Awk it's as simple as this one-liner:
awk -F: '{ print $1 }' /etc/passwd
This one-liner says, "Take each line from /etc/passwd, split it on the colon and print the first field of each line." Very straightforward and easy to write once you know Awk!
Suppose you didn't know Awk. Then you'd have to write it in some other language, such as C. Compare the example above with the example in C language:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define MAX_LINE_LEN 1024
int main () {
char line [MAX_LINE_LEN];
FILE *in = fopen ("/etc/passwd", "r");
if (!in) exit (EXIT_FAILURE);
while (fgets(line, MAX_LINE_LEN, in) != NULL) {
char *sep = strchr(line , ':');
if (!sep) exit (EXIT_FAILURE);
*sep = '\0';
printf("%s\n", line);
}
fclose(in);
return EXIT_SUCCESS ;
}
This is much longer and you have to compile the program first, only then you can run it. If you make any mistakes, you have to recompile it again. That's why one-liners are called one-liners! They are short, easy to write and they do one thing really well. I am pretty sure you're starting to see how mastering Awk and one-liners can make you much more efficient when working in the shell and with text files in general.
Once you read the e-book and work through the examples, you'll be able to solve the most common text processing tasks, such as joining lines in a file, numbering lines, replacing certain words and printing certain lines.
Book preview
I prepared a book preview that contains the first 11 pages of the book. It includes the table of contents, preface, introduction and the first page of the second chapter.
Buy it now!
The price of the e-book is just $5.95 and you can buy it through PayPal.
After you have made the payment, my automated e-book processing system will send the PDF e-book to you in a few minutes!
Testimonials
Iain Dooley, CEO and founder of Working Software LTD:
It never ceases to amaze me that, even though I spend 50% - 70% of my day using a *nix command line and have done so for the past 6 years, there are still countless thousands of useful tools and tips to learn, each with their corresponding little productivity boosts. The trouble, of course, is finding the time to organise and prioritise them, deciding which are the most important and useful to learn. "Awk One Liners Explained" is a fantastic resource of curated examples that helped me rapidly pick up a few cool tricks that have already provided many times the value I initially paid for the book. Any professional who spends time working with *nix systems can benefit from this book.
Tweet about my book!
I am really excited about my book and I would appreciate your help spreading the word via Twitter! Here is a quick link for tweeting:
My other e-books!
I am so passionate about programming and writing about programming that I have now written my second e-book called "Sed One-Liners Explained". It's written in the same style as this e-book and it explains sed, the Superman of Unix stream editing. Sed One-Liners Explained contains 100 well-explained one-liners and it's 98 pages long. Take a look!
And I am not stopping here - I am going to release several other books. My next e-book is called "Perl One-Liners Explained" and it's based on my "Famous Perl One-Liners Explained" article series.
Enjoy!
Enjoy the book and let me know what you think about it!
Sincerely,
Peteris Krumins (@pkrumins on Twitter)





Facebook
Plurk
more
GitHub
LinkedIn
FriendFeed
Google Plus
Amazon wish list
Comments
I'd rather buy it if it wasn't only in PDF format... (.mobi || .prc prefered, or I can help you to make a conversion if needed)
Is there a chance?
Thanks for the question. At the moment I only have the PDF but , yes, I'll have those formats available later. I'll announce it here on my blog when I have them ready!
Thanks for Your answer Peteris, then my next question is: if I order the PDF, will I get the other (Kindle (prc||mobi)) version for free?
If you buy the book right now, I'll send you the other version for free. Otherwise, it will be available for purchase on Kindle store!
But note that it will take me some time before I convert the book into those formats!
Fair enough! Order placed. May I recommend to try out Calibre [for conversion] which does a great job especially from HTML (or markdown) input? Or if you have a LaTeX source, there's pandoc which happily translates to several output format AFAIK.
Payment received, and my book system just sent you the book as attachment. If you don't see it in your inbox, please check your spam folder. (If still nothing, mail me!!!).
I have LaTeX source, I am gonna check out pandoc, or just prepare the document dimensions for Kindle (from their guidelines).
Some error reporting (but still, thanks for the great tutorial):
- @2.2:
awk 'BEGIN { ORS="\n\n" }; 1'the ; is not necessary AFAIK.- @3.4: same applies in
awk 'NF { $0=++ a " : " $0 }; { print }'- @3.7: ; before
END- @3.9-3.11,3.14: ; before
END- @4,17: "This is the first time we see the -F argument passed to Awk." Actually we had seen in @1.1.
Thanks for finding and reporting the errors!
There is nothing wrong with having ;, but yes, you're right, and they can be left out! One char shorter. :)
@4.17 - good catch!
It is good practise to program awk in C style. I would not want to work with anyone who believes code has to be as short as possible, sacrifying the readability.
I just bought the book. It looks very interesting. I would really like a .mobi version too.
Thanks for buying the book! I'll send you a .mobi for free when it's out!
Is this ebook available on the Amazon kindle store?
It's not yet available on the Amazon kindle store. I only have PDF for sale right now. I have planned on publishing a Kindle version later. I'll announce it on my blog when it's ready!
Kindle will display .pdf well enough, though one usually rotates the screen for good justice.
Thanks for your efforts, Peteris!
You're welcome! Thanks for buying a copy. :)
Talking about Kindle, I'd like it to be really fit for Kindle, so I'll take time to prepare a version just for Kindle.
Any progress on the Kindle version?
Yes, I wrote a parser that converts my LaTeX documents to Kindle. Now I need to format them nicely.
Any update on when other formats might be available? Would also be nice to offer the whole set as one item instead of three separate orders.
I've been so busy with Browserling lately that I haven't finished formatting them for Kindle. I'll announce it on my blog as soon as I'm done.
Yes, I'll be offering all 3 e-books packaged together.
If you follow me on Twitter, or have subscribed to my blog, you'll easily know when I do all this.
Wow! This is just what I need. I'm definitely going to buy a copy. But I've got to ask, will you have a version for Nook or the Sony Reader? I'm buying either of the two and of course I want to have a copy of your e-book uploaded into it. Thanks! Oh, congratulations!
Thank you! Yes, I'll have a version for Nook and Sony Reader but that will take some time to make. I don't have an estimate but I'll announce it on my blog when it's ready!
Nice! I like pdf and html, but will read epub with a FireFox addon (www.epubread.com).
Take a look at asciidoc. Documents, (e)books, etc., are written in text and than translated to html, pdf, epub, docbook, etc. I use it for documentation and notes, including math notes (w/the help of asciimathml js). Write it once in text without a lot of messy markup, but the final output can be any format. Good example at the link below --- although it supports most ebook readers, it looks like it doesn't support kindle.
http://www.methods.co.nz/asciidoc/publishing-ebooks-with-asciidoc.html
I wrote it in LaTeX. Looks like it will be trickier to convert it to epub, but still shouldn't be too difficult. LaTeX is LateX afterall.
I googled "latex to epub" and there seem to be lots of solutions.
I'd like to buy this book, and I will if it's an EPUB, but I'm not going to buy a PDF book.
Looks like it'll be great. I went ahead and paid for it on PayPal but have not yet received a link. Do you know how quickly that happens (it was a bit over an hour ago).
Hi Peter,
I just looked at my book processing system and I see that the book was generated for you:
And that it was sent for you successfully (your mail server accepted it):
It looks like your mail server didn't like that email for some reason and didn't deliver it further to you. Or if it did, can you check the spam folder? One other person who bought the book told me he found the book in his spam folder!
In any case, I just generated the book for you sent you an email with the book attached personally. Please confirm that you have received it.
And huge thanks for purchasing my book!
Update:
I just received an email from Peter, and indeed, the email (in fact both of them) were in the spam folder. Here is his response:
So for everyone who purchases the book - if you don't receive it in 10 mins, please check the spam folder, it must be there!
Maybe you can use some help with DKIM and more email white-listing techniques?
Congratulations pal on the new release, best of luck.
Thanks and this sounds exactly what I need. Let's talk!
Thank you! Could you please elaborate a bit on your hack "Prepared exclusively for foo (email address)"
Sure. The ebook is written in LaTeX and I use the
backgroundpackage with the following configuration options to add the message to the book:The
preparedforcolor is defined by the help ofxcolorpackage:I could not get your web page to load with Internet Explorer. Also what's up with the "congradulations you are the 100,000th visitor" at the bottom of the page? Must be some ad. It is annoying, because I hear the "congradulations" when I first load up your page in Chrome.
Besides all this, good luck with your book. Are you going to publish a hard copy version?
Sorry about that nasty ad. My ad company promised that such ads won't appear on my site. Looks like they broke their promise.
I have plans to publish the hard copy version, but much later. I'll announce it on my blog.
You showed AWK vs C.
Below is AWK vs Simple C++:
awk -F: '{ print $1 }' /etc/passwd scc -ni: 'F(0)' < /etc/passwdInteresting. I didn't know about Simple C++.
anyone got a torrent of this?
Nope.
LOL
The pdf email ended up in my Gmail spam folder. FYI :]
Not cool. I am investigating whitelist techniques right now. I need gmail to know that my mailserver is really the mailserver for catonmat.net. I have the reverse DNS record for my mail server but that's not yet enough.
We've been talking on IRC before, Peteris, so I won't tell anything new to you now, just congrats again for releasing your first book. :) But I want to write something for your visitors.
You must be aware that creating book is not such an easy thing. Those who are "creating" them in Word or LibreOffice Writer (for some publishers DOC/RTF is a common medium for interaction with authors) may be sometimes fooled. And on top of it, (La)TeX is not the easiest tool (and language) ever created, but it's still a really good and useful tool, that helps you to focus on writing, at least when you know it at acceptable level. I'm using LaTeX like ~8 years already (obviously not everyday though) and I am not proficient in it, because there is always something to learn. Moreover, I still haven't wrote and released any real book yet!
Peteris learned LaTeX (from 0!) enough to write a book within a month, which is quite a nice achievements. Surely it's not a mistake-free one, can be improved, etc. But there are always these early steps that must be done before proper walking (and sometimes unfortunately even running, which is rather bad, because creating books is the art and hastiness usually ruins the effect).
So don't be harsh on him even if you find some mistakes there. Errare humanum est, but per aspera ad astra!
Thank you sir!
can you provide private system let users who are interesting in sed or perl book in the future. if release can inform us to buy.
Sorry, I don't really understand what you mean? Can you explain it one more time?
I think he means:
can you provide { private_system """let users who are interesting in sed or perl book in the future""" if release: inform (us, to buy) }Ah, that explains it!
Peteris, Just thought I'd drop you a note to thank you for spending the time to produce this book. I find it's well written and explains things very clearly. It's also fun to work though. The only thing I struggled with was creating a suitable file to test the one liners on - I was in Vim staring at a blank screen for about 5 minutes. If I get writers block trying to create a 'somefile', then I can appreciate how hard it must be to write a book!
Great work, looking forward to any other works you publish.
Thanks for the comment and I am glad that you liked my book! I never thought about including a file to test one-liners on, and it's a great suggestion. I'll make sure to include the files in my next revision of this book and my future books.
Until then, I can share the files that I used to test one-liners on.
The first one contains just 10 lines. I call this file
n. I use it to test line spacing, matching lines, numbering lines, text conversion and other tasks.Then I have the
n2file that is double-spaced version ofn. I use it to test one liners such as 'number only non-blank lines in files' and 'delete all blank lines from a file'.Then I have this crazy file called
crazythat I used to test most everything else:That's about it. :)
Haha, thanks! You've made learning AWK an easy, but profitable job. Take care.
Hi Peteris i have a question
I have to delete a file by comparing today's date with the date in file if it is older than 7 days then delete it. Is it possible
Awk is not the language for this. You want to use the
findutility and combine it withrm.Here is how to do it:
find /path -ctime -7 -exec rm {} \;Note: If you use
rm, always substitute-execwith-lsto make find will remove the right files!nice work kit!
Peteris, let me thank you for making my fear of programming go away. Your 2 e-books (sed and awk) are priceless. Can't wait for Perl e-book.
Keep up the good work! and yes shame on google ... now I know why they suck! May be Apple is the company for you! :)
God Bless!
That's awesome! I'm glad I made your fear go away! :)
Hi Peteris,
Could you please tell some short command to delete lines from a file between two specific patterns/strings.
I am using the command : cat File | sed '/ABC/,/DEF/ d'
By using above command I am able to delete all the lines between ABC and DEF. But at the same time the first line containing ABC and last lilne containg DEF also get deleted, which i dont want.
Kindly help.
Here is how:
awk -vp=1 '/ABC/,/DEF/ { p=0; if (/ABC/ || /DEF/) p=1 } p'The
-vp=1sets variablep(p for print) to1before the awk script starts.If the range
/ABC/,/DEF/is matched, then setpto0(don't print). However if it's the/ABC/or/DEF/, then setpto1(print).Then evaluate
p. If it's true, awk prints the current line.Hi Peteris,
The sequence of commands you mentioned are extremely smart(and are working fine). I can also mange to get the same output but may be with the help of 10-15 lines. And you did in just one line. Great!!
Thanks You very much.
You're very welcome!
Great work peteris!!
Thanks!
Can I just buy all 3 ebooks in one go? Can't be arsed to place 3 separate orders from a mobile. Cheers.
Sure! I sent you an email about how to do it.
Dear Peteris,
1]when i buy the set of this 2/3 book ...is there any discount.
2]can we get the hard & soft copy of this book
3]how long it will deliver
Hi Shambhu!
Thanks for the questions.
1) I don't have a discount yet, but I'm thinking of offering all 3 e-books together soon with a nice discount. Actually I can manually send you all 3 e-books and give you a discount. Please email me peter@catonmat.net.
2) Hard & soft copy of this book is not available yet. I only offer an e-book.
3) It's delivered instantly to you through email as you buy it.
Hallo Peteris,
I happily found your webpage and I am really amused by reading all the above comments and your CV. I see your work almost as a humanitarian job by helping others:) I also belong to those who need your books and I would like to buy all three. What is the price and how can I pay please?
Thank you and keep have fun doing science.
Kind regards
Juraj
Hi Juraj, I just emailed you.
Just bought this e-book, will make a nice companion to my sed & awk Pocket Reference.
many thanks.
Hey Alyn, thanks for getting a copy of my book!
Hi,
I have written an one line awk, where in the use of RS="</Final>" does not return the </Final> xml tag in the output.
Thanks in advance,
Narasimha
Just bought your book. To be honest, partially for the one-liners but mostly for your awesome thumbnail photograph.
Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can’t wait to read lots of your posts.
Leave a new comment