I just had this quick idea to write a tcp port scanner in bash. Bash supports the special /dev/tcp/host/port file that you can read/write. Writing to this special file makes bash open a tcp connection to host:port. If writing to the port succeeds, the port is open, else the port is closed.

So at first I wrote this quick script:

for port in {1..65535}; do
  echo >/dev/tcp/google.com/$port &&
    echo "port $port is open" ||
    echo "port $port is closed"
done

This loops over ports 1-65535 and tries to open google.com:$port. However this doesn't work that well because if the port is closed, it takes bash like 2 minutes to realize that.

To solve this I needed something like alarm(2) to interrupt bash. Bash doesn't have a built-in alarm function, so I had to write my own using Perl:

alarm() {
  perl -e '
    eval {
      $SIG{ALRM} = sub { die };
      alarm shift;
      system(@ARGV);
    };
    if ($@) { exit 1 }
  ' "$@";
}

This alarm function takes two args: seconds for the alarm call, and the code to execute. If the code doesn't execute in the given time, the function fails.

Once I had this, I could take my earlier code and just call it through alarm:

for port in {1..65535}; do
  alarm 1 "echo >/dev/tcp/google.com/$port" &&
    echo "port $port is open" ||
    echo "port $port is closed"
done

This is working! Now if bash freezes because of a closed port, the alarm 1 will kill the probe in 1 second, and the script will move to the next port.

I went ahead and turned this into a proper scan function:

scan() {
  if [[ -z $1 || -z $2 ]]; then
    echo "Usage: $0 <host> <port, ports, or port-range>"
    return
  fi

  local host=$1
  local ports=()
  case $2 in
    *-*)
      IFS=- read start end <<< "$2"
      for ((port=start; port <= end; port++)); do
        ports+=($port)
      done
      ;;
    *,*)
      IFS=, read -ra ports <<< "$2"
      ;;
    *)
      ports+=($2)
      ;;
  esac


  for port in "${ports[@]}"; do
    alarm 1 "echo >/dev/tcp/$host/$port" &&
      echo "port $port is open" ||
      echo "port $port is closed"
  done
}

You can run the scan function from your shell. It takes two arguments: the host to scan, and a list of ports to scan (such as 22,80,443), or a range of ports to scan (such as 1-1024), or an individual port to scan (such as 80).

Here is what happens when I run scan google.com 78-82:

$ scan google.com 78-82 
port 78 is closed
port 79 is closed
port 80 is open
port 81 is closed
port 82 is closed

Similarly you can write an udp port scanner. Just replace /dev/tcp/ with /dev/udp/.

Update

It turns out GNU's coreutils include timeout utility that runs a command with a time limit. Using timeout we can rewrite the tcp proxy without using Perl for SIGALRM:

$ timeout 1 bash -c "echo >/dev/tcp/$host/$port" &&
    echo "port $port is open" ||
    echo "port $port is closed"
This article is part of the article series "Bash One-Liners Explained."
<- previous article next article ->

This is the third part of the Bash One-Liners Explained article series. In this part I'll teach you all about input/output redirection. I'll use only the best bash practices, various bash idioms and tricks. I want to illustrate how to get various tasks done with just bash built-in commands and bash programming language constructs.

See the first part of the series for introduction. After I'm done with the series I'll release an ebook (similar to my ebooks on awk, sed, and perl), and also bash1line.txt (similar to my perl1line.txt).

Also see my other articles about working fast in bash from 2007 and 2008:

Let's start.

Part III: Redirections

Working with redirections in bash is really easy once you realize that it's all about manipulating file descriptors. When bash starts it opens the three standard file descriptors: stdin (file descriptor 0), stdout (file descriptor 1), and stderr (file descriptor 2). You can open more file descriptors (such as 3, 4, 5, ...), and you can close them. You can also copy file descriptors. And you can write to them and read from them.

File descriptors always point to some file (unless they're closed). Usually when bash starts all three file descriptors, stdin, stdout, and stderr, point to your terminal. The input is read from what you type in the terminal and both outputs are sent to the terminal.

Assuming your terminal is /dev/tty0, here is how the file descriptor table looks like when bash starts:

When bash runs a command it forks a child process (see man 2 fork) that inherits all the file descriptors from the parent process, then it sets up the redirections that you specified, and execs the command (see man 3 exec).

To be a pro at bash redirections all you need to do is visualize how the file descriptors get changed when redirections happen. The graphics illustrations will help you.

1. Redirect the standard output of a command to a file

$ command >file

Operator > is the output redirection operator. Bash first tries to open the file for writing and if it succeeds it sends the stdout of command to the newly opened file. If it fails opening the file, the whole command fails.

Writing command >file is the same as writing command 1>file. The number 1 stands for stdout, which is the file descriptor number for standard output.

Here is how the file descriptor table changes. Bash opens file and replaces file descriptor 1 with the file descriptor that points to file. So all the output that gets written to file descriptor 1 from now on ends up being written to file:

In general you can write command n>file, which will redirect the file descriptor n to file.

For example,

$ ls > file_list

Redirects the output of the ls command to the file_list file.

2. Redirect the standard error of a command to a file

$ command 2> file

Here bash redirects the stderr to file. The number 2 stands for stderr.

Here is how the file descriptor table changes:

Bash opens file for writing, gets the file descriptor for this file, and it replaces file descriptor 2 with the file descriptor of this file. So now anything written to stderr gets written to file.

3. Redirect both standard output and standard error to a file

$ command &>file

This one-liner uses the &> operator to redirect both output streams - stdout and stderr - from command to file. This is bash's shortcut for quickly redirecting both streams to the same destination.

Here is how the file descriptor table looks like after bash has redirected both streams:

As you can see both stdout and stderr now point to file. So anything written to stdout and stderr gets written to file.

There are several ways to redirect both streams to the same destination. You can redirect each stream one after another:

$ command >file 2>&1

This is a much more common way to redirect both streams to a file. First stdout is redirected to file, and then stderr is duplicated to be the same as stdout. So both streams end up pointing to file.

When bash sees several redirections it processes them from left to right. Let's go through the steps and see how that happens. Before running any commands bash's file descriptor table looks like this:

Now bash processes the first redirection >file. We've seen this before and it makes stdout point to file:

Next bash sees the second redirection 2>&1. We haven't seen this redirection before. This one duplicates file descriptor 2 to be a copy of file descriptor 1 and we get:

Both streams have been redirected to file.

However be careful here! Writing:

command >file 2>&1

Is not the same as writing:

$ command 2>&1 >file

The order of redirects matters in bash! This command redirects only the standard output to the file. The stderr will still print to the terminal. To understand why that happens, let's go through the steps again. So before running the command the file descriptor table looks like this:

Now bash processes redirections left to right. It first sees 2>&1 so it duplicates stderr to stdout. The file descriptor table becomes:

Now bash sees the second redirect >file and it redirects stdout to file:

Do you see what happens here? Stdout now points to file but the stderr still points to the terminal! Everything that gets written to stderr still gets printed out to the screen! So be very, very careful with the order of redirects!

Also note that in bash, writing this:

$ command &>file

Is exactly the same as:

$ command >&file

The first form is preferred however.

4. Discard the standard output of a command

$ command > /dev/null

The special file /dev/null discards all data written to it. So what we're doing here is redirecting stdout to this special file and it gets discarded. Here is how it looks from the file descriptor table's perspective:

Similarly, by combining the previous one-liners, we can discard both stdout and stderr by doing:

$ command >/dev/null 2>&1

Or just simply:

$ command &>/dev/null

File descriptor table for this feat looks like this:

5. Redirect the contents of a file to the stdin of a command

$ command <file

Here bash tries to open the file for reading before running any commands. If opening the file fails, bash quits with error and doesn't run the command. If opening the file succeeds, bash uses the file descriptor of the opened file as the stdin file descriptor for the command.

After doing that the file descriptor table looks like this:

Here is an example. Suppose you want to read the first line of the file in a variable. You can simply do this:

$ read -r line < file

Bash's built-in read command reads a single line from standard input. By using the input redirection operator < we set it up to read the line from the file.

6. Redirect a bunch of text to the stdin of a command

$ command <<EOL
your
multi-line
text
goes
here
EOL

Here we use the here-document redirection operator <<MARKER. This operator instructs bash to read the input from stdin until a line containing only MARKER is found. At this point bash passes the all the input read so far to the stdin of the command.

Here is a common example. Suppose you've copied a bunch of URLs to the clipboard and you want to remove http:// part of them. A quick one-liner to do this would be:

$ sed 's|http://||' <<EOL
http://url1.com
http://url2.com
http://url3.com
EOL

Here the input of a list of URLs is redirected to the sed command that strips http:// from the input.

This example produces this output:

url1.com
url2.com
url3.com

7. Redirect a single line of text to the stdin of a command

$ command <<< "foo bar baz"

For example, let's say you quickly want to pass the text in your clipboard as the stdin to a command. Instead of doing something like:

$ echo "clipboard contents" | command

You can now just write:

$ command <<< "clipboard contents"

This trick changed my life when I learned it!

8. Redirect stderr of all commands to a file forever

$ exec 2>file
$ command1
$ command2
$ ...

This one-liner uses the built-in exec bash command. If you specify redirects after it, then they will last forever, meaning until you change them or exit the script/shell.

In this case the 2>file redirect is setup that redirects the stderr of the current shell to the file. Running commands after setting up this redirect will have the stderr of all of them redirected to file. It's really useful in situations when you want to have a complete log of all errors that happened in the script, but you don't want to specify 2>file after every single command!

In general exec can take an optional argument of a command. If it's specified, bash replaces itself with the command. So what you get is only that command running, and there is no more shell.

9. Open a file for reading using a custom file descriptor

$ exec 3<file

Here we use the exec command again and specify the 3<file redirect to it. What this does is opens the file for reading and assigns the opened file-descriptor to the shell's file descriptor number 3. The file descriptor table now looks like this:

Now you can read from the file descriptor 3, like this:

$ read -u 3 line

This reads a line from the file that we just opened as fd 3.

Or you can use regular shell commands such as grep and operate on file descriptor 3:

$ grep "foo" <&3

What happens here is file descriptor 3 gets duplicated to file descriptor 1 - stdin of grep. Just remember that once you read the file descriptor it's been exhausted and you need to close it and open it again to use it. (You can't rewind an fd in bash.)

After you're done using fd 3, you can close it this way:

$ exec 3>&-

Here the file descriptor 3 is duped to -, which is bash's special way to say "close this fd".

10. Open a file for writing using a custom file descriptor

$ exec 4>file

Here we simply tell bash to open file for writing and assign it number 4. The file descriptor table looks like this:

As you can see file descriptors don't have to be used in order, you can open any file descriptor number you like from 0 to 255.

Now we can simply write to the file descriptor 4:

$ echo "foo" >&4

And we can close the file descriptor 4:

$ exec 4>&-

It's so simple now once we learned how to work with custom file descriptors!

11. Open a file both for writing and reading

$ exec 3<>file

Here we use bash's diamond operator <>. The diamond operator opens a file descriptor for both reading and writing.

So for example, if you do this:

$ echo "foo bar" > file   # write string "foo bar" to file "file".
$ exec 5<> file           # open "file" for rw and assign it fd 5.
$ read -n 3 var <&5       # read the first 3 characters from fd 5.
$ echo $var

This will output foo as we just read the first 3 chars from the file.

Now we can write some stuff to the file:

$ echo -n + >&5           # write "+" at 4th position.
$ exec 5>&-               # close fd 5.
$ cat file

This will output foo+bar as we wrote the + char at 4th position in the file.

12. Send the output from multiple commands to a file

$ (command1; command2) >file

This one-liner uses the (commands) construct that runs the commands a sub-shell. A sub-shell is a child process launched by the current shell.

So what happens here is the commands command1 and command2 get executed in the sub-shell, and bash redirects their output to file.

13. Execute commands in a shell through a file

Open two shells. In shell 1 do this:

mkfifo fifo
exec < fifo

In shell 2 do this:

exec 3> fifo;
echo 'echo test' >&3

Now take a look in shell 1. It will execute echo test. You can keep writing commands to fifo and shell 1 will keep executing them.

Here is how it works.

In shell 1 we use the mkfifo command to create a named pipe called fifo. A named pipe (also called a FIFO) is similar to a regular pipe, except that it's accessed as part of the file system. It can be opened by multiple processes for reading or writing. When processes are exchanging data via the FIFO, the kernel passes all data internally without writing it to the file system. Thus, the FIFO special file has no contents on the file system; the file system entry merely serves as a reference point so that processes can access the pipe using a name in the file system.

Next we use exec < fifo to replace current shell's stdin with fifo.

Now in shell 2 we open the named pipe for writing and assign it a custom file descriptor 3. Next we simply write echo test to the file descriptor 3, which goes to fifo.

Since shell 1's stdin is connected to this pipe it executes it! Really simple!

14. Access a website through bash

$ exec 3<>/dev/tcp/www.google.com/80
$ echo -e "GET / HTTP/1.1\n\n" >&3
$ cat <&3

Bash treats the /dev/tcp/host/port as a special file. It doesn't need to exist on your system. This special file is for opening tcp connections through bash.

In this example we first open file descriptor 3 for reading and writing and point it to /dev/tcp/www.google.com/80 special file, which is a connection to www.google.com on port 80.

Next we write GET / HTTP/1.1\n\n to file descriptor 3. And then we simply read the response back from the same file descriptor by using cat.

Similarly you can create a UDP connection through /dev/udp/host/port special file.

With /dev/tcp/host/port you can even write things like port scanners in bash!

15. Prevent overwriting the contents of a file when redirecting output

$ set -o noclobber

This turns on the noclobber option for the current shell. The noclobber option prevents you from overwriting existing files with the > operator.

If you try redirecting output to a file that exists, you'll get an error:

$ program > file
bash: file: cannot overwrite existing file

If you're 100% sure that you want to overwrite the file, use the >| redirection operator:

$ program >| file

This succeeds as it overrides the noclobber option.

16. Redirect standard input to a file and print it to standard output

$ command | tee file

The tee command is super handy. It's not part of bash but you'll use it often. It takes an input stream and prints it both to standard output and to a file.

In this example it takes the stdout of command, puts it in file, and prints it to stdout.

Here is a graphical illustration of how it works:

17. Send stdout of one process to stdin of another process

$ command1 | command2

This is simple piping. I'm sure everyone is familiar with this. I'm just including it here for completeness. Just to remind you, a pipe connects stdout of command1 with the stdin of command2.

It can be illustrated with a graphic:

As you can see, everything sent to file descriptor 1 (stdout) of command1 gets redirected through a pipe to file descriptor 0 (stdin) of command2.

You can read more about pipes in man 2 pipe.

18. Send stdout and stderr of one process to stdin of another process

$ command1 |& command2

This works on bash versions starting 4.0. The |& redirection operator sends both stdout and stderr of command1 over a pipe to stdin of command2.

As the new features of bash 4.0 aren't widely used, the old, and more portable way to do the same is:

$ command1 2>&1 | command2

Here is an illustration that shows what happens with file descriptors:

First command1's stderr is redirected to stdout, and then a pipe is setup between command1's stdout and command2's stdin.

19. Give file descriptors names

$ exec {filew}>output_file

Named file descriptors is a feature of bash 4.1. Named file descriptors look like {varname}. You can use them just like regular, numeric, file descriptors. Bash internally chooses a free file descriptor and assigns it a name.

20. Order of redirections

You can put the redirections anywhere in the command you wish. Check out these 3 examples, they all do the same:

$ echo hello >/tmp/example

$ echo >/tmp/example hello

$ >/tmp/example echo hello

Got to love bash!

21. Swap stdout and stderr

$ command 3>&1 1>&2 2>&3

Here we first duplicate file descriptor 3 to be a copy of stdout. Then we duplicate stdout to be a copy of stderr, and finally we duplicate stderr to be a copy of file descriptor 3, which is stdout. As a result we've swapped stdout and stderr.

Let's go through each redirection with illustrations. Before running the command, we've file descriptors pointing to the terminal:

Next bash setups 3>&1 redirection. This creates file descriptor 3 to be a copy of file descriptor 1:

Next bash setups 1>&2 redirection. This makes file descriptor 1 to be a copy of file descriptor 2:

Next bash setups 2>&3 redirection. This makes file descriptor 2 to be a copy of file descriptor 3:

If we want to be nice citizens we can also close file descriptor 3 as it's no longer needed:

$ command 3>&1 1>&2 2>&3 3>&-

The file descriptor table then looks like this:

As you can see, file descriptors 1 and 2 have been swapped.

22. Send stdout to one process and stderr to another process

$ command > >(stdout_cmd) 2> >(stderr_cmd)

This one-liner uses process substitution. The >(...) operator runs the commands in ... with stdin connected to the read part of an anonymous named pipe. Bash replaces the operator with the filename of the anonymous pipe.

So for example, the first substitution >(stdout_cmd) might return /dev/fd/60, and the second substitution might return /dev/fd/61. Both of these files are named pipes that bash created on the fly. Both named pipes have the commands as readers. The commands wait for someone to write to the pipes so they can read the data.

The command then looks like this:

$ command > /dev/fd/60 2> /dev/fd/61

Now these are just simple redirections. Stdout gets redirected to /dev/fd/60, and stderr gets redirected to /dev/fd/61.

When the command writes to stdout, the process behind /dev/fd/60 (process stdout_cmd) reads the data. And when the command writes to stderr, the process behind /dev/fd/61 (process stderr_cmd) reads the data.

23. Find the exit codes of all piped commands

Let's say you run several commands all piped together:

$ cmd1 | cmd2 | cmd3 | cmd4

And you want to find out the exit status codes of all these commands. How do you do it? There is no easy way to get the exit codes of all commands as bash returns only the exit code of the last command.

Bash developers thought about this and they added a special PIPESTATUS array that saves the exit codes of all the commands in the pipe stream.

The elements of the PIPESTATUS array correspond to the exit codes of the commands. Here's an example:

$ echo 'pants are cool' | grep 'moo' | sed 's/o/x/' | awk '{ print $1 }'
$ echo ${PIPESTATUS[@]}
0 1 0 0

In this example grep 'moo' fails, and the 2nd element of the PIPESTATUS array indicates failure.

Shoutouts

Shoutouts to bash hackers wiki for their illustrated redirection tutorial and bash version changes.

Enjoy!

Enjoy the article and let me know in the comments what you think about it! If you think that I forgot some interesting bash one-liners related to redirections, let me know in the comments below!

A Year of BloggingHey everyone! Another year has passed and it's now 5 years since I've been blogging here on catonmat! In this post I wish to summarize this year's statistics.

See the one year of blogging, two years of blogging, three years of blogging, and four years of blogging for the previous year statistics.

Let's start this year's stats with traffic:


Traffic statistics from Google Analytics for Jul 1, 2011 - Jul 1, 2012.

This year: 988,295 uniques. 1,374,228 visits. 1,940,530 page views.
Last year: unknown uniques. 1,014,765 visits. 1,453,557 page views.
Delta: +350,000 visits, +490,000 page views.

Here is Feedburner statistics. I used my Feedburner graph generator to produce it.


Feedburner statistics Jul 1, 2011 - Jul 1, 2012.

At 14,362 rss subscribers this year. Last year 13,500. Delta +862. subscribe.

Github stats: At 1070 followers this year. Last year 686. Delta +385. follow me.
Twitter stats: At 3093 followers this year. Last year 1998. Delta +1095. follow me.

Total articles written: 41. Top 15 articles by views:

My top 5 favorite articles:

During this year I made 10 announcements:

And I wrote 3 e-books:

That's all folks! Now let's have some 5 year birthday cake,

And let's meet for the cake next year again! See you!

This article is part of the article series "Bash One-Liners Explained."
<- previous article next article ->

This is the second part of the Bash One-Liners Explained article series. In this part I'll show you how to do various string manipulations with bash. I'll use only the best bash practices, various bash idioms and tricks. I want to illustrate how to get various tasks done with just bash built-in commands and bash programming language constructs.

See the first part of the series for introduction. After I'm done with the series I'll release an ebook (similar to my ebooks on awk, sed, and perl), and also bash1line.txt (similar to my perl1line.txt).

Also see my other articles about working fast in bash from 2007 and 2008:

Let's start.

Part II: Working With Strings

1. Generate the alphabet from a-z

$ echo {a..z}

This one-liner uses brace expansion. Brace expansion is a mechanism for generating arbitrary strings. This one-liner uses a sequence expression of the form {x..y}, where x and y are single characters. The sequence expression expands to each character lexicographically between x and y, inclusive.

If you run it, you get all the letters from a-z:

$ echo {a..z}
a b c d e f g h i j k l m n o p q r s t u v w x y z

2. Generate the alphabet from a-z without spaces between characters

$ printf "%c" {a..z}

This is an awesome bash trick that 99.99% bash users don't know about. If you supply a list of items to the printf function it actually applies the format in a loop until the list is empty! printf as a loop! There is nothing more awesome than that!

In this one-liner the printf format is "%c", which means "a character" and the arguments are all letters from a-z separated by space. So what printf does is it iterates over the list outputting each character after character until it runs out of letters.

Here is the output if you run it:

abcdefghijklmnopqrstuvwxyz

This output is without a terminating newline because the format string was "%c" and it doesn't include \n. To have it newline terminated, just add $'\n' to the list of chars to print:

$ printf "%c" {a..z} $'\n'

$'\n' is bash idiomatic way to represent a newline character. printf then just prints chars a to z, and the newline character.

Another way to add a trailing newline character is to echo the output of printf:

$ echo $(printf "%c" {a..z})

This one-liner uses command substitution, which runs printf "%c" {a..z} and replaces the command with its output. Then echo prints this output and adds a newline itself.

Want to output all letters in a column instead? Add a newline after each character!

$ printf "%c\n" {a..z}

Output:

a
b
...
z

Want to put the output from printf in a variable quickly? Use the -v argument:

$ printf -v alphabet "%c" {a..z}

This puts abcdefghijklmnopqrstuvwxyz in the $alphabet variable.

Similarly you can generate a list of numbers. Let's say from 1 to 100:

$ echo {1..100}

Output:

1 2 3 ... 100

Alternatively, if you forget this method, you can use the external seq utility to generate a sequence of numbers:

$ seq 1 100

3. Pad numbers 0 to 9 with a leading zero

$ printf "%02d " {0..9}

Here we use the looping abilities of printf again. This time the format is "%02d ", which means "zero pad the integer up to two positions", and the items to loop through are the numbers 0-9, generated by the brace expansion (as explained in the previous one-liner).

Output:

00 01 02 03 04 05 06 07 08 09

If you use bash 4, you can do the same with the new feature of brace expansion:

$ echo {00..09}

Older bashes don't have this feature.

4. Produce 30 English words

$ echo {w,t,}h{e{n{,ce{,forth}},re{,in,fore,with{,al}}},ither,at}

This is an abuse of brace expansion. Just look at what this produces:

when whence whenceforth where wherein wherefore wherewith wherewithal whither what then thence thenceforth there therein therefore therewith therewithal thither that hen hence henceforth here herein herefore herewith herewithal hither hat

Crazy awesome!

Here is how it works - you can produce permutations of words/symbols with brace expansion. For example, if you do this,

$ echo {a,b,c}{1,2,3}

It will produce the result a1 a2 a3 b1 b2 b3 c1 c2 c3. It takes the first a, and combines it with {1,2,3}, producing a1 a2 a3. Then it takes b and combines it with {1,2,3}, and then it does the same for c.

So this one-liner is just a smart combination of braces that when expanded produce all these English words!

5. Produce 10 copies of the same string

$ echo foo{,,,,,,,,,,}

This one-liner uses the brace expansion again. What happens here is foo gets combined with 10 empty strings, so the output is 10 copies of foo:

foo foo foo foo foo foo foo foo foo foo foo

6. Join two strings

$ echo "$x$y"

This one-liner simply concatenates two variables together. If the variable x contains foo and y contains bar then the result is foobar.

Notice that "$x$y" were quoted. If we didn't quote it, echo would interpret the $x$y as regular arguments, and would first try to parse them to see if they contain command line switches. So if $x contains something beginning with -, it would be a command line argument rather than an argument to echo:

x=-n
y=" foo"
echo $x$y

Output:

foo

Versus the correct way:

x=-n
y=" foo"
echo "$x$y"

Output:

-n foo

If you need to put the two joined strings in a variable, you can omit the quotes:

var=$x$y

7. Split a string on a given character

Let's say you have a string foo-bar-baz in the variable $str and you wish to split it on the dash and iterate over it. You can simply combine IFS with read to do it:

$ IFS=- read -r x y z <<< "$str"

Here we use the read x command that reads data from stdin and puts the data in the x y z variables. We set IFS to - as this variable is used for field splitting. If multiple variable names are specified to read, IFS is used to split the line of input so that each variable gets a single field of the input.

In this one-liner $x gets foo, $y gets bar, $z gets baz.

Also notice the use of <<< operator. This is the here-string operator that allows strings to be passed to stdin of commands easily. In this case string $str is passed as stdin to read.

You can also put the split fields and put them in an array:

$ IFS=- read -ra parts <<< "foo-bar-baz"

The -a argument to read makes it put the split words in the given array. In this case the array is parts. You can access array elements through ${parts[0]}, ${parts[1]}, and ${parts[0]}. Or just access all of them through ${parts[@]}.

8. Process a string character by character

$ while IFS= read -rn1 c; do
    # do something with $c
done <<< "$str"

Here we use the -n1 argument to read command to make it read the input character at a time. Similarly we can use -n2 to read two chars at a time, etc.

9. Replace "foo" with "bar" in a string

$ echo ${str/foo/bar}

This one-liner uses parameter expansion of form ${var/find/replace}. It finds the string find in var and replaces it with replace. Really simple!

To replace all occurrences of "foo" with "bar", use the ${var//find/replace} form:

$ echo ${str//foo/bar}

10. Check if a string matches a pattern

$ if [[ $file = *.zip ]]; then
    # do something
fi

Here the one-liner does something if $file matches *.zip. This is a simple glob pattern matching, and you can use symbols * ? [...] to do matching. Code * matches any string, ? matches a single char, and [...] matches any character in ... or a character class.

Here is another example that matches if answer is Y or y:

$ if [[ $answer = [Yy]* ]]; then
    # do something
fi

11. Check if a string matches a regular expression

$ if [[ $str =~ [0-9]+\.[0-9]+ ]]; then
    # do something
fi

This one-liner tests if the string $str matches regex [0-9]+\.[0-9]+, which means match a number followed by a dot followed by number. The format for regular expressions is described in man 3 regex.

12. Find the length of the string

$ echo ${#str}

Here we use parameter expansion ${#str} which returns the length of the string in variable str. Really simple.

13. Extract a substring from a string

$ str="hello world"
$ echo ${str:6}

This one-liner extracts world from hello world. It uses the substring expansion. In general substring expansion looks like ${var:offset:length}, and it extracts length characters from var starting at index offset. In our one-liner we omit the length that makes it extract all characters starting at offset 6.

Here is another example:

$ echo ${str:7:2}

Output:

or

14. Uppercase a string

$ declare -u var
$ var="foo bar"

The declare command in bash declares variables and/or gives them attributes. In this case we give the variable var attribute -u, which upper-cases its content whenever it gets assigned something. Now if you echo it, the contents will be upper-cased:

$ echo $var
FOO BAR

Note that -u argument was introduced in bash 4. Similarly you can use another feature of bash 4, which is the ${var^^} parameter expansion that upper-cases a string in var:

$ str="zoo raw"
$ echo ${str^^}

Output:

ZOO RAW

15. Lowercase a string

$ declare -l var
$ var="FOO BAR"

Similar to the previous one-liner, -l argument to declare sets the lower-case attribute on var, which makes it always be lower-case:

$ echo $var
foo bar

The -l argument is also available only in bash 4 and later.

Another way to lowercase a string is to use ${var,,} parameter expansion:

$ str="ZOO RAW"
$ echo ${str,,}

Output:

zoo raw

Enjoy!

Enjoy the article and let me know in the comments what you think about it! If you think that I forgot some interesting bash one-liners related to string operations, let me know in the comments also!

This article is part of the article series "Bash One-Liners Explained."
<- previous article next article ->

I love being super fast in the shell so I decided to do a new article series called Bash One-Liners Explained. It's going to be similar to my other article series - Awk One-Liners Explained, Sed One-Liners Explained, and Perl One-Liners Explained. After I'm done with this bash series, I'll release an e-book by the same title, just as I did with awk, sed, and perl series. The e-book will be available as a pdf and in mobile formats (mobi and epub). I'll also be releasing bash1line.txt, similar to perl1line.txt that I made for the perl series.

In this series I'll use the best bash practices, various bash idioms and tricks. I want to illustrate how to get various tasks done with just bash built-in commands and bash programming language constructs.

Also see my other articles about working fast in bash:

Let's start.

Part I: Working With Files

1. Empty a file (truncate to 0 size)

$ > file

This one-liner uses the output redirection operator >. Redirection of output causes the file to be opened for writing. If the file does not exist it is created; if it does exist it is truncated to zero size. As we're not redirecting anything to the file it remains empty.

If you wish to replace the contents of a file with some string or create a file with specific content, you can do this:

$ echo "some string" > file

This puts the string "some string" in the file.

2. Append a string to a file

$ echo "foo bar baz" >> file

This one-liner uses a different output redirection operator >>, which appends to the file. If the file does not exist it is created. The string appended to the file is followed by a newline. If you don't want a newline appended after the string, add the -n argument to echo:

$ echo -n "foo bar baz" >> file

3. Read the first line from a file and put it in a variable

$ read -r line < file

This one-liner uses the built-in bash command read and the input redirection operator <. The read command reads one line from the standard input and puts it in the line variable. The -r parameter makes sure the input is read raw, meaning the backslashes won't get escaped (they'll be left as is). The redirection command < file opens file for reading and makes it the standard input to the read command.

The read command removes all characters present in the special IFS variable. IFS stands for Internal Field Separator that is used for word splitting after expansion and to split lines into words with the read built-in command. By default IFS contains space, tab, and newline, which means that the leading and trailing tabs and spaces will get removed. If you wish to preserve them, you can set IFS to nothing for the time being:

$ IFS= read -r line < file

This will change the value of IFS just for this command and will make sure the first line gets read into the line variable really raw with all the leading and trailing whitespaces.

Another way to read the first line from a file into a variable is to do this:

$ line=$(head -1 file)

This one-liner uses the command substitution operator $(...). It runs the command in ..., and returns its output. In this case the command is head -1 file that outputs the first line of the file. The output is then assigned to the line variable. Using $(...) is exactly the same as `...`, so you could have also written:

$ line=`head -1 file`

However $(...) is the preferred way in bash as it's cleaner and easier to nest.

4. Read a file line-by-line

$ while read -r line; do
    # do something with $line
done < file

This is the one and only right way to read lines from a file one-by-one. This method puts the read command in a while loop. When the read command encounters end-of-file, it returns a positive return code (code for failure) and the while loop stops.

Remember that read trims leading and trailing whitespace, so if you wish to preserve it, clear the IFS variable:

$ while IFS= read -r line; do
    # do something with $line
done < file

If you don't like the to put < file at the end, you can also pipe the contents of the file to the while loop:

$ cat file | while IFS= read -r line; do
    # do something with $line
done

5. Read a random line from a file and put it in a variable

$ read -r random_line < <(shuf file)

There is no clean way to read a random line from a file with just bash, so we'll need to use some external programs for help. If you're on a modern Linux machine, then it comes with the shuf utility that's in GNU coreutils.

This one-liner uses the process substitution <(...) operator. This process substitution operator creates an anonymous named pipe, and connects the stdout of the process to the write part of the named pipe. Then bash executes the process, and it replaces the whole process substitution expression with the filename of the anonymous named pipe.

When bash sees <(shuf file) it opens a special file /dev/fd/n, where n is a free file descriptor, then runs shuf file with its stdout connected to /dev/fd/n and replaces <(shuf file) with /dev/fd/n so the command effectively becomes:

$ read -r random_line < /dev/fd/n

Which reads the first line from the shuffled file.

Here is another way to do it with the help of GNU sort. GNU sort takes the -R option that randomizes the input.

$ read -r random_line < <(sort -R file)

Another way to get a random line in a variable is this:

$ random_line=$(sort -R file | head -1)

Here the file gets randomly sorted by sort -R and then head -1 takes the first line.

6. Read the first three columns/fields from a file into variables

$ while read -r field1 field2 field3 throwaway; do
    # do something with $field1, $field2, and $field3
done < file

If you specify more than one variable name to the read command, it shall split the line into fields (splitting is done based on what's in the IFS variable, which contains a whitespace, a tab, and a newline by default), and put the first field in the first variable, the second field in the second variable, etc., and it will put the remaining fields in the last variable. That's why we have the throwaway variable after the three field variables. if we didn't have it, and the file had more than three columns, the third field would also get the leftovers.

Sometimes it's shorter to just write _ for the throwaway variable:

$ while read -r field1 field2 field3 _; do
    # do something with $field1, $field2, and $field3
done < file

Or if you have a file with exactly three fields, then you don't need it at all:

$ while read -r field1 field2 field3; do
    # do something with $field1, $field2, and $field3
done < file

Here is an example. Let's say you wish to find out number of lines, number of words, and number of bytes in a file. If you run wc on a file you get these 3 numbers plus the filename as the fourth field:

$ cat file-with-5-lines
x 1
x 2
x 3
x 4
x 5

$ wc file-with-5-lines
 5 10 20 file-with-5-lines

So this file has 5 lines, 10 words, and 20 chars. We can use the read command to get this info into variables:

$ read lines words chars _ < <(wc file-with-5-lines)

$ echo $lines
5
$ echo $words
10
$ echo $chars
20

Similarly you can use here-strings to split strings into variables. Let's say you have a string "20 packets in 10 seconds" in a $info variable and you want to extract 20 and 10. Not too long ago I'd have written this:

$ packets=$(echo $info | awk '{ print $1 }')
$ time=$(echo $info | awk '{ print $4 }')

However given the power of read and our bash knowledge, we can now do this:

$ read packets _ _ time _ <<< "$info"

Here the <<< is a here-string, which lets you pass strings directly to the standard input of commands.

7. Find the size of a file, and put it in a variable

$ size=$(wc -c < file)

This one-liner uses the command substitution operator $(...) that I explained in one-liner #3. It runs the command in ..., and returns its output. In this case the command is wc -c < file that prints the number of chars (bytes) in the file. The output is then assigned to size variable.

8. Extract the filename from the path

Let's say you have a /path/to/file.ext, and you wish to extract just the filename file.ext. How do you do it? A good solution is to use the parameter expansion mechanism:

$ filename=${path##*/}

This one-liner uses the ${var##pattern} parameter expansion. This expansion tries to match the pattern at the beginning of the $var variable. If it matches, then the result of the expansion is the value of $var with the longest matching pattern deleted.

In this case the pattern is */ which matches at the beginning of /path/to/file.ext and as it's a greedy match, the pattern matches all the way till the last slash (it matches /path/to/). The result of this expansion is then just the filename file.ext as the matched pattern gets deleted.

9. Extract the directory name from the path

This is similar to the previous one-liner. Let's say you have a /path/to/file.ext, and you wish to extract just the path to the file /path/to. You can use the parameter expansion again:

$ dirname=${path%/*}

This time it's the ${var%pattern} parameter expansion that tries to match the pattern at the end of the $var variable. If the pattern matches, then the result of the expansion is the value of $var shortest matching pattern deleted.

In this case the pattern is /*, which matches at the end of /path/to/file.ext (it matches /file.ext). The result then is just the dirname /path/to as the matched pattern gets deleted.

10. Make a copy of a file quickly

Let's say you wish to copy the file at /path/to/file to /path/to/file_copy. Normally you'd write:

$ cp /path/to/file /path/to/file_copy

However you can do it much quicker by using the brace expansion {...}:

$ cp /path/to/file{,_copy}

Brace expansion is a mechanism by which arbitrary strings can be generated. In this particular case /path/to/file{,_copy} generates the string /path/to/file /path/to/file_copy, and the whole command becomes cp /path/to/file /path/to/file_copy.

Similarly you can move a file quickly:

$ mv /path/to/file{,_old}

This expands to mv /path/to/file /path/to/file_old.

Enjoy!

Enjoy the article and let me know in the comments what you think about it! If you think that I forgot some interesting bash one-liners related to file operations, let me know in to comments also!