I thought I'd do a shorter article on catonmat this time. It goes hand in hand with my upcoming article series on "100% technical guide to anonymity" and it's much easier to write larger articles in smaller pieces. Then I can edit them together and produce the final article.

This article will be interesting for those who didn't know it already -- you can turn any Linux computer into a SOCKS5 (and SOCKS4) proxy in just one command:

ssh -N -D 0.0.0.0:1080 localhost

And it doesn't require root privileges. The ssh command starts up dynamic -D port forwarding on port 1080 and talks to the clients via SOCSK5 or SOCKS4 protocols, just like a regular SOCKS5 proxy would! The -N option makes sure ssh stays idle and doesn't execute any commands on localhost.

If you also wish the command to go into background as a daemon, then add -f option:

ssh -f -N -D 0.0.0.0:1080 localhost

To use it, just make your software use SOCKS5 proxy on your Linux computer's IP, port 1080, and you're done, all your requests now get proxied.

Access control can be implemented via iptables. For example, to allow only people from the ip 1.2.3.4 to use the SOCKS5 proxy, add the following iptables rules:

iptables -A INPUT --src 1.2.3.4 -p tcp --dport 1080 -j ACCEPT
iptables -A INPUT -p tcp --dport 1080 -j REJECT

The first rule says, allow anyone from 1.2.3.4 to connect to port 1080, and the other rule says, deny everyone else from connecting to port 1080.

Surely, executing iptables requires root privileges. If you don't have root privileges, and you don't want to leave your proxy open (and you really don't want to do that), you'll have to use some kind of a simple TCP proxy wrapper to do access control.

Here, I wrote one in Perl. It's called tcp-proxy.pl and it uses IO::Socket::INET to abstract sockets, and IO::Select to do connection multiplexing.

#!/usr/bin/perl
#

use warnings;
use strict;

use IO::Socket::INET;
use IO::Select;

my @allowed_ips = ('1.2.3.4', '5.6.7.8', '127.0.0.1', '192.168.1.2');
my $ioset = IO::Select->new;
my %socket_map;

my $debug = 1;

sub new_conn {
    my ($host, $port) = @_;
    return IO::Socket::INET->new(
        PeerAddr => $host,
        PeerPort => $port
    ) || die "Unable to connect to $host:$port: $!";
}

sub new_server {
    my ($host, $port) = @_;
    my $server = IO::Socket::INET->new(
        LocalAddr => $host,
        LocalPort => $port,
        ReuseAddr => 1,
        Listen    => 100
    ) || die "Unable to listen on $host:$port: $!";
}

sub new_connection {
    my $server = shift;
    my $client = $server->accept;
    my $client_ip = client_ip($client);

    unless (client_allowed($client)) {
        print "Connection from $client_ip denied.\n" if $debug;
        $client->close;
        return;
    }
    print "Connection from $client_ip accepted.\n" if $debug;

    my $remote = new_conn('localhost', 55555);
    $ioset->add($client);
    $ioset->add($remote);

    $socket_map{$client} = $remote;
    $socket_map{$remote} = $client;
}

sub close_connection {
    my $client = shift;
    my $client_ip = client_ip($client);
    my $remote = $socket_map{$client};
    
    $ioset->remove($client);
    $ioset->remove($remote);

    delete $socket_map{$client};
    delete $socket_map{$remote};

    $client->close;
    $remote->close;

    print "Connection from $client_ip closed.\n" if $debug;
}

sub client_ip {
    my $client = shift;
    return inet_ntoa($client->sockaddr);
}

sub client_allowed {
    my $client = shift;
    my $client_ip = client_ip($client);
    return grep { $_ eq $client_ip } @allowed_ips;
}

print "Starting a server on 0.0.0.0:1080\n";
my $server = new_server('0.0.0.0', 1080);
$ioset->add($server);

while (1) {
    for my $socket ($ioset->can_read) {
        if ($socket == $server) {
            new_connection($server);
        }
        else {
            next unless exists $socket_map{$socket};
            my $remote = $socket_map{$socket};
            my $buffer;
            my $read = $socket->sysread($buffer, 4096);
            if ($read) {
                $remote->syswrite($buffer);
            }
            else {
                close_connection($socket);
            }
        }
    }
}

To use it, you'll have to make a change to the previous configuration. Instead of running ssh SOCKS5 proxy on 0.0.0.0:1080, you'll need to run it on localhost:55555,

ssh -f -N -D 55555 localhost

After that, run the tcp-proxy.pl,

perl tcp-proxy.pl &

The TCP proxy will start listening on 0.0.0.0:1080 and will redirect only the allowed IPs in @allowed_ips list to localhost:55555.

Another possibility is to use another computer instead of your own as exit node. What I mean is you can do the following:

ssh -f -N -D 1080 other_computer.com

This will set up a SOCKS5 proxy on localhost:1080 but when you use it, ssh will automatically tunnel your requests (encrypted) via other_computer.com. This way you can hide what you're doing on the Internet from anyone who might be sniffing your link. They will see that you're doing something but the traffic will be encrypted so they won't be able to tell what you're doing.

That's it. You're now the proxy king!

Download tcp-proxy.pl

Download link: tcp proxy (tcp-proxy.pl)
Download URL: http://www.catonmat.net/download/tcp-proxy.pl
Downloaded: 5139 times

I also pushed the tcp-proxy.pl to GitHub: tcp-proxy.pl on GitHub. This project is also pretty nifty to generalize and make a program that redirects between any number of hosts:ports, not just two.

PS. I will probably also write "A definitive guide to ssh port forwarding" some time in the future because it's an interesting but little understood topic.

var http = require('http');

http.createServer(function(request, response) {
  var proxy = http.createClient(80, request.headers['host'])
  var proxy_request = proxy.request(request.method, request.url, request.headers);
  proxy_request.addListener('response', function (proxy_response) {
    proxy_response.addListener('data', function(chunk) {
      response.write(chunk, 'binary');
    });
    proxy_response.addListener('end', function() {
      response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
  });
  request.addListener('data', function(chunk) {
    proxy_request.write(chunk, 'binary');
  });
  request.addListener('end', function() {
    proxy_request.end();
  });
}).listen(8080);

This is just amazing. In 20 lines of node.js code and 10 minutes of time I was able to write a HTTP proxy. And it scales well, too. It's not a blocking HTTP proxy, it's event driven and asynchronous, meaning hundreds of people can use simultaneously and it will work well.

To get the proxy running all you have to do is download node.js, compile it, and run the proxy program via the node program:

$ ./configure --prefix=/home/pkrumins/installs/nodejs-0.1.92
$ make
$ make install

$ PATH=$PATH:/home/pkrumins/installs/nodejs-0.1.92/bin

$ node proxy.js

And from here you can take this proxy wherever your imagination takes. For example, you can start by adding logging:

var http = require('http');
var sys  = require('sys');

http.createServer(function(request, response) {
  sys.log(request.connection.remoteAddress + ": " + request.method + " " + request.url);
  var proxy = http.createClient(80, request.headers['host'])
  var proxy_request = proxy.request(request.method, request.url, request.headers);
  proxy_request.addListener('response', function (proxy_response) {
    proxy_response.addListener('data', function(chunk) {
      response.write(chunk, 'binary');
    });
    proxy_response.addListener('end', function() {
      response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
  });
  request.addListener('data', function(chunk) {
    proxy_request.write(chunk, 'binary');
  });
  request.addListener('end', function() {
    proxy_request.end();
  });
}).listen(8080);

Next, you can add a regex-based host blacklist in 15 additional lines:

var http = require('http');
var sys  = require('sys');
var fs   = require('fs');

var blacklist = [];

fs.watchFile('./blacklist', function(c,p) { update_blacklist(); });

function update_blacklist() {
  sys.log("Updating blacklist.");
  blacklist = fs.readFileSync('./blacklist').split('\n')
              .filter(function(rx) { return rx.length })
              .map(function(rx) { return RegExp(rx) });
}

http.createServer(function(request, response) {
  for (i in blacklist) {
    if (blacklist[i].test(request.url)) {
      sys.log("Denied: " + request.method + " " + request.url);
      response.end();
      return;
    }
  }

  sys.log(request.connection.remoteAddress + ": " + request.method + " " + request.url);
  var proxy = http.createClient(80, request.headers['host'])
  var proxy_request = proxy.request(request.method, request.url, request.headers);
  proxy_request.addListener('response', function(proxy_response) {
    proxy_response.addListener('data', function(chunk) {
      response.write(chunk, 'binary');
    });
    proxy_response.addListener('end', function() {
      response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
  });
  request.addListener('data', function(chunk) {
    proxy_request.write(chunk, 'binary);
  });
  request.addListener('end', function() {
    proxy_request.end();
  });
}).listen(8080);

update_blacklist();

Now to block proxy users from using Facebook, just echo facebook.com to blacklist file:

$ echo 'facebook.com' >> blacklist

The proxy server will automatically notice the changes to the file and update the blacklist.

Surely, a proxy server without IP control is no proxy server, so let's add that as well:

var http = require('http');
var sys  = require('sys');
var fs   = require('fs');

var blacklist = [];
var iplist    = [];

fs.watchFile('./blacklist', function(c,p) { update_blacklist(); });
fs.watchFile('./iplist', function(c,p) { update_iplist(); });

function update_blacklist() {
  sys.log("Updating blacklist.");
  blacklist = fs.readFileSync('./blacklist').split('\n')
              .filter(function(rx) { return rx.length })
              .map(function(rx) { return RegExp(rx) });
}

function update_iplist() {
  sys.log("Updating iplist.");
  iplist = fs.readFileSync('./iplist').split('\n')
           .filter(function(ip) { return ip.length });
}

http.createServer(function(request, response) {
  var allowed_ip = false;
  for (i in iplist) {
    if (iplist[i] == request.connection.remoteAddress) {
      allowed_ip = true;
      break;
    }
  }

  if (!allowed_ip) {
    sys.log("IP " + request.connection.remoteAddress + " is not allowed");
    response.end();
    return;
  }

  for (i in blacklist) {
    if (blacklist[i].test(request.url)) {
      sys.log("Denied: " + request.method + " " + request.url);
      response.end();
      return;
    }
  }

  sys.log(request.connection.remoteAddress + ": " + request.method + " " + request.url);
  var proxy = http.createClient(80, request.headers['host'])
  var proxy_request = proxy.request(request.method, request.url, request.headers);
  proxy_request.addListener('response', function(proxy_response) {
    proxy_response.addListener('data', function(chunk) {
      response.write(chunk, 'binary');
    });
    proxy_response.addListener('end', function() {
      response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
  });
  request.addListener('data', function(chunk) {
    proxy_request.write(chunk, 'binary');
  });
  request.addListener('end', function() {
    proxy_request.end();
  });
}).listen(8080);

update_blacklist();
update_iplist();

By default the proxy server will not allow any connections, so add all the IPs you want the proxy to be accessible from to iplist file:

$ echo '1.2.3.4' >> iplist

Finally, let's refactor the code a little:

var http = require('http');
var sys  = require('sys');
var fs   = require('fs');

var blacklist = [];
var iplist    = [];

fs.watchFile('./blacklist', function(c,p) { update_blacklist(); });
fs.watchFile('./iplist', function(c,p) { update_iplist(); });

function update_blacklist() {
  sys.log("Updating blacklist.");
  blacklist = fs.readFileSync('./blacklist').split('\n')
              .filter(function(rx) { return rx.length })
              .map(function(rx) { return RegExp(rx) });
}

function update_iplist() {
  sys.log("Updating iplist.");
  iplist = fs.readFileSync('./iplist').split('\n')
           .filter(function(rx) { return rx.length });
}

function ip_allowed(ip) {
  for (i in iplist) {
    if (iplist[i] == ip) {
      return true;
    }
  }
  return false;
}

function host_allowed(host) {
  for (i in blacklist) {
    if (blacklist[i].test(host)) {
      return false;
    }
  }
  return true;
}

function deny(response, msg) {
  response.writeHead(401);
  response.write(msg);
  response.end();
}

http.createServer(function(request, response) {
  var ip = request.connection.remoteAddress;
  if (!ip_allowed(ip)) {
    msg = "IP " + ip + " is not allowed to use this proxy";
    deny(response, msg);
    sys.log(msg);
    return;
  }

  if (!host_allowed(request.url)) {
    msg = "Host " + request.url + " has been denied by proxy configuration";
    deny(response, msg);
    sys.log(msg);
    return;
  }

  sys.log(ip + ": " + request.method + " " + request.url);
  var proxy = http.createClient(80, request.headers['host'])
  var proxy_request = proxy.request(request.method, request.url, request.headers);
  proxy_request.addListener('response', function(proxy_response) {
    proxy_response.addListener('data', function(chunk) {
      response.write(chunk, 'binary');
    });
    proxy_response.addListener('end', function() {
      response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
  });
  request.addListener('data', function(chunk) {
    proxy_request.write(chunk, 'binary);
  });
  request.addListener('end', function() {
    proxy_request.end();
  });
}).listen(8080);

update_blacklist();
update_iplist();

Again, it's amazing how fast you can write server software in node.js and JavaScript. It would have probably taken me a day to write the same in C. I love how fast you can prototype the software nowadays.

Download proxy.js

Download link: proxy server written in node.js
Download URL: http://www.catonmat.net/download/proxy.js
Downloaded: 9158 times

I am gonna build this proxy up, so I also put it on GitHub: proxy.js on GitHub

Happy proxying!

This article is part of the article series "CommandLineFu One-Liners Explained."
<- previous article next article ->

CommandLineFu ExplainedAnother week and another top ten one-liners from commandlinefu explained.

This is the third post in the series already, covering one-liners 21-30. See the previous two posts for the introduction of the series and one-liners 1-20:

Update: Russian translation now available.

#21. Display currently mounted file systems nicely

$ mount | column -t

The file systems are not that important here. The column -t command is what is important. It takes the input and formats it into multiple columns so that all columns were aligned vertically.

Here is how the mounted filesystem list looks without column -t command:

$ mount

/dev/root on / type ext3 (rw)
/proc on /proc type proc (rw)
/dev/mapper/lvmraid-home on /home type ext3 (rw,noatime)

And now with column -t command:

$ mount | column -t

/dev/root                 on  /      type  ext3   (rw)
/proc                     on  /proc  type  proc   (rw)
/dev/mapper/lvmraid-home  on  /home  type  ext3   (rw,noatime)

You can improve this one-liner now by also adding column titles:

$ (echo "DEVICE - PATH - TYPE FLAGS" && mount) | column -t

DEVICE                    -   PATH   -     TYPE   FLAGS
/dev/root                 on  /      type  ext3   (rw)
/proc                     on  /proc  type  proc   (rw)
/dev/mapper/lvmraid-home  on  /home  type  ext3   (rw,noatime)

Columns 2 and 4 are not really necessary. We can use awk text processing utility to get rid of them:

$ (echo "DEVICE PATH TYPE FLAGS" && mount | awk '$2=$4="";1') | column -t

DEVICE                    PATH   TYPE   FLAGS
/dev/root                 /      ext3   (rw)
/proc                     /proc  proc   (rw)
/dev/mapper/lvmraid-home  /home  ext3   (rw,noatime)

Finally, we can make it an alias so that we always enjoyed the nice output from mount. Let's call this alias nicemount:

$ nicemount() { (echo "DEVICE PATH TYPE FLAGS" && mount | awk '$2=$4="";1') | column -t; }

Let's try it out:

$ nicemount

DEVICE                    PATH   TYPE   FLAGS
/dev/root                 /      ext3   (rw)
/proc                     /proc  proc   (rw)
/dev/mapper/lvmraid-home  /home  ext3   (rw,noatime)

It works!

#22. Run the previous shell command but replace every "foo" with "bar"

$ !!:gs/foo/bar

I explained this type of one-liners in one-liner #5 already. Please take a look for a longer discussion.

To summarize, what happens here is that the !! recalls the previous executed shell command and :gs/foo/bar substitutes (the :s flag) all (the g flag) occurrences of foo with bar. The !! construct is called an event designator.

#23. Top for files

$ watch -d -n 1 'df; ls -FlAt /path'

This one-liner watches for file changes in directory /path. It uses the watch command that executes the given command periodically. The -d flag tells watch to display differences between the command calls (so you saw what files get added or removed in /path). The -n 1 flag tells it to execute the command every second.

The command to execute is df; ls -FlAt /path that is actually two commands, executed one after other. First, df outputs the filesystem disk space usage, and then ls -FlAt lists the files in /path. The -F argument to ls tells it to classify files, appending */=>@| to the filenames to indicate whether they are executables *, directories /, sockets =, doors >, symlinks @, or named pipes |. The -l argument lists all files, -A hides . and .., and -t sorts the files by time.

Special note about doors - they are Solaris thing that act like pipes, except they launch the program that is supposed to be the receiving party. A plain pipe would block until the other party opens it, but a door launches the other party itself.

Actually the output is nicer if you specify -h argument to df so it was human readable. You can also join the arguments to watch together, making them -dn1. Here is the final version:

$ watch -dn1 'df -h; ls -FlAt /path'

Another note - -d in BSD is --differences

#24. Mount a remote folder through SSH

$ sshfs name@server:/path/to/folder /path/to/mount/point

That's right, you can mount a remote directory locally via SSH! You'll first need to install two programs however:

  • FUSE that allows to implement filesystems in userspace programs, and
  • sshfs client that uses FUSE and sftp (secure ftp - comes with OpenSSH, and is on your system already) to access the remote host.

And that's it, now you can use sshfs to mount remote directories via SSH.

To unmount, use fusermount:

fusermount -u /path/to/mount/point

#25. Read Wikipedia via DNS

$ dig +short txt <keyword>.wp.dg.cx

This is probably the most interesting one-liner today. David Leadbeater created a DNS server, which when queried the TXT record type, returns a short plain-text version of a Wikipedia article. Here is his presentation on he did it.

Here is an example, let's find out what "hacker" means:

$ dig +short txt hacker.wp.dg.cx

"Hacker may refer to: Hacker (computer security), someone involved
in computer security/insecurity, Hacker (programmer subculture), a
programmer subculture originating in the US academia in the 1960s,
which is nowadays mainly notable for the free software/" "open
source movement, Hacker (hobbyist), an enthusiastic home computer
hobbyist http://a.vu/w:Hacker"

The one-liner uses dig, the standard sysadmin's utility for DNS troubleshooting to do the DNS query. The +short option makes it output only the returned text response, and txt makes it query the TXT record type.

This one-liner is actually alias worthy, so let's make an alias:

wiki() { dig +short txt $1.wp.dg.cx; }

Try it out:

$ wiki hacker

"Hacker may refer to: Hacker (computer security), ..."

It works!

If you don't have dig, you may also use host that also performs DNS lookups:

host -t txt hacker.wp.dg.cx

#26. Download a website recursively with wget

$ wget --random-wait -r -p -e robots=off -U Mozilla www.example.com

This one-liner does what it says. Here is the explanation of the arguments:

  • --random-wait - wait between 0.5 to 1.5 seconds between requests.
  • -r - turn on recursive retrieving.
  • -e robots=off - ignore robots.txt.
  • -U Mozilla - set the "User-Agent" header to "Mozilla". Though a better choice is a real User-Agent like "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729)".

Some other useful options are:

  • --limit-rate=20k - limits download speed to 20kbps.
  • -o logfile.txt - log the downloads.
  • -l 0 - remove recursion depth (which is 5 by default).
  • --wait=1h - be sneaky, download one file every hour.

#27. Copy the arguments of the most recent command

ALT + . (or ESC + .)

This keyboard shortcut works in shell's emacs editing mode only, it copies the last argument form the last command to the current command. Here is an example:

$ echo a b c
a b c

$ echo <Press ALT + .>
$ echo c

If you repeat the command, it copies the last argument from the command before the last, then if you repeat again, it copies the last argument from command before the command before the last, etc.

Here is an example:

$ echo 1 2 3
1 2 3
$ echo a b c
a b c

$ echo <Press ALT + .>
$ echo c

$ echo <Press ALT + .> again
$ echo 3

However, if you wish to get 1st or 2nd or n-th argument, use the digit-argument command ALT + 1 (or ESC + 1) or ALT + 2 (or ESC +2), etc. Here is an example:

$ echo a b c
a b c

$ echo <Press ALT + 1> <Press ALT + .>
$ echo a
a

$ echo <Press ALT + 2> <Press ALT + .>
$ echo b
b

See my article on Emacs Editing Mode Keyboard Shortcuts for a tutorial and a cheat sheet of all the shortcuts.

#28. Execute a command without saving it in the history

$ <space>command

This one-liner works at least on bash, I haven't tested other shells.

If you start your command by a space, it won't be saved to bash history (~/.bash_history file). This behavior is controlled by $HISTIGNORE shell variable. Mine is set to HISTIGNORE="&:[ ]*", which means don't save repeated commands to history, and don't save commands that start with a space to history. The values in $HISTIGNORE are colon-separated.

If you're interested, see my article "The Definitive Guide to Bash Command Line History" for a short tutorial on how to work with shell history and a summary cheat sheet.

#29. Show the size of all sub folders in the current directory

$ du -h --max-depth=1

The --max-depth=1 causes du to summarize disk usage statistics for directories that are depth 1 from the current directory, that is, all directories in the current directory. The -h argument makes the summary human-readable, that is, displays 5MB instead of 5242880 (bytes).

If you are interested in both sub folder size and file size in the current directory, you can use the shorter:

$ du -sh *

#30. Display the top ten running processes sorted by memory usage

$ ps aux | sort -nk +4 | tail

This is certainly not the best way to display the top ten processes that consume the most memory, but, hey, it works.

It takes the output of ps aux, sorts it by 4th column numerically and then uses tail to output the last then lines which happen to be the processes with the biggest memory consumption.

If I was to find out who consumes the most memory, I'd simply use htop or top and not ps.

Bonus one-liner: Start an SMTP server

python -m smtpd -n -c DebuggingServer localhost:1025

This one-liner starts an SMTP server on port 1025. It uses Python's standard library smtpd (specified by -m smtpd) and passes it three arguments - -n, -c DebuggingServer and localhost:1025.

The -n argument tells Python not to setuid (change user) to "nobody" - it makes the code run under your user.

The -c DebuggingServer argument tells Python to use DebuggingServer class as the SMTP implementation that prints each message it receives to stdout.

The localhost:1025 argument tells Python to start the SMTP server on locahost, port 1025.

However, if you wish to start it on the standard port 25, you'll have to use sudo command, because only root is allowed to start services on ports 1-1024. These are also known as privileged ports.

sudo python -m smtpd -n -c DebuggingServer localhost:25

This one-liner was coined by Evan Culver. Thanks to him!

That's it for today,

but be sure to come back the next time for "Yet Another Ten One-Liners from CommandLineFu Explained!"

Remember my article on The Busy Beaver Problem? Well, someone built a real Turing Machine and decided to run the busy beaver with 4 states on it. Here is the video.

The Turing Machine in this video runs for 107 steps and halts with the total of 13 ones, as expected.

In my article on The Busy Beaver Problem, I also wrote a program that visualizes the tape changes. If you follow the video closely, you'll see that they match the visualization (black square stands for 1, white for 0).


Tape changes for 4 state busy beaver.

See A Turing Machine website for more videos and information about how this machine was actually built. Also see my article on Busy Beaver for a Turing Machine implementation in Python and C++.

This article is part of the article series "CommandLineFu One-Liners Explained."
<- previous article next article ->

CommandLineFu ExplainedHere are the next ten top one-liners from the commandlinefu website. The first post about the topic became massively popular and received over 100,000 views in the first two days.

Before I dive into the next ten one-liners, I want to take the chance and promote the other three article series on one-liners that I have written:

Update: Russian translation now available.

Alright, so here are today's one-liners:

#11. Edit the command you typed in your favorite editor

$ command <CTRL-x CTRL-e>

This one-liner opens the so-far typed command in your favorite text editor for further editing. This is handy if you are typing a lengthier shell command. After you have done editing the command, quit from your editor successfully to execute it. To cancel execution, just erase it. If you quit unsuccessfully, the command you had typed before diving into the editor will be executed.

Actually, I have to educate you, it's not a feature of the shell per se but a feature of the readline library that most shells use for command line processing. This particular binding CTRL-x CTRL-e only works in readline emacs editing mode. The other mode is readline vi editing mode, in which the same can be accomplished by pressing ESC and then v.

The emacs editing mode is the default in all the shells that use the readline library. The usual command to change between the modes is set -o vi to change to vi editing mode and set -o emacs to change back to emacs editing mode.

To change the editor, export the $EDITOR shell variable to your preference. For example, to set the default editor to pico, type export EDITOR=pico.

Another way to edit commands in a text editor is to use fc shell builtin (at least bash has this builtin). The fc command opens the previous edited command in your favorite text editor. It's easy to remember the fc command because it stands for "fix command."

Remember the ^foo^bar^ command from the first top ten one-liners? You can emulate this behavior by typing fc -s foo=bar. It will replace foo with bar in the previous command and execute it.

#12. Empty a file or create a new file

$ > file.txt

This one-liner either wipes the file called file.txt empty or creates a new file called file.txt.

The shell first checks if the file file.txt exists. If it does, the shell opens it and wipes it clean. If it doesn't exist, the shell creates the file and opens it. Next the shell proceeds to redirecting standard output to the opened file descriptor. Since there is nothing on the standard output, the command succeeds, closes the file descriptor, leaving the file empty.

Creating a new empty file is also called touching and can be done by $ touch file.txt command. The touch command can also be used for changing timestamps of the commands. Touch, however, won't wipe the file clean, it will only change the access and modification timestamps to the current time.

#13. Create a tunnel from localhost:2001 to somemachine:80

$ ssh -N -L2001:localhost:80 somemachine

This one-liner creates a tunnel from your computer's port 2001 to somemachine's port 80. Each time you connect to port 2001 on your machine, your connection gets tunneled to somemachine:80.

The -L option can be summarized as -L port:host:hostport. Whenever a connection is made to localhost:port, the connection is forwarded over the secure channel, and a connection is made to host:hostport from the remote machine.

The -N option makes sure you don't run shell as you connect to somemachine.

To make things more concrete, here is another example:

$ ssh -f -N -L2001:www.google.com:80 somemachine

This one-liner creates a tunnel from your computer's port 2001 to www.google.com:80 via somemachine. Each time you connect to localhost:2001, ssh tunnels your request via somemachine, where it tries to open a connection to www.google.com.

Notice the additional -f flag - it makes ssh daemonize (go into background) so it didn't consume a terminal.

#14. Reset terminal

$ reset

This command resets the terminal. You know, when you have accidentally output binary data to the console, it becomes messed up. The reset command usually cleans it up. It does that by sending a bunch of special byte sequences to the terminal. The terminal interprets them as special commands and executes them.

Here is what BusyBox's reset command does:

printf("\033c\033(K\033[J\033[0m\033[?25h");

It sends a bunch of escape codes and a bunch of CSI commands. Here is what they mean:

  • \033c: "ESC c" - sends reset to the terminal.
  • \033(K: "ESC ( K" - reloads the screen output mapping table.
  • \033[J: "ESC [ J" - erases display.
  • \033[0m: "ESC [ 0 m" - resets all display attributes to their defaults.
  • \033[?25h: "ESC [ ? 25 h" - makes cursor visible.

#15. Tweet from the shell

$ curl -u user:pass -d status='Tweeting from the shell' http://twitter.com/statuses/update.xml

This one-liner tweets your message from the terminal. It uses the curl program to HTTP POST your tweet via Twitter's API.

The -u user:pass argument sets the login and password to use for authentication. If you don't wish your password to be saved in the shell history, omit the :pass part and curl will prompt you for the password as it tries to authenticate. Oh, and while we are at shell history, another way to omit password from being saved in the history is to start the command with a space! For example, <space>curl ... won't save the curl command to the shell history.

The -d status='...' instructs curl to use the HTTP POST method for the request and send status=... as POST data.

Finally, http://twitter.com/statuses/update.xml is the API URL to POST the data to.

Talking about Twitter, I'd love if you followed me on Twitter! :)

#16. Execute a command at midnight

$ echo cmd | at midnight

This one-liner sends the shell command cmd to the at-daemon (atd) for execution at midnight.

The at command is light on the execution-time argument, you may write things like 4pm tomorrow to execute it at 4pm tomorrow, 9pm next year to run it on the same date at 9pm the next year, 6pm + 10 days to run it at 6pm after 10 days, or now +1minute to run it after a minute.

Use atq command to list all the jobs that are scheduled for execution and atrm to remove a job from the queue.

Compared to the universally known cron, at is suitable for one-time jobs. For example, you'd use cron to execute a job every day at midnight but you would use at to execute a job only today at midnight.

Also be aware that if the load is greater than some number (for one processor systems the default is 0.8), then atd will not execute the command! That can be fixed by specifying a greater max load to atd via -l argument.

#17. Output your microphone to other computer's speaker

$ dd if=/dev/dsp | ssh username@host dd of=/dev/dsp

The default sound device on Linux is /dev/dsp. It can be both written to and read from. If it's read from then the audio subsystem will read the data from the microphone. If it's written to, it will send audio to your speaker.

This one-liner reads audio from your microphone via the dd if=/dev/dsp command (if stands for input file) and pipes it as standard input to ssh. Ssh, in turn, opens a connection to a computer at host and runs the dd of=/dev/dsp (of stands for output file) on it. Dd of=/dev/dsp receives the standard input that ssh received from dd if=/dev/dsp. The result is that your microphone gets output on host computer's speaker.

Want to scare your colleague? Dump /dev/urandom to his speaker by dd if=/dev/urandom.

#18. Create and mount a temporary RAM partition

# mount -t tmpfs -o size=1024m tmpfs /mnt 

This command creates a temporary RAM filesystem of 1GB (1024m) and mounts it at /mnt. The -t flag to mount specifies the filesystem type and the -o size=1024m passes the size sets the filesystem size.

If it doesn't work, make sure your kernel was compiled to support the tmpfs. If tmpfs was compiled as a module, make sure to load it via modprobe tmpfs. If it still doesn't work, you'll have to recompile your kernel.

To unmount the ram disk, use the umount /mnt command (as root). But remember that mounting at /mnt is not the best practice. Better mount your drive to /mnt/tmpfs or a similar path.

If you wish your filesystem to grow dynamically, use ramfs filesystem type instead of tmpfs. Another note: tmpfs may use swap, while ramfs won't.

#19. Compare a remote file with a local file

$ ssh user@host cat /path/to/remotefile | diff /path/to/localfile -

This one-liner diffs the file /path/to/localfile on local machine with a file /path/to/remotefile on host machine.

It first opens a connection via ssh to host and executes the cat /path/to/remotefile command there. The shell then takes the output and pipes it to diff /path/to/localfile - command. The second argument - to diff tells it to diff the file /path/to/localfile against standard input. That's it.

#20. Find out which programs listen on which TCP ports

# netstat -tlnp

This is an easy one. Netstat is the standard utility for listing information about Linux networking subsystem. In this particular one-liner it's called with -tlnp arguments:

  • -t causes netstat to only list information about TCP sockets.
  • -l causes netstat to only list information about listening sockets.
  • -n causes netstat not to do reverse lookups on the IPs.
  • -p causes netstat to print the PID and name of the program to which the socket belongs (requires root).

To find more detailed info about open sockets on your computer, use the lsof utility. See my article "A Unix Utility You Should Know About: lsof" for more information.

That's it for today.

Tune in the next time for "Another Ten One-Liners from CommandLineFu Explained". There are many more nifty commands to write about. But for now, have fun and see ya!

PS. Follow me on twitter for updates!